Inspections

Robustness

class relai.vision.segmentation.inspect.robustness.RobustnessTest(model: Any, segmentation_dataset: RELAISegmentationDataset, framework: SegmentationFramework, test_types: List[str], save_to_path: str | Path, classes: List[str], num_severity: int = 6, batch_size: int = 10)

Bases: RELAIAlgorithm

The RobustnessTest class is designed to perform robustness analysis of image segmentation models. Input is a RELAISegmentationDataset. The input images are subjected to different corruptions with varying severity level. Model inference is performed on these corrupted input images and metrics are computed.

• Parameters:
  • model (Any) – An image segmentation model.
  • segmentation_dataset (RELAISegmentationDataset) – Segmentation dataset to be inspected.
  • framework (SegmentationFramework) – Supported segmentation framework. Current Supported frameworks: [mmsegmentation]
  • test_types (List *[*str ]) – List of corruptions to apply for the robustness analysis. Supported: Gaussian Noise, Shot Noise, Impulse Noise, Defocus Blur, Motion Blur, Zoom Blur Brightness, Contrast, Rain, Snow, SunFlare, Shadow, Fog
  • save_to_path (str) – absolute path to folder where to save the robustness analysis results.
  • classes (List *[*str ]) – List of class names corresponding to label ids in the model.
  • num_severity (int , default=6) – Number of severity levels to be considered. For example if num_severity=6, then severity in range [0, 5] are considered.
  • batch_size (int , default=10) – Batch size for the model inference.

Example:

# Example 1: Using mmsegmentation and RELAISegmentationDatasetCSV

from relai.vision.segmentation.inspect.robustness import (
    RobustnessTest,
    get_segmentation_model,
    SegmentationFramework
)
from relai.datasets import RELAISegmentationDatasetCSV

config_path = "/path/to/mmsegmentation/config_file"
ckpt = "/path/to/mmsegmentation/ckpy"
segmentation_dataset = RELAISegmentationDatasetCSV("/path/to/inspection/data")
classes = ["road", "sidewalk", ...]
model = get_segmentation_model(
    SegmentationFramework.MMSEGMENTATION,
    config=config_path,
    checkpoint=ckpt,
    device="cuda"
)

segmentation_model_inspector = ImageSegmentationRobustnessTest(
    model=model,
    segmentation_dataset=segmentation_dataset,
    framework=SegmentationFramework.MMSEGMENTATION,
    test_types=["Gaussian Noise", "Defocus Blur", "Brightness"],
    output_path="/absolute/path/to/output/dump/folder",
    classes=classes,
    batch_size=64
)
segmentation_model_inspector.run_inspection()

run_inspection()

class relai.vision.segmentation.inspect.robustness.SegmentationFramework(value, names=not given, *values, module=None, qualname=None, type=None, start=1, boundary=None)

Bases: StrEnum

Enum for supported segmentation framework

MMSEGMENTATION = 'mmsegmentation'

relai.vision.segmentation.inspect.robustness.get_segmentation_model(framework: SegmentationFramework, *args, **kwargs)