Appearance
relai.vision package
Subpackages
- Image Classification
- relai.vision.data_utils package
- Object Detection
- Image Segmentation
Submodules
relai.vision.robustness_test module
class relai.vision.robustness_test.CorruptionWrapper(test_type: str = 'Gaussian Noise', severity: int = 0, mode: str = 'PIL')
Bases: object
This provides a wide range of augmentations/perturbations/corruptions applicable to images, including “Gaussian Noise”, “Shot Noise”, “Impulse Noise”, “Defocus Blur”, “Motion Blur”, “Zoom Blur”, “Brightness”, “Contrast”, “Rain”, “Snow”, “SunFlare”, “Shadow”, “Fog”.
- Parameters:
- test_type (str) – to specify the type of perturbation/corruption to be used, must be a string in [“Gaussian Noise”, “Shot Noise”, “Impulse Noise”, “Defocus Blur”, “Motion Blur”, “Zoom Blur”, “Brightness”, “Contrast”, “Rain”, “Snow”, “SunFlare”, “Shadow”, “Fog”]
- severity (int) – For each test_type (i.e. each type of perturbation/corruption), there are 5 pre-defined severities indexed 1 to 5. The larger the severity, the stronger the augmentations. Set to 0 to evaluate without additional augmentation. Must be an integer between 0 to 6.
- mode (str) – The output format. Set to “PIL” when outputting PIL images and “numpy” when outputting numpy ndarrays. Defaults to “PIL”.
Example:
python
from relai.vision.data_utils import Robustness_Transform
from PIL import Image
img = Image.open('image.png')
T = Robustness_Transform("Gaussian Noise", severity = 3)
transformed_img = T(img)class relai.vision.robustness_test.RGB2BGR
Bases: object
class relai.vision.robustness_test.RobustnessTest(model: Any = None, model_name: str = None, model_ckpt: str | Path = None, dataset: RELAIDataset = None, dataset_format: str = None, dataset_path: str = None, dataset_kwargs: dict = {}, num_workers: int = 4, fabric: Fabric = None, test_type='Gaussian Noise', batch_size: int = 64, num_batches: int = 0, num_severity: int = 6, task: str = 'classification', iou_thrs: list[float] = [0.5, 0.75], nolightning: bool = False, num_sample_imgs: int = 0, obj_scale_thrs: list[int] = None, per_class: bool = False, tags: list[list[str]] = None, tags_min: int = 1)
Bases: RELAIAlgorithm
run()
summarize()
Method to summarize the results of the algorithm.
relai.vision.robustness_test.brightness(x, severity=1)
relai.vision.robustness_test.clipped_zoom(img, zoom_factor)
relai.vision.robustness_test.contrast(x, severity=1)
relai.vision.robustness_test.defocus_blur(x, severity=1)
relai.vision.robustness_test.disk(radius, alias_blur=0.1, dtype=class 'numpy.float32')
relai.vision.robustness_test.fog(x, severity=1)
relai.vision.robustness_test.gaussian_noise(x, severity=1)
relai.vision.robustness_test.get_average_precision(predictions, total_gts)
relai.vision.robustness_test.get_test_type_description(test_type)
relai.vision.robustness_test.impulse_noise(x, severity=1)
relai.vision.robustness_test.motion_blur(x, severity=1)
relai.vision.robustness_test.rain(x, severity=1)
relai.vision.robustness_test.robustness_analysis(*args, **kwargs)
This inspects the robustness of a model against multiple distribution shifts, including “Gaussian Noise”, “Shot Noise”, “Impulse Noise”, “Defocus Blur”, “Motion Blur”, “Zoom Blur”, “Brightness”, “Contrast”, “Rain”, “Snow”, “SunFlare”, “Shadow”, “Fog”. One can specify any subset of these with test_types and statistics with respect to individual distribution shifts will be returned.
- Parameters:
model (any) –
a loaded model; can be obtained through relai.models.get_model, relai.models.get_detection_model or other ways of one’s choice. For classification models, this should behave as a pytorch style model, meaning that:
- model(input) takes a batched torch tensor denoting a batch of images as input and outputs scores/logits for each class and each sample (the shape should be N * C, where N is the number of samples and C is the number of classes).
- model.transform should behave as a pytorch style transform, which defines how the inputs should be preprocessed.
For detection models, this should behave as a mmdet style inferencer (e.g. mmdet.apis.DetInferencer), meaning that:
- model(input) takes a list of numpy images (height * width * channel) as input
- model(input) outputs predictions in the following dictionary format: {
’predictions’ : [
Each instance corresponds to an input image
{
’labels’: […], # int list of length (N, ), N is the number of predicted bounding boxes ‘scores’: […], # float list of length (N, ) ‘bboxes’: […], # 2d list of shape (N, 4), format: [min_x, min_y, max_x, max_y]
]
}
model_name (str) – When $model is not specified: This specifies the name of model architecture to use; When $model is specified, this will only be used to specify the name used to denote the model when the results are displayed by relai frontend.
model_ckpt (str) – Path to checkpoint to load. Defaults to None.
dataset (RELAIDataset) – Dataset to use.
dataset_format (str) – Dataset format. Functions only when $dataset is not specified.
dataset_path (str) – Dataset path. Functions only when $dataset is not specified.
dataset_kwargs (dict) – A dict containing any additional arguments for the dataset. Functions only when $dataset is not specified.
test_types (list *[*str ]) – A list of strings to specify the set of perturbation/corruption types to be used. Each string must be in [“Gaussian Noise”, “Shot Noise”, “Impulse Noise”, “Defocus Blur”, “Motion Blur”, “Zoom Blur”, “Brightness”, “Contrast”, “Rain”, “Snow”, “SunFlare”, “Shadow”, “Fog”].
save_to_path (str) – The path to where the result file will be stored. If None, the result will still be returned by no result file will be generated. Defaults to None.
no_print (bool) – If True, helper messages explaining the progress will not be printed. Defaults to False.
num_workers (int) – number of workers to use for dataloaders
fabric (L.Fabric , optional) – The fabric to use for the algorithm. If not provided, a new fabric will be created.
num_severity (int) – The maximum strength (i.e. severity) of perturbation/corruption to be used. For each test_type (i.e. each type of perturbation/corruption), there are 5 pre-defined severities indexed 1 to 5. The larger the severity, the stronger the augmentations. The evaluation will incorporate augmented samples corresponding to severity 0 (i.e. no perturbation/corruption) through severity $num_severity. Set to 0 to evaluate without additional augmentation. Must be an integer between 0 to 6.
batch_size (int) – batch size to use for dataloaders
num_batches (int) – to specify how many batches of samples will be used for the test; if num_batches=0 or batch_size * num_batches >= dataset size, the entire dataset will be used.
task (str) – a string specifying the type of tasks that the model is for; Should be set to “classification” when evaluating classification models and “detection” when evaluating detection models. This will be set automatically based on whether relai.vision.classification.inspect.robustness_analysis or relai.vision.detection.inspect.robustness_analysis is called.
iou_thrs (list *[*float ]) – A set of IoU thresholds for evaluation. mAP (mean average precision) corresponding to each IoU threshold will be computed. The average mAP over all thresholds will also be returned. For evaluating detection models only.
nolightning (bool) – whether to disable the use of pytorch lightning. Must be enabled if evaluating detection models from mmdet. Defaults to False.
num_sample_imgs (int) – The number of examples of augmented images (per severity) to be included in the result, which can be visualized in relai frontend. Defaults to 0. Currently for evaluating classification models only.
Example:
python
from relai.vision.classification.inspect import robustness_analysis
result = robustness_analysis(
model_name = 'resnet50',
model_ckpt = checkpoint_path,
dataset_format = 'imagefolder',
dataset_path = dataset_path,
test_types = ["Gaussian Noise", "Motion Blur", "Rain"],
num_sample_imgs = 10,
save_to_path = './output/result'
)python
from relai.vision.detection.inspect import robustness_analysis
from relai.models import get_detection_model
model = get_detection_model(model_name = 'mmdet:yolox_tiny_8x8_300e_coco')
result = robustness_analysis(
model = model,
model_name = 'name of your model',
dataset_format = 'coco_format',
dataset_path = dataset_path,
test_types = ["Rain", "Snow"],
num_sample_imgs = 10,
save_to_path = './output/result'
)relai.vision.robustness_test.shadow(x, severity=1)
relai.vision.robustness_test.shot_noise(x, severity=1)
relai.vision.robustness_test.snow(x, severity=1)
relai.vision.robustness_test.sun_flare(x, severity=1)
relai.vision.robustness_test.versatile_analysis(*args, **kwargs)
This supports functionalities to evaluate classification and detection models, with the options to:
evaluate on augmented/perturbed/corrupted data; provide per-class statistics; provide statistics with respect to objects of different scales (for object detection); provide fine-grained statistics with respect to any given tags.
This supports multiple inspection methods and it is unnecessary to use this directly in most cases.
- Parameters:
model_name (str) – Name of model architecture to use.
model_ckpt (str) – Path to checkpoint to load. Defaults to None.
model (any) –
a loaded model; can be obtained through relai.models.get_model, relai.models.get_detection_model or other ways of one’s choice. For classification models, this should behave as a pytorch style model, meaning that:
- model(input) takes a batched torch tensor denoting a batch of images as input and outputs scores/logits for each class and each sample (the shape should be N * C, where N is the number of samples and C is the number of classes).
- model.transform should behave as a pytorch style transform, which defines how the inputs should be preprocessed.
For detection models, this should behave as a mmdet style inferencer (e.g. mmdet.apis.DetInferencer), meaning that:
- model(input) takes a list of numpy images (height * width * channel) as input
- model(input) outputs predictions in the following dictionary format: {
’predictions’ : [ {
’labels’: […], # int list of length (N, ), N is the number of predicted bounding boxes ‘scores’: […], # float list of length (N, ) ‘bboxes’: […], # 2d list of shape (N, 4), format: [min_x, min_y, max_x, max_y]
}, … (Each instance corresponds to an input image) ]
}
dataset (RELAIDataset) – Dataset to use.
dataset_format (str) – Dataset format. Functions only when $dataset is not specified.
dataset_path (str) – Dataset path. Functions only when $dataset is not specified.
dataset_kwargs (dict) – A dict containing any additional arguments for the dataset. Functions only when $dataset is not specified.
test_type (str) – to specify the type of perturbation/corruption to be used, must be a string in [“Gaussian Noise”, “Shot Noise”, “Impulse Noise”, “Defocus Blur”, “Motion Blur”, “Zoom Blur”, “Brightness”, “Contrast”, “Rain”, “Snow”, “SunFlare”, “Shadow”, “Fog”]
save_to_path (str) – The path to where the result file will be stored. If None, the result will still be returned by no result file will be generated. Defaults to None.
num_severity (int) – The maximum strength (i.e. severity) of perturbation/corruption to be used. For each test_type (i.e. each type of perturbation/corruption), there are 5 pre-defined severities indexed 1 to 5. The larger the severity, the stronger the augmentations. The evaluation will incorporate augmented samples corresponding to severity 0 (i.e. no perturbation/corruption) through severity $num_severity. Set to 0 to evaluate without additional augmentation. Must be an integer between 0 to 6.
num_workers (int) – number of workers to use for dataloaders
fabric (L.Fabric , optional) – The fabric to use for the algorithm. If not provided, a new fabric will be created.
batch_size (int) – batch size to use for dataloaders
num_batches (int) – to specify how many batches of samples will be used for the test; if num_batches=0 or batch_size * num_batches >= dataset size, the entire dataset will be used.
task (str) – a string specifying the type of tasks that the model is for; Should be set to “classification” when evaluating classification models and “detection” when evaluating detection models.
iou_thrs (list *[*float ]) – A set of IoU thresholds for evaluation. mAP (mean average precision) corresponding to each IoU threshold will be computed. The average mAP over all thresholds will also be returned. For evaluating detection models only.
nolightning (bool) – whether to disable the use of pytorch lightning. Must be enabled if evaluating detection models from mmdet. Defaults to False.
num_sample_imgs (int) – The number of examples of augmented images (per severity) to be included in the result, which can be visualized in relai frontend. Defaults to 0.
resize_sample_imgs (tuple *[*int , int ]) – Resize the resolution for examples of augmented images. If None, no additional resizing will be applied. Defaults to None.
obj_scale_thrs (list *[*int ]) – A list of threshold (in number of pixels) for the size of objects in object detection. Must be in an increasing order. Objects will be divided into len(obj_scale_thrs) + 1 groups based on the size of their bounding boxes and mAP per group will be computed. For evaluating detection models only.
per_class (bool) – If True, statistics per class will be computed. Defaults to False.
tags (list *[*list *[*str ] ]) – A list containing tags corresponding to individual samples. The i-th element of the list should be the list of tags associated with the i-th sample in the dataset. This can be obtained through relai.vision.data_utils.TagDataset or any other taggers of one’s choice. Fine-grained statistics with respect to different tags will be computed. Currently for evaluating detection models only. Defaults to None.
tags_min (int) – A threshold to filter out tags with low frequency. Statistics will be computed for a tag only if it appears in more than $tag_min samples.