Appearance
Inspections
Class Analysis
relai.vision.detection.inspect.class_analysis.class_analysis(*args, **kwargs)
This method inspects an object detection model with respect to different classes/categories. mAP per class/categroy will be computed and returned.
- Parameters:
model_name (str) – Name of model architecture to use.
model_ckpt (str) – Path to checkpoint to load. Defaults to None.
model (any) –
a loaded model; can be obtained through relai.models.get_detection_model or other ways of one’s choice. This should behave as a mmdet style inferencer (e.g. mmdet.apis.DetInferencer), meaning that:
- model(input) takes a list of numpy images (height * width * channel) as input
- model(input) outputs predictions in the following dictionary format: {
’predictions’ : [ {
’labels’: […], # int list of length (N, ), N is the number of predicted bounding boxes ‘scores’: […], # float list of length (N, ) ‘bboxes’: […], # 2d list of shape (N, 4), format: [min_x, min_y, max_x, max_y]
}, … (Each instance corresponds to an input image) ]
}
dataset (RELAIDataset) – Dataset to use.
dataset_format (str) – Dataset format. Functions only when $dataset is not specified.
dataset_path (str) – Dataset path. Functions only when $dataset is not specified.
dataset_kwargs (dict) – A dict containing any additional arguments for the dataset. Functions only when $dataset is not specified.
class_names (list) – A list containing names of different categories. This is only for visualization purposes in relai frontend and can be left as None. If None, class indices will be used as class names.
save_to_path (str) – The path to where the result file will be stored. If None, the result will still be returned by no result file will be generated. Defaults to None.
num_workers (int) – number of workers to use for dataloaders
batch_size (int) – batch size to use for dataloaders
num_batches (int) – to specify how many batches of samples will be used for the test; if num_batches=0 or batch_size * num_batches >= dataset size, the entire dataset will be used.
iou_thrs (list *[*float ]) – A set of IoU thresholds for evaluation. mAP (mean average precision) corresponding to each IoU threshold will be computed. The average mAP over all thresholds will also be returned. For evaluating detection models only.
nolightning (bool) – whether to disable the use of pytorch lightning. Must be enabled if evaluating detection models from mmdet. Defaults to True.
Example:
python
from relai.vision.detection.inspect import class_analysis
result = class_analysis(
model_name = model_name,
model_ckpt = checkpoint_path,
dataset_format = 'nuimages',
dataset_path = dataset_path,
dataset_kwargs = {'version':'v1.0-val'},
class_names = ['car', 'trunk', 'trailer', 'bus', 'construction vehicle',
'bicycle', 'motorcycle', 'pedestrian', 'traffic cone', 'barrier']
save_to_path = './output/result'
)Scale Analysis
relai.vision.detection.inspect.scale_analysis.scale_analysis(*args, **kwargs)
This method inspects an object detection model with respect to objects of different scales. One can use obj_scale_thrs (list[int]) to define a list of threshold (in number of pixels) for the size of objects in object detection and objects will then be divided into len(obj_scale_thrs) + 1 groups based on the size of their bounding boxes and mAP per group will be computed.
- Parameters:
model_name (str) – Name of model architecture to use.
model_ckpt (str) – Path to checkpoint to load. Defaults to None.
model (any) –
a loaded model; can be obtained through relai.models.get_detection_model or other ways of one’s choice. This should behave as a mmdet style inferencer (e.g. mmdet.apis.DetInferencer), meaning that:
- model(input) takes a list of numpy images (height * width * channel) as input
- model(input) outputs predictions in the following dictionary format: {
’predictions’ : [ {
’labels’: […], # int list of length (N, ), N is the number of predicted bounding boxes ‘scores’: […], # float list of length (N, ) ‘bboxes’: […], # 2d list of shape (N, 4), format: [min_x, min_y, max_x, max_y]
}, … (Each instance corresponds to an input image) ]
}
dataset (RELAIDataset) – Dataset to use.
dataset_format (str) – Dataset format. Functions only when $dataset is not specified.
dataset_path (str) – Dataset path. Functions only when $dataset is not specified.
dataset_kwargs (dict) – A dict containing any additional arguments for the dataset. Functions only when $dataset is not specified.
save_to_path (str) – The path to where the result file will be stored. If None, the result will still be returned by no result file will be generated. Defaults to None.
num_workers (int) – number of workers to use for dataloaders
batch_size (int) – batch size to use for dataloaders
num_batches (int) – to specify how many batches of samples will be used for the test; if num_batches=0 or batch_size * num_batches >= dataset size, the entire dataset will be used.
iou_thrs (list *[*float ]) – A set of IoU thresholds for evaluation. mAP (mean average precision) corresponding to each IoU threshold will be computed. The average mAP over all thresholds will also be returned. For evaluating detection models only.
nolightning (bool) – whether to disable the use of pytorch lightning. Must be enabled if evaluating detection models from mmdet. Defaults to True.
obj_scale_thrs (list *[*int ]) – A list of threshold (in number of pixels) for the size of objects in object detection. Must be in an increasing order. Objects will be divided into len(obj_scale_thrs) + 1 groups based on the size of their bounding boxes and mAP per group will be computed. For evaluating detection models only.
Example:
python
from relai.vision.detection.inspect import scale_analysis
result = scale_analysis(
model_name = model_name,
model_ckpt = checkpoint_path,
dataset_format = 'nuimages',
dataset_path = dataset_path,
dataset_kwargs = {'version':'v1.0-val'},
obj_scale_thrs = [16**2, 32**2, 64**2]
save_to_path = './output/result'
)Tag Analysis
relai.vision.detection.inspect.tag_analysis.tag_analysis(*args, **kwargs)
This method identifies potential biases/weaknesses of a detection model with tagged data samples. It computes statistics of the model over data with different tags and identify cases where the performance changes the most (either increase or decrease, depending on whether $reverse is set to False or True). If $reverse is False, the $tags_topk tags with the lowest mAP and their relevant statisitcs will be saved and viewable in relai frontend; If $reverse is True, the $tags_topk tags with the highest mAP and their relevant statisitcs will be saved and viewable in relai frontend.
Tags that appear in less than $tags_min images will be ignored.
- Parameters:
model_name (str) – Name of model architecture to use.
model_ckpt (str) – Path to checkpoint to load. Defaults to None.
model (any) –
a loaded model; can be obtained through relai.models.get_detection_model or other ways of one’s choice. This should behave as a mmdet style inferencer (e.g. mmdet.apis.DetInferencer), meaning that:
- model(input) takes a list of numpy images (height * width * channel) as input
- model(input) outputs predictions in the following dictionary format: {
’predictions’ : [ {
’labels’: […], # int list of length (N, ), N is the number of predicted bounding boxes ‘scores’: […], # float list of length (N, ) ‘bboxes’: […], # 2d list of shape (N, 4), format: [min_x, min_y, max_x, max_y]
}, … (Each instance corresponds to an input image) ]
}
dataset (RELAIDataset) – Dataset to use.
dataset_format (str) – Dataset format. Functions only when $dataset is not specified.
dataset_path (str) – Dataset path. Functions only when $dataset is not specified.
dataset_kwargs (dict) – A dict containing any additional arguments for the dataset. Functions only when $dataset is not specified.
class_names (list) – A list containing names of different categories. This is only for visualization purposes in relai frontend and can be left as None. If None, class indices will be used as class names.
save_to_path (str) – The path to where the result file will be stored. If None, the result will still be returned by no result file will be generated. Defaults to None.
num_workers (int) – number of workers to use for dataloaders
batch_size (int) – batch size to use for dataloaders
num_batches (int) – to specify how many batches of samples will be used for the test; if num_batches=0 or batch_size * num_batches >= dataset size, the entire dataset will be used.
iou_thrs (list *[*float ]) – A set of IoU thresholds for evaluation. mAP (mean average precision) corresponding to each IoU threshold will be computed. The average mAP over all thresholds will also be returned. For evaluating detection models only.
nolightning (bool) – whether to disable the use of pytorch lightning. Must be enabled if evaluating detection models from mmdet. Defaults to True.
tags (list *[*list *[*str ] ]) – A list containing tags corresponding to individual samples. The i-th element of the list should be the list of tags associated with the i-th sample in the dataset. This can be obtained through relai.vision.data_utils.TagDataset or any other taggers of one’s choice. Fine-grained statistics with respect to different tags will be computed. Currently for evaluating detection models only. Defaults to None.
tags_min (int) – A threshold to filter out tags with low frequency. Statistics will be computed for a tag only if it appears in more than $tag_min samples.
tags_topk (int) – If $reverse is False, the $tags_topk tags with the lowest mAP and their relevant statisitcs will be saved and viewable in relai frontend; If $reverse is True, the $tags_topk tags with the highest mAP and their relevant statisitcs will be saved and viewable in relai frontend.
reverse (bool) – If False, rank the tags in an increasing order with respect to mAP; If True, rank the tags in a decreasing order with respect to mAP.
rows_per_page (int) – Determine how many tags will be contained in each page when viewed in relai frontend.
Example:
python
from relai.vision.data_utils import TagDataset, get_tagger
from relai.datasets import get_dataset
from relai.vision.detection.inspect import tag_analysis
tagger = get_tagger('ram', "ram", "swin_l", 'ram_swin_large_14m')
dataset = get_dataset('nuimages', '/path/to/dataset', version = 'v1.0-val') #setup dataset here
tag_dataset = TagDataset(
dataset = dataset,
tagger = tagger
)
tags = tag_dataset.annotate()
result = tag_analysis(
model_name = model_name,
model_ckpt = checkpoint_path,
dataset = dataset,
tags = tags,
tags_min = 100,
tags_topk = 50,
save_to_path = './output/result'
)