relai.vision.data_utils package

Submodules

relai.vision.data_utils.filter module

relai.vision.data_utils.filter.tag_filtering(key: str | list[str], tags: list[list[str]])

A helper to identify the indices of samples with one or more tags of interest. The indices can be used easily with torch.utils.data.Subset for training/tuning/mitigations.

• Parameters:
  • key (Union *[*str , list *[*str ] ]) – a string or a list of strings correspond to one or more tags that one wants to filter for
  • tags (list *[*list *[*str ] ]) – A list containing tags corresponding to individual samples. This may be obtained through relai.inspect.TagDataset or any other taggers of one’s choice.

Example:

from relai.vision.data_utils import TagDataset, get_tagger
from relai.datasets import get_dataset
from relai.vision.data_utils import tag_filtering

tagger = get_tagger('ram', "ram", "swin_l", 'ram_swin_large_14m')
dataset = get_dataset('imagefolder', '/path/to/dataset', split='val') #setup dataset here

tag_dataset = TagDataset(dataset, tagger)
tags = tag_dataset.annotate()

filtered_indices = tag_filtering(key = "outdoor", tags = tags)

relai.vision.data_utils.image_sorter module

class relai.vision.data_utils.image_sorter.CLIPImageSorter(model_name: str, templates: list[str] = ['a photo of a {}'], batch_size: int = 128)

Bases: RELAIImageSorter

Image sorter using OpenAI’s CLIP model. The sorter takes in a batch of images and returns a dictionary of CLIP scores for each textual tag / sorting key provided by the user.

• Parameters:
  • model_name (str) – The name of the CLIP model to use. Will be passed to HuggingFace’s CLIPModel.from_pretrained.
  • templates (list *[*str ]) – A list of string templates to use for the CLIP model. Default: [‘a photo of a {}’]
  • batch_size (int) – The batch size to use for the CLIP model. Default: 128

Example:

from relai.vision.data_utils import CLIPImageSorter
sorter = CLIPImageSorter('ViT-B/32', ['a photo of a {}'])
sorter.set(['dog', 'cat'])

imgs = ... # load images here
imgs = sorter.transform(imgs) # transform them to the appropriate format
scores = sorter(imgs)

print(scores['dog']) # CLIP scores for the tag 'dog'
print(scores['cat']) # CLIP scores for the tag 'cat'

embed_tags(tags: list[str], fabric: Fabric)

Embeds the tags using the CLIP model’s text encoder and defined templates.

• Parameters:
  • tags (list *[*str ]) – A list of tags to be embedded.
  • fabric (L.Fabric) – Fabric object used for setup.
• Returns: A tensor of embeddings for the tags.
• Return type: torch.Tensor

forward(imgs: Tensor, tag_names: list[str] = [])

Forward pass of the model. Should return a dictionary of scores for each tag / sorting key (to be utilized for downstream sorting).

• Parameters:
  • imgs (torch.Tensor) – A batch of images to be scored.
  • tag_names (list *[*str ] , optional) – A list of tags to be scored. Defaults to [].

set(tag_names: list[str], fabric: Fabric = None)

Sets the tags to be used for sorting. The tags are embedded using the CLIP model’s text encoder and defined templates.

• Parameters:
  • tag_names (list *[*str ]) – A list of tags to be used for sorting.
  • fabric (L.Fabric , optional) – Fabric object used for setup. Defaults to None.

property transform

Default transform to be applied to the images before passing it to the model.

class relai.vision.data_utils.image_sorter.RELAIImageSorter(*args, **kwargs)

Bases: Module

Base class for image sorters. The sorter should be able to take in a batch of images and return a dictionary of scores for each tag / sorting key (to be utilized for downstream sorting).

Args: Defined by subclass

Example:

forward(imgs)

Forward pass of the model. Should return a dictionary of scores for each tag / sorting key (to be utilized for downstream sorting).

property transform

Default transform to be applied to the images before passing it to the model.

class relai.vision.data_utils.image_sorter.RobustNeuralImageSorter(model_name: str = None, model_ckpt: str = None, model=None, num_annotation_per_class: int = 5, transform=None, batch_size=128)

Bases: RELAIImageSorter

Image sorter using neural features from a pretrained model. The sorter takes in a batch of images and returns a dictionary of feature activation scores corresponding to the most influential features.

• Parameters:
  • model_name (str) – The name of model architecture to use. Functions only when $model is not specified. Defaults to “robust_resnet50_l2_eps3”.
  • model_ckpt (str) – Path to checkpoint to load. Functions only when $model is not specified. Defaults to None. When $model_name and $model_ckpt are both default, a default checkpoint will be used.
  • model (Any , optional) – a loaded model; This can be any customized object with a forward_features method implemented. model.forward_features(input) takes a batched torch tensor denoting a batch of images as input and outputs one feature vector per image (the shape should be N * D, where N is the number of samples and D is the dimension of the features). Defaults to None.
  • num_annotation_per_class (int) – The number of neural features to be included (for each class).
  • transform (Any) – This should behave as a pytorch style transform, which defines how the inputs should be preprocessed. Mandatory if not using the default checkpoint.

Example:

from relai.vision.data_utils import RobustNeuralImageSorter

sorter = RobustNeuralImageSorter(num_annotation_per_class = 5)

dataloader = ... # get a torch style dataloader
sorter.setup(dataloader = dataloader, classes_list = ["class 1", "class 2", ...])

cls = ... # a class label, the most influential neural features for this class will be used for sorting
sorter.set(cls)

imgs = ... # load images here
imgs = sorter.transform(imgs) # transform them to the appropriate format
scores = sorter(imgs)

forward(inputs)

Forward pass of the model. Should return a dictionary of scores for each tag / sorting key (to be utilized for downstream sorting).

set(cls, fabric=None)

Sets the class for the sorter.

• Parameters:
  • cls (str) – The class label to be used for sorting.
  • fabric (L.Fabric , optional) – Fabric object used for setup. Defaults to None.

setup(dataloader=None, fabric=None, classes_list=None)

property transform

Default transform to be applied to the images before passing it to the model.

relai.vision.data_utils.image_sorter.get_sorter(sorter_type, *sorter_args, **sorter_kwargs)

Returns an instance of an image sorter based on the specified sorter_type.

• Parameters:
  • sorter_type (str) – The type of image sorter to create.
  • *sorter_args – Variable length arguments to be passed to the image sorter constructor.
  • **sorter_kwargs – Keyword arguments to be passed to the image sorter constructor.
• Returns: An instance of the specified image sorter type.
• Return type: ImageSorter
• Raises:NotImplementedError – If the specified sorter_type is not implemented.

relai.vision.data_utils.image_tagger module

class relai.vision.data_utils.image_tagger.RAMImageTagger(model_type: str, vit_type: str, model_name: str, image_size=384, batch_size=16)

Bases: RELAIImageTagger

Image tagger using the RAM model. The tagger takes in a batch of images and returns a list of tags for each image.

• Parameters:
  • model_type (str) – The type of the model to use.
  • vit_type (str) – The type of the vision transformer to use.
  • model_path (str) – The path to the model to use.
  • image_size (int) – The size of the images to be passed to the model. Default: 384
  • batch_size (int) – The batch size to be used when passing images to the model. Default: 16

Example:

forward(imgs: Tensor)

Forward pass of the model. Should return a list of tags for each image.

property transform

Default transform to be applied to the images before passing it to the model.

class relai.vision.data_utils.image_tagger.RELAIImageTagger(*args, **kwargs)

Bases: Module

Base class for image taggers. The sorter should be able to take in a batch of images and return a list of tags for each image.

Args: Defined by subclass

Example:

forward(imgs)

Forward pass of the model. Should return a list of tags for each image.

property transform

Default transform to be applied to the images before passing it to the model.

relai.vision.data_utils.image_tagger.get_tagger(tagger_type, *tagger_args, **tagger_kwargs)

Get image tagger of specified type.

• Parameters:
  • tagger_type (str) – The type of image tagger to retrieve.
  • tagger_args – Additional positional arguments to pass to the image tagger constructor.
  • tagger_kwargs – Additional keyword arguments to pass to the image tagger constructor.
• Returns: An instance of the specified image tagger type.
• Return type:RELAIImageTagger

Example

>>> get_tagger("ram", "ram", "ram_swin_large_14m", image_size=384)
Returns an instance of RAMImageTagger with the provided arguments and keyword arguments.

relai.vision.data_utils.sort_dataset module

class relai.vision.data_utils.sort_dataset.SortDataset(dataset: RELAIDataset, sorter: RELAIImageSorter, fabric: Fabric = None, batch_size: int = 128, num_workers: int = 4)

Bases: RELAIAlgorithm

Sort a dataset of images using a sorter.

• Parameters:
  • dataset (RELAIDataset) – Dataset of images to be sorted.
  • sorter (RELAIImageSorter) – Sorter object used to sort the images.
  • fabric (L.Fabric , optional) – Fabric object used for setup. Defaults to None.
  • batch_size (int , optional) – Batch size for the dataloader. Defaults to 128.
  • num_workers (int , optional) – Number of workers for the dataloader. Defaults to 4.

Example:

from relai.vision.data_utils import SortDataset, get_sorter
from relai.datasets import get_dataset

sorter = get_sorter('clip', 'ViT-B/32', ['a photo of a {}']) # setup sorter here
dataset = get_dataset('imagefolder', '/path/to/dataset/val') # setup dataset here

sort_dataset = SortDataset(dataset, sorter, batch_size=16, num_workers=4)
sort_annotations = sort_dataset.annotate() # returns a dictionary of scores for each tag
print(sort_annotations) # prints a dictionary of tags to scores

annotate(*sorter_setup_args, **sorter_setup_kwargs)

Annotates the dataset with scores for each tag.

• Parameters:
  • *sorter_setup_args – Additional arguments to setup the sorter. Must be compatible with the setup method of
  • example (the sorter. For) –
  • sorter (for a CLIP) –
  • prompts (this could be a list of) –
  • RobustNeuralSorter (for) –
  • this –
  • indices. (could be a list of) –
  • **sorter_setup_kwargs – Additional keyword arguments to setup the sorter. Must be compatible with the setup
  • sorter. (method of the) –

Results: : self.score_annotations: A dictionary mapping each tag to a tensor of scores for each image.

get_images_for_tags(tags: list = None, n: int = 10, from_top: bool = True, save_results: bool = True, save_results_to_path: str = None)

Retrieves images associated with the given tags.

• Parameters:
  • tags (list) – List of tags to retrieve images for. If None, retrieves images for all tags. Default: None.
  • n (int , optional) – Number of images to retrieve per tag. Default: 10.
  • from_top (bool , optional) – Whether to retrieve images from the top or bottom of the sorted list. Default:
  • True. –
  • save_results (bool , optional) – Whether to generate a result file for displaying results. Default: True.
  • save_results_to_path (str , optional) – Path to save the results as a .relai file. Will be automatically
  • Default (generated if not provided.) – None.
• Returns: A dictionary mapping each tag to a list of images associated with that tag.
• Return type: tags_to_imgs

setup_robust()

Setup sorter prerequisites before calling annotate. (Only needed for RobustNeuralImageSorter)

relai.vision.data_utils.tag_dataset module

class relai.vision.data_utils.tag_dataset.TagDataset(dataset: RELAIDataset, tagger: RELAIImageTagger, fabric: Fabric = None, batch_size: int = 16, num_workers: int = 4)

Bases: RELAIAlgorithm

Annotate a dataset with tags using a tagger.

• Parameters:
  • dataset (RELAIDataset) – Dataset of images to be tagged.
  • tagger (RELAIImageTagger) – Tagger to use for annotating the dataset
  • fabric (L.Fabric , optional) – Fabric to use for the algorithm. If not provided, a new fabric will be created.
  • batch_size (int , optional) – Batch size to use for the tagger. Default: 16
  • num_workers (int , optional) – Number of workers to use for the dataloader. Default: 4

Example:

from relai.vision.data_utils import TagDataset, get_tagger
from relai.datasets import get_dataset

tagger = get_tagger('ram', "ram", "swin_l", 'ram_swin_large_14m')

dataset = get_dataset('imagefolder', '/path/to/dataset', split='val') #setup dataset here

tag_dataset = TagDataset(dataset, tagger, batch_size=16, num_workers=4)
tag_annotations = tag_dataset.annotate() # returns a list of tags for each image in the dataset
print(self.tag_to_index) # prints a dictionary of tags to indices

annotate(show_progress=False)

Annotation of the dataset with tags using the tagger. Produces a list of tags for each image in the dataset (tag_annotations), and a dictionary of tags to indices (self.tag_to_index).

• Parameters:show_progress (bool , optional) – Whether to show a progress bar. Defaults to False.
• Returns: A list of tags for each image in the dataset.
• Return type: list[list]

get_images_for_tags(tags=None, n=10, min_subpop_size=10, randomize=False, save_results=True, save_results_to_path=None, ask_feedback=False, save_query_to_path=None)

Get images for tags. If save_results_to_path is provided, saves the images to the specified path. If ask_feedback is True, generates a query file for feedback from the user.

• Parameters:
  • tags (list , optional) – List of tags to get images for. Defaults to all tags in self.tag_to_index.
  • n (int , optional) – Number of images to get for each tag. Defaults to 10.
  • min_subpop_size (int , optional) – Minimum subpopulation size for each tag. Defaults to 10.
  • randomize (bool , optional) – Whether to randomize the images. Defaults to False.
  • save_results (bool , optional) – Whether to generate a result file for displaying results. Defaults to True.
  • save_results_to_path (str , optional) – Path to save the results as a .relai file. Will be automatically
  • provided. (generated if not) –
  • ask_feedback (bool , optional) – Whether to generate a query file for feedback. Defaults to False.
  • save_query_to_path (str , optional) – Path to save the query file as a .relai file. Will be automatically
  • provided. –

get_tag_combinations(max_tag_len=3, min_subpop_size=10, add_to_tag_to_index=False)

Compute tag combinations of length up to max_tag_len, with a minimum subpopulation size of min_subpop_size. If add_to_tag_to_index is True, the tag combinations will be added to the tag_to_index dictionary. Returns a dictionary of tag combinations to indices.

• Parameters:
  • max_tag_len (int , optional) – Maximum length of tag combinations to compute. Defaults to 3.
  • min_subpop_size (int , optional) – Minimum subpopulation size for tag combinations. Defaults to 10.
  • add_to_tag_to_index (bool , optional) – Whether to add the tag combinations to the tag_to_index dictionary.
  • False. (Defaults to) –
• Returns: An OrderedDict of tag combinations to indices, in the order of increasing length of tag combinations (ties are broken arbitrarily).

process_feedback(feedback_file)

Process feedback from the user. If the user responds “No” to a tag, the tag will be removed from the tag_to_index dictionary.

• Parameters:feedback_file (str) – Path to the feedback file.

Module contents