Nested Labels and Augmentation

[kozlov721]

Labels and Augmentations refactor

• Nested Labels
• Augmentations
• Miscellaneous
• Breaking Changes
  • Migration

Nested Labels

• Added native support for arbitrarily nested labels
• Added new "metadata" field that can hold additional information about the label
  • Can be int, float, or string
• "task" moved up from the "annotation" section
  • An image without labels still belongs to some task
• Groundwork for future support of multiple images per annotation
• Added support for instance segmentation tasks
  • New "instance_segmentation" task type
• Mostly backwards compatible with LDF v1.0.0, no need for manual migration
  • However, for full compatibility the dataset has to be re-created using the latest luxonis-ml

Example

yield {
    "file": "img.png",
    "task": "vehicles",
    "annotation": {
        "class": "car",
        "boundingbox": {
            "x": x, "y": y, "w": w, "h": h
        },
        "keypoints": {
            "keypoints": [...]
        },
        "instance_segmentation": {
            "mask": ...
        },
        "array": {"path": ...},
        "metadata": {
            "id": 12,
        },
        "sub_detections": {
            "license_plate": {
                "boundingbox": {...},
                "metadata": {
                    "text": "123ABC"
                },
            }
        }
    }
}

The output of the loader for the above annotations would be:

{
    "vehicles/boundingbox": ...,
    "vehicles/keypoints": ...,
    "vehicles/instance_segmentation": ...,
    "vehicles/classification": ...,
    "vehicles/array": ...,
    "vehicles/metadata/id": ...,
    "vehicles/license_plate/boundingbox": ...,
    "vehicles/license_plate/metadata/text": ...,
}

Augmentations

• Upgraded albumentations from 1.3.x to 1.4.x
• Added support for multiple batched augmentations
  • e.g. MixUp together with Mosaic4
• Improved MixUp augmentation
  • Uses letterbox resizing by default
    • Can be configured by keep_aspect_ration parameter
  • Improved augmenting semantic segmentations
    • Masks are blended together, in case of a conflict, the mask associated with higher $\alpha$ takes precedence
• Native support for labels with multiple annotations of the same type
  • Instead of it being a hack on the LuxonisLoader part
• Cleaned up and removed unused code
• Overall performance increase for all types of augmentations (see the table below)
• Augmentations are now initialized by BaseLoader instead of by the user
  • This allows for augmenting all annotations in one go using albumentation's additional_targets logic
  • Simplified usage for users
    -Augmentations renamed to AlbumentationsEngine and inherit from AugmentationEngine
    • Groundwork for future support for more augmentation engines
    • Users can implement their own engine which registers into AUGMENTATION_ENGINES registry

Example
old.py

from luxonis_ml.data import LuxonisDataset, LuxonisLoader, Augmentations


dataset = LuxonisDataset(...)
augmentations = Augmentations((height, width), config, ...)
loader = LuxonisLoader(dataset, augmentations, ...)

new.py

from luxonis_ml.data import LuxonisDataset, LuxonisLoader

dataset = LuxonisDataset(...)
loader = LuxonisLoader(
    dataset, 
    height=...,
    width=...,
    # possible future support for other engines, e.g. Nvidia's DALI
    augmentation_engine="albumentations",
    augmentation_config=...,
    ...,
)

Benchmark

Benchmarking was done on Microsoft COCO 2017 Dataset. The size of the dataset is 92,325 images.
The new augmentations are faster in all categories.

Action	Old	New
Dataset Creation	118s	112s
Loader Init	184s	138s
Iterating No Augmentation	91s	89s
Iterating Standard Augmentations	620s	582s
Iterating MixUp Augmentation	401s	307s
Iterating Mosaic4 Augmentation	590s	505s

Miscellaneous

• Improved tests
  • 96 % coverage
• Added new directory tools to the repository that will contain various scripts for easier development
  • Currently contains benchmark.py
• Added new abstract property version to BaseDataset
• Added new luxonis_ml checkhealth command that prints a table with the state of individual sub-packages including possible import errors
• Custom types moved to luxonis_ml.typing
• Improved visualization functions for luxonis_ml data inspect
  • Nested labels rendered on top of each other
  • Occluded keypoints marked with a cross
  • Nested bounding boxes rendered using dashed lines
  • Added --keep_aspect_ratio (or -k) flag to use letterbox resizing with augmentations
  • Added --blend-all (or -b) flag to draw all labels on top of the same image (disregarding different tasks)
    

[klemen1999]

LGTM, thanks!
Let's fix the tests and then we can merge IMO