Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Nested Labels and Augmentation #211

Merged
merged 154 commits into from
Jan 5, 2025
Merged
Show file tree
Hide file tree
Changes from 144 commits
Commits
Show all changes
154 commits
Select commit Hold shift + click to select a range
05584fc
instance label
kozlov721 Nov 27, 2024
e6dc3be
nested labels
kozlov721 Nov 30, 2024
67e74a4
renamed
kozlov721 Dec 1, 2024
aaedfaf
label utils
kozlov721 Dec 1, 2024
8c0020b
renamed
kozlov721 Dec 1, 2024
94939ec
augmentations
kozlov721 Dec 2, 2024
0062f6f
fixed letterbox
kozlov721 Dec 2, 2024
10668e5
cleanup
kozlov721 Dec 2, 2024
9d2e65a
added option to keep aspect ratio
kozlov721 Dec 2, 2024
2c244c5
support for multiple batched augmentations
kozlov721 Dec 2, 2024
ce3e0fa
updated visualizations
kozlov721 Dec 3, 2024
66c8906
augmenting metadata
kozlov721 Dec 3, 2024
724da38
fixed augment with few labels
kozlov721 Dec 3, 2024
df48e20
fixing tests
kozlov721 Dec 3, 2024
9d410b4
updated parsers
kozlov721 Dec 3, 2024
feb0450
fixed solo parser
kozlov721 Dec 3, 2024
3d31321
classification annotation
kozlov721 Dec 3, 2024
85b9e7d
fixed ingestion
kozlov721 Dec 3, 2024
8d246ec
fixed compose
kozlov721 Dec 3, 2024
6829b45
typo
kozlov721 Dec 3, 2024
46602a9
fixed tests
kozlov721 Dec 3, 2024
a6c11e9
cleanup
kozlov721 Dec 3, 2024
b990375
moved base pipeline to separate file
kozlov721 Dec 3, 2024
20b8a24
registry for augmentations
kozlov721 Dec 3, 2024
e07118e
augmentaion cleanup
kozlov721 Dec 3, 2024
80b1459
updated alias
kozlov721 Dec 3, 2024
ca6fc04
fixed deep nested labels
kozlov721 Dec 3, 2024
81157cf
renamed
kozlov721 Dec 3, 2024
659942f
updated requirements.txt
kozlov721 Dec 3, 2024
316dcc3
optimization
kozlov721 Dec 4, 2024
1db7431
improved visualizations
kozlov721 Dec 4, 2024
2d22f66
fixed bug
kozlov721 Dec 4, 2024
017e155
fixed tasks in parsers
kozlov721 Dec 4, 2024
aa08614
updated type
kozlov721 Dec 5, 2024
91febca
moved augmentation initialization to loader
kozlov721 Dec 5, 2024
0e2308d
moved custom types to luxonis_ml.typing
kozlov721 Dec 5, 2024
c9b1efb
removed alias choices
kozlov721 Dec 5, 2024
bff4347
version checking
kozlov721 Dec 5, 2024
9d8100b
fixed old task name
kozlov721 Dec 5, 2024
c46290c
simplified batch compose
kozlov721 Dec 5, 2024
1a73b98
fixed parser
kozlov721 Dec 5, 2024
07f36ec
simplified batched transforms
kozlov721 Dec 5, 2024
66789d2
added semver to requirements.txt
kozlov721 Dec 5, 2024
5a3e144
added note
kozlov721 Dec 5, 2024
debafed
fixed type for albumentations 1.4.21
kozlov721 Dec 5, 2024
9a90d69
removed unused code
kozlov721 Dec 5, 2024
d004ccc
simplified
kozlov721 Dec 5, 2024
f37dfdf
optimized bbox augmentation
kozlov721 Dec 5, 2024
6558cfd
fixed type
kozlov721 Dec 5, 2024
1d7e83e
instance segmentation support
kozlov721 Dec 5, 2024
a338cee
renamed label_type to task_type
kozlov721 Dec 5, 2024
1374020
augmentations using additional_targets
kozlov721 Dec 6, 2024
a91afb7
set lookup
kozlov721 Dec 6, 2024
aef9f6f
various fixes
kozlov721 Dec 6, 2024
9a8c189
updated cli
kozlov721 Dec 6, 2024
4ea107b
fixed resize transform
kozlov721 Dec 7, 2024
c0b69bc
cleanup
kozlov721 Dec 7, 2024
f30b4ae
more efficient inspect
kozlov721 Dec 7, 2024
625062d
fix empty augmentation
kozlov721 Dec 7, 2024
025437a
fix zero width or height bboxes
kozlov721 Dec 9, 2024
291d0d1
updated requirements
kozlov721 Dec 9, 2024
dfcf4f2
option to set task_name in parser
kozlov721 Dec 9, 2024
6b015f9
fixed wrong path on windows
kozlov721 Dec 9, 2024
3e60581
simplified keypoints postprocessing
kozlov721 Dec 9, 2024
8818c43
fix wrong dir
kozlov721 Dec 9, 2024
06ca4ef
benchmark tool
kozlov721 Dec 9, 2024
5d43b7c
configurable visibility
kozlov721 Dec 9, 2024
5ea2257
migrate script
kozlov721 Dec 9, 2024
f4e107b
style
kozlov721 Dec 9, 2024
f5e983f
default visibility
kozlov721 Dec 9, 2024
340a914
remove type cast
kozlov721 Dec 9, 2024
6baef8c
backwards compatibility
kozlov721 Dec 9, 2024
59ce9d0
checkhealth command
kozlov721 Dec 9, 2024
b4df606
updated readme
kozlov721 Dec 10, 2024
aacda28
removed empty media directory
kozlov721 Dec 10, 2024
e462113
docs for ConfigItem
kozlov721 Dec 10, 2024
5ed32fd
docs and better color handling
kozlov721 Dec 10, 2024
31eef5e
cleaned up letterbox
kozlov721 Dec 10, 2024
43df18f
better aug config docs
kozlov721 Dec 10, 2024
ce8defb
Merge branch 'main' into feature/nested-labels
kozlov721 Dec 10, 2024
c55f56c
merge
kozlov721 Dec 10, 2024
2924a69
fix solo parser test
kozlov721 Dec 10, 2024
063616b
improved tests
kozlov721 Dec 10, 2024
dde9190
updated names
kozlov721 Dec 10, 2024
cad9fa5
fixed windows path issues
kozlov721 Dec 10, 2024
76358a7
pragma ignore in conftest
kozlov721 Dec 11, 2024
bff110b
default in registry metaclass
kozlov721 Dec 11, 2024
0223a99
more options for inspect command
kozlov721 Dec 11, 2024
5841ebf
renamed helper functions in mosaic
kozlov721 Dec 11, 2024
742756c
formatted long string
kozlov721 Dec 11, 2024
87b646e
fixed incorrect scaling of bboxes and keypoints in mixup
kozlov721 Dec 11, 2024
69dc0df
fixed incorrect bbox counting
kozlov721 Dec 11, 2024
ddaa3c7
added aug config documentation
kozlov721 Dec 11, 2024
3e149bd
improved batch augs test
kozlov721 Dec 11, 2024
ce5ab98
invalid loader test
kozlov721 Dec 11, 2024
ee5f6e9
bugfixes, improved tests
kozlov721 Dec 11, 2024
9001214
fix for empty labels
kozlov721 Dec 11, 2024
d6f6ee5
fix augmenting with missing labels
kozlov721 Dec 12, 2024
58d2065
nicer empty array comparison
kozlov721 Dec 12, 2024
79ff349
simplified loader output comparison
kozlov721 Dec 12, 2024
fd15e30
renamed augmentation classes and files
kozlov721 Dec 12, 2024
1c9b7aa
added doctest and coverage settings
kozlov721 Dec 12, 2024
4e0c792
moved all conftests to a single file
kozlov721 Dec 12, 2024
d818f3c
100 % annotations.py coverage
kozlov721 Dec 12, 2024
2746061
100 % task_utils.py coverage
kozlov721 Dec 12, 2024
2e06ad5
removed dict unpacking
kozlov721 Dec 12, 2024
407006f
added default value
kozlov721 Dec 12, 2024
f827606
fixed adding background
kozlov721 Dec 12, 2024
bffaf41
cleaned conftest
kozlov721 Dec 12, 2024
a5dd362
fixed batched augmentaions for instance segmentation
kozlov721 Dec 13, 2024
71e7f1b
changed default `keep_aspect_ratio`
kozlov721 Dec 13, 2024
d9808e4
complex dataset tests
kozlov721 Dec 13, 2024
837e58f
fix array annotation
kozlov721 Dec 13, 2024
ba73c25
fix array annotation error
kozlov721 Dec 13, 2024
ce452b2
added array handling to augmentations
kozlov721 Dec 13, 2024
6dbdd3a
added warnings
kozlov721 Dec 13, 2024
c823ea1
using dataset_name fixture
kozlov721 Dec 13, 2024
40b71f1
added `blend_all` option to `visualize` method
kozlov721 Dec 13, 2024
d5fc093
removed redundant check
kozlov721 Dec 13, 2024
083cca3
added extra tests
kozlov721 Dec 13, 2024
b924239
remove mark_default functionality
kozlov721 Dec 13, 2024
6acb13c
simplified mask thresholding
kozlov721 Dec 13, 2024
5c3f963
transform registry
kozlov721 Dec 13, 2024
92098c0
added roboflow key
kozlov721 Dec 13, 2024
2f0af5b
cleaned up test setups
kozlov721 Dec 13, 2024
8dc7f75
error raised when making splits with no images
kozlov721 Dec 13, 2024
9ce10ac
fixed integration tests
kozlov721 Dec 13, 2024
7b312d7
simplified instance segmentation augmentations
kozlov721 Dec 13, 2024
564bcb8
improved documentation
kozlov721 Dec 13, 2024
af240dc
fix resize error for keypoints in mixup
kozlov721 Dec 14, 2024
f4bd245
fix regular resizing
kozlov721 Dec 14, 2024
a67373f
updated mixup tests
kozlov721 Dec 14, 2024
e967ad8
fix test on windows
kozlov721 Dec 14, 2024
4230a6a
cleanup and docs
kozlov721 Dec 14, 2024
0410fcc
simplified augmentations
kozlov721 Dec 14, 2024
a5505db
fixed docs
kozlov721 Dec 14, 2024
997ff41
updated gitignore
kozlov721 Dec 14, 2024
d4c90ba
formatting
kozlov721 Dec 14, 2024
a8c893b
updated docs
kozlov721 Dec 14, 2024
88e4254
fixed type issues
kozlov721 Dec 14, 2024
6f840f2
using fixed seed in tests
kozlov721 Dec 14, 2024
90197f9
applying custom augmentations
kozlov721 Dec 14, 2024
0e305a8
fix windows tests
kozlov721 Dec 15, 2024
4d187c4
fix widnows test
kozlov721 Dec 15, 2024
ae8e7cd
moved augmentation tests
kozlov721 Dec 18, 2024
058d986
fixed windows test
kozlov721 Dec 18, 2024
2b9f8ab
automatic migration
kozlov721 Dec 27, 2024
bee6793
Merge branch 'main' into feature/nested-labels
kozlov721 Dec 27, 2024
dd240b1
updated requirements
kozlov721 Dec 27, 2024
52a751c
fix requirements syntax error
kozlov721 Dec 27, 2024
463a4f2
updated requirements
kozlov721 Dec 27, 2024
82b3946
updated excluded coverage lines
kozlov721 Dec 27, 2024
0586734
environ tests
kozlov721 Dec 27, 2024
1ac76dd
added pragma
kozlov721 Jan 5, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/workflows/ci.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -83,6 +83,7 @@ jobs:
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
AWS_S3_ENDPOINT_URL: ${{ secrets.AWS_S3_ENDPOINT_URL }}
ROBOFLOW_API_KEY: ${{ secrets.ROBOFLOW_API_KEY }}
LUXONISML_BUCKET: luxonis-test-bucket
run:
pytest --cov --junitxml=junit.xml -o junit_family=legacy
Expand Down
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -114,3 +114,5 @@ data/*
.vscode
tests/data
*.zip
.idea/*
apidocs/
38 changes: 35 additions & 3 deletions luxonis_ml/__main__.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,10 @@
from importlib.metadata import version

import rich
import rich.box
import typer
from rich.markup import escape
from rich.table import Table

from luxonis_ml.utils import setup_logging

Expand All @@ -9,7 +13,7 @@
add_completion=True,
pretty_exceptions_show_locals=False,
)
setup_logging(use_rich=True)
setup_logging(use_rich=True, rich_print=True)

try:
from luxonis_ml.data.__main__ import app as data_app
Expand All @@ -35,7 +39,7 @@

def version_callback(value: bool):
if value:
print(f"LuxonisML: {version(__package__)}")
typer.echo(f"LuxonisML: {version('luxonis_ml')}")
raise typer.Exit()


Expand All @@ -49,9 +53,37 @@ def main(
help="Show version and exit.",
),
):
# Do other global stuff, handle other global options here
return


@app.command()
def checkhealth():
"""Check the health of the Luxonis ML library."""
table = Table(
title="Health Check",
box=rich.box.ROUNDED,
)
table.add_column("Module", header_style="magenta i")
table.add_column("Status", header_style="magenta i")
table.add_column("Error", header_style="magenta i", max_width=50)
for submodule in ["data", "utils", "nn_archive"]:
error_message = ""
try:
__import__(f"luxonis_ml.{submodule}")
status = "✅"
style = "green"
except ImportError as e:
status = "❌"
style = "red"
error_message = escape(str(e.args[0]))
if len(e.args) > 1:
error_message += f" [bold]{escape(e.args[1])}[/bold]"

table.add_row(submodule, status, error_message, style=style)

console = rich.console.Console()
console.print(table)


if __name__ == "__main__":
app()
25 changes: 12 additions & 13 deletions luxonis_ml/data/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -130,18 +130,19 @@ def generator():
w, h = bbox["dimension"]

# get the class name of the bounding box
class_ = bbox["labelName"]
class_name = bbox["labelName"]
yield {
"file": image_path,
"annotation": {
"type": "boundingbox",
"class": class_,

# normalized bounding box
"x": x / W,
"y": y / H,
"w": w / W,
"h": h / H,
"class": class_name,

"boundingbox": {
# normalized bounding box
"x": x / W,
"y": y / H,
"w": w / W,
"h": h / H,
}
},
}
```
Expand Down Expand Up @@ -557,7 +558,6 @@ The following example demonstrates a simple augmentation pipeline:
'mask_pad_val': 0,
'fit_output': False,
'interpolation': 1,
'always_apply': False,
}
},
{
Expand All @@ -577,7 +577,6 @@ The following example demonstrates a simple augmentation pipeline:
'fit_output': False,
'keep_ratio': False,
'rotate_method': 'largest_box',
'always_apply': False,
}
},
]
Expand All @@ -593,13 +592,13 @@ For this example, we will assume the augmentation example above is stored in a J

```python

from luxonis_ml.data import LuxonisDataset, LuxonisLoader, Augmentations
from luxonis_ml.data import LuxonisDataset, LuxonisLoader, AlbumentationsEngine
import json

with open("augmentations.json") as f:
augmentations = json.load(f)

aug = Augmentations(image_size=[256, 320], augmentations=augmentations)
aug = AlbumentationsEngine(image_size=[256, 320], augmentations=augmentations)
dataset = LuxonisDataset("parking_lot")
loader = LuxonisLoader(dataset, view="train", augmentations=aug)

Expand Down
27 changes: 6 additions & 21 deletions luxonis_ml/data/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
from luxonis_ml.guard_extras import guard_missing_extra

with guard_missing_extra("data"):
from .augmentations import Augmentations
from .augmentations import AlbumentationsEngine
from .datasets import (
DATASETS_REGISTRY,
BaseDataset,
Expand All @@ -12,31 +12,19 @@
LuxonisDataset,
LuxonisSource,
)
from .loaders import (
LOADERS_REGISTRY,
BaseLoader,
Labels,
LuxonisLoader,
LuxonisLoaderOutput,
)
from .loaders import LOADERS_REGISTRY, BaseLoader, LuxonisLoader
from .parsers import LuxonisParser
from .utils.enums import (
BucketStorage,
BucketType,
ImageType,
LabelType,
MediaType,
)
from .utils.enums import BucketStorage, BucketType, ImageType, MediaType


def load_dataset_plugins() -> None:
def load_dataset_plugins() -> None: # pragma: no cover
"""Registers any external dataset BaseDataset class plugins."""
for entry_point in pkg_resources.iter_entry_points("dataset_plugins"):
plugin_class = entry_point.load()
DATASETS_REGISTRY.register_module(module=plugin_class)


def load_loader_plugins() -> None:
def load_loader_plugins() -> None: # pragma: no cover
"""Registers any external dataset BaseLoader class plugins."""
for entry_point in pkg_resources.iter_entry_points("loader_plugins"):
plugin_class = entry_point.load()
Expand All @@ -47,7 +35,7 @@ def load_loader_plugins() -> None:
load_loader_plugins()

__all__ = [
"Augmentations",
"AlbumentationsEngine",
"BaseDataset",
"BaseLoader",
"BucketStorage",
Expand All @@ -56,12 +44,9 @@ def load_loader_plugins() -> None:
"DATASETS_REGISTRY",
"LOADERS_REGISTRY",
"ImageType",
"LabelType",
"Labels",
"LuxonisComponent",
"LuxonisDataset",
"LuxonisLoader",
"LuxonisLoaderOutput",
"LuxonisParser",
"LuxonisSource",
"MediaType",
Expand Down
Loading
Loading