Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add faster-coco-eval metric #11776

Open
wants to merge 3 commits into
base: main
Choose a base branch
from

Conversation

MiXaiLL76
Copy link

Motivation

Faster COCO Eval

https://github.com/MiXaiLL76/faster_coco_eval

This package wraps a facebook C++ implementation of COCO-eval operations found in the pycocotools package. This implementation greatly speeds up the evaluation time for coco's AP metrics, especially when dealing with a high number of instances in an image.

Modification

I added a variable (use_faster_coco_eval) to the CocoMetric class that allows you to select an additional type of validation, namely COCOeval_faster

Use cases (Optional)

For our use case with a test dataset of 5000 images from the coco val dataset.
Testing was carried out using the mmdetection framework and the eval_metric.py script. The indicators are presented below.

Visualization of testing colab_example.ipynb available in directory examples/comparison
colab_example.ipynb in google collab
Tested with rtmdet model bbox + segm

Summary for 5000 imgs

Type faster-coco-eval pycocotools Profit
bbox 5.812 22.72 3.909
segm 7.413 24.434 3.296

Checklist

  1. Pre-commit or other linting tools are used to fix the potential lint issues.
mixaill76@HOME-SERVER:~/mmdetection$ pre-commit run --all-files
flake8...................................................................Passed
isort....................................................................Passed
yapf.....................................................................Passed
trim trailing whitespace.................................................Passed
check yaml...............................................................Passed
fix end of files.........................................................Passed
fix requirements.txt.....................................................Passed
fix double quoted strings................................................Passed
check for merge conflicts................................................Passed
fix python encoding pragma...............................................Passed
mixed line ending........................................................Passed
codespell................................................................Passed
mdformat.................................................................Passed
docformatter.............................................................Passed
check algorithm readme...................................................Passed
check copyright..........................................................Passed
  1. The modification is covered by complete unit tests. If not, please add more unit test to ensure the correctness.
mixaill76@HOME-SERVER:~/mmdetection$ python3 -m coverage run --branch --source mmdet -m pytest tests/test_evaluation/test_metrics/test_coco_metric.py 
========================= test session starts ==========================
platform linux -- Python 3.10.12, pytest-7.4.3, pluggy-1.3.0
rootdir: /home/mixaill76/mmdetection
configfile: pytest.ini
plugins: xdoctest-1.1.4, anyio-3.7.1
collected 17 items                                                                                                                                                                                                                                                             

tests/test_evaluation/test_metrics/test_coco_metric.py ................. [100%]

tests/test_evaluation/test_metrics/test_coco_metric.py::TestCocoMetric::test_evaluate_0
  /usr/lib/python3.10/tempfile.py:999: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmp606joo9e'>
    _warnings.warn(warn_message, ResourceWarning)

tests/test_evaluation/test_metrics/test_coco_metric.py::TestCocoMetric::test_evaluate_1
  /usr/lib/python3.10/tempfile.py:999: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpcnbtqpk6'>
    _warnings.warn(warn_message, ResourceWarning)

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
==================== 17 passed, 2 warnings in 3.47s ====================
  1. The documentation has been modified accordingly, like docstring or example tutorials.

@MiXaiLL76
Copy link
Author

@BIGWangYuDong Hello! I'm waiting for you Yudong Wang

@vansprosk8
Copy link

Good library to speed up calculations

@BIGWangYuDong
Copy link
Collaborator

LGTM, kindly ping @hhaAndroid have a look

@MiXaiLL76
Copy link
Author

Hello @hhaAndroid, I see that you are active on the site, please take a look at this PR

@MiXaiLL76
Copy link
Author

@RangiLyu Maybe you’re here and the project hasn’t been abandoned yet?

@@ -8,6 +8,8 @@

import numpy as np
import torch
from faster_coco_eval import COCO as FasterCOCO
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's best to make this dependency optional, to prevent affecting the previous usage pattern.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

@@ -1,3 +1,4 @@
faster-coco-eval
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's best to make this dependency optional, to prevent affecting the previous usage pattern.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

@hhaAndroid
Copy link
Collaborator

@MiXaiLL76 Thank you for your support

@MiXaiLL76
Copy link
Author

@MiXaiLL76 Thank you for your support

Thanks for a good framework!

@MiXaiLL76
Copy link
Author

@hhaAndroid Hi! Can you update your status?

@MiXaiLL76
Copy link
Author

@hhaAndroid @ZwwWayne Is anyone else working on the project? Maybe we can take it on as support?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants