How to use Detic instead of GroundingDINO as the detector?

[yashbhalgat]

Hi @siyuanliii , I want to perform open-vocabulary detection and tracking using Detic. However, I am not sure how to change the detector from GroundingDINO to Detic. Can you provide some instructions on how to do it?

I downloaded the Detic model from here to saved_models/masa_models directory. But how do I create a config for Detic, etc.? Would appreciate your help with this!

Also, if I want to use OWLv2 as the open-vocabulary 2D model, how would I do that?

Thanks!

[siyuanliii]

Hi, thanks for your question, and sorry for getting to you late! Yes, you can use any open-vocabulary detector! We have provided config files Yolo-x detector, as an example. In the demo/video_demo_with_text.py, try to set --det_config and --det_checkpoint, to be detic's config file and checkpoint. Similarly, if you want to use OWLv2, you can do follow two options: 1). first is to integrate it into mmdetection, then give the config and checkpoint like above. 2). Alternatively, you can modify the video_demo_with_text.py, to replace the results with your detections from OWLv2.

[EricHannus]

Hi @siyuanliii I am also interested in using Detic with MASA, but I am still confused about a few things after reading this thread.

First, is my understanding correct that the detic-masa model (https://huggingface.co/dereksiyuanli/masa/resolve/main/detic_masa.pth) which @yashbhalgat asked about above is an example of a unified model, and should thus be possible to run similarly to the provided Demo1? Similarly, the YOLOX demo showcases how to load the detector and MASA adapter separately, right? I managed to use some of the non-unified detic models listed here https://github.com/siyuanliii/masa/tree/main/projects/Detic_new with the corresponding config files like in the YOLOX example. However, I did not manage to find the files "demo/image_demo.py" and "tools/model_converters/detic_to_mmdet.py" referred to on this page.

I also tried running the unified detic-masa model like shown in Demo1 for GroundingDINO, using the config file “configs/masa-detic/open_vocabulary_mot_test/masa_detic_swinb_open_vocabulary_test.py”. However, I had to do two modifications before I was able to run Demo1 with Detic. First, I had to create the directory “data/metadata” and fill it with these files from Detic repository (https://github.com/facebookresearch/Detic/tree/main/datasets/metadata). Second, I had to add this piece of code to the Detic config file, which I had found in the “masa_gdino_swinb_inference.py“ config of Demo1 and “masa_r50_plug_and_play.py” config of the YOLOX demo (with different scale), both of which ran fine for me:
inference_pipeline = [
dict(
type='TransformBroadcaster',
transforms=[
dict(
type='Resize',
scale=(1333, 800),
keep_ratio=True),
]),
dict(type='PackTrackInputs')
]

Could you please clarify whether these two modifications are really necessary to get the unified detic-masa model running, and where the files "demo/image_demo.py" and "tools/model_converters/detic_to_mmdet.py" can be found?

[siyuanliii]

Thanks for the question! Yes, those two modifications are needed. I will provide an update to include those for detic. The doc you are referring Detic is from default mmdetection 3.3.0. This is for running object detection with detic. The two files can be found on original mmdetection repo. "demo/image_demo.py and "tools/model_converters/detic_to_mmdet.py"