libcudart error

[mattthelee]

I'm getting this error:

 bash dist_train.sh configs/mask_rcnn/vit-adapter-t-3x.py 4



Traceback (most recent call last):
  File "/workspace/./train.py", line 11, in <module>
    import mmdet_custom  # noqa: F401,F403
  File "/workspace/mmdet_custom/__init__.py", line 2, in <module>   
    from .models import *  # noqa: F401,F403
  File "/workspace/mmdet_custom/models/__init__.py", line 2, in <module>
    from .backbones import *  # noqa: F401,F403
  File "/workspace/mmdet_custom/models/backbones/__init__.py", line 2, in <module>
    from .beit_adapter import BEiTAdapter
  File "/workspace/mmdet_custom/models/backbones/beit_adapter.py", line 9, in <module>
    from ops.modules import MSDeformAttn
  File "/workspace/ops/modules/__init__.py", line 9, in <module>
    from .ms_deform_attn import MSDeformAttn
  File "/workspace/ops/modules/ms_deform_attn.py", line 19, in <module>
    from ..functions import MSDeformAttnFunction
  File "/workspace/ops/functions/__init__.py", line 9, in <module>  
    from .ms_deform_attn_func import MSDeformAttnFunction
  File "/workspace/ops/functions/ms_deform_attn_func.py", line 11, in <module>
    import MultiScaleDeformableAttention as MSDA
ImportError: libcudart.so.12: cannot open shared object file: No such file or directory
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 181) of binary: /opt/conda/bin/python

I have run the make file and used docker to ensure the right libraries, cuda and torch versions were installed.
The environment i'm running on is a docker container with the following dockerfile.

FROM pytorch/pytorch:1.13.1-cuda11.6-cudnn8-devel
RUN apt-get update && apt-get install ffmpeg libsm6 libxext6 gcc g++ build-essential -y
RUN pip install mmcv-full==1.7.0 -f https://download.openmmlab.com/mmcv/dist/cu116/torch1.13/index.html timm==0.4.12 mmdet==2.28.1 scipy
COPY svit /workspace/
ENV PYTHONPATH=/workspace/ops