torch.cuda.OutOfMemoryError on HuhhingFace NVidia 4xA10G Large

[lkthomas]

torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 288.00 MiB. GPU 0 has a total capacty of 21.99 GiB of which 59.00 MiB is free. Process 42083 has 21.92 GiB memory in use. Of the allocated memory 21.59 GiB is allocated by PyTorch, and 47.14 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

I am trying to run StarCoder and still getting this error after using Nvidia 4xA10G Large, it doesn't seems utilize VRAM more than 1 GPU, how could I fix this ?

[loubnabnl]

Can you try using the code in this section https://github.com/bigcode-project/starcoder?tab=readme-ov-file#inference-hardware-requirements this loads the model in ~16GB of RAM. Also the device_map="auto" is needed to dispatch the model on multiple GPUs.

[lkthomas]

Any idea how to load this model on hugging face Spaces? On 11 Feb 2024, at 8:15 AM, Loubna Ben Allal ***@***.***> wrote: Can you try using the code in this section https://github.com/bigcode-project/starcoder?tab=readme-ov-file#inference-hardware-requirements this loads the model in ~16GB of RAM —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you authored the thread.Message ID: ***@***.***>