-
|
Was trying to get Deepseek v3 running on across 3 machines and hit a 16 gpu limit, Is this related to Nvidia/CUDA, or just that Distributed inference is new and 1 machine would never have more than 16? |
Beta Was this translation helpful? Give feedback.
Answered by
slaren
Jan 13, 2025
Replies: 1 comment 1 reply
-
|
You should be able to use any number of devices by increasing the value of |
Beta Was this translation helpful? Give feedback.
1 reply
-
|
Bumped it up to 64 and I'm good to go, Thanks! |
Beta Was this translation helpful? Give feedback.
Answer selected by
justinjja
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
You should be able to use any number of devices by increasing the value of
GGML_SCHED_MAX_BACKENDS.