Seems that all models are loaded to the first GPU?

[DDDOH]

I use four A100 run python -m prover.launch --config=configs/RMaxTS.py --log_dir=logs/RMaxTS_results
But always get cuda out of memory error.

I added some print message:

Then i get

All models are tried to load in first GPU. Is this expected? Or maybe pytorch behavior is different for new version (I am using '2.5.1+cu121', and the version in requirement.txt is 2.2.1)

[Luobots]

Same error. VLLM can't set the cuda index.

[zzhisthebest]

@Luobots How to solve this problem?I meet the same error.