Multi GPU training

[zhangxd12]

Hello, I try to run the train process on 7 GPUs, it seems the GPUs has been found according to the log

But nvidia-smi shows only one GPU is used,

Do you have any suggestions?
Thank you !

[jishengpeng]

Hello, I try to run the train process on 7 GPUs, it seems the GPUs has been found according to the log

But nvidia-smi shows only one GPU is used,

Do you have any suggestions? Thank you !

You can try this: unset KUBERNETES_PORT

[zhangxd12]

Thank you ! It works!
But multi GPUs are even slower than single GPU, if I use 7 GPU , the log is like

But single GPU log is like