Enable multi-gpu training when "torch" is chosen as the RETURNN backend

[Judyxujj]

Enable to launch the torch DDP training using torchrun

[albertz]

I'm curious: What happens if you just use mpirun? I would have expected that this works as well, and torchrun internally just does the same. So, this is not the case?

[albertz]

I'm not sure if it is a good idea to just always do this for the PyTorch backend. Maybe we should just introduce a separate option for this, to make it explicit? There might be valid cases to use mpirun with PyTorch?

[Judyxujj]

I'm curious: What happens if you just use mpirun? I would have expected that this works as well, and torchrun internally just does the same. So, this is not the case?

I tried to use mpirun to launch the torch distributed data parallel (DDP) training but it gives ValueError: Error initializing torch.distributed using env:// rendezvous: environment variable RANK expected, but not set.

[albertz]

environment variable RANK expected, but not set.

It sounds like the env var RANK is expected but not set. Did you try to set it? Maybe you need to add it in DEFAULT_ENVIRONMENT_KEEP in the Sisyphus settings or so? Or maybe try the Sisyphus setting CLEANUP_ENVIRONMENT = False. Did you try that?

[christophmluscher]

@JackTemaki friendly ping :)

"Nick not in the topic"