How to use multi-gpu to train

[fire-python]

Hi, @Rayhane-mamah,
I have two gpus, and I want to assign training tasks parallelly to both. Could you give some advice, where to do some change to the code, such as inserting "with tf.device(...):", will allow me to do that? Thanks.

[begeekmyfriend]

You could use this fork #152

[Rayhane-mamah]

Hi @fire-python, thanks for reaching out!

I am not sure if you're requesting a multi-gpu training or a limitation on used GPUs for the model so that you can run multiple different Tacotron trainings at the same time.

• If it's the first: This fork has a good multi-gpu support, we plan on applying those enhancements to our work, but it will take some time. (You see I'm a lazy person.. :) )
• If it's the second: The easiest way of doing so is to constraint the CUDA visible devices by adding this line in the following location:
  • line: os.environ["CUDA_VISIBLE_DEVICES"] = "my_id" (then you can add one parameter inside "hparams.py" (my_id = hparams.gpu_id) or "args" (my_id = args.gpu_id) to control which device to use and simply select the device from command line when launching the training.)
  • location: between the following two lines:
    

    Tacotron-2/train.py

    Lines 36 to 37 in e244457

    	os.environ['TF_CPP_MIN_LOG_LEVEL'] = str(args.tf_log_level)
    	run_name = args.name or args.model

I hope this answers your question? In any case I will be adding both those enhancements to the code. Thus I am leaving your issue open until I upload both of them :)

Thanks for reaching out @fire-python!

[fire-python]

It's simply the first requesting. I will try the fork you mentioned. Thanks