I'm assuming by multi-GPU you mean something similar to DataParallel or DistributedDataParallel. We support multi-GPU to some extent -- The modules we provide are generally device agnostic so they could be put on any device, and should play nicely with DP or DDP. However:

1. The data modules and Executor do not support multi-GPU. You'll need to manually move data around.
2. It might not be possible to put different parts of a model on different GPUs, or at least, it's not possible automatically.

It's easy to run on multi-/distributed-GPUs, e.g., with Horovod. Here are two examples of using Texar with Horovod (by simply adding a couple of extra lines of code to your single-GPU Texar code): here and here.

The examples are on Texar-TF, but Horovod also supports PyTorch.

@ZhitingHu could you add an example for using horovod in texar-torch?