Update Scalability Tutorial

[jarlsondre]

Summary

This PR updates the scalability tutorial in multiple ways:

• Cleaner code
  • Removing many prints that strictly were not necessary
  • Reusing functionality such as the argparser or the train_epoch function for easier readability
  • General formatting
  • Rewritten tutorial
• SLURM scripts
  • Uses the new Python scripts for running the SLURM jobs, e.g. running all the strategies and also the scaling test

Current Limitations

The scalability tutorial was traditionally created with simplicity in mind, and therefore it tries to minimize its dependency to the itwinai project. Because of this, it does not use the itwinai trainer, which is required for using the updated scalability metrics. Therefore, the current implementation of the scalability tutorial only has the absolute and relative wall-clock time plots. This is, however, not so different from how it was before.

Examples of resulting plots

This is done using a subset size of 5000, so not the full ImageNet dataset.

Relative Scalability Plot

Absolute Scalability Plot

List of tasks

I am including this list of tasks not only for my own benefit, but also to make it easier for reviewers to see what has been done. Keep in mind that this list is not exhaustive: I started adding to it after already finishing some tasks.

• Fix deepspeed local rank problem
  • Turned out to be caused by some environment variables related to OMPI local rank
• Fix Horovod not wanting to load stuff (runs out of time?)
  • This turned out to be caused by me forgetting to train on a subset of imagenet instead of the whole thing...
• Change the name of the default SLURM log dir
• Refactor the tutorial files
  • No need to make them perfect but there is a lot of redundant stuff that just causes confusion IMO
• Update the tutorial readme
• Allow user to specify nodes for scalability analysis
• Allow user to specify imagenet subset size
• Allow use to specify folder for storing scalability metrics

Related issue :
#221, #263

[annaelisalappe]

I'm a little unsure if my build of the docs worked correctly, you might have to show me later where I can find the new tutorial in the docs :)