Rustification for Optimization and other thoughts for Performance Maximization

[BuildBackBuehler]

https://github.com/sensorial-systems/stable-diffusion/tree/main/stable-diffusion-trainer

To be honest, no clue how significant this is because I saw it on Reddit and it seemed more of a self-advertisement than anything. But considering SimpleTuner is about performance above all...besides, well, simplicity, I figured its worth a discussion.

I think besides the obvious improvements/speed/plain-english troubleshooting gainz to be had, I think it'll also temper/alleviate some issues with Torch/Dynamo. At least I know TVM-Unity has some Rust options/features that'd enable anyone (cough Mac users) to churn out those graphs as intended. TVM also has support for ARM Computing Library Graphing and some other stuff...if that has any relevance. I also wonder (with grandiosity of course) if one would be able to detach from Torch, use ONNX or some other Rust-friendly framework(s) for training...presuming one can convert models/LoRAs effortlessly enough. Though it may be less relevant because SD3 is purportedly less VRAM intensive than SDXL.

Other than that I was wondering if you @bghira had/would consider a change to the args template. I'm a huge fan of TOMLs and feel like one could get a much more organized/clean args script to feed into the training.py. It'd allow for some modularity. Further, I imagine it'd allow for some neat fusions/integrations. In my head, or moreso pulled out my arse, I can imagine linting tools/Poetry/Mise (formerly RTX) providing some utility.

These were just some pervasive thoughts of mine, I think I have some more ideas swirling around in the background which I'll add later.

Bonus non-trainer Rust-SD implementation https://github.com/Gadersd/stable-diffusion-xl-burn