SDXL Training, prediction_type = 'epsilon' + snr_gamma + rescale_betas_zero_snr will case NaN

[Jannchie]

I read the document at

https://huggingface.co/docs/diffusers/api/schedulers/ddim.

It mentions that using rescale_betas_zero_snr with v-prediction might be more scientifically sound. However, rescale_betas_zero_snr with e-prediction should also work. Yet, when rescale_betas_zero_snr is True, the calculated snr is 0 at timestamp 999.

And when snr_gamma is enabled, calculating the loss requires dividing by snr, which triggers a division by zero error, causing the loss to become NaN. (https://github.com/huggingface/diffusers/blob/6dfaec348780c6153a4cfd03a01972a291d67f82/examples/text_to_image/train_text_to_image_sdxl.py#L1147)

I think e-prediction, snr_gamma, and rescale_betas_zero_snr should be compatible options that can work together. I don’t have much knowledge about how these work, but maybe adding a small epsilon when calculating snr could solve the problem, though I’m not sure if it’s reasonable. Could someone offer some advice?

[bghira]

it must be v_prediction for zsnr

[Jannchie]

Thank you for the quick response on Christmas Eve.

Actually, I am learning based on the parameter documentation of this repository.

In the documentation related to rescale_betas_zero_snr, it is stated that using v-prediction is “recommended.”

SimpleTuner/OPTIONS.md

Lines 853 to 857 in c6cdbb0

	--rescale_betas_zero_snr
	If set, will rescale the betas to zero terminal SNR.
	This is recommended for training with v_prediction.
	For epsilon, this might help with fine details, but
	will not result in contrast improvements.

So I mistakenly assumed that it would work correctly under e-prediction as well.

Perhaps it might be necessary to raise an error when rescale_betas_zero_snr is set to True and the prediction type is not v-prediction?

[bghira]

nope, because some people insist that it works for them, and they complain when these limits are put into place.