Fix lora single device fine tune checkpoint saving & nan loss when use_dora=True

Fix lora single device fine tune checkpoint saving & nan loss when use_dora=True #5859