Could somebody share a sdxl-env.sh that works with a 24gb gpu (3090/4090)? I keep getting cuda OOM #219
-
|
Thanks in advance. Been following this repo's development on SDXL for a while now, excited to have time to sit down and test it out. If someone has successfully finetuned on a 3090/4090, please share your config so I can have a starting point to work with. Just trying to get it running. I've followed all the instructions I can find here on the github, configured accelerate/deepspeed, edited sdxl-env.sh. And yet I get cuda OOM errors on 3090/4090. Accelerate config according to example on: DEEPSPEED.MD In sdxl-env.sh, kept everything same except: Training machine is Ubuntu 22.04, 2x 3090, 1x 4090, 64gb ram. End goal is to use deepspeed to split the SDXL model across multiple gpus (3090/4090s). Any info towards that goal would be appreciated. Error output: |
Beta Was this translation helpful? Give feedback.
Replies: 2 comments 7 replies
-
|
i believe the pytorch version has a substantial impact here. can you share your library versions? and your deepspeed config? |
Beta Was this translation helpful? Give feedback.
-
|
Appreciate the reply. Everything should be according to pyproject.toml & DEEPSPEED.MD. Attached is deepspeed config, pip list, and edxl-env.sh export values. Figured put it all out at once and find the issue. System: Ubuntu 22.04, cuda 12.1, 64gb system ram, rtx 3090/4090 /home/rtxhome/.cache/huggingface/accelerate/default_config.yaml compute_environment: LOCAL_MACHINE
deepspeed_config:
gradient_accumulation_steps: 1
zero3_init_flag: false
zero_stage: 1
distributed_type: DEEPSPEED
downcast_bf16: 'no'
machine_rank: 0
main_training_function: main
mixed_precision: bf16
num_machines: 1
num_processes: 1
rdzv_backend: static
same_network: true
tpu_env: []
tpu_use_cluster: false
tpu_use_sudo: false
use_cpu: false(.venv) pip list: Package Version accelerate 0.21.0 sdxl-env.sh: export RESUME_CHECKPOINT="latest"
export CHECKPOINTING_STEPS=150
export CHECKPOINTING_LIMIT=2
export LEARNING_RATE=8e-7 #@param {type:"number"}
export MODEL_NAME="stabilityai/stable-diffusion-xl-base-1.0"
export DEBUG_EXTRA_ARGS=""
export TRACKER_PROJECT_NAME="sdxl-training"
export TRACKER_RUN_NAME="simpletuner-sdxl"
export VALIDATION_PROMPT="ethnographic photography of teddy bear at a picnic"
export VALIDATION_GUIDANCE=7.5
export VALIDATION_GUIDANCE_RESCALE=0.0
export VALIDATION_STEPS=100
export MAX_NUM_STEPS=30000
export NUM_EPOCHS=25
export BASE_DIR="/mnt/storage/training/datasets/test-out"
export INSTANCE_DIR="/mnt/storage/training/datasets/test-instance"
export OUTPUT_DIR="/mnt/storage/training/ckpts"
export RESOLUTION=1024
export MINIMUM_RESOLUTION=$RESOLUTION
export VALIDATION_RESOLUTION=$RESOLUTION
export RESOLUTION_TYPE="pixel"
export TRAIN_BATCH_SIZE=1
export GRADIENT_ACCUMULATION_STEPS=4
export STATE_PATH="${BASE_DIR}/training_state.json"
export SEEN_STATE_PATH="${BASE_DIR}/training_images_seen.json"
export LR_SCHEDULE="constant"
export LR_WARMUP_STEPS=$((MAX_NUM_STEPS / 10))
export CAPTION_DROPOUT_PROBABILITY=0.1
export TRAINER_EXTRA_ARGS="--allow_tf32 --use_adafactor_optimizer"
export TRAINING_SEED=420420420
export MIXED_PRECISION="bf16" # Might not be supported on all GPUs. fp32 will be needed for others.
export TRAINING_NUM_PROCESSES=1
export TRAINING_NUM_MACHINES=1
export ACCELERATE_EXTRA_ARGS="" # --multi_gpu or other similar flags for huggingface accelerate
export TRAINER_EXTRA_ARGS="${TRAINER_EXTRA_ARGS} --enable_xformers_memory_efficient_attention --set_grads_to_none"
export TRAINER_EXTRA_ARGS="${TRAINER_EXTRA_ARGS} --gradient_checkpointing --gradient_accumulation_steps=${GRADIENT_ACCUMULATION_STEPS}"
export TRAINING_DYNAMO_BACKEND='no' # or 'inductor' if you want to brave PyTorch 2 compile issues |
Beta Was this translation helpful? Give feedback.
-
|
does the same occur with zero stage 2 or 3? |
Beta Was this translation helpful? Give feedback.
-
|
Yes. It also cuda OOMs with zero stage 2 & 3. That was the first thing I tried. Currently, my plan is:
If you have any suggestions, or want me to test something, let me know. I'm in no rush. Currently still working on my dataset. Just getting started setting up the training environment. |
Beta Was this translation helpful? Give feedback.
-
compute_environment: LOCAL_MACHINE
deepspeed_config:
gradient_accumulation_steps: 1
offload_optimizer_device: cpu
offload_param_device: cpu
zero3_init_flag: false
zero_stage: 2
distributed_type: DEEPSPEED
downcast_bf16: 'no'
machine_rank: 0
main_training_function: main
mixed_precision: fp16
num_machines: 1
num_processes: 1
rdzv_backend: static
same_network: true
tpu_env: []
tpu_use_cluster: false
tpu_use_sudo: false
use_cpu: falseso this is what works on an 8GB laptop with a NVIDIA 3070. and then tested this on NVIDIA 4090 24G: ----------------------------------------------------------------------------------------------------------------------------
In which compute environment are you running?
This machine
----------------------------------------------------------------------------------------------------------------------------
Which type of machine are you using?
No distributed training
Do you want to run your training on CPU only (even if a GPU / Apple Silicon / Ascend NPU device is available)? [yes/NO]:NO
Do you wish to optimize your script with torch dynamo?[yes/NO]:NO
Do you want to use DeepSpeed? [yes/NO]: yes
Do you want to specify a json file to a DeepSpeed config? [yes/NO]: NO
----------------------------------------------------------------------------------------------------------------------------
What should be your DeepSpeed's ZeRO optimization stage?
1
How many gradient accumulation steps you're passing in your script? [1]: 4
Do you want to use gradient clipping? [yes/NO]: NO
Do you want to enable `deepspeed.zero.Init` when using ZeRO Stage-3 for constructing massive models? [yes/NO]: NO
How many GPU(s) should be used for distributed training? [1]: 1
----------------------------------------------------------------------------------------------------------------------------
Do you wish to use FP16 or BF16 (mixed precision)? bf16
accelerate configuration saved at /root/.cache/huggingface/accelerate/default_config.yaml |
Beta Was this translation helpful? Give feedback.
-
|
Is it still possible to train SDXL with 24GB vram using latest release branch? Now that the only optimizer supported is adam. I always get cuda OOM with deepspeed stage 1. Stage 2 is way too slow, 100+ s/it. |
Beta Was this translation helpful? Give feedback.
-
|
Also, training speed is much slower than before, probably because I was using adam8bit before. Wonder what's the reason behind the decision to only support adam? |
Beta Was this translation helpful? Give feedback.
-
|
bits and bytes isn't multi-platform not really sure why it's slower, it shouldn't be. |
Beta Was this translation helpful? Give feedback.
-
|
8bit adam actually wasn't the reason 24G was doable before, that was Adafactor but bf16 support can be introduced to any one of the optimizers with the exception of Bits and Bytes, which would require altering that upstream project. for any of the python optimizers, just open a pull request with a stochastic bf16 variant. |
Beta Was this translation helpful? Give feedback.
8bit adam actually wasn't the reason 24G was doable before, that was Adafactor
but bf16 support can be introduced to any one of the optimizers with the exception of Bits and Bytes, which would require altering that upstream project. for any of the python optimizers, just open a pull request with a stochastic bf16 variant.