this project support deepspeed pipeline

[mobilejammer]

when I creat a config json file to deepspeed, the memory out of also. and seem no part the model

[bghira]

please try and complete your thoughts before submitting issue reports or discussions. this doesnt contain information others would need to help you. i mentioned before that you need to at the least submit a config file and the steps taken to lead to the problem. read your own discussion post back; how do you expect anyone to help?

[mobilejammer]

I'm terribly sorry that I didn't provide enough information according to your previous reminder to enable you to help me with debugging. I'm training the Flux model with all parameters following the "distributed.md" and "sd3.md". After configuring the DeepSpeed in "accelerate config" to be zero3, I found that the model didn't seem to run in shards.

cuda out of memory. When I used Fully Sharded Data Parallel (FSDP), it would run in shards, but it got stuck after the training was completed.

My graphics card is an NVIDIA GeForce RTX 4090. My Flux FSDP configuration is as follows (the content of the file "~/.cache/huggingface/accelerate/default_config.yaml"):
compute_environment: LOCAL_MACHINE
debug: false
distributed_type: FSDP
downcast_bf16: 'no'
dynamo_config:
dynamo_backend: INDUCTOR
dynamo_mode: default
dynamo_use_dynamic: false
dynamo_use_fullgraph: false
enable_cpu_affinity: false
fsdp_config:
fsdp_activation_checkpointing: true
fsdp_auto_wrap_policy: TRANSFORMER_BASED_WRAP
fsdp_backward_prefetch: BACKWARD_PRE
fsdp_cpu_ram_efficient_loading: true
fsdp_forward_prefetch: true
fsdp_offload_params: true
fsdp_sharding_strategy: FULL_SHARD
fsdp_state_dict_type: SHARDED_STATE_DICT
fsdp_sync_module_states: true
fsdp_transformer_layer_cls_to_wrap: FluxTransformerBlock,FluxSingleTransformerBlock
fsdp_use_orig_params: true
machine_rank: 0
main_training_function: main
mixed_precision: bf16
num_machines: 1
num_processes: 8
rdzv_backend: static
same_network: true
tpu_env: []
tpu_use_cluster: false
tpu_use_sudo: false
use_cpu: false
When I tested DeepSpeed Zero3 with multiple machines and multiple GPUs, I still got the "cuda out of memory" error. So I feel that Zero3 doesn't seem to play the real role of splitting large models in this project.

[mobilejammer]

when I use fsdp config training:
0:14,190 [INFO] Training has completed.
-> global_step = 10, max_train_steps = 10, epoch = 1, num_train_epochs = 1
2024-12-28 21:00:14,191 [INFO] Exiting training loop. Beginning model unwind at epoch 1, step 10
192-168-1-86:2006783:2007685 [4] NCCL INFO [Service thread] Connection closed by localRank 4
192-168-1-86:2006784:2007683 [5] NCCL INFO [Service thread] Connection closed by localRank 5
192-168-1-86:2006786:2007682 [7] NCCL INFO [Service thread] Connection closed by localRank 7
192-168-1-86:2006785:2007684 [6] NCCL INFO [Service thread] Connection closed by localRank 6
192-168-1-86:2006781:2007690 [2] NCCL INFO [Service thread] Connection closed by localRank 2
192-168-1-86:2006780:2007687 [1] NCCL INFO [Service thread] Connection closed by localRank 1
192-168-1-86:2006782:2007686 [3] NCCL INFO [Service thread] Connection closed by localRank 3
192-168-1-86:2006786:2013858 [7] NCCL INFO comm 0x5580d734b5f0 rank 7 nranks 8 cudaDev 7 busId e1000 - Abort COMPLETE
192-168-1-86:2006783:2013852 [4] NCCL INFO comm 0x55ddfc00dd00 rank 4 nranks 8 cudaDev 4 busId 81000 - Abort COMPLETE
192-168-1-86:2006784:2013854 [5] NCCL INFO comm 0x5585191f4b60 rank 5 nranks 8 cudaDev 5 busId a1000 - Abort COMPLETE
192-168-1-86:2006785:2013856 [6] NCCL INFO comm 0x55763c6d2370 rank 6 nranks 8 cudaDev 6 busId c1000 - Abort COMPLETE
192-168-1-86:2006782:2013857 [3] NCCL INFO comm 0x561adcdb1580 rank 3 nranks 8 cudaDev 3 busId 61000 - Abort COMPLETE
192-168-1-86:2006781:2013853 [2] NCCL INFO comm 0x5639c2994ed0 rank 2 nranks 8 cudaDev 2 busId 41000 - Abort COMPLETE
192-168-1-86:2006780:2013855 [1] NCCL INFO comm 0x5558289100c0 rank 1 nranks 8 cudaDev 1 busId 23000 - Abort COMPLETE

when I enter ctrl+c to force quite

^CW1228 21:03:38.744000 2006575 torch/distributed/elastic/agent/server/api.py:704] Received Signals.SIGINT death signal, shutting down workers
W1228 21:03:38.747000 2006575 torch/distributed/elastic/multiprocessing/api.py:897] Sending process 2006779 closing signal SIGINT
^CW1228 21:03:39.344000 2006575 torch/distributed/elastic/multiprocessing/api.py:897] Sending process 2006779 closing signal SIGTERM
^CTraceback (most recent call last):
File "/home/ubuntu/SimpleTuner/.venv/lib/python3.10/site-packages/torch/distributed/elastic/agent/server/api.py", line 696, in run
result = self._invoke_run(role)
File "/home/ubuntu/SimpleTuner/.venv/lib/python3.10/site-packages/torch/distributed/elastic/agent/server/api.py", line 855, in _invoke_run
time.sleep(monitor_interval)
File "/home/ubuntu/SimpleTuner/.venv/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 84, in _terminate_process_handler
raise SignalException(f"Process {os.getpid()} got signal: {sigval}", sigval=sigval)
torch.distributed.elastic.multiprocessing.api.SignalException: Process 2006575 got signal: 2

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/ubuntu/SimpleTuner/.venv/lib/python3.10/site-packages/torch/distributed/elastic/agent/server/api.py", line 705, in run
self._shutdown(e.sigval)
File "/home/ubuntu/SimpleTuner/.venv/lib/python3.10/site-packages/torch/distributed/elastic/agent/server/local_elastic_agent.py", line 365, in _shutdown
self._pcontext.close(death_sig)
File "/home/ubuntu/SimpleTuner/.venv/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 572, in close
self._close(death_sig=death_sig, timeout=timeout)
File "/home/ubuntu/SimpleTuner/.venv/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 909, in _close
handler.proc.wait(time_to_wait)
File "/usr/lib/python3.10/subprocess.py", line 1209, in wait
return self._wait(timeout=timeout)
File "/usr/lib/python3.10/subprocess.py", line 1953, in _wait
time.sleep(delay)
File "/home/ubuntu/SimpleTuner/.venv/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 84, in _terminate_process_handler
raise SignalException(f"Process {os.getpid()} got signal: {sigval}", sigval=sigval)
torch.distributed.elastic.multiprocessing.api.SignalException: Process 2006575 got signal: 2

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/ubuntu/SimpleTuner/.venv/bin/accelerate", line 8, in
sys.exit(main())
File "/home/ubuntu/SimpleTuner/.venv/lib/python3.10/site-packages/accelerate/commands/accelerate_cli.py", line 48, in main
args.func(args)
File "/home/ubuntu/SimpleTuner/.venv/lib/python3.10/site-packages/accelerate/commands/launch.py", line 1155, in launch_command
multi_gpu_launcher(args)
File "/home/ubuntu/SimpleTuner/.venv/lib/python3.10/site-packages/accelerate/commands/launch.py", line 793, in multi_gpu_launcher
distrib_run.run(args)
File "/home/ubuntu/SimpleTuner/.venv/lib/python3.10/site-packages/torch/distributed/run.py", line 910, in run
elastic_launch(
File "/home/ubuntu/SimpleTuner/.venv/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 138, in call
return launch_agent(self._config, self._entrypoint, list(args))
File "/home/ubuntu/SimpleTuner/.venv/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 260, in launch_agent
result = agent.run()
File "/home/ubuntu/SimpleTuner/.venv/lib/python3.10/site-packages/torch/distributed/elastic/metrics/api.py", line 137, in wrapper
result = f(*args, **kwargs)
File "/home/ubuntu/SimpleTuner/.venv/lib/python3.10/site-packages/torch/distributed/elastic/agent/server/api.py", line 710, in run
self._shutdown()
File "/home/ubuntu/SimpleTuner/.venv/lib/python3.10/site-packages/torch/distributed/elastic/agent/server/local_elastic_agent.py", line 365, in _shutdown
self._pcontext.close(death_sig)
File "/home/ubuntu/SimpleTuner/.venv/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 572, in close
self._close(death_sig=death_sig, timeout=timeout)
File "/home/ubuntu/SimpleTuner/.venv/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 909, in _close
handler.proc.wait(time_to_wait)
File "/usr/lib/python3.10/subprocess.py", line 1209, in wait
return self._wait(timeout=timeout)
File "/usr/lib/python3.10/subprocess.py", line 1953, in _wait
time.sleep(delay)
File "/home/ubuntu/SimpleTuner/.venv/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 84, in _terminate_process_handler
raise SignalException(f"Process {os.getpid()} got signal: {sigval}", sigval=sigval)
torch.distributed.elastic.multiprocessing.api.SignalException: Process 2006575 got signal: 2

[bghira]

if you want to use FSDP i would just rely on checkpoints instead of the final model export. see if that works.

[bghira]

i think the deepspeed zero3 was definitely sharding things because when you enable it, the CLIP and T5 models start sharding and this caused the earlier bugs.

the transformer model is sharded or it wouldn't have been able to even create the optimiser states.

[mobilejammer]

if you want to use FSDP i would just rely on checkpoints instead of the final model export. see if that works.
When the Fully Sharded Data Parallel (FSDP) distributed training mode reaches the checkpoint, it gets stuck and doesn't continue to execute. There is no model in the output directory.

[bghira]

i guess that would have to be examined and fixed first, and i do not have the time