Wan2.2-TI2V-5B lora训练崩溃，全是噪声

[xinjiu612]

各位大佬好，我在训练Wan2.2-TI2V-5B model的时候想初步尝试能否成功，就用了自己收集的十几个video,来训练，结果训练出来的model，其实也就两个epoch之后就产生出来全是噪声。 我不清楚这是否是数据集数量过少或者是lora的其他设置不合理导致的崩溃，但是原理上小批量数据训练几个epoch应该不会导致模式崩塌吧。

dataset_repeat=10
--learning_rate 1e-4
--num_epochs 5
--remove_prefix_in_ckpt "pipe.dit."
--output_path "./models/train/Wan2.2-TI2V-5B_lora"
--lora_base_model "dit"
--lora_target_modules "q,k,v,o,ffn.0,ffn.2"
--lora_rank 32 \

[Artiprocher]

@xinjiu612 减小学习率到 1e-5 后，是否会有改善呢？

[xinjiu612]

学习率减小到1e-5能稍微好一点，但是还是能看得出来再接着训会继续出现噪声。很奇怪的是，在同样的setting下我训练Wan2.2-I2V-A14B就不会出现这个问题。

Wan2.2-TI2V-5B training:
export DIFFSYNTH_SKIP_DOWNLOAD=True
accelerate launch --config_file my_train/accelerate_config_zero3.yaml examples/wanvideo/model_training/train.py
--dataset_base_path data/example_video_dataset
--dataset_metadata_path data/example_video_dataset/metadata.csv
--height 480
--width 832
--num_frames 153
--dataset_repeat 100
--model_paths "["/beijing-c/workspace/hxj/Wan2.2/Wan2.2-TI2V-5B/models_t5_umt5-xxl-enc-bf16.pth",["/beijing-c/workspace/hxj/Wan2.2/Wan2.2-TI2V-5B/diffusion_pytorch_model-00001-of-00003.safetensors","/beijing-c/workspace/hxj/Wan2.2/Wan2.2-TI2V-5B/diffusion_pytorch_model-00002-of-00003.safetensors","/beijing-c/workspace/hxj/Wan2.2/Wan2.2-TI2V-5B/diffusion_pytorch_model-00003-of-00003.safetensors"],"/beijing-c/workspace/hxj/Wan2.2/Wan2.2-TI2V-5B/Wan2.2_VAE.pth"]"
--tokenizer_path /beijing-c/workspace/hxj/Wan2.2/Wan2.2-TI2V-5B/google/umt5-xxl
--learning_rate 1e-4
--num_epochs 5
--remove_prefix_in_ckpt "pipe.dit."
--output_path "./models/train/Wan2.2-TI2V-5B_lora"
--lora_base_model "dit"
--lora_target_modules "q,k,v,o,ffn.0,ffn.2"
--lora_rank 32
--extra_inputs "input_image" \

Wan2.2-I2V-A14B training:
accelerate launch --config_file my_train/accelerate_config_zero3.yaml examples/wanvideo/model_training/train.py
--dataset_base_path data/gate_dataset
--dataset_metadata_path data/gate_dataset/metadata.csv
--height 480
--width 832
--num_frames 153
--dataset_repeat 50
--model_paths "["/beijing-c/workspace/hxj/Wan2.2/Wan2.2-I2V-A14B/models_t5_umt5-xxl-enc-bf16.pth",["/beijing-c/workspace/hxj/Wan2.2/Wan2.2-I2V-A14B/high_noise_model/diffusion_pytorch_model-00001-of-00006.safetensors","/beijing-c/workspace/hxj/Wan2.2/Wan2.2-I2V-A14B/high_noise_model/diffusion_pytorch_model-00002-of-00006.safetensors","/beijing-c/workspace/hxj/Wan2.2/Wan2.2-I2V-A14B/high_noise_model/diffusion_pytorch_model-00003-of-00006.safetensors","/beijing-c/workspace/hxj/Wan2.2/Wan2.2-I2V-A14B/high_noise_model/diffusion_pytorch_model-00004-of-00006.safetensors","/beijing-c/workspace/hxj/Wan2.2/Wan2.2-I2V-A14B/high_noise_model/diffusion_pytorch_model-00005-of-00006.safetensors","/beijing-c/workspace/hxj/Wan2.2/Wan2.2-I2V-A14B/high_noise_model/diffusion_pytorch_model-00006-of-00006.safetensors"],"/beijing-c/workspace/hxj/Wan2.2/Wan2.2-I2V-A14B/Wan2.1_VAE.pth"]"
--tokenizer_path /beijing-c/workspace/hxj/Wan2.2/Wan2.2-I2V-A14B/google/umt5-xxl
--learning_rate 1e-4
--num_epochs 5
--remove_prefix_in_ckpt "pipe.dit."
--output_path "./models/train/Wan2.2-I2V-A14B_gate_high_noise_lora"
--lora_base_model "dit"
--lora_target_modules "q,k,v,o,ffn.0,ffn.2"
--lora_rank 32
--extra_inputs "input_image"
--max_timestep_boundary 0.358
--min_timestep_boundary 0

boundary corresponds to timesteps [900, 1000]

accelerate launch --config_file my_train/accelerate_config_zero3.yaml examples/wanvideo/model_training/train.py
--dataset_base_path data/gate_dataset
--dataset_metadata_path data/gate_dataset/metadata.csv
--height 480
--width 832
--num_frames 153
--dataset_repeat 50
--model_paths "["/beijing-c/workspace/hxj/Wan2.2/Wan2.2-I2V-A14B/models_t5_umt5-xxl-enc-bf16.pth",["/beijing-c/workspace/hxj/Wan2.2/Wan2.2-I2V-A14B/low_noise_model/diffusion_pytorch_model-00001-of-00006.safetensors","/beijing-c/workspace/hxj/Wan2.2/Wan2.2-I2V-A14B/low_noise_model/diffusion_pytorch_model-00002-of-00006.safetensors","/beijing-c/workspace/hxj/Wan2.2/Wan2.2-I2V-A14B/low_noise_model/diffusion_pytorch_model-00003-of-00006.safetensors","/beijing-c/workspace/hxj/Wan2.2/Wan2.2-I2V-A14B/low_noise_model/diffusion_pytorch_model-00004-of-00006.safetensors","/beijing-c/workspace/hxj/Wan2.2/Wan2.2-I2V-A14B/low_noise_model/diffusion_pytorch_model-00005-of-00006.safetensors","/beijing-c/workspace/hxj/Wan2.2/Wan2.2-I2V-A14B/low_noise_model/diffusion_pytorch_model-00006-of-00006.safetensors"],"/beijing-c/workspace/hxj/Wan2.2/Wan2.2-I2V-A14B/Wan2.1_VAE.pth"]"
--tokenizer_path /beijing-c/workspace/hxj/Wan2.2/Wan2.2-I2V-A14B/google/umt5-xxl
--learning_rate 1e-4
--num_epochs 5
--remove_prefix_in_ckpt "pipe.dit."
--output_path "./models/train/Wan2.2-I2V-A14B_gate_low_noise_lora"
--lora_base_model "dit"
--lora_target_modules "q,k,v,o,ffn.0,ffn.2"
--lora_rank 32
--extra_inputs "input_image" \
--max_timestep_boundary 1
--min_timestep_boundary 0.358

boundary corresponds to timesteps [0, 900)

下面这个事情是1e-5次在example的单个视频下讲datasets开到500,训练4个epoch之后的表现，依旧会出现噪声的现象
https://github.com/user-attachments/assets/f18080ff-53b7-4839-bda5-4016e75f7c2c