github
/
minimind-v
mirror of https://github.com/jingyaogong/minimind-v.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

fix: add find_unused_parameters=True to DDP for mixed text-image batches 

When a batch contains only text samples (pixel_values is None),
vision_proj parameters do not participate in the forward pass,
causing DDP to raise:
  'Expected to have finished reduction in the prior iteration'

Adding find_unused_parameters=True resolves this.
pull/112/head
王路宝 2026-04-01 14:20:30 +08:00
parent 14cfa88629
commit 66ad1c4b7f
1 changed files with 1 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
trainer/train_sft_vlm.py
View File

@ -150,7 +150,7 @@ if __name__ == "__main__":
Logger('torch.compile enabled') Logger('torch.compile enabled')
if dist.is_initialized(): if dist.is_initialized():
model._ddp_params_and_buffers_to_ignore = {"freqs_cos", "freqs_sin"} model._ddp_params_and_buffers_to_ignore = {"freqs_cos", "freqs_sin"}
model = DistributedDataParallel(model, device_ids=[local_rank]) model = DistributedDataParallel(model, device_ids=[local_rank], find_unused_parameters=True)
# ========== 8. 开始训练 ========== # ========== 8. 开始训练 ==========
for epoch in range(start_epoch, args.epochs): for epoch in range(start_epoch, args.epochs):