f07588a31b
[refactor] PPO cleanup
2026-07-23 11:11:05 +08:00
185baaa022
[fix] PPO rollout mode
...
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-23 11:05:51 +08:00
27b9c0082f
[fix] PPO: align mb_resp_logp with old_resp_logp under autocast
...
Move log_softmax for mb_resp_logp inside autocast_ctx so it is upcast
to fp32, matching the rollout path (compute_per_token_logps runs under
no_grad+autocast). Previously log_softmax ran outside autocast on
fp16/bf16 logits, introducing a ~1e-2 systematic bias between
mb_resp_logp and old_resp_logp, so ratio != 1 and KL != 0 even before
any weight update. Under sparse rule-based rewards (e.g. math PPO)
this bias destabilises training.
Also add:
- --rollout_eval_mode (default 1): switch actor to eval during rollout
to disable dropout and keep old/new logp consistent.
- --debug_log_ratio: optional diagnostic printing |mb-old| on the first
minibatch to verify ratio ~= 1.
2026-07-21 18:51:42 +08:00
13739bdbc5
[fix] ddp exit hang
2026-06-01 17:50:39 +08:00
6534447b62
[fix] issue#771
2026-05-19 17:40:03 +08:00
a22f1f1486
[feat] reduce RL memory
2026-05-06 15:07:28 +08:00
1c583deb28
fix: resolve CUDA OOM in train_grpo.py on GPUs with <=8GB VRAM
2026-05-06 14:14:15 +08:00
58959684c7
[fix] inference bug
2026-05-03 20:48:48 +08:00
4b85ba8b28
[fix] bugs
2026-04-27 19:16:08 +08:00
3b7a4d04a2
[fix] rollout bugs
2026-04-27 17:54:09 +08:00
568a863520
[fix] bugs
2026-04-21 13:03:34 +08:00
e612cc26e8
[fix] lora compile
2026-04-10 00:01:21 +08:00
862dd151f2
[fix] lora moe
2026-04-09 16:36:48 +08:00
b703844a60
[update] add a comment
2026-04-09 15:09:58 +08:00
2c24568176
[update] change default seq_len
2026-03-26 10:09:06 +08:00
b24fcdc1f9
[update] minimind-3
2026-03-25 23:57:45 +08:00
c88c497795
[fix] align log/save last-step check and ETA with 1-indexed step
2026-03-24 02:01:40 +01:00
9fe4343b7b
[fix] gradient accumulation step alignment
2026-03-24 01:45:04 +01:00
69180447af
[update] save interval
2026-01-30 20:30:50 +08:00
3d2f03cfff
[fix] data skip
2026-01-18 16:56:29 +08:00
f0aa3ba52e
[update] shuffle data
2026-01-18 16:39:34 +08:00
b189f0046a
[update] align loss
2026-01-15 00:56:32 +08:00
caf76fb884
[fix] compile unpack
2026-01-14 20:13:32 +08:00
6ef2105e34
[feat] add compile
2026-01-14 14:42:30 +08:00
379983849e
[update] params log
2026-01-07 23:08:45 +08:00
6bea38c741
[update] rename train tokenizer
2026-01-06 01:17:33 +08:00
cb6be70365
[update] rename reason
2026-01-05 23:12:29 +08:00
dbfeff23e3
[fix] dist cleanup
2026-01-02 22:25:55 +08:00
3c56aca1c1
[update] aux loss
2026-01-01 22:41:46 +08:00
5b87b9ad8a
[fix] experts unused
2025-12-31 21:47:04 +08:00
e82af4bf8f
[fix] layers set 8
2025-12-31 21:06:37 +08:00
1c333636c5
[fix] moe unused
2025-12-31 21:00:06 +08:00
73a2bcc412
[feat] get params
2025-12-31 20:46:59 +08:00
759036b822
[feat] get params
2025-12-31 20:44:34 +08:00
6ee867fffc
[feat] update lr
2025-12-31 10:27:09 +08:00
fc2eeff509
[feat] remove empty_cache
2025-12-27 07:14:36 +08:00
658ddcae3a
[feat] explicit left padding
2025-12-23 18:59:48 +08:00
dacd4e5121
[feat] adjust seq length
2025-12-14 20:41:58 +08:00
37300329c5
[fix] dtype & lr
2025-12-09 13:01:38 +08:00
8fcc81696d
[fix] Refactor get_lr function to include min_lr calculation
...
这里的退火算法会让参数里的lr的起始值变成原来lr的1.1倍，作出如下修改
2025-12-06 17:09:51 +08:00
95bcb7ba19
[fix] cuda memory #559
2025-12-01 16:17:43 +08:00
cafc3b22df
[feat] release memory
2025-11-27 19:39:49 +08:00
3382c90246
[fix] ppo mask
2025-11-19 23:39:02 +08:00
c8727733f3
[fix] prompt length calculate
2025-11-15 18:25:37 +08:00
86d5cc70ec
[feat] clear cache
2025-11-06 13:12:28 +08:00
7c3d660eec
[feat] update import
2025-10-31 23:45:55 +08:00
ccab16a9b9
[feat] add args
2025-10-30 10:05:12 +08:00
230b29b94a
[fix] model device
2025-10-29 10:36:28 +08:00
e29f695998
[feat] update trainer
2025-10-29 00:52:37 +08:00
cd576b58e8
[feat] update trainer
2025-10-28 23:30:10 +08:00
jingyaogong
FloatFrank
TKiteRunner
jingyaogong
readlnh
dyhuachi