baihaochen
|
46dabcfc19
|
chore: remove unused import
|
2026-05-23 16:42:02 +08:00 |
|
baihaochen
|
907f31f357
|
chore: add deepspeed suffix to save_weight, update default data_path
|
2026-05-23 16:38:49 +08:00 |
|
baihaochen
|
4f5cdfe59a
|
fix: avoid double init_process_group in DeepSpeed pretrain
|
2026-05-23 16:21:57 +08:00 |
|
baihaochen
|
ed006a254f
|
[feat] add DeepSpeed pretrain (ZeRO-2, BF16, auto LR, gradient accumulation)
|
2026-05-22 00:27:06 +08:00 |
|
baihaochen
|
09e491f940
|
Merge branch 'master' of github.com:daiweidaichenzi/minimind
|
2026-05-14 14:29:06 +08:00 |
|
baihaochen
|
8baa9c5f71
|
fix: add missing --use_mla to train_full_sft.py
|
2026-05-14 14:28:11 +08:00 |
rainpeach-m
|
de683eafae
|
merge local changes
|
2026-05-12 11:18:45 +00:00 |
|
baihaochen
|
49a05a1307
|
增加mla注意力模型
|
2026-05-12 18:57:53 +08:00 |
|
baihaochen
|
bc3d137248
|
[feat] add MLA with decoupled RoPE (DeepSeek-V2 style)
- Add model_minimind_mla.py with Q/KV low-rank compression and decoupled RoPE
- Add kv_norm on kv_latent, q_compress + q_up + q_rope_proj for Q path
- Cache format: (kv_latent, k_rope) tuple, 176 floats/tok/layer (4.4x compression)
- Add --use_mla flag to all training scripts and utils
- Add benchmark_gqa_vs_mla.py script
|
2026-05-12 18:38:47 +08:00 |
|
baihaochen
|
9c23a92922
|
add Chinese annotations pretrain
|
2026-05-08 20:44:02 +08:00 |
jingyaogong
|
802c15b2b4
|
[feat] reduce RL memory
|
2026-05-06 15:07:28 +08:00 |
TKiteRunner
|
10776417aa
|
fix: resolve CUDA OOM in train_grpo.py on GPUs with <=8GB VRAM
|
2026-05-06 14:14:15 +08:00 |
jingyaogong
|
bdee223036
|
[fix] inference bug
|
2026-05-03 20:48:48 +08:00 |
|
jingyaogong
|
773e451b11
|
[fix] bugs
|
2026-04-27 19:16:08 +08:00 |
|
jingyaogong
|
6361510016
|
[fix] rollout bugs
|
2026-04-27 17:54:09 +08:00 |
|
jingyaogong
|
5416a44471
|
[fix] bugs
|
2026-04-21 13:03:34 +08:00 |
|
jingyaogong
|
e796b8028a
|
[fix] lora compile
|
2026-04-10 00:01:21 +08:00 |
|
jingyaogong
|
5351424bf0
|
[fix] lora moe
|
2026-04-09 16:36:48 +08:00 |
|
jingyaogong
|
aa3e6affa1
|
[update] add a comment
|
2026-04-09 15:09:58 +08:00 |
|
jingyaogong
|
b8b3d35257
|
[update] change default seq_len
|
2026-03-26 10:09:06 +08:00 |
|
jingyaogong
|
101d7df2da
|
[update] minimind-3
|
2026-03-25 23:57:45 +08:00 |
readlnh
|
cf4b49a348
|
[fix] align log/save last-step check and ETA with 1-indexed step
|
2026-03-24 02:01:40 +01:00 |
|
readlnh
|
d25500d363
|
[fix] gradient accumulation step alignment
|
2026-03-24 01:45:04 +01:00 |
|
jingyaogong
|
11a44340ba
|
[update] save interval
|
2026-01-30 20:30:50 +08:00 |
|
jingyaogong
|
fea69cf338
|
[fix] data skip
|
2026-01-18 16:56:29 +08:00 |
|
jingyaogong
|
f7ffdf1fdb
|
[update] shuffle data
|
2026-01-18 16:39:34 +08:00 |
|
jingyaogong
|
c090b69c4d
|
[update] align loss
|
2026-01-15 00:56:32 +08:00 |
|
jingyaogong
|
e119db8478
|
[fix] compile unpack
|
2026-01-14 20:13:32 +08:00 |
|
jingyaogong
|
81d24a4f16
|
[feat] add compile
|
2026-01-14 14:42:30 +08:00 |
|
jingyaogong
|
df89069362
|
[update] params log
|
2026-01-07 23:08:45 +08:00 |
|
jingyaogong
|
07364c3fbe
|
[update] rename train tokenizer
|
2026-01-06 01:17:33 +08:00 |
|
jingyaogong
|
4e73f34823
|
[update] rename reason
|
2026-01-05 23:12:29 +08:00 |
|
jingyaogong
|
42a4e8c86a
|
[fix] dist cleanup
|
2026-01-02 22:25:55 +08:00 |
|
jingyaogong
|
9d898576ac
|
[update] aux loss
|
2026-01-01 22:41:46 +08:00 |
|
jingyaogong
|
c65335b56f
|
[fix] experts unused
|
2025-12-31 21:47:04 +08:00 |
|
jingyaogong
|
bc8fd82166
|
[fix] layers set 8
|
2025-12-31 21:06:37 +08:00 |
|
jingyaogong
|
5dd4df7e18
|
[fix] moe unused
|
2025-12-31 21:00:06 +08:00 |
|
jingyaogong
|
9236260a4a
|
[feat] get params
|
2025-12-31 20:46:59 +08:00 |
|
jingyaogong
|
288a1d7212
|
[feat] get params
|
2025-12-31 20:44:34 +08:00 |
|
jingyaogong
|
6242980917
|
[feat] update lr
|
2025-12-31 10:27:09 +08:00 |
|
jingyaogong
|
7eae14f3ce
|
[feat] remove empty_cache
|
2025-12-27 07:14:36 +08:00 |
|
jingyaogong
|
11b962da06
|
[feat] explicit left padding
|
2025-12-23 18:59:48 +08:00 |
|
jingyaogong
|
fe24501602
|
[feat] adjust seq length
|
2025-12-14 20:41:58 +08:00 |
|
jingyaogong
|
5129f0e2a2
|
[fix] dtype & lr
|
2025-12-09 13:01:38 +08:00 |
dyhuachi
|
bf3878ace8
|
[fix] Refactor get_lr function to include min_lr calculation
这里的退火算法会让参数里的lr的起始值变成原来lr的1.1倍，作出如下修改
|
2025-12-06 17:09:51 +08:00 |
|
jingyaogong
|
5e1447b913
|
[fix] cuda memory #559
|
2025-12-01 16:17:43 +08:00 |
|
jingyaogong
|
6b86ea399a
|
[feat] release memory
|
2025-11-27 19:39:49 +08:00 |
|
jingyaogong
|
d7f4f4eab8
|
[fix] ppo mask
|
2025-11-19 23:39:02 +08:00 |
|
jingyaogong
|
9c98cabc9a
|
[fix] prompt length calculate
|
2025-11-15 18:25:37 +08:00 |
|
jingyaogong
|
509d8dacf1
|
[feat] clear cache
|
2025-11-06 13:12:28 +08:00 |