baihaochen
|
4f5cdfe59a
|
fix: avoid double init_process_group in DeepSpeed pretrain
|
2026-05-23 16:21:57 +08:00 |
|
baihaochen
|
ed006a254f
|
[feat] add DeepSpeed pretrain (ZeRO-2, BF16, auto LR, gradient accumulation)
|
2026-05-22 00:27:06 +08:00 |
|
baihaochen
|
579cc79cd8
|
fix: extract tools and tool_calls before apply_chat_template in PPL eval
|
2026-05-16 11:33:55 +08:00 |
|
baihaochen
|
15645bbf56
|
[feat] add GQA vs MLA evaluation script (PPL + Q&A)
|
2026-05-16 11:27:49 +08:00 |
|
baihaochen
|
09e491f940
|
Merge branch 'master' of github.com:daiweidaichenzi/minimind
|
2026-05-14 14:29:06 +08:00 |
|
baihaochen
|
8baa9c5f71
|
fix: add missing --use_mla to train_full_sft.py
|
2026-05-14 14:28:11 +08:00 |
rainpeach-m
|
de683eafae
|
merge local changes
|
2026-05-12 11:18:45 +00:00 |
|
baihaochen
|
49a05a1307
|
增加mla注意力模型
|
2026-05-12 18:57:53 +08:00 |
|
baihaochen
|
bc3d137248
|
[feat] add MLA with decoupled RoPE (DeepSeek-V2 style)
- Add model_minimind_mla.py with Q/KV low-rank compression and decoupled RoPE
- Add kv_norm on kv_latent, q_compress + q_up + q_rope_proj for Q path
- Cache format: (kv_latent, k_rope) tuple, 176 floats/tok/layer (4.4x compression)
- Add --use_mla flag to all training scripts and utils
- Add benchmark_gqa_vs_mla.py script
|
2026-05-12 18:38:47 +08:00 |
|
baihaochen
|
9c23a92922
|
add Chinese annotations pretrain
|
2026-05-08 20:44:02 +08:00 |
jingyaogong
|
dddedc6881
|
[fix] repetition_penalty
|
2026-05-07 19:08:52 +08:00 |
|
jingyaogong
|
802c15b2b4
|
[feat] reduce RL memory
|
2026-05-06 15:07:28 +08:00 |
|
jingyaogong
|
e73a407f7a
|
Merge pull request #759 from TKiteRunner/fix/grpo-oom-reorder-rewards
fix: resolve CUDA OOM in train_grpo.py on GPUs with <=8GB VRAM
|
2026-05-06 15:00:17 +08:00 |
TKiteRunner
|
10776417aa
|
fix: resolve CUDA OOM in train_grpo.py on GPUs with <=8GB VRAM
|
2026-05-06 14:14:15 +08:00 |
|
jingyaogong
|
5020dc9dd4
|
[update] readme
|
2026-05-06 13:41:46 +08:00 |
jingyaogong
|
bdee223036
|
[fix] inference bug
|
2026-05-03 20:48:48 +08:00 |
|
jingyaogong
|
06d882e4ef
|
[update] readme
|
2026-05-01 12:06:47 +08:00 |
|
jingyaogong
|
da865af63d
|
[update] readme
|
2026-04-28 17:22:25 +08:00 |
|
jingyaogong
|
773e451b11
|
[fix] bugs
|
2026-04-27 19:16:08 +08:00 |
|
jingyaogong
|
6361510016
|
[fix] rollout bugs
|
2026-04-27 17:54:09 +08:00 |
|
jingyaogong
|
d4c6bc5c7e
|
[update] readme
|
2026-04-27 10:59:41 +08:00 |
|
jingyaogong
|
24896cd2c4
|
[update] readme
|
2026-04-24 20:16:06 +08:00 |
|
jingyaogong
|
e2fc397176
|
[update] readme
|
2026-04-24 20:14:10 +08:00 |
|
jingyaogong
|
693fb1ccf1
|
[update] readme
|
2026-04-21 14:34:46 +08:00 |
|
jingyaogong
|
5416a44471
|
[fix] bugs
|
2026-04-21 13:03:34 +08:00 |
|
jingyaogong
|
1718e9a44d
|
[fix] transformers-5.x
|
2026-04-19 23:48:54 +08:00 |
|
jingyaogong
|
5704766352
|
[update] tie embedding
|
2026-04-19 21:57:28 +08:00 |
|
jingyaogong
|
1ea113ea2c
|
[update] readme
|
2026-04-19 14:51:47 +08:00 |
|
jingyaogong
|
487f78754d
|
[update] readme
|
2026-04-10 10:55:24 +08:00 |
|
jingyaogong
|
e796b8028a
|
[fix] lora compile
|
2026-04-10 00:01:21 +08:00 |
|
jingyaogong
|
b2488e6440
|
[update] readme
|
2026-04-09 19:00:41 +08:00 |
|
jingyaogong
|
939dc8ff42
|
[update] readme
|
2026-04-09 18:54:17 +08:00 |
|
jingyaogong
|
cadacabecb
|
[update] bench
|
2026-04-09 17:12:48 +08:00 |
|
jingyaogong
|
5351424bf0
|
[fix] lora moe
|
2026-04-09 16:36:48 +08:00 |
|
jingyaogong
|
aa3e6affa1
|
[update] add a comment
|
2026-04-09 15:09:58 +08:00 |
|
jingyaogong
|
25a7edcd6f
|
[update] image
|
2026-04-04 14:54:13 +08:00 |
|
jingyaogong
|
cacf1d4cd0
|
[update] readme
|
2026-04-04 11:25:21 +08:00 |
|
jingyaogong
|
2ab6455d9d
|
[update] open causal
|
2026-04-02 15:28:58 +08:00 |
|
jingyaogong
|
9348fde743
|
[update] readme
|
2026-04-02 15:28:29 +08:00 |
|
jingyaogong
|
90cd275524
|
[update] readme
|
2026-04-01 14:00:21 +08:00 |
|
jingyaogong
|
b7e0ae21d6
|
[update] default model
|
2026-03-31 13:40:16 +08:00 |
|
jingyaogong
|
b1865f75c2
|
[update] random seed
|
2026-03-27 21:20:02 +08:00 |
|
jingyaogong
|
6b0b0c5e2f
|
[update] fp16 inference
|
2026-03-27 16:29:46 +08:00 |
|
jingyaogong
|
88e675dc2c
|
[update] image
|
2026-03-26 15:35:42 +08:00 |
|
jingyaogong
|
b8b3d35257
|
[update] change default seq_len
|
2026-03-26 10:09:06 +08:00 |
|
jingyaogong
|
101d7df2da
|
[update] minimind-3
|
2026-03-25 23:57:45 +08:00 |
jingyaogong
|
83e52f6a27
|
Merge pull request #698 from readlnh/master
[fix] 修复训练脚本中 1-indexed step 与 0-indexed 逻辑混用的问题
|
2026-03-24 13:41:20 +08:00 |
readlnh
|
cf4b49a348
|
[fix] align log/save last-step check and ETA with 1-indexed step
|
2026-03-24 02:01:40 +01:00 |
|
readlnh
|
d25500d363
|
[fix] gradient accumulation step alignment
|
2026-03-24 01:45:04 +01:00 |
|
jingyaogong
|
349e74ec7b
|
[update] empty_think_ratio
|
2026-02-06 19:15:21 +08:00 |