baihaochen
|
5012b232ff
|
fix: harden DeepSpeed SFT resume
|
2026-06-11 21:45:31 +08:00 |
|
baihaochen
|
eaf3a8cbec
|
feat: add pretrain evaluation scripts
|
2026-06-07 19:46:48 +08:00 |
|
baihaochen
|
4dcda45bfa
|
fix: recover DeepSpeed resume step from checkpoint tag
|
2026-06-04 14:20:21 +08:00 |
|
baihaochen
|
e9c93951cc
|
fix: keep only latest DeepSpeed checkpoint
|
2026-06-03 21:35:47 +08:00 |
|
baihaochen
|
0425c4f622
|
fix: make DeepSpeed resume checkpoints robust
|
2026-06-02 20:41:38 +08:00 |
|
baihaochen
|
33233fed6e
|
feat: add SearchShortQA DeepSpeed pipeline
|
2026-06-02 00:59:46 +08:00 |
|
baihaochen
|
bc3d137248
|
[feat] add MLA with decoupled RoPE (DeepSeek-V2 style)
- Add model_minimind_mla.py with Q/KV low-rank compression and decoupled RoPE
- Add kv_norm on kv_latent, q_compress + q_up + q_rope_proj for Q path
- Cache format: (kv_latent, k_rope) tuple, 176 floats/tok/layer (4.4x compression)
- Add --use_mla flag to all training scripts and utils
- Add benchmark_gqa_vs_mla.py script
|
2026-05-12 18:38:47 +08:00 |
|
baihaochen
|
9c23a92922
|
add Chinese annotations pretrain
|
2026-05-08 20:44:02 +08:00 |
jingyaogong
|
101d7df2da
|
[update] minimind-3
|
2026-03-25 23:57:45 +08:00 |
|
jingyaogong
|
e119db8478
|
[fix] compile unpack
|
2026-01-14 20:13:32 +08:00 |
|
jingyaogong
|
df89069362
|
[update] params log
|
2026-01-07 23:08:45 +08:00 |
|
jingyaogong
|
9236260a4a
|
[feat] get params
|
2025-12-31 20:46:59 +08:00 |
|
jingyaogong
|
288a1d7212
|
[feat] get params
|
2025-12-31 20:44:34 +08:00 |
|
jingyaogong
|
7eae14f3ce
|
[feat] remove empty_cache
|
2025-12-27 07:14:36 +08:00 |
|
jingyaogong
|
5129f0e2a2
|
[fix] dtype & lr
|
2025-12-09 13:01:38 +08:00 |
dyhuachi
|
bf3878ace8
|
[fix] Refactor get_lr function to include min_lr calculation
这里的退火算法会让参数里的lr的起始值变成原来lr的1.1倍，作出如下修改
|
2025-12-06 17:09:51 +08:00 |
|
jingyaogong
|
5e1447b913
|
[fix] cuda memory #559
|
2025-12-01 16:17:43 +08:00 |
|
jingyaogong
|
0323815729
|
[feat] update import
|
2025-10-31 23:45:55 +08:00 |
|
jingyaogong
|
e8484874f5
|
[feat] pause-training
|
2025-10-26 18:49:52 +08:00 |