baihaochen
|
eaf3a8cbec
|
feat: add pretrain evaluation scripts
|
2026-06-07 19:46:48 +08:00 |
|
baihaochen
|
bc3d137248
|
[feat] add MLA with decoupled RoPE (DeepSeek-V2 style)
- Add model_minimind_mla.py with Q/KV low-rank compression and decoupled RoPE
- Add kv_norm on kv_latent, q_compress + q_up + q_rope_proj for Q path
- Cache format: (kv_latent, k_rope) tuple, 176 floats/tok/layer (4.4x compression)
- Add --use_mla flag to all training scripts and utils
- Add benchmark_gqa_vs_mla.py script
|
2026-05-12 18:38:47 +08:00 |
jingyaogong
|
b1865f75c2
|
[update] random seed
|
2026-03-27 21:20:02 +08:00 |
|
jingyaogong
|
6b0b0c5e2f
|
[update] fp16 inference
|
2026-03-27 16:29:46 +08:00 |
|
jingyaogong
|
101d7df2da
|
[update] minimind-3
|
2026-03-25 23:57:45 +08:00 |
|
jingyaogong
|
05d0b216f6
|
[update] show speed
|
2026-01-07 23:33:47 +08:00 |
|
jingyaogong
|
288a1d7212
|
[feat] get params
|
2025-12-31 20:44:34 +08:00 |
|
jingyaogong
|
08ce3da228
|
[feat] update args
|
2025-10-30 10:48:31 +08:00 |
|
jingyaogong
|
bf123b585d
|
[feat] add args
|
2025-10-30 10:05:12 +08:00 |
|
jingyaogong
|
35bed8936d
|
[feat] update eval-llm
|
2025-10-26 18:52:01 +08:00 |
|
jingyaogong
|
4e35fb9da8
|
[fix] update model
|
2025-10-17 00:09:32 +08:00 |
|
jingyaogong
|
36159fb2ab
|
update app
|
2025-10-16 23:33:35 +08:00 |