minimind

Commit Graph

Author	SHA1	Message	Date
baihaochen	bc3d137248	[feat] add MLA with decoupled RoPE (DeepSeek-V2 style) - Add model_minimind_mla.py with Q/KV low-rank compression and decoupled RoPE - Add kv_norm on kv_latent, q_compress + q_up + q_rope_proj for Q path - Cache format: (kv_latent, k_rope) tuple, 176 floats/tok/layer (4.4x compression) - Add --use_mla flag to all training scripts and utils - Add benchmark_gqa_vs_mla.py script	2026-05-12 18:38:47 +08:00
jingyaogong	6b0b0c5e2f	[update] fp16 inference	2026-03-27 16:29:46 +08:00
jingyaogong	101d7df2da	[update] minimind-3	2026-03-25 23:57:45 +08:00

Author

SHA1

Message

Date

baihaochen

bc3d137248

[feat] add MLA with decoupled RoPE (DeepSeek-V2 style)

- Add model_minimind_mla.py with Q/KV low-rank compression and decoupled RoPE
- Add kv_norm on kv_latent, q_compress + q_up + q_rope_proj for Q path
- Cache format: (kv_latent, k_rope) tuple, 176 floats/tok/layer (4.4x compression)
- Add --use_mla flag to all training scripts and utils
- Add benchmark_gqa_vs_mla.py script

2026-05-12 18:38:47 +08:00

jingyaogong

6b0b0c5e2f

[update] fp16 inference

2026-03-27 16:29:46 +08:00

jingyaogong

101d7df2da

[update] minimind-3

2026-03-25 23:57:45 +08:00

3 Commits (de683eafae33d56eebb300aae78d7c3241e0b8d8)