baihaochen
|
bc3d137248
|
[feat] add MLA with decoupled RoPE (DeepSeek-V2 style)
- Add model_minimind_mla.py with Q/KV low-rank compression and decoupled RoPE
- Add kv_norm on kv_latent, q_compress + q_up + q_rope_proj for Q path
- Cache format: (kv_latent, k_rope) tuple, 176 floats/tok/layer (4.4x compression)
- Add --use_mla flag to all training scripts and utils
- Add benchmark_gqa_vs_mla.py script
|
2026-05-12 18:38:47 +08:00 |
jingyaogong
|
5704766352
|
[update] tie embedding
|
2026-04-19 21:57:28 +08:00 |
|
jingyaogong
|
5351424bf0
|
[fix] lora moe
|
2026-04-09 16:36:48 +08:00 |
|
jingyaogong
|
b7e0ae21d6
|
[update] default model
|
2026-03-31 13:40:16 +08:00 |
|
jingyaogong
|
101d7df2da
|
[update] minimind-3
|
2026-03-25 23:57:45 +08:00 |
|
jingyaogong
|
04616c41a5
|
[update] safe half
|
2026-01-30 20:29:31 +08:00 |
|
jingyaogong
|
eead9538b2
|
[feat] update config
|
2025-12-31 10:29:13 +08:00 |
|
jingyaogong
|
936d105e9b
|
[feat] compatible tokenizer
|
2025-12-31 10:26:46 +08:00 |
|
jingyaogong
|
28cc44579a
|
[feat] convert2llama
|
2025-10-23 20:22:42 +08:00 |
|
jingyaogong
|
2e118e9d3d
|
update convert script
|
2025-04-27 15:44:40 +08:00 |
|
jingyaogong
|
a62faf34bd
|
250426
|
2025-04-26 10:05:47 +08:00 |
gongjy
|
c9ca0f57dc
|
update config
|
2025-02-10 00:14:11 +08:00 |
|
gongjy
|
58e3af0359
|
add minimind2
|
2025-02-09 23:49:47 +08:00 |