b63a9565e3
[fix] restore top-1 router gradient #858
2026-09-18 15:28:50 +08:00
1c876435ef
[fix] keep the MoE router trainable at num_experts_per_tok=1
...
Normalising a single top-k weight by its own sum gives exactly 1.0 for every
token, so the gate leaves the graph entirely: topk_idx is a discrete index and
the weight is now a constant, which leaves the LM loss no differentiable path
to the router. It then trains on aux_loss alone, whose optimum is a uniform
distribution, and the experts never specialise.
Measured on MOEFeedForward with router_aux_loss_coef=0, so only the LM path can
contribute a gradient:
num_experts_per_tok=1, norm_topk_prob=True 7.02e-10 <- current default
num_experts_per_tok=1, norm_topk_prob=False 1.02e-02
num_experts_per_tok=2, norm_topk_prob=True 1.67e-02
Only normalise when there is more than one weight to normalise.
Note this changes the forward pass for existing num_experts_per_tok=1 MoE
checkpoints: the expert output was scaled by a constant 1.0 and is now scaled
by the gate probability.
Closes #858 .
2026-09-17 21:31:41 +05:30
4a68da72d5
[fix] lora dims
2026-05-31 13:38:51 +08:00
dddedc6881
[fix] repetition_penalty
2026-05-07 19:08:52 +08:00
1718e9a44d
[fix] transformers-5.x
2026-04-19 23:48:54 +08:00
5704766352
[update] tie embedding
2026-04-19 21:57:28 +08:00
2ab6455d9d
[update] open causal
2026-04-02 15:28:58 +08:00
101d7df2da
[update] minimind-3
2026-03-25 23:57:45 +08:00
c090b69c4d
[update] align loss
2026-01-15 00:56:32 +08:00
e119db8478
[fix] compile unpack
2026-01-14 20:13:32 +08:00
1279a61681
[update] prompt prefill
2026-01-13 17:46:54 +08:00
9d898576ac
[update] aux loss
2026-01-01 22:41:46 +08:00
c65335b56f
[fix] experts unused
2025-12-31 21:47:04 +08:00
a9c56b20e9
[fix] lora weight
2025-12-22 21:27:29 +08:00
3a18fdd666
Fix: support loading DDP-saved LoRA weights for inference
2025-12-22 20:50:25 +08:00
5129f0e2a2
[fix] dtype & lr
2025-12-09 13:01:38 +08:00
ecd1ae1563
[fix] reduce aux_loss_alpha
2025-12-05 23:08:29 +08:00
151fdf7e76
[feat] update yarn
2025-12-01 16:15:05 +08:00
6b86ea399a
[feat] release memory
2025-11-27 19:39:49 +08:00
f5374dc87f
[fix] model attn_mask
2025-11-19 22:26:53 +08:00
a044578d73
[fix] update model
2025-11-18 13:07:20 +08:00
7d02ce673c
fix: attn_forwad when is_causal=True assert attn_mask is None
2025-11-18 03:17:17 +00:00
4014e62cdf
[fix] restore
2025-10-23 19:00:06 +08:00
557bcc018d
[fix] issue-431
2025-10-23 18:56:30 +08:00
4e35fb9da8
[fix] update model
2025-10-17 00:09:32 +08:00
5ffde04b7c
update lora
2025-04-27 15:45:06 +08:00
29454c31af
fix bugs
2025-04-27 09:56:49 +08:00
274483cb1b
250426
2025-04-26 10:07:55 +08:00
a62faf34bd
250426
2025-04-26 10:05:47 +08:00
d9453ed9a3
update moe note
2025-04-09 17:38:31 +08:00
4a7c1c49e8
update rlaif
2025-04-05 16:06:08 +08:00
9e67798397
update generate
2025-04-05 15:53:55 +08:00
399d526fbd
add hidden state
2025-04-05 14:39:56 +08:00
ed01c5d84a
update inference
2025-04-05 12:03:04 +08:00
bf81fd5f5e
rmsnorm float convert
2025-04-01 16:03:44 +08:00
e369b33265
fix chat mask bug
2025-04-01 13:44:55 +08:00
258507ff89
delete __pycache__
2025-04-01 11:51:54 +08:00
844e79148c
update generate args
2025-02-15 23:56:09 +08:00
19b388cd87
update generate args
2025-02-15 23:55:10 +08:00
5b65bc767e
update cis init
2025-02-15 20:26:34 +08:00
58e3af0359
add minimind2
2025-02-09 23:49:47 +08:00
3ff66f7221
update model
2024-10-20 15:13:58 +08:00
772834148e
update readme
2024-10-08 23:40:29 +08:00
a87f628400
update model (fix loss bug)
2024-09-29 16:58:48 +08:00
75753ea765
Update data preprocessing methods
2024-09-27 17:19:03 +08:00
a8ae342775
Update data preprocessing methods
2024-09-27 16:19:30 +08:00
6759da45c1
update model mask
2024-09-21 20:00:25 +08:00
02297df3c1
Efficient implementation of Inference KV cache
2024-09-21 00:01:05 +08:00
9093519c37
Updated some explanations
2024-09-20 17:07:51 +08:00
ee218402cd
update some explain of the code
2024-09-20 17:04:16 +08:00
jingyaogong
basil-k-aji-dev
jingyaogong
whitesword
yuyu5333
gongjy