eefa663dd4
style(model): silence missing-import type warning
...
Pyright flags reportMissingImports on a third-party import. Add a targeted ignore rather than disabling the checker project-wide.
2026-10-24 00:45:27 +08:00
1e6e909f88
[update] trim redundant comments
2026-09-20 20:33:40 +08:00
ba10c4d17b
[fix] float16 下 attention mask 的 -1e9 溢出成 -inf，导致整条序列输出 NaN
...
Attention.forward 的手写 softmax 分支（flash SDPA 关闭时走这里）：
scores = (xq @ xk.transpose(-2, -1)) / math.sqrt(self.head_dim)
...
if attention_mask is not None: scores += (1.0 - mask) * -1e9
output = ... F.softmax(scores.float(), dim=-1) ...
scores 的 dtype 跟随 xq/xk。float16 模型下它是 float16，而
torch.finfo(torch.float16).min ≈ -65504，把 -1e9 累加进去会饱和成 -inf
（不是一个很大的有限负数）。当某个 query 在因果范围内可见的 key 全被 mask
掉时（左 padding 必然出现），softmax 整行 = NaN，并在下一层通过 K/V 投影
扩散到同序列里 mask=1 的有效 token 上。
.float() 加在 softmax 那一刻已经太晚——溢出发生在它之前。
修复：把 float32 提升提前到加 mask 之前，两处共 2 行。
fp32/bf16 路径数值等价（fp32 下原来的 .float() 本就是 no-op）。
2026-09-20 11:50:28 +08:00
cc312c1cc6
[fix] unify Ctrl+C handling
2026-09-18 16:33:09 +08:00
b63a9565e3
[fix] restore top-1 router gradient #858
2026-09-18 15:28:50 +08:00
1c876435ef
[fix] keep the MoE router trainable at num_experts_per_tok=1
...
Normalising a single top-k weight by its own sum gives exactly 1.0 for every
token, so the gate leaves the graph entirely: topk_idx is a discrete index and
the weight is now a constant, which leaves the LM loss no differentiable path
to the router. It then trains on aux_loss alone, whose optimum is a uniform
distribution, and the experts never specialise.
Measured on MOEFeedForward with router_aux_loss_coef=0, so only the LM path can
contribute a gradient:
num_experts_per_tok=1, norm_topk_prob=True 7.02e-10 <- current default
num_experts_per_tok=1, norm_topk_prob=False 1.02e-02
num_experts_per_tok=2, norm_topk_prob=True 1.67e-02
Only normalise when there is more than one weight to normalise.
Note this changes the forward pass for existing num_experts_per_tok=1 MoE
checkpoints: the expert output was scaled by a constant 1.0 and is now scaled
by the gate probability.
Closes #858 .
2026-09-17 21:31:41 +05:30
dddedc6881
[fix] repetition_penalty
2026-05-07 19:08:52 +08:00
1718e9a44d
[fix] transformers-5.x
2026-04-19 23:48:54 +08:00
5704766352
[update] tie embedding
2026-04-19 21:57:28 +08:00
2ab6455d9d
[update] open causal
2026-04-02 15:28:58 +08:00
101d7df2da
[update] minimind-3
2026-03-25 23:57:45 +08:00
c090b69c4d
[update] align loss
2026-01-15 00:56:32 +08:00
1279a61681
[update] prompt prefill
2026-01-13 17:46:54 +08:00
9d898576ac
[update] aux loss
2026-01-01 22:41:46 +08:00
c65335b56f
[fix] experts unused
2025-12-31 21:47:04 +08:00
5129f0e2a2
[fix] dtype & lr
2025-12-09 13:01:38 +08:00
ecd1ae1563
[fix] reduce aux_loss_alpha
2025-12-05 23:08:29 +08:00
151fdf7e76
[feat] update yarn
2025-12-01 16:15:05 +08:00
6b86ea399a
[feat] release memory
2025-11-27 19:39:49 +08:00
f5374dc87f
[fix] model attn_mask
2025-11-19 22:26:53 +08:00
a044578d73
[fix] update model
2025-11-18 13:07:20 +08:00
7d02ce673c
fix: attn_forwad when is_causal=True assert attn_mask is None
2025-11-18 03:17:17 +00:00
4e35fb9da8
[fix] update model
2025-10-17 00:09:32 +08:00
29454c31af
fix bugs
2025-04-27 09:56:49 +08:00
a62faf34bd
250426
2025-04-26 10:05:47 +08:00
Hwe1916
jingyaogong
Linxiushen
basil-k-aji-dev
jingyaogong
yuyu5333