From d9ad86500f72fc3f6569814497d8bc7df4970577 Mon Sep 17 00:00:00 2001
From: mingkun <mingkun502@gmail.com>
Date: Wed, 26 Feb 2025 09:10:23 +0800
Subject: [PATCH] fix bug and typo

---
 notebook/1_model.ipynb | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/notebook/1_model.ipynb b/notebook/1_model.ipynb
index c262827..3cf0faf 100644
--- a/notebook/1_model.ipynb
+++ b/notebook/1_model.ipynb
@@ -102,7 +102,7 @@
     "\n",
     "旋转位置编码是一种能将相对位置信息集成到 self-attention 中, 进而提升 transformer 架构性能的位置编码方式, 和绝对位置编码相比, RoPE 具有很好的外推性, 是目前的主流位置编码方式.\n",
     "\n",
-    "外推性的解释, 通俗来说就是训练的时候限制了 512 的上下文长度，那么推d理时如果面对超过该长度的文本，LLM 可能无法正确处理.\n",
+    "外推性的解释, 通俗来说就是训练的时候限制了 512 的上下文长度，那么推理时如果面对超过该长度的文本，LLM 可能无法正确处理.\n",
     "\n",
     "- **绝对位置编码**\n",
     "\n",
@@ -281,14 +281,14 @@
     "\n",
     "   attn_11 = softmax(Q1 * K1.T / sqrt(dim)) * V1, attn_12 = 0 (masked)\n",
     "   \n",
-    "   attn_21 = softmax(Q2 * K1.T / sqrt(dim)) * V2, attn_22 = softmax(Q2 * K2.T / sqrt(dim)) * V2\n",
+    "   attn_21 = softmax(Q2 * K1.T / sqrt(dim)) * V1, attn_22 = softmax(Q2 * K2.T / sqrt(dim)) * V2\n",
     "4. *seq = \\[tok1, tok2, tok3]:*\n",
     "\n",
     "   attn_11 = softmax(Q1 * K1.T / sqrt(dim)) * V1, attn_12 = 0 (masked), attn_13 = 0 (masked)\n",
     "   \n",
-    "   attn_21 = softmax(Q2 * K1.T / sqrt(dim)) * V2, attn_22 = softmax(Q2 * K2.T / sqrt(dim)) * V2, attn_23 = 0 (masked)\n",
+    "   attn_21 = softmax(Q2 * K1.T / sqrt(dim)) * V1, attn_22 = softmax(Q2 * K2.T / sqrt(dim)) * V2, attn_23 = 0 (masked)\n",
     "   \n",
-    "   attn_31 = softmax(Q3 * K1.T / sqrt(dim)) * V3, attn_32 = softmax(Q3 * K2.T / sqrt(dim)) * V3, attn_33 = softmax(Q3 * K3.T / sqrt(dim)) * V3\n",
+    "   attn_31 = softmax(Q3 * K1.T / sqrt(dim)) * V1, attn_32 = softmax(Q3 * K2.T / sqrt(dim)) * V2, attn_33 = softmax(Q3 * K3.T / sqrt(dim)) * V3\n",
     "5.  ··· ···\n",
     "\n",
     "不难发现，大模型生成一个 token 后的注意力计算中，总会用到 token 序列的历史 KV 值，导致重复计算，KV Cache 的设计正是为了通过缓存历史 KV 值，节省计算开销.\n",
@@ -861,7 +861,7 @@
     "\n",
     "- **Step 1**：在整个批次中对每个专家的路由值进行求和.\n",
     "\n",
-    "   Importance_per_token = *Prob* (What) + *Prob* (is) + *Prob* (Mixture) + *Prob* (of) + *Prob* (Experts)\n",
+    "   $$Importance \\, per \\, token = Prob (What) + Prob (is) + Prob (Mixture) + Prob (of) + Prob (Experts)$$\n",
     "\n",
     "   这个指标反映了 batch 维度上每个专家的重要性分数.\n",
     "\n",