docs: clarify how to read the GRPO policy loss curve

The GRPO section described Policy Loss as 'decreases steadily overall' next to the
reward trend, which reads like a supervised loss where a lower value means a
better model. GRPO's policy term is a TRPO-like surrogate whose advantage term is
renormalized within each group every iteration, so its absolute level mainly
tracks how far the policy has drifted from the sampling distribution.

Keep the reward observation and the single-network vs. PPO comparison, and state
that convergence should be judged from reward and generation quality instead.
Same change in README.md and README_en.md.

Refs #884

README.md

@@ -1153,7 +1153,9 @@ python train_grpo.py
 
 > MiniMind 在 GRPO 训练阶段的优化走势
 
-从训练曲线可以看出，GRPO的**reward呈现更加稳定的上升趋势**，达到4左右，说明GRPO本身能更好地利用RLAIF信号。Policy Loss整体下降平稳，相比PPO的双网络优化，GRPO单网络架构训练更稳定且收敛上限更高。
+从训练曲线可以看出，GRPO的**reward呈现更加稳定的上升趋势**，达到4左右，说明GRPO本身能更好地利用RLAIF信号。相比PPO的双网络优化，GRPO单网络架构训练更稳定且收敛上限更高。
+
+需要说明的是，Policy Loss 并不是监督学习里那种“越低越好”的 loss：它是按一阶导等于策略梯度构造的代理目标（TRPO like），并且每一轮都会做组内优势归一化，因此它的绝对值主要取决于当前策略与采样分布的偏离程度，“整体下降平稳”并不等于模型在变好。判断 GRPO 是否收敛，应该看 reward 和实际生成质量。
 
 #### 7.3 [Clipped Importance Sampling Policy Optimization](https://huggingface.co/papers/2506.13585)
 

README_en.md

@@ -1153,7 +1153,9 @@ python train_grpo.py
 
 > MiniMind optimization trends during the GRPO training stage
 
-From the training curves, it can be seen that GRPO's **reward shows a more stable upward trend**, reaching around 4, indicating that GRPO itself can better utilize RLAIF signals. Policy Loss decreases steadily overall, and compared to PPO's dual-network optimization, GRPO's single-network architecture trains more stably with a higher convergence ceiling.
+From the training curves, it can be seen that GRPO's **reward shows a more stable upward trend**, reaching around 4, indicating that GRPO itself can better utilize RLAIF signals. Compared to PPO's dual-network optimization, GRPO's single-network architecture trains more stably with a higher convergence ceiling.
+
+Worth noting: Policy Loss is not a supervised loss where "lower is better". It is a surrogate objective constructed so that its first-order derivative equals the policy gradient (TRPO like), and advantages are normalized within each group on every iteration. Its absolute value therefore mainly reflects how far the current policy has drifted from the sampling distribution, so a "steady overall decline" does not mean the model is getting better. Whether GRPO is converging should be judged from the reward and the actual generation quality.
 
 #### 7.3 [Clipped Importance Sampling Policy Optimization](https://huggingface.co/papers/2506.13585)