[update] minimind-3v

README.md

@@ -33,7 +33,7 @@
 
 </div>
 
-* 此项目旨在从0开始，仅用3块钱成本 + 2小时！即可训练出67M参数的超小多模态视觉语言模型**MiniMind-V**。
+* 此项目旨在从0开始，仅用3块钱成本 + 2小时！即可训练出65M参数的超小多模态视觉语言模型**MiniMind-V**。
 * **MiniMind-V**最小版本体积仅为 GPT3 的约 $\frac{1}{2600}$，力求做到个人GPU也可快速推理甚至训练。
 * **MiniMind-V**是[MiniMind](https://github.com/jingyaogong/minimind)纯语言模型的视觉能力额外拓展。
 * 项目同时包含了VLM大模型的极简结构、数据集清洗、Pretrain、SFT等全过程代码。
@@ -59,12 +59,12 @@
 训练过程究竟难不难？那么现在，探索它们的答案，一起感受创造的乐趣吧！
 
 > [!TIP]
-> （截至2026-04-20）MiniMind-V 系列已完成了以下型号模型训练，最小仅需67M (0.067B)，即可具备识图和对话的能力！
+> （截至2026-04-20）MiniMind-V 系列已完成了以下型号模型训练，最小仅需65M (0.065B)，即可具备识图和对话的能力！
 
 | 模型 (大小) | release |
 |---|---|
-| minimind-3v-moe (201M-A67M) | 2026.04.20 |
+| minimind-3v-moe (200M-A65M) | 2026.04.20 |
-| minimind-3v (67M) | 2026.04.20 |
+| minimind-3v (65M) | 2026.04.20 |
 | MiniMind2-V (104M) | 2025.02.20 |
 | MiniMind2-Small-V (26M) | 2025.02.20 |
 | minimind-v-v1-small (27M) | 2024.10.04 |
@@ -75,9 +75,9 @@
 <details>
 <summary> <b>2026-04-20</b> </summary>
 
-- 更新模型检查点：minimind-3v (67M) / minimind-3v-moe (201M-A67M)
+- 更新模型检查点：minimind-3v (65M) / minimind-3v-moe (200M-A65M)
-- Projector 更新：添加 `LayerNorm`，token 合并改为 2D pixel-shuffle
+- Projector 更新：添加 `LayerNorm`，去掉 reshape token 合并（P32 原生 64 token，无需下采样）
-- Vision Encoder 换为 `SiglipVisionModel`（固定 256×256）
+- Vision Encoder 换为 `SiglipVisionModel`（P32，固定 256×256）
 - 训练数据切到 ALLaVA-4V（Pretrain 127 万 / SFT 290 万，已合并为单阶段 SFT）
 - Freeze 策略更新：`freeze_llm=1` 解冻首末两层，Pretrain/SFT 默认改为 `2`/`1`；`max_seq_len` 360 → 450
 - 其他 bugfix 与小调整
@@ -155,9 +155,9 @@ git clone --depth 1 https://github.com/jingyaogong/minimind-v
 
 ```bash
 # 下载siglip2模型到 ./model 目录下
-git clone https://huggingface.co/jingyaogong/siglip2-base-p16-256-ve
+git clone https://huggingface.co/jingyaogong/siglip2-base-p32-256-ve
 # 或
-git clone https://modelscope.cn/models/gongjy/siglip2-base-p16-256-ve
+git clone https://modelscope.cn/models/gongjy/siglip2-base-p32-256-ve
 ```
 
 ```bash
@@ -351,7 +351,7 @@ python train_xxx.py --use_wandb
 
 MiniMind-V 的语言主干即孪生项目 [minimind](https://github.com/jingyaogong/minimind) 训练得到的 `llm_768.pth`，LLM 本身的结构、训练细节与实验分析不在本仓库重复，默认读者对 MiniMind LLM 已有基本了解。未接触过也不影响照着“快速开始”跑通 MiniMind-V，流程自洽。
 
-顶部 “从 0 训练” 和 “67M” 两个简化口号在这里也需要说明。“从 0 训练” 指 VLM 本身从零构建（Projection 随机初始化、LLM 首末层微调对齐），但 LLM 主干并非从零 pretrain，而是基于 MiniMind 的语言模型权重续训而来；若要严格意义上的 “完全从零 pretrain”，请先在 MiniMind 训一版 LLM 再迁回本项目。“67M” 指可训练部分的主干规模（LLM ~64M + Projection ~3M）；视觉编码器 SigLIP2 另有 ~93M 参数全程冻结、仅作图像特征提取，因此推理时整机参数量约 160M（dense）/ 294M（MoE）。
+顶部 “从 0 训练” 和 “65M” 两个简化口号在这里也需要说明。“从 0 训练” 指 VLM 本身从零构建（Projection 随机初始化、LLM 首末层微调对齐），但 LLM 主干并非从零 pretrain，而是基于 MiniMind 的语言模型权重续训而来；若要严格意义上的 “完全从零 pretrain”，请先在 MiniMind 训一版 LLM 再迁回本项目。“65M” 指可训练部分的主干规模（LLM ~64M + Projection ~1M）；视觉编码器 SigLIP2 另有 ~95M 参数全程冻结、仅作图像特征提取，因此推理时整机参数量约 160M（dense）/ 294M（MoE）。
 
 VLM 在 LLM 基础上增加 Visual Encoder 和特征投影两个子模块，引入模态混合分支以支持多模态输入：
 ![LLM-structure](./images/VLM-structure.jpg)
@@ -401,14 +401,14 @@ GPT模型根据现有token预测输出下一个下下一个下下下一个token
 
 "外语词典" 称之为Visual Encoder模型。
 和LlaVA、Qwen-VL等视觉语言模型类似，MiniMind-V当前选用开源SigLIP2系列模型作为Visual Encoder。
-具体使用[siglip2-base-p16-256-ve](https://huggingface.co/jingyaogong/siglip2-base-p16-256-ve)，
+具体使用[siglip2-base-p32-256-ve](https://huggingface.co/jingyaogong/siglip2-base-p32-256-ve)，
-一种基于 ViT-B/16 架构的Visual Encoder用于描述图像文本信息。
+一种基于 ViT-B/32 架构的Visual Encoder用于描述图像文本信息。
-当前使用的 SigLIP2 NaFlex 视觉编码器会根据预处理结果生成最多256个patch token作为encoder编码层的输入，
+当前使用的 SigLIP2 NaFlex 视觉编码器会根据预处理结果生成 64 个 patch token（256×256 图像 / patch_size 32 = 8×8 = 64）作为 encoder 编码层的输入，
 最终产生1×768维的嵌入向量用于和文本对计算误差。
 我们并不需要最终嵌入表示，因此只取encoder层的输出，也就是VIT核心主干的输出特征即可。
-它拿到前一层 256×768 大小的特征，通过 reshape 将每 4 个相邻 token 拼接为 1 个（256×768 → 64×3072），再经过 2 层 MLP（Linear→GELU→Linear）投影到 LLM 的隐藏维度，最终作为 64 个 visual token 输入 MiniMind-V——这一步完成的就是跨模态特征对齐：让原生视觉特征进入文本 token 所在的语义空间，使两者可以在同一空间里交互。
+它拿到前一层 64×768 大小的特征，经过 LayerNorm + 2 层 MLP（Linear→GELU→Linear）投影到 LLM 的隐藏维度，最终作为 64 个 visual token 输入 MiniMind-V——这一步完成的就是跨模态特征对齐：让原生视觉特征进入文本 token 所在的语义空间，使两者可以在同一空间里交互。
 
-[LlaVA-1](https://arxiv.org/pdf/2304.08485)使用简单的线性变换完成对齐，[LlaVA-1.5](https://arxiv.org/pdf/2310.03744)升级为2层MLP，MiniMind-V采用与LlaVA-1.5相同的MLP Projection方案，并结合reshape进行token压缩。
+[LlaVA-1](https://arxiv.org/pdf/2304.08485)使用简单的线性变换完成对齐，[LlaVA-1.5](https://arxiv.org/pdf/2310.03744)升级为2层MLP，MiniMind-V采用与LlaVA-1.5相同的MLP Projection方案（P32 原生输出 64 token，无需额外 reshape 压缩）。
 
 ![llava-structure](./images/llava-structure.png)
 
@@ -429,7 +429,7 @@ VLM的输入依然是一段文本，其中包含特殊的`<image>`占位符。
 <image>\n这个图像中有什么内容？
 ```
 
-在`minimind-v`中，使用64个`<|image_pad|>`组成的占位符代替图像（SigLIP2输出的256个patch特征经reshape+MLP压缩为64个token），因此`minimind-v`的prompt为：
+在`minimind-v`中，使用64个`<|image_pad|>`组成的占位符代替图像（SigLIP2 P32 直接输出 64 个 patch token，经 MLP 投影为 64 个 visual token），因此`minimind-v`的prompt为：
 
 ```text
 <|image_pad|><|image_pad|>...<|image_pad|>(×64)\n这个图片描述的是什么内容？
@@ -464,9 +464,9 @@ VLM的输入依然是一段文本，其中包含特殊的`<image>`占位符。
   - 任务形式混合了"围绕图片的推理式问答"、"caption 长描述"和"纯文本对话"，既考验细节追问/长链条推理，也兼顾图像描述与通用语言能力。
 
 合计约 290 万条样本，Pretrain 阶段可直接跳过（SFT 已把 Pretrain 作为子集吸收）。中英比例大致均衡。
-考虑到 MiniMind-V 的语言主干仅 67M，把英文和中文一起喂进去是比较稳妥的做法——中文语料对母语输出更友好，而英文原生描述通常更精细，两者互为补充。
+考虑到 MiniMind-V 的语言主干仅 65M，把英文和中文一起喂进去是比较稳妥的做法——中文语料对母语输出更友好，而英文原生描述通常更精细，两者互为补充。
 
-图像统一 `resize` 到 **256×256**（与 SigLIP2 NaFlex 编码器的 256 patch token 输入规格相对应），重新编码为 JPEG 打包进 parquet。
+图像统一 `resize` 到 **256×256**（与 SigLIP2 NaFlex 编码器的输入规格相对应，P32 产生 64 个 patch token），重新编码为 JPEG 打包进 parquet。
 
 (`pretrain_i2t.parquet`) Pretrain 数据集格式：
 
@@ -512,7 +512,7 @@ Pretrain 阶段从约 127 万条图文描述数据中学习图片的通用知识
 
 > train_sft_vlm
 
-SFT 阶段的数据即前述 `sft_i2t.parquet`，约 290 万条混合样本，涵盖 Pretrain 继承而来的图文 caption、自然图像上的推理问答、文档/图表的细节问答、Gemini/Claude/GPT-4o 合成的指令，以及约 23 万条纯文本对话（图像列填 8×8 黑图占位）。学习率降到 ~5e-5，最大序列 768。
+SFT 阶段的数据即前述 `sft_i2t.parquet`，约 290 万条混合样本，涵盖 Pretrain 继承而来的图文 caption、自然图像上的推理问答、文档/图表的细节问答、Gemini/Claude/GPT-4o 合成的指令，以及约 23 万条纯文本对话（图像列填 8×8 黑图占位）。学习率降到 ~5e-6，最大序列 768。
 
 常见做法是在 SFT 阶段把 LLM 全参解冻，但这通常建立在底座有几 B 参数、且 SFT 数据中混入大量纯文本这两个前提之上。MiniMind-V 的语言主干仅 64M，而当前 SFT 数据中 ~92% 与图像有关，若全参解冻，LLM 原有的通用语言能力极易被图文任务的梯度稀释。
 
@@ -557,57 +557,57 @@ Prompt: `<image>\n请描述这张图中的主要物体和场景。`
         <img src="./dataset/eval_images/image-01-golden-dog-balloons.jpg" alt="golden-dog">
         &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
       </td>
-      <td>这张图片中的主要物体是一只金毛寻回犬，以其金色的皮毛和友好的表情而闻名。金毛寻回犬是这幅图片中的主要对象，以其金色的皮毛和友好的表情而闻名。它们是图片的主要焦点，并在这幅图片中展示了这个狗的特征。</td>
+      <td>这张图中的主要物体是一只金毛寻回犬，以其金色的外套和友好的表情而闻名。金毛寻回犬以活泼的表情和友好而闻名，它们正在进行充满活力和活跃的互动。它的姿势和表情传达了一种快乐和积极的情绪。这只金毛寻回犬被描述为一个活泼的、充满活力的、充满活力的狗，在阳光明媚的户外环境中展现出来。它的眼睛是友好的，嘴巴是闭着的，这表明它在享受阳光明媚的日子里感到快乐和满足。在背景中，可以看到一片草地和晴朗的天空，暗示着这是一个宁静的户外环境。背景中的天空是淡蓝色的，有几朵零星的云，暗示着一个晴朗的日子。总体而言，这只金毛寻回犬的外表和活动充满了快乐和积极的情绪，使其成为一个充满活力和快乐的快乐时刻。</td>
-      <td>这张图片展示了一只金毛寻回犬在草地上享受一群彩色气球，很可能是在玩耍或散步的时候。这只狗的皮毛呈金黄色，尾巴尖端略带波浪状。这只狗的耳朵尖尖，尾巴垂下垂，表明它正在向前看，或者可能正在进行某种形式的玩耍或互动。背景是一片广阔的开阔景观，天空是晴朗的蓝色，零星分布着几朵云。在画面的左侧，有一条黄色和粉色的条纹，它们挂在草地上，为画面增添了色彩和质感。</td>
+      <td>这张图片展示了一只金毛寻回犬在草地上享受快乐。狗狗的毛色是丰富的棕色，带有白色和黄色的斑纹，它的耳朵竖立，嘴巴张开，看起来很快乐。它戴着一条项圈，表明它是一只宠物。背景是晴朗的蓝天，零星散落的云彩，表明照片是在一个阳光明媚的日子拍摄的。在狗狗的背后，有一片草地和几朵零星的云朵，表明这是一个宁静的自然环境。</td>
     </tr>
     <tr>
       <td>
         <img src="./dataset/eval_images/image-02-rainbow-umbrella-street.jpg" alt="rainbow-umbrella">
         &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
       </td>
-      <td>这张图片展示了一只雨伞在城市街道上停放的场景。雨伞主要是鲜艳的颜色，在黑色背景上形成鲜明的对比。它被一条多彩的线连接在一起，可能表明天气状况良好。雨伞的闪闪发光的外套增添了一丝动感，为场景增添了一种动感。雨伞的反射和反射暗示着它可能是为了街道或停车场的静止而被使用。它被放置在一条铺设的道路上，表明这个场景是在城市街道上拍摄的，可能是在一个城市公园或公寓区。</td>
+      <td>这张图中的主要物体是一把雨伞，根据其颜色和光泽，它是雨伞的特征之一。雨伞设计用于防止雨水进入，同时保持其透明度和可见性。它主要是绿色、红色、黄色和蓝色，还有一些白色的装饰，这在雨伞的背景下很常见。雨伞的反射和光线暗示着它可能是在一个有阳光的户外环境中拍摄的。雨伞是雨水的典型特征，表明它可能是雨天或湿润的天气条件。</td>
-      <td>这张图中的主要物体是雨伞，它在雨天被雨水滋润。雨伞是主要的物体，通过其设计和大小来识别，使其成为图片的焦点。雨伞的颜色鲜艳，有明亮的黄色、橙色、红色和蓝色，这些颜色通常与雨水相关联。雨伞的反光表面反射着光线，表明它是有光泽的，这在雨天是很常见的。雨伞的反射和颜色表明它可能是由雨水滴或雨水渗入地面，这在雨天是常见的。雨伞的设计，以其多彩的图案和颜色，与雨伞的鲜艳颜色相辅相成，使其成为图片的焦点。</td>
+      <td>这张图片展示了一把雨伞，放在湿润的表面上，可能是一条街道上。这把雨伞有着醒目的彩虹色图案，颜色有各种蓝色、红色、黄色和紫色，形成了一种彩虹般的效果。雨伞的边缘略微弯曲，表明它要么是从雨滴中滴落，要么是从一个有反射表面的表面上滴落。雨伞本身是黑色的，有光泽，表明它已经湿润了。背景中可以看到一个城市环境，有建筑物和街道，天空多云，暗示着阴天或阴凉的一天。场景中看不到人，暗示着静止或有静止的迹象。</td>
     </tr>
     <tr>
       <td>
         <img src="./dataset/eval_images/image-03-cherry-blossom-bike.jpg" alt="cherry-blossom-bike">
         &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
       </td>
-      <td>这张图片中的主要物体是一辆自行车，具体来说是一辆青绿色的自行车。这一识别是基于几个视觉线索：骑车者的服装，头盔和轮胎的存在，以及前部的后部轮胎，这些都是自行车的特征。此外，背景中有树木和晴朗的天空，进一步强调了这一场景。图片中没有任何可见的物体或人，进一步强调了这辆自行车的特点。</td>
+      <td>这张图中的主要物体是一辆自行车，具体来说是一辆青绿色的自行车。这辆自行车是图片的中心焦点，展示了它的设计和特点。自行车的座椅是鲜艳的蓝色，带有白色座椅，座椅和座椅上点缀着粉色花朵，与青绿色的背景形成鲜明对比。自行车的前部配有黑色轮胎，胎部和车轮上还有小的装饰，可能是为了增加了视觉吸引力。背景中可见一片水域，可能是湖泊或河流，这表明照片是在一个宁静的自然环境中拍摄的，很可能是在一个水域旁边，因为远处可以看到一条蜿蜒的河流。天空阴沉，光线柔和，没有明显的阴影，这表明照片是在太阳处于较低角度的时候拍摄的。</td>
-      <td>这张图片展示了一幅宁静的户外场景，前景是一条繁华的街道，上面挂满了粉色花朵和红色装饰，散落着各种粉色花朵和白色小花。在前景，有一棵开着粉色花朵的大树，树干略显皱纹，暗示着成熟或可能是春天的场景。树后面是一片郁郁葱葱的绿树，树叶浓密，表明可能是春天或夏天。前景是一条铺满粉色花朵的道路，通向画面左侧是一条长满花朵的小径。在道路的右侧，是一条修剪整齐的道路，两旁是修剪整齐的草坪，背景是晴朗的蓝天和远处的另一棵树。整个场景传达了一种宁静和自然美的感觉。</td>
+      <td>这张图片展示了一辆停在路边的古董自行车。自行车是鲜艳的蓝色，配有白色的车架和黑色的链条。它配有黑色的车把，表明它是一辆自行车。自行车上有两个踏板，一个位于左侧，一个位于右侧，都被遮挡着。背景是一片郁郁葱葱的绿色植被，暗示着这个地点可能靠近大自然或公园。天空晴朗，暗示着良好的天气条件。自行车上有黑色的座椅，与整体的蓝色和白色调相得益彰。自行车后面是郁郁葱葱的绿树，树叶浓密，与背景的宁静和郁郁葱葱形成鲜明对比。</td>
     </tr>
     <tr>
       <td>
         <img src="./dataset/eval_images/image-04-yellow-car.jpg" alt="yellow-car">
         &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
       </td>
-      <td>这张图片中的主要物体是一辆现代高尔夫球车，具体来说是一辆在沙漠或海滩上的高尔夫球车。这辆车是主要的焦点，设计时尚，具有流线型的车身，配备了高性能的后扰流板，并配备了赛车轮胎，表明这是一辆高性能跑车。车辆的风格和设计表明它可能是一款高性能跑车，可能是在沙漠或海滩上使用。</td>
+      <td>这张图中的主要物体是一辆现代高性能跑车。它以醒目的黄色为主色调，配有黑色轮毂和银色合金轮毂。车身设计时尚，前保险杠设计时尚，配有流线型的后扰流板，并配备了高性能轮胎。车轮是黑色的，带有明显的辐条，与黄色车身形成鲜明对比。背景中是晴朗的蓝天，偶尔有几朵零星的云，暗示着一个宁静的时刻。天空大部分晴朗，偶尔有几朵云，表明天气条件良好。没有可见的品牌标志或标识，使图片显得独特而不受欢迎。</td>
-      <td>这张图片展示了一辆明亮的黄色跑车在晴朗的天空下。这辆车看起来是一辆豪华轿车，设计时尚，有着流线型的车身和明显的车顶，暗示着运动性能。它拥有高性能的车身，配有大尺寸的大众Jet和前大灯，展现出时尚的外观。这辆车的车身设计流线型，有流线型的车身线条和明显的车顶线条，暗示着高性能的车型。背景是一片晴朗的天空，有几朵白云，表明天气条件良好。在图片的右侧，可以看到一辆明亮的黄色跑车，暗示着它是为休闲或运动设计的。</td>
+      <td>这张图片展示了一辆黄色的豪华跑车，停在一条道路上。这辆车是黄色的，具有独特的设计，前大灯设计，中央的LED大灯和尾灯，呈现出光滑的黄色。这辆车停在一个平坦的道路上，远处可见大海，天空是湛蓝色，零星云朵。背景中有大海和海岸线，天空晴朗，零星云朵，表明这是一个阳光明媚的日子。没有明显的特征表明这辆车是新的或正在进行的，而非最近的。</td>
     </tr>
     <tr>
       <td>
         <img src="./dataset/eval_images/image-05-superhero-rooftop.jpg" alt="superhero">
         &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
       </td>
-      <td>这张图中的主要物体是超级英雄的战斗姿态。超级英雄通常是最具威胁性或高能量的形象，以保护和控制英雄。他穿着一套经典的超级英雄服装，包括蓝色牛仔裤和黑色靴子，暗示着他是该品牌的忠实伙伴。超级英雄通常被描绘为一个具有攻击性的角色，通常被视为英雄的特征。超级英雄的服装和姿势，以及他的姿势，表明他可能代表着英雄的身份或身份，而非任何超级英雄。</td>
+      <td>这张图中的主要物体是一位超级英雄的个人，被描绘在黄金时刻，这可以从他的姿势和服装上看出。超级英雄以超人般的姿势站立，一条腿抬起，脚踩在空中，另一条腿弯曲，仿佛在进行着一场致命的战斗。他的服装是蓝色的，带有红色的装饰，暗示着一种奔跑的姿态。他的头发是黑色的，右手拿着一个类似超级英雄的标志，左手拿着一个红色的锁。背景是一片模糊的天空，有云彩和远处的城市景观，暗示着一个城市环境。天空被从顶部的深蓝色渐变到地平线附近的暖色调，增强了日落的色彩。</td>
-      <td>这张图中的主要物体似乎是一个超级英雄角色，可能是超级英雄，考虑到其服装和装甲的风格。这个角色穿着超级英雄服装，包括红色和白色的袍子，这在超级英雄服装中很常见。他手持一块大大的红色和白色的大胆胡须，这是超级英雄服装的标志性特征。这个角色还戴着一条带有红色和白色条纹的红色头盔，这表明他可能是一名战士或领导者。背景是城市天际线，有着现代建筑，天空呈现出蓝色和橙色的色调，暗示着一个城市环境。</td>
+      <td>这张图片展示了一个穿着超级英雄服装的男性角色。他被抬起在一个巨大的、圆形的金属结构上，类似于一个巨大的、流线型的舞台，与他的服装形成鲜明对比。他的面部表情严肃而紧张，他直视着摄像机，给人一种紧张和戏剧性的印象。他的服装包括一件蓝色和红色的卫衣，与他穿的服装形成鲜明对比。他的服装配有金色的装饰，包括一件红色的衣领和一条腰带，与他的服装相得益彰。背景显示了一个城市天际线，有各种高楼大厦，暗示着一个城市环境。天空从地平线处温暖的金黄色过渡到地平线附近温暖的黄色，表明这是日落或日出。</td>
     </tr>
     <tr>
       <td>
         <img src="./dataset/eval_images/image-06-racecar-drift.jpg" alt="racecar">
         &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
       </td>
-      <td>这张图片中的主要物体是一辆红色的赛车，很可能是赛车。赛车是赛车运动的典型特征，旨在提高速度、提高燃油效率和赛道的稳定性和安全性。赛车的特征包括前部有大型前部、后部有大型后部，以及与赛道的对峙部分。赛车的颜色和设计与赛车的整体风格相符，以赛车为主。赛车的赛道是静止的，表明这是一条赛道。赛车的赛道是从赛道上移动的，没有任何可见的轨迹或赛道的边缘。</td>
+      <td>这张图中的主要物体是一辆现代的赛车，具体来说是赛车。赛车是赛车运动的典型特征，具有高速、长长的后扰流板、高翼配置和独特的尾灯设计。赛车设计用于高速操控和高速操控，在赛道上形成一道弧线，增加了赛车的动感。赛车配备了空气动力学和空气动力学，这些特点在赛道上可见，表明这辆车可能是一辆高性能赛车。背景中可以看到赛车和观众，表明这辆赛车是高性能赛车的典型，而赛车的细节和风格表明这是一辆高性能赛车。赛车和车手身上没有可见的个人物品，使其成为图片的焦点。</td>
-      <td>这张图片中的主要物体是红色的，它在赛道上被捕捉到在运动中，这可以从赛车的后视图中看出。这个动作是红色的，通过其独特的尾部设计，可以识别出赛道的结构，这是赛道上常见的特征。赛车的赛车设计表明它是为高速赛道赛事设计的，这在赛道上很常见。背景中的观众也可见，但焦点完全集中在红色的赛车上，这表明这个场景是在一个正式的赛道赛事中拍摄的。</td>
+      <td>这张图片展示了一辆赛车在一条弯曲的赛道上。这辆车是一辆红色的赛车，具有流线型的车身，侧面醒目地显示着"NEWS"字样，暗示着这是一辆专业赛车。这辆车配备了赛车头盔，表明它是为高速赛车而设计的，因为能够提供保护。车轮和车身装饰有明显的红色装饰，包括一个大大的圆形标志，可能代表车队或品牌。背景中可以看到观众，表明这是一场赛车活动。天空晴朗，有几朵云，表明天气条件良好。赛道上没有明显的交通规则或安全措施，这表明这辆车是为赛车活动设计的。</td>
     </tr>
   </tbody>
 </table>
 
 ### 效果小结：
 
-两个模型都能在大多数样本上识别出主体对象（狗、雨伞、自行车、跑车、超级英雄、赛车等），但都伴随重复表述与细节幻觉，整体处于"能看懂大意、细节不准"的阶段。
+两个模型在 6 张样本上均能正确识别主体对象（狗、雨伞、自行车、跑车、超级英雄、赛车），主体识别率均为 6/6，但仍伴随部分重复表述与细节幻觉，整体处于"能看懂大意、细节不够精准"的阶段。
 
-在这 6 张样本上，MoE 版本的整体场景描述与颜色识别略优于 Dense（例如对樱花场景、彩色雨伞、黄色跑车的把握更完整），Dense 则在表达简洁性上占优；不过在更广的评测范围内，MoE 对少数样本会出现较明显的本体级幻觉或重复生成，方差大于 Dense，因此 Dense 仍是更稳妥的默认配置。
+MoE 版本的场景描述更丰富，对背景环境（城市街道、城市天际线、日落渐变色）和物体细节（彩虹色图案、蓝红服装配色、赛车车身标识）的捕捉优于 Dense。Dense 版本在表达简洁性上占优，重复表述更少。两者的幻觉程度接近，均会在局部细节上出现不准确的描述。
 
 视觉信号对 LLM 而言相当于一门特殊的"外语"，"学外语"的上限因此受制于 LLM 自身的语言能力。底座越强，同样的图文数据喂进去收益越大；把 MiniMind-V 的主干换成几 B 量级的 LLM，细节准确度与推理连贯性的提升会相当明显。
 

README_en.md

@@ -62,12 +62,12 @@ Is the training process difficult? Now, let's explore the answers and feel the j
 
 > [!TIP]
 > (As of 2026-04-20) The MiniMind-V series has completed the training of the following model versions, with the smallest
-> requiring only 67M (0.067B) parameters, capable of both image recognition and conversation!
+> requiring only 65M (0.065B) parameters, capable of both image recognition and conversation!
 
 | Model (Size) | Release |
 |---|---|
-| minimind-3v-moe (201M-A67M) | 2026.04.20 |
+| minimind-3v-moe (200M-A65M) | 2026.04.20 |
-| minimind-3v (67M) | 2026.04.20 |
+| minimind-3v (65M) | 2026.04.20 |
 | MiniMind2-V (104M) | 2025.02.20 |
 | MiniMind2-Small-V (26M) | 2025.02.20 |
 | minimind-v-v1-small (27M) | 2024.10.04 |
@@ -78,9 +78,9 @@ Is the training process difficult? Now, let's explore the answers and feel the j
 <details>
 <summary> <b>2026-04-20</b> </summary>
 
-- New checkpoints released: minimind-3v (67M) / minimind-3v-moe (201M-A67M)
+- New checkpoints released: minimind-3v (65M) / minimind-3v-moe (200M-A65M)
-- Projector: added `LayerNorm`, token merging switched to 2D pixel-shuffle
+- Projector: added `LayerNorm`, removed reshape token merging (P32 natively outputs 64 tokens, no downsampling needed)
-- Vision Encoder switched to `SiglipVisionModel` (fixed 256×256)
+- Vision Encoder switched to `SiglipVisionModel` (P32, fixed 256×256)
 - Training data moved to ALLaVA-4V (Pretrain 1.27M / SFT 2.9M, merged into a single-stage SFT)
 - Freeze strategy updated: `freeze_llm=1` unfreezes first + last layers; Pretrain/SFT defaults now `2`/`1`; `max_seq_len` 360 → 450
 - Misc bugfixes and small tweaks
@@ -158,9 +158,9 @@ git clone --depth 1 https://github.com/jingyaogong/minimind-v
 
 ```bash
 # Download the siglip2 model to the ./model directory
-git clone https://huggingface.co/jingyaogong/siglip2-base-p16-256-ve
+git clone https://huggingface.co/jingyaogong/siglip2-base-p32-256-ve
 # or
-git clone https://modelscope.cn/models/gongjy/siglip2-base-p16-256-ve
+git clone https://modelscope.cn/models/gongjy/siglip2-base-p32-256-ve
 ```
 
 ```bash
@@ -356,7 +356,7 @@ and `wandb_run_name` parameters.
 
 The language backbone of MiniMind-V is the `llm_768.pth` trained by the sibling project [minimind](https://github.com/jingyaogong/minimind). The LLM's own structure, training details and experimental analysis are not repeated here; the default assumption is that the reader has a basic understanding of MiniMind LLM. Not having touched the LLM project does not prevent following the "Quick Start" to get MiniMind-V running — the flow is self-contained.
 
-The two shorthand labels on the landing page — "from scratch" and "67M" — also need a stricter reading here. "From scratch" means the VLM itself is trained from zero (Projection randomly initialized, first/last LLM layers fine-tuned for alignment), but the LLM backbone is not pretrained from zero — it is continued from the weights of MiniMind. For a strictly "from-zero pretraining" setup, first pretrain an LLM in MiniMind and then plug it back here. "67M" refers to the trainable backbone (LLM ~64M + Projection ~3M); the SigLIP2 vision encoder contributes another ~93M parameters that stay frozen throughout and serve only as an image feature extractor, so the full model at inference time is roughly 160M (dense) / 294M (MoE).
+The two shorthand labels on the landing page — "from scratch" and "65M" — also need a stricter reading here. "From scratch" means the VLM itself is trained from zero (Projection randomly initialized, first/last LLM layers fine-tuned for alignment), but the LLM backbone is not pretrained from zero — it is continued from the weights of MiniMind. For a strictly "from-zero pretraining" setup, first pretrain an LLM in MiniMind and then plug it back here. "65M" refers to the trainable backbone (LLM ~64M + Projection ~1M); the SigLIP2 vision encoder contributes another ~95M parameters that stay frozen throughout and serve only as an image feature extractor, so the full model at inference time is roughly 160M (dense) / 294M (MoE).
 
 The VLM adds a Visual Encoder and a feature projection on top of the LLM, introducing a modality-mixing branch to support multimodal inputs:
 ![LLM-structure](./images/VLM-structure.jpg)
@@ -415,15 +415,15 @@ outputs the end token; here, the "token" doesn’t necessarily have to be text!
 The "foreign language dictionary" is referred to as the Visual Encoder model.  
 Like LlaVA, Qwen-VL, and other visual language models, MiniMind-V now uses the open-source SigLIP2 series models as the
 Visual Encoder.  
-Specifically, we use [siglip2-base-p16-256-ve](https://huggingface.co/jingyaogong/siglip2-base-p16-256-ve), a Visual
+Specifically, we use [siglip2-base-p32-256-ve](https://huggingface.co/jingyaogong/siglip2-base-p32-256-ve), a Visual
-Encoder based on the ViT-B/16 architecture for describing image-text information.  
+Encoder based on the ViT-B/32 architecture for describing image-text information.  
-The current SigLIP2 NaFlex vision encoder generates up to 256 patch tokens from the processor output as the input to the
+The current SigLIP2 NaFlex vision encoder generates 64 patch tokens (256×256 image / patch_size 32 = 8×8 = 64) from the processor output as the input to the
 encoder layer, which produces a 1×768 dimensional embedding vector for calculating error with the text.  
-We don’t need the final embedding representation, so we only take the output from the encoder layer, which is the output
+We don't need the final embedding representation, so we only take the output from the encoder layer, which is the output
 feature from the core ViT backbone.  
-It receives 256×768 features from the previous layer, which are then reshaped by concatenating every 4 adjacent tokens into 1 (256×768 → 64×3072), then projected to the LLM's hidden dimension via a 2-layer MLP (Linear→GELU→Linear), resulting in 64 visual tokens fed into MiniMind-V — this step is exactly cross-modal feature alignment: the native visual features are brought into the semantic space where text tokens live, so that the two can interact in the same space.
+It receives 64×768 features from the previous layer, which are projected to the LLM's hidden dimension via LayerNorm + a 2-layer MLP (Linear→GELU→Linear), resulting in 64 visual tokens fed into MiniMind-V — this step is exactly cross-modal feature alignment: the native visual features are brought into the semantic space where text tokens live, so that the two can interact in the same space.
 
-[LlaVA-1](https://arxiv.org/pdf/2304.08485) achieves good alignment with a simple linear transformation, [LlaVA-1.5](https://arxiv.org/pdf/2310.03744) upgrades to a 2-layer MLP. MiniMind-V adopts the same MLP Projection approach as LlaVA-1.5, combined with reshape for token compression.
+[LlaVA-1](https://arxiv.org/pdf/2304.08485) achieves good alignment with a simple linear transformation, [LlaVA-1.5](https://arxiv.org/pdf/2310.03744) upgrades to a 2-layer MLP. MiniMind-V adopts the same MLP Projection approach as LlaVA-1.5 (P32 natively outputs 64 tokens, no additional reshape compression needed).
 
 ![llava-structure](./images/llava-structure.png)
 
@@ -445,7 +445,7 @@ For example:
 <image>\nWhat is in this image?
 ```
 
-In `minimind-v`, the image is replaced by 64 `<|image_pad|>` tokens as placeholder (the 256 SigLIP2 patch features are compressed to 64 tokens via reshape+MLP),  
+In `minimind-v`, the image is replaced by 64 `<|image_pad|>` tokens as placeholder (SigLIP2 P32 directly outputs 64 patch tokens, projected to 64 visual tokens via MLP),  
 thus the `minimind-v` prompt becomes:
 
 ```text
@@ -481,9 +481,9 @@ It is composed of two image sources: a curated subset of LAION (mostly natural i
   - A blend of "image-grounded reasoning Q&A", "caption-style long descriptions" and "pure-text chat" — covering fine-grained follow-ups/long chains of thought as well as image description and general language ability.
 
 Roughly 2.9M samples in total. The Pretrain stage can be skipped entirely (SFT has absorbed it as a subset). Chinese and English are roughly balanced.
-Given MiniMind-V's trainable backbone is only 67M, mixing English and Chinese is a pragmatic choice: Chinese data helps native-language generation, while the original English descriptions tend to be more precise — the two complement each other.
+Given MiniMind-V's trainable backbone is only 65M, mixing English and Chinese is a pragmatic choice: Chinese data helps native-language generation, while the original English descriptions tend to be more precise — the two complement each other.
 
-All images are `resize`d to **256×256** (matching SigLIP2 NaFlex's 256 patch-token input) and re-encoded as JPEG, packed directly into parquet.
+All images are `resize`d to **256×256** (matching SigLIP2 NaFlex's input specification; P32 produces 64 patch tokens) and re-encoded as JPEG, packed directly into parquet.
 
 (`pretrain_i2t.parquet`) Pre-training dataset format:
 
@@ -529,7 +529,7 @@ Since SFT data already contains all Pretrain samples as a subset, this stage is
 
 > train_sft_vlm
 
-The SFT stage uses the aforementioned `sft_i2t.parquet` — about 2.9M mixed samples, covering the image captions inherited from Pretrain, reasoning-style Q&A on natural images, fine-grained Q&A on documents/charts, instructions synthesized by Gemini/Claude/GPT-4o, plus ~230K pure-text conversations (image column filled by an 8×8 black placeholder). Learning rate drops to ~5e-5, max sequence 768.
+The SFT stage uses the aforementioned `sft_i2t.parquet` — about 2.9M mixed samples, covering the image captions inherited from Pretrain, reasoning-style Q&A on natural images, fine-grained Q&A on documents/charts, instructions synthesized by Gemini/Claude/GPT-4o, plus ~230K pure-text conversations (image column filled by an 8×8 black placeholder). Learning rate drops to ~5e-6, max sequence 768.
 
 A common practice is to fully unfreeze the LLM during SFT, but this usually assumes a several-B-parameter base and a substantial amount of pure-text data mixed into SFT. MiniMind-V's language backbone is only 64M and ~92% of the current SFT data is image-related, so fully unfreezing the LLM would likely dilute its original general-language capability under the image-task gradients.
 
@@ -574,57 +574,57 @@ Prompt: `<image>\nPlease illustrate the image through your words.`
         <img src="./dataset/eval_images/image-01-golden-dog-balloons.jpg" alt="golden-dog">
         &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
       </td>
-      <td>The image shows a golden-brown dog in a park setting with a clear blue sky above. The dog appears to be a golden-brown dog with a cheerful expression, which is characteristic of a Golden Bay breed. The dog's fur is a lighter color, contrasting with the golden balloon, which is likely a balloon with a bright yellow color. The dog's tail is bald, and its tail is darker, indicating it might be a wild, possibly a festive holiday. The dog's bright yellow balloon, along with its tail, stands out against the lighter background of the park. There are no other people or identifying marks in the immediate vicinity of the dog, emphasizing its celebration and a sense of joy.</td>
+      <td>The image features a golden retriever in mid-motion, captured in mid-air, with its head tilted towards the camera. The dog, which appears to be a Golden Retriever, is in motion, with its tongue slightly out of focus, suggesting it's either waiting or turning. It has a distinctive blue tongue, which is a common feature in many dog breeds. The dog is standing on a grassy field with a few scattered clouds, and its mouth is open as if it is about to take a bite. The sky is partly cloudy, indicating fair weather. The lighting suggests it's daytime with ample sunlight, casting soft shadows on the grass and the grass.</td>
-      <td>The image depicts a golden-brown dog with a golden-brown coat, likely a Border Collie, standing on a grassy field with a clear blue sky. The dog appears to be a German Shepherd, given its size and the positioning of its tail. Its ears are perked up, and it has a golden-brown coat with a lighter golden hue. The dog's eyes are looking forward, and it seems to be looking down at it. The sky is partly cloudy, suggesting a fair weather day. The dog is standing on a grassy field, with some trees and others in the distance, indicating a calm and natural setting. The overall mood of the image is one of tranquility and enjoyment.</td>
+      <td>The image features a golden golden-brown dog with a slightly furrowed brow, likely in motion, as if in mid-motion. It's captured in a bright, natural setting, with a clear sky above and a clear blue sky above. The dog appears to be a German Shepherd, given its size and the prominence of its tongue and the bright red of its tongue. The dog's ears are perked up, suggesting it may be a pet. The grass is a vibrant green, and the sky is clear with no visible clouds, indicating good weather. There is a sense of motion in the photograph, with the dog's head tilted downwards and its eyes looking towards the sky.</td>
     </tr>
     <tr>
       <td>
         <img src="./dataset/eval_images/image-02-rainbow-umbrella-street.jpg" alt="rainbow-umbrella">
         &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
       </td>
-      <td>The image shows a rainy day with a rainbow-colored umbrella. The umbrella has a colorful mottled pattern of colorful, orange, yellow, red, green, purple, and blue. The umbrella is surrounded by a bright, irregular, bright daylight with irregular, warm lighting. The umbrella's bright colors range from pink, yellow, purple, green, blue, and red, creating a cool, wet atmosphere. The rainbow-colored umbrella is lit by the bright sky, indicating a sunny day. The umbrella is slightly warmer than the rainbow-colored umbrella, with a reflective surface and a blurred background that suggests a shaded area. The umbrella is in sharp focus, allowing the umbrella to be visible on the scene. The overall impression is one of a tropical or subtropical environment, with the rainbow-colored umbrella in a vibrant green color, creating a sense of depth and natural beauty.</td>
+      <td>The image shows a rain-shaped umbrella with a gradient of colors ranging from yellow to blue. The umbrella's umbrella is visible, showing a multicolored rainbow pattern with the gradient of colors. The water is illuminated by the rain, suggesting recent rain. The rainbow has a distinctive gradient of colors, primarily red, green, blue, yellow, and white, with the darker colors and the lighter colors on the umbrella's surface. In the background, there are buildings with red-roofed buildings, indicating a residential area. The weather appears overcast, and the lighting is soft, with no harsh shadows, indicating either early morning or late afternoon.</td>
-      <td>The image shows an outdoor scene with a rainy umbrella in the background, likely during a time of day when umbrellas are illuminated. The umbrellas are arranged in a turn, creating a dynamic and colorful umbrella. The umbrellas have a smooth, flat surface with some visible wrinkles and grouting, indicating a high level of detail. The umbrellas are capped with a slightly darker pink hue, providing a lighter gray color that contrasts with the umbrellas. They are positioned in the distance, adding to the umbrella's appearance. The umbrellas are the main subjects, and the umbrellas are positioned in the distance.</td>
+      <td>The image shows a rainbow umbrella in the midground, with a faint reflection of the umbrella. The umbrella is orange with a black fabric, and it is surrounded by a misty effect, suggesting the umbrella is illuminated. The umbrella is turned off, with its umbrella open in the middle ground, showing its umbrella open. The mist appears to be frosted, as indicated by the reflection of the umbrella. The background is blurred but shows an urban setting with buildings, possibly a street, and the sky is overcast. There is no discernible human activity in the image.</td>
     </tr>
     <tr>
       <td>
         <img src="./dataset/eval_images/image-03-cherry-blossom-bike.jpg" alt="cherry-blossom-bike">
         &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
       </td>
-      <td>The image depicts a vibrant pink bicycle, with its wicker basket attached to the bicycle. The bicycle's design is vibrant, with a sleek design that complements the vibrant colors of the pink and white flowers. The basket appears to be made of a material with a durable fabric, and it has a low-profile design, with a slight front wheel and a single rider, which are often used for pedestrian traffic. The bicycle's frame is not fully visible, but it has a black frame and a black handle. There are no visible people or distinctive features that suggest a specific location or event, such as a bike rack or a bike rack, which are not present in this image. The pink bicycle's body is partially visible and has a single seat, suggesting it is likely a bicycle. There are no discernible background elements that provide context or distraction from the image.</td>
+      <td>The image features a vibrant pink bicycle on a paved road, with a lush green palette dominated by pink and white blooms. The bicycle is positioned on a road, with a slightly elevated position and a brown handlebar on the right. The bicycle is positioned on a paved road, with a pedestal area visible on the left side of the frame, and a small, round, wooden frame with a rounded top. The bicycle is positioned in front of a white bicycle, and the background is softly blurred, with no discernible features that suggest a location or setting. The lighting is bright, and the overall mood of the image is peaceful and serene.</td>
-      <td>The image depicts a bright pink bicycle, captured from a perspective view from an elevated perspective. The bicycle is an iconic symbol of the bike, with its rider sporting a touch of white, white, and a touch of purple. The bike is a traditional style with a simple bike and a basketball hoop. It's a sunny day, and the lighting suggests it might be daytime. The scene is set against a pastoral scene, with the pathways leading towards a pathway that casts a warm glow, indicating the serene and warm atmosphere of the scene. The colors in the image are vibrant, with the pink of the pathway standing out against the pale pink of the bicycle. There's a sense of tranquility and freedom of movement.</td>
+      <td>The image features a vibrant pink pink scene on a street during the daytime. In the foreground, there's a large, classic bicycle with a sleek design, featuring a classic bike with a brown handlebar and a black frame. The bike has a blue handlebar and a chain strap, indicating it is designed for carrying luggage. Behind the bicycle, a series of pink blossoms and green foliage are visible, suggesting a garden or park setting. The background is blurred, with the focus on the pink and bicycle, which is the central subject of the image. The overall setting appears to be a serene, sunny day, with a clear sky and gentle lighting in the background.</td>
     </tr>
     <tr>
       <td>
         <img src="./dataset/eval_images/image-04-yellow-car.jpg" alt="yellow-car">
         &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
       </td>
-      <td>The image shows a yellow sports car positioned on a road with a clear blue sky. The car is positioned on a road with a prominent yellow roof, which is characteristic of a yellow sports car. The car is positioned in the front of the car, facing the camera, with its front grille and lower body lines visible, which suggests a sporty aesthetic. The car is parked on a road that appears to be a high-performance sports car, and the background is a parking lot with a few small boats, which is a common feature in such vehicles. There are no visible windows or other objects in the immediate vicinity of the car.</td>
+      <td>The image shows a bright yellow sports car positioned on a paved road. The car has a distinctive design with a long streamlined body, aerodynamic contours, and a long, streamlined body, typical of high-performance sports cars. The car's headlights are aerodynamic, and it has a low profile, aerodynamic brake, and a short, streamlined body. The windows appear to be tinted, and there's a clear sky above and a clear blue sky with few clouds. The car is parked on a road, with a hint of a flatbed traffic in the distance, indicating that this may be a busy road trip. The watermark "LOCKERS" is visible on the bottom right corner of the image, suggesting that this image may have been taken by a professional photographer or a promotional material.</td>
-      <td>The image features a bright yellow sports car parked on a sidewalk. The car is a paved road with a road that appears to be a parking lot, and the sidewalk is bordered by a thin line of trees and a clear blue sky. There are no visible passengers, and the car is parked on a road with a few cars, indicating it is likely a busy day. The car has a sleek design with a high-performance lines and a large round headlamps.</td>
+      <td>The image features a bright yellow sports car parked on a paved asphalt surface. The car is a sleek, aerodynamic design with a sleek, sporty body, aerodynamic lines, and aerodynamic bodywork. Its design suggests a modern, sporty aesthetic with a sporty appearance. The car's headlights are narrow and integrated into the design, with the headlights appearing to be short and streamlined, indicating a design that is likely sporty. The windows are tinted, and the vehicle is equipped with a black and white livery, which provides a contrast to the vibrant colors of the car. The sky is partly cloudy, with no clouds visible, suggesting a fair weather day. There are no people or other objects in the frame, placing the entire focus on the car.</td>
     </tr>
     <tr>
       <td>
         <img src="./dataset/eval_images/image-05-superhero-rooftop.jpg" alt="superhero">
         &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
       </td>
-      <td>The image shows a stylized silhouette of a character from a superhero film series. The character is wearing a long-sleeved blue suit with a blue collar and a red band on the left chest, paired with red trousers. The suit is equipped with a blue suit and a red band on top, and the suit is adorned with a red band, which is also in the midst of a superhero character. The suit is standing outdoors, and the character is on a city street, with a large, blue suit standing out. The sky is a gradient of warm colors, suggesting it is either dawn or dusk. The overall impression is one of a high-energy, urban nighttime scene, with a focus on the character and the character.</td>
+      <td>The image depicts a man standing on a raised, stern, ornate torch. He is dressed in a vibrant red and blue suit with a white shirt and a red swoosh on his head. His suit is adorned with a red cross, suggesting a superhero's personal style. The torch is illuminated with warm light, possibly from the sun, creating a soft, diffused light that enhances the three-dimensional effect. The sky is partly cloudy, indicating either early morning or late afternoon light. The terrain is populated with numerous high-rise buildings, some of which have a tall, slender building with a white roof. The buildings appear to be part of a cityscape, possibly in the United States. The sky is partly cloudy, suggesting it's either sunrise or sunset. The image is taken during the golden hour, with the sun setting behind the torch and the sky.</td>
-      <td>The image depicts a spaceship scene at what appears to be a cityscape during what appears to be sunset. The scene is dominated by the sky, with the silhouettes of a figure standing on a raised platform, which is the central subject of the image. The figure is dressed in a red and blue suit with a white shirt, which suggests a style commonly associated with the "UNITED SUNDER" superhero franchise. The suit is primarily red with a red hue and a white shirt. There is a figure with a red belt on the right, holding a red object that could be a shield or a wrench. The person is standing on a street with a couple of buildings, indicating that the location might be a city with buildings that are part of the UNITED SUNDER. The sky is filled with a warm orange hue, suggesting a sunset. The overall atmosphere is one of dynamism and fantasy.</td>
+      <td>The image features a female superhero standing on a steeplewheeled rooftop. She is dressed in a blue and red suit with a sleeveless blue top, which is red with a large, bold front opening. The suit is complemented by a red belt with a silver buckle, which is a characteristic feature of the superhero's superhero. Her hair is styled in a bun, and she has a confident stance with one hand on her hip and the other holding a weapon. The background is a clear sky with a gradient of warm colors, suggesting either dawn or dusk, with the sun setting in the background being the bright sunlight. The sky is a gradient of warm colors, transitioning from a warm yellow near the horizon to a deeper blue as it moves towards the sky.</td>
     </tr>
     <tr>
       <td>
         <img src="./dataset/eval_images/image-06-racecar-drift.jpg" alt="racecar">
         &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
       </td>
-      <td>The image captures a dynamic scene at what appears to be a race. In the foreground, a red sports car is mounted on a racetrack, its front face is turned slightly to the left. The car has a prominent rear wing and aerodynamic elements, which suggests a high-speed speed and aerodynamic profile. The track is adorned with multiple windows, one of which is visible, allowing a view of the ground level. The background shows a crowd of spectators, some dressed in red, while others are standing, possibly in the air, in the distance, indicating that this is likely the race of the race. The lighting is warm, and the ground is dim, which gives the racing a warm ambiance. There are no people visible in the image, and the image is taken at a time when the race is being captured and the car is in the air. The lighting conditions suggest that it is either early morning or late afternoon, as the sun is low on the horizon, casting a warm glow and creating a dramatic effect on the track.</td>
+      <td>The image captures a dynamic scene of a Ford Motor Company in mid-air, captured during twilight. The car is a Ford Motor Company, characterized by its sleek, aerodynamic design, with a high-performance tires suitable for diesel and aerodynamics. The car is equipped with a low-profile tires, which suggests it is designed for high-speed racing. The tires are marked with red and white markings, indicating a slightly rapid trajectory. The sky is clear with a soft gradient from light blue to orange near the horizon, suggesting either dawn or dusk. The racing has a modern design, with a prominent grille and side mirrors, which are characteristic of Motor Company's racing team. There are no visible race or other people in the immediate vicinity of the car. The car is on a track, indicating its position as a driver, and the background features a crowd of spectators, indicating the location might be in a competitive or high-speed racing area.</td>
-      <td>The image captures a dynamic scene at what appears to be a high-speed race, likely during the golden hour, given the warm lighting and the soft shadows. The track is bathed in a soft glow, indicating the sun is high and possibly at the moment of the race. There are no visible smoke, orbits, or disturbances in the sky, suggesting the race is either during the golden hour or during the golden hour. The race is a sizable form of aerodynamic body, with the front wheel being more pronounced, indicating it may have been used for speed and possibly aerodynamics. The track is bordered by a blurred audience, which may suggest a focus on the race's grandeur or the driver's presence. The track itself has a glossy finish, with some areas showing darker lines and the surrounding structures, which could be part of a track or a similar event.</td>
+      <td>The image depicts a dynamic scene at what appears to be a Formula One race car. The car is a two-person turbo-drive, indicated by the slight blur on the wheels, which suggests it is moving from left to right. The driver is wearing a full-face helmet with a visor, a white grille with a red and black design, and a red and black stripe on the left side of the front wheel. The car is sporty, with a sleek design, and is in motion, as indicated by the blur of the background. The background features a high-speed turn signal at the front, which is likely the car's trackside. The sky is clear with a few scattered clouds, suggesting the sun is either in the sky or the sun is low in the sky. There are no people visible in the image, focusing the attention solely on the driver and the car.</td>
     </tr>
   </tbody>
 </table>
 
 ### Effect Summary:
 
-Both models can identify the primary subject in most images (dog, umbrella, bicycle, sports car, superhero, racecar, etc.), but both exhibit repetitive phrasing and hallucinated details, placing overall performance at a stage of "understanding the gist but inaccurate on details".
+Both models correctly identify the primary subject across all 6 samples (dog, umbrella, bicycle, sports car, superhero, racecar) with 6/6 subject recognition, though both still exhibit some repetitive phrasing and hallucinated details, placing overall performance at a stage of "understanding the gist but inaccurate on details".
 
-Across these 6 samples, the MoE variant produces slightly richer scene descriptions and better color recognition (e.g. the cherry-blossom scene, rainbow umbrella, yellow sports car), while the Dense model tends to be more concise. Over a broader evaluation set, however, MoE occasionally suffers from more pronounced entity-level hallucinations or repetitive generation, showing higher variance than Dense — making Dense the safer default configuration.
+The MoE variant produces richer scene descriptions with better capture of background environments (urban streets, city skylines, sunset gradients) and object details (rainbow patterns, blue-red suit colors, racecar livery). The Dense model tends to be more concise with less repetition. Both exhibit similar levels of hallucination, with occasional inaccuracies in local details.
 
 Visual signals act as a special "foreign language" to the LLM, so the ceiling of "learning that language" is bounded by the LLM's own language ability. A stronger backbone extracts more value from the same image-text data; swapping MiniMind-V's backbone for a several-B-scale LLM yields clearly sharper details and more coherent reasoning.
 

eval_vlm.py

@@ -17,16 +17,15 @@ def init_model(args):
         ckp = f'./{args.save_dir}/{args.weight}_{args.hidden_size}{moe_suffix}.pth'
         model = MiniMindVLM(
             VLMConfig(hidden_size=args.hidden_size, num_hidden_layers=args.num_hidden_layers, use_moe=bool(args.use_moe)),
-            vision_model_path="./model/siglip2-base-p16-256-ve"
+            vision_model_path="./model/siglip2-base-p32-256-ve"
         )
         state_dict = torch.load(ckp, map_location=args.device)
         model.load_state_dict({k: v for k, v in state_dict.items() if 'mask' not in k}, strict=False)
     else:
         model = AutoModelForCausalLM.from_pretrained(args.load_from, trust_remote_code=True)
-        model.vision_encoder, model.processor = MiniMindVLM.get_vision_model("./model/siglip2-base-p16-256-ve")
+        model.vision_encoder, model.processor = MiniMindVLM.get_vision_model("./model/siglip2-base-p32-256-ve")
     get_model_params(model, model.config)
-    preprocess = model.processor
+    return model.half().eval().to(args.device), tokenizer, model.processor
-    return model.half().eval().to(args.device), tokenizer, preprocess
 
 
 def main():
@@ -54,7 +53,6 @@ def main():
     for image_file in sorted(os.listdir(args.image_dir)):
         if image_file.lower().endswith(('.png', '.jpg', '.jpeg', '.bmp')):
             setup_seed(random.randint(1, 31415926))
-            setup_seed(512)
             image_path = os.path.join(args.image_dir, image_file)
             image = Image.open(image_path).convert('RGB')
             pixel_values = {k: v.to(args.device) for k, v in MiniMindVLM.image2tensor(image, preprocess).items()}

model/model_init.md

@@ -1,23 +1,23 @@
 ## Download Vision Encoder (Required)
 
-This model requires an additional vision encoder `siglip2-base-p16-256-ve`
+This model requires an additional vision encoder `siglip2-base-p32-256-ve`
 
 
 ```bash
 git lfs install
 
 # from HuggingFace
-git clone https://huggingface.co/jingyaogong/siglip2-base-p16-256-ve
+git clone https://huggingface.co/jingyaogong/siglip2-base-p32-256-ve
 
 # or from ModelScope
-git clone https://modelscope.cn/models/gongjy/siglip2-base-p16-256-ve
+git clone https://modelscope.cn/models/gongjy/siglip2-base-p32-256-ve
 ```
 
 Directory Structure
 
 ```
 model/
-├── siglip2-base-p16-256-ve/        # Vision Encoder (Required)
+├── siglip2-base-p32-256-ve/        # Vision Encoder (Required)
 │   ├── config.json
 │   ├── model.safetensors           # ~180MB, float16
 │   └── preprocessor_config.json

model/model_vlm.py

@@ -21,28 +21,22 @@ class VLMConfig(MiniMindConfig):
         super().__init__(**kwargs)
 
 class MMVisionProjector(nn.Module):
-    def __init__(self, in_dim, out_dim, source_tokens=256, target_tokens=64):
+    def __init__(self, in_dim, out_dim, source_tokens=64, target_tokens=64):
         super().__init__()
-        self.target_tokens = target_tokens
-        self.merge = source_tokens // target_tokens
         self.mlp = nn.Sequential(
-            nn.LayerNorm(in_dim * self.merge),
+            nn.LayerNorm(in_dim),
-            nn.Linear(in_dim * self.merge, out_dim),
+            nn.Linear(in_dim, out_dim),
             nn.GELU(),
             nn.Linear(out_dim, out_dim),
         )
     def forward(self, x):
-        b, n, d = x.shape
-        side = int(n ** 0.5)
-        s = int(self.merge ** 0.5)
-        x = x.view(b, side // s, s, side // s, s, d).permute(0, 1, 3, 2, 4, 5).reshape(b, self.target_tokens, d * self.merge)
         return self.mlp(x)
 
 # 继承自语言模型
 class MiniMindVLM(MiniMindForCausalLM):
     config_class = VLMConfig
 
-    def __init__(self, config: VLMConfig = None, vision_model_path="./model/siglip2-base-p16-256-ve"):
+    def __init__(self, config: VLMConfig = None, vision_model_path="./model/siglip2-base-p32-256-ve"):
         self.config = config or VLMConfig()
         super().__init__(self.config)
         self.vision_encoder, self.processor = self.__class__.get_vision_model(vision_model_path)

scripts/convert_vlm.py

@@ -16,7 +16,7 @@ warnings.filterwarnings('ignore', category=UserWarning)
 def convert_torch2transformers_minimind(torch_path, transformers_path, dtype=torch.bfloat16):
     VLMConfig.register_for_auto_class()
     MiniMindVLM.register_for_auto_class("AutoModelForCausalLM")
-    lm_model = MiniMindVLM(lm_config, vision_model_path="../model/siglip2-base-p16-256-ve")
+    lm_model = MiniMindVLM(lm_config, vision_model_path="../model/siglip2-base-p32-256-ve")
     device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
     state_dict = torch.load(torch_path, map_location=device)
     lm_model.load_state_dict(state_dict, strict=False)
@@ -47,7 +47,7 @@ def convert_transformers2torch(transformers_path, torch_path):
 
 
 if __name__ == '__main__':
-    lm_config = VLMConfig(hidden_size=768, num_hidden_layers=8, max_seq_len=8192, use_moe=False)
+    lm_config = VLMConfig(hidden_size=768, num_hidden_layers=8, max_seq_len=8192, use_moe=True)
     torch_path = f"../out/sft_vlm_{lm_config.hidden_size}{'_moe' if lm_config.use_moe else ''}.pth"
-    transformers_path = '../minimind-3v'
+    transformers_path = '../minimind-3v-moe'
     convert_torch2transformers_minimind(torch_path, transformers_path)

scripts/web_demo_vlm.py

@@ -8,7 +8,7 @@ import torch
 import warnings
 import gradio as gr
 from queue import Queue
-from threading import Thread
+from threading import Thread, Lock
 from PIL import Image
 from transformers import AutoTokenizer, AutoModelForCausalLM, TextStreamer
 from model.model_vlm import MiniMindVLM, VLMConfig
@@ -16,6 +16,7 @@ from transformers import logging as hf_logging
 
 hf_logging.set_verbosity_error()
 warnings.filterwarnings('ignore')
+model_lock = Lock()
 
 
 def scan_vlm_models(base_dir):
@@ -34,18 +35,20 @@ def scan_vlm_models(base_dir):
 
 def load_vlm_model(model_path):
     global model, tokenizer, preprocess, lm_config, current_model_name
-    tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)
+    with model_lock:
-    model = AutoModelForCausalLM.from_pretrained(model_path, trust_remote_code=True)
+        [sys.modules.pop(k) for k in list(sys.modules) if 'transformers_modules' in k]
-    model.vision_encoder, model.processor = MiniMindVLM.get_vision_model(vision_model_path)
+        tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)
-    if model.vision_encoder is None: raise FileNotFoundError(f"视觉编码器未找到: {vision_model_path}")
+        model = AutoModelForCausalLM.from_pretrained(model_path, trust_remote_code=True)
-    preprocess = model.processor
+        model.vision_encoder, model.processor = MiniMindVLM.get_vision_model(vision_model_path)
-    lm_config = model.config
+        if model.vision_encoder is None: raise FileNotFoundError(f"视觉编码器未找到: {vision_model_path}")
-    model = model.half().eval().to(device)
+        preprocess = model.processor
-    model.vision_encoder = model.vision_encoder.to(device)
+        lm_config = model.config
-    current_model_name = os.path.basename(model_path)
+        model = model.half().eval().to(device)
-    param_str = f'{sum(p.numel() for p in model.parameters() if p.requires_grad) / 1e6:.2f}M'
+        model.vision_encoder = model.vision_encoder.to(device)
-    print(f'已加载 {current_model_name}，参数量：{param_str}')
+        current_model_name = os.path.basename(model_path)
-    return f"已加载: {current_model_name} ({param_str})"
+        param_str = f'{sum(p.numel() for p in model.parameters() if p.requires_grad) / 1e6:.2f}M'
+        print(f'已加载 {current_model_name}，参数量：{param_str}')
+        return f"已加载: {current_model_name} ({param_str})"
 
 
 class CustomStreamer(TextStreamer):
@@ -85,18 +88,19 @@ def chat(prompt, current_image_path=None):
         streamer = CustomStreamer(tokenizer, queue)
 
         def _generate():
-            model.generate(
+            with model_lock:
-                inputs.input_ids,
+                model.generate(
-                max_new_tokens=max_seq_len,
+                    inputs.input_ids,
-                do_sample=True,
+                    max_new_tokens=max_seq_len,
-                temperature=temperature,
+                    do_sample=True,
-                top_p=top_p,
+                    temperature=temperature,
-                attention_mask=inputs.attention_mask,
+                    top_p=top_p,
-                pad_token_id=tokenizer.pad_token_id,
+                    attention_mask=inputs.attention_mask,
-                eos_token_id=tokenizer.eos_token_id,
+                    pad_token_id=tokenizer.pad_token_id,
-                streamer=streamer,
+                    eos_token_id=tokenizer.eos_token_id,
-                pixel_values=pixel_values
+                    streamer=streamer,
-            )
+                    pixel_values=pixel_values
+                )
 
         Thread(target=_generate).start()
 
@@ -150,7 +154,7 @@ def launch_gradio_server(server_name="0.0.0.0", server_port=7788):
 if __name__ == '__main__':
     parser = argparse.ArgumentParser(description="Chat with MiniMind-V")
     parser.add_argument('--load_from', default='./', type=str, help="transformers模型扫描目录")
-    parser.add_argument('--vision_model', default='../model/siglip2-base-p16-256-ve', type=str, help="视觉编码器路径")
+    parser.add_argument('--vision_model', default='../model/siglip2-base-p32-256-ve', type=str, help="视觉编码器路径")
     parser.add_argument('--temperature', default=0.7, type=float, help="生成温度")
     parser.add_argument('--top_p', default=0.95, type=float, help="nucleus采样阈值")
     parser.add_argument('--device', default='cuda' if torch.cuda.is_available() else 'cpu', type=str, help="运行设备")

trainer/train_pretrain_vlm.py

@@ -90,7 +90,7 @@ if __name__ == "__main__":
     parser = argparse.ArgumentParser(description="MiniMind-V Pretrain")
     parser.add_argument("--save_dir", type=str, default="../out", help="模型保存目录")
     parser.add_argument('--save_weight', default='pretrain_vlm', type=str, help="保存权重的前缀名")
-    parser.add_argument("--epochs", type=int, default=3, help="训练轮数")
+    parser.add_argument("--epochs", type=int, default=2, help="训练轮数")
     parser.add_argument("--batch_size", type=int, default=16, help="batch size")
     parser.add_argument("--learning_rate", type=float, default=4e-4, help="初始学习率")
     parser.add_argument("--device", type=str, default="cuda:0" if torch.cuda.is_available() else "cpu", help="训练设备")

trainer/train_sft_vlm.py

@@ -90,7 +90,7 @@ if __name__ == "__main__":
     parser = argparse.ArgumentParser(description="MiniMind-V SFT")
     parser.add_argument("--save_dir", type=str, default="../out", help="模型保存目录")
     parser.add_argument('--save_weight', default='sft_vlm', type=str, help="保存权重的前缀名")
-    parser.add_argument("--epochs", type=int, default=6, help="训练轮数")
+    parser.add_argument("--epochs", type=int, default=2, help="训练轮数")
     parser.add_argument("--batch_size", type=int, default=4, help="batch size")
     parser.add_argument("--learning_rate", type=float, default=5e-6, help="初始学习率")
     parser.add_argument("--device", type=str, default="cuda:0" if torch.cuda.is_available() else "cpu", help="训练设备")

trainer/trainer_utils.py

@@ -63,7 +63,7 @@ def setup_seed(seed: int):
     torch.backends.cudnn.benchmark = False
 
 
-def init_vlm_model(vlm_config, from_weight='pretrain_vlm', tokenizer_path='../model', vision_model_path='../model/siglip2-base-p16-256-ve', save_dir='../out', device='cuda', freeze_llm=0):
+def init_vlm_model(vlm_config, from_weight='pretrain_vlm', tokenizer_path='../model', vision_model_path='../model/siglip2-base-p32-256-ve', save_dir='../out', device='cuda', freeze_llm=0):
     tokenizer = AutoTokenizer.from_pretrained(tokenizer_path)
     model = MiniMindVLM(vlm_config, vision_model_path=vision_model_path)