[update] minimind-3v
jingyaogong
parent
333654d6e9
commit
9c3343dd88
241
README.md
241
README.md
|
|
@ -33,14 +33,14 @@
|
||||||
|
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
* 此项目旨在从0开始，仅用1.3块钱成本 + 1小时！即可训练出67M参数的超小多模态视觉语言模型**MiniMind-V**。
|
* 此项目旨在从0开始，仅用3块钱成本 + 2小时！即可训练出67M参数的超小多模态视觉语言模型**MiniMind-V**。
|
||||||
* **MiniMind-V**最小版本体积仅为 GPT3 的约 $\frac{1}{2600}$，力求做到个人GPU也可快速推理甚至训练。
|
* **MiniMind-V**最小版本体积仅为 GPT3 的约 $\frac{1}{2600}$，力求做到个人GPU也可快速推理甚至训练。
|
||||||
* **MiniMind-V**是[MiniMind](https://github.com/jingyaogong/minimind)纯语言模型的视觉能力额外拓展。
|
* **MiniMind-V**是[MiniMind](https://github.com/jingyaogong/minimind)纯语言模型的视觉能力额外拓展。
|
||||||
* 项目同时包含了VLM大模型的极简结构、数据集清洗、预训练(Pretrain)、监督微调(SFT)等全过程代码。
|
* 项目同时包含了VLM大模型的极简结构、数据集清洗、Pretrain、SFT等全过程代码。
|
||||||
* 这不仅是一个开源VLM模型的最小实现，也是入门视觉语言模型的简明教程。
|
* 这不仅是一个开源VLM模型的最小实现，也是入门视觉语言模型的简明教程。
|
||||||
* 希望此项目能为所有人提供一个抛砖引玉的示例，一起感受创造的乐趣！推动更广泛AI社区的进步！
|
* 希望此项目能为所有人提供一个抛砖引玉的示例，一起感受创造的乐趣！推动更广泛AI社区的进步！
|
||||||
|
|
||||||
> 为防止误解，“1小时” 基于NVIDIA 3090硬件设备（单卡）测试`1 epoch`，“1.3块钱” 指GPU服务器租用成本。
|
> 注：本项目基于 Apache 2.0 协议开源，完全免费。“2 小时” 指 SFT 阶段在单张 NVIDIA 3090 上跑完 `1 epoch` 的实测耗时，“3 块钱” 指对应时段的 GPU 租用成本。
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
@ -61,23 +61,35 @@
|
||||||
> [!TIP]
|
> [!TIP]
|
||||||
> （截至2026-02-15）MiniMind-V 系列已完成了以下型号模型训练，最小仅需67M (0.067B)，即可具备识图和对话的能力！
|
> （截至2026-02-15）MiniMind-V 系列已完成了以下型号模型训练，最小仅需67M (0.067B)，即可具备识图和对话的能力！
|
||||||
|
|
||||||
| 模型 (大小) | 推理占用 | release |
|
| 模型 (大小) | release |
|
||||||
|---------------------------|--------|------------|
|
|---|---|
|
||||||
| minimind-3v-moe (201M-A67M) | 1.0 GB | 2026.04.01 |
|
| minimind-3v-moe (201M-A67M) | 2026.04.20 |
|
||||||
| minimind-3v (67M) | 0.5 GB | 2026.04.01 |
|
| minimind-3v (67M) | 2026.04.20 |
|
||||||
| MiniMind2-V (104M) | 1.1 GB | 2025.02.20 |
|
| MiniMind2-V (104M) | 2025.02.20 |
|
||||||
| MiniMind2-Small-V (26M) | 0.6 GB | 2025.02.20 |
|
| MiniMind2-Small-V (26M) | 2025.02.20 |
|
||||||
| minimind-v-v1-small (27M) | 0.6 GB | 2024.10.04 |
|
| minimind-v-v1-small (27M) | 2024.10.04 |
|
||||||
| minimind-v-v1 (109M) | 1.1 GB | 2024.10.04 |
|
| minimind-v-v1 (109M) | 2024.10.04 |
|
||||||
|
|
||||||
#### 👉 更新日志
|
#### 👉 更新日志
|
||||||
|
|
||||||
|
<details>
|
||||||
|
<summary> <b>2026-04-20</b> </summary>
|
||||||
|
|
||||||
|
- 更新模型检查点：minimind-3v (67M) / minimind-3v-moe (201M-A67M)
|
||||||
|
- Projector 更新：添加 `LayerNorm`，token 合并改为 2D pixel-shuffle
|
||||||
|
- Vision Encoder 换为 `SiglipVisionModel`（固定 256×256）
|
||||||
|
- 训练数据切到 ALLaVA-4V（Pretrain 127 万 / SFT 290 万，已合并为单阶段 SFT）
|
||||||
|
- Freeze 策略更新：`freeze_llm=1` 解冻首末两层，Pretrain/SFT 默认改为 `2`/`1`；`max_seq_len` 360 → 450
|
||||||
|
- 其他 bugfix 与小调整
|
||||||
|
|
||||||
|
</details>
|
||||||
|
|
||||||
<details>
|
<details>
|
||||||
<summary> <b>2026-04-01</b> </summary>
|
<summary> <b>2026-04-01</b> </summary>
|
||||||
|
|
||||||
- 新增 minimind-3v (67M) 和 minimind-3v-moe (201M-A67M) 模型
|
- 新增 minimind-3v (67M) 和 minimind-3v-moe (201M-A67M) 模型
|
||||||
- 统一使用768+8架构，支持dense和moe两种模式
|
- 统一使用768+8架构，支持dense和moe两种模式
|
||||||
- 视觉编码器从CLIP切换为SigLIP2（siglip2-base-p16-ve）
|
- 视觉编码器从CLIP切换为SigLIP2（siglip2-base-p16-256-ve）
|
||||||
- 投影模块从QFormer改为MLP Projection + reshape压缩
|
- 投影模块从QFormer改为MLP Projection + reshape压缩
|
||||||
- 数据集格式更新为parquet，混合数据源、更新tokenizer，图像占位符改为`<|image_pad|>`、新增WebUI：支持动态扫描模型目录、下拉菜单切换模型
|
- 数据集格式更新为parquet，混合数据源、更新tokenizer，图像占位符改为`<|image_pad|>`、新增WebUI：支持动态扫描模型目录、下拉菜单切换模型
|
||||||
- 模型代码重构，LLM/VLM统一适配Transformers格式
|
- 模型代码重构，LLM/VLM统一适配Transformers格式
|
||||||
|
|
@ -96,18 +108,15 @@
|
||||||
</details>
|
</details>
|
||||||
|
|
||||||
<details>
|
<details>
|
||||||
<summary> <b>2025-04-27</b> </summary>
|
|
||||||
|
<summary> <b>More...</b> </summary>
|
||||||
|
|
||||||
|
**2025-04-27**
|
||||||
|
|
||||||
- 兼容性更新
|
- 兼容性更新
|
||||||
- 适配[「minimind仓库新特性」](https://github.com/jingyaogong/minimind/issues/370)
|
- 适配[「minimind仓库新特性」](https://github.com/jingyaogong/minimind/issues/370)
|
||||||
- 规范化部分代码
|
- 规范化部分代码
|
||||||
|
|
||||||
</details>
|
|
||||||
|
|
||||||
<details>
|
|
||||||
|
|
||||||
<summary> <b>More...</b> </summary>
|
|
||||||
|
|
||||||
**2025-02-20**
|
**2025-02-20**
|
||||||
|
|
||||||
- MiniMind2-V伴随MiniMind2同步更新
|
- MiniMind2-V伴随MiniMind2同步更新
|
||||||
|
|
@ -146,9 +155,9 @@ git clone --depth 1 https://github.com/jingyaogong/minimind-v
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# 下载siglip2模型到 ./model 目录下
|
# 下载siglip2模型到 ./model 目录下
|
||||||
git clone https://huggingface.co/jingyaogong/siglip2-base-p16-ve
|
git clone https://huggingface.co/jingyaogong/siglip2-base-p16-256-ve
|
||||||
# 或
|
# 或
|
||||||
git clone https://modelscope.cn/models/gongjy/siglip2-base-p16-ve
|
git clone https://modelscope.cn/models/gongjy/siglip2-base-p16-256-ve
|
||||||
```
|
```
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
|
|
@ -223,40 +232,43 @@ print(torch.cuda.is_available())
|
||||||
|
|
||||||
【注2】Parquet是列式存储格式，支持高效压缩和快速读取。如果你对它感到陌生，可以预览数据内容，在`dataset/`目录下执行`python lm_dataset.py`可视化前5条图文对
|
【注2】Parquet是列式存储格式，支持高效压缩和快速读取。如果你对它感到陌生，可以预览数据内容，在`dataset/`目录下执行`python lm_dataset.py`可视化前5条图文对
|
||||||
|
|
||||||
Pretrain数据：
|
Pretrain 数据（可选；仅包含 caption 子集）：
|
||||||
```bash
|
```bash
|
||||||
wget https://hf-mirror.com/datasets/jingyaogong/minimind-v_dataset/resolve/main/pretrain_i2t.parquet
|
wget https://hf-mirror.com/datasets/jingyaogong/minimind-v_dataset/resolve/main/pretrain_i2t.parquet
|
||||||
```
|
```
|
||||||
|
|
||||||
SFT数据：
|
SFT 数据（必须；包含 caption + instruct + 纯文本全量合并）：
|
||||||
```bash
|
```bash
|
||||||
wget https://hf-mirror.com/datasets/jingyaogong/minimind-v_dataset/resolve/main/sft_i2t.parquet
|
wget https://hf-mirror.com/datasets/jingyaogong/minimind-v_dataset/resolve/main/sft_i2t.parquet
|
||||||
```
|
```
|
||||||
|
|
||||||
建议预留~2GB空间存放数据集，若无多余空间存放pretrain数据，可尝试跳过pretrain训练步骤直接进行sft训练。
|
SFT 单文件 290 万条已把 Pretrain 作为子集合并，经全局 dictionary encoding 去重后体积只比 SFT 原版多 ~10%，可覆盖所有训练阶段。
|
||||||
|
|
||||||
</details>
|
</details>
|
||||||
|
|
||||||
### 3' 开始训练
|
### 3' 开始训练
|
||||||
|
|
||||||
**3.1 预训练（学图像描述）**
|
**3.1 Pretrain（可选）**
|
||||||
|
|
||||||
|
> SFT 数据集 `sft_i2t.parquet` 已把 Pretrain 数据作为子集合并进来，**此阶段可跳过**，直接 `--from_weight llm` 进入 SFT 即可；若想让 Projector 提前完成对齐、SFT 收敛更稳，可以提前执行 Pretrain。
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# 基础训练命令（从LLM权重开始，仅训练vision_proj）
|
# 基础训练命令（从LLM权重开始，仅训练 vision_proj）
|
||||||
python train_pretrain_vlm.py --epochs 4 --from_weight llm
|
python train_pretrain_vlm.py --epochs 2 --from_weight llm
|
||||||
```
|
```
|
||||||
|
|
||||||
> 执行预训练，得到 `pretrain_vlm_*.pth` 作为预训练的输出权重（其中*为模型的dimension，默认为768）
|
> 执行 Pretrain，得到 `pretrain_vlm_*.pth` 作为 Pretrain 的输出权重（其中*为模型的dimension，默认为768）
|
||||||
|
|
||||||
|
|
||||||
**3.2 监督微调（学看图对话方式）**
|
**3.2 SFT（必须）**
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# 基础训练命令（从预训练权重开始，全参数微调）
|
# 基础训练命令（默认 --freeze_llm 1：解冻 proj + LLM 首尾层，保留中间层原 LLM 知识）
|
||||||
|
# 若已执行 3.1：--from_weight pretrain_vlm；若跳过 Pretrain：--from_weight llm
|
||||||
python train_sft_vlm.py --epochs 2 --from_weight pretrain_vlm
|
python train_sft_vlm.py --epochs 2 --from_weight pretrain_vlm
|
||||||
```
|
```
|
||||||
|
|
||||||
> 执行监督微调，得到 `sft_vlm_*.pth` 作为指令微调的输出权重
|
> 执行 SFT，得到 `sft_vlm_*.pth` 作为 SFT 的输出权重
|
||||||
|
|
||||||
<details>
|
<details>
|
||||||
<summary>注：训练须知</summary>
|
<summary>注：训练须知</summary>
|
||||||
|
|
@ -276,7 +288,7 @@ python train_sft_vlm.py --epochs 4 --from_resume 1
|
||||||
- `--from_weight`: 基础权重名称（llm, pretrain_vlm, none等）
|
- `--from_weight`: 基础权重名称（llm, pretrain_vlm, none等）
|
||||||
- `--save_weight`: 保存权重的前缀名
|
- `--save_weight`: 保存权重的前缀名
|
||||||
- `--from_resume`: 是否续训（0=从头开始，1=从检查点继续）
|
- `--from_resume`: 是否续训（0=从头开始，1=从检查点继续）
|
||||||
- `--freeze_llm`: 是否冻结LLM参数（仅pretrain使用）
|
- `--freeze_llm`: 冻结策略（0=全参可训，1=proj + LLM 首尾层，2=仅训 proj）。Pretrain 默认 2，SFT 默认 1
|
||||||
- 更多可直接参考代码
|
- 更多可直接参考代码
|
||||||
|
|
||||||
</details>
|
</details>
|
||||||
|
|
@ -300,7 +312,7 @@ python eval_vlm.py --weight pretrain_vlm
|
||||||
---
|
---
|
||||||
|
|
||||||
> [!TIP]
|
> [!TIP]
|
||||||
> 训练脚本均为Pytorch原生框架，均支持多卡加速，假设你的设备有N (N＞1) 张显卡：
|
> 训练脚本均为 PyTorch 原生框架，均支持多卡加速，假设你的设备有N (N＞1) 张显卡：
|
||||||
|
|
||||||
单机N卡启动训练方式 (DDP, 支持多机多卡集群)
|
单机N卡启动训练方式 (DDP, 支持多机多卡集群)
|
||||||
|
|
||||||
|
|
@ -337,14 +349,11 @@ python train_xxx.py --use_wandb
|
||||||
|
|
||||||
# 📌 模型细节
|
# 📌 模型细节
|
||||||
|
|
||||||
MiniMind-V (VLM)的基座语言模型MiniMind (LLM)来自孪生项目[minimind](https://github.com/jingyaogong/minimind)，
|
MiniMind-V 的语言主干即孪生项目 [minimind](https://github.com/jingyaogong/minimind) 训练得到的 `llm_768.pth`，LLM 本身的结构、训练细节与实验分析不在本仓库重复，默认读者对 MiniMind LLM 已有基本了解。未接触过也不影响照着“快速开始”跑通 MiniMind-V，流程自洽。
|
||||||
具体的模型结构、训练细节、原理、测试效果等均可移步[minimind](https://github.com/jingyaogong/minimind)项目查阅。
|
|
||||||
此处为减少冗余，省略讨论LLM的相关部分，默认您已对MiniMind (LLM)的细节有基本的了解。
|
|
||||||
|
|
||||||
> 即使您不太了解LLM的细节，也可参考“快速开始”流程训练一个MiniMind-V，
|
顶部 “从 0 训练” 和 “67M” 两个简化口号在这里也需要说明。“从 0 训练” 指 VLM 本身从零构建（Projection 随机初始化、LLM 首末层微调对齐），但 LLM 主干并非从零 pretrain，而是基于 MiniMind 的语言模型权重续训而来；若要严格意义上的 “完全从零 pretrain”，请先在 MiniMind 训一版 LLM 再迁回本项目。“67M” 指可训练部分的主干规模（LLM ~64M + Projection ~3M）；视觉编码器 SigLIP2 另有 ~93M 参数全程冻结、仅作图像特征提取，因此推理时整机参数量约 160M（dense）/ 294M（MoE）。
|
||||||
> 这并不受到影响，仓库致力于最低成本的开箱即用！
|
|
||||||
|
|
||||||
MiniMind-V的结构仅增加Visual Encoder和特征投影两个子模块，增加模态混合分支，以支持多种模态信息的输入：
|
VLM 在 LLM 基础上增加 Visual Encoder 和特征投影两个子模块，引入模态混合分支以支持多模态输入：
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
@ -392,15 +401,12 @@ GPT模型根据现有token预测输出下一个下下一个下下下一个token
|
||||||
|
|
||||||
"外语词典" 称之为Visual Encoder模型。
|
"外语词典" 称之为Visual Encoder模型。
|
||||||
和LlaVA、Qwen-VL等视觉语言模型类似，MiniMind-V当前选用开源SigLIP2系列模型作为Visual Encoder。
|
和LlaVA、Qwen-VL等视觉语言模型类似，MiniMind-V当前选用开源SigLIP2系列模型作为Visual Encoder。
|
||||||
具体使用[siglip2-base-p16-ve](https://huggingface.co/jingyaogong/siglip2-base-p16-ve)，
|
具体使用[siglip2-base-p16-256-ve](https://huggingface.co/jingyaogong/siglip2-base-p16-256-ve)，
|
||||||
一种基于 ViT-B/16 架构的Visual Encoder用于描述图像文本信息。
|
一种基于 ViT-B/16 架构的Visual Encoder用于描述图像文本信息。
|
||||||
当前使用的 SigLIP2 NaFlex 视觉编码器会根据预处理结果生成最多256个patch token作为encoder编码层的输入，
|
当前使用的 SigLIP2 NaFlex 视觉编码器会根据预处理结果生成最多256个patch token作为encoder编码层的输入，
|
||||||
最终产生1×768维的嵌入向量用于和文本对计算误差。
|
最终产生1×768维的嵌入向量用于和文本对计算误差。
|
||||||
我们并不需要最终嵌入表示，因此只取encoder层的输出，也就是VIT核心主干的输出特征即可。
|
我们并不需要最终嵌入表示，因此只取encoder层的输出，也就是VIT核心主干的输出特征即可。
|
||||||
它拿到前一层256×768大小的特征，通过reshape将每4个相邻token拼接为1个（256×768 → 64×3072），再经过2层MLP（Linear→GELU→Linear）投影到LLM的隐藏维度，最终作为64个visual token输入MiniMind-V。
|
它拿到前一层 256×768 大小的特征，通过 reshape 将每 4 个相邻 token 拼接为 1 个（256×768 → 64×3072），再经过 2 层 MLP（Linear→GELU→Linear）投影到 LLM 的隐藏维度，最终作为 64 个 visual token 输入 MiniMind-V——这一步完成的就是跨模态特征对齐：让原生视觉特征进入文本 token 所在的语义空间，使两者可以在同一空间里交互。
|
||||||
与LLM的结合在获取图像encoder特征后，一方面需要把视觉特征对齐到LLM的文本token维度，
|
|
||||||
另一方面，要将图像特征映射到与文本embedding相同的空间，即文本token和原生的视觉token需要磨合并不能直接地一视同仁，
|
|
||||||
可以称之为跨模态的特征对齐。
|
|
||||||
|
|
||||||
[LlaVA-1](https://arxiv.org/pdf/2304.08485)使用简单的线性变换完成对齐，[LlaVA-1.5](https://arxiv.org/pdf/2310.03744)升级为2层MLP，MiniMind-V采用与LlaVA-1.5相同的MLP Projection方案，并结合reshape进行token压缩。
|
[LlaVA-1](https://arxiv.org/pdf/2304.08485)使用简单的线性变换完成对齐，[LlaVA-1.5](https://arxiv.org/pdf/2310.03744)升级为2层MLP，MiniMind-V采用与LlaVA-1.5相同的MLP Projection方案，并结合reshape进行token压缩。
|
||||||
|
|
||||||
|
|
@ -439,61 +445,82 @@ VLM的输入依然是一段文本，其中包含特殊的`<image>`占位符。
|
||||||
|
|
||||||
## Ⅰ 数据集
|
## Ⅰ 数据集
|
||||||
|
|
||||||
原始来源：
|
本轮训练用到的图文数据全部来自 [ALLaVA-4V](https://huggingface.co/datasets/FreedomIntelligence/ALLaVA-4V) 系列。
|
||||||
- [Chinese-LLaVA-Vision](https://huggingface.co/datasets/LinkSoul/Chinese-LLaVA-Vision-Instructions)：包含约57万张预训练图像，来自CC-3M和COCO 2014
|
相比以往从几份 LLaVA 衍生集拼接得到的数据，ALLaVA-4V 的质量更整齐、中英双语原生对照，细粒度描述也更充分。
|
||||||
- [llava-en-zh-300k](https://huggingface.co/datasets/BUAADreamer/llava-en-zh-300k)：包含300k条指令微调数据和15万张图像
|
它由两个子源构成：一份是 LAION 里挑出来的高质量图片（自然图像为主），一份是 VFLAN 指令流里挑出来的图片（文档、图表、合成场景居多）。
|
||||||
- [LLaVA-SFT-665K](https://huggingface.co/datasets/csuhan/LLaVA-SFT-665K)：包含665k条指令微调数据
|
|
||||||
|
|
||||||
其中部分为中文数据，部分为英文数据。问答内容经过翻译，对中文支持更友好，进一步经过整理并`resize`（pretrain分辨率128×128，sft分辨率160×160）。
|
- **Pretrain（`pretrain_i2t.parquet`，约 127 万条 / ~64 万张唯一图像）**
|
||||||
|
- `ALLaVA-Caption-LAION-4V` 英/中：~47万 + ~44万
|
||||||
|
- `ALLaVA-Caption-VFLAN-4V` 英/中：~19万 + ~17万
|
||||||
|
- 任务形式为"请描述这张图片"类的单轮长描述，用于让模型建立视觉 token 到语言 token 的基础对齐。
|
||||||
|
|
||||||
(pretrain_i2t.parquet) 预训练数据集格式：
|
- **SFT（`sft_i2t.parquet`，约 290 万条 / ~65 万张唯一图像）**
|
||||||
|
- `ALLaVA-Instruct-LAION-4V` 英/中：~47万 + ~47万
|
||||||
|
- `ALLaVA-Instruct-VFLAN-4V` 英/中：~19万 + ~17万
|
||||||
|
- `Instruct-LAION-4v-gemini-claude-ensembled`（Gemini/Claude 合成的增广指令）：~5万
|
||||||
|
- `Instruct-LAION-4oiterative`（基于 GPT-4o 迭代润色的指令）：~5万
|
||||||
|
- 纯文本对话（保留基础语言能力，图像列填 8×8 黑图占位）：~23万
|
||||||
|
- **Pretrain caption 数据全量合并**（与 pretrain 同源、~99% 图像重叠）：~127万
|
||||||
|
- 任务形式混合了"围绕图片的推理式问答"、"caption 长描述"和"纯文本对话"，既考验细节追问/长链条推理，也兼顾图像描述与通用语言能力。
|
||||||
|
|
||||||
|
合计约 290 万条样本，Pretrain 阶段可直接跳过（SFT 已把 Pretrain 作为子集吸收）。中英比例大致均衡。
|
||||||
|
考虑到 MiniMind-V 的语言主干仅 67M，把英文和中文一起喂进去是比较稳妥的做法——中文语料对母语输出更友好，而英文原生描述通常更精细，两者互为补充。
|
||||||
|
|
||||||
|
图像统一 `resize` 到 **256×256**（与 SigLIP2 NaFlex 编码器的 256 patch token 输入规格相对应），重新编码为 JPEG 打包进 parquet。
|
||||||
|
|
||||||
|
(`pretrain_i2t.parquet`) Pretrain 数据集格式：
|
||||||
|
|
||||||
```text
|
```text
|
||||||
列名: conversations (json string), image_bytes (binary), image_names (string)
|
列名: conversations (json string), image_bytes (binary)
|
||||||
|
|
||||||
conversations 示例:
|
conversations 示例:
|
||||||
[
|
[
|
||||||
{"role": "user", "content": "提供给定图像的简要描述。\n<image>"},
|
{"role": "user", "content": "<image>\n请提供对图片的详细文字描述。"},
|
||||||
{"role": "assistant", "content": "橄榄油是自由使用的健康成分。"}
|
{"role": "assistant", "content": "这张图片展示的是一个..."}
|
||||||
]
|
]
|
||||||
image_bytes: <图像二进制数据>
|
image_bytes: <图像二进制数据>
|
||||||
```
|
```
|
||||||
|
|
||||||
(sft_i2t.parquet) 单图指令微调数据集格式：
|
(`sft_i2t.parquet`) 单图 SFT 数据集格式：
|
||||||
|
|
||||||
```text
|
```text
|
||||||
列名: conversations (json string), image_bytes (binary), image_names (string)
|
列名: conversations (json string), image_bytes (binary)
|
||||||
|
|
||||||
conversations 示例:
|
conversations 示例:
|
||||||
[
|
[
|
||||||
{"role": "user", "content": "闹钟的位置对睡眠质量有什么影响？<image>"},
|
{"role": "user", "content": "根据图片推断这个场景的大致时间？<image>"},
|
||||||
{"role": "assistant", "content": "把数字闹钟放在床头柜..."}
|
{"role": "assistant", "content": "从光线和阴影来看..."}
|
||||||
]
|
]
|
||||||
image_bytes: <图像二进制数据>
|
image_bytes: <图像二进制数据>
|
||||||
```
|
```
|
||||||
|
|
||||||
> 注：sft_i2t.parquet 共约 58 万条数据，其中约 23.6 万条为含图对话（i2t），约 34.6 万条为纯文本对话（t2t），后者用于保持模型的基础语言能力。
|
> 注：sft_i2t.parquet 共约 290 万条数据，其中约 140 万条为图像 instruct 对话、约 127 万条为图像 caption 描述（Pretrain 数据合并而来）、约 23 万条为纯文本对话（t2t，图像列填 8×8 黑图占位，用于保持模型的基础语言能力）。由于 Pretrain 已作为子集并入，可选择跳过 Pretrain 阶段直接进行 SFT。
|
||||||
|
|
||||||
数据集下载地址：([ModelScope](https://www.modelscope.cn/datasets/gongjy/minimind-v_dataset) | [HuggingFace](https://huggingface.co/datasets/jingyaogong/minimind-v_dataset))
|
数据集下载地址：([ModelScope](https://www.modelscope.cn/datasets/gongjy/minimind-v_dataset) | [HuggingFace](https://huggingface.co/datasets/jingyaogong/minimind-v_dataset))
|
||||||
|
|
||||||
## Ⅱ 训练
|
## Ⅱ 训练
|
||||||
|
|
||||||
训练分为两个阶段，均冻结Visual Encoder梯度，仅训练Projection和LLM部分。
|
训练分为两个阶段（Pretrain 可选；SFT 必需），均冻结 Visual Encoder 梯度，仅训练 Projection 和部分 LLM 层。
|
||||||
训练基于LLM预训练权重初始化，支持DDP多卡训练、混合精度（bfloat16）、torch.compile加速和swanlab日志记录。
|
训练基于 LLM Pretrain 权重初始化，支持 DDP 多卡训练、混合精度（bfloat16）、torch.compile 加速和 swanlab 日志记录。
|
||||||
|
|
||||||
> train_pretrain_vlm
|
> train_pretrain_vlm（可选）
|
||||||
|
|
||||||
预训练阶段从约113万条图文描述数据中学习图片的通用知识（如鹿是鹿，狗是狗）。
|
Pretrain 阶段从约 127 万条图文描述数据中学习图片的通用知识（如鹿是鹿，狗是狗）。
|
||||||
此阶段采用较高学习率（1e-4），最大序列长度360，冻结LLM主体参数，仅设置Projection和LLM的第0层可学习，
|
此阶段采用较高学习率（~4e-4），最大序列长度 450，**完全冻结 LLM 和 Visual Encoder，仅训练 Projection**（`--freeze_llm 2`）。
|
||||||
目的是让模型快速建立视觉特征到语言空间的基础映射，同时避免破坏LLM已有的语言能力。
|
目的是让 Projector 干净地完成视觉 token 到语言 token 的基础对齐，不扰动 LLM 原有权重。
|
||||||
|
由于 SFT 阶段数据已把 Pretrain 全部样本包含为子集，此步骤可选，跳过可节省时间，但**先跑一轮 Pretrain 让 Projector 先行对齐，SFT 收敛更稳**。
|
||||||
|
|
||||||
> train_sft_vlm
|
> train_sft_vlm
|
||||||
|
|
||||||
指令微调阶段从约58万条数据中学习真实问答格式，其中约23.6万条为图文多轮对话，约34.6万条为纯文本对话（用于保持LLM基础能力）。
|
SFT 阶段的数据即前述 `sft_i2t.parquet`，约 290 万条混合样本，涵盖 Pretrain 继承而来的图文 caption、自然图像上的推理问答、文档/图表的细节问答、Gemini/Claude/GPT-4o 合成的指令，以及约 23 万条纯文本对话（图像列填 8×8 黑图占位）。学习率降到 ~5e-5，最大序列 768。
|
||||||
此阶段采用较低学习率（1e-5~1e-6），最大序列长度768，解冻Projection和LLM全部参数进行全量微调，
|
|
||||||
使模型学会根据图片内容进行多轮对话，并通过混入的纯文本数据缓解灾难性遗忘。
|
|
||||||
|
|
||||||
> 训练时间和Loss走势（仅供参考）
|
常见做法是在 SFT 阶段把 LLM 全参解冻，但这通常建立在底座有几 B 参数、且 SFT 数据中混入大量纯文本这两个前提之上。MiniMind-V 的语言主干仅 64M，而当前 SFT 数据中 ~92% 与图像有关，若全参解冻，LLM 原有的通用语言能力极易被图文任务的梯度稀释。
|
||||||
|
|
||||||
|
这里采用 `--freeze_llm 1`：**只解冻 Projection 与 LLM 的首、末层，其余 N-2 层保持 Pretrain 时的权重**。首层是视觉 token 进入 LLM 后的第一道处理，直接承担跨模态融合；末层影响回答的格式与风格；中间层保留 LLM Pretrain 的知识，不被图文任务的梯度改写。那 23 万条纯文本样本对通用语言能力也起到类似正则的作用。
|
||||||
|
|
||||||
|
> 训练时间和 Loss 走势（仅供参考）
|
||||||
|
|
||||||
|
单张 NVIDIA 3090 上，SFT 跑完 `1 epoch` 实测约 2 小时，dense 与 MoE 用时接近（激活参数规模一致，差异主要来自 expert routing 的额外访存）；Pretrain 数据量约为 SFT 的 45%，单 epoch 耗时按比例粗估即可。按云上 3090 约 1.5 元/小时的行情，SFT 单轮成本落在 3 元上下。
|
||||||
|
|
||||||
Pretrain [768+8] (dense & moe)
|
Pretrain [768+8] (dense & moe)
|
||||||
|
|
||||||
|
|
@ -508,13 +535,13 @@ SFT [768+8] (dense & moe)
|
||||||
| 原生PyTorch (`*.pth`) | [minimind-3v-pytorch](https://www.modelscope.cn/models/gongjy/minimind-3v-pytorch) | [minimind-3v-pytorch](https://huggingface.co/jingyaogong/minimind-3v-pytorch) |
|
| 原生PyTorch (`*.pth`) | [minimind-3v-pytorch](https://www.modelscope.cn/models/gongjy/minimind-3v-pytorch) | [minimind-3v-pytorch](https://huggingface.co/jingyaogong/minimind-3v-pytorch) |
|
||||||
| Transformers 格式 | [minimind-v collection](https://modelscope.cn/collections/MiniMind-V-42b841dde22d41) | [minimind-v collection](https://huggingface.co/collections/jingyaogong/minimind-v-67000833fb60b3a2e1f3597d) |
|
| Transformers 格式 | [minimind-v collection](https://modelscope.cn/collections/MiniMind-V-42b841dde22d41) | [minimind-v collection](https://huggingface.co/collections/jingyaogong/minimind-v-67000833fb60b3a2e1f3597d) |
|
||||||
|
|
||||||
> 注：Transformers版本均为单图指令微调后的`MiniMind-V`模型
|
> 注：Transformers版本均为单图 SFT 后的`MiniMind-V`模型
|
||||||
|
|
||||||
# 📌 评估
|
# 📌 评估
|
||||||
|
|
||||||
### 效果测试
|
### 效果测试
|
||||||
|
|
||||||
#### 单图对话
|
Prompt: `<image>\n请描述这张图中的主要物体和场景。`
|
||||||
|
|
||||||
<table>
|
<table>
|
||||||
<thead>
|
<thead>
|
||||||
|
|
@ -527,92 +554,62 @@ SFT [768+8] (dense & moe)
|
||||||
<tbody>
|
<tbody>
|
||||||
<tr>
|
<tr>
|
||||||
<td>
|
<td>
|
||||||
<img src="./dataset/eval_images/airplane-flying-blue-sky.jpg" alt="airplane">
|
<img src="./dataset/eval_images/image-01-golden-dog-balloons.jpg" alt="golden-dog">
|
||||||
|
|
||||||
</td>
|
</td>
|
||||||
<td>在这幅图片中，一架白色的飞机正降落在一片广阔的天空中。天空中飘浮着许多流线型的建筑物，这些建筑物散落在天空中。除了主要的飞机外，还有两辆汽车在场景中各处停放，包括一辆巴士和一辆小型汽车。这架飞机似乎停在地面上，表明它们正在进行商业活动。</td>
|
<td>这张图片中的主要物体是一只金毛寻回犬，以其金色的皮毛和友好的表情而闻名。金毛寻回犬是这幅图片中的主要对象，以其金色的皮毛和友好的表情而闻名。它们是图片的主要焦点，并在这幅图片中展示了这个狗的特征。</td>
|
||||||
<td>在这张照片中，有一架大型的飞机正在降落，这表明它是为这架飞机而设计的。此外，它停在云层之下，这表明它在移动。天空中有云朵，暗示着这架飞机正在空中飞行。整个场景的背景显示出一种宁静祥和的气氛，暗示这架飞机正在飞行中作为一个机会来进行外交活动或与其他客机接触。</td>
|
<td>这张图片展示了一只金毛寻回犬在草地上享受一群彩色气球，很可能是在玩耍或散步的时候。这只狗的皮毛呈金黄色，尾巴尖端略带波浪状。这只狗的耳朵尖尖，尾巴垂下垂，表明它正在向前看，或者可能正在进行某种形式的玩耍或互动。背景是一片广阔的开阔景观，天空是晴朗的蓝色，零星分布着几朵云。在画面的左侧，有一条黄色和粉色的条纹，它们挂在草地上，为画面增添了色彩和质感。</td>
|
||||||
</tr>
|
</tr>
|
||||||
<tr>
|
<tr>
|
||||||
<td>
|
<td>
|
||||||
<img src="./dataset/eval_images/birthday-cake-candles-table.jpg" alt="birthday-cake">
|
<img src="./dataset/eval_images/image-02-rainbow-umbrella-street.jpg" alt="rainbow-umbrella">
|
||||||
|
|
||||||
</td>
|
</td>
|
||||||
<td>图片显示了一张大生日蛋糕，上面刻有各种生日和装饰物的生日和一束蜡烛。这些甜蜜而独特的气味使其成为这个特殊场合的焦点。蛋糕被放在餐桌上，有蛋糕放在上面，很可能是放在桌上，用蜡烛加上蜡烛来营造出诱人的场景。此外，这张生日蛋糕还配上了蜡烛作为装饰物，进一步丰富了整个生日的氛围。</td>
|
<td>这张图片展示了一只雨伞在城市街道上停放的场景。雨伞主要是鲜艳的颜色，在黑色背景上形成鲜明的对比。它被一条多彩的线连接在一起，可能表明天气状况良好。雨伞的闪闪发光的外套增添了一丝动感，为场景增添了一种动感。雨伞的反射和反射暗示着它可能是为了街道或停车场的静止而被使用。它被放置在一条铺设的道路上，表明这个场景是在城市街道上拍摄的，可能是在一个城市公园或公寓区。</td>
|
||||||
<td>这张照片捕捉到了一个生日聚会，各种美味可口的生日蛋糕和一把勺子放在那里。它被放置在一个充满生气的蛋糕盒子里，里面装着各种各样的草莓糖霜糖浆。蛋糕上有一个大号的蜡烛，给人一种温馨和诱人的图案。周围有很多小点心，比如小心地蜡烛和糖霜，让整个蛋糕看起来更加令人放松和诱人。</td>
|
<td>这张图中的主要物体是雨伞，它在雨天被雨水滋润。雨伞是主要的物体，通过其设计和大小来识别，使其成为图片的焦点。雨伞的颜色鲜艳，有明亮的黄色、橙色、红色和蓝色，这些颜色通常与雨水相关联。雨伞的反光表面反射着光线，表明它是有光泽的，这在雨天是很常见的。雨伞的反射和颜色表明它可能是由雨水滴或雨水渗入地面，这在雨天是常见的。雨伞的设计，以其多彩的图案和颜色，与雨伞的鲜艳颜色相辅相成，使其成为图片的焦点。</td>
|
||||||
</tr>
|
</tr>
|
||||||
<tr>
|
<tr>
|
||||||
<td>
|
<td>
|
||||||
<img src="./dataset/eval_images/pizza-on-wooden-board.jpg" alt="pizza">
|
<img src="./dataset/eval_images/image-03-cherry-blossom-bike.jpg" alt="cherry-blossom-bike">
|
||||||
|
|
||||||
</td>
|
</td>
|
||||||
<td>在这张图片中，有一个装饰着奶酪的比萨饼和一片新鲜的青绿野餐毯。这看起来像是一块沙拉，给人一种清新、诱人、美味的享受体验。披萨的大小和大小暗示着一种随意和随意的用餐体验。画面中，一群人围坐在一块砖块上，其中一些则分散在桌子上，周围摆放着不同种类的青绿野餐毯。此外，桌子上还放着一块披萨片，上面摆放着各种各样的青绿野餐毯，营造出一种轻松愉快的氛围。</td>
|
<td>这张图片中的主要物体是一辆自行车，具体来说是一辆青绿色的自行车。这一识别是基于几个视觉线索：骑车者的服装，头盔和轮胎的存在，以及前部的后部轮胎，这些都是自行车的特征。此外，背景中有树木和晴朗的天空，进一步强调了这一场景。图片中没有任何可见的物体或人，进一步强调了这辆自行车的特点。</td>
|
||||||
<td>图中，比萨饼在木桌上。它被切成了比萨饼的大小，而且披萨放置在木桌上。比萨有很多配料，包括奶酪和各种酱汁。披萨有多种口味，包括经典的牛排式和意大利式，还有一些加在比萨表面。比萨上有很多新鲜水果，如西红柿和莫吉托，以及奶酪，使这幅图片更加丰富和有吸引力。</td>
|
<td>这张图片展示了一幅宁静的户外场景，前景是一条繁华的街道，上面挂满了粉色花朵和红色装饰，散落着各种粉色花朵和白色小花。在前景，有一棵开着粉色花朵的大树，树干略显皱纹，暗示着成熟或可能是春天的场景。树后面是一片郁郁葱葱的绿树，树叶浓密，表明可能是春天或夏天。前景是一条铺满粉色花朵的道路，通向画面左侧是一条长满花朵的小径。在道路的右侧，是一条修剪整齐的道路，两旁是修剪整齐的草坪，背景是晴朗的蓝天和远处的另一棵树。整个场景传达了一种宁静和自然美的感觉。</td>
|
||||||
</tr>
|
</tr>
|
||||||
<tr>
|
<tr>
|
||||||
<td>
|
<td>
|
||||||
<img src="./dataset/eval_images/red-sports-car-road.jpg" alt="red-car">
|
<img src="./dataset/eval_images/image-04-yellow-car.jpg" alt="yellow-car">
|
||||||
|
|
||||||
</td>
|
</td>
|
||||||
<td>在这幅图片中，一辆白色的马车停泊在一条红色的路牌上。这个车辆位于一条绿油油的道路上，很可能是一个购物中心或高速公路。在这辆车的后部，可以看到一个绿色的马车停泊在路上，这是典型的户外场所。这辆马车可能是为了娱乐或观赏车辆而停放。</td>
|
<td>这张图片中的主要物体是一辆现代高尔夫球车，具体来说是一辆在沙漠或海滩上的高尔夫球车。这辆车是主要的焦点，设计时尚，具有流线型的车身，配备了高性能的后扰流板，并配备了赛车轮胎，表明这是一辆高性能跑车。车辆的风格和设计表明它可能是一款高性能跑车，可能是在沙漠或海滩上使用。</td>
|
||||||
<td>画面显示了一辆红色高性能赛车，停在一辆大型汽车后面。这辆车可以看到车身、汽车、汽车及汽车停在里面。它似乎是一辆大型红色汽车，有各种大小的汽车，表明它可能是汽车制造商生产的。此外，车辆周围的环境暗示了一种户外环境，因为一辆汽车也出现在场景中。</td>
|
<td>这张图片展示了一辆明亮的黄色跑车在晴朗的天空下。这辆车看起来是一辆豪华轿车，设计时尚，有着流线型的车身和明显的车顶，暗示着运动性能。它拥有高性能的车身，配有大尺寸的大众Jet和前大灯，展现出时尚的外观。这辆车的车身设计流线型，有流线型的车身线条和明显的车顶线条，暗示着高性能的车型。背景是一片晴朗的天空，有几朵白云，表明天气条件良好。在图片的右侧，可以看到一辆明亮的黄色跑车，暗示着它是为休闲或运动设计的。</td>
|
||||||
</tr>
|
</tr>
|
||||||
<tr>
|
<tr>
|
||||||
<td>
|
<td>
|
||||||
<img src="./dataset/eval_images/row-of-colorful-houses.jpg" alt="colorful-houses">
|
<img src="./dataset/eval_images/image-05-superhero-rooftop.jpg" alt="superhero">
|
||||||
|
|
||||||
</td>
|
</td>
|
||||||
<td>画面中，一座蓝色白色的大房子位于一条城市街道上，为这个地区增添了一丝自然与奇思妙想。墙上挂着一盏交通灯，为整个场景增添了特色和引人入胜。</td>
|
<td>这张图中的主要物体是超级英雄的战斗姿态。超级英雄通常是最具威胁性或高能量的形象，以保护和控制英雄。他穿着一套经典的超级英雄服装，包括蓝色牛仔裤和黑色靴子，暗示着他是该品牌的忠实伙伴。超级英雄通常被描绘为一个具有攻击性的角色，通常被视为英雄的特征。超级英雄的服装和姿势，以及他的姿势，表明他可能代表着英雄的身份或身份，而非任何超级英雄。</td>
|
||||||
<td>画面中，一座蓝色房子旁边有很多小花瓶。这表明这座房子里可能正在营运一些小型小型花盆或盆栽植物。一些人聚集在房子周围，可能正在享受户外生活中的某种乐趣或美景。有些人站在画面中，而其他人则散布于场景中。总体而言，这幅场景捕捉到了一个美丽而令人愉快的场景，展示出该房子里一个令人放松和宁静的环境。</td>
|
<td>这张图中的主要物体似乎是一个超级英雄角色，可能是超级英雄，考虑到其服装和装甲的风格。这个角色穿着超级英雄服装，包括红色和白色的袍子，这在超级英雄服装中很常见。他手持一块大大的红色和白色的大胆胡须，这是超级英雄服装的标志性特征。这个角色还戴着一条带有红色和白色条纹的红色头盔，这表明他可能是一名战士或领导者。背景是城市天际线，有着现代建筑，天空呈现出蓝色和橙色的色调，暗示着一个城市环境。</td>
|
||||||
</tr>
|
</tr>
|
||||||
<tr>
|
<tr>
|
||||||
<td>
|
<td>
|
||||||
<img src="./dataset/eval_images/snow-mountain-lake-view.jpg" alt="snow-mountain">
|
<img src="./dataset/eval_images/image-06-racecar-drift.jpg" alt="racecar">
|
||||||
|
|
||||||
</td>
|
</td>
|
||||||
<td>在这幅图片中，有一个高高的山，它看起来像是大森林中的一片高山。天空中闪烁着不同颜色的星星，给画面增加了一抹红点。天空中有两朵高大的树，树木高高地挂在一起，暗示着森林中的宁静与自然之美。在画面的中心，可以看到一棵高大的松树，树干上覆盖着一层薄薄的苔藓。这种高高的松树与周围的大山构成了一个有趣而引人入胜的背景，为这片自然之美增添了一丝神秘色彩。</td>
|
<td>这张图片中的主要物体是一辆红色的赛车，很可能是赛车。赛车是赛车运动的典型特征，旨在提高速度、提高燃油效率和赛道的稳定性和安全性。赛车的特征包括前部有大型前部、后部有大型后部，以及与赛道的对峙部分。赛车的颜色和设计与赛车的整体风格相符，以赛车为主。赛车的赛道是静止的，表明这是一条赛道。赛车的赛道是从赛道上移动的，没有任何可见的轨迹或赛道的边缘。</td>
|
||||||
<td>这幅图片展示了一个令人印象深刻的宁静湖面。湖水从天上飘浮着，暗示着一个令人平静和放松的水面。湖边上有几匹高大、形状各异的景象，它们在湖面上显得格外美丽。此外，在湖的边缘，有一座巨大的高山，为整个湖景增添了几分神秘色彩。湖景中的天空也被描述得如画，给整个画面增添了一种纯净和宁静的气氛。</td>
|
<td>这张图片中的主要物体是红色的，它在赛道上被捕捉到在运动中，这可以从赛车的后视图中看出。这个动作是红色的，通过其独特的尾部设计，可以识别出赛道的结构，这是赛道上常见的特征。赛车的赛车设计表明它是为高速赛道赛事设计的，这在赛道上很常见。背景中的观众也可见，但焦点完全集中在红色的赛车上，这表明这个场景是在一个正式的赛道赛事中拍摄的。</td>
|
||||||
</tr>
|
|
||||||
<tr>
|
|
||||||
<td>
|
|
||||||
<img src="./dataset/eval_images/street-food-hotpot-table.jpg" alt="street-food">
|
|
||||||
|
|
||||||
</td>
|
|
||||||
<td>图中，一大群人聚集在一张大餐桌旁欣赏着烤肉和热带水果，其中一人站在一碗里摆放着各种各样的盘子。桌子上放着几个碗，上面摆满了肉类和其他配料。有些放在盘子里，其他的则放在桌子上。在这张餐桌周围，有几个盘子，其中两个放着一杯酒，另一个放在靠近餐桌的左侧。</td>
|
|
||||||
<td>画面中，一群人聚集在一家大餐馆里，享受着一顿饭。这家餐厅的菜单上有一些生菜和猪肉，但它们已经被切成了四份，上面还有一个碗。他们拿着烤肉准备食用。在背景中，有几个瓶子在场景中。还有一把勺子位于桌子左侧，使盘子看起来更吸引人。一盆盆栽植物放在桌子左侧上方，为空间增添了一抹绿色。场景中的其他元素包括一个碗，里面放着葡萄酒和两根葡萄酒。</td>
|
|
||||||
</tr>
|
|
||||||
<tr>
|
|
||||||
<td>
|
|
||||||
<img src="./dataset/eval_images/three-kittens-basket.jpg" alt="kittens">
|
|
||||||
|
|
||||||
</td>
|
|
||||||
<td>图中，一只棕色的小灰猫正坐在篮子里。这只猫身上戴着一顶棕色的帽子，很可能是一个戴帽子的男人。在篮子里，一只棕色的紫色小灰猫正沿着篮子里去休息。这些小猫似乎也在享受这份温暖，但它们似乎并没有完全放过来。此外，背景中还可以看到一把剪刀。这把剪刀看起来是专门为小猫设计的，它可以用作家庭相册或礼物。在篮子的侧边，有几条篮子，其中一条是最亮的，另一条则是最暗的。在篮子中可以看到一只棕色小灰猫，而另一条则是更暗的。</td>
|
|
||||||
<td>在这张照片中，一只小猫坐在篮子里，紧挨着它坐在篮子里的那块木篮上。猫的身体上有九条纹毛发。这个场景描绘了它们之间的亲密关系，展示了它们在一起度过时光的不同场合。画面中，一群大的猫坐在篮子里，其中一只小猫也被描述为小猫，这可能表明他们正在享受与猫互动、与它们的互动或一起度过愉快时光。</td>
|
|
||||||
</tr>
|
|
||||||
<tr>
|
|
||||||
<td>
|
|
||||||
<img src="./dataset/eval_images/tropical-beach-palm-tree.jpg" alt="tropical-beach">
|
|
||||||
|
|
||||||
</td>
|
|
||||||
<td>在这张图片中，沙滩上有很多椅子，还有一些人站着，可以看到一把遮阳伞。虽然它看起来很大，但却没有任何特别的设计。沙滩上有许多椅子，表明这是一家餐馆或者服务员办公室。其中最引人注目的是一张海边椅子，椅子上放着一只热带海滩椅。这个椅子非常适合放松身心、享受海滩时光。此外，还有一些椅子和其他人在场景中，可能是为了放置食物或其他用途。靠近椅子的椅子表示该位置可供使用的其他人使用，也许也有一人在靠近那个椅子的地方。</td>
|
|
||||||
<td>图片显示了一个美丽的海滩场景，有很多椅子散布在天然的棕榈树上。其中一个椅子靠近海滩，而另一个则较小。沙滩上有两把椅子，其中一把靠近中间，另一把则稍微偏左，还有一些则在靠近边缘处。在海边的海滩周围，你可以看到几个人坐在海边的沙滩上，有的靠近海水中，还有一张沙滩椅。其中一张椅子靠近海滩，另一张椅子靠近海边。此外，还可以看到几只遮阳伞，为沙滩上的躺椅提供了遮阴。</td>
|
|
||||||
</tr>
|
|
||||||
<tr>
|
|
||||||
<td>
|
|
||||||
<img src="./dataset/eval_images/yellow-school-bus-road.jpg" alt="school-bus">
|
|
||||||
|
|
||||||
</td>
|
|
||||||
<td>画面中，一辆蓝色的黄色公共汽车正从一辆黄色公共汽车驶过，在道路上停泊着。这辆公共汽车看起来是在一个黄色的黄色高速公路上。图中有几个人，其中一些靠近前景，而另一些则靠后一些，但都没有看到。在黄色公共汽车附近，可以看到一辆停在路边，那辆停在路边。此外，还有两辆不同方向的巴士，一辆靠近前景，另一辆靠近前景，另一辆稍微靠后一些。</td>
|
|
||||||
<td>画面中，一辆黄色和黄色相间的黄色和蓝色交叉路口的蓝色公共汽车正在一条通往路缘上的红色公交车站。有几辆公共汽车正停在路边，它们离一排车道很近。在背景中，可以看到一些长凳，它们在城市里交叉起来。一个长凳位于图中最左侧，而另一个则稍微靠后一点，为画面增添了一些城市特色。整个场景中有很多人和车辆散落在场景各处，包括黄色和蓝色的交叉路口。整个场景给人一种忙碌和迷茫的感觉，这也突显了公共汽车在市区中的存在和目的。</td>
|
|
||||||
</tr>
|
</tr>
|
||||||
</tbody>
|
</tbody>
|
||||||
</table>
|
</table>
|
||||||
|
|
||||||
### 效果小结：
|
### 效果小结：
|
||||||
|
|
||||||
两个模型均能识别图像主体（飞机、蛋糕、汽车、海滩等），但普遍存在重复表述和幻觉细节。受限于模型和数据规模，整体处于"能看懂大意、细节不准"的阶段。
|
两个模型都能在大多数样本上识别出主体对象（狗、雨伞、自行车、跑车、超级英雄、赛车等），但都伴随重复表述与细节幻觉，整体处于"能看懂大意、细节不准"的阶段。
|
||||||
|
|
||||||
视觉信号对于LLM视作一种特殊的外语，因此"学习外语"的能力高低，很大程度上取决于LLM的能力。LLM性能越强，对应的VLM越强，此时效果增益会很明显。
|
在这 6 张样本上，MoE 版本的整体场景描述与颜色识别略优于 Dense（例如对樱花场景、彩色雨伞、黄色跑车的把握更完整），Dense 则在表达简洁性上占优；不过在更广的评测范围内，MoE 对少数样本会出现较明显的本体级幻觉或重复生成，方差大于 Dense，因此 Dense 仍是更稳妥的默认配置。
|
||||||
|
|
||||||
|
视觉信号对 LLM 而言相当于一门特殊的"外语"，"学外语"的上限因此受制于 LLM 自身的语言能力。底座越强，同样的图文数据喂进去收益越大；把 MiniMind-V 的主干换成几 B 量级的 LLM，细节准确度与推理连贯性的提升会相当明显。
|
||||||
|
|
||||||
#### 未来值得改进的方面：
|
#### 未来值得改进的方面：
|
||||||
|
|
||||||
|
|
|
||||||
256
README_en.md
256
README_en.md
|
|
@ -32,21 +32,19 @@
|
||||||
|
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
* This project aims to train a super-small multimodal vision-language model, **MiniMind-V**, with just a cost of 1.3 RMB
|
* This project aims to train a super-small multimodal vision-language model, **MiniMind-V**, with just a cost of 3 RMB
|
||||||
and 1 hours of work, starting from scratch!
|
and 2 hours of work, starting from scratch!
|
||||||
* The smallest version of **MiniMind-V** is only about $\frac{1}{2600}$ the size of GPT-3, designed to enable fast
|
* The smallest version of **MiniMind-V** is only about $\frac{1}{2600}$ the size of GPT-3, designed to enable fast
|
||||||
inference and even training on personal GPUs.
|
inference and even training on personal GPUs.
|
||||||
* **MiniMind-V** is an extension of the visual capabilities of the [MiniMind](https://github.com/jingyaogong/minimind)
|
* **MiniMind-V** is an extension of the visual capabilities of the [MiniMind](https://github.com/jingyaogong/minimind)
|
||||||
pure language model.
|
pure language model.
|
||||||
* The project includes full code for the minimalist structure of large VLM models, dataset cleaning, pretraining, and
|
* The project includes full code for the minimalist structure of large VLM models, dataset cleaning, Pretrain, and SFT.
|
||||||
supervised fine-tuning (SFT).
|
|
||||||
* This is not only the smallest implementation of an open-source VLM model but also a concise tutorial for beginners in
|
* This is not only the smallest implementation of an open-source VLM model but also a concise tutorial for beginners in
|
||||||
vision-language models.
|
vision-language models.
|
||||||
* The hope is that this project can provide a useful example to inspire others and share the joy of creation, helping to
|
* The hope is that this project can provide a useful example to inspire others and share the joy of creation, helping to
|
||||||
drive progress in the wider AI community!
|
drive progress in the wider AI community!
|
||||||
|
|
||||||
> To avoid misunderstandings, the "1 hours" is based on testing (`1 epoch`) with an NVIDIA 3090 hardware device (single GPU), and
|
> Note: this project is released under the Apache 2.0 license and is completely free. The "2 hours" refer to the measured time of running `1 epoch` of SFT on a single NVIDIA 3090, and the "3 RMB" refer to the GPU rental cost for that time slot.
|
||||||
> the "1.3 RMB" refers to GPU server rental costs.
|
|
||||||
|
|
||||||
<div align="center">
|
<div align="center">
|
||||||
|
|
||||||
|
|
@ -66,23 +64,35 @@ Is the training process difficult? Now, let's explore the answers and feel the j
|
||||||
> (As of 2026-02-15) The MiniMind-V series has completed the training of the following model versions, with the smallest
|
> (As of 2026-02-15) The MiniMind-V series has completed the training of the following model versions, with the smallest
|
||||||
> requiring only 67M (0.067B) parameters, capable of both image recognition and conversation!
|
> requiring only 67M (0.067B) parameters, capable of both image recognition and conversation!
|
||||||
|
|
||||||
| Model (Size) | Inference Memory | Release |
|
| Model (Size) | Release |
|
||||||
|---------------------------|------------------|------------|
|
|---|---|
|
||||||
| minimind-3v-moe (201M-A67M) | 1.0 GB | 2026.04.01 |
|
| minimind-3v-moe (201M-A67M) | 2026.04.20 |
|
||||||
| minimind-3v (67M) | 0.5 GB | 2026.04.01 |
|
| minimind-3v (67M) | 2026.04.20 |
|
||||||
| MiniMind2-V (104M) | 1.1 GB | 2025.02.20 |
|
| MiniMind2-V (104M) | 2025.02.20 |
|
||||||
| MiniMind2-Small-V (26M) | 0.6 GB | 2025.02.20 |
|
| MiniMind2-Small-V (26M) | 2025.02.20 |
|
||||||
| minimind-v-v1-small (27M) | 0.6 GB | 2024.10.04 |
|
| minimind-v-v1-small (27M) | 2024.10.04 |
|
||||||
| minimind-v-v1 (109M) | 1.1 GB | 2024.10.04 |
|
| minimind-v-v1 (109M) | 2024.10.04 |
|
||||||
|
|
||||||
### 👉**Recent Updates**
|
### 👉**Recent Updates**
|
||||||
|
|
||||||
|
<details>
|
||||||
|
<summary> <b>2026-04-20</b> </summary>
|
||||||
|
|
||||||
|
- New checkpoints released: minimind-3v (67M) / minimind-3v-moe (201M-A67M)
|
||||||
|
- Projector: added `LayerNorm`, token merging switched to 2D pixel-shuffle
|
||||||
|
- Vision Encoder switched to `SiglipVisionModel` (fixed 256×256)
|
||||||
|
- Training data moved to ALLaVA-4V (Pretrain 1.27M / SFT 2.9M, merged into a single-stage SFT)
|
||||||
|
- Freeze strategy updated: `freeze_llm=1` unfreezes first + last layers; Pretrain/SFT defaults now `2`/`1`; `max_seq_len` 360 → 450
|
||||||
|
- Misc bugfixes and small tweaks
|
||||||
|
|
||||||
|
</details>
|
||||||
|
|
||||||
<details>
|
<details>
|
||||||
<summary> <b>2026-04-01</b> </summary>
|
<summary> <b>2026-04-01</b> </summary>
|
||||||
|
|
||||||
- Added minimind-3v (67M) and minimind-3v-moe (201M-A67M) models
|
- Added minimind-3v (67M) and minimind-3v-moe (201M-A67M) models
|
||||||
- Unified 768+8 architecture, supporting both dense and moe modes
|
- Unified 768+8 architecture, supporting both dense and moe modes
|
||||||
- Switched Visual Encoder from CLIP to SigLIP2 (siglip2-base-p16-ve)
|
- Switched Visual Encoder from CLIP to SigLIP2 (siglip2-base-p16-256-ve)
|
||||||
- Replaced QFormer with MLP Projection + reshape compression
|
- Replaced QFormer with MLP Projection + reshape compression
|
||||||
- Dataset format updated to parquet, mixed data sources, updated tokenizer with image placeholder `<|image_pad|>`, new WebUI with dynamic model directory scanning and dropdown model switching
|
- Dataset format updated to parquet, mixed data sources, updated tokenizer with image placeholder `<|image_pad|>`, new WebUI with dynamic model directory scanning and dropdown model switching
|
||||||
- Model code refactored, LLM/VLM unified for Transformers format
|
- Model code refactored, LLM/VLM unified for Transformers format
|
||||||
|
|
@ -100,19 +110,16 @@ Is the training process difficult? Now, let's explore the answers and feel the j
|
||||||
|
|
||||||
</details>
|
</details>
|
||||||
|
|
||||||
<details>
|
|
||||||
<summary> <b>2025-04-27</b> </summary>
|
|
||||||
|
|
||||||
- Compatibility updates
|
|
||||||
- Adapted to the new feature in the "minimind" repository
|
|
||||||
- Standardized parts of the code
|
|
||||||
|
|
||||||
</details>
|
|
||||||
|
|
||||||
<details>
|
<details>
|
||||||
|
|
||||||
<summary> <b>More...</b> </summary>
|
<summary> <b>More...</b> </summary>
|
||||||
|
|
||||||
|
**2025-04-27**
|
||||||
|
|
||||||
|
- Compatibility updates
|
||||||
|
- Adapted to [MiniMind repository new features](https://github.com/jingyaogong/minimind/issues/370)
|
||||||
|
- Code normalization
|
||||||
|
|
||||||
**2025-02-20**
|
**2025-02-20**
|
||||||
|
|
||||||
- MiniMind2-V updated alongside MiniMind2
|
- MiniMind2-V updated alongside MiniMind2
|
||||||
|
|
@ -151,9 +158,9 @@ git clone --depth 1 https://github.com/jingyaogong/minimind-v
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# Download the siglip2 model to the ./model directory
|
# Download the siglip2 model to the ./model directory
|
||||||
git clone https://huggingface.co/jingyaogong/siglip2-base-p16-ve
|
git clone https://huggingface.co/jingyaogong/siglip2-base-p16-256-ve
|
||||||
# or
|
# or
|
||||||
git clone https://modelscope.cn/models/gongjy/siglip2-base-p16-ve
|
git clone https://modelscope.cn/models/gongjy/siglip2-base-p16-256-ve
|
||||||
```
|
```
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
|
|
@ -231,40 +238,42 @@ and place it under `./dataset`.
|
||||||
|
|
||||||
**[Note 2]** Parquet is a columnar storage format supporting efficient compression and fast reading. To preview data content, run `python lm_dataset.py` in the `dataset/` directory to visualize the first 5 image-text pairs.
|
**[Note 2]** Parquet is a columnar storage format supporting efficient compression and fast reading. To preview data content, run `python lm_dataset.py` in the `dataset/` directory to visualize the first 5 image-text pairs.
|
||||||
|
|
||||||
Pretrain data:
|
Pretrain data (optional; contains caption subset only):
|
||||||
```bash
|
```bash
|
||||||
wget https://hf-mirror.com/datasets/jingyaogong/minimind-v_dataset/resolve/main/pretrain_i2t.parquet
|
wget https://hf-mirror.com/datasets/jingyaogong/minimind-v_dataset/resolve/main/pretrain_i2t.parquet
|
||||||
```
|
```
|
||||||
|
|
||||||
SFT data:
|
SFT data (required; contains full caption + instruct + text merge):
|
||||||
```bash
|
```bash
|
||||||
wget https://hf-mirror.com/datasets/jingyaogong/minimind-v_dataset/resolve/main/sft_i2t.parquet
|
wget https://hf-mirror.com/datasets/jingyaogong/minimind-v_dataset/resolve/main/sft_i2t.parquet
|
||||||
```
|
```
|
||||||
|
|
||||||
Please reserve about ~2GB of space for the dataset. If there is insufficient space for pretrain data, you can try skipping the pretrain training step and proceed directly to SFT training.
|
The single `sft_i2t.parquet` file (2.9M rows) absorbs Pretrain as a subset, and after global dictionary encoding dedup it is only ~10% larger than the original SFT — enough to cover every training stage.
|
||||||
|
|
||||||
</details>
|
</details>
|
||||||
|
|
||||||
### 3' Start Training
|
### 3' Start Training
|
||||||
|
|
||||||
**3.1 Pretraining (Learning image description)**
|
**3.1 Pretrain (optional)**
|
||||||
|
|
||||||
|
> `sft_i2t.parquet` already contains the Pretrain data as a subset, so **this stage can be skipped** and you may go directly to SFT with `--from_weight llm`. If you prefer the Projector to be pre-aligned first and SFT to converge more stably, run a round of Pretrain separately.
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# Basic training command (start from LLM weights, train vision_proj only)
|
# Basic training command (start from LLM weights, train vision_proj only)
|
||||||
python train_pretrain_vlm.py --epochs 4 --from_weight llm
|
python train_pretrain_vlm.py --epochs 2 --from_weight llm
|
||||||
```
|
```
|
||||||
|
|
||||||
> Run pretraining to get `pretrain_vlm_*.pth` as the pretrained model's output weights (* represents the model
|
> Run Pretrain to get `pretrain_vlm_*.pth` as the Pretrain output weights (* is the model dimension, default 768).
|
||||||
> dimension, default is 768).
|
|
||||||
|
|
||||||
**3.2 Supervised Fine-Tuning (Learning image-caption dialogue style)**
|
**3.2 SFT (required)**
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# Basic training command (start from pretrain weights, full parameter fine-tuning)
|
# Basic command (default --freeze_llm 1: unfreeze proj + first/last LLM layers, keep middle layers' original LLM weights)
|
||||||
|
# If 3.1 was run: --from_weight pretrain_vlm; if Pretrain was skipped: --from_weight llm
|
||||||
python train_sft_vlm.py --epochs 2 --from_weight pretrain_vlm
|
python train_sft_vlm.py --epochs 2 --from_weight pretrain_vlm
|
||||||
```
|
```
|
||||||
|
|
||||||
> Perform supervised fine-tuning to get `sft_vlm_*.pth` as the output weights for the fine-tuned model.
|
> Run SFT to get `sft_vlm_*.pth` as the SFT output weights.
|
||||||
|
|
||||||
<details>
|
<details>
|
||||||
<summary>Note: Training Details</summary>
|
<summary>Note: Training Details</summary>
|
||||||
|
|
@ -284,7 +293,7 @@ python train_sft_vlm.py --epochs 4 --from_resume 1
|
||||||
- `--from_weight`: base weight name (llm, pretrain_vlm, none, etc.)
|
- `--from_weight`: base weight name (llm, pretrain_vlm, none, etc.)
|
||||||
- `--save_weight`: save weight prefix name
|
- `--save_weight`: save weight prefix name
|
||||||
- `--from_resume`: whether to resume training (0=start from scratch, 1=continue from checkpoint)
|
- `--from_resume`: whether to resume training (0=start from scratch, 1=continue from checkpoint)
|
||||||
- `--freeze_llm`: whether to freeze LLM parameters (pretrain use only)
|
- `--freeze_llm`: freezing strategy (0=all trainable, 1=proj + first/last LLM layers, 2=proj only). Default 2 for Pretrain, 1 for SFT
|
||||||
- More details can be found in the code
|
- More details can be found in the code
|
||||||
|
|
||||||
</details>
|
</details>
|
||||||
|
|
@ -345,17 +354,11 @@ and `wandb_run_name` parameters.
|
||||||
|
|
||||||
# 📌 VLM Detail
|
# 📌 VLM Detail
|
||||||
|
|
||||||
The base language model of MiniMind-V (VLM), MiniMind (LLM), comes from the twin
|
The language backbone of MiniMind-V is the `llm_768.pth` trained by the sibling project [minimind](https://github.com/jingyaogong/minimind). The LLM's own structure, training details and experimental analysis are not repeated here; the default assumption is that the reader has a basic understanding of MiniMind LLM. Not having touched the LLM project does not prevent following the "Quick Start" to get MiniMind-V running — the flow is self-contained.
|
||||||
project [minimind](https://github.com/jingyaogong/minimind). For detailed information on the model structure, training
|
|
||||||
specifics, principles, and testing results, please refer to the [minimind](https://github.com/jingyaogong/minimind)
|
|
||||||
project. To reduce redundancy, the discussion on LLM-related topics is omitted here, assuming you have a basic
|
|
||||||
understanding of MiniMind (LLM).
|
|
||||||
|
|
||||||
> Even if you are not very familiar with the details of LLMs, you can still follow the "Quick Start" guide to train a
|
The two shorthand labels on the landing page — "from scratch" and "67M" — also need a stricter reading here. "From scratch" means the VLM itself is trained from zero (Projection randomly initialized, first/last LLM layers fine-tuned for alignment), but the LLM backbone is not pretrained from zero — it is continued from the weights of MiniMind. For a strictly "from-zero pretraining" setup, first pretrain an LLM in MiniMind and then plug it back here. "67M" refers to the trainable backbone (LLM ~64M + Projection ~3M); the SigLIP2 vision encoder contributes another ~93M parameters that stay frozen throughout and serve only as an image feature extractor, so the full model at inference time is roughly 160M (dense) / 294M (MoE).
|
||||||
> MiniMind-V, as it remains unaffected and the repository focuses on the lowest cost for out-of-the-box use!
|
|
||||||
|
|
||||||
MiniMind-V's structure adds two submodules, a Visual Encoder and a feature projection, with a modality-mixing branch to
|
The VLM adds a Visual Encoder and a feature projection on top of the LLM, introducing a modality-mixing branch to support multimodal inputs:
|
||||||
support inputs from multiple modalities:
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
@ -412,16 +415,13 @@ outputs the end token; here, the "token" doesn’t necessarily have to be text!
|
||||||
The "foreign language dictionary" is referred to as the Visual Encoder model.
|
The "foreign language dictionary" is referred to as the Visual Encoder model.
|
||||||
Like LlaVA, Qwen-VL, and other visual language models, MiniMind-V now uses the open-source SigLIP2 series models as the
|
Like LlaVA, Qwen-VL, and other visual language models, MiniMind-V now uses the open-source SigLIP2 series models as the
|
||||||
Visual Encoder.
|
Visual Encoder.
|
||||||
Specifically, we use [siglip2-base-p16-ve](https://huggingface.co/jingyaogong/siglip2-base-p16-ve), a Visual
|
Specifically, we use [siglip2-base-p16-256-ve](https://huggingface.co/jingyaogong/siglip2-base-p16-256-ve), a Visual
|
||||||
Encoder based on the ViT-B/16 architecture for describing image-text information.
|
Encoder based on the ViT-B/16 architecture for describing image-text information.
|
||||||
The current SigLIP2 NaFlex vision encoder generates up to 256 patch tokens from the processor output as the input to the
|
The current SigLIP2 NaFlex vision encoder generates up to 256 patch tokens from the processor output as the input to the
|
||||||
encoder layer, which produces a 1×768 dimensional embedding vector for calculating error with the text.
|
encoder layer, which produces a 1×768 dimensional embedding vector for calculating error with the text.
|
||||||
We don’t need the final embedding representation, so we only take the output from the encoder layer, which is the output
|
We don’t need the final embedding representation, so we only take the output from the encoder layer, which is the output
|
||||||
feature from the core ViT backbone.
|
feature from the core ViT backbone.
|
||||||
It receives 256×768 features from the previous layer, which are then reshaped by concatenating every 4 adjacent tokens into 1 (256×768 → 64×3072), then projected to the LLM's hidden dimension via a 2-layer MLP (Linear→GELU→Linear), resulting in 64 visual tokens input into MiniMind-V.
|
It receives 256×768 features from the previous layer, which are then reshaped by concatenating every 4 adjacent tokens into 1 (256×768 → 64×3072), then projected to the LLM's hidden dimension via a 2-layer MLP (Linear→GELU→Linear), resulting in 64 visual tokens fed into MiniMind-V — this step is exactly cross-modal feature alignment: the native visual features are brought into the semantic space where text tokens live, so that the two can interact in the same space.
|
||||||
After obtaining the image encoder features, the integration with the LLM requires aligning the visual features to the LLM's text token dimension, and mapping the image features into the same space as text embeddings. In other
|
|
||||||
words, the image features and native visual tokens cannot be directly treated the same; they require cross-modal feature
|
|
||||||
alignment.
|
|
||||||
|
|
||||||
[LlaVA-1](https://arxiv.org/pdf/2304.08485) achieves good alignment with a simple linear transformation, [LlaVA-1.5](https://arxiv.org/pdf/2310.03744) upgrades to a 2-layer MLP. MiniMind-V adopts the same MLP Projection approach as LlaVA-1.5, combined with reshape for token compression.
|
[LlaVA-1](https://arxiv.org/pdf/2304.08485) achieves good alignment with a simple linear transformation, [LlaVA-1.5](https://arxiv.org/pdf/2310.03744) upgrades to a 2-layer MLP. MiniMind-V adopts the same MLP Projection approach as LlaVA-1.5, combined with reshape for token compression.
|
||||||
|
|
||||||
|
|
@ -462,63 +462,83 @@ At this point, all the details of `MiniMind-V` have been presented. The VLM mode
|
||||||
|
|
||||||
## Ⅰ Dataset
|
## Ⅰ Dataset
|
||||||
|
|
||||||
Original Source:
|
All image-text data used in this round come from the [ALLaVA-4V](https://huggingface.co/datasets/FreedomIntelligence/ALLaVA-4V) family.
|
||||||
- [Chinese-LLaVA-Vision](https://huggingface.co/datasets/LinkSoul/Chinese-LLaVA-Vision-Instructions): Contains approximately 570,000 pre-trained images from CC-3M and COCO 2014
|
Compared with data stitched together from earlier LLaVA-derived sets, ALLaVA-4V is more consistent in quality, natively paired in Chinese and English, and more thorough in its fine-grained descriptions.
|
||||||
- [llava-en-zh-300k](https://huggingface.co/datasets/BUAADreamer/llava-en-zh-300k): Contains 300k instruction fine-tuning data and 150k images
|
It is composed of two image sources: a curated subset of LAION (mostly natural images) and a curated subset of VFLAN (documents, charts, synthetic scenes).
|
||||||
- [LLaVA-SFT-665K](https://huggingface.co/datasets/csuhan/LLaVA-SFT-665K): Contains 665k instruction fine-tuning data
|
|
||||||
|
|
||||||
The dataset contains both Chinese and English data. The Q&A content has been translated, with better support for Chinese, further organized and resized (pretrain resolution 128×128, sft resolution 160×160).
|
- **Pretrain (`pretrain_i2t.parquet`, ~1.27M rows / ~640K unique images)**
|
||||||
|
- `ALLaVA-Caption-LAION-4V` en/zh: ~470K + ~440K
|
||||||
|
- `ALLaVA-Caption-VFLAN-4V` en/zh: ~195K + ~170K
|
||||||
|
- Single-turn "describe this image" style captions used to establish the basic alignment from visual tokens to language tokens.
|
||||||
|
|
||||||
(pretrain_i2t.parquet) Pre-training dataset format:
|
- **SFT (`sft_i2t.parquet`, ~2.90M rows / ~650K unique images)**
|
||||||
|
- `ALLaVA-Instruct-LAION-4V` en/zh: ~470K + ~470K
|
||||||
|
- `ALLaVA-Instruct-VFLAN-4V` en/zh: ~195K + ~165K
|
||||||
|
- `Instruct-LAION-4v-gemini-claude-ensembled` (synthesized by Gemini/Claude): ~50K
|
||||||
|
- `Instruct-LAION-4oiterative` (iteratively refined by GPT-4o): ~50K
|
||||||
|
- Pure-text conversations (8×8 black placeholder images, preserving base language ability): ~230K
|
||||||
|
- **Full Pretrain caption data merged in** (same source as pretrain, ~99% image overlap): ~1.27M
|
||||||
|
- A blend of "image-grounded reasoning Q&A", "caption-style long descriptions" and "pure-text chat" — covering fine-grained follow-ups/long chains of thought as well as image description and general language ability.
|
||||||
|
|
||||||
|
Roughly 2.9M samples in total. The Pretrain stage can be skipped entirely (SFT has absorbed it as a subset). Chinese and English are roughly balanced.
|
||||||
|
Given MiniMind-V's trainable backbone is only 67M, mixing English and Chinese is a pragmatic choice: Chinese data helps native-language generation, while the original English descriptions tend to be more precise — the two complement each other.
|
||||||
|
|
||||||
|
All images are `resize`d to **256×256** (matching SigLIP2 NaFlex's 256 patch-token input) and re-encoded as JPEG, packed directly into parquet.
|
||||||
|
|
||||||
|
(`pretrain_i2t.parquet`) Pre-training dataset format:
|
||||||
|
|
||||||
```text
|
```text
|
||||||
Columns: conversations (json string), image_bytes (binary), image_names (string)
|
Columns: conversations (json string), image_bytes (binary)
|
||||||
|
|
||||||
conversations example:
|
conversations example:
|
||||||
[
|
[
|
||||||
{"role": "user", "content": "Provide a brief description of the given image.\n<image>"},
|
{"role": "user", "content": "<image>\nPlease describe this image in detail."},
|
||||||
{"role": "assistant", "content": "Olive oil is a healthy ingredient for free use."}
|
{"role": "assistant", "content": "The image shows..."}
|
||||||
]
|
]
|
||||||
image_bytes: <binary image data>
|
image_bytes: <binary image data>
|
||||||
```
|
```
|
||||||
|
|
||||||
(sft_i2t.parquet) Single image instruction fine-tuning dataset format:
|
(`sft_i2t.parquet`) Single-image SFT dataset format:
|
||||||
|
|
||||||
```text
|
```text
|
||||||
Columns: conversations (json string), image_bytes (binary), image_names (string)
|
Columns: conversations (json string), image_bytes (binary)
|
||||||
|
|
||||||
conversations example:
|
conversations example:
|
||||||
[
|
[
|
||||||
{"role": "user", "content": "What impact does the location of the alarm clock have on sleep quality?<image>"},
|
{"role": "user", "content": "Based on the image, what time of day is it?<image>"},
|
||||||
{"role": "assistant", "content": "Place the digital alarm clock on the nightstand..."}
|
{"role": "assistant", "content": "Judging from the light and shadows..."}
|
||||||
]
|
]
|
||||||
image_bytes: <binary image data>
|
image_bytes: <binary image data>
|
||||||
```
|
```
|
||||||
|
|
||||||
> Note: sft_i2t.parquet contains ~580K samples in total, of which ~236K are image-text conversations (i2t) and ~346K are pure text conversations (t2t). The latter is used to preserve the model's base language capabilities.
|
> Note: sft_i2t.parquet contains ~2.9M samples, of which ~1.40M are image instruct conversations, ~1.27M are image caption descriptions (merged from Pretrain), and ~230K are pure-text conversations (t2t, image column filled by an 8×8 black placeholder) used to preserve the model's base language capabilities. Since Pretrain is already included as a subset, the Pretrain stage can be skipped and SFT run directly.
|
||||||
|
|
||||||
Dataset download
|
Dataset download link: ([ModelScope](https://www.modelscope.cn/datasets/gongjy/minimind-v_dataset) | [HuggingFace](https://huggingface.co/datasets/jingyaogong/minimind-v_dataset))
|
||||||
link: ([ModelScope](https://www.modelscope.cn/datasets/gongjy/minimind-v_dataset) | [HuggingFace](https://huggingface.co/datasets/jingyaogong/minimind-v_dataset))
|
|
||||||
|
|
||||||
## Ⅱ Training
|
## Ⅱ Training
|
||||||
|
|
||||||
Training is divided into two stages, both freezing the Visual Encoder gradients and only training the Projection and LLM parts.
|
Training has two stages (Pretrain optional; SFT required). Both freeze the Visual Encoder and train only the Projection and part of the LLM layers.
|
||||||
Training is initialized from LLM pre-trained weights, with support for DDP multi-GPU training, mixed precision (bfloat16), torch.compile acceleration, and swanlab logging.
|
Training is initialized from LLM Pretrain weights, with support for DDP multi-GPU training, mixed precision (bfloat16), torch.compile acceleration, and swanlab logging.
|
||||||
|
|
||||||
> train_pretrain_vlm
|
> train_pretrain_vlm (optional)
|
||||||
|
|
||||||
The pre-training stage learns general image knowledge from ~1.13M image-text description pairs (e.g., a deer is a deer, a dog is a dog).
|
The Pretrain stage learns general image knowledge from ~1.27M image-text description pairs (e.g., a deer is a deer, a dog is a dog).
|
||||||
This stage uses a higher learning rate (1e-4), max sequence length of 360, freezes the LLM main parameters, and only sets the Projection and LLM's layer 0 as learnable,
|
It uses a higher learning rate (~4e-4), max sequence length 450, and **fully freezes the LLM and Visual Encoder, training only the Projection** (`--freeze_llm 2`).
|
||||||
aiming to quickly establish a basic mapping from visual features to the language space while avoiding damage to the LLM's existing language capabilities.
|
The goal is to let the Projector cleanly align visual tokens to the language space without perturbing the original LLM weights.
|
||||||
|
Since SFT data already contains all Pretrain samples as a subset, this stage is optional; skipping it saves time, while **running one round of Pretrain first lets the Projector pre-align and makes SFT converge more steadily**.
|
||||||
|
|
||||||
> train_sft_vlm
|
> train_sft_vlm
|
||||||
|
|
||||||
The instruction fine-tuning stage learns real Q&A formats from ~580K samples, of which ~236K are image-text multi-turn conversations and ~346K are pure text conversations (to preserve LLM base capabilities).
|
The SFT stage uses the aforementioned `sft_i2t.parquet` — about 2.9M mixed samples, covering the image captions inherited from Pretrain, reasoning-style Q&A on natural images, fine-grained Q&A on documents/charts, instructions synthesized by Gemini/Claude/GPT-4o, plus ~230K pure-text conversations (image column filled by an 8×8 black placeholder). Learning rate drops to ~5e-5, max sequence 768.
|
||||||
This stage uses a lower learning rate (1e-5~1e-6), max sequence length of 768, unfreezes all Projection and LLM parameters for full fine-tuning,
|
|
||||||
enabling the model to conduct multi-turn conversations based on image content, while mitigating catastrophic forgetting through the mixed-in pure text data.
|
A common practice is to fully unfreeze the LLM during SFT, but this usually assumes a several-B-parameter base and a substantial amount of pure-text data mixed into SFT. MiniMind-V's language backbone is only 64M and ~92% of the current SFT data is image-related, so fully unfreezing the LLM would likely dilute its original general-language capability under the image-task gradients.
|
||||||
|
|
||||||
|
We therefore use `--freeze_llm 1`: **only the Projection and the first & last LLM layers are unfrozen, while the remaining N-2 layers keep their Pretrain weights**. The first layer is the first processing stage after visual tokens enter the LLM and thus bears the cross-modal fusion; the last layer shapes the format and style of the answer; the middle layers retain the knowledge from LLM Pretrain and are not overwritten by image-task gradients. The ~230K pure-text samples further act as a regularizer for general-language capability.
|
||||||
|
|
||||||
> Training Time and Loss Trend (for reference only)
|
> Training Time and Loss Trend (for reference only)
|
||||||
|
|
||||||
|
On a single NVIDIA 3090, SFT takes ~2 hours per `epoch` in practice; dense and MoE finish in similar time (activated parameters are on the same order, with the gap mostly coming from the extra memory traffic of expert routing). Pretrain data volume is ~45% of SFT's, so one Pretrain epoch can be roughly scaled by that ratio. At a typical cloud price of ~1.5 RMB/hour for a 3090, a full SFT round costs about 3 RMB.
|
||||||
|
|
||||||
Pretrain [768+8] (dense & moe)
|
Pretrain [768+8] (dense & moe)
|
||||||
|
|
||||||
|
|
||||||
|
|
@ -532,13 +552,13 @@ SFT [768+8] (dense & moe)
|
||||||
| Native PyTorch (`*.pth`) | [minimind-3v-pytorch](https://www.modelscope.cn/models/gongjy/minimind-3v-pytorch) | [minimind-3v-pytorch](https://huggingface.co/jingyaogong/minimind-3v-pytorch) |
|
| Native PyTorch (`*.pth`) | [minimind-3v-pytorch](https://www.modelscope.cn/models/gongjy/minimind-3v-pytorch) | [minimind-3v-pytorch](https://huggingface.co/jingyaogong/minimind-3v-pytorch) |
|
||||||
| Transformers | [minimind-v collection](https://modelscope.cn/collections/MiniMind-V-42b841dde22d41) | [minimind-v collection](https://huggingface.co/collections/jingyaogong/minimind-v-67000833fb60b3a2e1f3597d) |
|
| Transformers | [minimind-v collection](https://modelscope.cn/collections/MiniMind-V-42b841dde22d41) | [minimind-v collection](https://huggingface.co/collections/jingyaogong/minimind-v-67000833fb60b3a2e1f3597d) |
|
||||||
|
|
||||||
> Note: The Transformers version is the `MiniMind-V` model after single-image instruction fine-tuning
|
> Note: The Transformers version is the `MiniMind-V` model after single-image SFT
|
||||||
|
|
||||||
# 📌 Test
|
# 📌 Test
|
||||||
|
|
||||||
### Effect Test
|
### Effect Test
|
||||||
|
|
||||||
#### Single Image Dialogue
|
Prompt: `<image>\nPlease illustrate the image through your words.`
|
||||||
|
|
||||||
<table>
|
<table>
|
||||||
<thead>
|
<thead>
|
||||||
|
|
@ -551,92 +571,62 @@ SFT [768+8] (dense & moe)
|
||||||
<tbody>
|
<tbody>
|
||||||
<tr>
|
<tr>
|
||||||
<td>
|
<td>
|
||||||
<img src="./dataset/eval_images/airplane-flying-blue-sky.jpg" alt="airplane">
|
<img src="./dataset/eval_images/image-01-golden-dog-balloons.jpg" alt="golden-dog">
|
||||||
|
|
||||||
</td>
|
</td>
|
||||||
<td>The image features a white and black airplane parked on a grassy terrain. The layers of a building are likely to be filled with air traffic control, such as the building's tall building, the large building, or the overall pavement. There are also two airplanes parked in the background. The airplanes are displayed on a board, and the airplane is flying through the air while the black and white airplane is parked on the ground.</td>
|
<td>The image shows a golden-brown dog in a park setting with a clear blue sky above. The dog appears to be a golden-brown dog with a cheerful expression, which is characteristic of a Golden Bay breed. The dog's fur is a lighter color, contrasting with the golden balloon, which is likely a balloon with a bright yellow color. The dog's tail is bald, and its tail is darker, indicating it might be a wild, possibly a festive holiday. The dog's bright yellow balloon, along with its tail, stands out against the lighter background of the park. There are no other people or identifying marks in the immediate vicinity of the dog, emphasizing its celebration and a sense of joy.</td>
|
||||||
<td>The image features a large jetliner with a large airplane sitting on the ground. It is likely an airplane or an aircraft, possibly a flight jet or a runway.</td>
|
<td>The image depicts a golden-brown dog with a golden-brown coat, likely a Border Collie, standing on a grassy field with a clear blue sky. The dog appears to be a German Shepherd, given its size and the positioning of its tail. Its ears are perked up, and it has a golden-brown coat with a lighter golden hue. The dog's eyes are looking forward, and it seems to be looking down at it. The sky is partly cloudy, suggesting a fair weather day. The dog is standing on a grassy field, with some trees and others in the distance, indicating a calm and natural setting. The overall mood of the image is one of tranquility and enjoyment.</td>
|
||||||
</tr>
|
</tr>
|
||||||
<tr>
|
<tr>
|
||||||
<td>
|
<td>
|
||||||
<img src="./dataset/eval_images/birthday-cake-candles-table.jpg" alt="birthday-cake">
|
<img src="./dataset/eval_images/image-02-rainbow-umbrella-street.jpg" alt="rainbow-umbrella">
|
||||||
|
|
||||||
</td>
|
</td>
|
||||||
<td>The image features a white cake in an old-fashioned cake with cake placed on it. It is surrounded by a few cooked ingredients, including the wedding cake.</td>
|
<td>The image shows a rainy day with a rainbow-colored umbrella. The umbrella has a colorful mottled pattern of colorful, orange, yellow, red, green, purple, and blue. The umbrella is surrounded by a bright, irregular, bright daylight with irregular, warm lighting. The umbrella's bright colors range from pink, yellow, purple, green, blue, and red, creating a cool, wet atmosphere. The rainbow-colored umbrella is lit by the bright sky, indicating a sunny day. The umbrella is slightly warmer than the rainbow-colored umbrella, with a reflective surface and a blurred background that suggests a shaded area. The umbrella is in sharp focus, allowing the umbrella to be visible on the scene. The overall impression is one of a tropical or subtropical environment, with the rainbow-colored umbrella in a vibrant green color, creating a sense of depth and natural beauty.</td>
|
||||||
<td>The image features a white cake in an old-fashioned cake with cake placed on it. It is surrounded by a few cooked ingredients, including the wedding cake.</td>
|
<td>The image shows an outdoor scene with a rainy umbrella in the background, likely during a time of day when umbrellas are illuminated. The umbrellas are arranged in a turn, creating a dynamic and colorful umbrella. The umbrellas have a smooth, flat surface with some visible wrinkles and grouting, indicating a high level of detail. The umbrellas are capped with a slightly darker pink hue, providing a lighter gray color that contrasts with the umbrellas. They are positioned in the distance, adding to the umbrella's appearance. The umbrellas are the main subjects, and the umbrellas are positioned in the distance.</td>
|
||||||
</tr>
|
</tr>
|
||||||
<tr>
|
<tr>
|
||||||
<td>
|
<td>
|
||||||
<img src="./dataset/eval_images/pizza-on-wooden-board.jpg" alt="pizza">
|
<img src="./dataset/eval_images/image-03-cherry-blossom-bike.jpg" alt="cherry-blossom-bike">
|
||||||
|
|
||||||
</td>
|
</td>
|
||||||
<td>The image depicts a delicious pizza pizza with fresh toppings, which are likely present in a slice of pizza. The pizza is perfectly crispy, as it has a crispy crust and slightly crispy, making it a delightful pizza presentation. The pizza is filled with fresh toppings, adding to the crispy crust. The pizza is also a bit scrambled, as it has a fresh topping, while the crispy crust is cooked with a pizza pan. The pizza is likely to be a pizza with its crunchy texture and flavorfully. The pizza is also a popular choice for pizza with others, and it is a filling and crispy crust.</td>
|
<td>The image depicts a vibrant pink bicycle, with its wicker basket attached to the bicycle. The bicycle's design is vibrant, with a sleek design that complements the vibrant colors of the pink and white flowers. The basket appears to be made of a material with a durable fabric, and it has a low-profile design, with a slight front wheel and a single rider, which are often used for pedestrian traffic. The bicycle's frame is not fully visible, but it has a black frame and a black handle. There are no visible people or distinctive features that suggest a specific location or event, such as a bike rack or a bike rack, which are not present in this image. The pink bicycle's body is partially visible and has a single seat, suggesting it is likely a bicycle. There are no discernible background elements that provide context or distraction from the image.</td>
|
||||||
<td>The image features a scenic burning pizza in a pasta-style board, surrounded by a wooden cabinet, a garner topping for cheese and vegetables. There is a small cabinet nearby, with a pan-familt, a pizza flatter topping. The pizza is situated on the left side of the board, with a bowl of olives placed on top of the side.</td>
|
<td>The image depicts a bright pink bicycle, captured from a perspective view from an elevated perspective. The bicycle is an iconic symbol of the bike, with its rider sporting a touch of white, white, and a touch of purple. The bike is a traditional style with a simple bike and a basketball hoop. It's a sunny day, and the lighting suggests it might be daytime. The scene is set against a pastoral scene, with the pathways leading towards a pathway that casts a warm glow, indicating the serene and warm atmosphere of the scene. The colors in the image are vibrant, with the pink of the pathway standing out against the pale pink of the bicycle. There's a sense of tranquility and freedom of movement.</td>
|
||||||
</tr>
|
</tr>
|
||||||
<tr>
|
<tr>
|
||||||
<td>
|
<td>
|
||||||
<img src="./dataset/eval_images/red-sports-car-road.jpg" alt="red-car">
|
<img src="./dataset/eval_images/image-04-yellow-car.jpg" alt="yellow-car">
|
||||||
|
|
||||||
</td>
|
</td>
|
||||||
<td>The image shows a yellow yellow and red turf coastline on a street, with a couple of cars and a yellow and red traffic lights in the background.</td>
|
<td>The image shows a yellow sports car positioned on a road with a clear blue sky. The car is positioned on a road with a prominent yellow roof, which is characteristic of a yellow sports car. The car is positioned in the front of the car, facing the camera, with its front grille and lower body lines visible, which suggests a sporty aesthetic. The car is parked on a road that appears to be a high-performance sports car, and the background is a parking lot with a few small boats, which is a common feature in such vehicles. There are no visible windows or other objects in the immediate vicinity of the car.</td>
|
||||||
<td>The image features a couple of female vanity van lying on a car. The car is situated on the ground, surrounded by a wet furniture. The couple is seen in the middle of the car, possibly observing the scene.</td>
|
<td>The image features a bright yellow sports car parked on a sidewalk. The car is a paved road with a road that appears to be a parking lot, and the sidewalk is bordered by a thin line of trees and a clear blue sky. There are no visible passengers, and the car is parked on a road with a few cars, indicating it is likely a busy day. The car has a sleek design with a high-performance lines and a large round headlamps.</td>
|
||||||
</tr>
|
</tr>
|
||||||
<tr>
|
<tr>
|
||||||
<td>
|
<td>
|
||||||
<img src="./dataset/eval_images/row-of-colorful-houses.jpg" alt="colorful-houses">
|
<img src="./dataset/eval_images/image-05-superhero-rooftop.jpg" alt="superhero">
|
||||||
|
|
||||||
</td>
|
</td>
|
||||||
<td>IoT tree is a holiday that is often associated with Christmas culture, history, and celebration. The tree has a unique black and white striped pattern, which features a sweet treat, a budget-friendly chocolate cake with brown spots. The cake is burnt and has a balcony with a sweet treat, with the rich, vibrant colors of the striped pattern. The tree has a rich and burnt color, while the rich and vibrant colors are visually appealing. The tree has a striped pattern, which adds to the overall atmosphere of the image.</td>
|
<td>The image shows a stylized silhouette of a character from a superhero film series. The character is wearing a long-sleeved blue suit with a blue collar and a red band on the left chest, paired with red trousers. The suit is equipped with a blue suit and a red band on top, and the suit is adorned with a red band, which is also in the midst of a superhero character. The suit is standing outdoors, and the character is on a city street, with a large, blue suit standing out. The sky is a gradient of warm colors, suggesting it is either dawn or dusk. The overall impression is one of a high-energy, urban nighttime scene, with a focus on the character and the character.</td>
|
||||||
<td>The image is a colorful scene of a colorful vintage house, with a large pink roof of the wall and a red color scheme. The colorful vintage house has a pink color, and the color scheme appears to be fine, with a red color scheme.</td>
|
<td>The image depicts a spaceship scene at what appears to be a cityscape during what appears to be sunset. The scene is dominated by the sky, with the silhouettes of a figure standing on a raised platform, which is the central subject of the image. The figure is dressed in a red and blue suit with a white shirt, which suggests a style commonly associated with the "UNITED SUNDER" superhero franchise. The suit is primarily red with a red hue and a white shirt. There is a figure with a red belt on the right, holding a red object that could be a shield or a wrench. The person is standing on a street with a couple of buildings, indicating that the location might be a city with buildings that are part of the UNITED SUNDER. The sky is filled with a warm orange hue, suggesting a sunset. The overall atmosphere is one of dynamism and fantasy.</td>
|
||||||
</tr>
|
</tr>
|
||||||
<tr>
|
<tr>
|
||||||
<td>
|
<td>
|
||||||
<img src="./dataset/eval_images/snow-mountain-lake-view.jpg" alt="snow-mountain">
|
<img src="./dataset/eval_images/image-06-racecar-drift.jpg" alt="racecar">
|
||||||
|
|
||||||
</td>
|
</td>
|
||||||
<td>The image features a snowy mountain surrounded by a large, dense mountains. The large body of water is quite dense, with the body of water being hot and the water is sandy. The tall trees are swirls, and the tall trees are standing on the snow-covered ground. The body of water is also sink, creating a savanna-like appearance.</td>
|
<td>The image captures a dynamic scene at what appears to be a race. In the foreground, a red sports car is mounted on a racetrack, its front face is turned slightly to the left. The car has a prominent rear wing and aerodynamic elements, which suggests a high-speed speed and aerodynamic profile. The track is adorned with multiple windows, one of which is visible, allowing a view of the ground level. The background shows a crowd of spectators, some dressed in red, while others are standing, possibly in the air, in the distance, indicating that this is likely the race of the race. The lighting is warm, and the ground is dim, which gives the racing a warm ambiance. There are no people visible in the image, and the image is taken at a time when the race is being captured and the car is in the air. The lighting conditions suggest that it is either early morning or late afternoon, as the sun is low on the horizon, casting a warm glow and creating a dramatic effect on the track.</td>
|
||||||
<td>The image is an image of a large mountain visible in a lake, surrounded by the idyllic mountains and the mountains. It appears to be a blanket in the ocean, with the river and idylis watching the sea. The mountains are lined with fresh sand and waves, adding a sense of tranquility to the scene.</td>
|
<td>The image captures a dynamic scene at what appears to be a high-speed race, likely during the golden hour, given the warm lighting and the soft shadows. The track is bathed in a soft glow, indicating the sun is high and possibly at the moment of the race. There are no visible smoke, orbits, or disturbances in the sky, suggesting the race is either during the golden hour or during the golden hour. The race is a sizable form of aerodynamic body, with the front wheel being more pronounced, indicating it may have been used for speed and possibly aerodynamics. The track is bordered by a blurred audience, which may suggest a focus on the race's grandeur or the driver's presence. The track itself has a glossy finish, with some areas showing darker lines and the surrounding structures, which could be part of a track or a similar event.</td>
|
||||||
</tr>
|
|
||||||
<tr>
|
|
||||||
<td>
|
|
||||||
<img src="./dataset/eval_images/street-food-hotpot-table.jpg" alt="street-food">
|
|
||||||
|
|
||||||
</td>
|
|
||||||
<td>The image shows a variety of cooking options, including baking meat, cupcakes, and spinach. The baked vegetables are rich in a variety of flavors, including grilled, sautéed, and baked vegetables. The presence of a baked vegetable with a variety of vegetables in different parts suggests a variety of options, including baking, cooking, and baking. The cooking process is highly recommended, with a mix of vegetables and baking times, making it an ideal choice for those who prefer a variety of cooking options. The cooking process is also highly compatible, with a variety of flavors and textures enjoying the cooking process.</td>
|
|
||||||
<td>The image features a variety of freshwater salads, glasses, and coworkers displayed on a table. There is a mix of freshwater ingredients, likely a bun, which can be seen in the menu. The freshwater ingredients are placed on the table, and there is a portion of the coworkers displayed in the middle of the room. There is also a bowl filled with various ingredients.</td>
|
|
||||||
</tr>
|
|
||||||
<tr>
|
|
||||||
<td>
|
|
||||||
<img src="./dataset/eval_images/three-kittens-basket.jpg" alt="kittens">
|
|
||||||
|
|
||||||
</td>
|
|
||||||
<td>The image features a table filled with people standing together, casing bars, and a pair of brown cats. The table is filled with pink and white cats.</td>
|
|
||||||
<td>The image is a black and white detail of a miscellaneous bunch of toys, which is likely to be a part of a group or a similar artistic field.</td>
|
|
||||||
</tr>
|
|
||||||
<tr>
|
|
||||||
<td>
|
|
||||||
<img src="./dataset/eval_images/tropical-beach-palm-tree.jpg" alt="tropical-beach">
|
|
||||||
|
|
||||||
</td>
|
|
||||||
<td>The image features a brown wooden coat.</td>
|
|
||||||
<td>The image shows a sandy beach with an umbrella on top of a chair, providing a visual appeal for people to sit on.</td>
|
|
||||||
</tr>
|
|
||||||
<tr>
|
|
||||||
<td>
|
|
||||||
<img src="./dataset/eval_images/yellow-school-bus-road.jpg" alt="school-bus">
|
|
||||||
|
|
||||||
</td>
|
|
||||||
<td>The image displays a group of people sitting on the bus. They are waiting to be cautious and attentive to their feet, which indicates they are likely to be cautious and followed by the bus.</td>
|
|
||||||
<td>The image features a large collection of school buses, a brick-and-middle bus, and a stack of cars visible in the background. The school bus is situated next to a school bus, and there are several people watching the bus. The school bus is visible in the background, with one person standing behind the other, while the other person is watching the bus. The bus is positioned behind the school bus, creating a seamless and dynamic visual effect.</td>
|
|
||||||
</tr>
|
</tr>
|
||||||
</tbody>
|
</tbody>
|
||||||
</table>
|
</table>
|
||||||
|
|
||||||
### Effect Summary:
|
### Effect Summary:
|
||||||
|
|
||||||
Both models can identify image subjects (airplane, cake, car, beach, etc.), but commonly exhibit repetitive expressions and hallucinated details. Limited by model and data scale, the overall performance is at a stage of "understanding the gist but inaccurate on details".
|
Both models can identify the primary subject in most images (dog, umbrella, bicycle, sports car, superhero, racecar, etc.), but both exhibit repetitive phrasing and hallucinated details, placing overall performance at a stage of "understanding the gist but inaccurate on details".
|
||||||
|
|
||||||
Visual signals are treated as a special foreign language by LLMs, so the "language learning" ability highly depends on the LLM's capacity. The stronger the LLM, the more powerful the corresponding VLM, and the performance boost becomes significant.
|
Across these 6 samples, the MoE variant produces slightly richer scene descriptions and better color recognition (e.g. the cherry-blossom scene, rainbow umbrella, yellow sports car), while the Dense model tends to be more concise. Over a broader evaluation set, however, MoE occasionally suffers from more pronounced entity-level hallucinations or repetitive generation, showing higher variance than Dense — making Dense the safer default configuration.
|
||||||
|
|
||||||
|
Visual signals act as a special "foreign language" to the LLM, so the ceiling of "learning that language" is bounded by the LLM's own language ability. A stronger backbone extracts more value from the same image-text data; swapping MiniMind-V's backbone for a several-B-scale LLM yields clearly sharper details and more coherent reasoning.
|
||||||
|
|
||||||
#### Future Areas for Improvement:
|
#### Future Areas for Improvement:
|
||||||
|
|
||||||
|
|
@ -650,10 +640,8 @@ Visual signals are treated as a special foreign language by LLMs, so the "langua
|
||||||
# 📌 Acknowledge
|
# 📌 Acknowledge
|
||||||
|
|
||||||
> [!TIP]
|
> [!TIP]
|
||||||
> If you find `MiniMind-V` helpful, please consider giving it a ⭐ on GitHub. <br/>
|
> If `MiniMind-V` is useful to you, a ⭐ on GitHub is welcome. <br/>
|
||||||
> Given the limited expertise, there may be unknown issues, and we welcome everyone to discuss, correct, or submit PRs
|
> Issues and PRs are the best place to share problems or improvements found while using the project.
|
||||||
> to improve the project in Issues. <br/>
|
|
||||||
> Your support is the driving force behind continuous improvements to the project. Thank you!
|
|
||||||
|
|
||||||
## 🤝 [Contributors](https://github.com/jingyaogong/minimind-v/graphs/contributors)
|
## 🤝 [Contributors](https://github.com/jingyaogong/minimind-v/graphs/contributors)
|
||||||
|
|
||||||
|
|
|
||||||
Binary file not shown.
|
Before Width: | Height: | Size: 13 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 23 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 28 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 32 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 41 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 25 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 25 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 24 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 48 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 24 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 40 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 48 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 49 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 28 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 45 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 31 KiB |
|
|
@ -118,9 +118,9 @@ class VLMDataset(Dataset):
|
||||||
if __name__ == '__main__':
|
if __name__ == '__main__':
|
||||||
import matplotlib.pyplot as plt; plt.rcParams['font.sans-serif'] = ['Arial Unicode MS', 'SimHei']
|
import matplotlib.pyplot as plt; plt.rcParams['font.sans-serif'] = ['Arial Unicode MS', 'SimHei']
|
||||||
for path in ['pretrain_i2t.parquet', 'sft_i2t.parquet']:
|
for path in ['pretrain_i2t.parquet', 'sft_i2t.parquet']:
|
||||||
t = pa.Table.from_batches(pq.ParquetFile(path).iter_batches()); fig, ax = plt.subplots(1, 5, figsize=(20, 4))
|
pf = pq.ParquetFile(path); n = pf.num_row_groups; t = pa.concat_tables([pf.read_row_group(i * n // 5).slice(0, 1) for i in range(5)]); fig, ax = plt.subplots(1, 5, figsize=(20, 4))
|
||||||
for i in range(5):
|
for i in range(5):
|
||||||
img_data = t['image_bytes'][i].as_py(); img_data = img_data[0] if isinstance(img_data, list) else img_data
|
img_data = t['image_bytes'][i].as_py(); img_data = img_data[0] if isinstance(img_data, list) else img_data
|
||||||
ax[i].imshow(Image.open(io.BytesIO(img_data))); ax[i].axis('off')
|
ax[i].imshow(Image.open(io.BytesIO(img_data))); ax[i].axis('off')
|
||||||
ax[i].set_title(json.loads(t['conversations'][i].as_py())[1]['content'][:30], fontsize=8)
|
ax[i].set_title(json.loads(t['conversations'][i].as_py())[1]['content'][:30], fontsize=8)
|
||||||
out = path.replace('.parquet', '_preview.png'); plt.savefig(out); print(f'已保存{out}, 共{len(t)}条')
|
out = path.replace('.parquet', '_preview.png'); plt.savefig(out); print(f'已保存{out}, 共{pf.metadata.num_rows}条')
|
||||||
|
|
|
||||||
15
eval_vlm.py
15
eval_vlm.py
|
|
@ -11,19 +11,19 @@ from trainer.trainer_utils import setup_seed, get_model_params
|
||||||
warnings.filterwarnings('ignore')
|
warnings.filterwarnings('ignore')
|
||||||
|
|
||||||
def init_model(args):
|
def init_model(args):
|
||||||
tokenizer = AutoTokenizer.from_pretrained(args.load_from)
|
tokenizer = AutoTokenizer.from_pretrained(args.load_from, trust_remote_code=True)
|
||||||
if 'model' in args.load_from:
|
if 'model' in args.load_from:
|
||||||
moe_suffix = '_moe' if args.use_moe else ''
|
moe_suffix = '_moe' if args.use_moe else ''
|
||||||
ckp = f'./{args.save_dir}/{args.weight}_{args.hidden_size}{moe_suffix}.pth'
|
ckp = f'./{args.save_dir}/{args.weight}_{args.hidden_size}{moe_suffix}.pth'
|
||||||
model = MiniMindVLM(
|
model = MiniMindVLM(
|
||||||
VLMConfig(hidden_size=args.hidden_size, num_hidden_layers=args.num_hidden_layers, use_moe=bool(args.use_moe)),
|
VLMConfig(hidden_size=args.hidden_size, num_hidden_layers=args.num_hidden_layers, use_moe=bool(args.use_moe)),
|
||||||
vision_model_path="./model/siglip2-base-p16-ve"
|
vision_model_path="./model/siglip2-base-p16-256-ve"
|
||||||
)
|
)
|
||||||
state_dict = torch.load(ckp, map_location=args.device)
|
state_dict = torch.load(ckp, map_location=args.device)
|
||||||
model.load_state_dict({k: v for k, v in state_dict.items() if 'mask' not in k}, strict=False)
|
model.load_state_dict({k: v for k, v in state_dict.items() if 'mask' not in k}, strict=False)
|
||||||
else:
|
else:
|
||||||
model = AutoModelForCausalLM.from_pretrained(args.load_from, trust_remote_code=True)
|
model = AutoModelForCausalLM.from_pretrained(args.load_from, trust_remote_code=True)
|
||||||
model.vision_encoder, model.processor = MiniMindVLM.get_vision_model("./model/siglip2-base-p16-ve")
|
model.vision_encoder, model.processor = MiniMindVLM.get_vision_model("./model/siglip2-base-p16-256-ve")
|
||||||
get_model_params(model, model.config)
|
get_model_params(model, model.config)
|
||||||
preprocess = model.processor
|
preprocess = model.processor
|
||||||
return model.half().eval().to(args.device), tokenizer, preprocess
|
return model.half().eval().to(args.device), tokenizer, preprocess
|
||||||
|
|
@ -38,8 +38,8 @@ def main():
|
||||||
parser.add_argument('--num_hidden_layers', default=8, type=int, help="隐藏层数量")
|
parser.add_argument('--num_hidden_layers', default=8, type=int, help="隐藏层数量")
|
||||||
parser.add_argument('--use_moe', default=0, type=int, choices=[0, 1], help="是否使用MoE架构（0=否，1=是）")
|
parser.add_argument('--use_moe', default=0, type=int, choices=[0, 1], help="是否使用MoE架构（0=否，1=是）")
|
||||||
parser.add_argument('--max_new_tokens', default=512, type=int, help="最大生成长度")
|
parser.add_argument('--max_new_tokens', default=512, type=int, help="最大生成长度")
|
||||||
parser.add_argument('--temperature', default=0.9, type=float, help="生成温度，控制随机性（0-1，越大越随机）")
|
parser.add_argument('--temperature', default=0.7, type=float, help="生成温度，控制随机性（0-1，越大越随机）")
|
||||||
parser.add_argument('--top_p', default=0.9, type=float, help="nucleus采样阈值（0-1）")
|
parser.add_argument('--top_p', default=0.85, type=float, help="nucleus采样阈值（0-1）")
|
||||||
parser.add_argument('--image_dir', default='./dataset/eval_images/', type=str, help="测试图像目录")
|
parser.add_argument('--image_dir', default='./dataset/eval_images/', type=str, help="测试图像目录")
|
||||||
parser.add_argument('--show_speed', default=1, type=int, help="显示decode速度（tokens/s）")
|
parser.add_argument('--show_speed', default=1, type=int, help="显示decode速度（tokens/s）")
|
||||||
parser.add_argument('--device', default='cuda' if torch.cuda.is_available() else 'cpu', type=str, help="运行设备")
|
parser.add_argument('--device', default='cuda' if torch.cuda.is_available() else 'cpu', type=str, help="运行设备")
|
||||||
|
|
@ -49,11 +49,12 @@ def main():
|
||||||
model, tokenizer, preprocess = init_model(args)
|
model, tokenizer, preprocess = init_model(args)
|
||||||
streamer = TextStreamer(tokenizer, skip_prompt=True, skip_special_tokens=True)
|
streamer = TextStreamer(tokenizer, skip_prompt=True, skip_special_tokens=True)
|
||||||
# 自动测试image_dir中的所有图像
|
# 自动测试image_dir中的所有图像
|
||||||
prompt = "请仔细观察这张图片，描述你所看到的内容：\n\n<image>"
|
prompt = "<image>\n请描述这张图中的主要物体和场景。"
|
||||||
# prompt = "Describe the content of this image in detail:\n\n<image>"
|
# prompt = "<image>\nPlease illustrate the image through your words."
|
||||||
for image_file in sorted(os.listdir(args.image_dir)):
|
for image_file in sorted(os.listdir(args.image_dir)):
|
||||||
if image_file.lower().endswith(('.png', '.jpg', '.jpeg', '.bmp')):
|
if image_file.lower().endswith(('.png', '.jpg', '.jpeg', '.bmp')):
|
||||||
setup_seed(random.randint(1, 31415926))
|
setup_seed(random.randint(1, 31415926))
|
||||||
|
setup_seed(512)
|
||||||
image_path = os.path.join(args.image_dir, image_file)
|
image_path = os.path.join(args.image_dir, image_file)
|
||||||
image = Image.open(image_path).convert('RGB')
|
image = Image.open(image_path).convert('RGB')
|
||||||
pixel_values = {k: v.to(args.device) for k, v in MiniMindVLM.image2tensor(image, preprocess).items()}
|
pixel_values = {k: v.to(args.device) for k, v in MiniMindVLM.image2tensor(image, preprocess).items()}
|
||||||
|
|
|
||||||
Binary file not shown.
|
Before Width: | Height: | Size: 3.3 MiB After Width: | Height: | Size: 6.0 MiB |
|
|
@ -1,23 +1,23 @@
|
||||||
## Download Vision Encoder (Required)
|
## Download Vision Encoder (Required)
|
||||||
|
|
||||||
This model requires an additional vision encoder `siglip2-base-p16-ve`
|
This model requires an additional vision encoder `siglip2-base-p16-256-ve`
|
||||||
|
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
git lfs install
|
git lfs install
|
||||||
|
|
||||||
# from HuggingFace
|
# from HuggingFace
|
||||||
git clone https://huggingface.co/jingyaogong/siglip2-base-p16-ve
|
git clone https://huggingface.co/jingyaogong/siglip2-base-p16-256-ve
|
||||||
|
|
||||||
# or from ModelScope
|
# or from ModelScope
|
||||||
git clone https://modelscope.cn/models/gongjy/siglip2-base-p16-ve
|
git clone https://modelscope.cn/models/gongjy/siglip2-base-p16-256-ve
|
||||||
```
|
```
|
||||||
|
|
||||||
Directory Structure
|
Directory Structure
|
||||||
|
|
||||||
```
|
```
|
||||||
model/
|
model/
|
||||||
├── siglip2-base-p16-ve/ # Vision Encoder (Required)
|
├── siglip2-base-p16-256-ve/ # Vision Encoder (Required)
|
||||||
│ ├── config.json
|
│ ├── config.json
|
||||||
│ ├── model.safetensors # ~180MB, float16
|
│ ├── model.safetensors # ~180MB, float16
|
||||||
│ └── preprocessor_config.json
|
│ └── preprocessor_config.json
|
||||||
|
|
|
||||||
|
|
@ -27,6 +27,7 @@ class MiniMindConfig(PretrainedConfig):
|
||||||
self.max_position_embeddings = kwargs.get("max_position_embeddings", 32768)
|
self.max_position_embeddings = kwargs.get("max_position_embeddings", 32768)
|
||||||
self.rms_norm_eps = kwargs.get("rms_norm_eps", 1e-6)
|
self.rms_norm_eps = kwargs.get("rms_norm_eps", 1e-6)
|
||||||
self.rope_theta = kwargs.get("rope_theta", 1e6)
|
self.rope_theta = kwargs.get("rope_theta", 1e6)
|
||||||
|
self.tie_word_embeddings = kwargs.get("tie_word_embeddings", True)
|
||||||
self.inference_rope_scaling = kwargs.get("inference_rope_scaling", False)
|
self.inference_rope_scaling = kwargs.get("inference_rope_scaling", False)
|
||||||
self.rope_scaling = {
|
self.rope_scaling = {
|
||||||
"beta_fast": 32,
|
"beta_fast": 32,
|
||||||
|
|
@ -211,6 +212,10 @@ class MiniMindModel(nn.Module):
|
||||||
past_key_values = past_key_values or [None] * len(self.layers)
|
past_key_values = past_key_values or [None] * len(self.layers)
|
||||||
start_pos = past_key_values[0][0].shape[1] if past_key_values[0] is not None else 0
|
start_pos = past_key_values[0][0].shape[1] if past_key_values[0] is not None else 0
|
||||||
hidden_states = self.dropout(self.embed_tokens(input_ids))
|
hidden_states = self.dropout(self.embed_tokens(input_ids))
|
||||||
|
# Recompute RoPE buffers lost during meta-device init (transformers>=5.x)
|
||||||
|
if self.freqs_cos[0, 0] == 0:
|
||||||
|
freqs_cos, freqs_sin = precompute_freqs_cis(dim=self.config.head_dim, end=self.config.max_position_embeddings, rope_base=self.config.rope_theta, rope_scaling=self.config.rope_scaling)
|
||||||
|
self.freqs_cos, self.freqs_sin = freqs_cos.to(hidden_states.device), freqs_sin.to(hidden_states.device)
|
||||||
position_embeddings = (self.freqs_cos[start_pos:start_pos + seq_length], self.freqs_sin[start_pos:start_pos + seq_length])
|
position_embeddings = (self.freqs_cos[start_pos:start_pos + seq_length], self.freqs_sin[start_pos:start_pos + seq_length])
|
||||||
presents = []
|
presents = []
|
||||||
for layer, past_key_value in zip(self.layers, past_key_values):
|
for layer, past_key_value in zip(self.layers, past_key_values):
|
||||||
|
|
@ -228,12 +233,14 @@ class MiniMindModel(nn.Module):
|
||||||
|
|
||||||
class MiniMindForCausalLM(PreTrainedModel, GenerationMixin):
|
class MiniMindForCausalLM(PreTrainedModel, GenerationMixin):
|
||||||
config_class = MiniMindConfig
|
config_class = MiniMindConfig
|
||||||
|
_tied_weights_keys = {"lm_head.weight": "model.embed_tokens.weight"}
|
||||||
def __init__(self, config: MiniMindConfig = None):
|
def __init__(self, config: MiniMindConfig = None):
|
||||||
self.config = config or MiniMindConfig()
|
self.config = config or MiniMindConfig()
|
||||||
super().__init__(self.config)
|
super().__init__(self.config)
|
||||||
self.model = MiniMindModel(self.config)
|
self.model = MiniMindModel(self.config)
|
||||||
self.lm_head = nn.Linear(self.config.hidden_size, self.config.vocab_size, bias=False)
|
self.lm_head = nn.Linear(self.config.hidden_size, self.config.vocab_size, bias=False)
|
||||||
self.model.embed_tokens.weight = self.lm_head.weight
|
if self.config.tie_word_embeddings: self.model.embed_tokens.weight = self.lm_head.weight
|
||||||
|
self.post_init()
|
||||||
|
|
||||||
def forward(self, input_ids, attention_mask=None, past_key_values=None, use_cache=False, logits_to_keep=0, labels=None, **kwargs):
|
def forward(self, input_ids, attention_mask=None, past_key_values=None, use_cache=False, logits_to_keep=0, labels=None, **kwargs):
|
||||||
hidden_states, past_key_values, aux_loss = self.model(input_ids, attention_mask, past_key_values, use_cache, **kwargs)
|
hidden_states, past_key_values, aux_loss = self.model(input_ids, attention_mask, past_key_values, use_cache, **kwargs)
|
||||||
|
|
|
||||||
|
|
@ -4,7 +4,7 @@ import warnings
|
||||||
from .model_minimind import *
|
from .model_minimind import *
|
||||||
from typing import Optional, Tuple, List, Union
|
from typing import Optional, Tuple, List, Union
|
||||||
from torch import nn
|
from torch import nn
|
||||||
from transformers import Siglip2ImageProcessor, Siglip2VisionModel
|
from transformers import SiglipImageProcessor, SiglipVisionModel
|
||||||
from transformers.modeling_outputs import MoeCausalLMOutputWithPast
|
from transformers.modeling_outputs import MoeCausalLMOutputWithPast
|
||||||
|
|
||||||
warnings.filterwarnings('ignore')
|
warnings.filterwarnings('ignore')
|
||||||
|
|
@ -26,20 +26,23 @@ class MMVisionProjector(nn.Module):
|
||||||
self.target_tokens = target_tokens
|
self.target_tokens = target_tokens
|
||||||
self.merge = source_tokens // target_tokens
|
self.merge = source_tokens // target_tokens
|
||||||
self.mlp = nn.Sequential(
|
self.mlp = nn.Sequential(
|
||||||
|
nn.LayerNorm(in_dim * self.merge),
|
||||||
nn.Linear(in_dim * self.merge, out_dim),
|
nn.Linear(in_dim * self.merge, out_dim),
|
||||||
nn.GELU(),
|
nn.GELU(),
|
||||||
nn.Linear(out_dim, out_dim),
|
nn.Linear(out_dim, out_dim),
|
||||||
)
|
)
|
||||||
def forward(self, x):
|
def forward(self, x):
|
||||||
b, n, d = x.shape
|
b, n, d = x.shape
|
||||||
x = x.reshape(b, self.target_tokens, d * self.merge)
|
side = int(n ** 0.5)
|
||||||
|
s = int(self.merge ** 0.5)
|
||||||
|
x = x.view(b, side // s, s, side // s, s, d).permute(0, 1, 3, 2, 4, 5).reshape(b, self.target_tokens, d * self.merge)
|
||||||
return self.mlp(x)
|
return self.mlp(x)
|
||||||
|
|
||||||
# 继承自语言模型
|
# 继承自语言模型
|
||||||
class MiniMindVLM(MiniMindForCausalLM):
|
class MiniMindVLM(MiniMindForCausalLM):
|
||||||
config_class = VLMConfig
|
config_class = VLMConfig
|
||||||
|
|
||||||
def __init__(self, config: VLMConfig = None, vision_model_path="./model/siglip2-base-p16-ve"):
|
def __init__(self, config: VLMConfig = None, vision_model_path="./model/siglip2-base-p16-256-ve"):
|
||||||
self.config = config or VLMConfig()
|
self.config = config or VLMConfig()
|
||||||
super().__init__(self.config)
|
super().__init__(self.config)
|
||||||
self.vision_encoder, self.processor = self.__class__.get_vision_model(vision_model_path)
|
self.vision_encoder, self.processor = self.__class__.get_vision_model(vision_model_path)
|
||||||
|
|
@ -51,8 +54,11 @@ class MiniMindVLM(MiniMindForCausalLM):
|
||||||
hf_logging.set_verbosity_error()
|
hf_logging.set_verbosity_error()
|
||||||
if not os.path.exists(model_path):
|
if not os.path.exists(model_path):
|
||||||
return None, None
|
return None, None
|
||||||
model = Siglip2VisionModel.from_pretrained(model_path)
|
try:
|
||||||
processor = Siglip2ImageProcessor.from_pretrained(model_path)
|
model = SiglipVisionModel.from_pretrained(model_path)
|
||||||
|
except (RuntimeError, ValueError):
|
||||||
|
return None, None
|
||||||
|
processor = SiglipImageProcessor.from_pretrained(model_path)
|
||||||
# 冻结 vision_encoder 的所有参数
|
# 冻结 vision_encoder 的所有参数
|
||||||
for param in model.parameters():
|
for param in model.parameters():
|
||||||
param.requires_grad = False
|
param.requires_grad = False
|
||||||
|
|
@ -126,6 +132,10 @@ class MiniMindVLM(MiniMindForCausalLM):
|
||||||
vision_tensors = torch.stack([self.vision_proj(MiniMindVLM.get_image_embeddings(pixel_values[:, i, :, :, :], self.vision_encoder)) for i in range(num)], dim=1)
|
vision_tensors = torch.stack([self.vision_proj(MiniMindVLM.get_image_embeddings(pixel_values[:, i, :, :, :], self.vision_encoder)) for i in range(num)], dim=1)
|
||||||
hidden_states = self.count_vision_proj(tokens=input_ids, h=hidden_states, vision_tensors=vision_tensors, seqlen=input_ids.shape[1])
|
hidden_states = self.count_vision_proj(tokens=input_ids, h=hidden_states, vision_tensors=vision_tensors, seqlen=input_ids.shape[1])
|
||||||
|
|
||||||
|
# Recompute RoPE buffers lost during meta-device init (transformers>=5.x)
|
||||||
|
if self.model.freqs_cos[0, 0] == 0:
|
||||||
|
freqs_cos, freqs_sin = precompute_freqs_cis(dim=self.config.head_dim, end=self.config.max_position_embeddings, rope_base=self.config.rope_theta, rope_scaling=self.config.rope_scaling)
|
||||||
|
self.model.freqs_cos, self.model.freqs_sin = freqs_cos.to(hidden_states.device), freqs_sin.to(hidden_states.device)
|
||||||
position_embeddings = (
|
position_embeddings = (
|
||||||
self.model.freqs_cos[start_pos:start_pos + seq_length],
|
self.model.freqs_cos[start_pos:start_pos + seq_length],
|
||||||
self.model.freqs_sin[start_pos:start_pos + seq_length]
|
self.model.freqs_sin[start_pos:start_pos + seq_length]
|
||||||
|
|
|
||||||
|
|
@ -1,9 +1,11 @@
|
||||||
import os
|
import os
|
||||||
import sys
|
import sys
|
||||||
|
import json
|
||||||
|
|
||||||
__package__ = "scripts"
|
__package__ = "scripts"
|
||||||
sys.path.append(os.path.abspath(os.path.join(os.path.dirname(__file__), '..')))
|
sys.path.append(os.path.abspath(os.path.join(os.path.dirname(__file__), '..')))
|
||||||
import torch
|
import torch
|
||||||
|
import transformers
|
||||||
import warnings
|
import warnings
|
||||||
from transformers import AutoTokenizer, AutoModelForCausalLM, LlamaConfig, LlamaForCausalLM
|
from transformers import AutoTokenizer, AutoModelForCausalLM, LlamaConfig, LlamaForCausalLM
|
||||||
from model.model_vlm import MiniMindVLM, VLMConfig
|
from model.model_vlm import MiniMindVLM, VLMConfig
|
||||||
|
|
@ -14,7 +16,7 @@ warnings.filterwarnings('ignore', category=UserWarning)
|
||||||
def convert_torch2transformers_minimind(torch_path, transformers_path, dtype=torch.bfloat16):
|
def convert_torch2transformers_minimind(torch_path, transformers_path, dtype=torch.bfloat16):
|
||||||
VLMConfig.register_for_auto_class()
|
VLMConfig.register_for_auto_class()
|
||||||
MiniMindVLM.register_for_auto_class("AutoModelForCausalLM")
|
MiniMindVLM.register_for_auto_class("AutoModelForCausalLM")
|
||||||
lm_model = MiniMindVLM(lm_config, vision_model_path="../model/siglip2-base-p16-ve")
|
lm_model = MiniMindVLM(lm_config, vision_model_path="../model/siglip2-base-p16-256-ve")
|
||||||
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
|
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
|
||||||
state_dict = torch.load(torch_path, map_location=device)
|
state_dict = torch.load(torch_path, map_location=device)
|
||||||
lm_model.load_state_dict(state_dict, strict=False)
|
lm_model.load_state_dict(state_dict, strict=False)
|
||||||
|
|
@ -25,6 +27,16 @@ def convert_torch2transformers_minimind(torch_path, transformers_path, dtype=tor
|
||||||
lm_model.save_pretrained(transformers_path, safe_serialization=False)
|
lm_model.save_pretrained(transformers_path, safe_serialization=False)
|
||||||
tokenizer = AutoTokenizer.from_pretrained('../model/')
|
tokenizer = AutoTokenizer.from_pretrained('../model/')
|
||||||
tokenizer.save_pretrained(transformers_path)
|
tokenizer.save_pretrained(transformers_path)
|
||||||
|
# 显式写入 tie_word_embeddings（save_pretrained 默认不序列化与默认值相同的字段）
|
||||||
|
config_path = os.path.join(transformers_path, "config.json")
|
||||||
|
config = json.load(open(config_path, 'r', encoding='utf-8'))
|
||||||
|
config['tie_word_embeddings'] = True
|
||||||
|
# ======= transformers-5.0的兼容低版本写法 =======
|
||||||
|
if int(transformers.__version__.split('.')[0]) >= 5:
|
||||||
|
tokenizer_config_path = os.path.join(transformers_path, "tokenizer_config.json")
|
||||||
|
json.dump({**json.load(open(tokenizer_config_path, 'r', encoding='utf-8')), "tokenizer_class": "PreTrainedTokenizerFast", "extra_special_tokens": {}}, open(tokenizer_config_path, 'w', encoding='utf-8'), indent=2, ensure_ascii=False)
|
||||||
|
config['rope_theta'] = lm_config.rope_theta; config['rope_scaling'] = None; config.pop('rope_parameters', None)
|
||||||
|
json.dump(config, open(config_path, 'w', encoding='utf-8'), indent=2, ensure_ascii=False)
|
||||||
print(f"模型已保存为 Transformers-MiniMind-V 格式: {transformers_path}")
|
print(f"模型已保存为 Transformers-MiniMind-V 格式: {transformers_path}")
|
||||||
|
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -34,7 +34,7 @@ def scan_vlm_models(base_dir):
|
||||||
|
|
||||||
def load_vlm_model(model_path):
|
def load_vlm_model(model_path):
|
||||||
global model, tokenizer, preprocess, lm_config, current_model_name
|
global model, tokenizer, preprocess, lm_config, current_model_name
|
||||||
tokenizer = AutoTokenizer.from_pretrained(model_path)
|
tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)
|
||||||
model = AutoModelForCausalLM.from_pretrained(model_path, trust_remote_code=True)
|
model = AutoModelForCausalLM.from_pretrained(model_path, trust_remote_code=True)
|
||||||
model.vision_encoder, model.processor = MiniMindVLM.get_vision_model(vision_model_path)
|
model.vision_encoder, model.processor = MiniMindVLM.get_vision_model(vision_model_path)
|
||||||
if model.vision_encoder is None: raise FileNotFoundError(f"视觉编码器未找到: {vision_model_path}")
|
if model.vision_encoder is None: raise FileNotFoundError(f"视觉编码器未找到: {vision_model_path}")
|
||||||
|
|
@ -128,12 +128,12 @@ def launch_gradio_server(server_name="0.0.0.0", server_port=7788):
|
||||||
response += chunk
|
response += chunk
|
||||||
yield history + [{"role": "assistant", "content": response}]
|
yield history + [{"role": "assistant", "content": response}]
|
||||||
|
|
||||||
with gr.Blocks(title="minimind-3v", css="#chatbox img{max-width:120px!important;max-height:120px!important;border-radius:8px} div.full-container{display:flex!important;flex-direction:row!important;flex-wrap:nowrap!important;align-items:center!important} div.full-container>.thumbnails{flex-shrink:0!important;max-width:50px!important} div.full-container>.input-container{flex:1!important} .input-wrapper{display:flex!important;flex-direction:row!important;flex-wrap:nowrap!important;align-items:center!important} .input-wrapper>.thumbnails{flex-shrink:0!important;max-width:50px!important} .input-wrapper>.input-row{flex:1!important} .thumbnail-image{max-width:40px!important;max-height:40px!important} textarea{overflow-y:hidden!important}") as demo:
|
with gr.Blocks(title="minimind-3v", css="#chatbox img{max-width:120px!important;max-height:120px!important;border-radius:8px} div.full-container{display:flex!important;flex-direction:row!important;flex-wrap:nowrap!important;align-items:center!important} div.full-container>.thumbnails{flex-shrink:0!important;max-width:50px!important} div.full-container>.input-container{flex:1!important} .input-wrapper{display:flex!important;flex-direction:row!important;flex-wrap:nowrap!important;align-items:center!important} .input-wrapper>.thumbnails{flex-shrink:0!important;max-width:50px!important} .input-wrapper>.input-row{flex:1!important} .thumbnail-image{max-width:40px!important;max-height:40px!important} textarea{overflow-y:hidden!important} #chatbox,#chatbox *{scrollbar-width:none!important;-ms-overflow-style:none!important} #chatbox::-webkit-scrollbar,#chatbox *::-webkit-scrollbar{display:none!important;width:0!important;height:0!important}") as demo:
|
||||||
gr.HTML('<div style="display:flex;flex-direction:column;align-items:center;justify-content:center;margin:-8px 0 -6px 0"><div style="display:flex;align-items:center;justify-content:center;gap:8px"><img src="https://www.modelscope.cn/api/v1/studio/gongjy/MiniMind/repo?Revision=master&FilePath=images%2Flogo2.png&View=true" style="height:30px"><span style="font-size:1.5rem;font-weight:bold;font-style:italic">minimind-3v</span></div><div style="color:#bbb;font-style:italic;margin:0">AI-generated content may be inaccurate, please verify</div></div>')
|
gr.HTML('<div style="display:flex;flex-direction:column;align-items:center;justify-content:center;margin:-8px 0 -6px 0"><div style="display:flex;align-items:center;justify-content:center;gap:8px"><img src="https://www.modelscope.cn/api/v1/studio/gongjy/MiniMind/repo?Revision=master&FilePath=images%2Flogo2.png&View=true" style="height:30px"><span style="font-size:1.5rem;font-weight:bold;font-style:italic">minimind-3v</span></div><div style="color:#bbb;font-style:italic;margin:0">AI-generated content may be inaccurate, please verify</div></div>')
|
||||||
try:
|
try:
|
||||||
chatbot = gr.Chatbot(label="", height=520, elem_id="chatbox", type="messages")
|
chatbot = gr.Chatbot(label="", height=560, elem_id="chatbox", type="messages")
|
||||||
except TypeError:
|
except TypeError:
|
||||||
chatbot = gr.Chatbot(label="", height=520, elem_id="chatbox")
|
chatbot = gr.Chatbot(label="", height=560, elem_id="chatbox")
|
||||||
msg = gr.MultimodalTextbox(placeholder="输入问题，点击📎上传图片", show_label=False, submit_btn="发送")
|
msg = gr.MultimodalTextbox(placeholder="输入问题，点击📎上传图片", show_label=False, submit_btn="发送")
|
||||||
with gr.Row():
|
with gr.Row():
|
||||||
model_dropdown = gr.Dropdown(choices=list(model_dict.keys()), value=current_model_name, show_label=False, scale=1)
|
model_dropdown = gr.Dropdown(choices=list(model_dict.keys()), value=current_model_name, show_label=False, scale=1)
|
||||||
|
|
@ -150,9 +150,9 @@ def launch_gradio_server(server_name="0.0.0.0", server_port=7788):
|
||||||
if __name__ == '__main__':
|
if __name__ == '__main__':
|
||||||
parser = argparse.ArgumentParser(description="Chat with MiniMind-V")
|
parser = argparse.ArgumentParser(description="Chat with MiniMind-V")
|
||||||
parser.add_argument('--load_from', default='./', type=str, help="transformers模型扫描目录")
|
parser.add_argument('--load_from', default='./', type=str, help="transformers模型扫描目录")
|
||||||
parser.add_argument('--vision_model', default='../model/siglip2-base-p16-ve', type=str, help="视觉编码器路径")
|
parser.add_argument('--vision_model', default='../model/siglip2-base-p16-256-ve', type=str, help="视觉编码器路径")
|
||||||
parser.add_argument('--temperature', default=0.8, type=float, help="生成温度")
|
parser.add_argument('--temperature', default=0.7, type=float, help="生成温度")
|
||||||
parser.add_argument('--top_p', default=0.9, type=float, help="nucleus采样阈值")
|
parser.add_argument('--top_p', default=0.95, type=float, help="nucleus采样阈值")
|
||||||
parser.add_argument('--device', default='cuda' if torch.cuda.is_available() else 'cpu', type=str, help="运行设备")
|
parser.add_argument('--device', default='cuda' if torch.cuda.is_available() else 'cpu', type=str, help="运行设备")
|
||||||
parser.add_argument('--max_seq_len', default=8192, type=int, help="最大序列长度")
|
parser.add_argument('--max_seq_len', default=8192, type=int, help="最大序列长度")
|
||||||
args = parser.parse_args()
|
args = parser.parse_args()
|
||||||
|
|
|
||||||
|
|
@ -102,12 +102,12 @@ if __name__ == "__main__":
|
||||||
parser.add_argument("--save_interval", type=int, default=1000, help="模型保存间隔")
|
parser.add_argument("--save_interval", type=int, default=1000, help="模型保存间隔")
|
||||||
parser.add_argument('--hidden_size', default=768, type=int, help="隐藏层维度")
|
parser.add_argument('--hidden_size', default=768, type=int, help="隐藏层维度")
|
||||||
parser.add_argument('--num_hidden_layers', default=8, type=int, help="隐藏层数量")
|
parser.add_argument('--num_hidden_layers', default=8, type=int, help="隐藏层数量")
|
||||||
parser.add_argument('--max_seq_len', default=360, type=int, help="训练的最大截断长度")
|
parser.add_argument('--max_seq_len', default=450, type=int, help="训练的最大截断长度")
|
||||||
parser.add_argument('--use_moe', default=0, type=int, choices=[0, 1], help="是否使用MoE架构（0=否，1=是）")
|
parser.add_argument('--use_moe', default=0, type=int, choices=[0, 1], help="是否使用MoE架构（0=否，1=是）")
|
||||||
parser.add_argument("--data_path", type=str, default="../dataset/pretrain_i2t.parquet", help="训练数据路径")
|
parser.add_argument("--data_path", type=str, default="../dataset/pretrain_i2t.parquet", help="训练数据路径")
|
||||||
parser.add_argument('--from_weight', default='llm', type=str, help="基于哪个权重训练，为none则不基于任何权重训练")
|
parser.add_argument('--from_weight', default='llm', type=str, help="基于哪个权重训练，为none则不基于任何权重训练")
|
||||||
parser.add_argument('--from_resume', default=0, type=int, choices=[0, 1], help="是否自动检测&续训（0=否，1=是）")
|
parser.add_argument('--from_resume', default=0, type=int, choices=[0, 1], help="是否自动检测&续训（0=否，1=是）")
|
||||||
parser.add_argument('--freeze_llm', default=1, type=int, choices=[0, 1, 2], help="冻结策略（0=完全可训练，1=冻结+解冻第0层，2=完全冻结仅训练proj）")
|
parser.add_argument('--freeze_llm', default=2, type=int, choices=[0, 1, 2], help="冻结策略（0=完全可训练，1=冻结+解冻首尾层，2=完全冻结仅训练proj）")
|
||||||
parser.add_argument("--use_compile", default=0, type=int, choices=[0, 1], help="是否使用torch.compile加速（0=否，1=是）")
|
parser.add_argument("--use_compile", default=0, type=int, choices=[0, 1], help="是否使用torch.compile加速（0=否，1=是）")
|
||||||
parser.add_argument("--use_wandb", action="store_true", help="是否使用wandb")
|
parser.add_argument("--use_wandb", action="store_true", help="是否使用wandb")
|
||||||
parser.add_argument("--wandb_project", type=str, default="MiniMind-V-Pretrain", help="wandb项目名")
|
parser.add_argument("--wandb_project", type=str, default="MiniMind-V-Pretrain", help="wandb项目名")
|
||||||
|
|
|
||||||
|
|
@ -107,7 +107,7 @@ if __name__ == "__main__":
|
||||||
parser.add_argument("--data_path", type=str, default="../dataset/sft_i2t.parquet", help="训练数据路径")
|
parser.add_argument("--data_path", type=str, default="../dataset/sft_i2t.parquet", help="训练数据路径")
|
||||||
parser.add_argument('--from_weight', default='pretrain_vlm', type=str, help="基于哪个权重训练，为none则不基于任何权重训练")
|
parser.add_argument('--from_weight', default='pretrain_vlm', type=str, help="基于哪个权重训练，为none则不基于任何权重训练")
|
||||||
parser.add_argument('--from_resume', default=0, type=int, choices=[0, 1], help="是否自动检测&续训（0=否，1=是）")
|
parser.add_argument('--from_resume', default=0, type=int, choices=[0, 1], help="是否自动检测&续训（0=否，1=是）")
|
||||||
parser.add_argument('--freeze_llm', default=0, type=int, choices=[0, 1, 2], help="冻结策略（0=完全可训练，1=冻结+解冻第0层，2=完全冻结仅训练proj）")
|
parser.add_argument('--freeze_llm', default=1, type=int, choices=[0, 1, 2], help="冻结策略（0=完全可训练，1=冻结+解冻首尾层，2=完全冻结仅训练proj）")
|
||||||
parser.add_argument("--use_compile", default=0, type=int, choices=[0, 1], help="是否使用torch.compile加速（0=否，1=是）")
|
parser.add_argument("--use_compile", default=0, type=int, choices=[0, 1], help="是否使用torch.compile加速（0=否，1=是）")
|
||||||
parser.add_argument("--use_wandb", action="store_true", help="是否使用wandb")
|
parser.add_argument("--use_wandb", action="store_true", help="是否使用wandb")
|
||||||
parser.add_argument("--wandb_project", type=str, default="MiniMind-V-SFT", help="wandb项目名")
|
parser.add_argument("--wandb_project", type=str, default="MiniMind-V-SFT", help="wandb项目名")
|
||||||
|
|
|
||||||
|
|
@ -63,7 +63,7 @@ def setup_seed(seed: int):
|
||||||
torch.backends.cudnn.benchmark = False
|
torch.backends.cudnn.benchmark = False
|
||||||
|
|
||||||
|
|
||||||
def init_vlm_model(vlm_config, from_weight='pretrain_vlm', tokenizer_path='../model', vision_model_path='../model/siglip2-base-p16-ve', save_dir='../out', device='cuda', freeze_llm=0):
|
def init_vlm_model(vlm_config, from_weight='pretrain_vlm', tokenizer_path='../model', vision_model_path='../model/siglip2-base-p16-256-ve', save_dir='../out', device='cuda', freeze_llm=0):
|
||||||
tokenizer = AutoTokenizer.from_pretrained(tokenizer_path)
|
tokenizer = AutoTokenizer.from_pretrained(tokenizer_path)
|
||||||
model = MiniMindVLM(vlm_config, vision_model_path=vision_model_path)
|
model = MiniMindVLM(vlm_config, vision_model_path=vision_model_path)
|
||||||
|
|
||||||
|
|
@ -84,8 +84,9 @@ def init_vlm_model(vlm_config, from_weight='pretrain_vlm', tokenizer_path='../mo
|
||||||
if 'vision_encoder' not in name:
|
if 'vision_encoder' not in name:
|
||||||
param.requires_grad = True
|
param.requires_grad = True
|
||||||
elif freeze_llm == 1:
|
elif freeze_llm == 1:
|
||||||
|
last_idx = vlm_config.num_hidden_layers - 1
|
||||||
for name, param in model.model.named_parameters():
|
for name, param in model.model.named_parameters():
|
||||||
if 'layers.0.' in name:
|
if 'layers.0.' in name or f'layers.{last_idx}.' in name:
|
||||||
param.requires_grad = True
|
param.requires_grad = True
|
||||||
elif freeze_llm == 2:
|
elif freeze_llm == 2:
|
||||||
pass
|
pass
|
||||||
|
|
|
||||||
Loading…
Reference in New Issue