2025-04-27
+
+
+
-
-More...
+ +**2025-04-27** - 兼容性更新 - 适配[「minimind仓库新特性」](https://github.com/jingyaogong/minimind/issues/370) - 规范化部分代码 -
-
-
### 3' 开始训练
-**3.1 预训练（学图像描述）**
+**3.1 Pretrain（可选）**
+
+> SFT 数据集 `sft_i2t.parquet` 已把 Pretrain 数据作为子集合并进来，**此阶段可跳过**，直接 `--from_weight llm` 进入 SFT 即可；若想让 Projector 提前完成对齐、SFT 收敛更稳，可以提前执行 Pretrain。
```bash
-# 基础训练命令（从LLM权重开始，仅训练vision_proj）
-python train_pretrain_vlm.py --epochs 4 --from_weight llm
+# 基础训练命令（从LLM权重开始，仅训练 vision_proj）
+python train_pretrain_vlm.py --epochs 2 --from_weight llm
```
-> 执行预训练，得到 `pretrain_vlm_*.pth` 作为预训练的输出权重（其中*为模型的dimension，默认为768）
+> 执行 Pretrain，得到 `pretrain_vlm_*.pth` 作为 Pretrain 的输出权重（其中*为模型的dimension，默认为768）
-**3.2 监督微调（学看图对话方式）**
+**3.2 SFT（必须）**
```bash
-# 基础训练命令（从预训练权重开始，全参数微调）
+# 基础训练命令（默认 --freeze_llm 1：解冻 proj + LLM 首尾层，保留中间层原 LLM 知识）
+# 若已执行 3.1：--from_weight pretrain_vlm；若跳过 Pretrain：--from_weight llm
python train_sft_vlm.py --epochs 2 --from_weight pretrain_vlm
```
-> 执行监督微调，得到 `sft_vlm_*.pth` 作为指令微调的输出权重
+> 执行 SFT，得到 `sft_vlm_*.pth` 作为 SFT 的输出权重
More...
- **2025-02-20** - MiniMind2-V伴随MiniMind2同步更新 @@ -146,9 +155,9 @@ git clone --depth 1 https://github.com/jingyaogong/minimind-v ```bash # 下载siglip2模型到 ./model 目录下 -git clone https://huggingface.co/jingyaogong/siglip2-base-p16-ve +git clone https://huggingface.co/jingyaogong/siglip2-base-p16-256-ve # 或 -git clone https://modelscope.cn/models/gongjy/siglip2-base-p16-ve +git clone https://modelscope.cn/models/gongjy/siglip2-base-p16-256-ve ``` ```bash @@ -223,40 +232,43 @@ print(torch.cuda.is_available()) 【注2】Parquet是列式存储格式，支持高效压缩和快速读取。如果你对它感到陌生，可以预览数据内容，在`dataset/`目录下执行`python lm_dataset.py`可视化前5条图文对 -Pretrain数据： +Pretrain 数据（可选；仅包含 caption 子集）： ```bash wget https://hf-mirror.com/datasets/jingyaogong/minimind-v_dataset/resolve/main/pretrain_i2t.parquet ``` -SFT数据： +SFT 数据（必须；包含 caption + instruct + 纯文本全量合并）： ```bash wget https://hf-mirror.com/datasets/jingyaogong/minimind-v_dataset/resolve/main/sft_i2t.parquet ``` -建议预留~2GB空间存放数据集，若无多余空间存放pretrain数据，可尝试跳过pretrain训练步骤直接进行sft训练。 +SFT 单文件 290 万条已把 Pretrain 作为子集合并，经全局 dictionary encoding 去重后体积只比 SFT 原版多 ~10%，可覆盖所有训练阶段。注：训练须知
@@ -276,7 +288,7 @@ python train_sft_vlm.py --epochs 4 --from_resume 1 - `--from_weight`: 基础权重名称（llm, pretrain_vlm, none等） - `--save_weight`: 保存权重的前缀名 - `--from_resume`: 是否续训（0=从头开始，1=从检查点继续） -- `--freeze_llm`: 是否冻结LLM参数（仅pretrain使用） +- `--freeze_llm`: 冻结策略（0=全参可训，1=proj + LLM 首尾层，2=仅训 proj）。Pretrain 默认 2，SFT 默认 1 - 更多可直接参考代码
- 
+ 
|
- 在这幅图片中，一架白色的飞机正降落在一片广阔的天空中。天空中飘浮着许多流线型的建筑物，这些建筑物散落在天空中。除了主要的飞机外，还有两辆汽车在场景中各处停放，包括一辆巴士和一辆小型汽车。这架飞机似乎停在地面上，表明它们正在进行商业活动。 | -在这张照片中，有一架大型的飞机正在降落，这表明它是为这架飞机而设计的。此外，它停在云层之下，这表明它在移动。天空中有云朵，暗示着这架飞机正在空中飞行。整个场景的背景显示出一种宁静祥和的气氛，暗示这架飞机正在飞行中作为一个机会来进行外交活动或与其他客机接触。 | +这张图片中的主要物体是一只金毛寻回犬，以其金色的皮毛和友好的表情而闻名。金毛寻回犬是这幅图片中的主要对象，以其金色的皮毛和友好的表情而闻名。它们是图片的主要焦点，并在这幅图片中展示了这个狗的特征。 | +这张图片展示了一只金毛寻回犬在草地上享受一群彩色气球，很可能是在玩耍或散步的时候。这只狗的皮毛呈金黄色，尾巴尖端略带波浪状。这只狗的耳朵尖尖，尾巴垂下垂，表明它正在向前看，或者可能正在进行某种形式的玩耍或互动。背景是一片广阔的开阔景观，天空是晴朗的蓝色，零星分布着几朵云。在画面的左侧，有一条黄色和粉色的条纹，它们挂在草地上，为画面增添了色彩和质感。 |
- 
+ 
|
- 图片显示了一张大生日蛋糕，上面刻有各种生日和装饰物的生日和一束蜡烛。这些甜蜜而独特的气味使其成为这个特殊场合的焦点。蛋糕被放在餐桌上，有蛋糕放在上面，很可能是放在桌上，用蜡烛加上蜡烛来营造出诱人的场景。此外，这张生日蛋糕还配上了蜡烛作为装饰物，进一步丰富了整个生日的氛围。 | -这张照片捕捉到了一个生日聚会，各种美味可口的生日蛋糕和一把勺子放在那里。它被放置在一个充满生气的蛋糕盒子里，里面装着各种各样的草莓糖霜糖浆。蛋糕上有一个大号的蜡烛，给人一种温馨和诱人的图案。周围有很多小点心，比如小心地蜡烛和糖霜，让整个蛋糕看起来更加令人放松和诱人。 | +这张图片展示了一只雨伞在城市街道上停放的场景。雨伞主要是鲜艳的颜色，在黑色背景上形成鲜明的对比。它被一条多彩的线连接在一起，可能表明天气状况良好。雨伞的闪闪发光的外套增添了一丝动感，为场景增添了一种动感。雨伞的反射和反射暗示着它可能是为了街道或停车场的静止而被使用。它被放置在一条铺设的道路上，表明这个场景是在城市街道上拍摄的，可能是在一个城市公园或公寓区。 | +这张图中的主要物体是雨伞，它在雨天被雨水滋润。雨伞是主要的物体，通过其设计和大小来识别，使其成为图片的焦点。雨伞的颜色鲜艳，有明亮的黄色、橙色、红色和蓝色，这些颜色通常与雨水相关联。雨伞的反光表面反射着光线，表明它是有光泽的，这在雨天是很常见的。雨伞的反射和颜色表明它可能是由雨水滴或雨水渗入地面，这在雨天是常见的。雨伞的设计，以其多彩的图案和颜色，与雨伞的鲜艳颜色相辅相成，使其成为图片的焦点。 |
- 
+ 
|
- 在这张图片中，有一个装饰着奶酪的比萨饼和一片新鲜的青绿野餐毯。这看起来像是一块沙拉，给人一种清新、诱人、美味的享受体验。披萨的大小和大小暗示着一种随意和随意的用餐体验。画面中，一群人围坐在一块砖块上，其中一些则分散在桌子上，周围摆放着不同种类的青绿野餐毯。此外，桌子上还放着一块披萨片，上面摆放着各种各样的青绿野餐毯，营造出一种轻松愉快的氛围。 | -图中，比萨饼在木桌上。它被切成了比萨饼的大小，而且披萨放置在木桌上。比萨有很多配料，包括奶酪和各种酱汁。披萨有多种口味，包括经典的牛排式和意大利式，还有一些加在比萨表面。比萨上有很多新鲜水果，如西红柿和莫吉托，以及奶酪，使这幅图片更加丰富和有吸引力。 | +这张图片中的主要物体是一辆自行车，具体来说是一辆青绿色的自行车。这一识别是基于几个视觉线索：骑车者的服装，头盔和轮胎的存在，以及前部的后部轮胎，这些都是自行车的特征。此外，背景中有树木和晴朗的天空，进一步强调了这一场景。图片中没有任何可见的物体或人，进一步强调了这辆自行车的特点。 | +这张图片展示了一幅宁静的户外场景，前景是一条繁华的街道，上面挂满了粉色花朵和红色装饰，散落着各种粉色花朵和白色小花。在前景，有一棵开着粉色花朵的大树，树干略显皱纹，暗示着成熟或可能是春天的场景。树后面是一片郁郁葱葱的绿树，树叶浓密，表明可能是春天或夏天。前景是一条铺满粉色花朵的道路，通向画面左侧是一条长满花朵的小径。在道路的右侧，是一条修剪整齐的道路，两旁是修剪整齐的草坪，背景是晴朗的蓝天和远处的另一棵树。整个场景传达了一种宁静和自然美的感觉。 |
- 
+ 
|
- 在这幅图片中，一辆白色的马车停泊在一条红色的路牌上。这个车辆位于一条绿油油的道路上，很可能是一个购物中心或高速公路。在这辆车的后部，可以看到一个绿色的马车停泊在路上，这是典型的户外场所。这辆马车可能是为了娱乐或观赏车辆而停放。 | -画面显示了一辆红色高性能赛车，停在一辆大型汽车后面。这辆车可以看到车身、汽车、汽车及汽车停在里面。它似乎是一辆大型红色汽车，有各种大小的汽车，表明它可能是汽车制造商生产的。此外，车辆周围的环境暗示了一种户外环境，因为一辆汽车也出现在场景中。 | +这张图片中的主要物体是一辆现代高尔夫球车，具体来说是一辆在沙漠或海滩上的高尔夫球车。这辆车是主要的焦点，设计时尚，具有流线型的车身，配备了高性能的后扰流板，并配备了赛车轮胎，表明这是一辆高性能跑车。车辆的风格和设计表明它可能是一款高性能跑车，可能是在沙漠或海滩上使用。 | +这张图片展示了一辆明亮的黄色跑车在晴朗的天空下。这辆车看起来是一辆豪华轿车，设计时尚，有着流线型的车身和明显的车顶，暗示着运动性能。它拥有高性能的车身，配有大尺寸的大众Jet和前大灯，展现出时尚的外观。这辆车的车身设计流线型，有流线型的车身线条和明显的车顶线条，暗示着高性能的车型。背景是一片晴朗的天空，有几朵白云，表明天气条件良好。在图片的右侧，可以看到一辆明亮的黄色跑车，暗示着它是为休闲或运动设计的。 |
- 
+ 
|
- 画面中，一座蓝色白色的大房子位于一条城市街道上，为这个地区增添了一丝自然与奇思妙想。墙上挂着一盏交通灯，为整个场景增添了特色和引人入胜。 | -画面中，一座蓝色房子旁边有很多小花瓶。这表明这座房子里可能正在营运一些小型小型花盆或盆栽植物。一些人聚集在房子周围，可能正在享受户外生活中的某种乐趣或美景。有些人站在画面中，而其他人则散布于场景中。总体而言，这幅场景捕捉到了一个美丽而令人愉快的场景，展示出该房子里一个令人放松和宁静的环境。 | +这张图中的主要物体是超级英雄的战斗姿态。超级英雄通常是最具威胁性或高能量的形象，以保护和控制英雄。他穿着一套经典的超级英雄服装，包括蓝色牛仔裤和黑色靴子，暗示着他是该品牌的忠实伙伴。超级英雄通常被描绘为一个具有攻击性的角色，通常被视为英雄的特征。超级英雄的服装和姿势，以及他的姿势，表明他可能代表着英雄的身份或身份，而非任何超级英雄。 | +这张图中的主要物体似乎是一个超级英雄角色，可能是超级英雄，考虑到其服装和装甲的风格。这个角色穿着超级英雄服装，包括红色和白色的袍子，这在超级英雄服装中很常见。他手持一块大大的红色和白色的大胆胡须，这是超级英雄服装的标志性特征。这个角色还戴着一条带有红色和白色条纹的红色头盔，这表明他可能是一名战士或领导者。背景是城市天际线，有着现代建筑，天空呈现出蓝色和橙色的色调，暗示着一个城市环境。 |
- 
+ 
|
- 在这幅图片中，有一个高高的山，它看起来像是大森林中的一片高山。天空中闪烁着不同颜色的星星，给画面增加了一抹红点。天空中有两朵高大的树，树木高高地挂在一起，暗示着森林中的宁静与自然之美。在画面的中心，可以看到一棵高大的松树，树干上覆盖着一层薄薄的苔藓。这种高高的松树与周围的大山构成了一个有趣而引人入胜的背景，为这片自然之美增添了一丝神秘色彩。 | -这幅图片展示了一个令人印象深刻的宁静湖面。湖水从天上飘浮着，暗示着一个令人平静和放松的水面。湖边上有几匹高大、形状各异的景象，它们在湖面上显得格外美丽。此外，在湖的边缘，有一座巨大的高山，为整个湖景增添了几分神秘色彩。湖景中的天空也被描述得如画，给整个画面增添了一种纯净和宁静的气氛。 | -||
- 
-
- |
- 图中，一大群人聚集在一张大餐桌旁欣赏着烤肉和热带水果，其中一人站在一碗里摆放着各种各样的盘子。桌子上放着几个碗，上面摆满了肉类和其他配料。有些放在盘子里，其他的则放在桌子上。在这张餐桌周围，有几个盘子，其中两个放着一杯酒，另一个放在靠近餐桌的左侧。 | -画面中，一群人聚集在一家大餐馆里，享受着一顿饭。这家餐厅的菜单上有一些生菜和猪肉，但它们已经被切成了四份，上面还有一个碗。他们拿着烤肉准备食用。在背景中，有几个瓶子在场景中。还有一把勺子位于桌子左侧，使盘子看起来更吸引人。一盆盆栽植物放在桌子左侧上方，为空间增添了一抹绿色。场景中的其他元素包括一个碗，里面放着葡萄酒和两根葡萄酒。 | -||
- 
-
- |
- 图中，一只棕色的小灰猫正坐在篮子里。这只猫身上戴着一顶棕色的帽子，很可能是一个戴帽子的男人。在篮子里，一只棕色的紫色小灰猫正沿着篮子里去休息。这些小猫似乎也在享受这份温暖，但它们似乎并没有完全放过来。此外，背景中还可以看到一把剪刀。这把剪刀看起来是专门为小猫设计的，它可以用作家庭相册或礼物。在篮子的侧边，有几条篮子，其中一条是最亮的，另一条则是最暗的。在篮子中可以看到一只棕色小灰猫，而另一条则是更暗的。 | -在这张照片中，一只小猫坐在篮子里，紧挨着它坐在篮子里的那块木篮上。猫的身体上有九条纹毛发。这个场景描绘了它们之间的亲密关系，展示了它们在一起度过时光的不同场合。画面中，一群大的猫坐在篮子里，其中一只小猫也被描述为小猫，这可能表明他们正在享受与猫互动、与它们的互动或一起度过愉快时光。 | -||
- 
-
- |
- 在这张图片中，沙滩上有很多椅子，还有一些人站着，可以看到一把遮阳伞。虽然它看起来很大，但却没有任何特别的设计。沙滩上有许多椅子，表明这是一家餐馆或者服务员办公室。其中最引人注目的是一张海边椅子，椅子上放着一只热带海滩椅。这个椅子非常适合放松身心、享受海滩时光。此外，还有一些椅子和其他人在场景中，可能是为了放置食物或其他用途。靠近椅子的椅子表示该位置可供使用的其他人使用，也许也有一人在靠近那个椅子的地方。 | -图片显示了一个美丽的海滩场景，有很多椅子散布在天然的棕榈树上。其中一个椅子靠近海滩，而另一个则较小。沙滩上有两把椅子，其中一把靠近中间，另一把则稍微偏左，还有一些则在靠近边缘处。在海边的海滩周围，你可以看到几个人坐在海边的沙滩上，有的靠近海水中，还有一张沙滩椅。其中一张椅子靠近海滩，另一张椅子靠近海边。此外，还可以看到几只遮阳伞，为沙滩上的躺椅提供了遮阴。 | -||
- 
-
- |
- 画面中，一辆蓝色的黄色公共汽车正从一辆黄色公共汽车驶过，在道路上停泊着。这辆公共汽车看起来是在一个黄色的黄色高速公路上。图中有几个人，其中一些靠近前景，而另一些则靠后一些，但都没有看到。在黄色公共汽车附近，可以看到一辆停在路边，那辆停在路边。此外，还有两辆不同方向的巴士，一辆靠近前景，另一辆靠近前景，另一辆稍微靠后一些。 | -画面中，一辆黄色和黄色相间的黄色和蓝色交叉路口的蓝色公共汽车正在一条通往路缘上的红色公交车站。有几辆公共汽车正停在路边，它们离一排车道很近。在背景中，可以看到一些长凳，它们在城市里交叉起来。一个长凳位于图中最左侧，而另一个则稍微靠后一点，为画面增添了一些城市特色。整个场景中有很多人和车辆散落在场景各处，包括黄色和蓝色的交叉路口。整个场景给人一种忙碌和迷茫的感觉，这也突显了公共汽车在市区中的存在和目的。 | +这张图片中的主要物体是一辆红色的赛车，很可能是赛车。赛车是赛车运动的典型特征，旨在提高速度、提高燃油效率和赛道的稳定性和安全性。赛车的特征包括前部有大型前部、后部有大型后部，以及与赛道的对峙部分。赛车的颜色和设计与赛车的整体风格相符，以赛车为主。赛车的赛道是静止的，表明这是一条赛道。赛车的赛道是从赛道上移动的，没有任何可见的轨迹或赛道的边缘。 | +这张图片中的主要物体是红色的，它在赛道上被捕捉到在运动中，这可以从赛车的后视图中看出。这个动作是红色的，通过其独特的尾部设计，可以识别出赛道的结构，这是赛道上常见的特征。赛车的赛车设计表明它是为高速赛道赛事设计的，这在赛道上很常见。背景中的观众也可见，但焦点完全集中在红色的赛车上，这表明这个场景是在一个正式的赛道赛事中拍摄的。 |
@@ -66,23 +64,35 @@ Is the training process difficult? Now, let's explore the answers and feel the j
> (As of 2026-02-15) The MiniMind-V series has completed the training of the following model versions, with the smallest
> requiring only 67M (0.067B) parameters, capable of both image recognition and conversation!
-| Model (Size) | Inference Memory | Release |
-|---------------------------|------------------|------------|
-| minimind-3v-moe (201M-A67M) | 1.0 GB | 2026.04.01 |
-| minimind-3v (67M) | 0.5 GB | 2026.04.01 |
-| MiniMind2-V (104M) | 1.1 GB | 2025.02.20 |
-| MiniMind2-Small-V (26M) | 0.6 GB | 2025.02.20 |
-| minimind-v-v1-small (27M) | 0.6 GB | 2024.10.04 |
-| minimind-v-v1 (109M) | 1.1 GB | 2024.10.04 |
+| Model (Size) | Release |
+|---|---|
+| minimind-3v-moe (201M-A67M) | 2026.04.20 |
+| minimind-3v (67M) | 2026.04.20 |
+| MiniMind2-V (104M) | 2025.02.20 |
+| MiniMind2-Small-V (26M) | 2025.02.20 |
+| minimind-v-v1-small (27M) | 2024.10.04 |
+| minimind-v-v1 (109M) | 2024.10.04 |
### 👉**Recent Updates**
+
-
### 3' Start Training
-**3.1 Pretraining (Learning image description)**
+**3.1 Pretrain (optional)**
+
+> `sft_i2t.parquet` already contains the Pretrain data as a subset, so **this stage can be skipped** and you may go directly to SFT with `--from_weight llm`. If you prefer the Projector to be pre-aligned first and SFT to converge more stably, run a round of Pretrain separately.
```bash
# Basic training command (start from LLM weights, train vision_proj only)
-python train_pretrain_vlm.py --epochs 4 --from_weight llm
+python train_pretrain_vlm.py --epochs 2 --from_weight llm
```
-> Run pretraining to get `pretrain_vlm_*.pth` as the pretrained model's output weights (* represents the model
-> dimension, default is 768).
+> Run Pretrain to get `pretrain_vlm_*.pth` as the Pretrain output weights (* is the model dimension, default 768).
-**3.2 Supervised Fine-Tuning (Learning image-caption dialogue style)**
+**3.2 SFT (required)**
```bash
-# Basic training command (start from pretrain weights, full parameter fine-tuning)
+# Basic command (default --freeze_llm 1: unfreeze proj + first/last LLM layers, keep middle layers' original LLM weights)
+# If 3.1 was run: --from_weight pretrain_vlm; if Pretrain was skipped: --from_weight llm
python train_sft_vlm.py --epochs 2 --from_weight pretrain_vlm
```
-> Perform supervised fine-tuning to get `sft_vlm_*.pth` as the output weights for the fine-tuned model.
+> Run SFT to get `sft_vlm_*.pth` as the SFT output weights.
@@ -345,17 +354,11 @@ and `wandb_run_name` parameters.
# 📌 VLM Detail
-The base language model of MiniMind-V (VLM), MiniMind (LLM), comes from the twin
-project [minimind](https://github.com/jingyaogong/minimind). For detailed information on the model structure, training
-specifics, principles, and testing results, please refer to the [minimind](https://github.com/jingyaogong/minimind)
-project. To reduce redundancy, the discussion on LLM-related topics is omitted here, assuming you have a basic
-understanding of MiniMind (LLM).
+The language backbone of MiniMind-V is the `llm_768.pth` trained by the sibling project [minimind](https://github.com/jingyaogong/minimind). The LLM's own structure, training details and experimental analysis are not repeated here; the default assumption is that the reader has a basic understanding of MiniMind LLM. Not having touched the LLM project does not prevent following the "Quick Start" to get MiniMind-V running — the flow is self-contained.
-> Even if you are not very familiar with the details of LLMs, you can still follow the "Quick Start" guide to train a
-> MiniMind-V, as it remains unaffected and the repository focuses on the lowest cost for out-of-the-box use!
+The two shorthand labels on the landing page — "from scratch" and "67M" — also need a stricter reading here. "From scratch" means the VLM itself is trained from zero (Projection randomly initialized, first/last LLM layers fine-tuned for alignment), but the LLM backbone is not pretrained from zero — it is continued from the weights of MiniMind. For a strictly "from-zero pretraining" setup, first pretrain an LLM in MiniMind and then plug it back here. "67M" refers to the trainable backbone (LLM ~64M + Projection ~3M); the SigLIP2 vision encoder contributes another ~93M parameters that stay frozen throughout and serve only as an image feature extractor, so the full model at inference time is roughly 160M (dense) / 294M (MoE).
-MiniMind-V's structure adds two submodules, a Visual Encoder and a feature projection, with a modality-mixing branch to
-support inputs from multiple modalities:
+The VLM adds a Visual Encoder and a feature projection on top of the LLM, introducing a modality-mixing branch to support multimodal inputs:
@@ -412,16 +415,13 @@ outputs the end token; here, the "token" doesn’t necessarily have to be text!
The "foreign language dictionary" is referred to as the Visual Encoder model.
Like LlaVA, Qwen-VL, and other visual language models, MiniMind-V now uses the open-source SigLIP2 series models as the
Visual Encoder.
-Specifically, we use [siglip2-base-p16-ve](https://huggingface.co/jingyaogong/siglip2-base-p16-ve), a Visual
+Specifically, we use [siglip2-base-p16-256-ve](https://huggingface.co/jingyaogong/siglip2-base-p16-256-ve), a Visual
Encoder based on the ViT-B/16 architecture for describing image-text information.
The current SigLIP2 NaFlex vision encoder generates up to 256 patch tokens from the processor output as the input to the
encoder layer, which produces a 1×768 dimensional embedding vector for calculating error with the text.
We don’t need the final embedding representation, so we only take the output from the encoder layer, which is the output
feature from the core ViT backbone.
-It receives 256×768 features from the previous layer, which are then reshaped by concatenating every 4 adjacent tokens into 1 (256×768 → 64×3072), then projected to the LLM's hidden dimension via a 2-layer MLP (Linear→GELU→Linear), resulting in 64 visual tokens input into MiniMind-V.
-After obtaining the image encoder features, the integration with the LLM requires aligning the visual features to the LLM's text token dimension, and mapping the image features into the same space as text embeddings. In other
-words, the image features and native visual tokens cannot be directly treated the same; they require cross-modal feature
-alignment.
+It receives 256×768 features from the previous layer, which are then reshaped by concatenating every 4 adjacent tokens into 1 (256×768 → 64×3072), then projected to the LLM's hidden dimension via a 2-layer MLP (Linear→GELU→Linear), resulting in 64 visual tokens fed into MiniMind-V — this step is exactly cross-modal feature alignment: the native visual features are brought into the semantic space where text tokens live, so that the two can interact in the same space.
[LlaVA-1](https://arxiv.org/pdf/2304.08485) achieves good alignment with a simple linear transformation, [LlaVA-1.5](https://arxiv.org/pdf/2310.03744) upgrades to a 2-layer MLP. MiniMind-V adopts the same MLP Projection approach as LlaVA-1.5, combined with reshape for token compression.
@@ -462,63 +462,83 @@ At this point, all the details of `MiniMind-V` have been presented. The VLM mode
## Ⅰ Dataset
-Original Source:
-- [Chinese-LLaVA-Vision](https://huggingface.co/datasets/LinkSoul/Chinese-LLaVA-Vision-Instructions): Contains approximately 570,000 pre-trained images from CC-3M and COCO 2014
-- [llava-en-zh-300k](https://huggingface.co/datasets/BUAADreamer/llava-en-zh-300k): Contains 300k instruction fine-tuning data and 150k images
-- [LLaVA-SFT-665K](https://huggingface.co/datasets/csuhan/LLaVA-SFT-665K): Contains 665k instruction fine-tuning data
+All image-text data used in this round come from the [ALLaVA-4V](https://huggingface.co/datasets/FreedomIntelligence/ALLaVA-4V) family.
+Compared with data stitched together from earlier LLaVA-derived sets, ALLaVA-4V is more consistent in quality, natively paired in Chinese and English, and more thorough in its fine-grained descriptions.
+It is composed of two image sources: a curated subset of LAION (mostly natural images) and a curated subset of VFLAN (documents, charts, synthetic scenes).
-The dataset contains both Chinese and English data. The Q&A content has been translated, with better support for Chinese, further organized and resized (pretrain resolution 128×128, sft resolution 160×160).
+- **Pretrain (`pretrain_i2t.parquet`, ~1.27M rows / ~640K unique images)**
+ - `ALLaVA-Caption-LAION-4V` en/zh: ~470K + ~440K
+ - `ALLaVA-Caption-VFLAN-4V` en/zh: ~195K + ~170K
+ - Single-turn "describe this image" style captions used to establish the basic alignment from visual tokens to language tokens.
-(pretrain_i2t.parquet) Pre-training dataset format:
+- **SFT (`sft_i2t.parquet`, ~2.90M rows / ~650K unique images)**
+ - `ALLaVA-Instruct-LAION-4V` en/zh: ~470K + ~470K
+ - `ALLaVA-Instruct-VFLAN-4V` en/zh: ~195K + ~165K
+ - `Instruct-LAION-4v-gemini-claude-ensembled` (synthesized by Gemini/Claude): ~50K
+ - `Instruct-LAION-4oiterative` (iteratively refined by GPT-4o): ~50K
+ - Pure-text conversations (8×8 black placeholder images, preserving base language ability): ~230K
+ - **Full Pretrain caption data merged in** (same source as pretrain, ~99% image overlap): ~1.27M
+ - A blend of "image-grounded reasoning Q&A", "caption-style long descriptions" and "pure-text chat" — covering fine-grained follow-ups/long chains of thought as well as image description and general language ability.
+
+Roughly 2.9M samples in total. The Pretrain stage can be skipped entirely (SFT has absorbed it as a subset). Chinese and English are roughly balanced.
+Given MiniMind-V's trainable backbone is only 67M, mixing English and Chinese is a pragmatic choice: Chinese data helps native-language generation, while the original English descriptions tend to be more precise — the two complement each other.
+
+All images are `resize`d to **256×256** (matching SigLIP2 NaFlex's 256 patch-token input) and re-encoded as JPEG, packed directly into parquet.
+
+(`pretrain_i2t.parquet`) Pre-training dataset format:
```text
-Columns: conversations (json string), image_bytes (binary), image_names (string)
+Columns: conversations (json string), image_bytes (binary)
conversations example:
[
- {"role": "user", "content": "Provide a brief description of the given image.\n"},
- {"role": "assistant", "content": "Olive oil is a healthy ingredient for free use."}
+ {"role": "user", "content": "\nPlease describe this image in detail."},
+ {"role": "assistant", "content": "The image shows..."}
]
image_bytes:
```
-(sft_i2t.parquet) Single image instruction fine-tuning dataset format:
+(`sft_i2t.parquet`) Single-image SFT dataset format:
```text
-Columns: conversations (json string), image_bytes (binary), image_names (string)
+Columns: conversations (json string), image_bytes (binary)
conversations example:
[
- {"role": "user", "content": "What impact does the location of the alarm clock have on sleep quality?"},
- {"role": "assistant", "content": "Place the digital alarm clock on the nightstand..."}
+ {"role": "user", "content": "Based on the image, what time of day is it?"},
+ {"role": "assistant", "content": "Judging from the light and shadows..."}
]
image_bytes:
```
-> Note: sft_i2t.parquet contains ~580K samples in total, of which ~236K are image-text conversations (i2t) and ~346K are pure text conversations (t2t). The latter is used to preserve the model's base language capabilities.
+> Note: sft_i2t.parquet contains ~2.9M samples, of which ~1.40M are image instruct conversations, ~1.27M are image caption descriptions (merged from Pretrain), and ~230K are pure-text conversations (t2t, image column filled by an 8×8 black placeholder) used to preserve the model's base language capabilities. Since Pretrain is already included as a subset, the Pretrain stage can be skipped and SFT run directly.
-Dataset download
-link: ([ModelScope](https://www.modelscope.cn/datasets/gongjy/minimind-v_dataset) | [HuggingFace](https://huggingface.co/datasets/jingyaogong/minimind-v_dataset))
+Dataset download link: ([ModelScope](https://www.modelscope.cn/datasets/gongjy/minimind-v_dataset) | [HuggingFace](https://huggingface.co/datasets/jingyaogong/minimind-v_dataset))
## Ⅱ Training
-Training is divided into two stages, both freezing the Visual Encoder gradients and only training the Projection and LLM parts.
-Training is initialized from LLM pre-trained weights, with support for DDP multi-GPU training, mixed precision (bfloat16), torch.compile acceleration, and swanlab logging.
+Training has two stages (Pretrain optional; SFT required). Both freeze the Visual Encoder and train only the Projection and part of the LLM layers.
+Training is initialized from LLM Pretrain weights, with support for DDP multi-GPU training, mixed precision (bfloat16), torch.compile acceleration, and swanlab logging.
-> train_pretrain_vlm
+> train_pretrain_vlm (optional)
-The pre-training stage learns general image knowledge from ~1.13M image-text description pairs (e.g., a deer is a deer, a dog is a dog).
-This stage uses a higher learning rate (1e-4), max sequence length of 360, freezes the LLM main parameters, and only sets the Projection and LLM's layer 0 as learnable,
-aiming to quickly establish a basic mapping from visual features to the language space while avoiding damage to the LLM's existing language capabilities.
+The Pretrain stage learns general image knowledge from ~1.27M image-text description pairs (e.g., a deer is a deer, a dog is a dog).
+It uses a higher learning rate (~4e-4), max sequence length 450, and **fully freezes the LLM and Visual Encoder, training only the Projection** (`--freeze_llm 2`).
+The goal is to let the Projector cleanly align visual tokens to the language space without perturbing the original LLM weights.
+Since SFT data already contains all Pretrain samples as a subset, this stage is optional; skipping it saves time, while **running one round of Pretrain first lets the Projector pre-align and makes SFT converge more steadily**.
> train_sft_vlm
-The instruction fine-tuning stage learns real Q&A formats from ~580K samples, of which ~236K are image-text multi-turn conversations and ~346K are pure text conversations (to preserve LLM base capabilities).
-This stage uses a lower learning rate (1e-5~1e-6), max sequence length of 768, unfreezes all Projection and LLM parameters for full fine-tuning,
-enabling the model to conduct multi-turn conversations based on image content, while mitigating catastrophic forgetting through the mixed-in pure text data.
+The SFT stage uses the aforementioned `sft_i2t.parquet` — about 2.9M mixed samples, covering the image captions inherited from Pretrain, reasoning-style Q&A on natural images, fine-grained Q&A on documents/charts, instructions synthesized by Gemini/Claude/GPT-4o, plus ~230K pure-text conversations (image column filled by an 8×8 black placeholder). Learning rate drops to ~5e-5, max sequence 768.
+
+A common practice is to fully unfreeze the LLM during SFT, but this usually assumes a several-B-parameter base and a substantial amount of pure-text data mixed into SFT. MiniMind-V's language backbone is only 64M and ~92% of the current SFT data is image-related, so fully unfreezing the LLM would likely dilute its original general-language capability under the image-task gradients.
+
+We therefore use `--freeze_llm 1`: **only the Projection and the first & last LLM layers are unfrozen, while the remaining N-2 layers keep their Pretrain weights**. The first layer is the first processing stage after visual tokens enter the LLM and thus bears the cross-modal fusion; the last layer shapes the format and style of the answer; the middle layers retain the knowledge from LLM Pretrain and are not overwritten by image-task gradients. The ~230K pure-text samples further act as a regularizer for general-language capability.
> Training Time and Loss Trend (for reference only)
+On a single NVIDIA 3090, SFT takes ~2 hours per `epoch` in practice; dense and MoE finish in similar time (activated parameters are on the same order, with the gap mostly coming from the extra memory traffic of expert routing). Pretrain data volume is ~45% of SFT's, so one Pretrain epoch can be roughly scaled by that ratio. At a typical cloud price of ~1.5 RMB/hour for a 3090, a full SFT round costs about 3 RMB.
+
Pretrain [768+8] (dense & moe)
@@ -532,13 +552,13 @@ SFT [768+8] (dense & moe)
| Native PyTorch (`*.pth`) | [minimind-3v-pytorch](https://www.modelscope.cn/models/gongjy/minimind-3v-pytorch) | [minimind-3v-pytorch](https://huggingface.co/jingyaogong/minimind-3v-pytorch) |
| Transformers | [minimind-v collection](https://modelscope.cn/collections/MiniMind-V-42b841dde22d41) | [minimind-v collection](https://huggingface.co/collections/jingyaogong/minimind-v-67000833fb60b3a2e1f3597d) |
-> Note: The Transformers version is the `MiniMind-V` model after single-image instruction fine-tuning
+> Note: The Transformers version is the `MiniMind-V` model after single-image SFT
# 📌 Test
### Effect Test
-#### Single Image Dialogue
+Prompt: `\nPlease illustrate the image through your words.`
@@ -551,92 +571,62 @@ SFT [768+8] (dense & moe)
### Effect Summary:
-Both models can identify image subjects (airplane, cake, car, beach, etc.), but commonly exhibit repetitive expressions and hallucinated details. Limited by model and data scale, the overall performance is at a stage of "understanding the gist but inaccurate on details".
+Both models can identify the primary subject in most images (dog, umbrella, bicycle, sports car, superhero, racecar, etc.), but both exhibit repetitive phrasing and hallucinated details, placing overall performance at a stage of "understanding the gist but inaccurate on details".
-Visual signals are treated as a special foreign language by LLMs, so the "language learning" ability highly depends on the LLM's capacity. The stronger the LLM, the more powerful the corresponding VLM, and the performance boost becomes significant.
+Across these 6 samples, the MoE variant produces slightly richer scene descriptions and better color recognition (e.g. the cherry-blossom scene, rainbow umbrella, yellow sports car), while the Dense model tends to be more concise. Over a broader evaluation set, however, MoE occasionally suffers from more pronounced entity-level hallucinations or repetitive generation, showing higher variance than Dense — making Dense the safer default configuration.
+
+Visual signals act as a special "foreign language" to the LLM, so the ceiling of "learning that language" is bounded by the LLM's own language ability. A stronger backbone extracts more value from the same image-text data; swapping MiniMind-V's backbone for a several-B-scale LLM yields clearly sharper details and more coherent reasoning.
#### Future Areas for Improvement:
@@ -650,10 +640,8 @@ Visual signals are treated as a special foreign language by LLMs, so the "langua
# 📌 Acknowledge
> [!TIP]
-> If you find `MiniMind-V` helpful, please consider giving it a ⭐ on GitHub.
-> Given the limited expertise, there may be unknown issues, and we welcome everyone to discuss, correct, or submit PRs -> to improve the project in Issues.
-> Your support is the driving force behind continuous improvements to the project. Thank you! +> If `MiniMind-V` is useful to you, a ⭐ on GitHub is welcome.
+> Issues and PRs are the best place to share problems or improvements found while using the project. ## 🤝 [Contributors](https://github.com/jingyaogong/minimind-v/graphs/contributors) diff --git a/dataset/eval_images/airplane-flying-blue-sky.jpg b/dataset/eval_images/airplane-flying-blue-sky.jpg deleted file mode 100644 index 3ef3d6a..0000000 Binary files a/dataset/eval_images/airplane-flying-blue-sky.jpg and /dev/null differ diff --git a/dataset/eval_images/birthday-cake-candles-table.jpg b/dataset/eval_images/birthday-cake-candles-table.jpg deleted file mode 100644 index baacc28..0000000 Binary files a/dataset/eval_images/birthday-cake-candles-table.jpg and /dev/null differ diff --git a/dataset/eval_images/image-01-golden-dog-balloons.jpg b/dataset/eval_images/image-01-golden-dog-balloons.jpg new file mode 100644 index 0000000..53923c4 Binary files /dev/null and b/dataset/eval_images/image-01-golden-dog-balloons.jpg differ diff --git a/dataset/eval_images/image-02-rainbow-umbrella-street.jpg b/dataset/eval_images/image-02-rainbow-umbrella-street.jpg new file mode 100644 index 0000000..7b3201c Binary files /dev/null and b/dataset/eval_images/image-02-rainbow-umbrella-street.jpg differ diff --git a/dataset/eval_images/image-03-cherry-blossom-bike.jpg b/dataset/eval_images/image-03-cherry-blossom-bike.jpg new file mode 100644 index 0000000..df4f184 Binary files /dev/null and b/dataset/eval_images/image-03-cherry-blossom-bike.jpg differ diff --git a/dataset/eval_images/image-04-yellow-car.jpg b/dataset/eval_images/image-04-yellow-car.jpg new file mode 100644 index 0000000..1383b44 Binary files /dev/null and b/dataset/eval_images/image-04-yellow-car.jpg differ diff --git a/dataset/eval_images/image-05-superhero-rooftop.jpg b/dataset/eval_images/image-05-superhero-rooftop.jpg new file mode 100644 index 0000000..987b373 Binary files /dev/null and b/dataset/eval_images/image-05-superhero-rooftop.jpg differ diff --git a/dataset/eval_images/image-06-racecar-drift.jpg b/dataset/eval_images/image-06-racecar-drift.jpg new file mode 100644 index 0000000..3938c35 Binary files /dev/null and b/dataset/eval_images/image-06-racecar-drift.jpg differ diff --git a/dataset/eval_images/pizza-on-wooden-board.jpg b/dataset/eval_images/pizza-on-wooden-board.jpg deleted file mode 100644 index 1b3ceb4..0000000 Binary files a/dataset/eval_images/pizza-on-wooden-board.jpg and /dev/null differ diff --git a/dataset/eval_images/red-sports-car-road.jpg b/dataset/eval_images/red-sports-car-road.jpg deleted file mode 100644 index ba8695b..0000000 Binary files a/dataset/eval_images/red-sports-car-road.jpg and /dev/null differ diff --git a/dataset/eval_images/row-of-colorful-houses.jpg b/dataset/eval_images/row-of-colorful-houses.jpg deleted file mode 100644 index 206a395..0000000 Binary files a/dataset/eval_images/row-of-colorful-houses.jpg and /dev/null differ diff --git a/dataset/eval_images/snow-mountain-lake-view.jpg b/dataset/eval_images/snow-mountain-lake-view.jpg deleted file mode 100644 index 1bf1bb7..0000000 Binary files a/dataset/eval_images/snow-mountain-lake-view.jpg and /dev/null differ diff --git a/dataset/eval_images/street-food-hotpot-table.jpg b/dataset/eval_images/street-food-hotpot-table.jpg deleted file mode 100644 index 2c205b9..0000000 Binary files a/dataset/eval_images/street-food-hotpot-table.jpg and /dev/null differ diff --git a/dataset/eval_images/three-kittens-basket.jpg b/dataset/eval_images/three-kittens-basket.jpg deleted file mode 100644 index 6df31c6..0000000 Binary files a/dataset/eval_images/three-kittens-basket.jpg and /dev/null differ diff --git a/dataset/eval_images/tropical-beach-palm-tree.jpg b/dataset/eval_images/tropical-beach-palm-tree.jpg deleted file mode 100644 index 50f0b9d..0000000 Binary files a/dataset/eval_images/tropical-beach-palm-tree.jpg and /dev/null differ diff --git a/dataset/eval_images/yellow-school-bus-road.jpg b/dataset/eval_images/yellow-school-bus-road.jpg deleted file mode 100644 index f2c87bf..0000000 Binary files a/dataset/eval_images/yellow-school-bus-road.jpg and /dev/null differ diff --git a/dataset/lm_dataset.py b/dataset/lm_dataset.py index 5bc86c5..a22a1d5 100644 --- a/dataset/lm_dataset.py +++ b/dataset/lm_dataset.py @@ -118,9 +118,9 @@ class VLMDataset(Dataset): if __name__ == '__main__': import matplotlib.pyplot as plt; plt.rcParams['font.sans-serif'] = ['Arial Unicode MS', 'SimHei'] for path in ['pretrain_i2t.parquet', 'sft_i2t.parquet']: - t = pa.Table.from_batches(pq.ParquetFile(path).iter_batches()); fig, ax = plt.subplots(1, 5, figsize=(20, 4)) + pf = pq.ParquetFile(path); n = pf.num_row_groups; t = pa.concat_tables([pf.read_row_group(i * n // 5).slice(0, 1) for i in range(5)]); fig, ax = plt.subplots(1, 5, figsize=(20, 4)) for i in range(5): img_data = t['image_bytes'][i].as_py(); img_data = img_data[0] if isinstance(img_data, list) else img_data ax[i].imshow(Image.open(io.BytesIO(img_data))); ax[i].axis('off') ax[i].set_title(json.loads(t['conversations'][i].as_py())[1]['content'][:30], fontsize=8) - out = path.replace('.parquet', '_preview.png'); plt.savefig(out); print(f'已保存{out}, 共{len(t)}条') + out = path.replace('.parquet', '_preview.png'); plt.savefig(out); print(f'已保存{out}, 共{pf.metadata.num_rows}条') diff --git a/eval_vlm.py b/eval_vlm.py index 433ce6e..8865986 100644 --- a/eval_vlm.py +++ b/eval_vlm.py @@ -11,19 +11,19 @@ from trainer.trainer_utils import setup_seed, get_model_params warnings.filterwarnings('ignore') def init_model(args): - tokenizer = AutoTokenizer.from_pretrained(args.load_from) + tokenizer = AutoTokenizer.from_pretrained(args.load_from, trust_remote_code=True) if 'model' in args.load_from: moe_suffix = '_moe' if args.use_moe else '' ckp = f'./{args.save_dir}/{args.weight}_{args.hidden_size}{moe_suffix}.pth' model = MiniMindVLM( VLMConfig(hidden_size=args.hidden_size, num_hidden_layers=args.num_hidden_layers, use_moe=bool(args.use_moe)), - vision_model_path="./model/siglip2-base-p16-ve" + vision_model_path="./model/siglip2-base-p16-256-ve" ) state_dict = torch.load(ckp, map_location=args.device) model.load_state_dict({k: v for k, v in state_dict.items() if 'mask' not in k}, strict=False) else: model = AutoModelForCausalLM.from_pretrained(args.load_from, trust_remote_code=True) - model.vision_encoder, model.processor = MiniMindVLM.get_vision_model("./model/siglip2-base-p16-ve") + model.vision_encoder, model.processor = MiniMindVLM.get_vision_model("./model/siglip2-base-p16-256-ve") get_model_params(model, model.config) preprocess = model.processor return model.half().eval().to(args.device), tokenizer, preprocess @@ -38,8 +38,8 @@ def main(): parser.add_argument('--num_hidden_layers', default=8, type=int, help="隐藏层数量") parser.add_argument('--use_moe', default=0, type=int, choices=[0, 1], help="是否使用MoE架构（0=否，1=是）") parser.add_argument('--max_new_tokens', default=512, type=int, help="最大生成长度") - parser.add_argument('--temperature', default=0.9, type=float, help="生成温度，控制随机性（0-1，越大越随机）") - parser.add_argument('--top_p', default=0.9, type=float, help="nucleus采样阈值（0-1）") + parser.add_argument('--temperature', default=0.7, type=float, help="生成温度，控制随机性（0-1，越大越随机）") + parser.add_argument('--top_p', default=0.85, type=float, help="nucleus采样阈值（0-1）") parser.add_argument('--image_dir', default='./dataset/eval_images/', type=str, help="测试图像目录") parser.add_argument('--show_speed', default=1, type=int, help="显示decode速度（tokens/s）") parser.add_argument('--device', default='cuda' if torch.cuda.is_available() else 'cpu', type=str, help="运行设备") @@ -49,11 +49,12 @@ def main(): model, tokenizer, preprocess = init_model(args) streamer = TextStreamer(tokenizer, skip_prompt=True, skip_special_tokens=True) # 自动测试image_dir中的所有图像 - prompt = "请仔细观察这张图片，描述你所看到的内容：\n\n"
- # prompt = "Describe the content of this image in detail:\n\n"
+ prompt = "\n请描述这张图中的主要物体和场景。"
+ # prompt = "\nPlease illustrate the image through your words."
for image_file in sorted(os.listdir(args.image_dir)):
if image_file.lower().endswith(('.png', '.jpg', '.jpeg', '.bmp')):
setup_seed(random.randint(1, 31415926))
+ setup_seed(512)
image_path = os.path.join(args.image_dir, image_file)
image = Image.open(image_path).convert('RGB')
pixel_values = {k: v.to(args.device) for k, v in MiniMindVLM.image2tensor(image, preprocess).items()}
diff --git a/images/minimind-3v.gif b/images/minimind-3v.gif
index c91a875..43a0156 100644
Binary files a/images/minimind-3v.gif and b/images/minimind-3v.gif differ
diff --git a/model/model_init.md b/model/model_init.md
index b3b14f3..2538149 100644
--- a/model/model_init.md
+++ b/model/model_init.md
@@ -1,23 +1,23 @@
## Download Vision Encoder (Required)
-This model requires an additional vision encoder `siglip2-base-p16-ve`
+This model requires an additional vision encoder `siglip2-base-p16-256-ve`
```bash
git lfs install
# from HuggingFace
-git clone https://huggingface.co/jingyaogong/siglip2-base-p16-ve
+git clone https://huggingface.co/jingyaogong/siglip2-base-p16-256-ve
# or from ModelScope
-git clone https://modelscope.cn/models/gongjy/siglip2-base-p16-ve
+git clone https://modelscope.cn/models/gongjy/siglip2-base-p16-256-ve
```
Directory Structure
```
model/
-├── siglip2-base-p16-ve/ # Vision Encoder (Required)
+├── siglip2-base-p16-256-ve/ # Vision Encoder (Required)
│ ├── config.json
│ ├── model.safetensors # ~180MB, float16
│ └── preprocessor_config.json
diff --git a/model/model_minimind.py b/model/model_minimind.py
index d3518ad..35bb66f 100755
--- a/model/model_minimind.py
+++ b/model/model_minimind.py
@@ -27,6 +27,7 @@ class MiniMindConfig(PretrainedConfig):
self.max_position_embeddings = kwargs.get("max_position_embeddings", 32768)
self.rms_norm_eps = kwargs.get("rms_norm_eps", 1e-6)
self.rope_theta = kwargs.get("rope_theta", 1e6)
+ self.tie_word_embeddings = kwargs.get("tie_word_embeddings", True)
self.inference_rope_scaling = kwargs.get("inference_rope_scaling", False)
self.rope_scaling = {
"beta_fast": 32,
@@ -211,6 +212,10 @@ class MiniMindModel(nn.Module):
past_key_values = past_key_values or [None] * len(self.layers)
start_pos = past_key_values[0][0].shape[1] if past_key_values[0] is not None else 0
hidden_states = self.dropout(self.embed_tokens(input_ids))
+ # Recompute RoPE buffers lost during meta-device init (transformers>=5.x)
+ if self.freqs_cos[0, 0] == 0:
+ freqs_cos, freqs_sin = precompute_freqs_cis(dim=self.config.head_dim, end=self.config.max_position_embeddings, rope_base=self.config.rope_theta, rope_scaling=self.config.rope_scaling)
+ self.freqs_cos, self.freqs_sin = freqs_cos.to(hidden_states.device), freqs_sin.to(hidden_states.device)
position_embeddings = (self.freqs_cos[start_pos:start_pos + seq_length], self.freqs_sin[start_pos:start_pos + seq_length])
presents = []
for layer, past_key_value in zip(self.layers, past_key_values):
@@ -228,13 +233,15 @@ class MiniMindModel(nn.Module):
class MiniMindForCausalLM(PreTrainedModel, GenerationMixin):
config_class = MiniMindConfig
+ _tied_weights_keys = {"lm_head.weight": "model.embed_tokens.weight"}
def __init__(self, config: MiniMindConfig = None):
self.config = config or MiniMindConfig()
super().__init__(self.config)
self.model = MiniMindModel(self.config)
self.lm_head = nn.Linear(self.config.hidden_size, self.config.vocab_size, bias=False)
- self.model.embed_tokens.weight = self.lm_head.weight
-
+ if self.config.tie_word_embeddings: self.model.embed_tokens.weight = self.lm_head.weight
+ self.post_init()
+
def forward(self, input_ids, attention_mask=None, past_key_values=None, use_cache=False, logits_to_keep=0, labels=None, **kwargs):
hidden_states, past_key_values, aux_loss = self.model(input_ids, attention_mask, past_key_values, use_cache, **kwargs)
slice_indices = slice(-logits_to_keep, None) if isinstance(logits_to_keep, int) else logits_to_keep
diff --git a/model/model_vlm.py b/model/model_vlm.py
index eb5e0d9..35b6eba 100644
--- a/model/model_vlm.py
+++ b/model/model_vlm.py
@@ -4,7 +4,7 @@ import warnings
from .model_minimind import *
from typing import Optional, Tuple, List, Union
from torch import nn
-from transformers import Siglip2ImageProcessor, Siglip2VisionModel
+from transformers import SiglipImageProcessor, SiglipVisionModel
from transformers.modeling_outputs import MoeCausalLMOutputWithPast
warnings.filterwarnings('ignore')
@@ -26,20 +26,23 @@ class MMVisionProjector(nn.Module):
self.target_tokens = target_tokens
self.merge = source_tokens // target_tokens
self.mlp = nn.Sequential(
+ nn.LayerNorm(in_dim * self.merge),
nn.Linear(in_dim * self.merge, out_dim),
nn.GELU(),
nn.Linear(out_dim, out_dim),
)
def forward(self, x):
b, n, d = x.shape
- x = x.reshape(b, self.target_tokens, d * self.merge)
+ side = int(n ** 0.5)
+ s = int(self.merge ** 0.5)
+ x = x.view(b, side // s, s, side // s, s, d).permute(0, 1, 3, 2, 4, 5).reshape(b, self.target_tokens, d * self.merge)
return self.mlp(x)
# 继承自语言模型
class MiniMindVLM(MiniMindForCausalLM):
config_class = VLMConfig
- def __init__(self, config: VLMConfig = None, vision_model_path="./model/siglip2-base-p16-ve"):
+ def __init__(self, config: VLMConfig = None, vision_model_path="./model/siglip2-base-p16-256-ve"):
self.config = config or VLMConfig()
super().__init__(self.config)
self.vision_encoder, self.processor = self.__class__.get_vision_model(vision_model_path)
@@ -51,8 +54,11 @@ class MiniMindVLM(MiniMindForCausalLM):
hf_logging.set_verbosity_error()
if not os.path.exists(model_path):
return None, None
- model = Siglip2VisionModel.from_pretrained(model_path)
- processor = Siglip2ImageProcessor.from_pretrained(model_path)
+ try:
+ model = SiglipVisionModel.from_pretrained(model_path)
+ except (RuntimeError, ValueError):
+ return None, None
+ processor = SiglipImageProcessor.from_pretrained(model_path)
# 冻结 vision_encoder 的所有参数
for param in model.parameters():
param.requires_grad = False
@@ -126,6 +132,10 @@ class MiniMindVLM(MiniMindForCausalLM):
vision_tensors = torch.stack([self.vision_proj(MiniMindVLM.get_image_embeddings(pixel_values[:, i, :, :, :], self.vision_encoder)) for i in range(num)], dim=1)
hidden_states = self.count_vision_proj(tokens=input_ids, h=hidden_states, vision_tensors=vision_tensors, seqlen=input_ids.shape[1])
+ # Recompute RoPE buffers lost during meta-device init (transformers>=5.x)
+ if self.model.freqs_cos[0, 0] == 0:
+ freqs_cos, freqs_sin = precompute_freqs_cis(dim=self.config.head_dim, end=self.config.max_position_embeddings, rope_base=self.config.rope_theta, rope_scaling=self.config.rope_scaling)
+ self.model.freqs_cos, self.model.freqs_sin = freqs_cos.to(hidden_states.device), freqs_sin.to(hidden_states.device)
position_embeddings = (
self.model.freqs_cos[start_pos:start_pos + seq_length],
self.model.freqs_sin[start_pos:start_pos + seq_length]
diff --git a/scripts/convert_vlm.py b/scripts/convert_vlm.py
index 99e457f..55452f1 100644
--- a/scripts/convert_vlm.py
+++ b/scripts/convert_vlm.py
@@ -1,9 +1,11 @@
import os
import sys
+import json
__package__ = "scripts"
sys.path.append(os.path.abspath(os.path.join(os.path.dirname(__file__), '..')))
import torch
+import transformers
import warnings
from transformers import AutoTokenizer, AutoModelForCausalLM, LlamaConfig, LlamaForCausalLM
from model.model_vlm import MiniMindVLM, VLMConfig
@@ -14,7 +16,7 @@ warnings.filterwarnings('ignore', category=UserWarning)
def convert_torch2transformers_minimind(torch_path, transformers_path, dtype=torch.bfloat16):
VLMConfig.register_for_auto_class()
MiniMindVLM.register_for_auto_class("AutoModelForCausalLM")
- lm_model = MiniMindVLM(lm_config, vision_model_path="../model/siglip2-base-p16-ve")
+ lm_model = MiniMindVLM(lm_config, vision_model_path="../model/siglip2-base-p16-256-ve")
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
state_dict = torch.load(torch_path, map_location=device)
lm_model.load_state_dict(state_dict, strict=False)
@@ -25,6 +27,16 @@ def convert_torch2transformers_minimind(torch_path, transformers_path, dtype=tor
lm_model.save_pretrained(transformers_path, safe_serialization=False)
tokenizer = AutoTokenizer.from_pretrained('../model/')
tokenizer.save_pretrained(transformers_path)
+ # 显式写入 tie_word_embeddings（save_pretrained 默认不序列化与默认值相同的字段）
+ config_path = os.path.join(transformers_path, "config.json")
+ config = json.load(open(config_path, 'r', encoding='utf-8'))
+ config['tie_word_embeddings'] = True
+ # ======= transformers-5.0的兼容低版本写法 =======
+ if int(transformers.__version__.split('.')[0]) >= 5:
+ tokenizer_config_path = os.path.join(transformers_path, "tokenizer_config.json")
+ json.dump({**json.load(open(tokenizer_config_path, 'r', encoding='utf-8')), "tokenizer_class": "PreTrainedTokenizerFast", "extra_special_tokens": {}}, open(tokenizer_config_path, 'w', encoding='utf-8'), indent=2, ensure_ascii=False)
+ config['rope_theta'] = lm_config.rope_theta; config['rope_scaling'] = None; config.pop('rope_parameters', None)
+ json.dump(config, open(config_path, 'w', encoding='utf-8'), indent=2, ensure_ascii=False)
print(f"模型已保存为 Transformers-MiniMind-V 格式: {transformers_path}")
diff --git a/scripts/web_demo_vlm.py b/scripts/web_demo_vlm.py
index 41a7d58..a3eb1b6 100644
--- a/scripts/web_demo_vlm.py
+++ b/scripts/web_demo_vlm.py
@@ -34,7 +34,7 @@ def scan_vlm_models(base_dir):
def load_vlm_model(model_path):
global model, tokenizer, preprocess, lm_config, current_model_name
- tokenizer = AutoTokenizer.from_pretrained(model_path)
+ tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(model_path, trust_remote_code=True)
model.vision_encoder, model.processor = MiniMindVLM.get_vision_model(vision_model_path)
if model.vision_encoder is None: raise FileNotFoundError(f"视觉编码器未找到: {vision_model_path}")
@@ -128,12 +128,12 @@ def launch_gradio_server(server_name="0.0.0.0", server_port=7788):
response += chunk
yield history + [{"role": "assistant", "content": response}]
- with gr.Blocks(title="minimind-3v", css="#chatbox img{max-width:120px!important;max-height:120px!important;border-radius:8px} div.full-container{display:flex!important;flex-direction:row!important;flex-wrap:nowrap!important;align-items:center!important} div.full-container>.thumbnails{flex-shrink:0!important;max-width:50px!important} div.full-container>.input-container{flex:1!important} .input-wrapper{display:flex!important;flex-direction:row!important;flex-wrap:nowrap!important;align-items:center!important} .input-wrapper>.thumbnails{flex-shrink:0!important;max-width:50px!important} .input-wrapper>.input-row{flex:1!important} .thumbnail-image{max-width:40px!important;max-height:40px!important} textarea{overflow-y:hidden!important}") as demo:
+ with gr.Blocks(title="minimind-3v", css="#chatbox img{max-width:120px!important;max-height:120px!important;border-radius:8px} div.full-container{display:flex!important;flex-direction:row!important;flex-wrap:nowrap!important;align-items:center!important} div.full-container>.thumbnails{flex-shrink:0!important;max-width:50px!important} div.full-container>.input-container{flex:1!important} .input-wrapper{display:flex!important;flex-direction:row!important;flex-wrap:nowrap!important;align-items:center!important} .input-wrapper>.thumbnails{flex-shrink:0!important;max-width:50px!important} .input-wrapper>.input-row{flex:1!important} .thumbnail-image{max-width:40px!important;max-height:40px!important} textarea{overflow-y:hidden!important} #chatbox,#chatbox *{scrollbar-width:none!important;-ms-overflow-style:none!important} #chatbox::-webkit-scrollbar,#chatbox *::-webkit-scrollbar{display:none!important;width:0!important;height:0!important}") as demo:
gr.HTML('
minimind-3v')
try:
- chatbot = gr.Chatbot(label="", height=520, elem_id="chatbox", type="messages")
+ chatbot = gr.Chatbot(label="", height=560, elem_id="chatbox", type="messages")
except TypeError:
- chatbot = gr.Chatbot(label="", height=520, elem_id="chatbox")
+ chatbot = gr.Chatbot(label="", height=560, elem_id="chatbox")
msg = gr.MultimodalTextbox(placeholder="输入问题，点击📎上传图片", show_label=False, submit_btn="发送")
with gr.Row():
model_dropdown = gr.Dropdown(choices=list(model_dict.keys()), value=current_model_name, show_label=False, scale=1)
@@ -150,9 +150,9 @@ def launch_gradio_server(server_name="0.0.0.0", server_port=7788):
if __name__ == '__main__':
parser = argparse.ArgumentParser(description="Chat with MiniMind-V")
parser.add_argument('--load_from', default='./', type=str, help="transformers模型扫描目录")
- parser.add_argument('--vision_model', default='../model/siglip2-base-p16-ve', type=str, help="视觉编码器路径")
- parser.add_argument('--temperature', default=0.8, type=float, help="生成温度")
- parser.add_argument('--top_p', default=0.9, type=float, help="nucleus采样阈值")
+ parser.add_argument('--vision_model', default='../model/siglip2-base-p16-256-ve', type=str, help="视觉编码器路径")
+ parser.add_argument('--temperature', default=0.7, type=float, help="生成温度")
+ parser.add_argument('--top_p', default=0.95, type=float, help="nucleus采样阈值")
parser.add_argument('--device', default='cuda' if torch.cuda.is_available() else 'cpu', type=str, help="运行设备")
parser.add_argument('--max_seq_len', default=8192, type=int, help="最大序列长度")
args = parser.parse_args()
diff --git a/trainer/train_pretrain_vlm.py b/trainer/train_pretrain_vlm.py
index c4c76e0..330987e 100644
--- a/trainer/train_pretrain_vlm.py
+++ b/trainer/train_pretrain_vlm.py
@@ -102,12 +102,12 @@ if __name__ == "__main__":
parser.add_argument("--save_interval", type=int, default=1000, help="模型保存间隔")
parser.add_argument('--hidden_size', default=768, type=int, help="隐藏层维度")
parser.add_argument('--num_hidden_layers', default=8, type=int, help="隐藏层数量")
- parser.add_argument('--max_seq_len', default=360, type=int, help="训练的最大截断长度")
+ parser.add_argument('--max_seq_len', default=450, type=int, help="训练的最大截断长度")
parser.add_argument('--use_moe', default=0, type=int, choices=[0, 1], help="是否使用MoE架构（0=否，1=是）")
parser.add_argument("--data_path", type=str, default="../dataset/pretrain_i2t.parquet", help="训练数据路径")
parser.add_argument('--from_weight', default='llm', type=str, help="基于哪个权重训练，为none则不基于任何权重训练")
parser.add_argument('--from_resume', default=0, type=int, choices=[0, 1], help="是否自动检测&续训（0=否，1=是）")
- parser.add_argument('--freeze_llm', default=1, type=int, choices=[0, 1, 2], help="冻结策略（0=完全可训练，1=冻结+解冻第0层，2=完全冻结仅训练proj）")
+ parser.add_argument('--freeze_llm', default=2, type=int, choices=[0, 1, 2], help="冻结策略（0=完全可训练，1=冻结+解冻首尾层，2=完全冻结仅训练proj）")
parser.add_argument("--use_compile", default=0, type=int, choices=[0, 1], help="是否使用torch.compile加速（0=否，1=是）")
parser.add_argument("--use_wandb", action="store_true", help="是否使用wandb")
parser.add_argument("--wandb_project", type=str, default="MiniMind-V-Pretrain", help="wandb项目名")
diff --git a/trainer/train_sft_vlm.py b/trainer/train_sft_vlm.py
index 109f4d5..7c4b85f 100644
--- a/trainer/train_sft_vlm.py
+++ b/trainer/train_sft_vlm.py
@@ -107,7 +107,7 @@ if __name__ == "__main__":
parser.add_argument("--data_path", type=str, default="../dataset/sft_i2t.parquet", help="训练数据路径")
parser.add_argument('--from_weight', default='pretrain_vlm', type=str, help="基于哪个权重训练，为none则不基于任何权重训练")
parser.add_argument('--from_resume', default=0, type=int, choices=[0, 1], help="是否自动检测&续训（0=否，1=是）")
- parser.add_argument('--freeze_llm', default=0, type=int, choices=[0, 1, 2], help="冻结策略（0=完全可训练，1=冻结+解冻第0层，2=完全冻结仅训练proj）")
+ parser.add_argument('--freeze_llm', default=1, type=int, choices=[0, 1, 2], help="冻结策略（0=完全可训练，1=冻结+解冻首尾层，2=完全冻结仅训练proj）")
parser.add_argument("--use_compile", default=0, type=int, choices=[0, 1], help="是否使用torch.compile加速（0=否，1=是）")
parser.add_argument("--use_wandb", action="store_true", help="是否使用wandb")
parser.add_argument("--wandb_project", type=str, default="MiniMind-V-SFT", help="wandb项目名")
diff --git a/trainer/trainer_utils.py b/trainer/trainer_utils.py
index 7cc9ac3..0098ee6 100644
--- a/trainer/trainer_utils.py
+++ b/trainer/trainer_utils.py
@@ -63,7 +63,7 @@ def setup_seed(seed: int):
torch.backends.cudnn.benchmark = False
-def init_vlm_model(vlm_config, from_weight='pretrain_vlm', tokenizer_path='../model', vision_model_path='../model/siglip2-base-p16-ve', save_dir='../out', device='cuda', freeze_llm=0):
+def init_vlm_model(vlm_config, from_weight='pretrain_vlm', tokenizer_path='../model', vision_model_path='../model/siglip2-base-p16-256-ve', save_dir='../out', device='cuda', freeze_llm=0):
tokenizer = AutoTokenizer.from_pretrained(tokenizer_path)
model = MiniMindVLM(vlm_config, vision_model_path=vision_model_path)
@@ -84,8 +84,9 @@ def init_vlm_model(vlm_config, from_weight='pretrain_vlm', tokenizer_path='../mo
if 'vision_encoder' not in name:
param.requires_grad = True
elif freeze_llm == 1:
+ last_idx = vlm_config.num_hidden_layers - 1
for name, param in model.model.named_parameters():
- if 'layers.0.' in name:
+ if 'layers.0.' in name or f'layers.{last_idx}.' in name:
param.requires_grad = True
elif freeze_llm == 2:
pass
+
+
2026-04-20
+ +- New checkpoints released: minimind-3v (67M) / minimind-3v-moe (201M-A67M) +- Projector: added `LayerNorm`, token merging switched to 2D pixel-shuffle +- Vision Encoder switched to `SiglipVisionModel` (fixed 256×256) +- Training data moved to ALLaVA-4V (Pretrain 1.27M / SFT 2.9M, merged into a single-stage SFT) +- Freeze strategy updated: `freeze_llm=1` unfreezes first + last layers; Pretrain/SFT defaults now `2`/`1`; `max_seq_len` 360 → 450 +- Misc bugfixes and small tweaks + +2026-04-01
- Added minimind-3v (67M) and minimind-3v-moe (201M-A67M) models - Unified 768+8 architecture, supporting both dense and moe modes -- Switched Visual Encoder from CLIP to SigLIP2 (siglip2-base-p16-ve) +- Switched Visual Encoder from CLIP to SigLIP2 (siglip2-base-p16-256-ve) - Replaced QFormer with MLP Projection + reshape compression - Dataset format updated to parquet, mixed data sources, updated tokenizer with image placeholder `<|image_pad|>`, new WebUI with dynamic model directory scanning and dropdown model switching - Model code refactored, LLM/VLM unified for Transformers format @@ -100,19 +110,16 @@ Is the training process difficult? Now, let's explore the answers and feel the j
-
-
2025-04-27
- -- Compatibility updates -- Adapted to the new feature in the "minimind" repository -- Standardized parts of the code - -More...
+**2025-04-27** + +- Compatibility updates +- Adapted to [MiniMind repository new features](https://github.com/jingyaogong/minimind/issues/370) +- Code normalization + **2025-02-20** - MiniMind2-V updated alongside MiniMind2 @@ -151,9 +158,9 @@ git clone --depth 1 https://github.com/jingyaogong/minimind-v ```bash # Download the siglip2 model to the ./model directory -git clone https://huggingface.co/jingyaogong/siglip2-base-p16-ve +git clone https://huggingface.co/jingyaogong/siglip2-base-p16-256-ve # or -git clone https://modelscope.cn/models/gongjy/siglip2-base-p16-ve +git clone https://modelscope.cn/models/gongjy/siglip2-base-p16-256-ve ``` ```bash @@ -231,40 +238,42 @@ and place it under `./dataset`. **[Note 2]** Parquet is a columnar storage format supporting efficient compression and fast reading. To preview data content, run `python lm_dataset.py` in the `dataset/` directory to visualize the first 5 image-text pairs. -Pretrain data: +Pretrain data (optional; contains caption subset only): ```bash wget https://hf-mirror.com/datasets/jingyaogong/minimind-v_dataset/resolve/main/pretrain_i2t.parquet ``` -SFT data: +SFT data (required; contains full caption + instruct + text merge): ```bash wget https://hf-mirror.com/datasets/jingyaogong/minimind-v_dataset/resolve/main/sft_i2t.parquet ``` -Please reserve about ~2GB of space for the dataset. If there is insufficient space for pretrain data, you can try skipping the pretrain training step and proceed directly to SFT training. +The single `sft_i2t.parquet` file (2.9M rows) absorbs Pretrain as a subset, and after global dictionary encoding dedup it is only ~10% larger than the original SFT — enough to cover every training stage.Note: Training Details
@@ -284,7 +293,7 @@ python train_sft_vlm.py --epochs 4 --from_resume 1 - `--from_weight`: base weight name (llm, pretrain_vlm, none, etc.) - `--save_weight`: save weight prefix name - `--from_resume`: whether to resume training (0=start from scratch, 1=continue from checkpoint) -- `--freeze_llm`: whether to freeze LLM parameters (pretrain use only) +- `--freeze_llm`: freezing strategy (0=all trainable, 1=proj + first/last LLM layers, 2=proj only). Default 2 for Pretrain, 1 for SFT - More details can be found in the code|
-
+
|
- The image features a white and black airplane parked on a grassy terrain. The layers of a building are likely to be filled with air traffic control, such as the building's tall building, the large building, or the overall pavement. There are also two airplanes parked in the background. The airplanes are displayed on a board, and the airplane is flying through the air while the black and white airplane is parked on the ground. | -The image features a large jetliner with a large airplane sitting on the ground. It is likely an airplane or an aircraft, possibly a flight jet or a runway. | +The image shows a golden-brown dog in a park setting with a clear blue sky above. The dog appears to be a golden-brown dog with a cheerful expression, which is characteristic of a Golden Bay breed. The dog's fur is a lighter color, contrasting with the golden balloon, which is likely a balloon with a bright yellow color. The dog's tail is bald, and its tail is darker, indicating it might be a wild, possibly a festive holiday. The dog's bright yellow balloon, along with its tail, stands out against the lighter background of the park. There are no other people or identifying marks in the immediate vicinity of the dog, emphasizing its celebration and a sense of joy. | +The image depicts a golden-brown dog with a golden-brown coat, likely a Border Collie, standing on a grassy field with a clear blue sky. The dog appears to be a German Shepherd, given its size and the positioning of its tail. Its ears are perked up, and it has a golden-brown coat with a lighter golden hue. The dog's eyes are looking forward, and it seems to be looking down at it. The sky is partly cloudy, suggesting a fair weather day. The dog is standing on a grassy field, with some trees and others in the distance, indicating a calm and natural setting. The overall mood of the image is one of tranquility and enjoyment. |
|
-
+
|
- The image features a white cake in an old-fashioned cake with cake placed on it. It is surrounded by a few cooked ingredients, including the wedding cake. | -The image features a white cake in an old-fashioned cake with cake placed on it. It is surrounded by a few cooked ingredients, including the wedding cake. | +The image shows a rainy day with a rainbow-colored umbrella. The umbrella has a colorful mottled pattern of colorful, orange, yellow, red, green, purple, and blue. The umbrella is surrounded by a bright, irregular, bright daylight with irregular, warm lighting. The umbrella's bright colors range from pink, yellow, purple, green, blue, and red, creating a cool, wet atmosphere. The rainbow-colored umbrella is lit by the bright sky, indicating a sunny day. The umbrella is slightly warmer than the rainbow-colored umbrella, with a reflective surface and a blurred background that suggests a shaded area. The umbrella is in sharp focus, allowing the umbrella to be visible on the scene. The overall impression is one of a tropical or subtropical environment, with the rainbow-colored umbrella in a vibrant green color, creating a sense of depth and natural beauty. | +The image shows an outdoor scene with a rainy umbrella in the background, likely during a time of day when umbrellas are illuminated. The umbrellas are arranged in a turn, creating a dynamic and colorful umbrella. The umbrellas have a smooth, flat surface with some visible wrinkles and grouting, indicating a high level of detail. The umbrellas are capped with a slightly darker pink hue, providing a lighter gray color that contrasts with the umbrellas. They are positioned in the distance, adding to the umbrella's appearance. The umbrellas are the main subjects, and the umbrellas are positioned in the distance. |
|
-
+
|
- The image depicts a delicious pizza pizza with fresh toppings, which are likely present in a slice of pizza. The pizza is perfectly crispy, as it has a crispy crust and slightly crispy, making it a delightful pizza presentation. The pizza is filled with fresh toppings, adding to the crispy crust. The pizza is also a bit scrambled, as it has a fresh topping, while the crispy crust is cooked with a pizza pan. The pizza is likely to be a pizza with its crunchy texture and flavorfully. The pizza is also a popular choice for pizza with others, and it is a filling and crispy crust. | -The image features a scenic burning pizza in a pasta-style board, surrounded by a wooden cabinet, a garner topping for cheese and vegetables. There is a small cabinet nearby, with a pan-familt, a pizza flatter topping. The pizza is situated on the left side of the board, with a bowl of olives placed on top of the side. | +The image depicts a vibrant pink bicycle, with its wicker basket attached to the bicycle. The bicycle's design is vibrant, with a sleek design that complements the vibrant colors of the pink and white flowers. The basket appears to be made of a material with a durable fabric, and it has a low-profile design, with a slight front wheel and a single rider, which are often used for pedestrian traffic. The bicycle's frame is not fully visible, but it has a black frame and a black handle. There are no visible people or distinctive features that suggest a specific location or event, such as a bike rack or a bike rack, which are not present in this image. The pink bicycle's body is partially visible and has a single seat, suggesting it is likely a bicycle. There are no discernible background elements that provide context or distraction from the image. | +The image depicts a bright pink bicycle, captured from a perspective view from an elevated perspective. The bicycle is an iconic symbol of the bike, with its rider sporting a touch of white, white, and a touch of purple. The bike is a traditional style with a simple bike and a basketball hoop. It's a sunny day, and the lighting suggests it might be daytime. The scene is set against a pastoral scene, with the pathways leading towards a pathway that casts a warm glow, indicating the serene and warm atmosphere of the scene. The colors in the image are vibrant, with the pink of the pathway standing out against the pale pink of the bicycle. There's a sense of tranquility and freedom of movement. |
|
-
+
|
- The image shows a yellow yellow and red turf coastline on a street, with a couple of cars and a yellow and red traffic lights in the background. | -The image features a couple of female vanity van lying on a car. The car is situated on the ground, surrounded by a wet furniture. The couple is seen in the middle of the car, possibly observing the scene. | +The image shows a yellow sports car positioned on a road with a clear blue sky. The car is positioned on a road with a prominent yellow roof, which is characteristic of a yellow sports car. The car is positioned in the front of the car, facing the camera, with its front grille and lower body lines visible, which suggests a sporty aesthetic. The car is parked on a road that appears to be a high-performance sports car, and the background is a parking lot with a few small boats, which is a common feature in such vehicles. There are no visible windows or other objects in the immediate vicinity of the car. | +The image features a bright yellow sports car parked on a sidewalk. The car is a paved road with a road that appears to be a parking lot, and the sidewalk is bordered by a thin line of trees and a clear blue sky. There are no visible passengers, and the car is parked on a road with a few cars, indicating it is likely a busy day. The car has a sleek design with a high-performance lines and a large round headlamps. |
|
-
+
|
- IoT tree is a holiday that is often associated with Christmas culture, history, and celebration. The tree has a unique black and white striped pattern, which features a sweet treat, a budget-friendly chocolate cake with brown spots. The cake is burnt and has a balcony with a sweet treat, with the rich, vibrant colors of the striped pattern. The tree has a rich and burnt color, while the rich and vibrant colors are visually appealing. The tree has a striped pattern, which adds to the overall atmosphere of the image. | -The image is a colorful scene of a colorful vintage house, with a large pink roof of the wall and a red color scheme. The colorful vintage house has a pink color, and the color scheme appears to be fine, with a red color scheme. | +The image shows a stylized silhouette of a character from a superhero film series. The character is wearing a long-sleeved blue suit with a blue collar and a red band on the left chest, paired with red trousers. The suit is equipped with a blue suit and a red band on top, and the suit is adorned with a red band, which is also in the midst of a superhero character. The suit is standing outdoors, and the character is on a city street, with a large, blue suit standing out. The sky is a gradient of warm colors, suggesting it is either dawn or dusk. The overall impression is one of a high-energy, urban nighttime scene, with a focus on the character and the character. | +The image depicts a spaceship scene at what appears to be a cityscape during what appears to be sunset. The scene is dominated by the sky, with the silhouettes of a figure standing on a raised platform, which is the central subject of the image. The figure is dressed in a red and blue suit with a white shirt, which suggests a style commonly associated with the "UNITED SUNDER" superhero franchise. The suit is primarily red with a red hue and a white shirt. There is a figure with a red belt on the right, holding a red object that could be a shield or a wrench. The person is standing on a street with a couple of buildings, indicating that the location might be a city with buildings that are part of the UNITED SUNDER. The sky is filled with a warm orange hue, suggesting a sunset. The overall atmosphere is one of dynamism and fantasy. |
|
-
+
|
- The image features a snowy mountain surrounded by a large, dense mountains. The large body of water is quite dense, with the body of water being hot and the water is sandy. The tall trees are swirls, and the tall trees are standing on the snow-covered ground. The body of water is also sink, creating a savanna-like appearance. | -The image is an image of a large mountain visible in a lake, surrounded by the idyllic mountains and the mountains. It appears to be a blanket in the ocean, with the river and idylis watching the sea. The mountains are lined with fresh sand and waves, adding a sense of tranquility to the scene. | -||
|
-
-
- |
- The image shows a variety of cooking options, including baking meat, cupcakes, and spinach. The baked vegetables are rich in a variety of flavors, including grilled, sautéed, and baked vegetables. The presence of a baked vegetable with a variety of vegetables in different parts suggests a variety of options, including baking, cooking, and baking. The cooking process is highly recommended, with a mix of vegetables and baking times, making it an ideal choice for those who prefer a variety of cooking options. The cooking process is also highly compatible, with a variety of flavors and textures enjoying the cooking process. | -The image features a variety of freshwater salads, glasses, and coworkers displayed on a table. There is a mix of freshwater ingredients, likely a bun, which can be seen in the menu. The freshwater ingredients are placed on the table, and there is a portion of the coworkers displayed in the middle of the room. There is also a bowl filled with various ingredients. | -||
|
-
-
- |
- The image features a table filled with people standing together, casing bars, and a pair of brown cats. The table is filled with pink and white cats. | -The image is a black and white detail of a miscellaneous bunch of toys, which is likely to be a part of a group or a similar artistic field. | -||
|
-
-
- |
- The image features a brown wooden coat. | -The image shows a sandy beach with an umbrella on top of a chair, providing a visual appeal for people to sit on. | -||
|
-
-
- |
- The image displays a group of people sitting on the bus. They are waiting to be cautious and attentive to their feet, which indicates they are likely to be cautious and followed by the bus. | -The image features a large collection of school buses, a brick-and-middle bus, and a stack of cars visible in the background. The school bus is situated next to a school bus, and there are several people watching the bus. The school bus is visible in the background, with one person standing behind the other, while the other person is watching the bus. The bus is positioned behind the school bus, creating a seamless and dynamic visual effect. | +The image captures a dynamic scene at what appears to be a race. In the foreground, a red sports car is mounted on a racetrack, its front face is turned slightly to the left. The car has a prominent rear wing and aerodynamic elements, which suggests a high-speed speed and aerodynamic profile. The track is adorned with multiple windows, one of which is visible, allowing a view of the ground level. The background shows a crowd of spectators, some dressed in red, while others are standing, possibly in the air, in the distance, indicating that this is likely the race of the race. The lighting is warm, and the ground is dim, which gives the racing a warm ambiance. There are no people visible in the image, and the image is taken at a time when the race is being captured and the car is in the air. The lighting conditions suggest that it is either early morning or late afternoon, as the sun is low on the horizon, casting a warm glow and creating a dramatic effect on the track. | +The image captures a dynamic scene at what appears to be a high-speed race, likely during the golden hour, given the warm lighting and the soft shadows. The track is bathed in a soft glow, indicating the sun is high and possibly at the moment of the race. There are no visible smoke, orbits, or disturbances in the sky, suggesting the race is either during the golden hour or during the golden hour. The race is a sizable form of aerodynamic body, with the front wheel being more pronounced, indicating it may have been used for speed and possibly aerodynamics. The track is bordered by a blurred audience, which may suggest a focus on the race's grandeur or the driver's presence. The track itself has a glossy finish, with some areas showing darker lines and the surrounding structures, which could be part of a track or a similar event. |
-> Given the limited expertise, there may be unknown issues, and we welcome everyone to discuss, correct, or submit PRs -> to improve the project in Issues.
-> Your support is the driving force behind continuous improvements to the project. Thank you! +> If `MiniMind-V` is useful to you, a ⭐ on GitHub is welcome.
+> Issues and PRs are the best place to share problems or improvements found while using the project. ## 🤝 [Contributors](https://github.com/jingyaogong/minimind-v/graphs/contributors) diff --git a/dataset/eval_images/airplane-flying-blue-sky.jpg b/dataset/eval_images/airplane-flying-blue-sky.jpg deleted file mode 100644 index 3ef3d6a..0000000 Binary files a/dataset/eval_images/airplane-flying-blue-sky.jpg and /dev/null differ diff --git a/dataset/eval_images/birthday-cake-candles-table.jpg b/dataset/eval_images/birthday-cake-candles-table.jpg deleted file mode 100644 index baacc28..0000000 Binary files a/dataset/eval_images/birthday-cake-candles-table.jpg and /dev/null differ diff --git a/dataset/eval_images/image-01-golden-dog-balloons.jpg b/dataset/eval_images/image-01-golden-dog-balloons.jpg new file mode 100644 index 0000000..53923c4 Binary files /dev/null and b/dataset/eval_images/image-01-golden-dog-balloons.jpg differ diff --git a/dataset/eval_images/image-02-rainbow-umbrella-street.jpg b/dataset/eval_images/image-02-rainbow-umbrella-street.jpg new file mode 100644 index 0000000..7b3201c Binary files /dev/null and b/dataset/eval_images/image-02-rainbow-umbrella-street.jpg differ diff --git a/dataset/eval_images/image-03-cherry-blossom-bike.jpg b/dataset/eval_images/image-03-cherry-blossom-bike.jpg new file mode 100644 index 0000000..df4f184 Binary files /dev/null and b/dataset/eval_images/image-03-cherry-blossom-bike.jpg differ diff --git a/dataset/eval_images/image-04-yellow-car.jpg b/dataset/eval_images/image-04-yellow-car.jpg new file mode 100644 index 0000000..1383b44 Binary files /dev/null and b/dataset/eval_images/image-04-yellow-car.jpg differ diff --git a/dataset/eval_images/image-05-superhero-rooftop.jpg b/dataset/eval_images/image-05-superhero-rooftop.jpg new file mode 100644 index 0000000..987b373 Binary files /dev/null and b/dataset/eval_images/image-05-superhero-rooftop.jpg differ diff --git a/dataset/eval_images/image-06-racecar-drift.jpg b/dataset/eval_images/image-06-racecar-drift.jpg new file mode 100644 index 0000000..3938c35 Binary files /dev/null and b/dataset/eval_images/image-06-racecar-drift.jpg differ diff --git a/dataset/eval_images/pizza-on-wooden-board.jpg b/dataset/eval_images/pizza-on-wooden-board.jpg deleted file mode 100644 index 1b3ceb4..0000000 Binary files a/dataset/eval_images/pizza-on-wooden-board.jpg and /dev/null differ diff --git a/dataset/eval_images/red-sports-car-road.jpg b/dataset/eval_images/red-sports-car-road.jpg deleted file mode 100644 index ba8695b..0000000 Binary files a/dataset/eval_images/red-sports-car-road.jpg and /dev/null differ diff --git a/dataset/eval_images/row-of-colorful-houses.jpg b/dataset/eval_images/row-of-colorful-houses.jpg deleted file mode 100644 index 206a395..0000000 Binary files a/dataset/eval_images/row-of-colorful-houses.jpg and /dev/null differ diff --git a/dataset/eval_images/snow-mountain-lake-view.jpg b/dataset/eval_images/snow-mountain-lake-view.jpg deleted file mode 100644 index 1bf1bb7..0000000 Binary files a/dataset/eval_images/snow-mountain-lake-view.jpg and /dev/null differ diff --git a/dataset/eval_images/street-food-hotpot-table.jpg b/dataset/eval_images/street-food-hotpot-table.jpg deleted file mode 100644 index 2c205b9..0000000 Binary files a/dataset/eval_images/street-food-hotpot-table.jpg and /dev/null differ diff --git a/dataset/eval_images/three-kittens-basket.jpg b/dataset/eval_images/three-kittens-basket.jpg deleted file mode 100644 index 6df31c6..0000000 Binary files a/dataset/eval_images/three-kittens-basket.jpg and /dev/null differ diff --git a/dataset/eval_images/tropical-beach-palm-tree.jpg b/dataset/eval_images/tropical-beach-palm-tree.jpg deleted file mode 100644 index 50f0b9d..0000000 Binary files a/dataset/eval_images/tropical-beach-palm-tree.jpg and /dev/null differ diff --git a/dataset/eval_images/yellow-school-bus-road.jpg b/dataset/eval_images/yellow-school-bus-road.jpg deleted file mode 100644 index f2c87bf..0000000 Binary files a/dataset/eval_images/yellow-school-bus-road.jpg and /dev/null differ diff --git a/dataset/lm_dataset.py b/dataset/lm_dataset.py index 5bc86c5..a22a1d5 100644 --- a/dataset/lm_dataset.py +++ b/dataset/lm_dataset.py @@ -118,9 +118,9 @@ class VLMDataset(Dataset): if __name__ == '__main__': import matplotlib.pyplot as plt; plt.rcParams['font.sans-serif'] = ['Arial Unicode MS', 'SimHei'] for path in ['pretrain_i2t.parquet', 'sft_i2t.parquet']: - t = pa.Table.from_batches(pq.ParquetFile(path).iter_batches()); fig, ax = plt.subplots(1, 5, figsize=(20, 4)) + pf = pq.ParquetFile(path); n = pf.num_row_groups; t = pa.concat_tables([pf.read_row_group(i * n // 5).slice(0, 1) for i in range(5)]); fig, ax = plt.subplots(1, 5, figsize=(20, 4)) for i in range(5): img_data = t['image_bytes'][i].as_py(); img_data = img_data[0] if isinstance(img_data, list) else img_data ax[i].imshow(Image.open(io.BytesIO(img_data))); ax[i].axis('off') ax[i].set_title(json.loads(t['conversations'][i].as_py())[1]['content'][:30], fontsize=8) - out = path.replace('.parquet', '_preview.png'); plt.savefig(out); print(f'已保存{out}, 共{len(t)}条') + out = path.replace('.parquet', '_preview.png'); plt.savefig(out); print(f'已保存{out}, 共{pf.metadata.num_rows}条') diff --git a/eval_vlm.py b/eval_vlm.py index 433ce6e..8865986 100644 --- a/eval_vlm.py +++ b/eval_vlm.py @@ -11,19 +11,19 @@ from trainer.trainer_utils import setup_seed, get_model_params warnings.filterwarnings('ignore') def init_model(args): - tokenizer = AutoTokenizer.from_pretrained(args.load_from) + tokenizer = AutoTokenizer.from_pretrained(args.load_from, trust_remote_code=True) if 'model' in args.load_from: moe_suffix = '_moe' if args.use_moe else '' ckp = f'./{args.save_dir}/{args.weight}_{args.hidden_size}{moe_suffix}.pth' model = MiniMindVLM( VLMConfig(hidden_size=args.hidden_size, num_hidden_layers=args.num_hidden_layers, use_moe=bool(args.use_moe)), - vision_model_path="./model/siglip2-base-p16-ve" + vision_model_path="./model/siglip2-base-p16-256-ve" ) state_dict = torch.load(ckp, map_location=args.device) model.load_state_dict({k: v for k, v in state_dict.items() if 'mask' not in k}, strict=False) else: model = AutoModelForCausalLM.from_pretrained(args.load_from, trust_remote_code=True) - model.vision_encoder, model.processor = MiniMindVLM.get_vision_model("./model/siglip2-base-p16-ve") + model.vision_encoder, model.processor = MiniMindVLM.get_vision_model("./model/siglip2-base-p16-256-ve") get_model_params(model, model.config) preprocess = model.processor return model.half().eval().to(args.device), tokenizer, preprocess @@ -38,8 +38,8 @@ def main(): parser.add_argument('--num_hidden_layers', default=8, type=int, help="隐藏层数量") parser.add_argument('--use_moe', default=0, type=int, choices=[0, 1], help="是否使用MoE架构（0=否，1=是）") parser.add_argument('--max_new_tokens', default=512, type=int, help="最大生成长度") - parser.add_argument('--temperature', default=0.9, type=float, help="生成温度，控制随机性（0-1，越大越随机）") - parser.add_argument('--top_p', default=0.9, type=float, help="nucleus采样阈值（0-1）") + parser.add_argument('--temperature', default=0.7, type=float, help="生成温度，控制随机性（0-1，越大越随机）") + parser.add_argument('--top_p', default=0.85, type=float, help="nucleus采样阈值（0-1）") parser.add_argument('--image_dir', default='./dataset/eval_images/', type=str, help="测试图像目录") parser.add_argument('--show_speed', default=1, type=int, help="显示decode速度（tokens/s）") parser.add_argument('--device', default='cuda' if torch.cuda.is_available() else 'cpu', type=str, help="运行设备") @@ -49,11 +49,12 @@ def main(): model, tokenizer, preprocess = init_model(args) streamer = TextStreamer(tokenizer, skip_prompt=True, skip_special_tokens=True) # 自动测试image_dir中的所有图像 - prompt = "请仔细观察这张图片，描述你所看到的内容：\n\n
minimind-3vAI-generated content may be inaccurate, please verify