github
/
minimind-v
mirror of https://github.com/jingyaogong/minimind-v.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

[feat] update readme

pull/107/head
jingyaogong 2025-12-27 10:48:54 +08:00
parent 32cf4c5c01
commit 1863743ee9
2 changed files with 16 additions and 265 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

138
README.md
View File

@ -206,23 +206,21 @@ print(torch.cuda.is_available())
<details style="color:rgb(128,128,128)"> <details style="color:rgb(128,128,128)">
<summary>注：数据集须知</summary> <summary>注：数据集须知</summary>
【注1】之前需解压50万零碎的图像文件可能非常慢。2025-12-27起，数据集格式统一为Parquet，图文一体化存储，体积更小，无需解压，加载更快。
【注2】Parquet是列式存储格式，支持高效压缩和快速读取。如果你对它感到陌生，可以预览数据内容，在`dataset/`目录下执行`python lm_dataset.py`可视化前5条图文对
Pretrain数据： Pretrain数据：
```bash ```bash
wget https://hf-mirror.com/datasets/jingyaogong/minimind-v_dataset/resolve/main/pretrain_data.jsonl wget https://hf-mirror.com/datasets/jingyaogong/minimind-v_dataset/resolve/main/pretrain_data.parquet
wget https://hf-mirror.com/datasets/jingyaogong/minimind-v_dataset/resolve/main/pretrain_images.zip
unzip pretrain_images.zip && rm pretrain_images.zip
``` ```
SFT数据： SFT数据：
```bash ```bash
wget https://hf-mirror.com/datasets/jingyaogong/minimind-v_dataset/resolve/main/sft_data.jsonl wget https://hf-mirror.com/datasets/jingyaogong/minimind-v_dataset/resolve/main/sft_data.parquet
wget https://hf-mirror.com/datasets/jingyaogong/minimind-v_dataset/resolve/main/sft_images.zip
unzip sft_images.zip && rm sft_images.zip
``` ```
`*.jsonl`为问答文本，`*images`为配套的图片数据，下载完成后需要解压图像数据。 建议预留~2GB空间存放数据集，若无多余空间存放pretrain数据，可尝试跳过pretrain训练步骤直接进行sft训练。
请预留~5GB空间存放数据集，若无多余空间存放pretrain数据，可尝试跳过pretrain训练步骤直接进行sft训练。
</details> </details>
@ -424,73 +422,6 @@ VLM的输入依然是一段文本，其中包含特殊的`<image>`占位符。
![input](./images/minimind-v-input.png) ![input](./images/minimind-v-input.png)
一次性多图的实现方法就是通过注入多个`<image>`图像占位符进行实现，不需要修改任何框架。
<details>
<summary> 视频理解的拓展思路 </summary>
write by [@xinyanghuang7](https://github.com/xinyanghuang7)
对于多模态大模型的视频理解能力，一个可行的思路是参考现有MiniCPM-V 2.6 进行视频理解的Python示例。
主要思想是通过提取视频关键帧，而后进行多图推理。
因此，如果希望在MiniMind-V中添加视频理解能力，可以在现有多图训练的基础上，参考此python脚本中对于关键帧的提取方法，而后加大训练文件中支持图片的数量。
所支持的MAX_NUM_FRAMES越多，所消耗的显存越大。
```text
import torch
from PIL import Image
from transformers import AutoModel, AutoTokenizer
from decord import VideoReader, cpu # pip install decord
model = AutoModel.from_pretrained('openbmb/MiniCPM-V-2_6', trust_remote_code=True,
attn_implementation='sdpa',
torch_dtype=torch.bfloat16) # sdpa or flash_attention_2, no eager
model = model.eval().cuda()
tokenizer = AutoTokenizer.from_pretrained('openbmb/MiniCPM-V-2_6', trust_remote_code=True)
MAX_NUM_FRAMES = 64 # if cuda OOM set a smaller number
def encode_video(video_path):
def uniform_sample(l, n):
gap = len(l) / n
idxs = [int(i * gap + gap / 2) for i in range(n)]
return [l[i] for i in idxs]
vr = VideoReader(video_path, ctx=cpu(0))
sample_fps = round(vr.get_avg_fps() / 1) # FPS
frame_idx = [i for i in range(0, len(vr), sample_fps)]
if len(frame_idx) > MAX_NUM_FRAMES:
frame_idx = uniform_sample(frame_idx, MAX_NUM_FRAMES)
frames = vr.get_batch(frame_idx).asnumpy()
frames = [Image.fromarray(v.astype('uint8')) for v in frames]
print('num frames:', len(frames))
return frames
video_path = "video_test.mp4"
frames = encode_video(video_path)
question = "Describe the video"
msgs = [
{'role': 'user', 'content': frames + [question]},
]
# Set decode params for video
params = {}
params["use_image_id"] = False
params["max_slice_nums"] = 2 # 如果cuda OOM且视频分辨率大于448*448可设为1
answer = model.chat(
image=None,
msgs=msgs,
tokenizer=tokenizer,
**params
)
print(answer)
```
</details>
至此，`MiniMind-V`的所有细节已经呈现完毕。 至此，`MiniMind-V`的所有细节已经呈现完毕。
`MiniMind-V`的模型子类完全继承自`MiniMind`， `MiniMind-V`的模型子类完全继承自`MiniMind`，
仅基于后者做**最小**变更而产生， 仅基于后者做**最小**变更而产生，
@ -544,33 +475,6 @@ print(answer)
} }
``` ```
(sft_vlm_data_multi.jsonl) 多图指令微调数据集格式：
```json lines
{
"conversations": [
{
"role": "user",
"content": "context: Source Image: <image> Target Image: <image> Instruction: What is the correct image edit instruction that can transfrom the source image to target image?<image>"
},
{
"role": "assistant",
"content": "take the people out of the back in the photo. Remove the two people behind the woman in the white dress and the man in the blue suit. remove people behind the couple in the centre"
}
],
"image": "0.jpg, 1.jpg"
}
```
<details>
<summary> 数据说明 </summary>
* 多图数据集规模相对较小且为英文对话，数据集仅包含两图对比的场景，因此微调效果有限，这里只提供一种参考思路。
* `jsonl`均为文本指令，`images.zip`均为配套的图像数据（下载后需要解压）
</details>
数据集下载地址：([ModelScope](https://www.modelscope.cn/datasets/gongjy/minimind-v_dataset) | [HuggingFace](https://huggingface.co/datasets/jingyaogong/minimind-v_dataset)) 数据集下载地址：([ModelScope](https://www.modelscope.cn/datasets/gongjy/minimind-v_dataset) | [HuggingFace](https://huggingface.co/datasets/jingyaogong/minimind-v_dataset))
@ -584,10 +488,6 @@ print(answer)
指令微调从300K条真实对话数据集中学习对图片提问的真实问答格式，更符合与人类的交流习惯。 指令微调从300K条真实对话数据集中学习对图片提问的真实问答格式，更符合与人类的交流习惯。
> train_sft_vlm
多图微调提供demo：鸟类对比数据集，长度为13.6k的真实问答格式。
训练时均冻结visual encoder也就是clip模型梯度， 训练时均冻结visual encoder也就是clip模型梯度，
只训练Projection和LLM两部分。 只训练Projection和LLM两部分。
预训练中，只设置Projection和LLM的最后一层参数可学习。 预训练中，只设置Projection和LLM的最后一层参数可学习。
@ -694,27 +594,6 @@ SFT [512+8] & [768+16]
</tbody> </tbody>
</table> </table>
#### 多图对话（效果十分有限）
<table>
<thead>
<tr>
<th>图片1</th>
<th>图片2</th>
<th>512_sft_multi</th>
<th>768_sft_multi</th>
</tr>
</thead>
<tbody>
<tr>
<td><img src="./dataset/eval_multi_images/bird/0.jpg" alt="a-bird.png"></td>
<td><img src="./dataset/eval_multi_images/bird/1.jpg" alt="a-bird.png"></td>
<td>这幅图像显示了一种鸟簸戮的场景：一个女人站在红绿相间的红绿相间的紫色鸟簸戴在她身上。女人站在红色的鸟簸戴在她身上，而她的翻领上的那只红鸟则站在她身后。</td>
<td>这两只鸟在同一片树林中飞翔，有的位于画面中心，而另一些则较小，形成了鲜明对比。这种鸟类的出现突出了其飞行能力和适应性，因为它们能够在树林中快速迅速移动。此外，两只鸟的位置不同，一个在图像的左边，另一个在右边，这表明它们在同一片树林中移动得很近。这种鸟类的自然行为也有助于区分这两种鸟类物种。</td>
</tr>
</tbody>
</table>
### 效果小结： ### 效果小结：
视觉信号对于LLM视作一种特殊的外语， 视觉信号对于LLM视作一种特殊的外语，
@ -746,8 +625,7 @@ LLM性能越强，对应的VLM必然越强，此时效果增益会很明显。
## 😊鸣谢 ## 😊鸣谢
<a href="https://github.com/xinyanghuang7"><b>@xinyanghuang7</b></a>: <a href="https://github.com/xinyanghuang7"><b>@xinyanghuang7</b></a>: <a href="https://github.com/xinyanghuang7/minimind-v/tree/hxy">多图vlm分支</a> | <a href="https://github.com/jingyaogong/minimind-v/tree/32cf4c5c01337231fd907b92d513de8945594263">仓库截至此版本提供</a>
<a href="https://github.com/xinyanghuang7/minimind-v/tree/hxy">🔗实现了完整的多图分支</a>
<details close> <details close>
<summary> <b>参考链接 & 感谢以下优秀的论文或项目</b> </summary> <summary> <b>参考链接 & 感谢以下优秀的论文或项目</b> </summary>

143
README_en.md
View File

@ -214,23 +214,21 @@ and place it under `./dataset`.
<details style="color:rgb(128,128,128)"> <details style="color:rgb(128,128,128)">
<summary>Note: Dataset Details</summary> <summary>Note: Dataset Details</summary>
**[Note 1]** Previously, extracting 500k fragmented image files could be very slow. From 2025-12-27, dataset format is unified to Parquet with image-text integrated storage, smaller size, no decompression needed, faster loading.
**[Note 2]** Parquet is a columnar storage format supporting efficient compression and fast reading. To preview data content, run `python lm_dataset.py` in the `dataset/` directory to visualize the first 5 image-text pairs.
Pretrain data: Pretrain data:
```bash ```bash
wget https://hf-mirror.com/datasets/jingyaogong/minimind-v_dataset/resolve/main/pretrain_data.jsonl wget https://hf-mirror.com/datasets/jingyaogong/minimind-v_dataset/resolve/main/pretrain_data.parquet
wget https://hf-mirror.com/datasets/jingyaogong/minimind-v_dataset/resolve/main/pretrain_images.zip
unzip pretrain_images.zip && rm pretrain_images.zip
``` ```
SFT data: SFT data:
```bash ```bash
wget https://hf-mirror.com/datasets/jingyaogong/minimind-v_dataset/resolve/main/sft_data.jsonl wget https://hf-mirror.com/datasets/jingyaogong/minimind-v_dataset/resolve/main/sft_data.parquet
wget https://hf-mirror.com/datasets/jingyaogong/minimind-v_dataset/resolve/main/sft_images.zip
unzip sft_images.zip && rm sft_images.zip
``` ```
`*.jsonl` contains Q&A text, and `*images` are the accompanying image data. After downloading, decompress the image data. Please reserve about ~2GB of space for the dataset. If there is insufficient space for pretrain data, you can try skipping the pretrain training step and proceed directly to SFT training.
Please reserve about 5GB of space for the dataset. If there is insufficient space for pretrain data, you can try skipping the pretrain training step and proceed directly to SFT training.
</details> </details>
@ -450,77 +448,6 @@ output is no different from that of the LLM part.
![input](./images/minimind-v-input.png) ![input](./images/minimind-v-input.png)
For handling multiple images at once, this can be achieved by injecting multiple `<image>` placeholders without needing
to modify the framework at all.
<details>
<summary> Expansion Ideas for Video Understanding </summary>
written by [@xinyanghuang7](https://github.com/xinyanghuang7)
For the video understanding capabilities of multimodal large models, one feasible approach is to refer to the existing
MiniCPM-V 2.6 Python example for video understanding.
The main idea is to extract key frames from the video and then perform multi-image inference.
Therefore, if you want to add video understanding capabilities to MiniMind-V, you can base it on the existing
multi-image training, refer to the key frame extraction method in this Python script, and increase the number of images
supported in the training files.
The more MAX_NUM_FRAMES supported, the more GPU memory it will consume.
```text
import torch
from PIL import Image
from transformers import AutoModel, AutoTokenizer
from decord import VideoReader, cpu # pip install decord
model = AutoModel.from_pretrained('openbmb/MiniCPM-V-2_6', trust_remote_code=True,
attn_implementation='sdpa',
torch_dtype=torch.bfloat16) # sdpa or flash_attention_2, no eager
model = model.eval().cuda()
tokenizer = AutoTokenizer.from_pretrained('openbmb/MiniCPM-V-2_6', trust_remote_code=True)
MAX_NUM_FRAMES = 64 # if cuda OOM set a smaller number
def encode_video(video_path):
def uniform_sample(l, n):
gap = len(l) / n
idxs = [int(i * gap + gap / 2) for i in range(n)]
return [l[i] for i in idxs]
vr = VideoReader(video_path, ctx=cpu(0))
sample_fps = round(vr.get_avg_fps() / 1) # FPS
frame_idx = [i for i in range(0, len(vr), sample_fps)]
if len(frame_idx) > MAX_NUM_FRAMES:
frame_idx = uniform_sample(frame_idx, MAX_NUM_FRAMES)
frames = vr.get_batch(frame_idx).asnumpy()
frames = [Image.fromarray(v.astype('uint8')) for v in frames]
print('num frames:', len(frames))
return frames
video_path = "video_test.mp4"
frames = encode_video(video_path)
question = "Describe the video"
msgs = [
{'role': 'user', 'content': frames + [question]},
]
# Set decode params for video
params = {}
params["use_image_id"] = False
params["max_slice_nums"] = 2 # If cuda OOM and video resolution is greater than 448*448, set to 1
answer = model.chat(
image=None,
msgs=msgs,
tokenizer=tokenizer,
**params
)
print(answer)
```
</details>
At this point, all the details of `MiniMind-V` have been presented. At this point, all the details of `MiniMind-V` have been presented.
The `MiniMind-V` model subclass completely inherits from `MiniMind`, The `MiniMind-V` model subclass completely inherits from `MiniMind`,
and is generated with **minimal** changes based on the latter, and is generated with **minimal** changes based on the latter,
@ -573,34 +500,6 @@ The Q&A content has been translated, with better support for Chinese, further or
} }
``` ```
(sft_vlm_data_multi.jsonl) Multi-image instruction fine-tuning dataset format:
```json lines
{
"conversations": [
{
"role": "user",
"content": "context: Source Image: <image> Target Image: <image> Instruction: What is the correct image edit instruction that can transform the source image to target image?<image>"
},
{
"role": "assistant",
"content": "take the people out of the back in the photo. Remove the two people behind the woman in the white dress and the man in the blue suit. remove people behind the couple in the center"
}
],
"image": "0.jpg, 1.jpg"
}
```
<details>
<summary> Data Description </summary>
* The multi-image dataset is relatively small and contains English conversations, focusing only on scenes with two image
comparisons. Therefore, the fine-tuning effect is limited, and this is just one reference approach.
* `jsonl` contains textual instructions, and `images.zip` contains the corresponding image data (to be unzipped after
download).
</details>
Dataset download Dataset download
link: ([ModelScope](https://www.modelscope.cn/datasets/gongjy/minimind-v_dataset) | [HuggingFace](https://huggingface.co/datasets/jingyaogong/minimind-v_dataset)) link: ([ModelScope](https://www.modelscope.cn/datasets/gongjy/minimind-v_dataset) | [HuggingFace](https://huggingface.co/datasets/jingyaogong/minimind-v_dataset))
@ -616,10 +515,6 @@ Pre-training learns general image knowledge from a dataset of 595K samples, such
Instruction fine-tuning learns the real Q&A format for image-related questions from a dataset of 300K real Instruction fine-tuning learns the real Q&A format for image-related questions from a dataset of 300K real
conversations, which better aligns with human communication habits. conversations, which better aligns with human communication habits.
> train_sft_vlm
Multi-image fine-tuning provides a demo: a bird comparison dataset with 13.6k real Q&A formats.
During training, the visual encoder, i.e., the CLIP model's gradients, are frozen, and only the Projection and LLM parts During training, the visual encoder, i.e., the CLIP model's gradients, are frozen, and only the Projection and LLM parts
are trained. are trained.
In pre-training, only the last layer parameters of Projection and LLM are learnable. In pre-training, only the last layer parameters of Projection and LLM are learnable.
@ -726,27 +621,6 @@ Download link:
</tbody> </tbody>
</table> </table>
#### Multiple Image Dialogue (Effect is Limited)
<table>
<thead>
<tr>
<th>Image1</th>
<th>Image2</th>
<th>512_sft_multi</th>
<th>768_sft_multi</th>
</tr>
</thead>
<tbody>
<tr>
<td><img src="./dataset/eval_multi_images/bird/0.jpg" alt="a-bird.png"></td>
<td><img src="./dataset/eval_multi_images/bird/1.jpg" alt="a-bird.png"></td>
<td>This image displays a bird scenario: a woman standing with a red and green mixed purple bird perched on her. The woman stands with the bird on her shoulders, while the red bird on her collar stands behind her.</td>
<td>The two birds are flying in the same forest, some are in the center of the image, while others are smaller, creating a contrast. The birds’ presence highlights their flight ability and adaptability as they swiftly move through the woods. Additionally, the birds’ positions vary, with one on the left and the other on the right, indicating they are moving close within the same forest. Their natural behavior helps distinguish the two bird species.</td>
</tr>
</tbody>
</table>
### Effect Summary: ### Effect Summary:
Visual signals are treated as a special foreign language by LLMs, so the "language learning" ability highly depends on Visual signals are treated as a special foreign language by LLMs, so the "language learning" ability highly depends on
@ -778,8 +652,7 @@ significant.
## 😊 Acknowledgments ## 😊 Acknowledgments
<a href="https://github.com/xinyanghuang7"><b>@xinyanghuang7</b></a>: <a href="https://github.com/xinyanghuang7"><b>@xinyanghuang7</b></a>: <a href="https://github.com/xinyanghuang7/minimind-v/tree/hxy">Multi-image VLM branch</a> | <a href="https://github.com/jingyaogong/minimind-v/tree/32cf4c5c01337231fd907b92d513de8945594263">Repository provided up to this version</a>
<a href="https://github.com/xinyanghuang7/minimind-v/tree/hxy">🔗Implemented complete multi-graph branch</a>
<details close> <details close>
<summary> <b>Reference Links & Thanks to the following excellent papers or projects</b> </summary> <summary> <b>Reference Links & Thanks to the following excellent papers or projects</b> </summary>