-
-
至此，`MiniMind-V`的所有细节已经呈现完毕。
`MiniMind-V`的模型子类完全继承自`MiniMind`，
仅基于后者做**最小**变更而产生，
@@ -544,33 +475,6 @@ print(answer)
}
```
-(sft_vlm_data_multi.jsonl) 多图指令微调数据集格式：
-
-```json lines
-{
- "conversations": [
- {
- "role": "user",
- "content": "context: Source Image: 视频理解的拓展思路
- -write by [@xinyanghuang7](https://github.com/xinyanghuang7) - -对于多模态大模型的视频理解能力，一个可行的思路是参考现有MiniCPM-V 2.6 进行视频理解的Python示例。 -主要思想是通过提取视频关键帧，而后进行多图推理。 -因此，如果希望在MiniMind-V中添加视频理解能力，可以在现有多图训练的基础上，参考此python脚本中对于关键帧的提取方法，而后加大训练文件中支持图片的数量。 -所支持的MAX_NUM_FRAMES越多，所消耗的显存越大。 - -```text -import torch -from PIL import Image -from transformers import AutoModel, AutoTokenizer -from decord import VideoReader, cpu # pip install decord - -model = AutoModel.from_pretrained('openbmb/MiniCPM-V-2_6', trust_remote_code=True, - attn_implementation='sdpa', - torch_dtype=torch.bfloat16) # sdpa or flash_attention_2, no eager -model = model.eval().cuda() -tokenizer = AutoTokenizer.from_pretrained('openbmb/MiniCPM-V-2_6', trust_remote_code=True) - -MAX_NUM_FRAMES = 64 # if cuda OOM set a smaller number - - -def encode_video(video_path): - def uniform_sample(l, n): - gap = len(l) / n - idxs = [int(i * gap + gap / 2) for i in range(n)] - return [l[i] for i in idxs] - - vr = VideoReader(video_path, ctx=cpu(0)) - sample_fps = round(vr.get_avg_fps() / 1) # FPS - frame_idx = [i for i in range(0, len(vr), sample_fps)] - if len(frame_idx) > MAX_NUM_FRAMES: - frame_idx = uniform_sample(frame_idx, MAX_NUM_FRAMES) - frames = vr.get_batch(frame_idx).asnumpy() - frames = [Image.fromarray(v.astype('uint8')) for v in frames] - print('num frames:', len(frames)) - return frames - - -video_path = "video_test.mp4" -frames = encode_video(video_path) -question = "Describe the video" -msgs = [ - {'role': 'user', 'content': frames + [question]}, -] - -# Set decode params for video -params = {} -params["use_image_id"] = False -params["max_slice_nums"] = 2 # 如果cuda OOM且视频分辨率大于448*448可设为1 - -answer = model.chat( - image=None, - msgs=msgs, - tokenizer=tokenizer, - **params -) -print(answer) -``` - -
-
数据集下载地址：([ModelScope](https://www.modelscope.cn/datasets/gongjy/minimind-v_dataset) | [HuggingFace](https://huggingface.co/datasets/jingyaogong/minimind-v_dataset))
@@ -584,10 +488,6 @@ print(answer)
指令微调从300K条真实对话数据集中学习对图片提问的真实问答格式，更符合与人类的交流习惯。
-> train_sft_vlm
-
-多图微调提供demo：鸟类对比数据集，长度为13.6k的真实问答格式。
-
训练时均冻结visual encoder也就是clip模型梯度，
只训练Projection和LLM两部分。
预训练中，只设置Projection和LLM的最后一层参数可学习。
@@ -694,27 +594,6 @@ SFT [512+8] & [768+16]
-#### 多图对话（效果十分有限）
-
-数据说明
- -* 多图数据集规模相对较小且为英文对话，数据集仅包含两图对比的场景，因此微调效果有限，这里只提供一种参考思路。 - - -* `jsonl`均为文本指令，`images.zip`均为配套的图像数据（下载后需要解压） - -| 图片1 | -图片2 | -512_sft_multi | -768_sft_multi | -
|---|---|---|---|
 |
-  |
- 这幅图像显示了一种鸟簸戮的场景：一个女人站在红绿相间的红绿相间的紫色鸟簸戴在她身上。女人站在红色的鸟簸戴在她身上，而她的翻领上的那只红鸟则站在她身后。 | -这两只鸟在同一片树林中飞翔，有的位于画面中心，而另一些则较小，形成了鲜明对比。这种鸟类的出现突出了其飞行能力和适应性，因为它们能够在树林中快速迅速移动。此外，两只鸟的位置不同，一个在图像的左边，另一个在右边，这表明它们在同一片树林中移动得很近。这种鸟类的自然行为也有助于区分这两种鸟类物种。 | -
参考链接 & 感谢以下优秀的论文或项目
diff --git a/README_en.md b/README_en.md index f9a468a..b262189 100644 --- a/README_en.md +++ b/README_en.md @@ -214,23 +214,21 @@ and place it under `./dataset`.Note: Dataset Details
+**[Note 1]** Previously, extracting 500k fragmented image files could be very slow. From 2025-12-27, dataset format is unified to Parquet with image-text integrated storage, smaller size, no decompression needed, faster loading. + +**[Note 2]** Parquet is a columnar storage format supporting efficient compression and fast reading. To preview data content, run `python lm_dataset.py` in the `dataset/` directory to visualize the first 5 image-text pairs. + Pretrain data: ```bash -wget https://hf-mirror.com/datasets/jingyaogong/minimind-v_dataset/resolve/main/pretrain_data.jsonl -wget https://hf-mirror.com/datasets/jingyaogong/minimind-v_dataset/resolve/main/pretrain_images.zip -unzip pretrain_images.zip && rm pretrain_images.zip +wget https://hf-mirror.com/datasets/jingyaogong/minimind-v_dataset/resolve/main/pretrain_data.parquet ``` SFT data: ```bash -wget https://hf-mirror.com/datasets/jingyaogong/minimind-v_dataset/resolve/main/sft_data.jsonl -wget https://hf-mirror.com/datasets/jingyaogong/minimind-v_dataset/resolve/main/sft_images.zip -unzip sft_images.zip && rm sft_images.zip +wget https://hf-mirror.com/datasets/jingyaogong/minimind-v_dataset/resolve/main/sft_data.parquet ``` -`*.jsonl` contains Q&A text, and `*images` are the accompanying image data. After downloading, decompress the image data. - -Please reserve about 5GB of space for the dataset. If there is insufficient space for pretrain data, you can try skipping the pretrain training step and proceed directly to SFT training. +Please reserve about ~2GB of space for the dataset. If there is insufficient space for pretrain data, you can try skipping the pretrain training step and proceed directly to SFT training.
-
-
At this point, all the details of `MiniMind-V` have been presented.
The `MiniMind-V` model subclass completely inherits from `MiniMind`,
and is generated with **minimal** changes based on the latter,
@@ -573,34 +500,6 @@ The Q&A content has been translated, with better support for Chinese, further or
}
```
-(sft_vlm_data_multi.jsonl) Multi-image instruction fine-tuning dataset format:
-
-```json lines
-{
- "conversations": [
- {
- "role": "user",
- "content": "context: Source Image: Expansion Ideas for Video Understanding
- -written by [@xinyanghuang7](https://github.com/xinyanghuang7) - -For the video understanding capabilities of multimodal large models, one feasible approach is to refer to the existing -MiniCPM-V 2.6 Python example for video understanding. -The main idea is to extract key frames from the video and then perform multi-image inference. -Therefore, if you want to add video understanding capabilities to MiniMind-V, you can base it on the existing -multi-image training, refer to the key frame extraction method in this Python script, and increase the number of images -supported in the training files. -The more MAX_NUM_FRAMES supported, the more GPU memory it will consume. - -```text -import torch -from PIL import Image -from transformers import AutoModel, AutoTokenizer -from decord import VideoReader, cpu # pip install decord - -model = AutoModel.from_pretrained('openbmb/MiniCPM-V-2_6', trust_remote_code=True, - attn_implementation='sdpa', - torch_dtype=torch.bfloat16) # sdpa or flash_attention_2, no eager -model = model.eval().cuda() -tokenizer = AutoTokenizer.from_pretrained('openbmb/MiniCPM-V-2_6', trust_remote_code=True) - -MAX_NUM_FRAMES = 64 # if cuda OOM set a smaller number - - -def encode_video(video_path): - def uniform_sample(l, n): - gap = len(l) / n - idxs = [int(i * gap + gap / 2) for i in range(n)] - return [l[i] for i in idxs] - - vr = VideoReader(video_path, ctx=cpu(0)) - sample_fps = round(vr.get_avg_fps() / 1) # FPS - frame_idx = [i for i in range(0, len(vr), sample_fps)] - if len(frame_idx) > MAX_NUM_FRAMES: - frame_idx = uniform_sample(frame_idx, MAX_NUM_FRAMES) - frames = vr.get_batch(frame_idx).asnumpy() - frames = [Image.fromarray(v.astype('uint8')) for v in frames] - print('num frames:', len(frames)) - return frames - - -video_path = "video_test.mp4" -frames = encode_video(video_path) -question = "Describe the video" -msgs = [ - {'role': 'user', 'content': frames + [question]}, -] - -# Set decode params for video -params = {} -params["use_image_id"] = False -params["max_slice_nums"] = 2 # If cuda OOM and video resolution is greater than 448*448, set to 1 - -answer = model.chat( - image=None, - msgs=msgs, - tokenizer=tokenizer, - **params -) -print(answer) -``` - -
-
Dataset download
link: ([ModelScope](https://www.modelscope.cn/datasets/gongjy/minimind-v_dataset) | [HuggingFace](https://huggingface.co/datasets/jingyaogong/minimind-v_dataset))
@@ -616,10 +515,6 @@ Pre-training learns general image knowledge from a dataset of 595K samples, such
Instruction fine-tuning learns the real Q&A format for image-related questions from a dataset of 300K real
conversations, which better aligns with human communication habits.
-> train_sft_vlm
-
-Multi-image fine-tuning provides a demo: a bird comparison dataset with 13.6k real Q&A formats.
-
During training, the visual encoder, i.e., the CLIP model's gradients, are frozen, and only the Projection and LLM parts
are trained.
In pre-training, only the last layer parameters of Projection and LLM are learnable.
@@ -726,27 +621,6 @@ Download link:
-#### Multiple Image Dialogue (Effect is Limited)
-
-Data Description
- -* The multi-image dataset is relatively small and contains English conversations, focusing only on scenes with two image - comparisons. Therefore, the fine-tuning effect is limited, and this is just one reference approach. - -* `jsonl` contains textual instructions, and `images.zip` contains the corresponding image data (to be unzipped after - download). - -| Image1 | -Image2 | -512_sft_multi | -768_sft_multi | -
|---|---|---|---|
|
- |
- This image displays a bird scenario: a woman standing with a red and green mixed purple bird perched on her. The woman stands with the bird on her shoulders, while the red bird on her collar stands behind her. | -The two birds are flying in the same forest, some are in the center of the image, while others are smaller, creating a contrast. The birds’ presence highlights their flight ability and adaptability as they swiftly move through the woods. Additionally, the birds’ positions vary, with one on the left and the other on the right, indicating they are moving close within the same forest. Their natural behavior helps distinguish the two bird species. | -