github
/
minimind-v
mirror of https://github.com/jingyaogong/minimind-v.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

[update] readme

pull/121/head
jingyaogong 2026-05-02 00:35:40 +08:00
parent 1e86f201a2
commit b6a2532801
2 changed files with 128 additions and 153 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

138
README.md
View File

@ -141,139 +141,130 @@
* GPU: NVIDIA GeForce RTX 3090(24GB) * 8 * GPU: NVIDIA GeForce RTX 3090(24GB) * 8
* Ubuntu==20.04 * Ubuntu==20.04
* CUDA==12.2 * CUDA==12.2
* Python==3.10.16 * Python==3.10
* [requirements.txt](./requirements.txt) * [requirements.txt](./requirements.txt)
</details> </details>
### 第0步 ## 第0步（必须）
```bash
# 克隆代码仓库
git clone --depth 1 https://github.com/jingyaogong/minimind-v
```
```bash
# 下载siglip2模型到 ./model 目录下
git clone https://huggingface.co/jingyaogong/siglip2-base-p32-256-ve
# 或
git clone https://modelscope.cn/models/gongjy/siglip2-base-p32-256-ve
```
```bash
# 下载minimind语言模型权重到 ./out 目录下（作为训练VLM的基座语言模型）
# HuggingFace
https://huggingface.co/jingyaogong/minimind-3v-pytorch/blob/main/llm_768.pth
# 国内源
https://modelscope.cn/models/gongjy/minimind-3v-pytorch/resolve/master/llm_768.pth
```
## Ⅰ 测试已有模型效果
### 1' 环境准备 ### 1' 环境准备
```bash ```bash
# 克隆仓库代码
git clone --depth 1 https://github.com/jingyaogong/minimind-v
# 安装必要依赖
pip install -r requirements.txt -i https://pypi.tuna.tsinghua.edu.cn/simple pip install -r requirements.txt -i https://pypi.tuna.tsinghua.edu.cn/simple
``` ```
### 2' 下载模型 ### 2' 下载资源
```bash ```bash
git clone https://huggingface.co/jingyaogong/minimind-3v # 下载 SigLIP2 视觉编码器到 ./model/siglip2-base-p32-256-ve
modelscope download --model gongjy/siglip2-base-p32-256-ve --local_dir ./model/siglip2-base-p32-256-ve
# 下载 MiniMind 语言模型权重到 ./out 目录下（作为训练 VLM 的基座语言模型）
modelscope download --model gongjy/minimind-3v-pytorch llm_768.pth --local_dir ./out
``` ```
### 3' 命令行问答 注：也可从 [ModelScope Collection](https://modelscope.cn/collections/gongjy/MiniMind-V) 或 [HuggingFace Collection](https://huggingface.co/collections/jingyaogong/minimind-v-67000833fb60b3a2e1f3597d) 选择对应内容 `git clone`（需 LFS）下载，此处不再赘述。
完成后，结构应如下：
```text
minimind-v/
├── model/
│ ├── siglip2-base-p32-256-ve/
│ └── ...
├── out/
│ └── llm_768.pth
└── ...
```
## Ⅰ 🚀 模型推理
### 1' 下载发布权重
```bash
# 下载发布权重到 ./out 目录下
modelscope download --model gongjy/minimind-3v-pytorch --local_dir ./out
```
### 2' 命令行问答
```bash ```bash
# load_from='model': 加载原生PyTorch权重, load_from='其他路径': 加载transformers格式 # load_from='model': 加载原生PyTorch权重, load_from='其他路径': 加载transformers格式
python eval_vlm.py --load_from model --weight sft_vlm python eval_vlm.py --load_from model --weight sft_vlm
```
# 或使用transformers格式模型 如果使用 transformers 格式模型，可先下载模型目录：
```bash
git clone https://huggingface.co/jingyaogong/minimind-3v
python eval_vlm.py --load_from minimind-3v python eval_vlm.py --load_from minimind-3v
``` ```
### 4' 启动WebUI（可选） ### 3' 启动 WebUI（可选）
```bash ```bash
# ⚠️ 须先将 transformers 格式模型文件夹复制到 ./scripts/ 目录下（例如：cp -r minimind-3v ./scripts/minimind-3v），web_demo_vlm 脚本会自动扫描该目录下包含权重文件的子文件夹，如不存在则报错 # ⚠️ 须先将 transformers 格式模型文件夹复制到 ./scripts/ 目录下，web_demo_vlm 脚本会自动扫描该目录下包含权重文件的子文件夹，如不存在则报错
cp -r minimind-3v ./scripts/minimind-3v
cd scripts && python web_demo_vlm.py cd scripts && python web_demo_vlm.py
``` ```
## Ⅱ 从0开始自己训练 ## Ⅱ 🛠️ 模型训练
### 1' 环境准备 <details style="color:rgb(128,128,128)">
```bash
pip install -r requirements.txt -i https://pypi.tuna.tsinghua.edu.cn/simple
```
<details>
<summary>注：提前测试Torch是否可用cuda</summary> <summary>注：提前测试Torch是否可用cuda</summary>
```bash ```python
import torch import torch
print(torch.cuda.is_available()) print(torch.cuda.is_available())
``` ```
如果不可用，请自行去[torch_stable](https://download.pytorch.org/whl/torch_stable.html) 如果不可用，请自行去 [torch_stable](https://download.pytorch.org/whl/torch_stable.html) 下载 whl 文件安装。
下载whl文件安装。参考[链接](https://blog.csdn.net/weixin_45456738/article/details/141029610?ops_request_misc=&request_id=&biz_id=102&utm_term=%E5%AE%89%E8%A3%85torch&utm_medium=distribute.pc_search_result.none-task-blog-2~all~sobaiduweb~default-2-141029610.nonecase&spm=1018.2226.3001.4187)
</details> </details>
### 2' 数据下载 ### 1' 下载数据
从下文提供的[数据集链接](https://huggingface.co/datasets/jingyaogong/minimind-v_dataset) 快速开始时，直接从[数据集链接](https://huggingface.co/datasets/jingyaogong/minimind-v_dataset)下载 `sft_i2t.parquet`，并放到 `./dataset` 下即可。
下载所需内容并放到`./dataset`下。
<details> <details style="color:rgb(128,128,128)">
<summary>注：数据集须知</summary> <summary>注：数据集须知</summary>
【注1】之前需解压50万零碎的图像文件可能非常慢。2025-12-27起，数据集格式统一为Parquet，图文一体化存储，体积更小，无需解压，加载更快。 【注1】之前需解压50万零碎的图像文件可能非常慢。2025-12-27起，数据集格式统一为 Parquet，图文一体化存储，体积更小，无需解压，加载更快。
【注2】Parquet是列式存储格式，支持高效压缩和快速读取。如果你对它感到陌生，可以预览数据内容，在`dataset/`目录下执行`python lm_dataset.py`可视化前5条图文对 【注2】Parquet 是列式存储格式，支持高效压缩和快速读取。如果你对它感到陌生，可以预览数据内容，在 `dataset/` 目录下执行 `python lm_dataset.py` 可视化前5条图文对。
Pretrain 数据（可选；仅包含 caption 子集）： Pretrain 数据（可选；仅包含 caption 子集）：
```bash ```bash
wget https://hf-mirror.com/datasets/jingyaogong/minimind-v_dataset/resolve/main/pretrain_i2t.parquet wget https://hf-mirror.com/datasets/jingyaogong/minimind-v_dataset/resolve/main/pretrain_i2t.parquet -P ./dataset
``` ```
SFT 数据（必须；包含 caption + instruct + 纯文本全量合并）： SFT 单文件 290 万条已把 Pretrain 作为子集合并，经全局 dictionary encoding 去重后体积只比 SFT 原版多 ~10%，可覆盖所有训练阶段。因此快速复现时可以跳过 Pretrain，直接进入 SFT。
```bash
wget https://hf-mirror.com/datasets/jingyaogong/minimind-v_dataset/resolve/main/sft_i2t.parquet
```
SFT 单文件 290 万条已把 Pretrain 作为子集合并，经全局 dictionary encoding 去重后体积只比 SFT 原版多 ~10%，可覆盖所有训练阶段。
</details> </details>
### 3' 开始训练 ### 2' 开始训练
**3.1 Pretrain（可选）** 推荐直接执行 SFT。默认 `--freeze_llm 1`，即训练 `vision_proj` 和 LLM 首尾层，保留中间层原有语言能力：
> SFT 数据集 `sft_i2t.parquet` 已把 Pretrain 数据作为子集合并进来，**此阶段可跳过**，直接 `--from_weight llm` 进入 SFT 即可；若想让 Projector 提前完成对齐、SFT 收敛更稳，可以提前执行 Pretrain。
```bash ```bash
# 基础训练命令（从LLM权重开始，仅训练 vision_proj） python train_sft_vlm.py --epochs 2 --from_weight llm
python train_pretrain_vlm.py --epochs 2 --from_weight llm
``` ```
> 执行 Pretrain，得到 `pretrain_vlm_*.pth` 作为 Pretrain 的输出权重（其中*为模型的dimension，默认为768） 如果希望让 Projector 先完成一轮图文对齐，再进入 SFT，可额外执行 Pretrain：
**3.2 SFT（必须）**
```bash ```bash
# 基础训练命令（默认 --freeze_llm 1：解冻 proj + LLM 首尾层，保留中间层原 LLM 知识） python train_pretrain_vlm.py --epochs 2 --from_weight llm
# 若已执行 3.1：--from_weight pretrain_vlm；若跳过 Pretrain：--from_weight llm
python train_sft_vlm.py --epochs 2 --from_weight pretrain_vlm python train_sft_vlm.py --epochs 2 --from_weight pretrain_vlm
``` ```
> 执行 SFT，得到 `sft_vlm_*.pth` 作为 SFT 的输出权重 执行完成后，`out/` 下会生成 `sft_vlm_*.pth` 作为 SFT 权重。
<details> <details style="color:rgb(128,128,128)">
<summary>注：训练须知</summary> <summary>注：训练须知</summary>
**训练特性：**
- 支持断点续训：添加`--from_resume 1`参数可从上次中断处继续训练 - 支持断点续训：添加`--from_resume 1`参数可从上次中断处继续训练
- 支持GPU数量变化：续训时GPU数量改变会自动转换step - 支持GPU数量变化：续训时GPU数量改变会自动转换step
- 原子性保存：使用临时文件+替换机制，防止保存过程中断导致权重损坏 - 原子性保存：使用临时文件+替换机制，防止保存过程中断导致权重损坏
@ -293,12 +284,9 @@ python train_sft_vlm.py --epochs 4 --from_resume 1
</details> </details>
### 3' 测试已训练模型（可选）
--- 确保需要测试的模型 `*.pth` 文件位于 `./out/` 目录下。
### 4' 测试模型效果
确保需要测试的模型`*.pth`文件位于`./out/`目录下。
也可以直接去[此处](https://huggingface.co/jingyaogong/minimind-3v-pytorch)下载使用我训练的`*.pth`文件。 也可以直接去[此处](https://huggingface.co/jingyaogong/minimind-3v-pytorch)下载使用我训练的`*.pth`文件。
```bash ```bash
@ -312,7 +300,7 @@ python eval_vlm.py --weight pretrain_vlm
--- ---
> [!TIP] > [!TIP]
> 训练脚本均为 PyTorch 原生框架，均支持多卡加速，假设你的设备有N (N＞1) 张显卡： > 训练脚本均为 PyTorch 原生框架，均支持多卡加速，假设你的设备有 N (N＞1) 张显卡：
单机N卡启动训练方式 (DDP, 支持多机多卡集群) 单机N卡启动训练方式 (DDP, 支持多机多卡集群)

143
README_en.md
View File

@ -136,148 +136,137 @@ Is the training process difficult? Now, let's explore the answers and feel the j
# 📌 Quick Start # 📌 Quick Start
<details> <details>
<summary>Sharing my hardware and software configuration (for reference only)</summary> <summary>My software and hardware setup (for reference only)</summary>
* CPU: Intel(R) Core(TM) i9-10980XE CPU @ 3.00GHz * CPU: Intel(R) Core(TM) i9-10980XE CPU @ 3.00GHz
* RAM: 128 GB * RAM: 128 GB
* GPU: NVIDIA GeForce RTX 3090(24GB) * 8 * GPU: NVIDIA GeForce RTX 3090(24GB) * 8
* Ubuntu==20.04 * Ubuntu==20.04
* CUDA==12.2 * CUDA==12.2
* Python==3.10.16 * Python==3.10
* [requirements.txt](./requirements.txt) * [requirements.txt](./requirements.txt)
</details> </details>
### Step 0 ## Step 0 (required)
### 1' Environment setup
```bash ```bash
# Clone the code repository # Clone the repository
git clone --depth 1 https://github.com/jingyaogong/minimind-v git clone --depth 1 https://github.com/jingyaogong/minimind-v
# Install dependencies
pip install -r requirements.txt -i https://pypi.tuna.tsinghua.edu.cn/simple
``` ```
### 2' Download resources
```bash ```bash
# Download the siglip2 model to the ./model directory # Download the SigLIP2 vision encoder to ./model/siglip2-base-p32-256-ve
git clone https://huggingface.co/jingyaogong/siglip2-base-p32-256-ve modelscope download --model gongjy/siglip2-base-p32-256-ve --local_dir ./model/siglip2-base-p32-256-ve
# or # Download the MiniMind language model weight to ./out (used as the base language model for VLM training)
git clone https://modelscope.cn/models/gongjy/siglip2-base-p32-256-ve modelscope download --model gongjy/minimind-3v-pytorch llm_768.pth --local_dir ./out
``` ```
Alternatively, the same files can be selected from the [ModelScope Collection](https://modelscope.cn/collections/gongjy/MiniMind-V) or [HuggingFace Collection](https://huggingface.co/collections/jingyaogong/minimind-v-67000833fb60b3a2e1f3597d) and downloaded with `git clone` (LFS required).
The directory should look like this after the resources are ready:
```text
minimind-v/
├── model/
│ ├── siglip2-base-p32-256-ve/
│ └── ...
├── out/
│ └── llm_768.pth
└── ...
```
## Ⅰ 🚀 Model inference
### 1' Download released weights
```bash ```bash
# Download the minimind language model to the ./out directory (as the base language model for training VLM): # Download released weights to ./out
# HuggingFace modelscope download --model gongjy/minimind-3v-pytorch --local_dir ./out
https://huggingface.co/jingyaogong/minimind-3v-pytorch/blob/main/llm_768.pth
# Domestic source
https://modelscope.cn/models/gongjy/minimind-3v-pytorch/resolve/master/llm_768.pth
``` ```
### 2' Command-line Q&A
## Ⅰ Test an existing model's performance
### 1' Environment Preparation
```bash
pip install -r requirements.txt -i https://mirrors.aliyun.com/pypi/simple
```
### 2' Download the model
```bash
git clone https://huggingface.co/jingyaogong/minimind-3v
```
### 3' Command-line Q&A
```bash ```bash
# load_from='model': load native PyTorch weights, load_from='other path': load transformers format # load_from='model': load native PyTorch weights, load_from='other path': load transformers format
python eval_vlm.py --load_from model --weight sft_vlm python eval_vlm.py --load_from model --weight sft_vlm
```
# Or use transformers format model If using a transformers-format model, download the model directory first:
```bash
git clone https://huggingface.co/jingyaogong/minimind-3v
python eval_vlm.py --load_from minimind-3v python eval_vlm.py --load_from minimind-3v
``` ```
### 4' Or start the WebUI ### 3' Start WebUI (optional)
```bash ```bash
# ⚠️ You must first copy the transformers model folder to the ./scripts/ directory (e.g.: cp -r minimind-3v ./scripts/minimind-3v). The web_demo_vlm script will automatically scan subdirectories containing weight files; it will report an error if none are found. # ⚠️ The transformers-format model directory must be copied to ./scripts/ first. web_demo_vlm scans subdirectories under ./scripts/ that contain weight files and reports an error if none are found.
cp -r minimind-3v ./scripts/minimind-3v
cd scripts && python web_demo_vlm.py cd scripts && python web_demo_vlm.py
``` ```
## Ⅱ Train from scratch ## Ⅱ 🛠️ Model training
### 1' Environment Preparation <details style="color:rgb(128,128,128)">
<summary>Note: test whether Torch can use CUDA</summary>
```bash ```python
pip install -r requirements.txt -i https://mirrors.aliyun.com/pypi/simple
```
<details>
<summary>Note: Test if Torch can use CUDA</summary>
```bash
import torch import torch
print(torch.cuda.is_available()) print(torch.cuda.is_available())
``` ```
If unavailable, download the whl file from [torch_stable](https://download.pytorch.org/whl/torch_stable.html) for If CUDA is unavailable, download the matching whl file from [torch_stable](https://download.pytorch.org/whl/torch_stable.html) and install it manually.
installation. Refer
to [this link](https://blog.csdn.net/weixin_45456738/article/details/141029610?ops_request_misc=&request_id=&biz_id=102&utm_term=%E5%AE%89%E8%A3%85torch&utm_medium=distribute.pc_search_result.none-task-blog-2~all~sobaiduweb~default-2-141029610.nonecase&spm=1018.2226.3001.4187)
for help.
</details> </details>
### 2' Download Data ### 1' Download data
Download the required content from the [dataset link](https://huggingface.co/datasets/jingyaogong/minimind-v_dataset) For a quick start, download `sft_i2t.parquet` from the [dataset link](https://huggingface.co/datasets/jingyaogong/minimind-v_dataset) and place it under `./dataset`.
and place it under `./dataset`.
<details> <details style="color:rgb(128,128,128)">
<summary>Note: Dataset Details</summary> <summary>Note: dataset details</summary>
**[Note 1]** Previously, extracting 500k fragmented image files could be very slow. From 2025-12-27, dataset format is unified to Parquet with image-text integrated storage, smaller size, no decompression needed, faster loading. **[Note 1]** The older dataset required extracting 500k fragmented image files, which could be very slow. Since 2025-12-27, the dataset has been unified into Parquet with image and text stored together. It is smaller, requires no decompression, and loads faster.
**[Note 2]** Parquet is a columnar storage format supporting efficient compression and fast reading. To preview data content, run `python lm_dataset.py` in the `dataset/` directory to visualize the first 5 image-text pairs. **[Note 2]** Parquet is a columnar storage format with efficient compression and fast reading. If it is unfamiliar, run `python lm_dataset.py` under `dataset/` to visualize the first 5 image-text pairs.
Pretrain data (optional; contains caption subset only): Pretrain data (optional; contains caption subset only):
```bash ```bash
wget https://hf-mirror.com/datasets/jingyaogong/minimind-v_dataset/resolve/main/pretrain_i2t.parquet wget https://hf-mirror.com/datasets/jingyaogong/minimind-v_dataset/resolve/main/pretrain_i2t.parquet -P ./dataset
``` ```
SFT data (required; contains full caption + instruct + text merge): The single `sft_i2t.parquet` file contains 2.9M rows and absorbs Pretrain as a subset. After global dictionary encoding deduplication, it is only ~10% larger than the original SFT file and is enough to cover every training stage. For quick reproduction, Pretrain can be skipped and SFT can be started directly.
```bash
wget https://hf-mirror.com/datasets/jingyaogong/minimind-v_dataset/resolve/main/sft_i2t.parquet
```
The single `sft_i2t.parquet` file (2.9M rows) absorbs Pretrain as a subset, and after global dictionary encoding dedup it is only ~10% larger than the original SFT — enough to cover every training stage.
</details> </details>
### 3' Start Training ### 2' Start training
**3.1 Pretrain (optional)** SFT is the recommended starting point. By default, `--freeze_llm 1` trains `vision_proj` and the first/last LLM layers while keeping the middle layers' original language ability:
> `sft_i2t.parquet` already contains the Pretrain data as a subset, so **this stage can be skipped** and you may go directly to SFT with `--from_weight llm`. If you prefer the Projector to be pre-aligned first and SFT to converge more stably, run a round of Pretrain separately.
```bash ```bash
# Basic training command (start from LLM weights, train vision_proj only) python train_sft_vlm.py --epochs 2 --from_weight llm
python train_pretrain_vlm.py --epochs 2 --from_weight llm
``` ```
> Run Pretrain to get `pretrain_vlm_*.pth` as the Pretrain output weights (* is the model dimension, default 768). If the Projector should be aligned on image-text pairs before SFT, run Pretrain first:
**3.2 SFT (required)**
```bash ```bash
# Basic command (default --freeze_llm 1: unfreeze proj + first/last LLM layers, keep middle layers' original LLM weights) python train_pretrain_vlm.py --epochs 2 --from_weight llm
# If 3.1 was run: --from_weight pretrain_vlm; if Pretrain was skipped: --from_weight llm
python train_sft_vlm.py --epochs 2 --from_weight pretrain_vlm python train_sft_vlm.py --epochs 2 --from_weight pretrain_vlm
``` ```
> Run SFT to get `sft_vlm_*.pth` as the SFT output weights. After training, `sft_vlm_*.pth` will be written under `out/` as the SFT weight.
<details> <details style="color:rgb(128,128,128)">
<summary>Note: Training Details</summary> <summary>Note: training details</summary>
**Training Features:**
- Support checkpoint resumption: add `--from_resume 1` parameter to continue from last interruption - Support checkpoint resumption: add `--from_resume 1` parameter to continue from last interruption
- Support GPU count changes: automatically convert steps when GPU count changes during resumption - Support GPU count changes: automatically convert steps when GPU count changes during resumption
- Atomic saving: use temporary file + replacement mechanism to prevent weight corruption from interruption - Atomic saving: use temporary file + replacement mechanism to prevent weight corruption from interruption
@ -297,9 +286,7 @@ python train_sft_vlm.py --epochs 4 --from_resume 1
</details> </details>
--- ### 3' Test a trained model (optional)
### 4' Test the Model's Performance
Ensure that the model `*.pth` file you want to test is located in the `./out/` directory. Ensure that the model `*.pth` file you want to test is located in the `./out/` directory.
You can also directly download the pre-trained `*.pth` file You can also directly download the pre-trained `*.pth` file