Merge 31b57a0569 into 2b94dcb3d0
100apps
GitHub
commit
be1516ee25
|
|
@ -1,5 +1,10 @@
|
|||
model/__pycache__
|
||||
__pycache__
|
||||
out
|
||||
website/
|
||||
docs-minimind-v/
|
||||
测试.py
|
||||
.venv
|
||||
model
|
||||
minimind-3v
|
||||
dataset
|
||||
45
README.md
45
README.md
|
|
@ -142,7 +142,7 @@
|
|||
* Ubuntu==20.04
|
||||
* CUDA==12.2
|
||||
* Python==3.10
|
||||
* [requirements.txt](./requirements.txt)
|
||||
* [pyproject.toml](./pyproject.toml)（使用 [uv](https://docs.astral.sh/uv/) 管理依赖）
|
||||
|
||||
</details>
|
||||
|
||||
|
|
@ -153,17 +153,17 @@
|
|||
```bash
|
||||
# 克隆仓库代码
|
||||
git clone --depth 1 https://github.com/jingyaogong/minimind-v
|
||||
# 安装必要依赖
|
||||
pip install -r requirements.txt -i https://pypi.tuna.tsinghua.edu.cn/simple
|
||||
# 安装必要依赖（使用 uv）
|
||||
uv sync
|
||||
```
|
||||
|
||||
### 2' 下载资源
|
||||
|
||||
```bash
|
||||
# 下载 SigLIP2 视觉编码器到 ./model/siglip2-base-p32-256-ve
|
||||
modelscope download --model gongjy/siglip2-base-p32-256-ve --local_dir ./model/siglip2-base-p32-256-ve
|
||||
uv run modelscope download --model gongjy/siglip2-base-p32-256-ve --local_dir ./model/siglip2-base-p32-256-ve
|
||||
# 下载 MiniMind 语言模型权重到 ./out 目录下（作为训练 VLM 的基座语言模型）
|
||||
modelscope download --model gongjy/minimind-3v-pytorch llm_768.pth --local_dir ./out
|
||||
uv run modelscope download --model gongjy/minimind-3v-pytorch llm_768.pth --local_dir ./out
|
||||
```
|
||||
|
||||
注：也可从 [ModelScope Collection](https://modelscope.cn/collections/gongjy/MiniMind-V) 或 [HuggingFace Collection](https://huggingface.co/collections/jingyaogong/minimind-v-67000833fb60b3a2e1f3597d) 选择对应内容 `git clone`（需 LFS）下载，此处不再赘述。
|
||||
|
|
@ -186,29 +186,27 @@ minimind-v/
|
|||
|
||||
```bash
|
||||
# 下载发布权重到 ./out 目录下
|
||||
modelscope download --model gongjy/minimind-3v-pytorch --local_dir ./out
|
||||
uv run modelscope download --model gongjy/minimind-3v-pytorch --local_dir ./out
|
||||
```
|
||||
|
||||
### 2' 命令行问答
|
||||
|
||||
```bash
|
||||
# load_from='model': 加载原生PyTorch权重, load_from='其他路径': 加载transformers格式
|
||||
python eval_vlm.py --load_from model --weight sft_vlm
|
||||
uv run python eval_vlm.py --load_from model --weight sft_vlm
|
||||
```
|
||||
|
||||
如果使用 transformers 格式模型，可先下载模型目录：
|
||||
|
||||
```bash
|
||||
git clone https://huggingface.co/jingyaogong/minimind-3v
|
||||
python eval_vlm.py --load_from minimind-3v
|
||||
uv run python eval_vlm.py --load_from minimind-3v
|
||||
```
|
||||
|
||||
### 3' 启动 WebUI（可选）
|
||||
|
||||
```bash
|
||||
# ⚠️ 须先将 transformers 格式模型文件夹复制到 ./scripts/ 目录下，web_demo_vlm 脚本会自动扫描该目录下包含权重文件的子文件夹，如不存在则报错
|
||||
cp -r minimind-3v ./scripts/minimind-3v
|
||||
cd scripts && python web_demo_vlm.py
|
||||
uv run python scripts/web_demo_vlm.py
|
||||
```
|
||||
|
||||
## Ⅱ 🛠️ 模型训练
|
||||
|
|
@ -229,12 +227,17 @@ print(torch.cuda.is_available())
|
|||
|
||||
快速开始时，直接从[数据集链接](https://huggingface.co/datasets/jingyaogong/minimind-v_dataset)下载 `sft_i2t.parquet`，并放到 `./dataset` 下即可。
|
||||
|
||||
```bash
|
||||
# 完整下载
|
||||
hf download --repo-type dataset jingyaogong/minimind-v_dataset --local-dir dataset
|
||||
```
|
||||
|
||||
<details style="color:rgb(128,128,128)">
|
||||
<summary>注：数据集须知</summary>
|
||||
|
||||
【注1】之前需解压50万零碎的图像文件可能非常慢。2025-12-27起，数据集格式统一为 Parquet，图文一体化存储，体积更小，无需解压，加载更快。
|
||||
|
||||
【注2】Parquet 是列式存储格式，支持高效压缩和快速读取。如果你对它感到陌生，可以预览数据内容，在 `dataset/` 目录下执行 `python lm_dataset.py` 可视化前5条图文对。
|
||||
【注2】Parquet 是列式存储格式，支持高效压缩和快速读取。如果你对它感到陌生，可以预览数据内容，在 `dataset/` 目录下执行 `uv run python lm_dataset.py` 可视化前5条图文对。
|
||||
|
||||
Pretrain 数据（可选；仅包含 caption 子集）：
|
||||
```bash
|
||||
|
|
@ -250,14 +253,14 @@ SFT 单文件 290 万条已把 Pretrain 作为子集合并，经全局 dictionar
|
|||
推荐直接执行 SFT。默认 `--freeze_llm 1`，即训练 `vision_proj` 和 LLM 首尾层，保留中间层原有语言能力：
|
||||
|
||||
```bash
|
||||
python train_sft_vlm.py --epochs 2 --from_weight llm
|
||||
uv run python train_sft_vlm.py --epochs 2 --from_weight llm
|
||||
```
|
||||
|
||||
如果希望让 Projector 先完成一轮图文对齐，再进入 SFT，可额外执行 Pretrain：
|
||||
|
||||
```bash
|
||||
python train_pretrain_vlm.py --epochs 2 --from_weight llm
|
||||
python train_sft_vlm.py --epochs 2 --from_weight pretrain_vlm
|
||||
uv run python train_pretrain_vlm.py --epochs 2 --from_weight llm
|
||||
uv run python train_sft_vlm.py --epochs 2 --from_weight pretrain_vlm
|
||||
```
|
||||
|
||||
执行完成后，`out/` 下会生成 `sft_vlm_*.pth` 作为 SFT 权重。
|
||||
|
|
@ -272,7 +275,7 @@ python train_sft_vlm.py --epochs 2 --from_weight pretrain_vlm
|
|||
|
||||
```bash
|
||||
# 训练中断后，使用相同命令并添加 --from_resume 1
|
||||
python train_sft_vlm.py --epochs 4 --from_resume 1
|
||||
uv run python train_sft_vlm.py --epochs 4 --from_resume 1
|
||||
```
|
||||
|
||||
**参数说明：**
|
||||
|
|
@ -291,10 +294,10 @@ python train_sft_vlm.py --epochs 4 --from_resume 1
|
|||
|
||||
```bash
|
||||
# 测试SFT模型（默认）
|
||||
python eval_vlm.py --weight sft_vlm
|
||||
uv run python eval_vlm.py --weight sft_vlm
|
||||
|
||||
# 测试Pretrain模型
|
||||
python eval_vlm.py --weight pretrain_vlm
|
||||
uv run python eval_vlm.py --weight pretrain_vlm
|
||||
```
|
||||
|
||||
---
|
||||
|
|
@ -305,7 +308,7 @@ python eval_vlm.py --weight pretrain_vlm
|
|||
单机N卡启动训练方式 (DDP, 支持多机多卡集群)
|
||||
|
||||
```bash
|
||||
torchrun --nproc_per_node N train_xxx.py
|
||||
uv run torchrun --nproc_per_node N train_xxx.py
|
||||
```
|
||||
|
||||
<details>
|
||||
|
|
@ -323,9 +326,9 @@ deepspeed --master_port 29500 --num_gpus=N train_xxx.py
|
|||
|
||||
```bash
|
||||
# 需要登录: wandb login
|
||||
torchrun --nproc_per_node N train_xxx.py --use_wandb
|
||||
uv run torchrun --nproc_per_node N train_xxx.py --use_wandb
|
||||
# and
|
||||
python train_xxx.py --use_wandb
|
||||
uv run python train_xxx.py --use_wandb
|
||||
```
|
||||
|
||||
通过添加`--use_wandb`参数，可以记录训练过程，训练完成后，可以在wandb网站上查看训练过程。通过修改`wandb_project`
|
||||
|
|
|
|||
32
README_en.md
32
README_en.md
|
|
@ -144,7 +144,7 @@ Is the training process difficult? Now, let's explore the answers and feel the j
|
|||
* Ubuntu==20.04
|
||||
* CUDA==12.2
|
||||
* Python==3.10
|
||||
* [requirements.txt](./requirements.txt)
|
||||
* [pyproject.toml](./pyproject.toml) (dependencies managed by [uv](https://docs.astral.sh/uv/))
|
||||
|
||||
</details>
|
||||
|
||||
|
|
@ -155,8 +155,8 @@ Is the training process difficult? Now, let's explore the answers and feel the j
|
|||
```bash
|
||||
# Clone the repository
|
||||
git clone --depth 1 https://github.com/jingyaogong/minimind-v
|
||||
# Install dependencies
|
||||
pip install -r requirements.txt -i https://pypi.tuna.tsinghua.edu.cn/simple
|
||||
# Install dependencies (using uv)
|
||||
uv sync
|
||||
```
|
||||
|
||||
### 2' Download resources
|
||||
|
|
@ -195,14 +195,14 @@ modelscope download --model gongjy/minimind-3v-pytorch --local_dir ./out
|
|||
|
||||
```bash
|
||||
# load_from='model': load native PyTorch weights, load_from='other path': load transformers format
|
||||
python eval_vlm.py --load_from model --weight sft_vlm
|
||||
uv run python eval_vlm.py --load_from model --weight sft_vlm
|
||||
```
|
||||
|
||||
If using a transformers-format model, download the model directory first:
|
||||
|
||||
```bash
|
||||
git clone https://huggingface.co/jingyaogong/minimind-3v
|
||||
python eval_vlm.py --load_from minimind-3v
|
||||
uv run python eval_vlm.py --load_from minimind-3v
|
||||
```
|
||||
|
||||
### 3' Start WebUI (optional)
|
||||
|
|
@ -210,7 +210,7 @@ python eval_vlm.py --load_from minimind-3v
|
|||
```bash
|
||||
# ⚠️ The transformers-format model directory must be copied to ./scripts/ first. web_demo_vlm scans subdirectories under ./scripts/ that contain weight files and reports an error if none are found.
|
||||
cp -r minimind-3v ./scripts/minimind-3v
|
||||
cd scripts && python web_demo_vlm.py
|
||||
cd scripts && uv run python web_demo_vlm.py
|
||||
```
|
||||
|
||||
## Ⅱ 🛠️ Model training
|
||||
|
|
@ -236,7 +236,7 @@ For a quick start, download `sft_i2t.parquet` from the [dataset link](https://hu
|
|||
|
||||
**[Note 1]** The older dataset required extracting 500k fragmented image files, which could be very slow. Since 2025-12-27, the dataset has been unified into Parquet with image and text stored together. It is smaller, requires no decompression, and loads faster.
|
||||
|
||||
**[Note 2]** Parquet is a columnar storage format with efficient compression and fast reading. If it is unfamiliar, run `python lm_dataset.py` under `dataset/` to visualize the first 5 image-text pairs.
|
||||
**[Note 2]** Parquet is a columnar storage format with efficient compression and fast reading. If it is unfamiliar, run `uv run python lm_dataset.py` under `dataset/` to visualize the first 5 image-text pairs.
|
||||
|
||||
Pretrain data (optional; contains caption subset only):
|
||||
```bash
|
||||
|
|
@ -252,14 +252,14 @@ The single `sft_i2t.parquet` file contains 2.9M rows and absorbs Pretrain as a s
|
|||
SFT is the recommended starting point. By default, `--freeze_llm 1` trains `vision_proj` and the first/last LLM layers while keeping the middle layers' original language ability:
|
||||
|
||||
```bash
|
||||
python train_sft_vlm.py --epochs 2 --from_weight llm
|
||||
uv run python train_sft_vlm.py --epochs 2 --from_weight llm
|
||||
```
|
||||
|
||||
If the Projector should be aligned on image-text pairs before SFT, run Pretrain first:
|
||||
|
||||
```bash
|
||||
python train_pretrain_vlm.py --epochs 2 --from_weight llm
|
||||
python train_sft_vlm.py --epochs 2 --from_weight pretrain_vlm
|
||||
uv run python train_pretrain_vlm.py --epochs 2 --from_weight llm
|
||||
uv run python train_sft_vlm.py --epochs 2 --from_weight pretrain_vlm
|
||||
```
|
||||
|
||||
After training, `sft_vlm_*.pth` will be written under `out/` as the SFT weight.
|
||||
|
|
@ -274,7 +274,7 @@ After training, `sft_vlm_*.pth` will be written under `out/` as the SFT weight.
|
|||
|
||||
```bash
|
||||
# To resume training after interruption, use the same command and add --from_resume 1
|
||||
python train_sft_vlm.py --epochs 4 --from_resume 1
|
||||
uv run python train_sft_vlm.py --epochs 4 --from_resume 1
|
||||
```
|
||||
|
||||
**Parameter Description:**
|
||||
|
|
@ -294,10 +294,10 @@ from [here](https://huggingface.co/jingyaogong/minimind-3v-pytorch).
|
|||
|
||||
```bash
|
||||
# Test SFT model (default)
|
||||
python eval_vlm.py --weight sft_vlm
|
||||
uv run python eval_vlm.py --weight sft_vlm
|
||||
|
||||
# Test Pretrain model
|
||||
python eval_vlm.py --weight pretrain_vlm
|
||||
uv run python eval_vlm.py --weight pretrain_vlm
|
||||
```
|
||||
|
||||
---
|
||||
|
|
@ -309,7 +309,7 @@ python eval_vlm.py --weight pretrain_vlm
|
|||
Single-machine N-card training method (DDP, supports multi-machine multi-card cluster)
|
||||
|
||||
```bash
|
||||
torchrun --nproc_per_node N train_xxx.py
|
||||
uv run torchrun --nproc_per_node N train_xxx.py
|
||||
```
|
||||
|
||||
<details>
|
||||
|
|
@ -325,9 +325,9 @@ You can enable wandb logging during training:
|
|||
|
||||
```bash
|
||||
# You need to log in: wandb login
|
||||
torchrun --nproc_per_node N train_xxx.py --use_wandb
|
||||
uv run torchrun --nproc_per_node N train_xxx.py --use_wandb
|
||||
# and
|
||||
python train_xxx.py --use_wandb
|
||||
uv run python train_xxx.py --use_wandb
|
||||
```
|
||||
|
||||
By adding the `--use_wandb` parameter, you can log the training process, and after training is complete, you can view
|
||||
|
|
|
|||
|
|
@ -0,0 +1,62 @@
|
|||
[project]
|
||||
name = "minimind-v"
|
||||
version = "0.1.0"
|
||||
description = "MiniMind-V multimodal vision-language model training and inference scripts."
|
||||
readme = "README.md"
|
||||
requires-python = ">=3.10,<3.11"
|
||||
dependencies = [
|
||||
"datasets==3.6.0",
|
||||
"datasketch==1.6.4",
|
||||
"Flask==3.0.3",
|
||||
"Flask-Cors==4.0.0",
|
||||
"jieba==0.42.1",
|
||||
"jsonlines==4.0.0",
|
||||
"marshmallow==3.22.0",
|
||||
"matplotlib==3.5.1",
|
||||
"ngrok==1.4.0",
|
||||
"nltk==3.8",
|
||||
"numpy==1.26.4",
|
||||
"openai==1.59.6",
|
||||
"peft==0.7.1",
|
||||
"psutil==5.9.8",
|
||||
"pydantic==2.8.2",
|
||||
"rich==13.7.1",
|
||||
"scikit-learn==1.5.1",
|
||||
"sentence-transformers==2.3.1",
|
||||
"simhash==2.1.2",
|
||||
"tiktoken==0.10.0",
|
||||
"transformers==4.57.6",
|
||||
"jinja2==3.1.2",
|
||||
"trl==0.13.0",
|
||||
"ujson==5.1.0",
|
||||
"wandb==0.18.3",
|
||||
"streamlit==1.50.0",
|
||||
"gradio==5.49.1",
|
||||
"modelscope-studio==1.6.1",
|
||||
"python-multipart==0.0.22",
|
||||
"pyarrow==23.0.0",
|
||||
"Pillow",
|
||||
"swanlab==0.6.12",
|
||||
"torch==2.10.0",
|
||||
"torchvision==0.25.0",
|
||||
]
|
||||
|
||||
[tool.uv]
|
||||
package = false
|
||||
|
||||
[tool.uv.sources]
|
||||
torch = [
|
||||
{ index = "pytorch-cu128", marker = "sys_platform == 'win32' or sys_platform == 'linux'" },
|
||||
]
|
||||
torchvision = [
|
||||
{ index = "pytorch-cu128", marker = "sys_platform == 'win32' or sys_platform == 'linux'" },
|
||||
]
|
||||
|
||||
[[tool.uv.index]]
|
||||
url = "https://pypi.tuna.tsinghua.edu.cn/simple"
|
||||
default = true
|
||||
|
||||
[[tool.uv.index]]
|
||||
name = "pytorch-cu128"
|
||||
url = "https://download.pytorch.org/whl/cu128"
|
||||
explicit = true
|
||||
|
|
@ -1,34 +0,0 @@
|
|||
datasets==3.6.0
|
||||
datasketch==1.6.4
|
||||
Flask==3.0.3
|
||||
Flask_Cors==4.0.0
|
||||
jieba==0.42.1
|
||||
jsonlines==4.0.0
|
||||
marshmallow==3.22.0
|
||||
matplotlib==3.5.1
|
||||
ngrok==1.4.0
|
||||
nltk==3.8
|
||||
numpy==1.26.4
|
||||
openai==1.59.6
|
||||
peft==0.7.1
|
||||
psutil==5.9.8
|
||||
pydantic==2.8.2
|
||||
rich==13.7.1
|
||||
scikit_learn==1.5.1
|
||||
sentence_transformers==2.3.1
|
||||
simhash==2.1.2
|
||||
tiktoken==0.10.0
|
||||
transformers==4.57.6
|
||||
jinja2==3.1.2
|
||||
trl==0.13.0
|
||||
ujson==5.1.0
|
||||
wandb==0.18.3
|
||||
streamlit==1.50.0
|
||||
gradio==5.49.1
|
||||
modelscope-studio==1.6.1
|
||||
python-multipart==0.0.22
|
||||
pyarrow==23.0.0
|
||||
Pillow
|
||||
swanlab==0.6.12
|
||||
# torch==2.6.0
|
||||
# torchvision==0.21.0
|
||||
|
|
@ -154,7 +154,7 @@ def launch_gradio_server(server_name="0.0.0.0", server_port=7788):
|
|||
if __name__ == '__main__':
|
||||
parser = argparse.ArgumentParser(description="Chat with MiniMind-V")
|
||||
parser.add_argument('--load_from', default='./', type=str, help="transformers模型扫描目录")
|
||||
parser.add_argument('--vision_model', default='../model/siglip2-base-p32-256-ve', type=str, help="视觉编码器路径")
|
||||
parser.add_argument('--vision_model', default='./model/siglip2-base-p32-256-ve', type=str, help="视觉编码器路径")
|
||||
parser.add_argument('--temperature', default=0.7, type=float, help="生成温度")
|
||||
parser.add_argument('--top_p', default=0.95, type=float, help="nucleus采样阈值")
|
||||
parser.add_argument('--device', default='cuda' if torch.cuda.is_available() else 'cpu', type=str, help="运行设备")
|
||||
|
|
|
|||
|
|
@ -95,7 +95,7 @@ if __name__ == "__main__":
|
|||
parser.add_argument("--learning_rate", type=float, default=4e-4, help="初始学习率")
|
||||
parser.add_argument("--device", type=str, default="cuda:0" if torch.cuda.is_available() else "cpu", help="训练设备")
|
||||
parser.add_argument("--dtype", type=str, default="bfloat16", help="混合精度类型")
|
||||
parser.add_argument("--num_workers", type=int, default=2, help="数据加载线程数")
|
||||
parser.add_argument("--num_workers", type=int, default=0 if sys.platform == 'win32' else 8, help="数据加载线程数")
|
||||
parser.add_argument("--accumulation_steps", type=int, default=1, help="梯度累积步数")
|
||||
parser.add_argument("--grad_clip", type=float, default=1.0, help="梯度裁剪阈值")
|
||||
parser.add_argument("--log_interval", type=int, default=100, help="日志打印间隔")
|
||||
|
|
|
|||
|
|
@ -95,7 +95,7 @@ if __name__ == "__main__":
|
|||
parser.add_argument("--learning_rate", type=float, default=5e-6, help="初始学习率")
|
||||
parser.add_argument("--device", type=str, default="cuda:0" if torch.cuda.is_available() else "cpu", help="训练设备")
|
||||
parser.add_argument("--dtype", type=str, default="bfloat16", help="混合精度类型")
|
||||
parser.add_argument("--num_workers", type=int, default=2, help="数据加载线程数")
|
||||
parser.add_argument("--num_workers", type=int, default=0 if sys.platform == 'win32' else 8, help="数据加载线程数")
|
||||
parser.add_argument("--accumulation_steps", type=int, default=1, help="梯度累积步数")
|
||||
parser.add_argument("--grad_clip", type=float, default=1.0, help="梯度裁剪阈值")
|
||||
parser.add_argument("--log_interval", type=int, default=100, help="日志打印间隔")
|
||||
|
|
|
|||
Loading…
Reference in New Issue