diff --git a/README.md b/README.md index 901f34c..27810ab 100644 --- a/README.md +++ b/README.md @@ -141,139 +141,130 @@ * GPU: NVIDIA GeForce RTX 3090(24GB) * 8 * Ubuntu==20.04 * CUDA==12.2 -* Python==3.10.16 +* Python==3.10 * [requirements.txt](./requirements.txt) -### 第0步 - -```bash -# 克隆代码仓库 -git clone --depth 1 https://github.com/jingyaogong/minimind-v -``` - -```bash -# 下载siglip2模型到 ./model 目录下 -git clone https://huggingface.co/jingyaogong/siglip2-base-p32-256-ve -# 或 -git clone https://modelscope.cn/models/gongjy/siglip2-base-p32-256-ve -``` - -```bash -# 下载minimind语言模型权重到 ./out 目录下（作为训练VLM的基座语言模型） -# HuggingFace -https://huggingface.co/jingyaogong/minimind-3v-pytorch/blob/main/llm_768.pth -# 国内源 -https://modelscope.cn/models/gongjy/minimind-3v-pytorch/resolve/master/llm_768.pth -``` - -## Ⅰ 测试已有模型效果 +## 第0步（必须） ### 1' 环境准备 ```bash +# 克隆仓库代码 +git clone --depth 1 https://github.com/jingyaogong/minimind-v +# 安装必要依赖 pip install -r requirements.txt -i https://pypi.tuna.tsinghua.edu.cn/simple ``` -### 2' 下载模型 +### 2' 下载资源 ```bash -git clone https://huggingface.co/jingyaogong/minimind-3v +# 下载 SigLIP2 视觉编码器到 ./model/siglip2-base-p32-256-ve +modelscope download --model gongjy/siglip2-base-p32-256-ve --local_dir ./model/siglip2-base-p32-256-ve +# 下载 MiniMind 语言模型权重到 ./out 目录下（作为训练 VLM 的基座语言模型） +modelscope download --model gongjy/minimind-3v-pytorch llm_768.pth --local_dir ./out ``` -### 3' 命令行问答 +注：也可从 [ModelScope Collection](https://modelscope.cn/collections/gongjy/MiniMind-V) 或 [HuggingFace Collection](https://huggingface.co/collections/jingyaogong/minimind-v-67000833fb60b3a2e1f3597d) 选择对应内容 `git clone`（需 LFS）下载，此处不再赘述。 + +完成后，结构应如下： + +```text +minimind-v/ +├── model/ +│ ├── siglip2-base-p32-256-ve/ +│ └── ... +├── out/ +│ └── llm_768.pth +└── ... +``` + +## Ⅰ 🚀 模型推理 + +### 1' 下载发布权重 + +```bash +# 下载发布权重到 ./out 目录下 +modelscope download --model gongjy/minimind-3v-pytorch --local_dir ./out +``` + +### 2' 命令行问答 ```bash # load_from='model': 加载原生PyTorch权重, load_from='其他路径': 加载transformers格式 python eval_vlm.py --load_from model --weight sft_vlm +``` -# 或使用transformers格式模型 +如果使用 transformers 格式模型，可先下载模型目录： + +```bash +git clone https://huggingface.co/jingyaogong/minimind-3v python eval_vlm.py --load_from minimind-3v ``` -### 4' 启动WebUI（可选） +### 3' 启动 WebUI（可选） ```bash -# ⚠️ 须先将 transformers 格式模型文件夹复制到 ./scripts/ 目录下（例如：cp -r minimind-3v ./scripts/minimind-3v），web_demo_vlm 脚本会自动扫描该目录下包含权重文件的子文件夹，如不存在则报错 +# ⚠️ 须先将 transformers 格式模型文件夹复制到 ./scripts/ 目录下，web_demo_vlm 脚本会自动扫描该目录下包含权重文件的子文件夹，如不存在则报错 +cp -r minimind-3v ./scripts/minimind-3v cd scripts && python web_demo_vlm.py ``` -## Ⅱ 从0开始自己训练 +## Ⅱ 🛠️ 模型训练 -### 1' 环境准备 - -```bash -pip install -r requirements.txt -i https://pypi.tuna.tsinghua.edu.cn/simple -``` - -
+
注：提前测试Torch是否可用cuda -```bash +```python import torch print(torch.cuda.is_available()) ``` -如果不可用，请自行去[torch_stable](https://download.pytorch.org/whl/torch_stable.html) -下载whl文件安装。参考[链接](https://blog.csdn.net/weixin_45456738/article/details/141029610?ops_request_misc=&request_id=&biz_id=102&utm_term=%E5%AE%89%E8%A3%85torch&utm_medium=distribute.pc_search_result.none-task-blog-2~all~sobaiduweb~default-2-141029610.nonecase&spm=1018.2226.3001.4187) +如果不可用，请自行去 [torch_stable](https://download.pytorch.org/whl/torch_stable.html) 下载 whl 文件安装。
-### 2' 数据下载 +### 1' 下载数据 -从下文提供的[数据集链接](https://huggingface.co/datasets/jingyaogong/minimind-v_dataset) -下载所需内容并放到`./dataset`下。 +快速开始时，直接从[数据集链接](https://huggingface.co/datasets/jingyaogong/minimind-v_dataset)下载 `sft_i2t.parquet`，并放到 `./dataset` 下即可。 -
+
注：数据集须知 -【注1】之前需解压50万零碎的图像文件可能非常慢。2025-12-27起，数据集格式统一为Parquet，图文一体化存储，体积更小，无需解压，加载更快。 +【注1】之前需解压50万零碎的图像文件可能非常慢。2025-12-27起，数据集格式统一为 Parquet，图文一体化存储，体积更小，无需解压，加载更快。 -【注2】Parquet是列式存储格式，支持高效压缩和快速读取。如果你对它感到陌生，可以预览数据内容，在`dataset/`目录下执行`python lm_dataset.py`可视化前5条图文对 +【注2】Parquet 是列式存储格式，支持高效压缩和快速读取。如果你对它感到陌生，可以预览数据内容，在 `dataset/` 目录下执行 `python lm_dataset.py` 可视化前5条图文对。 Pretrain 数据（可选；仅包含 caption 子集）： ```bash -wget https://hf-mirror.com/datasets/jingyaogong/minimind-v_dataset/resolve/main/pretrain_i2t.parquet +wget https://hf-mirror.com/datasets/jingyaogong/minimind-v_dataset/resolve/main/pretrain_i2t.parquet -P ./dataset ``` -SFT 数据（必须；包含 caption + instruct + 纯文本全量合并）： -```bash -wget https://hf-mirror.com/datasets/jingyaogong/minimind-v_dataset/resolve/main/sft_i2t.parquet -``` - -SFT 单文件 290 万条已把 Pretrain 作为子集合并，经全局 dictionary encoding 去重后体积只比 SFT 原版多 ~10%，可覆盖所有训练阶段。 +SFT 单文件 290 万条已把 Pretrain 作为子集合并，经全局 dictionary encoding 去重后体积只比 SFT 原版多 ~10%，可覆盖所有训练阶段。因此快速复现时可以跳过 Pretrain，直接进入 SFT。
-### 3' 开始训练 +### 2' 开始训练 -**3.1 Pretrain（可选）** - -> SFT 数据集 `sft_i2t.parquet` 已把 Pretrain 数据作为子集合并进来，**此阶段可跳过**，直接 `--from_weight llm` 进入 SFT 即可；若想让 Projector 提前完成对齐、SFT 收敛更稳，可以提前执行 Pretrain。 +推荐直接执行 SFT。默认 `--freeze_llm 1`，即训练 `vision_proj` 和 LLM 首尾层，保留中间层原有语言能力： ```bash -# 基础训练命令（从LLM权重开始，仅训练 vision_proj） -python train_pretrain_vlm.py --epochs 2 --from_weight llm +python train_sft_vlm.py --epochs 2 --from_weight llm ``` -> 执行 Pretrain，得到 `pretrain_vlm_*.pth` 作为 Pretrain 的输出权重（其中*为模型的dimension，默认为768） - - -**3.2 SFT（必须）** +如果希望让 Projector 先完成一轮图文对齐，再进入 SFT，可额外执行 Pretrain： ```bash -# 基础训练命令（默认 --freeze_llm 1：解冻 proj + LLM 首尾层，保留中间层原 LLM 知识） -# 若已执行 3.1：--from_weight pretrain_vlm；若跳过 Pretrain：--from_weight llm +python train_pretrain_vlm.py --epochs 2 --from_weight llm python train_sft_vlm.py --epochs 2 --from_weight pretrain_vlm ``` -> 执行 SFT，得到 `sft_vlm_*.pth` 作为 SFT 的输出权重 +执行完成后，`out/` 下会生成 `sft_vlm_*.pth` 作为 SFT 权重。 -
+
注：训练须知 -**训练特性：** - 支持断点续训：添加`--from_resume 1`参数可从上次中断处继续训练 - 支持GPU数量变化：续训时GPU数量改变会自动转换step - 原子性保存：使用临时文件+替换机制，防止保存过程中断导致权重损坏 @@ -293,12 +284,9 @@ python train_sft_vlm.py --epochs 4 --from_resume 1
+### 3' 测试已训练模型（可选） ---- - -### 4' 测试模型效果 - -确保需要测试的模型`*.pth`文件位于`./out/`目录下。 +确保需要测试的模型 `*.pth` 文件位于 `./out/` 目录下。 也可以直接去[此处](https://huggingface.co/jingyaogong/minimind-3v-pytorch)下载使用我训练的`*.pth`文件。 ```bash @@ -312,7 +300,7 @@ python eval_vlm.py --weight pretrain_vlm --- > [!TIP] -> 训练脚本均为 PyTorch 原生框架，均支持多卡加速，假设你的设备有N (N＞1) 张显卡： +> 训练脚本均为 PyTorch 原生框架，均支持多卡加速，假设你的设备有 N (N＞1) 张显卡： 单机N卡启动训练方式 (DDP, 支持多机多卡集群) diff --git a/README_en.md b/README_en.md index e32ca44..40fe602 100644 --- a/README_en.md +++ b/README_en.md @@ -136,148 +136,137 @@ Is the training process difficult? Now, let's explore the answers and feel the j # 📌 Quick Start
-Sharing my hardware and software configuration (for reference only) +My software and hardware setup (for reference only) * CPU: Intel(R) Core(TM) i9-10980XE CPU @ 3.00GHz * RAM: 128 GB * GPU: NVIDIA GeForce RTX 3090(24GB) * 8 * Ubuntu==20.04 * CUDA==12.2 -* Python==3.10.16 +* Python==3.10 * [requirements.txt](./requirements.txt)
-### Step 0 +## Step 0 (required) + +### 1' Environment setup ```bash -# Clone the code repository +# Clone the repository git clone --depth 1 https://github.com/jingyaogong/minimind-v +# Install dependencies +pip install -r requirements.txt -i https://pypi.tuna.tsinghua.edu.cn/simple ``` +### 2' Download resources + ```bash -# Download the siglip2 model to the ./model directory -git clone https://huggingface.co/jingyaogong/siglip2-base-p32-256-ve -# or -git clone https://modelscope.cn/models/gongjy/siglip2-base-p32-256-ve +# Download the SigLIP2 vision encoder to ./model/siglip2-base-p32-256-ve +modelscope download --model gongjy/siglip2-base-p32-256-ve --local_dir ./model/siglip2-base-p32-256-ve +# Download the MiniMind language model weight to ./out (used as the base language model for VLM training) +modelscope download --model gongjy/minimind-3v-pytorch llm_768.pth --local_dir ./out ``` +Alternatively, the same files can be selected from the [ModelScope Collection](https://modelscope.cn/collections/gongjy/MiniMind-V) or [HuggingFace Collection](https://huggingface.co/collections/jingyaogong/minimind-v-67000833fb60b3a2e1f3597d) and downloaded with `git clone` (LFS required). + +The directory should look like this after the resources are ready: + +```text +minimind-v/ +├── model/ +│ ├── siglip2-base-p32-256-ve/ +│ └── ... +├── out/ +│ └── llm_768.pth +└── ... +``` + +## Ⅰ 🚀 Model inference + +### 1' Download released weights + ```bash -# Download the minimind language model to the ./out directory (as the base language model for training VLM): -# HuggingFace -https://huggingface.co/jingyaogong/minimind-3v-pytorch/blob/main/llm_768.pth -# Domestic source -https://modelscope.cn/models/gongjy/minimind-3v-pytorch/resolve/master/llm_768.pth +# Download released weights to ./out +modelscope download --model gongjy/minimind-3v-pytorch --local_dir ./out ``` - -## Ⅰ Test an existing model's performance - -### 1' Environment Preparation - -```bash -pip install -r requirements.txt -i https://mirrors.aliyun.com/pypi/simple -``` - -### 2' Download the model - -```bash -git clone https://huggingface.co/jingyaogong/minimind-3v -``` - -### 3' Command-line Q&A +### 2' Command-line Q&A ```bash # load_from='model': load native PyTorch weights, load_from='other path': load transformers format python eval_vlm.py --load_from model --weight sft_vlm +``` -# Or use transformers format model +If using a transformers-format model, download the model directory first: + +```bash +git clone https://huggingface.co/jingyaogong/minimind-3v python eval_vlm.py --load_from minimind-3v ``` -### 4' Or start the WebUI +### 3' Start WebUI (optional) ```bash -# ⚠️ You must first copy the transformers model folder to the ./scripts/ directory (e.g.: cp -r minimind-3v ./scripts/minimind-3v). The web_demo_vlm script will automatically scan subdirectories containing weight files; it will report an error if none are found. +# ⚠️ The transformers-format model directory must be copied to ./scripts/ first. web_demo_vlm scans subdirectories under ./scripts/ that contain weight files and reports an error if none are found. +cp -r minimind-3v ./scripts/minimind-3v cd scripts && python web_demo_vlm.py ``` -## Ⅱ Train from scratch +## Ⅱ 🛠️ Model training -### 1' Environment Preparation +
+Note: test whether Torch can use CUDA -```bash -pip install -r requirements.txt -i https://mirrors.aliyun.com/pypi/simple -``` - -
-Note: Test if Torch can use CUDA - -```bash +```python import torch print(torch.cuda.is_available()) ``` -If unavailable, download the whl file from [torch_stable](https://download.pytorch.org/whl/torch_stable.html) for -installation. Refer -to [this link](https://blog.csdn.net/weixin_45456738/article/details/141029610?ops_request_misc=&request_id=&biz_id=102&utm_term=%E5%AE%89%E8%A3%85torch&utm_medium=distribute.pc_search_result.none-task-blog-2~all~sobaiduweb~default-2-141029610.nonecase&spm=1018.2226.3001.4187) -for help. +If CUDA is unavailable, download the matching whl file from [torch_stable](https://download.pytorch.org/whl/torch_stable.html) and install it manually.
-### 2' Download Data +### 1' Download data -Download the required content from the [dataset link](https://huggingface.co/datasets/jingyaogong/minimind-v_dataset) -and place it under `./dataset`. +For a quick start, download `sft_i2t.parquet` from the [dataset link](https://huggingface.co/datasets/jingyaogong/minimind-v_dataset) and place it under `./dataset`. -
-Note: Dataset Details +
+Note: dataset details -**[Note 1]** Previously, extracting 500k fragmented image files could be very slow. From 2025-12-27, dataset format is unified to Parquet with image-text integrated storage, smaller size, no decompression needed, faster loading. +**[Note 1]** The older dataset required extracting 500k fragmented image files, which could be very slow. Since 2025-12-27, the dataset has been unified into Parquet with image and text stored together. It is smaller, requires no decompression, and loads faster. -**[Note 2]** Parquet is a columnar storage format supporting efficient compression and fast reading. To preview data content, run `python lm_dataset.py` in the `dataset/` directory to visualize the first 5 image-text pairs. +**[Note 2]** Parquet is a columnar storage format with efficient compression and fast reading. If it is unfamiliar, run `python lm_dataset.py` under `dataset/` to visualize the first 5 image-text pairs. Pretrain data (optional; contains caption subset only): ```bash -wget https://hf-mirror.com/datasets/jingyaogong/minimind-v_dataset/resolve/main/pretrain_i2t.parquet +wget https://hf-mirror.com/datasets/jingyaogong/minimind-v_dataset/resolve/main/pretrain_i2t.parquet -P ./dataset ``` -SFT data (required; contains full caption + instruct + text merge): -```bash -wget https://hf-mirror.com/datasets/jingyaogong/minimind-v_dataset/resolve/main/sft_i2t.parquet -``` - -The single `sft_i2t.parquet` file (2.9M rows) absorbs Pretrain as a subset, and after global dictionary encoding dedup it is only ~10% larger than the original SFT — enough to cover every training stage. +The single `sft_i2t.parquet` file contains 2.9M rows and absorbs Pretrain as a subset. After global dictionary encoding deduplication, it is only ~10% larger than the original SFT file and is enough to cover every training stage. For quick reproduction, Pretrain can be skipped and SFT can be started directly.
-### 3' Start Training +### 2' Start training -**3.1 Pretrain (optional)** - -> `sft_i2t.parquet` already contains the Pretrain data as a subset, so **this stage can be skipped** and you may go directly to SFT with `--from_weight llm`. If you prefer the Projector to be pre-aligned first and SFT to converge more stably, run a round of Pretrain separately. +SFT is the recommended starting point. By default, `--freeze_llm 1` trains `vision_proj` and the first/last LLM layers while keeping the middle layers' original language ability: ```bash -# Basic training command (start from LLM weights, train vision_proj only) -python train_pretrain_vlm.py --epochs 2 --from_weight llm +python train_sft_vlm.py --epochs 2 --from_weight llm ``` -> Run Pretrain to get `pretrain_vlm_*.pth` as the Pretrain output weights (* is the model dimension, default 768). - -**3.2 SFT (required)** +If the Projector should be aligned on image-text pairs before SFT, run Pretrain first: ```bash -# Basic command (default --freeze_llm 1: unfreeze proj + first/last LLM layers, keep middle layers' original LLM weights) -# If 3.1 was run: --from_weight pretrain_vlm; if Pretrain was skipped: --from_weight llm +python train_pretrain_vlm.py --epochs 2 --from_weight llm python train_sft_vlm.py --epochs 2 --from_weight pretrain_vlm ``` -> Run SFT to get `sft_vlm_*.pth` as the SFT output weights. +After training, `sft_vlm_*.pth` will be written under `out/` as the SFT weight. -
-Note: Training Details +
+Note: training details -**Training Features:** - Support checkpoint resumption: add `--from_resume 1` parameter to continue from last interruption - Support GPU count changes: automatically convert steps when GPU count changes during resumption - Atomic saving: use temporary file + replacement mechanism to prevent weight corruption from interruption @@ -297,9 +286,7 @@ python train_sft_vlm.py --epochs 4 --from_resume 1
---- - -### 4' Test the Model's Performance +### 3' Test a trained model (optional) Ensure that the model `*.pth` file you want to test is located in the `./out/` directory. You can also directly download the pre-trained `*.pth` file