diff --git a/README.md b/README.md
index 09a6c7e..21ef842 100644
--- a/README.md
+++ b/README.md
@@ -75,6 +75,8 @@
- bug修复:模型权重不对应
- 适配[「minimind-1024更新」](https://github.com/jingyaogong/minimind)
+- 代码重构:训练和评估脚本规范化
+- 新增完整的断点续训支持
@@ -159,8 +161,11 @@ git clone https://huggingface.co/jingyaogong/MiniMind2-V
### 3.命令行问答
```bash
-# load=0: load from pytorch model, load=1: load from transformers-hf model
-python eval_vlm.py --load 1
+# load_from='model': 加载原生PyTorch权重, load_from='其他路径': 加载transformers格式
+python eval_vlm.py --load_from model --weight sft_vlm
+
+# 或使用transformers格式模型
+python eval_vlm.py --load_from MiniMind2-V
```
### 4.或启动WebUI
@@ -223,7 +228,8 @@ unzip sft_images.zip && rm sft_images.zip
**3.1 预训练(学图像描述)**
```bash
-python train_pretrain_vlm.py --epochs 4
+# 基础训练命令(从LLM权重开始,仅训练vision_proj)
+python trainer/train_pretrain_vlm.py --epochs 4 --from_weight llm
```
> 执行预训练,得到 `pretrain_vlm_*.pth` 作为预训练的输出权重(其中*为模型的dimension,默认为512)
@@ -232,7 +238,8 @@ python train_pretrain_vlm.py --epochs 4
**3.2 监督微调(学看图对话方式)**
```bash
-python train_sft_vlm.py --epochs 4
+# 基础训练命令(从预训练权重开始,全参数微调)
+python trainer/train_sft_vlm.py --epochs 2 --from_weight pretrain_vlm
```
> 执行监督微调,得到 `sft_vlm_*.pth` 作为指令微调的输出权重
@@ -240,7 +247,23 @@ python train_sft_vlm.py --epochs 4
注:训练须知
-所有训练过程默认每隔100步保存1次参数到文件`./out/***.pth`(每次会覆盖掉旧权重文件)。
+**训练特性:**
+- 支持断点续训:添加`--from_resume 1`参数可从上次中断处继续训练
+- 支持GPU数量变化:续训时GPU数量改变会自动转换step
+- 原子性保存:使用临时文件+替换机制,防止保存过程中断导致权重损坏
+- 每次保存同时生成`out/**.pth`(模型权重)和`checkpoints/**_resume.pth`(训练状态)文件
+
+```bash
+# 训练中断后,使用相同命令并添加 --from_resume 1
+python trainer/train_sft_vlm.py --epochs 4 --from_resume 1
+```
+
+**参数说明:**
+- `--from_weight`: 基础权重名称(llm, pretrain_vlm, none等)
+- `--save_weight`: 保存权重的前缀名
+- `--from_resume`: 是否续训(0=从头开始,1=从检查点继续)
+- `--freeze_llm`: 是否冻结LLM参数(仅pretrain使用)
+- 更多可直接参考代码
@@ -253,7 +276,11 @@ python train_sft_vlm.py --epochs 4
也可以直接去[此处](https://huggingface.co/jingyaogong/MiniMind2-V-PyTorch)下载使用我训练的`*.pth`文件。
```bash
-python eval_vlm.py --model_mode 1 # 默认为0:测试pretrain模型效果,设置为1:测试sft模型效果
+# 测试SFT模型(默认)
+python eval_vlm.py --weight sft_vlm
+
+# 测试Pretrain模型
+python eval_vlm.py --weight pretrain_vlm
```
---
@@ -270,11 +297,13 @@ torchrun --nproc_per_node N train_xxx.py
注:其它须知
+
单机N卡启动训练 (DeepSpeed)
```bash
deepspeed --master_port 29500 --num_gpus=N train_xxx.py
```
+
可根据需要开启wandb记录训练过程
diff --git a/README_en.md b/README_en.md
index 499556f..ce68945 100644
--- a/README_en.md
+++ b/README_en.md
@@ -80,6 +80,8 @@ Is the training process difficult? Now, let's explore the answers and feel the j
- Bug fix: model weights mismatch
- Adapted to ["minimind-1024 update"](https://github.com/jingyaogong/minimind)
+- Code refactoring: training and evaluation scripts standardized
+- Added complete checkpoint resumption support
@@ -165,8 +167,11 @@ git clone https://huggingface.co/jingyaogong/MiniMind2-V
### 3. Command-line Q&A
```bash
-# load=0: load from pytorch model, load=1: load from transformers-hf model
-python eval_vlm.py --load 1
+# load_from='model': load native PyTorch weights, load_from='other path': load transformers format
+python eval_vlm.py --load_from model --weight sft_vlm
+
+# Or use transformers format model
+python eval_vlm.py --load_from MiniMind2-V
```
### 4. Or start the WebUI
@@ -231,7 +236,8 @@ Please reserve about 5GB of space for the dataset. If there is insufficient spac
**3.1 Pretraining (Learning image description)**
```bash
-python train_pretrain_vlm.py --epochs 4
+# Basic training command (start from LLM weights, train vision_proj only)
+python trainer/train_pretrain_vlm.py --epochs 4 --from_weight llm
```
> Run pretraining to get `pretrain_vlm_*.pth` as the pretrained model's output weights (* represents the model
@@ -240,7 +246,8 @@ python train_pretrain_vlm.py --epochs 4
**3.2 Supervised Fine-Tuning (Learning image-caption dialogue style)**
```bash
-python train_sft_vlm.py --epochs 4
+# Basic training command (start from pretrain weights, full parameter fine-tuning)
+python trainer/train_sft_vlm.py --epochs 2 --from_weight pretrain_vlm
```
> Perform supervised fine-tuning to get `sft_vlm_*.pth` as the output weights for the fine-tuned model.
@@ -248,8 +255,23 @@ python train_sft_vlm.py --epochs 4
Note: Training Details
-By default, the training process saves model parameters every 100 steps to the `./out/***.pth` file (it will overwrite
-previous weight files).
+**Training Features:**
+- Support checkpoint resumption: add `--from_resume 1` parameter to continue from last interruption
+- Support GPU count changes: automatically convert steps when GPU count changes during resumption
+- Atomic saving: use temporary file + replacement mechanism to prevent weight corruption from interruption
+- Each save generates `out/**.pth` (model weights) and `checkpoints/**_resume.pth` (training state) files
+
+```bash
+# To resume training after interruption, use the same command and add --from_resume 1
+python trainer/train_sft_vlm.py --epochs 4 --from_resume 1
+```
+
+**Parameter Description:**
+- `--from_weight`: base weight name (llm, pretrain_vlm, none, etc.)
+- `--save_weight`: save weight prefix name
+- `--from_resume`: whether to resume training (0=start from scratch, 1=continue from checkpoint)
+- `--freeze_llm`: whether to freeze LLM parameters (pretrain use only)
+- More details can be found in the code
@@ -262,7 +284,11 @@ You can also directly download the pre-trained `*.pth` file
from [here](https://huggingface.co/jingyaogong/MiniMind2-V-PyTorch).
```bash
-python eval_vlm.py --model_mode 1 # Default is 0: test pretrain model, set to 1: test sft model
+# Test SFT model (default)
+python eval_vlm.py --weight sft_vlm
+
+# Test Pretrain model
+python eval_vlm.py --weight pretrain_vlm
```
---