diff --git a/README.md b/README.md index 09a6c7e..21ef842 100644 --- a/README.md +++ b/README.md @@ -75,6 +75,8 @@ - bug修复:模型权重不对应 - 适配[「minimind-1024更新」](https://github.com/jingyaogong/minimind) +- 代码重构:训练和评估脚本规范化 +- 新增完整的断点续训支持 @@ -159,8 +161,11 @@ git clone https://huggingface.co/jingyaogong/MiniMind2-V ### 3.命令行问答 ```bash -# load=0: load from pytorch model, load=1: load from transformers-hf model -python eval_vlm.py --load 1 +# load_from='model': 加载原生PyTorch权重, load_from='其他路径': 加载transformers格式 +python eval_vlm.py --load_from model --weight sft_vlm + +# 或使用transformers格式模型 +python eval_vlm.py --load_from MiniMind2-V ``` ### 4.或启动WebUI @@ -223,7 +228,8 @@ unzip sft_images.zip && rm sft_images.zip **3.1 预训练(学图像描述)** ```bash -python train_pretrain_vlm.py --epochs 4 +# 基础训练命令(从LLM权重开始,仅训练vision_proj) +python trainer/train_pretrain_vlm.py --epochs 4 --from_weight llm ``` > 执行预训练,得到 `pretrain_vlm_*.pth` 作为预训练的输出权重(其中*为模型的dimension,默认为512) @@ -232,7 +238,8 @@ python train_pretrain_vlm.py --epochs 4 **3.2 监督微调(学看图对话方式)** ```bash -python train_sft_vlm.py --epochs 4 +# 基础训练命令(从预训练权重开始,全参数微调) +python trainer/train_sft_vlm.py --epochs 2 --from_weight pretrain_vlm ``` > 执行监督微调,得到 `sft_vlm_*.pth` 作为指令微调的输出权重 @@ -240,7 +247,23 @@ python train_sft_vlm.py --epochs 4
注:训练须知 -所有训练过程默认每隔100步保存1次参数到文件`./out/***.pth`(每次会覆盖掉旧权重文件)。 +**训练特性:** +- 支持断点续训:添加`--from_resume 1`参数可从上次中断处继续训练 +- 支持GPU数量变化:续训时GPU数量改变会自动转换step +- 原子性保存:使用临时文件+替换机制,防止保存过程中断导致权重损坏 +- 每次保存同时生成`out/**.pth`(模型权重)和`checkpoints/**_resume.pth`(训练状态)文件 + +```bash +# 训练中断后,使用相同命令并添加 --from_resume 1 +python trainer/train_sft_vlm.py --epochs 4 --from_resume 1 +``` + +**参数说明:** +- `--from_weight`: 基础权重名称(llm, pretrain_vlm, none等) +- `--save_weight`: 保存权重的前缀名 +- `--from_resume`: 是否续训(0=从头开始,1=从检查点继续) +- `--freeze_llm`: 是否冻结LLM参数(仅pretrain使用) +- 更多可直接参考代码
@@ -253,7 +276,11 @@ python train_sft_vlm.py --epochs 4 也可以直接去[此处](https://huggingface.co/jingyaogong/MiniMind2-V-PyTorch)下载使用我训练的`*.pth`文件。 ```bash -python eval_vlm.py --model_mode 1 # 默认为0:测试pretrain模型效果,设置为1:测试sft模型效果 +# 测试SFT模型(默认) +python eval_vlm.py --weight sft_vlm + +# 测试Pretrain模型 +python eval_vlm.py --weight pretrain_vlm ``` --- @@ -270,11 +297,13 @@ torchrun --nproc_per_node N train_xxx.py
注:其它须知 + 单机N卡启动训练 (DeepSpeed) ```bash deepspeed --master_port 29500 --num_gpus=N train_xxx.py ``` + 可根据需要开启wandb记录训练过程 diff --git a/README_en.md b/README_en.md index 499556f..ce68945 100644 --- a/README_en.md +++ b/README_en.md @@ -80,6 +80,8 @@ Is the training process difficult? Now, let's explore the answers and feel the j - Bug fix: model weights mismatch - Adapted to ["minimind-1024 update"](https://github.com/jingyaogong/minimind) +- Code refactoring: training and evaluation scripts standardized +- Added complete checkpoint resumption support
@@ -165,8 +167,11 @@ git clone https://huggingface.co/jingyaogong/MiniMind2-V ### 3. Command-line Q&A ```bash -# load=0: load from pytorch model, load=1: load from transformers-hf model -python eval_vlm.py --load 1 +# load_from='model': load native PyTorch weights, load_from='other path': load transformers format +python eval_vlm.py --load_from model --weight sft_vlm + +# Or use transformers format model +python eval_vlm.py --load_from MiniMind2-V ``` ### 4. Or start the WebUI @@ -231,7 +236,8 @@ Please reserve about 5GB of space for the dataset. If there is insufficient spac **3.1 Pretraining (Learning image description)** ```bash -python train_pretrain_vlm.py --epochs 4 +# Basic training command (start from LLM weights, train vision_proj only) +python trainer/train_pretrain_vlm.py --epochs 4 --from_weight llm ``` > Run pretraining to get `pretrain_vlm_*.pth` as the pretrained model's output weights (* represents the model @@ -240,7 +246,8 @@ python train_pretrain_vlm.py --epochs 4 **3.2 Supervised Fine-Tuning (Learning image-caption dialogue style)** ```bash -python train_sft_vlm.py --epochs 4 +# Basic training command (start from pretrain weights, full parameter fine-tuning) +python trainer/train_sft_vlm.py --epochs 2 --from_weight pretrain_vlm ``` > Perform supervised fine-tuning to get `sft_vlm_*.pth` as the output weights for the fine-tuned model. @@ -248,8 +255,23 @@ python train_sft_vlm.py --epochs 4
Note: Training Details -By default, the training process saves model parameters every 100 steps to the `./out/***.pth` file (it will overwrite -previous weight files). +**Training Features:** +- Support checkpoint resumption: add `--from_resume 1` parameter to continue from last interruption +- Support GPU count changes: automatically convert steps when GPU count changes during resumption +- Atomic saving: use temporary file + replacement mechanism to prevent weight corruption from interruption +- Each save generates `out/**.pth` (model weights) and `checkpoints/**_resume.pth` (training state) files + +```bash +# To resume training after interruption, use the same command and add --from_resume 1 +python trainer/train_sft_vlm.py --epochs 4 --from_resume 1 +``` + +**Parameter Description:** +- `--from_weight`: base weight name (llm, pretrain_vlm, none, etc.) +- `--save_weight`: save weight prefix name +- `--from_resume`: whether to resume training (0=start from scratch, 1=continue from checkpoint) +- `--freeze_llm`: whether to freeze LLM parameters (pretrain use only) +- More details can be found in the code
@@ -262,7 +284,11 @@ You can also directly download the pre-trained `*.pth` file from [here](https://huggingface.co/jingyaogong/MiniMind2-V-PyTorch). ```bash -python eval_vlm.py --model_mode 1 # Default is 0: test pretrain model, set to 1: test sft model +# Test SFT model (default) +python eval_vlm.py --weight sft_vlm + +# Test Pretrain model +python eval_vlm.py --weight pretrain_vlm ``` ---