github
/
minimind-v
mirror of https://github.com/jingyaogong/minimind-v.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

update readme

pull/19/head
gongjy 2024-10-05 10:47:21 +08:00
parent f8e868cbf2
commit be94af9e2c
2 changed files with 38 additions and 41 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

8
README.md
View File

@ -129,15 +129,15 @@ CPU: Intel(R) Core(TM) i9-10980XE CPU @ 3.00GHz
# 📌 Quick Start Train # 📌 Quick Start Train
* 0、环境安装 * 0.环境安装
```bash ```bash
pip install -r requirements.txt -i https://pypi.tuna.tsinghua.edu.cn/simple pip install -r requirements.txt -i https://pypi.tuna.tsinghua.edu.cn/simple
``` ```
* 1、克隆项目代码 * 1.克隆项目代码
```text ```text
git clone https://github.com/jingyaogong/minimind-v git clone https://github.com/jingyaogong/minimind-v
``` ```
* 2、如果需要自己训练 * 2.如果需要自己训练
* 2.1 下载数据集 ([百度网盘](https://pan.baidu.com/s/1Nz36OBBvVBGEx-PwIb7ofg?pwd=6666) or [HuggingFace](https://hf-mirror.com/datasets/jingyaogong/minimind-v_dataset))文件到`./dataset` * 2.1 下载数据集 ([百度网盘](https://pan.baidu.com/s/1Nz36OBBvVBGEx-PwIb7ofg?pwd=6666) or [HuggingFace](https://hf-mirror.com/datasets/jingyaogong/minimind-v_dataset))文件到`./dataset`
目录下，并解压`pretrain_images.zip` 和 `sft_images.zip` 目录下，并解压`pretrain_images.zip` 和 `sft_images.zip`
@ -148,7 +148,7 @@ CPU: Intel(R) Core(TM) i9-10980XE CPU @ 3.00GHz
* 2.4 `python 1-pretrain_vlm.py` 执行预训练，得到 `*_vlm_pretrain.pth` 作为预训练的输出权重 * 2.4 `python 1-pretrain_vlm.py` 执行预训练，得到 `*_vlm_pretrain.pth` 作为预训练的输出权重
* 2.5 `python 2-sft_vlm.py` 执行指令微调，得到 `*_vlm_sft.pth` 作为指令微调的输出权重 * 2.5 `python 2-sft_vlm.py` 执行指令微调，得到 `*_vlm_sft.pth` 作为指令微调的输出权重
* 3、测试自己训练的模型推理效果 * 3.测试自己训练的模型推理效果
* 确保需要使用的，训练完成的参数权重`*.pth`文件位于`./out/`目录下 * 确保需要使用的，训练完成的参数权重`*.pth`文件位于`./out/`目录下
* 也可以直接去[训练完成的模型权重](https://pan.baidu.com/s/1LE1SPoPYGS7VNtT1tpf7DA?pwd=6666) * 也可以直接去[训练完成的模型权重](https://pan.baidu.com/s/1LE1SPoPYGS7VNtT1tpf7DA?pwd=6666)
下载使用我训练好的`*.pth`权重文件 下载使用我训练好的`*.pth`权重文件

71
README_en.md
View File

@ -39,7 +39,7 @@
![](./images/modelscope-demo.gif) ![modelscope](./images/modelscope-demo.gif)
<div align="center"> <div align="center">
@ -98,35 +98,32 @@ Environment: python 3.9 + Torch 2.1.2 + DDP single-machine multi-GPU training
# 📌 Quick Test # 📌 Quick Test
1. Clone the project * 0. Clone the project
```bash
git clone https://github.com/jingyaogong/minimind-v & cd minimind-v
```
```bash * 1. Install the environment
# step 1 ```bash
git clone https://github.com/jingyaogong/minimind-v & cd minimind-v pip install -r requirements.txt -i https://pypi.tuna.tsinghua.edu.cn/simple
``` ```
* 2. Download the pre-trained model weights to the project root directory `minimind-v-v1`
```bash
git clone https://huggingface.co/jingyaogong/minimind-v-v1
```
2. Download the pre-trained model weight files to the root directory `minimind-v-v1` of the project * 3. Download the pre-trained `clip-vit-base-patch32` model to the `model/clip_model` directory:
```bash
cd model/clip_model & git clone https://hf-mirror.com/openai/clip-vit-base-patch32
```
```bash * 4. Start the chat web server for testing conversations
# step 2 ```bash
git clone https://huggingface.co/jingyaogong/minimind-v-v1 python web_server.py
``` ```
3. Download the pre-trained `clip-vit-base-patch32` model into the `model/clip_model` directory: ![web_server](images/web_server.gif)
```bash
# step 3
cd model/clip_model & git clone https://hf-mirror.com/openai/clip-vit-base-patch32
```
4. Start the chat web page to test the conversation
```bash
# step 4
python web_server.py
```
[web_server](images/web_server.gif)
# 📌 Quick Start Train # 📌 Quick Start Train
@ -140,7 +137,7 @@ python web_server.py
``` ```
* 2. If you want to train it yourself * 2. If you want to train it yourself
* 2.1 Download all contents of the [dataset](https://pan.baidu.com/s/1Nz36OBBvVBGEx-PwIb7ofg?pwd=6666) to the `./dataset` directory, and unzip `pretrain_images.zip` and `sft_images.zip` * 2.1 Download all contents of the dataset ([Baidu Netdisk](https://pan.baidu.com/s/1Nz36OBBvVBGEx-PwIb7ofg?pwd=6666) or [HuggingFace](https://hf-mirror.com/datasets/jingyaogong/minimind-v_dataset)) to the `./dataset` directory, and unzip `pretrain_images.zip` and `sft_images.zip`
* 2.2 Adjust the model parameters in `./model/LMConfig.py` * 2.2 Adjust the model parameters in `./model/LMConfig.py`
> Only need to adjust the dim and n_layers parameters, which are `(512+8)` or `(768+16)`, corresponding to `minimind-v-v1-small` and `minimind-v-v1` > Only need to adjust the dim and n_layers parameters, which are `(512+8)` or `(768+16)`, corresponding to `minimind-v-v1-small` and `minimind-v-v1`
* 2.3 Download the [pre-trained weight file](https://pan.baidu.com/s/1LE1SPoPYGS7VNtT1tpf7DA?pwd=6666) of the MiniMind language model and place it in the `./out/` directory, named `*_llm.pth` * 2.3 Download the [pre-trained weight file](https://pan.baidu.com/s/1LE1SPoPYGS7VNtT1tpf7DA?pwd=6666) of the MiniMind language model and place it in the `./out/` directory, named `*_llm.pth`
@ -186,8 +183,8 @@ The base language model MiniMind (LLM) for MiniMind-V (VLM) comes from the twin
> PS: Even if you do not wish to delve into the details of MiniMind (LLM), you can directly refer to Quick Test and Quick Start to quickly test or train MiniMind-V. This will not be significantly affected. > PS: Even if you do not wish to delve into the details of MiniMind (LLM), you can directly refer to Quick Test and Quick Start to quickly test or train MiniMind-V. This will not be significantly affected.
The structure of MiniMind-V remains almost unchanged, with only two additional sub-modules added: Visual Encoder and feature projection, as well as a multimodal fusion branch, to support input from multiple modalities: The structure of MiniMind-V remains almost unchanged, with only two additional sub-modules added: Visual Encoder and feature projection, as well as a multimodal fusion branch, to support input from multiple modalities:
[LLM-structure](./images/VLM-structure.png) ![LLM-structure](./images/VLM-structure.png)
[LLM-structure](./images/VLM-structure-moe.png) ![LLM-structure](./images/VLM-structure-moe.png)
At this point, it's interesting to ponder two questions: What is a **L**arge **L**anguage **M**odel (LLM)? And what is a multimodal model? At this point, it's interesting to ponder two questions: What is a **L**arge **L**anguage **M**odel (LLM)? And what is a multimodal model?
@ -246,7 +243,7 @@ After obtaining the image encoder features, on one hand, it is necessary to alig
on the other hand, the image features must be mapped to the same space as the text embeddings, i.e., the text tokens and the native visual tokens need to be aligned and cannot be treated equally; this can be called cross-modal feature alignment. on the other hand, the image features must be mapped to the same space as the text embeddings, i.e., the text tokens and the native visual tokens need to be aligned and cannot be treated equally; this can be called cross-modal feature alignment.
[LlaVA-1](https://arxiv.org/pdf/2304.08485) accomplished this with a simple unbiased linear transformation, and the results were excellent; MiniMind-V does the same. [LlaVA-1](https://arxiv.org/pdf/2304.08485) accomplished this with a simple unbiased linear transformation, and the results were excellent; MiniMind-V does the same.
[llava-structure](./images/llava-structure.png) ![llava-structure](./images/llava-structure.png)
With this, the internal structural changes of MiniMind-V have been presented. With this, the internal structural changes of MiniMind-V have been presented.
@ -274,7 +271,7 @@ Therefore, the prompt for minimind-v:
After computing the embedding and projection, and replacing the image part tokens, After computing the embedding and projection, and replacing the image part tokens,
the entire computation process to output is no different from the LLM part. the entire computation process to output is no different from the LLM part.
[input](./images/minimind-v-input.png) ![input](./images/minimind-v-input.png)
<u>With this, all the details of MiniMind-V have been presented.</u> <u>With this, all the details of MiniMind-V have been presented.</u>
@ -343,13 +340,13 @@ The instruction fine-tuning learns the real Q&A format for asking questions abou
During training, the visual encoder, which is the CLIP model, is frozen, and only the Projection and LLM parts are fine-tuned. During training, the visual encoder, which is the CLIP model, is frozen, and only the Projection and LLM parts are fine-tuned.
> Pretrain 512+8 Model (Training Time and Loss Reference Chart) > Pretrain 512+8 Model (Training Time and Loss Reference Chart)
[input](./images/1-pretrain-512.png) ![input](./images/1-pretrain-512.png)
> Pretrain 768+16 Model (Training Time and Loss Reference Chart) > Pretrain 768+16 Model (Training Time and Loss Reference Chart)
[input](./images/1-pretrain-768.png) ![input](./images/1-pretrain-768.png)
> SFT 512+8 Model (Training Time and Loss Reference Chart) > SFT 512+8 Model (Training Time and Loss Reference Chart)
[input](./images/2-sft-512.png) ![input](./images/2-sft-512.png)
> SFT 768+16 Model (Training Time and Loss Reference Chart) > SFT 768+16 Model (Training Time and Loss Reference Chart)
[input](./images/2-sft-768.png) ![input](./images/2-sft-768.png)
## Trained Model Weights ## Trained Model Weights
@ -459,8 +456,8 @@ Download link: [HuggingFace](https://huggingface.co/collections/jingyaogong/mini
python web_server.py python web_server.py
``` ```
[web_server1](./images/web_server1.png) ![web_server1](./images/web_server1.png)
[web_server2](./images/web_server2.png) ![web_server2](./images/web_server2.png)
### Summary of Effects ### Summary of Effects