commit 5f47204191603c5c52fe3a03f0845110d6f6d4ed Author: jingyaogong Date: Sun Oct 12 23:02:50 2025 +0800 init minimind-v docs diff --git a/.readthedocs.yaml b/.readthedocs.yaml new file mode 100644 index 0000000..0223969 --- /dev/null +++ b/.readthedocs.yaml @@ -0,0 +1,19 @@ +# Read the Docs 配置文件 +version: 2 + +# 构建配置 +build: + os: ubuntu-22.04 + tools: + python: "3.11" + +# MkDocs 配置 +mkdocs: + configuration: mkdocs.yml + fail_on_warning: false + +# Python 依赖 +python: + install: + - requirements: requirements.txt + diff --git a/MIGRATION_SUMMARY.md b/MIGRATION_SUMMARY.md new file mode 100644 index 0000000..3ac8a37 --- /dev/null +++ b/MIGRATION_SUMMARY.md @@ -0,0 +1,76 @@ +# MiniMind-V Documentation Migration Summary + +## Changes Made + +### 1. Configuration Files +- **mkdocs.yml**: Updated site name, description, and URL from "MiniMind" to "MiniMind-V" + +### 2. Images +- **Removed**: All old MiniMind LLM-related images +- **Added**: MiniMind-V VLM-specific images from `/images/`: + - logo.png + - minimind2-v.gif + - VLM-structure.png + - VLM-structure-moe.png + - llava-structure.png + - minimind-v-input.png + - pretrain_loss.png + - sft_loss.png + +### 3. Documentation Files (All in English) + +#### index.md +- Changed from LLM introduction to VLM introduction +- Updated model list to show MiniMind-V variants +- Added VLM-specific concepts and architecture +- Updated related links to minimind-v repository + +#### quickstart.md +- Updated from LLM quick start to VLM quick start +- Added CLIP model download instructions +- Changed evaluation script from `eval_model.py` to `eval_vlm.py` +- Updated model architecture section with Visual Encoder details +- Added image-text dialogue examples + +#### training.md +- Changed from LLM training to VLM training +- Added VLM dataset formats (pretrain_vlm_data, sft_vlm_data, sft_vlm_data_multi) +- Updated training scripts to `train_pretrain_vlm.py` and `train_sft_vlm.py` +- Added Visual Encoder and Projection layer explanations +- Updated training pipeline with VLM-specific steps +- Added multi-image fine-tuning section + +#### README.md +- Updated project description to MiniMind-V +- Changed documentation URL to minimind-v.readthedocs.io +- Updated GitHub links to minimind-v repository + +## Key Differences: MiniMind vs MiniMind-V + +| Aspect | MiniMind (LLM) | MiniMind-V (VLM) | +|--------|----------------|------------------| +| Type | Language Model | Vision-Language Model | +| Input | Text only | Text + Images | +| Components | Transformer Decoder | Transformer Decoder + Visual Encoder + Projection | +| Training Data | Text (pretrain.jsonl, sft.jsonl) | Text + Images (pretrain_vlm_data, sft_vlm_data) | +| Eval Script | eval_model.py | eval_vlm.py | +| Training Scripts | train_pretrain.py, train_full_sft.py | train_pretrain_vlm.py, train_sft_vlm.py | +| Model Weights | *.pth (LLM only) | *.pth (LLM + Visual components) | + +## Verification + +All documentation now correctly references: +- ✅ MiniMind-V project name +- ✅ VLM-specific architecture diagrams +- ✅ Vision-language training pipeline +- ✅ Correct repository URLs (minimind-v) +- ✅ VLM evaluation and inference scripts +- ✅ Image-text dataset formats +- ✅ English language throughout + +## Next Steps + +1. Deploy to ReadTheDocs at minimind-v.readthedocs.io +2. Ensure .readthedocs.yaml is properly configured +3. Test local build with `mkdocs serve` +4. Verify all image links work correctly diff --git a/README.md b/README.md new file mode 100644 index 0000000..f5274b5 --- /dev/null +++ b/README.md @@ -0,0 +1 @@ +### MiniMind-V Docs \ No newline at end of file diff --git a/docs/images/VLM-structure-moe.png b/docs/images/VLM-structure-moe.png new file mode 100644 index 0000000..dfa52a0 Binary files /dev/null and b/docs/images/VLM-structure-moe.png differ diff --git a/docs/images/VLM-structure.png b/docs/images/VLM-structure.png new file mode 100755 index 0000000..944c559 Binary files /dev/null and b/docs/images/VLM-structure.png differ diff --git a/docs/images/llava-structure.png b/docs/images/llava-structure.png new file mode 100644 index 0000000..342d1bb Binary files /dev/null and b/docs/images/llava-structure.png differ diff --git a/docs/images/logo.png b/docs/images/logo.png new file mode 100644 index 0000000..05d40a5 Binary files /dev/null and b/docs/images/logo.png differ diff --git a/docs/images/minimind-v-input.png b/docs/images/minimind-v-input.png new file mode 100644 index 0000000..26386e2 Binary files /dev/null and b/docs/images/minimind-v-input.png differ diff --git a/docs/images/minimind2-v.gif b/docs/images/minimind2-v.gif new file mode 100644 index 0000000..c115799 Binary files /dev/null and b/docs/images/minimind2-v.gif differ diff --git a/docs/images/pretrain_loss.png b/docs/images/pretrain_loss.png new file mode 100644 index 0000000..c1421da Binary files /dev/null and b/docs/images/pretrain_loss.png differ diff --git a/docs/images/sft_loss.png b/docs/images/sft_loss.png new file mode 100644 index 0000000..f815919 Binary files /dev/null and b/docs/images/sft_loss.png differ diff --git a/docs/index.md b/docs/index.md new file mode 100644 index 0000000..58c6bf1 --- /dev/null +++ b/docs/index.md @@ -0,0 +1,90 @@ +# Welcome to MiniMind-V! + +
+ ![logo](images/logo.png) +
"The Greatest Path is the Simplest"
+
+ +## 📌 Introduction + +MiniMind-V is a super-small multimodal vision-language model project trained completely from scratch, requiring **only 1.3 RMB + 1 hour** to train a **26M** parameter vision-language model! + +- **MiniMind-V** series is extremely lightweight, the smallest version is **1/7000** the size of GPT-3 +- **MiniMind-V** is an extension of the visual capabilities of the [MiniMind](https://github.com/jingyaogong/minimind) pure language model +- The project open-sources the minimalist structure of VLM models, including: + - Dataset cleaning + - Pretraining + - Supervised Fine-Tuning (SFT) + - Multi-image dialogue support +- All core algorithm code is reconstructed from scratch using native PyTorch, without relying on third-party abstract interfaces +- This is not only a full-stage open-source reproduction of vision-language models, but also a concise tutorial for getting started with VLMs + +!!! note "Training Cost" + "1 hour" is based on NVIDIA 3090 hardware (single card) testing `1 epoch`, "1.3 RMB" refers to GPU server rental cost + +## ✨ Key Features + +- **Ultra-low cost**: Single 3090, 1 hour, 1.3 RMB to train a vision-language model from scratch +- **Complete pipeline**: Covers Visual Encoder, Projection, Pretraining, SFT full process +- **Education-friendly**: Clean code, suitable for learning VLM principles +- **Ecosystem compatible**: Supports `transformers` and mainstream inference frameworks + +## 📊 Model List + +| Model (Size) | Inference Memory (Approx.) | Release | +|------------------------|----------------------------|------------| +| MiniMind2-V (104M) | 0.6 GB | 2025.02.20 | +| MiniMind2-Small-V (26M) | 1.1 GB | 2025.02.20 | +| minimind-v-v1-small (27M) | 0.6 GB | 2024.10.04 | +| minimind-v-v1 (109M) | 1.1 GB | 2024.10.04 | + +## 🚀 Quick Navigation + +- [Quick Start](quickstart.md) - Environment setup, model download, quick testing +- [Model Training](training.md) - Pretraining, SFT training full process + +## 🔗 Related Links + +- **GitHub**: [https://github.com/jingyaogong/minimind-v](https://github.com/jingyaogong/minimind-v) +- **HuggingFace**: [MiniMind-V Collection](https://huggingface.co/collections/jingyaogong/minimind-v-67000833fb60b3a2e1f3597d) +- **ModelScope**: [MiniMind-V Models](https://www.modelscope.cn/profile/gongjy) +- **Online Demo**: [ModelScope Studio](https://www.modelscope.cn/studios/gongjy/MiniMind-V) + +## 🎯 Project Highlights + +**"Building a plane with Legos is much more exciting than flying in first class!"** + +Is it really as complex as imagined to build a VLM-based multimodal large model? How is the code implementation done? Is the training process difficult? + +Now, let's explore the answers and feel the joy of creation together! + +
+ ![minimind2-v](images/minimind2-v.gif) +
+ +## 📚 Core Concepts + +### What is a Vision-Language Model (VLM)? + +A Vision-Language Model is a multimodal model that can understand both images and text simultaneously. MiniMind-V achieves this through: + +1. Using a **Visual Encoder** (CLIP model) to convert images into feature vectors +2. Aligning visual and text feature spaces through a **Projection layer** +3. Injecting image features into the language model to enable image-text understanding + +### MiniMind-V Design Philosophy + +- **Minimal modifications**: Based on the MiniMind language model, only adding Visual Encoder and Projection submodules +- **Core algorithm changes < 50 lines**: Extremely low migration difficulty +- **Simple yet effective**: Uses linear projection for cross-modal alignment, simple but effective + +## 🎓 Target Audience + +- Beginners who want to understand vision-language model principles +- Researchers who want to quickly build VLM prototypes +- Developers interested in multimodal large models +- Teams needing low-cost training of customized vision-language models + +## 💡 Get Started + +Ready? Let's start your VLM journey from the [Quick Start](quickstart.md) page! diff --git a/docs/quickstart.md b/docs/quickstart.md new file mode 100644 index 0000000..654f93a --- /dev/null +++ b/docs/quickstart.md @@ -0,0 +1,186 @@ +# Quick Start + +This page will help you quickly get started with the MiniMind-V project. + +## 📋 Requirements + +- **Python**: 3.10+ +- **PyTorch**: 1.12+ +- **CUDA**: 12.2+ (optional, for GPU acceleration) +- **VRAM**: At least 8GB (24GB recommended) + +!!! tip "Hardware Configuration Reference" + - CPU: Intel i9-10980XE @ 3.00GHz + - RAM: 128 GB + - GPU: NVIDIA GeForce RTX 3090 (24GB) + - Ubuntu: 20.04 + - CUDA: 12.2 + - Python: 3.10.16 + +## 🚀 Testing Existing Models + +### Step 0: Preparation + +```bash +# Clone the repository +git clone https://github.com/jingyaogong/minimind-v +cd minimind-v +``` + +```bash +# Download CLIP model to ./model/vision_model directory +git clone https://huggingface.co/openai/clip-vit-base-patch16 +# or +git clone https://www.modelscope.cn/models/openai-mirror/clip-vit-base-patch16 +``` + +### Step 1: Install Dependencies + +```bash +pip install -r requirements.txt -i https://pypi.tuna.tsinghua.edu.cn/simple +``` + +!!! warning "Torch CUDA Check" + After installation, test if Torch can use CUDA: + ```python + import torch + print(torch.cuda.is_available()) + ``` + If unavailable, download the whl file from [torch_stable](https://download.pytorch.org/whl/torch_stable.html) for installation. + +### Step 2: Download Model + +Download pretrained models from HuggingFace or ModelScope: + +```bash +# From HuggingFace +git clone https://huggingface.co/jingyaogong/MiniMind2-V + +# Or from ModelScope +git clone https://www.modelscope.cn/models/gongjy/MiniMind2-V.git +``` + +### Step 3: Command Line Q&A + +```bash +# load=0: load PyTorch model, load=1: load Transformers model +python eval_vlm.py --load 1 +``` + +### Step 4: Start WebUI (Optional) + +```bash +python scripts/web_demo_vlm.py +``` + +Visit `http://localhost:8501` to use the web interface for image-text dialogue. + +## 📝 Effect Testing + +### Single Image Dialogue Examples + +**Test Image 1: City Street Scene** + +```text +Q: Describe the content of this image +A: The image shows a busy city street with tall buildings on both sides of a long road. + The street is packed with cars, trucks, and buses, along with many other vehicles... +``` + +**Test Image 2: Panda** + +```text +Q: What animal is in this image? +A: The image shows a white brown bear sitting on the grass, next to a large bear with brown spots. + This bear seems shy or playful as it lies on the grass, resting... +``` + +### Model Performance + +| Model | Parameters | Inference Speed | Image Understanding | +|-------|-----------|-----------------|---------------------| +| MiniMind2-V | 104M | Fast | 😊😊😊😊😊😊 | +| MiniMind2-Small-V | 26M | Very Fast | 😊😊😊😊 | + +## 🔧 Loading from PyTorch Weights + +If you want to use native PyTorch model weights (`*.pth` files): + +### Download Weight Files + +Download the required weight files from: + +- [HuggingFace - MiniMind2-V-PyTorch](https://huggingface.co/jingyaogong/MiniMind2-V-PyTorch) +- [ModelScope - MiniMind2-V-PyTorch](https://www.modelscope.cn/models/gongjy/MiniMind2-V-PyTorch) + +Files needed: +- `sft_vlm_512.pth` or `sft_vlm_768.pth` (SFT model weights) +- Optional: `pretrain_vlm_512.pth` or `pretrain_vlm_768.pth` (pretrained model weights) + +### Run Testing + +```bash +# model_mode=0: test pretrain model, model_mode=1: test SFT model +python eval_vlm.py --load 0 --model_mode 1 +``` + +## 📊 Model Architecture + +MiniMind-V adds Visual Encoder and Projection layers on top of the MiniMind language model: + +![VLM-structure](images/VLM-structure.png) + +### Core Components + +1. **Visual Encoder (CLIP)** + - Uses `clip-vit-base-patch16` model + - Input image size: 224×224 + - Output: 196×768 dimensional visual tokens + +2. **Projection Layer** + - Simple linear transformation + - Aligns visual tokens to text embedding space + +3. **Language Model (MiniMind)** + - Inherits from MiniMind language model + - Supports text generation and dialogue + +### Model Parameter Configuration + +| Model Name | Params | d_model | n_layers | kv_heads | q_heads | +|-----------|--------|---------|----------|----------|---------| +| MiniMind2-Small-V | 26M | 512 | 8 | 2 | 8 | +| MiniMind2-V | 104M | 768 | 16 | 2 | 8 | + +## 🎯 Next Steps + +- Check [Model Training](training.md) to learn how to train your own vision-language model from scratch +- Read the source code to understand VLM implementation principles +- Try testing with your own images + +## ❓ Common Issues + +### 1. Model fails to load? + +Ensure all dependency files are downloaded: +- CLIP model weights +- MiniMind-V model weights +- tokenizer configuration files + +### 2. Out of memory? + +- Use the 26M Small version +- Reduce batch size +- Use CPU inference (slower) + +### 3. Poor image recognition? + +- Ensure image quality is clear +- Try adjusting image size +- Use more specific question descriptions + +## 🔗 Related Resources + +- **Online Demo**: [ModelScope Studio](https://www.modelscope.cn/studios/gongjy/MiniMind-V) +- **Video Introduction**: [Bilibili](https://www.bilibili.com/video/BV1Sh1vYBEzY) +- **Project Home**: [GitHub](https://github.com/jingyaogong/minimind-v) diff --git a/docs/training.md b/docs/training.md new file mode 100644 index 0000000..f22f3d9 --- /dev/null +++ b/docs/training.md @@ -0,0 +1,383 @@ +# Model Training + +This page introduces how to train MiniMind-V vision-language models from scratch. + +## 📊 Data Preparation + +### 1. Download Dataset + +Download datasets from the following addresses: + +- [ModelScope - minimind-v_dataset](https://www.modelscope.cn/datasets/gongjy/minimind-v_dataset) +- [HuggingFace - minimind-v_dataset](https://huggingface.co/datasets/jingyaogong/minimind-v_dataset) + +Create `./dataset` directory and place data files: + +```bash +./dataset/ +├── pretrain_vlm_data.jsonl # Pretrain data (~595K samples) +├── sft_vlm_data.jsonl # Single-image SFT data (~300K samples) +├── sft_vlm_data_multi.jsonl # Multi-image SFT data (~13.6K samples) +├── eval_images/ # Test images +│ ├── 城市车水马龙-city-traffic.jpg +│ ├── 熊猫草地-Panda-Grassland.jpg +│ └── ... +└── sft_multi_images_trans_image.zip # Multi-image data images (needs extraction) +``` + +!!! tip "Dataset Notes" + - `*.jsonl` are Q&A datasets + - `*images` are accompanying image data, need to be extracted after download + - Please reserve about 5GB space for the dataset + - If space is insufficient, try skipping pretrain and go directly to SFT training + +### 2. Data Format + +**Pretrain Data Format** (`pretrain_vlm_data.jsonl`): + +```json +{ + "conversations": [ + { + "role": "user", + "content": "Provide a brief description of the given image.\n" + }, + { + "role": "assistant", + "content": "Olive oil is a healthy ingredient for free use." + } + ], + "image": "GCC_train_002582585.jpg" +} +``` + +**Single-Image SFT Data Format** (`sft_vlm_data.jsonl`): + +```json +{ + "conversations": [ + { + "role": "user", + "content": "What impact does the location of the alarm clock have on sleep quality?" + }, + { + "role": "assistant", + "content": "Place the digital alarm clock on the nightstand..." + } + ], + "image": "train-00000-of-00001_image_0_0.jpg" +} +``` + +**Multi-Image SFT Data Format** (`sft_vlm_data_multi.jsonl`): + +```json +{ + "conversations": [ + { + "role": "user", + "content": "context: Source Image: Target Image: Instruction: What is the correct image edit instruction?" + }, + { + "role": "assistant", + "content": "take the people out of the back in the photo..." + } + ], + "image": "0.jpg, 1.jpg" +} +``` + +### 3. Data Source + +- **Pretrain Data**: [Chinese-LLaVA-Vision](https://huggingface.co/datasets/LinkSoul/Chinese-LLaVA-Vision-Instructions) + - Approximately 570,000 images from CC-3M and COCO 2014 + +- **SFT Data**: [llava-en-zh-300k](https://huggingface.co/datasets/BUAADreamer/llava-en-zh-300k) + - 300K instruction fine-tuning data + - 150K images + - Translated content, better Chinese support + +## 🎯 Training Pipeline + +All training scripts are located in the `./trainer` directory. + +### Step 0: Prepare Base Language Model + +Download pure language model weights to the `./out` directory (as the base language model for training VLM): + +```bash +# Download 512-dim model +wget https://huggingface.co/jingyaogong/MiniMind2-V-PyTorch/blob/main/llm_512.pth + +# Or download 768-dim model +wget https://huggingface.co/jingyaogong/MiniMind2-V-PyTorch/blob/main/llm_768.pth +``` + +### Step 1: Pretraining (Learning Image Description) + +The pretraining stage teaches the model general image knowledge, such as a deer is a deer, a dog is a dog. + +```bash +cd trainer +python train_pretrain_vlm.py --epochs 4 + +# Multi-GPU training +torchrun --nproc_per_node 2 train_pretrain_vlm.py --epochs 4 +``` + +**Output weights**: `./out/pretrain_vlm_*.pth` (* is the model dimension, default is 512) + +!!! info "Training Duration" + - MiniMind2-Small-V (26M): ~1h (single 3090) + - MiniMind2-V (104M): ~3h (single 3090) + +**Training Strategy**: +- Freeze Visual Encoder (CLIP model) gradients +- Only train the last layer parameters of Projection and LLM + +**Loss Curve**: + +![Pretrain Loss](images/pretrain_loss.png) + +### Step 2: Supervised Fine-Tuning (Learning Image-Caption Dialogue Style) + +The SFT stage teaches the model real image-text dialogue format, better aligning with human communication habits. + +```bash +python train_sft_vlm.py --epochs 4 + +# Multi-GPU training +torchrun --nproc_per_node 2 train_sft_vlm.py --epochs 4 +``` + +**Output weights**: `./out/sft_vlm_*.pth` + +!!! info "Training Duration" + - MiniMind2-Small-V: ~1h (single 3090) + - MiniMind2-V: ~3h (single 3090) + +**Training Strategy**: +- Freeze Visual Encoder (CLIP model) gradients +- Set all parameters of Projection and LLM as learnable + +**Loss Curve**: + +![SFT Loss](images/sft_loss.png) + +### Step 3 (Optional): Multi-Image Fine-Tuning + +Multi-image fine-tuning provides a demo example based on bird comparison dataset. + +```bash +python train_sft_vlm.py --epochs 4 --use_multi_image +``` + +**Notes**: +- Multi-image dataset is relatively small and contains English conversations +- Only includes two-image comparison scenarios +- Fine-tuning effect is limited, provided as a reference approach + +!!! warning "Training Notes" + By default, training process saves parameters every 100 steps to `./out/***.pth` (will overwrite old weight files) + +## 📈 Model Architecture Details + +MiniMind-V's structure only adds Visual Encoder and feature projection submodules: + +![VLM Structure](images/VLM-structure.png) + +### Core Components + +1. **Visual Encoder** + - Uses [clip-vit-base-patch16](https://huggingface.co/openai/clip-vit-base-patch16) + - Based on ViT-B/16 architecture + - Input image: 224×224 + - Patch size: 16×16 + - Output: 196×768 dimensional features (196 visual tokens) + +2. **Projection Layer** + - Simple unbiased linear transformation + - Functions: + - Aligns 768-dimensional visual tokens to LLM text tokens + - Maps image features to the same space as text embeddings + - Achieves cross-modal feature alignment + +3. **Language Model** + - Fully inherits from MiniMind + - Minimal modifications (core algorithm changes < 50 lines) + +### Input-Output Mechanism + +**Input Format**: + +In `minimind-v`, a 196-character `@@@...@@@` placeholder is used to replace the image: + +```text +@@@......@@@\nWhat is this image describing? +``` + +Why 196 characters? Because any image is encoded by the CLIP model as 196×768 dimensional tokens. + +![Input Mechanism](images/minimind-v-input.png) + +**Multi-Image Implementation**: + +Achieved by injecting multiple `` placeholders, no framework modification needed. + +### Model Parameter Configuration + +| Model Name | Params | d_model | n_layers | kv_heads | q_heads | Visual Token | +|-----------|--------|---------|----------|----------|---------|--------------| +| MiniMind2-Small-V | 26M | 512 | 8 | 2 | 8 | 196×768 | +| MiniMind2-V | 104M | 768 | 16 | 2 | 8 | 196×768 | + +## 🧪 Test Model + +### Test Trained Model + +Ensure the model `*.pth` file to be tested is in the `./out/` directory. + +```bash +# model_mode=0: test pretrain model +python eval_vlm.py --model_mode 0 + +# model_mode=1: test SFT model +python eval_vlm.py --model_mode 1 +``` + +### Use Pre-Trained Model + +You can also directly download and use pre-trained `*.pth` files: + +- [HuggingFace - MiniMind2-V-PyTorch](https://huggingface.co/jingyaogong/MiniMind2-V-PyTorch) +- [ModelScope - MiniMind2-V-PyTorch](https://www.modelscope.cn/models/gongjy/MiniMind2-V-PyTorch) + +## 🔧 Multi-GPU Training + +### DDP Method + +```bash +torchrun --nproc_per_node N train_xxx.py +``` + +### DeepSpeed Method + +```bash +deepspeed --master_port 29500 --num_gpus=N train_xxx.py +``` + +### Wandb Monitoring + +```bash +# Login first +wandb login + +# Enable wandb +torchrun --nproc_per_node N train_xxx.py --use_wandb +``` + +By adding the `--use_wandb` parameter, you can log the training process. After training, you can view the process on the wandb website. + +You can specify the project name and run name by modifying `wandb_project` and `wandb_run_name` parameters. + +## 💰 Training Cost + +Based on single NVIDIA 3090: + +| Dataset Combination | Training Time | Cost (Approx.) | Effect | +|---------------------|---------------|----------------|---------| +| pretrain (1 epoch) + sft (1 epoch) | 2h | ≈1.3 RMB | 😊😊😊 Basic dialogue | +| pretrain (4 epochs) + sft (4 epochs) | 8h | ≈5.4 RMB | 😊😊😊😊😊 Better effect | + +!!! success "Quick Reproduction" + Using single epoch of pretrain and SFT, single 3090 only needs **2 hours + 1.3 RMB** to train a vision-language ChatBot! + +## 📝 Training Tips + +### 1. Reduce Memory Usage + +- Use smaller batch size +- Use 512-dim model instead of 768 +- Enable gradient checkpointing (if implemented) +- Use DeepSpeed ZeRO optimization + +### 2. Accelerate Training + +- Use multi-GPU training (DDP or DeepSpeed) +- Use mixed precision training (FP16) +- Reduce image resolution (but may affect performance) + +### 3. Improve Performance + +- Increase training epochs +- Use larger model (768 vs 512) +- Use higher quality datasets +- Adjust learning rate + +## 🎓 Core Principles + +### Why Pretraining? + +Pretraining teaches the model basic image description capabilities, establishing fundamental mapping relationships between image features and text. + +### Why SFT? + +SFT teaches the model real dialogue formats, making its outputs more aligned with human communication habits rather than simple image descriptions. + +### Why Freeze CLIP? + +CLIP is already a powerful pre-trained visual encoder. Freezing its parameters can: +- Significantly reduce trainable parameters +- Speed up training +- Prevent overfitting +- Lower training costs + +### Future Improvement Directions + +```text +> More complex Projection layers (e.g., Cross-Attention) may bring better cross-modal alignment +> Use larger CLIP models (e.g., large series) for finer-grained image features +> Increase image resolution (currently only 224×224, dataset uses 128×128) +> Expand multi-image datasets to support more complex multi-image understanding scenarios +``` + +## 📚 Related Resources + +- **Base Language Model**: [MiniMind](https://github.com/jingyaogong/minimind) +- **Reference Paper**: [LlaVA](https://arxiv.org/pdf/2304.08485) +- **Visual Encoder**: [CLIP](https://huggingface.co/openai/clip-vit-base-patch16) + +## ❓ Common Issues + +### 1. Out of memory during training? + +- Reduce batch_size +- Use 512-dim model +- Use DeepSpeed ZeRO +- Single-GPU training instead of multi-GPU + +### 2. Training loss not decreasing? + +- Check if dataset path is correct +- Check if learning rate is appropriate +- Confirm base LLM model is loaded correctly +- Check for gradient explosion/vanishing + +### 3. Multi-GPU training error? + +- Ensure all GPUs are visible and CUDA versions are consistent +- Check if port is occupied (modify `--master_port`) +- Try using DeepSpeed instead of DDP + +### 4. How to use custom dataset? + +Prepare jsonl files and corresponding images according to the data format above, and modify the data path in the training script. + +## 🎯 Next Steps + +Congratulations! You've learned the complete training process of MiniMind-V. Now you can: + +- Start training your own vision-language model +- Try using different datasets +- Explore the source code implementation +- Contribute your improvements to the project diff --git a/mkdocs.yml b/mkdocs.yml new file mode 100644 index 0000000..cef3ffb --- /dev/null +++ b/mkdocs.yml @@ -0,0 +1,66 @@ +site_name: MiniMind-V +site_description: MiniMind-V - 超小多模态视觉语言模型 / Super-Small Multimodal Vision-Language Model +site_author: jingyaogong +site_url: https://minimind-v.readthedocs.io/ + +# 搜索插件配置 +plugins: + - search: + lang: en + +# 主题配置 +theme: + name: material + favicon: images/logo.png + icon: + logo: material/book-open-page-variant + palette: + # 浅色模式 + - scheme: default + primary: white + accent: blue + toggle: + icon: material/brightness-7 + name: 切换至深色模式 + # 深色模式 + - scheme: slate + primary: black + accent: blue + toggle: + icon: material/brightness-4 + name: 切换至浅色模式 + features: + # - navigation.instant # 与多语言切换不兼容，已禁用 + - navigation.tracking # 锚点跟踪 + - navigation.sections # 导航分组 + - navigation.expand # 默认展开导航 + - navigation.top # 返回顶部按钮 + - search.suggest # 搜索建议 + - search.highlight # 搜索高亮 + - content.code.copy # 代码复制按钮 + - toc.follow # 目录跟随 + - toc.integrate # 目录集成到左侧边栏 + language: en + +# 导航结构 +nav: + - Home: index.md + - Quick Start: quickstart.md + - Model Training: training.md + +# Markdown 扩展 +markdown_extensions: + - toc: + permalink: true + - admonition + - pymdownx.highlight: + anchor_linenums: true + - pymdownx.inlinehilite + - pymdownx.snippets + - pymdownx.superfences + - pymdownx.details + - pymdownx.tabbed: + alternate_style: true + - attr_list + - md_in_html + diff --git a/requirements.txt b/requirements.txt new file mode 100644 index 0000000..a007367 --- /dev/null +++ b/requirements.txt @@ -0,0 +1,2 @@ +mkdocs>=1.5.0 +mkdocs-material>=9.0.0