187 lines
4.9 KiB
Markdown
187 lines
4.9 KiB
Markdown
# Quick Start
|
||
|
||
This page will help you quickly get started with the MiniMind-V project.
|
||
|
||
## 📋 Requirements
|
||
|
||
- **Python**: 3.10+
|
||
- **PyTorch**: 1.12+
|
||
- **CUDA**: 12.2+ (optional, for GPU acceleration)
|
||
- **VRAM**: At least 8GB (24GB recommended)
|
||
|
||
!!! tip "Hardware Configuration Reference"
|
||
- CPU: Intel i9-10980XE @ 3.00GHz
|
||
- RAM: 128 GB
|
||
- GPU: NVIDIA GeForce RTX 3090 (24GB)
|
||
- Ubuntu: 20.04
|
||
- CUDA: 12.2
|
||
- Python: 3.10.16
|
||
|
||
## 🚀 Testing Existing Models
|
||
|
||
### Step 0: Preparation
|
||
|
||
```bash
|
||
# Clone the repository
|
||
git clone https://github.com/jingyaogong/minimind-v
|
||
cd minimind-v
|
||
```
|
||
|
||
```bash
|
||
# Download CLIP model to ./model/vision_model directory
|
||
git clone https://huggingface.co/openai/clip-vit-base-patch16
|
||
# or
|
||
git clone https://www.modelscope.cn/models/openai-mirror/clip-vit-base-patch16
|
||
```
|
||
|
||
### Step 1: Install Dependencies
|
||
|
||
```bash
|
||
pip install -r requirements.txt -i https://pypi.tuna.tsinghua.edu.cn/simple
|
||
```
|
||
|
||
!!! warning "Torch CUDA Check"
|
||
After installation, test if Torch can use CUDA:
|
||
```python
|
||
import torch
|
||
print(torch.cuda.is_available())
|
||
```
|
||
If unavailable, download the whl file from [torch_stable](https://download.pytorch.org/whl/torch_stable.html) for installation.
|
||
|
||
### Step 2: Download Model
|
||
|
||
Download pretrained models from HuggingFace or ModelScope:
|
||
|
||
```bash
|
||
# From HuggingFace
|
||
git clone https://huggingface.co/jingyaogong/MiniMind2-V
|
||
|
||
# Or from ModelScope
|
||
git clone https://www.modelscope.cn/models/gongjy/MiniMind2-V.git
|
||
```
|
||
|
||
### Step 3: Command Line Q&A
|
||
|
||
```bash
|
||
# load=0: load PyTorch model, load=1: load Transformers model
|
||
python eval_vlm.py --load 1
|
||
```
|
||
|
||
### Step 4: Start WebUI (Optional)
|
||
|
||
```bash
|
||
python scripts/web_demo_vlm.py
|
||
```
|
||
|
||
Visit `http://localhost:8501` to use the web interface for image-text dialogue.
|
||
|
||
## 📝 Effect Testing
|
||
|
||
### Single Image Dialogue Examples
|
||
|
||
**Test Image 1: City Street Scene**
|
||
|
||
```text
|
||
Q: Describe the content of this image
|
||
A: The image shows a busy city street with tall buildings on both sides of a long road.
|
||
The street is packed with cars, trucks, and buses, along with many other vehicles...
|
||
```
|
||
|
||
**Test Image 2: Panda**
|
||
|
||
```text
|
||
Q: What animal is in this image?
|
||
A: The image shows a white brown bear sitting on the grass, next to a large bear with brown spots.
|
||
This bear seems shy or playful as it lies on the grass, resting...
|
||
```
|
||
|
||
### Model Performance
|
||
|
||
| Model | Parameters | Inference Speed | Image Understanding |
|
||
|-------|-----------|-----------------|---------------------|
|
||
| MiniMind2-V | 104M | Fast | 😊😊😊😊😊😊 |
|
||
| MiniMind2-Small-V | 26M | Very Fast | 😊😊😊😊 |
|
||
|
||
## 🔧 Loading from PyTorch Weights
|
||
|
||
If you want to use native PyTorch model weights (`*.pth` files):
|
||
|
||
### Download Weight Files
|
||
|
||
Download the required weight files from:
|
||
|
||
- [HuggingFace - MiniMind2-V-PyTorch](https://huggingface.co/jingyaogong/MiniMind2-V-PyTorch)
|
||
- [ModelScope - MiniMind2-V-PyTorch](https://www.modelscope.cn/models/gongjy/MiniMind2-V-PyTorch)
|
||
|
||
Files needed:
|
||
- `sft_vlm_512.pth` or `sft_vlm_768.pth` (SFT model weights)
|
||
- Optional: `pretrain_vlm_512.pth` or `pretrain_vlm_768.pth` (pretrained model weights)
|
||
|
||
### Run Testing
|
||
|
||
```bash
|
||
# model_mode=0: test pretrain model, model_mode=1: test SFT model
|
||
python eval_vlm.py --load 0 --model_mode 1
|
||
```
|
||
|
||
## 📊 Model Architecture
|
||
|
||
MiniMind-V adds Visual Encoder and Projection layers on top of the MiniMind language model:
|
||
|
||
|
||
|
||
### Core Components
|
||
|
||
1. **Visual Encoder (CLIP)**
|
||
- Uses `clip-vit-base-patch16` model
|
||
- Input image size: 224×224
|
||
- Output: 196×768 dimensional visual tokens
|
||
|
||
2. **Projection Layer**
|
||
- Simple linear transformation
|
||
- Aligns visual tokens to text embedding space
|
||
|
||
3. **Language Model (MiniMind)**
|
||
- Inherits from MiniMind language model
|
||
- Supports text generation and dialogue
|
||
|
||
### Model Parameter Configuration
|
||
|
||
| Model Name | Params | d_model | n_layers | kv_heads | q_heads |
|
||
|-----------|--------|---------|----------|----------|---------|
|
||
| MiniMind2-Small-V | 26M | 512 | 8 | 2 | 8 |
|
||
| MiniMind2-V | 104M | 768 | 16 | 2 | 8 |
|
||
|
||
## 🎯 Next Steps
|
||
|
||
- Check [Model Training](training.md) to learn how to train your own vision-language model from scratch
|
||
- Read the source code to understand VLM implementation principles
|
||
- Try testing with your own images
|
||
|
||
## ❓ Common Issues
|
||
|
||
### 1. Model fails to load?
|
||
|
||
Ensure all dependency files are downloaded:
|
||
- CLIP model weights
|
||
- MiniMind-V model weights
|
||
- tokenizer configuration files
|
||
|
||
### 2. Out of memory?
|
||
|
||
- Use the 26M Small version
|
||
- Reduce batch size
|
||
- Use CPU inference (slower)
|
||
|
||
### 3. Poor image recognition?
|
||
|
||
- Ensure image quality is clear
|
||
- Try adjusting image size
|
||
- Use more specific question descriptions
|
||
|
||
## 🔗 Related Resources
|
||
|
||
- **Online Demo**: [ModelScope Studio](https://www.modelscope.cn/studios/gongjy/MiniMind-V)
|
||
- **Video Introduction**: [Bilibili](https://www.bilibili.com/video/BV1Sh1vYBEzY)
|
||
- **Project Home**: [GitHub](https://github.com/jingyaogong/minimind-v)
|