github
/
minimind
mirror of https://github.com/jingyaogong/minimind.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

[feat] update docs

v2
jingyaogong 2025-10-21 22:24:30 +08:00
parent 941db7a5e6
commit f44ee7a1b0
8 changed files with 899 additions and 170 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

BIN
docs/images/train_grpo_512.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 214 KiB

BIN
docs/images/train_grpo_768.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 246 KiB

BIN
docs/images/train_ppo_512.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 246 KiB

BIN
docs/images/train_ppo_768.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 241 KiB

BIN
docs/images/train_spo_768.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 234 KiB

131
docs/index.md
View File

@ -1,4 +1,4 @@
# <strong>Welcome to MiniMind!</strong> # Welcome to MiniMind!
<figure markdown> <figure markdown>
![logo](images/logo.png) ![logo](images/logo.png)
@ -7,47 +7,118 @@
## 📌 Introduction ## 📌 Introduction
MiniMind is a super-small language model project trained completely from scratch, requiring **only $0.5 + 2 hours** to train a **26M** language model! **MiniMind** is a complete, open-source project for training ultra-small language models from scratch with minimal cost. Train a **26M** ChatBot in just **2 hours** with only **$3** on a single 3090 GPU!
- **MiniMind** series is extremely lightweight, the smallest version is **1/7000** the size of GPT-3 - **MiniMind** series is extremely lightweight, the smallest version is **1/7000** the size of GPT-3
- The project open-sources the minimalist structure of large models, including: - Complete implementation covering:
- Mixture of Experts (MoE) - **Tokenizer training** with custom vocabulary
- Dataset cleaning - **Pretraining** (knowledge learning)
- Pretraining - **Supervised Fine-Tuning (SFT)** (conversation patterns)
- Supervised Fine-Tuning (SFT) - **LoRA fine-tuning** (parameter-efficient adaptation)
- LoRA fine-tuning - **Direct Preference Optimization (DPO)** (human preference alignment)
- Direct Preference Optimization (DPO) - **RLAIF algorithms** (PPO/GRPO/SPO - reinforcement learning)
- Model distillation - **Knowledge distillation** (compress large model knowledge)
- All core algorithm code is reconstructed from scratch using native PyTorch, without relying on third-party abstract interfaces - **Model reasoning distillation** (DeepSeek-R1 style)
- This is not only a full-stage open-source reproduction of large language models, but also a tutorial for getting started with LLMs - **YaRN algorithm** (context length extrapolation)
- **Pure PyTorch implementation**: All core algorithms are implemented from scratch using native PyTorch, without relying on third-party abstract interfaces
- **Educational value**: This is not only a full-stage open-source reproduction of large language models, but also a comprehensive tutorial for getting started with LLMs
- **Extended capabilities**: MiniMind now supports [MiniMind-V](https://github.com/jingyaogong/minimind-v) for vision multimodal tasks
!!! note "Training Cost" !!! note "Training Cost & Time"
"2 hours" is based on NVIDIA 3090 hardware (single card) testing, "$0.5" refers to GPU server rental cost "2 hours" is based on **NVIDIA 3090** hardware (single card) testing
"$3" refers to GPU server rental cost
With 8× RTX 4090 GPUs, training time can be compressed to **under 10 minutes**
## ✨ Key Features ## ✨ Key Highlights
- **Ultra-low cost**: Single 3090, 2 hours, $0.5 to train a ChatBot from scratch - **Ultra-low cost**: Single 3090, 2 hours, $3 to train a fully functional ChatBot from scratch
- **Complete pipeline**: Covers Tokenizer, pretraining, SFT, LoRA, DPO, distillation full process - **Complete pipeline**: Tokenizer → Pretraining → SFT → LoRA → DPO/RLAIF → Distillation → Reasoning
- **Education-friendly**: Clean code, suitable for learning LLM principles - **Latest algorithms**: Implements cutting-edge techniques including GRPO, SPO, and YaRN
- **Ecosystem compatible**: Supports `transformers`, `llama.cpp`, `vllm`, `ollama` and other mainstream frameworks - **Education-friendly**: Clean, well-documented code suitable for learning LLM principles
- **Ecosystem compatible**: Seamless support for `transformers`, `trl`, `peft`, `llama.cpp`, `vllm`, `ollama`, and `Llama-Factory`
- **Full capabilities**: Supports multi-GPU training (DDP/DeepSpeed), model visualization (Wandb/SwanLab), and dynamic checkpoint management
- **Production-ready**: OpenAI API protocol support for easy integration with third-party UIs (FastGPT, Open-WebUI, etc.)
- **Multimodal extension**: Extended to vision with [MiniMind-V](https://github.com/jingyaogong/minimind-v)
## 📊 Model List ## 📊 Model Series
| Model (Size) | Inference Memory (Approx.) | Release | ### MiniMind2 Series (Latest - 2025.04.26)
|------------|----------|---------|
| MiniMind2-small (26M) | 0.5 GB | 2025.04.26 | | Model | Parameters | Vocabulary | Layers | Hidden Dim | Context | Inference Memory |
| MiniMind2-MoE (145M) | 1.0 GB | 2025.04.26 | |-------|-----------|------------|--------|-----------|---------|-----------------|
| MiniMind2 (104M) | 1.0 GB | 2025.04.26 | | MiniMind2-small | 26M | 6,400 | 8 | 512 | 2K | ~0.5 GB |
| MiniMind2-MoE | 145M | 6,400 | 8 | 640 | 2K | ~1.0 GB |
| MiniMind2 | 104M | 6,400 | 16 | 768 | 2K | ~1.0 GB |
### MiniMind-V1 Series (Legacy - 2024.09.01)
| Model | Parameters | Vocabulary | Layers | Hidden Dim | Context |
|-------|-----------|------------|--------|-----------|---------|
| minimind-v1-small | 26M | 6,400 | 8 | 512 | 2K |
| minimind-v1-moe | 104M | 6,400 | 8 | 512 | 2K |
| minimind-v1 | 108M | 6,400 | 16 | 768 | 2K |
## 📅 Latest Updates (2025-10-24)
🔥 **RLAIF Training Algorithms**: Native implementation of PPO, GRPO, and SPO
- **YaRN Algorithm**: RoPE length extrapolation for improved long-sequence handling
- **Adaptive Thinking**: Reasoning models support optional thinking chains
- **Full template support**: Tool calling and reasoning tags (`<tool_call>`, `<think>`, etc.)
- **Visualization**: Switched from WandB to [SwanLab](https://swanlab.cn/) (China-friendly)
- **Reasoning models**: Complete MiniMind-Reason series based on DeepSeek-R1 distillation
## 🎯 Project Contents
- Complete MiniMind-LLM architecture code (Dense + MoE models)
- Detailed Tokenizer training code
- Full training pipeline: Pretrain → SFT → LoRA → RLHF/RLAIF → Distillation
- High-quality, curated and deduplicated datasets at all stages
- Native PyTorch implementation of key algorithms, minimal third-party dependencies
- Multi-GPU training support (single-machine multi-card DDP, DeepSpeed, distributed clusters)
- Visualization with Wandb/SwanLab
- Model evaluation on third-party benchmarks (C-Eval, C-MMLU, OpenBookQA)
- YaRN algorithm for RoPE context length extrapolation
- OpenAI API protocol server for easy integration
- Streamlit web UI for chat
- Full compatibility with community tools: llama.cpp, vllm, ollama, Llama-Factory
- MiniMind-Reason models: Complete open-source data + weights for reasoning distillation
## 🚀 Quick Navigation ## 🚀 Quick Navigation
- [Quick Start](quickstart.md) - Environment setup, model download, quick testing - **[Quick Start](quickstart.md)** - Environment setup, model download, quick testing
- [Model Training](training.md) - Pretraining, SFT, LoRA, DPO training process - **[Model Training](training.md)** - Pretraining, SFT, LoRA, RLHF, RLAIF, and reasoning training
## 🔗 Related Links ## 🔗 Links & Resources
**Project Repositories**:
- **GitHub**: [https://github.com/jingyaogong/minimind](https://github.com/jingyaogong/minimind) - **GitHub**: [https://github.com/jingyaogong/minimind](https://github.com/jingyaogong/minimind)
- **HuggingFace**: [MiniMind Collection](https://huggingface.co/collections/jingyaogong/minimind-66caf8d999f5c7fa64f399e5) - **HuggingFace**: [MiniMind Collection](https://huggingface.co/collections/jingyaogong/minimind-66caf8d999f5c7fa64f399e5)
- **ModelScope**: [MiniMind Models](https://www.modelscope.cn/profile/gongjy) - **ModelScope**: [MiniMind Profile](https://www.modelscope.cn/profile/gongjy)
- **Online Demo**: [ModelScope Studio](https://www.modelscope.cn/studios/gongjy/MiniMind)
**Online Demos**:
- [ModelScope Studio - Standard Chat](https://www.modelscope.cn/studios/gongjy/MiniMind)
- [ModelScope Studio - Reasoning Model](https://www.modelscope.cn/studios/gongjy/MiniMind-Reasoning)
- [Bilibili Video Introduction](https://www.bilibili.com/video/BV12dHPeqE72/)
**Vision Extension**:
- [MiniMind-V](https://github.com/jingyaogong/minimind-v) - Multimodal vision language models
## 💡 Why MiniMind?
The AI community is flooded with high-cost, complex frameworks that abstract away the fundamentals. MiniMind aims to democratize LLM learning by:
1. **Lowering the barrier**: No need for expensive GPUs or cloud services
2. **Understanding, not just using**: Learn every detail from tokenization to inference
3. **End-to-end learning**: Train from scratch, not just fine-tune existing models
4. **Code clarity**: Pure PyTorch implementations you can read and understand
5. **Practical results**: Get a working ChatBot with minimal resources
As we say: **"Building a Lego airplane is far more exciting than flying first class!"**
---
Next: [Get Started →](quickstart.md)

249
docs/quickstart.md
View File

@ -1,114 +1,279 @@
# Quick Start # Quick Start
This page will help you quickly get started with the MiniMind project. Get MiniMind up and running in minutes!
## 📋 Requirements ## 📋 Requirements
### Hardware
- **GPU Memory**: 8GB minimum (24GB recommended for comfortable development)
- **Recommended GPU**: NVIDIA RTX 3090 (24GB)
### Software
- **Python**: 3.10+ - **Python**: 3.10+
- **PyTorch**: 1.12+ - **PyTorch**: 2.0+ (with CUDA 12.2+ for GPU support)
- **CUDA**: 12.2+ (optional, for GPU acceleration) - **CUDA**: 12.2+ (optional, for GPU acceleration)
- **VRAM**: At least 8GB (24GB recommended)
!!! tip "Hardware Configuration Reference" !!! tip "Hardware Configuration Reference"
- CPU: Intel i9-10980XE @ 3.00GHz - **CPU**: Intel i9-10980XE @ 3.00GHz
- RAM: 128 GB - **RAM**: 128 GB
- GPU: NVIDIA GeForce RTX 3090 (24GB) - **GPU**: NVIDIA GeForce RTX 3090 (24GB) × 8
- **OS**: Ubuntu 20.04
- **CUDA**: 12.2
- **Python**: 3.10.16
## 🚀 Testing Existing Models ## 🚀 Step 0: Clone the Repository
### 1. Clone the Project
```bash ```bash
git clone https://github.com/jingyaogong/minimind.git git clone https://github.com/jingyaogong/minimind.git
cd minimind cd minimind
``` ```
### 2. Install Dependencies ## 🎯 Section I: Testing Existing Models
### 1. Environment Setup
```bash ```bash
pip install -r requirements.txt -i https://pypi.tuna.tsinghua.edu.cn/simple pip install -r requirements.txt -i https://pypi.tuna.tsinghua.edu.cn/simple
``` ```
!!! warning "Torch CUDA Check" !!! warning "Verify CUDA Support"
After installation, test if Torch can use CUDA: After installation, verify PyTorch can access CUDA:
```python ```python
import torch import torch
print(torch.cuda.is_available()) print(torch.cuda.is_available())
print(torch.cuda.get_device_name(0))
``` ```
If `False`, download the correct PyTorch version from [PyTorch Official](https://download.pytorch.org/whl/torch_stable.html)
### 3. Download Model ### 2. Download Pretrained Models
Download pretrained models from HuggingFace or ModelScope: Choose one option:
**From HuggingFace** (recommended for international users):
```bash ```bash
# From HuggingFace
git clone https://huggingface.co/jingyaogong/MiniMind2 git clone https://huggingface.co/jingyaogong/MiniMind2
```
# Or from ModelScope **From ModelScope** (recommended for China users):
```bash
git clone https://www.modelscope.cn/models/gongjy/MiniMind2.git git clone https://www.modelscope.cn/models/gongjy/MiniMind2.git
``` ```
### 4. Command Line Q&A ### 3. Command-Line Chat
```bash ```bash
# load=0: load PyTorch model, load=1: load Transformers model # load=0: load PyTorch model, load=1: load transformers model
python eval_model.py --load 1 --model_mode 2 python eval_model.py --load 1 --model_mode 2
``` ```
### 5. Start WebUI (Optional) **Model Modes**:
- `model_mode 0`: Pretrain model (word continuation)
- `model_mode 1`: SFT Chat model (conversation)
- `model_mode 2`: RLHF model (refined responses, currently same as SFT for small models)
- `model_mode 3`: Reasoning model (with thinking chains)
- `model_mode 4/5`: RLAIF models (PPO/GRPO trained)
**Example Session**:
```text
👶: Hello, please introduce yourself.
🤖️: I am MiniMind, an AI assistant developed by Jingyao Gong.
I use natural language processing and machine learning algorithms to interact with users.
👶: What's the capital of France?
🤖️: The capital of France is Paris, which is located in the northern central part of France.
It is the largest city in France and serves as its political, economic, and cultural center.
```
### 4. Web UI Demo (Optional)
```bash ```bash
# Requires Python >= 3.10 # Requires Python >= 3.10
pip install streamlit pip install streamlit
cd scripts cd scripts
streamlit run web_demo.py streamlit run web_demo.py
``` ```
Visit `http://localhost:8501` to use the web interface. Visit `http://localhost:8501` to use the interactive web interface.
## 🔧 Third-party Inference Frameworks ### 5. Rope Length Extrapolation with YaRN
MiniMind supports multiple mainstream inference frameworks: Extend context length beyond training with RoPE extrapolation:
### Ollama ```bash
python eval_model.py --inference_rope_scaling True
```
This enables the YaRN algorithm to handle sequences longer than the 2K training context, useful for processing documents and long conversations.
## 🔧 Third-Party Inference Frameworks
MiniMind is compatible with popular inference engines:
### Ollama (Easiest)
```bash ```bash
ollama run jingyaogong/minimind2 ollama run jingyaogong/minimind2
``` ```
### vLLM ### vLLM (Fastest)
```bash ```bash
vllm serve ./MiniMind2/ --served-model-name "minimind" vllm serve ./MiniMind2/ --served-model-name "minimind" --port 8000
# Test with curl
curl http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "minimind",
"messages": [{"role": "user", "content": "Hello!"}],
"temperature": 0.7,
"max_tokens": 512
}'
``` ```
### llama.cpp ### llama.cpp (CPU-Friendly)
```bash ```bash
# Convert model # Convert to GGUF format
python convert_hf_to_gguf.py ./MiniMind2/ python scripts/convert_model.py ./MiniMind2/ --output ./MiniMind2.gguf
# Quantize model # Quantize for size reduction
./build/bin/llama-quantize ./MiniMind2/MiniMind2-109M-F16.gguf ./Q4-MiniMind2.gguf Q4_K_M ./llama-quantize ./MiniMind2.gguf ./MiniMind2-Q4.gguf Q4_K_M
# Inference # Run inference
./build/bin/llama-cli -m ./Q4-MiniMind2.gguf --chat-template chatml ./llama-cli -m ./MiniMind2-Q4.gguf -p "Hello" -n 128
``` ```
## 📝 Effect Testing ## 🔌 OpenAI API Server (For Integration)
Run MiniMind as an OpenAI API-compatible service:
```bash
python scripts/serve_openai_api.py
```
Test the API:
```bash
# In another terminal
python scripts/chat_openai_api.py
```
**cURL Example**:
```bash
curl http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "minimind",
"messages": [
{"role": "user", "content": "Explain machine learning in one sentence."}
],
"temperature": 0.7,
"max_tokens": 256,
"stream": true
}'
```
This enables integration with:
- [FastGPT](https://fastgpt.run/)
- [Open-WebUI](https://github.com/open-webui/open-webui)
- [Dify](https://dify.ai/)
- Any OpenAI API-compatible client
## 📊 Model Selection Guide
| Use Case | Recommended Model | Memory | Speed |
|----------|------------------|--------|-------|
| Learning/Testing | MiniMind2-small (26M) | ~0.5 GB | Fastest |
| Balanced | MiniMind2 (104M) | ~1.0 GB | Fast |
| Expert System (MoE) | MiniMind2-MoE (145M) | ~1.0 GB | Dynamic |
| Reasoning/Complex | MiniMind-Reason (104M) | ~1.0 GB | Standard |
## ⚡ Quick Test Results
**Model**: MiniMind2 (104M parameters)
```text ```text
👶: Hello, please introduce yourself. Q: What is photosynthesis?
🤖️: Hello! I'm MiniMind, an AI assistant developed by Jingyao Gong. A: Photosynthesis is a process in which plants convert light energy from the sun
I interact with users through natural language processing and algorithm training. into chemical energy to produce glucose. This process occurs mainly in leaves
and is essential for plant growth and survival.
👶: What is the highest mountain in the world? Q: Write a Python function to calculate Fibonacci numbers.
🤖️: Mount Everest is the highest mountain in the world, located in the Himalayas, A: def fibonacci(n):
with an elevation of 8,848.86 meters (29,031.7 feet). if n <= 1:
return n
return fibonacci(n-1) + fibonacci(n-2)
# For better performance, use dynamic programming:
def fibonacci_dp(n):
dp = [0] * (n + 1)
for i in range(2, n + 1):
dp[i] = dp[i-1] + dp[i-2]
return dp[n]
Q: 世界上最高的山峰是什么? (What is the highest mountain?)
A: 珠穆朗玛峰（Mount Everest）是世界上最高的山峰，位于喜马拉雅山脉...
(Mount Everest is the world's highest mountain, located in the Himalayas...)
``` ```
## 🎯 Next Steps ## 🆘 Troubleshooting
- Check [Model Training](training.md) to learn how to train your own model from scratch ### Issue: CUDA Out of Memory
- Read the source code to understand LLM implementation principles
**Solution**:
```bash
# Reduce batch size
python eval_model.py --batch_size 1
# Or use CPU (slow but works)
python eval_model.py --device cpu
```
### Issue: Slow Inference
**Solutions**:
- Use vLLM or llama.cpp for faster inference
- Enable quantization (4-bit, 8-bit)
- Use GPU instead of CPU
- Reduce `max_tokens` parameter
### Issue: Model Responses Are Poor Quality
**Possible Causes**:
- Using pretrain model (`model_mode 0`) instead of SFT (`model_mode 1`)
- Model is undertrained - download the full checkpoint instead
- Input prompt is too short - provide more context
### Issue: Python/PyTorch Version Mismatch
**Solution**:
```bash
# Use conda for clean environment
conda create -n minimind python=3.10
conda activate minimind
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu122
pip install -r requirements.txt
```
## 📖 Next Steps
- **[Model Training Guide](training.md)** - Train your own MiniMind from scratch
- **[Source Code](https://github.com/jingyaogong/minimind)** - Explore and learn LLM implementation
- **[Inference Benchmarks](https://huggingface.co/collections/jingyaogong/minimind-66caf8d999f5c7fa64f399e5)** - See model performance comparisons
## 💡 Pro Tips
1. **GPU Memory Optimization**: Use `torch.cuda.empty_cache()` periodically
2. **Batch Processing**: For efficiency, process multiple prompts in batches
3. **Temperature Tuning**: Lower (0.3-0.7) = more consistent, Higher (0.8-1.0) = more creative
4. **Prompt Engineering**: Better prompts → better results, even for small models
5. **Model Quantization**: Use 4-bit quantization to run on smaller GPUs
---
Done! Now you're ready to use MiniMind. Start with the Quick Start, then move to [Model Training](training.md) to learn how to train your own models.

689
docs/training.md
View File

@ -1,155 +1,419 @@
# Model Training # Model Training Guide
This page introduces how to train MiniMind language models from scratch. Learn how to train MiniMind language models from scratch using pure PyTorch.
## 📊 Data Preparation ## 📊 Training Overview
### 1. Download Dataset MiniMind implements a complete training pipeline:
Download datasets from [ModelScope](https://www.modelscope.cn/datasets/gongjy/minimind_dataset/files) or [HuggingFace](https://huggingface.co/datasets/jingyaogong/minimind_dataset). ```
Tokenizer Training
Create `./dataset` directory and place data files: ↓
Pretraining (Learn knowledge)
```bash ↓
./dataset/ SFT (Learn conversation)
├── pretrain_hq.jsonl (1.6GB, ✨Recommended) ↓
├── sft_mini_512.jsonl (1.2GB, ✨Recommended) ┌───────────────────┬─────────────────────┬──────────────┐
├── sft_512.jsonl (7.5GB) ↓ ↓ ↓ ↓
├── sft_1024.jsonl (5.6GB) LoRA DPO/RLHF RLAIF (PPO/GRPO/SPO) Distillation
├── sft_2048.jsonl (9GB) (Domain adapt) (Preference) (Reinforcement Learn) (Reasoning)
├── dpo.jsonl (909MB)
├── r1_mix_1024.jsonl (340MB)
└── lora_*.jsonl
``` ```
!!! tip "Recommended Combination" ## 💰 Training Costs (Single NVIDIA 3090)
Fastest reproduction: `pretrain_hq.jsonl` + `sft_mini_512.jsonl`
| Model | Dataset | Duration | Cost (RMB) | Quality |
|-------|---------|----------|-----------|---------|
| MiniMind2-Small | pretrain_hq + sft_mini_512 | 2.1h | ≈3 | 😊😊 |
| MiniMind2-Small | Full dataset | 38h | ≈50 | 😊😊😊😊😊😊 |
| MiniMind2 | pretrain_hq + sft_mini_512 | 3.3h | ≈5 | 😊😊 |
| MiniMind2 | Full dataset | 122h | ≈160 | 😊😊😊😊😊😊😊 |
!!! success "Ultra-Fast Training"
**Just 2.1 hours + $3 = Functional ChatBot!**
**Single 3090 only needs 2 hours + $0.5!** Use `pretrain_hq.jsonl` + `sft_mini_512.jsonl` for fastest reproduction
### 2. Data Format ## 📋 Data Preparation
**Pretrain Data** (`pretrain_hq.jsonl`): ### 1. Download Datasets
Download from [ModelScope](https://www.modelscope.cn/datasets/gongjy/minimind_dataset) or [HuggingFace](https://huggingface.co/datasets/jingyaogong/minimind_dataset):
```bash
mkdir -p dataset
cd dataset
# Download required files
```
### 2. Dataset Directory Structure
```
./dataset/
├── pretrain_hq.jsonl ✨ (1.6GB, required for pretraining)
├── sft_mini_512.jsonl ✨ (1.2GB, fastest SFT)
├── sft_512.jsonl (7.5GB, standard SFT)
├── sft_1024.jsonl (5.6GB, longer SFT)
├── sft_2048.jsonl (9GB, very long SFT)
├── dpo.jsonl (909MB, DPO training)
├── r1_mix_1024.jsonl (340MB, reasoning distillation)
├── rlaif-mini.jsonl (1MB, RLAIF algorithms)
├── lora_identity.jsonl (22.8KB, identity LoRA)
└── lora_medical.jsonl (34MB, medical domain LoRA)
```
### 3. Data Formats
**Pretraining Data** (`pretrain_hq.jsonl`):
```json ```json
{"text": "How to overcome procrastination? Overcoming procrastination is not easy..."} {"text": "How to overcome procrastination? Overcoming procrastination is not easy, but these suggestions may help..."}
``` ```
**SFT Data** (`sft_*.jsonl`): **SFT Data** (`sft_*.jsonl`):
```json ```json
{ {
"conversations": [ "conversations": [
{"role": "user", "content": "Hello"}, {"role": "user", "content": "Hello!"},
{"role": "assistant", "content": "Hello!"} {"role": "assistant", "content": "Hello! How can I help?"},
{"role": "user", "content": "Tell me a joke."},
{"role": "assistant", "content": "Why did the scarecrow win an award? Because he was outstanding in his field!"}
] ]
} }
``` ```
## 🎯 Training Pipeline **DPO Data** (`dpo.jsonl`):
```json
{
"chosen": [
{"role": "user", "content": "What is 2+2?"},
{"role": "assistant", "content": "2+2 equals 4."}
],
"rejected": [
{"role": "user", "content": "What is 2+2?"},
{"role": "assistant", "content": "2+2 equals 5."}
]
}
```
All training scripts are located in the `./trainer` directory. **LoRA Domain Data** (`lora_*.jsonl`):
```json
{
"conversations": [
{"role": "user", "content": "What's the treatment for cervical spondylosis?"},
{"role": "assistant", "content": "Cervical spondylosis treatment typically includes..."}
]
}
```
### 1. Pretraining ## 🎯 Complete Training Pipeline
The pretraining stage lets the model learn basic knowledge, the goal is to **learn word continuation**. All training scripts are in the `./trainer` directory.
```bash ```bash
cd trainer cd trainer
python train_pretrain.py
# Multi-GPU training
torchrun --nproc_per_node 2 train_pretrain.py
``` ```
Output weights: `./out/pretrain_*.pth` ### Stage 1: Pretraining
!!! info "Training Duration" **Purpose**: Learn foundational knowledge (word continuation)
- MiniMind2-Small (26M): ~1.1h (single 3090)
- MiniMind2 (104M): ~3.9h (single 3090)
### 2. Supervised Fine-Tuning (SFT)
The SFT stage teaches the model conversation patterns and adapts to chat templates.
```bash ```bash
# Single GPU
python train_pretrain.py
# Multi-GPU (DDP)
torchrun --nproc_per_node 2 train_pretrain.py
# Multi-GPU (DeepSpeed)
deepspeed --master_port 29500 --num_gpus=2 train_pretrain.py
```
**Key Parameters**:
- `max_seq_len`: 512 (adjust based on GPU memory)
- `learning_rate`: 1e-4
- `epochs`: Adjust based on dataset size
**Output**: `./out/pretrain_*.pth`
**Training Duration**:
- MiniMind2-Small (26M): ~1.1h
- MiniMind2 (104M): ~3.9h
!!! tip "Pretraining Tips"
- Start with `pretrain_hq.jsonl` for best results
- Quality > Quantity for pretraining data
- Monitor loss curve to detect overfitting
### Stage 2: Supervised Fine-Tuning (SFT)
**Purpose**: Teach conversation patterns and chat templates
```bash
# Single GPU
python train_full_sft.py python train_full_sft.py
# Multi-GPU training # Multi-GPU
torchrun --nproc_per_node 2 train_full_sft.py torchrun --nproc_per_node 2 train_full_sft.py
``` ```
Output weights: `./out/full_sft_*.pth` **Configuration**:
- Load pretrained model from Stage 1
- Use SFT dataset (`sft_mini_512.jsonl` or `sft_512.jsonl`)
- Adjust `max_seq_len` to match training data
!!! info "Training Duration" **Output**: `./out/full_sft_*.pth`
- MiniMind2-Small: ~1h (using sft_mini_512)
- MiniMind2: ~3.3h (using sft_mini_512)
### 3. LoRA Fine-tuning (Optional) **Training Duration**:
- With sft_mini_512: 1-3 hours
- With full sft_512: 20-25 hours
LoRA is a parameter-efficient fine-tuning method, suitable for domain adaptation. !!! warning "SFT Data Selection"
- `sft_mini_512.jsonl`: Fastest, ~1.2GB, 512 tokens max
- `sft_512.jsonl`: Standard, ~7.5GB, 512 tokens max
- `sft_1024.jsonl`: Longer, ~5.6GB, 1024 tokens max
- `sft_2048.jsonl`: Extended, ~9GB, 2048 tokens max
### Stage 3: LoRA Fine-Tuning (Optional)
**Purpose**: Parameter-efficient domain adaptation
**Use Cases**:
- Medical Q&A knowledge
- Personal identity/self-awareness
- Proprietary domain knowledge
```bash ```bash
# Edit train_lora.py to set correct dataset and base model
python train_lora.py
# Multi-GPU
torchrun --nproc_per_node 2 train_lora.py
```
**Output**: `./out/lora/lora_*.pth`
**Example 1: Medical Domain**
Prepare `dataset/lora_medical.jsonl`:
```json
{
"conversations": [
{"role": "user", "content": "What's the correct pillow height for cervical spondylosis?"},
{"role": "assistant", "content": "For cervical spondylosis, pillow height should be..."}
]
}
```
Train:
```bash
# Modify train_lora.py: lora_name = 'medical'
python train_lora.py python train_lora.py
``` ```
**Use Cases**: **Example 2: Identity/Self-Awareness**
- Medical Q&A: use `lora_medical.jsonl`
- Self-awareness: use `lora_identity.jsonl`
Output weights: `./out/lora/lora_*.pth` Prepare `dataset/lora_identity.jsonl`:
```json
{
"conversations": [
{"role": "user", "content": "Who are you?"},
{"role": "assistant", "content": "I am MiniMind..."}
]
}
```
### 4. DPO Reinforcement Learning (Optional) ### Stage 4: Direct Preference Optimization (DPO)
DPO is used to optimize model response quality to better align with human preferences. **Purpose**: Align model responses with human preferences
DPO eliminates the need for separate reward models by directly optimizing preference pairs.
```bash ```bash
python train_dpo.py python train_dpo.py
# Multi-GPU
torchrun --nproc_per_node 2 train_dpo.py
``` ```
Output weights: `./out/rlhf_*.pth` **Output**: `./out/rlhf_*.pth`
### 5. Reasoning Model Distillation (Optional) **Key Features**:
- Off-policy training (reuse data across epochs)
- No separate reward model needed
- Better sample efficiency than PPO
- Stable training convergence
Distill reasoning capabilities from DeepSeek-R1. **Training Duration**: ~1-3 hours
### Stage 5: Reinforcement Learning from AI Feedback (RLAIF)
RLAIF is an advanced training approach using AI-generated rewards instead of human annotations. MiniMind implements three modern algorithms:
#### 5.1 PPO (Proximal Policy Optimization)
Classical on-policy RL algorithm with proven stability.
```bash
python train_ppo.py
# Multi-GPU
torchrun --nproc_per_node 2 train_ppo.py
```
**Algorithm**:
$$\mathcal{L}_{PPO} = -\mathbb{E}\left[\min(r_t \cdot A_t, \text{clip}(r_t, 1-\varepsilon, 1+\varepsilon) \cdot A_t)\right] + \beta \cdot \mathbb{E}[\text{KL}]$$
**Characteristics**:
- Stable but slower reward improvement
- Requires both Actor and Critic networks
- High memory usage (1.5-2× single network)
- Good for exploration
**Output**: `./out/ppo_actor_*.pth`
**Training Duration**: ~1-3 hours
#### 5.2 GRPO (Group Relative Policy Optimization)
Modern algorithm used in DeepSeek-R1, with faster convergence.
```bash
python train_grpo.py
# Multi-GPU
torchrun --nproc_per_node 2 train_grpo.py
```
**Algorithm**:
$$\mathcal{L}_{GRPO} = -\mathbb{E}\left[r_t \cdot A_t - \beta \cdot \text{KL}_t\right]$$
Where advantage is computed as:
$$A_t = \frac{R - \mu_{group}}{\sigma_{group}}$$
**Characteristics**:
- Single-network design (memory efficient)
- Faster reward improvement
- Group normalization removes bias
- Better convergence stability
**Output**: `./out/grpo_*.pth`
**Training Duration**: ~1-3 hours
#### 5.3 SPO (Single-stream Policy Optimization)
Newest algorithm (2025) addressing GRPO's degenerate group problem.
```bash
python train_spo.py
# Multi-GPU
torchrun --nproc_per_node 2 train_spo.py
```
**Algorithm**:
$$\mathcal{L}_{SPO} = -\mathbb{E}\left[\log \pi_\theta(a_t|s) \cdot A_t - \beta \cdot \text{KL}_t\right]$$
With adaptive baseline: $B_t^{adaptive}$
**Characteristics**:
- No group dependency (1 input → 1 training sample)
- Adaptive value tracking
- Better handling of difficult examples
- Experimental on small models
**Output**: `./out/spo_*.pth`
**Training Duration**: ~1-3 hours
#### RLAIF Dataset Preparation
All RLAIF algorithms use `rlaif-mini.jsonl` (1MB, 10k examples):
```bash
# Download dataset
# Format: Same as SFT, but assistant content is "无" (none)
{
"conversations": [
{"role": "user", "content": "Explain photosynthesis briefly."},
{"role": "assistant", "content": "无"}
]
}
```
The model generates completions during training, which are scored by a **Reward Model** (e.g., InternLM2-1.8B-Reward).
**Reward Model Setup**:
```bash
# Download reward model to parent directory
cd ../
git clone https://huggingface.co/internlm/internlm2-1_8b-reward
# Directory structure should be:
# project/
# ├── minimind/
# └── internlm2-1_8b-reward/
```
#### RLAIF vs DPO Comparison
| Aspect | DPO | RLAIF (PPO/GRPO/SPO) |
|--------|-----|---------------------|
| Training Type | Off-policy | On-policy |
| Data Freshness | Static pairs | Dynamic (generated) |
| Reward Source | Implicit | Explicit model |
| Convergence | Fast | Slower |
| Memory Usage | Lower | Higher |
| Best For | Preference refinement | Capability improvement |
### Stage 6: Reasoning Model Distillation
**Purpose**: Distill DeepSeek-R1-style reasoning into MiniMind
```bash ```bash
python train_distill_reason.py python train_distill_reason.py
# Multi-GPU
torchrun --nproc_per_node 2 train_distill_reason.py
``` ```
Output weights: `./out/reason_*.pth` **Data Format** (`r1_mix_1024.jsonl`):
```json
## 📈 Model Architecture {
"conversations": [
MiniMind uses Transformer Decoder-Only architecture (similar to Llama3): {
"role": "user",
![structure](images/LLM-structure.png) "content": "Solve: 5 + 3 = ?"
},
### Model Parameter Configuration {
"role": "assistant",
| Model Name | params | d_model | n_layers | kv_heads | q_heads | "content": "<think>\nI need to add 5 and 3.\n5 + 3 = 8\n</think>\n<answer>\n5 + 3 = 8\n</answer>"
|------------|--------|---------|----------|----------|---------| }
| MiniMind2-Small | 26M | 512 | 8 | 2 | 8 | ]
| MiniMind2-MoE | 145M | 640 | 8 | 2 | 8 | }
| MiniMind2 | 104M | 768 | 16 | 2 | 8 |
## 🧪 Test Model
```bash
# model_mode: 0=pretrain, 1=sft, 2=rlhf, 3=reason
python eval_model.py --model_mode 1
# Test LoRA model
python eval_model.py --lora_name 'lora_medical' --model_mode 2
``` ```
**Output**: `./out/reason_*.pth`
**Training Features**:
- Enforces `<think>` and `<answer>` tags
- Penalty loss for format violations
- Mixed data (reasoning + multi-turn + English)
## 🔧 Multi-GPU Training ## 🔧 Multi-GPU Training
### DDP Method ### DDP (Distributed Data Parallel)
Best for single-machine multi-GPU:
```bash ```bash
torchrun --nproc_per_node N train_xxx.py torchrun --nproc_per_node N train_xxx.py
# N = number of GPUs
``` ```
### DeepSpeed Method ### DeepSpeed
For advanced optimization:
```bash ```bash
deepspeed --master_port 29500 --num_gpus=N train_xxx.py deepspeed --master_port 29500 --num_gpus=N train_xxx.py
@ -157,30 +421,259 @@ deepspeed --master_port 29500 --num_gpus=N train_xxx.py
### Wandb Monitoring ### Wandb Monitoring
Track training progress:
```bash ```bash
# Login first # Login first
wandb login wandb login
# Enable wandb # Enable wandb logging
torchrun --nproc_per_node N train_xxx.py --use_wandb torchrun --nproc_per_node N train_xxx.py --use_wandb
# Or SwanLab (China-friendly alternative)
python train_xxx.py --use_wandb # Automatically uses SwanLab if available
``` ```
## 💰 Training Cost ## 🧪 Model Testing
Based on single NVIDIA 3090: ### Evaluate Pretrain Model
| Dataset Combination | Duration | Cost | Effect | ```bash
|-----------|------|------|------| python eval_model.py --model_mode 0
| pretrain_hq + sft_mini_512 | 2.1h | ≈$0.35 | 😊😊 Basic chat | ```
| Full dataset (MiniMind2-Small) | 38h | ≈$6.50 | 😊😊😊😊😊😊 Complete capabilities |
| Full dataset (MiniMind2) | 122h | ≈$20.80 | 😊😊😊😊😊😊😊😊 Best performance |
!!! success "Quick Reproduction" ### Evaluate Chat Model
Using `pretrain_hq` + `sft_mini_512`, single 3090 only needs **2 hours + $0.5** to train a ChatBot!
## 📝 Common Issues ```bash
python eval_model.py --model_mode 1
```
- **Out of memory**: Reduce `batch_size` or use DeepSpeed ### Evaluate with LoRA
- **Training not converging**: Adjust learning rate or check data quality
- **Multi-GPU training error**: Ensure all GPUs are visible and CUDA versions are consistent ```bash
python eval_model.py --lora_name 'lora_medical' --model_mode 1
```
### Evaluate Reasoning Model
```bash
python eval_model.py --model_mode 3
```
### Evaluate RLAIF Models
```bash
# PPO model
python eval_model.py --model_mode 4
# GRPO model
python eval_model.py --model_mode 4
```
### RoPE Length Extrapolation
Test with extended context:
```bash
python eval_model.py --model_mode 1 --inference_rope_scaling True
```
## 📐 Model Architecture
### MiniMind Structure
**Decoder-Only Transformer** (similar to Llama3):
```
Input Tokens
↓
Token Embedding (6400 vocab)
↓
Rotary Embeddings (RoPE) [with YaRN for length extrapolation]
↓
[Transformer Blocks] ×N
├─ Attention (Multi-Head)
├─ RMSNorm
├─ SwiGLU FFN [or MoE for MoE variant]
└─ Residual Connections
↓
RMSNorm
↓
LM Head (→ 6400 vocab logits)
↓
Output Probabilities
```
### Model Configurations
| Config | MiniMind2-Small | MiniMind2 | MiniMind2-MoE |
|--------|-----------------|----------|---------------|
| Parameters | 26M | 104M | 145M |
| Hidden Dim | 512 | 768 | 640 |
| Layers | 8 | 16 | 8 |
| KV Heads | 2 | 2 | 2 |
| Q Heads | 8 | 8 | 8 |
| Vocab Size | 6,400 | 6,400 | 6,400 |
| Context Length | 2,048 | 2,048 | 2,048 |
### Modify Architecture
Edit `./model/LMConfig.py`:
```python
class LMConfig:
hidden_size: int = 768
num_layers: int = 16
num_heads: int = 8
num_kv_heads: int = 2
# ... other configs
```
## 🔍 Training Tips & Best Practices
### Data Quality > Quantity
- High-quality pretraining data accelerates convergence
- `pretrain_hq.jsonl` is carefully curated for quality
- Consider data deduplication and cleaning
### Learning Rate Scheduling
```python
# Recommended schedules
- Linear warmup then decay
- Initial: 1e-4 to 5e-4
- Warmup steps: 10% of total
- Final: 10% of initial LR
```
### Batch Size & Sequence Length
```python
# Balance between GPU memory and convergence
- Pretraining: max_seq_len=512, batch_size=32
- SFT: max_seq_len=512, batch_size=16
- LoRA: max_seq_len=512, batch_size=16
```
### Memory Optimization
```bash
# Reduce batch size if OOM
python train_xxx.py --batch_size 8
# Or use gradient accumulation
python train_xxx.py --gradient_accumulation_steps 4
```
### Checkpoint Management
- Saves every 100 steps by default
- Each new save overwrites the old one
- Automatic backup before training
## 🚨 Common Issues & Solutions
### Issue: CUDA Out of Memory
```bash
# Solution 1: Reduce batch size
python train_xxx.py --batch_size 4
# Solution 2: Use gradient accumulation
python train_xxx.py --batch_size 16 --gradient_accumulation_steps 2
# Solution 3: Use smaller model
# Edit trainer script to use MiniMind2-Small instead
```
### Issue: Training Not Converging
```python
# Possible causes:
1. Learning rate too high/low
2. Data quality issues
3. Model capacity mismatch
# Solutions:
- Reduce learning rate: --learning_rate 1e-5
- Check data format and quality
- Try smaller model first
```
### Issue: Multi-GPU Sync Errors
```bash
# Ensure:
1. All GPUs visible: nvidia-smi
2. Same CUDA version across all GPUs
3. Network connectivity for distributed training
# Debug:
torchrun --nproc_per_node 2 train_xxx.py --debug
```
### Issue: Different Results Than Expected
```python
# Check:
1. Random seed set (reproducibility)
2. Correct model checkpoint loaded
3. Correct dataset being used
4. Same hyperparameters as reference
```
## 📈 Training Progression
Typical training curves:
```
Pretraining Loss: ↘↘↘ (steep decline, then plateau)
SFT Loss: ↘ (steady decline)
PPO Reward: ↗ (rising, may plateau)
GRPO Reward: ↗↗ (faster rise, more stable)
```
## 🎓 Advanced Topics
### Custom Datasets
Create your own dataset:
```python
# Format: JSONL with conversations list
# Each line is one training example
# Ensure consistent quality and format
```
### Model Quantization (Post-training)
```bash
# 4-bit quantization for inference
# Use tools like:
# - llama.cpp (gguf format)
# - bitsandbytes (dynamic quantization)
# - AutoGPTQ (static quantization)
```
### Model Merging
```python
# Merge base model + LoRA weights
# Use tools like: peft, llama.cpp
```
## 📚 References
- [Scaling Laws](https://arxiv.org/pdf/2001.08361.pdf)
- [RoPE Position Embeddings](https://arxiv.org/abs/2104.09864)
- [YaRN Length Extrapolation](https://arxiv.org/abs/2309.00071)
- [PPO Algorithm](https://arxiv.org/abs/1707.06347)
- [GRPO (DeepSeek)](https://arxiv.org/pdf/2402.03300)
- [SPO Algorithm](https://arxiv.org/abs/2509.13232)
- [DPO](https://arxiv.org/abs/2305.18290)
---
**Next**: Deploy your trained model or explore [advanced inference options](quickstart.md#third-party-inference-frameworks)