fix i18n
jingyaogong
parent
3c5feebf3f
commit
941db7a5e6
|
|
@ -1,53 +0,0 @@
|
||||||
# <strong>Welcome to MiniMind!</strong>
|
|
||||||
|
|
||||||
<figure markdown>
|
|
||||||
|
|
||||||
<figcaption><strong>"Simplicity is the ultimate sophistication"</strong></figcaption>
|
|
||||||
</figure>
|
|
||||||
|
|
||||||
## 📌 Introduction
|
|
||||||
|
|
||||||
MiniMind is a super-small language model project trained completely from scratch, requiring **only $0.5 + 2 hours** to train a **26M** language model!
|
|
||||||
|
|
||||||
- **MiniMind** series is extremely lightweight, the smallest version is **1/7000** the size of GPT-3
|
|
||||||
- The project open-sources the minimalist structure of large models, including:
|
|
||||||
- Mixture of Experts (MoE)
|
|
||||||
- Dataset cleaning
|
|
||||||
- Pretraining
|
|
||||||
- Supervised Fine-Tuning (SFT)
|
|
||||||
- LoRA fine-tuning
|
|
||||||
- Direct Preference Optimization (DPO)
|
|
||||||
- Model distillation
|
|
||||||
- All core algorithm code is reconstructed from scratch using native PyTorch, without relying on third-party abstract interfaces
|
|
||||||
- This is not only a full-stage open-source reproduction of large language models, but also a tutorial for getting started with LLMs
|
|
||||||
|
|
||||||
!!! note "Training Cost"
|
|
||||||
"2 hours" is based on NVIDIA 3090 hardware (single card) testing, "$0.5" refers to GPU server rental cost
|
|
||||||
|
|
||||||
## ✨ Key Features
|
|
||||||
|
|
||||||
- **Ultra-low cost**: Single 3090, 2 hours, $0.5 to train a ChatBot from scratch
|
|
||||||
- **Complete pipeline**: Covers Tokenizer, pretraining, SFT, LoRA, DPO, distillation full process
|
|
||||||
- **Education-friendly**: Clean code, suitable for learning LLM principles
|
|
||||||
- **Ecosystem compatible**: Supports `transformers`, `llama.cpp`, `vllm`, `ollama` and other mainstream frameworks
|
|
||||||
|
|
||||||
## 📊 Model List
|
|
||||||
|
|
||||||
| Model (Size) | Inference Memory (Approx.) | Release |
|
|
||||||
|------------|----------|---------|
|
|
||||||
| MiniMind2-small (26M) | 0.5 GB | 2025.04.26 |
|
|
||||||
| MiniMind2-MoE (145M) | 1.0 GB | 2025.04.26 |
|
|
||||||
| MiniMind2 (104M) | 1.0 GB | 2025.04.26 |
|
|
||||||
|
|
||||||
## 🚀 Quick Navigation
|
|
||||||
|
|
||||||
- [Quick Start](quickstart.en.md) - Environment setup, model download, quick testing
|
|
||||||
- [Model Training](training.en.md) - Pretraining, SFT, LoRA, DPO training process
|
|
||||||
|
|
||||||
## 🔗 Related Links
|
|
||||||
|
|
||||||
- **GitHub**: [https://github.com/jingyaogong/minimind](https://github.com/jingyaogong/minimind)
|
|
||||||
- **HuggingFace**: [MiniMind Collection](https://huggingface.co/collections/jingyaogong/minimind-66caf8d999f5c7fa64f399e5)
|
|
||||||
- **ModelScope**: [MiniMind Models](https://www.modelscope.cn/profile/gongjy)
|
|
||||||
- **Online Demo**: [ModelScope Studio](https://www.modelscope.cn/studios/gongjy/MiniMind)
|
|
||||||
|
|
||||||
|
|
@ -1,53 +1,53 @@
|
||||||
# <strong>Welcome to MiniMind!</strong>
|
# <strong>Welcome to MiniMind!</strong>
|
||||||
|
|
||||||
|
|
||||||
<figure markdown>
|
<figure markdown>
|
||||||
|
|
||||||
|
<figcaption><strong>"Simplicity is the ultimate sophistication"</strong></figcaption>
|
||||||
</figure>
|
</figure>
|
||||||
|
|
||||||
## 📌 项目简介
|
## 📌 Introduction
|
||||||
|
|
||||||
MiniMind 是一个完全从 0 开始训练的超小语言模型项目，**仅需 3 块钱成本 + 2 小时**即可训练出仅为 **26M** 的语言模型！
|
MiniMind is a super-small language model project trained completely from scratch, requiring **only $0.5 + 2 hours** to train a **26M** language model!
|
||||||
|
|
||||||
- **MiniMind** 系列极其轻量，最小版本体积是 GPT-3 的 **1/7000**
|
- **MiniMind** series is extremely lightweight, the smallest version is **1/7000** the size of GPT-3
|
||||||
- 项目开源了大模型的极简结构，包含：
|
- The project open-sources the minimalist structure of large models, including:
|
||||||
- 混合专家模型（MoE）
|
- Mixture of Experts (MoE)
|
||||||
- 数据集清洗
|
- Dataset cleaning
|
||||||
- 预训练（Pretrain）
|
- Pretraining
|
||||||
- 监督微调（SFT）
|
- Supervised Fine-Tuning (SFT)
|
||||||
- LoRA 微调
|
- LoRA fine-tuning
|
||||||
- 直接偏好优化（DPO）
|
- Direct Preference Optimization (DPO)
|
||||||
- 模型蒸馏
|
- Model distillation
|
||||||
- 所有核心算法代码均从 0 使用 PyTorch 原生重构，不依赖第三方抽象接口
|
- All core algorithm code is reconstructed from scratch using native PyTorch, without relying on third-party abstract interfaces
|
||||||
- 这不仅是大语言模型的全阶段开源复现，也是一个入门 LLM 的教程
|
- This is not only a full-stage open-source reproduction of large language models, but also a tutorial for getting started with LLMs
|
||||||
|
|
||||||
!!! note "训练成本"
|
!!! note "Training Cost"
|
||||||
"2小时" 基于 NVIDIA 3090 硬件设备（单卡）测试，"3块钱" 指 GPU 服务器租用成本
|
"2 hours" is based on NVIDIA 3090 hardware (single card) testing, "$0.5" refers to GPU server rental cost
|
||||||
|
|
||||||
## ✨ 主要特点
|
## ✨ Key Features
|
||||||
|
|
||||||
- **超低成本**：单卡 3090，2 小时，3 块钱即可从 0 训练 ChatBot
|
- **Ultra-low cost**: Single 3090, 2 hours, $0.5 to train a ChatBot from scratch
|
||||||
- **完整流程**：涵盖 Tokenizer、预训练、SFT、LoRA、DPO、蒸馏全流程
|
- **Complete pipeline**: Covers Tokenizer, pretraining, SFT, LoRA, DPO, distillation full process
|
||||||
- **教育友好**：代码简洁，适合学习 LLM 原理
|
- **Education-friendly**: Clean code, suitable for learning LLM principles
|
||||||
- **生态兼容**：支持 `transformers`、`llama.cpp`、`vllm`、`ollama` 等主流框架
|
- **Ecosystem compatible**: Supports `transformers`, `llama.cpp`, `vllm`, `ollama` and other mainstream frameworks
|
||||||
|
|
||||||
## 📊 模型列表
|
## 📊 Model List
|
||||||
|
|
||||||
| 模型 (大小) | 推理占用 (约) | Release |
|
| Model (Size) | Inference Memory (Approx.) | Release |
|
||||||
|------------|----------|---------|
|
|------------|----------|---------|
|
||||||
| MiniMind2-small (26M) | 0.5 GB | 2025.04.26 |
|
| MiniMind2-small (26M) | 0.5 GB | 2025.04.26 |
|
||||||
| MiniMind2-MoE (145M) | 1.0 GB | 2025.04.26 |
|
| MiniMind2-MoE (145M) | 1.0 GB | 2025.04.26 |
|
||||||
| MiniMind2 (104M) | 1.0 GB | 2025.04.26 |
|
| MiniMind2 (104M) | 1.0 GB | 2025.04.26 |
|
||||||
|
|
||||||
## 🚀 快速导航
|
## 🚀 Quick Navigation
|
||||||
|
|
||||||
- [快速开始](quickstart.md) - 环境安装、模型下载、快速测试
|
- [Quick Start](quickstart.md) - Environment setup, model download, quick testing
|
||||||
- [模型训练](training.md) - 预训练、SFT、LoRA、DPO 等训练流程
|
- [Model Training](training.md) - Pretraining, SFT, LoRA, DPO training process
|
||||||
|
|
||||||
## 🔗 相关链接
|
## 🔗 Related Links
|
||||||
|
|
||||||
- **GitHub**: [https://github.com/jingyaogong/minimind](https://github.com/jingyaogong/minimind)
|
- **GitHub**: [https://github.com/jingyaogong/minimind](https://github.com/jingyaogong/minimind)
|
||||||
- **HuggingFace**: [MiniMind Collection](https://huggingface.co/collections/jingyaogong/minimind-66caf8d999f5c7fa64f399e5)
|
- **HuggingFace**: [MiniMind Collection](https://huggingface.co/collections/jingyaogong/minimind-66caf8d999f5c7fa64f399e5)
|
||||||
- **ModelScope**: [MiniMind 模型](https://www.modelscope.cn/profile/gongjy)
|
- **ModelScope**: [MiniMind Models](https://www.modelscope.cn/profile/gongjy)
|
||||||
- **在线体验**: [ModelScope 创空间](https://www.modelscope.cn/studios/gongjy/MiniMind)
|
- **Online Demo**: [ModelScope Studio](https://www.modelscope.cn/studios/gongjy/MiniMind)
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -1,114 +0,0 @@
|
||||||
# Quick Start
|
|
||||||
|
|
||||||
This page will help you quickly get started with the MiniMind project.
|
|
||||||
|
|
||||||
## 📋 Requirements
|
|
||||||
|
|
||||||
- **Python**: 3.10+
|
|
||||||
- **PyTorch**: 1.12+
|
|
||||||
- **CUDA**: 12.2+ (optional, for GPU acceleration)
|
|
||||||
- **VRAM**: At least 8GB (24GB recommended)
|
|
||||||
|
|
||||||
!!! tip "Hardware Configuration Reference"
|
|
||||||
- CPU: Intel i9-10980XE @ 3.00GHz
|
|
||||||
- RAM: 128 GB
|
|
||||||
- GPU: NVIDIA GeForce RTX 3090 (24GB)
|
|
||||||
|
|
||||||
## 🚀 Testing Existing Models
|
|
||||||
|
|
||||||
### 1. Clone the Project
|
|
||||||
|
|
||||||
```bash
|
|
||||||
git clone https://github.com/jingyaogong/minimind.git
|
|
||||||
cd minimind
|
|
||||||
```
|
|
||||||
|
|
||||||
### 2. Install Dependencies
|
|
||||||
|
|
||||||
```bash
|
|
||||||
pip install -r requirements.txt -i https://pypi.tuna.tsinghua.edu.cn/simple
|
|
||||||
```
|
|
||||||
|
|
||||||
!!! warning "Torch CUDA Check"
|
|
||||||
After installation, test if Torch can use CUDA:
|
|
||||||
```python
|
|
||||||
import torch
|
|
||||||
print(torch.cuda.is_available())
|
|
||||||
```
|
|
||||||
|
|
||||||
### 3. Download Model
|
|
||||||
|
|
||||||
Download pretrained models from HuggingFace or ModelScope:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# From HuggingFace
|
|
||||||
git clone https://huggingface.co/jingyaogong/MiniMind2
|
|
||||||
|
|
||||||
# Or from ModelScope
|
|
||||||
git clone https://www.modelscope.cn/models/gongjy/MiniMind2.git
|
|
||||||
```
|
|
||||||
|
|
||||||
### 4. Command Line Q&A
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# load=0: load PyTorch model, load=1: load Transformers model
|
|
||||||
python eval_model.py --load 1 --model_mode 2
|
|
||||||
```
|
|
||||||
|
|
||||||
### 5. Start WebUI (Optional)
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Requires Python >= 3.10
|
|
||||||
pip install streamlit
|
|
||||||
cd scripts
|
|
||||||
streamlit run web_demo.py
|
|
||||||
```
|
|
||||||
|
|
||||||
Visit `http://localhost:8501` to use the web interface.
|
|
||||||
|
|
||||||
## 🔧 Third-party Inference Frameworks
|
|
||||||
|
|
||||||
MiniMind supports multiple mainstream inference frameworks:
|
|
||||||
|
|
||||||
### Ollama
|
|
||||||
|
|
||||||
```bash
|
|
||||||
ollama run jingyaogong/minimind2
|
|
||||||
```
|
|
||||||
|
|
||||||
### vLLM
|
|
||||||
|
|
||||||
```bash
|
|
||||||
vllm serve ./MiniMind2/ --served-model-name "minimind"
|
|
||||||
```
|
|
||||||
|
|
||||||
### llama.cpp
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Convert model
|
|
||||||
python convert_hf_to_gguf.py ./MiniMind2/
|
|
||||||
|
|
||||||
# Quantize model
|
|
||||||
./build/bin/llama-quantize ./MiniMind2/MiniMind2-109M-F16.gguf ./Q4-MiniMind2.gguf Q4_K_M
|
|
||||||
|
|
||||||
# Inference
|
|
||||||
./build/bin/llama-cli -m ./Q4-MiniMind2.gguf --chat-template chatml
|
|
||||||
```
|
|
||||||
|
|
||||||
## 📝 Effect Testing
|
|
||||||
|
|
||||||
```text
|
|
||||||
👶: Hello, please introduce yourself.
|
|
||||||
🤖️: Hello! I'm MiniMind, an AI assistant developed by Jingyao Gong.
|
|
||||||
I interact with users through natural language processing and algorithm training.
|
|
||||||
|
|
||||||
👶: What is the highest mountain in the world?
|
|
||||||
🤖️: Mount Everest is the highest mountain in the world, located in the Himalayas,
|
|
||||||
with an elevation of 8,848.86 meters (29,031.7 feet).
|
|
||||||
```
|
|
||||||
|
|
||||||
## 🎯 Next Steps
|
|
||||||
|
|
||||||
- Check [Model Training](training.en.md) to learn how to train your own model from scratch
|
|
||||||
- Read the source code to understand LLM implementation principles
|
|
||||||
|
|
||||||
|
|
@ -1,74 +1,74 @@
|
||||||
# 快速开始
|
# Quick Start
|
||||||
|
|
||||||
本页面将帮助你快速上手 MiniMind 项目。
|
This page will help you quickly get started with the MiniMind project.
|
||||||
|
|
||||||
## 📋 环境要求
|
## 📋 Requirements
|
||||||
|
|
||||||
- **Python**: 3.10+
|
- **Python**: 3.10+
|
||||||
- **PyTorch**: 1.12+
|
- **PyTorch**: 1.12+
|
||||||
- **CUDA**: 12.2+（可选，用于 GPU 加速）
|
- **CUDA**: 12.2+ (optional, for GPU acceleration)
|
||||||
- **显存**: 至少 8GB（推荐 24GB）
|
- **VRAM**: At least 8GB (24GB recommended)
|
||||||
|
|
||||||
!!! tip "硬件配置参考"
|
!!! tip "Hardware Configuration Reference"
|
||||||
- CPU: Intel i9-10980XE @ 3.00GHz
|
- CPU: Intel i9-10980XE @ 3.00GHz
|
||||||
- RAM: 128 GB
|
- RAM: 128 GB
|
||||||
- GPU: NVIDIA GeForce RTX 3090 (24GB)
|
- GPU: NVIDIA GeForce RTX 3090 (24GB)
|
||||||
|
|
||||||
## 🚀 测试已有模型
|
## 🚀 Testing Existing Models
|
||||||
|
|
||||||
### 1. 克隆项目
|
### 1. Clone the Project
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
git clone https://github.com/jingyaogong/minimind.git
|
git clone https://github.com/jingyaogong/minimind.git
|
||||||
cd minimind
|
cd minimind
|
||||||
```
|
```
|
||||||
|
|
||||||
### 2. 安装依赖
|
### 2. Install Dependencies
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
pip install -r requirements.txt -i https://pypi.tuna.tsinghua.edu.cn/simple
|
pip install -r requirements.txt -i https://pypi.tuna.tsinghua.edu.cn/simple
|
||||||
```
|
```
|
||||||
|
|
||||||
!!! warning "Torch CUDA 检查"
|
!!! warning "Torch CUDA Check"
|
||||||
安装后请测试 Torch 是否可用 CUDA：
|
After installation, test if Torch can use CUDA:
|
||||||
```python
|
```python
|
||||||
import torch
|
import torch
|
||||||
print(torch.cuda.is_available())
|
print(torch.cuda.is_available())
|
||||||
```
|
```
|
||||||
|
|
||||||
### 3. 下载模型
|
### 3. Download Model
|
||||||
|
|
||||||
从 HuggingFace 或 ModelScope 下载预训练模型：
|
Download pretrained models from HuggingFace or ModelScope:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# 从 HuggingFace 下载
|
# From HuggingFace
|
||||||
git clone https://huggingface.co/jingyaogong/MiniMind2
|
git clone https://huggingface.co/jingyaogong/MiniMind2
|
||||||
|
|
||||||
# 或从 ModelScope 下载
|
# Or from ModelScope
|
||||||
git clone https://www.modelscope.cn/models/gongjy/MiniMind2.git
|
git clone https://www.modelscope.cn/models/gongjy/MiniMind2.git
|
||||||
```
|
```
|
||||||
|
|
||||||
### 4. 命令行问答
|
### 4. Command Line Q&A
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# load=0: 加载 PyTorch 模型, load=1: 加载 Transformers 模型
|
# load=0: load PyTorch model, load=1: load Transformers model
|
||||||
python eval_model.py --load 1 --model_mode 2
|
python eval_model.py --load 1 --model_mode 2
|
||||||
```
|
```
|
||||||
|
|
||||||
### 5. 启动 WebUI（可选）
|
### 5. Start WebUI (Optional)
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# 需要 Python >= 3.10
|
# Requires Python >= 3.10
|
||||||
pip install streamlit
|
pip install streamlit
|
||||||
cd scripts
|
cd scripts
|
||||||
streamlit run web_demo.py
|
streamlit run web_demo.py
|
||||||
```
|
```
|
||||||
|
|
||||||
访问 `http://localhost:8501` 即可使用 Web 界面。
|
Visit `http://localhost:8501` to use the web interface.
|
||||||
|
|
||||||
## 🔧 第三方推理框架
|
## 🔧 Third-party Inference Frameworks
|
||||||
|
|
||||||
MiniMind 支持多种主流推理框架：
|
MiniMind supports multiple mainstream inference frameworks:
|
||||||
|
|
||||||
### Ollama
|
### Ollama
|
||||||
|
|
||||||
|
|
@ -85,30 +85,30 @@ vllm serve ./MiniMind2/ --served-model-name "minimind"
|
||||||
### llama.cpp
|
### llama.cpp
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# 转换模型
|
# Convert model
|
||||||
python convert_hf_to_gguf.py ./MiniMind2/
|
python convert_hf_to_gguf.py ./MiniMind2/
|
||||||
|
|
||||||
# 量化模型
|
# Quantize model
|
||||||
./build/bin/llama-quantize ./MiniMind2/MiniMind2-109M-F16.gguf ./Q4-MiniMind2.gguf Q4_K_M
|
./build/bin/llama-quantize ./MiniMind2/MiniMind2-109M-F16.gguf ./Q4-MiniMind2.gguf Q4_K_M
|
||||||
|
|
||||||
# 推理
|
# Inference
|
||||||
./build/bin/llama-cli -m ./Q4-MiniMind2.gguf --chat-template chatml
|
./build/bin/llama-cli -m ./Q4-MiniMind2.gguf --chat-template chatml
|
||||||
```
|
```
|
||||||
|
|
||||||
## 📝 效果测试
|
## 📝 Effect Testing
|
||||||
|
|
||||||
```text
|
```text
|
||||||
👶: 你好，请介绍一下自己。
|
👶: Hello, please introduce yourself.
|
||||||
🤖️: 你好！我是 MiniMind，一个由 Jingyao Gong 开发的人工智能助手。
|
🤖️: Hello! I'm MiniMind, an AI assistant developed by Jingyao Gong.
|
||||||
我通过自然语言处理和算法训练来与用户进行交互。
|
I interact with users through natural language processing and algorithm training.
|
||||||
|
|
||||||
👶: 世界上最高的山峰是什么？
|
👶: What is the highest mountain in the world?
|
||||||
🤖️: 珠穆朗玛峰是世界上最高的山峰，位于喜马拉雅山脉，
|
🤖️: Mount Everest is the highest mountain in the world, located in the Himalayas,
|
||||||
海拔 8,848.86 米（29,031.7 英尺）。
|
with an elevation of 8,848.86 meters (29,031.7 feet).
|
||||||
```
|
```
|
||||||
|
|
||||||
## 🎯 下一步
|
## 🎯 Next Steps
|
||||||
|
|
||||||
- 查看 [模型训练](training.md) 了解如何从 0 开始训练自己的模型
|
- Check [Model Training](training.md) to learn how to train your own model from scratch
|
||||||
- 阅读源码了解 LLM 的实现原理
|
- Read the source code to understand LLM implementation principles
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -1,186 +0,0 @@
|
||||||
# Model Training
|
|
||||||
|
|
||||||
This page introduces how to train MiniMind language models from scratch.
|
|
||||||
|
|
||||||
## 📊 Data Preparation
|
|
||||||
|
|
||||||
### 1. Download Dataset
|
|
||||||
|
|
||||||
Download datasets from [ModelScope](https://www.modelscope.cn/datasets/gongjy/minimind_dataset/files) or [HuggingFace](https://huggingface.co/datasets/jingyaogong/minimind_dataset).
|
|
||||||
|
|
||||||
Create `./dataset` directory and place data files:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
./dataset/
|
|
||||||
├── pretrain_hq.jsonl (1.6GB, ✨Recommended)
|
|
||||||
├── sft_mini_512.jsonl (1.2GB, ✨Recommended)
|
|
||||||
├── sft_512.jsonl (7.5GB)
|
|
||||||
├── sft_1024.jsonl (5.6GB)
|
|
||||||
├── sft_2048.jsonl (9GB)
|
|
||||||
├── dpo.jsonl (909MB)
|
|
||||||
├── r1_mix_1024.jsonl (340MB)
|
|
||||||
└── lora_*.jsonl
|
|
||||||
```
|
|
||||||
|
|
||||||
!!! tip "Recommended Combination"
|
|
||||||
Fastest reproduction: `pretrain_hq.jsonl` + `sft_mini_512.jsonl`
|
|
||||||
|
|
||||||
**Single 3090 only needs 2 hours + $0.5!**
|
|
||||||
|
|
||||||
### 2. Data Format
|
|
||||||
|
|
||||||
**Pretrain Data** (`pretrain_hq.jsonl`):
|
|
||||||
```json
|
|
||||||
{"text": "How to overcome procrastination? Overcoming procrastination is not easy..."}
|
|
||||||
```
|
|
||||||
|
|
||||||
**SFT Data** (`sft_*.jsonl`):
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"conversations": [
|
|
||||||
{"role": "user", "content": "Hello"},
|
|
||||||
{"role": "assistant", "content": "Hello!"}
|
|
||||||
]
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
## 🎯 Training Pipeline
|
|
||||||
|
|
||||||
All training scripts are located in the `./trainer` directory.
|
|
||||||
|
|
||||||
### 1. Pretraining
|
|
||||||
|
|
||||||
The pretraining stage lets the model learn basic knowledge, the goal is to **learn word continuation**.
|
|
||||||
|
|
||||||
```bash
|
|
||||||
cd trainer
|
|
||||||
python train_pretrain.py
|
|
||||||
|
|
||||||
# Multi-GPU training
|
|
||||||
torchrun --nproc_per_node 2 train_pretrain.py
|
|
||||||
```
|
|
||||||
|
|
||||||
Output weights: `./out/pretrain_*.pth`
|
|
||||||
|
|
||||||
!!! info "Training Duration"
|
|
||||||
- MiniMind2-Small (26M): ~1.1h (single 3090)
|
|
||||||
- MiniMind2 (104M): ~3.9h (single 3090)
|
|
||||||
|
|
||||||
### 2. Supervised Fine-Tuning (SFT)
|
|
||||||
|
|
||||||
The SFT stage teaches the model conversation patterns and adapts to chat templates.
|
|
||||||
|
|
||||||
```bash
|
|
||||||
python train_full_sft.py
|
|
||||||
|
|
||||||
# Multi-GPU training
|
|
||||||
torchrun --nproc_per_node 2 train_full_sft.py
|
|
||||||
```
|
|
||||||
|
|
||||||
Output weights: `./out/full_sft_*.pth`
|
|
||||||
|
|
||||||
!!! info "Training Duration"
|
|
||||||
- MiniMind2-Small: ~1h (using sft_mini_512)
|
|
||||||
- MiniMind2: ~3.3h (using sft_mini_512)
|
|
||||||
|
|
||||||
### 3. LoRA Fine-tuning (Optional)
|
|
||||||
|
|
||||||
LoRA is a parameter-efficient fine-tuning method, suitable for domain adaptation.
|
|
||||||
|
|
||||||
```bash
|
|
||||||
python train_lora.py
|
|
||||||
```
|
|
||||||
|
|
||||||
**Use Cases**:
|
|
||||||
- Medical Q&A: use `lora_medical.jsonl`
|
|
||||||
- Self-awareness: use `lora_identity.jsonl`
|
|
||||||
|
|
||||||
Output weights: `./out/lora/lora_*.pth`
|
|
||||||
|
|
||||||
### 4. DPO Reinforcement Learning (Optional)
|
|
||||||
|
|
||||||
DPO is used to optimize model response quality to better align with human preferences.
|
|
||||||
|
|
||||||
```bash
|
|
||||||
python train_dpo.py
|
|
||||||
```
|
|
||||||
|
|
||||||
Output weights: `./out/rlhf_*.pth`
|
|
||||||
|
|
||||||
### 5. Reasoning Model Distillation (Optional)
|
|
||||||
|
|
||||||
Distill reasoning capabilities from DeepSeek-R1.
|
|
||||||
|
|
||||||
```bash
|
|
||||||
python train_distill_reason.py
|
|
||||||
```
|
|
||||||
|
|
||||||
Output weights: `./out/reason_*.pth`
|
|
||||||
|
|
||||||
## 📈 Model Architecture
|
|
||||||
|
|
||||||
MiniMind uses Transformer Decoder-Only architecture (similar to Llama3):
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
### Model Parameter Configuration
|
|
||||||
|
|
||||||
| Model Name | params | d_model | n_layers | kv_heads | q_heads |
|
|
||||||
|------------|--------|---------|----------|----------|---------|
|
|
||||||
| MiniMind2-Small | 26M | 512 | 8 | 2 | 8 |
|
|
||||||
| MiniMind2-MoE | 145M | 640 | 8 | 2 | 8 |
|
|
||||||
| MiniMind2 | 104M | 768 | 16 | 2 | 8 |
|
|
||||||
|
|
||||||
## 🧪 Test Model
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# model_mode: 0=pretrain, 1=sft, 2=rlhf, 3=reason
|
|
||||||
python eval_model.py --model_mode 1
|
|
||||||
|
|
||||||
# Test LoRA model
|
|
||||||
python eval_model.py --lora_name 'lora_medical' --model_mode 2
|
|
||||||
```
|
|
||||||
|
|
||||||
## 🔧 Multi-GPU Training
|
|
||||||
|
|
||||||
### DDP Method
|
|
||||||
|
|
||||||
```bash
|
|
||||||
torchrun --nproc_per_node N train_xxx.py
|
|
||||||
```
|
|
||||||
|
|
||||||
### DeepSpeed Method
|
|
||||||
|
|
||||||
```bash
|
|
||||||
deepspeed --master_port 29500 --num_gpus=N train_xxx.py
|
|
||||||
```
|
|
||||||
|
|
||||||
### Wandb Monitoring
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Login first
|
|
||||||
wandb login
|
|
||||||
|
|
||||||
# Enable wandb
|
|
||||||
torchrun --nproc_per_node N train_xxx.py --use_wandb
|
|
||||||
```
|
|
||||||
|
|
||||||
## 💰 Training Cost
|
|
||||||
|
|
||||||
Based on single NVIDIA 3090:
|
|
||||||
|
|
||||||
| Dataset Combination | Duration | Cost | Effect |
|
|
||||||
|-----------|------|------|------|
|
|
||||||
| pretrain_hq + sft_mini_512 | 2.1h | ≈$0.35 | 😊😊 Basic chat |
|
|
||||||
| Full dataset (MiniMind2-Small) | 38h | ≈$6.50 | 😊😊😊😊😊😊 Complete capabilities |
|
|
||||||
| Full dataset (MiniMind2) | 122h | ≈$20.80 | 😊😊😊😊😊😊😊😊 Best performance |
|
|
||||||
|
|
||||||
!!! success "Quick Reproduction"
|
|
||||||
Using `pretrain_hq` + `sft_mini_512`, single 3090 only needs **2 hours + $0.5** to train a ChatBot!
|
|
||||||
|
|
||||||
## 📝 Common Issues
|
|
||||||
|
|
||||||
- **Out of memory**: Reduce `batch_size` or use DeepSpeed
|
|
||||||
- **Training not converging**: Adjust learning rate or check data quality
|
|
||||||
- **Multi-GPU training error**: Ensure all GPUs are visible and CUDA versions are consistent
|
|
||||||
|
|
||||||
136
docs/training.md
136
docs/training.md
|
|
@ -1,19 +1,19 @@
|
||||||
# 模型训练
|
# Model Training
|
||||||
|
|
||||||
本页面介绍如何从 0 开始训练 MiniMind 语言模型。
|
This page introduces how to train MiniMind language models from scratch.
|
||||||
|
|
||||||
## 📊 数据准备
|
## 📊 Data Preparation
|
||||||
|
|
||||||
### 1. 下载数据集
|
### 1. Download Dataset
|
||||||
|
|
||||||
从 [ModelScope](https://www.modelscope.cn/datasets/gongjy/minimind_dataset/files) 或 [HuggingFace](https://huggingface.co/datasets/jingyaogong/minimind_dataset) 下载数据集。
|
Download datasets from [ModelScope](https://www.modelscope.cn/datasets/gongjy/minimind_dataset/files) or [HuggingFace](https://huggingface.co/datasets/jingyaogong/minimind_dataset).
|
||||||
|
|
||||||
创建 `./dataset` 目录并放入数据文件：
|
Create `./dataset` directory and place data files:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
./dataset/
|
./dataset/
|
||||||
├── pretrain_hq.jsonl (1.6GB, ✨推荐)
|
├── pretrain_hq.jsonl (1.6GB, ✨Recommended)
|
||||||
├── sft_mini_512.jsonl (1.2GB, ✨推荐)
|
├── sft_mini_512.jsonl (1.2GB, ✨Recommended)
|
||||||
├── sft_512.jsonl (7.5GB)
|
├── sft_512.jsonl (7.5GB)
|
||||||
├── sft_1024.jsonl (5.6GB)
|
├── sft_1024.jsonl (5.6GB)
|
||||||
├── sft_2048.jsonl (9GB)
|
├── sft_2048.jsonl (9GB)
|
||||||
|
|
@ -22,108 +22,108 @@
|
||||||
└── lora_*.jsonl
|
└── lora_*.jsonl
|
||||||
```
|
```
|
||||||
|
|
||||||
!!! tip "推荐组合"
|
!!! tip "Recommended Combination"
|
||||||
最快速度复现：`pretrain_hq.jsonl` + `sft_mini_512.jsonl`
|
Fastest reproduction: `pretrain_hq.jsonl` + `sft_mini_512.jsonl`
|
||||||
|
|
||||||
**单卡 3090 仅需 2 小时 + 3 块钱！**
|
**Single 3090 only needs 2 hours + $0.5!**
|
||||||
|
|
||||||
### 2. 数据格式
|
### 2. Data Format
|
||||||
|
|
||||||
**预训练数据** (`pretrain_hq.jsonl`):
|
**Pretrain Data** (`pretrain_hq.jsonl`):
|
||||||
```json
|
```json
|
||||||
{"text": "如何才能摆脱拖延症？治愈拖延症并不容易..."}
|
{"text": "How to overcome procrastination? Overcoming procrastination is not easy..."}
|
||||||
```
|
```
|
||||||
|
|
||||||
**SFT 数据** (`sft_*.jsonl`):
|
**SFT Data** (`sft_*.jsonl`):
|
||||||
```json
|
```json
|
||||||
{
|
{
|
||||||
"conversations": [
|
"conversations": [
|
||||||
{"role": "user", "content": "你好"},
|
{"role": "user", "content": "Hello"},
|
||||||
{"role": "assistant", "content": "你好！"}
|
{"role": "assistant", "content": "Hello!"}
|
||||||
]
|
]
|
||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
## 🎯 训练流程
|
## 🎯 Training Pipeline
|
||||||
|
|
||||||
所有训练脚本位于 `./trainer` 目录。
|
All training scripts are located in the `./trainer` directory.
|
||||||
|
|
||||||
### 1. 预训练（Pretrain）
|
### 1. Pretraining
|
||||||
|
|
||||||
预训练阶段让模型学习基础知识，目标是**学会词语接龙**。
|
The pretraining stage lets the model learn basic knowledge, the goal is to **learn word continuation**.
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
cd trainer
|
cd trainer
|
||||||
python train_pretrain.py
|
python train_pretrain.py
|
||||||
|
|
||||||
# 多卡训练
|
# Multi-GPU training
|
||||||
torchrun --nproc_per_node 2 train_pretrain.py
|
torchrun --nproc_per_node 2 train_pretrain.py
|
||||||
```
|
```
|
||||||
|
|
||||||
输出权重：`./out/pretrain_*.pth`
|
Output weights: `./out/pretrain_*.pth`
|
||||||
|
|
||||||
!!! info "训练时长"
|
!!! info "Training Duration"
|
||||||
- MiniMind2-Small (26M): ~1.1h (单卡 3090)
|
- MiniMind2-Small (26M): ~1.1h (single 3090)
|
||||||
- MiniMind2 (104M): ~3.9h (单卡 3090)
|
- MiniMind2 (104M): ~3.9h (single 3090)
|
||||||
|
|
||||||
### 2. 监督微调（SFT）
|
### 2. Supervised Fine-Tuning (SFT)
|
||||||
|
|
||||||
SFT 阶段让模型学习对话方式，适应聊天模板。
|
The SFT stage teaches the model conversation patterns and adapts to chat templates.
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
python train_full_sft.py
|
python train_full_sft.py
|
||||||
|
|
||||||
# 多卡训练
|
# Multi-GPU training
|
||||||
torchrun --nproc_per_node 2 train_full_sft.py
|
torchrun --nproc_per_node 2 train_full_sft.py
|
||||||
```
|
```
|
||||||
|
|
||||||
输出权重：`./out/full_sft_*.pth`
|
Output weights: `./out/full_sft_*.pth`
|
||||||
|
|
||||||
!!! info "训练时长"
|
!!! info "Training Duration"
|
||||||
- MiniMind2-Small: ~1h (使用 sft_mini_512)
|
- MiniMind2-Small: ~1h (using sft_mini_512)
|
||||||
- MiniMind2: ~3.3h (使用 sft_mini_512)
|
- MiniMind2: ~3.3h (using sft_mini_512)
|
||||||
|
|
||||||
### 3. LoRA 微调（可选）
|
### 3. LoRA Fine-tuning (Optional)
|
||||||
|
|
||||||
LoRA 是一种参数高效的微调方法，适合领域适配。
|
LoRA is a parameter-efficient fine-tuning method, suitable for domain adaptation.
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
python train_lora.py
|
python train_lora.py
|
||||||
```
|
```
|
||||||
|
|
||||||
**应用场景**：
|
**Use Cases**:
|
||||||
- 医疗问答：使用 `lora_medical.jsonl`
|
- Medical Q&A: use `lora_medical.jsonl`
|
||||||
- 自我认知：使用 `lora_identity.jsonl`
|
- Self-awareness: use `lora_identity.jsonl`
|
||||||
|
|
||||||
输出权重：`./out/lora/lora_*.pth`
|
Output weights: `./out/lora/lora_*.pth`
|
||||||
|
|
||||||
### 4. DPO 强化学习（可选）
|
### 4. DPO Reinforcement Learning (Optional)
|
||||||
|
|
||||||
DPO 用于优化模型回复质量，使其更符合人类偏好。
|
DPO is used to optimize model response quality to better align with human preferences.
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
python train_dpo.py
|
python train_dpo.py
|
||||||
```
|
```
|
||||||
|
|
||||||
输出权重：`./out/rlhf_*.pth`
|
Output weights: `./out/rlhf_*.pth`
|
||||||
|
|
||||||
### 5. 推理模型蒸馏（可选）
|
### 5. Reasoning Model Distillation (Optional)
|
||||||
|
|
||||||
蒸馏 DeepSeek-R1 的推理能力。
|
Distill reasoning capabilities from DeepSeek-R1.
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
python train_distill_reason.py
|
python train_distill_reason.py
|
||||||
```
|
```
|
||||||
|
|
||||||
输出权重：`./out/reason_*.pth`
|
Output weights: `./out/reason_*.pth`
|
||||||
|
|
||||||
## 📈 模型结构
|
## 📈 Model Architecture
|
||||||
|
|
||||||
MiniMind 使用 Transformer Decoder-Only 结构（类似 Llama3）：
|
MiniMind uses Transformer Decoder-Only architecture (similar to Llama3):
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
### 模型参数配置
|
### Model Parameter Configuration
|
||||||
|
|
||||||
| Model Name | params | d_model | n_layers | kv_heads | q_heads |
|
| Model Name | params | d_model | n_layers | kv_heads | q_heads |
|
||||||
|------------|--------|---------|----------|----------|---------|
|
|------------|--------|---------|----------|----------|---------|
|
||||||
|
|
@ -131,56 +131,56 @@ MiniMind 使用 Transformer Decoder-Only 结构（类似 Llama3）：
|
||||||
| MiniMind2-MoE | 145M | 640 | 8 | 2 | 8 |
|
| MiniMind2-MoE | 145M | 640 | 8 | 2 | 8 |
|
||||||
| MiniMind2 | 104M | 768 | 16 | 2 | 8 |
|
| MiniMind2 | 104M | 768 | 16 | 2 | 8 |
|
||||||
|
|
||||||
## 🧪 测试模型
|
## 🧪 Test Model
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# model_mode: 0=pretrain, 1=sft, 2=rlhf, 3=reason
|
# model_mode: 0=pretrain, 1=sft, 2=rlhf, 3=reason
|
||||||
python eval_model.py --model_mode 1
|
python eval_model.py --model_mode 1
|
||||||
|
|
||||||
# 测试 LoRA 模型
|
# Test LoRA model
|
||||||
python eval_model.py --lora_name 'lora_medical' --model_mode 2
|
python eval_model.py --lora_name 'lora_medical' --model_mode 2
|
||||||
```
|
```
|
||||||
|
|
||||||
## 🔧 多卡训练
|
## 🔧 Multi-GPU Training
|
||||||
|
|
||||||
### DDP 方式
|
### DDP Method
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
torchrun --nproc_per_node N train_xxx.py
|
torchrun --nproc_per_node N train_xxx.py
|
||||||
```
|
```
|
||||||
|
|
||||||
### DeepSpeed 方式
|
### DeepSpeed Method
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
deepspeed --master_port 29500 --num_gpus=N train_xxx.py
|
deepspeed --master_port 29500 --num_gpus=N train_xxx.py
|
||||||
```
|
```
|
||||||
|
|
||||||
### Wandb 监控
|
### Wandb Monitoring
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# 需要先登录
|
# Login first
|
||||||
wandb login
|
wandb login
|
||||||
|
|
||||||
# 启用 wandb
|
# Enable wandb
|
||||||
torchrun --nproc_per_node N train_xxx.py --use_wandb
|
torchrun --nproc_per_node N train_xxx.py --use_wandb
|
||||||
```
|
```
|
||||||
|
|
||||||
## 💰 训练成本
|
## 💰 Training Cost
|
||||||
|
|
||||||
基于单卡 NVIDIA 3090：
|
Based on single NVIDIA 3090:
|
||||||
|
|
||||||
| 数据集组合 | 时长 | 成本 | 效果 |
|
| Dataset Combination | Duration | Cost | Effect |
|
||||||
|-----------|------|------|------|
|
|-----------|------|------|------|
|
||||||
| pretrain_hq + sft_mini_512 | 2.1h | ≈2.73￥ | 😊😊 基础对话 |
|
| pretrain_hq + sft_mini_512 | 2.1h | ≈$0.35 | 😊😊 Basic chat |
|
||||||
| 完整数据集 (MiniMind2-Small) | 38h | ≈49.61￥ | 😊😊😊😊😊😊 完整能力 |
|
| Full dataset (MiniMind2-Small) | 38h | ≈$6.50 | 😊😊😊😊😊😊 Complete capabilities |
|
||||||
| 完整数据集 (MiniMind2) | 122h | ≈158.6￥ | 😊😊😊😊😊😊😊😊 最强性能 |
|
| Full dataset (MiniMind2) | 122h | ≈$20.80 | 😊😊😊😊😊😊😊😊 Best performance |
|
||||||
|
|
||||||
!!! success "极速复现"
|
!!! success "Quick Reproduction"
|
||||||
使用 `pretrain_hq` + `sft_mini_512`，单卡 3090 仅需 **2 小时 + 3 块钱**即可训练出能对话的 ChatBot！
|
Using `pretrain_hq` + `sft_mini_512`, single 3090 only needs **2 hours + $0.5** to train a ChatBot!
|
||||||
|
|
||||||
## 📝 常见问题
|
## 📝 Common Issues
|
||||||
|
|
||||||
- **显存不足**：减小 `batch_size` 或使用 DeepSpeed
|
- **Out of memory**: Reduce `batch_size` or use DeepSpeed
|
||||||
- **训练不收敛**：调整学习率或检查数据质量
|
- **Training not converging**: Adjust learning rate or check data quality
|
||||||
- **多卡训练报错**：确保所有卡都可见且 CUDA 版本一致
|
- **Multi-GPU training error**: Ensure all GPUs are visible and CUDA versions are consistent
|
||||||
|
|
||||||
|
|
|
||||||
29
mkdocs.yml
29
mkdocs.yml
|
|
@ -3,27 +3,10 @@ site_description: MiniMind - 轻量级语言模型训练框架 / Lightweight Lan
|
||||||
site_author: jingyaogong
|
site_author: jingyaogong
|
||||||
site_url: https://minimind.readthedocs.io/
|
site_url: https://minimind.readthedocs.io/
|
||||||
|
|
||||||
# 多语言配置
|
# 搜索插件配置
|
||||||
plugins:
|
plugins:
|
||||||
- search:
|
- search:
|
||||||
lang:
|
lang: en
|
||||||
- zh
|
|
||||||
- en
|
|
||||||
- i18n:
|
|
||||||
docs_structure: folder
|
|
||||||
fallback_to_default: true
|
|
||||||
reconfigure_material: true
|
|
||||||
reconfigure_search: true
|
|
||||||
languages:
|
|
||||||
- locale: zh
|
|
||||||
default: true
|
|
||||||
name: 简体中文
|
|
||||||
build: true
|
|
||||||
site_name: MiniMind
|
|
||||||
- locale: en
|
|
||||||
name: English
|
|
||||||
build: true
|
|
||||||
site_name: MiniMind
|
|
||||||
|
|
||||||
# 主题配置
|
# 主题配置
|
||||||
theme:
|
theme:
|
||||||
|
|
@ -57,13 +40,13 @@ theme:
|
||||||
- content.code.copy # 代码复制按钮
|
- content.code.copy # 代码复制按钮
|
||||||
- toc.follow # 目录跟随
|
- toc.follow # 目录跟随
|
||||||
- toc.integrate # 目录集成到左侧边栏
|
- toc.integrate # 目录集成到左侧边栏
|
||||||
language: zh
|
language: en
|
||||||
|
|
||||||
# 导航结构
|
# 导航结构
|
||||||
nav:
|
nav:
|
||||||
- 首页: index.md
|
- Home: index.md
|
||||||
- 快速开始: quickstart.md
|
- Quick Start: quickstart.md
|
||||||
- 模型训练: training.md
|
- Model Training: training.md
|
||||||
|
|
||||||
# Markdown 扩展
|
# Markdown 扩展
|
||||||
markdown_extensions:
|
markdown_extensions:
|
||||||
|
|
|
||||||
|
|
@ -1,3 +1,2 @@
|
||||||
mkdocs>=1.5.0
|
mkdocs>=1.5.0
|
||||||
mkdocs-material>=9.0.0
|
mkdocs-material>=9.0.0
|
||||||
mkdocs-static-i18n>=1.0.0
|
|
||||||
|
|
|
||||||
Loading…
Reference in New Issue