2.3 KiB
2.3 KiB
Welcome to MiniMind!
📌 Introduction
MiniMind is a super-small language model project trained completely from scratch, requiring only $0.5 + 2 hours to train a 26M language model!
- MiniMind series is extremely lightweight, the smallest version is 1/7000 the size of GPT-3
- The project open-sources the minimalist structure of large models, including:
- Mixture of Experts (MoE)
- Dataset cleaning
- Pretraining
- Supervised Fine-Tuning (SFT)
- LoRA fine-tuning
- Direct Preference Optimization (DPO)
- Model distillation
- All core algorithm code is reconstructed from scratch using native PyTorch, without relying on third-party abstract interfaces
- This is not only a full-stage open-source reproduction of large language models, but also a tutorial for getting started with LLMs
!!! note "Training Cost" "2 hours" is based on NVIDIA 3090 hardware (single card) testing, "$0.5" refers to GPU server rental cost
✨ Key Features
- Ultra-low cost: Single 3090, 2 hours, $0.5 to train a ChatBot from scratch
- Complete pipeline: Covers Tokenizer, pretraining, SFT, LoRA, DPO, distillation full process
- Education-friendly: Clean code, suitable for learning LLM principles
- Ecosystem compatible: Supports
transformers,llama.cpp,vllm,ollamaand other mainstream frameworks
📊 Model List
| Model (Size) | Inference Memory (Approx.) | Release |
|---|---|---|
| MiniMind2-small (26M) | 0.5 GB | 2025.04.26 |
| MiniMind2-MoE (145M) | 1.0 GB | 2025.04.26 |
| MiniMind2 (104M) | 1.0 GB | 2025.04.26 |
🚀 Quick Navigation
- Quick Start - Environment setup, model download, quick testing
- Model Training - Pretraining, SFT, LoRA, DPO training process
🔗 Related Links
- GitHub: https://github.com/jingyaogong/minimind
- HuggingFace: MiniMind Collection
- ModelScope: MiniMind Models
- Online Demo: ModelScope Studio