- 
-
- 
- 
- 
- 
- 
- 
+
-
-
-
-
-
-
+
+
+
+
-
-
-
+ +
-
"The Greatest Path is the Simplest"
- 🔗 MiniMind Github Repo 🔗 + +
+
+
+
+
-
+
+
+ +
+
-
+ Train a 26M ChatBot from zero.
+ 2 hours. ¥3. One 3090.
+ That's it.
+
-
-
- The MiniMind series is extremely lightweight, with the smallest version being 1/7000 the - size of GPT-3, making it possible to train quickly on even the most ordinary personal GPUs. - -
- The project also open-sources the minimalist structure of the large model, including extensions for shared - mixed experts (MoE), dataset cleaning, pretraining, supervised fine-tuning (SFT), LoRA fine-tuning, direct - preference optimization (DPO) algorithms, and model distillation algorithms, along with the full code of the - process. - -
- MiniMind also expands into vision multimodal VLM: MiniMind-V. - -
- All core algorithm code is reconstructed from scratch using native PyTorch! It does not rely on abstract - interfaces provided by third-party libraries. - -
- This is not only a full-stage open-source reproduction of a large language model but also a tutorial for - beginners in LLM. - -
- We hope this project will serve as an inspiring example for everyone, helping to enjoy the fun of creation - and promoting the progress of the wider AI community! - - -
- To avoid misunderstanding, the "2 hours" test is based on NVIDIA 3090 hardware (single - GPU), and the "3 - RMB" refers to the GPU server rental cost. Details of the specifications can be found below. - -
-
-
- 
- 
- 
+
+
-
+
+
-
-
-
+
+
+
- 🔗🍓Reason Model | - 🔗🤖Standard Model | - 🔗🎞️Video Introduction -
-
+
+
+
+
+
- 26M
+Parameters
- 🔗🍓Reason Model | - 🔗🤖Standard Model | - 🔗🎞️Video Introduction -
-
+
+
+ 2h
+Training
+
+
+ ¥3
+Cost
+
+
+ 1/7000
+vs GPT-3
+✨ What You Get
+
+
+
+
+
+
+ 💰 Ultra-Cheap
+One 3090. Zero to ChatBot in 2 hours, costing just ¥3.
+
+
+ 📦 Full Stack
+Complete pipeline: Tokenizer → Pretrain → SFT → LoRA → PPO/GRPO/SPO
+
+
+ 🔬 Latest RL (2025)
+PPO, GRPO, SPO + YaRN length extrapolation. Native PyTorch implementation.
+
+
+ 📖 Learn by Reading
+Pure PyTorch. No black boxes. Understand every line of code.
+
+
+ 🔌 Plug & Play
+Compatible with vLLM, ollama, llama.cpp, transformers.
+
+
+ ⚡ OpenAI API
+Drop-in replacement for FastGPT, Open-WebUI, Dify.
+📦 Models
+
+
+
+
+
+
+
+ | Model | +Parameters | +Hidden Dim | +Layers | +Memory | +
|---|---|---|---|---|
| MiniMind2-Small | +26M | +512 | +8 | +~0.5 GB | +
| MiniMind2 | +104M | +768 | +16 | +~1.0 GB | +
| MiniMind2-MoE | +145M | +640 | +8 | +~1.0 GB | +
📰 What's New
+
+
+
+
+
+
+
+
+ 🔥 2025-10-24 (Latest)
+
+
+
+
+ -
+
- 🔥 RLAIF algorithms: PPO, GRPO, SPO (native PyTorch) +
- YaRN algorithm for RoPE length extrapolation +
- Adaptive Thinking in reasoning models +
- Tool Calling & Reasoning tags support +
- SwanLab integration (WandB alternative for China) +
- Bug fixes and performance improvements +
+
+
+
+ ⚙️ 2025-04-26 (Major Refactor)
+
+
+
+
+ -
+
- Model parameter renaming (align with Transformers) +
- Generate method refactor (GenerationMixin) +
- ✅ llama.cpp, vllm, ollama support +
- Vocab update: <s></s> → <|im_start|><|im_end|> +
- Code structure standardization +
+
+
+
+ 🎉 2025-02-09 (MiniMind2 Release)
+
+
+
+
+ -
+
- Complete codebase rewrite +
- MiniMind2 series: 26M, 104M, 145M models +
- JSONL data format (no preprocessing needed) +
- LoRA from scratch (no peft dependency) +
- DPO native PyTorch implementation +
- White-box model distillation +
- DeepSeek-R1 distillation models +
- HQ pretraining data (2h on single 3090) +
+
+
+
+ 🎬 2024-10-05 (Vision Multimodal)
+
+
+
+
+ -
+
- Vision multimodal support +
- MiniMind-V project launch +
+
+
+
+ 🔧 2024-09-27 (Data Preprocessing)
+
+
+
+
+ -
+
- Pretrain dataset preprocessing update +
- Text integrity preservation +
- Code cleanup +
+
+
+
+ 📝 2024-09-17 (MoE & Tokenizer)
+
+
+
+
+ -
+
- MiniMind-V1-MoE model release +
- Standardized minimind_tokenizer +
- Removed mistral_tokenizer variants +
+
+
+
+ 🚀 2024-09-01 (V1 Release)
+
+
+
+
+ -
+
- MiniMind-V1 (108M) release +
- 3 epochs pretrain + 10 epochs SFT +
- ModelScope online demo +
+
+
+ ⭐ 2024-08-27 (First Release)
+
+
+
+
+ -
+
- 🚀 First open-source release +
- MiniMind project launched +
🎮 Inside MiniMind
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ 🎯 Try It Online
+ + + + +📦 Get the Code
+ +
+
+
+
+
💡 Why MiniMind?
+-
+
- Train from zero, not just finetune +
- Pure PyTorch—no magic, no black boxes +
- Understand by building, not by reading docs +
- Works on your laptop—no cloud GPUs needed +
- 2025 RLAIF algorithms: PPO, GRPO, SPO +
- OpenAI API compatible—plug into any UI +
- Vision support via MiniMind-V +
- Code you can actually read and modify +
+ 💭 "Building a Lego plane beats flying first class." +
+