---

---

"The Greatest Path is the Simplest"

🔗MiniMind Github Repo🔗

• This open-source project aims to train a miniature language model MiniMind from scratch, with a size of just 26MB.

• MiniMind is extremely lightweight, approximately 1/7000 the size of GPT-3, designed to enable fast inference and even training on CPUs.

• MiniMind is an improvement on the DeepSeek-V2 and Llama3 architectures. The project includes all stages of data processing, pretraining, SFT, and DPO, and features a Mixture of Experts (MoE) model.

• This is not only the implementation of an open-source model, but also a tutorial for getting started with large language models (LLMs).

• We hope that this project serves as a stepping stone for researchers and developers, providing an introductory example to help them quickly get started and foster more exploration and innovation in the LLM field.

  To avoid any misunderstanding, "fastest 3 hours" refers to the requirement of using hardware with higher specifications than the author's setup. Detailed specifications will be provided below.

---

ModelScope Online | Bilibili Video Link

---