diff --git a/RESOURCES.md b/RESOURCES.md new file mode 100644 index 0000000..c274cb7 --- /dev/null +++ b/RESOURCES.md @@ -0,0 +1,137 @@ +# MiniMind Resources Collection + +> A curated collection of tutorials, tools, and learning resources for MiniMind - Train a 26M-parameter GPT from scratch in just 2h! + +## 📚 Official Documentation + +- **MiniMind Docs**: https://minimind.readthedocs.io +- **GitHub Pages**: https://jingyaogong.github.io/minimind +- **Main Repository**: https://github.com/jingyaogong/minimind + +## 🎯 Tutorials & Deep Dives + +### Core Concepts +- **MiniMind In-Depth**: https://github.com/hanz0809/MiniMind-in-Depth + - Comprehensive guide covering tokenizer, RoPE, MoE, KV Cache + - Pretraining, SFT, LoRA, and DPO techniques + +- **Mini RWKV Implementation**: https://github.com/AliC-Li/Mini_RWKV_7 + - Alternative architecture exploration + - Self-implemented mini-RWKV-7 model + +### Step-by-Step Guides +1. **Getting Started with GPT Training** + - Understanding the 26M parameter architecture + - Setting up your training environment + - Data preparation and tokenization + +2. **Attention Mechanisms in MiniMind** + - Multi-head attention explained + - KV cache optimization + - Rotary Position Embeddings (RoPE) + +3. **Efficient Training Techniques** + - Mixed precision training + - Gradient accumulation strategies + - Memory-efficient implementations + +## 💻 Code Examples + +### Training Scenarios +```python +# Quick Start: Train MiniMind on your dataset +python train.py --config configs/minimind_26m.yaml --data_path ./your_data +``` + +### Fine-tuning Examples +- Custom dataset fine-tuning +- Domain-specific adaptation +- Few-shot learning scenarios + +### Evaluation Scripts +```python +# Evaluate model performance +python eval.py --model_path ./checkpoints/best_model.pt --eval_tasks ["perplexity", "accuracy"] +``` + +## 🛠️ Advanced Techniques + +### Mixture of Experts (MoE) +- Understanding sparse models +- Router network design +- Load balancing strategies + +### LoRA (Low-Rank Adaptation) +- Parameter-efficient fine-tuning +- Adapter modules implementation +- Multi-task LoRA setups + +### DPO (Direct Preference Optimization) +- Human feedback integration +- Preference learning techniques +- Alignment strategies + +## 🎓 Community Resources + +### Research Papers +- **GPT Architecture**: "Attention Is All You Need" +- **Parameter-Efficient Fine-tuning**: LoRA and related methods +- **Small Model Optimization**: Techniques for efficient training + +### Video Tutorials +- YouTube: MiniMind training walkthroughs +- Bilibili: Chinese language tutorials +- Conference talks and presentations + +### Discussion Forums +- GitHub Discussions: https://github.com/jingyaogong/minimind/discussions +- Issues tracker for Q&A +- Community Discord/Slack channels + +## ❓ Frequently Asked Questions + +**Q: Can I run MiniMind on Google Colab?** +A: Yes! MiniMind is optimized to run on free tier GPUs. Check the examples folder for Colab notebooks. + +**Q: What's the minimum GPU requirement?** +A: You can train on NVIDIA GPUs with 8GB+ VRAM. For inference, even CPUs work with quantization. + +**Q: How do I add my own dataset?** +A: Follow the data preparation guide in docs/data_prep.md. Convert your text to the required format. + +**Q: Can I use MiniMind for production?** +A: Yes, under Apache 2.0 license. Consider additional safety checks and alignments for production use. + +## 📦 Related Projects + +- **nanoGPT**: Minimal GPT implementation by Andrej Karpathy +- **minGPT**: Educational GPT codebase +- **GPT-2 from scratch**: Complete implementation tutorials + +## 🔗 Useful Links + +- **Hugging Face Models**: Pre-trained checkpoints +- **Datasets**: Common training corpora +- **Benchmarks**: Performance comparisons + +## 📝 Contributing + +Want to add more resources? Please: +1. Fork the repository +2. Add your resource with description +3. Submit a pull request +4. Follow the contribution guidelines + +## 💬 Community + +- **GitHub Issues**: Bug reports and feature requests +- **Discussions**: General Q&A and ideas +- **Pull Requests**: Code contributions welcome! + +--- + +**Last Updated**: October 2025 +**Maintainers**: MiniMind Community +**License**: Apache-2.0 + +*This resource collection is community-driven. Contributions are welcome!*