Implemented Day 1 of DriveMind-V plan: parameter-efficient training with LoRA and retrieval-augmented generation with FAISS.
## LoRA (Low-Rank Adaptation)
- Created trainer/lora_utils.py with LoRA configuration helpers
- Created trainer/train_sft_vlm_lora.py for LoRA-enabled SFT training
- Reduces trainable parameters by 98-99% (r=16 → ~1% trainable)
- Memory savings: ~50% during training
- Checkpoint size: 2-5MB vs 100-400MB for full model
## RAG (Retrieval-Augmented Generation)
- Created rag/ package with vector search infrastructure
- rag/vector_store.py: FAISS-based vector storage
- rag/retriever.py: CLIP ViT-B/16 based image retrieval
- rag/rag_vlm.py: RAG-enhanced VLM wrapper
- scripts/build_knowledge_base.py: Build FAISS index from images
- scripts/demo_rag.py: Interactive retrieval demo
## Demo & Testing
- Built demo knowledge base from 8 eval images
- Tested retrieval: perfect match at distance 0.0
- Fixed tensor shape bug in batch encoding
## Stats
- Time: ~1.5 hours
- Files created: 11 new files
- Dependencies added: faiss-cpu==1.13.2
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
dynorth
jingyaogong
MiniMind-V User