62f47fb844
[fix] LoRA/RAG: correct target modules, vlm_checkpoint call, and gitignore
...
- lora_utils.py: fix target_modules to match MiniMind arch (q/k/v/o_proj
instead of GPT-2 style c_attn/c_proj/c_fc)
- train_sft_vlm_lora.py: fix broken vlm_checkpoint call in train_epoch
(was passing model/epoch/step/args instead of vlm_config); now saves
inline like train_sft_vlm.py; remove unused SkipBatchSampler import
- .gitignore: add __pycache__, .claude/, dataset/sft_vlm.parquet
2026-03-06 04:49:40 +00:00
a86648ed66
[feat] Implement LoRA + RAG for DriveMind-V
...
Implemented Day 1 of DriveMind-V plan: parameter-efficient training with LoRA and retrieval-augmented generation with FAISS.
## LoRA (Low-Rank Adaptation)
- Created trainer/lora_utils.py with LoRA configuration helpers
- Created trainer/train_sft_vlm_lora.py for LoRA-enabled SFT training
- Reduces trainable parameters by 98-99% (r=16 → ~1% trainable)
- Memory savings: ~50% during training
- Checkpoint size: 2-5MB vs 100-400MB for full model
## RAG (Retrieval-Augmented Generation)
- Created rag/ package with vector search infrastructure
- rag/vector_store.py: FAISS-based vector storage
- rag/retriever.py: CLIP ViT-B/16 based image retrieval
- rag/rag_vlm.py: RAG-enhanced VLM wrapper
- scripts/build_knowledge_base.py: Build FAISS index from images
- scripts/demo_rag.py: Interactive retrieval demo
## Demo & Testing
- Built demo knowledge base from 8 eval images
- Tested retrieval: perfect match at distance 0.0
- Fixed tensor shape bug in batch encoding
## Stats
- Time: ~1.5 hours
- Files created: 11 new files
- Dependencies added: faiss-cpu==1.13.2
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-03-05 01:13:59 +00:00
56ab6ebd13
[update] readme
2026-02-05 00:23:20 +08:00
33db76fe10
[update] dataset name
2026-02-05 00:22:58 +08:00
b28715e6fd
[update] vlm params
2026-02-04 16:41:17 +08:00
7e094afa59
[update] shuffle data
2026-01-18 17:02:24 +08:00
326b24727e
[update] sync minimind
2026-01-15 21:51:58 +08:00
d2bdff3269
[feat] test parquet
2025-12-27 10:56:07 +08:00
fef284776f
[feat] delete images
2025-12-27 10:53:38 +08:00
3684285a91
[feat] support parquet data
2025-12-27 10:49:50 +08:00
1863743ee9
[feat] update readme
2025-12-27 10:48:54 +08:00
32cf4c5c01
[fix] add gc
2025-12-27 01:09:33 +08:00
b7cb98c136
[fix] reduce aux_loss_alpha
2025-12-05 23:07:55 +08:00
13c3a2c093
[fix] cuda memory
2025-12-01 16:35:05 +08:00
e03af46d12
[feat] update yarn
2025-12-01 16:34:00 +08:00
a157b66ace
[feat] del cuda memory
2025-11-27 19:37:39 +08:00
e613613a13
[feat] update model
2025-11-27 19:36:41 +08:00
d698d98dc5
[feat] update readme
2025-11-11 11:12:09 +08:00
92ca9959bb
[feat] update readme
2025-10-30 23:32:12 +08:00
ca15b0440f
[feat] update readme
2025-10-30 23:31:19 +08:00
2e548ba08e
[feat] update datasets
2025-10-30 11:08:30 +08:00
25d25383e7
[feat] update requirements
2025-10-30 11:06:20 +08:00
7605850e1d
[feat] update args
2025-10-30 10:50:54 +08:00
88b0cd89ea
[feat] update readme
2025-10-29 12:13:44 +08:00
778aeb2946
[feat] update trainer
2025-10-28 23:34:16 +08:00
2ce5c31170
[feat] update sft
2025-10-26 21:24:19 +08:00
c2bfead000
[feat] update readme
2025-10-26 20:55:02 +08:00
54fcbb04c8
[feat] update eval-vlm
2025-10-26 20:54:37 +08:00
aebc6fb191
[feat] pause-training
2025-10-26 20:54:13 +08:00
4c3b5bacbf
[fix] 1024-bugs
2025-10-24 00:35:56 +08:00
094f88a087
[feat] update readme
2025-10-22 10:58:52 +08:00
8bed2ec00b
[feat] update readme
2025-10-21 22:53:00 +08:00
974a666741
[feat] update readme
2025-10-17 00:04:48 +08:00
4d2accbcef
[feat] update images
2025-10-17 00:03:58 +08:00
58dfb9779c
[feat] update minimind model
2025-10-17 00:03:13 +08:00
a8572190a8
update eval
2025-04-27 18:38:51 +08:00
623a130c34
250427
2025-04-27 18:13:22 +08:00
26de815253
update mask
2025-04-10 22:36:34 +08:00
5f130d142d
moe note
2025-04-09 17:37:51 +08:00
139cc98bb9
moe note
2025-04-09 17:37:06 +08:00
bb0ec7a420
update readme
2025-04-09 17:03:13 +08:00
6f00fc7954
update dataset path
2025-04-03 17:55:43 +08:00
9f1d67246b
update readme
2025-04-01 17:45:55 +08:00
5c7ac3a090
update structure image
2025-04-01 16:31:54 +08:00
c1a069b532
fix a bug
2025-02-23 18:59:35 +08:00
694ce87e39
update readme
2025-02-20 23:47:34 +08:00
38a84c2dd2
update minimind2-v
2025-02-20 23:38:48 +08:00
62e999fbda
update minimind2-v
2025-02-20 23:37:00 +08:00
60b265d01c
update minimind2-v
2025-02-20 23:34:51 +08:00
61a4468bab
update readme
2025-02-10 20:45:41 +08:00
MiniMind-V User
jingyaogong
gongjy