1b5c16f674
feat: use HuggingFace datasets for memory-mapped data loading
...
Replace PyArrow Table with HuggingFace datasets.Dataset.from_parquet()
for memory-mapped Arrow access. This avoids loading the entire parquet
file into RAM per DataLoader worker.
Before: 8 workers × 4GB parquet = ~32GB RAM (OOM on 30GB machines)
After: memory-mapped cache shared across all workers via OS page cache
- First load builds Arrow cache (~76s one-time cost)
- Subsequent loads: <1s (reads from cached .arrow file)
- Process RSS: ~2.1GB vs ~4.1GB+ per worker previously
- Enables using num_workers=8 on machines with 30GB RAM
Tested: full pipeline (dataset → dataloader → forward pass) verified
working with 4 workers on g6.2xlarge (L4 GPU, 30GB RAM).
2026-05-11 18:22:19 +00:00
ee18831dce
[fix] repetition_penalty
2026-05-07 19:08:14 +08:00
b6a2532801
[update] readme
2026-05-02 00:35:40 +08:00
1e86f201a2
[update] readme
2026-05-01 12:11:48 +08:00
6fa6127b69
[update] readme
2026-04-28 17:26:02 +08:00
0a9c06fa68
[update] readme
2026-04-27 10:57:35 +08:00
fb38500054
[update] readme
2026-04-27 10:51:46 +08:00
ff81f88833
[update] minimind-3v
2026-04-20 16:54:49 +08:00
92ae3f963a
[update] readme
2026-04-20 00:06:27 +08:00
9c3343dd88
[update] minimind-3v
2026-04-19 23:51:48 +08:00
333654d6e9
[update] image
2026-04-04 14:57:45 +08:00
f74124f6b9
[fix] bugs
2026-04-04 12:43:13 +08:00
5098d67d9f
[fix] issue#114
2026-04-04 12:13:28 +08:00
41c6d9e2a5
Merge pull request #115 from CgggggY/fix-single-batch-multi-image-shape
...
修复 bs=1 多图输入时 vision_tensors 维度不一致的问题
2026-04-04 12:06:04 +08:00
9e59a77080
Fix vision_tensors shape for bs=1 multi-image inputs
2026-04-04 11:32:38 +08:00
c840cdea35
Personal study notes for model_vlm (local only)
2026-04-04 11:22:27 +08:00
4bc6524ac2
[fix] sync minimind
2026-04-02 16:34:38 +08:00
887a42d9cb
[update] model
2026-04-02 15:29:50 +08:00
28acd77d42
[fix] dummy grad
2026-04-01 17:56:44 +08:00
48adb0fda0
[fix] pr113
2026-04-01 15:40:25 +08:00
ac4c506eeb
Merge pull request #113 from JX446/master
...
fix: 修复 eval_vlm.py 中 f-string 引号嵌套导致的语法错误
2026-04-01 15:31:33 +08:00
eac29745da
fix: 修复 eval_vlm.py 中 f-string 引号嵌套导致的语法错误
...
- 修正第 66 行 print 语句中单双引号嵌套冲突的问题。
- 解决在部分 Python 版本下因反斜杠处理导致的 SyntaxError。
2026-04-01 15:11:38 +08:00
14cfa88629
[update] requirement
2026-03-30 11:51:18 +08:00
9089310ede
[update] fp16 inference
2026-03-27 16:22:47 +08:00
1a6d650e9a
[update] readme
2026-03-26 11:46:05 +08:00
e51dcf0fda
[update] minimind-3v
2026-03-26 00:09:02 +08:00
56ab6ebd13
[update] readme
2026-02-05 00:23:20 +08:00
33db76fe10
[update] dataset name
2026-02-05 00:22:58 +08:00
b28715e6fd
[update] vlm params
2026-02-04 16:41:17 +08:00
7e094afa59
[update] shuffle data
2026-01-18 17:02:24 +08:00
326b24727e
[update] sync minimind
2026-01-15 21:51:58 +08:00
d2bdff3269
[feat] test parquet
2025-12-27 10:56:07 +08:00
fef284776f
[feat] delete images
2025-12-27 10:53:38 +08:00
3684285a91
[feat] support parquet data
2025-12-27 10:49:50 +08:00
1863743ee9
[feat] update readme
2025-12-27 10:48:54 +08:00
32cf4c5c01
[fix] add gc
2025-12-27 01:09:33 +08:00
b7cb98c136
[fix] reduce aux_loss_alpha
2025-12-05 23:07:55 +08:00
13c3a2c093
[fix] cuda memory
2025-12-01 16:35:05 +08:00
e03af46d12
[feat] update yarn
2025-12-01 16:34:00 +08:00
a157b66ace
[feat] del cuda memory
2025-11-27 19:37:39 +08:00
e613613a13
[feat] update model
2025-11-27 19:36:41 +08:00
d698d98dc5
[feat] update readme
2025-11-11 11:12:09 +08:00
92ca9959bb
[feat] update readme
2025-10-30 23:32:12 +08:00
ca15b0440f
[feat] update readme
2025-10-30 23:31:19 +08:00
2e548ba08e
[feat] update datasets
2025-10-30 11:08:30 +08:00
25d25383e7
[feat] update requirements
2025-10-30 11:06:20 +08:00
7605850e1d
[feat] update args
2025-10-30 10:50:54 +08:00
88b0cd89ea
[feat] update readme
2025-10-29 12:13:44 +08:00
778aeb2946
[feat] update trainer
2025-10-28 23:34:16 +08:00
2ce5c31170
[feat] update sft
2025-10-26 21:24:19 +08:00
Shawn Tsai
jingyaogong
jingyaogong
jingyaogong
CgggggY
DJY