Shawn Tsai 1b5c16f674 feat: use HuggingFace datasets for memory-mapped data loading ... Replace PyArrow Table with HuggingFace datasets.Dataset.from_parquet() for memory-mapped Arrow access. This avoids loading the entire parquet file into RAM per DataLoader worker. Before: 8 workers × 4GB parquet = ~32GB RAM (OOM on 30GB machines) After: memory-mapped cache shared across all workers via OS page cache - First load builds Arrow cache (~76s one-time cost) - Subsequent loads: <1s (reads from cached .arrow file) - Process RSS: ~2.1GB vs ~4.1GB+ per worker previously - Enables using num_workers=8 on machines with 30GB RAM Tested: full pipeline (dataset → dataloader → forward pass) verified working with 4 workers on g6.2xlarge (L4 GPU, 30GB RAM).		2026-05-11 18:22:19 +00:00
..
eval_images	[update] minimind-3v	2026-04-19 23:51:48 +08:00
__init__.py	250427	2025-04-27 18:13:22 +08:00
lm_dataset.py	feat: use HuggingFace datasets for memory-mapped data loading	2026-05-11 18:22:19 +00:00