[feat] Implement DPO training for driving safety alignment
- dataset/driving_preferences.jsonl: 56 (prompt, chosen, rejected) pairs
across 8 safety categories: pedestrian, speed, lane change, weather,
following distance, traffic signals, emergency vehicles, night driving
- scripts/generate_driving_preferences.py: generator script for the dataset
- trainer/train_dpo_driving.py: custom DPO trainer
* implements DPO loss from scratch (no trl dependency)
* frozen reference model + LoRA policy (1.07% trainable params)
* logs preference accuracy per step
* saves LoRA adapters or full weights
* 1 epoch on 56 pairs: loss 0.11, pref_acc 96.4%
Usage:
uv run python trainer/train_dpo_driving.py --data_path dataset/driving_preferences.jsonl --use_lora --epochs 3
823ce1e4fc