Compare commits
10 Commits
6689137375
...
a458e70363
| Author | SHA1 | Date |
|---|---|---|
AmirX0
|
a458e70363 | |
jingyaogong
|
7a9137d2e9 | |
jingyaogong
|
7988bc81a3 | |
basil-k-aji-dev
|
37ac206e9a | |
|
jingyaogong
|
21ec325dba | |
|
jingyaogong
|
cdf0a93e02 | |
jingyaogong
|
c367572702 | |
|
basil-k-aji-dev
|
9c088dd3a5 | |
tkgcc
|
0f6803e551 | |
AmirX0
|
66aef88185 |
|
|
@ -27,7 +27,7 @@
|
|||
|
||||
<div align="center">
|
||||
|
||||
中文 | [English](./README_en.md)
|
||||
中文 | [English](./README_en.md) | [العربية](./README_i18n.md#arabic) | [Français](./README_i18n.md#french) | [Español](./README_i18n.md#spanish) | [Português](./README_i18n.md#portuguese) | [Deutsch](./README_i18n.md#german) | [فارسی](./README_i18n.md#persian) | [日本語](./README_i18n.md#japanese) | [한국어](./README_i18n.md#korean)
|
||||
|
||||
</div>
|
||||
|
||||
|
|
|
|||
|
|
@ -27,7 +27,7 @@
|
|||
|
||||
<div align="center">
|
||||
|
||||
[中文](./README.md) | English
|
||||
[中文](./README.md) | English | [العربية](./README_i18n.md#arabic) | [Français](./README_i18n.md#french) | [Español](./README_i18n.md#spanish) | [Português](./README_i18n.md#portuguese) | [Deutsch](./README_i18n.md#german) | [فارسی](./README_i18n.md#persian) | [日本語](./README_i18n.md#japanese) | [한국어](./README_i18n.md#korean)
|
||||
|
||||
</div>
|
||||
|
||||
|
|
|
|||
|
|
@ -0,0 +1,197 @@
|
|||
# MiniMind in other languages
|
||||
|
||||
[中文](./README.md) | [English](./README_en.md) | [العربية](#arabic) | [Français](#french) | [Español](#spanish) | [Português](#portuguese) | [Deutsch](#german) | [فارسی](#persian) | [日本語](#japanese) | [한국어](#korean)
|
||||
|
||||
These localized introductions provide a concise overview and the fastest path to running MiniMind. The complete and most current technical documentation is maintained in [English](./README_en.md) and [Chinese](./README.md).
|
||||
|
||||
<a id="arabic"></a>
|
||||
|
||||
<h2 dir="rtl">العربية</h2>
|
||||
|
||||
<p dir="rtl">MiniMind هو مشروع مفتوح المصدر يتيح تدريب نموذج لغوي صغير يضم نحو 64 مليون مُعلَمة من الصفر. يمكن إكمال مرحلة الضبط الدقيق الخاضع للإشراف خلال ساعتين تقريبًا على بطاقة NVIDIA 3090 واحدة، بتكلفة استئجار تقارب 3 يوانات صينية.</p>
|
||||
|
||||
<ul dir="rtl">
|
||||
<li>تنفيذ مبسّط ومباشر باستخدام PyTorch لبنية Dense وMoE.</li>
|
||||
<li>مسار تدريب متكامل يشمل التدريب المسبق وSFT وLoRA وDPO وPPO وGRPO وCISPO واستخدام الأدوات وAgentic RL والتقطير.</li>
|
||||
<li>دعم Transformers وllama.cpp وvLLM وOllama وواجهة API متوافقة مع OpenAI وواجهة WebUI.</li>
|
||||
<li>بيانات وأوزان ونصوص تدريب مفتوحة لتسهيل التعلم وإعادة إنتاج النتائج.</li>
|
||||
</ul>
|
||||
|
||||
<p dir="rtl"><strong>البدء السريع</strong></p>
|
||||
|
||||
```bash
|
||||
git clone --depth 1 https://github.com/jingyaogong/minimind
|
||||
cd minimind
|
||||
pip install -r requirements.txt
|
||||
modelscope download --model gongjy/minimind-3 --local_dir ./minimind-3
|
||||
python eval_llm.py --load_from ./minimind-3
|
||||
```
|
||||
|
||||
<p dir="rtl">للحصول على تعليمات التدريب ومجموعات البيانات والتقييم وتحويل النموذج، راجع <a href="./README_en.md">الدليل الإنجليزي الكامل</a>.</p>
|
||||
|
||||
<a id="french"></a>
|
||||
|
||||
## Français
|
||||
|
||||
MiniMind est un projet open source permettant d'entraîner de zéro un petit modèle de langage d'environ 64 millions de paramètres. L'étape de fine-tuning supervisé peut être réalisée en environ deux heures sur une seule NVIDIA 3090, pour un coût de location proche de 3 RMB.
|
||||
|
||||
- Implémentation PyTorch minimale et lisible des architectures Dense et MoE.
|
||||
- Pipeline complet : préentraînement, SFT, LoRA, DPO, PPO, GRPO, CISPO, Tool Use, Agentic RL et distillation.
|
||||
- Compatibilité avec Transformers, llama.cpp, vLLM et Ollama, ainsi qu'une API compatible OpenAI et une WebUI.
|
||||
- Jeux de données, poids et scripts d'entraînement ouverts pour apprendre et reproduire les résultats.
|
||||
|
||||
**Démarrage rapide**
|
||||
|
||||
```bash
|
||||
git clone --depth 1 https://github.com/jingyaogong/minimind
|
||||
cd minimind
|
||||
pip install -r requirements.txt
|
||||
modelscope download --model gongjy/minimind-3 --local_dir ./minimind-3
|
||||
python eval_llm.py --load_from ./minimind-3
|
||||
```
|
||||
|
||||
Pour les instructions complètes sur l'entraînement, les données, l'évaluation et la conversion des modèles, consultez le [guide complet en anglais](./README_en.md).
|
||||
|
||||
<a id="spanish"></a>
|
||||
|
||||
## Español
|
||||
|
||||
MiniMind es un proyecto de código abierto para entrenar desde cero un modelo de lenguaje pequeño de unos 64 millones de parámetros. La etapa de ajuste supervisado puede completarse en unas dos horas con una sola NVIDIA 3090, con un coste de alquiler cercano a 3 RMB.
|
||||
|
||||
- Implementación mínima y legible en PyTorch de arquitecturas Dense y MoE.
|
||||
- Flujo completo: preentrenamiento, SFT, LoRA, DPO, PPO, GRPO, CISPO, uso de herramientas, Agentic RL y destilación.
|
||||
- Compatibilidad con Transformers, llama.cpp, vLLM y Ollama, además de una API compatible con OpenAI y una WebUI.
|
||||
- Conjuntos de datos, pesos y scripts de entrenamiento abiertos para aprender y reproducir resultados.
|
||||
|
||||
**Inicio rápido**
|
||||
|
||||
```bash
|
||||
git clone --depth 1 https://github.com/jingyaogong/minimind
|
||||
cd minimind
|
||||
pip install -r requirements.txt
|
||||
modelscope download --model gongjy/minimind-3 --local_dir ./minimind-3
|
||||
python eval_llm.py --load_from ./minimind-3
|
||||
```
|
||||
|
||||
Para obtener instrucciones completas sobre entrenamiento, datos, evaluación y conversión de modelos, consulta la [documentación completa en inglés](./README_en.md).
|
||||
|
||||
<a id="portuguese"></a>
|
||||
|
||||
## Português
|
||||
|
||||
MiniMind é um projeto de código aberto para treinar do zero um pequeno modelo de linguagem com cerca de 64 milhões de parâmetros. A etapa de ajuste fino supervisionado pode ser concluída em aproximadamente duas horas com uma única NVIDIA 3090, por um custo de locação próximo de 3 RMB.
|
||||
|
||||
- Implementação PyTorch mínima e legível de arquiteturas Dense e MoE.
|
||||
- Pipeline completo: pré-treinamento, SFT, LoRA, DPO, PPO, GRPO, CISPO, uso de ferramentas, Agentic RL e destilação.
|
||||
- Compatibilidade com Transformers, llama.cpp, vLLM e Ollama, além de API compatível com OpenAI e WebUI.
|
||||
- Conjuntos de dados, pesos e scripts de treinamento abertos para aprendizado e reprodução dos resultados.
|
||||
|
||||
**Início rápido**
|
||||
|
||||
```bash
|
||||
git clone --depth 1 https://github.com/jingyaogong/minimind
|
||||
cd minimind
|
||||
pip install -r requirements.txt
|
||||
modelscope download --model gongjy/minimind-3 --local_dir ./minimind-3
|
||||
python eval_llm.py --load_from ./minimind-3
|
||||
```
|
||||
|
||||
Para instruções completas sobre treinamento, dados, avaliação e conversão de modelos, consulte a [documentação completa em inglês](./README_en.md).
|
||||
|
||||
<a id="german"></a>
|
||||
|
||||
## Deutsch
|
||||
|
||||
MiniMind ist ein Open-Source-Projekt, mit dem sich ein kleines Sprachmodell mit rund 64 Millionen Parametern von Grund auf trainieren lässt. Das überwachte Fine-Tuning kann auf einer einzelnen NVIDIA 3090 in ungefähr zwei Stunden abgeschlossen werden; die entsprechenden Mietkosten liegen bei etwa 3 RMB.
|
||||
|
||||
- Kleine, gut lesbare PyTorch-Implementierung von Dense- und MoE-Architekturen.
|
||||
- Vollständige Pipeline: Pretraining, SFT, LoRA, DPO, PPO, GRPO, CISPO, Tool Use, Agentic RL und Distillation.
|
||||
- Kompatibel mit Transformers, llama.cpp, vLLM und Ollama; zusätzlich stehen eine OpenAI-kompatible API und eine WebUI bereit.
|
||||
- Offene Datensätze, Gewichte und Trainingsskripte zum Lernen und Reproduzieren der Ergebnisse.
|
||||
|
||||
**Schnellstart**
|
||||
|
||||
```bash
|
||||
git clone --depth 1 https://github.com/jingyaogong/minimind
|
||||
cd minimind
|
||||
pip install -r requirements.txt
|
||||
modelscope download --model gongjy/minimind-3 --local_dir ./minimind-3
|
||||
python eval_llm.py --load_from ./minimind-3
|
||||
```
|
||||
|
||||
Vollständige Anleitungen zu Training, Datensätzen, Evaluation und Modellkonvertierung enthält die [englische Dokumentation](./README_en.md).
|
||||
|
||||
<a id="persian"></a>
|
||||
|
||||
<h2 dir="rtl">فارسی</h2>
|
||||
|
||||
<p dir="rtl">MiniMind یک پروژهٔ متنباز برای آموزش یک مدل زبانی کوچک با حدود ۶۴ میلیون پارامتر از صفر است. مرحلهٔ تنظیم دقیق نظارتشده را میتوان با یک کارت NVIDIA 3090 در حدود دو ساعت و با هزینهٔ اجارهای نزدیک به ۳ یوان چین انجام داد.</p>
|
||||
|
||||
<ul dir="rtl">
|
||||
<li>پیادهسازی ساده و خوانای معماریهای Dense و MoE با PyTorch.</li>
|
||||
<li>زنجیرهٔ کامل آموزش شامل پیشآموزش، SFT، LoRA، DPO، PPO، GRPO، CISPO، استفاده از ابزار، Agentic RL و تقطیر مدل.</li>
|
||||
<li>سازگار با Transformers، llama.cpp، vLLM و Ollama، همراه با API سازگار با OpenAI و WebUI.</li>
|
||||
<li>دادهها، وزنها و اسکریپتهای آموزشی باز برای یادگیری و بازتولید نتایج.</li>
|
||||
</ul>
|
||||
|
||||
<p dir="rtl"><strong>شروع سریع</strong></p>
|
||||
|
||||
```bash
|
||||
git clone --depth 1 https://github.com/jingyaogong/minimind
|
||||
cd minimind
|
||||
pip install -r requirements.txt
|
||||
modelscope download --model gongjy/minimind-3 --local_dir ./minimind-3
|
||||
python eval_llm.py --load_from ./minimind-3
|
||||
```
|
||||
|
||||
<p dir="rtl">برای دستورالعمل کامل آموزش، دادهها، ارزیابی و تبدیل مدل، <a href="./README_en.md">مستندات کامل انگلیسی</a> را ببینید.</p>
|
||||
|
||||
<a id="japanese"></a>
|
||||
|
||||
## 日本語
|
||||
|
||||
MiniMind は、約 6,400 万パラメータの小規模言語モデルをゼロから学習できるオープンソースプロジェクトです。教師ありファインチューニングは、NVIDIA 3090 1枚で約2時間、レンタル費用約3人民元で完了できます。
|
||||
|
||||
- Dense および MoE アーキテクチャを、読みやすい最小限の PyTorch コードで実装。
|
||||
- 事前学習、SFT、LoRA、DPO、PPO、GRPO、CISPO、Tool Use、Agentic RL、蒸留までを網羅。
|
||||
- Transformers、llama.cpp、vLLM、Ollama に対応し、OpenAI互換APIとWebUIも提供。
|
||||
- 学習と再現のためのデータセット、重み、学習スクリプトを公開。
|
||||
|
||||
**クイックスタート**
|
||||
|
||||
```bash
|
||||
git clone --depth 1 https://github.com/jingyaogong/minimind
|
||||
cd minimind
|
||||
pip install -r requirements.txt
|
||||
modelscope download --model gongjy/minimind-3 --local_dir ./minimind-3
|
||||
python eval_llm.py --load_from ./minimind-3
|
||||
```
|
||||
|
||||
学習、データセット、評価、モデル変換の詳しい手順については、[英語版の完全なドキュメント](./README_en.md)を参照してください。
|
||||
|
||||
<a id="korean"></a>
|
||||
|
||||
## 한국어
|
||||
|
||||
MiniMind는 약 6,400만 개의 파라미터를 가진 소형 언어 모델을 처음부터 학습할 수 있는 오픈 소스 프로젝트입니다. 지도 미세 조정 단계는 NVIDIA 3090 한 장에서 약 2시간, 약 3위안의 대여 비용으로 완료할 수 있습니다.
|
||||
|
||||
- Dense 및 MoE 아키텍처를 간결하고 읽기 쉬운 PyTorch 코드로 구현합니다.
|
||||
- 사전 학습, SFT, LoRA, DPO, PPO, GRPO, CISPO, 도구 사용, Agentic RL, 모델 증류를 포함한 전체 파이프라인을 제공합니다.
|
||||
- Transformers, llama.cpp, vLLM, Ollama와 호환되며 OpenAI 호환 API와 WebUI를 제공합니다.
|
||||
- 학습과 결과 재현을 위한 데이터셋, 가중치, 학습 스크립트를 공개합니다.
|
||||
|
||||
**빠른 시작**
|
||||
|
||||
```bash
|
||||
git clone --depth 1 https://github.com/jingyaogong/minimind
|
||||
cd minimind
|
||||
pip install -r requirements.txt
|
||||
modelscope download --model gongjy/minimind-3 --local_dir ./minimind-3
|
||||
python eval_llm.py --load_from ./minimind-3
|
||||
```
|
||||
|
||||
학습, 데이터셋, 평가, 모델 변환에 대한 전체 안내는 [영문 문서](./README_en.md)를 참고하세요.
|
||||
|
||||
---
|
||||
|
||||
These translations are community-maintained summaries. If you find an inaccurate phrase or want to maintain a complete translation, please open a pull request.
|
||||
|
|
@ -12,7 +12,7 @@ from datetime import datetime
|
|||
from transformers import AutoTokenizer, AutoModelForCausalLM, TextStreamer
|
||||
from openai import OpenAI
|
||||
from model.model_minimind import MiniMindConfig, MiniMindForCausalLM
|
||||
from trainer.trainer_utils import setup_seed, get_model_params
|
||||
from trainer.trainer_utils import setup_seed, get_model_params, safe_math_eval
|
||||
warnings.filterwarnings('ignore')
|
||||
|
||||
TOOLS = [
|
||||
|
|
@ -27,7 +27,7 @@ TOOLS = [
|
|||
]
|
||||
|
||||
MOCK_RESULTS = {
|
||||
"calculate_math": lambda args: {"result": str(eval(str(args.get("expression", "0")).replace("^", "**").replace("×", "*").replace("÷", "/").replace("−", "-").replace("²", "**2").replace("³", "**3").replace("（", "(").replace("）", ")")))},
|
||||
"calculate_math": lambda args: {"result": str(safe_math_eval(args.get("expression", "0")))},
|
||||
"get_current_time": lambda args: {"datetime": datetime.now().strftime("%Y-%m-%d %H:%M:%S"), "timezone": args.get("timezone", "Asia/Shanghai")},
|
||||
"random_number": lambda args: {"result": random.randint(int(args.get("min", 0)), int(args.get("max", 100)))},
|
||||
"text_length": lambda args: {"characters": len(args.get("text", "")), "words": len(args.get("text", "").split())},
|
||||
|
|
|
|||
|
|
@ -2,6 +2,7 @@ import random
|
|||
import re
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
from threading import Thread
|
||||
|
||||
import torch
|
||||
|
|
@ -9,6 +10,9 @@ import numpy as np
|
|||
import streamlit as st
|
||||
from transformers import AutoModelForCausalLM, AutoTokenizer, TextIteratorStreamer
|
||||
|
||||
sys.path.append(os.path.abspath(os.path.join(os.path.dirname(__file__), '..')))
|
||||
from trainer.trainer_utils import safe_math_eval
|
||||
|
||||
st.set_page_config(page_title="MiniMind", initial_sidebar_state="collapsed")
|
||||
|
||||
st.markdown("""
|
||||
|
|
@ -125,7 +129,7 @@ def execute_tool(tool_name, args):
|
|||
import datetime
|
||||
try:
|
||||
if tool_name == 'calculate_math':
|
||||
return {"result": eval(args.get('expression', '0'))}
|
||||
return {"result": safe_math_eval(args.get('expression', '0'))}
|
||||
elif tool_name == 'get_current_time':
|
||||
tz = args.get('timezone', 'Asia/Shanghai')
|
||||
return {"result": datetime.datetime.now().strftime('%Y-%m-%d %H:%M:%S')}
|
||||
|
|
|
|||
|
|
@ -10,7 +10,6 @@ import gc
|
|||
import json
|
||||
import math
|
||||
import random
|
||||
import signal
|
||||
import argparse
|
||||
import warnings
|
||||
import torch
|
||||
|
|
@ -24,7 +23,7 @@ from torch.optim.lr_scheduler import CosineAnnealingLR
|
|||
from transformers import AutoTokenizer
|
||||
from model.model_minimind import MiniMindConfig, MiniMindForCausalLM
|
||||
from dataset.lm_dataset import AgentRLDataset
|
||||
from trainer.trainer_utils import Logger, is_main_process, lm_checkpoint, init_distributed_mode, setup_seed, SkipBatchSampler, init_model, LMForRewardModel
|
||||
from trainer.trainer_utils import Logger, is_main_process, lm_checkpoint, init_distributed_mode, setup_seed, SkipBatchSampler, init_model, LMForRewardModel, safe_math_eval
|
||||
from trainer.rollout_engine import create_rollout_engine, compute_per_token_logps
|
||||
|
||||
warnings.filterwarnings('ignore')
|
||||
|
|
@ -55,7 +54,7 @@ UNIT_DATA = {"km_miles": 0.621371, "miles_km": 1.60934, "kg_pounds": 2.20462, "p
|
|||
|
||||
# ======== 模拟执行 ========
|
||||
MOCK_RESULTS = {
|
||||
"calculate_math": lambda args: {"result": str(eval(str(args.get("expression", "0")).replace("^", "**").replace("×", "*").replace("÷", "/").replace("−", "-").replace("（", "(").replace("）", ")"), {"__builtins__": {}, "math": math}))},
|
||||
"calculate_math": lambda args: {"result": str(safe_math_eval(args.get("expression", "0")))},
|
||||
"unit_converter": lambda args: {"result": round(float(args.get("value", 0)) * UNIT_DATA.get(f"{args.get('from_unit', '').lower()}_{args.get('to_unit', '').lower()}", 1), 4)},
|
||||
"get_current_weather": lambda args: (lambda w: {"city": args.get("location"), "temperature": w[0], "humidity": "65%", "condition": w[1]})(WEATHER_DATA.get(args.get("location"), ("22°C", "晴"))),
|
||||
"get_current_time": lambda args: {"datetime": TIME_DATA.get(args.get("timezone", "Asia/Shanghai"), "2025-03-07 14:30:00"), "timezone": args.get("timezone", "Asia/Shanghai")},
|
||||
|
|
@ -85,14 +84,9 @@ def execute_tool(name, args):
|
|||
fn = MOCK_RESULTS.get(name)
|
||||
if not fn: return None
|
||||
try:
|
||||
signal.signal(signal.SIGALRM, lambda *_: (_ for _ in ()).throw(TimeoutError()))
|
||||
signal.alarm(1)
|
||||
return fn(args)
|
||||
except:
|
||||
except Exception:
|
||||
return None
|
||||
finally:
|
||||
try: signal.alarm(0)
|
||||
except: pass
|
||||
|
||||
# ======== 多轮 Rollout ========
|
||||
def rollout_single(rollout_engine, tokenizer, messages, tools, max_turns=3, max_new_tokens=256, thinking_ratio=0.5, device="cuda"):
|
||||
|
|
@ -278,9 +272,10 @@ def rl_train_epoch(epoch, loader, iters, rollout_engine, ref_model, reward_model
|
|||
|
||||
rewards = calculate_rewards(prompts, completions, gt_batch, tools_batch, args.num_generations, reward_model, device=args.device, turn_outputs_batch=turn_outputs_batch, unfinished_batch=unfinished_batch)
|
||||
|
||||
model_unwrapped = model.module if isinstance(model, DistributedDataParallel) else model
|
||||
with autocast_ctx:
|
||||
res = model_unwrapped(input_ids, attention_mask=full_mask)
|
||||
# 反向传播必须经过 DDP 包装后的模块：直接调用 .module 会跳过
|
||||
# DDP 的 prepare_for_backward，梯度不会 all-reduce，各卡静默发散。
|
||||
res = model(input_ids, attention_mask=full_mask)
|
||||
aux_loss = res.aux_loss if lm_config.use_moe else torch.tensor(0.0, device=args.device)
|
||||
logits = res.logits[:, :-1, :]
|
||||
per_token_logps = F.log_softmax(logits, dim=-1).gather(2, input_ids[:, 1:].unsqueeze(-1)).squeeze(-1)
|
||||
|
|
@ -478,7 +473,8 @@ if __name__ == "__main__":
|
|||
Logger('torch.compile enabled')
|
||||
rollout_engine.update_policy(model)
|
||||
if dist.is_initialized():
|
||||
model = DistributedDataParallel(model, device_ids=[local_rank])
|
||||
# 同 train_ppo：RoPE buffer 各 rank 一致，每步广播纯属浪费
|
||||
model = DistributedDataParallel(model, device_ids=[local_rank], broadcast_buffers=False)
|
||||
rollout_engine.update_policy(model)
|
||||
|
||||
for epoch in range(start_epoch, args.epochs):
|
||||
|
|
|
|||
|
|
@ -95,9 +95,10 @@ def grpo_train_epoch(epoch, loader, iters, rollout_engine, ref_model, reward_mod
|
|||
|
||||
rewards = calculate_rewards(prompts, completions, reward_model).to(args.device) # [B*num_gen]
|
||||
|
||||
model_unwrapped = model.module if isinstance(model, DistributedDataParallel) else model
|
||||
with autocast_ctx:
|
||||
res = model_unwrapped(outputs, attention_mask=full_mask)
|
||||
# 反向传播必须经过 DDP 包装后的模块：直接调用 .module 会跳过
|
||||
# DDP 的 prepare_for_backward，梯度不会 all-reduce，各卡静默发散。
|
||||
res = model(outputs, attention_mask=full_mask)
|
||||
aux_loss = res.aux_loss if lm_config.use_moe else torch.tensor(0.0, device=args.device)
|
||||
per_token_logps = F.log_softmax(res.logits[:, :-1, :], dim=-1).gather(2, outputs[:, 1:].unsqueeze(-1)).squeeze(-1).gather(1, logp_pos)
|
||||
|
||||
|
|
@ -313,7 +314,8 @@ if __name__ == "__main__":
|
|||
Logger('torch.compile enabled')
|
||||
rollout_engine.update_policy(model)
|
||||
if dist.is_initialized():
|
||||
model = DistributedDataParallel(model, device_ids=[local_rank])
|
||||
# 同 train_ppo：RoPE buffer 各 rank 一致，每步广播纯属浪费
|
||||
model = DistributedDataParallel(model, device_ids=[local_rank], broadcast_buffers=False)
|
||||
rollout_engine.update_policy(model)
|
||||
|
||||
# ========== 8. 开始训练 ==========
|
||||
|
|
|
|||
|
|
@ -36,7 +36,7 @@ def rep_penalty(text, n=3, cap=0.5):
|
|||
class CriticModel(MiniMindForCausalLM):
|
||||
def __init__(self, params):
|
||||
super().__init__(params)
|
||||
# 替换lm_head为输出单一价值的线性层
|
||||
# lm_head 不参与 forward，仅靠 tie_word_embeddings 与 embed_tokens 共享权重，解绑后 DDP 会报未使用参数
|
||||
self.value_head = nn.Linear(params.hidden_size, 1)
|
||||
|
||||
def forward(self, input_ids=None, attention_mask=None, **kwargs):
|
||||
|
|
@ -159,8 +159,6 @@ def ppo_train_epoch(epoch, loader, iters, rollout_engine, ref_model, actor_sched
|
|||
clipfrac_sum = 0.0
|
||||
aux_loss_sum = 0.0
|
||||
log_count = 0
|
||||
actor_unwrapped = actor_model.module if isinstance(actor_model, DistributedDataParallel) else actor_model
|
||||
critic_unwrapped = critic_model.module if isinstance(critic_model, DistributedDataParallel) else critic_model
|
||||
for ppo_epoch in range(args.ppo_update_iters):
|
||||
if stop_ppo:
|
||||
break
|
||||
|
|
@ -168,11 +166,13 @@ def ppo_train_epoch(epoch, loader, iters, rollout_engine, ref_model, actor_sched
|
|||
for i in range(0, B, mb_size):
|
||||
inds = b_inds[i:i + mb_size]
|
||||
|
||||
mb_values_seq = critic_unwrapped(input_ids=gen_out[inds], attention_mask=full_mask[inds])
|
||||
# 反向传播必须经过 DDP 包装后的模块：直接调用 .module 会跳过
|
||||
# DDP 的 prepare_for_backward，梯度不会 all-reduce，各卡静默发散。
|
||||
mb_values_seq = critic_model(input_ids=gen_out[inds], attention_mask=full_mask[inds])
|
||||
mb_resp_values = mb_values_seq.gather(1, logp_pos[inds])
|
||||
|
||||
with autocast_ctx:
|
||||
res = actor_unwrapped(input_ids=gen_out[inds], attention_mask=full_mask[inds])
|
||||
res = actor_model(input_ids=gen_out[inds], attention_mask=full_mask[inds])
|
||||
aux_loss = res.aux_loss if lm_config.use_moe else torch.tensor(0.0, device=args.device)
|
||||
# 在 autocast 内计算 log_softmax，避免直接对 fp16/bf16 logits
|
||||
# 计算造成额外数值偏差。
|
||||
|
|
@ -191,7 +191,7 @@ def ppo_train_epoch(epoch, loader, iters, rollout_engine, ref_model, actor_sched
|
|||
f"ratio_max={torch.exp(_lrv).max().item():.6f} "
|
||||
f"ratio_min={torch.exp(_lrv).min().item():.6f} "
|
||||
f"dropout={getattr(lm_config, 'dropout', None)} "
|
||||
f"training={actor_unwrapped.training}")
|
||||
f"training={actor_model.training}")
|
||||
approx_kl = (0.5 * (log_ratio ** 2) * resp_policy_mask[inds]).sum() / resp_policy_mask[inds].sum().clamp(min=1)
|
||||
|
||||
# 同步各卡的 approx_kl，防止某卡 break 而其它卡继续导致 DDP 死锁
|
||||
|
|
@ -429,8 +429,9 @@ if __name__ == "__main__":
|
|||
Logger('torch.compile enabled')
|
||||
rollout_engine.update_policy(actor_model)
|
||||
if dist.is_initialized():
|
||||
actor_model = DistributedDataParallel(actor_model, device_ids=[local_rank])
|
||||
critic_model = DistributedDataParallel(critic_model, device_ids=[local_rank])
|
||||
# freqs_cos/freqs_sin 各 rank 由 config 确定性算出，默认每步广播一次纯属浪费
|
||||
actor_model = DistributedDataParallel(actor_model, device_ids=[local_rank], broadcast_buffers=False)
|
||||
critic_model = DistributedDataParallel(critic_model, device_ids=[local_rank], broadcast_buffers=False)
|
||||
rollout_engine.update_policy(actor_model)
|
||||
|
||||
# ========== 8. 开始训练 ==========
|
||||
|
|
|
|||
|
|
@ -5,6 +5,8 @@ import os
|
|||
import sys
|
||||
__package__ = "trainer"
|
||||
sys.path.append(os.path.abspath(os.path.join(os.path.dirname(__file__), '..')))
|
||||
import ast
|
||||
import operator
|
||||
import random
|
||||
import math
|
||||
import numpy as np
|
||||
|
|
@ -175,3 +177,33 @@ class LMForRewardModel:
|
|||
]
|
||||
score = self.model.get_score(self.tokenizer, eval_messages)
|
||||
return max(min(score, 3.0), -3.0)
|
||||
|
||||
|
||||
# ===== 数学表达式安全求值：替代 eval，只放行算术运算与 math 白名单（长度上限 512） =====
|
||||
def safe_math_eval(expression):
|
||||
"""对模型生成的数学表达式求值：支持 + - * / // % **、math 白名单函数与 pi/e/tau 常量。"""
|
||||
def pow_guard(base, exp): # 幂运算统一走这里，拦截 9**9**9 / 10**99999 这类把进程算死的输入
|
||||
if abs(exp) > 1e4 or (abs(base) > 1 and abs(exp) * math.log10(abs(base)) > 100): raise ValueError('幂运算结果过大')
|
||||
return base ** exp
|
||||
def resolve(node): # 同时兼容 sqrt(4) 与 math.sqrt(4) 两种写法
|
||||
if isinstance(node, ast.Name): return node.id
|
||||
if isinstance(node, ast.Attribute) and getattr(node.value, 'id', '') == 'math': return node.attr
|
||||
def walk(node):
|
||||
if isinstance(node, ast.Constant) and type(node.value) in (int, float): return node.value
|
||||
name = resolve(node)
|
||||
if name in consts: return consts[name]
|
||||
if isinstance(node, ast.UnaryOp): return unary_ops[type(node.op)](walk(node.operand))
|
||||
if isinstance(node, ast.BinOp): return bin_ops[type(node.op)](walk(node.left), walk(node.right))
|
||||
if isinstance(node, ast.Call) and not node.keywords: return funcs[resolve(node.func)](*map(walk, node.args))
|
||||
raise ValueError(f'不支持的表达式语法: {type(node).__name__}')
|
||||
consts = {'pi': math.pi, 'e': math.e, 'tau': math.tau}
|
||||
funcs = {'pow': pow_guard, **{n: getattr(math, n) for n in 'sqrt exp log log2 log10 sin cos tan asin acos atan atan2 sinh cosh tanh floor ceil trunc fabs fmod hypot gcd degrees radians'.split()}}
|
||||
bin_ops = {ast.Add: operator.add, ast.Sub: operator.sub, ast.Mult: operator.mul, ast.Div: operator.truediv, ast.FloorDiv: operator.floordiv, ast.Mod: operator.mod, ast.Pow: pow_guard}
|
||||
unary_ops = {ast.UAdd: operator.pos, ast.USub: operator.neg}
|
||||
chars = str.maketrans({'^': '**', '×': '*', '÷': '/', '−': '-', '²': '**2', '³': '**3', '（': '(', '）': ')'})
|
||||
expr = str(expression).translate(chars).strip()
|
||||
if not expr or len(expr) > 512: raise ValueError('表达式为空或过长')
|
||||
try:
|
||||
return walk(ast.parse(expr, mode='eval').body)
|
||||
except (KeyError, SyntaxError, TypeError):
|
||||
raise ValueError('不支持的表达式语法') from None
|
||||
|
|
|
|||
Loading…
Reference in New Issue