增加mla注意力模型
baihaochen
parent
bc3d137248
commit
49a05a1307
|
|
@ -0,0 +1,103 @@
|
|||
# CLAUDE.md
|
||||
|
||||
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
|
||||
|
||||
## Project Overview
|
||||
|
||||
MiniMind is an ultra-small language model (~26M-104M params) designed for training from scratch on consumer GPUs. It demonstrates the complete LLM training pipeline at minimal cost. The codebase is a Chinese-language educational project with English code.
|
||||
|
||||
## Common Commands
|
||||
|
||||
### Training (all run from `trainer/` directory)
|
||||
|
||||
```bash
|
||||
# Stage 1: Pretraining
|
||||
cd trainer && torchrun --nproc_per_node=1 train_pretrain.py
|
||||
|
||||
# Stage 2: Supervised Fine-Tuning
|
||||
cd trainer && torchrun --nproc_per_node=1 train_full_sft.py
|
||||
|
||||
# Optional: LoRA / DPO / PPO / GRPO / Agent RL / Distillation
|
||||
cd trainer && torchrun --nproc_per_node=1 train_lora.py
|
||||
cd trainer && torchrun --nproc_per_node=1 train_dpo.py
|
||||
cd trainer && torchrun --nproc_per_node=1 train_ppo.py
|
||||
cd trainer && torchrun --nproc_per_node=1 train_grpo.py
|
||||
cd trainer && torchrun --nproc_per_node=1 train_agent.py
|
||||
cd trainer && torchrun --nproc_per_node=1 train_distillation.py
|
||||
```
|
||||
|
||||
Multi-GPU: change `--nproc_per_node=N`. Resume training with `--from_resume 1`.
|
||||
|
||||
### Inference
|
||||
|
||||
```bash
|
||||
python eval_llm.py # CLI inference
|
||||
python eval_llm.py --from_transformers # Load transformers-format model
|
||||
python eval_llm.py --lora_path <path> # With LoRA adapter
|
||||
python scripts/serve_openai_api.py # OpenAI-compatible API server
|
||||
```
|
||||
|
||||
### Model Conversion
|
||||
|
||||
```bash
|
||||
python scripts/convert_model.py # torch <-> transformers format, LoRA merge
|
||||
```
|
||||
|
||||
### Dependencies
|
||||
|
||||
```bash
|
||||
pip install -r requirements.txt
|
||||
# PyTorch must be installed separately (commented out in requirements.txt)
|
||||
```
|
||||
|
||||
## Architecture
|
||||
|
||||
### Training Pipeline (sequential)
|
||||
|
||||
`Pretrain` → `Full SFT` → optional (`LoRA` / `DPO` / `PPO` / `GRPO` / `Agent RL` / `Distillation`)
|
||||
|
||||
### Model (`model/model_minimind.py`)
|
||||
|
||||
- `MiniMindConfig`: HuggingFace PretrainedConfig with hidden_size=768, 8 layers, vocab_size=6400
|
||||
- `MiniMindForCausalLM`: Main model class (inherits PreTrainedModel + GenerationMixin)
|
||||
- Components: RMSNorm, RoPE + YaRN extrapolation, GQA (8 query heads, 4 KV heads), SwiGLU FFN, MoE with top-K routing and aux loss
|
||||
- Weight tying between input embedding and LM head (`tie_word_embeddings`)
|
||||
- Custom `generate()` with top-k/top-p sampling, repetition penalty, KV-cache
|
||||
- Model sizes controlled by `hidden_size`: 512 (minimind-3, 4 layers), 640 (minimind-2, 6 layers), 768 (default, 8 layers)
|
||||
|
||||
### LoRA (`model/model_lora.py`)
|
||||
|
||||
Pure PyTorch LoRA implementation (rank-16 A*B matrices). Functions: `apply_lora()`, `load_lora()`, `save_lora()`, `merge_lora()`.
|
||||
|
||||
### Datasets (`dataset/lm_dataset.py`)
|
||||
|
||||
- `PretrainDataset`: plain text JSONL
|
||||
- `SFTDataset`: multi-turn chat conversations
|
||||
- `DPODataset`: chosen/rejected pairs
|
||||
- `RLAIFDataset`: prompts for RL training
|
||||
- `AgentRLDataset`: multi-turn with tool calls
|
||||
|
||||
All data is JSONL format in `dataset/`.
|
||||
|
||||
### Shared Utilities (`trainer/trainer_utils.py`)
|
||||
|
||||
- `init_model()`: model + tokenizer initialization, handles DDP unwrapping and torch.compile
|
||||
- `lm_checkpoint()`: atomic checkpoint save/load with `os.replace()`, handles multi-GPU world_size changes on resume
|
||||
- `init_distributed_mode()`: DDP setup
|
||||
- `setup_seed()`: deterministic seeding
|
||||
- `LMForRewardModel`: wrapper for reward models (uses InternLM2-1.8B-Reward)
|
||||
|
||||
### Logging
|
||||
|
||||
SwanLab (default) or WandB (`--use_wandb`). Checkpoints store wandb_id for resume.
|
||||
|
||||
### Tokenizer (`model/`)
|
||||
|
||||
Custom BPE tokenizer with 6,400 vocab, 36 special tokens. Chat template supports tools, multi-turn, and reasoning_content. `train_tokenizer.py` is reference-only (pre-trained tokenizer is used).
|
||||
|
||||
## Key Conventions
|
||||
|
||||
- No test framework exists; `scripts/eval_toolcall.py` provides tool call evaluation
|
||||
- All training scripts use argparse with defaults matching the minimind-768 config
|
||||
- Checkpoints are saved to `../checkpoints/` relative to `trainer/`
|
||||
- Code comments are in Chinese
|
||||
|
|
@ -4,8 +4,10 @@ import json
|
|||
import os
|
||||
import random
|
||||
from datasets import load_dataset, Features, Sequence, Value
|
||||
|
||||
os.environ["TOKENIZERS_PARALLELISM"] = "false"
|
||||
|
||||
|
||||
def pre_processing_chat(conversations, add_system_ratio=0.2):
|
||||
# tool use 数据完整保留不做处理
|
||||
if any(conv.get('tools') for conv in conversations): return conversations
|
||||
|
|
@ -28,18 +30,20 @@ def pre_processing_chat(conversations, add_system_ratio=0.2):
|
|||
return [{'role': 'system', 'content': random.choice(SYSTEM_PROMPTS)}] + conversations
|
||||
return conversations
|
||||
|
||||
|
||||
def post_processing_chat(prompt_content, empty_think_ratio=0.2):
|
||||
# 以80%概率移除空思考标签
|
||||
if '<think>\n\n</think>\n\n' in prompt_content and random.random() > empty_think_ratio:
|
||||
prompt_content = prompt_content.replace('<think>\n\n</think>\n\n', '')
|
||||
return prompt_content
|
||||
|
||||
|
||||
class PretrainDataset(Dataset):
|
||||
def __init__(self, data_path, tokenizer, max_length=512):
|
||||
super().__init__()
|
||||
self.tokenizer = tokenizer
|
||||
self.max_length = max_length
|
||||
#load_dataset是HuggingFace datasets库上用来加载数据集的函数，可以加载HuggingFace Hub上的数据，也可以加载本地的json/csv/txt等文件
|
||||
# load_dataset是HuggingFace datasets库上用来加载数据集的函数，可以加载HuggingFace Hub上的数据，也可以加载本地的json/csv/txt等文件
|
||||
self.samples = load_dataset('json', data_files=data_path, split='train')
|
||||
|
||||
def __len__(self):
|
||||
|
|
@ -47,7 +51,8 @@ class PretrainDataset(Dataset):
|
|||
|
||||
def __getitem__(self, index):
|
||||
sample = self.samples[index]
|
||||
tokens = self.tokenizer(str(sample['text']), add_special_tokens=False, max_length=self.max_length - 2, truncation=True).input_ids
|
||||
tokens = self.tokenizer(str(sample['text']), add_special_tokens=False, max_length=self.max_length - 2,
|
||||
truncation=True).input_ids
|
||||
tokens = [self.tokenizer.bos_token_id] + tokens + [self.tokenizer.eos_token_id]
|
||||
input_ids = tokens + [self.tokenizer.pad_token_id] * (self.max_length - len(tokens))
|
||||
input_ids = torch.tensor(input_ids, dtype=torch.long)
|
||||
|
|
@ -61,7 +66,9 @@ class SFTDataset(Dataset):
|
|||
super().__init__()
|
||||
self.tokenizer = tokenizer
|
||||
self.max_length = max_length
|
||||
features = Features({'conversations': [{'role': Value('string'), 'content': Value('string'), 'reasoning_content': Value('string'), 'tools': Value('string'), 'tool_calls': Value('string')}]})
|
||||
features = Features({'conversations': [
|
||||
{'role': Value('string'), 'content': Value('string'), 'reasoning_content': Value('string'),
|
||||
'tools': Value('string'), 'tool_calls': Value('string')}]})
|
||||
self.samples = load_dataset('json', data_files=jsonl_path, split='train', features=features)
|
||||
self.bos_id = tokenizer(f'{tokenizer.bos_token}assistant\n', add_special_tokens=False).input_ids
|
||||
self.eos_id = tokenizer(f'{tokenizer.eos_token}\n', add_special_tokens=False).input_ids
|
||||
|
|
@ -74,15 +81,21 @@ class SFTDataset(Dataset):
|
|||
tools = None
|
||||
for message in conversations:
|
||||
message = dict(message)
|
||||
# 如果消息是system角色，且带了tools字段
|
||||
if message.get("role") == "system" and message.get("tools"):
|
||||
# json.loads把json格式字符串解析为python对象
|
||||
tools = json.loads(message["tools"]) if isinstance(message["tools"], str) else message["tools"]
|
||||
if message.get("tool_calls") and isinstance(message["tool_calls"], str):
|
||||
message["tool_calls"] = json.loads(message["tool_calls"])
|
||||
messages.append(message)
|
||||
return self.tokenizer.apply_chat_template(
|
||||
# messages是多轮对话列表
|
||||
messages,
|
||||
# 返回文本字符串，不做分词，如果是true则返回文本token id 列表
|
||||
tokenize=False,
|
||||
|
||||
add_generation_prompt=False,
|
||||
# 传入工具定义，模版会在开头插入工具描述
|
||||
tools=tools
|
||||
)
|
||||
|
||||
|
|
@ -215,6 +228,7 @@ class RLAIFDataset(Dataset):
|
|||
open_thinking=use_thinking,
|
||||
add_generation_prompt=True
|
||||
)
|
||||
|
||||
def __getitem__(self, index):
|
||||
sample = self.samples[index]
|
||||
prompt = self.create_chat_prompt(sample['conversations'])
|
||||
|
|
@ -224,6 +238,7 @@ class RLAIFDataset(Dataset):
|
|||
'answer': ""
|
||||
}
|
||||
|
||||
|
||||
class AgentRLDataset(Dataset):
|
||||
def __init__(self, jsonl_path, tokenizer, max_length=1024):
|
||||
super().__init__()
|
||||
|
|
|
|||
|
|
@ -14,9 +14,9 @@ from torch import optim, nn
|
|||
from torch.nn.parallel import DistributedDataParallel
|
||||
from torch.utils.data import DataLoader, DistributedSampler
|
||||
from model.model_minimind import MiniMindConfig
|
||||
from model.model_minimind_mla import MiniMindMLAConfig
|
||||
from dataset.lm_dataset import SFTDataset
|
||||
from trainer.trainer_utils import get_lr, Logger, is_main_process, lm_checkpoint, init_distributed_mode, setup_seed, init_model, SkipBatchSampler, get_model_suffix
|
||||
from trainer.trainer_utils import get_lr, Logger, is_main_process, lm_checkpoint, init_distributed_mode, setup_seed, \
|
||||
init_model, SkipBatchSampler
|
||||
|
||||
warnings.filterwarnings('ignore')
|
||||
|
||||
|
|
@ -55,19 +55,21 @@ def train_epoch(epoch, loader, iters, start_step=0, wandb=None):
|
|||
current_logits_loss = current_loss - current_aux_loss
|
||||
current_lr = optimizer.param_groups[-1]['lr']
|
||||
eta_min = spend_time / max(step - start_step, 1) * (iters - step) // 60
|
||||
Logger(f'Epoch:[{epoch + 1}/{args.epochs}]({step}/{iters}), loss: {current_loss:.4f}, logits_loss: {current_logits_loss:.4f}, aux_loss: {current_aux_loss:.4f}, lr: {current_lr:.8f}, epoch_time: {eta_min:.1f}min')
|
||||
if wandb: wandb.log({"loss": current_loss, "logits_loss": current_logits_loss, "aux_loss": current_aux_loss, "learning_rate": current_lr, "epoch_time": eta_min})
|
||||
Logger(
|
||||
f'Epoch:[{epoch + 1}/{args.epochs}]({step}/{iters}), loss: {current_loss:.4f}, logits_loss: {current_logits_loss:.4f}, aux_loss: {current_aux_loss:.4f}, lr: {current_lr:.8f}, epoch_time: {eta_min:.1f}min')
|
||||
if wandb: wandb.log({"loss": current_loss, "logits_loss": current_logits_loss, "aux_loss": current_aux_loss,
|
||||
"learning_rate": current_lr, "epoch_time": eta_min})
|
||||
|
||||
if (step % args.save_interval == 0 or step == iters) and is_main_process():
|
||||
model.eval()
|
||||
model_suffix = get_model_suffix(lm_config)
|
||||
ckp = f'{args.save_dir}/{args.save_weight}_{lm_config.hidden_size}{model_suffix}.pth'
|
||||
moe_suffix = '_moe' if lm_config.use_moe else ''
|
||||
ckp = f'{args.save_dir}/{args.save_weight}_{lm_config.hidden_size}{moe_suffix}.pth'
|
||||
raw_model = model.module if isinstance(model, DistributedDataParallel) else model
|
||||
raw_model = getattr(raw_model, '_orig_mod', raw_model)
|
||||
state_dict = raw_model.state_dict()
|
||||
torch.save({k: v.half().cpu() for k, v in state_dict.items()}, ckp)
|
||||
lm_checkpoint(lm_config, weight=args.save_weight, model=model, optimizer=optimizer,
|
||||
epoch=epoch, step=step, wandb=wandb, save_dir='../checkpoints', scaler=scaler)
|
||||
epoch=epoch, step=step, wandb=wandb, save_dir='../checkpoints', scaler=scaler)
|
||||
model.train()
|
||||
del state_dict
|
||||
|
||||
|
|
@ -86,7 +88,7 @@ if __name__ == "__main__":
|
|||
parser.add_argument("--save_dir", type=str, default="../out", help="模型保存目录")
|
||||
parser.add_argument('--save_weight', default='full_sft', type=str, help="保存权重的前缀名")
|
||||
parser.add_argument("--epochs", type=int, default=2, help="训练轮数")
|
||||
parser.add_argument("--batch_size", type=int, default=16, help="batch size")
|
||||
parser.add_argument("--batch_size", type=int, default=48, help="batch size")
|
||||
parser.add_argument("--learning_rate", type=float, default=1e-5, help="初始学习率")
|
||||
parser.add_argument("--device", type=str, default="cuda:0" if torch.cuda.is_available() else "cpu", help="训练设备")
|
||||
parser.add_argument("--dtype", type=str, default="bfloat16", help="混合精度类型")
|
||||
|
|
@ -99,14 +101,14 @@ if __name__ == "__main__":
|
|||
parser.add_argument('--num_hidden_layers', default=8, type=int, help="隐藏层数量")
|
||||
parser.add_argument('--max_seq_len', default=768, type=int, help="训练的最大截断长度（中文1token≈1.5~1.7字符）")
|
||||
parser.add_argument('--use_moe', default=0, type=int, choices=[0, 1], help="是否使用MoE架构（0=否，1=是）")
|
||||
parser.add_argument('--use_mla', default=0, type=int, choices=[0, 1], help="是否使用MLA注意力架构（0=否，1=是）")
|
||||
parser.add_argument('--kv_lora_rank', default=128, type=int, help="MLA的KV压缩秩（仅use_mla=1时生效）")
|
||||
parser.add_argument("--data_path", type=str, default="../dataset/sft_t2t_mini.jsonl", help="训练数据路径")
|
||||
parser.add_argument('--from_weight', default='pretrain', type=str, help="基于哪个权重训练，为none则不基于任何权重训练")
|
||||
parser.add_argument('--from_weight', default='pretrain', type=str,
|
||||
help="基于哪个权重训练，为none则不基于任何权重训练")
|
||||
parser.add_argument('--from_resume', default=0, type=int, choices=[0, 1], help="是否自动检测&续训（0=否，1=是）")
|
||||
parser.add_argument("--use_wandb", action="store_true", help="是否使用wandb")
|
||||
parser.add_argument("--wandb_project", type=str, default="MiniMind-Full-SFT", help="wandb项目名")
|
||||
parser.add_argument("--use_compile", default=0, type=int, choices=[0, 1], help="是否使用torch.compile加速（0=否，1=是）")
|
||||
parser.add_argument("--use_compile", default=0, type=int, choices=[0, 1],
|
||||
help="是否使用torch.compile加速（0=否，1=是）")
|
||||
args = parser.parse_args()
|
||||
|
||||
# ========== 1. 初始化环境和随机种子 ==========
|
||||
|
|
@ -116,11 +118,10 @@ if __name__ == "__main__":
|
|||
|
||||
# ========== 2. 配置目录、模型参数、检查ckp ==========
|
||||
os.makedirs(args.save_dir, exist_ok=True)
|
||||
if args.use_mla:
|
||||
lm_config = MiniMindMLAConfig(hidden_size=args.hidden_size, num_hidden_layers=args.num_hidden_layers, use_moe=bool(args.use_moe), kv_lora_rank=args.kv_lora_rank)
|
||||
else:
|
||||
lm_config = MiniMindConfig(hidden_size=args.hidden_size, num_hidden_layers=args.num_hidden_layers, use_moe=bool(args.use_moe))
|
||||
ckp_data = lm_checkpoint(lm_config, weight=args.save_weight, save_dir='../checkpoints') if args.from_resume==1 else None
|
||||
lm_config = MiniMindConfig(hidden_size=args.hidden_size, num_hidden_layers=args.num_hidden_layers,
|
||||
use_moe=bool(args.use_moe))
|
||||
ckp_data = lm_checkpoint(lm_config, weight=args.save_weight,
|
||||
save_dir='../checkpoints') if args.from_resume == 1 else None
|
||||
|
||||
# ========== 3. 设置混合精度 ==========
|
||||
device_type = "cuda" if "cuda" in args.device else "cpu"
|
||||
|
|
@ -131,6 +132,7 @@ if __name__ == "__main__":
|
|||
wandb = None
|
||||
if args.use_wandb and is_main_process():
|
||||
import swanlab as wandb
|
||||
|
||||
wandb_id = ckp_data.get('wandb_id') if ckp_data else None
|
||||
resume = 'must' if wandb_id else None
|
||||
wandb_run_name = f"MiniMind-Full-SFT-Epoch-{args.epochs}-BatchSize-{args.batch_size}-LearningRate-{args.learning_rate}"
|
||||
|
|
@ -162,7 +164,8 @@ if __name__ == "__main__":
|
|||
# ========== 8. 开始训练 ==========
|
||||
for epoch in range(start_epoch, args.epochs):
|
||||
train_sampler and train_sampler.set_epoch(epoch)
|
||||
setup_seed(42 + epoch); indices = torch.randperm(len(train_ds)).tolist()
|
||||
setup_seed(42 + epoch);
|
||||
indices = torch.randperm(len(train_ds)).tolist()
|
||||
skip = start_step if (epoch == start_epoch and start_step > 0) else 0
|
||||
batch_sampler = SkipBatchSampler(train_sampler or indices, args.batch_size, skip)
|
||||
loader = DataLoader(train_ds, batch_sampler=batch_sampler, num_workers=args.num_workers, pin_memory=True)
|
||||
|
|
|
|||
Loading…
Reference in New Issue