Merge 56a21ed3c2 into 1e6e909f88

fix: guard against empty/filtered LLM responses in chat_api.py

README.md

@@ -486,6 +486,8 @@ torchrun --nproc_per_node N train_xxx.py
 
 MiniMind训练数据集下载地址： [ModelScope](https://www.modelscope.cn/datasets/gongjy/minimind_dataset/files) | [HuggingFace](https://huggingface.co/datasets/jingyaogong/minimind_dataset/tree/main)
 
+下载、文件用途、最终格式与自定义数据检查清单可参考[数据集双语速查指南](./dataset/dataset.md)。
+
 > 无需全部clone，可单独下载所需的文件
 
 将下载的数据集文件放到`./dataset/`目录下（✨为推荐的必须项）

README_en.md

@@ -485,6 +485,8 @@ Besides this, other RL data maintains the same format as SFT data, typically fil
 
 MiniMind training dataset download links: [ModelScope](https://www.modelscope.cn/datasets/gongjy/minimind_dataset/files) | [HuggingFace](https://huggingface.co/datasets/jingyaogong/minimind_dataset/tree/main)
 
+For downloads, file-to-trainer mapping, final schemas, and a custom-data checklist, see the [bilingual dataset quick reference](./dataset/dataset.md).
+
 > No need to clone everything, you can download individual files as needed
 
 Place the downloaded dataset files in the `./dataset/` directory (✨ indicates recommended essentials)

dataset/dataset.md

@@ -1,5 +1,219 @@
-# MiniMind Datasets
+# MiniMind 数据集指南 / Dataset Guide
-
+
-将所有下载的数据集文件放置到当前目录.
+[中文](#user-content-chinese) | [English](#user-content-english)
-
+
-Place the downloaded dataset file in the current directory.
+<a id="chinese"></a>
+
+## 中文
+
+本目录存放 MiniMind 各训练阶段使用的最终 JSONL 文件。官方发布的数据已经完成收集、蒸馏、清洗、去重、长度控制与格式统一，可直接由 `dataset/lm_dataset.py` 中的数据加载器读取。
+
+### 下载与最小复现
+
+数据集可从 [ModelScope](https://www.modelscope.cn/datasets/gongjy/minimind_dataset/files) 或 [Hugging Face](https://huggingface.co/datasets/jingyaogong/minimind_dataset/tree/main) 按文件下载，无需克隆全部数据。
+
+将文件放在当前目录：
+
+```text
+dataset/
+├── pretrain_t2t_mini.jsonl
+├── sft_t2t_mini.jsonl
+├── pretrain_t2t.jsonl
+├── sft_t2t.jsonl
+├── dpo.jsonl
+├── rlaif.jsonl
+├── agent_rl.jsonl
+└── agent_rl_math.jsonl
+```
+
+只想快速跑通从预训练到对话模型的流程时，下载以下两个文件即可：
+
+- `pretrain_t2t_mini.jsonl`
+- `sft_t2t_mini.jsonl`
+
+然后从仓库根目录运行：
+
+```bash
+cd trainer
+python train_pretrain.py
+python train_full_sft.py
+```
+
+### 文件与训练脚本对应关系
+
+| 文件 | 用途 | 默认使用者 |
+| --- | --- | --- |
+| `pretrain_t2t_mini.jsonl` | 快速预训练 | `train_pretrain.py` |
+| `pretrain_t2t.jsonl` | 完整 MiniMind-3 预训练 | 通过 `train_pretrain.py --data_path ...` 指定 |
+| `sft_t2t_mini.jsonl` | 快速 SFT、蒸馏及 LoRA 自定义数据格式参考 | `train_full_sft.py`、`train_distillation.py` |
+| `sft_t2t.jsonl` | 完整 MiniMind-3 SFT | 通过 `train_full_sft.py --data_path ...` 指定 |
+| `dpo.jsonl` | 偏好优化 | `train_dpo.py` |
+| `rlaif.jsonl` | PPO、GRPO 与 CISPO | `train_ppo.py`、`train_grpo.py` |
+| `agent_rl.jsonl` | 多轮 Tool-Use Agentic RL | `train_agent.py` |
+| `agent_rl_math.jsonl` | 带可验证最终答案的数学 Agentic RL | 通过 `train_agent.py --data_path ...` 指定 |
+
+上表中的训练脚本都支持 `--data_path`，因此也可直接指向自己的 JSONL 文件。
+
+### 最终数据格式
+
+预训练数据每行包含一个 `text` 字段：
+
+```jsonl
+{"text": "Transformer 通过自注意力机制建模上下文关系。"}
+```
+
+SFT、RLAIF 与常规对话数据使用 `conversations` 数组：
+
+```json
+{
+  "conversations": [
+    {"role": "user", "content": "你好"},
+    {"role": "assistant", "content": "你好！"}
+  ]
+}
+```
+
+DPO 数据分别提供偏好回复与非偏好回复：
+
+```json
+{
+  "chosen": [
+    {"role": "user", "content": "Q"},
+    {"role": "assistant", "content": "good answer"}
+  ],
+  "rejected": [
+    {"role": "user", "content": "Q"},
+    {"role": "assistant", "content": "bad answer"}
+  ]
+}
+```
+
+Agentic RL 数据还可在 system 消息中包含序列化的 `tools`，并用顶层 `gt` 字段提供可验证目标。具体解析逻辑以 [`lm_dataset.py`](./lm_dataset.py) 为准。
+
+### 来源、处理与自定义数据
+
+当前公开版本使用的主要社区来源包括：
+
+- [匠数大模型数据集](https://www.modelscope.cn/datasets/deepctrl/deepctrl-sft-data)
+- [Magpie-Align](https://www.modelscope.cn/organization/Magpie-Align)
+- [R1-Distill-SFT](https://www.modelscope.cn/datasets/AI-ModelScope/R1-Distill-SFT)
+- [COIG](https://huggingface.co/datasets/BAAI/COIG)
+- [Step-3.5-Flash-SFT](https://huggingface.co/datasets/stepfun-ai/Step-3.5-Flash-SFT)
+- [DPO-En-Zh-20k](https://huggingface.co/datasets/llamafactory/DPO-En-Zh-20k)
+
+本仓库目前提供训练所需的最终数据、格式说明和加载器；原始数据的完整收集、蒸馏与清洗流水线并未作为可执行脚本包含在本目录中。准备自定义数据时应：
+
+1. 确认每个来源允许当前用途与再分发方式，并保留来源和许可证记录。
+2. 完成去重、质量过滤、隐私信息清理与长度控制。
+3. 转换为上面的最终 JSONL 格式。
+4. 先用小样本运行对应训练脚本，确认 schema 和 chat template 正确，再开始完整训练。
+
+数据来源、推荐组合、Tool Calling 格式与 token 长度说明以仓库根目录的[中文 README](../README.md#-数据介绍)为准。
+
+<a id="english"></a>
+
+## English
+
+This directory stores the final JSONL files consumed by each MiniMind training stage. The published datasets have already gone through collection, distillation, cleaning, deduplication, length control, and format normalization, and can be read directly by the loaders in `dataset/lm_dataset.py`.
+
+### Download and minimal reproduction
+
+Download individual files from [ModelScope](https://www.modelscope.cn/datasets/gongjy/minimind_dataset/files) or [Hugging Face](https://huggingface.co/datasets/jingyaogong/minimind_dataset/tree/main). You do not need to clone the full dataset repository.
+
+Place the downloaded files in this directory:
+
+```text
+dataset/
+├── pretrain_t2t_mini.jsonl
+├── sft_t2t_mini.jsonl
+├── pretrain_t2t.jsonl
+├── sft_t2t.jsonl
+├── dpo.jsonl
+├── rlaif.jsonl
+├── agent_rl.jsonl
+└── agent_rl_math.jsonl
+```
+
+For the quickest pretraining-to-chat-model reproduction, only these two files are required:
+
+- `pretrain_t2t_mini.jsonl`
+- `sft_t2t_mini.jsonl`
+
+Then run from the repository root:
+
+```bash
+cd trainer
+python train_pretrain.py
+python train_full_sft.py
+```
+
+### Dataset-to-trainer map
+
+| File | Purpose | Default consumer |
+| --- | --- | --- |
+| `pretrain_t2t_mini.jsonl` | Quick pretraining | `train_pretrain.py` |
+| `pretrain_t2t.jsonl` | Full MiniMind-3 pretraining | Pass with `train_pretrain.py --data_path ...` |
+| `sft_t2t_mini.jsonl` | Quick SFT, distillation, and the reference schema for custom LoRA data | `train_full_sft.py`, `train_distillation.py` |
+| `sft_t2t.jsonl` | Full MiniMind-3 SFT | Pass with `train_full_sft.py --data_path ...` |
+| `dpo.jsonl` | Preference optimization | `train_dpo.py` |
+| `rlaif.jsonl` | PPO, GRPO, and CISPO | `train_ppo.py`, `train_grpo.py` |
+| `agent_rl.jsonl` | Multi-turn Tool-Use Agentic RL | `train_agent.py` |
+| `agent_rl_math.jsonl` | Math Agentic RL with verifiable final answers | Pass with `train_agent.py --data_path ...` |
+
+Every trainer listed above accepts `--data_path`, so it can also read a custom JSONL file directly.
+
+### Final data schemas
+
+Each pretraining line contains a `text` field:
+
+```jsonl
+{"text": "Transformers model context with self-attention."}
+```
+
+SFT, RLAIF, and regular conversation data use a `conversations` array:
+
+```json
+{
+  "conversations": [
+    {"role": "user", "content": "Hello"},
+    {"role": "assistant", "content": "Hello!"}
+  ]
+}
+```
+
+DPO data provides preferred and rejected conversations separately:
+
+```json
+{
+  "chosen": [
+    {"role": "user", "content": "Q"},
+    {"role": "assistant", "content": "good answer"}
+  ],
+  "rejected": [
+    {"role": "user", "content": "Q"},
+    {"role": "assistant", "content": "bad answer"}
+  ]
+}
+```
+
+Agentic RL records may also contain serialized `tools` on the system message and a top-level `gt` field for verifiable targets. See [`lm_dataset.py`](./lm_dataset.py) for the authoritative parsing behavior.
+
+### Sources, processing, and custom data
+
+Major community sources used by the current published datasets include:
+
+- [Craftsman LLM Dataset](https://www.modelscope.cn/datasets/deepctrl/deepctrl-sft-data)
+- [Magpie-Align](https://www.modelscope.cn/organization/Magpie-Align)
+- [R1-Distill-SFT](https://www.modelscope.cn/datasets/AI-ModelScope/R1-Distill-SFT)
+- [COIG](https://huggingface.co/datasets/BAAI/COIG)
+- [Step-3.5-Flash-SFT](https://huggingface.co/datasets/stepfun-ai/Step-3.5-Flash-SFT)
+- [DPO-En-Zh-20k](https://huggingface.co/datasets/llamafactory/DPO-En-Zh-20k)
+
+This repository currently provides the final training datasets, their schemas, and the loaders. The complete raw-data collection, distillation, and cleaning pipeline is not included here as an executable script. When preparing custom data:
+
+1. Confirm that each source permits the intended use and redistribution, and retain provenance and license records.
+2. Deduplicate, quality-filter, remove private information, and control sample lengths.
+3. Convert records to the final JSONL schemas above.
+4. Run the matching trainer on a small sample to validate the schema and chat template before starting a full training run.
+
+For source details, recommended dataset combinations, Tool Calling schemas, and token-length guidance, see the [English README](../README_en.md#-data-introduction).

dataset/lm_dataset.py

@@ -28,10 +28,13 @@ def pre_processing_chat(conversations, add_system_ratio=0.2):
             return [{'role': 'system', 'content': random.choice(SYSTEM_PROMPTS)}] + conversations
     return conversations
 
-def post_processing_chat(prompt_content, empty_think_ratio=0.2):
+def post_processing_chat(prompt_content, empty_think_ratio=0.2, remove_empty_think=None):
     # 以80%概率移除空思考标签
-    if '<think>\n\n</think>\n\n' in prompt_content and random.random() > empty_think_ratio:
+    if '<think>\n\n</think>\n\n' in prompt_content:
-        prompt_content = prompt_content.replace('<think>\n\n</think>\n\n', '')
+        if remove_empty_think is None:
+            remove_empty_think = random.random() > empty_think_ratio
+        if remove_empty_think:
+            prompt_content = prompt_content.replace('<think>\n\n</think>\n\n', '')
     return prompt_content
 
 class PretrainDataset(Dataset):
@@ -139,12 +142,13 @@ class DPODataset(Dataset):
         chosen_prompt = self.tokenizer.apply_chat_template(
             chosen, tokenize=False, add_generation_prompt=False
         )
-        chosen_prompt = post_processing_chat(chosen_prompt)
 
         rejected_prompt = self.tokenizer.apply_chat_template(
             rejected, tokenize=False, add_generation_prompt=False
         )
-        rejected_prompt = post_processing_chat(rejected_prompt)
+        remove_empty_think = random.random() > 0.2
+        chosen_prompt = post_processing_chat(chosen_prompt, remove_empty_think=remove_empty_think)
+        rejected_prompt = post_processing_chat(rejected_prompt, remove_empty_think=remove_empty_think)
         chosen_encoding = self.tokenizer(
             chosen_prompt, truncation=True, max_length=self.max_length, padding='max_length'
         )
@@ -253,4 +257,4 @@ class AgentRLDataset(Dataset):
 
 
 if __name__ == "__main__":
-    pass
+    pass

model/model_minimind.py

@@ -125,10 +125,10 @@ class Attention(nn.Module):
         if self.flash and (seq_len > 1) and (not self.is_causal or past_key_value is None) and (attention_mask is None or torch.all(attention_mask == 1)):
             output = F.scaled_dot_product_attention(xq, xk, xv, dropout_p=self.dropout if self.training else 0.0, is_causal=self.is_causal)
         else:
-            scores = (xq @ xk.transpose(-2, -1)) / math.sqrt(self.head_dim)
+            scores = (xq @ xk.transpose(-2, -1)).float() / math.sqrt(self.head_dim)
             if self.is_causal: scores[:, :, :, -seq_len:] += torch.full((seq_len, seq_len), float("-inf"), device=scores.device).triu(1)
             if attention_mask is not None: scores += (1.0 - attention_mask.unsqueeze(1).unsqueeze(2)) * -1e9
-            output = self.attn_dropout(F.softmax(scores.float(), dim=-1).type_as(xq)) @ xv
+            output = self.attn_dropout(F.softmax(scores, dim=-1).type_as(xq)) @ xv
         output = output.transpose(1, 2).reshape(bsz, seq_len, -1)
         output = self.resid_dropout(self.o_proj(output))
         return output, past_kv

requirements.txt

@@ -28,6 +28,8 @@ streamlit==1.50.0
 einops==0.8.1
 swanlab==0.9.8
 modelscope==1.37.0
+fastapi==0.120.1
+uvicorn==0.38.0
 # torch==2.6.0
 # torchvision==0.21.0
 # peft==0.7.1

scripts/serve_openai_api.py

@@ -202,7 +202,7 @@ async def chat_completions(request: ChatRequest):
             with torch.no_grad():
                 generated_ids = model.generate(
                     inputs["input_ids"],
-                    max_length=inputs["input_ids"].shape[1] + request.max_tokens,
+                    max_new_tokens=request.max_tokens,
                     do_sample=True,
                     attention_mask=inputs["attention_mask"],
                     pad_token_id=tokenizer.pad_token_id,

trainer/train_agent.py

@@ -273,8 +273,6 @@ def rl_train_epoch(epoch, loader, iters, rollout_engine, ref_model, reward_model
         rewards = calculate_rewards(prompts, completions, gt_batch, tools_batch, args.num_generations, reward_model, device=args.device, turn_outputs_batch=turn_outputs_batch, unfinished_batch=unfinished_batch)
 
         with autocast_ctx:
-            # 反向传播必须经过 DDP 包装后的模块：直接调用 .module 会跳过
-            # DDP 的 prepare_for_backward，梯度不会 all-reduce，各卡静默发散。
             res = model(input_ids, attention_mask=full_mask)
             aux_loss = res.aux_loss if lm_config.use_moe else torch.tensor(0.0, device=args.device)
             logits = res.logits[:, :-1, :]
@@ -471,7 +469,6 @@ if __name__ == "__main__":
     if args.use_compile == 1:
         model = torch.compile(model)
         Logger('torch.compile enabled')
-        rollout_engine.update_policy(model)
     if dist.is_initialized():
         # 同 train_ppo：RoPE buffer 各 rank 一致，每步广播纯属浪费
         model = DistributedDataParallel(model, device_ids=[local_rank], broadcast_buffers=False)

trainer/train_grpo.py

@@ -96,8 +96,6 @@ def grpo_train_epoch(epoch, loader, iters, rollout_engine, ref_model, reward_mod
         rewards = calculate_rewards(prompts, completions, reward_model).to(args.device)  # [B*num_gen]
 
         with autocast_ctx:
-            # 反向传播必须经过 DDP 包装后的模块：直接调用 .module 会跳过
-            # DDP 的 prepare_for_backward，梯度不会 all-reduce，各卡静默发散。
             res = model(outputs, attention_mask=full_mask)
             aux_loss = res.aux_loss if lm_config.use_moe else torch.tensor(0.0, device=args.device)
             per_token_logps = F.log_softmax(res.logits[:, :-1, :], dim=-1).gather(2, outputs[:, 1:].unsqueeze(-1)).squeeze(-1).gather(1, logp_pos)
@@ -312,7 +310,6 @@ if __name__ == "__main__":
     if args.use_compile == 1:
         model = torch.compile(model)
         Logger('torch.compile enabled')
-        rollout_engine.update_policy(model)
     if dist.is_initialized():
         # 同 train_ppo：RoPE buffer 各 rank 一致，每步广播纯属浪费
         model = DistributedDataParallel(model, device_ids=[local_rank], broadcast_buffers=False)

trainer/train_ppo.py

@@ -166,16 +166,12 @@ def ppo_train_epoch(epoch, loader, iters, rollout_engine, ref_model, actor_sched
             for i in range(0, B, mb_size):
                 inds = b_inds[i:i + mb_size]
                 
-                # 反向传播必须经过 DDP 包装后的模块：直接调用 .module 会跳过
-                # DDP 的 prepare_for_backward，梯度不会 all-reduce，各卡静默发散。
                 mb_values_seq = critic_model(input_ids=gen_out[inds], attention_mask=full_mask[inds])
                 mb_resp_values = mb_values_seq.gather(1, logp_pos[inds])
 
                 with autocast_ctx:
                     res = actor_model(input_ids=gen_out[inds], attention_mask=full_mask[inds])
                     aux_loss = res.aux_loss if lm_config.use_moe else torch.tensor(0.0, device=args.device)
-                    # 在 autocast 内计算 log_softmax，避免直接对 fp16/bf16 logits
-                    # 计算造成额外数值偏差。
                     mb_resp_logp = F.log_softmax(res.logits[:, :-1], dim=-1).gather(2, labels[inds].unsqueeze(-1)).squeeze(-1).gather(1, logp_pos[inds])
 
                 log_ratio = mb_resp_logp - old_resp_logp[inds]
@@ -427,7 +423,6 @@ if __name__ == "__main__":
     if args.use_compile == 1:
         actor_model = torch.compile(actor_model)
         Logger('torch.compile enabled')
-        rollout_engine.update_policy(actor_model)
     if dist.is_initialized():
         # freqs_cos/freqs_sin 各 rank 由 config 确定性算出，默认每步广播一次纯属浪费
         actor_model = DistributedDataParallel(actor_model, device_ids=[local_rank], broadcast_buffers=False)

trainer/train_tokenizer.py

@@ -8,18 +8,39 @@ DATA_PATH = '../dataset/sft_t2t_mini.jsonl'
 TOKENIZER_DIR = '../model_learn_tokenizer/'
 VOCAB_SIZE = 6400
 SPECIAL_TOKENS_NUM = 36
+MAX_LINES = 0  # 0 表示读取全部；设为正数则只取前 N 条（快速试跑）
 
-def get_texts(data_path):
+def get_texts(data_path, max_lines=MAX_LINES):
+    """逐行产出用于训练 BPE 的文本，兼容两种主线数据格式。
+
+    pretrain_t2t(_mini).jsonl 是 {"text": ...}，
+    sft_t2t(_mini).jsonl 是 {"conversations": [{"role":..., "content":...}, ...]}。
+    原先只解析 conversations，所以把 DATA_PATH 指向预训练语料时一条文本都取不到，
+    BPE 仅基于 initial_alphabet 训练，产出一个没有任何 merge 的词表，且全程不报错。
+    """
+    used = 0
     with open(data_path, 'r', encoding='utf-8', errors='ignore') as f:
-        for i, line in enumerate(f):
+        for line in f:
-            if i >= 10000: break # 选10000行测试
+            if max_lines and used >= max_lines:
+                break
             try:
                 data = json.loads(line)
-                contents = [item.get('content') for item in data.get('conversations', []) if item.get('content')]
-                if contents:
-                    yield "\n".join(contents)
             except json.JSONDecodeError:
                 continue
+            if 'text' in data:
+                text = str(data['text'])
+            else:
+                contents = [item.get('content') for item in data.get('conversations', []) if item.get('content')]
+                text = "\n".join(contents) if contents else ''
+            if text.strip():
+                used += 1
+                yield text
+    if used == 0:
+        raise ValueError(
+            f'{data_path} 中没有可用文本：每行应为 {{"text": ...}} 或 '
+            f'{{"conversations": [{{"role":..., "content":...}}, ...]}}。'
+            f'继续训练只会得到一个空词表。'
+        )
 
 def train_tokenizer(data_path, tokenizer_dir, vocab_size, special_tokens_num=SPECIAL_TOKENS_NUM):
     tokenizer = Tokenizer(models.BPE())