Merge d9ad86500f into 3812a16d20
Daisy
GitHub
commit
c4a1b4ada9
|
|
@ -1,3 +1,4 @@
|
|||
/model/__pycache__
|
||||
**/__pycache__
|
||||
/dataset
|
||||
/out
|
||||
/out
|
||||
**/.ipynb_checkpoints
|
||||
|
|
@ -0,0 +1,448 @@
|
|||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"vscode": {
|
||||
"languageId": "plaintext"
|
||||
}
|
||||
},
|
||||
"source": [
|
||||
"# 0-Tokenizer\n",
|
||||
"\n",
|
||||
"Tokenizer（分词器）在 NLP 领域扮演着基础且关键的作用，它将文本分割成单词或子词并转化为数组编号，为模型提供可处理的输入，在文本预处理、语义理解及适配不同语言和任务等方面奠定基础，是连接自然语言文本与计算机可处理数据的重要桥梁."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"vscode": {
|
||||
"languageId": "plaintext"
|
||||
}
|
||||
},
|
||||
"source": [
|
||||
"## 子词分词算法\n",
|
||||
"\n",
|
||||
"常见的子词分词算法有三种：\n",
|
||||
"\n",
|
||||
"1. 字节对编码（Byte Pair Encoding，BPE）\n",
|
||||
"2. WordPiece\n",
|
||||
"3. Unigram\n",
|
||||
"\n",
|
||||
"### BPE\n",
|
||||
"\n",
|
||||
"BPE 是一种简单的数据压缩技术，它会迭代地替换序列中最频繁出现的字节对。BPE 依赖一个预分词器，该预分词器会将训练数据分割成单词（在本项目中，我们使用按空格分词的方法作为预分词方法）.\n",
|
||||
"\n",
|
||||
"在预分词之后，会创建一组唯一的单词，并确定它们在数据中的出现频率。接下来，BPE 会创建一个基础词表，该词表包含预分词器最初生成的数据中所有唯一单词的符号。然后，会将这对符号从词表中移除，新形成的符号将加入词表。在迭代过程中，BPE 算法会合并频繁出现的符号对.\n",
|
||||
"\n",
|
||||
"给定词表的大小，BPE（字节对编码）算法最终会合并出现频率最高的符号对，直到收敛到该大小.\n",
|
||||
"\n",
|
||||
"### WordPiece\n",
|
||||
"\n",
|
||||
"WordPiece 算法与 BPE 非常相似。WordPiece 首先将词表初始化为包含训练数据中出现的每个字符，然后逐步学习给定数量的合并规则. 与 BPE 不同的是，WordPiece 并不选择最频繁出现的符号对，而是选择那个加入词表后能使训练数据出现的可能性最大化的符号对.\n",
|
||||
"\n",
|
||||
"### Unigram\n",
|
||||
"\n",
|
||||
"Unigram 算法将其基础词表初始化为大量的符号，然后逐步削减每个符号，以获得一个更小的词表。它会在训练数据上定义一个对数似然损失，以此来确定是否从词表中移除某个符号."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 训练一个最简单的分词器\n",
|
||||
"\n",
|
||||
"在本节中，我们将学习基于 transformers 库来训练你自己的分词器."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### 初始化\n",
|
||||
"\n",
|
||||
"首先，我们应该初始化我们的分词器，并确定选择哪种方法。我们将使用字节对编码（BPE）算法."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from tokenizers import (\n",
|
||||
" decoders,\n",
|
||||
" models,\n",
|
||||
" normalizers,\n",
|
||||
" pre_tokenizers,\n",
|
||||
" processors,\n",
|
||||
" trainers,\n",
|
||||
" Tokenizer,\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"tokenizer = Tokenizer(models.BPE())\n",
|
||||
"tokenizer.pre_tokenizer = pre_tokenizers.ByteLevel(add_prefix_space=False)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### 定义特殊标记\n",
|
||||
"\n",
|
||||
"数据集中存在一些我们不希望被分词的特殊标记，我们会将这些标记定义为特殊标记，并将它们传递给分词器训练器，以防止出现错误的分词情况."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 2,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"special_tokens = [\"<unk>\", \"<s>\", \"</s>\"]\n",
|
||||
"\n",
|
||||
"trainer = trainers.BpeTrainer(\n",
|
||||
" vocab_size=256,\n",
|
||||
" special_tokens=special_tokens,\n",
|
||||
" show_progress=True,\n",
|
||||
" initial_alphabet=pre_tokenizers.ByteLevel.alphabet()\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### 从文件中读取数据\n",
|
||||
"\n",
|
||||
"在本次实验中，我们使用 JSON Lines（jsonl）格式来存储 Tokenizer 训练数据，分词器内置的训练函数要求训练数据以迭代器的形式传入，因此，我们首先获取一个数据读取的生成器."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 3,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import json\n",
|
||||
"\n",
|
||||
"def read_texts_from_jsonl(file_path):\n",
|
||||
" with open(file_path, 'r', encoding='utf-8') as f:\n",
|
||||
" for line in f:\n",
|
||||
" data = json.loads(line)\n",
|
||||
" yield data['text']\n",
|
||||
"\n",
|
||||
"data_path = './toydata/tokenizer_data.jsonl'"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Row 1: <s>近年来，人工智能技术迅速发展，深刻改变了各行各业的面貌。机器学习、自然语言处理、计算机视觉等领域的突破性进展，使得智能产品和服务越来越普及。从智能家居到自动驾驶，再到智能医疗，AI的应用场景正在快速拓展。随着技术的不断进步，未来的人工智能将更加智能、更加贴近人类生活。</s>\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"data_iter = read_texts_from_jsonl(data_path)\n",
|
||||
"print(f'Row 1: {next(data_iter)}')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### 开始训练!\n",
|
||||
"\n",
|
||||
"我们使用分词器的内置函数 `tokenizer.train_from_iterator` 来训练分词器."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 5,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"tokenizer.train_from_iterator(data_iter, trainer=trainer)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### 设置解码器"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 6,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"tokenizer.decoder = decoders.ByteLevel()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"接下来，检查一下特殊标记是否得到了妥善处理。"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 7,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"assert tokenizer.token_to_id('<unk>') == 0\n",
|
||||
"assert tokenizer.token_to_id('<s>') == 1\n",
|
||||
"assert tokenizer.token_to_id('</s>') == 2"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### 将训练好的分词器保存到磁盘"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 8,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"['./model/toy_tokenizer\\\\vocab.json', './model/toy_tokenizer\\\\merges.txt']"
|
||||
]
|
||||
},
|
||||
"execution_count": 8,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"import os\n",
|
||||
"\n",
|
||||
"tokenizer_dir = \"./model/toy_tokenizer\"\n",
|
||||
"os.makedirs(tokenizer_dir, exist_ok=True)\n",
|
||||
"tokenizer.save(os.path.join(tokenizer_dir, \"tokenizer.json\")) # At this point, you will see a file named tokenizer.json under tokenizer_dir\n",
|
||||
"tokenizer.model.save(tokenizer_dir) # generate vocab.json & merges.txt"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### 手动创建一份配置文件"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 9,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Tokenizer training completed and saved.\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"config = {\n",
|
||||
" \"add_bos_token\": False,\n",
|
||||
" \"add_eos_token\": False,\n",
|
||||
" \"add_prefix_space\": False,\n",
|
||||
" \"added_tokens_decoder\": {\n",
|
||||
" \"0\": {\n",
|
||||
" \"content\": \"<unk>\",\n",
|
||||
" \"lstrip\": False,\n",
|
||||
" \"normalized\": False,\n",
|
||||
" \"rstrip\": False,\n",
|
||||
" \"single_word\": False,\n",
|
||||
" \"special\": True\n",
|
||||
" },\n",
|
||||
" \"1\": {\n",
|
||||
" \"content\": \"<s>\",\n",
|
||||
" \"lstrip\": False,\n",
|
||||
" \"normalized\": False,\n",
|
||||
" \"rstrip\": False,\n",
|
||||
" \"single_word\": False,\n",
|
||||
" \"special\": True\n",
|
||||
" },\n",
|
||||
" \"2\": {\n",
|
||||
" \"content\": \"</s>\",\n",
|
||||
" \"lstrip\": False,\n",
|
||||
" \"normalized\": False,\n",
|
||||
" \"rstrip\": False,\n",
|
||||
" \"single_word\": False,\n",
|
||||
" \"special\": True\n",
|
||||
" }\n",
|
||||
" },\n",
|
||||
" \"additional_special_tokens\": [],\n",
|
||||
" \"bos_token\": \"<s>\",\n",
|
||||
" \"clean_up_tokenization_spaces\": False,\n",
|
||||
" \"eos_token\": \"</s>\",\n",
|
||||
" \"legacy\": True,\n",
|
||||
" \"model_max_length\": 32768,\n",
|
||||
" \"pad_token\": \"<unk>\",\n",
|
||||
" \"sp_model_kwargs\": {},\n",
|
||||
" \"spaces_between_special_tokens\": False,\n",
|
||||
" \"tokenizer_class\": \"PreTrainedTokenizerFast\",\n",
|
||||
" \"unk_token\": \"<unk>\",\n",
|
||||
" \"chat_template\": \"{{ '<s>' + messages[0]['text'] + '</s>' }}\"\n",
|
||||
"}\n",
|
||||
"\n",
|
||||
"with open(os.path.join(tokenizer_dir, \"tokenizer_config.json\"), \"w\", encoding=\"utf-8\") as config_file:\n",
|
||||
" json.dump(config, config_file, ensure_ascii=False, indent=4)\n",
|
||||
"\n",
|
||||
"print(\"Tokenizer training completed and saved.\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"现在我们已经训练了一个简单的分词器，并将其进行保存，接下来，我们试着加载它，并使用其帮助我们对文本进行编解码."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 10,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"原始文本：[{'text': '失去的东西就要学着去接受，学着放下。'}]\n",
|
||||
"修改文本：[{'text': '失去的东西就要学着去接受，学着放下。'}] (添加自定义聊天模板)\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"from transformers import AutoTokenizer\n",
|
||||
"\n",
|
||||
"tokenizer = AutoTokenizer.from_pretrained(\"./model/toy_tokenizer\")\n",
|
||||
"msg = [{\"text\": \"失去的东西就要学着去接受，学着放下。\"}]\n",
|
||||
"new_msg = tokenizer.apply_chat_template(\n",
|
||||
" msg,\n",
|
||||
" tokenize=False\n",
|
||||
")\n",
|
||||
"print(f'原始文本：{msg}')\n",
|
||||
"print(f'修改文本：{msg} (添加自定义聊天模板)')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 11,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"分词器词表大小：259\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"print(f'分词器词表大小：{tokenizer.vocab_size}')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 12,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"查看分词结果：\n",
|
||||
"{'input_ids': [1, 164, 100, 112, 164, 239, 122, 166, 251, 229, 163, 119, 253, 167, 101, 126, 164, 111, 112, 167, 102, 226, 164, 258, 102, 166, 254, 225, 164, 239, 122, 165, 239, 101, 164, 240, 248, 174, 123, 237, 164, 258, 102, 166, 254, 225, 165, 245, 125, 163, 119, 236, 162, 225, 227, 2], 'token_type_ids': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]}\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"model_inputs = tokenizer(new_msg)\n",
|
||||
"print(f'查看分词结果：\\n{model_inputs}')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 13,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"对分词结果进行解码：<s>失去的东西就要学着去接受，学着放下。</s> (保留特殊字符)\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"response = tokenizer.decode(model_inputs['input_ids'], skip_special_tokens=False)\n",
|
||||
"print(f'对分词结果进行解码：{response} (保留特殊字符)' )"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 14,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"对分词结果进行解码：失去的东西就要学着去接受，学着放下。 (移除特殊字符)\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"response = tokenizer.decode(model_inputs['input_ids'], skip_special_tokens=True)\n",
|
||||
"print(f'对分词结果进行解码：{response} (移除特殊字符)' )"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 参考资料\n",
|
||||
"\n",
|
||||
"- [Hugging Face NLP Course](https://huggingface.co/learn/nlp-course/zh-CN/chapter2/4)"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3 (ipykernel)",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.10.14"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 4
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
|
|
@ -0,0 +1,562 @@
|
|||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "44db347d-fef7-4e56-aeab-ce55bdab8eab",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# 2-Dataset\n",
|
||||
"\n",
|
||||
"到这里我们便完成了对于 MiniMind Tokenizer 和 Model 部分的全部了解，我们所熟悉的大语言模型正是由这个组件构成的，接下来，我们需要对大模型训练所使用的数据集结构有个基本的认识。\n",
|
||||
"\n",
|
||||
"想要训练一个能够正常对话，并且符合人类对话偏好的大模型一般需要经过以下几个训练阶段：\n",
|
||||
"\n",
|
||||
"- 预训练（Pre-training）\n",
|
||||
"- 有监督微调（Supervised Fine-tuning，SFT）\n",
|
||||
"- 人类反馈强化学习（Reinforcement Learning from Human Feedback，RLHF）\n",
|
||||
"\n",
|
||||
"在不同训练阶段使用的数据集有所不同，下面会从 MiniMind 代码出发进行介绍和解读。"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"id": "e41156fd-ca0e-4083-a7a4-a5f98019ab88",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import json\n",
|
||||
"import random\n",
|
||||
"import re\n",
|
||||
"\n",
|
||||
"import pandas as pd\n",
|
||||
"import numpy as np\n",
|
||||
"from torch.utils.data import Dataset, DataLoader\n",
|
||||
"import torch\n",
|
||||
"from sklearn.model_selection import train_test_split\n",
|
||||
"import os\n",
|
||||
"import ast\n",
|
||||
"from transformers import AutoTokenizer\n",
|
||||
"\n",
|
||||
"os.environ[\"TOKENIZERS_PARALLELISM\"] = \"false\""
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 2,
|
||||
"id": "27d18ad2-3f71-435c-af03-ad9d347aa048",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"6400\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"# 从 ../model 目录加载分词器\n",
|
||||
"\n",
|
||||
"tokenizer = AutoTokenizer.from_pretrained('../model/minimind_tokenizer')\n",
|
||||
"print(tokenizer.vocab_size)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "ad08fe92-1d04-4b9c-b689-6ad9d763a7c2",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 预训练数据集\n",
|
||||
"\n",
|
||||
"预训练是模型在大规模语料上进行无监督学习的训练阶段，在该阶段，模型主要学习下一词预测的能力，简单的来说就是学会说话，而不是胡言乱语。因此，该阶段训练的模型不会具有问答能力，而是根据用户输入进行简单的词语接龙。\n",
|
||||
"\n",
|
||||
"我们可以看一看预训练的数据集格式：\n",
|
||||
"\n",
|
||||
"```\n",
|
||||
"{\"text\": \"如何才能摆脱拖延症？ 治愈拖延症并不容易，但以下建议可能有所帮助...\"}\n",
|
||||
"```\n",
|
||||
"\n",
|
||||
"为了降低该 demo 的运行门槛，在 `./demo` 文件夹下提供了包含两条训练数据的 `pretrain_data.jsonl` 文件作为熟悉训练流程的数据集 demo。"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 3,
|
||||
"id": "cdcac4aa-0ced-4a74-bba0-bc248923c7bc",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Row 1: {'text': 'LLM首先要学习的并非直接与人交流，而是让网络参数中充满知识的墨水，“墨水” 理论上喝的越饱越好，产生大量的对世界的知识积累。 预训练就是让Model先埋头苦学大量基本的知识，例如从Wiki百科、新闻、书籍整理大规模的高质量训练数据。 这个过程是“无监督”的，即人类不需要在过程中做任何“有监督”的校正，而是由模型自己从大量文本中总结规律学习知识点。 模型此阶段目的只有一个：学会词语接龙。例如我们输入“秦始皇”四个字，它可以接龙“是中国的第一位皇帝”。'}\n",
|
||||
"\n",
|
||||
"Row 2: {'text': '经过预训练，LLM此时已经掌握了大量知识，然而此时它只会无脑地词语接龙，还不会与人聊天。 SFT阶段就需要把半成品LLM施加一个自定义的聊天模板进行微调。 例如模型遇到这样的模板【问题->回答，问题->回答】后不再无脑接龙，而是意识到这是一段完整的对话结束。 称这个过程为指令微调，就如同让已经学富五车的「牛顿」先生适应21世纪智能手机的聊天习惯，学习屏幕左侧是对方消息，右侧是本人消息这个规律。 在训练时，MiniMind的指令和回答长度被截断在512，是为了节省显存空间。就像我们学习时，会先从短的文章开始，当学会写作200字作文后，800字文章也可以手到擒来。 在需要长度拓展时，只需要准备少量的2k/4k/8k长度对话数据进行进一步微调即可（此时最好配合RoPE-NTK的基准差值）。'}\n",
|
||||
"\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"# 我们可以查看一下 demo 中提供的数据\n",
|
||||
"path_pretrain = './toydata/pretrain_data.jsonl'\n",
|
||||
"\n",
|
||||
"with open(path_pretrain, 'r', encoding='utf-8') as f:\n",
|
||||
" for line_num, line in enumerate(f, 1):\n",
|
||||
" data = json.loads(line.strip())\n",
|
||||
" print(f'Row {line_num}: {data}\\n')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "0c0707e3-3ff9-424b-a5be-dab9db55aa44",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"我们知道，构建一个深度学习数据集需要继承 `torch.utils.data.dataset`，并构建 DataLoader 数据迭代器进行迭代访问。下面，我们来看看 MiniMind 是如何抽象一个预训练数据集的。"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"id": "fb2458e3-3661-40de-96be-aa036291a8fb",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"class PretrainDataset(Dataset):\n",
|
||||
" def __init__(self, data_path, tokenizer, max_length=512):\n",
|
||||
" super().__init__()\n",
|
||||
" self.tokenizer = tokenizer\n",
|
||||
" self.max_length = max_length\n",
|
||||
" self.samples = self.load_data(data_path)\n",
|
||||
"\n",
|
||||
" def load_data(self, path):\n",
|
||||
" \"\"\"按行读取 jsonl 文件，并存储在列表中\"\"\"\n",
|
||||
" samples = []\n",
|
||||
" with open(path, 'r', encoding='utf-8') as f:\n",
|
||||
" for line_num, line in enumerate(f, 1):\n",
|
||||
" data = json.loads(line.strip())\n",
|
||||
" samples.append(data)\n",
|
||||
" return samples\n",
|
||||
"\n",
|
||||
" def __len__(self):\n",
|
||||
" return len(self.samples)\n",
|
||||
"\n",
|
||||
" def __getitem__(self, index):\n",
|
||||
" sample = self.samples[index]\n",
|
||||
"\n",
|
||||
" text = f\"{self.tokenizer.bos_token}{str(sample['text'])}{self.tokenizer.eos_token}\"\n",
|
||||
" # print(text) # uncomment to see formating prompt\n",
|
||||
" encoding = self.tokenizer(\n",
|
||||
" text,\n",
|
||||
" max_length=self.max_length,\n",
|
||||
" padding='max_length',\n",
|
||||
" truncation=True,\n",
|
||||
" return_tensors='pt'\n",
|
||||
" )\n",
|
||||
" # print(encoding) # uncomment to see encoding result\n",
|
||||
" input_ids = encoding.input_ids.squeeze()\n",
|
||||
" loss_mask = (input_ids != self.tokenizer.pad_token_id) # mask to ignore loss on pad token\n",
|
||||
"\n",
|
||||
" X = torch.tensor(input_ids[:-1], dtype=torch.long) # <eos> token not included\n",
|
||||
" Y = torch.tensor(input_ids[1:], dtype=torch.long) # <bos> token not included\n",
|
||||
" loss_mask = torch.tensor(loss_mask[1:], dtype=torch.long) # align to tensor X\n",
|
||||
" return X, Y, loss_mask"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 5,
|
||||
"id": "677a1bbb-0df7-45e5-a3ed-d4431cddf43e",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"预训练数据集长度2\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"pretrain_dataset = PretrainDataset(path_pretrain, tokenizer)\n",
|
||||
"print(f'预训练数据集长度{len(pretrain_dataset)}')\n",
|
||||
"# x, y, lm = pretrain_dataset[0]\n",
|
||||
"# print(x.shape, y.shape, lm.shape)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "ebb160cb-0e96-45a7-a1c0-be145895e70f",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 有监督微调数据集\n",
|
||||
"\n",
|
||||
"有监督微调（Supervised Fine Tuning，SFT）对预训练后得到的基座 LLM 施加一个自定义聊天模板进行微调，由于在这一阶段，模型训练的目标是根据用户指令生成响应（构建问答体系），故又称为指令微调。\n",
|
||||
"\n",
|
||||
"我们可以看一看有监督微调的数据集格式：\n",
|
||||
"\n",
|
||||
"```\n",
|
||||
"{\n",
|
||||
" \"conversations\": [\n",
|
||||
" {\"role\": \"user\", \"content\": \"你好\"},\n",
|
||||
" {\"role\": \"assistant\", \"content\": \"你好！\"},\n",
|
||||
" {\"role\": \"user\", \"content\": \"再见\"},\n",
|
||||
" {\"role\": \"assistant\", \"content\": \"再见！\"}\n",
|
||||
" ]\n",
|
||||
"}\n",
|
||||
"```\n",
|
||||
"\n",
|
||||
"为了降低该 demo 的运行门槛，在 `./demo` 文件夹下提供了包含两条 conversation 问答数据的 `sft_data.jsonl` 文件作为熟悉训练流程的数据集 demo。"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 6,
|
||||
"id": "7e025e25-c937-4468-ae66-b32447440658",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Row 1: {'conversations': [{'role': 'user', 'content': '你好吗？'}, {'role': 'assistant', 'content': '我很好，谢谢！你呢？'}, {'role': 'user', 'content': '我也很好，谢谢！'}, {'role': 'assistant', 'content': '太好了！祝你今天愉快！'}]}\n",
|
||||
"\n",
|
||||
"Row 2: {'conversations': [{'role': 'user', 'content': '你喜欢什么运动？'}, {'role': 'assistant', 'content': '我喜欢跑步和游泳。你呢？'}, {'role': 'user', 'content': '我喜欢打篮球！'}, {'role': 'assistant', 'content': '篮球很棒！是一个很好的团队运动。'}]}\n",
|
||||
"\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"# 我们可以查看一下 demo 中提供的数据\n",
|
||||
"path_sft = './toydata/sft_data.jsonl'\n",
|
||||
"\n",
|
||||
"with open(path_sft, 'r', encoding='utf-8') as f:\n",
|
||||
" for line_num, line in enumerate(f, 1):\n",
|
||||
" data = json.loads(line.strip())\n",
|
||||
" print(f'Row {line_num}: {data}\\n')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "5f988233-530f-4ddf-8d64-9532442a2535",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"接下来，我们尝试构造一个数据集对象，实现对 sft 格式数据的读取与处理。"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 7,
|
||||
"id": "8b25f0c6-b5b5-47b1-974f-15a2f0d2760e",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"class SFTDataset(Dataset):\n",
|
||||
" def __init__(self, jsonl_path, tokenizer, max_length=512):\n",
|
||||
" super().__init__()\n",
|
||||
" self.tokenizer = tokenizer\n",
|
||||
" self.max_length = max_length\n",
|
||||
" self.samples = self.load_data(jsonl_path)\n",
|
||||
" self.bos_id = tokenizer('<s>assistant\\n', add_special_tokens=False).input_ids # set bos token\n",
|
||||
" self.eos_id = tokenizer('</s>\\n', add_special_tokens=False).input_ids # set eos token\n",
|
||||
"\n",
|
||||
" def load_data(self, path):\n",
|
||||
" samples = []\n",
|
||||
" with open(path, 'r', encoding='utf-8') as f:\n",
|
||||
" for line_num, line in enumerate(f, 1):\n",
|
||||
" data = json.loads(line.strip())\n",
|
||||
" samples.append(data)\n",
|
||||
" return samples\n",
|
||||
" \n",
|
||||
" def __len__(self):\n",
|
||||
" return len(self.samples)\n",
|
||||
"\n",
|
||||
" def _create_chat_prompt(self, conversations):\n",
|
||||
" \"\"\"构建符合 ChatML 格式的对话\"\"\"\n",
|
||||
" messages = []\n",
|
||||
" for i, turn in enumerate(conversations): # for each speaker in one conversation\n",
|
||||
" role = 'user' if i % 2 == 0 else 'assistant'\n",
|
||||
" messages.append({\"role\": role, \"content\": turn['content']})\n",
|
||||
" return self.tokenizer.apply_chat_template(\n",
|
||||
" messages,\n",
|
||||
" tokenize=False,\n",
|
||||
" add_generation_prompt=False\n",
|
||||
" )\n",
|
||||
"\n",
|
||||
" def _generate_loss_mask(self, input_ids):\n",
|
||||
" loss_mask = [0] * len(input_ids)\n",
|
||||
" i = 0\n",
|
||||
" while i < len(input_ids):\n",
|
||||
" if input_ids[i:i + len(self.bos_id)] == self.bos_id: # check if reach bos token\n",
|
||||
" ########### find content start & end point ###########\n",
|
||||
" start = i + len(self.bos_id) # set start point at the end of bos token\n",
|
||||
" end = start\n",
|
||||
" while end < len(input_ids):\n",
|
||||
" if input_ids[end:end + len(self.eos_id)] == self.eos_id: # check if reach eos token\n",
|
||||
" break\n",
|
||||
" end += 1\n",
|
||||
" ######################################################\n",
|
||||
" for j in range(start + 1, min(end + len(self.eos_id) + 1, self.max_length)): # ignore tokens that reach max input length\n",
|
||||
" loss_mask[j] = 1\n",
|
||||
" i = end + len(self.eos_id) if end < len(input_ids) else len(input_ids) # update i to exit current conversation turn\n",
|
||||
" else:\n",
|
||||
" i += 1\n",
|
||||
" return loss_mask\n",
|
||||
" \n",
|
||||
" def __getitem__(self, index):\n",
|
||||
" sample = self.samples[index]\n",
|
||||
" prompt = self._create_chat_prompt(sample['conversations'])\n",
|
||||
" # print(prompt) # uncomment to see formating prompt\n",
|
||||
" input_ids = self.tokenizer(prompt).input_ids[:self.max_length] # encode input\n",
|
||||
" input_ids += [self.tokenizer.pad_token_id] * (self.max_length - len(input_ids)) # encode padding\n",
|
||||
" # print(input_ids) # uncomment to see encoded prompt\n",
|
||||
" \n",
|
||||
" loss_mask = self._generate_loss_mask(input_ids)\n",
|
||||
"\n",
|
||||
" X = torch.tensor(input_ids[:-1], dtype=torch.long)\n",
|
||||
" Y = torch.tensor(input_ids[1:], dtype=torch.long)\n",
|
||||
" loss_mask = torch.tensor(loss_mask[1:], dtype=torch.long) # align with pred pos\n",
|
||||
"\n",
|
||||
" return X, Y, loss_mask"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 8,
|
||||
"id": "67a62dbf-6a50-4faf-8ad1-5228eb8aa7c9",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"2\n",
|
||||
"样本 shape = torch.Size([511]), 标签 shape = torch.Size([511]), loss_mask shape torch.Size([511])\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"sft_dataset = SFTDataset(path_sft, tokenizer)\n",
|
||||
"print(len(sft_dataset))\n",
|
||||
"x, y, lm = sft_dataset[0]\n",
|
||||
"print(f'样本 shape = {x.shape}, 标签 shape = {y.shape}, loss_mask shape {lm.shape}')\n",
|
||||
"# print(lm) # 打印 loss mask，你会发现在序列中有两处以 1 填充的序列，这是因为我们在一个 conversation 中开展了两轮对话，其中只有 assistant 回复计算损失"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "a7094ff6-4657-4483-8c34-785ffe9adbd4",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 人类反馈强化学习数据集\n",
|
||||
"\n",
|
||||
"在 MiniMind 项目中，采用直接偏好优化（Direct Parameter Optimization，DPO）训练大模型对齐人类偏好。在这一训练阶段，模型将会根据提供的问答正反例进行偏好优化，从而降低让人类不满意的答案出现的几率。\n",
|
||||
"\n",
|
||||
"与PPO(Proximal Policy Optimization)这种需要奖励模型、价值模型的RL算法不同； DPO通过推导PPO奖励模型的显式解，把在线奖励模型换成离线数据，Ref模型输出可以提前保存。 DPO性能几乎不变，只用跑 actor_model 和 ref_model 两个模型，大大节省显存开销和增加训练稳定性。\n",
|
||||
"\n",
|
||||
"我们可以看一看有监督微调的数据集格式：\n",
|
||||
"\n",
|
||||
"```\n",
|
||||
"{\n",
|
||||
" \"chosen\": [\n",
|
||||
" {\"content\": \"Query\", \"role\": \"user\"}, \n",
|
||||
" {\"content\": \"good answer\", \"role\": \"assistant\"}\n",
|
||||
" ], \n",
|
||||
" \"rejected\": [\n",
|
||||
" {\"content\": \"Query\", \"role\": \"user\"}, \n",
|
||||
" {\"content\": \"bad answer\", \"role\": \"assistant\"}\n",
|
||||
" ]\n",
|
||||
"}\n",
|
||||
"```\n",
|
||||
"\n",
|
||||
"为了降低该 demo 的运行门槛，在 ./demo 文件夹下提供了包含两条 conversation 问答数据的 sft_data.jsonl 文件作为熟悉训练流程的数据集 demo。"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 9,
|
||||
"id": "81682b5f-0d3e-4fa2-aec4-0be2b19d5c05",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Row 1: {'conversations': [{'role': 'user', 'content': '你好吗？'}, {'role': 'assistant', 'content': '我很好，谢谢！你呢？'}, {'role': 'user', 'content': '我也很好，谢谢！'}, {'role': 'assistant', 'content': '太好了！祝你今天愉快！'}]}\n",
|
||||
"\n",
|
||||
"Row 2: {'conversations': [{'role': 'user', 'content': '你喜欢什么运动？'}, {'role': 'assistant', 'content': '我喜欢跑步和游泳。你呢？'}, {'role': 'user', 'content': '我喜欢打篮球！'}, {'role': 'assistant', 'content': '篮球很棒！是一个很好的团队运动。'}]}\n",
|
||||
"\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"# 我们可以查看一下 demo 中提供的数据\n",
|
||||
"path_dpo = './toydata/dpo_data.jsonl'\n",
|
||||
"\n",
|
||||
"with open(path_sft, 'r', encoding='utf-8') as f:\n",
|
||||
" for line_num, line in enumerate(f, 1):\n",
|
||||
" data = json.loads(line.strip())\n",
|
||||
" print(f'Row {line_num}: {data}\\n')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "ebb54557-54f4-4004-8a7b-443b8671002b",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"接下来，我们尝试构造 json 对象，实现对 dpo 格式数据的读取和处理"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 10,
|
||||
"id": "efa98777-5d3a-4b3f-a9fb-e0f05e4af81d",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"class DPODataset(Dataset):\n",
|
||||
" def __init__(self, file_path, tokenizer, max_length=512):\n",
|
||||
" super().__init__()\n",
|
||||
" self.tokenizer = tokenizer\n",
|
||||
" self.max_length = max_length\n",
|
||||
" self.padding = tokenizer.pad_token_id if tokenizer.pad_token_id is not None else 0\n",
|
||||
" self.bos_id = tokenizer('<s>assistant\\n', add_special_tokens=False).input_ids # content prefix\n",
|
||||
" self.eos_id = tokenizer('</s>\\n', add_special_tokens=False).input_ids # content suffix\n",
|
||||
" with open(file_path, 'r', encoding='utf-8') as f: # load data\n",
|
||||
" self.data = []\n",
|
||||
" for line in f:\n",
|
||||
" line = line.strip()\n",
|
||||
" obj = json.loads(line)\n",
|
||||
" self.data.append(obj)\n",
|
||||
"\n",
|
||||
" def __len__(self):\n",
|
||||
" return len(self.data)\n",
|
||||
"\n",
|
||||
" def _generate_loss_mask(self, input_ids):\n",
|
||||
" \"\"\"此处的损失掩码生成函数与 SFT 阶段逻辑一致\"\"\"\n",
|
||||
" loss_mask = [0] * len(input_ids)\n",
|
||||
" i = 0\n",
|
||||
" while i < len(input_ids):\n",
|
||||
" if input_ids[i:i + len(self.bos_id)] == self.bos_id:\n",
|
||||
" start = i + len(self.bos_id)\n",
|
||||
" end = start\n",
|
||||
" while end < len(input_ids):\n",
|
||||
" if input_ids[end:end + len(self.eos_id)] == self.eos_id:\n",
|
||||
" break\n",
|
||||
" end += 1\n",
|
||||
" for j in range(start + 1, min(end + len(self.eos_id) + 1, self.max_length)):\n",
|
||||
" loss_mask[j] = 1\n",
|
||||
" i = end + len(self.eos_id) if end < len(input_ids) else len(input_ids)\n",
|
||||
" else:\n",
|
||||
" i += 1\n",
|
||||
" return loss_mask\n",
|
||||
" \n",
|
||||
" def __getitem__(self, index):\n",
|
||||
" item = self.data[index]\n",
|
||||
" chosen = item['chosen'] # 一个 list，里面包含若干 {role, content}\n",
|
||||
" rejected = item['rejected'] # 同上\n",
|
||||
" # format prompt\n",
|
||||
" chosen_prompt = self.tokenizer.apply_chat_template(\n",
|
||||
" chosen, tokenize=False, add_generation_prompt=False\n",
|
||||
" )\n",
|
||||
" print(chosen_prompt) # uncomment to see formating prompt\n",
|
||||
" rejected_prompt = self.tokenizer.apply_chat_template(\n",
|
||||
" rejected, tokenize = False, add_gerneration_prompt=False\n",
|
||||
" )\n",
|
||||
" print(rejected_prompt) # uncomment to see formating prompt\n",
|
||||
" # tokenize\n",
|
||||
" chosen_encoding = self.tokenizer(\n",
|
||||
" chosen_prompt, truncation=True, max_length=self.max_length, padding='max_length'\n",
|
||||
" )\n",
|
||||
" rejected_encoding = self.tokenizer(\n",
|
||||
" rejected_prompt, truncation=True, max_length=self.max_length, padding='max_length'\n",
|
||||
" )\n",
|
||||
" # generate loss mask\n",
|
||||
" chosen_input_ids = chosen_encoding['input_ids']\n",
|
||||
" chosen_loss_mask = self._generate_loss_mask(chosen_input_ids)\n",
|
||||
" rejected_input_ids = rejected_encoding['input_ids']\n",
|
||||
" rejected_loss_mask = self._generate_loss_mask(rejected_input_ids)\n",
|
||||
" # same as sft / pretrain\n",
|
||||
" x_chosen = torch.tensor(chosen_input_ids[:-1], dtype=torch.long)\n",
|
||||
" y_chosen = torch.tensor(chosen_input_ids[1:], dtype=torch.long)\n",
|
||||
" mask_chosen = torch.tensor(chosen_loss_mask[1:], dtype=torch.long)\n",
|
||||
" x_rejected = torch.tensor(rejected_input_ids[:-1], dtype=torch.long)\n",
|
||||
" y_rejected = torch.tensor(rejected_input_ids[1:], dtype=torch.long)\n",
|
||||
" mask_rejected = torch.tensor(rejected_loss_mask[1:], dtype=torch.long)\n",
|
||||
"\n",
|
||||
" return {\n",
|
||||
" 'x_chosen': x_chosen,\n",
|
||||
" 'y_chosen': y_chosen,\n",
|
||||
" 'mask_chosen': mask_chosen,\n",
|
||||
" 'x_rejected': x_rejected,\n",
|
||||
" 'y_rejected': y_rejected,\n",
|
||||
" 'mask_rejected': mask_rejected\n",
|
||||
" }"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 11,
|
||||
"id": "5ba205c4-0dd2-48fd-8aa2-26a42afc90d8",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"DPO 数据集长度：2\n",
|
||||
"<s>system\n",
|
||||
"你是 MiniMind，是一个有用的人工智能助手。</s>\n",
|
||||
"<s>user\n",
|
||||
"你好吗？</s>\n",
|
||||
"<s>assistant\n",
|
||||
"我很好，谢谢！你呢？</s>\n",
|
||||
"<s>user\n",
|
||||
"今天过得怎么样？</s>\n",
|
||||
"<s>assistant\n",
|
||||
"挺好的，去跑步了，心情不错。</s>\n",
|
||||
"\n",
|
||||
"<s>system\n",
|
||||
"你是 MiniMind，是一个有用的人工智能助手。</s>\n",
|
||||
"<s>user\n",
|
||||
"你好吗？</s>\n",
|
||||
"<s>assistant\n",
|
||||
"不好，我很累。</s>\n",
|
||||
"<s>user\n",
|
||||
"你喜欢什么运动？</s>\n",
|
||||
"<s>assistant\n",
|
||||
"我不喜欢运动，没兴趣。</s>\n",
|
||||
"\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"dpo_dataset = DPODataset(path_dpo, tokenizer)\n",
|
||||
"print(f'DPO 数据集长度：{len(dpo_dataset)}')\n",
|
||||
"res = dpo_dataset[0]\n",
|
||||
"# 如有需要，请自主查看 res 中的元素"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3 (ipykernel)",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.10.14"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 5
|
||||
}
|
||||
|
|
@ -0,0 +1,376 @@
|
|||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "aba635c1-cdcd-4d00-a123-00045edbd10f",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# 3-Pretrain\n",
|
||||
"\n",
|
||||
"预训练是模型经历的第一个阶段，在该阶段，模型将会吸收知识，学习尽可能正确的下一词语预测范式。在这个笔记本中，我们仅对预训练的训练流程进行展示和学习，因此只给出必要的代码片段，如 wandb 和 ddp 不会在此笔记本中涉及。\n",
|
||||
"\n",
|
||||
"此笔记本的完整实现见主仓库 `/minimind/train_pretrain.py`"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"id": "98cb706f-2e03-4d8e-8e45-3663c9ab3b64",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# 导入依赖\n",
|
||||
"import os\n",
|
||||
"import platform\n",
|
||||
"import argparse\n",
|
||||
"import time\n",
|
||||
"import math\n",
|
||||
"import warnings\n",
|
||||
"import pandas as pd\n",
|
||||
"import torch\n",
|
||||
"import torch.distributed as dist\n",
|
||||
"from torch import optim, nn\n",
|
||||
"from torch.nn.parallel import DistributedDataParallel\n",
|
||||
"from torch.optim.lr_scheduler import CosineAnnealingLR\n",
|
||||
"from torch.utils.data import DataLoader, DistributedSampler\n",
|
||||
"from contextlib import nullcontext\n",
|
||||
"\n",
|
||||
"from transformers import AutoTokenizer\n",
|
||||
"\n",
|
||||
"from model.model import MiniMindLM\n",
|
||||
"from model.LMConfig import LMConfig\n",
|
||||
"from model.dataset import PretrainDataset"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 2,
|
||||
"id": "c3b6f015-0e5c-4e88-bb0d-32092993933f",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"warnings.filterwarnings('ignore')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "a46c9b5b-12eb-4cad-a859-c5101cdcba82",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 可选参数设置\n",
|
||||
"\n",
|
||||
"首先，查看训练的可选参数，这些参数在实际使用时通过解析命令行进行导入，我们用 class 进行包装."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 3,
|
||||
"id": "3f07e381-961e-4a4f-9f37-ff2f0bc2918c",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"class args:\n",
|
||||
" # out_dir: str = \"out\" # pytorch 格式权重文件保存位置 我们只展示训练过程 所以不使用\n",
|
||||
" epochs: int = 1 # 训练轮数\n",
|
||||
" batch_size: int = 2 # pretrain 数据集仅两个样本，设置 batch 为 2\n",
|
||||
" learning_rate: float = 5e-4 # 学习率\n",
|
||||
" device: str = 'cuda' if torch.cuda.is_available() else 'cpu'\n",
|
||||
" dtype: str = 'bfloat16' # 16 bit 浮点数：8 bit 指数 + 7 bit 尾数\n",
|
||||
" # use_wandb: bool = False # 是否使用 wandb 我们不使用\n",
|
||||
" wandb_project: str = 'MiniMind-Notebook'\n",
|
||||
" num_workers: int = 1 # 工作进程数\n",
|
||||
" # ddp：bool = False # 单机多卡\n",
|
||||
" accumulation_steps: int = 1 # 梯度累积步数\n",
|
||||
" grad_clip: float = 1.0 # 梯度剪裁\n",
|
||||
" warmup_iters: int = 0 # 学习率热启动\n",
|
||||
" log_interval: int = 1 # 每一步打印日志 仅用于观察\n",
|
||||
" # save_interval: int = 100 # checkpoint 保存点 我们不使用\n",
|
||||
" local_rank: int = 1 # device 设备号\n",
|
||||
" dim: int = 512 # 词嵌入维度 模型超参数\n",
|
||||
" n_layers: int = 2 # MiniMind Block 数量 模型超参数\n",
|
||||
" max_seq_len: int = 512 # 序列长度阈值\n",
|
||||
" use_moe: bool = False # 是否启用混合专家\n",
|
||||
" data_path: str = './toydata/pretrain_data.jsonl' # 数据集路径"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"id": "8f73286e-48a0-4a5b-b7c4-8c5fefea5f14",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"查看工作设备 cuda\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"print(f'查看工作设备 {args.device}')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "7e4b0ea8-0f41-47fe-8125-93629d03d451",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 初始化训练\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"接下来，我们对一些重要模块进行初始化，我们已经了解过，分词器，模型和数据集是大模型的基本组件，我们对其进行初始化。"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 5,
|
||||
"id": "9818e376-7f1e-4347-b8c3-c8e30f7d587f",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def init_model(lm_config):\n",
|
||||
" tokenizer = AutoTokenizer.from_pretrained('../model/minimind_tokenizer')\n",
|
||||
" model = MiniMindLM(lm_config).to(args.device)\n",
|
||||
" print(f'LLM总参数量：{sum(p.numel() for p in model.parameters() if p.requires_grad) / 1e6:.3f} 百万')\n",
|
||||
" return model, tokenizer"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 6,
|
||||
"id": "24d15001-b670-4333-9fbe-b224ab1993c5",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"LLM总参数量：8.915 百万\n",
|
||||
"模型位于设备：cuda:0, 词表长度：6400, DataLoader：<torch.utils.data.dataloader.DataLoader object at 0x000001C6BF33F280>\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"lm_config = LMConfig(dim=args.dim, n_layers=args.n_layers, max_seq_len=args.max_seq_len, use_moe=args.use_moe)\n",
|
||||
"model, tokenizer = init_model(lm_config)\n",
|
||||
"\n",
|
||||
"train_ds = PretrainDataset(args.data_path, tokenizer, max_length=lm_config.max_seq_len)\n",
|
||||
"\n",
|
||||
"train_loader = DataLoader(\n",
|
||||
" train_ds,\n",
|
||||
" batch_size=args.batch_size,\n",
|
||||
" pin_memory=True,\n",
|
||||
" drop_last=False,\n",
|
||||
" shuffle=False,\n",
|
||||
" num_workers=args.num_workers,\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"print(f'模型位于设备：{model.device}, 词表长度：{tokenizer.vocab_size}, DataLoader：{train_loader}')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 7,
|
||||
"id": "60c8c970-f448-4ea0-b21b-8f929e03f6fa",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"打印一个 iter 的数据:\n",
|
||||
"[tensor([[ 1, 46, 46, ..., 0, 0, 0],\n",
|
||||
" [ 1, 5349, 1619, ..., 0, 0, 0]]), tensor([[ 46, 46, 47, ..., 0, 0, 0],\n",
|
||||
" [5349, 1619, 2875, ..., 0, 0, 0]]), tensor([[1, 1, 1, ..., 0, 0, 0],\n",
|
||||
" [1, 1, 1, ..., 0, 0, 0]])]\n",
|
||||
"\n",
|
||||
"数据集大小：2, DataLoader 大小：1\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"loader = iter(train_loader)\n",
|
||||
"print(f'打印一个 iter 的数据:\\n{next(loader)}\\n')\n",
|
||||
"print(f'数据集大小：{len(train_ds)}, DataLoader 大小：{len(loader)}')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "0d91904d-7ddf-4105-a5f5-65b84315a73b",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"我们发现，train loader 的每一个 iter 都包含一个长度为 3 的张量列表，这是因为 train_dataset 每一次取数据都会返回三个张量，分别为:\n",
|
||||
"\n",
|
||||
"- 样本 X: 包含 \\<bos> 在内的输入 content\n",
|
||||
"- 标签 Y: 包含 \\<eos> 在内的输出 content\n",
|
||||
"- 掩码 loss_mask: 指示需要计算损失的 token 位置\n",
|
||||
"\n",
|
||||
"由于我们的数据集只有两条数据，而 batch size 设置为 2，因此我们的 dataloader 只有一个 iter."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "40764309-9c98-45d5-ada5-e31c1aaa02c5",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 启动训练\n",
|
||||
"\n",
|
||||
"训练一个深度学习模型，还涉及到了优化器，损失函数和学习率调度。接下来，我们查看 MiniMind 训练部分的代码，并进行一轮简单的训练。"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 8,
|
||||
"id": "4f70b9cf-fcff-4242-bd96-ce34a5da55e4",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# 学习率调度方面 采用余弦退火学习率\n",
|
||||
"def get_lr(current_step, total_steps, lr):\n",
|
||||
" return lr / 10 + 0.5 * lr * (1 + math.cos(math.pi * current_step / total_steps))\n",
|
||||
"\n",
|
||||
"# 优化器方面 选择 AdamW 优化器 并在混精度场景下创建 scaler 进行梯度缩放避免数值下溢\n",
|
||||
"scaler = torch.cuda.amp.GradScaler(enabled=(args.dtype in ['float16', 'bfloat16']))\n",
|
||||
"optimizer = optim.AdamW(model.parameters(), lr=args.learning_rate)\n",
|
||||
"\n",
|
||||
"device_type = \"cuda\" if \"cuda\" in args.device else \"cpu\"\n",
|
||||
"ctx = nullcontext() if device_type == \"cpu\" else torch.cuda.amp.autocast() # 在 cuda 上启动混精度训练，否则空白上下文"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "c101ffb4-1a97-4e29-8612-caf40a8aa05d",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"接下来，我们来看看 MiniMind 的训练函数"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 9,
|
||||
"id": "03043d8e-e90b-42a8-ba4d-de6187f6eb81",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def train_epoch(epoch):\n",
|
||||
" loss_fct = nn.CrossEntropyLoss(reduction='none') # 损失函数 采用交叉熵损失\n",
|
||||
" start_time = time.time()\n",
|
||||
" for step, (X, Y, loss_mask) in enumerate(train_loader):\n",
|
||||
" X = X.to(args.device)\n",
|
||||
" Y = Y.to(args.device)\n",
|
||||
" loss_mask = loss_mask.to(args.device)\n",
|
||||
"\n",
|
||||
" lr = get_lr(epoch * iter_per_epoch + step, args.epochs * iter_per_epoch, args.learning_rate)\n",
|
||||
" for param_group in optimizer.param_groups:\n",
|
||||
" param_group['lr'] = lr\n",
|
||||
"\n",
|
||||
" with ctx:\n",
|
||||
" res = model(X) # 前向推理\n",
|
||||
" loss = loss_fct(\n",
|
||||
" res.logits.view(-1, res.logits.size(-1)),\n",
|
||||
" Y.view(-1)\n",
|
||||
" ).view(Y.size()) # 取生成的最后一个 token 的 logits 计算损失\n",
|
||||
" loss = (loss * loss_mask).sum() / loss_mask.sum()\n",
|
||||
" loss += res.aux_loss # 若为混合专家 则将 MOE 辅助损失纳入考虑\n",
|
||||
" loss = loss / args.accumulation_steps # 梯度累积\n",
|
||||
"\n",
|
||||
" scaler.scale(loss).backward() # 梯度缩放\n",
|
||||
"\n",
|
||||
" if (step + 1) % args.accumulation_steps == 0:\n",
|
||||
" scaler.unscale_(optimizer)\n",
|
||||
" torch.nn.utils.clip_grad_norm_(model.parameters(), args.grad_clip) # 梯度剪裁\n",
|
||||
"\n",
|
||||
" scaler.step(optimizer)\n",
|
||||
" scaler.update()\n",
|
||||
"\n",
|
||||
" optimizer.zero_grad(set_to_none=True) # 单步更新\n",
|
||||
"\n",
|
||||
" if step % args.log_interval == 0:\n",
|
||||
" spend_time = time.time() - start_time\n",
|
||||
" print(\n",
|
||||
" 'Epoch:[{}/{}]({}/{}) loss:{:.3f} lr:{:.12f} epoch_Time:{}min:'.format(\n",
|
||||
" epoch + 1,\n",
|
||||
" args.epochs,\n",
|
||||
" step,\n",
|
||||
" iter_per_epoch,\n",
|
||||
" loss.item() * args.accumulation_steps,\n",
|
||||
" optimizer.param_groups[-1]['lr'],\n",
|
||||
" spend_time / (step + 1) * iter_per_epoch // 60 - spend_time // 60))\n",
|
||||
"\n",
|
||||
" # 到达指定保存步数时，save as PyTorch\n",
|
||||
" # if (step + 1) % args.save_interval == 0 and (not ddp or dist.get_rank() == 0):\n",
|
||||
" # model.eval()\n",
|
||||
" # moe_path = '_moe' if lm_config.use_moe else ''\n",
|
||||
" # ckp = f'{args.save_dir}/pretrain_{lm_config.dim}{moe_path}.pth'\n",
|
||||
"\n",
|
||||
" # if isinstance(model, torch.nn.parallel.DistributedDataParallel):\n",
|
||||
" # state_dict = model.module.state_dict()\n",
|
||||
" # else:\n",
|
||||
" # state_dict = model.state_dict()\n",
|
||||
"\n",
|
||||
" # torch.save(state_dict, ckp)\n",
|
||||
" # model.train()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "7792bf9c-95a1-4c0d-bb53-fac2a1132ee0",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"准备完毕，我们尝试一轮长度 1 个 iter 的训练."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 10,
|
||||
"id": "17f51549-2009-4322-86c8-7aa783169e0b",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Epoch:[1/1](0/1) loss:8.959 lr:0.000550000000 epoch_Time:0.0min:\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"iter_per_epoch = len(train_loader)\n",
|
||||
"for epoch in range(args.epochs):\n",
|
||||
" train_epoch(epoch)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 11,
|
||||
"id": "6f257b9e-3a1e-4686-9516-3f9eeb7d0654",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"del model"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3 (ipykernel)",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.10.14"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 5
|
||||
}
|
||||
|
|
@ -0,0 +1,386 @@
|
|||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "70560a4d-ff57-4699-afb2-52f6a145b97e",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# 4-SFT\n",
|
||||
"\n",
|
||||
"在预训练阶段后，我们应该能够获得一个下一词预测模型，此时的模型已经掌握了大量的知识。不过，仅仅具备下一词预测能力是不够的，我们希望大模型能够获得问答能力，这一能力便是在有监督微调（Supervised Fine Tuning，SFT）阶段获得的.\n",
|
||||
"\n",
|
||||
"在这个笔记本中，我们仅对 SFT 的训练流程进行展示和学习，因此只给出必要的代码片段，如 wandb 和 ddp 不会在此笔记本中涉及.\n",
|
||||
"\n",
|
||||
"此笔记本的完整实现见主仓库 `/minimind/train_full_sft.py`"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"id": "2508c8c7-eb44-4ac0-918c-41479f41d1ac",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# 导入依赖\n",
|
||||
"import os\n",
|
||||
"import platform\n",
|
||||
"import argparse\n",
|
||||
"import time\n",
|
||||
"import math\n",
|
||||
"import warnings\n",
|
||||
"\n",
|
||||
"import pandas as pd\n",
|
||||
"import torch\n",
|
||||
"import torch.nn.functional as F\n",
|
||||
"import torch.distributed as dist\n",
|
||||
"from contextlib import nullcontext\n",
|
||||
"\n",
|
||||
"from torch import optim, nn\n",
|
||||
"from torch.nn.parallel import DistributedDataParallel\n",
|
||||
"from torch.utils.data import DataLoader, DistributedSampler\n",
|
||||
"from transformers import AutoTokenizer, AutoModelForCausalLM\n",
|
||||
"from model.model import MiniMindLM\n",
|
||||
"from model.LMConfig import LMConfig\n",
|
||||
"from model.dataset import SFTDataset"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 2,
|
||||
"id": "4a79e140-23c7-4498-9a4d-ef7459692996",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"warnings.filterwarnings('ignore')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "dbe1d40e-e074-4856-8c67-da8019f250ab",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 可选参数设置\n",
|
||||
"\n",
|
||||
"首先，查看训练的可选参数，这些参数在实际使用时通过命令行导入，为了保持笔记本的易用性，选择用 class 进行包装."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 3,
|
||||
"id": "98cbd6cf-081f-4c31-8dce-52566004801a",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"class args:\n",
|
||||
" # out_dir: str = \"out\" # pytorch 格式权重文件保存位置 我们只展示训练过程 所以不使用\n",
|
||||
" epochs: int = 1 # 训练轮数\n",
|
||||
" batch_size: int = 2 # pretrain 数据集仅两个样本，设置 batch 为 2\n",
|
||||
" learning_rate: float = 5e-4 # 学习率\n",
|
||||
" device: str = 'cuda' if torch.cuda.is_available() else 'cpu'\n",
|
||||
" dtype: str = 'bfloat16' # 16 bit 浮点数：8 bit 指数 + 7 bit 尾数\n",
|
||||
" # use_wandb: bool = False # 是否使用 wandb 我们不使用\n",
|
||||
" wandb_project: str = 'MiniMind-Notebook'\n",
|
||||
" num_workers: int = 1 # 工作进程数\n",
|
||||
" # ddp：bool = False # 单机多卡\n",
|
||||
" accumulation_steps: int = 1 # 梯度累积步数\n",
|
||||
" grad_clip: float = 1.0 # 梯度剪裁\n",
|
||||
" warmup_iters: int = 0 # 学习率热启动\n",
|
||||
" log_interval: int = 1 # 每一步打印日志 仅用于观察\n",
|
||||
" # save_interval: int = 100 # checkpoint 保存点 我们不使用\n",
|
||||
" local_rank: int = 1 # device 设备号\n",
|
||||
" dim: int = 512 # 词嵌入维度 模型超参数\n",
|
||||
" n_layers: int = 2 # MiniMind Block 数量 模型超参数\n",
|
||||
" max_seq_len: int = 512 # 序列长度阈值\n",
|
||||
" use_moe: bool = False # 是否启用混合专家\n",
|
||||
" data_path: str = './toydata/sft_data.jsonl' # 数据集路径"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"id": "642f5107-aa9b-49cc-9759-e8731326edc3",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"查看工作设备 cuda\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"print(f'查看工作设备 {args.device}')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "b70fadb6-f78f-45a0-ad3e-d41d7253c72b",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 初始化训练\n",
|
||||
"\n",
|
||||
"接下来，我们对一些重要模块进行初始化，我们已经了解过，分词器，模型和数据集是大模型的基本组件，我们对其进行初始化.\n",
|
||||
"\n",
|
||||
"> 注意 与预训练阶段不同的是 在 sft 阶段 我们实际上是在上一阶段训练获得的模型的基础上修改数据集进行接续训练 因此需要载入上一阶段的模型权重 出于展示的目的 载入权重的代码在此笔记本中只作展示 并不执行"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 5,
|
||||
"id": "c64a0e37-a9d0-4ce6-b5ee-7b2728de879b",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def init_model(lm_config):\n",
|
||||
" tokenizer = AutoTokenizer.from_pretrained('../model/minimind_tokenizer')\n",
|
||||
" model = MiniMindLM(lm_config).to(args.device)\n",
|
||||
" moe_path = '_moe' if lm_config.use_moe else ''\n",
|
||||
" # ckp = f'./out/pretrain_{lm_config.dim}{moe_path}.pth' # 指示上一阶段训练保存的模型文件位置\n",
|
||||
" # state_dict = torch.load(ckp, map_location=args.device) # 载入模型状态字典\n",
|
||||
" # model.laod_state_dict(state_dict, strict=False) # 装入模型\n",
|
||||
" print(f'LLM总参数量：{sum(p.numel() for p in model.parameters() if p.requires_grad) / 1e6:.3f} 百万')\n",
|
||||
" model = model.to(args.device)\n",
|
||||
" return model, tokenizer"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 6,
|
||||
"id": "2eec33f3-2ac2-4c75-b58e-844d89d8add9",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"LLM总参数量：8.915 百万\n",
|
||||
"模型位于设备：cuda:0, 词表长度：6400, DataLoader：<torch.utils.data.dataloader.DataLoader object at 0x0000024234BBE950>\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"lm_config = LMConfig(dim=args.dim, n_layers=args.n_layers, max_seq_len=args.max_seq_len, use_moe=args.use_moe)\n",
|
||||
"model, tokenizer = init_model(lm_config)\n",
|
||||
"\n",
|
||||
"train_ds = SFTDataset(args.data_path, tokenizer, max_length=lm_config.max_seq_len)\n",
|
||||
"\n",
|
||||
"train_loader = DataLoader(\n",
|
||||
" train_ds,\n",
|
||||
" batch_size=args.batch_size,\n",
|
||||
" pin_memory=True,\n",
|
||||
" drop_last=False,\n",
|
||||
" shuffle=False,\n",
|
||||
" num_workers=args.num_workers,\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"print(f'模型位于设备：{model.device}, 词表长度：{tokenizer.vocab_size}, DataLoader：{train_loader}')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 7,
|
||||
"id": "b02644e0-e61d-4df0-9203-fa841cd83635",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"打印一个 iter 的数据:\n",
|
||||
"[tensor([[ 1, 85, 736, ..., 0, 0, 0],\n",
|
||||
" [ 1, 85, 736, ..., 0, 0, 0]]), tensor([[ 85, 736, 201, ..., 0, 0, 0],\n",
|
||||
" [ 85, 736, 201, ..., 0, 0, 0]]), tensor([[0, 0, 0, ..., 0, 0, 0],\n",
|
||||
" [0, 0, 0, ..., 0, 0, 0]])]\n",
|
||||
"\n",
|
||||
"数据集大小：2, DataLoader 大小：1\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"loader = iter(train_loader)\n",
|
||||
"print(f'打印一个 iter 的数据:\\n{next(loader)}\\n')\n",
|
||||
"print(f'数据集大小：{len(train_ds)}, DataLoader 大小：{len(loader)}')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "13ac8a47-d26a-49da-82fb-468ef5ac91a4",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"我们发现，train loader 的每一个 iter 都包含一个长度为 3 的张量列表，这是因为 train_dataset 每一次取数据都会返回三个张量，分别为:\n",
|
||||
"\n",
|
||||
"- 样本 X: 包含 \\<bos> 在内的输入 conversation\n",
|
||||
"- 标签 Y: 包含 \\<eos> 在内的输出 conversation\n",
|
||||
"- 掩码 loss_mask: 指示需要计算损失的 token 位置\n",
|
||||
"\n",
|
||||
"由于我们的数据集只有两条数据，而 batch size 设置为 2，因此我们的 dataloader 只有一个 iter."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "ee23f937-f333-4c02-a3b4-a768818c0e2b",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 启动训练\n",
|
||||
"\n",
|
||||
"训练一个深度学习模型，还涉及到了优化器，损失函数和学习率调度. 接下来，我们查看 MiniMind 训练部分的代码，并进行一轮简单的训练.\n",
|
||||
"\n",
|
||||
"> 不难发现 pretrain 阶段和 sft 阶段的训练主体差不多 因为这两个阶段的差异体现在数据集格式 而数据集在经过 chat template 格式化后差异小了很多"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 8,
|
||||
"id": "5c32a59a-be0b-4426-b47e-c522322bef69",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# 学习率调度方面 采用余弦退火学习率\n",
|
||||
"def get_lr(current_step, total_steps, lr):\n",
|
||||
" return lr / 10 + 0.5 * lr * (1 + math.cos(math.pi * current_step / total_steps))\n",
|
||||
"\n",
|
||||
"# 优化器方面 选择 AdamW 优化器 并在混精度场景下创建 scaler 进行梯度缩放避免数值下溢\n",
|
||||
"scaler = torch.cuda.amp.GradScaler(enabled=(args.dtype in ['float16', 'bfloat16']))\n",
|
||||
"optimizer = optim.AdamW(model.parameters(), lr=args.learning_rate)\n",
|
||||
"\n",
|
||||
"device_type = \"cuda\" if \"cuda\" in args.device else \"cpu\"\n",
|
||||
"ctx = nullcontext() if device_type == \"cpu\" else torch.cuda.amp.autocast() # 在 cuda 上启动混精度训练，否则空白上下文"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "13963ec0-d22e-4120-b0f2-8f5e5abfcda1",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"接下来，我们来看看 MiniMind 的训练函数"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 9,
|
||||
"id": "74499592-2065-4fff-97d1-762cd08b77b6",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def train_epoch(epoch):\n",
|
||||
" loss_fct = nn.CrossEntropyLoss(reduction='none') # 损失函数 采用交叉熵损失\n",
|
||||
" start_time = time.time()\n",
|
||||
" for step, (X, Y, loss_mask) in enumerate(train_loader):\n",
|
||||
" X = X.to(args.device)\n",
|
||||
" Y = Y.to(args.device)\n",
|
||||
" loss_mask = loss_mask.to(args.device)\n",
|
||||
"\n",
|
||||
" lr = get_lr(epoch * iter_per_epoch + step, args.epochs * iter_per_epoch, args.learning_rate)\n",
|
||||
" for param_group in optimizer.param_groups:\n",
|
||||
" param_group['lr'] = lr\n",
|
||||
"\n",
|
||||
" with ctx:\n",
|
||||
" res = model(X) # 前向推理\n",
|
||||
" loss = loss_fct(\n",
|
||||
" res.logits.view(-1, res.logits.size(-1)),\n",
|
||||
" Y.view(-1)\n",
|
||||
" ).view(Y.size()) # 取生成的最后一个 token 的 logits 计算损失\n",
|
||||
" loss = (loss * loss_mask).sum() / loss_mask.sum()\n",
|
||||
" loss += res.aux_loss # 若为混合专家 则将 MOE 辅助损失纳入考虑\n",
|
||||
" loss = loss / args.accumulation_steps # 梯度累积\n",
|
||||
"\n",
|
||||
" scaler.scale(loss).backward() # 梯度缩放\n",
|
||||
"\n",
|
||||
" if (step + 1) % args.accumulation_steps == 0:\n",
|
||||
" scaler.unscale_(optimizer)\n",
|
||||
" torch.nn.utils.clip_grad_norm_(model.parameters(), args.grad_clip) # 梯度剪裁\n",
|
||||
"\n",
|
||||
" scaler.step(optimizer)\n",
|
||||
" scaler.update()\n",
|
||||
"\n",
|
||||
" optimizer.zero_grad(set_to_none=True) # 单步更新\n",
|
||||
"\n",
|
||||
" if step % args.log_interval == 0:\n",
|
||||
" spend_time = time.time() - start_time\n",
|
||||
" print(\n",
|
||||
" 'Epoch:[{}/{}]({}/{}) loss:{:.3f} lr:{:.12f} epoch_Time:{}min:'.format(\n",
|
||||
" epoch + 1,\n",
|
||||
" args.epochs,\n",
|
||||
" step,\n",
|
||||
" iter_per_epoch,\n",
|
||||
" loss.item() * args.accumulation_steps,\n",
|
||||
" optimizer.param_groups[-1]['lr'],\n",
|
||||
" spend_time / (step + 1) * iter_per_epoch // 60 - spend_time // 60))\n",
|
||||
"\n",
|
||||
" # 到达指定保存步数时，save as PyTorch\n",
|
||||
" # if (step + 1) % args.save_interval == 0 and (not ddp or dist.get_rank() == 0):\n",
|
||||
" # model.eval()\n",
|
||||
" # moe_path = '_moe' if lm_config.use_moe else ''\n",
|
||||
" # ckp = f'{args.save_dir}/pretrain_{lm_config.dim}{moe_path}.pth'\n",
|
||||
"\n",
|
||||
" # if isinstance(model, torch.nn.parallel.DistributedDataParallel):\n",
|
||||
" # state_dict = model.module.state_dict()\n",
|
||||
" # else:\n",
|
||||
" # state_dict = model.state_dict()\n",
|
||||
"\n",
|
||||
" # torch.save(state_dict, ckp)\n",
|
||||
" # model.train()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "546f28c2-418e-453c-88aa-695468f9216c",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"准备完毕，我们尝试一轮长度 1 个 iter 的训练."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 10,
|
||||
"id": "341200c9-0d3a-4ed1-9dd2-d2fc35d02564",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Epoch:[1/1](0/1) loss:8.902 lr:0.000550000000 epoch_Time:0.0min:\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"iter_per_epoch = len(train_loader)\n",
|
||||
"for epoch in range(args.epochs):\n",
|
||||
" train_epoch(epoch)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 11,
|
||||
"id": "b0480212-1500-49fb-b679-7bdf2c100c87",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"del model"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3 (ipykernel)",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.10.14"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 5
|
||||
}
|
||||
|
|
@ -0,0 +1,457 @@
|
|||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "f53f31fb-4795-49cf-9a68-a13f8f7e5cae",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# 5-DPO\n",
|
||||
"\n",
|
||||
"直接偏好优化（Direct Preference Optimization，DPO）是后训练阶段中，使用正反样例激励大模型产生符合人类偏好的回答的策略，为人类反馈强化学习（Reinforcement Learning from Human Feedback, RLHF）提供了一个高效简化的替代方案。通过这一阶段的训练，大模型将会学会依照人类的喜好生成回复.\n",
|
||||
"\n",
|
||||
"在这个笔记本中，我们仅对 DPO 的训练流程进行展示和学习，因此只给出必要的代码片段，如 wandb 和 ddp 不会在此笔记本中涉及.\n",
|
||||
"\n",
|
||||
"此笔记本的完整实现见主仓库 `/minimind/train_dpo.py`"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"id": "3e09f234-5b8b-4bfb-8d42-bb766a457acc",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# 导入依赖\n",
|
||||
"import os\n",
|
||||
"import platform\n",
|
||||
"import argparse\n",
|
||||
"import time\n",
|
||||
"import math\n",
|
||||
"import warnings\n",
|
||||
"\n",
|
||||
"import pandas as pd\n",
|
||||
"import torch\n",
|
||||
"import torch.nn.functional as F\n",
|
||||
"import torch.distributed as dist\n",
|
||||
"from contextlib import nullcontext\n",
|
||||
"\n",
|
||||
"from torch import optim, nn\n",
|
||||
"from torch.nn.parallel import DistributedDataParallel\n",
|
||||
"from torch.utils.data import DataLoader, DistributedSampler\n",
|
||||
"from transformers import AutoTokenizer, AutoModelForCausalLM\n",
|
||||
"from model.model import MiniMindLM\n",
|
||||
"from model.LMConfig import LMConfig\n",
|
||||
"from model.dataset import DPODataset"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 2,
|
||||
"id": "eb946301-eb7e-45c6-9e1c-16cc0ee4db82",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"warnings.filterwarnings('ignore')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "d70c65d3-14f7-4569-bebd-cddb48f1851f",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 可选参数设置\n",
|
||||
"\n",
|
||||
"首先，查看训练的可选参数，这些参数在实际使用时通过命令行导入，为了保持笔记本的易用性，选择用 class 进行包装."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 3,
|
||||
"id": "b78b11a1-9189-4eaf-a094-07c0497c4d38",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"class args:\n",
|
||||
" # out_dir: str = \"out\" # pytorch 格式权重文件保存位置 我们只展示训练过程 所以不使用\n",
|
||||
" epochs: int = 1 # 训练轮数\n",
|
||||
" batch_size: int = 2 # pretrain 数据集仅两个样本，设置 batch 为 2\n",
|
||||
" # sft阶段学习率为 「5e-6」->「5e-7」长度512，建议离线正负样本「概率」偏好对齐阶段lr <=「1e-8」长度3000，否则很容易遗忘训坏\n",
|
||||
" learning_rate: float = 5e-4 # 学习率\n",
|
||||
" device: str = 'cuda' if torch.cuda.is_available() else 'cpu'\n",
|
||||
" dtype: str = 'bfloat16' # 16 bit 浮点数：8 bit 指数 + 7 bit 尾数\n",
|
||||
" # use_wandb: bool = False # 是否使用 wandb 我们不使用\n",
|
||||
" wandb_project: str = 'MiniMind-Notebook'\n",
|
||||
" num_workers: int = 1 # 工作进程数\n",
|
||||
" # ddp：bool = False # 单机多卡\n",
|
||||
" accumulation_steps: int = 1 # 梯度累积步数\n",
|
||||
" grad_clip: float = 1.0 # 梯度剪裁\n",
|
||||
" warmup_iters: int = 0 # 学习率热启动\n",
|
||||
" log_interval: int = 1 # 每一步打印日志 仅用于观察\n",
|
||||
" # save_interval: int = 100 # checkpoint 保存点 我们不使用\n",
|
||||
" local_rank: int = 1 # device 设备号\n",
|
||||
" dim: int = 512 # 词嵌入维度 模型超参数\n",
|
||||
" n_layers: int = 1 # MiniMind Block 数量 模型超参数 | 由于 dpo 要加载两个模型 我们出于演示目的设定 n_layers = 1\n",
|
||||
" max_seq_len: int = 512 # 序列长度阈值\n",
|
||||
" use_moe: bool = False # 是否启用混合专家\n",
|
||||
" data_path: str = './toydata/dpo_data.jsonl' # 数据集路径"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"id": "0a72c8aa-ca60-4b0d-aabb-2c9d16b48175",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"查看工作设备 cuda\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"print(f'查看工作设备 {args.device}')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "f8fb31ff-524b-4788-a25f-699edb887704",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 初始化训练\n",
|
||||
"\n",
|
||||
"接下来，我们对一些重要模块进行初始化，我们已经了解过，分词器，模型和数据集是大模型的基本组件，我们对其进行初始化.\n",
|
||||
"\n",
|
||||
"> 在这一阶段 我们调整的是大模型的问答偏好 因此与 sft 阶段同理 我们需要载入在 sft 阶段微调好的问答模型"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 5,
|
||||
"id": "5d5971fc-4794-4903-b418-76f266bf19d9",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def init_model(lm_config):\n",
|
||||
" \"\"\"注意 在此处我们注释了加载模型部分代码 但应当认识到 actor 和 ref 两个模型都具备 sft 阶段获得的权重\"\"\"\n",
|
||||
" tokenizer = AutoTokenizer.from_pretrained('../model/minimind_tokenizer')\n",
|
||||
" model = MiniMindLM(lm_config)\n",
|
||||
" moe_path = '_moe' if lm_config.use_moe else ''\n",
|
||||
" # ckp = f'./out/full_sft_{lm_config.dim}{moe_path}.pth' # 指示上一阶段训练保存的模型文件位置\n",
|
||||
" # state_dict = torch.load(ckp, map_location=args.device) # 载入模型状态字典\n",
|
||||
" # model.load_state_dict(state_dict, strict=False) # 装入模型\n",
|
||||
" # 初始化参考模型\n",
|
||||
" ref_model = MiniMindLM(lm_config)\n",
|
||||
" # ref_model.load_state_dict(state_dict, strict=False)\n",
|
||||
" ref_model.eval()\n",
|
||||
" ref_model.requires_grad_(False)\n",
|
||||
"\n",
|
||||
" print(f'LLM总参数量：{sum(p.numel() for p in model.parameters() if p.requires_grad) / 1e6:.3f} 百万')\n",
|
||||
" model = model.to(args.device)\n",
|
||||
" ref_model = ref_model.to(args.device)\n",
|
||||
"\n",
|
||||
" return model, ref_model, tokenizer"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 6,
|
||||
"id": "fbd7bd15-a060-4aba-8c5d-277705ec6b52",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"LLM总参数量：6.096 百万\n",
|
||||
"模型位于设备：cuda:0, 词表长度：6400, DataLoader：<torch.utils.data.dataloader.DataLoader object at 0x000001AF0BBB71C0>\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"lm_config = LMConfig(dim=args.dim, n_layers=args.n_layers, max_seq_len=args.max_seq_len, use_moe=args.use_moe)\n",
|
||||
"model, ref_model ,tokenizer = init_model(lm_config)\n",
|
||||
"\n",
|
||||
"train_ds = DPODataset(args.data_path, tokenizer, max_length=lm_config.max_seq_len)\n",
|
||||
"\n",
|
||||
"train_loader = DataLoader(\n",
|
||||
" train_ds,\n",
|
||||
" batch_size=args.batch_size,\n",
|
||||
" pin_memory=True,\n",
|
||||
" drop_last=False,\n",
|
||||
" shuffle=False,\n",
|
||||
" num_workers=args.num_workers,\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"print(f'模型位于设备：{model.device}, 词表长度：{tokenizer.vocab_size}, DataLoader：{train_loader}')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 7,
|
||||
"id": "d807a2af-41e9-4115-b470-a05f6a04613c",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"打印一个 iter 的数据:\n",
|
||||
"{'x_chosen': tensor([[ 1, 85, 736, ..., 0, 0, 0],\n",
|
||||
" [ 1, 85, 736, ..., 0, 0, 0]]), 'y_chosen': tensor([[ 85, 736, 201, ..., 0, 0, 0],\n",
|
||||
" [ 85, 736, 201, ..., 0, 0, 0]]), 'mask_chosen': tensor([[0, 0, 0, ..., 0, 0, 0],\n",
|
||||
" [0, 0, 0, ..., 0, 0, 0]]), 'x_rejected': tensor([[ 1, 85, 736, ..., 0, 0, 0],\n",
|
||||
" [ 1, 85, 736, ..., 0, 0, 0]]), 'y_rejected': tensor([[ 85, 736, 201, ..., 0, 0, 0],\n",
|
||||
" [ 85, 736, 201, ..., 0, 0, 0]]), 'mask_rejected': tensor([[0, 0, 0, ..., 0, 0, 0],\n",
|
||||
" [0, 0, 0, ..., 0, 0, 0]])}\n",
|
||||
"\n",
|
||||
"数据集大小：2, DataLoader 大小：1\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"loader = iter(train_loader)\n",
|
||||
"print(f'打印一个 iter 的数据:\\n{next(loader)}\\n')\n",
|
||||
"print(f'数据集大小：{len(train_ds)}, DataLoader 大小：{len(loader)}')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "665ad506-ab12-4186-ad63-ada143a33d52",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"我们发现，train loader 的每一个 iter 都包含一个拥有六个键值对的字典，这是因为 train_dataset 每一次取数据都会返回:\n",
|
||||
"\n",
|
||||
"- chosen 样本 X: 包含 \\<bos> 在内的输入 content\n",
|
||||
"- chosen 标签 Y: 包含 \\<eos> 在内的输出 content\n",
|
||||
"- chosen 掩码 loss_mask: 指示需要计算损失的 token 位置\n",
|
||||
"- rejected 样本 X: 包含 \\<bos> 在内的输入 content\n",
|
||||
"- rejected 标签 Y: 包含 \\<eos> 在内的输出 content\n",
|
||||
"- rejected 掩码 loss_mask: 指示需要计算损失的 token 位置\n",
|
||||
"\n",
|
||||
"由于我们的数据集只有两条数据，而 batch size 设置为 2，因此我们的 dataloader 只有一个 iter."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "32462eb9-7008-4389-a54b-47ef1c1c4732",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# 启动训练\n",
|
||||
"\n",
|
||||
"训练一个深度学习模型，还涉及到了优化器，损失函数和学习率调度. 接下来，我们查看 MiniMind 训练部分的代码，并进行一轮简单的训练.\n",
|
||||
"\n",
|
||||
"> DPO 阶段涉及 DPO 损失函数涉及 因此与前两个阶段相比内容略有增加 不过整体流程与逻辑类似"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 8,
|
||||
"id": "1752f117-33c0-4810-886f-fbd1b4d44c28",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# 学习率调度方面 采用余弦退火学习率\n",
|
||||
"def get_lr(current_step, total_steps, lr):\n",
|
||||
" return lr / 10 + 0.5 * lr * (1 + math.cos(math.pi * current_step / total_steps))\n",
|
||||
"\n",
|
||||
"# 优化器方面 选择 AdamW 优化器 并在混精度场景下创建 scaler 进行梯度缩放避免数值下溢\n",
|
||||
"scaler = torch.cuda.amp.GradScaler(enabled=(args.dtype in ['float16', 'bfloat16']))\n",
|
||||
"optimizer = optim.AdamW(model.parameters(), lr=args.learning_rate)\n",
|
||||
"\n",
|
||||
"device_type = \"cuda\" if \"cuda\" in args.device else \"cpu\"\n",
|
||||
"ctx = nullcontext() if device_type == \"cpu\" else torch.cuda.amp.autocast() # 在 cuda 上启动混精度训练，否则空白上下文"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "5f2c3ac1-92bf-4eae-8d22-7fac4cfa69df",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"DPO 的原理是增加偏好样本的对数概率与减小非偏好样本响应的对数概率.\n",
|
||||
"\n",
|
||||
"该阶段引入 DPO 损失函数，通过计算选择样本和拒绝样本的对数比率，然后基于这些比率计算 DPO 损失，适用于偏好学习任务."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 9,
|
||||
"id": "2c07f54a-496e-4b1d-9394-dd2c266ab5d3",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def logits_to_probs(logits, labels):\n",
|
||||
" # logits shape: (batch_size, seq_len, vocab_size)\n",
|
||||
" # labels shape: (batch_size, seq_len)\n",
|
||||
" # probs shape: (batch_size, seq_len)\n",
|
||||
" log_probs = F.log_softmax(logits, dim=2)\n",
|
||||
" probs = torch.gather(log_probs, dim=2, index=labels.unsqueeze(2)).squeeze(-1)\n",
|
||||
" return probs\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"def dpo_loss(ref_probs, probs, beta):\n",
|
||||
" # ref_probs 和 probs 都是 shape: (batch_size, seq_len)\n",
|
||||
" # 计算每个样本的平均概率\n",
|
||||
" ref_probs = ref_probs.mean(dim=1)\n",
|
||||
" probs = probs.mean(dim=1)\n",
|
||||
"\n",
|
||||
" # 将 chosen 和 rejected 数据分开\n",
|
||||
" batch_size = ref_probs.shape[0]\n",
|
||||
" chosen_ref_probs = ref_probs[:batch_size // 2]\n",
|
||||
" reject_ref_probs = ref_probs[batch_size // 2:]\n",
|
||||
" chosen_probs = probs[:batch_size // 2]\n",
|
||||
" reject_probs = probs[batch_size // 2:]\n",
|
||||
"\n",
|
||||
" # 计算对数比率，比较偏好差异\n",
|
||||
" pi_logratios = chosen_probs - reject_probs\n",
|
||||
" ref_logratios = chosen_ref_probs - reject_ref_probs\n",
|
||||
" logits = pi_logratios - ref_logratios\n",
|
||||
" loss = -F.logsigmoid(beta * logits)\n",
|
||||
" return loss.mean()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "e743562c-6e34-488d-bd84-f62eaee0ae20",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"接下来，我们来看看 MiniMind 的训练函数"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 10,
|
||||
"id": "493b7368-d82f-48a6-b5bd-7737b7a23bee",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def train_epoch(epoch):\n",
|
||||
" start_time = time.time()\n",
|
||||
" for step, batch in enumerate(train_loader):\n",
|
||||
" # 提取数据\n",
|
||||
" x_chosen = batch['x_chosen'].to(args.device)\n",
|
||||
" x_rejected = batch['x_rejected'].to(args.device)\n",
|
||||
" y_chosen = batch['y_chosen'].to(args.device)\n",
|
||||
" y_rejected = batch['y_rejected'].to(args.device)\n",
|
||||
" mask_chosen = batch['mask_chosen'].to(args.device)\n",
|
||||
" mask_rejected = batch['mask_rejected'].to(args.device)\n",
|
||||
" # 正反例拼接\n",
|
||||
" x = torch.cat([x_chosen, x_rejected], dim=0)\n",
|
||||
" y = torch.cat([y_chosen, y_rejected], dim=0)\n",
|
||||
" mask = torch.cat([mask_chosen, mask_rejected], dim=0)\n",
|
||||
"\n",
|
||||
" lr = get_lr(epoch * iter_per_epoch + step, args.epochs * iter_per_epoch, args.learning_rate)\n",
|
||||
" for param_group in optimizer.param_groups:\n",
|
||||
" param_group['lr'] = lr\n",
|
||||
"\n",
|
||||
" with ctx:\n",
|
||||
" with torch.no_grad(): # 计算 ref 模型输出\n",
|
||||
" ref_outputs = ref_model(x)\n",
|
||||
" ref_logits = ref_outputs.logits\n",
|
||||
" ref_probs = logits_to_probs(ref_logits, y)\n",
|
||||
" ref_probs = ref_probs * mask # 得到 ref 概率\n",
|
||||
" outputs = model(x) # 计算 actor 模型输出\n",
|
||||
" logits = outputs.logits\n",
|
||||
" probs = logits_to_probs(logits, y)\n",
|
||||
" probs = probs * mask # 得到 actor 概率\n",
|
||||
" loss = dpo_loss(ref_probs, probs, beta=0.1) # dpo 损失\n",
|
||||
" loss = loss / args.accumulation_steps\n",
|
||||
"\n",
|
||||
" scaler.scale(loss).backward() # 梯度缩放\n",
|
||||
"\n",
|
||||
" if (step + 1) % args.accumulation_steps == 0:\n",
|
||||
" scaler.unscale_(optimizer)\n",
|
||||
" torch.nn.utils.clip_grad_norm_(model.parameters(), args.grad_clip) # 梯度剪裁\n",
|
||||
" scaler.step(optimizer)\n",
|
||||
" scaler.update()\n",
|
||||
" optimizer.zero_grad(set_to_none=True)\n",
|
||||
"\n",
|
||||
" if step % args.log_interval == 0:\n",
|
||||
" spend_time = time.time() - start_time\n",
|
||||
" print(\n",
|
||||
" 'Epoch:[{}/{}]({}/{}) loss:{:.3f} lr:{:.12f} epoch_Time:{}min:'.format(\n",
|
||||
" epoch + 1,\n",
|
||||
" args.epochs,\n",
|
||||
" step,\n",
|
||||
" iter_per_epoch,\n",
|
||||
" loss.item(),\n",
|
||||
" optimizer.param_groups[-1]['lr'],\n",
|
||||
" spend_time / (step + 1) * iter_per_epoch // 60 - spend_time // 60))\n",
|
||||
"\n",
|
||||
" # 到达指定保存步数时，save as PyTorch\n",
|
||||
" # if (step + 1) % args.save_interval == 0 and (not ddp or dist.get_rank() == 0):\n",
|
||||
" # model.eval()\n",
|
||||
" # moe_path = '_moe' if lm_config.use_moe else ''\n",
|
||||
" # ckp = f'{args.save_dir}/rlhf_{lm_config.dim}{moe_path}.pth'\n",
|
||||
"\n",
|
||||
" # if isinstance(model, torch.nn.parallel.DistributedDataParallel):\n",
|
||||
" # state_dict = model.module.state_dict()\n",
|
||||
" # else:\n",
|
||||
" # state_dict = model.state_dict()\n",
|
||||
"\n",
|
||||
" # torch.save(state_dict, ckp)\n",
|
||||
" # model.train()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "49386dc7-2919-428a-a073-f37d576afbcf",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"准备完毕，我们尝试一轮长度 1 个 iter 的训练."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 11,
|
||||
"id": "0ac86572-ee04-43b5-b5fa-b1c65cbaebec",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Epoch:[1/1](0/1) loss:0.694 lr:0.000550000000 epoch_Time:0.0min:\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"iter_per_epoch = len(train_loader)\n",
|
||||
"for epoch in range(args.epochs):\n",
|
||||
" train_epoch(epoch)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 12,
|
||||
"id": "5215ef67-2647-4519-966b-074c46c225ac",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"del model"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3 (ipykernel)",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.10.14"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 5
|
||||
}
|
||||
|
|
@ -0,0 +1,599 @@
|
|||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "fd56e0ed-d6e3-4fdd-bed2-2a0c32fb61f9",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# 6-LoRA\n",
|
||||
"\n",
|
||||
"大语言模型的低秩适应(LoRA, Low-Rank Adaptation of Large Language Models) 是一项大模型参数高效微调技术，其可以显著减少可训练参数的数量.\n",
|
||||
"\n",
|
||||
"由于大模型参数量较大，直接进行全参微调需要消耗大量硬件资源，LoRA 的工作原理是将少量的新权重插入倒模型中，并且仅训练这些权重，这使得使用 LoRA 进行训练的速度更快、内存效率更高，并生成更小的模型权重.\n",
|
||||
"\n",
|
||||
"具体来说，LoRA 它冻结了预训练模型 W 的权重，并注入可训练的秩分解矩阵 A 与 B，在微调时，只训练降维矩阵 A 和 升维矩阵 B，微调结束后，将 AB 与 W 进行叠加.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"其中，用随机高斯分布进行初始化 A，用 0 矩阵初始化 B，从而保证训练开始时旁路矩阵为 0 矩阵.\n",
|
||||
"\n",
|
||||
"具体来看，假设模型经过预训练主干的输出为 $W_0 x$，在 LoRA 微调阶段，我们可以用如下形式对输出进行表示.\n",
|
||||
"\n",
|
||||
"$$h=W_0x + \\Delta Wx = W_0x + BA x=(W_0 + BA)x$$\n",
|
||||
"\n",
|
||||
"其中, $B \\in \\mathbb{R}^{d \\times r},A \\in \\mathbb{R}^{r\\times k}$，r 为 LoRA 低秩矩阵的维数，$r << min(d, k)$."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"id": "dad11c0c-043d-44cb-a9a7-4c36c2e909da",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import torch\n",
|
||||
"from torch import optim, nn"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "3031399b-9342-454f-86df-9a96b8c6ff50",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## LoRA Adapter\n",
|
||||
"\n",
|
||||
"简单的来说，LoRA 矩阵就是具有一个隐藏层的全连接网络，其挂接在主干网络边侧进行参数更新，我们来看看 MiniMind 模型是如何在主干网络外部定义 LoRA 网络结构的."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 2,
|
||||
"id": "2f9bc623-f13d-4d67-bf2e-dce0784b43f8",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"class LoRA(nn.Module):\n",
|
||||
" def __init__(self, in_features, out_features, rank):\n",
|
||||
" super().__init__()\n",
|
||||
" self.rank = rank # LoRA 秩 控制低秩矩阵大小\n",
|
||||
" self.A = nn.Linear(in_features, rank, bias=False)\n",
|
||||
" self.B = nn.Linear(rank, out_features, bias=False)\n",
|
||||
" # 矩阵 A 高斯分布初始化\n",
|
||||
" self.A.weight.data.normal_(mean=0.0, std=0.02)\n",
|
||||
" # 矩阵 B 全零初始化\n",
|
||||
" self.B.weight.data.zero_()\n",
|
||||
"\n",
|
||||
" def forward(self, x):\n",
|
||||
" return self.B(self.A(x))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "4a3557bd-d34d-4a15-b789-22afea60a87d",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"可以看到，LoRA 的网络结构非常简单直观，我们接下来定义一个方法，将 LoRA 网络应用到 MiniMind 模型的特定线性层."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 3,
|
||||
"id": "4d52e346-af62-40e2-a5b0-5cc0cf8ae6e8",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def apply_lora(model, rank=16):\n",
|
||||
" \"\"\"将 LoRA 模块与目标模块进行绑定\"\"\"\n",
|
||||
" for name, module in model.named_modules():\n",
|
||||
" if isinstance(module, nn.Linear) and module.weight.shape[0] == module.weight.shape[1]:\n",
|
||||
" lora = LoRA(module.weight.shape[0], module.weight.shape[1], rank=rank).to(model.device)\n",
|
||||
" setattr(module, 'lora', lora) # 显式添加 LoRA 模块\n",
|
||||
" original_forward = module.forward\n",
|
||||
"\n",
|
||||
" # 修改目标模块的 forward 函数\n",
|
||||
" def forward_with_lora(x, layer1=original_forward, layer2=lora):\n",
|
||||
" return layer1(x) + layer2(x)\n",
|
||||
" \n",
|
||||
" module.forward = forward_with_lora\n",
|
||||
" # 打印 LoRA 绑定的模块名称\n",
|
||||
" print(f'apply lora on module: {name}')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "753a54c6-a442-4ce5-be89-3bc601bc8d26",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"我们可以声明一个小模型，对于 LoRA 的绑定进行测试."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"id": "f830221b-891e-4e27-b401-20b6fe5d3c59",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"class TestModel(nn.Module):\n",
|
||||
" def __init__(self):\n",
|
||||
" super().__init__()\n",
|
||||
" self.linear1 = nn.Linear(64, 512)\n",
|
||||
" self.linear2 = nn.Linear(512, 512)\n",
|
||||
" self.linear3 = nn.Linear(512, 64)\n",
|
||||
"\n",
|
||||
" @property\n",
|
||||
" def device(self):\n",
|
||||
" return next(self.parameters()).device\n",
|
||||
" \n",
|
||||
" def forward(self, x):\n",
|
||||
" out = self.linear3(self.linear2(self.linear1))\n",
|
||||
" return out"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "1a8edbc2-f36c-4127-be3b-3c598872621a",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"按照 apply_lora 的函数逻辑，LoRA 模块会应用在主干网络中满足 input_feature == output_feature 的模块上."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 5,
|
||||
"id": "eb24e24f-da09-4cd7-b4ce-7fe482194f75",
|
||||
"metadata": {
|
||||
"scrolled": true
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"apply lora on module: linear2\n",
|
||||
"TestModel(\n",
|
||||
" (linear1): Linear(in_features=64, out_features=512, bias=True)\n",
|
||||
" (linear2): Linear(\n",
|
||||
" in_features=512, out_features=512, bias=True\n",
|
||||
" (lora): LoRA(\n",
|
||||
" (A): Linear(in_features=512, out_features=16, bias=False)\n",
|
||||
" (B): Linear(in_features=16, out_features=512, bias=False)\n",
|
||||
" )\n",
|
||||
" )\n",
|
||||
" (linear3): Linear(in_features=512, out_features=64, bias=True)\n",
|
||||
")\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"test_model = TestModel()\n",
|
||||
"apply_lora(test_model)\n",
|
||||
"print(test_model)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 6,
|
||||
"id": "f882e6c0-65a6-47b2-9ee6-4a3fca988636",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"del test_model"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "9771475e-0c60-42db-b0cd-7e193de23f7c",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"完成了 LoRA 模块在主干网络特定模块的绑定后，我们便可以冻结主干网络参数进行微调了，不过，考虑到主干网络权重在训练过程中并不会做任何参数更新，我们可以只保存 LoRA 模块的参数来节省内存，下面给出加载/保存 LoRA 权重的方法."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 7,
|
||||
"id": "95a453d7-2011-4575-9a51-36d91876874f",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def load_lora(model, path):\n",
|
||||
" state_dict = torch.load(path, map_location=model.device)\n",
|
||||
" for name, module in model.named_modules():\n",
|
||||
" if hasattr(module, 'lora'):\n",
|
||||
" lora_state = {k.replace(f'{name}.lora.', ''): v for k, v in state_dict.items() if f'{name}.lora.' in k}\n",
|
||||
" module.lora.load_state_dict(lora_state)\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"def save_lora(model, path):\n",
|
||||
" state_dict = {}\n",
|
||||
" for name, module in model.named_modules():\n",
|
||||
" if hasattr(module, 'lora'):\n",
|
||||
" lora_state = {f'{name}.lora.{k}': v for k, v in module.lora.state_dict().items()}\n",
|
||||
" state_dict.update(lora_state)\n",
|
||||
" torch.save(state_dict, path)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "854b8afb-39b1-4175-8e69-4a9d5b25a3e5",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Fine-Tuning MiniMind with LoRA"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 8,
|
||||
"id": "10baf158-6653-49e3-9ba3-22b2e341730f",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import os\n",
|
||||
"import platform\n",
|
||||
"import argparse\n",
|
||||
"import random\n",
|
||||
"import time\n",
|
||||
"import math\n",
|
||||
"import warnings\n",
|
||||
"import torch.distributed as dist\n",
|
||||
"from contextlib import nullcontext\n",
|
||||
"from torch.utils.data import DataLoader, DistributedSampler\n",
|
||||
"from transformers import AutoTokenizer, AutoModelForCausalLM\n",
|
||||
"from model.model import MiniMindLM\n",
|
||||
"from model.LMConfig import LMConfig\n",
|
||||
"from model.dataset import SFTDataset\n",
|
||||
"\n",
|
||||
"warnings.filterwarnings('ignore')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "f0b93272-80b3-45ce-9dec-4583fca8f87a",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### 可选参数设置\n",
|
||||
"\n",
|
||||
"首先，查看训练的可选参数，这些参数在实际使用时通过解析命令行进行导入，我们用 class 进行包装."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 9,
|
||||
"id": "bb9d4bac-8a30-4ca9-b293-94df3f880826",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"class args:\n",
|
||||
" # out_dir: str = \"out\" # pytorch 格式权重文件保存位置 我们只展示训练过程 所以不使用\n",
|
||||
" epochs: int = 1 # 训练轮数\n",
|
||||
" batch_size: int = 2 # pretrain 数据集仅两个样本，设置 batch 为 2\n",
|
||||
" learning_rate: float = 5e-4 # 学习率\n",
|
||||
" device: str = 'cuda' if torch.cuda.is_available() else 'cpu'\n",
|
||||
" dtype: str = 'bfloat16' # 16 bit 浮点数：8 bit 指数 + 7 bit 尾数\n",
|
||||
" # use_wandb: bool = False # 是否使用 wandb 我们不使用\n",
|
||||
" wandb_project: str = 'MiniMind-Notebook'\n",
|
||||
" num_workers: int = 1 # 工作进程数\n",
|
||||
" # ddp：bool = False # 单机多卡\n",
|
||||
" accumulation_steps: int = 1 # 梯度累积步数\n",
|
||||
" grad_clip: float = 1.0 # 梯度剪裁\n",
|
||||
" warmup_iters: int = 0 # 学习率热启动\n",
|
||||
" log_interval: int = 1 # 每一步打印日志 仅用于观察\n",
|
||||
" # save_interval: int = 100 # checkpoint 保存点 我们不使用\n",
|
||||
" local_rank: int = 1 # device 设备号\n",
|
||||
" dim: int = 512 # 词嵌入维度 模型超参数\n",
|
||||
" n_layers: int = 2 # MiniMind Block 数量 模型超参数\n",
|
||||
" max_seq_len: int = 512 # 序列长度阈值\n",
|
||||
" use_moe: bool = False # 是否启用混合专家\n",
|
||||
" data_path: str = './toydata/lora_data.jsonl' # 数据集路径\n",
|
||||
" lora_name: str = 'lora_identity' # 根据任务保存成lora_(英文/医学/心理...)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 10,
|
||||
"id": "3b496134-ac9a-4146-a236-fd8867f45bc6",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"查看工作设备 cuda\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"print(f'查看工作设备 {args.device}')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "32b956e6-e81b-49d9-bbf7-b78fbbc7eff5",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"接下来，我们对分词器、MiniMindLM 和数据迭代器执行初始化."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 11,
|
||||
"id": "2fb7c880-b86e-4329-aeef-2827837425a3",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def init_model(lm_config):\n",
|
||||
" tokenizer = AutoTokenizer.from_pretrained('../model/minimind_tokenizer')\n",
|
||||
" model = MiniMindLM(lm_config)\n",
|
||||
" moe_path = '_moe' if lm_config.use_moe else ''\n",
|
||||
" # 热启动\n",
|
||||
" # ckp = f'./out/rlhf_{lm_config.dim}{moe_path}.pth'\n",
|
||||
" # state_dict = torch.load(ckp, map_location=args.device)\n",
|
||||
" # model.load_state_dict(state_dict, strict=False)\n",
|
||||
" return model.to(args.device), tokenizer"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 12,
|
||||
"id": "7f0c74d0-a191-40c1-b922-104f241aa879",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"apply lora on module: layers.0.attention.wq\n",
|
||||
"apply lora on module: layers.0.attention.wo\n",
|
||||
"apply lora on module: layers.1.attention.wq\n",
|
||||
"apply lora on module: layers.1.attention.wo\n",
|
||||
"模型位于设备：cuda:0, 词表长度：6400, DataLoader：<torch.utils.data.dataloader.DataLoader object at 0x0000022F5786D4E0>\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"lm_config = LMConfig(dim=args.dim, n_layers=args.n_layers, max_seq_len=args.max_seq_len, use_moe=args.use_moe)\n",
|
||||
"model, tokenizer = init_model(lm_config)\n",
|
||||
"apply_lora(model)\n",
|
||||
"\n",
|
||||
"# 由于 MiniMind 用于 LoRA 微调的数据集和 SFT 数据集格式相同，可以用 SFT 数据集进行加载\n",
|
||||
"train_ds = SFTDataset(args.data_path, tokenizer, max_length=lm_config.max_seq_len)\n",
|
||||
"\n",
|
||||
"train_loader = DataLoader(\n",
|
||||
" train_ds,\n",
|
||||
" batch_size=args.batch_size,\n",
|
||||
" pin_memory=True,\n",
|
||||
" drop_last=False,\n",
|
||||
" shuffle=False,\n",
|
||||
" num_workers=args.num_workers,\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"print(f'模型位于设备：{model.device}, 词表长度：{tokenizer.vocab_size}, DataLoader：{train_loader}')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "c35bb3ae-0b31-42ff-88d1-04d7e945bd31",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"可以看到，LoRA 模块挂接在 Attention Block 的 Query 与 Output 线性层上，下面查看 LoRA 微调下可学习参数的占比："
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 13,
|
||||
"id": "766ef2d5-1f24-4efb-9eb9-f969a57a85b0",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"LLM 总参数量: 8980992\n",
|
||||
"LoRA 参数量: 65536\n",
|
||||
"LoRA 参数占比: 0.73%\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"total_params = sum(p.numel() for p in model.parameters()) # 总参数数量\n",
|
||||
"lora_params_count = sum(p.numel() for name, p in model.named_parameters() if 'lora' in name) # LoRA 参数数量\n",
|
||||
"print(f\"LLM 总参数量: {total_params}\")\n",
|
||||
"print(f\"LoRA 参数量: {lora_params_count}\")\n",
|
||||
"print(f\"LoRA 参数占比: {lora_params_count / total_params * 100:.2f}%\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "8a487807-65da-4e19-ac2f-fce7077494c0",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"接下来，冻结 MiniMindLM 主干网络的参数，做好 LoRA 微调准备."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 14,
|
||||
"id": "db99fd52-4f07-4278-939f-47b91aeeedbe",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"for name, param in model.named_parameters():\n",
|
||||
" if 'lora' not in name:\n",
|
||||
" param.requires_grad = False\n",
|
||||
"lora_params = [] # 收集 LoRA 模块的可学习参数, 提供给优化器\n",
|
||||
"for name, param in model.named_parameters():\n",
|
||||
" if 'lora' in name:\n",
|
||||
" lora_params.append(param)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "6f865b22-35a7-4c2c-85a1-a13d1ada12a7",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### 启动训练\n",
|
||||
"\n",
|
||||
"接下来，我们定义 MiniMind LoRA 微调所使用的优化器，损失函数和学习率调度，并进行一轮简单的训练."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 15,
|
||||
"id": "2b57beaf-c04a-4b88-9617-7142054f2927",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# 学习率调度方面 采用余弦退火学习率\n",
|
||||
"def get_lr(current_step, total_steps, lr):\n",
|
||||
" return lr / 10 + 0.5 * lr * (1 + math.cos(math.pi * current_step / total_steps))\n",
|
||||
"\n",
|
||||
"# 优化器方面 选择 AdamW 优化器 并在混精度场景下创建 scaler 进行梯度缩放避免数值下溢\n",
|
||||
"scaler = torch.cuda.amp.GradScaler(enabled=(args.dtype in ['float16', 'bfloat16']))\n",
|
||||
"# 只训练 LoRA 模块参数\n",
|
||||
"optimizer = optim.AdamW(lora_params, lr=args.learning_rate)\n",
|
||||
"\n",
|
||||
"device_type = \"cuda\" if \"cuda\" in args.device else \"cpu\"\n",
|
||||
"ctx = nullcontext() if device_type == \"cpu\" else torch.cuda.amp.autocast() # 在 cuda 上启动混精度训练，否则空白上下文"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "8037d2b2-1fa3-4e19-9ad5-9674ff866a36",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"接下来，我们来看训练函数."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 16,
|
||||
"id": "8f50d42d-da6d-4f8b-aa35-853b3900f8b5",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# 代码和 SFT 几乎一致\n",
|
||||
"def train_epoch(epoch):\n",
|
||||
" loss_fct = nn.CrossEntropyLoss(reduction='none')\n",
|
||||
" start_time = time.time()\n",
|
||||
" for step, (X, Y, loss_mask) in enumerate(train_loader):\n",
|
||||
" X = X.to(args.device)\n",
|
||||
" Y = Y.to(args.device)\n",
|
||||
" loss_mask = loss_mask.to(args.device)\n",
|
||||
" lr = get_lr(epoch * iter_per_epoch + step, args.epochs * iter_per_epoch, args.learning_rate)\n",
|
||||
" for param_group in optimizer.param_groups:\n",
|
||||
" param_group['lr'] = lr\n",
|
||||
"\n",
|
||||
" with ctx:\n",
|
||||
" res = model(X)\n",
|
||||
" loss = loss_fct(\n",
|
||||
" res.logits.view(-1, res.logits.size(-1)),\n",
|
||||
" Y.view(-1)\n",
|
||||
" ).view(Y.size())\n",
|
||||
" loss = (loss * loss_mask).sum() / loss_mask.sum()\n",
|
||||
" loss += res.aux_loss\n",
|
||||
" loss = loss / args.accumulation_steps\n",
|
||||
"\n",
|
||||
" scaler.scale(loss).backward() # 梯度缩放\n",
|
||||
"\n",
|
||||
" if (step + 1) % args.accumulation_steps == 0:\n",
|
||||
" scaler.unscale_(optimizer)\n",
|
||||
" torch.nn.utils.clip_grad_norm_(lora_params, args.grad_clip) # 梯度剪裁\n",
|
||||
"\n",
|
||||
" scaler.step(optimizer)\n",
|
||||
" scaler.update()\n",
|
||||
"\n",
|
||||
" optimizer.zero_grad(set_to_none=True)\n",
|
||||
"\n",
|
||||
" if step % args.log_interval == 0:\n",
|
||||
" spend_time = time.time() - start_time\n",
|
||||
" print(\n",
|
||||
" 'Epoch:[{}/{}]({}/{}) loss:{:.3f} lr:{:.12f} epoch_Time:{}min:'.format(\n",
|
||||
" epoch + 1,\n",
|
||||
" args.epochs,\n",
|
||||
" step,\n",
|
||||
" iter_per_epoch,\n",
|
||||
" loss.item(),\n",
|
||||
" optimizer.param_groups[-1]['lr'],\n",
|
||||
" spend_time / (step + 1) * iter_per_epoch // 60 - spend_time // 60))\n",
|
||||
"\n",
|
||||
" # 只保存 LoRA 模块的权重\n",
|
||||
" # if (step + 1) % args.save_interval == 0 and (not ddp or dist.get_rank() == 0):\n",
|
||||
" # model.eval()\n",
|
||||
" # # 【区别1】只保存lora权重即可\n",
|
||||
" # save_lora(model, f'{args.save_dir}/lora/{args.lora_name}_{lm_config.dim}.pth')\n",
|
||||
" # model.train()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "618876fa-7f0f-43b3-9eed-384be6cd66be",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"接下来，我们启动一个 Epoch 的训练进行观察."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 17,
|
||||
"id": "38584341-9ef6-4ebd-9024-0c44341de7c8",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Epoch:[1/1](0/1) loss:8.883 lr:0.000550000000 epoch_Time:0.0min:\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"iter_per_epoch = len(train_loader)\n",
|
||||
"for epoch in range(args.epochs):\n",
|
||||
" train_epoch(epoch)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 18,
|
||||
"id": "61150c72-39eb-4b90-b321-d4c7aa6c885a",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"del model"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "dd5e9aa8-b5a5-4380-989d-e54948f33a1b",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 参考资料\n",
|
||||
"\n",
|
||||
"- [LoRA: Low-Rank Adaptation of Large Language Models](https://arxiv.org/abs/2106.09685)\n",
|
||||
"- [HuggingFace LoRA](https://huggingface.co/docs/diffusers/en/training/lora)\n",
|
||||
"- [LoRA 微调和低秩矩阵](https://www.cnblogs.com/ghj1976/p/18032882/lora-finetuning-he-di-zhi-ju-zhen)"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3 (ipykernel)",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.10.14"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 5
|
||||
}
|
||||
|
|
@ -0,0 +1,430 @@
|
|||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "e0f42399-b424-41a3-b179-a6f1165a92e7",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# 7-Distill\n",
|
||||
"\n",
|
||||
"模型蒸馏 (Knowledge Distillation, KD) 是一种机器学习模型压缩方法，它用于将大型模型（教师模型）的知识迁移到较小的模型（学生模型）中.\n",
|
||||
"\n",
|
||||
"KD 背后的核心思想是将教师模型的综合知识转化为更精简、更有效的表示. 学生模型是一个较小的模型，目标是学习教师模型的行为，而不是直接从原始数据中学习.\n",
|
||||
"\n",
|
||||
"大模型的 KD 有白盒蒸馏与黑盒蒸馏两个派别，对于本次实验代码中两个模型均为 MiniMind 开源模型，支持对教师模型内部结构的访问，因此在训练过程中，我们能够获取教师模型的 softmax 概率分布并用作软标签（soft labels），让小模型学习软标签，并使用 KL-Loss 来优化模型的参数，而不是直接学习输出 Token 的硬标签. 对于下一章蒸馏推理模型中，由于我们面向推理数据集进行蒸馏，并不存在输出 Token 的概率分布让我们学习，这种面向输出数据学习的蒸馏方式被称为黑盒蒸馏.\n",
|
||||
"\n",
|
||||
"此笔记本的完整实现见主仓库 `/minimind/train_distillation.py`"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"id": "fb823b9f-c0d3-4566-ad9c-f9ec6f95c446",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import os\n",
|
||||
"import argparse\n",
|
||||
"import time\n",
|
||||
"import math\n",
|
||||
"import warnings\n",
|
||||
"\n",
|
||||
"import pandas as pd\n",
|
||||
"import torch\n",
|
||||
"import torch.nn.functional as F\n",
|
||||
"import torch.distributed as dist\n",
|
||||
"from contextlib import nullcontext\n",
|
||||
"\n",
|
||||
"from torch import optim, nn\n",
|
||||
"from torch.nn.parallel import DistributedDataParallel\n",
|
||||
"from torch.utils.data import DataLoader, DistributedSampler\n",
|
||||
"from transformers import AutoTokenizer, AutoModelForCausalLM\n",
|
||||
"from model.model import MiniMindLM\n",
|
||||
"from model.LMConfig import LMConfig\n",
|
||||
"from model.dataset import SFTDataset\n",
|
||||
"\n",
|
||||
"warnings.filterwarnings('ignore')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "5f4546ae-9520-4f67-8530-0dfed996fa35",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 可选参数设置\n",
|
||||
"\n",
|
||||
"首先，查看训练的可选参数，这些参数在实际使用时通过解析命令行进行导入，我们用 class 进行包装."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 2,
|
||||
"id": "c7907dbe-ca43-4904-b598-ab9e5f019982",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"class args:\n",
|
||||
" # out_dir: str = \"out\" # pytorch 格式权重文件保存位置 我们只展示训练过程 所以不使用\n",
|
||||
" epochs: int = 1 # 训练轮数\n",
|
||||
" batch_size: int = 2 # pretrain 数据集仅两个样本，设置 batch 为 2\n",
|
||||
" learning_rate: float = 5e-4 # 学习率\n",
|
||||
" device: str = 'cuda' if torch.cuda.is_available() else 'cpu'\n",
|
||||
" dtype: str = 'bfloat16' # 16 bit 浮点数：8 bit 指数 + 7 bit 尾数\n",
|
||||
" # use_wandb: bool = False # 是否使用 wandb 我们不使用\n",
|
||||
" wandb_project: str = 'MiniMind-Notebook'\n",
|
||||
" num_workers: int = 1 # 工作进程数\n",
|
||||
" # ddp：bool = False # 单机多卡\n",
|
||||
" accumulation_steps: int = 1 # 梯度累积步数\n",
|
||||
" grad_clip: float = 1.0 # 梯度剪裁\n",
|
||||
" warmup_iters: int = 0 # 学习率热启动\n",
|
||||
" log_interval: int = 1 # 每一步打印日志 仅用于观察\n",
|
||||
" # save_interval: int = 100 # checkpoint 保存点 我们不使用\n",
|
||||
" local_rank: int = 1 # device 设备号\n",
|
||||
" dim: int = 512 # 词嵌入维度 模型超参数\n",
|
||||
" n_layers: int = 2 # MiniMind Block 数量 模型超参数\n",
|
||||
" max_seq_len: int = 512 # 序列长度阈值\n",
|
||||
" use_moe: bool = False # 是否启用混合专家\n",
|
||||
" data_path: str = './toydata/sft_data.jsonl' # 数据集路径"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 3,
|
||||
"id": "044bb66f-f878-4785-b860-a9a89b52f94c",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"查看工作设备 cuda\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"print(f'查看工作设备 {args.device}')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "68f3ea54-b23c-4d2c-abe7-63bbe9f1e5dd",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"接下来，我们对分词器、MiniMind 教师/学生模型以及数据迭代器执行初始化."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"id": "f355d146-ee89-41c6-ba79-7aba75e3d151",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def init_student_model(lm_config):\n",
|
||||
" tokenizer = AutoTokenizer.from_pretrained('../model/minimind_tokenizer')\n",
|
||||
" model = MiniMindLM(lm_config)\n",
|
||||
" moe_path = '_moe' if lm_config.use_moe else ''\n",
|
||||
" # 学生模型热启动\n",
|
||||
" # ckp = f'./out/full_sft_{lm_config.dim}{moe_path}.pth'\n",
|
||||
" # state_dict = torch.load(ckp, map_location=args.device)\n",
|
||||
" # model.load_state_dict(state_dict, strict=False)\n",
|
||||
" print(f'学生模型(LLM)总参数量：{sum(p.numel() for p in model.parameters() if p.requires_grad) / 1e6:.3f} 百万')\n",
|
||||
" model = model.to(args.device)\n",
|
||||
" return model, tokenizer\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"def init_teacher_model(lm_config):\n",
|
||||
" model = MiniMindLM(lm_config)\n",
|
||||
" moe_path = '_moe' if lm_config.use_moe else ''\n",
|
||||
" # 教师模型热启动\n",
|
||||
" # ckp = f'./out/full_sft_{lm_config.dim}{moe_path}.pth'\n",
|
||||
" # state_dict = torch.load(ckp, map_location=args.device)\n",
|
||||
" # model.load_state_dict(state_dict, strict=False)\n",
|
||||
" print(f'教师模型(LLM)总参数量：{sum(p.numel() for p in model.parameters() if p.requires_grad) / 1e6:.3f} 百万')\n",
|
||||
" model = model.to(args.device)\n",
|
||||
" return model"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 5,
|
||||
"id": "5592c247-4424-45e0-949a-0a56b1604f83",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"学生模型(LLM)总参数量：6.096 百万\n",
|
||||
"教师模型(LLM)总参数量：17.305 百万\n",
|
||||
"模型位于设备：cuda:0, 词表长度：6400, DataLoader：<torch.utils.data.dataloader.DataLoader object at 0x0000021E5B6E7010>\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"# 初始化模型配置，一般学生模型比较小，从大的教师模型那里学知识\n",
|
||||
"lm_config_student = LMConfig(dim=512, n_layers=1, max_seq_len=512)\n",
|
||||
"lm_config_teacher = LMConfig(dim=768, n_layers=2, max_seq_len=512)\n",
|
||||
"\n",
|
||||
"model, tokenizer = init_student_model(lm_config_student)\n",
|
||||
"teacher_model = init_teacher_model(lm_config_teacher)\n",
|
||||
"\n",
|
||||
"train_ds = SFTDataset(args.data_path, tokenizer, max_length=lm_config_student.max_seq_len)\n",
|
||||
"train_loader = DataLoader(\n",
|
||||
" train_ds,\n",
|
||||
" batch_size=args.batch_size,\n",
|
||||
" pin_memory=True,\n",
|
||||
" drop_last=False,\n",
|
||||
" shuffle=False,\n",
|
||||
" num_workers=args.num_workers,\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"print(f'模型位于设备：{model.device}, 词表长度：{tokenizer.vocab_size}, DataLoader：{train_loader}')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "9af25f3d-6127-4053-92e6-533bd889a917",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 启动训练\n",
|
||||
"\n",
|
||||
"接下来，我们定义 MiniMind LoRA 微调所使用的优化器，损失函数和学习率调度，并进行一轮简单的训练."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 6,
|
||||
"id": "ae344278-4f4c-495e-8c1b-e5c542ae7a92",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# 学习率调度方面 采用余弦退火学习率\n",
|
||||
"def get_lr(current_step, total_steps, lr):\n",
|
||||
" return lr / 10 + 0.5 * lr * (1 + math.cos(math.pi * current_step / total_steps))\n",
|
||||
"\n",
|
||||
"# 优化器方面 选择 AdamW 优化器 并在混精度场景下创建 scaler 进行梯度缩放避免数值下溢\n",
|
||||
"scaler = torch.cuda.amp.GradScaler(enabled=(args.dtype in ['float16', 'bfloat16']))\n",
|
||||
"# 优化学生模型参数\n",
|
||||
"optimizer = optim.AdamW(model.parameters(), lr=args.learning_rate)\n",
|
||||
"\n",
|
||||
"device_type = \"cuda\" if \"cuda\" in args.device else \"cpu\"\n",
|
||||
"ctx = nullcontext() if device_type == \"cpu\" else torch.cuda.amp.autocast() # 在 cuda 上启动混精度训练，否则空白上下文"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "ceadf787-ab7c-4e3e-958a-4b3cd3126a08",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"损失函数方面，使用 KL Loss 方法. \n",
|
||||
"\n",
|
||||
"KL Loss 中，损失是 KL 散度，衡量学生模型和教师模型在面对相同输入时，在输出层产生的分类 logits 分布之间的距离. 直观理解上，就是让学生模型的输出尽量向教师模型的输出概率靠近.\n",
|
||||
"\n",
|
||||
"$$D_{KL}(P||Q)=\\sum_i P(i)\\log\\frac{P(i)}{Q(i)}$$\n",
|
||||
"\n",
|
||||
"其中，$P(i)$ 代表教师模型的概率分布，$Q(i)$ 代表学生模型的预测分布."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 7,
|
||||
"id": "d273971f-50ec-4319-807d-1a8e59ea9813",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def distillation_loss_fn(student_logits, teacher_logits, temperature=1.0, reduction='batchmean'):\n",
|
||||
" with torch.no_grad():\n",
|
||||
" teacher_probs = F.softmax(teacher_logits / temperature, dim=-1).detach()\n",
|
||||
"\n",
|
||||
" student_log_probs = F.log_softmax(student_logits / temperature, dim=-1)\n",
|
||||
"\n",
|
||||
" kl = F.kl_div(\n",
|
||||
" student_log_probs,\n",
|
||||
" teacher_probs,\n",
|
||||
" reduction=reduction # 对各批次损失求平均值\n",
|
||||
" )\n",
|
||||
" return (temperature ** 2) * kl # 尺度不变"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "2ce6f00d-1cbd-47ab-8736-991eeb904cd9",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"接下来，我们来看训练函数."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 8,
|
||||
"id": "cd01790d-6a18-40ab-b133-da31b3ca6b66",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# 作者在训练函数上的注释浅显易懂 故不作额外注释\n",
|
||||
"def train_epoch(epoch, alpha=0.0, temperature=1.0):\n",
|
||||
" start_time = time.time()\n",
|
||||
"\n",
|
||||
" if teacher_model is not None: # 禁用教师模型梯度\n",
|
||||
" teacher_model.eval()\n",
|
||||
" teacher_model.requires_grad_(False)\n",
|
||||
"\n",
|
||||
" for step, (X, Y, loss_mask) in enumerate(train_loader):\n",
|
||||
" X = X.to(args.device)\n",
|
||||
" Y = Y.to(args.device)\n",
|
||||
" loss_mask = loss_mask.to(args.device)\n",
|
||||
" lr = get_lr(epoch * iter_per_epoch + step,\n",
|
||||
" args.epochs * iter_per_epoch,\n",
|
||||
" args.learning_rate)\n",
|
||||
" for param_group in optimizer.param_groups:\n",
|
||||
" param_group['lr'] = lr\n",
|
||||
"\n",
|
||||
" # 前向传播（学生模型）\n",
|
||||
" with ctx:\n",
|
||||
" res = model(X)\n",
|
||||
" student_logits = res.logits\n",
|
||||
"\n",
|
||||
" # 教师模型前向传播（只在eval & no_grad）\n",
|
||||
" if teacher_model is not None:\n",
|
||||
" with torch.no_grad():\n",
|
||||
" teacher_logits = teacher_model(X).logits\n",
|
||||
" vocab_size_student = student_logits.size(-1) # N\n",
|
||||
" teacher_logits = teacher_logits[..., :vocab_size_student] # ... 保留除了最后一个维度外的所有维度\n",
|
||||
"\n",
|
||||
" # ========== 计算损失 ==========\n",
|
||||
" # 1) Ground-Truth CE Loss（可选）\n",
|
||||
" loss_mask_flat = loss_mask.view(-1)\n",
|
||||
" ce_loss = F.cross_entropy(\n",
|
||||
" student_logits.view(-1, student_logits.size(-1)),\n",
|
||||
" Y.view(-1),\n",
|
||||
" ignore_index=0,\n",
|
||||
" reduction='none'\n",
|
||||
" )\n",
|
||||
" ce_loss = torch.sum(ce_loss * loss_mask_flat) / loss_mask_flat.sum()\n",
|
||||
" if lm_config_student.use_moe:\n",
|
||||
" ce_loss += res.aux_loss\n",
|
||||
"\n",
|
||||
" # 2) Distillation Loss（可选）\n",
|
||||
" if teacher_model is not None:\n",
|
||||
" # 只在有效token位置做蒸馏\n",
|
||||
" distill_loss = distillation_loss_fn(\n",
|
||||
" student_logits.view(-1, student_logits.size(-1))[loss_mask_flat == 1],\n",
|
||||
" teacher_logits.view(-1, teacher_logits.size(-1))[loss_mask_flat == 1],\n",
|
||||
" temperature=temperature\n",
|
||||
" )\n",
|
||||
" else:\n",
|
||||
" distill_loss = torch.tensor(0.0, device=args.device)\n",
|
||||
"\n",
|
||||
" # 3) 总损失 = alpha * CE + (1-alpha) * Distill\n",
|
||||
" loss = alpha * ce_loss + (1 - alpha) * distill_loss\n",
|
||||
"\n",
|
||||
" scaler.scale(loss).backward()\n",
|
||||
"\n",
|
||||
" if (step + 1) % args.accumulation_steps == 0:\n",
|
||||
" scaler.unscale_(optimizer)\n",
|
||||
" torch.nn.utils.clip_grad_norm_(model.parameters(), args.grad_clip)\n",
|
||||
" scaler.step(optimizer)\n",
|
||||
" scaler.update()\n",
|
||||
" optimizer.zero_grad(set_to_none=True)\n",
|
||||
"\n",
|
||||
" if step % args.log_interval == 0:\n",
|
||||
" spend_time = time.time() - start_time\n",
|
||||
" print(\n",
|
||||
" 'Epoch:[{}/{}]({}/{}) loss:{:.4f} lr:{:.12f} epoch_Time:{}min:'.format(\n",
|
||||
" epoch,\n",
|
||||
" args.epochs - 1,\n",
|
||||
" step,\n",
|
||||
" iter_per_epoch,\n",
|
||||
" loss.item(),\n",
|
||||
" optimizer.param_groups[-1]['lr'],\n",
|
||||
" spend_time / (step + 1) * iter_per_epoch // 60 - spend_time // 60\n",
|
||||
" )\n",
|
||||
" )\n",
|
||||
"\n",
|
||||
" # 到达指定保存步数时，save as PyTorch\n",
|
||||
" # if (step + 1) % args.save_interval == 0 and (not ddp or dist.get_rank() == 0):\n",
|
||||
" # model.eval()\n",
|
||||
" # moe_path = '_moe' if lm_config_student.use_moe else ''\n",
|
||||
" # ckp = f'{args.save_dir}/full_dist_{lm_config_student.dim}{moe_path}.pth'\n",
|
||||
" # if isinstance(model, torch.nn.parallel.DistributedDataParallel):\n",
|
||||
" # state_dict = model.module.state_dict()\n",
|
||||
" # else:\n",
|
||||
" # state_dict = model.state_dict()\n",
|
||||
" # torch.save(state_dict, ckp)\n",
|
||||
" # model.train()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "b5369c12-e315-4a21-8f23-42dc0151909e",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"接下来，我们启动一个 Epoch 的训练进行观察."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 9,
|
||||
"id": "e73d59fc-677a-4795-8377-cdef06e8e1d9",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Epoch:[0/0](0/1) loss:0.3257 lr:0.000550000000 epoch_Time:0.0min:\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"iter_per_epoch = len(train_loader)\n",
|
||||
"for epoch in range(args.epochs):\n",
|
||||
" train_epoch(epoch)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 10,
|
||||
"id": "94cd3e5c-e37b-4b37-919f-7a1066b0f482",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"del model, teacher_model"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "e3ff1681-962d-4f4b-a75d-e4673978e5d3",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 参考资料\n",
|
||||
"\n",
|
||||
"- [大模型知识蒸馏概述](https://zhuanlan.zhihu.com/p/659943824)\n",
|
||||
"- [使用知识蒸馏将大模型能力克隆到小模型](https://zhuanlan.zhihu.com/p/691672620)\n",
|
||||
"- [理解知识蒸馏中的散度损失函数](https://deepseek.csdn.net/67ab1c3f79aaf67875cb9664.html)"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3 (ipykernel)",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.10.14"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 5
|
||||
}
|
||||
|
|
@ -0,0 +1,359 @@
|
|||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "2edfeb4e-03ea-406a-a345-6fe19d509c37",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# 8-Reason\n",
|
||||
"\n",
|
||||
"参数太小的模型直接通过冷启动SFT+GRPO几乎不可能获得任何推理效果，因此，使用冷启动 SFT + GRPO 训练方法对小模型推理能力的作用有限.因此，MiniMind 项目作者使用推理数据集对 MiniMind 系列模型进行黑盒蒸馏来训练推理模型.\n",
|
||||
"\n",
|
||||
"使用的推理数据格式:\n",
|
||||
"\n",
|
||||
"```\n",
|
||||
"{\n",
|
||||
" \"conversations\": [\n",
|
||||
" {\"role\": \"user\", \"content\": \"Q1?\"},\n",
|
||||
" {\"role\": \"assistant\", \"content\": \"<think>T1</think>\\n<answer>A1</answer>\"},\n",
|
||||
" {\"role\": \"user\", \"content\": \"Q2?\"},\n",
|
||||
" {\"role\": \"assistant\", \"content\": \"<think>T2</think>\\n<answer>A2</answer>\"}\n",
|
||||
" ]\n",
|
||||
"}\n",
|
||||
"```\n",
|
||||
"\n",
|
||||
"此笔记本的完整实现见主仓库 `/minimind/train_distill_reason.py`"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"id": "f7eb789f-18d9-45c5-acab-ca3167950911",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import os\n",
|
||||
"import platform\n",
|
||||
"import argparse\n",
|
||||
"import time\n",
|
||||
"import math\n",
|
||||
"import warnings\n",
|
||||
"\n",
|
||||
"import pandas as pd\n",
|
||||
"import torch\n",
|
||||
"import torch.nn.functional as F\n",
|
||||
"import torch.distributed as dist\n",
|
||||
"from contextlib import nullcontext\n",
|
||||
"\n",
|
||||
"from torch import optim, nn\n",
|
||||
"from torch.nn.parallel import DistributedDataParallel\n",
|
||||
"from torch.utils.data import DataLoader, DistributedSampler\n",
|
||||
"from transformers import AutoTokenizer, AutoModelForCausalLM\n",
|
||||
"from model.model import MiniMindLM\n",
|
||||
"from model.LMConfig import LMConfig\n",
|
||||
"from model.dataset import SFTDataset\n",
|
||||
"\n",
|
||||
"warnings.filterwarnings('ignore')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "24cefe29-fdf0-44c7-86fa-97206bc18161",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 可选参数设置\n",
|
||||
"\n",
|
||||
"首先，查看训练的可选参数，这些参数在实际使用时通过解析命令行进行导入，我们用 class 进行包装."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 2,
|
||||
"id": "22643ad2-fc7a-4efd-ac53-81c2bcc88cd1",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"class args:\n",
|
||||
" # out_dir: str = \"out\" # pytorch 格式权重文件保存位置 我们只展示训练过程 所以不使用\n",
|
||||
" epochs: int = 1 # 训练轮数\n",
|
||||
" batch_size: int = 2 # pretrain 数据集仅两个样本，设置 batch 为 2\n",
|
||||
" learning_rate: float = 5e-4 # 学习率\n",
|
||||
" device: str = 'cuda' if torch.cuda.is_available() else 'cpu'\n",
|
||||
" dtype: str = 'bfloat16' # 16 bit 浮点数：8 bit 指数 + 7 bit 尾数\n",
|
||||
" # use_wandb: bool = False # 是否使用 wandb 我们不使用\n",
|
||||
" wandb_project: str = 'MiniMind-Notebook'\n",
|
||||
" num_workers: int = 1 # 工作进程数\n",
|
||||
" # ddp：bool = False # 单机多卡\n",
|
||||
" accumulation_steps: int = 1 # 梯度累积步数\n",
|
||||
" grad_clip: float = 1.0 # 梯度剪裁\n",
|
||||
" warmup_iters: int = 0 # 学习率热启动\n",
|
||||
" log_interval: int = 1 # 每一步打印日志 仅用于观察\n",
|
||||
" # save_interval: int = 100 # checkpoint 保存点 我们不使用\n",
|
||||
" local_rank: int = 1 # device 设备号\n",
|
||||
" dim: int = 512 # 词嵌入维度 模型超参数\n",
|
||||
" n_layers: int = 2 # MiniMind Block 数量 模型超参数\n",
|
||||
" max_seq_len: int = 512 # 序列长度阈值\n",
|
||||
" use_moe: bool = False # 是否启用混合专家\n",
|
||||
" data_path: str = './toydata/r1_data.jsonl' # 数据集路径"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 3,
|
||||
"id": "d2dc3bf2-215f-4fc0-b7ce-6b4a09b0f2d9",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"查看工作设备 cuda\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"print(f'查看工作设备 {args.device}')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "82a5bce8-a578-4e56-9dd5-9cb43321da0c",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"接下来，我们对分词器、MiniMind 学生模型以及数据迭代器执行初始化."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"id": "db202500-e265-434e-b0a4-c352854de648",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def init_model(lm_config):\n",
|
||||
" tokenizer = AutoTokenizer.from_pretrained('../model/minimind_tokenizer')\n",
|
||||
" model = MiniMindLM(lm_config)\n",
|
||||
" moe_path = '_moe' if lm_config.use_moe else ''\n",
|
||||
" # 热启动\n",
|
||||
" # ckp = f'./out/rlhf_{lm_config.dim}{moe_path}.pth'\n",
|
||||
" # state_dict = torch.load(ckp, map_location=args.device)\n",
|
||||
" # model.load_state_dict(state_dict, strict=False)\n",
|
||||
" print(f'LLM总参数量：{sum(p.numel() for p in model.parameters() if p.requires_grad) / 1e6:.3f} 百万')\n",
|
||||
" model = model.to(args.device)\n",
|
||||
" return model, tokenizer"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 5,
|
||||
"id": "e1948aa4-381d-485a-9b19-e0e566e0b5da",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"LLM总参数量：8.915 百万\n",
|
||||
"模型位于设备：cuda:0, 词表长度：6400, DataLoader：<torch.utils.data.dataloader.DataLoader object at 0x000001DAD0FFD630>\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"lm_config = LMConfig(dim=args.dim, n_layers=args.n_layers, max_seq_len=args.max_seq_len, use_moe=args.use_moe)\n",
|
||||
"model, tokenizer = init_model(lm_config)\n",
|
||||
"\n",
|
||||
"train_ds = SFTDataset(args.data_path, tokenizer, max_length=lm_config.max_seq_len)\n",
|
||||
"\n",
|
||||
"train_loader = DataLoader(\n",
|
||||
" train_ds,\n",
|
||||
" batch_size=args.batch_size,\n",
|
||||
" pin_memory=True,\n",
|
||||
" drop_last=False,\n",
|
||||
" shuffle=False,\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"print(f'模型位于设备：{model.device}, 词表长度：{tokenizer.vocab_size}, DataLoader：{train_loader}')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "9be67e55-0d1b-4c5d-9304-3b65d5b81c4d",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 启动训练\n",
|
||||
"\n",
|
||||
"接下来，我们定义 MiniMind LoRA 微调所使用的优化器，损失函数和学习率调度，并进行一轮简单的训练."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 6,
|
||||
"id": "59084328-05e4-49d3-8459-54c549131511",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# 学习率调度方面 采用余弦退火学习率\n",
|
||||
"def get_lr(current_step, total_steps, lr):\n",
|
||||
" return lr / 10 + 0.5 * lr * (1 + math.cos(math.pi * current_step / total_steps))\n",
|
||||
"\n",
|
||||
"# 优化器方面 选择 AdamW 优化器 并在混精度场景下创建 scaler 进行梯度缩放避免数值下溢\n",
|
||||
"scaler = torch.cuda.amp.GradScaler(enabled=(args.dtype in ['float16', 'bfloat16']))\n",
|
||||
"# 优化学生模型参数\n",
|
||||
"optimizer = optim.AdamW(model.parameters(), lr=args.learning_rate)\n",
|
||||
"\n",
|
||||
"device_type = \"cuda\" if \"cuda\" in args.device else \"cpu\"\n",
|
||||
"ctx = nullcontext() if device_type == \"cpu\" else torch.cuda.amp.autocast() # 在 cuda 上启动混精度训练，否则空白上下文"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "7a241f42-32ec-4070-9fda-1cb39860089f",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"接下来，我们来看看训练函数.\n",
|
||||
"\n",
|
||||
"蒸馏思考数据集的训练过程与 SFT 类似，区别在于模型生成序列中，思考标签位置的预测错误惩罚被放大."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 7,
|
||||
"id": "e705c796-938e-4e0e-b813-3d6ffc68635e",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def train_epoch(epoch):\n",
|
||||
" # 思考标签占位符\n",
|
||||
" start_of_think_ids = tokenizer('<think>').input_ids\n",
|
||||
" end_of_think_ids = tokenizer('</think>').input_ids\n",
|
||||
" start_of_answer_ids = tokenizer('<answer>').input_ids\n",
|
||||
" end_of_answer_ids = tokenizer('</answer>').input_ids\n",
|
||||
" loss_fct = nn.CrossEntropyLoss(reduction='none') # ce 损失\n",
|
||||
" start_time = time.time()\n",
|
||||
" for step, (X, Y, loss_mask) in enumerate(train_loader):\n",
|
||||
" X = X.to(args.device)\n",
|
||||
" Y = Y.to(args.device)\n",
|
||||
" loss_mask = loss_mask.to(args.device)\n",
|
||||
" lr = get_lr(epoch * iter_per_epoch + step, args.epochs * iter_per_epoch, args.learning_rate)\n",
|
||||
" for param_group in optimizer.param_groups:\n",
|
||||
" param_group['lr'] = lr\n",
|
||||
"\n",
|
||||
" with ctx:\n",
|
||||
" res = model(X)\n",
|
||||
" loss = loss_fct(\n",
|
||||
" res.logits.view(-1, res.logits.size(-1)),\n",
|
||||
" Y.view(-1)\n",
|
||||
" ).view(Y.size())\n",
|
||||
" # 判断思考标签是否存在于 Y 中，并返回一个 bool 类型张量，指示思考标签在 Y 中的位置\n",
|
||||
" sp_ids = torch.isin(Y.view(-1),\n",
|
||||
" torch.tensor(start_of_think_ids + end_of_think_ids\n",
|
||||
" + start_of_answer_ids + end_of_answer_ids\n",
|
||||
" ).to(args.device))\n",
|
||||
" # 在 sp_ids 对应的位置增加额外的惩罚\n",
|
||||
" loss_mask = loss_mask.view(-1)\n",
|
||||
" loss_mask_sum = loss_mask.sum()\n",
|
||||
" loss_mask[sp_ids] = 10\n",
|
||||
" loss_mask = loss_mask.view(Y.size())\n",
|
||||
" loss = (loss * loss_mask).sum() / loss_mask_sum # 思考标签相应位置的惩罚被放大十倍\n",
|
||||
" loss += res.aux_loss # 负载均衡损失\n",
|
||||
" loss = loss / args.accumulation_steps\n",
|
||||
"\n",
|
||||
" scaler.scale(loss).backward()\n",
|
||||
"\n",
|
||||
" if (step + 1) % args.accumulation_steps == 0:\n",
|
||||
" scaler.unscale_(optimizer)\n",
|
||||
" torch.nn.utils.clip_grad_norm_(model.parameters(), args.grad_clip)\n",
|
||||
"\n",
|
||||
" scaler.step(optimizer)\n",
|
||||
" scaler.update()\n",
|
||||
"\n",
|
||||
" optimizer.zero_grad(set_to_none=True)\n",
|
||||
"\n",
|
||||
" if step % args.log_interval == 0:\n",
|
||||
" spend_time = time.time() - start_time\n",
|
||||
" print(\n",
|
||||
" 'Epoch:[{}/{}]({}/{}) loss:{:.3f} lr:{:.12f} epoch_Time:{}min:'.format(\n",
|
||||
" epoch + 1,\n",
|
||||
" args.epochs,\n",
|
||||
" step,\n",
|
||||
" iter_per_epoch,\n",
|
||||
" loss.item(),\n",
|
||||
" optimizer.param_groups[-1]['lr'],\n",
|
||||
" spend_time / (step + 1) * iter_per_epoch // 60 - spend_time // 60))\n",
|
||||
"\n",
|
||||
" # 到达指定步数后 save as torch\n",
|
||||
" # if (step + 1) % args.save_interval == 0 and (not ddp or dist.get_rank() == 0):\n",
|
||||
" # model.eval()\n",
|
||||
" # moe_path = '_moe' if lm_config.use_moe else ''\n",
|
||||
" # ckp = f'{args.save_dir}/reason_{lm_config.dim}{moe_path}.pth'\n",
|
||||
"\n",
|
||||
" # if isinstance(model, torch.nn.parallel.DistributedDataParallel):\n",
|
||||
" # state_dict = model.module.state_dict()\n",
|
||||
" # else:\n",
|
||||
" # state_dict = model.state_dict()\n",
|
||||
"\n",
|
||||
" # torch.save(state_dict, ckp)\n",
|
||||
" # model.train()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "a3b5aaf7-feac-42d9-adfe-9a583f26879d",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"接下来，我们启动一个 Epoch 的训练进行观察."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 8,
|
||||
"id": "f021611c-6291-4c50-acf0-9a24dde87e77",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Epoch:[1/1](0/1) loss:11.656 lr:0.000550000000 epoch_Time:0.0min:\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"iter_per_epoch = len(train_loader)\n",
|
||||
"for epoch in range(args.epochs):\n",
|
||||
" train_epoch(epoch)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 9,
|
||||
"id": "fba06e90-680e-47ff-99e3-49fd91d74dc9",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"del model"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3 (ipykernel)",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.10.14"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 5
|
||||
}
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 64 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 12 KiB |
|
|
@ -0,0 +1,61 @@
|
|||
from transformers import PretrainedConfig
|
||||
from typing import List
|
||||
|
||||
|
||||
class LMConfig(PretrainedConfig):
|
||||
model_type = "minimind"
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
dim: int = 512,
|
||||
n_layers: int = 1, # n_layers set to 1 to enable local practice
|
||||
n_heads: int = 8,
|
||||
n_kv_heads: int = 2,
|
||||
vocab_size: int = 6400,
|
||||
hidden_dim: int = None,
|
||||
multiple_of: int = 64,
|
||||
norm_eps: float = 1e-5,
|
||||
max_seq_len: int = 8192,
|
||||
rope_theta: int = 1e6,
|
||||
dropout: float = 0.0,
|
||||
flash_attn: bool = True,
|
||||
####################################################
|
||||
# Here are the specific configurations of MOE
|
||||
# When use_moe is false, the following is invalid
|
||||
####################################################
|
||||
use_moe: bool = False,
|
||||
####################################################
|
||||
num_experts_per_tok: int = 2,
|
||||
n_routed_experts: int = 4,
|
||||
n_shared_experts: bool = True,
|
||||
scoring_func: str = 'softmax',
|
||||
aux_loss_alpha: float = 0.1,
|
||||
seq_aux: bool = True,
|
||||
norm_topk_prob: bool = True,
|
||||
**kwargs,
|
||||
):
|
||||
self.dim = dim
|
||||
self.n_layers = n_layers
|
||||
self.n_heads = n_heads
|
||||
self.n_kv_heads = n_kv_heads
|
||||
self.vocab_size = vocab_size
|
||||
self.hidden_dim = hidden_dim
|
||||
self.multiple_of = multiple_of
|
||||
self.norm_eps = norm_eps
|
||||
self.max_seq_len = max_seq_len
|
||||
self.rope_theta = rope_theta
|
||||
self.dropout = dropout
|
||||
self.flash_attn = flash_attn
|
||||
####################################################
|
||||
# Here are the specific configurations of MOE
|
||||
# When use_moe is false, the following is invalid
|
||||
####################################################
|
||||
self.use_moe = use_moe
|
||||
self.num_experts_per_tok = num_experts_per_tok # 每个token选择的专家数量
|
||||
self.n_routed_experts = n_routed_experts # 总的专家数量
|
||||
self.n_shared_experts = n_shared_experts # 共享专家
|
||||
self.scoring_func = scoring_func # 评分函数，默认为'softmax'
|
||||
self.aux_loss_alpha = aux_loss_alpha # 辅助损失的alpha参数
|
||||
self.seq_aux = seq_aux # 是否在序列级别上计算辅助损失
|
||||
self.norm_topk_prob = norm_topk_prob # 是否标准化top-k概率
|
||||
super().__init__(**kwargs)
|
||||
|
|
@ -0,0 +1,200 @@
|
|||
import json
|
||||
import random
|
||||
import re
|
||||
|
||||
import pandas as pd
|
||||
import numpy as np
|
||||
from torch.utils.data import Dataset, DataLoader
|
||||
import torch
|
||||
from sklearn.model_selection import train_test_split
|
||||
import os
|
||||
import ast
|
||||
|
||||
os.environ["TOKENIZERS_PARALLELISM"] = "false"
|
||||
|
||||
|
||||
class PretrainDataset(Dataset):
|
||||
def __init__(self, data_path, tokenizer, max_length=512):
|
||||
super().__init__()
|
||||
self.tokenizer = tokenizer
|
||||
self.max_length = max_length
|
||||
self.samples = self.load_data(data_path)
|
||||
|
||||
def load_data(self, path):
|
||||
samples = []
|
||||
with open(path, 'r', encoding='utf-8') as f:
|
||||
for line_num, line in enumerate(f, 1):
|
||||
data = json.loads(line.strip())
|
||||
samples.append(data)
|
||||
return samples
|
||||
|
||||
def __len__(self):
|
||||
return len(self.samples)
|
||||
|
||||
def __getitem__(self, index):
|
||||
sample = self.samples[index]
|
||||
|
||||
# 构建输入文本
|
||||
text = f"{self.tokenizer.bos_token}{str(sample['text'])}{self.tokenizer.eos_token}"
|
||||
encoding = self.tokenizer(
|
||||
text,
|
||||
max_length=self.max_length,
|
||||
padding='max_length',
|
||||
truncation=True,
|
||||
return_tensors='pt'
|
||||
)
|
||||
input_ids = encoding.input_ids.squeeze()
|
||||
loss_mask = (input_ids != self.tokenizer.pad_token_id)
|
||||
|
||||
X = torch.tensor(input_ids[:-1], dtype=torch.long)
|
||||
Y = torch.tensor(input_ids[1:], dtype=torch.long)
|
||||
loss_mask = torch.tensor(loss_mask[1:], dtype=torch.long)
|
||||
return X, Y, loss_mask
|
||||
|
||||
|
||||
class SFTDataset(Dataset):
|
||||
def __init__(self, jsonl_path, tokenizer, max_length=1024):
|
||||
super().__init__()
|
||||
self.tokenizer = tokenizer
|
||||
self.max_length = max_length
|
||||
self.samples = self.load_data(jsonl_path)
|
||||
self.bos_id = tokenizer('<s>assistant\n', add_special_tokens=False).input_ids
|
||||
self.eos_id = tokenizer('</s>\n', add_special_tokens=False).input_ids
|
||||
|
||||
def __len__(self):
|
||||
return len(self.samples)
|
||||
|
||||
def load_data(self, path):
|
||||
samples = []
|
||||
with open(path, 'r', encoding='utf-8') as f:
|
||||
for line_num, line in enumerate(f, 1):
|
||||
data = json.loads(line.strip())
|
||||
samples.append(data)
|
||||
return samples
|
||||
|
||||
def _create_chat_prompt(self, conversations):
|
||||
"""构建符合ChatML格式的对话"""
|
||||
messages = []
|
||||
for i, turn in enumerate(conversations):
|
||||
role = 'user' if i % 2 == 0 else 'assistant'
|
||||
messages.append({"role": role, "content": turn['content']})
|
||||
return self.tokenizer.apply_chat_template(
|
||||
messages,
|
||||
tokenize=False,
|
||||
add_generation_prompt=False
|
||||
)
|
||||
|
||||
def _generate_loss_mask(self, input_ids):
|
||||
loss_mask = [0] * len(input_ids)
|
||||
i = 0
|
||||
while i < len(input_ids):
|
||||
if input_ids[i:i + len(self.bos_id)] == self.bos_id:
|
||||
start = i + len(self.bos_id)
|
||||
end = start
|
||||
while end < len(input_ids):
|
||||
if input_ids[end:end + len(self.eos_id)] == self.eos_id:
|
||||
break
|
||||
end += 1
|
||||
for j in range(start + 1, min(end + len(self.eos_id) + 1, self.max_length)):
|
||||
loss_mask[j] = 1
|
||||
i = end + len(self.eos_id) if end < len(input_ids) else len(input_ids)
|
||||
else:
|
||||
i += 1
|
||||
return loss_mask
|
||||
|
||||
def __getitem__(self, index):
|
||||
sample = self.samples[index]
|
||||
# 构建对话提示
|
||||
prompt = self._create_chat_prompt(sample['conversations'])
|
||||
input_ids = self.tokenizer(prompt).input_ids[:self.max_length]
|
||||
input_ids += [self.tokenizer.pad_token_id] * (self.max_length - len(input_ids))
|
||||
|
||||
# 生成动态损失掩码
|
||||
loss_mask = self._generate_loss_mask(input_ids)
|
||||
|
||||
# 构建训练数据
|
||||
X = torch.tensor(input_ids[:-1], dtype=torch.long)
|
||||
Y = torch.tensor(input_ids[1:], dtype=torch.long)
|
||||
loss_mask = torch.tensor(loss_mask[1:], dtype=torch.long) # 对齐预测位置
|
||||
|
||||
return X, Y, loss_mask
|
||||
|
||||
|
||||
class DPODataset(Dataset):
|
||||
def __init__(self, file_path, tokenizer, max_length=4096):
|
||||
super().__init__()
|
||||
self.tokenizer = tokenizer
|
||||
self.max_length = max_length
|
||||
self.padding = tokenizer.pad_token_id if tokenizer.pad_token_id is not None else 0
|
||||
self.bos_id = tokenizer('<s>assistant\n', add_special_tokens=False).input_ids
|
||||
self.eos_id = tokenizer('</s>\n', add_special_tokens=False).input_ids
|
||||
with open(file_path, 'r', encoding='utf-8') as f:
|
||||
self.data = []
|
||||
for line in f:
|
||||
line = line.strip()
|
||||
obj = json.loads(line)
|
||||
self.data.append(obj)
|
||||
|
||||
def __len__(self):
|
||||
return len(self.data)
|
||||
|
||||
def __getitem__(self, index):
|
||||
item = self.data[index]
|
||||
chosen = item['chosen'] # 是一个 list，里面包含若干 {role, content}
|
||||
rejected = item['rejected'] # 同上
|
||||
chosen_prompt = self.tokenizer.apply_chat_template(
|
||||
chosen, tokenize=False, add_generation_prompt=False
|
||||
)
|
||||
|
||||
rejected_prompt = self.tokenizer.apply_chat_template(
|
||||
rejected, tokenize=False, add_generation_prompt=False
|
||||
)
|
||||
chosen_encoding = self.tokenizer(
|
||||
chosen_prompt, truncation=True, max_length=self.max_length, padding='max_length'
|
||||
)
|
||||
rejected_encoding = self.tokenizer(
|
||||
rejected_prompt, truncation=True, max_length=self.max_length, padding='max_length'
|
||||
)
|
||||
|
||||
chosen_input_ids = chosen_encoding['input_ids']
|
||||
chosen_loss_mask = self._generate_loss_mask(chosen_input_ids)
|
||||
|
||||
rejected_input_ids = rejected_encoding['input_ids']
|
||||
rejected_loss_mask = self._generate_loss_mask(rejected_input_ids)
|
||||
x_chosen = torch.tensor(chosen_input_ids[:-1], dtype=torch.long)
|
||||
y_chosen = torch.tensor(chosen_input_ids[1:], dtype=torch.long)
|
||||
mask_chosen = torch.tensor(chosen_loss_mask[1:], dtype=torch.long)
|
||||
x_rejected = torch.tensor(rejected_input_ids[:-1], dtype=torch.long)
|
||||
y_rejected = torch.tensor(rejected_input_ids[1:], dtype=torch.long)
|
||||
mask_rejected = torch.tensor(rejected_loss_mask[1:], dtype=torch.long)
|
||||
|
||||
return {
|
||||
'x_chosen': x_chosen,
|
||||
'y_chosen': y_chosen,
|
||||
'mask_chosen': mask_chosen,
|
||||
'x_rejected': x_rejected,
|
||||
'y_rejected': y_rejected,
|
||||
'mask_rejected': mask_rejected
|
||||
}
|
||||
|
||||
def _generate_loss_mask(self, input_ids):
|
||||
loss_mask = [0] * len(input_ids)
|
||||
i = 0
|
||||
while i < len(input_ids):
|
||||
if input_ids[i:i + len(self.bos_id)] == self.bos_id:
|
||||
start = i + len(self.bos_id)
|
||||
end = start
|
||||
while end < len(input_ids):
|
||||
if input_ids[end:end + len(self.eos_id)] == self.eos_id:
|
||||
break
|
||||
end += 1
|
||||
for j in range(start + 1, min(end + len(self.eos_id) + 1, self.max_length)):
|
||||
loss_mask[j] = 1
|
||||
i = end + len(self.eos_id) if end < len(input_ids) else len(input_ids)
|
||||
else:
|
||||
i += 1
|
||||
return loss_mask
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
pass
|
||||
|
|
@ -0,0 +1,376 @@
|
|||
import math
|
||||
import struct
|
||||
import inspect
|
||||
import time
|
||||
|
||||
from .LMConfig import LMConfig
|
||||
from typing import Any, Optional, Tuple, List
|
||||
import numpy as np
|
||||
import torch
|
||||
import torch.nn.functional as F
|
||||
from torch import nn
|
||||
from transformers import PreTrainedModel
|
||||
from transformers.modeling_outputs import CausalLMOutputWithPast
|
||||
|
||||
|
||||
class RMSNorm(torch.nn.Module):
|
||||
def __init__(self, dim: int, eps: float):
|
||||
super().__init__()
|
||||
self.eps = eps
|
||||
self.weight = nn.Parameter(torch.ones(dim))
|
||||
|
||||
def forward(self, x):
|
||||
return self.weight * (x.float() * torch.rsqrt(x.pow(2).mean(-1, keepdim=True) + self.eps)).type_as(x)
|
||||
|
||||
|
||||
def precompute_pos_cis(dim: int, end: int = int(32 * 1024), theta: float = 1e6):
|
||||
freqs = 1.0 / (theta ** (torch.arange(0, dim, 2)[: (dim // 2)].float() / dim))
|
||||
t = torch.arange(end, device=freqs.device) # type: ignore
|
||||
freqs = torch.outer(t, freqs).float() # type: ignore
|
||||
pos_cis = torch.polar(torch.ones_like(freqs), freqs) # complex64
|
||||
return pos_cis
|
||||
|
||||
|
||||
def apply_rotary_emb(xq, xk, pos_cis):
|
||||
def unite_shape(pos_cis, x):
|
||||
ndim = x.ndim
|
||||
assert 0 <= 1 < ndim
|
||||
assert pos_cis.shape == (x.shape[1], x.shape[-1])
|
||||
shape = [d if i == 1 or i == ndim - 1 else 1 for i, d in enumerate(x.shape)]
|
||||
return pos_cis.view(*shape)
|
||||
|
||||
xq_ = torch.view_as_complex(xq.float().reshape(*xq.shape[:-1], -1, 2))
|
||||
xk_ = torch.view_as_complex(xk.float().reshape(*xk.shape[:-1], -1, 2))
|
||||
pos_cis = unite_shape(pos_cis, xq_)
|
||||
xq_out = torch.view_as_real(xq_ * pos_cis).flatten(3)
|
||||
xk_out = torch.view_as_real(xk_ * pos_cis).flatten(3)
|
||||
return xq_out.type_as(xq), xk_out.type_as(xk)
|
||||
|
||||
|
||||
def repeat_kv(x: torch.Tensor, n_rep: int) -> torch.Tensor:
|
||||
"""torch.repeat_interleave(x, dim=2, repeats=n_rep)"""
|
||||
bs, slen, n_kv_heads, head_dim = x.shape
|
||||
if n_rep == 1:
|
||||
return x
|
||||
return (
|
||||
x[:, :, :, None, :]
|
||||
.expand(bs, slen, n_kv_heads, n_rep, head_dim)
|
||||
.reshape(bs, slen, n_kv_heads * n_rep, head_dim)
|
||||
)
|
||||
|
||||
|
||||
class Attention(nn.Module):
|
||||
def __init__(self, args: LMConfig):
|
||||
super().__init__()
|
||||
self.n_kv_heads = args.n_heads if args.n_kv_heads is None else args.n_kv_heads
|
||||
assert args.n_heads % self.n_kv_heads == 0
|
||||
self.n_local_heads = args.n_heads
|
||||
self.n_local_kv_heads = self.n_kv_heads
|
||||
self.n_rep = self.n_local_heads // self.n_local_kv_heads
|
||||
self.head_dim = args.dim // args.n_heads
|
||||
self.wq = nn.Linear(args.dim, args.n_heads * self.head_dim, bias=False)
|
||||
self.wk = nn.Linear(args.dim, self.n_kv_heads * self.head_dim, bias=False)
|
||||
self.wv = nn.Linear(args.dim, self.n_kv_heads * self.head_dim, bias=False)
|
||||
self.wo = nn.Linear(args.n_heads * self.head_dim, args.dim, bias=False)
|
||||
self.attn_dropout = nn.Dropout(args.dropout)
|
||||
self.resid_dropout = nn.Dropout(args.dropout)
|
||||
self.dropout = args.dropout
|
||||
self.flash = hasattr(torch.nn.functional, 'scaled_dot_product_attention') and args.flash_attn
|
||||
# print("WARNING: using slow attention. Flash Attention requires PyTorch >= 2.0")
|
||||
mask = torch.full((1, 1, args.max_seq_len, args.max_seq_len), float("-inf"))
|
||||
mask = torch.triu(mask, diagonal=1)
|
||||
self.register_buffer("mask", mask, persistent=False)
|
||||
|
||||
def forward(self,
|
||||
x: torch.Tensor,
|
||||
pos_cis: torch.Tensor,
|
||||
past_key_value: Optional[Tuple[torch.Tensor, torch.Tensor]] = None,
|
||||
use_cache=False):
|
||||
bsz, seq_len, _ = x.shape
|
||||
xq, xk, xv = self.wq(x), self.wk(x), self.wv(x)
|
||||
xq = xq.view(bsz, seq_len, self.n_local_heads, self.head_dim)
|
||||
xk = xk.view(bsz, seq_len, self.n_local_kv_heads, self.head_dim)
|
||||
xv = xv.view(bsz, seq_len, self.n_local_kv_heads, self.head_dim)
|
||||
|
||||
xq, xk = apply_rotary_emb(xq, xk, pos_cis)
|
||||
# kv_cache实现
|
||||
if past_key_value is not None:
|
||||
xk = torch.cat([past_key_value[0], xk], dim=1)
|
||||
xv = torch.cat([past_key_value[1], xv], dim=1)
|
||||
past_kv = (xk, xv) if use_cache else None
|
||||
|
||||
xq, xk, xv = (
|
||||
xq.transpose(1, 2),
|
||||
repeat_kv(xk, self.n_rep).transpose(1, 2),
|
||||
repeat_kv(xv, self.n_rep).transpose(1, 2)
|
||||
)
|
||||
if self.flash and seq_len != 1:
|
||||
dropout_p = self.dropout if self.training else 0.0
|
||||
output = F.scaled_dot_product_attention(
|
||||
xq, xk, xv,
|
||||
attn_mask=None,
|
||||
dropout_p=dropout_p,
|
||||
is_causal=True
|
||||
)
|
||||
else:
|
||||
scores = (xq @ xk.transpose(-2, -1)) / math.sqrt(self.head_dim)
|
||||
scores += self.mask[:, :, :seq_len, :seq_len]
|
||||
scores = F.softmax(scores.float(), dim=-1).type_as(xq)
|
||||
scores = self.attn_dropout(scores)
|
||||
output = scores @ xv
|
||||
|
||||
output = output.transpose(1, 2).reshape(bsz, seq_len, -1)
|
||||
output = self.resid_dropout(self.wo(output))
|
||||
return output, past_kv
|
||||
|
||||
|
||||
class FeedForward(nn.Module):
|
||||
def __init__(self, config: LMConfig):
|
||||
super().__init__()
|
||||
if config.hidden_dim is None:
|
||||
hidden_dim = 4 * config.dim
|
||||
hidden_dim = int(2 * hidden_dim / 3)
|
||||
config.hidden_dim = config.multiple_of * ((hidden_dim + config.multiple_of - 1) // config.multiple_of)
|
||||
self.w1 = nn.Linear(config.dim, config.hidden_dim, bias=False)
|
||||
self.w2 = nn.Linear(config.hidden_dim, config.dim, bias=False)
|
||||
self.w3 = nn.Linear(config.dim, config.hidden_dim, bias=False)
|
||||
self.dropout = nn.Dropout(config.dropout)
|
||||
|
||||
def forward(self, x):
|
||||
return self.dropout(self.w2(F.silu(self.w1(x)) * self.w3(x)))
|
||||
|
||||
|
||||
class MoEGate(nn.Module):
|
||||
def __init__(self, config: LMConfig):
|
||||
super().__init__()
|
||||
self.config = config
|
||||
self.top_k = config.num_experts_per_tok
|
||||
self.n_routed_experts = config.n_routed_experts
|
||||
|
||||
self.scoring_func = config.scoring_func
|
||||
self.alpha = config.aux_loss_alpha
|
||||
self.seq_aux = config.seq_aux
|
||||
|
||||
self.norm_topk_prob = config.norm_topk_prob
|
||||
self.gating_dim = config.dim
|
||||
self.weight = nn.Parameter(torch.empty((self.n_routed_experts, self.gating_dim)))
|
||||
self.reset_parameters()
|
||||
|
||||
def reset_parameters(self) -> None:
|
||||
import torch.nn.init as init
|
||||
init.kaiming_uniform_(self.weight, a=math.sqrt(5))
|
||||
|
||||
def forward(self, hidden_states):
|
||||
bsz, seq_len, h = hidden_states.shape
|
||||
hidden_states = hidden_states.view(-1, h)
|
||||
logits = F.linear(hidden_states, self.weight, None)
|
||||
if self.scoring_func == 'softmax':
|
||||
scores = logits.softmax(dim=-1)
|
||||
else:
|
||||
raise NotImplementedError(f'insupportable scoring function for MoE gating: {self.scoring_func}')
|
||||
|
||||
topk_weight, topk_idx = torch.topk(scores, k=self.top_k, dim=-1, sorted=False)
|
||||
|
||||
if self.top_k > 1 and self.norm_topk_prob:
|
||||
denominator = topk_weight.sum(dim=-1, keepdim=True) + 1e-20
|
||||
topk_weight = topk_weight / denominator
|
||||
|
||||
if self.training and self.alpha > 0.0:
|
||||
scores_for_aux = scores
|
||||
aux_topk = self.top_k
|
||||
topk_idx_for_aux_loss = topk_idx.view(bsz, -1)
|
||||
if self.seq_aux:
|
||||
scores_for_seq_aux = scores_for_aux.view(bsz, seq_len, -1)
|
||||
ce = torch.zeros(bsz, self.n_routed_experts, device=hidden_states.device)
|
||||
ce.scatter_add_(1, topk_idx_for_aux_loss,
|
||||
torch.ones(bsz, seq_len * aux_topk, device=hidden_states.device)).div_(
|
||||
seq_len * aux_topk / self.n_routed_experts)
|
||||
aux_loss = (ce * scores_for_seq_aux.mean(dim=1)).sum(dim=1).mean() * self.alpha
|
||||
else:
|
||||
mask_ce = F.one_hot(topk_idx_for_aux_loss.view(-1), num_classes=self.n_routed_experts)
|
||||
ce = mask_ce.float().mean(0)
|
||||
Pi = scores_for_aux.mean(0)
|
||||
fi = ce * self.n_routed_experts
|
||||
aux_loss = (Pi * fi).sum() * self.alpha
|
||||
else:
|
||||
aux_loss = 0
|
||||
return topk_idx, topk_weight, aux_loss
|
||||
|
||||
|
||||
class MOEFeedForward(nn.Module):
|
||||
def __init__(self, config: LMConfig):
|
||||
super().__init__()
|
||||
self.config = config
|
||||
self.experts = nn.ModuleList([
|
||||
FeedForward(config)
|
||||
for _ in range(config.n_routed_experts)
|
||||
])
|
||||
self.gate = MoEGate(config)
|
||||
if config.n_shared_experts is not None:
|
||||
self.shared_experts = FeedForward(config)
|
||||
|
||||
def forward(self, x):
|
||||
identity = x
|
||||
orig_shape = x.shape
|
||||
bsz, seq_len, _ = x.shape
|
||||
# 使用门控机制选择专家
|
||||
topk_idx, topk_weight, aux_loss = self.gate(x)
|
||||
x = x.view(-1, x.shape[-1])
|
||||
flat_topk_idx = topk_idx.view(-1)
|
||||
if self.training:
|
||||
# 训练模式下，重复输入数据
|
||||
x = x.repeat_interleave(self.config.num_experts_per_tok, dim=0)
|
||||
y = torch.empty_like(x, dtype=torch.float16)
|
||||
for i, expert in enumerate(self.experts):
|
||||
y[flat_topk_idx == i] = expert(x[flat_topk_idx == i]).to(y.dtype) # 确保类型一致
|
||||
y = (y.view(*topk_weight.shape, -1) * topk_weight.unsqueeze(-1)).sum(dim=1)
|
||||
y = y.view(*orig_shape)
|
||||
else:
|
||||
# 推理模式下，只选择最优专家
|
||||
y = self.moe_infer(x, flat_topk_idx, topk_weight.view(-1, 1)).view(*orig_shape)
|
||||
if self.config.n_shared_experts is not None:
|
||||
y = y + self.shared_experts(identity)
|
||||
self.aux_loss = aux_loss
|
||||
return y
|
||||
|
||||
@torch.no_grad()
|
||||
def moe_infer(self, x, flat_expert_indices, flat_expert_weights):
|
||||
expert_cache = torch.zeros_like(x)
|
||||
idxs = flat_expert_indices.argsort()
|
||||
tokens_per_expert = flat_expert_indices.bincount().cpu().numpy().cumsum(0)
|
||||
token_idxs = idxs // self.config.num_experts_per_tok
|
||||
# 例如当tokens_per_expert=[6, 15, 20, 26, 33, 38, 46, 52]
|
||||
# 当token_idxs=[3, 7, 19, 21, 24, 25, 4, 5, 6, 10, 11, 12...]
|
||||
# 意味着当token_idxs[:6] -> [3, 7, 19, 21, 24, 25, 4]位置的token都由专家0处理，token_idxs[6:15]位置的token都由专家1处理......
|
||||
for i, end_idx in enumerate(tokens_per_expert):
|
||||
start_idx = 0 if i == 0 else tokens_per_expert[i - 1]
|
||||
if start_idx == end_idx:
|
||||
continue
|
||||
expert = self.experts[i]
|
||||
exp_token_idx = token_idxs[start_idx:end_idx]
|
||||
expert_tokens = x[exp_token_idx]
|
||||
expert_out = expert(expert_tokens).to(expert_cache.dtype)
|
||||
expert_out.mul_(flat_expert_weights[idxs[start_idx:end_idx]])
|
||||
# 使用 scatter_add_ 进行 sum 操作
|
||||
expert_cache.scatter_add_(0, exp_token_idx.view(-1, 1).repeat(1, x.shape[-1]), expert_out)
|
||||
|
||||
return expert_cache
|
||||
|
||||
|
||||
class MiniMindBlock(nn.Module):
|
||||
def __init__(self, layer_id: int, config: LMConfig):
|
||||
super().__init__()
|
||||
self.n_heads = config.n_heads
|
||||
self.dim = config.dim
|
||||
self.head_dim = config.dim // config.n_heads
|
||||
self.attention = Attention(config)
|
||||
|
||||
self.layer_id = layer_id
|
||||
self.attention_norm = RMSNorm(config.dim, eps=config.norm_eps)
|
||||
self.ffn_norm = RMSNorm(config.dim, eps=config.norm_eps)
|
||||
self.feed_forward = FeedForward(config) if not config.use_moe else MOEFeedForward(config)
|
||||
|
||||
def forward(self, x, pos_cis, past_key_value=None, use_cache=False):
|
||||
h_attn, past_kv = self.attention(
|
||||
self.attention_norm(x),
|
||||
pos_cis,
|
||||
past_key_value=past_key_value,
|
||||
use_cache=use_cache
|
||||
)
|
||||
h = x + h_attn
|
||||
out = h + self.feed_forward(self.ffn_norm(h))
|
||||
return out, past_kv
|
||||
|
||||
|
||||
class MiniMindLM(PreTrainedModel):
|
||||
config_class = LMConfig
|
||||
|
||||
def __init__(self, params: LMConfig = None):
|
||||
self.params = params or LMConfig()
|
||||
super().__init__(self.params)
|
||||
self.vocab_size, self.n_layers = params.vocab_size, params.n_layers
|
||||
self.tok_embeddings = nn.Embedding(params.vocab_size, params.dim)
|
||||
self.dropout = nn.Dropout(params.dropout)
|
||||
self.layers = nn.ModuleList([MiniMindBlock(l, params) for l in range(self.n_layers)])
|
||||
self.norm = RMSNorm(params.dim, eps=params.norm_eps)
|
||||
self.output = nn.Linear(params.dim, params.vocab_size, bias=False)
|
||||
self.tok_embeddings.weight = self.output.weight
|
||||
self.register_buffer("pos_cis",
|
||||
precompute_pos_cis(dim=params.dim // params.n_heads, theta=params.rope_theta),
|
||||
persistent=False)
|
||||
self.OUT = CausalLMOutputWithPast()
|
||||
|
||||
def forward(self,
|
||||
input_ids: Optional[torch.Tensor] = None,
|
||||
past_key_values: Optional[List[Tuple[torch.Tensor, torch.Tensor]]] = None,
|
||||
use_cache: bool = False,
|
||||
**args):
|
||||
past_key_values = past_key_values or [None] * len(self.layers)
|
||||
start_pos = args.get('start_pos', 0)
|
||||
h = self.dropout(self.tok_embeddings(input_ids))
|
||||
pos_cis = self.pos_cis[start_pos:start_pos + input_ids.size(1)]
|
||||
past_kvs = []
|
||||
for l, layer in enumerate(self.layers):
|
||||
h, past_kv = layer(
|
||||
h, pos_cis,
|
||||
past_key_value=past_key_values[l],
|
||||
use_cache=use_cache
|
||||
)
|
||||
past_kvs.append(past_kv)
|
||||
logits = self.output(self.norm(h))
|
||||
aux_loss = sum(l.feed_forward.aux_loss for l in self.layers if isinstance(l.feed_forward, MOEFeedForward))
|
||||
self.OUT.__setitem__('logits', logits)
|
||||
self.OUT.__setitem__('aux_loss', aux_loss)
|
||||
self.OUT.__setitem__('past_key_values', past_kvs)
|
||||
return self.OUT
|
||||
|
||||
@torch.inference_mode()
|
||||
def generate(self, input_ids, eos_token_id=2, max_new_tokens=1024, temperature=0.75, top_p=0.90,
|
||||
stream=False, rp=1., use_cache=True, pad_token_id=0, **args):
|
||||
# 流式生成
|
||||
if stream:
|
||||
return self._stream(input_ids, eos_token_id, max_new_tokens, temperature, top_p, rp, use_cache, **args)
|
||||
|
||||
# 直接生成
|
||||
generated = []
|
||||
for i in range(input_ids.size(0)):
|
||||
non_pad = input_ids[i][input_ids[i] != pad_token_id].unsqueeze(0)
|
||||
out = self._stream(non_pad, eos_token_id, max_new_tokens, temperature, top_p, rp, use_cache, **args)
|
||||
tokens_list = [tokens[:, -1:] for tokens in out]
|
||||
gen = torch.cat(tokens_list, dim=-1) if tokens_list else non_pad
|
||||
full_sequence = torch.cat([non_pad, gen], dim=-1)
|
||||
generated.append(full_sequence)
|
||||
max_length = max(seq.size(1) for seq in generated)
|
||||
generated = [
|
||||
torch.cat(
|
||||
[seq, torch.full((1, max_length - seq.size(1)), pad_token_id, dtype=seq.dtype, device=seq.device)],
|
||||
dim=-1)
|
||||
for seq in generated
|
||||
]
|
||||
return torch.cat(generated, dim=0)
|
||||
|
||||
def _stream(self, input_ids, eos_token_id, max_new_tokens, temperature, top_p, rp, use_cache, **args):
|
||||
start, first_seq, past_kvs = input_ids.shape[1], True, None
|
||||
while input_ids.shape[1] < max_new_tokens - 1:
|
||||
if first_seq or not use_cache:
|
||||
out, first_seq = self(input_ids, past_key_values=past_kvs, use_cache=use_cache, **args), False
|
||||
else:
|
||||
out = self(input_ids[:, -1:], past_key_values=past_kvs, use_cache=use_cache,
|
||||
start_pos=input_ids.shape[1] - 1, **args)
|
||||
logits, past_kvs = out.logits[:, -1, :], out.past_key_values
|
||||
logits[:, list(set(input_ids.tolist()[0]))] /= rp
|
||||
logits /= (temperature + 1e-9)
|
||||
if top_p is not None and top_p < 1.0:
|
||||
sorted_logits, sorted_indices = torch.sort(logits, descending=True, dim=-1)
|
||||
sorted_probs = F.softmax(sorted_logits, dim=-1)
|
||||
cumulative_probs = torch.cumsum(sorted_probs, dim=-1)
|
||||
sorted_indices_to_remove = cumulative_probs > top_p
|
||||
sorted_indices_to_remove[:, 1:] = sorted_indices_to_remove[:, :-1].clone()
|
||||
sorted_indices_to_remove[:, 0] = False
|
||||
indices_to_remove = sorted_indices_to_remove.scatter(1, sorted_indices, sorted_indices_to_remove)
|
||||
logits[indices_to_remove] = -float('Inf')
|
||||
input_ids_next = torch.multinomial(F.softmax(logits, dim=-1), num_samples=1)
|
||||
input_ids = torch.cat((input_ids, input_ids_next), dim=1)
|
||||
yield input_ids[:, start:]
|
||||
if input_ids_next.item() == eos_token_id:
|
||||
break
|
||||
|
|
@ -0,0 +1,2 @@
|
|||
{"chosen":[{"content":"你好吗？","role":"user"},{"content":"我很好，谢谢！你呢？","role":"assistant"},{"content":"今天过得怎么样？","role":"user"},{"content":"挺好的，去跑步了，心情不错。","role":"assistant"}],"rejected":[{"content":"你好吗？","role":"user"},{"content":"不好，我很累。","role":"assistant"},{"content":"你喜欢什么运动？","role":"user"},{"content":"我不喜欢运动，没兴趣。","role":"assistant"}]}
|
||||
{"chosen":[{"content":"你喜欢旅行吗？","role":"user"},{"content":"我喜欢，尤其是去海边。","role":"assistant"},{"content":"你去过哪些地方？","role":"user"},{"content":"我去过欧洲、美国和日本，都是很棒的经历。","role":"assistant"}],"rejected":[{"content":"你喜欢旅行吗？","role":"user"},{"content":"不太喜欢，觉得太麻烦。","role":"assistant"},{"content":"你去过哪些地方？","role":"user"},{"content":"没去过太多地方，主要是待在家里。","role":"assistant"}]}
|
||||
|
|
@ -0,0 +1,2 @@
|
|||
{"conversations": [{"role": "user", "content": "hi"}, {"role": "assistant", "content": "Hello! I am MiniMind, an AI assistant developed by Jingyao Gong. How can I assist you today?"}]}
|
||||
{"conversations": [{"role": "user", "content": "hello"}, {"role": "assistant", "content": "Hello! I am MiniMind, an AI assistant developed by Jingyao Gong. How can I assist you today?"}]}
|
||||
|
|
@ -0,0 +1,2 @@
|
|||
{"text": "LLM首先要学习的并非直接与人交流，而是让网络参数中充满知识的墨水，“墨水” 理论上喝的越饱越好，产生大量的对世界的知识积累。 预训练就是让Model先埋头苦学大量基本的知识，例如从Wiki百科、新闻、书籍整理大规模的高质量训练数据。 这个过程是“无监督”的，即人类不需要在过程中做任何“有监督”的校正，而是由模型自己从大量文本中总结规律学习知识点。 模型此阶段目的只有一个：学会词语接龙。例如我们输入“秦始皇”四个字，它可以接龙“是中国的第一位皇帝”。"}
|
||||
{"text": "经过预训练，LLM此时已经掌握了大量知识，然而此时它只会无脑地词语接龙，还不会与人聊天。 SFT阶段就需要把半成品LLM施加一个自定义的聊天模板进行微调。 例如模型遇到这样的模板【问题->回答，问题->回答】后不再无脑接龙，而是意识到这是一段完整的对话结束。 称这个过程为指令微调，就如同让已经学富五车的「牛顿」先生适应21世纪智能手机的聊天习惯，学习屏幕左侧是对方消息，右侧是本人消息这个规律。 在训练时，MiniMind的指令和回答长度被截断在512，是为了节省显存空间。就像我们学习时，会先从短的文章开始，当学会写作200字作文后，800字文章也可以手到擒来。 在需要长度拓展时，只需要准备少量的2k/4k/8k长度对话数据进行进一步微调即可（此时最好配合RoPE-NTK的基准差值）。"}
|
||||
|
|
@ -0,0 +1,2 @@
|
|||
{"conversations": [{"role": "user", "content": "请用一段话描述阿里巴巴集团的企业文化。"}, {"role": "assistant", "content": "<think>\n嗯，用户让我用一段话描述阿里巴巴集团的企业文化。首先，我需要明确阿里巴巴集团的平均文化是什么。企业文化的定义是企业内部发生、指导管理方向的准则和规范性的精神或规范。\n\n阿里巴巴集团在2000年代中期开始发展，之前也有类似的概念，比如微软的人工智能理念。所以，我可以从这个发展中来寻找关键词和结构。\n\n接下来，我要考虑用户可能的意图。用户可能想了解阿里巴巴的企业文化，或者需要用于某个报告、宣传材料或者内部交流中。因此，我需要确保描述既全面又有深度，涵盖战略、创新、效率等各个方面。\n\n我还需要用简洁的语言来表达，避免过于复杂的词汇，让段落易于理解。同时，要注意逻辑连贯，段落之间要有过渡，让读者能够顺畅地跟随观点。\n\n可能还需要引用一些关键点，比如战略协作、用户体验、创新、效率、数字化转型、客户支持、全球化等，这些都是阿里巴巴成功的关键点，能够帮助用户全面了解企业文化。\n\n最后，检查语法和结构是否正确，确保专业性和准确性，这样用户可以直接使用这段文字，或者稍作修改，就能得到满意的效果。\n</think>\n<answer>\n阿里巴巴集团的企业文化以战略协作为核心，倡导员工之间的资源整合与创新思维的碰撞，旨在实现企业价值的提升和持续发展。阿里巴巴将其视为一个协同共生的整体，致力于通过技术创新、用户体验优化和数字化转型，致力于为客户创造最大价值。企业内外部力量协同作战，推动企业从单纯的业务拓展延伸至价值创造和社会服务，并在全球范围内进行全方位的数字化转型，以满足多样化、个性化、高端化的客户需求。阿里巴巴集团ix platform的建立旨在帮助员工实现高效协作，激发创新精神，吸引更多优秀人才加入，共同推动企业不断向前发展。\n</answer>"}]}
|
||||
{"conversations": [{"role": "user", "content": "你好，我是张华，今年15岁，是北京四中的学生。我喜欢编程和篮球，我梦想成为一名软件工程师。很高兴认识你！"}, {"role": "assistant", "content": "<think>\n好，用户是张华，15岁，来自北京四中，喜欢编程和篮球，梦想成为软件工程师。我需要回应他的兴趣，强调他的优势，比如编程基础和对篮球的热爱，有针对性地引导他规划未来。可以建议他关注 Font Awesome，展示他的技术背景，同时鼓励他扩展兴趣，并辅以鼓励自己。\n</think>\n<answer>\n你好！很高兴认识你。既然你对编程和篮球都很感兴趣，结合你的梦想成为一名软件工程师，这可能是一个很好的切入点。你编程的能力很强，可能在某个领域表现出色，这也是你成为 software engineer的有利条件。\n\n其实，除了你的编程技能，你对篮球的热爱也是挺有趣的，如果你能将这些技能结合使用，可能是一个很好的方向。例如，你可以利用你的篮球知识来设计游戏或解决某种技术问题，或者利用你的编程能力来优化自己的篮球战术。\n\n如果你对 Font Awesome 这样的开源 hesitate，这将是你的一个很好的起点。它可以让你快速开发一个简单的网站，展示你的编程能力和对足球的热爱。这样的话，你不仅能展示你的技术能力，还能展示你的 Otherwise Half。\n\n说到 else half，你应该更 Later来扩展你的兴趣，比如学习 Game Dev, Game Engine, 项目管理等，这样可以让你的探索更加多元化。或许你对 AI 或数据科学也有一定的兴趣，如果有机会，可以加入一些相关的话题和团队，体验运动与技术结合的有趣场景。\n\n总之，你很有潜力成为 software工程师！比方说，你可以尝试以下步骤：\n\n1. 关注 Font Awesome，展示你的编程能力。\n2. 尝试开发某个功能，结合你的篮球知识。\n3. 现在是好好学习 else half 的机会。\n\n希望这是一个鼓励的方向。祝你未来一切顺利！\n</answer>"}]}
|
||||
|
|
@ -0,0 +1,2 @@
|
|||
{"conversations":[{"role":"user","content":"你好吗？"},{"role":"assistant","content":"我很好，谢谢！你呢？"},{"role":"user","content":"我也很好，谢谢！"},{"role":"assistant","content":"太好了！祝你今天愉快！"}]}
|
||||
{"conversations":[{"role":"user","content":"你喜欢什么运动？"},{"role":"assistant","content":"我喜欢跑步和游泳。你呢？"},{"role":"user","content":"我喜欢打篮球！"},{"role":"assistant","content":"篮球很棒！是一个很好的团队运动。"}]}
|
||||
|
|
@ -0,0 +1,3 @@
|
|||
{"text": "<s>近年来，人工智能技术迅速发展，深刻改变了各行各业的面貌。机器学习、自然语言处理、计算机视觉等领域的突破性进展，使得智能产品和服务越来越普及。从智能家居到自动驾驶，再到智能医疗，AI的应用场景正在快速拓展。随着技术的不断进步，未来的人工智能将更加智能、更加贴近人类生活。</s>"}
|
||||
{"text": "<s>在数据时代，信息的流通和处理速度决定了一个国家的竞争力。随着大数据技术和云计算的普及，企业和政府能够更高效地利用信息资源，实现精细化管理和决策，1234567890qwertyuiopasdfhgjklzxbcdljfasjdfaflsdjffa深度发掘的链接发的饭卡京东方卡拉快递费发动快速开发设计大大大大大大顶顶顶。</s>"}
|
||||
{"text": "<s>一个卖瓷碗的老人挑着扁担在路上走着，突然一个瓷碗掉到地上摔碎了，但是老人头也不回继续向前走。路人看到很奇怪，便问：“为什么你的碗摔碎了你却不看一下呢？”老人答到：“我再怎么回头看，碗是碎的。”失去的东西就要学着去接受，学着放下。很多事并不会因为你的悲伤就会回来，结果就会被改变。</s>"}
|
||||
Loading…
Reference in New Issue