{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "vscode": {
     "languageId": "plaintext"
    }
   },
   "source": [
    "# 0-Tokenizer\n",
    "\n",
    "对一段文本进行分词就是将其分割成单词或子词，然后通过查找表将这些单词或子词转换为对应的编号。\n",
    "\n",
    "分词器的就是把一段文本分割成单词或子词的工具。 "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "vscode": {
     "languageId": "plaintext"
    }
   },
   "source": [
    "## 子词分词算法\n",
    "\n",
    "常见的子词分词算法有三种：\n",
    "\n",
    "1. 字节对编码（Byte Pair Encoding，BPE）\n",
    "2. WordPiece\n",
    "3. Unigram\n",
    "\n",
    "### BPE\n",
    "\n",
    "BPE 是一种简单的数据压缩技术，它会迭代地替换序列中最频繁出现的字节对。BPE 依赖一个预分词器，该预分词器会将训练数据分割成单词（在本项目中，我们使用按空格分词的方法作为预分词方法）。\n",
    "\n",
    "在预分词之后，会创建一组唯一的单词，并确定它们在数据中的出现频率。接下来，BPE 会创建一个基础词表，该词表包含预分词器最初生成的数据中所有唯一单词的符号。然后，会将这对符号从词表中移除，新形成的符号将加入词表。在迭代过程中，BPE 算法会合并频繁出现的符号对。\n",
    "\n",
    "给定词表的大小，BPE（字节对编码）算法最终会合并出现频率最高的符号对，直到收敛到该大小。\n",
    "\n",
    "### WordPiece\n",
    "\n",
    "WordPiece 算法与 BPE 非常相似。WordPiece 首先将词表初始化为包含训练数据中出现的每个字符，然后逐步学习给定数量的合并规则。与 BPE 不同的是，WordPiece 并不选择最频繁出现的符号对，而是选择那个加入词表后能使训练数据出现的可能性最大化的符号对。\n",
    "\n",
    "### Unigram\n",
    "\n",
    "Unigram 算法将其基础词表初始化为大量的符号，然后逐步削减每个符号，以获得一个更小的词表。它会在训练数据上定义一个对数似然损失，以此来确定是否从词表中移除某个符号。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 训练一个最简单的分词器\n",
    "\n",
    "在本节中，我们将学习基于 transformers 库来训练你自己的分词器。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 初始化\n",
    "\n",
    "首先，我们应该初始化我们的分词器，并确定选择哪种方法。我们将使用字节对编码（BPE）算法。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tokenizers import (\n",
    "    decoders,\n",
    "    models,\n",
    "    normalizers,\n",
    "    pre_tokenizers,\n",
    "    processors,\n",
    "    trainers,\n",
    "    Tokenizer,\n",
    ")\n",
    "\n",
    "tokenizer = Tokenizer(models.BPE())\n",
    "tokenizer.pre_tokenizer = pre_tokenizers.ByteLevel(add_prefix_space=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 定义特殊标记\n",
    "\n",
    "数据集中存在一些我们不希望被分词的特殊标记，我们会将这些标记定义为特殊标记，并将它们传递给分词器训练器，以防止出现错误的分词情况。 "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "special_tokens = [\"<unk>\", \"<s>\", \"</s>\"]\n",
    "\n",
    "trainer = trainers.BpeTrainer(\n",
    "        vocab_size=256,\n",
    "        special_tokens=special_tokens,\n",
    "        show_progress=True,\n",
    "        initial_alphabet=pre_tokenizers.ByteLevel.alphabet()\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 从文件中读取数据\n",
    "我们使用 JSON Lines（jsonl）格式来存储训练数据，让我们编写函数来处理这些数据。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "\n",
    "def read_texts_from_jsonl(file_path):\n",
    "    with open(file_path, 'r', encoding='utf-8') as f:\n",
    "        for line in f:\n",
    "            data = json.loads(line)\n",
    "            yield data['text']\n",
    "\n",
    "data_path = './demo/tokenizer_data.jsonl'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'<s>近年来，人工智能技术迅速发展，深刻改变了各行各业的面貌。机器学习、自然语言处理、计算机视觉等领域的突破性进展，使得智能产品和服务越来越普及。从智能家居到自动驾驶，再到智能医疗，AI的应用场景正在快速拓展。随着技术的不断进步，未来的人工智能将更加智能、更加贴近人类生活。</s>'"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_iter = read_texts_from_jsonl(data_path)\n",
    "next(data_iter)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 开始训练!\n",
    "\n",
    "我们使用分词器的内置函数来训练分词器，轻松愉快。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "tokenizer.train_from_iterator(data_iter, trainer=trainer)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 设置解码器"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "tokenizer.decoder = decoders.ByteLevel()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "接下来，检查一下特殊标记是否得到了妥善处理。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "assert tokenizer.token_to_id('<unk>') == 0\n",
    "assert tokenizer.token_to_id('<s>') == 1\n",
    "assert tokenizer.token_to_id('</s>') == 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 将训练好的分词器保存到磁盘"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['./demo/tokenizer_demo\\\\vocab.json', './demo/tokenizer_demo\\\\merges.txt']"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import os\n",
    "\n",
    "tokenizer_dir = \"./demo/tokenizer_demo\"\n",
    "os.makedirs(tokenizer_dir, exist_ok=True)\n",
    "tokenizer.save(os.path.join(tokenizer_dir, \"tokenizer.json\")) # At this point, you will see a file named tokenizer.json under tokenizer_dir\n",
    "tokenizer.model.save(tokenizer_dir) # generate vocab.json & merges.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 手动创建一份配置文件"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tokenizer training completed and saved.\n"
     ]
    }
   ],
   "source": [
    "config = {\n",
    "    \"add_bos_token\": False,\n",
    "    \"add_eos_token\": False,\n",
    "    \"add_prefix_space\": False,\n",
    "    \"added_tokens_decoder\": {\n",
    "        \"0\": {\n",
    "            \"content\": \"<unk>\",\n",
    "            \"lstrip\": False,\n",
    "            \"normalized\": False,\n",
    "            \"rstrip\": False,\n",
    "            \"single_word\": False,\n",
    "            \"special\": True\n",
    "        },\n",
    "        \"1\": {\n",
    "            \"content\": \"<s>\",\n",
    "            \"lstrip\": False,\n",
    "            \"normalized\": False,\n",
    "            \"rstrip\": False,\n",
    "            \"single_word\": False,\n",
    "            \"special\": True\n",
    "        },\n",
    "        \"2\": {\n",
    "            \"content\": \"</s>\",\n",
    "            \"lstrip\": False,\n",
    "            \"normalized\": False,\n",
    "            \"rstrip\": False,\n",
    "            \"single_word\": False,\n",
    "            \"special\": True\n",
    "        }\n",
    "    },\n",
    "    \"additional_special_tokens\": [],\n",
    "    \"bos_token\": \"<s>\",\n",
    "    \"clean_up_tokenization_spaces\": False,\n",
    "    \"eos_token\": \"</s>\",\n",
    "    \"legacy\": True,\n",
    "    \"model_max_length\": 32768,\n",
    "    \"pad_token\": \"<unk>\",\n",
    "    \"sp_model_kwargs\": {},\n",
    "    \"spaces_between_special_tokens\": False,\n",
    "    \"tokenizer_class\": \"PreTrainedTokenizerFast\",\n",
    "    \"unk_token\": \"<unk>\",\n",
    "    \"chat_template\": \"{{ '<s>' + messages[0]['text'] + '</s>' }}\"\n",
    "}\n",
    "\n",
    "with open(os.path.join(tokenizer_dir, \"tokenizer_config.json\"), \"w\", encoding=\"utf-8\") as config_file:\n",
    "    json.dump(config, config_file, ensure_ascii=False, indent=4)\n",
    "\n",
    "print(\"Tokenizer training completed and saved.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "现在我们已经训练了一个简单的分词器，并将其进行保存，接下来，我们试着加载它，并使用其帮助我们对文本进行编解码"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'<s>失去的东西就要学着去接受，学着放下。</s>'"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from transformers import AutoTokenizer\n",
    "\n",
    "tokenizer = AutoTokenizer.from_pretrained(\"./demo/tokenizer_demo\")\n",
    "msg = [{\"text\": \"失去的东西就要学着去接受，学着放下。\"}]\n",
    "new_msg = tokenizer.apply_chat_template(\n",
    "    msg,\n",
    "    tokenize=False\n",
    ")\n",
    "new_msg"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "actural length of tokenizer： 259\n"
     ]
    }
   ],
   "source": [
    "actual_vocab_size = len(tokenizer)\n",
    "print('actural length of tokenizer：', actual_vocab_size)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'input_ids': [1, 164, 100, 112, 164, 239, 122, 166, 251, 229, 163, 119, 253, 167, 101, 126, 164, 111, 112, 167, 102, 226, 164, 258, 102, 166, 254, 225, 164, 239, 122, 165, 239, 101, 164, 240, 248, 174, 123, 237, 164, 258, 102, 166, 254, 225, 165, 245, 125, 163, 119, 236, 162, 225, 227, 2], 'token_type_ids': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]}"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_inputs = tokenizer(new_msg)\n",
    "model_inputs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'<s>失去的东西就要学着去接受，学着放下。</s>'"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "response = tokenizer.decode(model_inputs['input_ids'], skip_special_tokens=False)\n",
    "response"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'失去的东西就要学着去接受，学着放下。'"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "response = tokenizer.decode(model_inputs['input_ids'], skip_special_tokens=True)\n",
    "response"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.14"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}