fix ( model/dataset.py )

在预训练阶段,pretrain_hq.jsonl文件中已经用和将句子分割,在dataset.py文件构建输入文本时,再次在头部和尾部添加了和,有重复。

fix #314
pull/315/head
cn-farmer 2025-04-02 10:37:58 +08:00
parent 278ec760a1
commit ad741354b0
1 changed files with 1 additions and 1 deletions

View File

@ -35,7 +35,7 @@ class PretrainDataset(Dataset):
sample = self.samples[index]
# 构建输入文本
text = f"{self.tokenizer.bos_token}{str(sample['text'])}{self.tokenizer.eos_token}"
text = sample['text']
encoding = self.tokenizer(
text,
max_length=self.max_length,