train_qwen2.ipynb中process_func()缺失<|im_end|>的问题 #10

yangyifei-gogogo · 2025-04-08T09:09:56Z

用复旦中文新闻微调qwen2时，在对数据进行tokenization的时候，response中最后并没有添加<|im_end|>的token，是否应该加上？源代码如下：

def process_func(example):
    """
    将数据集进行预处理
    """
    MAX_LENGTH = 384 
    input_ids, attention_mask, labels = [], [], []
    instruction = tokenizer(
        f"<|im_start|>system\n你是一个文本分类领域的专家，你会接收到一段文本和几个潜在的分类选项，请输出文本内容的正确类型<|im_end|>\n<|im_start|>user\n{example['input']}<|im_end|>\n<|im_start|>assistant\n",
        add_special_tokens=False,
    )
    response = tokenizer(f"{example['output']}", add_special_tokens=False)
    input_ids = instruction["input_ids"] + response["input_ids"] + [tokenizer.pad_token_id]
    attention_mask = (
        instruction["attention_mask"] + response["attention_mask"] + [1]
    )
    labels = [-100] * len(instruction["input_ids"]) + response["input_ids"] + [tokenizer.pad_token_id]
    if len(input_ids) > MAX_LENGTH:  # 做一个截断
        input_ids = input_ids[:MAX_LENGTH]
        attention_mask = attention_mask[:MAX_LENGTH]
        labels = labels[:MAX_LENGTH]
    return {"input_ids": input_ids, "attention_mask": attention_mask, "labels": labels}

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

train_qwen2.ipynb中process_func()缺失<|im_end|>的问题 #10

train_qwen2.ipynb中process_func()缺失<|im_end|>的问题 #10

train_qwen2.ipynb中process_func()缺失<|im_end|>的问题 #10

train_qwen2.ipynb中process_func()缺失<|im_end|>的问题 #10

Comments