mmlu_pro bug in fewshot + chat_template

[Moreh-LeeJunhyeok]

Hi.
using chat_template option in mmlu_pro makes awkward input, which user question contains answer content
(lm_harness commit hash 4890e88)

{'role': 'system', 'content': 'you are a helpful assistant\n\nThe following are multiple choice questions (with answers) about biology. Think step by step and then finish your answer with "the answer is (X)" where X is the correct letter choice.\n'}, 
{'role': 'user', 'content': "Question:\nWhich of the following represents an accurate statement concerning arthropods?\nOptions:\nA. They possess an exoskeleton composed primarily of peptidoglycan.\nB. They possess an open circulatory system with a dorsal heart.\nC. They are members of a biologically unsuccessful phylum incapable of exploiting diverse habitats and nutrition sources.\nD. They lack paired, jointed appendages.\nE. N/A\nF. N/A\nG. N/A\nH. N/A\nI. N/A\nJ. N/A
Answer: Let's think step by step.  Peptidoglycan is known to comprise the plasma membrane of most bacteria, rather than the exoskeleton of arthropods, which is made of chitin, which rules out (A). The answer (C) is false because arthropods are a highly successful phylum. Likewise, arthropods have paired, jointed appendages, which rules out (D). The only remaining option is (B) 
...(much more!)

changing below makes input makes sense

fewshot_config:
  ...
  doc_to_target: !function utils.fewshot_to_target

adding new function, fewshot_to_target

def format_cot_target(example, including_answer=True):
   cot_content = example["cot_content"].replace(
        "A: Let's think step by step. ", ""
   )
    return cot_content

fewshot_to_target = partial(format_cot_target, including_answer=True)

input after changes

{'role': 'system', 'content': 'you are a helpful assistant\n\nThe following are multiple choice questions (with answers) about biology. Think step by step and then finish your answer with "the answer is (X)" where X is the correct letter choice.\n'}, 
{'role': 'user', 'content': "Question:\nWhich of the following represents an accurate statement concerning arthropods?\nOptions:\nA. They possess an exoskeleton composed primarily of peptidoglycan.\nB. They possess an open circulatory system with a dorsal heart.\nC. They are members of a biologically unsuccessful phylum incapable of exploiting diverse habitats and nutrition sources.\nD. They lack paired, jointed appendages.\nE. N/A\nF. N/A\nG. N/A\nH. N/A\nI. N/A\nJ. N/A\nAnswer: Let's think step by step."}, 
{'role': 'assistant', 'content': 'Peptidoglycan is known to comprise the plasma membrane of most bacteria, rather than the exoskeleton of arthropods, which is made of chitin, which rules out (A). The answer (C) is false because arthropods are a highly successful phylum. Likewise, arthropods have paired, jointed appendages, which rules out (D). The only remaining option is (B), as arthropods have an open circulatory system with a dorsal tubular heart. The answer is (B).'}, 
...