Different tokenization with same tokenizer from 4.57.3 to 5.0

[awni]

System Info

Moving from transformers 4.57.3 to 5.0+ introduces a different and seemingly incorrect tokenization when using the same tokenizer.

I believe the new version is incorrect because when using it, we get bad results (the model starts to introduce unexpected artifacts in the response).

Who can help?

@ArthurZucker

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

Run the following with the two versions and compare the tokenized prompt.

from transformers import AutoTokenizer                                         
                                                                               
tokenizer = AutoTokenizer.from_pretrained("mlx-community/MiniMax-M2.1-4bit")   
messages = [                                                                   
    {                                                                          
        "role": "system",                                                      
        "content": '"You are opencode, an interactive CLI tool that helps users with software engineering tasks. Use the instructions below and the tools available to you to assist the user.\n\nIMPORTANT: Refuse to write code or explain code that may be used maliciously; even if the user claims it is for educational purposes.'
    }
]
    
prompt = tokenizer.apply_chat_template(                                        
    messages, add_generation_prompt=True, tokenize=True, return_dict=False     
)
print(prompt) 

Expected behavior

They should be the same.

[ArthurZucker]

In [12]: print(tokenizer.convert_ids_to_tokens(prompt))
[']~!b[', ']~b]', 'system', 'Ċ', '"You', 'Ġare', 'Ġopen', 'code', ',', 'Ġan', 'Ġinteractive', 'ĠCLI', 'Ġtool', 'Ġthat', 'Ġhelps', 'Ġusers', 'Ġwith', 'Ġsoftware', 'Ġengineering', 'Ġtasks', '.', 'ĠUse', 'Ġthe', 'Ġinstructions', 'Ġbelow', 'Ġand', 'Ġthe', 'Ġtools', 'Ġavailable', 'Ġto', 'Ġyou', 'Ġto', 'Ġassist', 'Ġthe', 'Ġuser', '.ĊĊ', 'IMPORTANT', ':', 'ĠRef', 'use', 'Ġto', 'Ġwrite', 'Ġcode', 'Ġor', 'Ġexplain', 'Ġcode', 'Ġthat', 'Ġmay', 'Ġbe', 'Ġused', 'Ġmalicious', 'ly', ';', 'Ġeven', 'Ġif', 'Ġthe', 'Ġuser', 'Ġclaims', 'Ġit', 'Ġis', 'Ġfor', 'Ġeducational', 'Ġpurposes', '.', '[e~[', 'Ċ', ']~b]', 'ai', 'Ċ', '<think>', 'Ċ']

In [13]: print(tokenizer.convert_ids_to_tokens(b))
[']~!b[', ']~b]', 'system', 'Ċ', '"', 'You', 'Ġare', 'Ġopen', 'code', ',', 'Ġan', 'Ġinteractive', 'ĠCLI', 'Ġtool', 'Ġthat', 'Ġhelps', 'Ġusers', 'Ġwith', 'Ġsoftware', 'Ġengineering', 'Ġtasks', '.', 'ĠUse', 'Ġthe', 'Ġinstructions', 'Ġbelow', 'Ġand', 'Ġthe', 'Ġtools', 'Ġavailable', 'Ġto', 'Ġyou', 'Ġto', 'Ġassist', 'Ġthe', 'Ġuser', '.', 'Ċ', 'Ċ', 'IMPORTANT', ':', 'ĠRef', 'use', 'Ġto', 'Ġwrite', 'Ġcode', 'Ġor', 'Ġexplain', 'Ġcode', 'Ġthat', 'Ġmay', 'Ġbe', 'Ġused', 'Ġmalicious', 'ly', ';', 'Ġeven', 'Ġif', 'Ġthe', 'Ġuser', 'Ġclaims', 'Ġit', 'Ġis', 'Ġfor', 'Ġeducational', 'Ġpurposes', '.', '[e~[', 'Ċ', ']~b]', 'ai', 'Ċ', '<think>', 'Ċ']

I think what helps more is if you show token to token comparison.
In that case it seems '"You' is now split into '"', 'You'.

Usually this means pre_tokenizer / normalizer issues.

# v4.57.3
In [17]: tokenizer._tokenizer.pre_tokenizer
Out[17]: Sequence(pretokenizers=[Split(pattern=Regex("[^\r\n\p{L}\p{N}]?[\p{Lu}\p{Lt}\p{Lm}\p{Lo}\p{M}]*[\p{Ll}\p{Lm}\p{Lo}\p{M}]+(?i:'s|'t|'re|'ve|'m|'ll|'d)?|[^\r\n\p{L}\p{N}]?[\p{Lu}\p{Lt}\p{Lm}\p{Lo}\p{M}]+[\p{Ll}\p{Lm}\p{Lo}\p{M}]*(?i:'s|'t|'re|'ve|'m|'ll|'d)?|\p{N}{1,3}| ?[^\s\p{L}\p{N}]+[\r\n/]*|\s*[\r\n]+|\s+(?!\S)|\s+"), behavior=Removed, invert=True), ByteLevel(add_prefix_space=False, trim_offsets=True, use_regex=False)])

In [18]: tokenizer._tokenizer.normalizer
Out[18]: NFC()

vs

# main
In [2]: tokenizer._tokenizer.normalizer

In [3]: tokenizer._tokenizer.pre_tokenizer
Out[3]: ByteLevel(add_prefix_space=False, trim_offsets=True, use_regex=True)

So I can reproduce the issue.
If you just do:

from tokenizers import Tokenizer
tokenizer = Tokenizer.from_pretrained("mlx-community/MiniMax-M2.1-4bit")

We realize the tokenizer.json is absolutely correct, and the issue is that the class they used is GPT2Tokenizer when it should be PreTrainedTokenizerFast in the tokenizer_config.json....

cc @itazap I am afraid MANY other repos have this bug

[ArthurZucker]

https://huggingface.co/mlx-community/MiniMax-M2.1-4bit/discussions/3 is the solution

[ArthurZucker]

#42894 will also get the fix, in this cas the minimax was mapped to GPT2, it should not