Add missing tokens from added_tokens_decoder into added_tokens

[sayanshaw24]

Fixes tokenizer issue where special tokens added into added_tokens_decoder attribute in tokenizer_config.json but missing in added_tokens in tokenizer.json are still populated correctly into added tokens map such that they are not tokenized incorrectly. This ensures compatibility even with HuggingFace models whose files have inconsistent tokenizer.json and tokenizer_config.json files.

Validation

• Ran C++ test locally with Phi tokenizer files and validated with different tokens
• Tested correct population of added_tokens with missing and complete cases

[wenbingl]

Fixes tokenizer issue where special tokens added into added_tokens_decoder attribute in tokenizer_config.json but missing in added_tokens in tokenizer.json are still populated correctly into added tokens map such that they are not tokenized incorrectly. This ensures compatibility even with HuggingFace models whose files have inconsistent tokenizer.json and tokenizer_config.json files.

Validation

• Ran C++ test locally with PhiO tokenizer files and validated on numerous cases
• Tested correct population of added_tokens with missing and complete cases

Good finding, thanks!

[wenbingl]

Add a sentence which contains one or more tokens from added_tokens_decoder in the current unit test?