Skip to content

Allow untokenized messages when using the prevent tokenization mode. #3864

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
ArEsKay3 wants to merge 2 commits into
base: main
Choose a base branch
from
Open

Allow untokenized messages when using the prevent tokenization mode. #3864

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
43c9da7
Allow untokenized messages when using the prevent tokenization mode.
Mar 13, 2026
c906b5b
Merge branch 'main' into rkirby/allow_untokenized_when_using_prevent
ArEsKay3 Mar 17, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 7 additions & 7 deletions ...re/inference/text_generation_server/dynamic_text_gen_server/endpoints/chat_completions.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -264,7 +264,12 @@ async def chat_completions():
break

# If there was a previous assistant message, we need to replace the prefix tokens with the tokens from the previous generation
if last_assistant_message_idx is not None:
if (
last_assistant_message_idx is not None and
"prompt_token_ids" in template_messages[last_assistant_message_idx] and
"generation_token_ids" in template_messages[last_assistant_message_idx]
):
last_assistant_message = template_messages[last_assistant_message_idx]
messages_to_last_assistant_message = template_messages[
: last_assistant_message_idx + 1
]
Expand All @@ -279,13 +284,8 @@ async def chat_completions():
)

# Replace the prefix tokens with the tokens from the previous generation
last_assistant_message = template_messages[last_assistant_message_idx]
assert (
"prompt_token_ids" in last_assistant_message
and "generation_token_ids" in last_assistant_message
), "Last assistant message must have prompt_token_ids and generation_token_ids from previous generation to avoid prefix retokenization"
previous_turn_token_ids = (
last_assistant_message["prompt_token_ids"]
last_assistant_message["prompt_token_ids"]
+ last_assistant_message["generation_token_ids"]
)
prompt_tokens = _replace_prefix_tokens(
Expand Down
Loading