fix(routing): classify tokenizer text before binary formats - #1629
mldangelo-oai merged 18 commits into
Conversation
|
@codex review |
Performance BenchmarksCompared
|
|
Security review completed. No security issues were found in this pull request. ℹ️ About Codex security reviews in GitHubThis is an experimental Codex feature. Security reviews are triggered when:
Once complete, Codex will leave suggestions, or a comment if no findings are found. |
|
Codex Review: Didn't find any major issues. More of your lovely PRs please. ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
|
New pinned top-1000 routing QA on main
Please include this exact pinned Local audit: |
|
Additional pinned tokenizer-routing QA on main
Please include this exact pinned Audit: |
|
Pinned real-model QA from the rolling top-1000 scan reproduces the
Six issues are informational README URL/domain detections. The only incomplete-coverage condition is ordinary BPE The 3.10 GB SafeTensors model and every other selected artifact completed without a coverage error. This is a real-world false positive, not a legitimate model security finding. Please add this immutable model/revision as another end-to-end regression and verify the exact PR head exits cleanly without selecting Flax for |
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 45360b7622
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6d2621a4b3
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 752bfdf82a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 40c514f79a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6e9eef5f6b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex review |
|
Codex Review: Didn't find any major issues. Can't wait for the next one! ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
|
Exact-head independent review of P3: Hugging Face streaming routes large tokenizer text as Flax MessagePack The HF prefix probe returns non-text whenever it has only a prefix, then an inconclusive MessagePack probe selects Three pinned real Validation: targeted core tests |
…owner routing in Hugging Face streaming selection so complete tokenizer/config text is not promoted to binary model scanners while preserving binary/protobuf fail-closed candidates.
|
@codex review |
|
Codex Review: Didn't find any major issues. Another round soon, please! ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0edfb7dcb8
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6c2e39f697
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 78668aaa39
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Resolve tokenizer text routing conflicts.
|
@codex review |
|
Codex Review: Didn't find any major issues. Another round soon, please! ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
|
@codex review |
|
Addressed the four open routing review threads in head 6fbf79f: bounded text-owner helpers now check the 2/10 MiB cap before reading, inline-colon MessagePack scalar candidates have local/core/directory/HF malicious-positive coverage, weak ASCII protobuf field-5 varint text now stays text-owned locally and remotely, and the remote inline-colon Flax candidate is covered in streaming selection. Local validation: focused routing boundary 55 passed, media 10 passed, ONNX 4 passed, broader adjacent suite 2290 passed/56 skipped, ruff format/check, mypy, git diff --check, and independent no-uv reviewer pass. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6fbf79fdfe
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
Addressed the latest HF routing review threads in head b6ca1fc: SafeTensors-to-Flax overlap is limited to text-owner suffixes, and remote text-owner checks no longer expand to full known-small files before binary probes. Local validation: focused latest-review set 44 passed; full Hugging Face source tests 352 passed, 2 skipped; ruff format/check, mypy, and git diff --check passed; independent no-uv reviewer pass. |
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b6ca1fc4bf
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
Addressed the remote text-owner P1 in head 4c04dcf. Prefix-plus-known-size no longer proves complete text; ordinary text prefixes now run a bounded post-prefix/tail binary guard before being skipped; confirmed Flax and protobuf candidate bytes after text prefixes are retained, while ordinary non-ASCII BPE/tokenizer text remains skipped. Local validation: focused reviewer-derived set 16 passed; full Hugging Face source tests 356 passed, 2 skipped; adjacent routing regression set 38 passed; ruff format/check, mypy, and git diff --check passed; independent reviewer pass with blockers=[]. |
|
@codex review |
|
Codex Review: Didn't find any major issues. 👍 Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
Summary
Root Cause
The bounded printable-text fast path only accepted ASCII bytes. UTF-8 tokenizer/docs with non-ASCII bytes fell through to renamed Flax MessagePack routing and produced S902 incomplete-analysis findings.
Validation
Pinned Hugging Face QA
Downloaded only named small artifacts for nvidia/LocateAnything-3B@272068e81a31e88a48ea03c20a09decba2b62ed6 and bosonai/higgs-audio-v3-tts-4b@5402f019e7f316ff513e265f0431e145afcd2cc1. The bosonai pinned tree has README.md but no merges.txt.
Security Tradeoff
Invalid UTF-8 and binary control bytes still fail the text proof, so ambiguous binary content remains eligible for binary/MessagePack routing.