Skip to content

fix(routing): classify tokenizer text before binary formats - #1629

Merged
mldangelo-oai merged 18 commits into
mainfrom
mdangelo/codex/hf-fp-t05-tokenizer-text-routing-20260610
Jun 12, 2026
Merged

mldangelo-oai merged 18 commits into
mainfrom
mdangelo/codex/hf-fp-t05-tokenizer-text-routing-20260610

Conversation

@mldangelo-oai

Copy link
Copy Markdown
Contributor

Summary

  • classify bounded UTF-8 tokenizer/documentation text before renamed binary format routing
  • keep real MessagePack and malformed binary controls on their owning scanners

Root Cause

The bounded printable-text fast path only accepted ASCII bytes. UTF-8 tokenizer/docs with non-ASCII bytes fell through to renamed Flax MessagePack routing and produced S902 incomplete-analysis findings.

Validation

  • PROMPTFOO_DISABLE_TELEMETRY=1 UV_CACHE_DIR=/tmp/modelaudit-uv-cache uv run pytest tests/test_core.py -k "bounded_utf8_tokenizer_text or real_msgpack_fixture or malformed_binary_control" -q # 4 passed
  • PROMPTFOO_DISABLE_TELEMETRY=1 UV_CACHE_DIR=/tmp/modelaudit-uv-cache uv run pytest tests/test_core.py tests/scanners/test_flax_msgpack_scanner.py tests/scanners/test_pytorch_binary_scanner.py -q # 835 passed, 22 skipped
  • UV_CACHE_DIR=/tmp/modelaudit-uv-cache uv run ruff format --check modelaudit/ packages/modelaudit-picklescan/src packages/modelaudit-picklescan/tests tests/
  • UV_CACHE_DIR=/tmp/modelaudit-uv-cache uv run ruff check modelaudit/ packages/modelaudit-picklescan/src packages/modelaudit-picklescan/tests tests/
  • UV_CACHE_DIR=/tmp/modelaudit-uv-cache uv run mypy modelaudit/ packages/modelaudit-picklescan/src packages/modelaudit-picklescan/tests tests/
  • PROMPTFOO_DISABLE_TELEMETRY=1 UV_CACHE_DIR=/tmp/modelaudit-uv-cache uv run pytest -n auto -m "not slow and not integration" --maxfail=1 # inherited current-main failure: tests/test_cli.py::test_scan_multiple_cross_directory_shards_reconciles_complete_family; reproduced on edited and clean-main worktrees

Pinned Hugging Face QA

Downloaded only named small artifacts for nvidia/LocateAnything-3B@272068e81a31e88a48ea03c20a09decba2b62ed6 and bosonai/higgs-audio-v3-tts-4b@5402f019e7f316ff513e265f0431e145afcd2cc1. The bosonai pinned tree has README.md but no merges.txt.

  • baseline named-path CLI QA: nvidia merges.txt and bosonai README.md routed to flax_msgpack with S902 MessagePack findings
  • patched named-path CLI QA: scanner_names=[text]; nvidia merges.txt clean; no pinned artifact had MessagePack checks/findings

Security Tradeoff

Invalid UTF-8 and binary control bytes still fail the text proof, so ambiguous binary content remains eligible for binary/MessagePack routing.

@mldangelo-oai

Copy link
Copy Markdown
Contributor Author

@codex review

@github-actions

github-actions Bot commented Jun 10, 2026 •

Copy link
Copy Markdown
Contributor

Workflow run and artifacts

Performance Benchmarks

Compared 12 shared benchmarks with a regression threshold of 15%.
Status: 0 regressions, 0 improved, 12 stable, 0 new, 0 missing.
Aggregate shared-benchmark median: 1.473s -> 1.464s (-0.6%).

Workload Benchmark Target Size Files Baseline Current Change Status
suspicious-pickle-intake tests/benchmarks/test_scan_benchmarks.py::test_scan_suspicious_pickle_intake suspicious-intake 183.8 KiB 4 150.49ms 144.37ms -4.1% stable
nested-payload-review tests/benchmarks/test_picklescan_benchmarks.py::test_picklescan_nested_payload_review[nested_hex] nested_hex 130 B 1 548.4us 562.1us +2.5% stable
warm-cache-rescan tests/benchmarks/test_scan_benchmarks.py::test_scan_warm_cached_repository_rescan release-candidate 547.3 KiB 32 110.79ms 108.50ms -2.1% stable
padded-multi-stream-upload tests/benchmarks/test_picklescan_benchmarks.py::test_picklescan_padded_multi_stream_upload multi_stream_padded 4.1 KiB 1 569.5us 581.1us +2.0% stable
direct-malicious-upload tests/benchmarks/test_picklescan_benchmarks.py::test_picklescan_direct_malicious_upload malicious_reduce 52 B 1 469.4us 460.3us -1.9% stable
single-checkpoint-preflight tests/benchmarks/test_scan_benchmarks.py::test_scan_single_checkpoint_before_load single_checkpoint.pkl 183.0 KiB 1 75.35ms 74.61ms -1.0% stable
nested-payload-review tests/benchmarks/test_picklescan_benchmarks.py::test_picklescan_nested_payload_review[nested_raw] nested_raw 78 B 1 513.4us 517.2us +0.7% stable
chunked-upload-stream tests/benchmarks/test_picklescan_benchmarks.py::test_picklescan_chunked_upload_stream chunked_stream 278.2 KiB 1 116.74ms 115.89ms -0.7% stable
duplicate-heavy-registry tests/benchmarks/test_scan_benchmarks.py::test_scan_duplicate_registry_snapshot registry-snapshot 915.2 KiB 13 413.72ms 414.86ms +0.3% stable
clean-training-checkpoint tests/benchmarks/test_picklescan_benchmarks.py::test_picklescan_clean_training_checkpoint safe_large 278.2 KiB 1 112.71ms 112.42ms -0.3% stable
mixed-model-repository tests/benchmarks/test_scan_benchmarks.py::test_scan_release_candidate_repository release-candidate 547.3 KiB 32 490.34ms 490.87ms +0.1% stable
nested-payload-review tests/benchmarks/test_picklescan_benchmarks.py::test_picklescan_nested_payload_review[nested_base64] nested_base64 98 B 1 517.5us 517.5us +0.0% stable

@chatgpt-codex-connector

Copy link
Copy Markdown

Security review completed. No security issues were found in this pull request.

View security finding report

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. More of your lovely PRs please.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@mldangelo-oai
mldangelo-oai requested a review from mldangelo June 11, 2026 00:51
@mldangelo-oai
mldangelo-oai enabled auto-merge (squash) June 11, 2026 00:53
@mldangelo-oai

Copy link
Copy Markdown
Contributor Author

New pinned top-1000 routing QA on main 8d6c4864:

Qwen/Qwen2-VL-2B-Instruct@895c3a49bc3fa70a340399125c650a463535e71c routes its ordinary 1,671,839-byte tokenizer merges.txt through flax_msgpack and emits flax_msgpack_routing_incomplete. The full repository scan completed all 4.43 GB in 470.45 seconds; this is a deterministic text-versus-binary ownership defect, not a download truncation.

Please include this exact pinned merges.txt in current-head QA and retain renamed/binary MessagePack controls.

Local audit: modelaudit-hf-scan-swarm/outputs/hf-audit-r257-r258-20260611.md.

@mldangelo-oai

Copy link
Copy Markdown
Contributor Author

Additional pinned tokenizer-routing QA on main 8d6c4864:

Hcompany/Holo-3.1-4B@b63fb9bacd968a20b31d0c704b14429c0aad3a84 routes its ordinary 3,353,259-byte merges.txt through flax_msgpack and emits flax_msgpack_routing_incomplete. The full 15-file / 10.37 GB repository scan completed, so this is deterministic text-versus-binary ownership noise rather than a partial download.

Please include this exact pinned merges.txt in current-head QA while retaining renamed/binary MessagePack and text/binary polyglot controls.

Audit: modelaudit-hf-scan-swarm/outputs/hf-audit-r259-20260611.md.

@mldangelo-oai

Copy link
Copy Markdown
Contributor Author

Pinned real-model QA from the rolling top-1000 scan reproduces the merges.txt routing defect again.

  • Model: Qwen/Qwen2-1.5B-Instruct
  • Revision: ba1cf1846d7df0a0591d6c00649f57e798519da8
  • Exact ModelAudit main: 8d6c4864fe2ea833ceaef1b9803d225afb1e8d69
  • Coverage: 10/10 selected files, 3,098,962,420 declared bytes
  • Runtime: 490 seconds
  • Result: exit 2, valid JSON, seven issues

Six issues are informational README URL/domain detections. The only incomplete-coverage condition is ordinary BPE merges.txt being routed to flax_msgpack, producing S902 Flax MessagePack analysis incomplete because bounded routing inspection could not complete with scan_outcome=inconclusive and flax_msgpack_routing_incomplete.

The 3.10 GB SafeTensors model and every other selected artifact completed without a coverage error. This is a real-world false positive, not a legitimate model security finding. Please add this immutable model/revision as another end-to-end regression and verify the exact PR head exits cleanly without selecting Flax for merges.txt.

@mldangelo-oai

Copy link
Copy Markdown
Contributor Author

@codex review

@mldangelo-oai
mldangelo-oai disabled auto-merge June 11, 2026 07:03

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 45360b7622

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread modelaudit/utils/file/detection.py Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6d2621a4b3

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread modelaudit/utils/file/detection.py Outdated
Comment thread modelaudit/utils/file/detection.py Outdated
@mldangelo-oai

Copy link
Copy Markdown
Contributor Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 752bfdf82a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread modelaudit/utils/file/detection.py
@mldangelo-oai

Copy link
Copy Markdown
Contributor Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 40c514f79a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread modelaudit/utils/file/detection.py
@mldangelo-oai

Copy link
Copy Markdown
Contributor Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6e9eef5f6b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread modelaudit/utils/file/detection.py Outdated
Comment thread modelaudit/utils/file/detection.py
@mldangelo-oai

Copy link
Copy Markdown
Contributor Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Can't wait for the next one!

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@mldangelo-oai
mldangelo-oai enabled auto-merge (squash) June 11, 2026 12:09
@mldangelo-oai
mldangelo-oai disabled auto-merge June 11, 2026 12:42
@mldangelo-oai

Copy link
Copy Markdown
Contributor Author

Exact-head independent review of 3b286cc22919bae7a20ef8e65fa632e7fec26600 found no P0/P1/P2 issues, but remains blocked on one reproducible P3 parity/resource defect.

P3: Hugging Face streaming routes large tokenizer text as Flax MessagePack

The HF prefix probe returns non-text whenever it has only a prefix, then an inconclusive MessagePack probe selects flax_msgpack. That remote path was not updated to mirror the new local bounded text-owner logic.

Three pinned real merges.txt files scanned locally as unknown, success=true, exit 0. Mocked HF range reads over the same bytes classified all three as flax_msgpack; _select_streamable_hf_files(... scannable_scanner_ids={"flax_msgpack"}) selected both pytorch_model.bin and merges.txt. This is a local/remote parity and resource-budget issue, not a demonstrated binary scanner bypass.

Validation: targeted core tests 22 passed; filetype tests 22 passed; file-filter/directory tests 6 passed; independent probes covered tokenizer/vocab/merges text, misleading suffixes, malformed UTF-8/control bytes, pickle, XGBoost, JAX, nested ZIP, executable/network text, and local resource bounds. All exact-head CI passed and all review threads are resolved. Please mirror the local bounded text-owner logic in the HF route, add source-level regressions, and re-request exact-head review.

…owner routing in Hugging Face streaming selection so complete tokenizer/config text is not promoted to binary model scanners while preserving binary/protobuf fail-closed candidates.
@mldangelo-oai

Copy link
Copy Markdown
Contributor Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Another round soon, please!

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 0edfb7dcb8

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread modelaudit/utils/file/detection.py

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6c2e39f697

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread modelaudit/utils/file/detection.py
Comment thread modelaudit/utils/file/detection.py

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 78668aaa39

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread modelaudit/utils/sources/huggingface.py
Resolve tokenizer text routing conflicts.
@mldangelo-oai

Copy link
Copy Markdown
Contributor Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Another round soon, please!

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@mldangelo-oai

Copy link
Copy Markdown
Contributor Author

@codex review

@mldangelo-oai

Copy link
Copy Markdown
Contributor Author

Addressed the four open routing review threads in head 6fbf79f: bounded text-owner helpers now check the 2/10 MiB cap before reading, inline-colon MessagePack scalar candidates have local/core/directory/HF malicious-positive coverage, weak ASCII protobuf field-5 varint text now stays text-owned locally and remotely, and the remote inline-colon Flax candidate is covered in streaming selection. Local validation: focused routing boundary 55 passed, media 10 passed, ONNX 4 passed, broader adjacent suite 2290 passed/56 skipped, ruff format/check, mypy, git diff --check, and independent no-uv reviewer pass.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6fbf79fdfe

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread modelaudit/utils/sources/huggingface.py Outdated
Comment thread modelaudit/utils/sources/huggingface.py Outdated
@mldangelo-oai

Copy link
Copy Markdown
Contributor Author

Addressed the latest HF routing review threads in head b6ca1fc: SafeTensors-to-Flax overlap is limited to text-owner suffixes, and remote text-owner checks no longer expand to full known-small files before binary probes. Local validation: focused latest-review set 44 passed; full Hugging Face source tests 352 passed, 2 skipped; ruff format/check, mypy, and git diff --check passed; independent no-uv reviewer pass.

@mldangelo-oai

Copy link
Copy Markdown
Contributor Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b6ca1fc4bf

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread modelaudit/utils/sources/huggingface.py Outdated
@mldangelo-oai

Copy link
Copy Markdown
Contributor Author

Addressed the remote text-owner P1 in head 4c04dcf. Prefix-plus-known-size no longer proves complete text; ordinary text prefixes now run a bounded post-prefix/tail binary guard before being skipped; confirmed Flax and protobuf candidate bytes after text prefixes are retained, while ordinary non-ASCII BPE/tokenizer text remains skipped. Local validation: focused reviewer-derived set 16 passed; full Hugging Face source tests 356 passed, 2 skipped; adjacent routing regression set 38 passed; ruff format/check, mypy, and git diff --check passed; independent reviewer pass with blockers=[].

@mldangelo-oai

Copy link
Copy Markdown
Contributor Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. 👍

Reviewed commit: 4c04dcf860

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@mldangelo-oai
mldangelo-oai merged commit 067717d into main Jun 12, 2026
29 checks passed
@mldangelo-oai
mldangelo-oai deleted the mdangelo/codex/hf-fp-t05-tokenizer-text-routing-20260610 branch June 12, 2026 03:40
@github-actions github-actions Bot mentioned this pull request Jun 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant