fix: deduplicate onnx custom domain findings

[mldangelo-oai]

Summary

• Collapse repeated ONNX custom-operator domain S1111 checks into one bounded per-domain, per-file finding.
• Preserve affected-file evidence at the scanned ONNX file path with occurrence_count, bounded operator_samples, bounded operator_identities, and bounded representative_nodes.
• Preserve explicit custom-operator (domain, op_type, overload) identity through final JSON and SARIF serialization.
• Preserve distinct custom-domain/custom-identity checks through final check consolidation using raw-identity SHA-256 keys.
• Harden long-name and NUL-boundary identity handling: dedup/counting uses raw identities, hashes are length-framed, and display truncation no longer collapses evidence.
• Keep ONNX trust/classification policy unchanged from PR fix: calibrate onnx custom domain findings #1639: no newly trusted domains, no com.microsoft allowlist, and no change to schema/local-function/Python-operator classification.

Root Cause

OnnxScanner._check_custom_ops() deduplicated metadata["custom_domains"], but emitted a failed Custom Operator Domain Check inside the per-node loop. Optimized ONNX exports with repeated runtime kernels therefore produced one S1111 issue/check per node even when the domain evidence was identical.

Review follow-up found final-output identity risks after aggregation: explicit standard-domain custom_op checks needed domain/overload/hash identity in emitted results, core check consolidation needed per-domain/per-identity keys, long custom operator/domain display truncation could collide, and NUL-delimited hashes were ambiguous for protobuf strings. This PR now keeps raw identities for dedup/counting, emits bounded display values plus SHA-256 identity evidence, and length-frames every hashed field.

Security Tradeoff

The change only aggregates reporting after existing classification has decided a node is an external custom operator or explicit custom operator. It does not weaken custom-domain detection, suppress distinct untrusted domains, or change CRITICAL Python operator detection. Malicious/malformed controls cover repeated custom domains plus PyOp, missing custom-domain opset imports, external-data findings, multi-file evidence, mixed domains, final check/issue consolidation, explicit custom_op overload/domain identities, long display-prefix collisions, and NUL-containing identity hash boundaries.

Pinned Real-Model QA

Baseline reproduction on starting SHA 8d6c4864fe2ea833ceaef1b9803d225afb1e8d69 used the exact pinned revision:

PROMPTFOO_DISABLE_TELEMETRY=1 HF_HUB_DISABLE_TELEMETRY=1 HF_HOME=/tmp/modelaudit-hf-t31-cache uv run python -u - <<'PY'
from collections import Counter
from pathlib import Path
from huggingface_hub import HfApi, hf_hub_download
import onnx
from modelaudit.scanners.base import CheckStatus
from modelaudit.scanners.onnx_scanner import OnnxScanner
repo_id = "intfloat/multilingual-e5-large"
revision = "3d7cfbdacd47fdda877c5cd8a79fbcc4f2a574f3"
files = HfApi().list_repo_files(repo_id=repo_id, revision=revision)
onnx_files = [name for name in files if name.startswith("onnx/") and name.endswith(".onnx")]
filename = "onnx/model_O4.onnx"
path = Path(hf_hub_download(repo_id=repo_id, revision=revision, filename=filename))
model = onnx.load(str(path), load_external_data=False)
custom_nodes = [(node.domain, node.op_type, node.name) for node in model.graph.node if node.domain]
result = OnnxScanner().scan(str(path))
failed_custom = [c for c in result.checks if c.name == "Custom Operator Domain Check" and c.status == CheckStatus.FAILED]
print("onnx_files", onnx_files)
print("selected", filename, "bytes", path.stat().st_size)
print("domain_node_counts", dict(Counter(domain for domain, _op, _name in custom_nodes)))
print("com_microsoft_ops", dict(Counter(op for domain, op, _name in custom_nodes if domain == "com.microsoft")))
print("failed_custom_count", len(failed_custom))
print("failed_custom_domains", dict(Counter(check.details.get("domain") for check in failed_custom)))
print("metadata_custom_domains", result.metadata.get("custom_domains", []))
PY

Baseline outcome: onnx/model_O4.onnx is 80,602 bytes with domain_node_counts {'com.microsoft': 96} and com_microsoft_ops {'Attention': 24, 'SkipLayerNormalization': 48, 'FastGelu': 24}. Current main emitted failed_custom_count 96, failed_custom_domains {'com.microsoft': 96}, and metadata_custom_domains ['com.microsoft']. The same revision's onnx/model.onnx has no com.microsoft graph nodes; onnx/model_qint8_avx512_vnni.onnx is 561,845,741 bytes and was not downloaded for bounded local QA.

Post-fix exact-head QA on ea8c7cc115fdb59c048ec9dddc4f7ace469b22bb:

PROMPTFOO_DISABLE_TELEMETRY=1 HF_HUB_DISABLE_TELEMETRY=1 HF_HOME=/tmp/modelaudit-hf-t31-cache uv run python - <<'PY'
from collections import Counter
import json
from pathlib import Path
import onnx
from huggingface_hub import hf_hub_download
from modelaudit.scanners.base import CheckStatus
from modelaudit.scanners.onnx_scanner import OnnxScanner
repo_id = "intfloat/multilingual-e5-large"
revision = "3d7cfbdacd47fdda877c5cd8a79fbcc4f2a574f3"
filename = "onnx/model_O4.onnx"
path = Path(hf_hub_download(repo_id=repo_id, revision=revision, filename=filename, local_files_only=True))
model = onnx.load(str(path), load_external_data=False)
custom_nodes = [node for node in model.graph.node if node.domain == "com.microsoft"]
result = OnnxScanner().scan(str(path))
custom_checks = [check for check in result.checks if check.name == "Custom Operator Domain Check" and check.status == CheckStatus.FAILED]
com_ms_checks = [check for check in custom_checks if check.details.get("domain") == "com.microsoft"]
print(json.dumps({
    "size_bytes": path.stat().st_size,
    "com_microsoft_node_count": len(custom_nodes),
    "com_microsoft_ops": dict(sorted(Counter(node.op_type for node in custom_nodes).items())),
    "failed_custom_check_count": len(custom_checks),
    "failed_custom_domains": sorted({check.details.get("domain") for check in custom_checks}),
    "com_microsoft_occurrence_count": com_ms_checks[0].details["occurrence_count"],
    "com_microsoft_distinct_operator_identity_count": com_ms_checks[0].details["distinct_operator_identity_count"],
    "com_microsoft_domain_hash_present": bool(com_ms_checks[0].details.get("domain_hash")),
}, sort_keys=True))
PY

Post-fix outcome: failed_custom_check_count=1, failed_custom_domains=['com.microsoft'], com_microsoft_node_count=96, com_microsoft_occurrence_count=96, com_microsoft_ops={'Attention': 24, 'FastGelu': 24, 'SkipLayerNormalization': 48}, com_microsoft_distinct_operator_identity_count=3, and com_microsoft_domain_hash_present=true.

Additional bounded QA: sentence-transformers/all-MiniLM-L12-v2@a50ef00143b4d5391434df20ae11632588ac25be, files onnx/model_O2.onnx, onnx/model_O3.onnx, and onnx/model_O4.onnx, each now emits exactly one com.microsoft S1111 check with occurrence_count=36 and two distinct operator identities.

Validation

• uv sync --extra all-ci
• PROMPTFOO_DISABLE_TELEMETRY=1 uv run pytest tests/scanners/test_onnx_scanner.py -k "long_custom_domain_operator_identities or long_custom_domains_survive or identity_hash_length_frames or custom_domain_aggregate_reports or custom_domain_custom_op_overloads or custom_domains_survive_core or long_explicit_custom_op" -m "not slow and not integration" --maxfail=1 -> 7 passed
• PROMPTFOO_DISABLE_TELEMETRY=1 uv run pytest tests/scanners/test_onnx_scanner.py -m "not slow and not integration" --maxfail=1 -> 258 passed, 1 deselected, 1 warning
• MODELAUDIT_RUN_HF_REAL_MODEL_TESTS=1 PROMPTFOO_DISABLE_TELEMETRY=1 HF_HUB_DISABLE_TELEMETRY=1 HF_HOME=/tmp/modelaudit-hf-t31-cache uv run pytest tests/scanners/test_onnx_scanner.py -k "pinned_hf_multilingual" -m "slow and integration" -s --maxfail=1 -> 1 passed, 258 deselected
• uv run ruff format --check modelaudit/ packages/modelaudit-picklescan/src packages/modelaudit-picklescan/tests tests/ -> 419 files already formatted
• uv run ruff check modelaudit/ packages/modelaudit-picklescan/src packages/modelaudit-picklescan/tests tests/ -> all checks passed
• uv run mypy modelaudit/ packages/modelaudit-picklescan/src packages/modelaudit-picklescan/tests tests/ -> success, 474 source files
• PROMPTFOO_DISABLE_TELEMETRY=1 uv run pytest -n auto -m "not slow and not integration" --maxfail=1 -> 18,551 passed, 793 skipped, 40 warnings
• Post-fetch smoke: PROMPTFOO_DISABLE_TELEMETRY=1 uv run pytest tests/scanners/test_onnx_scanner.py -k "long_custom_domain_operator_identities or long_custom_domains_survive or identity_hash_length_frames" -m "not slow and not integration" --maxfail=1 -> 3 passed
• git diff --check -> clean

Final fetch before validation: origin/main remained 8d6c4864fe2ea833ceaef1b9803d225afb1e8d69, so no merge was needed.

[mldangelo-oai]

@codex review

[github-actions]

Workflow run and artifacts

Performance Benchmarks

Compared 12 shared benchmarks with a regression threshold of 15%.
Status: 0 regressions, 0 improved, 12 stable, 0 new, 0 missing.
Aggregate shared-benchmark median: 1.432s -> 1.456s (+1.7%).

Workload	Benchmark	Target	Size	Files	Baseline	Current	Change	Status
warm-cache-rescan	tests/benchmarks/test_scan_benchmarks.py::test_scan_warm_cached_repository_rescan	release-candidate	547.3 KiB	32	111.93ms	120.56ms	+7.7%	stable
direct-malicious-upload	tests/benchmarks/test_picklescan_benchmarks.py::test_picklescan_direct_malicious_upload	malicious_reduce	52 B	1	519.4us	541.1us	+4.2%	stable
suspicious-pickle-intake	tests/benchmarks/test_scan_benchmarks.py::test_scan_suspicious_pickle_intake	suspicious-intake	183.8 KiB	4	143.23ms	146.17ms	+2.1%	stable
mixed-model-repository	tests/benchmarks/test_scan_benchmarks.py::test_scan_release_candidate_repository	release-candidate	547.3 KiB	32	482.72ms	490.93ms	+1.7%	stable
chunked-upload-stream	tests/benchmarks/test_picklescan_benchmarks.py::test_picklescan_chunked_upload_stream	chunked_stream	278.2 KiB	1	113.68ms	115.18ms	+1.3%	stable
clean-training-checkpoint	tests/benchmarks/test_picklescan_benchmarks.py::test_picklescan_clean_training_checkpoint	safe_large	278.2 KiB	1	111.71ms	110.66ms	-0.9%	stable
duplicate-heavy-registry	tests/benchmarks/test_scan_benchmarks.py::test_scan_duplicate_registry_snapshot	registry-snapshot	915.2 KiB	13	394.53ms	397.92ms	+0.9%	stable
padded-multi-stream-upload	tests/benchmarks/test_picklescan_benchmarks.py::test_picklescan_padded_multi_stream_upload	multi_stream_padded	4.1 KiB	1	639.1us	643.8us	+0.7%	stable
nested-payload-review	tests/benchmarks/test_picklescan_benchmarks.py::test_picklescan_nested_payload_review[nested_hex]	nested_hex	130 B	1	617.8us	622.1us	+0.7%	stable
nested-payload-review	tests/benchmarks/test_picklescan_benchmarks.py::test_picklescan_nested_payload_review[nested_raw]	nested_raw	78 B	1	589.7us	593.7us	+0.7%	stable
nested-payload-review	tests/benchmarks/test_picklescan_benchmarks.py::test_picklescan_nested_payload_review[nested_base64]	nested_base64	98 B	1	587.3us	583.5us	-0.6%	stable
single-checkpoint-preflight	tests/benchmarks/test_scan_benchmarks.py::test_scan_single_checkpoint_before_load	single_checkpoint.pkl	183.0 KiB	1	71.69ms	71.89ms	+0.3%	stable

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. More of your lovely PRs please.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[mldangelo-oai]

PR #1656 independent review

Exact head: cb4afc8df30d04ce09cf89c39b23b4de5c651dad

P2: preserve domain and overload identity in explicit custom-operator findings

modelaudit/scanners/onnx_scanner.py:2930-2942 aggregates explicit custom
operators by (domain, op_type, overload), but the emitted issue message omits
the domain and overload and every issue uses the same file location. Final JSON
deduplication therefore collapses distinct operators, and SARIF loses the same
records.

An exact-head reproduction containing custom_op overloads float and int
produced two S1111 issues before CLI deduplication but only one afterward. The
base branch retained both. Include domain and overload in the emitted finding
identity and add end-to-end JSON and SARIF regressions that prove both records
survive final serialization.

[mldangelo-oai]

@codex address that feedback on exact head cb4afc8d. Preserve (domain, op_type, overload) identity through final JSON and SARIF deduplication, add end-to-end overload/domain regressions, push the fix, resolve the blocker, and request a fresh exact-head review.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: cb4afc8df3

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[mldangelo-oai]

Addressed the exact-head feedback in 0555e6962d4083de6fab37238a305c7eb929aa7d: explicit ONNX custom_op findings now include domain and overload in the emitted finding identity/details, with final JSON and SARIF regressions proving distinct identities survive serialization. @codex review

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 0555e6962d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[mldangelo-oai]

Pinned dedup QA input: sentence-transformers/all-MiniLM-L12-v2@a50ef00143b4d5391434df20ae11632588ac25be emits 108 S1111 records that collapse to one root condition: 36 com.microsoft nodes in each of model_O2.onnx, model_O3.onnx, and model_O4.onnx. Preserve file/domain/operator identity, bounded representative nodes, and occurrence counts; do not collapse distinct domains or overloads.

[mldangelo-oai]

Addressed the follow-up review feedback in 9f415c7cbf894deebd79ffed3166c42bed36e19d: domain aggregates now carry bounded distinct operator identities, ONNX custom-domain/custom-identity checks include consolidation keys so final JSON checks stay distinct, and the existing explicit custom_op JSON/SARIF identity regressions remain passing. @codex review

[mldangelo-oai]

Final test-only follow-up is in c81728e828f58fd6e566a6e88e73ea27b9f7b214: added JSON/SARIF regression proving custom-domain custom_op overload identities survive inside the domain aggregate evidence. @codex review

[mldangelo-oai]

Additional bounded QA for sentence-transformers/all-MiniLM-L12-v2@a50ef00143b4d5391434df20ae11632588ac25be completed with telemetry disabled. Scanned only onnx/model_O2.onnx, onnx/model_O3.onnx, and onnx/model_O4.onnx: each contains 36 com.microsoft nodes and now emits exactly one S1111 custom-domain check with occurrence_count=36; distinct operator identity samples are preserved in the aggregate details (2 identities per file).

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c81728e828

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[mldangelo-oai]

Addressed the long-name collision feedback in 2ad045ab25091df01856b30f1d1a4e2adf89d37a: explicit custom-op messages/details now include a raw identity hash, with JSON/SARIF regression for two long names sharing the bounded display prefix. @codex review

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 2ad045ab25

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[mldangelo-oai]

@codex address these exact-head independent-review blockers before merge:

• P2: custom-operator identities are deduplicated after 256-character display truncation. Two checker-valid 257-character op types differing only at the end collapse to one identity while operator_identities_truncated=false.
• P2: per-domain check_consolidation_key uses the truncated domain, so two checker-valid long domains with the same 256-character prefix merge into one generic top-level check.
• P2: NUL-delimited identity hashing is ambiguous because protobuf strings may contain NUL. Distinct tuples such as ("", "custom_op\0x", "") and ("", "custom_op", "x\0") produce the same discriminator.

Use raw identities or collision-resistant digests for dedup, length-frame every hashed field, and truncate only display values. Add public JSON/check/SARIF regressions for all three boundaries. Independent report: /Users/mdangelo/modelaudit-pr-swarm/outputs/pr-1656-independent-review-20260611.md.

[chatgpt-codex-connector]

Summary

• Added length-framed SHA-256 identity hashing for ONNX custom operator tuples and custom domains, avoiding NUL-delimiter ambiguity while hashing raw values before truncation. modelaudit/scanners/onnx_scanner.pyL426-L440
• Updated custom-operator aggregation to deduplicate/count raw (domain, op_type, overload) identities, while storing bounded display fields plus an operator_identity_hash in public evidence. modelaudit/scanners/onnx_scanner.pyL454-L482
• Switched per-domain check_consolidation_key generation to a raw-domain digest so long domains with identical 256-character display prefixes remain distinct through core consolidation. modelaudit/scanners/onnx_scanner.pyL2965-L2982
• Added/updated ONNX regressions covering check, JSON, and SARIF preservation for custom-domain overload evidence, long op-type truncation boundaries, long domain consolidation boundaries, and NUL-containing identity tuples. tests/scanners/test_onnx_scanner.pyL2430-L2496 tests/scanners/test_onnx_scanner.pyL2499-L2650

Commit / PR

• Committed changes on the current branch: a38fd8a fix: hash onnx custom identities before truncation.
• Created PR: fix: hash ONNX custom identities before truncation.

Testing

• ✅ uv run ruff format --check modelaudit/ packages/modelaudit-picklescan/src packages/modelaudit-picklescan/tests tests/
• ✅ uv run ruff check modelaudit/ packages/modelaudit-picklescan/src packages/modelaudit-picklescan/tests tests/
• ✅ uv run mypy modelaudit/ packages/modelaudit-picklescan/src packages/modelaudit-picklescan/tests tests/
• ✅ uv run python -m py_compile modelaudit/scanners/onnx_scanner.py tests/scanners/test_onnx_scanner.py
• ✅ uv run python - <<'PY' ... _custom_operator_identity_hash('', 'custom_op\0x', '') != _custom_operator_identity_hash('', 'custom_op', 'x\0') ... PY
• ⚠️ uv sync --extra all-ci (warning due environment/network tunnel failure fetching nvidia-nvjitlink)
• ⚠️ uv sync --extra onnx (warning due environment/network tunnel failure fetching the onnx wheel)
• ⚠️ PROMPTFOO_DISABLE_TELEMETRY=1 uv run pytest tests/scanners/test_onnx_scanner.py -k "operator_identity_uses_raw_values_before_truncation or long_custom_domains_survive_core_check_consolidation or identity_hash_length_frames_nul_fields or custom_domain_custom_op_overloads_survive_json_and_sarif_serialization" -m "not slow and not integration" --maxfail=1 (warning: ONNX is not installed locally; collection skipped with No module named 'onnx')

View task →

[mldangelo-oai]

Addressed the exact-head collision blockers in ea8c7cc115fdb59c048ec9dddc4f7ace469b22bb: ONNX custom operator identity hashes are now length-framed SHA-256 digests, domain consolidation keys use raw-domain hashes, domain messages include the digest so issue/SARIF dedup stays distinct, and domain aggregate identity counting uses raw (domain, op_type, overload) while keeping display values bounded. Added JSON/check/SARIF regressions for long operator identities, long domain collisions, and NUL-containing identity tuples. Validation rerun locally: focused collision tests 7 passed, full ONNX scanner file 258 passed/1 deselected, pinned HF multilingual test 1 passed, full Ruff check/format, full mypy, and broad pytest 18,551 passed/793 skipped. @codex review

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. Breezy!

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[mldangelo-oai]

Independent merge-gate review: promptfoo/modelaudit PR #1656

Reviewed live state at 2026-06-11T05:29:47Z.

Findings

No actionable changed-code finding survived adversarial source review, security diff discovery, or runtime validation on the exact current head.

I found no P0, P1, P2, or P3 defect introduced by this PR. The previously reported identity-loss and collision defects are fixed at the reviewed head and covered through final JSON, check consolidation, and SARIF output.

Changed-code versus pre-existing issues

• Changed code: no validated regression or security weakening found.
• Pre-existing code: no separate actionable pre-existing issue was discovered within the supporting context reviewed for this PR.
• Residual integration risk: PR fix: calibrate onnx custom domain findings #1639 is still open and overlaps ONNX custom-domain classification. It is not in this PR or its base. If fix: calibrate onnx custom domain findings #1639 lands first, PR fix: deduplicate onnx custom domain findings #1656 should be rebased and the custom-domain/pinned-model suite rerun because both changes touch _check_custom_ops() policy behavior.

Merge disposition

Code disposition: approve after live gates clear. The exact head is technically merge-ready based on this review, but GitHub currently reports BLOCKED / REVIEW_REQUIRED. At the review snapshot, Python 3.10 and Windows test jobs were still running; no check was failing. Do not merge until those exact-head jobs complete successfully and the repository’s required approving review is present.

This is a conditional approval, not an unconditional merge authorization while GitHub’s gates remain blocked.

Exact target

Field	Value
Repository	promptfoo/modelaudit
PR	#1656
Title	fix: deduplicate onnx custom domain findings
State	open, non-draft
Base branch	main
Base SHA	8d6c4864fe2ea833ceaef1b9803d225afb1e8d69
Head branch	mdangelo/codex/hf-fp-t31-onnx-custom-domain-dedup-20260610
Head SHA	ea8c7cc115fdb59c048ec9dddc4f7ace469b22bb
Mergeability	MERGEABLE
Merge state	BLOCKED
Review decision	REVIEW_REQUIRED
Diff	4 files, +858 / -30

The review used a detached clone at /private/tmp/modelaudit-pr1656-independent-ea8c7cc; the user’s primary checkout was not accessed or modified.

Change assessment

The PR changes reporting after ONNX operator classification:

• repeated external custom operators are aggregated once per raw domain and scanned file;
• bounded samples and representative nodes retain useful evidence without one finding per node;
• raw (domain, op_type, overload) identities are length-framed and SHA-256 hashed before display truncation;
• per-domain and explicit-identity consolidation keys preserve distinct checks through core aggregation;
• issue messages include the raw-identity digest so final issue deduplication and SARIF fingerprints remain distinct;
• ONNX domain trust, schema validation, local-function handling, and Python-operator classification are unchanged.

The generic check_consolidation_key support in modelaudit/core_results.py only partitions existing per-asset check groups. It does not alter issue generation, severity, security-finding precedence, or exit-code behavior.

Prior feedback reconciliation

GitHub GraphQL returned seven review threads: 7 resolved, 0 unresolved. Four resolved threads remain attached to current lines; three are outdated. All formal bot reviews are COMMENTED, not approvals.

Prior blocker	Current-head disposition	Evidence
Per-domain aggregation could lose distinct operator/overload identity	Resolved	Domain aggregates retain bounded raw-identity-derived hashes and a truncation signal; overload regression passes through JSON and SARIF.
Multiple custom domains collapsed during core check consolidation	Resolved	check_consolidation_key now includes the full raw-domain SHA-256 digest; long-prefix collision regression passes.
Standard-domain explicit custom_op overloads deduplicated in final output	Resolved	Explicit findings include domain, overload, full identity hash, and distinct message/fingerprint evidence.
Custom-domain explicit custom_op overloads collapsed to one opaque domain record	Resolved	The one per-domain record contains bounded operator_identities with per-identity hashes and a distinct-count/truncation contract.
Long explicit operator names collided after 256-character display truncation	Resolved	Dedup uses raw tuples and message identity uses full SHA-256; display truncation no longer decides identity.
Long custom domains collided in check_consolidation_key	Resolved	Consolidation key hashes the raw domain before truncation.
NUL-delimited tuple hashing was ambiguous	Resolved	Each UTF-8 field is length-framed with an 8-byte length before hashing; the NUL-boundary regression passes.
Earlier independent issue comment: explicit domain/overload identity was lost	Resolved	Fixed beginning with 0555e696; exact-head JSON/SARIF tests remain green.
Exact-head issue comment containing three collision blockers	Resolved	Fixed by ea8c7cc; the current Codex review comment reports no major issue, and independent tests below reproduce the fixes.

No prior blocker remains actionable on ea8c7cc115fdb59c048ec9dddc4f7ace469b22bb.

Security review

The security-sensitive surface is attacker-controlled ONNX protobuf metadata flowing through scanner classification, aggregation, final issue/check deduplication, and SARIF serialization.

No security candidate survived discovery. Specifically:

• no domain was newly trusted or allowlisted;
• custom operators are still classified before aggregation;
• each scanned file remains a separate asset group;
• distinct raw domains remain distinct through SHA-256 consolidation keys and issue messages;
• explicit standard-domain operator identities remain distinct through raw tuple hashes;
• external custom-domain identities remain available as bounded samples with explicit truncation indicators;
• Python-like operators still generate independent CRITICAL S902 checks;
• malformed/missing-structure ONNX input remains unsuccessful and exit code 2;
• failed INFO S1111 aggregation does not suppress WARNING/CRITICAL findings or alter security exit-code precedence.

Security diff coverage closed 2/2 source-file worklist rows with no candidate findings. The validated security scan artifacts are:

• Markdown: /tmp/codex-security-scans/modelaudit/ea8c7cc115fdb59c048ec9dddc4f7ace469b22bb_20260611T051812Z/report.md
• HTML: /tmp/codex-security-scans/modelaudit/ea8c7cc115fdb59c048ec9dddc4f7ace469b22bb_20260611T051812Z/report.html

Independent validation

All commands used the detached exact-head clone with PYTHONPATH forced to that clone and telemetry disabled.

Validation	Result
git diff --check 8d6c486...HEAD	clean
Focused custom_domain or custom_op or pyop ONNX tests	27 passed, 232 deselected
Full non-slow/non-integration ONNX scanner test file	258 passed, 1 deselected, 1 warning
Ruff check on changed Python/test files	passed
Ruff format check on changed Python/test files	3 files already formatted
Mypy on changed Python/test files	passed, no issues
Security worklist reconciliation	2 rows, 2 completion receipts, closed
Security report validator	passed
Security HTML renderer	passed

Pinned Hugging Face ONNX model

Pinned artifact:

• repository: intfloat/multilingual-e5-large
• revision: 3d7cfbdacd47fdda877c5cd8a79fbcc4f2a574f3
• file: onnx/model_O4.onnx
• size: approximately 80.6 KB

Independent base/head comparison:

Revision	com.microsoft nodes	S1111 custom-domain checks	Evidence
Base 8d6c486	96	96	Attention=24, FastGelu=24, SkipLayerNormalization=48
Head ea8c7cc	96	1	same operator inventory, occurrence_count=96

The exact-head opt-in integration test passed: 1 passed, 258 deselected. The first attempt was blocked by the local SOCKS proxy dependency; rerunning with only the unsupported SOCKS proxy variables unset used the configured HTTP proxy and passed.

Malformed and fail-closed controls

A separate adversarial model contained 25 distinct custom operators plus a PyOp in one external domain and omitted the custom-domain opset import. Exact-head results:

• one bounded domain aggregate;
• occurrence_count=26;
• identity lower bound 20 with distinct_operator_identity_count_truncated=true;
• one independent CRITICAL Python-operator check;
• aggregate exit code 1.

A serialized ONNX ModelProto with ir_version=8 and no graph remained unsuccessful with aggregate exit code 2.

The changed regression suite also covers missing opset imports, external-data findings, mixed domains, multiple files, long names/domains, NUL-containing identities, final JSON, check consolidation, and SARIF fingerprints.

CI status

Snapshot for exact head ea8c7cc115fdb59c048ec9dddc4f7ace469b22bb:

• No failed checks.
• Pending: Test Python 3.10, Windows Tests (Python 3.11).
• Newly completed during this review: Test Python 3.13 passed in 26m12s.
• Passed: Quick Feedback Python 3.12, type check, lint/format, package build, dependency audit, vendored-proto checks, CodeQL, Python/GitHub Actions analysis, Docker lanes, documentation formatting, title validation, and benchmarks.
• Skipped by workflow conditions: NumPy matrix, extras smoke, license, lockfile, standalone pickle package, and full Docker image lanes.
• Performance benchmark result: 0 regressions across 12 shared benchmarks; aggregate median +1.4%.

Workflow run: 27324929590.

Remaining validation gaps

• The independent local run intentionally did not rerun the repository-wide 18k-test suite; the exact-head Quick Feedback CI job provides the broad fast-suite signal and passed.
• Python 3.10 and Windows CI were still in progress at the final snapshot; Python 3.13 completed successfully during the review. Their completion is a hard merge prerequisite.
• No separate Windows machine or alternate ONNX version was exercised locally.
• PR fix: calibrate onnx custom domain findings #1639 remains an overlapping, unmerged change and requires rebase/revalidation if it changes the base before fix: deduplicate onnx custom domain findings #1656 merges.