fix: paginate large Hugging Face inventories by mldangelo-oai · Pull Request #1659 · promptfoo/modelaudit

mldangelo-oai · 2026-06-11T03:10:20Z

Summary

Fixes ModelAudit false-positive task 32 by replacing the hard include_all_files streaming failure above 128 unfiltered Hugging Face candidates with a bounded, deterministic, revision-pinned repository inventory.

Root cause: default/header-routed Hugging Face streaming uses include_all_files=True; _select_streamable_hf_files() counted every unknown or extensionless filename against _MAX_HF_STREAMING_UNFILTERED_FILES = 128 and raised before any scan or aggregate size accounting. The pinned xai-org/grok-1 revision declares 773 files, so it failed before reaching a truthful bounded terminal outcome.

Security tradeoffs

Inventory is now fetched from the Hugging Face tree API at an immutable resolved commit SHA, across all pages, before any download starts.
Repository filenames are validated, deduplicated, sorted deterministically, and bounded by explicit inventory/page caps.
The old unfiltered 128-file abort is removed only after inventory acceptance; extensionless candidate caps, content-sniff file/byte caps, scanner selection filters, and no-scannable fail-closed behavior remain intact.
Selected-file size metadata is batched and conflict-checked before transfer, preserving aggregate --max-size enforcement for large selections.
Malformed/truncated/unstable paginated inventories fail closed before download rather than producing partial clean coverage.

Pinned real-model QA

Pre-fix proof on exact main SHA 8d6c4864fe2ea833ceaef1b9803d225afb1e8d69:

HF_HUB_DISABLE_IMPLICIT_TOKEN=1 PROMPTFOO_DISABLE_TELEMETRY=1 uv run python - <<'PY'
from huggingface_hub import HfApi
from modelaudit.utils.sources.huggingface import _select_streamable_hf_files
repo_id = "xai-org/grok-1"
revision = "5de83eb225f49624b424f1c8aa74f96983b5885c"
files = HfApi().list_repo_files(repo_id, repo_type="model", revision=revision)
print("declared_files", len(files))
try:
    _select_streamable_hf_files(repo_id, files, revision, include_all_files=True)
except Exception as exc:
    print(type(exc).__name__, str(exc))
PY

Outcome before fix: declared_files 773; Refusing to stream-download unfiltered files ... bounded unfiltered candidate limit (128).

Post-fix selector proof:

HF_HUB_DISABLE_IMPLICIT_TOKEN=1 PROMPTFOO_DISABLE_TELEMETRY=1 uv run python - <<'PY'
from huggingface_hub import HfApi
from modelaudit.utils.sources.huggingface import _select_streamable_hf_files
repo_id = "xai-org/grok-1"
revision = "5de83eb225f49624b424f1c8aa74f96983b5885c"
files = HfApi().list_repo_files(repo_id, repo_type="model", revision=revision)
selected = _select_streamable_hf_files(repo_id, files, revision, include_all_files=True)
print("declared_files", len(files))
print("selected_stream_files", len(selected))
PY

Outcome after fix: declared_files 773; selected_stream_files 773.

Post-fix all-format bounded planning proof:

HF_HUB_DISABLE_IMPLICIT_TOKEN=1 PROMPTFOO_DISABLE_TELEMETRY=1 uv run python - <<'PY'
from modelaudit.utils.sources.huggingface import (
    _ensure_huggingface_selection_within_max_size,
    _list_huggingface_repo_files_at_revision,
    _select_streamable_hf_files,
)
repo_id = "xai-org/grok-1"
revision = "5de83eb225f49624b424f1c8aa74f96983b5885c"
files, resolved_revision = _list_huggingface_repo_files_at_revision(repo_id, requested_revision=revision, timeout_seconds=30)
selected = _select_streamable_hf_files(repo_id, files, resolved_revision, include_all_files=True)
print("declared_files", len(files))
print("selected_stream_files", len(selected))
try:
    _ensure_huggingface_selection_within_max_size(repo_id, selected, 10 * 1024, resolved_revision=resolved_revision)
except Exception as exc:
    print("terminal_outcome", type(exc).__name__, str(exc))
PY

Outcome: declared_files 773; selected_stream_files 773; terminal outcome is an aggregate max-size refusal at 3221229327 bytes > 10240 bytes, before downloading weights.

Metadata-only real streaming test:

HF_HUB_DISABLE_IMPLICIT_TOKEN=1 PROMPTFOO_DISABLE_TELEMETRY=1 uv run pytest tests/utils/sources/test_huggingface.py::TestModelDownloadStreaming::test_pinned_grok_large_inventory_metadata_streaming_reaches_terminal -q

Outcome: 1 passed; fetched the pinned 773-file inventory and downloaded only README.md under a 10KB cap.

Validation

PROMPTFOO_DISABLE_TELEMETRY=1 uv run pytest tests/utils/sources/test_huggingface.py -q -m 'not integration'
# 278 passed, 1 skipped, 1 deselected

PROMPTFOO_DISABLE_TELEMETRY=1 uv run pytest tests/test_scanner_selection.py -q
# 42 passed, 2 skipped

PROMPTFOO_DISABLE_TELEMETRY=1 uv run pytest tests/test_streaming_scan.py -q
# 80 passed

PROMPTFOO_DISABLE_TELEMETRY=1 uv run pytest tests/test_cli.py -q -k 'huggingface or stream'
# 43 passed, 214 deselected

uv run ruff format modelaudit/ packages/modelaudit-picklescan/src packages/modelaudit-picklescan/tests tests/
uv run ruff check --fix modelaudit/ packages/modelaudit-picklescan/src packages/modelaudit-picklescan/tests tests/
uv run mypy modelaudit/ packages/modelaudit-picklescan/src packages/modelaudit-picklescan/tests tests/
# Success: no issues found in 474 source files

PROMPTFOO_DISABLE_TELEMETRY=1 uv run pytest -n auto -m "not slow and not integration" --maxfail=1
# 17377 passed, 1292 skipped, 39 warnings

uv run ruff check modelaudit/ packages/modelaudit-picklescan/src packages/modelaudit-picklescan/tests tests/
uv run ruff format --check modelaudit/ packages/modelaudit-picklescan/src packages/modelaudit-picklescan/tests tests/
git diff --check

Malformed controls added

Duplicate paginated names dedupe deterministically.
Traversal, absolute, empty-component, backslash, and NUL/control filenames fail closed.
Repeated pagination cursors fail closed.
Later page failures/truncation fail closed before download.
Excessive inventory totals fail closed via explicit bounds.
Conflicting duplicate selected-file size metadata fails closed before transfer.
Existing extensionless and renamed-content probe caps remain covered.

mldangelo-oai · 2026-06-11T03:11:07Z

@codex review

github-actions · 2026-06-11T03:12:39Z

Workflow run and artifacts

Performance Benchmarks

Compared 12 shared benchmarks with a regression threshold of 15%.
Status: 0 regressions, 0 improved, 12 stable, 0 new, 0 missing.
Aggregate shared-benchmark median: 1.461s -> 1.468s (+0.5%).

Workload	Benchmark	Target	Size	Files	Baseline	Current	Change	Status
`warm-cache-rescan`	`tests/benchmarks/test_scan_benchmarks.py::test_scan_warm_cached_repository_rescan`	`release-candidate`	547.3 KiB	32	128.30ms	123.27ms	-3.9%	stable
`single-checkpoint-preflight`	`tests/benchmarks/test_scan_benchmarks.py::test_scan_single_checkpoint_before_load`	`single_checkpoint.pkl`	183.0 KiB	1	74.39ms	71.90ms	-3.3%	stable
`mixed-model-repository`	`tests/benchmarks/test_scan_benchmarks.py::test_scan_release_candidate_repository`	`release-candidate`	547.3 KiB	32	490.01ms	499.17ms	+1.9%	stable
`nested-payload-review`	`tests/benchmarks/test_picklescan_benchmarks.py::test_picklescan_nested_payload_review[nested_base64]`	`nested_base64`	98 B	1	592.9us	600.8us	+1.3%	stable
`duplicate-heavy-registry`	`tests/benchmarks/test_scan_benchmarks.py::test_scan_duplicate_registry_snapshot`	`registry-snapshot`	915.2 KiB	13	398.22ms	401.98ms	+0.9%	stable
`clean-training-checkpoint`	`tests/benchmarks/test_picklescan_benchmarks.py::test_picklescan_clean_training_checkpoint`	`safe_large`	278.2 KiB	1	109.26ms	110.20ms	+0.9%	stable
`nested-payload-review`	`tests/benchmarks/test_picklescan_benchmarks.py::test_picklescan_nested_payload_review[nested_hex]`	`nested_hex`	130 B	1	620.9us	624.8us	+0.6%	stable
`suspicious-pickle-intake`	`tests/benchmarks/test_scan_benchmarks.py::test_scan_suspicious_pickle_intake`	`suspicious-intake`	183.8 KiB	4	144.50ms	145.30ms	+0.6%	stable
`nested-payload-review`	`tests/benchmarks/test_picklescan_benchmarks.py::test_picklescan_nested_payload_review[nested_raw]`	`nested_raw`	78 B	1	596.3us	594.6us	-0.3%	stable
`direct-malicious-upload`	`tests/benchmarks/test_picklescan_benchmarks.py::test_picklescan_direct_malicious_upload`	`malicious_reduce`	52 B	1	528.0us	527.1us	-0.2%	stable
`padded-multi-stream-upload`	`tests/benchmarks/test_picklescan_benchmarks.py::test_picklescan_padded_multi_stream_upload`	`multi_stream_padded`	4.1 KiB	1	649.1us	650.1us	+0.1%	stable
`chunked-upload-stream`	`tests/benchmarks/test_picklescan_benchmarks.py::test_picklescan_chunked_upload_stream`	`chunked_stream`	278.2 KiB	1	113.14ms	113.13ms	-0.0%	stable

chatgpt-codex-connector · 2026-06-11T03:14:01Z

Codex Review: Didn't find any major issues. Another round soon, please!

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

mldangelo-oai · 2026-06-11T03:30:45Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 713877a765

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

mldangelo-oai · 2026-06-11T04:10:17Z

@codex review

chatgpt-codex-connector · 2026-06-11T04:12:56Z

Codex Review: Didn't find any major issues. 👍

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

mldangelo-oai · 2026-06-11T04:19:09Z

Independent exact-head review of 45f27835c186f09266e7da415aec3b10ee5834db: request changes.

P1: _extract_huggingface_tree_page_files() silently skips every item whose type is not exactly file. A page containing benign.bin plus {"type":"unexpected","path":"malicious.pkl"} is accepted as a complete inventory containing only benign.bin. Missing/unknown types can therefore omit a malicious artifact while the remaining repository scans clean.

Explicitly accept known directory item types, validate file, and fail closed on missing/unknown types before any download. Add missing/unknown/new-type regressions. The pagination, immutable revision, duplicate, size metadata, and pinned 773-file inventory behavior otherwise validated successfully.

Full report: modelaudit-pr-swarm/outputs/pr-1659-independent-review-20260611.md.

@codex fix the type handling, add regressions, push, and request exact-head review after CI.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 45f27835c1

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

mldangelo-oai · 2026-06-11T04:47:48Z

@codex review

chatgpt-codex-connector · 2026-06-11T04:50:49Z

Codex Review: Didn't find any major issues. What shall we delve into next?

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

mldangelo-oai · 2026-06-11T06:48:17Z

@codex review

chatgpt-codex-connector · 2026-06-11T06:51:50Z

Codex Review: Didn't find any major issues. You're on a roll.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

…t32-hf-large-inventory-pagination-20260610 # Conflicts: # modelaudit/utils/sources/_huggingface_download_worker.py # modelaudit/utils/sources/huggingface.py # tests/utils/sources/test_huggingface.py

…t32-hf-large-inventory-pagination-20260610

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 51c35a0008

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

…t32-hf-large-inventory-pagination-20260610

…t32-hf-large-inventory-pagination-20260610 # Conflicts: # modelaudit/utils/sources/huggingface.py

…t32-hf-large-inventory-pagination-20260610

fix: paginate large Hugging Face inventories

d55a382

github-code-quality Bot found potential problems Jun 11, 2026

View reviewed changes

Comment thread tests/utils/sources/test_huggingface.py Fixed

test: use single Hugging Face import style

713877a

chatgpt-codex-connector Bot reviewed Jun 11, 2026

View reviewed changes

Comment thread modelaudit/utils/sources/huggingface.py Outdated

fix: allow Hugging Face cache blob symlinks

45f2783

chatgpt-codex-connector Bot reviewed Jun 11, 2026

View reviewed changes

Comment thread modelaudit/utils/sources/huggingface.py Outdated

fix: reject unknown Hugging Face tree item types

e643d43

mldangelo-oai added 3 commits June 11, 2026 06:21

fix: bound Hugging Face tree inventory pages

6b250e3

fix: use configured Hugging Face tree session

f7e4e95

fix: type legacy Hugging Face tree fallback

ec69832

mldangelo-oai requested a review from mldangelo June 11, 2026 07:20

mldangelo-oai enabled auto-merge (squash) June 11, 2026 07:20

mldangelo-oai added 3 commits June 12, 2026 00:58

Merge remote-tracking branch 'origin/main' into mdangelo/codex/hf-fp-…

48cd4db

…t32-hf-large-inventory-pagination-20260610 # Conflicts: # modelaudit/utils/sources/_huggingface_download_worker.py # modelaudit/utils/sources/huggingface.py # tests/utils/sources/test_huggingface.py

fix: validate huggingface pagination links

af025e1

Merge remote-tracking branch 'origin/main' into mdangelo/codex/hf-fp-…

51c35a0

…t32-hf-large-inventory-pagination-20260610

chatgpt-codex-connector Bot reviewed Jun 12, 2026

View reviewed changes

Comment thread modelaudit/utils/sources/huggingface.py

Comment thread modelaudit/utils/sources/huggingface.py

mldangelo-oai added 3 commits June 12, 2026 03:00

Merge remote-tracking branch 'origin/main' into mdangelo/codex/hf-fp-…

3b16ff0

…t32-hf-large-inventory-pagination-20260610

fix: bound huggingface include-all streaming

42513b8

Merge remote-tracking branch 'origin/main' into mdangelo/codex/hf-fp-…

2d5595a

…t32-hf-large-inventory-pagination-20260610 # Conflicts: # modelaudit/utils/sources/huggingface.py

mldangelo-oai added 3 commits June 12, 2026 03:33

Merge remote-tracking branch 'origin/main' into mdangelo/codex/hf-fp-…

62acb1d

…t32-hf-large-inventory-pagination-20260610

Merge remote-tracking branch 'origin/main' into mdangelo/codex/hf-fp-…

632a90c

…t32-hf-large-inventory-pagination-20260610

Merge remote-tracking branch 'origin/main' into mdangelo/codex/hf-fp-…

7438f9f

…t32-hf-large-inventory-pagination-20260610

mldangelo-oai merged commit e393dc0 into main Jun 12, 2026
29 checks passed

mldangelo-oai deleted the mdangelo/codex/hf-fp-t32-hf-large-inventory-pagination-20260610 branch June 12, 2026 05:06

Conversation

mldangelo-oai commented Jun 11, 2026

Summary

Security tradeoffs

Pinned real-model QA

Validation

Malformed controls added

Uh oh!

mldangelo-oai commented Jun 11, 2026

Uh oh!

github-actions Bot commented Jun 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Performance Benchmarks

Uh oh!

chatgpt-codex-connector Bot commented Jun 11, 2026

Uh oh!

Uh oh!

mldangelo-oai commented Jun 11, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

mldangelo-oai commented Jun 11, 2026

Uh oh!

chatgpt-codex-connector Bot commented Jun 11, 2026

Uh oh!

mldangelo-oai commented Jun 11, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

mldangelo-oai commented Jun 11, 2026

Uh oh!

chatgpt-codex-connector Bot commented Jun 11, 2026

Uh oh!

mldangelo-oai commented Jun 11, 2026

Uh oh!

chatgpt-codex-connector Bot commented Jun 11, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

github-actions Bot commented Jun 11, 2026 •

edited

Loading