Skip to content

Preflight coded geometry on the guarded RGB8 route - #10

Closed
laurens-pilot wants to merge 2 commits into
mainfrom
fix/hevc-rgb8-coded-prefixed-preflight
Closed

laurens-pilot wants to merge 2 commits into
mainfrom
fix/hevc-rgb8-coded-prefixed-preflight

Conversation

@laurens-pilot

Copy link
Copy Markdown
Owner

What

The guarded RGB8 coded route now runs the same property-validation and coded-geometry preflight the RGBA route uses, before decoding. Stacked on the framing fix (#8, upstream ente#20); the diff against main shows both commits.

Why

Every other guardrail-bearing HEVC route — coded RGBA/PNG, slices, image hooks, grids, alpha — checks the coded SPS geometry against max_pixels and the decoded-byte budget before pixel decoding. The RGB8 coded arm decoded first and only checked the cropped result afterwards, so a visible 16384×16384 SPS cropped to the ispe size allocated the full native planes under max_pixels and still returned a valid-looking image: measured, a 536,870,912-byte allocation under max_pixels: 1024 returning Ok(16x16) via decode_bytes_to_rgb8_with_guardrails, contradicting the documented coded-area guardrail ("cropping cannot hide an oversized coded frame"). No in-repo or ente caller passes guardrails on this route today.

Verification

  • Regression places the oversized SPS visibly and asserts PixelCountExceeded with the largest allocation under 1 MiB (fails at the parent commit with the allocation above); a clean RGB8 control still decodes.
  • fmt, clippy --all-features --all-targets -- -D warnings, all-features and no-default-features tests green on Rust 1.90.0; 272-file corpus unchanged.

Stream assembly always writes four-byte length-prefixed NAL units, but the
backend re-detected the framing per stream. A first NAL unit of length 1 or
256-511 produces a length prefix that looks like an Annex B start code, and
the detector then scanned start codes across the whole stream, exposing
parameter sets hidden inside any NAL to the backend while every estimator,
metadata walker and pixel-limit check saw only the visible NALs. A hidden SPS
coded 16384x16384 with a conformance window cropping to the visible geometry
allocated native planes far beyond max_pixels while returning a valid-looking
image (measured: a 536870912-byte allocation under max_pixels 1024 with a
16x16 primary).

Decode with fixed four-byte framing and remove the format detector, the
Annex B parser and the delegating wrapper. hevc::decode has one caller inside
a private module and no other code referenced the removed parsers. Streams
with these first-NAL lengths now decode strictly by their length prefixes:
their visible parameter sets decode, or they fail cleanly; previously most
failed while crafted ones decoded attacker-chosen hidden geometry.

Regressions cover the hidden-parameter fixture (fails at base with the
allocation above), first-NAL boundary lengths 1/255/256/511/512, the
auxiliary-alpha route and the grid-tile route; all compare pixels against
clean controls and bound the largest allocation. The 272-file corpus is
unchanged, including option-less PNG outputs.

Known pre-existing gap disclosed for a follow-up: the guarded RGB8 coded
route still lacks the SPS preflight the RGBA route has, so a visible
oversized cropped SPS can still allocate past max_pixels there (measured
536870912 bytes via decode_bytes_to_rgb8_with_guardrails); no in-repo or
ente caller passes guardrails on that route today.
The RGBA, slice, hook, grid and alpha routes check the coded SPS geometry
against max_pixels and the decoded-byte budget before decoding, but the RGB8
coded arm decoded first and only checked the cropped result afterwards. A
visible 16384x16384 SPS cropped to the ispe size therefore allocated the
full native planes under max_pixels (measured: 536870912 bytes via
decode_bytes_to_rgb8_with_guardrails with max_pixels 1024 returning a valid
16x16 image), contradicting the documented coded-area guardrail.

Run the same property-validation and geometry preflight the RGBA coded arm
uses before decoding. The regression places the oversized SPS visibly and
asserts PixelCountExceeded with the largest allocation under 1 MiB; a clean
RGB8 control still decodes.
@laurens-pilot

Copy link
Copy Markdown
Owner Author

@codex review

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 1, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-01T12:02:55.858795Z af7fa84 Manual request
🔒 Security Review ✅ Completed 2026-11-01T12:04:47.039274Z af7fa84 Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Another round soon, please!

Reviewed commit: af7fa84a20

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@chatgpt-codex-connector

Copy link
Copy Markdown

🛡️ Codex Security Review · Automatically triggered

Security review completed. No security issues were found in this pull request.

Reviewed commit: af7fa84a20

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

@laurens-pilot

Copy link
Copy Markdown
Owner Author

Superseded by ente#21.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants