fix(review): structured review gate reads a terminated run as a clean pass - #2813
Open
chongwon83 wants to merge 1 commit into
Open
fix(review): structured review gate reads a terminated run as a clean pass#2813chongwon83 wants to merge 1 commit into
chongwon83 wants to merge 1 commit into
Conversation
… pass
The `codex review --base` pass in `DEFAULT_REVIEW_STEP` never captures the
wrapper's exit status, and its gate asks only whether `[P1]` appears:
_gstack_codex_timeout_wrapper 540 codex review --base <base> ... 2>"$TMPERR"
Check for `[P1]` markers: found → `GATE: FAIL`, not found → `GATE: PASS`.
So when the 540s wrapper terminates the review, output is empty, no `[P1]` is
present, and the run is recorded as `GATE: PASS` — then `gstack-review-log`
stores `"status":"clean"`. A review that produced nothing becomes evidence that
the branch was reviewed and found clean. The same is true for expired auth, a
rejected flag, or a model-entitlement 400: every non-completion reads as a pass.
The adversarial pass a few lines above already gets this right — it treats
exit 124 as "MISSING COVERAGE, not a pass". Only the structured path is
fail-open, which is why it went unnoticed.
Changes:
- Capture `_CODEX_EXIT=$?` after the wrapper (it was not captured at all, so the
gate had no way to know whether codex completed).
- Replace the single-condition gate with four rules, first match wins: non-zero
exit → FAIL, empty/whitespace output → FAIL, `[P1]` → FAIL, otherwise PASS.
- Ledger: `STATUS = "clean"` only when every pass COMPLETED. A run that hit
rule 1 or 2 is missing coverage and must not be logged as clean.
Rules 1 and 2 are the point: absence of a bad marker is not evidence of a good
result unless the check actually ran to completion.
Goldens regenerated (`factory-ship-SKILL.md`, 9 lines — gate text only).
`bun test test/host-config.test.ts test/skill-validation.test.ts
test/gen-skill-docs.test.ts` → 833 pass / 0 fail.
|
Merging to
After your PR is submitted to the merge queue, this comment will be automatically updated with its status. If the PR fails, failure details will also be posted here |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The
codex review --basepass inDEFAULT_REVIEW_STEPnever captures thewrapper's exit status, and its gate asks only whether
[P1]appears:So when the 540s wrapper terminates the review, output is empty, no
[P1]ispresent, and the run is recorded as
GATE: PASS— thengstack-review-logstores
"status":"clean". A review that produced nothing becomes evidence thatthe branch was reviewed and found clean. The same is true for expired auth, a
rejected flag, or a model-entitlement 400: every non-completion reads as a pass.
The adversarial pass a few lines above already gets this right — it treats
exit 124 as "MISSING COVERAGE, not a pass". Only the structured path is
fail-open, which is why it went unnoticed.
Changes:
_CODEX_EXIT=$?after the wrapper (it was not captured at all, so thegate had no way to know whether codex completed).
exit → FAIL, empty/whitespace output → FAIL,
[P1]→ FAIL, otherwise PASS.STATUS = "clean"only when every pass COMPLETED. A run that hitrule 1 or 2 is missing coverage and must not be logged as clean.
Rules 1 and 2 are the point: absence of a bad marker is not evidence of a good
result unless the check actually ran to completion.
Goldens regenerated (
factory-ship-SKILL.md, 9 lines — gate text only).bun test test/host-config.test.ts test/skill-validation.test.ts test/gen-skill-docs.test.ts→ 833 pass / 0 fail.