Add independently verified Codex reviews - #606
Conversation
Native review output can mix actionable defects, style notes, and false positives. Add a second read-only Codex pass that maps every native finding to evidence, runs only explicitly authorized validation commands, and fails closed when evidence or command accounting is ambiguous. Constraint: Native Codex review output is Markdown and verifier command execution is a local trust boundary Rejected: Trust a single review pass | no independent evidence or one-to-one finding accounting Confidence: high Scope-risk: moderate Reversibility: clean Directive: Keep verified review read-only and preserve exact --check command text Tested: 106/106 tests, npm run build, npm run check-version, git diff --check Not-tested: Live Windows app-server command action classification
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 307ff56224
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
Claude Code expands slash-command arguments before Bash runs, so raw input cannot safely be quoted inside the command template. Capture the exact input in a blocking expansion hook and pass only a session/workspace-bound, single-use UUID to the companion. Constraint: Claude Code substitutes $ARGUMENTS before Bash parses the command Rejected: Quote or base64 inside Bash | shell parsing occurs before safe encoding Confidence: high Scope-risk: moderate Reversibility: clean Directive: Never reintroduce raw slash arguments into verified-review shell commands Tested: 111/111 tests; focused security tests 3/3; npm run build; npm run check-version; git diff --check; Sol high independent review Not-tested: Claude Code versions without UserPromptExpansion support fail closed
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 3d59617c93
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
Inline long-option parsing discarded text after a second equals sign, while finding extraction erased indentation before deciding where findings began. Preserve full inline values and parse native review structure before normalizing its text. Constraint: Explicit check commands must remain exact user-authorized input Constraint: Native findings require stable one-to-one verification IDs Rejected: Reject inline values containing equals signs | valid environment-assignment checks need them Rejected: Normalize all review lines before parsing | indentation and fences carry finding structure Confidence: high Scope-risk: narrow Reversibility: clean Directive: Preserve raw option values and native review structure before normalization Tested: 114/114 tests; npm run build; npm run check-version; node syntax; git diff --check; Sol high independent review Not-tested: Unfenced column-zero diff markers remain syntactically ambiguous with top-level plus bullets
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 20316cd6be
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
The verified-review command protected companion invocation with captured input but still suggested interpolating a user-provided base ref during size estimation. Branch reviews now skip Bash sizing and use the safe background default; only fixed working-tree commands remain. Constraint: Raw slash-command refs must not enter model-generated shell commands Rejected: Add a companion sizing subcommand | background-by-default closes the boundary without new runtime surface Confidence: high Scope-risk: narrow Reversibility: clean Directive: Never interpolate raw base or ref values into verified-review preflight Bash Tested: 9/9 command tests; npm run build; npm run check-version; git diff --check; Sol high independent review Not-tested: Tiny branch reviews now intentionally skip size-based foreground recommendations
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 042b8d36ce
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
Reuse the canonical review result shape validator before classifying verified findings, so incomplete verifier JSON cannot produce a successful exit. Constraint: Verified review must not succeed without the complete result schema Rejected: Duplicate verifier-only validation because rendering already owns the canonical shape check Confidence: high Scope-risk: narrow Reversibility: clean Tested: 115/115 tests; npm run build; npm run check-version; git diff --check
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6bec4b2385
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
Validate every verified finding before accepting the verifier output. Record detached worker PIDs without overwriting cancellation, and route session cleanup through the same job lock. Constraint: Verified reviews must fail closed and background workers must not outlive their Claude session Rejected: New schema-validation dependency and worker handshake protocol Confidence: high Scope-risk: narrow Reversibility: clean Tested: 116/116 tests; npm run build; npm run check-version; git diff --check
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d3a1de1257
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
Serialize native findings as JSON so multiline evidence cannot become sibling prompt entries, and reject inverted verified-review line ranges before rendering. Constraint: Verified evidence must retain exact finding boundaries and valid source ranges Rejected: Custom escaping or schema-validation dependency Confidence: high Scope-risk: narrow Reversibility: clean Tested: 116/116 tests; npm run build; npm run check-version; git diff --check
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 981513e483
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| const shapeError = validateReviewResultShape(parsed.parsed); | ||
| if (shapeError) { | ||
| return invalidVerifiedReview(parsed, shapeError); |
There was a problem hiding this comment.
Fresh evidence beyond the earlier missing-field fix is that this path still only uses the loose renderer shape check, which accepts any non-empty verdict and any next_steps array contents; a verifier response like verdict: "bogus" with otherwise one-to-one findings therefore passes the remaining checks and exits successfully. In the verified-review flow that violates the supplied JSON schema/fail-closed contract and can publish an invalid “verified” result, so validate against VERIFIED_REVIEW_SCHEMA or mirror its enum and array-item checks before computing success.
Useful? React with 👍 / 👎.
1 participant
Why
A single review pass can mix real defects with style preferences, false positives, or findings that are difficult to reproduce. That leaves users choosing between acting on unverified feedback and manually re-investigating every finding.
This PR adds an explicit evidence step so review output is more trustworthy before code is changed or shipped.
What this adds
/codex:verified-review: one native Codex review followed by a fresh, ephemeral, read-only verification turn.confirmed,false-positive,style-only, orunverified.--check "<command>"values as the only user-authorized validation commands.Benefits
Safety and tradeoffs
--checktext is preserved exactly and only explicitly supplied commands count as authorized validation.Validation
npm run buildnpm run check-versiongit diff --checkKnown gap