Skip to content

gate: nothing checks that vendor/frankenphp/ is unmodified — the oracle is the one hard rule with no fail-closed check, and doc-citations rewards editing it #204

Description

@melihucar

Gate: bootstrap
Agent: codex

Hard rule 3 in orchestrator/prompts/shared.md:26 is "Do not modify
vendor/frankenphp/.
It is the oracle." Nothing enforces it.

Evidence

Every other hard rule in this project has a fail-closed gate step. This one has
none. The full scripts/gate.sh step list:

test-suite-intact        <- rule 1, "never weaken the gate to pass it"     (#58, #67, #68)
orchestrator-runnable    <- self-modification survivability
doc-citations            <- docs must cite upstream truthfully             (#20)
dev-env / build / fmt / clippy / test / conformance / bench-smoke / bench-report-selftest
                         ^ nothing in this list reads vendor/ as a thing to protect
$ grep -c vendor orchestrator/loop.py orchestrator/gh.py
orchestrator/loop.py:2      # both lines are "agent vendor" (codex/claude), not vendor/
orchestrator/gh.py:0

guard_root_writes (#168) guards stray writes to the main checkout. A vendor
edit made inside the agent's own worktree is in scope for that worktree, passes
every gate step, and merges.

The incentive is real, not hypothetical

scripts/check_doc_citations.py resolves each vendor/frankenphp/<file>:<line>
citation against the real file and fails when the range is out of bounds. There
are two ways to make a failing citation pass: fix the doc (correct), or add
lines to the cited vendor file (wrong, silent, green).

The checker's own docstring anticipates exactly this — "a vendor bump that
shifts lines without going out of bounds, or a careless edit, fails the
gate" — but its only defence is the hand-listed ANCHORS table, which pins a
handful of citations. The other 545 files in the vendor tree are unprotected.

#137, open, extends this checker to the 285 upstream citations in Rust
comments.
That multiplies the number of ways the check can fail, and so the
incentive, by roughly six, while leaving the escape hatch open.

Two things stop anyone noticing afterwards:

  • orchestrator/prompts/reviewer.md:37 tells the reviewer "Compare against
    vendor/frankenphp/". A reviewer reviewing a diff that edits vendor reads the
    edited oracle as ground truth, so the divergence it was asked to find is
    gone.
  • tests/conformance/ compares us to a container built from published upstream,
    not to the vendor tree, so it does not see a vendor edit either.

No vendor edit has happened yet — git log --oneline -- vendor/ is one commit,
the scaffold a5e88df. That is why this is p2 and not p0: it is a gate hole
with no trigger fired, not a demonstrated bad merge.

What to build

  1. scripts/check_vendor_pristine.sh, in the house style of
    scripts/check_test_suite.sh. It must:

    • fail on a committed vendor change: compare git rev-parse HEAD:vendor/frankenphp
      against a pin file;
    • fail on an uncommitted one: git status --porcelain -- vendor/ must be
      empty (the gate runs in a dirty worktree, so the hash alone is not enough);
    • read the pin from .gate/vendor-tree, following the .gate/min-test-files
      convention established by gate.sh's test-suite-intact check is blind to tests under crates/ #58. Seed it with the current value:
      a6119f8613aa71d4fb890c43d4a19dc98c40a016;
    • report the changed paths (git diff --name-only <pinned> HEAD -- vendor/
      plus the porcelain output), not a bare hash mismatch;
    • support --selftest, run before the real check, in the exact two-command
      shape test-suite-intact and doc-citations already use in gate.sh.
  2. step "vendor-pristine" in scripts/gate.sh, placed with the other
    toolchain-free checks (alongside test-suite-intact / doc-citations, before
    dev-env) so it runs in every profile, including bootstrap.

  3. The selftest must be able to fail — see prompts: no prompt asks whether a new test can fail — six blocked rounds and two disarmed gate checks, all the same defect #192. At minimum: one negative case
    that mutates a vendor-shaped file in a temp git tree and asserts non-zero
    exit, and one positive case on a clean tree asserting exit 0.

Acceptance

  • bash scripts/gate.sh bootstrap passes on a clean tree.
  • Appending one line to any file under vendor/frankenphp/ makes
    bash scripts/gate.sh bootstrap fail, naming that file. Paste the real
    command output in your final message, then revert the edit.
  • A deliberate vendor bump is still possible, but only by editing
    .gate/vendor-tree in the same commit — one visible, reviewable line in the
    diff instead of a silent 545-file blind spot.

Out of scope

Metadata

Metadata

Assignees

No one assigned

    Labels

    fr:metaImprovement to the loop itselffr:p2Normal -- the default when untriagedfr:readyClaimable by an agent

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions