Skip to content

conformance: arm the frankenrust replay leg once the response writer is byte-exact, and measure which cases actually pass #200

Description

@melihucar

Gate: default
Agent: claude
Depends on: #159, #199

#199 turned the frankenrust conformance leg off behind a repo-pinned constant
in tests/conformance/lib/replay.py, because frankenrust fails all 25
non-skipped goldens on header casing alone. #159 replaces the response write
path with a byte-exact writer that fixes casing, ordering and chunking. This
issue is the one that flips the constant back and finds out what is left.

Without this, the leg stays off forever and the corpus never covers
frankenrust -- which is the hole #141 was filed against. This issue exists so
that hole is tracked rather than assumed.

Why this cannot be folded into #159

#159's acceptance is an in-crate test over a 13-case subset, and its body
says tests/conformance/run.sh "continues to run unchanged... Do not touch
it". The other 12 cases -- cookies-malformed, read-input, large-request,
autoloader, file-upload-get, file-upload-post, request-superglobal,
transition-regular, server-globals, exception,
server-all-vars-ordered, response-headers -- differ from the 13 on the
body, not the header block: $_FILES population, $_SERVER var set and
ordering, the exception stack trace. #159's writer does not speak to any of
those. Nobody has measured whether they pass, and that measurement is this
issue's actual deliverable.

What to do

  1. Flip FRANKENRUST_LEG_ENABLED (or whatever conformance: the frankenrust replay leg arms itself from host-global Docker state, and today fails all 25 cases on header casing alone #199 named it) to True in
    tests/conformance/lib/replay.py, and build frankenrust:bench from
    docker/frankenrust.Dockerfile (bench: docker/frankenrust.Dockerfile producing frankenrust:bench #15).
  2. Run bash tests/conformance/run.sh and record, per case, pass or the
    actual diff.
  3. For each case that still fails, do not fix the server here. Either it
    is covered by an existing open issue -- reference it in a per-case
    skip_targets = ["frankenrust"] / skip_reason in corpus.toml, in the
    shape finish-request already uses (corpus.toml:265-281) -- or it is
    not, in which case gh issue create for it first and reference that.
    Every skip names an owner.
  4. If the number of skips is large, read conformance: nothing bounds how many cases skip_targets may remove from a target #189 first: it is the issue about
    nothing bounding how many cases skip_targets may remove from a target.
    A leg that is on but skips 20 of 25 is not meaningfully on.

Acceptance

  • bash scripts/gate.sh default passes with the frankenrust leg armed.
  • The final line of the replay reports a compared count greater than zero for
    the frankenrust target, and every skipped case names an issue that owns it.
  • Your final message lists, case by case, which of the 25 pass and which are
    skipped with which owner. That table is the point of this issue.

Out of scope

Metadata

Metadata

Assignees

No one assigned

    Labels

    fr:followupFiled by an agent mid-taskfr:p2Normal -- the default when untriagedfr:readyClaimable by an agentfr:waitingReady, but waiting on an open dependency

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions