Publish the 1 September matrix: 101/114, one execution date - #73
Merged
Conversation
The first clean full matrix under the current base prompt. 114 cells, six experiments, nineteen scenarios, every row executed on 1 September — which is what #66 asks for before publishing resumes, and the export confirms it: no carried rows, no provenance warnings, one execution date across the file. Replaces a snapshot from 24 August that was measured under the previous base prompt and produced by a local sweep rather than a CI run, so the published numbers now link to a workflow run a reader can open. Published by hand from the run's own artifacts rather than by the workflow. `publish-results` was skipped on 1 September because `EVALS_PUBLISH` was `false`; re-running that job today exported correctly — the log shows the same 114 rows and the same 101/13 split — and then failed to push. A re-run checks out the SHA the original run used, and main has moved six merges since, so the push was a non-fast-forward and the rebase-and-retry loop did not recover it. The artifacts are the same either way; only the commit path differs. Filed separately. What the run says: claude-code-sonnet-5 19/19 codex-gpt-5.6 18/19 codex-gpt-5.6-no-skills 18/19 claude-code-sonnet-5-no-skills 17/19 codex-gpt-5.4-mini-no-skills 15/19 codex-gpt-5.4-mini 14/19 Skills delta +2, 0 and -1. The weak model's figure has now read -3, -2 and -1 on successive clean measurements, so the sign has replicated and the magnitude has not. Thirteen failures, and the largest cluster is real: four of six arms failed `verification-001-stripe-express`, three of them on the same check. Triaged before publishing, because that is the shape a scorer defect makes. It is not one — exactly one run configured the Hookdeck source with the secret sitting in its own `.env`, and the four that failed used placeholders like `whsec_xxx` and `whsec_replace_me`. That is the failure this scenario was built to catch. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MQzUoMAwEBJWpEGVvVzSjK
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Clears the hold on #66.
What this publishes
The first clean full matrix under the current base prompt: 114 cells, six experiments, nineteen scenarios, every row executed on 1 September. The export reports no carried rows and no provenance warnings, and every
ranAtin the file is the same date — so this is a run, not a merge of six.It replaces a snapshot from 24 August that was measured under the previous base prompt and produced by a local sweep, not a CI run — its own provenance line said so. The published numbers now link to a workflow run a reader can open.
Skills delta +2, 0, −1. The weak model has now read −3, −2, −1 on successive clean measurements: the sign replicates, the magnitude does not.
The failure cluster was triaged before publishing
Four of six arms failed
verification-001-stripe-express, three on the identical check — the shape a scorer defect makes, and this repository has been wrong about that shape before. It is not one:.envwhsec_xxx,whsec_new,whsec_replace_me, …Exactly one run used the credential it was given. Full triage on #66.
Why this is a hand-publish
publish-resultswas skipped on 1 September becauseEVALS_PUBLISHwasfalse. Re-running that job today exported correctly — same 114 rows, same 101/13 split — and then failed to push: a re-run checks out the SHA of the original run, main has moved six merges since, and the rebase-and-retry loop did not recover the non-fast-forward. Same artifacts, different commit path. Worth fixing so a held run can be published later without this dance; filing separately.pnpm checkclean, formatting clean.🤖 Generated with Claude Code
https://claude.ai/code/session_01MQzUoMAwEBJWpEGVvVzSjK