docs: trace cookbook + trace-autopsy series opener + competition proposal draft - #285
Open
reacher-z wants to merge 1 commit into
Open
docs: trace cookbook + trace-autopsy series opener + competition proposal draft#285reacher-z wants to merge 1 commit into
reacher-z wants to merge 1 commit into
Conversation
…draft - docs/trace-cookbook.md: what researchers can build on the five-layer trace datasets (SFT/distillation, process rewards, failure taxonomies, security audits, judge research, site-drift), with download/load recipes; cites TurnOPD / ClawTrack / RedundancyBench as existing precedents. - docs/blog/: Trace Autopsy #1 — evidence-backed breakdown of an Opus 4.6 fail on the Uber Eats task (mid-verification abandonment lockout), every quote verified against run 001-...-claude-opus-4-6-20260329-110845 in the public V1Trace dump. Includes an X-thread version. - docs/proposals/clawbench-competition.md: internal draft for a WAB shared task / NeurIPS competition — not submitted anywhere yet.
Perry2004
requested changes
Aug 18, 2026
Perry2004
left a comment
Perry2004
left a comment
Collaborator
There was a problem hiding this comment.
Internal proposal leaked to PR: docs/proposals/clawbench-competition.md, please remove it.
Also I believe the website is a better place for the blog instead in the repo's documentation folder.
The trace-cookbook looks good to me.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Publicity workstreams #3/#4 (per maintainer's 2026-08-16 directive), plus the #6 competition proposal as an internal draft.
docs/blog/): why claude-opus-4-6 failed the Uber Eats task — it beat the bot wall, signup, and a two-tab OTP relay, then abandoned mandatory phone verification mid-flow and got the account locked by Uber's anti-fraud. Every quote is verbatim from the public trace (001-daily-life-food-uber-eats-claude-opus-4-6-20260329-110845); includes a ready-to-post X-thread version. Intended as the first of a series.docs/trace-cookbook.md): positions the V1/V2 trace datasets as a research resource (SFT/distillation, process rewards, failure taxonomies, security auditing, judge robustness, site drift) with copy-paste download/load recipes. Cites TurnOPD, ClawTrack, and RedundancyBench as existing external uses.docs/proposals/): WAB shared-task first, NeurIPS competition track second; organizers-run-everything execution model; budget envelope. Draft only — nothing has been sent to any organizer.Review focus: factual claims in the autopsy are all trace-backed, but tone/length are editorial calls; the cookbook's task counts use the paper numbers (153/130) — will sync with #283's corrected counts (281/163) if that merges first.