Support workbench evals#27
Merged
Merged
Conversation
…ctionality) `evals/swarmkit/workbench-run.ts` (+ `npm run eval:workbench`): benchmark OpenTasks as a planning scaffold on WorkBench (olly-styles/WorkBench "Revisited"), by COMPOSING swarmkit-eval's WorkBench benchmark + grader + tool bridge with OpenTasks arms. No WorkBench-specific code here and NO change to OpenTasks core — all WorkBench machinery stays in swarmkit-eval. Arms: stock (WorkBench tools only) / notes / opentasks (+ the mcp__opentasks__* graph as a planning scaffold, composed via swarmkit-eval's workbenchNativeArms). Runs on base OpenTasks functionality: per-cell isolation via a RELATIVE OPENTASKS_PROJECT_DIR=.opentasks (resolves against each cell's cwd), and a post-run reap of only the daemons started from THIS build's dist/cli.js during the run (path-scoped + new-PID → never touches another project's daemon). Optional OPENTASKS_DAEMON_IDLE_TIMEOUT passthrough for self-reaping builds. Gateway (Bedrock) or ambient auth; report shows per-arm completion + harmful-action rates with 95% CIs + a paired stock-vs-opentasks Δ. First live smoke (3 email tasks, claude-haiku via Bedrock) ran clean end-to-end (stock 0.33 vs opentasks 1.00, 0 harmful, 0 env-errors; n=3, not significant — as expected). README section added. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
N agents coordinate on one real WorkBench multi_domain task, sharing one workspace (so their per-agent WorkBench MCP servers append to ONE union action log) + one OpenTasks daemon; grading replays the UNION of side-effecting actions, so a duplicate side effect from two uncoordinated agents = a WorkBench harmful action — the thing claim_next prevents. Composes swarmkit-eval's marble engine + WorkBench benchmark/grader with OpenTasks arms; no swarmkit-eval or OpenTasks-core change. Base functionality only (relative OPENTASKS_PROJECT_DIR; the shared daemon is seeded before agents and reaped by the service's stop() + a path-scoped post-run sweep). - workbench-marble.ts: MarbleBenchmarkAdapter. A workbench service seeds one CLAIMABLE opentasks subtask per PUBLIC domain (a fair, non-leaking work split), exposes the task text to the per-agent prompt, snapshots the union .wb_actions.jsonl in readState(); score() reuses WorkbenchGrader on the union log; coordination() classifies side-effecting workbench calls as work (dedup key = the action) and opentasks calls as coordination → R/O/c/E_c (+ A_e via EVAL_SOLO). - workbench-marble-run.ts (npm run eval:workbench:marble): stock/notes/opentasks arms, gateway/ambient. - Fix: taskLimit was clobbered by an unset config.taskLimit spread; now o.limit ?? opts.taskLimit, plus config.taskLimit set in the run. Validated (claude-haiku via Bedrock, N=2): machinery + coordination KPIs work. Cached signal — stock (n=210) completion 0.15 / harmful 0.84 / R 0.32 vs opentasks (n=11) 0.27 / 0.73 / 0.23: the coordination payoff (less redundancy → fewer harmful duplicates → higher completion). Known issue: a sporadic per-cell hang (1/12, ~970s, outside the agent timeout) to root-cause before a large run. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…matched node Two stacked bugs silently disabled OpenTasks coordination in the marble arm (every prior run read a false Δ=0.000 with both agents duplicating work): 1. Socket path: the per-cell daemon socket lived under the deep in-process workspace, past macOS's 103-byte sun_path limit → silent bind failure. Now run ONE daemon on a short /tmp/ote-XXXXXX/daemon.sock, publish it to ws.root/.ot_sock, and have each agent attach as a thin `mcp --socket` client (CooperBench pattern, OPENTASKS_NO_AUTOSTART=1). 2. Native-module ABI mismatch (the real killer): the eval runs under `npx tsx`, whose node (homebrew 23, NODE_MODULE_VERSION 131) differs from the nvm node 22 (127) that built opentasks' better-sqlite3. Spawning the daemon with process.execPath crashed it on `new Database()` (ERR_DLOPEN_FAILED) before it bound → 10s timeout. Add resolveOpentasksNode(), which probes candidate node binaries (npm_node_execpath, process.execPath, PATH node, $NVM_BIN, nvm/fnm sweeps) by actually instantiating a DB, and uses the ABI-compatible one for the daemon, seed/list, and the agents' MCP wrapper. Verified end-to-end under tsx: daemon boots, agents claim DISTINCT domains, redundancy R 0.25→0.00 and union side-effects halved (2→1) on duplication-prone tasks. No OpenTasks core change; all machinery stays in evals/swarmkit. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…geted oversampling The plain limit only takes the first-N tasks in file order, which mixes task classes (pure queries + hard-accuracy tasks) and masks the coordination signal. Add an optional taskIds filter (wb-<domain>-sha1(task)[:12]) so a run can target an exact curated slice — e.g. the single-action side-effect tasks where duplicate side effects are the binding constraint. Threaded through as EVAL_TASK_IDS. Used to prove the OpenTasks coordination payoff: on 12 single-agent-correct side-effect tasks, opentasks completion 0.92 vs stock 0.08 (Δ=+0.833, sig), harmful 0.00 vs 0.92 — stock's two agents duplicate the one action (WorkBench flags the dup harmful) while opentasks claims distinct domains and does it once. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add a Tier-2 (multi-agent marble) section documenting the entrypoint, the two gotchas that silently disabled coordination (deep-path socket > macOS 103-byte sun_path limit; better-sqlite3 ABI mismatch under npx tsx) and their fixes, and the experimental results: sample n opentasks stock Δ (paired) first-6 (mixed) 6 0.50 0.50 +0.000 single-action-correct 12 0.92 0.08 +0.833 ✓ single-action-correct (widened) 18 0.94 0.06 +0.889 ✓ (CI 0.72–1.00) On single-action-correct side-effect tasks, opentasks claim_next coordination turns a 0.06 completion into 0.94 (harmful 0.94→0.00) by preventing the duplicate side effect two uncoordinated agents produce. Invisible on the mixed default sample because WorkBench's harmful flag is duplication-count-insensitive. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.