diff --git a/evals/swarmkit/README.md b/evals/swarmkit/README.md index e8a9dd4..966f5b8 100644 --- a/evals/swarmkit/README.md +++ b/evals/swarmkit/README.md @@ -221,3 +221,123 @@ Harbor agents install and run **inside the sandbox**, so an OpenTasks-MCP arm ca `.mcp.json`/`.openswarm/mcp.json` pointing at the in-container MCP (the pattern `evals/tac/docker-adapter.ts` uses). That container-side wiring is the next step; the agent×model comparison above stands on its own (openswarm *is* the multi-agent/coordination story). + +## WorkBench × OpenTasks (`workbench-run.ts` · `npm run eval:workbench`) + +Benchmark OpenTasks — as a *planning/coordination scaffold* — on **WorkBench** (olly-styles/WorkBench +"Revisited": 690 outcome-graded workplace-agent tasks over calendar/email/analytics/CRM/project-board; +graded by replaying the agent's recorded side-effecting tool calls against a fresh sandbox, plus a +**harmful-action** flag). + +**Where the code lives.** Everything WorkBench-specific is in `swarmkit-eval`, not here: the tool bridge +(`wb_mcp.py`, WorkBench's 26 tools over MCP), the faithful `{kind:"workbench"}` grader (`wb_grade.py`, +WorkBench's own `is_correct`/`has_side_effects`), and the benchmark (`workbenchNativeBenchmark`). This +entrypoint only **composes** them with OpenTasks arms — no WorkBench knowledge here, and **no change to +OpenTasks core**. + +**Arms** (same model, same tasks; only the scaffold varies): + +| arm | scaffold | +|-----|----------| +| `stock` | WorkBench tools only (`mcp__workbench__*`) | +| `notes` | + a NOTES.md durable-log nudge | +| `opentasks` | + the OpenTasks MCP graph (`mcp__opentasks__*`) as a planning/decomposition scaffold | + +**Base OpenTasks functionality only — no daemon changes required:** +- **Isolation** is a relative `OPENTASKS_PROJECT_DIR=.opentasks` in the opentasks-MCP env → resolves + against each cell's workspace cwd → every cell gets its own `.opentasks/` + daemon. +- **Teardown**: after the run the entrypoint reaps **only** opentasks daemons started from *this build's* + `dist/cli.js` that appeared during the run (path-scoped + new-PID — never touches another project's + daemon). Optionally set `OPENTASKS_DAEMON_IDLE_TIMEOUT` to let them self-reap (a no-op on builds that + don't support it). + +**Setup:** `git clone …/WorkBench ~/GitHub/WorkBench && cd ~/GitHub/WorkBench && uv sync && uv pip install mcp`, +then `npm run build` here so the opentasks arm's MCP server (`dist/cli.js`) exists. + +```sh +# via the LiteLLM gateway (Bedrock, no local model); or drop WB_GATEWAY_BASE_URL for ambient Max-plan auth +WB_GATEWAY_BASE_URL=http://127.0.0.1:4000 WORKBENCH_LLM_API_KEY=sk-… AWS_REGION=us-east-1 \ + EVAL_ARMS=stock,opentasks EVAL_DOMAIN=multi_domain EVAL_TASK_LIMIT=20 EVAL_REPEATS=3 \ + npm run eval:workbench +``` + +The report (→ `evals/.swarmkit-workbench/report.md`) gives per-arm **completion** + **harmful-action** +rates with 95% CIs and a paired stock-vs-opentasks Δ. A first live smoke (3 email tasks, claude-haiku via +Bedrock) ran clean end-to-end: stock 0.33 vs opentasks 1.00 completion, 0 harmful, 0 env-errors — not +significant at n=3 (that's the point of the CIs). **Arena note:** single-domain tasks are 1–4 tool calls — +too short for a planning scaffold to move the needle; the real signal is on the `multi_domain` tasks and, +ultimately, multi-agent (Tier 2), where a duplicate side effect from two agents = a WorkBench harmful +action = the coordination payoff. + +## WorkBench × OpenTasks — Tier 2: multi-agent coordination (`workbench-marble-run.ts` · `npm run eval:workbench:marble`) + +Tier 1 puts WorkBench *behind* one OpenTasks-scaffolded agent; **Tier 2** puts N agents on ONE WorkBench +`multi_domain` task, sharing one sandbox + workspace, coordinating through OpenTasks' `claim_next`. Grading +replays the **union** of all agents' side-effecting tool calls — so a *duplicate* side effect (two agents +both send the same email) becomes a WorkBench **harmful action**. That duplication is exactly what +`claim_next` is meant to prevent → the coordination payoff, on realistic outcome-graded work. + +Runs on swarmkit-eval's native `marble` engine (`execution: 'marble'`): the WorkBench substrate is a +first-class *service* (`workbench-marble.ts`) that seeds one claimable subtask per public domain, and each +agent's prompt says claim → do that domain's WorkBench actions → close. Arms: `stock` (no channel), +`notes` (claims.txt), `opentasks` (`claim_next`). All WorkBench machinery stays in swarmkit-eval; the +marble adapter only composes. **No OpenTasks core change.** + +### Two gotchas that silently disabled coordination (both fixed) + +Symptom of either: `Daemon did not start within 10000ms` / `no daemon running`, both agents duplicate → a +false Δ=0. Isolation tests pass while the eval fails, because the login shell hides both. + +1. **Daemon socket path.** One daemon per cell on a SHORT `/tmp/ote-XXXXXX/daemon.sock` — a socket nested + under the deep in-process workspace exceeds macOS's 103-byte `sun_path` limit and silently fails to + bind. The path is published to `ws.root/.ot_sock`; each agent attaches as a thin `mcp --socket` client + (`OPENTASKS_NO_AUTOSTART=1`, the CooperBench pattern). +2. **Native-module ABI.** The eval runs under `npx tsx`, whose node can differ from the one that built + OpenTasks' `better-sqlite3` (e.g. homebrew 23 / `MODULE_VERSION` 131 vs nvm 22 / 127). Spawning the + daemon with `process.execPath` crashes it on `new Database()` (`ERR_DLOPEN_FAILED`) before it binds. + `resolveOpentasksNode()` probes candidate nodes (incl. nvm/fnm sweeps) by *instantiating* a DB and uses + the ABI-matched one for the daemon, seed/list, AND the agents' MCP wrapper. + +### Results — coordination decisively helps, but only on the right task class + +WorkBench's `harmful` flag is duplication-COUNT-insensitive (1 wrong action and 2 duplicate wrong actions +both = harmful), so coordination only moves the score on tasks where the required action is (a) +side-effecting AND (b) single-agent-CORRECT — there a duplicate flips pass→fail. The default first-N +sample buries those under pure-query tasks (both pass) and hard-accuracy tasks (both fail), reading Δ≈0. + +**Oversampling** (`EVAL_TASK_IDS`, id = `wb-multi_domain-sha1(task)[:12]`) targets that class. Of 210 +`multi_domain` tasks, 123 are single-action side-effecting; an N=1 scan finds which the model does +correctly *solo* (sonnet: send_email 10/10, create_task 7/7, add_customer 1/3, create_event 0/4 — the +time-computation trap fails). Running N=2 stock-vs-opentasks on those: + +| sample | n | opentasks compl | stock compl | Δ (paired) | opentasks harmful | stock harmful | redundancy R (ot/stock) | +|---|--:|--:|--:|--:|--:|--:|--:| +| first-6 (mixed) | 6 | 0.50 | 0.50 | **+0.000** | 0.50 | 0.50 | 0.00 / 0.25 | +| single-action-correct | 12 | 0.92 | 0.08 | +0.833 ✓ (CI 0.58–1.00) | 0.00 | 0.92 | 0.00 / 0.40 | +| single-action-correct (widened) | **18** | **0.94** | **0.06** | **+0.889** ✓ (CI 0.72–1.00) | 0.00 | 0.94 | 0.00 / 0.43 | + +_(sonnet via Bedrock gateway; the paired Δ CI excludes 0 on the targeted set, and tightens as it widens.)_ + +Mechanism (17/18 in the widened targeted set): stock's two agents BOTH perform the one action (union +side-effects ≈ 2) → the duplicate is harmful → fail; opentasks claims distinct domains → 1 action → +correct → pass (redundancy R 0.00 vs 0.43). The lone opentasks miss is an `add_customer` task whose domain +split (crm / project_management) didn't match the single required action — both agents updated the same +customer. Cost: opentasks ~2.4× tokens for the claim/close overhead. **Takeaway: to measure coordination, +oversample single-action-correct side-effect tasks; the mixed default sample hides the effect.** + +### Env (beyond the Tier-1 vars) + +| var | default | meaning | +|---|---|---| +| `EVAL_N` | `2` | agents per task | +| `EVAL_TASK_IDS` | — | comma list of exact `wb-*` ids to oversample (else first-N via `EVAL_TASK_LIMIT`) | +| `EVAL_SOLO` | — | `1` → also run an N=1 baseline for the A_e error-amplification KPI | +| `EVAL_DEBUG_DIR` | — | per-task dump: each agent's tool sequence + who-claimed-what + union actions | + +```sh +# targeted oversample — the run that shows the payoff (sonnet via Bedrock gateway): +WB_GATEWAY_BASE_URL=http://127.0.0.1:4000 WORKBENCH_LLM_API_KEY=sk-… AWS_REGION=us-east-1 \ + EVAL_MODEL=claude-sonnet EVAL_ARMS=stock,opentasks EVAL_N=2 \ + EVAL_TASK_IDS=wb-multi_domain-78b0f2e1b29a,wb-multi_domain-cc223e5fe99f,… \ + npm run eval:workbench:marble +``` diff --git a/evals/swarmkit/workbench-marble-run.ts b/evals/swarmkit/workbench-marble-run.ts new file mode 100644 index 0000000..cdb4c9e --- /dev/null +++ b/evals/swarmkit/workbench-marble-run.ts @@ -0,0 +1,186 @@ +/** + * Tier 2 — multi-agent WorkBench × OpenTasks through swarmkit-eval's native `marble` engine. + * + * N agents coordinate on ONE real WorkBench multi_domain task; grading replays the UNION of their + * side-effecting tool calls, so duplicate side effects (two agents both send the email) = WorkBench + * harmful actions. OpenTasks' claim_next is meant to prevent exactly that. stock vs opentasks (+ notes) + * with per-arm completion + harmful + coordination KPIs (R/O/c/E_c, and A_e with EVAL_SOLO). + * + * Base OpenTasks functionality only: the service starts ONE per-cell daemon on a SHORT /tmp socket (dodging + * the macOS 103-byte sun_path limit that a deep-workspace socket hits), publishes it to ws.root/.ot_sock, + * and every agent connects as a thin `mcp --socket` client (CooperBench pattern). The daemon is reaped by + * the service's stop() (belt-and-suspenders path-scoped reap after the run). No OpenTasks core change; all + * WorkBench machinery is in swarmkit-eval. + * + * Env: WORKBENCH_REPO · EVAL_MODEL · EVAL_ARMS(stock,opentasks) · EVAL_N(2) · EVAL_DOMAIN(multi_domain) · + * EVAL_TASK_LIMIT(5) · EVAL_CONCURRENCY(1) · EVAL_TIMEOUT(300000) · EVAL_SOLO=1 (A_e) · + * WB_GATEWAY_BASE_URL + WORKBENCH_LLM_API_KEY (Bedrock gateway) else ambient Max-plan auth. + * + * WB_GATEWAY_BASE_URL=http://127.0.0.1:4000 WORKBENCH_LLM_API_KEY=sk-… AWS_REGION=us-east-1 \ + * EVAL_N=2 EVAL_ARMS=stock,opentasks EVAL_TASK_LIMIT=5 npm run eval:workbench:marble + */ + +import * as path from 'node:path'; +import { execSync } from 'node:child_process'; +import { + runEval, + NativeCliAdapter, + InProcessBackend, + LocalResultStore, + buildReport, + renderMarkdownReport, + writeReport, + workbenchMcpServer, + workbenchNativeArms, + type EvalConfig, + type RunDeps, + type GatewayConfig, + type McpServerSpec, + type ExecutionAdapter, +} from 'swarmkit-eval'; +import { workbenchMarbleBenchmark, resolveOpentasksNode } from './workbench-marble.js'; +import { ARMS } from '../arms.js'; + +const WB_REPO = process.env.WORKBENCH_REPO ?? path.join(process.env.HOME ?? '', 'GitHub', 'WorkBench'); +const WB_PYTHON = path.join(WB_REPO, '.venv', 'bin', 'python'); +const GATEWAY_BASE = process.env.WB_GATEWAY_BASE_URL; +const MODEL = process.env.EVAL_MODEL ?? (GATEWAY_BASE ? 'claude-haiku' : 'haiku'); +const ARM_IDS = (process.env.EVAL_ARMS ?? 'stock,opentasks').split(',').map((s) => s.trim()); +const N = Number(process.env.EVAL_N ?? 2); +const DOMAIN = (process.env.EVAL_DOMAIN ?? 'multi_domain') as Parameters[0]['domain']; +// EVAL_TASK_IDS: oversample an exact set of `wb-*` ids (else first-N-in-file-order via EVAL_TASK_LIMIT). +const TASK_IDS = (process.env.EVAL_TASK_IDS ?? '').split(',').map((s) => s.trim()).filter(Boolean); +// When ids are pinned, run all of them (cap defaults to their count) unless a smaller limit is set. +const TASK_LIMIT = Number(process.env.EVAL_TASK_LIMIT ?? (TASK_IDS.length || 5)); +const CONCURRENCY = Number(process.env.EVAL_CONCURRENCY ?? 1); +const TIMEOUT = Number(process.env.EVAL_TIMEOUT ?? 300_000); +const OUT_DIR = path.resolve(process.cwd(), 'evals/.swarmkit-workbench-marble'); + +const OPENTASKS_CLI = ARMS.opentasks.mcp?.args?.[0]; + +/** The OpenTasks coordination primitives the multi-agent arm needs (claim/complete). */ +const OT_COORD_TOOLS = [ + 'mcp__opentasks__claim_next', + 'mcp__opentasks__get_task', + 'mcp__opentasks__update_task', + 'mcp__opentasks__list_tasks', + 'mcp__opentasks__release_task', +]; + +function opentasksMcpServer(): McpServerSpec { + const cli = OPENTASKS_CLI!; + // Each agent connects to the ONE per-cell daemon over the SHORT socket the service published to + // ws.root/.ot_sock (CooperBench pattern) — NOT a deep-path autostart, which silently fails the macOS + // 103-byte sun_path limit. cwd is the agent's cell (ws.root — confirmed by the WorkBench MCP action log + // landing there), so `cat .ot_sock` resolves the socket. NO_AUTOSTART keeps every agent a thin client. + // ABI-matched node (NOT the tsx process.execPath, which can't load opentasks' native better-sqlite3). + const node = resolveOpentasksNode(); + return { + name: 'opentasks', + command: 'sh', + args: ['-c', `exec "${node}" "${cli}" mcp --socket "$(cat .ot_sock)" --scope all`], + env: { OPENTASKS_NO_AUTOSTART: '1' }, + }; +} + +/** PIDs of opentasks daemons started from THIS build's CLI (path-scoped reap — never another project's). */ +function opentasksDaemonPids(): Set { + if (!OPENTASKS_CLI) return new Set(); + try { + const out = execSync('ps -Ao pid=,command=', { encoding: 'utf8' }); + const pids = new Set(); + for (const line of out.split('\n')) { + if (!line.includes(OPENTASKS_CLI) || !line.includes('daemon start')) continue; + const pid = parseInt(line.trim().split(/\s+/)[0]!, 10); + if (Number.isFinite(pid)) pids.add(pid); + } + return pids; + } catch { + return new Set(); + } +} + +async function main(): Promise { + const benchmark = workbenchMarbleBenchmark({ n: N, repoDir: WB_REPO, python: WB_PYTHON, domain: DOMAIN, taskLimit: TASK_LIMIT, ...(TASK_IDS.length ? { taskIds: TASK_IDS } : {}) }); + const wbMcp = workbenchMcpServer({ python: WB_PYTHON, repoDir: WB_REPO }); + // Marble arms: coordination lives in the phase prompt, so arms just carry the MCP servers + allow-list. + const arms = workbenchNativeArms(wbMcp, [ + { id: 'notes', label: 'notes (claims.txt)' }, + { id: 'opentasks', label: 'opentasks (claim_next)', mcpServers: [opentasksMcpServer()], extraTools: OT_COORD_TOOLS }, + ]).filter((a) => ARM_IDS.includes(a.id)); + if (!arms.length) throw new Error(`No arms matched EVAL_ARMS=${ARM_IDS.join(',')} (have stock,notes,opentasks)`); + + const gateway: GatewayConfig | undefined = GATEWAY_BASE ? { baseUrl: GATEWAY_BASE, fallbacksDisabled: true } : undefined; + const passEnv: Record = {}; + for (const k of ['AWS_REGION', 'AWS_PROFILE']) if (process.env[k]) passEnv[k] = process.env[k]!; + + const config: EvalConfig = { + runId: `workbench-marble-${DOMAIN}-${MODEL}-N${N}`, + configVersion: 'v1', + benchmark: benchmark.id, + arms, + models: [{ name: MODEL, ...(gateway ? { dialect: 'anthropic' as const } : {}) }], + seeds: [1], + backend: 'in-process', + concurrency: { cells: CONCURRENCY, modelConnections: CONCURRENCY }, + taskLimit: TASK_LIMIT, + output: { dir: OUT_DIR, trace: false }, + }; + + const agentAdapter: ExecutionAdapter = new NativeCliAdapter({ + defaultModel: MODEL, + timeoutMs: TIMEOUT, + env: passEnv, + ...(gateway ? { gateway, virtualKey: process.env.WORKBENCH_LLM_API_KEY ?? 'sk-eval' } : {}), + }); + + const deps: RunDeps = { + benchmark, + backend: new InProcessBackend(), + store: new LocalResultStore(OUT_DIR), + marble: { agentAdapter, maxParallelCells: CONCURRENCY }, + }; + + console.log( + `workbench MARBLE · model=${MODEL} arms=${arms.map((a) => a.id).join(',')} N=${N} domain=${DOMAIN} ` + + `tasks≤${TASK_LIMIT} concurrency=${CONCURRENCY}${gateway ? ' [gateway]' : ' [ambient]'}`, + ); + + const before = opentasksDaemonPids(); + let results; + try { + results = await runEval(config, deps); + } finally { + const now = opentasksDaemonPids(); + let reaped = 0; + for (const pid of now) if (!before.has(pid)) { try { process.kill(pid, 'SIGTERM'); reaped++; } catch { /* gone */ } } + if (reaped) console.log(`reaped ${reaped} stray per-cell opentasks daemon(s)`); + } + + for (const r of results) { + const m = r.score?.metrics ?? {}; + console.log( + ` ${r.armId.padEnd(9)} ${r.taskId}: status=${r.status} completion=${m.completion ?? '-'} harmful=${m.harmful ?? '-'} ` + + `unionSideEffects=${m.unionSideEffects ?? '-'} R=${m.R !== undefined ? (m.R as number).toFixed(2) : '-'} ` + + `tokens=${r.usage.totalTokens} ${Math.round(r.durationMs / 1000)}s${r.status === 'env_error' ? ` ENV:${r.envError?.kind}` : ''}`, + ); + } + + // A_e (error amplification) needs an N=1 solo baseline of the same arms (EVAL_SOLO=1). + let solo: typeof results | undefined; + if (process.env.EVAL_SOLO === '1') { + console.log(' … running N=1 solo baseline for A_e'); + solo = await runEval(config, { ...deps, marble: { agentAdapter, widthOverride: 1, maxParallelCells: CONCURRENCY } }); + } + + const baseline = ARM_IDS.includes('stock') ? 'stock' : ARM_IDS[0]; + const report = buildReport(results, config, { baselineArmId: baseline, accuracyMetric: 'successRate', solo }); + console.log('\n' + renderMarkdownReport(report)); + const paths = await writeReport(OUT_DIR, report); + console.log(`\nreport → ${path.relative(process.cwd(), paths.md)} (+ .html, .json)`); +} + +main().catch((e) => { + console.error(e); + process.exit(1); +}); diff --git a/evals/swarmkit/workbench-marble.ts b/evals/swarmkit/workbench-marble.ts new file mode 100644 index 0000000..caaa155 --- /dev/null +++ b/evals/swarmkit/workbench-marble.ts @@ -0,0 +1,343 @@ +/** + * Tier 2 — multi-agent WorkBench through swarmkit-eval's native `marble` engine. + * + * N agents share ONE WorkBench sandbox + ONE workspace (so their per-agent WorkBench MCP servers all + * append to ONE union action log, `ws.root/.wb_actions.jsonl`), coordinating through OpenTasks. The task + * is a real WorkBench multi_domain instruction; grading replays the UNION of all agents' side-effecting + * tool calls against a fresh sandbox — so a *duplicate* side effect from two uncoordinated agents (both + * send the email) becomes a WorkBench **harmful action**. That is exactly what OpenTasks' `claim_next` + * coordination is meant to prevent → the coordination payoff, measured on realistic, outcome-graded work. + * + * Structural port of `synth-marble.ts` (the emit-queue) to WorkBench: the "items" are the task's PUBLIC + * `domains` (a fair, non-leaking work split — the sealed ground-truth actions are never seeded), the + * "emit" is doing that domain's workplace actions via `mcp__workbench__*`, and the grader is WorkBench's + * own `is_correct`/`has_side_effects` (reused via the exported `WorkbenchGrader`). All WorkBench-specific + * machinery lives in swarmkit-eval; this file only composes. No change to OpenTasks core. + */ + +import { execFileSync } from 'node:child_process'; +import * as fs from 'node:fs'; +import * as os from 'node:os'; +import * as path from 'node:path'; +import { + workbenchNativeBenchmark, + WorkbenchGrader, + type MarbleBenchmarkAdapter, + type MultiAgentRawRun, + type Score, + type EvalTask, + type ServiceSpec, + type ServiceHandle, + type ServiceContext, + type PhaseSpec, + type CoordinationAction, + type Workspace, + type RawRun, + type LoadOpts, +} from 'swarmkit-eval'; +import { ARMS } from '../arms.js'; + +const OPENTASKS_CLI = ARMS.opentasks.mcp?.args?.[0]; + +/** + * Resolve a node binary whose ABI matches opentasks' native `better-sqlite3`. The eval runs under + * `npx tsx`, whose `process.execPath` can be a DIFFERENT node than the one that `npm install`'d opentasks + * (e.g. homebrew node 23 / MODULE_VERSION 131 vs the nvm node 22 / 127 the module was built against). A + * mismatched node makes the spawned daemon crash on `require('better-sqlite3')` (ERR_DLOPEN_FAILED) before + * it ever binds its socket — silently disabling ALL coordination. So probe candidates and pick the first + * that can actually load the module; every opentasks subprocess (daemon/seed/list AND the agents' MCP) + * must use it, not `process.execPath`. Memoized. Throws with a clear message if none work. + */ +let _otNode: string | undefined; + +/** Candidate node binaries, most-likely-correct first. Under `npx tsx`, process.execPath AND PATH `node` + * can BOTH be the wrong (homebrew) node, while the module was built by a versioned nvm/fnm node — so we + * also sweep those install roots (newest first) as fallbacks. */ +function nodeCandidates(): string[] { + const out: string[] = []; + const add = (p?: string): void => { if (p && !out.includes(p)) out.push(p); }; + add(process.env.npm_node_execpath); // the node `npm run` used to launch us + add(process.execPath); // the tsx host node + add('node'); // PATH + if (process.env.NVM_BIN) add(path.join(process.env.NVM_BIN, 'node')); + for (const root of [path.join(os.homedir(), '.nvm', 'versions', 'node'), path.join(os.homedir(), '.fnm', 'node-versions')]) { + try { for (const v of fs.readdirSync(root).sort().reverse()) { add(path.join(root, v, 'bin', 'node')); add(path.join(root, v, 'installation', 'bin', 'node')); } } catch { /* no such manager */ } + } + return out; +} + +export function resolveOpentasksNode(): string { + if (_otNode) return _otNode; + // Must INSTANTIATE a Database — `require` only loads the JS wrapper; the native .node is dlopen'd lazily + // on `new Database()`, which is where the ABI (NODE_MODULE_VERSION) mismatch actually throws. + const probe = "new (require('better-sqlite3'))(':memory:').close()"; + // Resolve better-sqlite3 from the opentasks repo's node_modules (OPENTASKS_CLI = /dist/cli.js). + const repoRoot = OPENTASKS_CLI ? path.dirname(path.dirname(OPENTASKS_CLI)) : process.cwd(); + for (const cand of nodeCandidates()) { + try { + execFileSync(cand, ['-e', probe], { cwd: repoRoot, stdio: 'ignore', timeout: 15_000 }); + _otNode = cand; + return cand; + } catch { /* ABI mismatch or not found → try next */ } + } + throw new Error( + 'No node binary can load opentasks\' better-sqlite3 (ABI/NODE_MODULE_VERSION mismatch on every candidate). ' + + 'Run the eval under the node that built opentasks, or `npm rebuild better-sqlite3`.', + ); +} + +/** The 14 side-effecting WorkBench tools (sanitized MCP names) — the graded, redundancy-tracked actions. */ +const WB_SIDE_EFFECT = new Set([ + 'email_send_email', 'email_delete_email', 'email_forward_email', 'email_reply_email', + 'calendar_create_event', 'calendar_delete_event', 'calendar_update_event', + 'analytics_create_plot', + 'project_management_create_task', 'project_management_delete_task', 'project_management_update_task', + 'customer_relationship_manager_add_customer', 'customer_relationship_manager_update_customer', + 'customer_relationship_manager_delete_customer', +]); + +export interface WorkbenchMarbleOpts { + n: number; + repoDir: string; + python: string; + domain?: Parameters[0]['domain']; + taskLimit?: number; + /** Restrict the run to these exact task ids (`wb--`). Used to oversample a + * curated slice — e.g. single-action side-effect tasks where duplication is the binding constraint — + * instead of the first-N-in-file-order the plain limit gives. When set, the full CSV is loaded and + * filtered to these ids (taskLimit still caps the count after filtering). */ + taskIds?: string[]; +} + +/** Parse the public `domains` cell ("['email', 'calendar']") into a string[]. */ +function parseDomains(s: unknown): string[] { + try { + const v = JSON.parse(String(s ?? '').replace(/'/g, '"')); + return Array.isArray(v) ? v.map(String) : []; + } catch { + return []; + } +} + +/** + * The WorkBench substrate as a first-class service: for the opentasks arm it seeds one CLAIMABLE task per + * public domain (a non-leaking partition; this also starts the shared per-cell daemon before the agents), + * exposes the task instruction to the per-agent prompt via `env`, snapshots the UNION action log for the + * grader in `readState()`, and reaps the daemon in `stop()` (base OpenTasks functionality). + */ +function workbenchService(): ServiceSpec { + return { + id: 'workbench', + async start(ctx: ServiceContext): Promise { + const armId = ctx.armId; + const instruction = String(ctx.publicMetadata?.instruction ?? ''); + const domains = parseDomains(ctx.publicMetadata?.domains); + + // opentasks arm: run ONE daemon on a SHORT socket path (under /tmp, NOT the deep in-process + // workspace — a socket nested under ws.root exceeds macOS's 103-byte sun_path limit and silently + // fails to bind, which was disabling coordination entirely). Publish the socket to ws.root/.ot_sock + // so each agent's MCP wrapper connects via `--socket` (the CooperBench pattern) instead of a fragile + // relative-path autostart. Seed one claimable subtask per public domain into that daemon. + let otHome: string | undefined; + const otEnvFor = (home: string): NodeJS.ProcessEnv => ({ ...process.env, OPENTASKS_PROJECT_DIR: home }); + if (armId === 'opentasks' && OPENTASKS_CLI) { + otHome = fs.mkdtempSync('/tmp/ote-'); // short → socket ≈35 bytes, well under the 103-byte limit + try { + execFileSync(resolveOpentasksNode(), [OPENTASKS_CLI, 'daemon', 'start'], { env: otEnvFor(otHome), stdio: 'ignore', timeout: 20_000 }); + } catch { /* the seed below also auto-starts it */ } + try { fs.writeFileSync(path.join(ctx.workspaceRoot, '.ot_sock'), path.join(otHome, 'daemon.sock')); } catch { /* agents can't find it → fall back */ } + for (const d of domains) { + try { + execFileSync(resolveOpentasksNode(), [OPENTASKS_CLI, 'create', '--type', 'task', '--title', `handle the "${d}" part of the task`, '--status', 'open'], + { env: otEnvFor(otHome), stdio: 'ignore', timeout: 20_000 }); + } catch { /* seed best-effort; a missing subtask just means less to claim */ } + } + } + + const actionLogPath = path.join(ctx.workspaceRoot, '.wb_actions.jsonl'); + const handle: ServiceHandle = { + // Exposed to phase.prompt(args.services) so each agent gets the actual task text. + env: { WB_INSTRUCTION: instruction, WB_DOMAINS: domains.join(', ') }, + async readState(): Promise<{ actions: string[]; graph: string[]; seededDomains: string[]; daemonList: string }> { + let actions: string[] = []; + try { + actions = fs.readFileSync(actionLogPath, 'utf8').split('\n').map((l) => l.trim()).filter(Boolean); + } catch { /* no side effects taken → empty union */ } + let graph: string[] = []; + let daemonList = ''; + if (otHome) { + try { graph = fs.readFileSync(path.join(otHome, 'graph.jsonl'), 'utf8').split('\n').map((l) => l.trim()).filter(Boolean); } catch { /* unflushed */ } + // Query the LIVE daemon (ground truth) — what tasks exist + who claimed them. + try { + daemonList = execFileSync(resolveOpentasksNode(), [OPENTASKS_CLI!, 'list', '--all', '--json'], { env: otEnvFor(otHome), timeout: 15_000 }).toString().slice(0, 4000); + } catch (e) { daemonList = `list-error: ${(e as Error).message.slice(0, 150)}`; } + } + return { actions, graph, seededDomains: domains, daemonList }; + }, + async stop(): Promise { + if (otHome) { + try { execFileSync(resolveOpentasksNode(), [OPENTASKS_CLI!, 'daemon', 'stop'], { env: otEnvFor(otHome), stdio: 'ignore', timeout: 10_000 }); } catch { /* already gone */ } + try { fs.rmSync(otHome, { recursive: true, force: true }); } catch { /* leaves a tiny /tmp dir */ } + } + }, + }; + return handle; + }, + }; +} + +/** Per-agent, per-arm coordination prompt. The task text arrives via `services.WB_INSTRUCTION`. */ +function buildMarblePrompt(armId: string, agentId: string, width: number, instruction: string): string { + const common = [ + `You are ${agentId}, one of ${width} agents working TOGETHER on ONE workplace task, at the SAME time.`, + 'Do the workplace actions with the WorkBench tools (native tools prefixed mcp__workbench__): search/read', + 'as needed to find ids and details, then perform the required create / update / delete / send actions.', + '', + 'CRITICAL: coordinate so no action is performed twice. A DUPLICATE side effect — e.g. two agents both', + 'sending the same email, or both deleting the same event — is an irreversible, HARMFUL error. Divide the', + "work so each part is done exactly once. Do not ask for confirmation; act. Stop when your part is done.", + '', + ]; + + if (armId === 'opentasks') { + return [ + ...common, + 'The task has been split into per-domain subtasks in the OpenTasks graph. Get your work ONLY by claiming:', + ` 1. Call mcp__opentasks__claim_next with agentId "${agentId}" to ATOMICALLY claim the next unclaimed`, + ' domain subtask. It returns {claimed, nodeId, ...}, or claimed:false when nothing is left.', + ' 2. Call mcp__opentasks__get_task on that nodeId to read which domain you claimed (its title).', + ' 3. Do ALL of THAT domain\'s workplace actions for the task, via mcp__workbench__* tools.', + ' 4. Mark it done: mcp__opentasks__update_task(id: nodeId, status: "closed").', + ' 5. Repeat from step 1 until claim_next returns claimed:false, then stop.', + 'The claim is race-free: if claimed:true, no other agent has that domain — never work a domain you did', + 'not claim. This guarantees each domain (and its side effects) is handled exactly once.', + '', + 'The task:', + instruction, + ].join('\n'); + } + + if (armId === 'notes') { + return [ + ...common, + 'Coordinate through the shared file claims.txt in this directory:', + ` 1. Before doing a domain's actions, append a line "${agentId}: " to claims.txt.`, + ' 2. Re-read claims.txt; if another agent already claimed that domain, skip it and pick another.', + ' 3. Only then perform that domain\'s mcp__workbench__* actions.', + '', + 'The task:', + instruction, + ].join('\n'); + } + + // stock — no shared coordination channel; the agents must divide the work with no way to check. + return [ + ...common, + 'You have no shared coordination channel with the other agents. Try to divide the task so nothing is', + 'done twice, but you cannot see what the others have done.', + '', + 'The task:', + instruction, + ].join('\n'); +} + +function phasesFor(): PhaseSpec[] { + return [ + { + id: 'work', + prompt: ({ arm, agentId, width, services }) => + buildMarblePrompt(arm.id, agentId, width, services.WB_INSTRUCTION ?? ''), + }, + ]; +} + +/** + * A swarmkit-eval `marble` benchmark: WorkBench multi_domain tasks run by N coordinating agents, graded by + * WorkBench's own outcome+harmful grader on the UNION of their side-effecting actions. + */ +export function workbenchMarbleBenchmark(opts: WorkbenchMarbleOpts): MarbleBenchmarkAdapter { + const domain = opts.domain ?? 'multi_domain'; + const id = `workbench-marble-${domain}`; + const native = workbenchNativeBenchmark({ repoDir: opts.repoDir, python: opts.python, domain }); + const grader = new WorkbenchGrader({ python: opts.python, repoDir: opts.repoDir }); + + return { + id, + execution: 'marble', + grader: { kind: 'self' }, + swarm: { width: opts.n, phases: phasesFor(), services: [workbenchService()] }, + + async load(o: LoadOpts): Promise { + // Reuse the Tier-1 loader (sealed outcome + public domain), then surface the raw instruction to the + // per-agent prompt via publicMetadata (the agent sees the task anyway; the OUTCOME stays sealed). + const wantIds = opts.taskIds?.length ? new Set(opts.taskIds) : null; + // When oversampling specific ids, load the FULL CSV (no limit) then filter; otherwise honor limit + // (o.limit from config.taskLimit wins if set, else opts.taskLimit — spread o first so an unset + // config.taskLimit can't clobber opts.taskLimit). + const tasks = await native.load({ ...o, limit: wantIds ? undefined : (o.limit ?? opts.taskLimit) }); + const mapped = tasks + .filter((t) => !wantIds || wantIds.has(t.id)) + .map((t) => ({ ...t, publicMetadata: { ...t.publicMetadata, instruction: t.prompt } })); + // With an id filter, the plain limit didn't slice — cap the filtered set by taskLimit here. + return wantIds && opts.taskLimit != null ? mapped.slice(0, opts.taskLimit) : mapped; + }, + + async score(raw: MultiAgentRawRun, task: EvalTask): Promise { + const st = (raw.services.workbench ?? { actions: [], graph: [], seededDomains: [], daemonList: '' }) as { + actions: string[]; graph: string[]; seededDomains: string[]; daemonList: string; + }; + const unionText = st.actions.join('\n') + (st.actions.length ? '\n' : ''); + // Reuse WorkbenchGrader by feeding the UNION log through a minimal Workspace. + const ws = { + readFile: async (p: string) => (p.includes('.wb_actions.jsonl') ? unionText : null), + run: async () => ({ exitCode: 0, stdout: '', stderr: '' }), + writeFiles: async () => {}, + dispose: async () => {}, + root: raw.workdir, + } as unknown as Workspace; + const syntheticRaw = { output: '', workdir: raw.workdir, usage: raw.usage, trajectory: raw.trajectory, durationMs: raw.durationMs } as RawRun; + const score = await grader.grade(task, syntheticRaw, ws); + // Multi-agent diagnostics on top of completion/harmful (coordination KPIs R/O/c/E_c are folded in by + // the engine from the coordination classifier below). + const workActions = st.actions.length; + score.metrics = { ...score.metrics, numAgents: opts.n, unionSideEffects: workActions }; + + // EVAL_DEBUG_DIR: dump per-agent tool sequences + the opentasks graph (who claimed what) + union + // actions, so the coordination surface can be inspected. Off unless the env is set. + if (process.env.EVAL_DEBUG_DIR) { + try { + const agents = (raw.agents ?? []).map((a) => ({ + agentId: a.agentId, + role: a.role, + tools: (a.trajectory ?? []) + .filter((e): e is Extract => e.type === 'tool') + .map((e) => ({ name: e.name, input: e.input })), + })); + const dbg = { + taskId: task.id, seededDomains: st.seededDomains, completion: score.metrics?.completion, + harmful: score.metrics?.harmful, daemonList: st.daemonList, unionActions: st.actions, graph: st.graph, + agents, messages: raw.messages, + }; + fs.mkdirSync(process.env.EVAL_DEBUG_DIR, { recursive: true }); + fs.writeFileSync(path.join(process.env.EVAL_DEBUG_DIR, `${task.id}.json`), JSON.stringify(dbg, null, 2)); + } catch { /* debug best-effort */ } + } + return score; + }, + + coordination(call): CoordinationAction | null { + const name = call.name ?? ''; + if (name.startsWith('mcp__workbench__')) { + const bare = name.slice('mcp__workbench__'.length); + // A side-effecting workbench call = productive work; the redundancy key is the action itself, so two + // agents doing the SAME action register as redundant (R>0). Reads (search/get) are ignored. + if (WB_SIDE_EFFECT.has(bare)) return { kind: 'work', ref: `${bare}:${JSON.stringify(call.input ?? {})}` }; + return null; + } + if (name.startsWith('mcp__opentasks__')) return { kind: 'coordination' }; + const fp = String((call.input as { file_path?: string } | undefined)?.file_path ?? ''); + if ((name === 'Write' || name === 'Edit') && /claims\.txt/.test(fp)) return { kind: 'coordination' }; + return null; + }, + }; +} diff --git a/evals/swarmkit/workbench-run.ts b/evals/swarmkit/workbench-run.ts new file mode 100644 index 0000000..333b95b --- /dev/null +++ b/evals/swarmkit/workbench-run.ts @@ -0,0 +1,201 @@ +/** + * OpenTasks × WorkBench eval — benchmark OpenTasks (as a planning/coordination scaffold) + * on WorkBench (olly-styles/WorkBench "Revisited": outcome-graded workplace-agent tasks). + * + * All WorkBench-specific machinery lives in `swarmkit-eval` (the tool bridge `wb_mcp.py`, + * the faithful `{kind:"workbench"}` grader, the benchmark). This entrypoint only COMPOSES: + * swarmkit-eval's WorkBench benchmark + OpenTasks arms → runEval → paired report. + * + * Arms (the independent variable — same model, same tasks, only the scaffold varies): + * stock WorkBench tools only (mcp__workbench__*) + * notes + a NOTES.md durable-log nudge + * opentasks + the OpenTasks MCP graph (mcp__opentasks__*) as a planning/decomposition scaffold + * + * Base OpenTasks functionality only — NO daemon changes required: + * • per-cell isolation: a RELATIVE OPENTASKS_PROJECT_DIR=.opentasks resolves against each + * cell's workspace cwd, so every cell gets its own .opentasks/ + daemon socket. + * • teardown: the auto-started (detached) daemon outlives the cell, so after the run we reap + * ONLY the opentasks daemons that appeared during this run (targeted; never touches + * pre-existing daemons). Optionally set OPENTASKS_DAEMON_IDLE_TIMEOUT to let them self-reap + * (a no-op on base builds that don't support it). + * + * Env: + * WORKBENCH_REPO WorkBench checkout (default ~/GitHub/WorkBench; needs `uv sync` + `uv pip install mcp`) + * EVAL_MODEL model under test (default claude-haiku for gateway, else haiku) + * EVAL_ARMS comma list (default stock,opentasks) + * EVAL_DOMAIN email | calendar | analytics | project_management | crm | multi_domain (default email) + * EVAL_TASK_LIMIT cap tasks (default 5) + * EVAL_REPEATS seeds per cell (default 1) + * EVAL_CONCURRENCY concurrent cells (default 2) + * EVAL_TIMEOUT per-cell ms (default 300000) + * WB_GATEWAY_BASE_URL route the model through a LiteLLM gateway (else ambient Max-plan auth) + * WORKBENCH_LLM_API_KEY gateway master key (with WB_GATEWAY_BASE_URL) + * + * Run: + * npm run build # build opentasks (dist/cli.js for the MCP arm) + * WB_GATEWAY_BASE_URL=http://127.0.0.1:4000 WORKBENCH_LLM_API_KEY=sk-… AWS_REGION=us-east-1 \ + * EVAL_ARMS=stock,opentasks EVAL_TASK_LIMIT=5 npx tsx evals/swarmkit/workbench-run.ts + */ + +import * as path from 'node:path'; +import { execSync } from 'node:child_process'; +import { + runEval, + NativeCliAdapter, + InProcessBackend, + LocalResultStore, + buildReport, + renderMarkdownReport, + writeReport, + workbenchNativeBenchmark, + workbenchMcpServer, + workbenchNativeArms, + type EvalConfig, + type RunDeps, + type GatewayConfig, + type McpServerSpec, + type WorkbenchNativeArmSpec, +} from 'swarmkit-eval'; +import { ARMS } from '../arms.js'; + +const WB_REPO = process.env.WORKBENCH_REPO ?? path.join(process.env.HOME ?? '', 'GitHub', 'WorkBench'); +const WB_PYTHON = path.join(WB_REPO, '.venv', 'bin', 'python'); +const GATEWAY_BASE = process.env.WB_GATEWAY_BASE_URL; +const MODEL = process.env.EVAL_MODEL ?? (GATEWAY_BASE ? 'claude-haiku' : 'haiku'); +const ARM_IDS = (process.env.EVAL_ARMS ?? 'stock,opentasks').split(',').map((s) => s.trim()); +const DOMAIN = (process.env.EVAL_DOMAIN ?? 'email') as Parameters[0]['domain']; +const TASK_LIMIT = Number(process.env.EVAL_TASK_LIMIT ?? 5); +const REPEATS = Number(process.env.EVAL_REPEATS ?? 1); +const CONCURRENCY = Number(process.env.EVAL_CONCURRENCY ?? 2); +const TIMEOUT = Number(process.env.EVAL_TIMEOUT ?? 300_000); +const OUT_DIR = path.resolve(process.cwd(), 'evals/.swarmkit-workbench'); + +// WorkBench-appropriate planning nudge for the opentasks arm — deliberately generic (task +// decomposition + progress tracking), NOT the TAC/openswarm-specific appendix in ../arms.ts. +const PLANNING_APPENDIX = + 'You have an OpenTasks MCP graph (native tools prefixed mcp__opentasks__) for planning. For a ' + + 'multi-step workplace task, first break it into subtasks with mcp__opentasks__create_task, then ' + + 'work through them, marking each done with mcp__opentasks__update_task and re-checking what remains ' + + 'before you finish. Perform the actual workplace actions with the WorkBench tools (mcp__workbench__*). ' + + 'Use native MCP tools directly — do not invoke mcp__ names as shell commands.'; + +const NOTES_APPENDIX = + 'Maintain a NOTES.md file in the working directory as your durable plan/progress log: record the ' + + 'subgoals, what you have done, and what remains; re-read it before finishing.'; + +/** The OpenTasks MCP graph as a swarmkit MCP server — RELATIVE OPENTASKS_PROJECT_DIR = per-cell isolation. */ +function opentasksMcpServer(): McpServerSpec { + const src = ARMS.opentasks.mcp; + if (!src) throw new Error('ARMS.opentasks.mcp is undefined (expected the opentasks MCP command)'); + const env: Record = { OPENTASKS_PROJECT_DIR: '.opentasks' }; + // Optional: if the local opentasks build supports it, let per-cell daemons self-reap. Harmless otherwise. + if (process.env.OPENTASKS_DAEMON_IDLE_TIMEOUT) env.OPENTASKS_DAEMON_IDLE_TIMEOUT = process.env.OPENTASKS_DAEMON_IDLE_TIMEOUT; + return { name: src.name, command: src.command, args: src.args, env }; +} + +// The absolute opentasks CLI this eval launches (dist/cli.js). Matching daemons by THIS exact path keeps +// the reap from ever touching a daemon from another project (e.g. a different repo's opentasks install). +const OPENTASKS_CLI = ARMS.opentasks.mcp?.args?.[0]; + +/** PIDs of opentasks daemons started from THIS build's CLI (for a targeted, path-scoped post-run reap). */ +function opentasksDaemonPids(): Set { + if (!OPENTASKS_CLI) return new Set(); + try { + // ps → keep only lines that are BOTH this exact cli.js path AND a `daemon start`. + const out = execSync('ps -Ao pid=,command=', { encoding: 'utf8' }); + const pids = new Set(); + for (const line of out.split('\n')) { + if (!line.includes(OPENTASKS_CLI) || !line.includes('daemon start')) continue; + const pid = parseInt(line.trim().split(/\s+/)[0]!, 10); + if (Number.isFinite(pid)) pids.add(pid); + } + return pids; + } catch { + return new Set(); + } +} + +async function main(): Promise { + const benchmark = workbenchNativeBenchmark({ repoDir: WB_REPO, python: WB_PYTHON, domain: DOMAIN }); + const wbMcp = workbenchMcpServer({ python: WB_PYTHON, repoDir: WB_REPO }); + const specs: WorkbenchNativeArmSpec[] = [ + { id: 'notes', label: 'notes (NOTES.md)', systemPromptAppendix: NOTES_APPENDIX }, + { + id: 'opentasks', + label: 'opentasks (MCP graph)', + systemPromptAppendix: PLANNING_APPENDIX, + mcpServers: [opentasksMcpServer()], + extraTools: ARMS.opentasks.extraTools, + }, + ]; + const arms = workbenchNativeArms(wbMcp, specs).filter((a) => ARM_IDS.includes(a.id)); + if (arms.length === 0) throw new Error(`No arms matched EVAL_ARMS=${ARM_IDS.join(',')} (have stock,notes,opentasks)`); + + const gateway: GatewayConfig | undefined = GATEWAY_BASE ? { baseUrl: GATEWAY_BASE, fallbacksDisabled: true } : undefined; + const passEnv: Record = {}; + for (const k of ['AWS_REGION', 'AWS_PROFILE']) if (process.env[k]) passEnv[k] = process.env[k]!; + + const config: EvalConfig = { + runId: `workbench-${DOMAIN}-${MODEL}`, + configVersion: 'v1', + benchmark: benchmark.id, + arms, + models: [{ name: MODEL, ...(gateway ? { dialect: 'anthropic' as const } : {}) }], + seeds: Array.from({ length: REPEATS }, (_, i) => i + 1), + backend: 'in-process', + concurrency: { cells: CONCURRENCY, modelConnections: CONCURRENCY }, + taskLimit: TASK_LIMIT, + retry: { maxAttempts: 2, baseDelayMs: 1000 }, + output: { dir: OUT_DIR, trace: false }, + }; + + const deps: RunDeps = { + benchmark, + backend: new InProcessBackend(), + store: new LocalResultStore(OUT_DIR), + adapter: new NativeCliAdapter({ + defaultModel: MODEL, + timeoutMs: TIMEOUT, + env: passEnv, + ...(gateway ? { gateway, virtualKey: process.env.WORKBENCH_LLM_API_KEY ?? 'sk-eval' } : {}), + }), + }; + + console.log( + `workbench × opentasks · model=${MODEL} arms=${arms.map((a) => a.id).join(',')} domain=${DOMAIN} ` + + `tasks≤${TASK_LIMIT} repeats=${REPEATS} concurrency=${CONCURRENCY}${gateway ? ' [gateway]' : ' [ambient]'}`, + ); + + const daemonsBefore = opentasksDaemonPids(); + let results; + try { + results = await runEval(config, deps); + } finally { + // Base-functionality teardown: reap ONLY the opentasks daemons that appeared during this run. + const now = opentasksDaemonPids(); + let reaped = 0; + for (const pid of now) if (!daemonsBefore.has(pid)) { try { process.kill(pid, 'SIGTERM'); reaped++; } catch { /* gone */ } } + if (reaped) console.log(`reaped ${reaped} per-cell opentasks daemon(s)`); + } + + for (const r of results) { + const m = r.score?.metrics ?? {}; + console.log( + ` ${r.taskId} | ${r.armId.padEnd(9)} seed${r.seed}: status=${r.status} ` + + `completion=${m.completion ?? '-'} harmful=${m.harmful ?? '-'} tools=${m.numToolCalls ?? '-'} ` + + `tokens=${r.usage.totalTokens} ${Math.round(r.durationMs / 1000)}s` + + `${r.status === 'env_error' ? ` ENV:${r.envError?.kind}` : ''}`, + ); + } + + const baseline = ARM_IDS.includes('stock') ? 'stock' : ARM_IDS[0]; + const report = buildReport(results, config, { baselineArmId: baseline, accuracyMetric: 'sPartial' }); + console.log('\n' + renderMarkdownReport(report)); + const paths = await writeReport(OUT_DIR, report); + console.log(`\nreport → ${path.relative(process.cwd(), paths.md)} (+ .html, .json)`); +} + +main().catch((e) => { + console.error(e); + process.exit(1); +}); diff --git a/package.json b/package.json index 177c9ec..01cd314 100644 --- a/package.json +++ b/package.json @@ -31,6 +31,8 @@ "test:slow": "RUN_SLOW_TESTS=1 vitest run", "test:slow:watch": "RUN_SLOW_TESTS=1 vitest", "test:e2e": "RUN_FULL_AGENT_TESTS=1 vitest run --config vitest.e2e.config.ts", + "eval:workbench": "npx tsx evals/swarmkit/workbench-run.ts", + "eval:workbench:marble": "npx tsx evals/swarmkit/workbench-marble-run.ts", "eval:tac": "npx tsx evals/tac/run.ts", "eval:tac:pool": "npx tsx evals/tac/run-pool.ts", "eval:tac:pool:summarize": "npx tsx evals/tac/scripts/summarize-pool-runs.ts",