Skip to content

DI-4855 Add Deep Agents Code benchmark runner - #154

Draft
wiggzz wants to merge 2 commits into
mainfrom
wj/di-4855-dcode-agent
Draft

DI-4855 Add Deep Agents Code benchmark runner#154
wiggzz wants to merge 2 commits into
mainfrom
wj/di-4855-dcode-agent

Conversation

@wiggzz

@wiggzz wiggzz commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

Why

We need a directly comparable ADE-Bench baseline for LangChain's Deep Agents Code (dcode) while evaluating Deep Agents for Wizard.

What

Adds dcode as a first-class installed ADE-Bench agent, pinned to deepagents-code==0.1.45. It runs headlessly with explicit model/reasoning configuration, reuses the benchmark AGENTS.md, preserves transcripts, records runtime, and parses the CLI usage table into approximate token/request metrics.

dcode's headless output rounds compact token values and does not expose cache-token or cost breakdowns, so token totals are explicitly approximate and cost remains unavailable.

Validation

  • Adapter tests, including public task execution and usage parsing: 3 passed.
  • Ruff and Black checks: passed.
  • Full suite: 85 passed; one unrelated test hard-codes allowed checkout directory names and rejects this shared checkout name.
  • Completed the GPT-5.6 Sol, low-reasoning, 75-task × 3-attempt run: 167/225 resolved (74.2%), two agent timeouts, ~78.46M reported input tokens, ~570.9K output tokens.
  • The paired Codex run resolved 165/225 (73.3%); dcode was better on 12 task-level three-attempt scores, Codex on 10, and 53 tied. dcode's evaluation took 1h40m25s versus Codex's 1h09m25s.

Refs

Drafted by Codex (GPT-5) under direction of @wiggzz.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant