Support CLAUDE_CODE_EFFORT and CLAUDE_CODE_OAUTH_TOKEN for the Claude Code agent - #153
Open
rrailton wants to merge 2 commits into
Open
Support CLAUDE_CODE_EFFORT and CLAUDE_CODE_OAUTH_TOKEN for the Claude Code agent#153rrailton wants to merge 2 commits into
rrailton wants to merge 2 commits into
Conversation
Mirrors OPENAI_CODEX_REASONING_EFFORT (dbt-labs#149): an optional env-var override for Claude Code's --effort level (low|medium|high|xhigh|max), useful when comparing agents or model configurations at matched effort levels. Set via env var rather than a constructor kwarg so the existing harness/factory plumbing doesn't need to change.
When set, the Claude Code agent authenticates the sandboxed CLI with a long-lived subscription OAuth token (from `claude setup-token`) instead of an API key, letting subscription holders run benchmarks against their plan's usage quota rather than metered API billing. ANTHROPIC_API_KEY remains the default when the token is not set. Also teaches the `ab check environment anthropic` preflight to accept the token, so OAuth-only environments don't report missing credentials.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Two opt-in env vars for the Claude Code agent, in two separable commits:
CLAUDE_CODE_EFFORT— appends--effort <value>(low|medium|high|xhigh|max) to theclaudeinvocation. This is the Claude Code counterpart ofOPENAI_CODEX_REASONING_EFFORT(DI-4253: support OPENAI_CODEX_REASONING_EFFORT env var #149), with the same design: an env var rather than a constructor kwarg, so no harness/factory plumbing changes. Value isshlex.quoted. Note--effortrequires a recent Claude Code CLI; the setup script installs latest, so this can't regress existing runs.CLAUDE_CODE_OAUTH_TOKEN— when set (non-whitespace), the sandboxed CLI authenticates with a long-lived subscription OAuth token (fromclaude setup-token) instead ofANTHROPIC_API_KEY, letting subscription holders run benchmarks against their plan quota rather than metered API billing. Theab check environment anthropicpreflight accepts the token too. The API key remains the default; this commit is separable if a single auth path is preferred.Both are covered by unit tests mirroring
test_openai_codex_agent.py(no-env-var, value emitted, shell-quoting; plus auth-precedence cases including whitespace-only tokens). Neither changes behavior when the env vars are unset.Why
Same motivation as #149: comparing agents/models at matched effort levels. As a working example, we used this to run all 60 ready duckdb+dbt tasks — 75 trials per arm once per-task prompt variants are included — at 1 attempt per trial (Claude Code pinned to 2.1.207), comparing
claude-fable-5 --effort lowvsclaude-opus-4-8 --effort xhigh:i.e. statistical parity on outcomes with a ~2x latency and ~4x token gap — the kind of readout this flag makes easy to produce.
Testing
pytest tests/agents/installed_agents/— 18 passed (11 existing + 7 new)--effortflag is present in the recorded container commands, and OAuth auth confirmed via subscription usage accountingmodelUsageaudit across all 75 fable-5 trials confirmed no cross-model fallback occurred🤖 Generated with Claude Code