Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 10 additions & 3 deletions docs/v6/cookbooks/coding-agent.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -66,9 +66,16 @@ To run the `claude` CLI over SSH, select the `claude_cli` agent:
hud eval env.py claude_cli --gateway
```

The `claude` executable must already be installed in the host or environment image. Pinning it in
the image keeps runs reproducible; `ClaudeCLIAgent` does not install or update it. The equivalent
Python API is:
Codex uses the same environment through the `codex_cli` agent:

```bash
hud eval env.py codex_cli --gateway
```

The SSH runtime must expose the selected executable, either as a managed runtime bundle or through
the environment image. The agent validates the effective runtime OS against the live SSH target,
prefers a compatible managed bundle, and otherwise resolves the executable from `PATH`. CLI agents
do not download or update executables. The equivalent Claude Python API is:

```python run.py
import asyncio
Expand Down
8 changes: 5 additions & 3 deletions docs/v6/guides/running-an-eval.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -38,9 +38,11 @@ and launch - no CLI required.

### Choosing an agent

The agent name (`claude`, `openai`, `gemini`) selects a built-in harness and routes calls through the
[HUD gateway](/v6/reference/agents), where one `HUD_API_KEY` covers every provider. Switching models is a
single flag, and `hud models list` shows every model the gateway knows.
The agent name (`claude`, `openai`, `gemini`) selects a provider harness and routes calls through the
[HUD gateway](/v6/reference/agents), where one `HUD_API_KEY` covers every provider. The `claude_cli` and
`codex_cli` harnesses instead run an installed CLI inside the environment; pass `--gateway` to route
their model calls through HUD. Switching models is a single flag, and `hud models list` shows every
model the gateway knows.

```bash
hud eval "My Taskset" claude --model claude-haiku-4-5 # a cheaper model for fast iteration
Expand Down
18 changes: 11 additions & 7 deletions docs/v6/reference/agents.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -60,15 +60,19 @@ agent = ClaudeAgent(ClaudeConfig(model="claude-sonnet-4-5", max_steps=30))
| `GeminiAgent` | `GeminiConfig` | `gemini-3-pro-preview` |
| `OpenAIChatAgent` | `OpenAIChatConfig` | `gpt-5.4-mini` |
| `ClaudeCLIAgent` | `ClaudeCLIConfig` | `claude-sonnet-5` |
| `CodexCLIAgent` | `CodexCLIConfig` | `gpt-5.6-sol` |

Each config lives in `hud.agents.types`. `OpenAIChatAgent` speaks the OpenAI Chat Completions API, so it
points at any compatible server (vLLM, a local model) via `base_url`; `ClaudeCLIAgent` runs the `claude`
CLI over an `ssh` capability, against the env's filesystem. SSH-only runs support POSIX and Windows
workspaces; computer use over an `rfb` capability currently requires a POSIX workspace. Every knob
points at any compatible server (vLLM, a local model) via `base_url`. `ClaudeCLIAgent` and
`CodexCLIAgent` run their respective CLIs over an `ssh` capability against the env's filesystem and
stream the CLI's structured events into the HUD trace. They prefer a compatible managed runtime
bundle and fall back to the environment's `PATH`; they never install or update the executable.
SSH-only runs support POSIX and Windows workspaces; Claude computer use over an `rfb` capability
currently requires a POSIX workspace. Every knob
(`model`, `max_steps`, `timeout_seconds`, `tool_timeout_seconds`, `system_prompt`, `citations_enabled`, `stop_on`) lives on the
config; `__call__(run)` takes only the run.

`timeout_seconds` bounds the complete agent phase. For provider tool agents, `tool_timeout_seconds` bounds each complete SSH-backed tool call, including multi-operation editor calls. It is unset by default except on `ClaudeConfig`, where it defaults to 120 seconds. A timeout is returned to the model as a tool error so the agent can continue. `ClaudeSDKAgent` does not apply this setting because its SSH process is the complete Claude Code agent, not one tool call.
`timeout_seconds` bounds the complete agent phase. For provider tool agents, `tool_timeout_seconds` bounds each complete SSH-backed tool call, including multi-operation editor calls. It is unset by default except on `ClaudeConfig`, where it defaults to 120 seconds. A timeout is returned to the model as a tool error so the agent can continue. `ClaudeCLIAgent` does not apply this setting because its SSH process is the complete Claude Code agent, not one tool call.

```python
agent = OpenAIChatAgent(
Expand Down Expand Up @@ -100,8 +104,8 @@ A model id maps to one of four gateway agent types (`AgentType`), each a provide
| `gemini` | `GeminiAgent` |
| `openai_compatible` | `OpenAIChatAgent` |

For a provider key instead of the gateway, or for `ClaudeCLIAgent` (not a gateway shortcut), construct
the agent directly.
For a provider key instead of the gateway, or for a CLI agent (not a gateway shortcut), construct the
agent directly.

## Agent

Expand All @@ -126,7 +130,7 @@ print(job.reward)
```

**From the CLI**, `hud eval` takes a task source and an agent name (`claude`, `openai`, `gemini`,
`openai_compatible`); see [running an eval](/v6/guides/running-an-eval) for the walkthrough and the
`openai_compatible`, `claude_cli`, `codex_cli`); see [running an eval](/v6/guides/running-an-eval) for the walkthrough and the
[CLI reference](/v6/reference/cli#hud-eval) for the full flag set.

## Bring your own harness
Expand Down
5 changes: 5 additions & 0 deletions hud/agents/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@
from typing import TypeAlias

from hud.agents.claude import ClaudeAgent, ClaudeCLIAgent, ClaudeCLIConfig
from hud.agents.codex import CodexCLIAgent, CodexCLIConfig
from hud.agents.gemini import GeminiAgent
from hud.agents.openai import OpenAIAgent
from hud.agents.openai_compatible import OpenAIChatAgent
Expand Down Expand Up @@ -124,6 +125,8 @@ def create_agent(model: str, **kwargs: Any) -> GatewayAgent:
"ClaudeAgent": ("hud.agents.claude", "ClaudeAgent"),
"ClaudeCLIAgent": ("hud.agents.claude", "ClaudeCLIAgent"),
"ClaudeCLIConfig": ("hud.agents.claude", "ClaudeCLIConfig"),
"CodexCLIAgent": ("hud.agents.codex", "CodexCLIAgent"),
"CodexCLIConfig": ("hud.agents.codex", "CodexCLIConfig"),
"GeminiAgent": ("hud.agents.gemini", "GeminiAgent"),
"MCPAgent": ("hud.agents.tool_agent", "ToolAgent"),
"OpenAIAgent": ("hud.agents.openai", "OpenAIAgent"),
Expand All @@ -134,6 +137,8 @@ def create_agent(model: str, **kwargs: Any) -> GatewayAgent:
"ClaudeAgent",
"ClaudeCLIAgent",
"ClaudeCLIConfig",
"CodexCLIAgent",
"CodexCLIConfig",
"GeminiAgent",
"MCPAgent",
"OpenAIAgent",
Expand Down
4 changes: 2 additions & 2 deletions hud/agents/claude/agent.py
Original file line number Diff line number Diff line change
Expand Up @@ -257,10 +257,10 @@ async def get_response(
if response is None:
raise ValueError("Claude response missing after retries")

return self._message_to_agent_step(response, citations_enabled=citations_enabled)
return self.message_to_agent_step(response, citations_enabled=citations_enabled)

@classmethod
def _message_to_agent_step(
def message_to_agent_step(
cls,
response: BetaMessage,
*,
Expand Down
Loading
Loading