diff --git a/docs/v6/cookbooks/coding-agent.mdx b/docs/v6/cookbooks/coding-agent.mdx index 04bb3736a..be81fcfcb 100644 --- a/docs/v6/cookbooks/coding-agent.mdx +++ b/docs/v6/cookbooks/coding-agent.mdx @@ -66,9 +66,16 @@ To run the `claude` CLI over SSH, select the `claude_cli` agent: hud eval env.py claude_cli --gateway ``` -The `claude` executable must already be installed in the host or environment image. Pinning it in -the image keeps runs reproducible; `ClaudeCLIAgent` does not install or update it. The equivalent -Python API is: +Codex uses the same environment through the `codex_cli` agent: + +```bash +hud eval env.py codex_cli --gateway +``` + +The SSH runtime must expose the selected executable, either as a managed runtime bundle or through +the environment image. The agent validates the effective runtime OS against the live SSH target, +prefers a compatible managed bundle, and otherwise resolves the executable from `PATH`. CLI agents +do not download or update executables. The equivalent Claude Python API is: ```python run.py import asyncio diff --git a/docs/v6/guides/running-an-eval.mdx b/docs/v6/guides/running-an-eval.mdx index 9305efc0a..a124e9eec 100644 --- a/docs/v6/guides/running-an-eval.mdx +++ b/docs/v6/guides/running-an-eval.mdx @@ -38,9 +38,11 @@ and launch - no CLI required. ### Choosing an agent -The agent name (`claude`, `openai`, `gemini`) selects a built-in harness and routes calls through the -[HUD gateway](/v6/reference/agents), where one `HUD_API_KEY` covers every provider. Switching models is a -single flag, and `hud models list` shows every model the gateway knows. +The agent name (`claude`, `openai`, `gemini`) selects a provider harness and routes calls through the +[HUD gateway](/v6/reference/agents), where one `HUD_API_KEY` covers every provider. The `claude_cli` and +`codex_cli` harnesses instead run an installed CLI inside the environment; pass `--gateway` to route +their model calls through HUD. Switching models is a single flag, and `hud models list` shows every +model the gateway knows. ```bash hud eval "My Taskset" claude --model claude-haiku-4-5 # a cheaper model for fast iteration diff --git a/docs/v6/reference/agents.mdx b/docs/v6/reference/agents.mdx index de18580d9..2db5f5527 100644 --- a/docs/v6/reference/agents.mdx +++ b/docs/v6/reference/agents.mdx @@ -60,15 +60,19 @@ agent = ClaudeAgent(ClaudeConfig(model="claude-sonnet-4-5", max_steps=30)) | `GeminiAgent` | `GeminiConfig` | `gemini-3-pro-preview` | | `OpenAIChatAgent` | `OpenAIChatConfig` | `gpt-5.4-mini` | | `ClaudeCLIAgent` | `ClaudeCLIConfig` | `claude-sonnet-5` | +| `CodexCLIAgent` | `CodexCLIConfig` | `gpt-5.6-sol` | Each config lives in `hud.agents.types`. `OpenAIChatAgent` speaks the OpenAI Chat Completions API, so it -points at any compatible server (vLLM, a local model) via `base_url`; `ClaudeCLIAgent` runs the `claude` -CLI over an `ssh` capability, against the env's filesystem. SSH-only runs support POSIX and Windows -workspaces; computer use over an `rfb` capability currently requires a POSIX workspace. Every knob +points at any compatible server (vLLM, a local model) via `base_url`. `ClaudeCLIAgent` and +`CodexCLIAgent` run their respective CLIs over an `ssh` capability against the env's filesystem and +stream the CLI's structured events into the HUD trace. They prefer a compatible managed runtime +bundle and fall back to the environment's `PATH`; they never install or update the executable. +SSH-only runs support POSIX and Windows workspaces; Claude computer use over an `rfb` capability +currently requires a POSIX workspace. Every knob (`model`, `max_steps`, `timeout_seconds`, `tool_timeout_seconds`, `system_prompt`, `citations_enabled`, `stop_on`) lives on the config; `__call__(run)` takes only the run. -`timeout_seconds` bounds the complete agent phase. For provider tool agents, `tool_timeout_seconds` bounds each complete SSH-backed tool call, including multi-operation editor calls. It is unset by default except on `ClaudeConfig`, where it defaults to 120 seconds. A timeout is returned to the model as a tool error so the agent can continue. `ClaudeSDKAgent` does not apply this setting because its SSH process is the complete Claude Code agent, not one tool call. +`timeout_seconds` bounds the complete agent phase. For provider tool agents, `tool_timeout_seconds` bounds each complete SSH-backed tool call, including multi-operation editor calls. It is unset by default except on `ClaudeConfig`, where it defaults to 120 seconds. A timeout is returned to the model as a tool error so the agent can continue. `ClaudeCLIAgent` does not apply this setting because its SSH process is the complete Claude Code agent, not one tool call. ```python agent = OpenAIChatAgent( @@ -100,8 +104,8 @@ A model id maps to one of four gateway agent types (`AgentType`), each a provide | `gemini` | `GeminiAgent` | | `openai_compatible` | `OpenAIChatAgent` | -For a provider key instead of the gateway, or for `ClaudeCLIAgent` (not a gateway shortcut), construct -the agent directly. +For a provider key instead of the gateway, or for a CLI agent (not a gateway shortcut), construct the +agent directly. ## Agent @@ -126,7 +130,7 @@ print(job.reward) ``` **From the CLI**, `hud eval` takes a task source and an agent name (`claude`, `openai`, `gemini`, -`openai_compatible`); see [running an eval](/v6/guides/running-an-eval) for the walkthrough and the +`openai_compatible`, `claude_cli`, `codex_cli`); see [running an eval](/v6/guides/running-an-eval) for the walkthrough and the [CLI reference](/v6/reference/cli#hud-eval) for the full flag set. ## Bring your own harness diff --git a/hud/agents/__init__.py b/hud/agents/__init__.py index 382b25331..b4c0cb97b 100644 --- a/hud/agents/__init__.py +++ b/hud/agents/__init__.py @@ -21,6 +21,7 @@ from typing import TypeAlias from hud.agents.claude import ClaudeAgent, ClaudeCLIAgent, ClaudeCLIConfig + from hud.agents.codex import CodexCLIAgent, CodexCLIConfig from hud.agents.gemini import GeminiAgent from hud.agents.openai import OpenAIAgent from hud.agents.openai_compatible import OpenAIChatAgent @@ -124,6 +125,8 @@ def create_agent(model: str, **kwargs: Any) -> GatewayAgent: "ClaudeAgent": ("hud.agents.claude", "ClaudeAgent"), "ClaudeCLIAgent": ("hud.agents.claude", "ClaudeCLIAgent"), "ClaudeCLIConfig": ("hud.agents.claude", "ClaudeCLIConfig"), + "CodexCLIAgent": ("hud.agents.codex", "CodexCLIAgent"), + "CodexCLIConfig": ("hud.agents.codex", "CodexCLIConfig"), "GeminiAgent": ("hud.agents.gemini", "GeminiAgent"), "MCPAgent": ("hud.agents.tool_agent", "ToolAgent"), "OpenAIAgent": ("hud.agents.openai", "OpenAIAgent"), @@ -134,6 +137,8 @@ def create_agent(model: str, **kwargs: Any) -> GatewayAgent: "ClaudeAgent", "ClaudeCLIAgent", "ClaudeCLIConfig", + "CodexCLIAgent", + "CodexCLIConfig", "GeminiAgent", "MCPAgent", "OpenAIAgent", diff --git a/hud/agents/claude/agent.py b/hud/agents/claude/agent.py index 7aad09954..e3e5d2c84 100644 --- a/hud/agents/claude/agent.py +++ b/hud/agents/claude/agent.py @@ -257,10 +257,10 @@ async def get_response( if response is None: raise ValueError("Claude response missing after retries") - return self._message_to_agent_step(response, citations_enabled=citations_enabled) + return self.message_to_agent_step(response, citations_enabled=citations_enabled) @classmethod - def _message_to_agent_step( + def message_to_agent_step( cls, response: BetaMessage, *, diff --git a/hud/agents/claude/cli/agent.py b/hud/agents/claude/cli/agent.py index 676d3e001..7046dfdd4 100644 --- a/hud/agents/claude/cli/agent.py +++ b/hud/agents/claude/cli/agent.py @@ -2,9 +2,6 @@ from __future__ import annotations -import asyncio -import base64 -import contextlib import json import logging import shlex @@ -17,6 +14,13 @@ from hud.agents.base import Agent from hud.agents.claude.agent import ClaudeAgent +from hud.agents.cli import ( + WINDOWS_SHELLS, + powershell, + powershell_quote, + resolve_executable, + run_jsonl, +) from hud.agents.types import ClaudeCLIConfig, ToolStep from hud.settings import settings from hud.telemetry.context import get_current_trace_id @@ -31,25 +35,27 @@ logger = logging.getLogger(__name__) -_WINDOWS_SHELLS = ("cmd", "powershell") -_PROMPT_PATH = ".hud_prompt.txt" -_MCP_CONFIG_PATH = ".hud_mcp_config.json" -_RUN_SCRIPT_PATH = ".hud_run.bat" -_PROCESS_CLOSE_TIMEOUT_S = 5.0 +INPUT_PATH = ".hud_input.jsonl" +MCP_CONFIG_PATH = ".hud_mcp_config.json" +RUN_SCRIPT_PATH = ".hud_run.bat" +_MANAGED_CLAUDE_PATHS = { + "linux-x64": "/media/hud/bin/claude/linux-x64/claude", + "linux-x64-musl": "/media/hud/bin/claude/linux-x64-musl/claude", +} -class _ClaudeStreamParser: + +class ClaudeEvents: """Translate Claude CLI stream messages into canonical HUD steps.""" def __init__(self, run: Run, *, started_at: str) -> None: - self._run = run - self._agent_started_at = started_at - self._pending_calls: dict[str, tuple[MCPToolCall, str]] = {} - self._last_agent_content = "" - self._saw_result = False - self._result_error: str | None = None - - def feed_line(self, line: str) -> None: + self.run = run + self.agent_started_at = started_at + self.pending_calls: dict[str, tuple[MCPToolCall, str]] = {} + self.saw_result = False + self.error: str | None = None + + def consume(self, line: str) -> None: line = line.strip() if not line: return @@ -59,139 +65,245 @@ def feed_line(self, line: str) -> None: received_at = now_iso() match message.get("type"): case "system" if message.get("subtype") == "init": - self._agent_started_at = received_at + self.agent_started_at = received_at case "assistant": - self._record_assistant(message, received_at) + step = ClaudeAgent.message_to_agent_step( + BetaMessage.model_validate(message["message"]) + ) + step.started_at = self.agent_started_at + step.ended_at = received_at + if step.content: + self.run.trace.content = step.content + self.run.record(step) + for call in step.tool_calls: + self.pending_calls[call.id] = (call, received_at) case "user": - self._record_tool_results(message, received_at) + saw_result = False + for block in message["message"]["content"]: + if block["type"] != "tool_result": + continue + call_id = block["tool_use_id"] + try: + call, started_at = self.pending_calls.pop(call_id) + except KeyError: + raise ValueError( + f"Claude returned a result for unknown tool call {call_id!r}" + ) from None + + raw_result = block.get("content") + raw_items = raw_result if isinstance(raw_result, list) else [raw_result] + content: list[mcp_types.ContentBlock] = [] + for item in raw_items: + if isinstance(item, str): + content.append(mcp_types.TextContent(type="text", text=item)) + elif item["type"] == "text": + content.append(mcp_types.TextContent(type="text", text=item["text"])) + elif item["type"] == "image": + source = item["source"] + content.append( + mcp_types.ImageContent( + type="image", + data=source["data"], + mimeType=source["media_type"], + ) + ) + else: + raise ValueError(f"unsupported Claude tool result block: {item!r}") + + self.run.record( + ToolStep( + call=call, + result=MCPToolResult( + call_id=call_id, + content=content, + isError=block.get("is_error") is True, + ), + started_at=started_at, + ended_at=received_at, + ) + ) + saw_result = True + if saw_result: + self.agent_started_at = received_at case "result": - self._record_result(message) + self.saw_result = True + trace = self.run.trace + result = message.get("result") + if isinstance(result, str): + trace.content = result + if message.get("is_error") is True: + self.error = trace.content or "claude CLI reported an error" + for key in ( + "subtype", + "session_id", + "duration_ms", + "duration_api_ms", + "stop_reason", + "num_turns", + "total_cost_usd", + ): + if (value := message.get(key)) is not None: + trace.extra[key] = value def finish(self, *, returncode: int, stderr: str) -> None: - trace = self._run.trace - error = self._result_error + trace = self.run.trace + error = self.error if returncode != 0: trace.extra["returncode"] = returncode error = stderr.strip() or f"claude CLI exited with return code {returncode}" - elif not self._saw_result: + elif not self.saw_result: error = "claude CLI exited without a result event" - elif self._pending_calls: - missing = ", ".join(sorted(self._pending_calls)) + elif self.pending_calls: + missing = ", ".join(sorted(self.pending_calls)) error = f"claude CLI exited without results for tool calls: {missing}" - if not trace.content and self._last_agent_content: - trace.content = self._last_agent_content if error is not None and stderr: trace.extra["stderr"] = stderr if error is not None: raise RuntimeError(error) - def _record_assistant(self, event: dict[str, Any], received_at: str) -> None: - message = BetaMessage.model_validate(event["message"]) - step = ClaudeAgent._message_to_agent_step(message) - step.started_at = self._agent_started_at - step.ended_at = received_at - if step.content: - self._last_agent_content = step.content - self._run.record(step) - for call in step.tool_calls: - self._pending_calls[call.id] = (call, received_at) - - def _record_tool_results(self, event: dict[str, Any], received_at: str) -> None: - message = event.get("message") - if not isinstance(message, dict): - return - content = message.get("content") - if not isinstance(content, list): - return - saw_result = False - for raw_block in content: - if not isinstance(raw_block, dict) or raw_block.get("type") != "tool_result": - continue - call_id = raw_block.get("tool_use_id") - if not isinstance(call_id, str): - raise ValueError("Claude tool result is missing tool_use_id") - try: - call, started_at = self._pending_calls.pop(call_id) - except KeyError: - raise ValueError( - f"Claude returned a result for unknown tool call {call_id!r}" - ) from None - - raw_result = raw_block.get("content") - raw_items = raw_result if isinstance(raw_result, list) else [raw_result] - result_content: list[mcp_types.ContentBlock] = [] - for item in raw_items: - if isinstance(item, str): - result_content.append(mcp_types.TextContent(type="text", text=item)) - elif isinstance(item, dict) and item.get("type") == "text": - result_content.append(mcp_types.TextContent(type="text", text=item["text"])) - elif isinstance(item, dict) and item.get("type") == "image": - source = item["source"] - result_content.append( - mcp_types.ImageContent( - type="image", - data=source["data"], - mimeType=source["media_type"], - ) - ) - elif item is not None: - raise ValueError(f"unsupported Claude tool result block: {item!r}") - - saw_result = True - self._run.record( - ToolStep( - call=call, - result=MCPToolResult( - call_id=call_id, - content=result_content, - isError=raw_block.get("is_error") is True, - ), - started_at=started_at, - ended_at=received_at, - ) - ) - if saw_result: - self._agent_started_at = received_at - - def _record_result(self, event: dict[str, Any]) -> None: - self._saw_result = True - trace = self._run.trace - result = event.get("result") - trace.content = result if isinstance(result, str) else self._last_agent_content - if event.get("is_error") is True: - self._result_error = trace.content or "claude CLI reported an error" - for key in ( - "subtype", - "session_id", - "duration_ms", - "duration_api_ms", - "stop_reason", - "num_turns", - "total_cost_usd", +def claude_command( + config: ClaudeCLIConfig, + shell: str, + mcp_config_path: str | None = None, + executable: str = "claude", +) -> str: + env: dict[str, str] = {} + use_hud_gateway = config.use_hud_gateway + if use_hud_gateway is None: + use_hud_gateway = settings.api_key is not None + + if use_hud_gateway: + if not settings.api_key: + raise ValueError("HUD_API_KEY is required for HUD gateway routing") + env["ANTHROPIC_BASE_URL"] = settings.hud_gateway_url + env["ANTHROPIC_API_KEY"] = settings.api_key + env["CLAUDE_CODE_DISABLE_EXPERIMENTAL_BETAS"] = "1" + env["DISABLE_AUTO_COMPACT"] = "1" + if trace_id := get_current_trace_id(): + env["ANTHROPIC_CUSTOM_HEADERS"] = f"Trace-Id: {trace_id}" + elif settings.anthropic_api_key: + env["ANTHROPIC_API_KEY"] = settings.anthropic_api_key + + env["ANTHROPIC_MODEL"] = config.model + env["ANTHROPIC_SMALL_FAST_MODEL"] = config.model + + # A custom base URL must own every model tier; otherwise background calls + # can escape to Anthropic instead of using the configured gateway. + if "ANTHROPIC_BASE_URL" in env: + for name in ( + "ANTHROPIC_DEFAULT_SONNET_MODEL", + "ANTHROPIC_DEFAULT_OPUS_MODEL", + "ANTHROPIC_DEFAULT_HAIKU_MODEL", + "CLAUDE_CODE_SUBAGENT_MODEL", ): - value = event.get(key) - if value is not None: - trace.extra[key] = value - - -def _powershell_quote(value: str) -> str: - return "'" + value.replace("'", "''") + "'" - - -def _powershell(script: str) -> str: - encoded = base64.b64encode(script.encode("utf-16-le")).decode("ascii") - return f"powershell -NoProfile -NonInteractive -EncodedCommand {encoded}" + env[name] = config.model + + env["CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC"] = "1" + env["DISABLE_AUTOUPDATER"] = "1" + env["IS_SANDBOX"] = "1" + + args: list[str] = [ + executable, + "--verbose", + "--input-format=stream-json", + "--output-format=stream-json", + "--print", + f"--permission-mode={config.permission_mode}", + ] + if config.max_steps > 0: + args.append(f"--max-turns={config.max_steps}") + if config.system_prompt: + args.extend(["--system-prompt", config.system_prompt]) + for tool in config.allowed_tools: + args.extend(["--allowedTools", tool]) + if mcp_config_path: + args.extend(["--mcp-config", mcp_config_path]) + + if shell in WINDOWS_SHELLS: + script = ";".join( + [ + *(f"$env:{key}={powershell_quote(value)}" for key, value in env.items()), + f"Get-Content -Raw -Encoding UTF8 {powershell_quote(INPUT_PATH)}" + f" | & {powershell_quote(executable)} " + f"{' '.join(powershell_quote(arg) for arg in args[1:])}", + "exit $LASTEXITCODE", + ] + ) + return powershell(script) + + command = " ".join(shlex.quote(arg) for arg in args) + env_prefix = " ".join(f"{key}={shlex.quote(value)}" for key, value in env.items()) + return f'export PATH="$HOME/.local/bin:$PATH"; {env_prefix} {command}' + + +async def run_claude( + config: ClaudeCLIConfig, + run: Run, + *, + ssh: SSHClient, + shell: str, + mcp_servers: dict[str, dict[str, Any]], + prompt: str, + executable: str = "claude", +) -> None: + files: dict[str, str] = {} + input_text = ( + json.dumps( + { + "type": "user", + "message": { + "role": "user", + "content": [{"type": "text", "text": prompt}], + }, + } + ) + + "\n" + ) + mcp_config_path = MCP_CONFIG_PATH if mcp_servers else None + if mcp_servers: + files[MCP_CONFIG_PATH] = json.dumps({"mcpServers": mcp_servers}, indent=2) + if shell in WINDOWS_SHELLS: + files[INPUT_PATH] = input_text + + command = claude_command( + config, + shell, + mcp_config_path=mcp_config_path, + executable=executable, + ) + if shell in WINDOWS_SHELLS: + files[RUN_SCRIPT_PATH] = f"@echo off\r\n{command}\r\n" + command = f"cmd /c {RUN_SCRIPT_PATH}" + + try: + for path, content in files.items(): + await ssh.write_text(path, content) + logger.info("SSH exec claude CLI (%d chars)", len(command)) + events = ClaudeEvents(run, started_at=now_iso()) + returncode, stderr = await run_jsonl( + ssh, + command, + events.consume, + input_text=None if shell in WINDOWS_SHELLS else input_text, + ) + logger.info("exit=%s stderr=%d", returncode, len(stderr)) + events.finish(returncode=returncode, stderr=stderr) + finally: + if files: + if shell in WINDOWS_SHELLS: + cleanup = f"cmd /c del /f /q {' '.join(files)} 2>nul" + else: + cleanup = "rm -f -- " + " ".join(shlex.quote(path) for path in files) + try: + await ssh.run(cleanup, check=False) + except (OSError, asyncssh.Error): + logger.warning("Failed to remove Claude CLI runtime files") class ClaudeCLIAgent(Agent): - """Runs ``claude`` CLI over SSH inside the env workspace. - - Stateless w.r.t. the env: driven by ``await agent(run)``. SSH is opened - live off the run. Environment MCP bindings are used directly; computer MCP - servers are bridged over the run's SSH connection. - """ + """Runs ``claude`` CLI over SSH inside the environment workspace.""" config: ClaudeCLIConfig @@ -205,6 +317,12 @@ async def __call__(self, run: Run) -> None: assert manifest is not None bindings = manifest.bindings shell = ssh.capability.params.get("shell", "bash") + executable = await resolve_executable( + ssh, + "claude", + _MANAGED_CLAUDE_PATHS, + run.runtime_config, + ) rfb_bindings = [cap for cap in bindings if cap.protocol.split("/", 1)[0] == "rfb"] async with AsyncExitStack() as resources: @@ -235,152 +353,15 @@ async def __call__(self, run: Run) -> None: ) ) - await self._run_cli( + await run_claude( + self.config, run, ssh=ssh, shell=shell, mcp_servers=mcp_servers, prompt=run.prompt_text, + executable=executable, ) - async def _run_cli( - self, - run: Run, - *, - ssh: SSHClient, - shell: str, - mcp_servers: dict[str, dict[str, Any]], - prompt: str, - ) -> None: - files: dict[str, str] = {} - mcp_config_path = _MCP_CONFIG_PATH if mcp_servers else None - if mcp_servers: - files[_MCP_CONFIG_PATH] = json.dumps({"mcpServers": mcp_servers}, indent=2) - if shell in _WINDOWS_SHELLS: - files[_PROMPT_PATH] = prompt - - command = self._build_command( - shell=shell, - prompt=prompt, - mcp_config_path=mcp_config_path, - ) - if shell in _WINDOWS_SHELLS: - files[_RUN_SCRIPT_PATH] = f"@echo off\r\n{command}\r\n" - command = f"cmd /c {_RUN_SCRIPT_PATH}" - - try: - for path, content in files.items(): - await ssh.write_text(path, content) - logger.info("SSH exec claude CLI (%d chars)", len(command)) - await self._stream_cli(run, ssh, command) - finally: - if files: - if shell in _WINDOWS_SHELLS: - cleanup = f"cmd /c del /f /q {' '.join(files)} 2>nul" - else: - cleanup = "rm -f -- " + " ".join(shlex.quote(path) for path in files) - try: - await ssh.run(cleanup, check=False) - except (OSError, asyncssh.Error): - logger.warning("Failed to remove Claude CLI runtime files") - - async def _stream_cli(self, run: Run, ssh: SSHClient, command: str) -> None: - parser = _ClaudeStreamParser(run, started_at=now_iso()) - process = await ssh.create_process(command) - stderr_task = asyncio.create_task(process.stderr.read()) - try: - while line := await process.stdout.readline(): - parser.feed_line(line.decode(errors="replace")) - await process.wait_closed() - stderr_output = await stderr_task - except BaseException: - process.close() - if not stderr_task.done(): - stderr_task.cancel() - await asyncio.gather(stderr_task, return_exceptions=True) - with contextlib.suppress(OSError, TimeoutError, asyncssh.Error): - async with asyncio.timeout(_PROCESS_CLOSE_TIMEOUT_S): - await process.wait_closed() - raise - - stderr = stderr_output.decode(errors="replace") - returncode = process.returncode - if returncode is None: - raise RuntimeError("claude CLI process closed without an exit status") - logger.info("exit=%s stderr=%d", returncode, len(stderr)) - parser.finish(returncode=returncode, stderr=stderr) - - def _build_command( - self, - *, - shell: str, - prompt: str, - mcp_config_path: str | None = None, - ) -> str: - env: dict[str, str] = {} - use_hud_gateway = self.config.use_hud_gateway - if use_hud_gateway is None: - use_hud_gateway = settings.api_key is not None - - if use_hud_gateway: - if not settings.api_key: - raise ValueError("HUD_API_KEY is required for HUD gateway routing") - env["ANTHROPIC_BASE_URL"] = settings.hud_gateway_url - env["ANTHROPIC_API_KEY"] = settings.api_key - if trace_id := get_current_trace_id(): - env["ANTHROPIC_CUSTOM_HEADERS"] = f"Trace-Id: {trace_id}" - elif settings.anthropic_api_key: - env["ANTHROPIC_API_KEY"] = settings.anthropic_api_key - - env["ANTHROPIC_MODEL"] = self.config.model - env["ANTHROPIC_SMALL_FAST_MODEL"] = self.config.model - - # When using a custom base URL, alias all model tiers to the same model - # so the CLI doesn't try to reach Anthropic for background requests. - if "ANTHROPIC_BASE_URL" in env: - for name in ( - "ANTHROPIC_DEFAULT_SONNET_MODEL", - "ANTHROPIC_DEFAULT_OPUS_MODEL", - "ANTHROPIC_DEFAULT_HAIKU_MODEL", - "CLAUDE_CODE_SUBAGENT_MODEL", - ): - env[name] = self.config.model - - env["CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC"] = "1" - env["IS_SANDBOX"] = "1" - - base_args: list[str] = [ - "claude", - "--verbose", - "--output-format=stream-json", - "--print", - f"--permission-mode={self.config.permission_mode}", - ] - if self.config.max_steps > 0: - base_args.append(f"--max-turns={self.config.max_steps}") - if self.config.system_prompt: - base_args.extend(["--system-prompt", self.config.system_prompt]) - for tool in self.config.allowed_tools: - base_args.extend(["--allowedTools", tool]) - if mcp_config_path: - base_args.extend(["--mcp-config", mcp_config_path]) - - if shell in _WINDOWS_SHELLS: - script = ";".join( - [ - *(f"$env:{key}={_powershell_quote(value)}" for key, value in env.items()), - f"Get-Content -Raw -Encoding UTF8 {_powershell_quote(_PROMPT_PATH)}" - f" | & claude {' '.join(_powershell_quote(arg) for arg in base_args[1:])}", - "exit $LASTEXITCODE", - ] - ) - return _powershell(script) - - cli_parts = [shlex.quote(a) for a in base_args] - cli_parts.extend(["--", shlex.quote(prompt)]) - cli_cmd = " ".join(cli_parts) - env_prefix = " ".join(f"{key}={shlex.quote(value)}" for key, value in env.items()) - return f'export PATH="$HOME/.local/bin:$PATH"; {env_prefix} {cli_cmd}' - __all__ = ["ClaudeCLIAgent"] diff --git a/hud/agents/cli.py b/hud/agents/cli.py new file mode 100644 index 000000000..d626c024e --- /dev/null +++ b/hud/agents/cli.py @@ -0,0 +1,163 @@ +"""Process boundary for JSONL CLI agents.""" + +from __future__ import annotations + +import asyncio +import base64 +import contextlib +import shlex +from typing import TYPE_CHECKING + +import asyncssh + +if TYPE_CHECKING: + from collections.abc import Callable + + from hud.capabilities import SSHClient + from hud.eval.runtime import RuntimeConfig + +WINDOWS_SHELLS = ("cmd", "powershell") +PROCESS_CLOSE_TIMEOUT_S = 5.0 + + +async def resolve_executable( + ssh: SSHClient, + command: str, + managed_paths: dict[str, str], + runtime_config: RuntimeConfig | None, +) -> str: + """Resolve a CLI against the live SSH target and its declared runtime config.""" + platform = await _runtime_platform(ssh) + _validate_runtime_os(runtime_config, platform.partition("-")[0]) + + managed = managed_paths.get(platform) + if managed is not None: + result = await ssh.run( + f"test -x {shlex.quote(managed)}", + check=False, + encoding=None, + ) + if result.returncode == 0: + return managed + + if platform.startswith("windows-"): + result = await ssh.run(f"where.exe {command}", check=False, encoding=None) + else: + result = await ssh.run(f"command -v -- {command}", check=False, encoding=None) + if result.returncode == 0: + stdout = _output_text(result.stdout) + if path := stdout.splitlines()[0].strip(): + return path + + raise RuntimeError( + f"{command} is unavailable for runtime platform {platform}; " + "install it in the environment or provide a managed runtime bundle" + ) + + +async def _runtime_platform(ssh: SSHClient) -> str: + shell = ssh.capability.params.get("shell", "bash") + if shell in WINDOWS_SHELLS: + result = await ssh.run( + powershell("[System.Runtime.InteropServices.RuntimeInformation]::OSArchitecture"), + check=True, + encoding=None, + ) + arch = _output_text(result.stdout).strip().lower() + return f"windows-{_normalize_arch(arch)}" + + result = await ssh.run( + "uname -s; uname -m; " + "if ls /lib/ld-musl-*.so.1 >/dev/null 2>&1; then echo musl; else echo gnu; fi", + check=True, + encoding=None, + ) + lines = _output_text(result.stdout).splitlines() + if len(lines) != 3: + raise RuntimeError("SSH runtime platform probe returned an invalid response") + system, machine, libc = (line.strip().lower() for line in lines) + os_name = {"darwin": "darwin", "linux": "linux"}.get(system) + if os_name is None: + raise RuntimeError(f"unsupported SSH runtime operating system {system!r}") + platform = f"{os_name}-{_normalize_arch(machine)}" + return f"{platform}-musl" if os_name == "linux" and libc == "musl" else platform + + +def _normalize_arch(value: str) -> str: + normalized = { + "amd64": "x64", + "x86_64": "x64", + "arm64": "arm64", + "aarch64": "arm64", + }.get(value) + if normalized is None: + raise RuntimeError(f"unsupported SSH runtime architecture {value!r}") + return normalized + + +def _output_text(value: bytes | str | None) -> str: + if isinstance(value, bytes): + return value.decode(errors="replace") + return value or "" + + +def _validate_runtime_os(runtime_config: RuntimeConfig | None, actual: str) -> None: + if runtime_config is None or runtime_config.resources is None: + return + declared = runtime_config.resources.os + if declared is None: + return + normalized = { + "darwin": "darwin", + "linux": "linux", + "macos": "darwin", + "windows": "windows", + }.get(declared.lower()) + if normalized is not None and normalized != actual: + raise RuntimeError( + f"runtime_config.resources.os requested {declared!r}, " + f"but the SSH runtime reports {actual!r}" + ) + + +def powershell_quote(value: str) -> str: + return "'" + value.replace("'", "''") + "'" + + +def powershell(script: str) -> str: + encoded = base64.b64encode(script.encode("utf-16-le")).decode("ascii") + return f"powershell -NoProfile -NonInteractive -EncodedCommand {encoded}" + + +async def run_jsonl( + ssh: SSHClient, + command: str, + consume: Callable[[str], None], + *, + input_text: str | None = None, +) -> tuple[int, str]: + """Stream one remote JSONL process and own its cancellation cleanup.""" + process = await ssh.create_process(command) + stderr_task = asyncio.create_task(process.stderr.read()) + try: + if input_text is not None: + process.stdin.write(input_text.encode()) + await process.stdin.drain() + process.stdin.write_eof() + while line := await process.stdout.readline(): + consume(line.decode(errors="replace")) + await process.wait_closed() + stderr = (await stderr_task).decode(errors="replace") + except BaseException: + process.close() + if not stderr_task.done(): + stderr_task.cancel() + await asyncio.gather(stderr_task, return_exceptions=True) + with contextlib.suppress(OSError, TimeoutError, asyncssh.Error): + async with asyncio.timeout(PROCESS_CLOSE_TIMEOUT_S): + await process.wait_closed() + raise + + if process.returncode is None: + raise RuntimeError("CLI process closed without an exit status") + return process.returncode, stderr diff --git a/hud/agents/codex/__init__.py b/hud/agents/codex/__init__.py new file mode 100644 index 000000000..0b5e99611 --- /dev/null +++ b/hud/agents/codex/__init__.py @@ -0,0 +1,7 @@ +"""Codex CLI agent.""" + +from hud.agents.types import CodexCLIConfig + +from .agent import CodexCLIAgent + +__all__ = ["CodexCLIAgent", "CodexCLIConfig"] diff --git a/hud/agents/codex/agent.py b/hud/agents/codex/agent.py new file mode 100644 index 000000000..ed4f6946b --- /dev/null +++ b/hud/agents/codex/agent.py @@ -0,0 +1,328 @@ +"""Codex CLI harness over a workspace SSH capability.""" + +from __future__ import annotations + +import json +import logging +import shlex +from typing import TYPE_CHECKING, Any, cast + +import mcp.types as mcp_types + +from hud.agents.base import Agent +from hud.agents.cli import ( + WINDOWS_SHELLS, + powershell, + powershell_quote, + resolve_executable, + run_jsonl, +) +from hud.agents.types import AgentStep, CodexCLIConfig, ToolStep +from hud.settings import settings +from hud.telemetry.context import get_current_trace_id +from hud.types import MCPToolCall, MCPToolResult, Step +from hud.utils.time import now_iso + +if TYPE_CHECKING: + from hud.capabilities import SSHClient + from hud.eval.run import Run + +logger = logging.getLogger(__name__) + +_MANAGED_CODEX_PATHS = { + "linux-x64": "/media/hud/bin/codex/bin/codex", + "linux-x64-musl": "/media/hud/bin/codex/bin/codex", +} + + +class CodexEvents: + """Translate ``codex exec --json`` events into canonical HUD steps.""" + + def __init__(self, run: Run, *, model: str, started_at: str) -> None: + self.run = run + self.model = model + self.agent_started_at = started_at + self.item_started_at: dict[str, str] = {} + self.saw_completion = False + self.error: str | None = None + + def consume(self, line: str) -> None: + line = line.strip() + if not line: + return + event = json.loads(line) + if not isinstance(event, dict): + raise ValueError("Codex stream event must be an object") + + received_at = now_iso() + match event.get("type"): + case "thread.started": + self.run.trace.extra["codex_thread_id"] = event["thread_id"] + case "turn.started": + self.agent_started_at = received_at + case "item.started": + item = event["item"] + self.item_started_at[item["id"]] = received_at + case "item.completed": + self.record(event["item"], received_at) + case "turn.completed": + self.run.trace.extra["usage"] = event["usage"] + self.saw_completion = True + case "turn.failed": + self.error = event["error"]["message"] + case "error": + self.error = event["message"] + + def finish(self, *, returncode: int, stderr: str) -> None: + trace = self.run.trace + error = self.error + if returncode != 0: + trace.extra["returncode"] = returncode + error = error or stderr.strip() or f"codex CLI exited with return code {returncode}" + elif error is None and not self.saw_completion: + error = "codex CLI exited without a turn.completed event" + + if error is not None and stderr and self.error is None: + trace.extra["stderr"] = stderr + if error is not None: + raise RuntimeError(error) + + def record(self, item: dict[str, Any], received_at: str) -> None: + item_id = item["id"] + started_at = self.item_started_at.pop(item_id, self.agent_started_at) + match item["type"]: + case "agent_message": + text = item["text"] + self.run.trace.content = text + self.run.record( + AgentStep( + content=text, + model=self.model, + raw=item, + started_at=started_at, + ended_at=received_at, + ) + ) + case "reasoning": + self.run.record( + AgentStep( + reasoning=item["text"], + model=self.model, + raw=item, + started_at=started_at, + ended_at=received_at, + ) + ) + case "command_execution" | "file_change" | "mcp_tool_call" | "web_search": + self.record_tool(item, started_at, received_at) + case _: + self.run.record( + Step( + source="agent", + extra={"codex_item": item}, + started_at=started_at, + ended_at=received_at, + ) + ) + self.agent_started_at = received_at + + def record_tool(self, item: dict[str, Any], started_at: str, ended_at: str) -> None: + call_id = item["id"] + match item["type"]: + case "command_execution": + call = MCPToolCall( + id=call_id, + name="shell", + arguments={"command": item["command"]}, + ) + result = MCPToolResult( + call_id=call_id, + content=[mcp_types.TextContent(type="text", text=item["aggregated_output"])], + isError=item["status"] != "completed" or item["exit_code"] not in (None, 0), + ) + case "file_change": + changes = item["changes"] + call = MCPToolCall( + id=call_id, + name="apply_patch", + arguments={"changes": changes}, + ) + result = MCPToolResult( + call_id=call_id, + content=[ + mcp_types.TextContent( + type="text", + text="\n".join( + f"{change['kind']}: {change['path']}" for change in changes + ), + ) + ], + isError=item["status"] != "completed", + ) + case "mcp_tool_call": + call = MCPToolCall( + id=call_id, + name=item["tool"], + provider_name=f"{item['server']}.{item['tool']}", + arguments=item["arguments"], + ) + raw_result = item.get("result") or {} + error = item.get("error") + result = MCPToolResult.model_validate( + { + "call_id": call_id, + "content": raw_result.get("content") + or ([{"type": "text", "text": error["message"]}] if error else []), + "structuredContent": raw_result.get("structured_content"), + "_meta": raw_result.get("_meta"), + "isError": item["status"] == "failed", + } + ) + case "web_search": + call = MCPToolCall( + id=call_id, + name="web_search", + arguments={"query": item["query"], "action": item["action"]}, + ) + result = MCPToolResult( + call_id=call_id, + content=[ + mcp_types.TextContent( + type="text", + text=json.dumps(item["action"], separators=(",", ":")), + ) + ], + isError=False, + ) + case _: + raise ValueError(f"unsupported Codex tool item {item['type']!r}") + + self.run.record( + ToolStep( + call=call, + result=result, + extra={"codex_item": item}, + started_at=started_at, + ended_at=ended_at, + ) + ) + + +def codex_command(config: CodexCLIConfig, shell: str, executable: str = "codex") -> str: + env: dict[str, str] = {} + args = [ + executable, + "exec", + "--json", + "--ephemeral", + "--skip-git-repo-check", + "--color", + "never", + "--sandbox", + config.sandbox, + "--model", + config.model, + ] + + use_hud_gateway = config.use_hud_gateway + if use_hud_gateway is None: + use_hud_gateway = settings.api_key is not None + if use_hud_gateway: + if not settings.api_key: + raise ValueError("HUD_API_KEY is required for HUD gateway routing") + env["HUD_API_KEY"] = settings.api_key + overrides = { + "model_provider": "hud", + "model_providers.hud.name": "HUD", + "model_providers.hud.base_url": settings.hud_gateway_url, + "model_providers.hud.env_key": "HUD_API_KEY", + "model_providers.hud.wire_api": "responses", + } + for key, value in overrides.items(): + args.extend(["-c", f"{key}={json.dumps(value)}"]) + if trace_id := get_current_trace_id(): + args.extend( + [ + "-c", + f'model_providers.hud.http_headers={{"Trace-Id"={json.dumps(trace_id)}}}', + ] + ) + elif settings.openai_api_key: + env["CODEX_API_KEY"] = settings.openai_api_key + + args.append("-") + if shell in WINDOWS_SHELLS: + script = ";".join( + [ + "$codexHome=Join-Path ([System.IO.Path]::GetTempPath()) " + "('hud-codex-' + [System.Guid]::NewGuid())", + "New-Item -ItemType Directory -Force -Path $codexHome | Out-Null", + "$env:CODEX_HOME=$codexHome", + *(f"$env:{key}={powershell_quote(value)}" for key, value in env.items()), + f"try {{ & {powershell_quote(executable)} " + f"{' '.join(powershell_quote(arg) for arg in args[1:])}; " + "$hudExitCode=$LASTEXITCODE } finally { Remove-Item -Recurse -Force " + "$codexHome }", + "exit $hudExitCode", + ] + ) + return powershell(script) + + command = " ".join(shlex.quote(arg) for arg in args) + env_prefix = " ".join(f"{key}={shlex.quote(value)}" for key, value in env.items()) + invocation = f"{env_prefix} {command}" if env_prefix else command + return "; ".join( + [ + 'codex_home=$(mktemp -d "${TMPDIR:-/tmp}/hud-codex.XXXXXX") || exit 1', + "trap 'rm -rf -- \"$codex_home\"' EXIT", + 'export CODEX_HOME="$codex_home"', + 'export PATH="$HOME/.local/bin:$PATH"', + invocation, + ] + ) + + +async def run_codex( + config: CodexCLIConfig, + run: Run, + *, + ssh: SSHClient, + shell: str, + prompt: str, + executable: str = "codex", +) -> None: + command = codex_command(config, shell, executable) + logger.info("SSH exec codex CLI (%d chars)", len(command)) + events = CodexEvents(run, model=config.model, started_at=now_iso()) + returncode, stderr = await run_jsonl(ssh, command, events.consume, input_text=prompt) + logger.info("exit=%s stderr=%d", returncode, len(stderr)) + events.finish(returncode=returncode, stderr=stderr) + + +class CodexCLIAgent(Agent): + """Runs ``codex exec`` over SSH inside the environment workspace.""" + + config: CodexCLIConfig + + def __init__(self, config: CodexCLIConfig | None = None) -> None: + self.config = config or CodexCLIConfig() + + async def __call__(self, run: Run) -> None: + ssh = cast("SSHClient", await run.client.open("ssh")) + executable = await resolve_executable( + ssh, + "codex", + _MANAGED_CODEX_PATHS, + run.runtime_config, + ) + await run_codex( + self.config, + run, + ssh=ssh, + shell=ssh.capability.params.get("shell", "bash"), + prompt=run.prompt_text, + executable=executable, + ) + + +__all__ = ["CodexCLIAgent"] diff --git a/hud/agents/tests/test_base.py b/hud/agents/tests/test_base.py index b88faf0e0..4af61fe76 100644 --- a/hud/agents/tests/test_base.py +++ b/hud/agents/tests/test_base.py @@ -13,6 +13,7 @@ from hud.agents import ( ClaudeCLIAgent, + CodexCLIAgent, OpenAIAgent, OpenAIChatAgent, create_agent, @@ -51,6 +52,8 @@ def test_agent_type_maps_value_to_class_and_provider() -> None: def test_agent_type_registers_cli_agent() -> None: assert AgentType("claude_cli").cls is ClaudeCLIAgent assert AgentType.of(ClaudeCLIAgent()) == AgentType.CLAUDE_CLI + assert AgentType("codex_cli").cls is CodexCLIAgent + assert AgentType.of(CodexCLIAgent()) == AgentType.CODEX_CLI def test_cli_agent_round_trips_through_registered_wire_format() -> None: @@ -65,6 +68,18 @@ def test_cli_agent_round_trips_through_registered_wire_format() -> None: assert loaded.config == agent.config +def test_codex_cli_agent_round_trips_through_registry() -> None: + agent = CodexCLIAgent() + spec = dump_agent(agent) + + loaded = load_agent(spec) + + assert spec["type"] == "codex_cli" + assert spec["config"]["model"] == "gpt-5.6-sol" + assert isinstance(loaded, CodexCLIAgent) + assert loaded.config == agent.config + + def test_dump_agent_rejects_unregistered_agent() -> None: with pytest.raises(ValueError, match="registered types"): dump_agent(_FillingAgent()) diff --git a/hud/agents/tests/test_claude_cli_agent.py b/hud/agents/tests/test_claude_cli_agent.py index cb9ff492a..007e1a35c 100644 --- a/hud/agents/tests/test_claude_cli_agent.py +++ b/hud/agents/tests/test_claude_cli_agent.py @@ -24,7 +24,7 @@ from mcp.types import ImageContent, TextContent from hud.agents.claude.cli import computer_mcp -from hud.agents.claude.cli.agent import ClaudeCLIAgent +from hud.agents.claude.cli.agent import ClaudeCLIAgent, claude_command, run_claude from hud.agents.types import AgentStep, ClaudeCLIConfig, ToolStep from hud.capabilities import Capability, SSHClient from hud.capabilities.rfb import WebPScreenshotEncoding @@ -40,23 +40,27 @@ def _clear_api_keys(monkeypatch: pytest.MonkeyPatch) -> None: monkeypatch.setattr(settings, "api_key", None) monkeypatch.setattr(settings, "anthropic_api_key", None) + monkeypatch.setattr( + "hud.agents.claude.cli.agent.resolve_executable", + AsyncMock(return_value="claude"), + ) def test_command_follows_explicit_gateway_routing(monkeypatch: pytest.MonkeyPatch) -> None: monkeypatch.setattr(settings, "api_key", "hud-key") monkeypatch.setattr(settings, "anthropic_api_key", "anthropic-key") - gateway = ClaudeCLIAgent(ClaudeCLIConfig(use_hud_gateway=True))._build_command( - shell="bash", prompt="run", mcp_config_path=None - ) - provider = ClaudeCLIAgent(ClaudeCLIConfig(use_hud_gateway=False))._build_command( - shell="bash", prompt="run", mcp_config_path=None - ) + gateway = claude_command(ClaudeCLIConfig(use_hud_gateway=True), "bash") + provider = claude_command(ClaudeCLIConfig(use_hud_gateway=False), "bash") assert f"ANTHROPIC_BASE_URL={settings.hud_gateway_url}" in gateway assert "ANTHROPIC_API_KEY=hud-key" in gateway + assert "CLAUDE_CODE_DISABLE_EXPERIMENTAL_BETAS=1" in gateway + assert "DISABLE_AUTO_COMPACT=1" in gateway assert "ANTHROPIC_API_KEY=anthropic-key" in provider assert "ANTHROPIC_BASE_URL" not in provider + assert "CLAUDE_CODE_DISABLE_EXPERIMENTAL_BETAS" not in provider + assert "DISABLE_AUTO_COMPACT" not in provider assert "ANTHROPIC_MODEL=claude-sonnet-5" in provider @@ -64,25 +68,21 @@ def test_windows_command_encodes_environment_and_arguments( monkeypatch: pytest.MonkeyPatch, ) -> None: monkeypatch.setattr(settings, "api_key", "hud&key's") - agent = ClaudeCLIAgent( - ClaudeCLIConfig( - use_hud_gateway=True, - max_steps=3, - system_prompt="don't $expand", - ) - ) - - command = agent._build_command( - shell="powershell", - prompt="not embedded", + config = ClaudeCLIConfig( + use_hud_gateway=True, + max_steps=3, + system_prompt="don't $expand", ) + command = claude_command(config, "powershell") encoded = command.rsplit(" ", 1)[1] script = base64.b64decode(encoded).decode("utf-16-le") assert "$env:ANTHROPIC_API_KEY='hud&key''s'" in script + assert "$env:CLAUDE_CODE_DISABLE_EXPERIMENTAL_BETAS='1'" in script + assert "$env:DISABLE_AUTO_COMPACT='1'" in script assert "'--system-prompt' 'don''t $expand'" in script - assert "Get-Content -Raw -Encoding UTF8 '.hud_prompt.txt' | & claude" in script - assert "not embedded" not in script + assert "Get-Content -Raw -Encoding UTF8 '.hud_input.jsonl' | & 'claude'" in script + assert "'--input-format=stream-json'" in script assert "python" not in script @@ -120,6 +120,7 @@ def __init__( returncode: int | None = None, pause_after: int | None = None, ) -> None: + self.stdin = _FakeWriter() self.stdout = _FakeReader(stdout, pause_after=pause_after) self.stderr = _FakeReader(stderr) self.exit_status = exit_status @@ -137,6 +138,21 @@ async def wait_closed(self) -> None: pass +class _FakeWriter: + def __init__(self) -> None: + self.data = bytearray() + self.eof = False + + def write(self, data: bytes) -> None: + self.data.extend(data) + + async def drain(self) -> None: + pass + + def write_eof(self) -> None: + self.eof = True + + class _FakeCompletedProcess: async def wait(self, *, check: bool, **kwargs: Any) -> Any: del check @@ -180,7 +196,7 @@ async def create_process(self, cmd: str, **kwargs: Any) -> Any: script = base64.b64decode(match.group(1)).decode("utf-16-le") name = next( path - for path in (".hud_prompt.txt", ".hud_run.bat", ".hud_mcp_config.json") + for path in (".hud_input.jsonl", ".hud_run.bat", ".hud_mcp_config.json") if path in script ) if input_value is not None: @@ -234,18 +250,26 @@ def _ssh_with_conn(shell: str, conn: _FakeConn) -> SSHClient: async def test_exec_on_windows_writes_batch_and_execs_via_cmd() -> None: sink: dict[str, bytes] = {} conn = _FakeConn(sink, _FakeStreamProcess(_STREAM_JSON)) - agent = ClaudeCLIAgent() ssh = _ssh_with_conn("cmd", conn) run = _fake_run() - await agent._run_cli(run, ssh=ssh, shell="cmd", mcp_servers={}, prompt="build it") + await run_claude( + ClaudeCLIConfig(), run, ssh=ssh, shell="cmd", mcp_servers={}, prompt="build it" + ) assert conn.ran == ["cmd /c .hud_run.bat"] assert all(command.startswith("powershell ") for command in conn.write_commands) assert conn.written[".hud_run.bat"].startswith(b"@echo off\r\n") - assert conn.written[".hud_prompt.txt"] == b"build it" + assert conn.written[".hud_input.jsonl"].endswith(b"\n") + assert json.loads(conn.written[".hud_input.jsonl"]) == { + "type": "user", + "message": { + "role": "user", + "content": [{"type": "text", "text": "build it"}], + }, + } assert sink == {} - assert set(conn.deleted) == {".hud_prompt.txt", ".hud_run.bat"} + assert set(conn.deleted) == {".hud_input.jsonl", ".hud_run.bat"} assert run.trace.status is None assert run.trace.content == "done" assert "messages" not in run.trace.extra @@ -253,18 +277,31 @@ async def test_exec_on_windows_writes_batch_and_execs_via_cmd() -> None: async def test_exec_on_bash_runs_inline_without_batch() -> None: sink: dict[str, bytes] = {} - conn = _FakeConn(sink, _FakeStreamProcess(_STREAM_JSON)) - agent = ClaudeCLIAgent() + process = _FakeStreamProcess(_STREAM_JSON) + conn = _FakeConn(sink, process) ssh = _ssh_with_conn("bash", conn) run = _fake_run() - await agent._run_cli(run, ssh=ssh, shell="bash", mcp_servers={}, prompt="build it") + await run_claude( + ClaudeCLIConfig(), run, ssh=ssh, shell="bash", mcp_servers={}, prompt="build it" + ) assert sink == {} assert conn.write_commands == [] assert conn.deleted == [] assert len(conn.ran) == 1 assert "claude" in conn.ran[0] + assert "--input-format=stream-json" in conn.ran[0] + assert "build it" not in conn.ran[0] + assert process.stdin.data.endswith(b"\n") + assert json.loads(process.stdin.data) == { + "type": "user", + "message": { + "role": "user", + "content": [{"type": "text", "text": "build it"}], + }, + } + assert process.stdin.eof is True assert run.trace.status is None assert run.trace.content == "done" assert "messages" not in run.trace.extra @@ -273,9 +310,8 @@ async def test_exec_on_bash_runs_inline_without_batch() -> None: async def test_exec_removes_mcp_config_after_run() -> None: sink: dict[str, bytes] = {} conn = _FakeConn(sink, _FakeStreamProcess(_STREAM_JSON)) - agent = ClaudeCLIAgent() - - await agent._run_cli( + await run_claude( + ClaudeCLIConfig(), _fake_run(), ssh=_ssh_with_conn("bash", conn), shell="bash", @@ -293,12 +329,18 @@ async def test_exec_removes_mcp_config_after_run() -> None: async def test_exec_records_steps_before_process_exit() -> None: process = _FakeStreamProcess(_STREAM_JSON, pause_after=1) conn = _FakeConn({}, process) - agent = ClaudeCLIAgent() ssh = _ssh_with_conn("bash", conn) run = _fake_run() execution = asyncio.create_task( - agent._run_cli(run, ssh=ssh, shell="bash", mcp_servers={}, prompt="edit it") + run_claude( + ClaudeCLIConfig(), + run, + ssh=ssh, + shell="bash", + mcp_servers={}, + prompt="edit it", + ) ) await process.stdout.blocked.wait() @@ -336,9 +378,10 @@ async def test_exec_forwards_trace_id_only_to_hud_gateway( monkeypatch.setattr(settings, "anthropic_api_key", "anthropic-key") gateway_conn = _FakeConn({}, _FakeStreamProcess(_STREAM_JSON)) - gateway = ClaudeCLIAgent(ClaudeCLIConfig(use_hud_gateway=True)) + gateway = ClaudeCLIConfig(use_hud_gateway=True) with set_trace_context("trace-123"): - await gateway._run_cli( + await run_claude( + gateway, _fake_run(), ssh=_ssh_with_conn("bash", gateway_conn), shell="bash", @@ -348,9 +391,10 @@ async def test_exec_forwards_trace_id_only_to_hud_gateway( assert "ANTHROPIC_CUSTOM_HEADERS='Trace-Id: trace-123'" in gateway_conn.ran[0] provider_conn = _FakeConn({}, _FakeStreamProcess(_STREAM_JSON)) - provider = ClaudeCLIAgent(ClaudeCLIConfig(use_hud_gateway=False)) + provider = ClaudeCLIConfig(use_hud_gateway=False) with set_trace_context("trace-123"): - await provider._run_cli( + await run_claude( + provider, _fake_run(), ssh=_ssh_with_conn("bash", provider_conn), shell="bash", @@ -363,10 +407,9 @@ async def test_exec_forwards_trace_id_only_to_hud_gateway( async def test_exec_closes_streaming_process_when_cancelled() -> None: process = _FakeStreamProcess(_STREAM_JSON, pause_after=0) conn = _FakeConn({}, process) - agent = ClaudeCLIAgent() - execution = asyncio.create_task( - agent._run_cli( + run_claude( + ClaudeCLIConfig(), _fake_run(), ssh=_ssh_with_conn("bash", conn), shell="bash", @@ -386,12 +429,11 @@ async def test_exec_closes_streaming_process_when_cancelled() -> None: async def test_exec_nonzero_exit_with_no_stdout_raises() -> None: sink: dict[str, bytes] = {} conn = _FakeConn(sink, _FakeStreamProcess("", stderr="boom", exit_status=1)) - agent = ClaudeCLIAgent() ssh = _ssh_with_conn("cmd", conn) run = _fake_run() with pytest.raises(RuntimeError, match="boom"): - await agent._run_cli(run, ssh=ssh, shell="cmd", mcp_servers={}, prompt="x") + await run_claude(ClaudeCLIConfig(), run, ssh=ssh, shell="cmd", mcp_servers={}, prompt="x") assert run.trace.extra["returncode"] == 1 @@ -402,12 +444,11 @@ async def test_exec_signal_exit_records_the_returncode() -> None: sink, _FakeStreamProcess("", exit_status=None, returncode=-15), ) - agent = ClaudeCLIAgent() ssh = _ssh_with_conn("bash", conn) run = _fake_run() with pytest.raises(RuntimeError, match="return code -15"): - await agent._run_cli(run, ssh=ssh, shell="bash", mcp_servers={}, prompt="x") + await run_claude(ClaudeCLIConfig(), run, ssh=ssh, shell="bash", mcp_servers={}, prompt="x") assert run.trace.extra["returncode"] == -15 @@ -418,12 +459,11 @@ async def test_exec_nonzero_exit_with_result_stream_remains_an_error() -> None: sink, _FakeStreamProcess(_STREAM_JSON, stderr="transport failed", exit_status=1), ) - agent = ClaudeCLIAgent() ssh = _ssh_with_conn("bash", conn) run = _fake_run() with pytest.raises(RuntimeError, match="transport failed"): - await agent._run_cli(run, ssh=ssh, shell="bash", mcp_servers={}, prompt="x") + await run_claude(ClaudeCLIConfig(), run, ssh=ssh, shell="bash", mcp_servers={}, prompt="x") assert run.trace.content == "done" assert run.trace.extra["returncode"] == 1 @@ -435,12 +475,11 @@ async def test_exec_zero_exit_without_result_event_is_an_error() -> None: sink: dict[str, bytes] = {} stdout = _STREAM_JSON.rsplit('{"type":"result"', 1)[0] conn = _FakeConn(sink, _FakeStreamProcess(stdout)) - agent = ClaudeCLIAgent() ssh = _ssh_with_conn("bash", conn) run = _fake_run() with pytest.raises(RuntimeError, match="without a result event"): - await agent._run_cli(run, ssh=ssh, shell="bash", mcp_servers={}, prompt="x") + await run_claude(ClaudeCLIConfig(), run, ssh=ssh, shell="bash", mcp_servers={}, prompt="x") assert run.trace.content == "done" @@ -476,12 +515,12 @@ async def open(self, ref: str) -> SSHClient: agent = ClaudeCLIAgent() execute = AsyncMock() - monkeypatch.setattr(agent, "_run_cli", execute) + monkeypatch.setattr("hud.agents.claude.cli.agent.run_claude", execute) await agent( cast( "Any", - SimpleNamespace(client=Client(), prompt_text="call the tool"), + SimpleNamespace(client=Client(), prompt_text="call the tool", runtime_config=None), ) ) @@ -547,12 +586,12 @@ async def execute(*_args: Any, **_kwargs: Any) -> None: execute_mock = AsyncMock(side_effect=execute) monkeypatch.setattr(computer_mcp, "bridge_computer_mcp", bridge) - monkeypatch.setattr(agent, "_run_cli", execute_mock) + monkeypatch.setattr("hud.agents.claude.cli.agent.run_claude", execute_mock) await agent( cast( "Any", - SimpleNamespace(client=Client(), prompt_text="use the computer"), + SimpleNamespace(client=Client(), prompt_text="use the computer", runtime_config=None), ) ) @@ -634,9 +673,18 @@ async def execute(*_args: Any, **kwargs: Any) -> None: agent = ClaudeCLIAgent() monkeypatch.setattr(computer_mcp, "bridge_computer_mcp", bridge) - monkeypatch.setattr(agent, "_run_cli", execute) + monkeypatch.setattr("hud.agents.claude.cli.agent.run_claude", execute) - await agent(cast("Any", SimpleNamespace(client=Client(), prompt_text="use both screens"))) + await agent( + cast( + "Any", + SimpleNamespace( + client=Client(), + prompt_text="use both screens", + runtime_config=None, + ), + ) + ) assert bridged == [] @@ -883,6 +931,7 @@ async def open(self, ref: str) -> SSHClient: seen: list[tuple[Any, SSHClient, str]] = [] async def execute( + _config: ClaudeCLIConfig, run: Any, *, ssh: SSHClient, @@ -897,9 +946,11 @@ async def execute( await release_first.wait() agent = ClaudeCLIAgent() - monkeypatch.setattr(agent, "_run_cli", execute) - run_a = SimpleNamespace(client=Client(shell_a, ssh_a), prompt_text="first") - run_b = SimpleNamespace(client=Client(shell_b, ssh_b), prompt_text="second") + monkeypatch.setattr("hud.agents.claude.cli.agent.run_claude", execute) + run_a = SimpleNamespace(client=Client(shell_a, ssh_a), prompt_text="first", runtime_config=None) + run_b = SimpleNamespace( + client=Client(shell_b, ssh_b), prompt_text="second", runtime_config=None + ) first = asyncio.create_task(agent(cast("Any", run_a))) await first_entered.wait() diff --git a/hud/agents/tests/test_codex_cli_agent.py b/hud/agents/tests/test_codex_cli_agent.py new file mode 100644 index 000000000..53f739f0b --- /dev/null +++ b/hud/agents/tests/test_codex_cli_agent.py @@ -0,0 +1,399 @@ +"""CodexCLIAgent command construction and JSONL trajectory mapping.""" + +from __future__ import annotations + +import asyncio +import base64 +from types import SimpleNamespace +from typing import Any, cast +from unittest.mock import AsyncMock + +import pytest +from mcp.types import TextContent + +from hud.agents.cli import resolve_executable +from hud.agents.codex import CodexCLIAgent +from hud.agents.codex.agent import codex_command, run_codex +from hud.agents.types import AgentStep, CodexCLIConfig, ToolStep +from hud.capabilities import Capability, SSHClient +from hud.eval.runtime import RuntimeConfig, RuntimeResources +from hud.settings import settings +from hud.telemetry.context import set_trace_context + + +@pytest.fixture(autouse=True) +def _clear_api_keys(monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.setattr(settings, "api_key", None) + monkeypatch.setattr(settings, "openai_api_key", None) + monkeypatch.setattr( + "hud.agents.codex.agent.resolve_executable", + AsyncMock(return_value="codex"), + ) + + +class _FakeReader: + def __init__(self, value: str, *, pause_after: int | None = None) -> None: + self._raw = value.encode() + self._lines = self._raw.splitlines(keepends=True) + self._pause_after = pause_after + self._index = 0 + self.blocked = asyncio.Event() + self.release = asyncio.Event() + + async def readline(self) -> bytes: + if self._pause_after == self._index: + self.blocked.set() + await self.release.wait() + self._pause_after = None + if self._index == len(self._lines): + return b"" + line = self._lines[self._index] + self._index += 1 + return line + + async def read(self) -> bytes: + return self._raw + + +class _FakeWriter: + def __init__(self) -> None: + self.data = b"" + self.eof = False + + def write(self, data: bytes) -> None: + self.data += data + + async def drain(self) -> None: + pass + + def write_eof(self) -> None: + self.eof = True + + +class _FakeProcess: + def __init__( + self, + stdout: str, + *, + stderr: str = "", + returncode: int | None = 0, + pause_after: int | None = None, + ) -> None: + self.stdin = _FakeWriter() + self.stdout = _FakeReader(stdout, pause_after=pause_after) + self.stderr = _FakeReader(stderr) + self.returncode = returncode + self.closed = False + + def close(self) -> None: + self.closed = True + + async def wait_closed(self) -> None: + pass + + +class _FakeSSH: + def __init__(self, process: _FakeProcess, *, shell: str = "bash") -> None: + self.process = process + self.capability = Capability( + name="shell", + protocol="ssh/2", + url="ssh://localhost:22", + params={"shell": shell}, + ) + self.commands: list[str] = [] + + async def create_process(self, command: str) -> _FakeProcess: + self.commands.append(command) + return self.process + + +def _fake_run() -> Any: + trace = SimpleNamespace(status=None, content="", extra={}) + steps: list[Any] = [] + return SimpleNamespace(trace=trace, record=steps.append, steps=steps) + + +_STREAM_JSON = ( + '{"type":"thread.started","thread_id":"thread-1"}\n' + '{"type":"turn.started"}\n' + '{"type":"item.started","item":{"id":"cmd-1","type":"command_execution",' + '"command":"pytest -q","aggregated_output":"","exit_code":null,' + '"status":"in_progress"}}\n' + '{"type":"item.completed","item":{"id":"cmd-1","type":"command_execution",' + '"command":"pytest -q","aggregated_output":"1 passed\\n","exit_code":0,' + '"status":"completed"}}\n' + '{"type":"item.completed","item":{"id":"patch-1","type":"file_change",' + '"changes":[{"path":"calc.py","kind":"update"}],"status":"completed"}}\n' + '{"type":"item.started","item":{"id":"mcp-1","type":"mcp_tool_call",' + '"server":"db","tool":"query","arguments":{"sql":"select 42"},' + '"result":null,"error":null,"status":"in_progress"}}\n' + '{"type":"item.completed","item":{"id":"mcp-1","type":"mcp_tool_call",' + '"server":"db","tool":"query","arguments":{"sql":"select 42"},' + '"result":{"content":[{"type":"text","text":"42"}],' + '"structured_content":{"answer":42}},"error":null,"status":"completed"}}\n' + '{"type":"item.completed","item":{"id":"search-1","type":"web_search",' + '"query":"HUD evals","action":{"type":"search"}}}\n' + '{"type":"item.completed","item":{"id":"reason-1","type":"reasoning",' + '"text":"The test now passes."}}\n' + '{"type":"item.completed","item":{"id":"message-1","type":"agent_message",' + '"text":"Implemented and verified."}}\n' + '{"type":"turn.completed","usage":{"input_tokens":20,"cached_input_tokens":5,' + '"output_tokens":8,"reasoning_output_tokens":3}}\n' +) + + +def test_command_follows_explicit_gateway_routing(monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.setattr(settings, "api_key", "hud-key") + monkeypatch.setattr(settings, "openai_api_key", "openai-key") + + with set_trace_context("trace-123"): + gateway = codex_command(CodexCLIConfig(use_hud_gateway=True), "bash") + provider = codex_command(CodexCLIConfig(use_hud_gateway=False), "bash") + + assert "HUD_API_KEY=hud-key" in gateway + assert 'model_provider="hud"' in gateway + assert f'model_providers.hud.base_url="{settings.hud_gateway_url}"' in gateway + assert "Trace-Id" in gateway + assert "CODEX_API_KEY=openai-key" in provider + assert "model_provider" not in provider + for command in (gateway, provider): + assert "codex exec" in command + assert "--json" in command + assert "--ephemeral" in command + assert "--ignore-user-config" not in command + assert "mktemp -d" in command + assert 'export CODEX_HOME="$codex_home"' in command + assert "--sandbox workspace-write" in command + assert "--model gpt-5.6-sol" in command + assert command.endswith(" -") + + +def test_windows_command_encodes_environment_and_arguments( + monkeypatch: pytest.MonkeyPatch, +) -> None: + monkeypatch.setattr(settings, "openai_api_key", "key&value's") + config = CodexCLIConfig(use_hud_gateway=False, sandbox="danger-full-access") + command = codex_command(config, "powershell") + + script = base64.b64decode(command.rsplit(" ", 1)[1]).decode("utf-16-le") + assert "$env:CODEX_API_KEY='key&value''s'" in script + assert "$env:CODEX_HOME=$codexHome" in script + assert "[System.Guid]::NewGuid()" in script + assert "Remove-Item -Recurse -Force $codexHome" in script + assert "--ignore-user-config" not in script + assert "'--sandbox' 'danger-full-access'" in script + assert "& 'codex' 'exec'" in script + assert script.endswith(";exit $hudExitCode") + + +async def test_exec_streams_prompt_and_records_codex_items() -> None: + process = _FakeProcess(_STREAM_JSON) + ssh = _FakeSSH(process) + run = _fake_run() + + await run_codex( + CodexCLIConfig(), + run, + ssh=cast("SSHClient", ssh), + shell="bash", + prompt="Fix the failing test", + ) + + assert process.stdin.data == b"Fix the failing test" + assert process.stdin.eof + assert [type(step) for step in run.steps] == [ + ToolStep, + ToolStep, + ToolStep, + ToolStep, + AgentStep, + AgentStep, + ] + command = cast("ToolStep", run.steps[0]) + assert command.call is not None + assert command.call.name == "shell" + assert command.call.arguments == {"command": "pytest -q"} + assert command.result is not None + assert command.result.isError is False + output = command.result.content[0] + assert isinstance(output, TextContent) + assert output.text == "1 passed\n" + patch = cast("ToolStep", run.steps[1]) + assert patch.call is not None + assert patch.call.name == "apply_patch" + mcp = cast("ToolStep", run.steps[2]) + assert mcp.call is not None + assert mcp.call.name == "query" + assert mcp.call.provider_name == "db.query" + assert mcp.result is not None + assert mcp.result.structuredContent == {"answer": 42} + search = cast("ToolStep", run.steps[3]) + assert search.call is not None + assert search.call.name == "web_search" + assert cast("AgentStep", run.steps[4]).reasoning == "The test now passes." + assert cast("AgentStep", run.steps[5]).content == "Implemented and verified." + assert run.trace.content == "Implemented and verified." + assert run.trace.extra["codex_thread_id"] == "thread-1" + assert run.trace.extra["usage"]["cached_input_tokens"] == 5 + assert run.trace.status is None + + +async def test_exec_records_completed_items_before_process_exit() -> None: + process = _FakeProcess(_STREAM_JSON, pause_after=4) + ssh = _FakeSSH(process) + run = _fake_run() + execution = asyncio.create_task( + run_codex( + CodexCLIConfig(), + run, + ssh=cast("SSHClient", ssh), + shell="bash", + prompt="Fix it", + ) + ) + await process.stdout.blocked.wait() + + assert not execution.done() + assert len(run.steps) == 1 + assert isinstance(run.steps[0], ToolStep) + + process.stdout.release.set() + await execution + + +async def test_exec_turn_failure_raises() -> None: + stream = ( + '{"type":"thread.started","thread_id":"thread-1"}\n' + '{"type":"turn.started"}\n' + '{"type":"turn.failed","error":{"message":"model unavailable"}}\n' + ) + run = _fake_run() + + with pytest.raises(RuntimeError, match="model unavailable"): + await run_codex( + CodexCLIConfig(), + run, + ssh=cast("SSHClient", _FakeSSH(_FakeProcess(stream))), + shell="bash", + prompt="Fix it", + ) + + +async def test_exec_nonzero_exit_raises_stderr() -> None: + run = _fake_run() + + with pytest.raises(RuntimeError, match="authentication failed"): + await run_codex( + CodexCLIConfig(), + run, + ssh=cast( + "SSHClient", + _FakeSSH(_FakeProcess("", stderr="authentication failed", returncode=1)), + ), + shell="bash", + prompt="Fix it", + ) + + assert run.trace.extra["returncode"] == 1 + + +async def test_exec_nonzero_exit_prefers_structured_error() -> None: + run = _fake_run() + stream = '{"type":"error","message":"gateway rejected streaming"}\n' + + with pytest.raises(RuntimeError, match="gateway rejected streaming"): + await run_codex( + CodexCLIConfig(), + run, + ssh=cast( + "SSHClient", + _FakeSSH(_FakeProcess(stream, stderr="noisy warning", returncode=1)), + ), + shell="bash", + prompt="Fix it", + ) + + assert "stderr" not in run.trace.extra + + +async def test_exec_closes_process_when_cancelled() -> None: + process = _FakeProcess(_STREAM_JSON, pause_after=0) + execution = asyncio.create_task( + run_codex( + CodexCLIConfig(), + _fake_run(), + ssh=cast("SSHClient", _FakeSSH(process)), + shell="bash", + prompt="Fix it", + ) + ) + await process.stdout.blocked.wait() + execution.cancel() + + with pytest.raises(asyncio.CancelledError): + await execution + + assert process.closed + + +async def test_agent_opens_ssh_and_uses_workspace_prompt(monkeypatch: pytest.MonkeyPatch) -> None: + ssh = _FakeSSH(_FakeProcess(_STREAM_JSON), shell="powershell") + + class Client: + async def open(self, ref: str) -> _FakeSSH: + assert ref == "ssh" + return ssh + + agent = CodexCLIAgent() + execute = AsyncMock() + monkeypatch.setattr("hud.agents.codex.agent.run_codex", execute) + run = SimpleNamespace(client=Client(), prompt_text="Fix it", runtime_config=None) + + await agent(cast("Any", run)) + + execute.assert_awaited_once_with( + agent.config, + run, + ssh=ssh, + shell="powershell", + prompt="Fix it", + executable="codex", + ) + + +async def test_executable_resolution_prefers_matching_managed_bundle() -> None: + ssh = SimpleNamespace( + capability=Capability.ssh(url="ssh://localhost:22", host_pubkey="key", shell="bash"), + run=AsyncMock( + side_effect=[ + SimpleNamespace(returncode=0, stdout=b"Linux\nx86_64\ngnu\n"), + SimpleNamespace(returncode=0, stdout=b""), + ] + ), + ) + + executable = await resolve_executable( + cast("Any", ssh), + "codex", + {"linux-x64": "/media/hud/bin/codex/bin/codex"}, + RuntimeConfig(resources=RuntimeResources(os="linux")), + ) + + assert executable == "/media/hud/bin/codex/bin/codex" + assert ssh.run.await_count == 2 + + +async def test_executable_resolution_rejects_runtime_os_mismatch() -> None: + ssh = SimpleNamespace( + capability=Capability.ssh(url="ssh://localhost:22", host_pubkey="key", shell="bash"), + run=AsyncMock(return_value=SimpleNamespace(returncode=0, stdout=b"Linux\nx86_64\ngnu\n")), + ) + + with pytest.raises(RuntimeError, match=r"requested 'windows'.*reports 'linux'"): + await resolve_executable( + cast("Any", ssh), + "codex", + {}, + RuntimeConfig(resources=RuntimeResources(os="windows")), + ) diff --git a/hud/agents/types.py b/hud/agents/types.py index f8103a497..4dbf3e3bd 100644 --- a/hud/agents/types.py +++ b/hud/agents/types.py @@ -176,6 +176,20 @@ class ClaudeCLIConfig(AgentConfig): ) +# ----------------------------------------------------------------------------- +# Codex CLI (over SSH) +# ----------------------------------------------------------------------------- + + +class CodexCLIConfig(AgentConfig): + """Configuration for CodexCLIAgent (runs ``codex exec`` over SSH).""" + + model_name: str = "Codex CLI" + model: str = Field(default="gpt-5.6-sol", validation_alias=_model_alias) + use_hud_gateway: bool | None = None + sandbox: Literal["read-only", "workspace-write", "danger-full-access"] = "workspace-write" + + # ----------------------------------------------------------------------------- # Browser Use # ----------------------------------------------------------------------------- diff --git a/hud/capabilities/ssh.py b/hud/capabilities/ssh.py index 93b6f062f..fe95b7565 100644 --- a/hud/capabilities/ssh.py +++ b/hud/capabilities/ssh.py @@ -5,12 +5,19 @@ import asyncio import base64 import contextlib +import ntpath +import os +import secrets import shlex -from typing import Any, ClassVar, Self +from pathlib import Path +from typing import TYPE_CHECKING, Any, ClassVar, Self, cast from urllib.parse import urlsplit import asyncssh +if TYPE_CHECKING: + from typing import BinaryIO + from .base import Capability, CapabilityClient SSH_RECONNECT_ATTEMPTS = 3 @@ -220,6 +227,85 @@ def remaining_timeout() -> float | None: run_kwargs["stdin"] = asyncssh.DEVNULL await self.run(f"cat > {shlex.quote(path)}", check=True, timeout=timeout_s, **run_kwargs) + async def upload( + self, + source: str | os.PathLike[str], + destination: str, + *, + executable: bool = False, + timeout_s: float | None = None, + ) -> None: + """Stream a local file into the SSH namespace and install it atomically.""" + source_path = os.fspath(source) + temporary = f"{destination}.hud-upload-{secrets.token_hex(8)}" + if self._is_windows: + parent = ntpath.dirname(destination) or "." + command = _powershell( + f"New-Item -ItemType Directory -Force -Path {_powershell_quote(parent)} " + "| Out-Null;" + f"$out=[IO.File]::Open({_powershell_quote(temporary)}," + "[IO.FileMode]::CreateNew,[IO.FileAccess]::Write,[IO.FileShare]::None);" + "try{[Console]::OpenStandardInput().CopyTo($out)}finally{$out.Dispose()}" + ) + else: + parent = os.path.dirname(destination) or "." + command = f"mkdir -p -- {shlex.quote(parent)} && cat > {shlex.quote(temporary)}" + + source_file = cast( + "BinaryIO", + await asyncio.to_thread(Path(source_path).open, "rb"), + ) + try: + process = await self.create_process(command) + try: + async with asyncio.timeout(timeout_s): + while chunk := await asyncio.to_thread(source_file.read, 1024 * 1024): + process.stdin.write(chunk) + await process.stdin.drain() + process.stdin.write_eof() + completed = await process.wait(check=True, timeout=None) + if completed.returncode is None: + raise SSHConnectionError("SSH upload ended without an exit status") + + if self._is_windows: + install = ( + f"Move-Item -Force -LiteralPath {_powershell_quote(temporary)} " + f"-Destination {_powershell_quote(destination)}" + ) + install_result = await self.run( + _powershell(install), + check=False, + encoding=None, + timeout=timeout_s, + ) + else: + mode = "0555" if executable else "0444" + install_result = await self.run( + f"chmod {mode} {shlex.quote(temporary)} && " + f"mv -f -- {shlex.quote(temporary)} {shlex.quote(destination)}", + check=False, + encoding=None, + timeout=timeout_s, + ) + if install_result.returncode != 0: + message = _decode(install_result.stderr).strip() + raise RuntimeError(f"SSH upload install failed: {message}") + except BaseException: + process.close() + cleanup = ( + _powershell( + "Remove-Item -Force -ErrorAction SilentlyContinue " + f"-LiteralPath {_powershell_quote(temporary)}" + ) + if self._is_windows + else f"rm -f -- {shlex.quote(temporary)}" + ) + with contextlib.suppress(OSError, TimeoutError, asyncssh.Error): + await self.run(cleanup, check=False, timeout=SSH_SESSION_CLOSE_TIMEOUT_S) + raise + finally: + await asyncio.to_thread(source_file.close) + async def listdir(self, path: str, *, timeout_s: float | None = None) -> list[str]: """List direct children through the exec channel.""" if self._is_windows: diff --git a/hud/cli/tests/test_eval_config.py b/hud/cli/tests/test_eval_config.py index 5148d6a16..e1e521155 100644 --- a/hud/cli/tests/test_eval_config.py +++ b/hud/cli/tests/test_eval_config.py @@ -375,6 +375,25 @@ def test_build_agent_preserves_claude_cli_gateway_config() -> None: assert agent.config.use_hud_gateway is True +def test_build_agent_constructs_codex_cli() -> None: + from hud.agents.codex import CodexCLIAgent + + cfg = EvalConfig(agent_type="codex_cli") + agent = eval_mod._build_agent(cfg) + + assert isinstance(agent, CodexCLIAgent) + assert agent.config.model == "gpt-5.6-sol" + assert agent.config.use_hud_gateway is False + + +def test_build_agent_routes_hosted_codex_cli_through_gateway() -> None: + cfg = EvalConfig(agent_type="codex_cli", remote=True) + + agent = eval_mod._build_agent(cfg) + + assert agent.config.use_hud_gateway is True + + def test_spawn_target_serves_single_file_env(tmp_path: Path) -> None: env_py = tmp_path / "tasks.py" env_py.write_text( diff --git a/hud/environment/tests/test_workspace.py b/hud/environment/tests/test_workspace.py index 5b857ab59..fa860ce3a 100644 --- a/hud/environment/tests/test_workspace.py +++ b/hud/environment/tests/test_workspace.py @@ -199,6 +199,25 @@ async def test_file_operations_use_the_exec_channel(tmp_path: Path) -> None: await ws.stop() +@pytest.mark.asyncio +async def test_upload_streams_binary_into_the_session_namespace(tmp_path: Path) -> None: + source = tmp_path / "source.bin" + source.write_bytes(bytes(range(256)) * 8192) + root = tmp_path / "root" + ws = Workspace(root) + await ws.start() + try: + async with await _connect(ws) as conn: + client = SSHClient(ws.capability(), conn) + await client.upload(source, "managed/tool", executable=True) + finally: + await ws.stop() + + uploaded = root / "managed/tool" + assert uploaded.read_bytes() == source.read_bytes() + assert uploaded.stat().st_mode & 0o777 == 0o555 + + @pytest.mark.asyncio async def test_output_arrives_while_the_command_is_still_running(tmp_path: Path) -> None: """Held until exit, a long build tells the agent nothing while it runs and diff --git a/hud/eval/run.py b/hud/eval/run.py index d3ac5c741..839d106e9 100644 --- a/hud/eval/run.py +++ b/hud/eval/run.py @@ -199,11 +199,13 @@ def __init__( args: dict[str, Any], *, best_effort_grade: bool = False, + runtime_config: RuntimeConfig | None = None, ) -> None: self._client = client self._task_id = task_id self._args = args self._best_effort_grade = best_effort_grade + self.runtime_config = runtime_config #: The task's opening prompt as ``tasks.start`` returned it: plain #: text, or a list of message dicts (``{"role", "content"}``) for #: chat-style / multi-turn prompts. Agents consume the normalized @@ -476,11 +478,12 @@ async def rollout( """ from .runtime.core import resolve_runtime_config + actor_runtime_config = resolve_runtime_config(runtime, task) agent_timeout = validate_rollout_timeouts( task, agent, rollout_timeout, - actor_runtime_config=resolve_runtime_config(runtime, task), + actor_runtime_config=actor_runtime_config, verifier_runtime_config=( resolve_runtime_config(runtime, task.verifier) if task.verifier is not None else None ), @@ -539,6 +542,7 @@ async def close_actor() -> None: task.id, task.args, best_effort_grade=task.verifier is not None, + runtime_config=addr.config or actor_runtime_config, ) live._runtime = addr.url # the placement record for the receipt async with live: # start on enter; complete on exit diff --git a/hud/integrations/harbor/env.py b/hud/integrations/harbor/env.py index df8137e83..40f2fb973 100644 --- a/hud/integrations/harbor/env.py +++ b/hud/integrations/harbor/env.py @@ -260,6 +260,7 @@ def home(user_id: int | None, *, root: Path | None = None) -> str | None: ) agent_mounts = ( *harness_mounts, + Mount("ro", src=str(ROOT / "bin"), dst=str(ROOT / "bin")), Mount("tmpfs", dst=str(TESTS)), Mount("tmpfs", dst=str(VERIFIER_LOGS)), Mount("ro", src="/dev/null", dst=str(AGENT_ANSWER)), diff --git a/hud/integrations/harbor/tests/test_contract.py b/hud/integrations/harbor/tests/test_contract.py index 58124a04f..531bf4c55 100644 --- a/hud/integrations/harbor/tests/test_contract.py +++ b/hud/integrations/harbor/tests/test_contract.py @@ -130,6 +130,7 @@ def test_adapt_packages_an_image_task_as_a_compose_project(tmp_path: Path) -> No served = (context / "env.py").read_text(encoding="utf-8") assert f'Environment("{context.name}")' in served assert 'Environment(CONFIG["name"])' not in served + assert 'Mount("ro", src=str(ROOT / "bin"), dst=str(ROOT / "bin"))' in served project_root = context / "compose-project" assert _tree_snapshot(project_root / "environment") == authored_environment payload = project_root / "hud" diff --git a/hud/types.py b/hud/types.py index c697727ee..05982985d 100644 --- a/hud/types.py +++ b/hud/types.py @@ -53,13 +53,14 @@ class AgentType(StrEnum): CLAUDE = "claude" CLAUDE_CLI = "claude_cli" + CODEX_CLI = "codex_cli" OPENAI = "openai" GEMINI = "gemini" OPENAI_COMPATIBLE = "openai_compatible" @property def is_cli(self) -> bool: - return self is AgentType.CLAUDE_CLI + return self in (AgentType.CLAUDE_CLI, AgentType.CODEX_CLI) @property def cls(self) -> type[Agent]: @@ -72,6 +73,10 @@ def cls(self) -> type[Agent]: from hud.agents import ClaudeCLIAgent return ClaudeCLIAgent + case AgentType.CODEX_CLI: + from hud.agents import CodexCLIAgent + + return CodexCLIAgent case AgentType.OPENAI: from hud.agents import OpenAIAgent @@ -91,6 +96,7 @@ def config_cls(self) -> type[AgentConfig]: from hud.agents.types import ( ClaudeCLIConfig, ClaudeConfig, + CodexCLIConfig, GeminiConfig, OpenAIChatConfig, OpenAIConfig, @@ -101,6 +107,8 @@ def config_cls(self) -> type[AgentConfig]: return ClaudeConfig case AgentType.CLAUDE_CLI: return ClaudeCLIConfig + case AgentType.CODEX_CLI: + return CodexCLIConfig case AgentType.OPENAI: return OpenAIConfig case AgentType.GEMINI: @@ -119,6 +127,8 @@ def gateway_provider(self) -> str: return "anthropic" case AgentType.CLAUDE_CLI: return "anthropic" + case AgentType.CODEX_CLI: + return "openai" case AgentType.OPENAI: return "openai" case AgentType.GEMINI: