Fix Remote Runtime API Authentication #1090

xingyaoww · 2025-11-07T00:07:54Z

Fix remote runtime API authentication and add regression test as part of the "test-examples" workflow

Agent Server images for this PR

• GHCR package: https://github.com/OpenHands/agent-sdk/pkgs/container/agent-server

Variants & Base Images

Variant	Architectures	Base Image	Docs / Tags
java	amd64, arm64	`eclipse-temurin:17-jdk`	Link
python	amd64, arm64	`nikolaik/python-nodejs:python3.12-nodejs22`	Link
golang	amd64, arm64	`golang:1.21-bookworm`	Link

Pull (multi-arch manifest)

# Each variant is a multi-arch manifest supporting both amd64 and arm64
docker pull ghcr.io/openhands/agent-server:55be574-python

Run

docker run -it --rm \
  -p 8000:8000 \
  --name agent-server-55be574-python \
  ghcr.io/openhands/agent-server:55be574-python

All tags pushed for this build

ghcr.io/openhands/agent-server:55be574-golang-amd64
ghcr.io/openhands/agent-server:55be574-golang_tag_1.21-bookworm-amd64
ghcr.io/openhands/agent-server:55be574-golang-arm64
ghcr.io/openhands/agent-server:55be574-golang_tag_1.21-bookworm-arm64
ghcr.io/openhands/agent-server:55be574-java-amd64
ghcr.io/openhands/agent-server:55be574-eclipse-temurin_tag_17-jdk-amd64
ghcr.io/openhands/agent-server:55be574-java-arm64
ghcr.io/openhands/agent-server:55be574-eclipse-temurin_tag_17-jdk-arm64
ghcr.io/openhands/agent-server:55be574-python-amd64
ghcr.io/openhands/agent-server:55be574-nikolaik_s_python-nodejs_tag_python3.12-nodejs22-amd64
ghcr.io/openhands/agent-server:55be574-python-arm64
ghcr.io/openhands/agent-server:55be574-nikolaik_s_python-nodejs_tag_python3.12-nodejs22-arm64
ghcr.io/openhands/agent-server:55be574-golang
ghcr.io/openhands/agent-server:55be574-java
ghcr.io/openhands/agent-server:55be574-python

About Multi-Architecture Support

Each variant tag (e.g., 55be574-python) is a multi-arch manifest supporting both amd64 and arm64
Docker automatically pulls the correct architecture for your platform
Individual architecture tags (e.g., 55be574-python-amd64) are also available if needed

xingyaoww · 2025-11-07T00:08:57Z

@OpenHands review diff from main

I got these error when running DEBUG=1 uv run examples/02_remote_agent_server/04_convo_with_api_sandboxed_server.py

[11/06/25 19:00:12] ERROR    HTTP request failed (422 Unprocessable Entity): remote_conversation.py:60
                             {'detail': [{'type': 'extra_forbidden', 'loc':                           
                             ['body', 'agent', 'llm', 'extra_headers'],                               
                             'msg': 'Extra inputs are not permitted',                                 
                             'input': None}, {'type': 'extra_forbidden',                              
                             'loc': ['body', 'agent', 'llm',                                          
                             'reasoning_summary'], 'msg': 'Extra inputs are                           
                             not permitted', 'input': None}, {'type':                                 
                             'extra_forbidden', 'loc': ['body', 'agent',                              
                             'llm', 'litellm_extra_body'], 'msg': 'Extra                              
                             inputs are not permitted', 'input': {}},                                 
                             {'type': 'extra_forbidden', 'loc': ['body',                              
                             'agent', 'condenser',                                                    
                             'LLMSummarizingCondenser', 'llm',                                        
                             'extra_headers'], 'msg': 'Extra inputs are not                           
                             permitted', 'input': None}, {'type':                                     
                             'extra_forbidden', 'loc': ['body', 'agent',                              
                             'condenser', 'LLMSummarizingCondenser', 'llm',                           
                             'reasoning_summary'], 'msg': 'Extra inputs are                           
                             not permitted', 'input': None}, {'type':                                 
                             'extra_forbidden', 'loc': ['body', 'agent',                              
                             'condenser', 'LLMSummarizingCondenser', 'llm',                           
                             'litellm_extra_body'], 'msg': 'Extra inputs are                          
                             not permitted', 'input': {}}]}                                           
                             ╭───── Traceback (most recent call last) ─────╮                          
                             │ /Users/xingyaow/Projects/All-Hands-AI/softw │                          
                             │ are-agent-sdk/openhands-sdk/openhands/sdk/c │                          
                             │ onversation/impl/remote_conversation.py:52  │                          
                             │ in _send_request                            │                          
                             │                                             │                          
                             │    49 │   │   response = client.request(met │                          
                             │    50 │   │   if acceptable_status_codes an │                          
                             │    51 │   │   │   return response           │                          
                             │ ❱  52 │   │   response.raise_for_status()   │                          
                             │    53 │   │   return response               │                          
                             │    54 │   except httpx.HTTPStatusError as e │                          
                             │    55 │   │   content = None                │                          
                             │                                             │                          
                             │ /Users/xingyaow/Projects/All-Hands-AI/softw │                          
                             │ are-agent-sdk/.venv/lib/python3.12/site-pac │                          
                             │ kages/httpx/_models.py:829 in               │                          
                             │ raise_for_status                            │                          
                             │                                             │                          
                             │    826 │   │   }                            │                          
                             │    827 │   │   error_type = error_types.get │                          
                             │    828 │   │   message = message.format(sel │                          
                             │ ❱  829 │   │   raise HTTPStatusError(messag │                          
                             │    830 │                                    │                          
                             │    831 │   def json(self, **kwargs: typing. │                          
                             │    832 │   │   return jsonlib.loads(self.co │                          
                             ╰─────────────────────────────────────────────╯                          
                             HTTPStatusError: Client error '422                                       
                             Unprocessable Entity' for url                                            
                             'https://zcsubdqmuglmhpsc.eval-runtime.all-hand                          
                             s.dev/api/conversations'                                                 
                             For more information check:                                              
                             https://developer.mozilla.org/en-US/docs/Web/HT                          
                             TP/Status/422                                                            
[11/06/25 19:00:12] INFO     Stoping runtime zcsubdqmuglmhpsc                         workspace.py:289
[11/06/25 19:00:12] DEBUG    Sending POST request to                                  workspace.py:263
                             https://runtime.eval.all-hands.dev/stop

Can you come up with a few hypothesis what might cause this error?

openhands-ai · 2025-11-07T00:09:07Z

I'm on it! xingyaoww can track my progress at all-hands.dev

neubig

Can we add a regression test?

openhands-ai · 2025-11-07T00:12:16Z

Summary: Analysis of 422 Error in Remote Agent Server

I've analyzed the error you encountered when running the example with DEBUG=1 uv run examples/02_remote_agent_server/04_convo_with_api_sandboxed_server.py.

The Error

The remote agent server returned 422 Unprocessable Entity when trying to create a conversation, rejecting these LLM fields as "Extra inputs are not permitted":

extra_headers
reasoning_summary
litellm_extra_body

These fields appear in both the main agent's LLM and the condenser's LLM.

Root Cause Hypotheses (Ordered by Likelihood)

1. Version Mismatch Between SDK and Remote Agent Server ⭐ (Most Likely)

Your local SDK code has newer LLM fields (extra_headers, reasoning_summary, litellm_extra_body) that were recently added
The remote agent-server image (ghcr.io/openhands/agent-server:main-python) was built from an older commit that doesn't have these fields
When the SDK serializes the agent with these fields and sends it to the older server, Pydantic's extra="forbid" validation rejects them

2. Serialization Including Default Values

Fields with default values (e.g., litellm_extra_body = Field(default_factory=dict)) are being included in the JSON payload even when empty
The server's stricter validation rejects these

3. Field Exclusion Not Configured

These newer fields might need explicit exclude=True or custom serialization logic for backward compatibility with older agent-server versions

4. Pydantic Model Config Inconsistency

The LLM class has extra="forbid" in its ConfigDict, causing strict validation failures when schemas don't match exactly

5. PR Changes Side Effects (Least Likely)

Your PR's HTTP client header changes shouldn't affect JSON payload serialization, but worth considering

Recommended Next Steps

To diagnose:

Check git history to see when these LLM fields were added
Verify the agent-server image version/build date
Add debug logging before line 467 in remote_conversation.py to see the exact JSON payload being sent

To fix (short-term):

Use explicit field exclusion when serializing the agent for remote servers
Or explicitly set these fields to exclude from serialization for backward compatibility

To fix (long-term):

Ensure remote agent-server images are rebuilt with the latest SDK version
Implement version negotiation or graceful degradation for field compatibility

The error is not directly related to your authentication PR changes, but rather a schema compatibility issue between the SDK and the remote agent-server version.

_{View full conversation}

github-actions · 2025-11-07T16:07:27Z

🔄 Running Examples with `openhands/claude-haiku-4-5-20251001`

Last updated: 2025-11-07 16:09:23 UTC

Example	Status	Duration	Cost
02_remote_agent_server/04_vscode_with_docker_sandboxed_server.py	❌ FAIL (exit: 1)	116s	$0.00

❌ Some tests failed

Total: 1 | Passed: 0 | Failed: 1 | Total Cost: $0.00

View full workflow run

github-actions · 2025-11-07T16:08:57Z

🔄 Running Examples with `openhands/claude-haiku-4-5-20251001`

Last updated: 2025-11-07 16:09:03 UTC

Example	Status	Duration	Cost
02_remote_agent_server/04_convo_with_api_sandboxed_server.py	❌ FAIL (exit: 1)	6s	$0.00

❌ Some tests failed

Total: 1 | Passed: 0 | Failed: 1 | Total Cost: $0.00

View full workflow run

github-actions · 2025-11-07T16:19:46Z

🔄 Running Examples with `openhands/claude-haiku-4-5-20251001`

Last updated: 2025-11-07 16:25:01 UTC

Example	Status	Duration	Cost
02_remote_agent_server/04_convo_with_api_sandboxed_server.py	❌ FAIL (exit: 1)	315s	$0.00

❌ Some tests failed

Total: 1 | Passed: 0 | Failed: 1 | Total Cost: $0.00

View full workflow run

github-actions · 2025-11-07T16:31:33Z

🔄 Running Examples with `openhands/claude-haiku-4-5-20251001`

Last updated: 2025-11-07 16:36:47 UTC

Example	Status	Duration	Cost
02_remote_agent_server/04_convo_with_api_sandboxed_server.py	❌ FAIL (exit: 1)	314s	$0.00

❌ Some tests failed

Total: 1 | Passed: 0 | Failed: 1 | Total Cost: $0.00

View full workflow run

github-actions · 2025-11-07T18:49:25Z

🔄 Running Examples with `openhands/claude-haiku-4-5-20251001`

Last updated: 2025-11-07 18:51:11 UTC

Example	Status	Duration	Cost
02_remote_agent_server/04_convo_with_api_sandboxed_server.py	❌ FAIL (exit: 1)	106s	$0.00

❌ Some tests failed

Total: 1 | Passed: 0 | Failed: 1 | Total Cost: $0.00

View full workflow run

This reverts commit bbae87d.

github-actions · 2025-11-07T19:05:01Z

🔄 Running Examples with `openhands/claude-haiku-4-5-20251001`

Last updated: 2025-11-07 19:06:38 UTC

Example	Status	Duration	Cost
02_remote_agent_server/04_convo_with_api_sandboxed_server.py	✅ PASS	96s	$0.04

✅ All tests passed!

Total: 1 | Passed: 1 | Failed: 0 | Total Cost: $0.04

View full workflow run

This reverts commit 3701e0f.

xingyaoww · 2025-11-07T19:22:57Z

@neubig added and it works!

enyst · 2025-11-07T19:32:21Z

openhands-workspace/openhands/workspace/remote_api/workspace.py

        self.host = self._runtime_url.rstrip("/")
        self.api_key = self._session_api_key
+        self._client = None  # Reset HTTP client with new host and API key
+        _ = self.client  # Initialize client by accessing the property


This looks a bit weird, maybe error-prone, though I see where it's coming from...

@OpenHands review the codebase and understand how we can do this better. Then create an issue about it as we can resolve it in a future PR

I'm on it! xingyaoww can track my progress at all-hands.dev

Summary

I've reviewed the codebase and created a GitHub issue documenting the architectural concern identified in PR #1090.

What I Did

Analyzed the codebase to understand the authentication pattern in APIRemoteWorkspace (lines 114-116 in openhands-workspace/openhands/workspace/remote_api/workspace.py)

Identified the issue: The current pattern for updating HTTP client credentials is fragile and error-prone:

self.api_key = self._session_api_key self._client = None # Reset HTTP client with new host and API key _ = self.client # Initialize client by accessing the property

Created GitHub Issue Refactor: Improve HTTP client credential update pattern in Remote Workspace #1101: Refactor: Improve HTTP client credential update pattern in Remote Workspace

Key Problems Documented

The issue describes why this pattern is concerning:

Non-obvious side effects: Accessing .client triggers lazy initialization

Order dependency: Operations must happen in exact sequence

Fragility: Changes to parent class could break this silently

Code smell: Using _ = self.client just for side effects

Mixed concerns: Two authentication mechanisms (Runtime API vs Agent Server) are intertwined

Proposed Solutions

I documented four potential solutions in the issue, with Option 2 (dynamic client recreation) recommended as it:

Requires minimal code changes

Makes credential updates transparent

Is backwards compatible

Handles both credential AND host changes automatically

Option 3 (separate clients for Runtime API vs Agent Server) would be ideal for a larger refactoring to make separation of concerns explicit.

No Code Changes Made

As requested, I did not modify any code - this analysis is captured in the GitHub issue for future resolution in a separate PR.

_{View full conversation}

openhands-workspace/openhands/workspace/remote_api/workspace.py

Only log header keys (not values) to avoid exposing API keys in logs. This applies to both request kwargs and error response headers. Co-authored-by: openhands <[email protected]>

openhands-ai · 2025-11-07T19:47:47Z

Looks like there are a few issues preventing this PR from being merged!

GitHub Actions are failing:
- Agent Server

If you'd like me to help, just leave a comment, like

@OpenHands please fix the failing actions on PR #1090 at branch `xw/remote-api-auth`

Feel free to include any additional details that might help me get this PR into a better state.

_{^{You can manage your notification settings}}

neubig · 2025-11-07T22:02:06Z

examples/02_remote_agent_server/04_convo_with_api_sandboxed_server.py

+# If GITHUB_SHA is set (e.g. running in CI of a PR), use that to ensure consistency
+# Otherwise, get the latest commit SHA from main branch (images are built on main)
+server_image_sha = os.getenv("GITHUB_SHA") or get_latest_commit_sha(
+    "OpenHands/software-agent-sdk", "main"
+)
+server_image = f"ghcr.io/openhands/agent-server:{server_image_sha[:7]}-python-amd64"
+logger.info(f"Using server image: {server_image}")


I don't know if this is necessarily blocking, but this is a bit complex for an example. Maybe we could move this off into some other utility function in another file?

@OpenHands please create a new github issue tracking this - we can simplify this after we are able to customize image pull policy on runtime API (after https://github.com/OpenHands/runtime-api/pull/356)

I'm on it! xingyaoww can track my progress at all-hands.dev

Summary

I've successfully created a GitHub issue to track the simplification work as requested.

Issue Created: #1104

Title: "Simplify image resolution logic in remote agent server example after runtime-api image pull policy support"

What the issue tracks:

The complex get_latest_commit_sha() function and related image resolution logic in examples/02_remote_agent_server/04_convo_with_api_sandboxed_server.py (lines 49-79)

Plans to simplify this code once runtime-api PR Unify tool initialization — Step 1: global registry + lazy materialization #356 is merged, which will add support for customizable image pull policies

The simplification will remove the GitHub API querying logic and use simpler, stable image tags instead

This will make the example more focused on demonstrating core functionality rather than workaround logic

The issue has been labeled with enhancement and technical-debt tags, includes references to the runtime-api PR #356 and the current PR #1090, and contains a task checklist for tracking the work.

_{View full conversation}

openhands-workspace/openhands/workspace/remote_api/workspace.py

xingyaoww added 3 commits November 6, 2025 18:52

fix remote api workspace

c53bee7

figure out headers

f2d5322

fix auth for agent-server requests

28fd153

neubig reviewed Nov 7, 2025

View reviewed changes

xingyaoww added 3 commits November 7, 2025 11:04

fetch latest agent-server image in the example

363dd4a

add to example tests

be97d23

Merge branch 'main' into xw/remote-api-auth

bec7b32

xingyaoww added the test-examples Run all applicable "examples/" files. Expensive operation. label Nov 7, 2025

fix example name

2c3dc35