Add sandbox_harbor example: evaluate Harbor benchmarks on AgentCore Runtime - #112
Merged
Merged
Conversation
…untime Adds examples/sandbox_harbor/ — one command to evaluate a whole Harbor benchmark on AgentCore Runtime, with two interchangeable agent harnesses (strands and co-located claude_code), a benchmark-agnostic ECR image builder, and per-task sandbox lifecycle via HarborSandboxClient. Also adds read_timeout/connect_timeout to SandboxClient (default 900s read) so long, silent commands (heavy verifiers, runtime warm-up) aren't cut off by boto3's short default; the example relies on this. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Teq2412
force-pushed
the
add-sandbox-harbor-example
branch
from
August 21, 2026 07:28
a9ee166 to
e279deb
Compare
Teq2412
commented
Aug 21, 2026
| _SDK_KEYS = ("status_code", "input_id", "s3_bucket", "result_key", "payload") | ||
|
|
||
|
|
||
| def harness_arn(agent: str) -> str: |
Contributor
Author
There was a problem hiding this comment.
I realized that this is a unstable way to locate harness runtime, we should have it to be provided as a mapping from {claude-code : claude-code-arn}, will fix in next revision
Two blockers for training against the tmax harness (see tmax15k_handoff.md): - build the LLM from trainer-injected _rollout (OpenAIModel, api_key session contract) when base_url is present; Bedrock eval path unchanged otherwise - return plural 'rewards' alongside 'reward' (training backends read plural) - Dockerfile: strands-agents[openai] for the OpenAI provider Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Leasing the shared per-task runtime (name = hash of image URI) breaks under concurrent same-task rollouts (GRPO groups): the first release() deletes the runtime out from under the siblings (InvokeAgentRuntimeCommand -> DELETING). lease now creates sb_<code>_<hash12>_<rand8> via create(unique=True); the shared deterministic name remains for the pool / task_runtime_arn pattern. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
luyuzhe111
approved these changes
Aug 25, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds
examples/sandbox_harbor/— one command to evaluate a whole Harbor benchmark on AgentCore Runtime:uv run python bench.py --benchmark tmax/TMax-15K-Harbor --task-root ./tasks \ --agent claude-code --model us.anthropic.claude-sonnet-4-6One rollout per task: each task gets a fresh sandbox, the agent does the work, and the task's own shipped verifier grades it. You get a solve rate and one result record per task.
Pieces
harbor_sandbox/— turns each task into a runnable sandbox:build.pypackages every task image and pushes it to ECR;HarborSandboxClientcreates/removes a task's runtime on demand (naming single-source-of-truth innaming.py).harness/— two interchangeable agents:strands(framework agent,bashtool =sb.exec) andclaude_code(Claude Code co-located in the task box). A harness deploys once as a long-lived runtime and is reused for every task.bench.py— the entrypoint: hands each task to the harness, collects results, prints the solve rate.Runs entirely on the serverless arm64 microVM substrate (PUBLIC network).
Also
SandboxClientgainsread_timeout/connect_timeout(default 900s read) so long, silent commands (heavy verifiers, AgentCore runtime warm-up) aren't cut off by boto3's short default. The example relies on this to give each task's verifier its full declared budget.Notes
.env(template committed as.env.example); nothing account-specific is committed.examples/sandbox_harbor/TODO.md.🤖 Generated with Claude Code