Add sandbox_harbor example: evaluate Harbor benchmarks on AgentCore Runtime

examples/sandbox_harbor/.gitignore

@@ -0,0 +1,12 @@
+# Toolkit + harbor_sandbox staged into a harness build context by build_push.sh
+harness/*/_toolkit/
+harness/*/harbor_sandbox/
+# sandboxd health-shim binary is staged from the repo-root sandboxd/, not vendored
+harbor_sandbox/wrapper/agentcore-sandboxd-linux-*
+# real account-specific values (copy .env.example); never committed
+harbor_sandbox/.env
+# Run outputs
+*.jsonl
+# Standalone uv project: users run `uv sync` themselves; examples don't track lockfiles
+uv.lock
+__pycache__/

examples/sandbox_harbor/README.md

@@ -0,0 +1,86 @@
+# Harbor on AgentCore Runtime
+
+Harbor support easy evaluation on popular benchmarks on infra such as Daytona, Modal, LangSmith, Blaxel, and Novita Sandbox:
+```
+harbor run -d "<dataset@version>" -m "<model>" -a "<agent>"
+```
+This example shows how to enable similar, convenient evaluation on Harbor benchmarks on AgentCore Runtime.
+
+one command that evaluates a whole [Harbor](https://harborframework.com) benchmark on AgentCore Runtime:
+
+```bash
+uv run python bench.py --benchmark tmax/TMax-15K-Harbor --task-root ./tasks \
+                       --agent claude-code --model us.anthropic.claude-sonnet-4-6
+```
+
+That runs one rollout per task: for each task the agent gets its own fresh
+sandbox, does the work, and is graded by the task's own tests. You get a solve
+rate and one result record per task.
+
+## How it fits together
+
+A Harbor benchmark is a folder of tasks. Each task ships a container image, an
+instruction, and hidden tests. Evaluating it on AgentCore Runtime takes three
+pieces, all in this folder:
+
+- **`harbor_sandbox/`** — turns each task into a runnable sandbox. `build.py`
+  packages every task image and pushes it to ECR; `HarborSandboxClient` creates
+  (and later removes) a task's runtime on demand.
+- **`harness/`** — the agent. Two interchangeable ones: `strands` and
+  `claude-code` (Claude Code co-located in the box). A harness is deployed once
+  as a long-lived runtime and reused for every task.
+- **`bench.py`** — the entrypoint above. It hands each task to the harness,
+  collects results, and prints the summary.
+
+## Setup
+
+```bash
+uv sync    # installs this example (and its harbor_sandbox package) into ./.venv
+cp harbor_sandbox/.env.example harbor_sandbox/.env   # then fill in your account/region/bucket
+```
+
+`.env` holds your account-specific values (git-ignored, never committed); it is
+read by `harbor_sandbox/config.py` and sourced by `harness/build_push.sh`.
+
+## Steps
+
+1. **Build the task images** (once per benchmark):
+
+   ```bash
+   uv run python -m harbor_sandbox.build --task-root ./tasks \
+          --benchmark tmax/TMax-15K-Harbor --arch arm64
+   ```
+
+2. **Deploy an agent harness** (once):
+
+   ```bash
+   (cd harness && ./build_push.sh claude_code)
+   uv run python harness/deploy_harness.py --agent claude-code --image-tag claude-code
+   ```
+
+3. **Run the benchmark**:
+
+   ```bash
+   uv run python bench.py --benchmark tmax/TMax-15K-Harbor --task-root ./tasks \
+          --agent claude-code --model us.anthropic.claude-sonnet-4-6
+   ```
+
+## Runtime
+
+Tasks run on the serverless arm64 microVM substrate — fast cold start, PUBLIC
+network. Build the task images for arm64 (`--arch arm64`); that is the substrate
+this example targets end to end. For now, AgentCore Runtime microVM only support arm64, with upcoming support on x86 microVM, we'll update to support once the infra is available.
+
+## Notes
+
+- The harness needs no credentials injected: each task runtime uses its own IAM
+  role.
+- By default a task's runtime is removed right after its rollout, so a large
+  benchmark only ever holds a handful of runtimes at once. Alternatively, one can just raise up AgentCore Runtime quota, and keep the runtime deployed to achieve a faster start.
+- Restrict a run with `--tasks` / `--exclude` (a comma list or `@file`) and
+  `--limit`.
+- `bench.py` reads tasks from `--task-root`; if that dir is empty it auto-pulls
+  them from the Harbor registry, which needs the optional `harbor` package
+  (`uv sync --extra harbor`). With the tasks already on disk this is skipped.
+
+Known limits on the terminal-bench-2 benchmark are tracked in [TODO.md](TODO.md).

examples/sandbox_harbor/TODO.md

@@ -0,0 +1,33 @@
+# TODO — current limits on terminal-bench-2
+
+We can run terminal-bench-2 end to end on AgentCore Runtime today (serverless
+arm64 microVM). A handful of tasks still don't grade cleanly. These are the
+known limits, and none of them is a problem with the agent or the scoring —
+they're environment limits.
+
+## 1. Long silent commands time out
+
+A few heavy tasks run a single command that works for a long time while printing
+nothing (for example, compiling a large library from source). The connection
+that carries output has an idle limit, so after enough silence it gives up and
+the task is dropped instead of being scored.
+
+**Fix:** keep the connection alive during silent commands — send a heartbeat, or
+start the command in the background and poll it. Not done yet.
+
+## 2. One command can't run longer than an hour
+
+A single command in a sandbox is capped at one hour by the service. Tasks that
+ask for more are stopped at the cap and graded on whatever they finished. This
+mostly overlaps with limit #1.
+
+## Future: x86 / EC2
+
+This example runs arm64-only. An earlier version could also place tasks on an
+x86 EC2 capacity provider; it was removed to keep the code simple, since it
+didn't improve results and added a lot of complexity (a bare x86 image ships no
+`curl`/`python`, and its root is capability-stripped, so the harness had to
+inject a static `curl` + CA bundle before anything could run). Re-adding x86
+would also have to handle a per-image size limit — a few x86 images come out
+around 6 GB, over the limit, so their runtime can't be created (the same tasks
+are smaller and run fine on arm64).