feat: add Harbor multi-harness SDK demo

.github/workflows/ci_python.yml

@@ -86,5 +86,5 @@ jobs:
       - name: Run pytest
         run: |
           set -euo pipefail
-          uv sync --group test --no-group dev --extra hermes --extra relay
+          uv sync --group test --no-group dev --extra harbor --extra hermes --extra relay
           uv run pytest

.gitignore

@@ -18,6 +18,8 @@ build/
 
 examples/*/artifacts/
 tests/fixtures/*/artifacts/
+integrations/harbor/demo/runs/
+integrations/harbor/demo/task/environment/vendor/
 
 .DS_Store
 
@@ -27,4 +29,3 @@ tests/fixtures/*/artifacts/
 # UV Lock
 # TBD if we want to track this in git
 uv.lock
-

README.md

@@ -5,8 +5,8 @@ SPDX-License-Identifier: Apache-2.0
 
 # NVIDIA NeMo Fabric
 
-Fabric is the harness-management layer that turns many agent runtimes into one
-configurable, observable execution surface.
+Fabric is a runtime execution layer for agents. It turns multiple agent
+harnesses into one configurable, observable lifecycle surface.
 
 <p align="center">
   <img src="assets/fabric-hero.png" alt="NeMo Fabric connects deployment platforms, evaluation harnesses, and RL rollout harnesses to multiple agent runtimes through one observable execution surface." width="1000">
@@ -30,10 +30,10 @@ Fabric provides:
 ```mermaid
 flowchart TB
   Consumer["Consumer\nCLI | Python SDK | integrations"]
-  Config["Agent package\nagent.yaml + profiles"]
-  Core["Fabric Rust core\nvalidate | resolve | plan | run"]
-  Adapter["Selected harness adapter\nHermes SDK | Hermes CLI | Codex CLI"]
-  Harness["Agent harness runtime\nHermes | Codex"]
+  Config["Agent source\nagent.yaml or FabricConfig + profiles"]
+  Core["Fabric Rust core\nresolve | plan | create | invoke | destroy"]
+  Adapter["Selected Fabric adapter"]
+  Harness["Agent harness runtime\nHermes | Codex | custom"]
   Artifacts["Artifact manifest\noutput | logs | patches | telemetry refs"]
   Relay["NeMo Relay\nATOF / ATIF when enabled"]
 
@@ -101,24 +101,32 @@ under `examples/code-review-agent/artifacts/hermes-sdk/`.
 
 ## Core Concepts
 
-- **Agent package:** an `agent.yaml` file plus optional profiles, skills, repos,
-  and artifacts. Start with `examples/code-review-agent/agent.yaml`.
-- **Typed config:** the SDK can pass an in-memory config directly to Fabric.
-  `agent.yaml` is the portable representation for CLI, examples, CI, and
-  reproducible runs.
+- **Agent source:** callers provide either an agent package path or a typed
+  `FabricConfig`. An agent package contains `agent.yaml` plus optional profiles,
+  skills, repos, and artifacts. Start with
+  `examples/code-review-agent/agent.yaml`.
+- **Typed config:** SDK consumers can construct configuration in memory without
+  materializing an agent directory. `agent.yaml` remains the portable
+  representation for CLI use, examples, CI, and reproducible runs.
 - **Profiles:** named variations of the base config. Use profiles to vary the
   harness, model, MCP, tools, skills, telemetry, or environment context without
   editing `agent.yaml`.
 - **Adapters:** harness-specific integrations selected by `harness.adapter_id`.
   The Hermes SDK and CLI adapters live under `adapters/hermes-sdk/` and
   `adapters/hermes-cli/`; the Codex CLI adapter lives under
-  `adapters/codex-cli/`.
+  `adapters/codex-cli/`. Harness-specific extensions belong under
+  `harness.settings` so the normalized contract can remain stable.
 - **Artifacts:** normalized output, logs, patches, and telemetry references
   returned through an `ArtifactManifest`.
 
 Fabric applies profiles in caller order and validates the final effective config
 before planning or running.
 
+Path sources select profiles by name. Typed `FabricConfig` sources use ordered
+`FabricProfileConfig` objects; the SDK rejects mixed profile stacks. See the
+[Python SDK contract](docs/python-sdk-contract.md) for the complete public API,
+type definitions, lifecycle semantics, and compatibility rules.
+
 ## Use Fabric
 
 Inspect the run plan before invoking a harness:
@@ -140,11 +148,13 @@ async def main():
     agent = Path("examples/code-review-agent")
 
     async with FabricClient() as client:
+        resolved = client.resolve(agent, profiles=["hermes_sdk"])
         plan = client.plan(agent, profiles=["hermes_sdk"])
         report = await client.doctor(agent, profiles=["hermes_sdk"])
 
-    print(plan["agent_name"])
-    print(report["checks"])
+    print(resolved.agent_name)
+    print(plan.agent_name)
+    print(report.checks)
 
 asyncio.run(main())
 ```
@@ -259,6 +269,10 @@ result. Runtime updates and cancellation are capability-gated and raise
 `FabricCapabilityError` when the selected runtime does not support them.
 Session APIs require `runtime.mode: session`.
 
+Service mode is part of the forward SDK contract but is not implemented by the
+current runtime. `start_service(...)` raises `FabricCapabilityError` rather than
+silently emulating server or tenancy behavior outside Fabric's execution scope.
+
 ### Interactive CLI Chat
 
 For local manual multi-turn testing, use `fabric chat` with a session-mode
@@ -303,6 +317,37 @@ adapter execution contract. The CLI is a separate interface over the same Rust
 core. For source-tree development, install the package with
 `python3 -m pip install -e .` before using the SDK.
 
+## Harbor Integration
+
+Harbor can use Fabric as one external agent while Fabric selects the execution
+harness from its ordered profile stack. Harbor retains task, environment,
+verification, reward, and job ownership. `FabricAgent` invokes the Fabric Python
+SDK inside the Harbor task environment; it does not invoke the Fabric CLI.
+
+After preparing the demo build context as described in the
+[Harbor multi-harness demo](integrations/harbor/demo/README.md), run the
+credential-free integration example:
+
+```bash
+DEMO_DIR="$PWD/integrations/harbor/demo"
+
+uv run --extra harbor harbor run \
+  --path "$DEMO_DIR/task" \
+  --agent nemo_fabric.integrations.harbor:FabricAgent \
+  --ak fabric_config_path=/opt/fabric-demo/agent.yaml \
+  --ak 'fabric_profile_paths=["/opt/fabric-demo/profiles/smoke.yaml"]' \
+  --job-name fabric-smoke \
+  --jobs-dir "$DEMO_DIR/runs" \
+  --n-concurrent 1 \
+  --n-attempts 1 \
+  --force-build
+```
+
+The same Harbor agent can switch between the smoke, Hermes, Hermes with Relay
+telemetry, and Codex profiles. See the
+[Harbor integration guide](integrations/harbor/README.md) for ownership and
+installation details, and the demo guide for the complete command matrix.
+
 ## Other Runs
 
 Run one isolated Codex CLI turn using Codex's existing authentication and

docs/guides/harbor-evaluation.mdx

@@ -0,0 +1,108 @@
+---
+title: "Evaluate agents with Harbor"
+description: "Run multiple agent harnesses through one Harbor integration and inspect normalized results, artifacts, and telemetry."
+---
+{/* SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+SPDX-License-Identifier: Apache-2.0 */}
+
+# Evaluate multiple harnesses through one Harbor agent
+
+Use `nemo_fabric.integrations.harbor:FabricAgent` when Harbor should own the
+evaluation workflow while Fabric owns agent harness execution. The Harbor agent
+class stays fixed as Fabric profiles select Hermes CLI, Codex CLI, or another
+adapter.
+
+```text
+Harbor task -> FabricAgent -> Fabric SDK -> selected adapter -> agent harness
+     |                                                        |
+     +----- verifier, reward, and run layout <- RunResult -----+
+```
+
+## Ownership boundary
+
+| Harbor owns | Fabric owns |
+| --- | --- |
+| Task and dataset materialization | Fabric config and ordered profile resolution |
+| Environment and container lifecycle | Harness adapter selection and invocation |
+| Verifier execution and reward calculation | Normalized requests, results, and artifacts |
+| Job, trial, log, and artifact layout | Telemetry configuration and references |
+
+Fabric runs inside the Harbor task environment. It does not replace Harbor's
+container management, verifier, or evaluation semantics.
+
+## Install the integration
+
+Install Fabric with the optional Harbor dependency:
+
+```bash
+python3 -m pip install "nemo-fabric[harbor]"
+```
+
+For a source checkout, `uv run --extra harbor` installs the same optional
+dependency before invoking Harbor.
+
+## Run a Fabric-backed Harbor task
+
+Point Harbor at the Fabric agent class, then pass the base config and ordered
+profile paths as agent constructor arguments:
+
+```bash
+uv run --extra harbor harbor run \
+  --path "$TASK_DIR" \
+  --agent nemo_fabric.integrations.harbor:FabricAgent \
+  --ak fabric_config_path=/opt/fabric-demo/agent.yaml \
+  --ak 'fabric_profile_paths=["/opt/fabric-demo/profiles/hermes.yaml"]' \
+  --model nvidia/nemotron-3-nano-30b-a3b \
+  --ae "NVIDIA_API_KEY=$NVIDIA_API_KEY"
+```
+
+The config and profile paths are paths inside the Harbor task container.
+`--ak` passes constructor arguments to `FabricAgent`; these are not Fabric CLI
+flags. The integration loads the YAML into typed SDK config objects and calls
+`FabricClient.run()` inside the task environment.
+
+## Switch harnesses and telemetry with profiles
+
+The runnable demo keeps the Harbor agent and task fixed. Only the Fabric profile
+stack, model, and required credentials change:
+
+| Profile stack | Execution path | What it demonstrates |
+| --- | --- | --- |
+| `smoke.yaml` | Deterministic scripted adapter | Credential-free Harbor, Fabric, workspace, and verifier pipeline |
+| `hermes.yaml` | Hermes CLI | A real model-backed harness selected through Fabric |
+| `hermes.yaml`, `telemetry.yaml` | Hermes CLI with NeMo Relay | Phoenix OpenInference traces plus ATOF events and an ATIF trajectory |
+| `codex.yaml` | Codex CLI | A second real harness using an existing Codex login mounted by Harbor |
+
+Profiles are applied in caller order. The telemetry profile composes with the
+Hermes profile without requiring another Harbor agent implementation.
+
+## Inspect results and evidence
+
+Each trial stores Fabric's normalized result in the Harbor agent logs as
+`fabric-result.json`. It includes status, selected profiles, harness and adapter
+identity, runtime and invocation IDs, artifacts, telemetry references, and any
+structured error.
+
+Open the Harbor viewer for rewards, exceptions, and trial logs:
+
+```bash
+uv run --extra harbor harbor view "$RUNS_DIR"
+```
+
+For Relay-enabled runs, inspect the same Harbor run directory for portable ATOF
+and ATIF records:
+
+```bash
+find "$RUNS_DIR" -path '*/agent/fabric-artifacts/*/relay/events.atof.jsonl'
+find "$RUNS_DIR" -path '*/agent/fabric-artifacts/*/relay/*.atif.json'
+```
+
+Open Phoenix separately to inspect the corresponding OpenInference trace.
+
+## Run the complete demo
+
+The repository demo contains the task image, Fabric config, profile matrix,
+portable Phoenix routing, exact commands for every variant, expected results,
+and a recording flow:
+
+[Run the Harbor multi-harness demo](https://github.com/NVIDIA/NeMo-Fabric/tree/main/integrations/harbor/demo)

docs/index.yml

@@ -6,6 +6,10 @@ navigation:
     contents:
       - page: Overview
         path: ./getting-started/overview.mdx
+  - section: Guides
+    contents:
+      - page: Evaluate agents with Harbor
+        path: ./guides/harbor-evaluation.mdx
   - section: Reference
     contents:
       - section: API

integrations/harbor/README.md

@@ -30,7 +30,7 @@ Fabric owns:
 The integration shape is:
 
 ```text
-Harbor task/env -> FabricAgent -> fabric run -> harness runtime -> Fabric result -> Harbor metadata/verifier
+Harbor task/env -> FabricAgent -> Fabric SDK runner -> harness runtime -> Fabric result -> Harbor metadata/verifier
 ```
 
 ## Install
@@ -50,32 +50,40 @@ python3 -m pip install -e ../harbor
 
 ## Using FabricAgent
 
-`FabricAgent` is selected like any other custom Harbor agent:
+`FabricAgent` follows Harbor 0.16.1's external-agent contract. The runner and
+config files must be installed or copied into the task environment:
 
 ```bash
 harbor run --path <dataset-or-task-dir> \
-  --agent-import-path nemo_fabric.integrations.harbor:FabricAgent \
+  --agent nemo_fabric.integrations.harbor:FabricAgent \
   --model nvidia/nemotron-3-nano-30b-a3b \
-  --ak fabric_agent_path=/workspace/code-review-agent \
-  --ak fabric_profiles=hermes_sdk \
-  --ak fabric_cli=fabric
+  --ak fabric_config_path=/opt/fabric/agent.yaml \
+  --ak 'fabric_profile_paths=["/opt/fabric/profiles/hermes.yaml"]' \
+  --ae NVIDIA_API_KEY="$NVIDIA_API_KEY"
 ```
 
 Important kwargs:
 
-- `fabric_agent_path`: path to the Fabric agent package or `agent.yaml` visible
-  inside the Harbor environment.
-- `fabric_profiles`: profile name or profile list applied in order.
-- `fabric_cli`: Fabric CLI command visible inside the Harbor environment.
+- `fabric_config_path`: YAML config path visible inside the Harbor environment.
+- `fabric_profile_paths`: YAML profile path or ordered profile-path list.
+- `fabric_python`: Python command used for the sandbox-local SDK runner.
 - `fabric_cwd`: optional working directory for Fabric commands.
 - `fabric_install_command`: optional explicit install/bootstrap command.
 - `fabric_timeout_sec`: optional timeout for Fabric install/run commands.
 
 Harbor passes the task instruction to Fabric as `RunRequest.input`. Harbor
 metadata such as model name, skills directory, and MCP server definitions are
-included under `RunRequest.context`. Fabric writes the normalized result to the
-Harbor logs directory as `fabric-result.json`, and summary fields are copied into
-`context.metadata["fabric"]`.
+included under `RunRequest.context`. The sandbox-local runner loads YAML into
+`FabricConfig` and `FabricProfileConfig`, then calls `FabricClient.run()`.
+The normalized result is saved as `fabric-result.json`, and summary fields are
+copied into `context.metadata["fabric"]`.
+
+## Multi-Harness Demo
+
+The runnable MVP demo includes explicit Harbor CLI commands for a
+credential-free pipeline check plus real Hermes, Hermes-with-Relay, and Codex
+variants. See [`demo/README.md`](demo/README.md) for the commands and recording
+flow.
 
 ## Local Smoke