[codex] Add Hermes capability hardening coverage

[AjayThorve]

Summary

Closes FABRIC-37

Adds focused coverage for the remaining Hermes capability-demonstration hardening gaps without introducing a standalone closeout smoke:

• extends tests/smoke_cli.py with the Hermes SDK/CLI profile matrix and Relay on/off plan assertions
• extends python/tests/smoke_sdk.py with SDK-fallback vs direct CLI normalized RunResult parity for Python and process adapters
• extends existing CLI/SDK smokes to verify Relay-disabled runs still expose native stdout artifacts, lifecycle events, and disabled telemetry metadata
• tightens the Harbor smoke so populate_context_from_result is exercised with non-empty Fabric artifacts, using lightweight stubs when the real Harbor package is not installed

Validation

• python3 python/tests/smoke_sdk.py
• python3 tests/smoke_cli.py
• python3 tests/smoke_hermes_cli.py
• python3 python/tests/smoke_harbor_integration.py
• git diff --check

Summary by CodeRabbit

• Bug Fixes

  • Improved Harbor integration smoke-test reliability when the Harbor package isn’t installed, and strengthened validation of returned artifact metadata.
  • Enhanced SDK-to-CLI parity checks for run results, including event structure, artifact contents, and telemetry fields.

• Tests

  • Expanded Hermes shim/relay smoke coverage across multiple profiles with stricter configuration and telemetry validation.
  • Added tighter assertions for “native observability” behavior when relay is disabled (stdout/stderr, events, and telemetry).

• Chores

  • Updated CI smoke-test selection for the dependency-free suite to include the Harbor integration smoke test.

[linear]

FABRIC-37 Close FABRIC-4 Hermes hardening coverage gaps

FABRIC-4 Demonstrate capabilities using Hermes Agent

Review in Linear

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 7af84448-f475-4fef-a5b4-b91edd705b58

📥 Commits

Reviewing files that changed from the base of the PR and between 76dd14a and 58fbf6e.

📒 Files selected for processing (1)

• .github/workflows/ci_python.yml

---

📝 Walkthrough

Walkthrough

Smoke tests now stub Harbor imports when Harbor is unavailable, validate Fabric artifact names, expand CLI profile planning checks, add relay-disabled observability assertions, compare SDK and CLI results for Hermes and process adapters, and update CI smoke selection.

Changes

Smoke test coverage updates

Layer / File(s)	Summary
Harbor import and metadata python/tests/smoke_harbor_integration.py	Minimal Harbor stubs are installed on import failure, the fabricated Fabric result includes named artifacts, and the test asserts those artifact names.
CLI profile and observability tests/smoke_cli.py, tests/smoke_hermes_cli.py	Multiple Hermes-related profiles are planned with adapter and telemetry assertions, and the Hermes CLI smoke run reuses the relay-disabled observability helper.
Native observability helpers tests/_utils/utils.py	Shared helpers validate relay-disabled artifact, event, and telemetry fields, and extend that check with process adapter returncode and stderr assertions.
SDK command and parity support python/tests/smoke_sdk.py	A shared CLI command tuple, process fixture path, CLI JSON runner, and SDK-vs-CLI parity helper are added for adapter comparisons.
SDK adapter runs python/tests/smoke_sdk.py	The smoke test copies Hermès and process adapter fixtures, runs both adapters through the SDK client and CLI, invokes observability checks, and parses the process adapter response payload.
CI smoke selection .github/workflows/ci_python.yml	The workflow narrows the dependency comment and adds the Harbor integration smoke to the dependency-free smoke list.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Suggested labels

enhancement

Suggested reviewers

• dagardner-nv

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 6.25% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title matches the main change: added Hermes hardening coverage across CLI, SDK, and Harbor smoke tests.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch ajay/fabric-37-close-fabric-4-hermes-hardening-coverage-gaps

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 1

🧹 Nitpick comments (1)

tests/smoke_hermes_cli.py (1)

67-85: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Extract this observability helper into shared test utilities.

This contract is now duplicated here, in tests/smoke_cli.py, and in the SDK smoke path. Keeping three near-copies in sync will drift quickly; a shared helper with optional process-only assertions would make future hardening changes safer.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/smoke_hermes_cli.py` around lines 67 - 85, The observability contract
asserted in assert_relay_disabled_native_observability is duplicated across
multiple smoke tests and will drift; extract the shared checks into a common
test utility and reuse it from the existing helper in this file,
tests/smoke_cli.py, and the SDK smoke path. Keep the helper focused on the
common artifact/event/telemetry assertions, and add an optional flag or separate
wrapper for process-only assertions so each caller can preserve its specific
stdout/stderr/returncode checks.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@tests/smoke_cli.py`:
- Around line 78-80: Update the smoke test in the CLI relay matrix to cover both
branches of the relay flag. In the test that inspects telemetry_plan, keep the
existing assertion for relay_enabled using telemetry_plan["relay_output_dir"],
and add a corresponding assertion for the disabled case to ensure that field is
absent or falsy when relay is off. Use the relay_enabled conditional in the test
body to locate the branch and make sure both enabled and disabled profiles are
validated.

---

Nitpick comments:
In `@tests/smoke_hermes_cli.py`:
- Around line 67-85: The observability contract asserted in
assert_relay_disabled_native_observability is duplicated across multiple smoke
tests and will drift; extract the shared checks into a common test utility and
reuse it from the existing helper in this file, tests/smoke_cli.py, and the SDK
smoke path. Keep the helper focused on the common artifact/event/telemetry
assertions, and add an optional flag or separate wrapper for process-only
assertions so each caller can preserve its specific stdout/stderr/returncode
checks.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: d8e19f29-9814-43db-b109-87a63bdec8b2

📥 Commits

Reviewing files that changed from the base of the PR and between 4b0488a and f0b4f9e.

📒 Files selected for processing (4)

• python/tests/smoke_harbor_integration.py
• python/tests/smoke_sdk.py
• tests/smoke_cli.py
• tests/smoke_hermes_cli.py