Skip to content

Add SDK guardrail smokes - #9

Merged
AjayThorve merged 4 commits into
NVIDIA:mainfrom
AjayThorve:ajay/ws4-guardrails
Jun 23, 2026
Merged

Add SDK guardrail smokes#9
AjayThorve merged 4 commits into
NVIDIA:mainfrom
AjayThorve:ajay/ws4-guardrails

Conversation

@AjayThorve

@AjayThorve AjayThorve commented Jun 23, 2026

Copy link
Copy Markdown
Collaborator

What

Adds three dependency-light SDK smoke tests that pin WS4 contract guarantees (test-only; no runtime code changes):

Smoke Guarantees
python/tests/smoke_typed_config.py Typed (in-memory) config is first-class: plan_config/doctor_config resolve a maintained adapter with no agent directory, run_config drives an inline run from an adapter-only dir (no agent package), and the *_config methods raise a clear FabricNativeUnavailableError on the CLI fallback (native-only).
python/tests/smoke_consumer_neutral.py The SDK core imports only the standard library and nemo_fabric (consumer glue stays under nemo_fabric.integrations); a plain import nemo_fabric pulls in no consumer/harness package. Also pins the zero-dependency contract (dependencies = []).
python/tests/smoke_readme_examples.py The README "Use Fabric" Python examples (plan/doctor/plan_config, the source-tree CLI-command form) execute against the example agent, with a drift guard asserting the README documents each invocation verbatim.

Why

These are the WS4 "keep ..." guardrails: typed config first-class, a consumer-neutral SDK API, and accurate README examples. They harden the contract a consumer (e.g. Platform) depends on, with near-zero risk.

Tests

No external setup or secrets required (native extension only):

python3 -m pip install -e .
python3 python/tests/smoke_typed_config.py
python3 python/tests/smoke_consumer_neutral.py
python3 python/tests/smoke_readme_examples.py

All three print smoke_<name> ok. The CLI README snippets remain covered by tests/smoke_cli.py.

Summary by CodeRabbit

Release Notes

  • Tests
    • Added a consumer-neutral smoke test ensuring the SDK core stays isolated from unwanted external packages.
    • Added a README “Use Fabric” smoke test that verifies documented snippets and executes the mirrored examples against the code-review agent.
    • Added a typed-config smoke test validating in-memory config resolution, diagnostics, and execution, including expected native-extension behavior for CLI-path usage.
  • Chores
    • Updated the CI Python smoke test suite to run the new scripts.

@coderabbitai

coderabbitai Bot commented Jun 23, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: c2e084bc-cbdd-4369-a343-c57ba6a3d4bf

📥 Commits

Reviewing files that changed from the base of the PR and between 29f857d and 69ef919.

📒 Files selected for processing (1)
  • python/tests/smoke_consumer_neutral.py
🚧 Files skipped from review as they are similar to previous changes (1)
  • python/tests/smoke_consumer_neutral.py

📝 Walkthrough

Walkthrough

Three new standalone smoke test scripts are added under python/tests/. One enforces that the nemo_fabric SDK core imports only stdlib via static AST analysis and a runtime subprocess check. One verifies README snippets appear verbatim and the documented FabricClient examples execute correctly. One validates typed in-memory config resolution, diagnosis, and inline runs without an on-disk agent package. The CI workflow is updated to execute all three tests in the dependency-free smoke loop.

Changes

SDK Smoke Tests

Layer / File(s) Summary
Consumer-neutrality static and runtime checks
python/tests/smoke_consumer_neutral.py
Defines SDK_ROOT, ALLOWED, CONSUMER_SPECIFIC; adds _top_level_imports AST walker; implements core_imports_only_stdlib_and_self (file-level import scan excluding integrations) and importing_the_sdk_pulls_in_no_consumer_package (subprocess sys.modules intersection); wires both via main().
README snippet presence and runnable examples
python/tests/smoke_readme_examples.py
Defines DOCUMENTED_SNIPPETS and README_PLAN_CONFIG typed dict; readme_documents_each_example asserts each snippet appears verbatim in README.md; readme_python_examples_run exercises plan/doctor/plan_config via FabricClient against examples/code-review-agent and asserts returned shapes; wires via async main().
Typed in-memory config resolution, diagnosis, and run
python/tests/smoke_typed_config.py
Defines _repository_adapter_config and _shim_adapter_config builders; resolves_and_diagnoses_without_a_directory calls plan_config/doctor_config with base_dir=None and an empty dir; runs_without_an_agent_package stages shim adapter into a temp dir and asserts run_config result; typed_config_requires_native confirms FabricNativeUnavailableError on CLI client path; wires via async main().
CI workflow smoke test loop
.github/workflows/ci_python.yml
Extends smokes array with three new test script paths for CI execution.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Possibly related PRs

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 69.23% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title 'Add SDK guardrail smokes' accurately describes the main change: adding three smoke test modules that validate SDK contracts and guarantees.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 4

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@python/tests/smoke_consumer_neutral.py`:
- Around line 15-17: The current test in smoke_consumer_neutral.py performs an
AST check on imports but does not directly verify the zero-dependency contract
declared in python/pyproject.toml. Add an assertion that reads the
pyproject.toml file, parses the dependencies field, and explicitly verifies that
dependencies is an empty list. This ensures that the declared packaging contract
matches the actual implementation and prevents accidental addition of
dependencies that could circumvent the static import check.
- Around line 91-93: The subprocess.run call that executes the probe without a
timeout parameter can hang indefinitely if the import blocks. Add a timeout
parameter to the subprocess.run call to enforce a maximum execution time, then
wrap it in a try-except block to catch subprocess.TimeoutExpired exceptions.
When handling both the TimeoutExpired exception and any other failures, include
the captured stderr in the error message to provide debugging context about what
went wrong during the probe execution.

In `@python/tests/smoke_readme_examples.py`:
- Line 62: The README.read_text() call uses the process default encoding instead
of explicitly specifying UTF-8, which can cause the test to fail on systems with
non-UTF-8 locale settings. Modify the read_text() method call to explicitly pass
the encoding parameter set to UTF-8 to ensure deterministic behavior across
different system locales and make the smoke test more reliable.

In `@python/tests/smoke_typed_config.py`:
- Around line 123-131: The test function typed_config_requires_native() only
validates that plan_config() raises FabricNativeUnavailableError when using CLI
fallback, but the WS4 contract requires this behavior for all typed-config
methods including doctor_config() and run_config(). Expand the test to also
invoke doctor_config() and run_config() on the cli_client instance to ensure all
three typed-config entrypoints properly raise FabricNativeUnavailableError and
prevent regressions in the other methods.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: a98be8b7-7457-465a-8540-bd498b051e15

📥 Commits

Reviewing files that changed from the base of the PR and between 7c9b0c1 and 0604444.

📒 Files selected for processing (3)
  • python/tests/smoke_consumer_neutral.py
  • python/tests/smoke_readme_examples.py
  • python/tests/smoke_typed_config.py

Comment thread python/tests/smoke_consumer_neutral.py
Comment thread python/tests/smoke_consumer_neutral.py
Comment thread python/tests/smoke_readme_examples.py Outdated
Comment thread python/tests/smoke_typed_config.py Outdated
@AjayThorve
AjayThorve marked this pull request as draft June 23, 2026 06:33
@AjayThorve AjayThorve changed the title Add WS4 SDK guardrail smokes Add SDK guardrail smokes Jun 23, 2026
Three dependency-light smokes that pin WS4 contract guarantees:

- smoke_typed_config: typed (in-memory) config is first-class. plan_config and
  doctor_config resolve a maintained adapter with no agent directory; run_config
  drives an inline run from an adapter-only dir (no agent package); and the
  *_config methods raise a clear error on the CLI fallback (native-only).
- smoke_consumer_neutral: the SDK core imports only the standard library and
  nemo_fabric (consumer glue stays under nemo_fabric.integrations), and a plain
  `import nemo_fabric` pulls in no consumer/harness package. Also pins the
  zero-dependency contract.
- smoke_readme_examples: the README "Use Fabric" Python examples execute against
  the example agent, with a drift guard that the README documents each verbatim.

Signed-off-by: Ajay Thorve <athorve@nvidia.com>
The Python CI runs an explicit smoke list; add the new dependency-free
guardrail smokes so they are enforced on every PR, not just present in the repo:
smoke_typed_config, smoke_consumer_neutral, smoke_readme_examples.

Signed-off-by: Ajay Thorve <athorve@nvidia.com>
@AjayThorve
AjayThorve force-pushed the ajay/ws4-guardrails branch from 0604444 to b4ac6aa Compare June 23, 2026 16:21
@AjayThorve
AjayThorve marked this pull request as ready for review June 23, 2026 16:24
- consumer_neutral: assert the declared zero-dependency contract directly
  (pyproject dependencies == []), which the docstring already claimed but the
  import scan alone could miss; bound the import-probe subprocess with a timeout.
- readme_examples: read the README as UTF-8 for locale-independent behavior.
- typed_config: extend the native-only check to doctor_config and run_config,
  not just plan_config (all three *_config methods are native-only).

Signed-off-by: Ajay Thorve <athorve@nvidia.com>

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@python/tests/smoke_consumer_neutral.py`:
- Around line 84-86: The current code uses chained .get() calls with default
values that silently default to an empty dependency list if either the [project]
section or dependencies key is missing from the pyproject.toml metadata. This
allows metadata regressions to go undetected. Replace the current approach of
using .get("project", {}).get("dependencies", []) with explicit assertions that
verify both the "project" key and the nested "dependencies" key exist in the
pyproject dictionary before accessing them, then extract the dependencies list.
This ensures the assertion properly validates the metadata structure while still
checking that dependencies is empty.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 9944c3ea-40f4-4f28-8046-847f7064acd8

📥 Commits

Reviewing files that changed from the base of the PR and between b4ac6aa and 29f857d.

📒 Files selected for processing (3)
  • python/tests/smoke_consumer_neutral.py
  • python/tests/smoke_readme_examples.py
  • python/tests/smoke_typed_config.py
🚧 Files skipped from review as they are similar to previous changes (2)
  • python/tests/smoke_readme_examples.py
  • python/tests/smoke_typed_config.py

Comment thread python/tests/smoke_consumer_neutral.py
@AjayThorve
AjayThorve requested a review from dagardner-nv June 23, 2026 16:41
A missing [project] table or dependencies key would silently default to an
empty list and pass the zero-dependency guard; assert both keys exist so a
metadata regression fails loudly rather than going undetected.

Signed-off-by: Ajay Thorve <athorve@nvidia.com>
@AjayThorve
AjayThorve merged commit 20b0b8a into NVIDIA:main Jun 23, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants