E2E policy test evidence (2026-03-22) by jesseturner21 · Pull Request #10 · jesseturner21/agentcore-cli

jesseturner21 · 2026-03-22T18:42:21Z

Policy E2E Test Evidence (2026-03-22)

Full end-to-end test of the policy engine lifecycle through the CLI TUI. All 13 steps completed successfully.

1. Main Menu & Add Resource

2. Gateway Creation (OAuth)

Gateway creation success output

✓ Added gateway: TestGateway
Gateway defined in agentcore/mcp.json. Next: Use 'add gateway-target' to route targets through this gateway.

3. Gateway Target Creation (ExaSearch MCP)

4. Agent Creation (SearchAgent - Strands/Python/Bedrock)

5. First Deploy — Verified: 4 tools available, web search works

Deploy completed output

✓ Deploy to AWS Complete [████████████████████] 8/8
Deployed 1 stack(s): AgentCore-policytest-default
Gateway Targets: ExaSearch: ✓ synced

6. First Invoke — Web search works

7. Policy Engine Creation (TestEngine → TestGateway)

8. Deploy with Policy Engine (ENFORCE mode) — Tools blocked by default deny

Deploy with enforce output

✓ Deploy to AWS Complete
Deployed 1 stack(s): AgentCore-policytest-default

9. Invoke — Default Deny Verified (only add_numbers available)

Agent response: "I don't have access to a web search tool to look up the latest AWS news. The only tool available to me is an add_numbers function."

10. Policy Generation via Natural Language

Prompt: "permit all web search calls unless the input contains Jesse Turner"

Generated Cedar Policy

permit(
  principal,
  action in [AgentCore::Action::"ExaSearch___web_search_exa",AgentCore::Action::"x_amz_bedrock_agentcore_search"],
  resource == AgentCore::Gateway::"arn:aws:bedrock-agentcore:us-west-2:325335451438:gateway/policytest-testgateway-8ckurs50xo"
) when {
  !(((context has input) && ((context.input) has query)) && (((context.input).query) like "*Jesse Turner*"))
};

11. Final Deploy with Policy

12. ✅ Invoke PERMITTED — "cloud computing" search returns results

13. ✅ Invoke BLOCKED — "Jesse Turner" search restricted by policy

Issues Found

Issue	Severity	Description
Progress bar overflow	Minor UI	Counter goes beyond total (6/5, 7/5...) causing `String.repeat` error with negative value
LOG_ONLY default	UX	Policy engine mode defaults to LOG_ONLY; must manually change to ENFORCE
TTY requirement	Expected	`agentcore deploy` requires interactive terminal for Ink TUI

Progress Flow

See progress.txt for the full step-by-step test log.

🤖 Generated with Claude Code

Add Cedar authorization policy support to AgentCore CLI: - Schema: PolicyEngine and Policy schemas with Zod validation - TUI: Full add/remove wizards for policy engines and policies - Source methods: Cedar file, inline statement, or AI generation - Gateway selection for generation flow - Expandable text input for generation prompts - CLI: Non-interactive add/remove commands with all flags - agentcore add policy-engine --name <name> - agentcore add policy --name <name> --engine <engine> --source/--statement/--generate - agentcore remove policy-engine/policy --name <name> - Deploy: CDK construct integration, CloudFormation output parsing, deployed state tracking with composite engine/policy keys - Status: Policy engines and policies shown in status command and ResourceGraph TUI with correct deployment state diffing - Generation: StartPolicyGeneration + waiter integration with deployed engine ID and gateway ARN resolution - Validation: Schema validation for names, statements, validation modes Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

…e collisions Policies are nested under engines, so the same policy name can exist in multiple engines. Switch getRemovable/remove/previewRemove to use an "engineName/policyName" composite key so the generic TUI remove flow can uniquely identify policies with a single string. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Regenerate lock file with npm@10 to resolve missing yaml@2.8.2 dependency entry that caused `npm ci` failures on Node 20.x and 22.x. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

- Merge duplicate imports in policy-generation.ts - Use dot notation instead of bracket notation in outputs test - Replace Array<T> with T[] in outputs.ts and useDeployFlow.ts - Add void operator for floating promises in AddPolicyFlow - Wrap async handlers with void for no-misused-promises - Escape quotes in JSX text in AddPolicyScreen - Fix prettier formatting across all changed files Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

… policy Previously, passing multiple source flags (e.g. --statement + --source) was silently accepted with an implicit precedence order. Now the command returns a clear error if more than one is provided. Also fix pre-existing type errors in dev config tests by adding the required policyEngines field to test fixtures. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Add interactive TUI support for removing policy engines and policies, including menu entries, selection screens, confirmation, and success states. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

…h credentials The createManagedOAuthCredential method was only writing the client secret with an incorrect env var name. Now correctly writes both _CLIENT_ID and _CLIENT_SECRET suffixed env vars, matching the pattern used by CredentialPrimitive. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

The "Generate a Cedar policy" option's disabled description was too long ("Requires deployed engine — run `deploy` first") and got cut off in narrow terminals. Shortened to "Deploy engine first". Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

…ion-truncation fix: shorten disabled policy generate description to prevent truncation

Screenshots of the complete TUI flow: create project, add agent, deploy, invoke, remove agent, deploy removal. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Full end-to-end test of policy engine lifecycle: - Gateway creation with OAuth (screenshots 01-10) - Gateway target creation - ExaSearch MCP (screenshot 11) - Agent creation - SearchAgent Strands/Python (screenshot 12) - Deploy and invoke - 4 tools available, web search works (screenshots 13-14) - Policy engine creation and ENFORCE mode deploy (screenshots 15-16) - Default deny verification - tools blocked (screenshot 17) - Natural language policy generation for Cedar (screenshot 18) - Final deploy with policy and invoke verification (screenshots 19-21) - Permitted: "cloud computing" search returns results - Blocked: "Jesse Turner" search restricted by policy Issues found: progress bar counter overflow, LOG_ONLY default mode, TTY requirement. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

github-actions · 2026-03-22T18:43:15Z

Package Tarball

aws-agentcore-0.3.0-preview.6.1.tgz

How to install

npm install https://github.com/jesseturner21/agentcore-cli/releases/download/pr-10-tarball/aws-agentcore-0.3.0-preview.6.1.tgz

jesseturner21 and others added 15 commits March 19, 2026 18:34

fix: sync package-lock.json for npm@10 compatibility

27457a2

Regenerate lock file with npm@10 to resolve missing yaml@2.8.2 dependency entry that caused `npm ci` failures on Node 20.x and 22.x. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

feat: add policy engine and policy support to TUI remove flow

4c56ed6

Add interactive TUI support for removing policy engines and policies, including menu entries, selection screens, confirmation, and success states. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

feat: add PolicyEngineConfiguration support for gateways

ad88e2c

feat: add policy engine selection to gateway TUI wizard

f4040ee

Merge branch 'aws:main' into main

9858591

chore: sync package-lock.json

c756de6

Merge pull request #7 from jesseturner21/fix/policy-generate-descript…

b2869e9

…ion-truncation fix: shorten disabled policy generate description to prevent truncation

Add e2e CLI TUI test evidence for full agent lifecycle

4fd437f

Screenshots of the complete TUI flow: create project, add agent, deploy, invoke, remove agent, deploy removal. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

github-actions bot added the size/xl PR size: XL label Mar 22, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

E2E policy test evidence (2026-03-22)#10

E2E policy test evidence (2026-03-22)#10
jesseturner21 wants to merge 15 commits intomainfrom
e2e-test-evidence-20260322

jesseturner21 commented Mar 22, 2026 •

edited

Loading

Uh oh!

github-actions bot commented Mar 22, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

jesseturner21 commented Mar 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Policy E2E Test Evidence (2026-03-22)

1. Main Menu & Add Resource

2. Gateway Creation (OAuth)

3. Gateway Target Creation (ExaSearch MCP)

4. Agent Creation (SearchAgent - Strands/Python/Bedrock)

5. First Deploy — Verified: 4 tools available, web search works

6. First Invoke — Web search works

7. Policy Engine Creation (TestEngine → TestGateway)

8. Deploy with Policy Engine (ENFORCE mode) — Tools blocked by default deny

9. Invoke — Default Deny Verified (only add_numbers available)

10. Policy Generation via Natural Language

11. Final Deploy with Policy

12. ✅ Invoke PERMITTED — "cloud computing" search returns results

13. ✅ Invoke BLOCKED — "Jesse Turner" search restricted by policy

Issues Found

Progress Flow

Uh oh!

github-actions bot commented Mar 22, 2026

Package Tarball

How to install

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

jesseturner21 commented Mar 22, 2026 •

edited

Loading