add check for kms key in token vault before create one

Author: jesseturner21

manual-kms-test.md

@@ -0,0 +1,222 @@
+# Manual KMS Key Fix Verification
+
+Tests for issue #337 — orphaned KMS keys on deploy.
+
+The fix uses `GetTokenVaultCommand` to check if the account's token vault already has a customer-managed KMS key before creating a new one. This prevents orphaned keys across deploys and across different projects in the same account.
+
+KMS keys are only created when the project uses a **non-Bedrock model provider** (Anthropic, OpenAI, Gemini), because those require storing an API key in the token vault via `ApiKeyCredentialProvider`. Bedrock uses IAM auth and skips this path entirely.
+
+## Setup
+
+```bash
+# Get credentials for the test account
+ada credentials update --account 325335451438 --provider isengard --role Admin --once
+
+# Set region
+export AWS_REGION=us-west-2
+
+# Create a throwaway test directory
+cd /tmp
+rm -rf kms-test-project && mkdir kms-test-project && cd kms-test-project
+
+# Build latest from source
+(cd /Volumes/workplace/kmskeybug/agentcore-cli && npm run build)
+
+# Use the local build of the CLI
+export AGENTCORE_CLI="/Volumes/workplace/kmskeybug/agentcore-cli/dist/cli/index.mjs"
+alias agentcore="node $AGENTCORE_CLI"
+
+# Create project with Anthropic model provider (triggers KMS path)
+agentcore create \
+  --name kmstest \
+  --framework Strands \
+  --model-provider Anthropic \
+  --api-key sk-ant-fake-key-for-testing-12345 \
+  --memory none \
+  --skip-git \
+  --skip-python-setup
+cd kmstest
+
+# Add a deployment target (the CLI does not create one automatically)
+cat > agentcore/aws-targets.json << 'EOF'
+[
+  {
+    "name": "default",
+    "account": "325335451438",
+    "region": "us-west-2"
+  }
+]
+EOF
+```
+
+## Helper: check token vault KMS config
+
+```bash
+# Check what key the token vault is currently using (source of truth)
+check_vault_kms() {
+  aws bedrock-agentcore-control get-token-vault \
+    --region $AWS_REGION \
+    --query 'kmsConfiguration' \
+    --output json
+}
+
+# Read KMS key ARN from local deployed state
+read_kms_arn() {
+  python3 -c "
+import json
+with open('agentcore/.cli/deployed-state.json') as f:
+    d = json.load(f)
+for t in d['targets'].values():
+    arn = (t.get('resources') or {}).get('identityKmsKeyArn')
+    if arn:
+        print(arn)
+        break
+"
+}
+```
+
+## Test 1: Key reuse across deploys (same project)
+
+Verify that a second deploy reuses the existing KMS key instead of creating a new one.
+
+```bash
+# First deploy
+agentcore deploy -y
+
+# Record the KMS key ARN
+KEY_ARN_1=$(read_kms_arn)
+echo "First deploy key: $KEY_ARN_1"
+
+if [ -z "$KEY_ARN_1" ]; then
+  echo "FAIL: No KMS key ARN found in agentcore/.cli/deployed-state.json after first deploy"
+  echo "Check that the project has ApiKeyCredentialProvider credentials."
+  exit 1
+fi
+
+# Check the vault directly
+echo "Vault KMS config after first deploy:"
+check_vault_kms
+
+# Second deploy
+agentcore deploy -y
+
+# Record the key ARN again
+KEY_ARN_2=$(read_kms_arn)
+echo "Second deploy key: $KEY_ARN_2"
+
+# VERIFY: both ARNs should be identical
+if [ "$KEY_ARN_1" = "$KEY_ARN_2" ]; then
+  echo "PASS: Key was reused"
+else
+  echo "FAIL: New key created on second deploy"
+  echo "  First:  $KEY_ARN_1"
+  echo "  Second: $KEY_ARN_2"
+fi
+
+# VERIFY: only one key tagged for this project exists
+KEY_COUNT=$(aws resourcegroupstaggingapi get-resources \
+  --tag-filters Key=agentcore:project,Values=kmstest \
+  --resource-type-filters kms:key \
+  --region $AWS_REGION \
+  --query 'ResourceTagMappingList | length(@)' \
+  --output text)
+echo "Keys tagged for this project: $KEY_COUNT"
+if [ "$KEY_COUNT" = "1" ]; then
+  echo "PASS: Only one KMS key exists"
+else
+  echo "FAIL: Expected 1 key, found $KEY_COUNT"
+fi
+```
+
+## Test 2: Key reuse across different projects (same account)
+
+Verify that a second project in the same account reuses the vault's existing CMK
+instead of creating a new orphaned key.
+
+```bash
+# Save the key ARN from project 1
+KEY_ARN_PROJECT1=$(read_kms_arn)
+echo "Project 1 key: $KEY_ARN_PROJECT1"
+
+# Create a second project in the same account
+cd /tmp/kms-test-project
+agentcore create \
+  --name kmstest2 \
+  --framework Strands \
+  --model-provider Anthropic \
+  --api-key sk-ant-fake-key-for-testing-67890 \
+  --memory none \
+  --skip-git \
+  --skip-python-setup
+cd kmstest2
+
+# Add a deployment target
+cat > agentcore/aws-targets.json << 'EOF'
+[
+  {
+    "name": "default",
+    "account": "325335451438",
+    "region": "us-west-2"
+  }
+]
+EOF
+
+# Deploy project 2
+agentcore deploy -y
+
+KEY_ARN_PROJECT2=$(read_kms_arn)
+echo "Project 2 key: $KEY_ARN_PROJECT2"
+
+# VERIFY: both projects should use the same key (the vault's CMK)
+if [ "$KEY_ARN_PROJECT1" = "$KEY_ARN_PROJECT2" ]; then
+  echo "PASS: Both projects share the same vault CMK"
+else
+  echo "FAIL: Project 2 created a new key instead of reusing the vault's CMK"
+  echo "  Project 1: $KEY_ARN_PROJECT1"
+  echo "  Project 2: $KEY_ARN_PROJECT2"
+fi
+
+# VERIFY: vault still has the same key
+echo "Vault KMS config after project 2 deploy:"
+check_vault_kms
+
+# Count total agentcore KMS keys in the account — should still be 1
+TOTAL_KEYS=$(aws resourcegroupstaggingapi get-resources \
+  --tag-filters Key=agentcore:project \
+  --resource-type-filters kms:key \
+  --region $AWS_REGION \
+  --query 'ResourceTagMappingList | length(@)' \
+  --output text)
+echo "Total agentcore KMS keys in account: $TOTAL_KEYS"
+if [ "$TOTAL_KEYS" = "1" ]; then
+  echo "PASS: No orphaned keys created"
+else
+  echo "FAIL: Expected 1 key total, found $TOTAL_KEYS"
+fi
+```
+
+## Cleanup
+
+```bash
+# Delete test project directories
+cd /tmp
+rm -rf kms-test-project
+
+# List any leftover KMS keys tagged for test projects
+aws resourcegroupstaggingapi get-resources \
+  --tag-filters Key=agentcore:project \
+  --resource-type-filters kms:key \
+  --region $AWS_REGION \
+  --query 'ResourceTagMappingList[].ResourceARN' \
+  --output table
+
+# To manually clean up KMS keys after all testing is done:
+# aws kms schedule-key-deletion --key-id KEY_ID --pending-window-in-days 7 --region $AWS_REGION
+```
+
+## Expected results summary
+
+| Test | Expected |
+|------|----------|
+| Test 1: Same project redeploy | Same KMS key ARN, only 1 key exists |
+| Test 2: Cross-project deploy | Both projects share the vault's CMK, no orphaned keys |