diff --git a/.gitignore b/.gitignore new file mode 100644 index 00000000..4c842e40 --- /dev/null +++ b/.gitignore @@ -0,0 +1,2 @@ +.env +.DS_Store \ No newline at end of file diff --git a/caipe/README.md b/caipe/README.md new file mode 100644 index 00000000..3d4e4cde --- /dev/null +++ b/caipe/README.md @@ -0,0 +1,237 @@ +# CAIPE - Cloud AI Platform Engineering + +CAIPE provides AI-powered platform engineering capabilities with support for multiple LLM providers and automated secret management. + +## Quick Start + +### 1. Deploy CAIPE Platform + +```bash +# Deploy the basic peer-to-peer configuration +./idpbuilder create --name caipe-basic-p2p \ + --use-path-routing \ + --package https://github.com/cnoe-io/stacks//ref-implementation \ + --package https://github.com/sriaradhyula/stacks//caipe/caipe-basic-p2p +``` + +### 2. Setup LLM Credentials + +Run the interactive setup script to configure your LLM provider: + +```bash +# Make the script executable +chmod +x setup-llm-credentials.sh + +# Run the setup script +./setup-llm-credentials.sh +``` + +**Note**: The script uses clean input handling - simply type or paste your credentials and press Enter. No special key combinations needed. + +The script supports the following LLM providers: + +#### Azure OpenAI +- API Key +- Endpoint URL +- API Version (default: 2024-02-15-preview) +- Deployment Name + +#### OpenAI +- API Key +- Endpoint (default: https://api.openai.com/v1) +- Model Name (default: gpt-4) + +#### AWS Bedrock +- Access Key ID +- Secret Access Key +- Region (default: us-east-1) +- Model ID (default: anthropic.claude-3-sonnet-20240229-v1:0) +- Provider (default: anthropic) + +#### Google Gemini +- API Key +- Model Name (default: gemini-pro) + +#### GCP Vertex AI +- Project ID +- Location (default: us-central1) +- Model Name (default: gemini-pro) + +### 3. Setup Agent Secrets + +Configure API keys and tokens for active agents: + +```bash +# Make the script executable +chmod +x setup-agent-secrets.sh + +# Run the setup script +./setup-agent-secrets.sh +``` + +**Note**: The script uses clean input handling - simply type or paste your credentials and press Enter. No special key combinations needed. + +The script supports the following agents and their required credentials: + +#### GitHub Agent +- **Personal Access Token**: GitHub API access token with repo permissions +- **Webhook Secret**: Optional secret for webhook validation + +#### GitLab Agent +- **Personal Access Token**: GitLab API access token +- **Webhook Secret**: Optional secret for webhook validation + +#### Jira Agent +- **API Token**: Jira API token for authentication +- **Base URL**: Jira instance URL (e.g., https://company.atlassian.net) +- **Username**: Jira username/email + +#### Slack Agent +- **Bot Token**: Slack bot token (xoxb-...) +- **App Token**: Slack app token (xapp-...) +- **Signing Secret**: Slack signing secret for request verification + +#### AWS Agent +- **Access Key ID**: AWS access key ID +- **Secret Access Key**: AWS secret access key +- **Region**: AWS region (default: us-east-1) + +### 4. Access Services + +After deployment, access the platform services: + +- **ArgoCD**: https://cnoe.localtest.me:8443/argocd +- **Backstage**: https://cnoe.localtest.me:8443/backstage +- **Vault**: https://vault.cnoe.localtest.me:8443/ui +- **Gitea**: https://gitea.cnoe.localtest.me:8443 + +### 5. Verify Setup + +Check that your credentials are properly stored: + +#### LLM Credentials +1. Access Vault UI: https://vault.cnoe.localtest.me:8443/ui +2. Navigate to: `secret/ai-platform-engineering/global` +3. Verify your LLM provider configuration + +#### Agent Secrets +1. Access Vault UI: https://vault.cnoe.localtest.me:8443/ui +2. Navigate to: `secret/ai-platform-engineering/agent-secrets` +3. Verify your agent API keys and tokens + +## Architecture + +CAIPE includes: + +- **Vault**: Secret management with automated token rotation +- **ArgoCD**: GitOps deployment with API token automation +- **Backstage**: Developer portal with API authentication +- **External Secrets**: Kubernetes secret synchronization +- **AI Agents**: Platform engineering automation + +## Security Features + +- Automated ArgoCD API token generation and rotation +- Vault-based secret management +- Secure credential storage with encryption +- RBAC for cross-namespace access +- No secrets exposed in logs + +## Utility Scripts + +### Refresh Secrets and Restart Deployments + +After updating secrets in Vault, use this script to refresh Kubernetes secrets and restart deployments: + +```bash +# Make the script executable +chmod +x refresh-secrets.sh + +# Run the refresh script +./refresh-secrets.sh +``` + +This script will: +- Check if Vault secrets exist and have data +- Delete Kubernetes secrets and wait for External Secrets to recreate them +- Restart corresponding deployments with rollout status verification + +### Sync ArgoCD Applications + +Ensure all ArgoCD applications are synced and healthy: + +```bash +# Make the script executable +chmod +x sync-apps.sh + +# Run the sync script +./sync-apps.sh +``` + +This script will sync: +- **backstage** - Developer portal +- **vault** - Secret management +- **argocd** - GitOps controller +- **ai-platform-engineering** - CAIPE stack +- **external-secrets** - Secret synchronization +- **ingress-nginx** - Ingress controller +- **gitea** - Git repository + +## Raw Script Access + +Download and run the setup scripts directly: + +### LLM Credentials Setup +```bash +# Download the script +curl -sSL https://raw.githubusercontent.com/sriaradhyula/stacks/main/caipe/setup-llm-credentials.sh -o setup-llm-credentials.sh + +# Make it executable and run +chmod +x setup-llm-credentials.sh +./setup-llm-credentials.sh +``` + +### Agent Secrets Setup +```bash +# Download the script +curl -sSL https://raw.githubusercontent.com/sriaradhyula/stacks/main/caipe/setup-agent-secrets.sh -o setup-agent-secrets.sh + +# Make it executable and run +chmod +x setup-agent-secrets.sh +./setup-agent-secrets.sh +``` + +**Note**: Both scripts use clean input handling - simply type or paste your credentials and press Enter. No special key combinations needed. + +### Utility Scripts + +#### Refresh Secrets +```bash +curl -sSL https://raw.githubusercontent.com/sriaradhyula/stacks/main/caipe/refresh-secrets.sh -o refresh-secrets.sh +chmod +x refresh-secrets.sh +./refresh-secrets.sh +``` + +#### Sync Applications +```bash +curl -sSL https://raw.githubusercontent.com/sriaradhyula/stacks/main/caipe/sync-apps.sh -o sync-apps.sh +chmod +x sync-apps.sh +./sync-apps.sh +``` + +## Troubleshooting + +### Prerequisites +- `kubectl` CLI installed and configured +- `vault` CLI installed +- Access to the CAIPE cluster + +### Common Issues + +1. **Vault connection failed**: Ensure port-forward is working and Vault is running +2. **Permission denied**: Check that you have access to the vault namespace +3. **Invalid provider**: Select a number from 1-5 for supported providers + +### Support + +For issues and questions, please refer to the [CNOE documentation](https://cnoe.io/docs/) or open an issue in the repository. diff --git a/ai-platform-engineering/ai-platform-engineering.yaml b/caipe/base/ai-platform-engineering.yaml similarity index 67% rename from ai-platform-engineering/ai-platform-engineering.yaml rename to caipe/base/ai-platform-engineering.yaml index 8a33ea4f..83a33f51 100644 --- a/ai-platform-engineering/ai-platform-engineering.yaml +++ b/caipe/base/ai-platform-engineering.yaml @@ -11,19 +11,17 @@ spec: # Main chart from GHCR - chart: ai-platform-engineering repoURL: ghcr.io/cnoe-io/helm-charts - targetRevision: 0.1.10 + targetRevision: 0.3.3 helm: + parameters: + - name: tags.basic + value: "true" valueFiles: - - $values/helm/values.yaml - - $values/helm/values-external-secrets.yaml + - $values/values.yaml # Values files from Git repository (your custom branch) - - repoURL: https://github.com/cnoe-io/ai-platform-engineering.git - targetRevision: idpbuilder-values + - repoURL: cnoe://ai-platform-engineering + path: manifests ref: values - # Raw manifests from Git - - repoURL: https://github.com/cnoe-io/stacks.git - targetRevision: main - path: ai-platform-engineering/ai-platform-engineering/manifests destination: server: https://kubernetes.default.svc namespace: ai-platform-engineering diff --git a/ai-platform-engineering/ai-platform-engineering/manifests/ingress.yaml b/caipe/base/ai-platform-engineering/manifests/ingress.yaml similarity index 86% rename from ai-platform-engineering/ai-platform-engineering/manifests/ingress.yaml rename to caipe/base/ai-platform-engineering/manifests/ingress.yaml index e01283bc..52c1e505 100644 --- a/ai-platform-engineering/ai-platform-engineering/manifests/ingress.yaml +++ b/caipe/base/ai-platform-engineering/manifests/ingress.yaml @@ -16,7 +16,7 @@ spec: pathType: ImplementationSpecific backend: service: - name: ai-platform-engineering + name: ai-platform-engineering-supervisor-agent port: number: 8000 - host: cnoe.localtest.me @@ -26,6 +26,6 @@ spec: pathType: ImplementationSpecific backend: service: - name: ai-platform-engineering + name: ai-platform-engineering-supervisor-agent port: number: 8000 diff --git a/caipe/base/ai-platform-engineering/values.yaml b/caipe/base/ai-platform-engineering/values.yaml new file mode 100644 index 00000000..224adfd5 --- /dev/null +++ b/caipe/base/ai-platform-engineering/values.yaml @@ -0,0 +1,468 @@ +# Override values for idpbuilder local deployment - Base configuration +# Global configuration shared across all subcharts +global: + slim: + enabled: false + + externalSecrets: + enabled: true + secretStoreRef: + name: "vault-secret-store" + kind: ClusterSecretStore + + agentSecrets: + create: true + + llmSecrets: + create: false + secretName: "llm-secret" + externalSecrets: + secretStoreRef: + name: "vault-secret-store" + kind: ClusterSecretStore + data: + - secretKey: LLM_PROVIDER + remoteRef: + key: secret/ai-platform-engineering/global + property: LLM_PROVIDER + + # Azure OpenAI configuration + - secretKey: AZURE_OPENAI_API_KEY + remoteRef: + conversionStrategy: Default + decodingStrategy: None + key: secret/ai-platform-engineering/global + property: AZURE_OPENAI_API_KEY + - secretKey: AZURE_OPENAI_ENDPOINT + remoteRef: + conversionStrategy: Default + decodingStrategy: None + key: secret/ai-platform-engineering/global + property: AZURE_OPENAI_ENDPOINT + - secretKey: AZURE_OPENAI_API_VERSION + remoteRef: + conversionStrategy: Default + decodingStrategy: None + key: secret/ai-platform-engineering/global + property: AZURE_OPENAI_API_VERSION + - secretKey: OPENAI_API_VERSION + remoteRef: + conversionStrategy: Default + decodingStrategy: None + key: secret/ai-platform-engineering/global + property: AZURE_OPENAI_API_VERSION + - secretKey: AZURE_OPENAI_DEPLOYMENT + remoteRef: + conversionStrategy: Default + decodingStrategy: None + key: secret/ai-platform-engineering/global + property: AZURE_OPENAI_DEPLOYMENT + # OpenAI configuration + - secretKey: OPENAI_API_KEY + remoteRef: + conversionStrategy: Default + decodingStrategy: None + key: secret/ai-platform-engineering/global + property: OPENAI_API_KEY + - secretKey: OPENAI_ENDPOINT + remoteRef: + conversionStrategy: Default + decodingStrategy: None + key: secret/ai-platform-engineering/global + property: OPENAI_ENDPOINT + - secretKey: OPENAI_MODEL_NAME + remoteRef: + conversionStrategy: Default + decodingStrategy: None + key: secret/ai-platform-engineering/global + property: OPENAI_MODEL_NAME + # AWS Bedrock configuration + - secretKey: AWS_ACCESS_KEY_ID + remoteRef: + conversionStrategy: Default + decodingStrategy: None + key: secret/ai-platform-engineering/global + property: AWS_ACCESS_KEY_ID + - secretKey: AWS_SECRET_ACCESS_KEY + remoteRef: + conversionStrategy: Default + decodingStrategy: None + key: secret/ai-platform-engineering/global + property: AWS_SECRET_ACCESS_KEY + - secretKey: AWS_REGION + remoteRef: + conversionStrategy: Default + decodingStrategy: None + key: secret/ai-platform-engineering/global + property: AWS_REGION + - secretKey: AWS_BEDROCK_MODEL_ID + remoteRef: + conversionStrategy: Default + decodingStrategy: None + key: secret/ai-platform-engineering/global + property: AWS_BEDROCK_MODEL_ID + - secretKey: AWS_BEDROCK_PROVIDER + remoteRef: + conversionStrategy: Default + decodingStrategy: None + key: secret/ai-platform-engineering/global + property: AWS_BEDROCK_PROVIDER + +supervisor-agent: + nameOverride: "supervisor-agent" + image: + repository: "ghcr.io/cnoe-io/ai-platform-engineering" + tag: "latest" + env: + EXTERNAL_URL: "https://cnoe.localtest.me:8443/ai-platform-engineering" + RAG_AGENT_PORT: 8099 +agent-argocd: + nameOverride: "agent-argocd" + image: + repository: "ghcr.io/cnoe-io/agent-argocd" + tag: "latest" + pullPolicy: "Always" + mcp: + image: + repository: "ghcr.io/cnoe-io/mcp-argocd" + tag: "latest" + pullPolicy: "Always" + mode: "http" # Options: stdio, http + port: 8000 + agentSecrets: + secretName: "agent-argocd-secret" + externalSecrets: + data: + - secretKey: ARGOCD_TOKEN + remoteRef: + key: secret/ai-platform-engineering/argocd-secret + property: ARGOCD_TOKEN + - secretKey: ARGOCD_API_URL + remoteRef: + key: secret/ai-platform-engineering/argocd-secret + property: ARGOCD_API_URL + - secretKey: ARGOCD_VERIFY_SSL + remoteRef: + key: secret/ai-platform-engineering/argocd-secret + property: ARGOCD_VERIFY_SSL + +agent-aws: + nameOverride: "agent-aws" + image: + repository: "ghcr.io/cnoe-io/agent-aws" + pullPolicy: "Always" + tag: "latest" + mcp: + mode: "stdio" # Options: stdio, http + # image: # HOT FIX: will be modified in the helm chart itself in the future + # repository: "public.ecr.aws/awslabs-mcp/awslabs/eks-mcp-server" + # tag: "sha-e18497bbc401f08334ea8b966db1f235b647bfed" + agentSecrets: + secretName: "agent-aws-secret" + externalSecrets: + data: + - secretKey: AWS_ACCESS_KEY_ID + remoteRef: + key: secret/ai-platform-engineering/aws-secret + property: AWS_ACCESS_KEY_ID + - secretKey: AWS_SECRET_ACCESS_KEY + remoteRef: + key: secret/ai-platform-engineering/aws-secret + property: AWS_SECRET_ACCESS_KEY + - secretKey: AWS_REGION + remoteRef: + key: secret/ai-platform-engineering/aws-secret + property: AWS_REGION + +agent-backstage: + nameOverride: "agent-backstage" + image: + repository: "ghcr.io/cnoe-io/agent-backstage" + pullPolicy: "Always" + tag: "latest" + mcp: + image: + repository: "ghcr.io/cnoe-io/mcp-backstage" + tag: "latest" + pullPolicy: "Always" + mode: "http" # Options: stdio, http + port: 8000 + agentSecrets: + secretName: "agent-backstage-secret" + externalSecrets: + data: + - secretKey: BACKSTAGE_API_TOKEN + remoteRef: + key: secret/ai-platform-engineering/backstage-secret + property: BACKSTAGE_API_TOKEN + - secretKey: BACKSTAGE_URL + remoteRef: + key: secret/ai-platform-engineering/backstage-secret + property: BACKSTAGE_URL + +agent-confluence: + nameOverride: "agent-confluence" + image: + repository: "ghcr.io/cnoe-io/agent-confluence" + pullPolicy: "Always" + tag: "latest" + mcp: + image: + repository: "ghcr.io/cnoe-io/mcp-confluence" + tag: "latest" + pullPolicy: "Always" + mode: "http" # Options: stdio, http + port: 8000 + agentSecrets: + secretName: "agent-confluence-secret" + externalSecrets: + data: + - secretKey: ATLASSIAN_TOKEN + remoteRef: + key: secret/ai-platform-engineering/confluence-secret + property: ATLASSIAN_TOKEN + - secretKey: ATLASSIAN_EMAIL + remoteRef: + key: secret/ai-platform-engineering/confluence-secret + property: ATLASSIAN_EMAIL + - secretKey: CONFLUENCE_API_URL + remoteRef: + key: secret/ai-platform-engineering/confluence-secret + property: CONFLUENCE_API_URL + +agent-github: + nameOverride: "agent-github" + image: + repository: "ghcr.io/cnoe-io/agent-github" + pullPolicy: "Always" + tag: "latest" + mcp: + useRemoteMcpServer: true + agentSecrets: + secretName: "agent-github-secret" + externalSecrets: + data: + - secretKey: GITHUB_PERSONAL_ACCESS_TOKEN + remoteRef: + key: secret/ai-platform-engineering/github-secret + property: GITHUB_PERSONAL_ACCESS_TOKEN + +agent-jira: + nameOverride: "agent-jira" + image: + repository: "ghcr.io/cnoe-io/agent-jira" + pullPolicy: "Always" + tag: "latest" + mcp: + image: + repository: "ghcr.io/cnoe-io/mcp-jira" + tag: "latest" + pullPolicy: "Always" + mode: "http" # Options: stdio, http + port: 8000 + agentSecrets: + secretName: "agent-jira-secret" + externalSecrets: + data: + - secretKey: ATLASSIAN_TOKEN + remoteRef: + key: secret/ai-platform-engineering/jira-secret + property: ATLASSIAN_TOKEN + - secretKey: ATLASSIAN_EMAIL + remoteRef: + key: secret/ai-platform-engineering/jira-secret + property: ATLASSIAN_EMAIL + - secretKey: ATLASSIAN_API_URL + remoteRef: + key: secret/ai-platform-engineering/jira-secret + property: ATLASSIAN_API_URL + + +agent-komodor: + nameOverride: "agent-komodor" + image: + repository: "ghcr.io/cnoe-io/agent-komodor" + pullPolicy: "Always" + tag: "latest" + mcp: + image: + repository: "ghcr.io/cnoe-io/mcp-komodor" + tag: "latest" + pullPolicy: "Always" + mode: "http" # Options: stdio, http + port: 8000 + agentSecrets: + secretName: "agent-komodor-secret" + externalSecrets: + data: + - secretKey: KOMODOR_TOKEN + remoteRef: + key: secret/ai-platform-engineering/komodor-secret + property: KOMODOR_TOKEN + - secretKey: KOMODOR_API_URL + remoteRef: + key: secret/ai-platform-engineering/komodor-secret + property: KOMODOR_API_URL + +agent-pagerduty: + nameOverride: "agent-pagerduty" + image: + repository: "ghcr.io/cnoe-io/agent-pagerduty" + pullPolicy: "Always" + tag: "latest" + mcp: + image: + repository: "ghcr.io/cnoe-io/mcp-pagerduty" + tag: "latest" + pullPolicy: "Always" + mode: "http" # Options: stdio, http + port: 8000 + agentSecrets: + secretName: "agent-pagerduty-secret" + externalSecrets: + data: + - secretKey: PAGERDUTY_API_KEY + remoteRef: + key: secret/ai-platform-engineering/pagerduty-secret + property: PAGERDUTY_API_KEY + - secretKey: PAGERDUTY_API_URL + remoteRef: + key: secret/ai-platform-engineering/pagerduty-secret + property: PAGERDUTY_API_URL + +agent-slack: + nameOverride: "agent-slack" + image: + repository: "ghcr.io/cnoe-io/agent-slack" + pullPolicy: "Always" + tag: "latest" + mcp: + image: + repository: "ghcr.io/cnoe-io/mcp-slack" + tag: "latest" + pullPolicy: "Always" + mode: "http" # Options: stdio, http + port: 8000 + pullPolicy: "Always" + agentSecrets: + secretName: "agent-slack-secret" + externalSecrets: + data: + - secretKey: SLACK_BOT_TOKEN + remoteRef: + key: secret/ai-platform-engineering/slack-secret + property: SLACK_BOT_TOKEN + - secretKey: SLACK_APP_TOKEN + remoteRef: + key: secret/ai-platform-engineering/slack-secret + property: SLACK_APP_TOKEN + - secretKey: SLACK_SIGNING_SECRET + remoteRef: + key: secret/ai-platform-engineering/slack-secret + property: SLACK_SIGNING_SECRET + - secretKey: SLACK_CLIENT_SECRET + remoteRef: + key: secret/ai-platform-engineering/slack-secret + property: SLACK_CLIENT_SECRET + - secretKey: SLACK_TEAM_ID + remoteRef: + key: secret/ai-platform-engineering/slack-secret + property: SLACK_TEAM_ID + +agent-splunk: + nameOverride: "agent-splunk" + image: + repository: "ghcr.io/cnoe-io/agent-splunk" + pullPolicy: "Always" + tag: "latest" + mcp: + image: + repository: "ghcr.io/cnoe-io/mcp-splunk" + tag: "latest" + pullPolicy: "Always" + mode: "http" # Options: stdio, http + port: 8000 + agentSecrets: + secretName: "agent-splunk-secret" + externalSecrets: + data: + - secretKey: SPLUNK_TOKEN + remoteRef: + key: secret/ai-platform-engineering/splunk-secret + property: SPLUNK_TOKEN + - secretKey: SPLUNK_API_URL + remoteRef: + key: secret/ai-platform-engineering/splunk-secret + property: SPLUNK_API_URL + +agent-webex: + nameOverride: "agent-webex" + image: + repository: "ghcr.io/cnoe-io/agent-webex" + pullPolicy: "Always" + tag: "latest" + mcp: + image: + repository: "ghcr.io/cnoe-io/mcp-webex" + tag: "latest" + pullPolicy: "Always" + mode: "http" # Options: stdio, http + port: 8000 + agentSecrets: + secretName: "agent-webex-secret" + externalSecrets: + data: + - secretKey: WEBEX_BOT_TOKEN + remoteRef: + key: secret/ai-platform-engineering/webex-secret + property: WEBEX_BOT_TOKEN + - secretKey: WEBEX_WEBHOOK_SECRET + remoteRef: + key: secret/ai-platform-engineering/webex-secret + property: WEBEX_WEBHOOK_SECRET + - secretKey: WEBEX_TOKEN + remoteRef: + key: secret/ai-platform-engineering/webex-secret + property: WEBEX_TOKEN + +rag-stack: + rag-webui: + ingress: + enabled: true + className: "nginx" + hosts: + - host: rag-webui.cnoe.localtest.me + paths: + - path: / + pathType: Prefix + tls: + - hosts: + - rag-webui.cnoe.localtest.me + secretName: rag-webui-tls + + # Neo4j storage class override for KIND compatibility + neo4j: + volumes: + data: + mode: "dynamic" + dynamic: + storageClassName: standard # Override from gp2 to standard for KIND + + # Neo4j Ontology storage class override for KIND compatibility + neo4j-ontology: + volumes: + data: + mode: "dynamic" + dynamic: + storageClassName: standard # Override from gp2 to standard for KIND + + milvus: + # Only override what differs from chart defaults + pulsarv3: + enabled: false # Default is true, we need false + woodpecker: + enabled: true # Default is false, we need true + + # Jarvis node scheduling + + # Performance overrides only for compute nodes \ No newline at end of file diff --git a/ai-platform-engineering/backstage.yaml b/caipe/base/backstage.yaml similarity index 100% rename from ai-platform-engineering/backstage.yaml rename to caipe/base/backstage.yaml diff --git a/ai-platform-engineering/backstage/manifests/argocd-secrets.yaml b/caipe/base/backstage/manifests/argocd-secrets.yaml similarity index 100% rename from ai-platform-engineering/backstage/manifests/argocd-secrets.yaml rename to caipe/base/backstage/manifests/argocd-secrets.yaml diff --git a/caipe/base/backstage/manifests/install.yaml b/caipe/base/backstage/manifests/install.yaml new file mode 100644 index 00000000..b6cf75cf --- /dev/null +++ b/caipe/base/backstage/manifests/install.yaml @@ -0,0 +1,579 @@ +apiVersion: v1 +kind: Namespace +metadata: + name: backstage + annotations: + argocd.argoproj.io/sync-wave: "-10" +--- +apiVersion: v1 +kind: ServiceAccount +metadata: + name: backstage + namespace: backstage + annotations: + argocd.argoproj.io/sync-wave: "-10" +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + name: backstage-argo-worfklows + annotations: + argocd.argoproj.io/sync-wave: "-10" +rules: + - apiGroups: + - argoproj.io + resources: + - workflows + verbs: + - create +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + name: read-all + annotations: + argocd.argoproj.io/sync-wave: "-10" +rules: + - apiGroups: + - '*' + resources: + - '*' + verbs: + - get + - list + - watch +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRoleBinding +metadata: + name: backstage-argo-worfklows + annotations: + argocd.argoproj.io/sync-wave: "-10" +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: backstage-argo-worfklows +subjects: + - kind: ServiceAccount + name: backstage + namespace: backstage +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRoleBinding +metadata: + name: backstage-read-all + annotations: + argocd.argoproj.io/sync-wave: "-10" +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: read-all +subjects: + - kind: ServiceAccount + name: backstage + namespace: backstage +--- +apiVersion: v1 +kind: ConfigMap +metadata: + name: backstage-config + namespace: backstage +data: + app-config.yaml: | + app: + title: CNOE Backstage + baseUrl: https://cnoe.localtest.me:8443 + organization: + name: CNOE + backend: + # Used for enabling authentication, secret is shared by all backend plugins + # See https://backstage.io/docs/tutorials/backend-to-backend-auth for + # information on the format + auth: + externalAccess: + - options: + subject: caipe-api-access + token: ${BACKSTAGE_API_TOKEN} + type: static + keys: + - secret: ${BACKSTAGE_API_TOKEN} + baseUrl: https://cnoe.localtest.me:8443 + listen: + port: 7007 + # Uncomment the following host directive to bind to specific interfaces + # host: 127.0.0.1 + csp: + connect-src: ["'self'", 'http:', 'https:'] + # Content-Security-Policy directives follow the Helmet format: https://helmetjs.github.io/#reference + # Default Helmet Content-Security-Policy values can be removed by setting the key to false + cors: + origin: https://cnoe.localtest.me:8443 + methods: [GET, HEAD, PATCH, POST, PUT, DELETE] + credentials: true + database: + client: pg + connection: + host: ${POSTGRES_HOST} + port: ${POSTGRES_PORT} + user: ${POSTGRES_USER} + password: ${POSTGRES_PASSWORD} + cache: + store: memory + # workingDirectory: /tmp # Use this to configure a working directory for the scaffolder, defaults to the OS temp-dir + + integrations: + gitea: + - baseUrl: https://cnoe.localtest.me:8443/gitea + host: cnoe.localtest.me:8443 + username: ${GITEA_USERNAME} + password: ${GITEA_PASSWORD} + - baseUrl: https://cnoe.localtest.me/gitea + host: cnoe.localtest.me + username: ${GITEA_USERNAME} + password: ${GITEA_PASSWORD} + # github: + # - host: github.com + # apps: + # - $include: github-integration.yaml + # - host: github.com + # # This is a Personal Access Token or PAT from GitHub. You can find out how to generate this token, and more information + # # about setting up the GitHub integration here: https://backstage.io/docs/getting-started/configuration#setting-up-a-github-integration + # token: ${GITHUB_TOKEN} + ### Example for how to add your GitHub Enterprise instance using the API: + # - host: ghe.example.net + # apiBaseUrl: https://ghe.example.net/api/v3 + # token: ${GHE_TOKEN} + + # Reference documentation http://backstage.io/docs/features/techdocs/configuration + # Note: After experimenting with basic setup, use CI/CD to generate docs + # and an external cloud storage when deploying TechDocs for production use-case. + # https://backstage.io/docs/features/techdocs/how-to-guides#how-to-migrate-from-techdocs-basic-to-recommended-deployment-approach + techdocs: + builder: 'local' # Alternatives - 'external' + generator: + runIn: 'local' + publisher: + type: 'local' # Alternatives - 'googleGcs' or 'awsS3'. Read documentation for using alternatives. + + auth: + environment: development + session: + secret: MW2sV-sIPngEl26vAzatV-6VqfsgAx4bPIz7PuE_2Lk= + providers: + keycloak-oidc: + development: + metadataUrl: ${KEYCLOAK_NAME_METADATA} + clientId: backstage + clientSecret: ${KEYCLOAK_CLIENT_SECRET} + prompt: auto + + scaffolder: + # see https://backstage.io/docs/features/software-templates/configuration for software template options + defaultAuthor: + name: backstage-scaffolder + email: noreply + defaultCommitMessage: "backstage scaffolder" + catalog: + import: + entityFilename: catalog-info.yaml + pullRequestBranchName: backstage-integration + rules: + - allow: [Component, System, API, Resource, Location, Template] + locations: + # Examples from a public GitHub repository. + - type: url + target: https://cnoe.localtest.me/gitea/giteaAdmin/idpbuilder-localdev-backstage-templates-entities/raw/branch/main/catalog-info.yaml + rules: + - allow: [Component, System, API, Resource, Location, Template, User, Group] + kubernetes: + serviceLocatorMethod: + type: 'multiTenant' + clusterLocatorMethods: + - $include: k8s-config.yaml + argocd: + username: admin + password: ${ARGOCD_ADMIN_PASSWORD} + appLocatorMethods: + - type: 'config' + instances: + - name: in-cluster + url: https://cnoe.localtest.me:8443/argocd + username: admin + password: ${ARGOCD_ADMIN_PASSWORD} + argoWorkflows: + baseUrl: ${ARGO_WORKFLOWS_URL} + agentForge: + baseUrl: ${AGENT_FORGE_URL} + botName: CAIPE + infoPage: https://cnoe-io.github.io/ai-platform-engineering/ + showOptions: true + useOpenIDToken: true # Set to true to use OpenIdConnectApi.getIdToken(), false to use IdentityApi.getCredentials() + enableStreaming: false # Set to true to enable streaming responses, false for non-streaming + requestTimeout: 300 + headerTitle: CAIPE + headerSubtitle: AI Platform Engineer Assistant + inputPlaceholder: Ask CAIPE anything... + initialSuggestions: + - Create GitHub Repository + - Deploy ArgoCD Application + - Create AWS Resources + - Get LLM keys + - Add MyID Groups + - Invite users to Github Organization + thinkingMessagesInterval: 7000 # Interval in milliseconds (default: 7000) + thinkingMessages: + - "โš™๏ธ Processing query โ€” even great thoughts start with a single thread." + - "๐Ÿค– Contacting agents โ€” because collaboration beats computation." + - "๐Ÿ“ฆ Carrying bits โ€” somewhere in here lies meaningโ€ฆ and metadata." + - "๐Ÿ“Š Analyzing data โ€” the truth hides between the averages." + - "๐Ÿง  Reflecting on the prompt โ€” clarity is the best optimization." + - "๐ŸŒ Syncing thoughts across nodes โ€” wisdom prefers low latency." + - "๐Ÿงฉ Assembling context โ€” the art of not missing the missing piece." + - "๐Ÿ” Searching the void โ€” if you stare long enough into the logs, the logs stare back." + - "๐Ÿชž Reflecting reasoning โ€” recursive thinking: now 30% more self-aware." + - "๐Ÿงญ Following agent breadcrumbs โ€” because even distributed minds need a map." + - "๐Ÿ”ง Calibrating models โ€” what is alignment, if not purpose with precision?" + - "๐Ÿ’ญ Daydreaming in embeddings โ€” vectors have feelings too." + - "๐Ÿšฆ Negotiating with rate limits โ€” patience is also a form of throughput." + - "๐Ÿ“š Reading context โ€” a good agent never stops learning." + - "๐Ÿ•ธ๏ธ Weaving reasoning chains โ€” because one thought alone never scales." + - "๐Ÿ’ก Illuminating intent โ€” signal, not noise โ€” that's the real bandwidth." + - "๐Ÿงฎ Counting tokens โ€” every byte a step toward enlightenment." + - "๐Ÿ•ฐ๏ธ Waiting for consensus โ€” distributed wisdom takes time." + - "๐Ÿงฐ Debugging the universe โ€” there's always one more missing semicolon." + - "๐ŸŒŒ Manifesting insights โ€” the cosmos runs on context, too." + +--- +apiVersion: v1 +kind: Secret +metadata: + name: k8s-config + namespace: backstage +stringData: + k8s-config.yaml: "type: 'config'\nclusters:\n - url: https://kubernetes.default.svc.cluster.local\n + \ name: local\n authProvider: 'serviceAccount'\n skipTLSVerify: true\n + \ skipMetricsLookup: true\n serviceAccountToken: \n $file: /var/run/secrets/kubernetes.io/serviceaccount/token\n + \ caData: \n $file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt\n" +--- +apiVersion: v1 +kind: Service +metadata: + name: backstage + namespace: backstage +spec: + ports: + - name: http + port: 7007 + targetPort: http + selector: + app: backstage +--- +apiVersion: v1 +kind: Service +metadata: + labels: + app: postgresql + name: postgresql + namespace: backstage +spec: + clusterIP: None + ports: + - name: postgres + port: 5432 + selector: + app: postgresql +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + name: backstage + namespace: backstage + annotations: + argocd.argoproj.io/sync-wave: "20" +spec: + replicas: 1 + selector: + matchLabels: + app: backstage + template: + metadata: + labels: + app: backstage + spec: + containers: + - command: + - node + - packages/backend + - --config + - config/app-config.yaml + env: + - name: LOG_LEVEL + value: debug + - name: NODE_TLS_REJECT_UNAUTHORIZED + value: "0" + envFrom: + - secretRef: + name: backstage-env-vars + - secretRef: + name: gitea-credentials + - secretRef: + name: argocd-credentials + - secretRef: + name: backstage-api-token + image: ghcr.io/sriaradhyula/backstage-app:35e30d165584e583b205c365d44cb938864896c7 + name: backstage + ports: + - containerPort: 7007 + name: http + volumeMounts: + - mountPath: /app/config + name: backstage-config + readOnly: true + serviceAccountName: backstage + volumes: + - name: backstage-config + projected: + sources: + - configMap: + items: + - key: app-config.yaml + path: app-config.yaml + name: backstage-config + - secret: + items: + - key: k8s-config.yaml + path: k8s-config.yaml + name: k8s-config +--- +apiVersion: apps/v1 +kind: StatefulSet +metadata: + labels: + app: postgresql + name: postgresql + namespace: backstage + annotations: + argocd.argoproj.io/sync-wave: "10" +spec: + replicas: 1 + selector: + matchLabels: + app: postgresql + serviceName: service-postgresql + template: + metadata: + labels: + app: postgresql + spec: + containers: + - env: + - name: POSTGRES_DB + valueFrom: + secretKeyRef: + name: backstage-env-vars + key: POSTGRES_DB + - name: POSTGRES_USER + valueFrom: + secretKeyRef: + name: backstage-env-vars + key: POSTGRES_USER + - name: POSTGRES_PASSWORD + valueFrom: + secretKeyRef: + name: backstage-env-vars + key: POSTGRES_PASSWORD + image: docker.io/library/postgres:15.3-alpine3.18 + name: postgres + ports: + - containerPort: 5432 + name: postgresdb + resources: + limits: + memory: 500Mi + requests: + cpu: 100m + memory: 300Mi + volumeMounts: + - name: data + mountPath: /var/lib/postgresql/data + volumeClaimTemplates: + - metadata: + name: data + spec: + accessModes: ["ReadWriteOnce"] + resources: + requests: + storage: "500Mi" +--- +apiVersion: batch/v1 +kind: Job +metadata: + name: backstage-dependency-wait + namespace: backstage + annotations: + argocd.argoproj.io/sync-wave: "-1" +spec: + template: + spec: + serviceAccountName: backstage + containers: + - name: dependency-wait + image: bitnami/kubectl:latest + command: + - /bin/bash + - -c + - | + echo "Waiting for dependencies to be ready..." + + while true; do + echo "Checking vault-config job status in vault namespace..." + + # Check if vault-config job exists and is complete + VAULT_JOB_STATUS=$(kubectl get job vault-config -n vault -o jsonpath='{.status.conditions[?(@.type=="Complete")].status}' 2>/dev/null || echo "NotFound") + + echo "Checking keycloak deployment status in keycloak namespace..." + + # Check if keycloak deployment exists and is ready + KEYCLOAK_READY=$(kubectl get deployment keycloak -n keycloak -o jsonpath='{.status.conditions[?(@.type=="Available")].status}' 2>/dev/null || echo "NotFound") + + echo "Vault job status: $VAULT_JOB_STATUS" + echo "Keycloak deployment ready: $KEYCLOAK_READY" + + if [[ "$VAULT_JOB_STATUS" == "True" && "$KEYCLOAK_READY" == "True" ]]; then + echo "All dependencies are ready! Exiting..." + exit 0 + else + echo "Dependencies not ready yet. Waiting 30 seconds before next check..." + sleep 30 + fi + done + restartPolicy: OnFailure + backoffLimit: 60 # Allow retries for up to 30 minutes +--- +apiVersion: generators.external-secrets.io/v1alpha1 +kind: Password +metadata: + name: backstage + namespace: backstage +spec: + length: 36 + digits: 5 + symbols: 5 + symbolCharacters: "/-+" + noUpper: false + allowRepeat: true +--- +apiVersion: external-secrets.io/v1beta1 +kind: ExternalSecret +metadata: + name: backstage-oidc + namespace: backstage +spec: + secretStoreRef: + name: keycloak + kind: ClusterSecretStore + refreshInterval: "0" + target: + name: backstage-env-vars + template: + engineVersion: v2 + data: + BACKSTAGE_FRONTEND_URL: https://cnoe.localtest.me:8443/backstage + POSTGRES_HOST: postgresql.backstage.svc.cluster.local + POSTGRES_PORT: '5432' + POSTGRES_DB: backstage + POSTGRES_USER: backstage + POSTGRES_PASSWORD: "{{.POSTGRES_PASSWORD}}" + ARGO_WORKFLOWS_URL: https://cnoe.localtest.me:8443/argo-workflows + KEYCLOAK_NAME_METADATA: https://cnoe.localtest.me:8443/keycloak/realms/cnoe/.well-known/openid-configuration + KEYCLOAK_CLIENT_SECRET: "{{.BACKSTAGE_CLIENT_SECRET}}" + ARGOCD_AUTH_TOKEN: "argocd.token={{.ARGOCD_SESSION_TOKEN}}" + ARGO_CD_URL: 'https://argocd-server.argocd.svc.cluster.local/api/v1/' + AGENT_FORGE_URL: https://cnoe.localtest.me:8443/ai-platform-engineering + data: + - secretKey: ARGOCD_SESSION_TOKEN + remoteRef: + key: keycloak-clients + property: ARGOCD_SESSION_TOKEN + - secretKey: BACKSTAGE_CLIENT_SECRET + remoteRef: + key: keycloak-clients + property: BACKSTAGE_CLIENT_SECRET + dataFrom: + - sourceRef: + generatorRef: + apiVersion: generators.external-secrets.io/v1alpha1 + kind: Password + name: backstage + rewrite: + - transform: + template: "POSTGRES_PASSWORD" +--- +apiVersion: external-secrets.io/v1beta1 +kind: ExternalSecret +metadata: + name: gitea-credentials + namespace: backstage +spec: + secretStoreRef: + name: gitea + kind: ClusterSecretStore + refreshInterval: "0" + target: + name: gitea-credentials + data: + - secretKey: GITEA_USERNAME + remoteRef: + key: gitea-credential + property: username + - secretKey: GITEA_PASSWORD + remoteRef: + key: gitea-credential + property: password +--- +apiVersion: networking.k8s.io/v1 +kind: Ingress +metadata: + name: backstage + namespace: backstage +spec: + ingressClassName: "nginx" + rules: + - host: localhost + http: + paths: + - path: / + pathType: Prefix + backend: + service: + name: backstage + port: + name: http + - host: cnoe.localtest.me + http: + paths: + - path: / + pathType: Prefix + backend: + service: + name: backstage + port: + name: http +--- +apiVersion: external-secrets.io/v1beta1 +kind: ExternalSecret +metadata: + name: backstage-api-token + namespace: backstage +spec: + secretStoreRef: + name: vault-secret-store + kind: ClusterSecretStore + refreshInterval: "0" + target: + name: backstage-api-token + data: + - secretKey: BACKSTAGE_API_TOKEN + remoteRef: + key: ai-platform-engineering/backstage-secret + property: BACKSTAGE_API_TOKEN diff --git a/caipe/base/backstage/manifests/preinstall-job.yaml b/caipe/base/backstage/manifests/preinstall-job.yaml new file mode 100644 index 00000000..e69de29b diff --git a/ai-platform-engineering/cluster.yaml b/caipe/base/cluster.yaml similarity index 100% rename from ai-platform-engineering/cluster.yaml rename to caipe/base/cluster.yaml diff --git a/ai-platform-engineering/cluster/manifests/coredns-custom.yaml b/caipe/base/cluster/manifests/coredns-custom.yaml similarity index 100% rename from ai-platform-engineering/cluster/manifests/coredns-custom.yaml rename to caipe/base/cluster/manifests/coredns-custom.yaml diff --git a/ai-platform-engineering/vault.yaml b/caipe/base/vault.yaml similarity index 79% rename from ai-platform-engineering/vault.yaml rename to caipe/base/vault.yaml index f6588e2b..3f5b0ccb 100644 --- a/ai-platform-engineering/vault.yaml +++ b/caipe/base/vault.yaml @@ -13,10 +13,10 @@ spec: targetRevision: "0.27.0" helm: valueFiles: - - $values/ai-platform-engineering/vault/values.yaml - - repoURL: https://github.com/cnoe-io/stacks.git - targetRevision: main - path: ai-platform-engineering/vault/manifests + - $values/values.yaml + - repoURL: cnoe://vault + targetRevision: HEAD + path: manifests ref: values destination: server: https://kubernetes.default.svc diff --git a/caipe/base/vault/manifests/argocd-token-cronjob.yaml b/caipe/base/vault/manifests/argocd-token-cronjob.yaml new file mode 100644 index 00000000..b7329874 --- /dev/null +++ b/caipe/base/vault/manifests/argocd-token-cronjob.yaml @@ -0,0 +1,115 @@ +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + name: vault-config-argocd +rules: +- apiGroups: [""] + resources: ["secrets"] + verbs: ["get"] +- apiGroups: ["apps"] + resources: ["deployments"] + verbs: ["get"] +- apiGroups: [""] + resources: ["pods"] + verbs: ["get", "list"] +- apiGroups: [""] + resources: ["pods/exec"] + verbs: ["create"] +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRoleBinding +metadata: + name: vault-config-argocd +subjects: +- kind: ServiceAccount + name: vault-config + namespace: vault +roleRef: + kind: ClusterRole + name: vault-config-argocd + apiGroup: rbac.authorization.k8s.io +--- +apiVersion: batch/v1 +kind: CronJob +metadata: + name: argocd-token-sync + namespace: vault + annotations: + argocd.argoproj.io/sync-wave: "2" +spec: + schedule: "*/10 * * * *" # Every 10 minutes + jobTemplate: + spec: + template: + spec: + serviceAccountName: vault-config + restartPolicy: OnFailure + containers: + - name: argocd-token-sync + image: docker.io/library/ubuntu:22.04 + command: ["/bin/bash", "-c"] + args: + - | + #!/bin/bash + set -ex -o pipefail + + # Install required tools + apt-get update -qq + apt-get install -y -qq curl netcat-openbsd wget jq libcap2-bin gnupg lsb-release + + # Install Vault CLI + wget -O- https://apt.releases.hashicorp.com/gpg | gpg --dearmor > /usr/share/keyrings/hashicorp-archive-keyring.gpg + echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com jammy main" > /etc/apt/sources.list.d/hashicorp.list + apt-get update + apt-get install vault -y + setcap cap_ipc_lock= /usr/bin/vault + + # Install kubectl + curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl" + chmod +x kubectl + mv kubectl /usr/local/bin/ + + # Authenticate to Vault + export VAULT_ADDR="http://vault:8200" + set +x # Disable command echoing for sensitive operations + export VAULT_TOKEN=$(kubectl get secret vault-root-token -n vault -o jsonpath='{.data.token}' | base64 -d) + set -x # Re-enable command echoing + + # Wait for ArgoCD to be ready (max 5 minutes) + echo "Waiting for ArgoCD to be ready..." + timeout=300 + while [ $timeout -gt 0 ]; do + if kubectl get secret -n argocd argocd-initial-admin-secret >/dev/null 2>&1; then + echo "ArgoCD admin secret found" + break + fi + echo "Waiting for ArgoCD admin secret... ($timeout seconds remaining)" + sleep 10 + timeout=$((timeout - 10)) + done + + # Try to create ArgoCD API token + ARGOCD_PASSWORD=$(kubectl get secret -n argocd argocd-initial-admin-secret -o jsonpath='{.data.password}' 2>/dev/null | base64 -d 2>/dev/null || echo "") + + if [[ -n "$ARGOCD_PASSWORD" ]]; then + echo "Creating ArgoCD API token..." + ARGOCD_TOKEN=$(kubectl exec -n argocd deployment/argocd-server -- sh -c " + argocd login localhost:8080 --username admin --password '$ARGOCD_PASSWORD' --plaintext >/dev/null 2>&1 + argocd account generate-token --account developer --id vault-sync-$(date +%s) + " 2>/dev/null || echo "") + + if [[ -n "$ARGOCD_TOKEN" && "$ARGOCD_TOKEN" != *"error"* ]]; then + vault kv put secret/ai-platform-engineering/argocd-secret \ + ARGOCD_TOKEN="$ARGOCD_TOKEN" \ + ARGOCD_API_URL="http://argocd-server.argocd.svc.cluster.local" \ + ARGOCD_VERIFY_SSL="false" + echo "ArgoCD API token created and stored in Vault" + else + echo "Failed to create ArgoCD token" + exit 1 + fi + else + echo "ArgoCD password not found" + exit 1 + fi diff --git a/ai-platform-engineering/vault/manifests/ingress.yaml b/caipe/base/vault/manifests/ingress.yaml similarity index 100% rename from ai-platform-engineering/vault/manifests/ingress.yaml rename to caipe/base/vault/manifests/ingress.yaml diff --git a/ai-platform-engineering/vault/manifests/secret-store.yaml b/caipe/base/vault/manifests/secret-store.yaml similarity index 100% rename from ai-platform-engineering/vault/manifests/secret-store.yaml rename to caipe/base/vault/manifests/secret-store.yaml diff --git a/caipe/base/vault/manifests/vault-config-job.yaml b/caipe/base/vault/manifests/vault-config-job.yaml new file mode 100644 index 00000000..0059c536 --- /dev/null +++ b/caipe/base/vault/manifests/vault-config-job.yaml @@ -0,0 +1,199 @@ +apiVersion: batch/v1 +kind: Job +metadata: + name: vault-config + namespace: vault + annotations: + argocd.argoproj.io/sync-wave: "1" + argocd.argoproj.io/hook-delete-policy: HookSucceeded +spec: + template: + spec: + serviceAccountName: vault-config + restartPolicy: OnFailure + containers: + - name: vault-config + image: docker.io/library/ubuntu:22.04 + command: ["/bin/bash", "-c"] + args: + - | + #!/bin/bash + + # Install required tools FIRST + echo "Installing required tools..." + apt-get update -qq || { echo "ERROR: apt-get update failed"; exit 1; } + apt-get install -y -qq curl netcat-openbsd software-properties-common wget jq libcap2-bin || { echo "ERROR: package installation failed"; exit 1; } + + # Install Vault CLI + echo "Installing Vault CLI..." + wget -O- https://apt.releases.hashicorp.com/gpg | gpg --dearmor > /usr/share/keyrings/hashicorp-archive-keyring.gpg + echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" > /etc/apt/sources.list.d/hashicorp.list + apt-get update + apt-get install vault -y + setcap cap_ipc_lock= /usr/bin/vault + + # Install kubectl + echo "Installing kubectl..." + curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl" || { echo "ERROR: kubectl download failed"; exit 1; } + chmod +x kubectl + mv kubectl /usr/local/bin/ + + echo "Tools installed successfully. Checking connectivity..." + + # Wait for the root token to be available + until kubectl get secret vault-root-token -n vault >/dev/null 2>&1; do + echo "Waiting for vault-root-token secret to be created..." + sleep 5 + done + + echo "Vault root token available. Authenticating..." + # Authenticate to Vault + export VAULT_ADDR="http://vault:8200" + export VAULT_TOKEN=$(kubectl get secret vault-root-token -n vault -o jsonpath='{.data.token}' | base64 -d) + + # Enable KV v2 secrets engine (idempotent) + echo "Enabling KV v2 secrets engine at path 'secret'..." + vault secrets enable -path=secret kv-v2 >/dev/null 2>&1 || echo "Secrets engine 'secret' already enabled." + + # Create agent-specific secrets based on values-secrets.yaml.example structure + echo "Creating agent-specific secrets..." + + # Global secrets (shared across all agents) + echo "Creating global secrets..." + vault kv put secret/ai-platform-engineering/global \ + LLM_PROVIDER="azure-openai" \ + AZURE_OPENAI_API_KEY="" \ + AZURE_OPENAI_ENDPOINT="" \ + AZURE_OPENAI_API_VERSION="" \ + AZURE_OPENAI_DEPLOYMENT="" \ + OPENAI_API_KEY="" \ + OPENAI_ENDPOINT="" \ + OPENAI_MODEL_NAME="" \ + AWS_ACCESS_KEY_ID="" \ + AWS_SECRET_ACCESS_KEY="" \ + AWS_REGION="" \ + AWS_BEDROCK_MODEL_ID="" \ + AWS_BEDROCK_PROVIDER="" >/dev/null + + # ArgoCD agent secrets + echo "Creating argocd-secret..." + vault kv put secret/ai-platform-engineering/argocd-secret \ + ARGOCD_TOKEN="tbd" \ + ARGOCD_API_URL="http://argocd-server.argocd.svc.cluster.local" \ + ARGOCD_VERIFY_SSL="false" >/dev/null + + # AWS agent secrets + echo "Creating aws-secret..." + vault kv put secret/ai-platform-engineering/aws-secret \ + AWS_ACCESS_KEY_ID="" \ + AWS_SECRET_ACCESS_KEY="" \ + AWS_REGION="" >/dev/null + + # Backstage agent secrets + echo "Creating backstage-secret..." + BACKSTAGE_API_TOKEN=$(openssl rand -base64 32 | tr -d "=+/" | cut -c1-32) + vault kv put secret/ai-platform-engineering/backstage-secret \ + BACKSTAGE_API_TOKEN="$BACKSTAGE_API_TOKEN" \ + BACKSTAGE_URL="http://backstage.backstage.svc.cluster.local:7007" >/dev/null + + + # Confluence agent secrets + echo "Creating confluence-secret..." + vault kv put secret/ai-platform-engineering/confluence-secret \ + CONFLUENCE_API_URL="" \ + ATLASSIAN_TOKEN="" \ + ATLASSIAN_EMAIL="" \ + ATLASSIAN_VERIFY_SSL="true" >/dev/null + + # Jira agent secrets + echo "Creating jira-secret..." + vault kv put secret/ai-platform-engineering/jira-secret \ + ATLASSIAN_TOKEN="" \ + ATLASSIAN_EMAIL="" \ + ATLASSIAN_API_URL="" \ + ATLASSIAN_VERIFY_SSL="true" >/dev/null + + # Komodor agent secrets + echo "Creating komodor-secret..." + vault kv put secret/ai-platform-engineering/komodor-secret \ + KOMODOR_TOKEN="" \ + KOMODOR_API_URL="" >/dev/null + + # GitHub agent secrets + echo "Creating github-secret..." + vault kv put secret/ai-platform-engineering/github-secret \ + GITHUB_PERSONAL_ACCESS_TOKEN="" >/dev/null + + # Jira agent secrets + echo "Creating jira-secret..." + vault kv put secret/ai-platform-engineering/jira-secret \ + ATLASSIAN_TOKEN="" \ + ATLASSIAN_EMAIL="" \ + ATLASSIAN_API_URL="" \ + ATLASSIAN_VERIFY_SSL="true" >/dev/null + + # PagerDuty agent secrets + echo "Creating pagerduty-secret..." + vault kv put secret/ai-platform-engineering/pagerduty-secret \ + PAGERDUTY_API_KEY="" \ + PAGERDUTY_API_URL="" >/dev/null + + # Slack agent secrets + echo "Creating slack-secret..." + vault kv put secret/ai-platform-engineering/slack-secret \ + SLACK_BOT_TOKEN="" \ + SLACK_APP_TOKEN="" \ + SLACK_SIGNING_SECRET="" \ + SLACK_CLIENT_SECRET="" \ + SLACK_TEAM_ID="" >/dev/null + + # Splunk agent secrets + echo "Creating splunk-secret..." + vault kv put secret/ai-platform-engineering/splunk-secret \ + SPLUNK_TOKEN="" \ + SPLUNK_API_URL="" >/dev/null + + # Webex agent secrets + echo "Creating webex-secret..." + vault kv put secret/ai-platform-engineering/webex-secret \ + WEBEX_BOT_TOKEN="" \ + WEBEX_WEBHOOK_SECRET="" \ + WEBEX_TOKEN="" >/dev/null + + # KB RAG agent secrets + # This is a fake secret as for now MILVUS does not actually utilise a secret + echo "Creating kb-rag-secret..." + vault kv put secret/ai-platform-engineering/kb-rag-secret \ + MILVUS_SECRET="foo" >/dev/null + + echo "Vault configuration complete." +--- +apiVersion: v1 +kind: ServiceAccount +metadata: + name: vault-config + namespace: vault +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: Role +metadata: + name: vault-config + namespace: vault +rules: +- apiGroups: [""] + resources: ["secrets"] + verbs: ["get"] +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: RoleBinding +metadata: + name: vault-config + namespace: vault +subjects: +- kind: ServiceAccount + name: vault-config + namespace: vault +roleRef: + kind: Role + name: vault-config + apiGroup: rbac.authorization.k8s.io diff --git a/ai-platform-engineering/vault/manifests/vault-init-job.yaml b/caipe/base/vault/manifests/vault-init-job.yaml similarity index 100% rename from ai-platform-engineering/vault/manifests/vault-init-job.yaml rename to caipe/base/vault/manifests/vault-init-job.yaml diff --git a/ai-platform-engineering/vault/manifests/vault-unsealer.yaml b/caipe/base/vault/manifests/vault-unsealer.yaml similarity index 100% rename from ai-platform-engineering/vault/manifests/vault-unsealer.yaml rename to caipe/base/vault/manifests/vault-unsealer.yaml diff --git a/ai-platform-engineering/vault/values.yaml b/caipe/base/vault/values.yaml similarity index 100% rename from ai-platform-engineering/vault/values.yaml rename to caipe/base/vault/values.yaml diff --git a/caipe/caipe-complete-agentgateway/agentgateway.yaml b/caipe/caipe-complete-agentgateway/agentgateway.yaml new file mode 100644 index 00000000..b5fbf64e --- /dev/null +++ b/caipe/caipe-complete-agentgateway/agentgateway.yaml @@ -0,0 +1,23 @@ +apiVersion: argoproj.io/v1alpha1 +kind: Application +metadata: + name: agentgateway + namespace: argocd + finalizers: + - resources-finalizer.argocd.argoproj.io +spec: + project: default + source: + repoURL: https://github.com/sriaradhyula/stacks.git + targetRevision: main + path: agentgateway/agentgateway/manifests + destination: + server: https://kubernetes.default.svc + namespace: agentgateway + syncPolicy: + automated: + prune: true + selfHeal: true + syncOptions: + - CreateNamespace=true + diff --git a/caipe/caipe-complete-agentgateway/agentgateway/manifests/configmap.yaml b/caipe/caipe-complete-agentgateway/agentgateway/manifests/configmap.yaml new file mode 100644 index 00000000..a2e92c24 --- /dev/null +++ b/caipe/caipe-complete-agentgateway/agentgateway/manifests/configmap.yaml @@ -0,0 +1,31 @@ +apiVersion: v1 +kind: ConfigMap +metadata: + name: agentgateway-config + namespace: agentgateway + annotations: + argocd.argoproj.io/sync-wave: "0" +data: + config.yaml: | + adminAddr: + SocketAddr: "0.0.0.0:15000" + binds: + - port: 3000 + listeners: + - routes: + - policies: + cors: + allowOrigins: + - "*" + allowHeaders: + - mcp-protocol-version + - content-type + - cache-control + backends: + - mcp: + targets: + - name: everything + stdio: + cmd: npx + args: ["@modelcontextprotocol/server-everything"] + diff --git a/caipe/caipe-complete-agentgateway/agentgateway/manifests/deployment.yaml b/caipe/caipe-complete-agentgateway/agentgateway/manifests/deployment.yaml new file mode 100644 index 00000000..6e4e7ca4 --- /dev/null +++ b/caipe/caipe-complete-agentgateway/agentgateway/manifests/deployment.yaml @@ -0,0 +1,56 @@ +apiVersion: apps/v1 +kind: Deployment +metadata: + name: agentgateway + namespace: agentgateway + labels: + app: agentgateway + annotations: + argocd.argoproj.io/sync-wave: "1" +spec: + replicas: 2 + selector: + matchLabels: + app: agentgateway + template: + metadata: + labels: + app: agentgateway + spec: + containers: + - name: agentgateway + image: ghcr.io/agentgateway/agentgateway:0.8.3 + ports: + - containerPort: 15000 + name: ui + - containerPort: 3000 + name: proxy + args: ["-f", "/etc/agentgateway/config.yaml"] + env: + - name: ADMIN_ADDR + value: "0.0.0.0:15000" + securityContext: + allowPrivilegeEscalation: false + capabilities: + drop: + - ALL + readOnlyRootFilesystem: true + runAsNonRoot: true + runAsUser: 65532 + volumeMounts: + - name: config + mountPath: /etc/agentgateway + readOnly: true + resources: + requests: + memory: "128Mi" + cpu: "100m" + limits: + memory: "512Mi" + cpu: "500m" + ephemeral-storage: "1Gi" + volumes: + - name: config + configMap: + name: agentgateway-config + diff --git a/caipe/caipe-complete-agentgateway/agentgateway/manifests/ingress.yaml b/caipe/caipe-complete-agentgateway/agentgateway/manifests/ingress.yaml new file mode 100644 index 00000000..ceb3d883 --- /dev/null +++ b/caipe/caipe-complete-agentgateway/agentgateway/manifests/ingress.yaml @@ -0,0 +1,32 @@ +apiVersion: networking.k8s.io/v1 +kind: Ingress +metadata: + name: agentgateway-ingress + namespace: agentgateway + annotations: + argocd.argoproj.io/sync-wave: "0" + nginx.ingress.kubernetes.io/backend-protocol: HTTP + nginx.ingress.kubernetes.io/use-regex: "true" +spec: + ingressClassName: "nginx" + rules: + - host: localhost + http: + paths: + - path: /agentgateway + pathType: Prefix + backend: + service: + name: agentgateway + port: + number: 15000 + - host: cnoe.localtest.me + http: + paths: + - path: /agentgateway + pathType: Prefix + backend: + service: + name: agentgateway + port: + number: 15000 diff --git a/caipe/caipe-complete-agentgateway/agentgateway/manifests/service.yaml b/caipe/caipe-complete-agentgateway/agentgateway/manifests/service.yaml new file mode 100644 index 00000000..e393e80c --- /dev/null +++ b/caipe/caipe-complete-agentgateway/agentgateway/manifests/service.yaml @@ -0,0 +1,23 @@ +apiVersion: v1 +kind: Service +metadata: + name: agentgateway + namespace: agentgateway + labels: + app: agentgateway + annotations: + argocd.argoproj.io/sync-wave: "2" +spec: + type: ClusterIP + ports: + - port: 15000 + targetPort: 15000 + protocol: TCP + name: ui + - port: 3000 + targetPort: 3000 + protocol: TCP + name: proxy + selector: + app: agentgateway + diff --git a/caipe/caipe-complete-agentgateway/kgateway/manifests/kgateway-agentgateway-ingress.yaml b/caipe/caipe-complete-agentgateway/kgateway/manifests/kgateway-agentgateway-ingress.yaml new file mode 100644 index 00000000..74ab2728 --- /dev/null +++ b/caipe/caipe-complete-agentgateway/kgateway/manifests/kgateway-agentgateway-ingress.yaml @@ -0,0 +1,19 @@ +apiVersion: networking.k8s.io/v1 +kind: Ingress +metadata: + name: kgateway-agentgateway + namespace: kgateway-system + annotations: + nginx.ingress.kubernetes.io/rewrite-target: /$2 +spec: + rules: + - host: cnoe.localtest.me + http: + paths: + - path: /kgateway(/|$)(.*) + pathType: ImplementationSpecific + backend: + service: + name: agentgateway + port: + number: 8080 diff --git a/caipe/caipe-complete-slim-agentgateway/agentgateway.yaml b/caipe/caipe-complete-slim-agentgateway/agentgateway.yaml new file mode 100644 index 00000000..b5fbf64e --- /dev/null +++ b/caipe/caipe-complete-slim-agentgateway/agentgateway.yaml @@ -0,0 +1,23 @@ +apiVersion: argoproj.io/v1alpha1 +kind: Application +metadata: + name: agentgateway + namespace: argocd + finalizers: + - resources-finalizer.argocd.argoproj.io +spec: + project: default + source: + repoURL: https://github.com/sriaradhyula/stacks.git + targetRevision: main + path: agentgateway/agentgateway/manifests + destination: + server: https://kubernetes.default.svc + namespace: agentgateway + syncPolicy: + automated: + prune: true + selfHeal: true + syncOptions: + - CreateNamespace=true + diff --git a/caipe/caipe-complete-slim-agentgateway/agentgateway/manifests/configmap.yaml b/caipe/caipe-complete-slim-agentgateway/agentgateway/manifests/configmap.yaml new file mode 100644 index 00000000..a2e92c24 --- /dev/null +++ b/caipe/caipe-complete-slim-agentgateway/agentgateway/manifests/configmap.yaml @@ -0,0 +1,31 @@ +apiVersion: v1 +kind: ConfigMap +metadata: + name: agentgateway-config + namespace: agentgateway + annotations: + argocd.argoproj.io/sync-wave: "0" +data: + config.yaml: | + adminAddr: + SocketAddr: "0.0.0.0:15000" + binds: + - port: 3000 + listeners: + - routes: + - policies: + cors: + allowOrigins: + - "*" + allowHeaders: + - mcp-protocol-version + - content-type + - cache-control + backends: + - mcp: + targets: + - name: everything + stdio: + cmd: npx + args: ["@modelcontextprotocol/server-everything"] + diff --git a/caipe/caipe-complete-slim-agentgateway/agentgateway/manifests/deployment.yaml b/caipe/caipe-complete-slim-agentgateway/agentgateway/manifests/deployment.yaml new file mode 100644 index 00000000..6e4e7ca4 --- /dev/null +++ b/caipe/caipe-complete-slim-agentgateway/agentgateway/manifests/deployment.yaml @@ -0,0 +1,56 @@ +apiVersion: apps/v1 +kind: Deployment +metadata: + name: agentgateway + namespace: agentgateway + labels: + app: agentgateway + annotations: + argocd.argoproj.io/sync-wave: "1" +spec: + replicas: 2 + selector: + matchLabels: + app: agentgateway + template: + metadata: + labels: + app: agentgateway + spec: + containers: + - name: agentgateway + image: ghcr.io/agentgateway/agentgateway:0.8.3 + ports: + - containerPort: 15000 + name: ui + - containerPort: 3000 + name: proxy + args: ["-f", "/etc/agentgateway/config.yaml"] + env: + - name: ADMIN_ADDR + value: "0.0.0.0:15000" + securityContext: + allowPrivilegeEscalation: false + capabilities: + drop: + - ALL + readOnlyRootFilesystem: true + runAsNonRoot: true + runAsUser: 65532 + volumeMounts: + - name: config + mountPath: /etc/agentgateway + readOnly: true + resources: + requests: + memory: "128Mi" + cpu: "100m" + limits: + memory: "512Mi" + cpu: "500m" + ephemeral-storage: "1Gi" + volumes: + - name: config + configMap: + name: agentgateway-config + diff --git a/caipe/caipe-complete-slim-agentgateway/agentgateway/manifests/ingress.yaml b/caipe/caipe-complete-slim-agentgateway/agentgateway/manifests/ingress.yaml new file mode 100644 index 00000000..ceb3d883 --- /dev/null +++ b/caipe/caipe-complete-slim-agentgateway/agentgateway/manifests/ingress.yaml @@ -0,0 +1,32 @@ +apiVersion: networking.k8s.io/v1 +kind: Ingress +metadata: + name: agentgateway-ingress + namespace: agentgateway + annotations: + argocd.argoproj.io/sync-wave: "0" + nginx.ingress.kubernetes.io/backend-protocol: HTTP + nginx.ingress.kubernetes.io/use-regex: "true" +spec: + ingressClassName: "nginx" + rules: + - host: localhost + http: + paths: + - path: /agentgateway + pathType: Prefix + backend: + service: + name: agentgateway + port: + number: 15000 + - host: cnoe.localtest.me + http: + paths: + - path: /agentgateway + pathType: Prefix + backend: + service: + name: agentgateway + port: + number: 15000 diff --git a/caipe/caipe-complete-slim-agentgateway/agentgateway/manifests/service.yaml b/caipe/caipe-complete-slim-agentgateway/agentgateway/manifests/service.yaml new file mode 100644 index 00000000..e393e80c --- /dev/null +++ b/caipe/caipe-complete-slim-agentgateway/agentgateway/manifests/service.yaml @@ -0,0 +1,23 @@ +apiVersion: v1 +kind: Service +metadata: + name: agentgateway + namespace: agentgateway + labels: + app: agentgateway + annotations: + argocd.argoproj.io/sync-wave: "2" +spec: + type: ClusterIP + ports: + - port: 15000 + targetPort: 15000 + protocol: TCP + name: ui + - port: 3000 + targetPort: 3000 + protocol: TCP + name: proxy + selector: + app: agentgateway + diff --git a/caipe/caipe-complete-slim-agentgateway/kgateway/manifests/kgateway-agentgateway-ingress.yaml b/caipe/caipe-complete-slim-agentgateway/kgateway/manifests/kgateway-agentgateway-ingress.yaml new file mode 100644 index 00000000..74ab2728 --- /dev/null +++ b/caipe/caipe-complete-slim-agentgateway/kgateway/manifests/kgateway-agentgateway-ingress.yaml @@ -0,0 +1,19 @@ +apiVersion: networking.k8s.io/v1 +kind: Ingress +metadata: + name: kgateway-agentgateway + namespace: kgateway-system + annotations: + nginx.ingress.kubernetes.io/rewrite-target: /$2 +spec: + rules: + - host: cnoe.localtest.me + http: + paths: + - path: /kgateway(/|$)(.*) + pathType: ImplementationSpecific + backend: + service: + name: agentgateway + port: + number: 8080 diff --git a/caipe/catalog-info.yaml b/caipe/catalog-info.yaml new file mode 100644 index 00000000..82a26e4a --- /dev/null +++ b/caipe/catalog-info.yaml @@ -0,0 +1,131 @@ +apiVersion: backstage.io/v1alpha1 +kind: System +metadata: + name: caipe-platform + description: Cloud AI Platform Engineering - Complete AI-powered platform engineering solution + labels: + platform: caipe + annotations: + backstage.io/managed-by-location: url:https://github.com/sriaradhyula/stacks/tree/main/caipe/catalog-info.yaml +spec: + owner: platform-team + domain: platform-engineering +--- +apiVersion: backstage.io/v1alpha1 +kind: Component +metadata: + name: ai-platform-engineering + description: AI Platform Engineering - Main CAIPE stack + labels: + platform: caipe + environment: production + annotations: + backstage.io/managed-by-location: url:https://github.com/sriaradhyula/stacks/tree/main/caipe + argocd/app-name: ai-platform-engineering +spec: + type: service + lifecycle: production + owner: platform-team + system: caipe-platform +--- +apiVersion: backstage.io/v1alpha1 +kind: Component +metadata: + name: github-agent + description: GitHub Agent - Repository management + labels: + platform: caipe + environment: production + annotations: + backstage.io/managed-by-location: url:https://github.com/sriaradhyula/stacks/tree/main/caipe + argocd/app-name: github-agent +spec: + type: service + lifecycle: production + owner: platform-team + system: caipe-platform +--- +apiVersion: backstage.io/v1alpha1 +kind: Component +metadata: + name: jira-agent + description: Jira Agent - Issue tracking integration + labels: + platform: caipe + environment: production + annotations: + backstage.io/managed-by-location: url:https://github.com/sriaradhyula/stacks/tree/main/caipe + argocd/app-name: jira-agent +spec: + type: service + lifecycle: production + owner: platform-team + system: caipe-platform +--- +apiVersion: backstage.io/v1alpha1 +kind: Component +metadata: + name: slack-agent + description: Slack Agent - Team communication + labels: + platform: caipe + environment: production + annotations: + backstage.io/managed-by-location: url:https://github.com/sriaradhyula/stacks/tree/main/caipe + argocd/app-name: slack-agent +spec: + type: service + lifecycle: production + owner: platform-team + system: caipe-platform +--- +apiVersion: backstage.io/v1alpha1 +kind: Component +metadata: + name: aws-agent + description: AWS Agent - Cloud resource management + labels: + platform: caipe + environment: production + annotations: + backstage.io/managed-by-location: url:https://github.com/sriaradhyula/stacks/tree/main/caipe + argocd/app-name: aws-agent +spec: + type: service + lifecycle: production + owner: platform-team + system: caipe-platform +--- +apiVersion: backstage.io/v1alpha1 +kind: Component +metadata: + name: argocd-agent + description: ArgoCD Agent - GitOps deployment + labels: + platform: caipe + environment: production + annotations: + backstage.io/managed-by-location: url:https://github.com/sriaradhyula/stacks/tree/main/caipe + argocd/app-name: argocd-agent +spec: + type: service + lifecycle: production + owner: platform-team + system: caipe-platform +--- +apiVersion: backstage.io/v1alpha1 +kind: Component +metadata: + name: backstage-agent + description: Backstage Agent - Developer portal integration + labels: + platform: caipe + environment: production + annotations: + backstage.io/managed-by-location: url:https://github.com/sriaradhyula/stacks/tree/main/caipe + argocd/app-name: backstage-agent +spec: + type: service + lifecycle: production + owner: platform-team + system: caipe-platform diff --git a/caipe/complete-slim/ai-platform-engineering.yaml b/caipe/complete-slim/ai-platform-engineering.yaml new file mode 100644 index 00000000..260c8e84 --- /dev/null +++ b/caipe/complete-slim/ai-platform-engineering.yaml @@ -0,0 +1,44 @@ +apiVersion: argoproj.io/v1alpha1 +kind: Application +metadata: + name: ai-platform-engineering + namespace: argocd + finalizers: + - resources-finalizer.argocd.argoproj.io +spec: + project: default + sources: + # Main chart from GHCR + - chart: ai-platform-engineering + repoURL: ghcr.io/cnoe-io/helm-charts + targetRevision: 0.2.5 + helm: + parameters: + - name: tags.complete + value: "true" + - name: global.slim.enabled # TODO: maybe move this to tag and we make cm called slim-env + value: "true" + valueFiles: + - $values/values.yaml + # Values files from Git repository (your custom branch) + - repoURL: cnoe://../base/ai-platform-engineering + path: manifests + ref: values + destination: + server: https://kubernetes.default.svc + namespace: ai-platform-engineering + syncPolicy: + automated: + prune: true + selfHeal: true + syncOptions: + - CreateNamespace=true + - PrunePropagationPolicy=foreground + - PruneLast=true + retry: + limit: 5 + backoff: + duration: 5s + factor: 2 + maxDuration: 3m + revisionHistoryLimit: 10 diff --git a/caipe/complete/ai-platform-engineering.yaml b/caipe/complete/ai-platform-engineering.yaml new file mode 100644 index 00000000..a77a16d6 --- /dev/null +++ b/caipe/complete/ai-platform-engineering.yaml @@ -0,0 +1,42 @@ +apiVersion: argoproj.io/v1alpha1 +kind: Application +metadata: + name: ai-platform-engineering + namespace: argocd + finalizers: + - resources-finalizer.argocd.argoproj.io +spec: + project: default + sources: + # Main chart from GHCR + - chart: ai-platform-engineering + repoURL: ghcr.io/cnoe-io/helm-charts + targetRevision: 0.3.0 + helm: + parameters: + - name: tags.complete + value: "true" + valueFiles: + - $values/values.yaml + # Values files from Git repository (your custom branch) + - repoURL: cnoe://../base/ai-platform-engineering + path: manifests + ref: values + destination: + server: https://kubernetes.default.svc + namespace: ai-platform-engineering + syncPolicy: + automated: + prune: true + selfHeal: true + syncOptions: + - CreateNamespace=true + - PrunePropagationPolicy=foreground + - PruneLast=true + retry: + limit: 5 + backoff: + duration: 5s + factor: 2 + maxDuration: 3m + revisionHistoryLimit: 10 diff --git a/caipe/docs/argocd-api.md b/caipe/docs/argocd-api.md new file mode 100644 index 00000000..a8f28685 --- /dev/null +++ b/caipe/docs/argocd-api.md @@ -0,0 +1,185 @@ +# ArgoCD API Integration + +This document describes the ArgoCD API integration for the CAIPE platform, including authentication, endpoints, and usage examples. + +## Overview + +ArgoCD provides a REST API for managing GitOps deployments programmatically. The CAIPE platform includes automated token management and API access configuration. + +## API Configuration + +### Endpoints +- **Internal API URL**: `http://argocd-server.argocd.svc.cluster.local` +- **External Web UI**: `https://cnoe.localtest.me:8443/argocd` +- **API Port**: `80` (HTTP internal), `443` (HTTPS external) + +### Authentication + +The platform automatically manages ArgoCD API tokens through a CronJob that: +1. Generates API tokens for the `developer` account +2. Stores tokens securely in Vault at `secret/ai-platform-engineering/argocd-secret` +3. Rotates tokens every 10 minutes for security + +#### Retrieving API Token + +```bash +# From Vault (requires vault CLI and access) +vault kv get -field=ARGOCD_TOKEN secret/ai-platform-engineering/argocd-secret + +# From Kubernetes secret (for agents) +kubectl get secret agent-argocd-secret -n ai-platform-engineering -o jsonpath='{.data.ARGOCD_TOKEN}' | base64 -d +``` + +## Current Applications + +The ArgoCD instance manages the following applications: + +| Application | Status | Health | Description | +|-------------|--------|--------|-------------| +| ai-platform-engineering | Synced | Healthy | AI platform engineering agents | +| argo-workflows | OutOfSync | Missing | Workflow orchestration | +| argocd | Synced | Healthy | ArgoCD itself | +| backstage | Synced | Healthy | Developer portal | +| backstage-templates | Synced | Healthy | Backstage templates | +| cluster-config | Synced | Healthy | Cluster configuration | +| external-secrets | Synced | Healthy | Secret synchronization | +| gitea | Synced | Healthy | Git repository server | +| keycloak | Synced | Healthy | Identity and access management | +| metric-server | Synced | Healthy | Metrics collection | +| nginx | Synced | Healthy | Ingress controller | +| spark-operator | Synced | Healthy | Apache Spark operator | +| vault | Synced | Healthy | Secret management | + +## API Usage Examples + +### Authentication Header + +```bash +# Set the API token +ARGOCD_TOKEN="your-api-token-here" +ARGOCD_API_URL="http://argocd-server.argocd.svc.cluster.local" + +# Use in API calls +curl -H "Authorization: Bearer $ARGOCD_TOKEN" \ + -H "Content-Type: application/json" \ + "$ARGOCD_API_URL/api/v1/applications" +``` + +### Common API Endpoints + +#### List Applications +```bash +GET /api/v1/applications +``` + +#### Get Application Details +```bash +GET /api/v1/applications/{app-name} +``` + +#### Sync Application +```bash +POST /api/v1/applications/{app-name}/sync +``` + +#### Get Application Resources +```bash +GET /api/v1/applications/{app-name}/resource-tree +``` + +### Python Example + +```python +import requests +import os + +# Get token from environment or Kubernetes secret +argocd_token = os.getenv('ARGOCD_TOKEN') +argocd_url = "http://argocd-server.argocd.svc.cluster.local" + +headers = { + 'Authorization': f'Bearer {argocd_token}', + 'Content-Type': 'application/json' +} + +# List all applications +response = requests.get(f"{argocd_url}/api/v1/applications", headers=headers) +applications = response.json() + +for app in applications['items']: + print(f"App: {app['metadata']['name']}, Status: {app['status']['sync']['status']}") +``` + +### Shell Script Example + +```bash +#!/bin/bash + +# Get ArgoCD token from Vault +ARGOCD_TOKEN=$(vault kv get -field=ARGOCD_TOKEN secret/ai-platform-engineering/argocd-secret) +ARGOCD_API_URL="http://argocd-server.argocd.svc.cluster.local" + +# Function to call ArgoCD API +argocd_api() { + local endpoint="$1" + curl -s -H "Authorization: Bearer $ARGOCD_TOKEN" \ + -H "Content-Type: application/json" \ + "$ARGOCD_API_URL$endpoint" +} + +# List applications +echo "Applications:" +argocd_api "/api/v1/applications" | jq -r '.items[].metadata.name' + +# Get specific application status +echo "Vault application status:" +argocd_api "/api/v1/applications/vault" | jq -r '.status.sync.status' +``` + +## Security Considerations + +1. **Token Rotation**: API tokens are automatically rotated every 10 minutes +2. **Internal Access**: API is accessible only within the cluster by default +3. **RBAC**: The `developer` account has limited permissions for safety +4. **Vault Storage**: Tokens are stored encrypted in Vault +5. **No SSL Verification**: Internal API uses HTTP (SSL verification disabled) + +## Troubleshooting + +### Token Issues +```bash +# Check if token is valid +curl -H "Authorization: Bearer $ARGOCD_TOKEN" \ + "$ARGOCD_API_URL/api/v1/account" + +# Check token generation logs +kubectl logs -n vault job/argocd-token-sync-$(date +%Y%m%d%H%M | cut -c1-10) +``` + +### API Connectivity +```bash +# Test internal connectivity +kubectl run test-pod --rm -i --tty --image=curlimages/curl -- \ + curl -H "Authorization: Bearer $ARGOCD_TOKEN" \ + http://argocd-server.argocd.svc.cluster.local/api/v1/version +``` + +### Application Sync Issues +```bash +# Force sync an application +curl -X POST -H "Authorization: Bearer $ARGOCD_TOKEN" \ + -H "Content-Type: application/json" \ + "$ARGOCD_API_URL/api/v1/applications/vault/sync" \ + -d '{"prune": false, "dryRun": false}' +``` + +## Integration with AI Agents + +The ArgoCD API is integrated with AI platform engineering agents for: + +- **Deployment Monitoring**: Track application sync status and health +- **Automated Remediation**: Trigger syncs when applications drift +- **Resource Management**: Query application resources and configurations +- **GitOps Workflows**: Coordinate with Git repositories for deployments + +Agents can access the API using the automatically managed tokens stored in Vault and synchronized to Kubernetes secrets. diff --git a/caipe/env.example b/caipe/env.example new file mode 100644 index 00000000..52a44db2 --- /dev/null +++ b/caipe/env.example @@ -0,0 +1,58 @@ +# Example .env file for setup-secrets.sh +# Copy this to ~/.env or .env in your current directory + +# LLM Provider Configuration +LLM_PROVIDER=azure-openai +AZURE_OPENAI_API_KEY=your-azure-openai-api-key-here +AZURE_OPENAI_ENDPOINT=https://your-resource.openai.azure.com/ +AZURE_OPENAI_API_VERSION=2024-02-15-preview +AZURE_OPENAI_DEPLOYMENT=your-deployment-name + +# GitHub Agent +GITHUB_PERSONAL_ACCESS_TOKEN=ghp_your-github-token-here + +# Jira Agent +ATLASSIAN_TOKEN=your-atlassian-token-here +ATLASSIAN_API_URL=https://your-company.atlassian.net +ATLASSIAN_EMAIL=your-email@company.com +ATLASSIAN_VERIFY_SSL=true + +# Slack Agent +SLACK_BOT_TOKEN=xoxb-your-slack-bot-token-here +SLACK_TOKEN=xoxp-your-slack-token-here +SLACK_APP_TOKEN=xapp-your-slack-app-token-here +SLACK_SIGNING_SECRET=your-slack-signing-secret-here +SLACK_CLIENT_SECRET=your-slack-client-secret-here +SLACK_TEAM_ID=your-slack-team-id-here + +# AWS Agent +AWS_ACCESS_KEY_ID=your-aws-access-key-id-here +AWS_SECRET_ACCESS_KEY=your-aws-secret-access-key-here +AWS_REGION=us-east-1 + +# ArgoCD Agent (usually auto-populated from k8s secrets) +ARGOCD_TOKEN=your-argocd-token-here +ARGOCD_API_URL=http://argocd-server.argocd.svc.cluster.local +ARGOCD_VERIFY_SSL=false + +# Backstage Agent (usually auto-populated from k8s secrets) +BACKSTAGE_API_TOKEN=your-backstage-api-token-here +BACKSTAGE_URL=http://backstage.backstage.svc.cluster.local:7007 + +# PagerDuty Agent +PAGERDUTY_API_KEY=your-pagerduty-api-key-here +PAGERDUTY_API_URL=https://api.pagerduty.com + +# Confluence Agent +CONFLUENCE_API_URL=https://your-company.atlassian.net/wiki + +# Splunk Agent +SPLUNK_TOKEN=your-splunk-token-here +SPLUNK_API_URL=https://splunk.your-company.com + +# Webex Agent +WEBEX_TOKEN=your-webex-token-here + +# Komodor Agent +KOMODOR_TOKEN=your-komodor-token-here +KOMODOR_API_URL=https://api.komodor.com diff --git a/caipe/scripts/populate-backstage-catalog.sh b/caipe/scripts/populate-backstage-catalog.sh new file mode 100755 index 00000000..72cc8f12 --- /dev/null +++ b/caipe/scripts/populate-backstage-catalog.sh @@ -0,0 +1,117 @@ +#!/bin/bash + +set -e + +log() { + echo "[$(date '+%Y-%m-%d %H:%M:%S')] $1" +} + +warn() { + echo "โš ๏ธ $1" +} + +log "๐Ÿ”ง Registering CAIPE catalog location in Backstage" + +# Get Backstage API token from Kubernetes secret +log "๐Ÿ”‘ Retrieving Backstage API token..." +BACKSTAGE_TOKEN=$(kubectl get secret backstage-api-token -n backstage -o jsonpath='{.data.BACKSTAGE_API_TOKEN}' | base64 -d) + +if [[ -z "$BACKSTAGE_TOKEN" ]]; then + warn "No Backstage API token found" + exit 1 +else + log "โœ… Backstage API token retrieved (${#BACKSTAGE_TOKEN} chars)" +fi + +# Start Backstage port forward +log "๐Ÿ”— Starting Backstage port forward..." +kubectl port-forward -n backstage svc/backstage 7007:7007 & +BACKSTAGE_PID=$! +sleep 3 + +# Function to call Backstage API +backstage_api() { + local method="$1" + local endpoint="$2" + local data="$3" + + if [[ -n "$data" ]]; then + curl -s -X "$method" \ + -H "Authorization: Bearer $BACKSTAGE_TOKEN" \ + -H "Content-Type: application/json" \ + -d "$data" \ + "http://localhost:7007$endpoint" + else + curl -s -X "$method" \ + -H "Authorization: Bearer $BACKSTAGE_TOKEN" \ + -H "Content-Type: application/json" \ + "http://localhost:7007$endpoint" + fi +} + +# Register GitHub catalog-info.yaml as a location +log "๐Ÿ“ Registering CAIPE catalog location..." + +location_data='{ + "type": "url", + "target": "https://raw.githubusercontent.com/sriaradhyula/stacks/main/caipe/catalog-info.yaml" +}' + +response=$(backstage_api "POST" "/api/catalog/locations" "$location_data") + +if echo "$response" | jq -e '.location.id' >/dev/null 2>&1; then + location_id=$(echo "$response" | jq -r '.location.id') + log "โœ… Successfully registered CAIPE catalog location: $location_id" +else + log "โš ๏ธ Location registration response: $response" +fi + +# Trigger catalog refresh to process the new location +log "๐Ÿ”„ Triggering catalog refresh..." +refresh_response=$(backstage_api "POST" "/api/catalog/refresh") +log "โœ… Catalog refresh triggered" + +# Wait a moment for processing +sleep 5 + +# Check current locations +log "๐Ÿ“‹ Checking registered locations..." +locations=$(backstage_api "GET" "/api/catalog/locations") +github_locations=$(echo "$locations" | jq -r '.[] | select(.target | contains("github.com/sriaradhyula/stacks")) | .target' 2>/dev/null || echo "") + +if [[ -n "$github_locations" ]]; then + log "โœ… Found GitHub CAIPE location:" + echo "$github_locations" +else + log "โš ๏ธ GitHub CAIPE location not found in registered locations" +fi + +# List all entities to see if CAIPE components are now available +log "๐Ÿ“Š Checking catalog entities..." +all_entities=$(backstage_api "GET" "/api/catalog/entities") +total_entities=$(echo "$all_entities" | jq '. | length' 2>/dev/null || echo "0") +log "๐Ÿ“Š Total entities in catalog: $total_entities" + +# Look for CAIPE entities +caipe_entities=$(echo "$all_entities" | jq -r '.[] | select(.metadata.labels.platform == "caipe") | "\(.kind): \(.metadata.name) - \(.metadata.description // "No description")"' 2>/dev/null || echo "") + +if [[ -n "$caipe_entities" ]]; then + log "โœ… Found CAIPE entities:" + echo "$caipe_entities" +else + log "โš ๏ธ CAIPE entities not yet visible (may take a few minutes to process)" + # Show any entities that might be related + echo "$all_entities" | jq -r '.[] | select(.metadata.name | contains("caipe") or contains("agent")) | "\(.kind): \(.metadata.name)"' 2>/dev/null | head -5 || echo "No related entities found" +fi + +# Cleanup +kill $BACKSTAGE_PID 2>/dev/null || true + +echo "" +echo "โœ… CAIPE CATALOG REGISTRATION COMPLETE:" +echo "- Location: https://raw.githubusercontent.com/sriaradhyula/stacks/main/caipe/catalog-info.yaml" +echo "- Entities will appear in UI within a few minutes" +echo "- Web UI: https://cnoe.localtest.me:8443/backstage/catalog" +echo "- Filter by: platform=caipe" + +log "๐ŸŽ‰ Backstage catalog location registration complete!" diff --git a/caipe/scripts/recreate-idpbuilder.sh b/caipe/scripts/recreate-idpbuilder.sh new file mode 100755 index 00000000..668bd627 --- /dev/null +++ b/caipe/scripts/recreate-idpbuilder.sh @@ -0,0 +1,25 @@ +#!/bin/bash + +set -e + +echo "๐Ÿ”„ Recreating idpbuilder with latest GitHub changes..." + +# Destroy existing cluster +echo "๐Ÿ—‘๏ธ Destroying existing cluster..." +kind delete cluster --name localdev || echo "Cluster doesn't exist or already deleted" + +# Wait a moment for cleanup +sleep 5 + +# Recreate with latest packages +echo "๐Ÿš€ Creating new cluster with latest packages..." +idpbuilder create \ + --use-path-routing \ + --package https://github.com/cnoe-io/stacks//ref-implementation \ + --package https://github.com/sriaradhyula/stacks//caipe/base \ + --package https://github.com/sriaradhyula/stacks//caipe/complete + +echo "โœ… Cluster recreated successfully!" +echo "๐ŸŒ ArgoCD: https://cnoe.localtest.me:8443/argocd" +echo "๐Ÿ  Backstage: https://cnoe.localtest.me:8443/backstage" +echo "๐Ÿค– AI Platform: https://cnoe.localtest.me:8443/ai-platform-engineering" diff --git a/caipe/scripts/refresh-secrets.sh b/caipe/scripts/refresh-secrets.sh new file mode 100755 index 00000000..52fbecfd --- /dev/null +++ b/caipe/scripts/refresh-secrets.sh @@ -0,0 +1,29 @@ +#!/bin/bash + +set -euo pipefail + +echo "๐Ÿš€ Starting AI Platform Engineering cleanup process..." +echo "" + +echo "๐Ÿ”‘ Deleting all secrets in ai-platform-engineering namespace..." +kubectl delete secret --all -n ai-platform-engineering + +echo "" +echo "โฑ๏ธ Waiting 5 second for cleanup to complete..." +sleep 5 + +echo "" +echo "๐Ÿ“ Command executed: kubectl delete secret --all -n ai-platform-engineering" +echo "" + +echo "๐Ÿ—‘๏ธ Deleting all pods in ai-platform-engineering namespace..." +kubectl delete pod --all -n ai-platform-engineering + +echo "" +echo "๐Ÿ“Š Current pods in ai-platform-engineering namespace:" +echo "==================================================" +kubectl get pods -n ai-platform-engineering + +echo "" +echo "โœ… Cleanup process completed successfully!" +echo "๐ŸŽฏ All secrets and pods have been refreshed in the ai-platform-engineering namespace" diff --git a/caipe/scripts/setup-agent-secrets.sh b/caipe/scripts/setup-agent-secrets.sh new file mode 100755 index 00000000..6ed15d07 --- /dev/null +++ b/caipe/scripts/setup-agent-secrets.sh @@ -0,0 +1,742 @@ +#!/bin/bash + +set -e + +log() { + echo "[$(date '+%Y-%m-%d %H:%M:%S')] $1" +} + +# Check dependencies +for cmd in kubectl vault jq; do + if ! command -v $cmd &> /dev/null; then + log "โŒ $cmd is required but not installed" + exit 1 + fi +done + +# Parse command line arguments +OVERRIDE_ALL=false +ENV_FILE="" +while [[ $# -gt 0 ]]; do + case $1 in + --override-all) + OVERRIDE_ALL=true + shift + ;; + --envFile) + ENV_FILE="$2" + shift 2 + ;; + -h|--help) + echo "Usage: $0 [--override-all] [--envFile ]" + echo "" + echo "Options:" + echo " --override-all Prompt for ArgoCD and Backstage secrets even if they exist" + echo " --envFile Read environment variables from specified file" + echo " -h, --help Show this help message" + exit 0 + ;; + *) + echo "Unknown option: $1" + echo "Use --help for usage information" + exit 1 + ;; + esac +done + +# Function to read and load environment variables from file +load_env_file() { + local env_file="$1" + if [[ -n "$env_file" ]]; then + if [[ -f "$env_file" ]]; then + log "๐Ÿ“„ Loading environment variables from: $env_file" + # Read the file line by line and export variables + while IFS= read -r line || [[ -n "$line" ]]; do + # Skip empty lines and comments + if [[ -n "$line" && ! "$line" =~ ^[[:space:]]*# ]]; then + # Check if line contains = + if [[ "$line" =~ ^[[:space:]]*([^=]+)=(.*)$ ]]; then + local var_name="${BASH_REMATCH[1]// /}" # Remove spaces + local var_value="${BASH_REMATCH[2]}" + + # Remove quotes if present + if [[ "$var_value" =~ ^\"(.*)\"$ ]] || [[ "$var_value" =~ ^\'(.*)\'$ ]]; then + var_value="${BASH_REMATCH[1]}" + fi + + # Export the variable if it's not already set or if we have a value + if [[ -n "$var_value" ]]; then + export "$var_name"="$var_value" + log " โœ“ Loaded $var_name from env file" + fi + fi + fi + done < "$env_file" + else + log "โš ๏ธ Environment file not found: $env_file" + exit 1 + fi + fi +} + +log "๐Ÿ”ง Setting up agent secrets based on active agents" + +# Setup Vault connection +VAULT_TOKEN=$(kubectl get secret vault-root-token -n vault -o jsonpath='{.data.token}' | base64 -d) +export VAULT_ADDR="http://localhost:8200" +export VAULT_TOKEN + +# Start port forward +log "๐Ÿ”— Starting Vault port forward..." +kubectl port-forward -n vault svc/vault 8200:8200 & +VAULT_PID=$! +sleep 3 + +# Single-line, exact-byte prompt helper (no newline added, no stripping) +# Usage: prompt_with_env "" VAR_NAME is_secret [default_value] +prompt_with_env() { + local prompt="$1" var_name="$2" is_secret="$3" default_value="$4" + local env_value="${!var_name}" result + + # If we have an env file and the variable has a value, auto-populate + if [[ -n "$ENV_FILE" && -n "$env_value" ]]; then + echo "[$(date '+%Y-%m-%d %H:%M:%S')] โœ“ Using existing value detected for $prompt in env file. Auto-populating..." >&2 + result="$env_value" + elif [[ -n "$env_value" ]]; then + if [[ "$is_secret" == "true" ]]; then + local hint="${env_value:0:5}..." + printf "%s (env: %s) [Enter to use, type new]: " "$prompt" "$hint" > /dev/tty + IFS= read -r choice < /dev/tty + if [[ -z "$choice" ]]; then + result="$env_value" + else + IFS= read -rs -p "$prompt: " result < /dev/tty + printf "\n" > /dev/tty + fi + else + IFS= read -r -p "$prompt (env: $env_value) [Enter to use, type new]: " choice < /dev/tty + if [[ -z "$choice" ]]; then + result="$env_value" + else + IFS= read -r -p "$prompt: " result < /dev/tty + fi + fi + else + if [[ "$is_secret" == "true" ]]; then + IFS= read -rs -p "$prompt: " result < /dev/tty + printf "\n" > /dev/tty + else + IFS= read -r -p "$prompt: " result < /dev/tty + fi + fi + + # Normalize only a trailing CR (some terminals send \r) + result=${result%$'\r'} + + # Use default value if result is empty and default is provided + if [[ -z "$result" && -n "$default_value" ]]; then + result="$default_value" + fi + + # Output EXACTLY the bytes, no newline + printf '%s' "$result" +} + +# Helper function to fetch existing secrets from Vault +# Usage: fetch_vault_secret "" "" +fetch_vault_secret() { + local vault_path="$1" field_name="$2" + local value + + # Try to fetch the secret, suppress errors if it doesn't exist + value=$(vault kv get -field="$field_name" "$vault_path" 2>/dev/null || echo "") + printf '%s' "$value" +} + +# Helper function to confirm override when value exists +# Usage: confirm_override "" +# Returns: 0 if user wants to override, 1 if not +confirm_override() { + local field_desc="$1" + local choice + + printf "%s is already populated. Are you sure you want to override? (Y/N): " "$field_desc" > /dev/tty + IFS= read -r choice < /dev/tty + + case "${choice,,}" in # Convert to lowercase + y|yes) + return 0 + ;; + *) + return 1 + ;; + esac +} + +# Check which agents are active +log "๐Ÿ” Checking active agents..." +active_agents=() + +# Check for GitHub agent (look for GitHub-related deployments or configs) +if kubectl get deployment -n ai-platform-engineering ai-platform-engineering-agent-github 2>/dev/null || \ + kubectl get configmap -n ai-platform-engineering | grep -i github >/dev/null 2>&1; then + active_agents+=("github") + log "โœ… GitHub agent detected" +fi + +# Check for GitLab agent +if kubectl get deployment -n ai-platform-engineering ai-platform-engineering-agent-gitlab 2>/dev/null || \ + kubectl get configmap -n ai-platform-engineering | grep -i gitlab >/dev/null 2>&1; then + active_agents+=("gitlab") + log "โœ… GitLab agent detected" +fi + +# Check for Jira agent +if kubectl get deployment -n ai-platform-engineering ai-platform-engineering-agent-jira 2>/dev/null || \ + kubectl get configmap -n ai-platform-engineering | grep -i jira >/dev/null 2>&1; then + active_agents+=("jira") + log "โœ… Jira agent detected" +fi + +# Check for Slack agent +if kubectl get deployment -n ai-platform-engineering ai-platform-engineering-agent-slack 2>/dev/null || \ + kubectl get configmap -n ai-platform-engineering | grep -i slack >/dev/null 2>&1; then + active_agents+=("slack") + log "โœ… Slack agent detected" +fi + +# Check for AWS agent +if kubectl get deployment -n ai-platform-engineering ai-platform-engineering-agent-aws 2>/dev/null || \ + kubectl get configmap -n ai-platform-engineering | grep -i aws >/dev/null 2>&1; then + active_agents+=("aws") + log "โœ… AWS agent detected" +fi + +# Check for ArgoCD agent +if kubectl get deployment -n ai-platform-engineering ai-platform-engineering-agent-argocd 2>/dev/null || \ + kubectl get configmap -n ai-platform-engineering | grep -i argocd >/dev/null 2>&1; then + active_agents+=("argocd") + log "โœ… ArgoCD agent detected" +fi + +# Check for Backstage agent +if kubectl get deployment -n ai-platform-engineering ai-platform-engineering-agent-backstage 2>/dev/null || \ + kubectl get configmap -n ai-platform-engineering | grep -i backstage >/dev/null 2>&1; then + active_agents+=("backstage") + log "โœ… Backstage agent detected" +fi + +# Check for PagerDuty agent +if kubectl get deployment -n ai-platform-engineering ai-platform-engineering-agent-pagerduty 2>/dev/null || \ + kubectl get configmap -n ai-platform-engineering | grep -i pagerduty >/dev/null 2>&1; then + active_agents+=("pagerduty") + log "โœ… PagerDuty agent detected" +fi + +# Check for Confluence agent +if kubectl get deployment -n ai-platform-engineering ai-platform-engineering-agent-confluence 2>/dev/null || \ + kubectl get configmap -n ai-platform-engineering | grep -i confluence >/dev/null 2>&1; then + active_agents+=("confluence") + log "โœ… Confluence agent detected" +fi + +# Check for Splunk agent +if kubectl get deployment -n ai-platform-engineering ai-platform-engineering-agent-splunk 2>/dev/null || \ + kubectl get configmap -n ai-platform-engineering | grep -i splunk >/dev/null 2>&1; then + active_agents+=("splunk") + log "โœ… Splunk agent detected" +fi + +# Check for Webex agent +if kubectl get deployment -n ai-platform-engineering ai-platform-engineering-agent-webex 2>/dev/null || \ + kubectl get configmap -n ai-platform-engineering | grep -i webex >/dev/null 2>&1; then + active_agents+=("webex") + log "โœ… Webex agent detected" +fi + +# Check for Komodor agent +if kubectl get deployment -n ai-platform-engineering ai-platform-engineering-agent-komodor 2>/dev/null || \ + kubectl get configmap -n ai-platform-engineering | grep -i komodor >/dev/null 2>&1; then + active_agents+=("komodor") + log "โœ… Komodor agent detected" +fi + +# If no agents detected, ask user to select +if [[ ${#active_agents[@]} -eq 0 ]]; then + log "๐Ÿค” No active agents detected. Please select which agents to configure:" + echo "" + echo "Available agents:" + echo "1) GitHub" + echo "2) Jira" + echo "3) Slack" + echo "4) AWS" + echo "5) ArgoCD" + echo "6) Backstage" + echo "7) PagerDuty" + echo "8) Confluence" + echo "9) Splunk" + echo "10) Webex" + echo "11) Komodor" + echo "12) All of the above" + echo "" + read -p "Select agents (comma-separated numbers, e.g., 1,3,4): " agent_selection + + IFS=',' read -ra selected <<< "$agent_selection" + for choice in "${selected[@]}"; do + case $choice in + 1) active_agents+=("github") ;; + 2) active_agents+=("jira") ;; + 3) active_agents+=("slack") ;; + 4) active_agents+=("aws") ;; + 5) active_agents+=("argocd") ;; + 6) active_agents+=("backstage") ;; + 7) active_agents+=("pagerduty") ;; + 8) active_agents+=("confluence") ;; + 9) active_agents+=("splunk") ;; + 10) active_agents+=("webex") ;; + 11) active_agents+=("komodor") ;; + 12) active_agents=("github" "jira" "slack" "aws" "argocd" "backstage" "pagerduty" "confluence" "splunk" "webex" "komodor") ;; + esac + done +fi + +log "๐Ÿ“ Configuring secrets for agents: ${active_agents[*]}" +echo "" +log "๐Ÿ”’ Note: Sensitive credentials will not be displayed on screen" + +# Initialize all fields as empty +GITHUB_PERSONAL_ACCESS_TOKEN="" +ATLASSIAN_TOKEN="" +ATLASSIAN_API_URL="" +ATLASSIAN_EMAIL="" +ATLASSIAN_VERIFY_SSL="" +SLACK_BOT_TOKEN="" +SLACK_TOKEN="" +SLACK_APP_TOKEN="" +SLACK_SIGNING_SECRET="" +SLACK_CLIENT_SECRET="" +SLACK_TEAM_ID="" +AWS_ACCESS_KEY_ID="" +AWS_SECRET_ACCESS_KEY="" +AWS_REGION="" +ARGOCD_TOKEN="" +ARGOCD_API_URL="" +ARGOCD_VERIFY_SSL="" +BACKSTAGE_API_TOKEN="" +BACKSTAGE_URL="" +PAGERDUTY_API_URL="" +PAGERDUTY_API_KEY="" +CONFLUENCE_API_URL="" +SPLUNK_API_URL="" +SPLUNK_TOKEN="" +WEBEX_TOKEN="" +KOMODOR_TOKEN="" +KOMODOR_API_URL="" + +# Load environment file if specified (after initialization) +load_env_file "$ENV_FILE" + +# Collect credentials based on active agents +for agent in "${active_agents[@]}"; do + case $agent in + "github") + echo "" + log "๐Ÿ™ Configuring GitHub agent secrets..." + GITHUB_PERSONAL_ACCESS_TOKEN="$(prompt_with_env 'GitHub Personal Access Token' 'GITHUB_PERSONAL_ACCESS_TOKEN' 'true')" + ;; + "jira") + echo "" + log "๐ŸŽซ Configuring Jira agent secrets..." + ATLASSIAN_TOKEN=$(prompt_with_env "Atlassian API Token" "ATLASSIAN_TOKEN" "true") + ATLASSIAN_API_URL=$(prompt_with_env "Atlassian API URL (e.g., https://company.atlassian.net)" "ATLASSIAN_API_URL" "false") + ATLASSIAN_EMAIL=$(prompt_with_env "Atlassian Email" "ATLASSIAN_EMAIL" "false") + ATLASSIAN_VERIFY_SSL=$(prompt_with_env "Verify SSL (true/false)" "ATLASSIAN_VERIFY_SSL" "false" "true") + ;; + "slack") + echo "" + log "๐Ÿ’ฌ Configuring Slack agent secrets..." + SLACK_BOT_TOKEN=$(prompt_with_env "Slack Bot Token (xoxb-...)" "SLACK_BOT_TOKEN" "true") + SLACK_TOKEN=$(prompt_with_env "Slack Token" "SLACK_TOKEN" "true") + SLACK_APP_TOKEN=$(prompt_with_env "Slack App Token (xapp-...)" "SLACK_APP_TOKEN" "true") + SLACK_SIGNING_SECRET=$(prompt_with_env "Slack Signing Secret" "SLACK_SIGNING_SECRET" "true") + SLACK_CLIENT_SECRET=$(prompt_with_env "Slack Client Secret" "SLACK_CLIENT_SECRET" "true") + SLACK_TEAM_ID=$(prompt_with_env "Slack Team ID" "SLACK_TEAM_ID" "false") + ;; + "aws") + echo "" + log "โ˜๏ธ Configuring AWS agent secrets..." + AWS_ACCESS_KEY_ID=$(prompt_with_env "AWS Access Key ID" "AWS_ACCESS_KEY_ID" "false") + AWS_SECRET_ACCESS_KEY=$(prompt_with_env "AWS Secret Access Key" "AWS_SECRET_ACCESS_KEY" "true") + AWS_REGION=$(prompt_with_env "AWS Region" "AWS_REGION" "false" "us-east-1") + ;; + "argocd") + echo "" + log "๐Ÿš€ Configuring ArgoCD agent secrets..." + + # Try to fetch existing secrets from Vault first + existing_token=$(fetch_vault_secret "secret/ai-platform-engineering/argocd-secret" "ARGOCD_TOKEN") + existing_api_url=$(fetch_vault_secret "secret/ai-platform-engineering/argocd-secret" "ARGOCD_API_URL") + existing_verify_ssl=$(fetch_vault_secret "secret/ai-platform-engineering/argocd-secret" "ARGOCD_VERIFY_SSL") + + # Handle ArgoCD Token + should_prompt_token=false + if [[ -z "$existing_token" ]]; then + should_prompt_token=true + elif [[ "$OVERRIDE_ALL" == "true" ]]; then + # Check if we have env file value and should use it + if [[ -n "$ENV_FILE" && -n "${ARGOCD_TOKEN:-}" ]]; then + log " Using ArgoCD Token from env file (override-all mode)" + # Value already loaded from env file + else + if confirm_override "ArgoCD Token"; then + should_prompt_token=true + else + log " Keeping existing ArgoCD Token from Vault" + ARGOCD_TOKEN="$existing_token" + fi + fi + else + log " Using existing ArgoCD Token from Vault" + ARGOCD_TOKEN="$existing_token" + fi + + if [[ "$should_prompt_token" == "true" ]]; then + [[ -n "$existing_token" ]] && export ARGOCD_TOKEN="$existing_token" + ARGOCD_TOKEN=$(prompt_with_env "ArgoCD Token" "ARGOCD_TOKEN" "true") + fi + + # Handle ArgoCD API URL + should_prompt_url=false + if [[ -z "$existing_api_url" ]]; then + should_prompt_url=true + elif [[ "$OVERRIDE_ALL" == "true" ]]; then + # Check if we have env file value and should use it + if [[ -n "$ENV_FILE" && -n "${ARGOCD_API_URL:-}" ]]; then + log " Using ArgoCD API URL from env file (override-all mode)" + # Value already loaded from env file + else + if confirm_override "ArgoCD API URL"; then + should_prompt_url=true + else + log " Keeping existing ArgoCD API URL from Vault" + ARGOCD_API_URL="$existing_api_url" + fi + fi + else + log " Using existing ArgoCD API URL from Vault" + ARGOCD_API_URL="$existing_api_url" + fi + + if [[ "$should_prompt_url" == "true" ]]; then + [[ -n "$existing_api_url" ]] && export ARGOCD_API_URL="$existing_api_url" + ARGOCD_API_URL=$(prompt_with_env "ArgoCD API URL" "ARGOCD_API_URL" "false") + [[ -z "$ARGOCD_API_URL" ]] && ARGOCD_API_URL="http://argocd-server.argocd.svc.cluster.local" + fi + + # Handle ArgoCD Verify SSL + should_prompt_ssl=false + if [[ -z "$existing_verify_ssl" ]]; then + should_prompt_ssl=true + elif [[ "$OVERRIDE_ALL" == "true" ]]; then + # Check if we have env file value and should use it + if [[ -n "$ENV_FILE" && -n "${ARGOCD_VERIFY_SSL:-}" ]]; then + log " Using ArgoCD Verify SSL setting from env file (override-all mode)" + # Value already loaded from env file + else + if confirm_override "ArgoCD Verify SSL setting"; then + should_prompt_ssl=true + else + log " Keeping existing ArgoCD Verify SSL setting from Vault" + ARGOCD_VERIFY_SSL="$existing_verify_ssl" + fi + fi + else + log " Using existing ArgoCD Verify SSL setting from Vault" + ARGOCD_VERIFY_SSL="$existing_verify_ssl" + fi + + if [[ "$should_prompt_ssl" == "true" ]]; then + [[ -n "$existing_verify_ssl" ]] && export ARGOCD_VERIFY_SSL="$existing_verify_ssl" + ARGOCD_VERIFY_SSL=$(prompt_with_env "Verify SSL (true/false)" "ARGOCD_VERIFY_SSL" "false") + [[ -z "$ARGOCD_VERIFY_SSL" ]] && ARGOCD_VERIFY_SSL="false" + fi + ;; + "backstage") + echo "" + log "๐ŸŽญ Configuring Backstage agent secrets..." + + # Try to fetch existing secrets from Vault first + existing_api_token=$(fetch_vault_secret "secret/ai-platform-engineering/backstage-secret" "BACKSTAGE_API_TOKEN") + existing_url=$(fetch_vault_secret "secret/ai-platform-engineering/backstage-secret" "BACKSTAGE_URL") + + # Handle Backstage API Token + should_prompt_token=false + if [[ -z "$existing_api_token" ]]; then + should_prompt_token=true + elif [[ "$OVERRIDE_ALL" == "true" ]]; then + # Check if we have env file value and should use it + if [[ -n "$ENV_FILE" && -n "${BACKSTAGE_API_TOKEN:-}" ]]; then + log " Using Backstage API Token from env file (override-all mode)" + # Value already loaded from env file + else + if confirm_override "Backstage API Token"; then + should_prompt_token=true + else + log " Keeping existing Backstage API Token from Vault" + BACKSTAGE_API_TOKEN="$existing_api_token" + fi + fi + else + log " Using existing Backstage API Token from Vault" + BACKSTAGE_API_TOKEN="$existing_api_token" + fi + + if [[ "$should_prompt_token" == "true" ]]; then + [[ -n "$existing_api_token" ]] && export BACKSTAGE_API_TOKEN="$existing_api_token" + BACKSTAGE_API_TOKEN=$(prompt_with_env "Backstage API Token" "BACKSTAGE_API_TOKEN" "true") + fi + + # Handle Backstage URL + should_prompt_url=false + if [[ -z "$existing_url" ]]; then + should_prompt_url=true + elif [[ "$OVERRIDE_ALL" == "true" ]]; then + # Check if we have env file value and should use it + if [[ -n "$ENV_FILE" && -n "${BACKSTAGE_URL:-}" ]]; then + log " Using Backstage URL from env file (override-all mode)" + # Value already loaded from env file + else + if confirm_override "Backstage URL"; then + should_prompt_url=true + else + log " Keeping existing Backstage URL from Vault" + BACKSTAGE_URL="$existing_url" + fi + fi + else + log " Using existing Backstage URL from Vault" + BACKSTAGE_URL="$existing_url" + fi + + if [[ "$should_prompt_url" == "true" ]]; then + [[ -n "$existing_url" ]] && export BACKSTAGE_URL="$existing_url" + BACKSTAGE_URL=$(prompt_with_env "Backstage URL" "BACKSTAGE_URL" "false") + [[ -z "$BACKSTAGE_URL" ]] && BACKSTAGE_URL="http://backstage.backstage.svc.cluster.local:7007" + fi + ;; + "pagerduty") + echo "" + log "๐Ÿ“Ÿ Configuring PagerDuty agent secrets..." + PAGERDUTY_API_KEY=$(prompt_with_env "PagerDuty API Key" "PAGERDUTY_API_KEY" "true") + PAGERDUTY_API_URL=$(prompt_with_env "PagerDuty API URL" "PAGERDUTY_API_URL" "false" "https://api.pagerduty.com") + ;; + "confluence") + echo "" + log "๐Ÿ“š Configuring Confluence agent secrets..." + CONFLUENCE_API_URL=$(prompt_with_env "Confluence API URL (e.g., https://company.atlassian.net/wiki)" "CONFLUENCE_API_URL" "false") + if [[ -z "$ATLASSIAN_TOKEN" ]]; then + ATLASSIAN_TOKEN=$(prompt_with_env "Atlassian API Token" "ATLASSIAN_TOKEN" "true") + ATLASSIAN_EMAIL=$(prompt_with_env "Atlassian Email" "ATLASSIAN_EMAIL" "false") + ATLASSIAN_VERIFY_SSL=$(prompt_with_env "Verify SSL (true/false)" "ATLASSIAN_VERIFY_SSL" "false" "true") + fi + ;; + "splunk") + echo "" + log "๐Ÿ” Configuring Splunk agent secrets..." + SPLUNK_TOKEN=$(prompt_with_env "Splunk Token" "SPLUNK_TOKEN" "true") + SPLUNK_API_URL=$(prompt_with_env "Splunk API URL (e.g., https://splunk.company.com)" "SPLUNK_API_URL" "false") + ;; + "webex") + echo "" + log "๐Ÿ“น Configuring Webex agent secrets..." + WEBEX_TOKEN=$(prompt_with_env "Webex Token" "WEBEX_TOKEN" "true") + ;; + "komodor") + echo "" + log "๐Ÿ”ง Configuring Komodor agent secrets..." + KOMODOR_TOKEN=$(prompt_with_env "Komodor Token" "KOMODOR_TOKEN" "true") + KOMODOR_API_URL=$(prompt_with_env "Komodor API URL" "KOMODOR_API_URL" "false" "https://api.komodor.com") + ;; + esac +done + +# Store all secrets in Vault +log "๐Ÿ’พ Storing agent secrets in Vault..." + +# Store secrets individually for each active agent +for agent in "${active_agents[@]}"; do + case $agent in + "github") + if [[ -n "$GITHUB_PERSONAL_ACCESS_TOKEN" ]]; then + vault kv put secret/ai-platform-engineering/github-secret \ + GITHUB_PERSONAL_ACCESS_TOKEN="$GITHUB_PERSONAL_ACCESS_TOKEN" >/dev/null + log "โœ… GitHub secrets stored" + fi + ;; + "jira") + if [[ -n "$ATLASSIAN_TOKEN" ]]; then + vault kv put secret/ai-platform-engineering/jira-secret \ + ATLASSIAN_TOKEN="$ATLASSIAN_TOKEN" \ + ATLASSIAN_API_URL="$ATLASSIAN_API_URL" \ + ATLASSIAN_EMAIL="$ATLASSIAN_EMAIL" \ + ATLASSIAN_VERIFY_SSL="$ATLASSIAN_VERIFY_SSL" >/dev/null + log "โœ… Jira secrets stored" + fi + ;; + "slack") + if [[ -n "$SLACK_BOT_TOKEN" ]]; then + vault kv put secret/ai-platform-engineering/slack-secret \ + SLACK_BOT_TOKEN="$SLACK_BOT_TOKEN" \ + SLACK_TOKEN="$SLACK_TOKEN" \ + SLACK_APP_TOKEN="$SLACK_APP_TOKEN" \ + SLACK_SIGNING_SECRET="$SLACK_SIGNING_SECRET" \ + SLACK_CLIENT_SECRET="$SLACK_CLIENT_SECRET" \ + SLACK_TEAM_ID="$SLACK_TEAM_ID" >/dev/null + log "โœ… Slack secrets stored" + fi + ;; + "aws") + if [[ -n "$AWS_ACCESS_KEY_ID" ]]; then + vault kv put secret/ai-platform-engineering/aws-secret \ + AWS_ACCESS_KEY_ID="$AWS_ACCESS_KEY_ID" \ + AWS_SECRET_ACCESS_KEY="$AWS_SECRET_ACCESS_KEY" \ + AWS_REGION="$AWS_REGION" >/dev/null + log "โœ… AWS secrets stored" + fi + ;; + "argocd") + if [[ -n "$ARGOCD_TOKEN" ]]; then + vault kv put secret/ai-platform-engineering/argocd-secret \ + ARGOCD_TOKEN="$ARGOCD_TOKEN" \ + ARGOCD_API_URL="$ARGOCD_API_URL" \ + ARGOCD_VERIFY_SSL="$ARGOCD_VERIFY_SSL" >/dev/null + log "โœ… ArgoCD secrets stored" + fi + ;; + "backstage") + if [[ -n "$BACKSTAGE_API_TOKEN" ]]; then + vault kv put secret/ai-platform-engineering/backstage-secret \ + BACKSTAGE_API_TOKEN="$BACKSTAGE_API_TOKEN" \ + BACKSTAGE_URL="$BACKSTAGE_URL" >/dev/null + log "โœ… Backstage secrets stored" + fi + ;; + "pagerduty") + if [[ -n "$PAGERDUTY_API_KEY" ]]; then + vault kv put secret/ai-platform-engineering/pagerduty-secret \ + PAGERDUTY_API_KEY="$PAGERDUTY_API_KEY" \ + PAGERDUTY_API_URL="$PAGERDUTY_API_URL" >/dev/null + log "โœ… PagerDuty secrets stored" + fi + ;; + "confluence") + if [[ -n "$CONFLUENCE_API_URL" ]]; then + vault kv put secret/ai-platform-engineering/confluence-secret \ + CONFLUENCE_API_URL="$CONFLUENCE_API_URL" \ + ATLASSIAN_TOKEN="$ATLASSIAN_TOKEN" \ + ATLASSIAN_EMAIL="$ATLASSIAN_EMAIL" \ + ATLASSIAN_VERIFY_SSL="$ATLASSIAN_VERIFY_SSL" >/dev/null + log "โœ… Confluence secrets stored" + fi + ;; + "splunk") + if [[ -n "$SPLUNK_TOKEN" ]]; then + vault kv put secret/ai-platform-engineering/splunk-secret \ + SPLUNK_TOKEN="$SPLUNK_TOKEN" \ + SPLUNK_API_URL="$SPLUNK_API_URL" >/dev/null + log "โœ… Splunk secrets stored" + fi + ;; + "webex") + if [[ -n "$WEBEX_TOKEN" ]]; then + vault kv put secret/ai-platform-engineering/webex-secret \ + WEBEX_TOKEN="$WEBEX_TOKEN" >/dev/null + log "โœ… Webex secrets stored" + fi + ;; + "komodor") + if [[ -n "$KOMODOR_TOKEN" ]]; then + vault kv put secret/ai-platform-engineering/komodor-secret \ + KOMODOR_TOKEN="$KOMODOR_TOKEN" \ + KOMODOR_API_URL="$KOMODOR_API_URL" >/dev/null + log "โœ… Komodor secrets stored" + fi + ;; + esac +done + +log "โœ… Agent secrets successfully stored in Vault" +echo "" +log "๐Ÿ” You can verify individual agent secrets at:" +for agent in "${active_agents[@]}"; do + case $agent in + "github") log " ๐Ÿ™ GitHub: https://vault.cnoe.localtest.me:8443/ui/vault/secrets/secret/kv/ai-platform-engineering%2Fgithub-secret" ;; + "jira") log " ๐ŸŽซ Jira: https://vault.cnoe.localtest.me:8443/ui/vault/secrets/secret/kv/ai-platform-engineering%2Fjira-secret" ;; + "slack") log " ๐Ÿ’ฌ Slack: https://vault.cnoe.localtest.me:8443/ui/vault/secrets/secret/kv/ai-platform-engineering%2Fslack-secret" ;; + "aws") log " โ˜๏ธ AWS: https://vault.cnoe.localtest.me:8443/ui/vault/secrets/secret/kv/ai-platform-engineering%2Faws-secret" ;; + "argocd") log " ๐Ÿš€ ArgoCD: https://vault.cnoe.localtest.me:8443/ui/vault/secrets/secret/kv/ai-platform-engineering%2Fargocd-secret" ;; + "backstage") log " ๐ŸŽญ Backstage: https://vault.cnoe.localtest.me:8443/ui/vault/secrets/secret/kv/ai-platform-engineering%2Fbackstage-secret" ;; + "pagerduty") log " ๐Ÿ“Ÿ PagerDuty: https://vault.cnoe.localtest.me:8443/ui/vault/secrets/secret/kv/ai-platform-engineering%2Fpagerduty-secret" ;; + "confluence") log " ๐Ÿ“š Confluence: https://vault.cnoe.localtest.me:8443/ui/vault/secrets/secret/kv/ai-platform-engineering%2Fconfluence-secret" ;; + "splunk") log " ๐Ÿ” Splunk: https://vault.cnoe.localtest.me:8443/ui/vault/secrets/secret/kv/ai-platform-engineering%2Fsplunk-secret" ;; + "webex") log " ๐Ÿ“น Webex: https://vault.cnoe.localtest.me:8443/ui/vault/secrets/secret/kv/ai-platform-engineering%2Fwebex-secret" ;; + "komodor") log " ๐Ÿ”ง Komodor: https://vault.cnoe.localtest.me:8443/ui/vault/secrets/secret/kv/ai-platform-engineering%2Fkomodor-secret" ;; + esac +done + +# Create Kubernetes secret for agents +log "๐Ÿ”„ Creating Kubernetes secret for agents..." +kubectl create secret generic agent-secrets -n ai-platform-engineering \ + --from-literal=GITHUB_PERSONAL_ACCESS_TOKEN="$GITHUB_PERSONAL_ACCESS_TOKEN" \ + --from-literal=ATLASSIAN_TOKEN="$ATLASSIAN_TOKEN" \ + --from-literal=ATLASSIAN_API_URL="$ATLASSIAN_API_URL" \ + --from-literal=ATLASSIAN_EMAIL="$ATLASSIAN_EMAIL" \ + --from-literal=ATLASSIAN_VERIFY_SSL="$ATLASSIAN_VERIFY_SSL" \ + --from-literal=SLACK_BOT_TOKEN="$SLACK_BOT_TOKEN" \ + --from-literal=SLACK_TOKEN="$SLACK_TOKEN" \ + --from-literal=SLACK_APP_TOKEN="$SLACK_APP_TOKEN" \ + --from-literal=SLACK_SIGNING_SECRET="$SLACK_SIGNING_SECRET" \ + --from-literal=SLACK_CLIENT_SECRET="$SLACK_CLIENT_SECRET" \ + --from-literal=SLACK_TEAM_ID="$SLACK_TEAM_ID" \ + --from-literal=AWS_ACCESS_KEY_ID="$AWS_ACCESS_KEY_ID" \ + --from-literal=AWS_SECRET_ACCESS_KEY="$AWS_SECRET_ACCESS_KEY" \ + --from-literal=AWS_REGION="$AWS_REGION" \ + --from-literal=ARGOCD_TOKEN="$ARGOCD_TOKEN" \ + --from-literal=ARGOCD_API_URL="$ARGOCD_API_URL" \ + --from-literal=ARGOCD_VERIFY_SSL="$ARGOCD_VERIFY_SSL" \ + --from-literal=BACKSTAGE_API_TOKEN="$BACKSTAGE_API_TOKEN" \ + --from-literal=BACKSTAGE_URL="$BACKSTAGE_URL" \ + --from-literal=PAGERDUTY_API_KEY="$PAGERDUTY_API_KEY" \ + --from-literal=PAGERDUTY_API_URL="$PAGERDUTY_API_URL" \ + --from-literal=CONFLUENCE_API_URL="$CONFLUENCE_API_URL" \ + --from-literal=SPLUNK_TOKEN="$SPLUNK_TOKEN" \ + --from-literal=SPLUNK_API_URL="$SPLUNK_API_URL" \ + --from-literal=WEBEX_TOKEN="$WEBEX_TOKEN" \ + --from-literal=KOMODOR_TOKEN="$KOMODOR_TOKEN" \ + --from-literal=KOMODOR_API_URL="$KOMODOR_API_URL" \ + --dry-run=client -o yaml | kubectl apply -f - + +log "โœ… Kubernetes secret created/updated" + +# Summary +echo "" +log "๐Ÿ“Š Configuration Summary:" +for agent in "${active_agents[@]}"; do + case $agent in + "github") log " ๐Ÿ™ GitHub: Personal Access Token configured" ;; + "jira") log " ๐ŸŽซ Jira: Atlassian Token and API URL configured" ;; + "slack") log " ๐Ÿ’ฌ Slack: Bot Token, App Token, and additional tokens configured" ;; + "aws") log " โ˜๏ธ AWS: Access Keys and Region configured" ;; + "argocd") log " ๐Ÿš€ ArgoCD: Token and API URL configured" ;; + "backstage") log " ๐ŸŽญ Backstage: API Token and URL configured" ;; + "pagerduty") log " ๐Ÿ“Ÿ PagerDuty: API Key and URL configured" ;; + "confluence") log " ๐Ÿ“š Confluence: API URL and Atlassian credentials configured" ;; + "splunk") log " ๐Ÿ” Splunk: Token and API URL configured" ;; + "webex") log " ๐Ÿ“น Webex: Token configured" ;; + "komodor") log " ๐Ÿ”ง Komodor: Token and API URL configured" ;; + esac +done + +# Cleanup +kill $VAULT_PID 2>/dev/null +log "๐ŸŽ‰ Agent secrets setup complete!" diff --git a/caipe/scripts/setup-all.sh b/caipe/scripts/setup-all.sh new file mode 100755 index 00000000..0bc3fa8b --- /dev/null +++ b/caipe/scripts/setup-all.sh @@ -0,0 +1,61 @@ +#!/bin/bash + +set -euo pipefail + +script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" + +OVERRIDE_ALL=false +ENV_FILE="" + +while [[ $# -gt 0 ]]; do + case "$1" in + --override-all) + OVERRIDE_ALL=true + shift + ;; + --envFile) + ENV_FILE="$2" + shift 2 + ;; + -h|--help) + echo "Usage: $0 [--override-all] [--envFile ]" + echo "" + echo "Options:" + echo " --override-all Force prompts for existing secrets in agent setup" + echo " --envFile Path to env file to source for values" + exit 0 + ;; + *) + echo "Unknown option: $1" >&2 + echo "Use --help for usage information" >&2 + exit 1 + ;; + esac +done + +# Build argument lists safely +llm_args=() +agent_args=() + +if [[ -n "$ENV_FILE" ]]; then + llm_args+=(--envFile "$ENV_FILE") + agent_args+=(--envFile "$ENV_FILE") +fi + +if [[ "$OVERRIDE_ALL" == "true" ]]; then + agent_args+=(--override-all) +fi + +echo "๐Ÿงฉ Running setup-llm-credentials.sh..." +bash "$script_dir/setup-llm-credentials.sh" "${llm_args[@]}" + +echo "๐Ÿงฉ Running setup-agent-secrets.sh..." +bash "$script_dir/setup-agent-secrets.sh" "${agent_args[@]}" + +echo "โณ Waiting 2 seconds before refreshing secrets..." +sleep 2 + +echo "๐Ÿ”„ Running refresh-secrets.sh..." +bash "$script_dir/refresh-secrets.sh" + +echo "โœ… All done." diff --git a/caipe/scripts/setup-llm-credentials.sh b/caipe/scripts/setup-llm-credentials.sh new file mode 100755 index 00000000..75e6c917 --- /dev/null +++ b/caipe/scripts/setup-llm-credentials.sh @@ -0,0 +1,262 @@ +#!/bin/bash + +set -e + +log() { + echo "[$(date '+%Y-%m-%d %H:%M:%S')] $1" +} + +# Check if kubectl is available +if ! command -v kubectl &> /dev/null; then + log "โŒ kubectl is required but not installed" + exit 1 +fi + +# Check if vault CLI is available +if ! command -v vault &> /dev/null; then + log "โŒ vault CLI is required but not installed" + exit 1 +fi + +ENV_FILE="" +while [[ $# -gt 0 ]]; do + case $1 in + --envFile) + ENV_FILE="$2" + shift 2 + ;; + -h|--help) + echo "Usage: $0 [--envFile ]" + echo "" + echo "Options:" + echo " --envFile Read environment variables from specified file" + echo " -h, --help Show this help message" + exit 0 + ;; + *) + echo "Unknown option: $1" + echo "Use --help for usage information" + exit 1 + ;; + esac +done + +# Function to read and load environment variables from file +load_env_file() { + local env_file="$1" + if [[ -n "$env_file" ]]; then + if [[ -f "$env_file" ]]; then + log "๐Ÿ“„ Loading environment variables from: $env_file" + # Read the file line by line and export variables + while IFS= read -r line || [[ -n "$line" ]]; do + # Skip empty lines and comments + if [[ -n "$line" && ! "$line" =~ ^[[:space:]]*# ]]; then + # Check if line contains = + if [[ "$line" =~ ^[[:space:]]*([^=]+)=(.*)$ ]]; then + local var_name="${BASH_REMATCH[1]// /}" # Remove spaces + local var_value="${BASH_REMATCH[2]}" + + # Remove quotes if present + if [[ "$var_value" =~ ^\"(.*)\"$ ]] || [[ "$var_value" =~ ^\'(.*)\'$ ]]; then + var_value="${BASH_REMATCH[1]}" + fi + + # Export the variable if it's not already set or if we have a value + if [[ -n "$var_value" ]]; then + export "$var_name"="$var_value" + log " โœ“ Loaded $var_name from env file" + fi + fi + fi + done < "$env_file" + else + log "โš ๏ธ Environment file not found: $env_file" + exit 1 + fi + fi +} + +log "๐Ÿ”ง Setting up LLM credentials for AI Platform Engineering" + +# Get vault token and setup connection +VAULT_TOKEN=$(kubectl get secret vault-root-token -n vault -o jsonpath='{.data.token}' | base64 -d) +export VAULT_ADDR="http://localhost:8200" +export VAULT_TOKEN + +# Start port forward in background +log "๐Ÿ”— Starting Vault port forward..." +kubectl port-forward -n vault svc/vault 8200:8200 & +VAULT_PID=$! +sleep 3 + +# Initialize all fields as empty +AZURE_OPENAI_API_KEY="" +AZURE_OPENAI_ENDPOINT="" +AZURE_OPENAI_API_VERSION="" +AZURE_OPENAI_DEPLOYMENT="" +OPENAI_API_KEY="" +OPENAI_ENDPOINT="" +OPENAI_MODEL_NAME="" +AWS_ACCESS_KEY_ID="" +AWS_SECRET_ACCESS_KEY="" +AWS_REGION="" +AWS_BEDROCK_MODEL_ID="" +AWS_BEDROCK_PROVIDER="" +GOOGLE_API_KEY="" +GOOGLE_MODEL_NAME="" +GCP_PROJECT_ID="" +GCP_LOCATION="" +GCP_MODEL_NAME="" + +if [[ -n "$ENV_FILE" ]]; then + load_env_file "$ENV_FILE" +fi + +# see if LLM_PROVIDER is set in the env file +if [[ -n "${LLM_PROVIDER:-}" ]]; then + LLM_PROVIDER="$LLM_PROVIDER" + log "๐Ÿ“ Using provider from env file: $LLM_PROVIDER" +else + # Prompt for LLM provider + echo "" + echo "Supported LLM Providers:" + echo "1) azure-openai" + echo "2) openai" + echo "3) aws-bedrock" + echo "4) google-gemini" + echo "5) gcp-vertex" + echo "" + read -p "Select LLM provider (1-5): " provider_choice + + case $provider_choice in + 1) LLM_PROVIDER="azure-openai" ;; + 2) LLM_PROVIDER="openai" ;; + 3) LLM_PROVIDER="aws-bedrock" ;; + 4) LLM_PROVIDER="google-gemini" ;; + 5) LLM_PROVIDER="gcp-vertex" ;; + *) log "โŒ Invalid choice"; kill $VAULT_PID 2>/dev/null; exit 1 ;; + esac + + log "๐Ÿ“ Selected provider: $LLM_PROVIDER" +fi + +echo "" +log "๐Ÿ”’ Note: Sensitive credentials will not be displayed on screen" + +# Load environment file if specified (after initialization) +load_env_file "$ENV_FILE" + +# Single-line, exact-byte prompt helper (no newline added, no stripping) +# Usage: prompt_with_env "" VAR_NAME is_secret +prompt_with_env() { + local prompt="$1" var_name="$2" is_secret="$3" default_value="$4" + local env_value="${!var_name}" result + + # If we have an env file and the variable has a value, auto-populate + if [[ -n "$ENV_FILE" && -n "$env_value" ]]; then + echo "[$(date '+%Y-%m-%d %H:%M:%S')] โœ“ Using existing value detected for $prompt in env file. Auto-populating..." >&2 + result="$env_value" + elif [[ -n "$env_value" ]]; then + if [[ "$is_secret" == "true" ]]; then + local hint="${env_value:0:5}..." + printf "%s (env: %s) [Enter to use, type new]: " "$prompt" "$hint" > /dev/tty + IFS= read -r choice < /dev/tty + if [[ -z "$choice" ]]; then + result="$env_value" + else + IFS= read -rs -p "$prompt: " result < /dev/tty + printf "\n" > /dev/tty + fi + else + IFS= read -r -p "$prompt (env: $env_value) [Enter to use, type new]: " choice < /dev/tty + if [[ -z "$choice" ]]; then + result="$env_value" + else + IFS= read -r -p "$prompt: " result < /dev/tty + fi + fi + else + if [[ "$is_secret" == "true" ]]; then + IFS= read -rs -p "$prompt: " result < /dev/tty + printf "\n" > /dev/tty + else + IFS= read -r -p "$prompt: " result < /dev/tty + fi + fi + + # Normalize only a trailing CR (some terminals send \r) + result=${result%$'\r'} + + # Use default value if result is empty and default is provided + if [[ -z "$result" && -n "$default_value" ]]; then + result="$default_value" + fi + + # Output EXACTLY the bytes, no newline + printf '%s' "$result" +} + +# Collect credentials based on provider +case $LLM_PROVIDER in + "azure-openai") + echo "" + AZURE_OPENAI_API_KEY="$(prompt_with_env 'Azure OpenAI API Key' 'AZURE_OPENAI_API_KEY' 'true')" + AZURE_OPENAI_ENDPOINT="$(prompt_with_env 'Azure OpenAI Endpoint' 'AZURE_OPENAI_ENDPOINT' 'false')" + AZURE_OPENAI_API_VERSION="$(prompt_with_env 'Azure OpenAI API Version' 'AZURE_OPENAI_API_VERSION' 'false')" + AZURE_OPENAI_DEPLOYMENT="$(prompt_with_env 'Azure OpenAI Deployment Name' 'AZURE_OPENAI_DEPLOYMENT' 'false')" + ;; + "openai") + echo "" + OPENAI_API_KEY="$(prompt_with_env 'OpenAI API Key' 'OPENAI_API_KEY' 'true')" + OPENAI_ENDPOINT="$(prompt_with_env 'OpenAI Endpoint' 'OPENAI_ENDPOINT' 'false')" + OPENAI_MODEL_NAME="$(prompt_with_env 'OpenAI Model Name' 'OPENAI_MODEL_NAME' 'false')" + ;; + "aws-bedrock") + echo "" + AWS_ACCESS_KEY_ID="$(prompt_with_env 'AWS Access Key ID' 'AWS_ACCESS_KEY_ID' 'false')" + AWS_SECRET_ACCESS_KEY="$(prompt_with_env 'AWS Secret Access Key' 'AWS_SECRET_ACCESS_KEY' 'true')" + AWS_REGION="$(prompt_with_env 'AWS Region' 'AWS_REGION' 'false')" + AWS_BEDROCK_MODEL_ID="$(prompt_with_env 'AWS Bedrock Model ID' 'AWS_BEDROCK_MODEL_ID' 'false')" + AWS_BEDROCK_PROVIDER="$(prompt_with_env 'AWS Bedrock Provider' 'AWS_BEDROCK_PROVIDER' 'false')" + ;; + "google-gemini") + echo "" + GOOGLE_API_KEY="$(prompt_with_env 'Google API Key' 'GOOGLE_API_KEY' 'true')" + GOOGLE_MODEL_NAME="$(prompt_with_env 'Google Model Name' 'GOOGLE_MODEL_NAME' 'false')" + ;; + "gcp-vertex") + echo "" + GCP_PROJECT_ID="$(prompt_with_env 'GCP Project ID' 'GCP_PROJECT_ID' 'false')" + GCP_LOCATION="$(prompt_with_env 'GCP Location' 'GCP_LOCATION' 'false')" + GCP_MODEL_NAME="$(prompt_with_env 'GCP Model Name' 'GCP_MODEL_NAME' 'false')" + ;; +esac + +# Store credentials in Vault +log "๐Ÿ’พ Storing credentials in Vault..." +vault kv put secret/ai-platform-engineering/global \ + LLM_PROVIDER="$LLM_PROVIDER" \ + AZURE_OPENAI_API_KEY="$AZURE_OPENAI_API_KEY" \ + AZURE_OPENAI_ENDPOINT="$AZURE_OPENAI_ENDPOINT" \ + AZURE_OPENAI_API_VERSION="$AZURE_OPENAI_API_VERSION" \ + AZURE_OPENAI_DEPLOYMENT="$AZURE_OPENAI_DEPLOYMENT" \ + OPENAI_API_KEY="$OPENAI_API_KEY" \ + OPENAI_ENDPOINT="$OPENAI_ENDPOINT" \ + OPENAI_MODEL_NAME="$OPENAI_MODEL_NAME" \ + AWS_ACCESS_KEY_ID="$AWS_ACCESS_KEY_ID" \ + AWS_SECRET_ACCESS_KEY="$AWS_SECRET_ACCESS_KEY" \ + AWS_REGION="$AWS_REGION" \ + AWS_BEDROCK_MODEL_ID="$AWS_BEDROCK_MODEL_ID" \ + AWS_BEDROCK_PROVIDER="$AWS_BEDROCK_PROVIDER" \ + GOOGLE_API_KEY="$GOOGLE_API_KEY" \ + GOOGLE_MODEL_NAME="$GOOGLE_MODEL_NAME" \ + GCP_PROJECT_ID="$GCP_PROJECT_ID" \ + GCP_LOCATION="$GCP_LOCATION" \ + GCP_MODEL_NAME="$GCP_MODEL_NAME" >/dev/null + +log "โœ… LLM credentials successfully stored in Vault" +log "๐Ÿ” You can verify at: https://vault.cnoe.localtest.me:8443/ui/vault/secrets/secret/kv/ai-platform-engineering%2Fglobal" + +# Cleanup +kill $VAULT_PID 2>/dev/null +log "๐ŸŽ‰ Setup complete!" diff --git a/caipe/scripts/setup-ubuntu-prerequisites.sh b/caipe/scripts/setup-ubuntu-prerequisites.sh new file mode 100755 index 00000000..66aa6387 --- /dev/null +++ b/caipe/scripts/setup-ubuntu-prerequisites.sh @@ -0,0 +1,851 @@ +#!/bin/bash +# Complete CAIPE + i3 VNC Setup Script +# Combines i3 desktop environment with IDPBuilder platform setup +# Run with: bash setup-ubuntu-prerequisites.sh +# +# This script runs in non-interactive mode to avoid package configuration prompts. +# It preconfigures keyboard layout (US English), timezone (America/New_York), +# locale (en_US.UTF-8), and display manager (lightdm) to prevent interactive dialogs during installation. + +set -e + +# Colors for output +RED='\033[0;31m' +GREEN='\033[0;32m' +YELLOW='\033[1;33m' +BLUE='\033[0;34m' +NC='\033[0m' # No Color + +# Function to print colored output +print_status() { + echo -e "${BLUE}[INFO]${NC} $1" +} + +print_success() { + echo -e "${GREEN}[SUCCESS]${NC} $1" +} + +print_warning() { + echo -e "${YELLOW}[WARNING]${NC} $1" +} + +print_error() { + echo -e "${RED}[ERROR]${NC} $1" +} + +echo "๐Ÿš€ Setting up CAIPE Ubuntu prerequisites..." + +# Function to handle package installation with error recovery +install_package() { + local package_name="$1" + local description="${2:-$package_name}" + + # Check if package is already installed + if dpkg -l | grep -q "^ii.*$package_name "; then + print_success "$description is already installed" + return 0 + fi + + print_status "Installing $description..." + + # First attempt with non-interactive flags + if DEBIAN_FRONTEND=noninteractive sudo apt install -y -q "$package_name"; then + print_success "$description installed successfully" + return 0 + fi + + # If first attempt fails, try to fix dependencies + print_warning "Failed to install $description, attempting to fix dependencies..." + DEBIAN_FRONTEND=noninteractive sudo apt --fix-broken install -y -q || true + DEBIAN_FRONTEND=noninteractive sudo apt autoremove -y -q || true + sudo apt update -q || true + + # Second attempt with non-interactive flags + if DEBIAN_FRONTEND=noninteractive sudo apt install -y -q "$package_name"; then + print_success "$description installed successfully on second attempt" + return 0 + fi + + # If still failing, try to remove conflicting packages and retry + print_warning "Still failing, attempting to remove conflicting packages..." + DEBIAN_FRONTEND=noninteractive sudo apt remove -y -q amazon-q 2>/dev/null || true + DEBIAN_FRONTEND=noninteractive sudo apt autoremove -y -q || true + DEBIAN_FRONTEND=noninteractive sudo apt --fix-broken install -y -q || true + + # Third attempt with non-interactive flags + if DEBIAN_FRONTEND=noninteractive sudo apt install -y -q "$package_name"; then + print_success "$description installed successfully after cleanup" + return 0 + fi + + print_error "Failed to install $description after multiple attempts" + return 1 +} + +# Function to handle command execution with error recovery +run_command() { + local description="$1" + local command="$2" + + print_status "$description..." + if eval "$command"; then + print_success "$description completed successfully" + else + print_warning "$description failed, continuing..." + return 1 + fi +} + +# Function to clean up duplicate repositories +cleanup_duplicate_repositories() { + print_status "Cleaning up duplicate repositories..." + + # Remove duplicate HashiCorp repositories + sudo rm -f /etc/apt/sources.list.d/archive_uri-https_apt_releases_hashicorp_com-*.list + sudo rm -f /etc/apt/sources.list.d/hashicorp.list + + # Remove duplicate GitHub CLI repositories + sudo rm -f /etc/apt/sources.list.d/archive_uri-https_cli_github_com_packages-*.list + sudo rm -f /etc/apt/sources.list.d/github-cli.list + + # Remove duplicate Docker repositories + sudo rm -f /etc/apt/sources.list.d/archive_uri-https_download_docker_com_linux_ubuntu-*.list + sudo rm -f /etc/apt/sources.list.d/docker.list + + print_success "Repository cleanup completed" +} + +# Function to aggressively clean up conflicting packages +cleanup_conflicting_packages() { + print_status "Cleaning up conflicting packages..." + + # Check if amazon-q is causing issues + if dpkg -l | grep -q amazon-q; then + print_status "Found amazon-q package, attempting removal..." + + # First, try to fix broken dependencies + print_status "Fixing broken dependencies..." + DEBIAN_FRONTEND=noninteractive sudo apt --fix-broken install -y -q || true + + # Try normal removal first + print_status "Attempting normal removal of Amazon packages..." + print_status "Removing only amazon-q package (other Amazon packages are snaps)..." + DEBIAN_FRONTEND=noninteractive sudo apt remove --purge -y -q amazon-q || true + + # Force remove if normal removal failed + print_status "Force removing Amazon packages..." + sudo dpkg --remove --force-remove-reinstreq amazon-q 2>/dev/null || true + + # Alternative: Install the missing dependency to resolve the conflict + print_status "Installing missing WebKit dependency to resolve conflict..." + DEBIAN_FRONTEND=noninteractive sudo apt install -y -q libwebkit2gtk-4.1-0 || true + + # Clean up any remaining broken dependencies + print_status "Final cleanup of broken dependencies..." + DEBIAN_FRONTEND=noninteractive sudo apt --fix-broken install -y -q || true + DEBIAN_FRONTEND=noninteractive sudo apt autoremove -y -q || true + DEBIAN_FRONTEND=noninteractive sudo apt autoclean || true + + # Update package lists + sudo apt update -q || true + + # Verify the fix worked + if DEBIAN_FRONTEND=noninteractive sudo apt install -y -q curl >/dev/null 2>&1; then + print_success "Package cleanup completed successfully" + else + print_warning "Package cleanup completed with warnings - some issues may persist" + fi + else + print_status "No conflicting Amazon packages found, performing standard cleanup..." + DEBIAN_FRONTEND=noninteractive sudo apt --fix-broken install -y -q || true + DEBIAN_FRONTEND=noninteractive sudo apt autoremove -y -q || true + sudo apt update -q || true + print_success "Standard cleanup completed" + fi +} + +# Check if running as root +if [[ $EUID -eq 0 ]]; then + print_error "This script should not be run as root" + exit 1 +fi + +# Detect OS +if [[ "$OSTYPE" == "linux-gnu"* ]]; then + OS="linux" +elif [[ "$OSTYPE" == "darwin"* ]]; then + OS="mac" +else + print_error "Unsupported OS: $OSTYPE" + exit 1 +fi + +print_status "Detected OS: $OS" + +# ============================================================================= +# PRE-FLIGHT: FIX ANY EXISTING DEPENDENCY ISSUES +# ============================================================================= + +if [[ "$OS" == "linux" ]]; then + print_status "Configuring non-interactive installation mode..." + + # Set non-interactive mode to prevent prompts during package installation + export DEBIAN_FRONTEND=noninteractive + export NEEDRESTART_MODE=a + export NEEDRESTART_SUSPEND=1 + + # Prevent automatic service restarts during package installation + echo '$nrconf{restart} = "a";' | sudo tee /etc/needrestart/conf.d/50local.conf >/dev/null 2>&1 || true + + # Preconfigure keyboard to avoid interactive prompt + print_status "Preconfiguring keyboard layout (US English)..." + echo 'keyboard-configuration keyboard-configuration/layoutcode string us' | sudo debconf-set-selections + echo 'keyboard-configuration keyboard-configuration/modelcode string pc105' | sudo debconf-set-selections + echo 'keyboard-configuration keyboard-configuration/variant select USA' | sudo debconf-set-selections + echo 'keyboard-configuration keyboard-configuration/layout select English (US)' | sudo debconf-set-selections + echo 'keyboard-configuration keyboard-configuration/store_defaults_in_debconf_db boolean true' | sudo debconf-set-selections + + # Preconfigure other common interactive prompts + echo 'tzdata tzdata/Areas select America' | sudo debconf-set-selections + echo 'tzdata tzdata/Zones/America select New_York' | sudo debconf-set-selections + echo 'locales locales/locales_to_be_generated multiselect en_US.UTF-8 UTF-8' | sudo debconf-set-selections + echo 'locales locales/default_environment_locale select en_US.UTF-8' | sudo debconf-set-selections + + # Preconfigure display manager (lightdm for i3 setup) + echo 'lightdm shared/default-x-display-manager select lightdm' | sudo debconf-set-selections + echo 'gdm3 shared/default-x-display-manager select lightdm' | sudo debconf-set-selections + + # Preconfigure other common interactive packages + echo 'wireshark-common wireshark-common/install-setuid boolean false' | sudo debconf-set-selections + echo 'console-setup console-setup/charmap47 select UTF-8' | sudo debconf-set-selections + echo 'console-setup console-setup/codeset47 select # Latin1 and Latin5 - western Europe and Turkic languages' | sudo debconf-set-selections + echo 'console-setup console-setup/codesetcode string Lat15' | sudo debconf-set-selections + echo 'console-setup console-setup/fontface47 select Fixed' | sudo debconf-set-selections + echo 'console-setup console-setup/fontsize-text47 select 16' | sudo debconf-set-selections + echo 'console-setup console-setup/fontsize-fb47 select 16' | sudo debconf-set-selections + + print_status "Performing pre-flight dependency check..." + + # Clean up duplicate repositories first + cleanup_duplicate_repositories + + # Force remove any remaining duplicate repository files + print_status "Removing any remaining duplicate repository files..." + sudo rm -f /etc/apt/sources.list.d/archive_uri-https_apt_releases_hashicorp_com-*.list + sudo rm -f /etc/apt/sources.list.d/archive_uri-https_cli_github_com_packages-*.list + sudo rm -f /etc/apt/sources.list.d/archive_uri-https_download_docker_com_linux_ubuntu-*.list + + # Check for broken dependencies + if ! DEBIAN_FRONTEND=noninteractive sudo apt install -y -q curl >/dev/null 2>&1; then + print_warning "Detected broken dependencies, attempting to fix..." + cleanup_conflicting_packages + else + print_success "No dependency issues detected" + fi +fi + +# ============================================================================= +# PART 1: SYSTEM PREREQUISITES +# ============================================================================= + +print_status "Installing system prerequisites..." + +if [[ "$OS" == "linux" ]]; then + # Aggressively clean up conflicting packages first + cleanup_conflicting_packages + + # Install basic tools + install_package "git" "git" + install_package "vim" "vim" + install_package "jq" "jq" + install_package "software-properties-common" "software-properties-common" + install_package "curl" "curl" + install_package "wget" "wget" + + # Install Docker + print_status "Installing Docker..." + DEBIAN_FRONTEND=noninteractive sudo apt install -y -q ca-certificates curl + sudo install -m 0755 -d /etc/apt/keyrings + sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc + sudo chmod a+r /etc/apt/keyrings/docker.asc + + echo \ + "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \ + $(. /etc/os-release && echo "${UBUNTU_CODENAME:-$VERSION_CODENAME}") stable" | \ + sudo tee /etc/apt/sources.list.d/docker.list > /dev/null + + sudo apt update -q + install_package "docker-ce" "Docker CE" + install_package "docker-ce-cli" "Docker CLI" + install_package "containerd.io" "containerd" + install_package "docker-buildx-plugin" "Docker Buildx" + install_package "docker-compose-plugin" "Docker Compose" + + # Add user to docker group + sudo groupadd docker 2>/dev/null || true + sudo usermod -aG docker $USER + + # Install kubectl + print_status "Installing kubectl..." + curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl" + chmod +x kubectl + sudo mv kubectl /usr/local/bin/ + + # Create .kube directory with proper permissions + print_status "Setting up kubectl configuration directory..." + mkdir -p ~/.kube + chmod 755 ~/.kube + # Ensure the user owns the .kube directory + chown $USER:$USER ~/.kube 2>/dev/null || true + + # Create other common directories that might be needed + print_status "Setting up additional user directories..." + mkdir -p ~/.local/bin ~/.cache ~/.config + chmod 755 ~/.local/bin ~/.cache ~/.config + chown $USER:$USER ~/.local ~/.cache ~/.config 2>/dev/null || true + + # Install Vault + print_status "Installing Vault..." + # Clean up any existing HashiCorp repositories to avoid duplicates + sudo rm -f /etc/apt/sources.list.d/archive_uri-https_apt_releases_hashicorp_com-*.list + sudo rm -f /etc/apt/sources.list.d/hashicorp.list + + # Use modern keyring method instead of deprecated apt-key + curl -fsSL https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor --yes -o /etc/apt/keyrings/hashicorp-archive-keyring.gpg + echo "deb [signed-by=/etc/apt/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list + sudo apt update -q + install_package "vault" "HashiCorp Vault" + + # Install GitHub CLI + print_status "Installing GitHub CLI..." + curl -fsSL https://cli.github.com/packages/githubcli-archive-keyring.gpg | sudo dd of=/usr/share/keyrings/githubcli-archive-keyring.gpg + sudo chmod go+r /usr/share/keyrings/githubcli-archive-keyring.gpg + echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/githubcli-archive-keyring.gpg] https://cli.github.com/packages stable main" | sudo tee /etc/apt/sources.list.d/github-cli.list > /dev/null + sudo apt update -q + install_package "gh" "GitHub CLI" + + # Install K9s + print_status "Installing K9s..." + run_command "Downloading K9s" "wget https://github.com/derailed/k9s/releases/download/v0.50.12/k9s_linux_amd64.deb" + run_command "Installing K9s" "sudo dpkg -i k9s_linux_amd64.deb || DEBIAN_FRONTEND=noninteractive sudo apt --fix-broken install -y -q" + rm -f k9s_linux_amd64.deb + +elif [[ "$OS" == "mac" ]]; then + # Check if Homebrew is installed + if ! command -v brew &> /dev/null; then + print_status "Installing Homebrew..." + /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)" + fi + + # Install tools via Homebrew + brew install git docker kind kubectl vault gh k9s +fi + +# Install IDPBuilder +print_status "Installing IDPBuilder..." +if [[ "$OS" == "mac" ]]; then + brew install cnoe-io/tap/idpbuilder +else + arch=$(if [[ "$(uname -m)" == "x86_64" ]]; then echo "amd64"; else uname -m; fi) + os=$(uname -s | tr '[:upper:]' '[:lower:]') + idpbuilder_latest_tag=$(curl --silent "https://api.github.com/repos/cnoe-io/idpbuilder/releases/latest" | grep '"tag_name":' | sed -E 's/.*"([^"]+)".*/\1/') + curl -LO https://github.com/cnoe-io/idpbuilder/releases/download/$idpbuilder_latest_tag/idpbuilder-$os-$arch.tar.gz + tar xvzf idpbuilder-$os-$arch.tar.gz + chmod +x idpbuilder + sudo mv idpbuilder /usr/local/bin + rm idpbuilder-linux-amd64.tar.gz LICENSE README.md 2>/dev/null || true +fi + +# Install Kind +print_status "Installing Kind..." +if [[ "$OS" == "mac" ]]; then + brew install kind +else + curl -Lo ./kind https://kind.sigs.k8s.io/dl/v0.20.0/kind-linux-amd64 + chmod +x ./kind + sudo mv ./kind /usr/local/bin/kind +fi + +print_success "System prerequisites installed!" + +# ============================================================================= +# GIT AND GITHUB CLI SETUP +# ============================================================================= + +print_status "Setting up Git and GitHub CLI..." + +# Check if git is configured +if ! git config --global user.name >/dev/null 2>&1 || ! git config --global user.email >/dev/null 2>&1; then + print_warning "Git is not configured yet. You'll need to set up your git identity:" + echo " Run these commands to configure git:" + echo " git config --global user.name \"Your Name\"" + echo " git config --global user.email \"your.email@example.com\"" + echo "" +else + print_success "Git is already configured" + echo " Name: $(git config --global user.name)" + echo " Email: $(git config --global user.email)" + echo "" +fi + +# Check GitHub CLI authentication +if command -v gh &> /dev/null; then + if gh auth status >/dev/null 2>&1; then + print_success "GitHub CLI is already authenticated" + else + print_warning "GitHub CLI is installed but not authenticated" + echo " To authenticate with GitHub, run:" + echo " gh auth login" + echo " This will guide you through the authentication process" + echo "" + fi +fi + +# ============================================================================= +# PART 2: i3 DESKTOP ENVIRONMENT SETUP +# ============================================================================= + +if [[ "$OS" == "linux" ]]; then + print_status "Setting up i3 desktop environment..." + + # Remove GNOME (if present) + sudo apt remove --purge ubuntu-desktop gnome-shell gnome-session gdm3 -y 2>/dev/null || true + sudo apt autoremove --purge -y + + # Remove Firefox snap and configure Mozilla PPA for better i3 compatibility + print_status "Removing Firefox snap packages for i3 compatibility..." + if snap list | grep -q firefox; then + print_status "Found Firefox snap, removing..." + sudo snap remove firefox 2>/dev/null || true + print_success "Firefox snap removed" + else + print_status "No Firefox snap found, continuing..." + fi + + # Add Mozilla PPA repository for newer Firefox versions + print_status "Adding Mozilla PPA repository..." + sudo add-apt-repository -y ppa:mozillateam/ppa + sudo apt update -q + + # Set apt priority over snap to prevent future snap reinstalls + print_status "Configuring apt priority to prevent snap Firefox reinstall..." + cat << 'EOF' | sudo tee /etc/apt/preferences.d/mozilla-firefox > /dev/null +Package: * +Pin: release o=LP-PPA-mozillateam +Pin-Priority: 1001 +EOF + print_success "Mozilla PPA configured with priority over snap" + + # Install i3 and VNC packages + # Install required dependencies for webkit first + install_package "libwebkit2gtk-4.1-0" "WebKit dependencies" + + install_package "i3" "i3 window manager" + install_package "i3status" "i3 status bar" + install_package "i3lock" "i3 screen locker" + install_package "dmenu" "dmenu" + install_package "rofi" "rofi launcher" + install_package "xorg" "X.Org server" + install_package "lightdm" "LightDM display manager" + install_package "xterm" "xterm terminal" + install_package "terminator" "Terminator terminal" + install_package "xclip" "xclip clipboard utility" + install_package "parcellite" "Parcellite clipboard manager" + # Install Firefox from Mozilla PPA with downgrade allowance + print_status "Installing Firefox from Mozilla PPA..." + if DEBIAN_FRONTEND=noninteractive sudo apt install -y -q --allow-downgrades firefox; then + print_success "Firefox installed successfully from Mozilla PPA" + else + print_warning "Failed to install Firefox from PPA, trying standard installation..." + install_package "firefox" "Firefox browser (fallback)" + fi + + # Configure AppArmor for Firefox to prevent security policy violations + print_status "Configuring AppArmor for Firefox compatibility..." + if command -v aa-status &> /dev/null; then + if aa-status | grep -q "firefox"; then + print_status "Setting Firefox AppArmor profile to complain mode..." + sudo aa-complain snap.firefox.firefox 2>/dev/null || true + sudo aa-complain /usr/bin/firefox 2>/dev/null || true + print_success "AppArmor configured for Firefox" + else + print_status "No Firefox AppArmor profile found, skipping..." + fi + else + print_status "AppArmor not installed, installing..." + install_package "apparmor-utils" "AppArmor utilities" + fi + + install_package "tigervnc-standalone-server" "TigerVNC server" + install_package "openbox" "Openbox window manager (fallback)" + + # Verify i3 installation immediately after package installation + print_status "Verifying i3 packages installation..." + if ! command -v i3 &> /dev/null; then + print_warning "i3 not found after installation, trying alternative packages..." + install_package "i3-wm" "i3-wm window manager (alternative)" + # Force refresh PATH + export PATH="/usr/bin:/usr/local/bin:$PATH" + hash -r 2>/dev/null || true + fi + + # Final verification with detailed output + if command -v i3 &> /dev/null; then + print_success "i3 is available: $(which i3)" + print_status "i3 version: $(i3 --version 2>/dev/null || echo 'version check failed')" + else + print_warning "i3 still not found, VNC will use fallback window manager" + fi + + # Create i3 config + print_status "Creating i3 configuration..." + mkdir -p ~/.config/i3 + cat > ~/.config/i3/config << 'EOF' +# i3 config - Mac compatible (Alt key) +set $mod Mod1 +font pango:monospace 8 +floating_modifier $mod + +# Terminal shortcuts +bindsym $mod+Return exec terminator +bindsym $mod+t exec terminator + +# Application shortcuts +bindsym $mod+Shift+q kill +bindsym $mod+d exec rofi -show run +bindsym $mod+space exec rofi -show drun +bindsym $mod+f exec "bash -c 'if [ -x ~/.local/bin/firefox-i3 ]; then ~/.local/bin/firefox-i3; else firefox --no-sandbox; fi'" + +# Navigation +bindsym $mod+Left focus left +bindsym $mod+Down focus down +bindsym $mod+Up focus up +bindsym $mod+Right focus right + +# Move windows +bindsym $mod+Shift+Left move left +bindsym $mod+Shift+Down move down +bindsym $mod+Shift+Up move up +bindsym $mod+Shift+Right move right + +# Splits and layout +bindsym $mod+h split h +bindsym $mod+v split v +bindsym $mod+F11 fullscreen toggle +bindsym $mod+Shift+space floating toggle +bindsym $mod+Tab focus mode_toggle + +# Workspaces +set $ws1 "1" +set $ws2 "2" +set $ws3 "3" +set $ws4 "4" +set $ws5 "5" + +bindsym $mod+1 workspace number $ws1 +bindsym $mod+2 workspace number $ws2 +bindsym $mod+3 workspace number $ws3 +bindsym $mod+4 workspace number $ws4 +bindsym $mod+5 workspace number $ws5 + +bindsym $mod+Shift+1 move container to workspace number $ws1 +bindsym $mod+Shift+2 move container to workspace number $ws2 +bindsym $mod+Shift+3 move container to workspace number $ws3 +bindsym $mod+Shift+4 move container to workspace number $ws4 +bindsym $mod+Shift+5 move container to workspace number $ws5 + +# System +bindsym $mod+Shift+c reload +bindsym $mod+Shift+r restart +bindsym $mod+Shift+e exec "i3-nagbar -t warning -m 'Exit i3?' -B 'Yes' 'i3-msg exit'" + +# Firefox-specific window management for i3 compatibility +for_window [class="firefox"] floating disable +for_window [class="Firefox"] floating disable +for_window [class="Firefox-esr"] floating disable + +# Status bar +bar { + status_command i3status +} +EOF + + # Verify i3 installation before creating VNC startup script + print_status "Verifying i3 installation..." + if ! command -v i3 &> /dev/null; then + print_warning "i3 not found in PATH, attempting to reinstall..." + install_package "i3" "i3 window manager (retry)" + + # If still not found, check common installation locations + if ! command -v i3 &> /dev/null; then + print_warning "i3 still not found, checking installation paths..." + if [ -f "/usr/bin/i3" ]; then + print_success "Found i3 at /usr/bin/i3" + elif [ -f "/usr/local/bin/i3" ]; then + print_success "Found i3 at /usr/local/bin/i3" + else + print_error "i3 installation verification failed" + print_status "Installing i3-wm as alternative..." + install_package "i3-wm" "i3-wm window manager" + fi + fi + else + print_success "i3 is properly installed and accessible" + fi + + # Create VNC startup script + print_status "Setting up VNC..." + mkdir -p ~/.vnc + cat > ~/.vnc/xstartup << 'EOF' +#!/bin/bash +export DISPLAY=:1 +xhost +local: & +xsetroot -solid grey & +parcellite & +terminator -g 80x24+10+10 & +# Use optimized Firefox startup script if available, fallback to regular Firefox +if [ -x ~/.local/bin/firefox-i3 ]; then + ~/.local/bin/firefox-i3 & +else + firefox --no-sandbox & +fi +exec i3 +EOF + chmod +x ~/.vnc/xstartup + + print_success "i3 desktop environment configured!" +fi + +# ============================================================================= +# FIREFOX TROUBLESHOOTING UTILITIES +# ============================================================================= + +# Function to clean up Firefox cache and temporary files +cleanup_firefox_cache() { + print_status "Cleaning up Firefox cache and temporary files..." + + # Kill any hanging Firefox processes + pkill -f firefox 2>/dev/null || true + + # Clean up Firefox cache directories + if [ -d "$HOME/.cache/mozilla/firefox/" ]; then + print_status "Removing Firefox cache directory..." + rm -rf "$HOME/.cache/mozilla/firefox/" || true + fi + + # Clean up temporary Firefox files + print_status "Removing temporary Firefox files..." + rm -rf /tmp/firefox* 2>/dev/null || true + rm -rf /tmp/rust_mozprofile* 2>/dev/null || true + + print_success "Firefox cache cleanup completed" +} + +# Create Firefox startup script with i3-optimized settings +create_firefox_startup_script() { + print_status "Creating Firefox startup script for i3 compatibility..." + + mkdir -p ~/.local/bin + cat > ~/.local/bin/firefox-i3 << 'EOF' +#!/bin/bash +# Firefox startup script optimized for i3 window manager +# This script addresses common Firefox/i3 compatibility issues + +# Clean up any hanging Firefox processes +pkill -f firefox 2>/dev/null || true + +# Clear cache if Firefox previously crashed +if [ -f ~/.mozilla/firefox/*/sessionstore-backups/recovery.jsonlz4 ]; then + echo "Detected Firefox crash recovery files, cleaning up..." + rm -rf ~/.cache/mozilla/firefox/ 2>/dev/null || true +fi + +# Start Firefox with i3-optimized flags +exec firefox \ + --no-sandbox \ + --disable-gpu-sandbox \ + --disable-seccomp-filter-sandbox \ + --disable-namespace-sandbox \ + --disable-setuid-sandbox \ + "$@" +EOF + chmod +x ~/.local/bin/firefox-i3 + + # Create a desktop entry for the optimized Firefox + mkdir -p ~/.local/share/applications + cat > ~/.local/share/applications/firefox-i3.desktop << 'EOF' +[Desktop Entry] +Name=Firefox (i3 Optimized) +Comment=Firefox Web Browser optimized for i3 window manager +GenericName=Web Browser +X-GNOME-FullName=Firefox Web Browser (i3 Optimized) +Exec=/home/%u/.local/bin/firefox-i3 %u +Terminal=false +X-MultipleArgs=false +Type=Application +Icon=firefox +Categories=GNOME;GTK;Network;WebBrowser; +MimeType=text/html;text/xml;application/xhtml+xml;application/xml;application/rss+xml;application/rdf+xml;image/gif;image/jpeg;image/png;x-scheme-handler/http;x-scheme-handler/https;x-scheme-handler/ftp;x-scheme-handler/chrome;video/webm;application/x-xpinstall; +StartupNotify=true +Actions=NewWindow;NewPrivateWindow; + +[Desktop Action NewWindow] +Name=Open a New Window +Exec=/home/%u/.local/bin/firefox-i3 -new-window + +[Desktop Action NewPrivateWindow] +Name=Open a New Private Window +Exec=/home/%u/.local/bin/firefox-i3 -private-window +EOF + + print_success "Firefox i3 startup script created at ~/.local/bin/firefox-i3" +} + +if [[ "$OS" == "linux" ]]; then + # Run Firefox optimizations + cleanup_firefox_cache + create_firefox_startup_script +fi + +# ============================================================================= +# PART 3: IDPBuilder CLUSTER CREATION +# ============================================================================= + +print_status "IDPBuilder cluster creation will be handled separately by the user" +print_status "To create a cluster, run: idpbuilder create --use-path-routing --package " + +# ============================================================================= +# PART 4: VERIFICATION AND ACCESS INFORMATION +# ============================================================================= + +print_status "Verifying system setup..." + +# Final cleanup and verification +print_status "Performing final cleanup and verification..." + +# Fix any remaining broken dependencies +DEBIAN_FRONTEND=noninteractive sudo apt --fix-broken install -y -q || true + +# Clean up package cache +DEBIAN_FRONTEND=noninteractive sudo apt autoremove -y -q +DEBIAN_FRONTEND=noninteractive sudo apt autoclean + +# Verify critical tools are installed +print_status "Verifying installation..." +for tool in git docker kubectl vault gh k9s idpbuilder kind; do + if command -v "$tool" &> /dev/null; then + print_success "$tool is installed" + else + print_warning "$tool is not installed or not in PATH" + fi +done + +print_success "Prerequisites setup complete! ๐ŸŽ‰" + +# ============================================================================= +# PART 5: VNC SETUP (FINAL STEP) +# ============================================================================= + +print_status "Setting up VNC access..." + +# # Set VNC password +# print_status "Setting VNC password (you'll be prompted)..." +# vncpasswd + +# Start VNC server +print_status "Starting VNC server..." + +# Check if VNC server is already running +if pgrep -f "Xtigervnc.*:1" > /dev/null || pgrep -f "vncserver.*:1" > /dev/null; then + print_success "VNC server is already running on display :1" + echo " (Detected existing VNC process)" +else + # Start VNC server with timeout to prevent hanging + print_status "Attempting to start VNC server..." + if timeout 10 vncserver :1 -geometry 2560x1400 -depth 24 -localhost yes 2>/dev/null; then + print_success "VNC server started successfully" + else + print_warning "VNC server startup timed out or failed, but continuing..." + echo " (This is normal if VNC was already running)" + fi +fi +echo "" +echo "๐Ÿ–ฅ๏ธ VNC Desktop Access:" +echo " Start VNC: vncserver :1 -geometry 2560x1400 -depth 24 -localhost yes" +echo " SSH Tunnel: ssh -i ~/.ssh/private.pem -L 5903:localhost:5901 ubuntu@ -f -N" +echo " VNC Client: Connect to localhost:5903" +echo "" + +echo "======================================================================" +echo " ๐Ÿ–ฅ๏ธ VNC ACCESS INSTRUCTIONS & SECURITY RECOMMENDATIONS " +echo "======================================================================" +echo "" +echo "๐Ÿ”‘ NOTE: You must set a VNC password before connecting with TigerVNC, VNC Viewer, or using screen sharing clients." +echo " To set your VNC password, run:" +echo " vncpasswd" +echo "" +echo "๐Ÿ’ป To connect from your local machine:" +echo " - On Mac:" +echo " 1. Open Finder, press Cmd+K, and enter: vnc://localhost:5903" +echo " 2. Or use a VNC client like TigerVNC or RealVNC Viewer and connect to localhost:5903" +echo " - On Windows:" +echo " 1. Download and install TigerVNC or RealVNC Viewer" +echo " 2. Connect to: localhost:5903" +echo "" +echo "๐Ÿ”’ For better security and compression, tunnel VNC via SSH:" +echo " Example command:" +echo " ssh -i ~/.ssh/private.pem -L 5903:localhost:5901 ubuntu@ -f -N" +echo " This forwards your local port 5903 to the remote VNC server's port 5901." +echo " Then connect your VNC client to localhost:5903." +echo "" +echo " (Make sure to set up the SSH tunnel as shown above before connecting!)" +echo "" +echo "๐Ÿ”ง Git and GitHub CLI Setup:" +echo " If git is not configured, run:" +echo " git config --global user.name \"Your Name\"" +echo " git config --global user.email \"your.email@example.com\"" +echo "" +echo " To authenticate GitHub CLI, run:" +echo " gh auth login" +echo " This enables you to clone repositories, create issues, and manage GitHub resources from the command line." +echo "" +echo "๐Ÿ› VNC Troubleshooting:" +echo " If VNC fails with 'i3: not found' error:" +echo " 1. Verify i3 installation: which i3" +echo " 2. Check if i3-wm is available: which i3-wm" +echo " 3. Test VNC startup manually: bash ~/.vnc/xstartup" +echo " 4. Use fallback window manager: tigervncserver -xstartup /usr/bin/xterm" +echo " 5. Check VNC logs: cat ~/.vnc/*.log" +echo "" +echo " If you need to restart VNC server:" +echo " vncserver -kill :1" +echo " vncserver :1 -geometry 2560x1400 -depth 24 -localhost yes" +echo "" +echo "๐Ÿ”ง Firefox/i3 Troubleshooting:" +echo " If Firefox freezes or crashes in i3:" +echo " 1. Kill hanging Firefox processes: pkill -f firefox" +echo " 2. Clear Firefox cache: rm -rf ~/.cache/mozilla/firefox/" +echo " 3. Clear temporary files: rm -rf /tmp/firefox*" +echo " 4. Use optimized Firefox launcher: ~/.local/bin/firefox-i3" +echo " 5. Check AppArmor status: sudo aa-status | grep firefox" +echo "" +echo " If Firefox snap causes issues:" +echo " 1. Remove Firefox snap: sudo snap remove firefox" +echo " 2. Add Mozilla PPA: sudo add-apt-repository -y ppa:mozillateam/ppa" +echo " 3. Set apt priority: echo 'Package: * Pin: release o=LP-PPA-mozillateam Pin-Priority: 1001' | sudo tee /etc/apt/preferences.d/mozilla-firefox" +echo " 4. Install Firefox: sudo apt install -y --allow-downgrades firefox" +echo " 5. Configure AppArmor: sudo aa-complain /usr/bin/firefox" +echo "" +echo " Firefox optimized launcher available at: ~/.local/bin/firefox-i3" +echo " This launcher includes --no-sandbox and other i3-compatible flags" +echo "" +echo "๐Ÿ”ง KUBECONFIG Permission Issues:" +echo " If you get 'permission denied' when writing KUBECONFIG:" +echo " mkdir -p ~/.kube" +echo " chmod 755 ~/.kube" +echo " chown \$USER:\$USER ~/.kube" +echo " Or if the config file exists but is not writable:" +echo " chmod 644 ~/.kube/config" +echo " chown \$USER:\$USER ~/.kube/config" +echo "" +echo "======================================================================" diff --git a/caipe/scripts/sync-apps.sh b/caipe/scripts/sync-apps.sh new file mode 100755 index 00000000..cc4ca89f --- /dev/null +++ b/caipe/scripts/sync-apps.sh @@ -0,0 +1,191 @@ +#!/bin/bash + +set -euo pipefail + +# Colors for output +RED='\033[0;31m' +GREEN='\033[0;32m' +YELLOW='\033[1;33m' +BLUE='\033[0;34m' +NC='\033[0m' # No Color + +log() { + echo -e "${BLUE}[$(date +'%Y-%m-%d %H:%M:%S')]${NC} $1" +} + +success() { + echo -e "${GREEN}โœ… $1${NC}" +} + +warn() { + echo -e "${YELLOW}โš ๏ธ $1${NC}" +} + +error() { + echo -e "${RED}โŒ $1${NC}" +} + +# Check if kubectl is available +if ! command -v kubectl &> /dev/null; then + error "kubectl is not installed or not in PATH" + exit 1 +fi + +# Check if argocd CLI is available +if ! command -v argocd &> /dev/null; then + warn "argocd CLI not found, will use kubectl for ArgoCD operations" + USE_KUBECTL=true +else + USE_KUBECTL=false +fi + +# ArgoCD applications to sync +APPS=( + "backstage" + "vault" + "argocd" + "ai-platform-engineering" + "external-secrets" + "ingress-nginx" + "gitea" +) + +# Function to check if app exists +app_exists() { + local app_name="$1" + kubectl get application "$app_name" -n argocd >/dev/null 2>&1 +} + +# Function to get app sync status +get_app_status() { + local app_name="$1" + kubectl get application "$app_name" -n argocd -o jsonpath='{.status.sync.status}' 2>/dev/null || echo "Unknown" +} + +# Function to get app health status +get_app_health() { + local app_name="$1" + kubectl get application "$app_name" -n argocd -o jsonpath='{.status.health.status}' 2>/dev/null || echo "Unknown" +} + +# Function to sync app using kubectl +sync_app_kubectl() { + local app_name="$1" + log "Syncing $app_name using kubectl..." + + # Trigger sync by adding annotation + kubectl annotate application "$app_name" -n argocd argocd.argoproj.io/refresh=normal --overwrite + + # Wait a moment for the annotation to take effect + sleep 2 + + # Remove the annotation + kubectl annotate application "$app_name" -n argocd argocd.argoproj.io/refresh- || true +} + +# Function to sync app using argocd CLI +sync_app_argocd() { + local app_name="$1" + log "Syncing $app_name using argocd CLI..." + + # Login to ArgoCD (assuming port-forward is available) + argocd login argocd.cnoe.localtest.me:8443 --username admin --password "$(kubectl get secret argocd-initial-admin-secret -n argocd -o jsonpath='{.data.password}' | base64 -d)" --insecure + + # Sync the application + argocd app sync "$app_name" --timeout 300 +} + +# Function to wait for app to be synced and healthy +wait_for_app_sync() { + local app_name="$1" + local timeout=300 + local count=0 + + log "Waiting for $app_name to sync and become healthy..." + + while [[ $count -lt $timeout ]]; do + local sync_status=$(get_app_status "$app_name") + local health_status=$(get_app_health "$app_name") + + if [[ "$sync_status" == "Synced" && "$health_status" == "Healthy" ]]; then + success "$app_name is synced and healthy" + return 0 + fi + + if [[ $((count % 30)) -eq 0 ]]; then + log "$app_name status: sync=$sync_status, health=$health_status" + fi + + sleep 5 + ((count+=5)) + done + + warn "$app_name did not become synced and healthy within ${timeout}s" + return 1 +} + +# Main sync process +log "Starting ArgoCD application sync process..." + +# Check if ArgoCD is available +if ! kubectl get namespace argocd >/dev/null 2>&1; then + error "ArgoCD namespace not found. Is ArgoCD installed?" + exit 1 +fi + +echo "" +log "Checking application status before sync..." + +# Show current status +for app in "${APPS[@]}"; do + if app_exists "$app"; then + sync_status=$(get_app_status "$app") + health_status=$(get_app_health "$app") + log "$app: sync=$sync_status, health=$health_status" + else + warn "$app: Application not found" + fi +done + +echo "" +log "Starting sync process..." + +# Sync each application +for app in "${APPS[@]}"; do + if app_exists "$app"; then + echo "" + log "Processing application: $app" + + if [[ "$USE_KUBECTL" == "true" ]]; then + sync_app_kubectl "$app" + else + sync_app_argocd "$app" + fi + + # Wait for sync to complete + wait_for_app_sync "$app" + else + warn "Skipping $app - application not found" + fi +done + +echo "" +log "Final application status check..." + +# Show final status +for app in "${APPS[@]}"; do + if app_exists "$app"; then + sync_status=$(get_app_status "$app") + health_status=$(get_app_health "$app") + + if [[ "$sync_status" == "Synced" && "$health_status" == "Healthy" ]]; then + success "$app: sync=$sync_status, health=$health_status" + else + warn "$app: sync=$sync_status, health=$health_status" + fi + fi +done + +echo "" +success "Application sync process completed!" +log "All available applications have been processed" diff --git a/caipe/scripts/sync-github-to-gitea.sh b/caipe/scripts/sync-github-to-gitea.sh new file mode 100755 index 00000000..cc6272d4 --- /dev/null +++ b/caipe/scripts/sync-github-to-gitea.sh @@ -0,0 +1,87 @@ +#!/bin/bash + +set -e + +# Configuration +GITEA_URL="https://cnoe.localtest.me:8443/gitea" +GITEA_USER="giteaAdmin" +GITHUB_REPO="https://github.com/sriaradhyula/stacks.git" +TEMP_DIR="/tmp/stacks-sync" + +echo "๐Ÿ”„ Syncing GitHub stacks to Gitea repositories..." + +# Get Gitea admin password +GITEA_PASSWORD=$(kubectl get secret gitea-credential -n gitea -o jsonpath='{.data.password}' | base64 -d) + +# Clone latest from GitHub +echo "๐Ÿ“ฅ Cloning latest from GitHub..." +rm -rf $TEMP_DIR +git clone $GITHUB_REPO $TEMP_DIR + +# Function to sync a specific path to Gitea repo +sync_to_gitea() { + local path=$1 + local repo_name=$2 + + echo "๐Ÿ”„ Syncing $path to $repo_name..." + + # Check if Gitea repo exists + if curl -k -s -u "$GITEA_USER:$GITEA_PASSWORD" "$GITEA_URL/api/v1/repos/$GITEA_USER/$repo_name" > /dev/null 2>&1; then + echo "๐Ÿ“‚ Repository $repo_name exists, updating..." + + # Clone Gitea repo using kubectl port-forward + local gitea_dir="/tmp/gitea-$repo_name" + rm -rf $gitea_dir + + # Start port-forward in background + kubectl port-forward -n gitea svc/my-gitea-http 3000:3000 & + local pf_pid=$! + sleep 3 + + # Clone using localhost + git -c http.sslVerify=false clone "http://$GITEA_USER:$GITEA_PASSWORD@localhost:3000/$GITEA_USER/$repo_name.git" $gitea_dir + + # Kill port-forward + kill $pf_pid 2>/dev/null || true + + # Clear existing content and copy new + cd $gitea_dir + find . -maxdepth 1 ! -name '.git' ! -name '.' -exec rm -rf {} + + cp -r "$TEMP_DIR/$path"/* . + + # Commit and push changes + git add . + if git diff --staged --quiet; then + echo "โœ… No changes to sync for $repo_name" + else + git commit -m "Sync from GitHub $(date)" + + # Start port-forward for push + kubectl port-forward -n gitea svc/my-gitea-http 3000:3000 & + local pf_pid2=$! + sleep 3 + + git push origin main + + # Kill port-forward + kill $pf_pid2 2>/dev/null || true + + echo "โœ… Synced $repo_name successfully" + fi + + rm -rf $gitea_dir + else + echo "โŒ Repository $repo_name not found in Gitea" + fi +} + +# Sync backstage manifests +sync_to_gitea "caipe/base/backstage" "idpbuilder-localdev-backstage-manifests" + +# Sync ai-platform-engineering manifests +sync_to_gitea "caipe/base" "idpbuilder-localdev-ai-platform-engineering-ai-platform-engineering" + +# Clean up +rm -rf $TEMP_DIR + +echo "๐ŸŽ‰ GitHub to Gitea sync completed!" diff --git a/caipe/scripts/update-gitea-repos.sh b/caipe/scripts/update-gitea-repos.sh new file mode 100755 index 00000000..ad431737 --- /dev/null +++ b/caipe/scripts/update-gitea-repos.sh @@ -0,0 +1,35 @@ +#!/bin/bash + +set -e + +echo "๐Ÿ”„ Updating Gitea repositories with latest GitHub changes..." + +# Clone latest from GitHub +TEMP_DIR="/tmp/stacks-sync" +rm -rf $TEMP_DIR +git clone https://github.com/sriaradhyula/stacks.git $TEMP_DIR + +# Function to update ArgoCD application source +update_argocd_app() { + local app_name=$1 + echo "๐Ÿ”„ Refreshing $app_name application..." + kubectl patch application $app_name -n argocd --type merge -p '{"operation":{"sync":{"syncOptions":["CreateNamespace=true"]}}}' + kubectl patch application $app_name -n argocd --type merge -p '{"spec":{"source":{"targetRevision":"HEAD"}}}' +} + +# Force ArgoCD to refresh from source +echo "๐Ÿ”„ Forcing ArgoCD to refresh applications..." +kubectl patch application backstage -n argocd --type merge -p '{"metadata":{"annotations":{"argocd.argoproj.io/refresh":"hard"}}}' +kubectl patch application ai-platform-engineering -n argocd --type merge -p '{"metadata":{"annotations":{"argocd.argoproj.io/refresh":"hard"}}}' + +# Trigger manual sync +echo "๐Ÿ”„ Triggering manual sync..." +kubectl patch application backstage -n argocd --type merge -p '{"operation":{"sync":{"syncOptions":["CreateNamespace=true"]}}}' +kubectl patch application ai-platform-engineering -n argocd --type merge -p '{"operation":{"sync":{"syncOptions":["CreateNamespace=true"]}}}' + +echo "โœ… Repository refresh completed!" +echo "โ„น๏ธ Note: Changes will only appear if the source repositories in Gitea are updated." +echo "โ„น๏ธ For full sync, consider running: ./recreate-idpbuilder.sh" + +# Clean up +rm -rf $TEMP_DIR diff --git a/caipe/workshop/ai-platform-engineering.yaml b/caipe/workshop/ai-platform-engineering.yaml new file mode 100644 index 00000000..746a0b27 --- /dev/null +++ b/caipe/workshop/ai-platform-engineering.yaml @@ -0,0 +1,39 @@ +apiVersion: argoproj.io/v1alpha1 +kind: Application +metadata: + name: ai-platform-engineering + namespace: argocd + finalizers: + - resources-finalizer.argocd.argoproj.io +spec: + project: default + sources: + # Main chart from GHCR + - chart: ai-platform-engineering + repoURL: ghcr.io/cnoe-io/helm-charts + targetRevision: 0.2.2 + helm: + valueFiles: + - $values/values.yaml + - repoURL: cnoe://ai-platform-engineering + targetRevision: main + path: manifests + ref: values + destination: + server: https://kubernetes.default.svc + namespace: ai-platform-engineering + syncPolicy: + automated: + prune: true + selfHeal: true + syncOptions: + - CreateNamespace=true + - PrunePropagationPolicy=foreground + - PruneLast=true + retry: + limit: 5 + backoff: + duration: 5s + factor: 2 + maxDuration: 3m + revisionHistoryLimit: 10 diff --git a/caipe/workshop/ai-platform-engineering/manifests/ingress-graphrag-neo4j.yaml b/caipe/workshop/ai-platform-engineering/manifests/ingress-graphrag-neo4j.yaml new file mode 100644 index 00000000..6f822dbf --- /dev/null +++ b/caipe/workshop/ai-platform-engineering/manifests/ingress-graphrag-neo4j.yaml @@ -0,0 +1,47 @@ +apiVersion: networking.k8s.io/v1 +kind: Ingress +metadata: + name: graphrag-neo4j-ingress + namespace: ai-platform-engineering + annotations: + argocd.argoproj.io/sync-wave: "0" + nginx.ingress.kubernetes.io/rewrite-target: /$2 +spec: + ingressClassName: "nginx" + rules: + - host: localhost + http: + paths: + - path: /neo4j(/|$)(.*) + pathType: ImplementationSpecific + backend: + service: + name: graphrag-neo4j + port: + number: 7474 + - path: /nexigraph(/|$)(.*) + pathType: ImplementationSpecific + backend: + service: + name: ai-platform-engineering-graphrag-nexigraph-server + port: + number: 8095 + - host: cnoe.localtest.me + http: + paths: + - path: /neo4j(/|$)(.*) + pathType: ImplementationSpecific + backend: + service: + name: graphrag-neo4j + port: + number: 7474 + - path: /nexigraph(/|$)(.*) + pathType: ImplementationSpecific + backend: + service: + name: ai-platform-engineering-graphrag-nexigraph-server + port: + number: 8095 + + diff --git a/caipe/workshop/ai-platform-engineering/manifests/ingress.yaml b/caipe/workshop/ai-platform-engineering/manifests/ingress.yaml new file mode 100644 index 00000000..52c1e505 --- /dev/null +++ b/caipe/workshop/ai-platform-engineering/manifests/ingress.yaml @@ -0,0 +1,31 @@ +apiVersion: networking.k8s.io/v1 +kind: Ingress +metadata: + name: ai-platform-engineering-ingress + namespace: ai-platform-engineering + annotations: + argocd.argoproj.io/sync-wave: "0" + nginx.ingress.kubernetes.io/rewrite-target: /$2 +spec: + ingressClassName: "nginx" + rules: + - host: localhost + http: + paths: + - path: /ai-platform-engineering(/|$)(.*) + pathType: ImplementationSpecific + backend: + service: + name: ai-platform-engineering-supervisor-agent + port: + number: 8000 + - host: cnoe.localtest.me + http: + paths: + - path: /ai-platform-engineering(/|$)(.*) + pathType: ImplementationSpecific + backend: + service: + name: ai-platform-engineering-supervisor-agent + port: + number: 8000 diff --git a/caipe/workshop/ai-platform-engineering/manifests/presync-local-path-provisioner.yaml b/caipe/workshop/ai-platform-engineering/manifests/presync-local-path-provisioner.yaml new file mode 100644 index 00000000..4f4f5c48 --- /dev/null +++ b/caipe/workshop/ai-platform-engineering/manifests/presync-local-path-provisioner.yaml @@ -0,0 +1,59 @@ +apiVersion: v1 +kind: ServiceAccount +metadata: + name: local-path-provisioner-service-account + namespace: ai-platform-engineering + annotations: + argocd.argoproj.io/hook: PreSync + argocd.argoproj.io/sync-wave: "-2" +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRoleBinding +metadata: + name: local-path-provisioner-service-account-cluster-admin + annotations: + argocd.argoproj.io/hook: PreSync + argocd.argoproj.io/sync-wave: "-1" +subjects: + - kind: ServiceAccount + name: local-path-provisioner-service-account + namespace: ai-platform-engineering +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: cluster-admin +--- +apiVersion: batch/v1 +kind: Job +metadata: + name: install-local-path-provisioner + namespace: ai-platform-engineering + annotations: + argocd.argoproj.io/hook: PreSync + argocd.argoproj.io/sync-wave: "0" + argocd.argoproj.io/hook-delete-policy: HookSucceeded +spec: + backoffLimit: 1 + template: + spec: + serviceAccountName: local-path-provisioner-service-account + restartPolicy: Never + containers: + - name: kubectl + image: alpine/k8s:1.29.0 # move back to docker.io/bitnami/kubectl + imagePullPolicy: IfNotPresent + command: + - /bin/sh + - -c + - | + set -eu + echo "Applying local-path-provisioner manifests..." + kubectl apply -f https://raw.githubusercontent.com/rancher/local-path-provisioner/master/deploy/local-path-storage.yaml + echo "Waiting for local-path-provisioner to be ready..." + kubectl -n local-path-storage rollout status deployment/local-path-provisioner --timeout=180s || \ + kubectl -n local-path-storage rollout status daemonset/local-path-provisioner --timeout=180s || true + echo "Setting local-path as default StorageClass..." + kubectl annotate sc local-path storageclass.kubernetes.io/is-default-class=true --overwrite || true + echo "Current StorageClasses:" + kubectl get sc -o wide + diff --git a/caipe/workshop/ai-platform-engineering/values.yaml b/caipe/workshop/ai-platform-engineering/values.yaml new file mode 100644 index 00000000..9b1c9b3f --- /dev/null +++ b/caipe/workshop/ai-platform-engineering/values.yaml @@ -0,0 +1,597 @@ +# Global configuration shared across all subcharts +global: + + enabledSubAgents: + argocd: true + backstage: true + confluence: false + github: true + jira: false + pagerduty: false + slack: false + aws: false + weather: true + splunk: false + + slim: + enabled: false + endpoint: "http://ai-platform-engineering-slim:46357" + transport: "slim" + + externalSecrets: + enabled: true + secretStoreRef: + name: "vault-secret-store" + kind: ClusterSecretStore + + agentSecrets: + create: true + + llmSecrets: + create: false # do not create llm secret in subcharts (use global or existing llm secret) + secretName: "llm-secret" + externalSecrets: + data: + - secretKey: LLM_PROVIDER + remoteRef: + conversionStrategy: Default + decodingStrategy: None + key: ai-platform-engineering/global + property: LLM_PROVIDER + + # Azure OpenAI configuration + - secretKey: AZURE_OPENAI_API_KEY + optional: true + remoteRef: + conversionStrategy: Default + decodingStrategy: None + key: ai-platform-engineering/global + property: AZURE_OPENAI_API_KEY + - secretKey: AZURE_OPENAI_ENDPOINT + optional: true + remoteRef: + conversionStrategy: Default + decodingStrategy: None + key: ai-platform-engineering/global + property: AZURE_OPENAI_ENDPOINT + - secretKey: AZURE_OPENAI_API_VERSION + optional: true + remoteRef: + conversionStrategy: Default + decodingStrategy: None + key: ai-platform-engineering/global + property: AZURE_OPENAI_API_VERSION + - secretKey: AZURE_OPENAI_DEPLOYMENT + optional: true + remoteRef: + conversionStrategy: Default + decodingStrategy: None + key: ai-platform-engineering/global + property: AZURE_OPENAI_DEPLOYMENT + # OpenAI configuration + - secretKey: OPENAI_API_KEY + optional: true + remoteRef: + conversionStrategy: Default + decodingStrategy: None + key: ai-platform-engineering/global + property: OPENAI_API_KEY + - secretKey: OPENAI_ENDPOINT + optional: true + remoteRef: + conversionStrategy: Default + decodingStrategy: None + key: ai-platform-engineering/global + property: OPENAI_ENDPOINT + - secretKey: OPENAI_MODEL_NAME + optional: true + remoteRef: + conversionStrategy: Default + decodingStrategy: None + key: ai-platform-engineering/global + property: OPENAI_MODEL_NAME + # AWS Bedrock configuration + - secretKey: AWS_ACCESS_KEY_ID + optional: true + remoteRef: + conversionStrategy: Default + decodingStrategy: None + key: ai-platform-engineering/global + property: AWS_ACCESS_KEY_ID + - secretKey: AWS_SECRET_ACCESS_KEY + optional: true + remoteRef: + conversionStrategy: Default + decodingStrategy: None + key: ai-platform-engineering/global + property: AWS_SECRET_ACCESS_KEY + - secretKey: AWS_REGION + optional: true + remoteRef: + conversionStrategy: Default + decodingStrategy: None + key: ai-platform-engineering/global + property: AWS_REGION + - secretKey: AWS_BEDROCK_MODEL_ID + optional: true + remoteRef: + conversionStrategy: Default + decodingStrategy: None + key: ai-platform-engineering/global + property: AWS_BEDROCK_MODEL_ID + - secretKey: AWS_BEDROCK_PROVIDER + optional: true + remoteRef: + conversionStrategy: Default + decodingStrategy: None + key: ai-platform-engineering/global + property: AWS_BEDROCK_PROVIDER + +# AI Platform Engineer Multi-Agent +supervisor-agent: + nameOverride: "supervisor-agent" + image: + repository: "ghcr.io/cnoe-io/prebuild/ai-platform-engineering" + tag: "supervisor-hot-fix-2" + pullPolicy: "Always" + args: ["platform-engineer"] + env: + EXTERNAL_URL: "https://cnoe.localtest.me:8443/ai-platform-engineering" # Agent url for the client + SKIP_AGENT_CONNECTIVITY_CHECK: "false" + AGENT_CONNECTIVITY_ENABLE_BACKGROUND: "true" + multiAgentConfig: + protocol: "a2a" + port: "8000" + +# Agent configurations using aliases from Chart.yaml +agent-argocd: + nameOverride: "agent-argocd" + image: + repository: "ghcr.io/cnoe-io/agent-argocd" + pullPolicy: "Always" + tag: "0.1.15" + mcp: + image: + repository: "ghcr.io/cnoe-io/mcp-argocd" + tag: "0.1.15" + pullPolicy: "Always" + mode: "http" # Options: stdio, http + port: 8000 + agentSecrets: + secretName: "agent-argocd-secret" + externalSecrets: + data: + - secretKey: ARGOCD_TOKEN + remoteRef: + conversionStrategy: Default + decodingStrategy: None + key: ai-platform-engineering/argocd-secret + property: ARGOCD_TOKEN + - secretKey: ARGOCD_API_URL + remoteRef: + conversionStrategy: Default + decodingStrategy: None + key: ai-platform-engineering/argocd-secret + property: ARGOCD_API_URL + - secretKey: ARGOCD_VERIFY_SSL + remoteRef: + conversionStrategy: Default + decodingStrategy: None + key: ai-platform-engineering/argocd-secret + property: ARGOCD_VERIFY_SSL + +agent-backstage: + nameOverride: "agent-backstage" + image: + repository: "ghcr.io/cnoe-io/agent-backstage" + pullPolicy: "Always" + tag: "0.1.15" + mcp: + image: + repository: "ghcr.io/cnoe-io/mcp-backstage" + tag: "0.1.15" + pullPolicy: "Always" + mode: "http" # Options: stdio, http + port: 8000 + agentSecrets: + secretName: "agent-backstage-secret" + externalSecrets: + data: + - secretKey: BACKSTAGE_API_TOKEN + remoteRef: + conversionStrategy: Default + decodingStrategy: None + key: ai-platform-engineering/backstage-secret + property: BACKSTAGE_API_TOKEN + - secretKey: BACKSTAGE_URL + remoteRef: + conversionStrategy: Default + decodingStrategy: None + key: ai-platform-engineering/backstage-secret + property: BACKSTAGE_URL + +agent-confluence: + nameOverride: "agent-confluence" + image: + repository: "ghcr.io/cnoe-io/agent-confluence" + pullPolicy: "Always" + tag: "0.1.15" + mcp: + image: + repository: "ghcr.io/cnoe-io/mcp-confluence" + tag: "0.1.15" + pullPolicy: "Always" + mode: "http" # Options: stdio, http + port: 8000 + agentSecrets: + secretName: "agent-confluence-secret" + externalSecrets: + data: + - secretKey: ATLASSIAN_TOKEN + remoteRef: + conversionStrategy: Default + decodingStrategy: None + key: ai-platform-engineering/jira-secret + property: ATLASSIAN_TOKEN + - secretKey: ATLASSIAN_EMAIL + remoteRef: + conversionStrategy: Default + decodingStrategy: None + key: ai-platform-engineering/jira-secret + property: ATLASSIAN_EMAIL + - secretKey: ATLASSIAN_API_URL + remoteRef: + conversionStrategy: Default + decodingStrategy: None + key: ai-platform-engineering/jira-secret + property: ATLASSIAN_API_URL + - secretKey: ATLASSIAN_VERIFY_SSL + remoteRef: + conversionStrategy: Default + decodingStrategy: None + key: ai-platform-engineering/jira-secret + property: ATLASSIAN_VERIFY_SSL + +agent-github: + nameOverride: "agent-github" + image: + repository: "ghcr.io/cnoe-io/agent-github" + pullPolicy: "Always" + tag: "0.1.15" + mcp: + useRemoteMcpServer: true + agentSecrets: + secretName: "agent-github-secret" + externalSecrets: + data: + - secretKey: GITHUB_PERSONAL_ACCESS_TOKEN + remoteRef: + conversionStrategy: Default + decodingStrategy: None + key: ai-platform-engineering/github-secret + property: GITHUB_PERSONAL_ACCESS_TOKEN + +agent-jira: + nameOverride: "agent-jira" + image: + repository: "ghcr.io/cnoe-io/agent-jira" + pullPolicy: "Always" + tag: "0.1.15" + mcp: + image: + repository: "ghcr.io/cnoe-io/mcp-jira" + tag: "0.1.15" + pullPolicy: "Always" + mode: "http" # Options: stdio, http + port: 8000 + agentSecrets: + secretName: "agent-jira-secret" + externalSecrets: + data: + - secretKey: ATLASSIAN_TOKEN + remoteRef: + conversionStrategy: Default + decodingStrategy: None + key: ai-platform-engineering/jira-secret + property: ATLASSIAN_TOKEN + - secretKey: ATLASSIAN_EMAIL + remoteRef: + conversionStrategy: Default + decodingStrategy: None + key: ai-platform-engineering/jira-secret + property: ATLASSIAN_EMAIL + - secretKey: ATLASSIAN_API_URL + remoteRef: + conversionStrategy: Default + decodingStrategy: None + key: ai-platform-engineering/jira-secret + property: ATLASSIAN_API_URL + - secretKey: ATLASSIAN_VERIFY_SSL + remoteRef: + conversionStrategy: Default + decodingStrategy: None + key: ai-platform-engineering/jira-secret + property: ATLASSIAN_VERIFY_SSL + +agent-pagerduty: + nameOverride: "agent-pagerduty" + image: + repository: "ghcr.io/cnoe-io/agent-pagerduty" + pullPolicy: "Always" + tag: "0.1.15" + mcp: + image: + repository: "ghcr.io/cnoe-io/mcp-pagerduty" + tag: "0.1.15" + pullPolicy: "Always" + mode: "http" # Options: stdio, http + port: 8000 + agentSecrets: + secretName: "agent-pagerduty-secret" + externalSecrets: + data: + - secretKey: PAGERDUTY_API_KEY + remoteRef: + conversionStrategy: Default + decodingStrategy: None + key: ai-platform-engineering/pagerduty-secret + property: PAGERDUTY_API_KEY + - secretKey: PAGERDUTY_API_URL + remoteRef: + conversionStrategy: Default + decodingStrategy: None + key: ai-platform-engineering/pagerduty-secret + property: PAGERDUTY_API_URL + +agent-slack: + nameOverride: "agent-slack" + image: + repository: "ghcr.io/cnoe-io/agent-slack" + pullPolicy: "Always" + mcp: + image: + repository: "ghcr.io/cnoe-io/mcp-slack" + tag: "0.1.15" + pullPolicy: "Always" + mode: "http" # Options: stdio, http + port: 8000 + pullPolicy: "Always" + agentSecrets: + secretName: "agent-slack-secret" + externalSecrets: + data: + - secretKey: SLACK_BOT_TOKEN + remoteRef: + conversionStrategy: Default + decodingStrategy: None + key: ai-platform-engineering/slack-secret + property: SLACK_BOT_TOKEN + - secretKey: SLACK_APP_TOKEN + remoteRef: + conversionStrategy: Default + decodingStrategy: None + key: ai-platform-engineering/slack-secret + property: SLACK_APP_TOKEN + - secretKey: SLACK_SIGNING_SECRET + remoteRef: + conversionStrategy: Default + decodingStrategy: None + key: ai-platform-engineering/slack-secret + property: SLACK_SIGNING_SECRET + - secretKey: SLACK_CLIENT_SECRET + remoteRef: + conversionStrategy: Default + decodingStrategy: None + key: ai-platform-engineering/slack-secret + property: SLACK_CLIENT_SECRET + - secretKey: SLACK_TEAM_ID + remoteRef: + conversionStrategy: Default + decodingStrategy: None + key: ai-platform-engineering/slack-secret + property: SLACK_TEAM_ID + +# Backstage plugin agent forge +backstage-plugin-agent-forge: + enabled: false + nameOverride: "backstage-plugin-agent-forge" + image: + repository: "ghcr.io/cnoe-io/backstage-plugin-agent-forge" + tag: "latest" + pullPolicy: "Always" + service: + ports: + - name: http + port: 3000 + protocol: TCP + - name: backend + port: 7007 + protocol: TCP + +graphrag: + enabled: false + neo4jNameOverride: "graphrag-neo4j" # This needs to match the value in neo4j.fullnameOverride + +# neo4j is used by graphrag to store the graph data. If graphrag.enabled is false, neo4j is not deployed. +neo4j: + fullnameOverride: "graphrag-neo4j" + config: + server.directories.plugins: "/var/lib/neo4j/labs" + dbms.security.procedures.unrestricted: "apoc.*" + dbms.security.procedures.allowlist: "apoc.*" + server.config.strict_validation.enabled: "false" + apoc_config: + apoc.trigger.enabled: "true" + apoc.import.file.enabled: "true" + disableLookups: true + services: + neo4j: + enabled: false + neo4j: + name: graphrag-neo4j + password: "dummy_password" # This dummy password needs to match the value in graphrag.neo4j.auth.password + resources: + cpu: "1" + memory: "2Gi" + + volumes: + data: + mode: "dynamic" + dynamic: + storageClassName: local-path # Use local-path storage class for kind cluster + +# KB-RAG Stack - Complete stack including web, server, agent, Redis, and Milvus +kb-rag-stack: + enabled: false + + # KB-RAG Web configuration + kb-rag-web: + enabled: true + image: + repository: "ghcr.io/cnoe-io/kb-rag-web" + tag: "latest" + pullPolicy: "Always" + + service: + type: ClusterIP + port: 80 + + ingress: + enabled: false + className: "" + annotations: {} + hosts: + - host: kb-rag-web.local + paths: + - path: / + pathType: Prefix + tls: [] + + resources: + requests: + cpu: 100m + memory: 128Mi + limits: + cpu: 500m + memory: 512Mi + + # KB-RAG Server configuration + kb-rag-server: + enabled: true + image: + repository: "ghcr.io/cnoe-io/kb-rag-server" + tag: "latest" + pullPolicy: "Always" + + service: + type: ClusterIP + port: 9446 + + resources: + requests: + cpu: 100m + memory: 128Mi + ephemeral-storage: 256Mi + limits: + cpu: 500m + memory: 512Mi + ephemeral-storage: 1Gi + + # Agent configuration (standard agent chart) + kb-rag-agent: + enabled: true + nameOverride: "kb-rag-agent" + image: + repository: "ghcr.io/cnoe-io/kb-rag-agent-a2a" + tag: "latest" + pullPolicy: "Always" + + service: + type: ClusterIP + port: 8000 + + resources: + requests: + cpu: 100m + memory: 128Mi + ephemeral-storage: 256Mi + limits: + cpu: 500m + memory: 2Gi + ephemeral-storage: 1Gi + + agentSecrets: + secretName: "kb-rag-agent-secret" + externalSecrets: + data: + - secretKey: MILVUS_SECRET + remoteRef: + conversionStrategy: Default + decodingStrategy: None + key: ai-platform-engineering/milvus-secret + property: MILVUS_SECRET + + # KB-RAG Redis configuration + kb-rag-redis: + enabled: true + image: + repository: "redis" + tag: "7.2-alpine" + pullPolicy: "Always" + + service: + type: ClusterIP + port: 6379 + + persistence: + enabled: true + size: 1Gi + storageClass: "" + + resources: + requests: + cpu: 100m + memory: 128Mi + limits: + cpu: 200m + memory: 256Mi + + # Redis configuration + redis: + maxmemory: "256mb" + maxmemoryPolicy: "allkeys-lru" + save: "60 1" + appendonly: "yes" + +# Minimal Milvus cluster configuration using Zilliz Helm chart defaults +milvus: + # Only override what differs from chart defaults + pulsarv3: + enabled: false # Default is true, we need false + woodpecker: + enabled: true # Default is false, we need true + + # Performance overrides only for compute nodes + queryNode: + resources: + limits: + cpu: 200m + memory: 256Mi + + dataNode: + resources: + limits: + cpu: 200m + memory: 256Mi + + # MinIO storage overrides for kind cluster - drastically reduce from 500Gi default + minio: + persistence: + size: 2Gi # Down from 500Gi default + resources: + requests: + memory: 128Mi + cpu: 100m + limits: + memory: 256Mi + cpu: 200m diff --git a/caipe/workshop/backstage.yaml b/caipe/workshop/backstage.yaml new file mode 100644 index 00000000..40d82c6c --- /dev/null +++ b/caipe/workshop/backstage.yaml @@ -0,0 +1,21 @@ +apiVersion: argoproj.io/v1alpha1 +kind: Application +metadata: + name: backstage + namespace: argocd + labels: + env: dev +spec: + project: default + source: + repoURL: cnoe://backstage + targetRevision: HEAD + path: manifests + destination: + server: "https://kubernetes.default.svc" + namespace: backstage + syncPolicy: + syncOptions: + - CreateNamespace=true + automated: + selfHeal: true diff --git a/caipe/workshop/backstage/manifests/argocd-secrets.yaml b/caipe/workshop/backstage/manifests/argocd-secrets.yaml new file mode 100644 index 00000000..28ab8558 --- /dev/null +++ b/caipe/workshop/backstage/manifests/argocd-secrets.yaml @@ -0,0 +1,77 @@ +--- +apiVersion: v1 +kind: ServiceAccount +metadata: + name: eso-store + namespace: argocd +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: Role +metadata: + name: eso-store + namespace: argocd +rules: + - apiGroups: [""] + resources: + - secrets + verbs: + - get + - list + - watch + - apiGroups: + - authorization.k8s.io + resources: + - selfsubjectrulesreviews + verbs: + - create +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: RoleBinding +metadata: + name: eso-store + namespace: argocd +subjects: + - kind: ServiceAccount + name: eso-store + namespace: argocd +roleRef: + kind: Role + name: eso-store + apiGroup: rbac.authorization.k8s.io +--- +apiVersion: external-secrets.io/v1beta1 +kind: ClusterSecretStore +metadata: + name: argocd +spec: + provider: + kubernetes: + remoteNamespace: argocd + server: + caProvider: + type: ConfigMap + name: kube-root-ca.crt + namespace: argocd + key: ca.crt + auth: + serviceAccount: + name: eso-store + namespace: argocd +--- +apiVersion: external-secrets.io/v1beta1 +kind: ExternalSecret +metadata: + name: argocd-credentials + namespace: backstage +spec: + secretStoreRef: + name: argocd + kind: ClusterSecretStore + refreshInterval: "0" + target: + name: argocd-credentials + data: + - secretKey: ARGOCD_ADMIN_PASSWORD + remoteRef: + key: argocd-initial-admin-secret + property: password diff --git a/ai-platform-engineering/backstage/manifests/install.yaml b/caipe/workshop/backstage/manifests/install.yaml similarity index 88% rename from ai-platform-engineering/backstage/manifests/install.yaml rename to caipe/workshop/backstage/manifests/install.yaml index 5d3d050c..a4b16f63 100644 --- a/ai-platform-engineering/backstage/manifests/install.yaml +++ b/caipe/workshop/backstage/manifests/install.yaml @@ -2,17 +2,23 @@ apiVersion: v1 kind: Namespace metadata: name: backstage + annotations: + argocd.argoproj.io/sync-wave: "0" --- apiVersion: v1 kind: ServiceAccount metadata: name: backstage namespace: backstage + annotations: + argocd.argoproj.io/sync-wave: "0" --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: backstage-argo-worfklows + annotations: + argocd.argoproj.io/sync-wave: "1" rules: - apiGroups: - argoproj.io @@ -25,6 +31,8 @@ apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: read-all + annotations: + argocd.argoproj.io/sync-wave: "1" rules: - apiGroups: - '*' @@ -34,11 +42,21 @@ rules: - get - list - watch + - apiGroups: + - "" + resources: + - secrets + verbs: + - create + - update + - patch --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: backstage-argo-worfklows + annotations: + argocd.argoproj.io/sync-wave: "2" roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole @@ -52,6 +70,8 @@ apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: backstage-read-all + annotations: + argocd.argoproj.io/sync-wave: "2" roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole @@ -74,12 +94,14 @@ data: organization: name: CNOE backend: - # Used for enabling authentication, secret is shared by all backend plugins - # See https://backstage.io/docs/tutorials/backend-to-backend-auth for - # information on the format - # auth: - # keys: - # - secret: ${BACKEND_SECRET} + auth: + externalAccess: + - options: + subject: dev-api-access + token: ${AUTH_API_TOKEN_TEST} + type: static + keys: + - secret: ${BACKEND_AUTH_SECRET} baseUrl: https://cnoe.localtest.me:8443 listen: port: 7007 @@ -199,6 +221,36 @@ stringData: \ skipMetricsLookup: true\n serviceAccountToken: \n $file: /var/run/secrets/kubernetes.io/serviceaccount/token\n \ caData: \n $file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt\n" --- +apiVersion: batch/v1 +kind: Job +metadata: + name: backstage-auth-secret-generator + namespace: backstage + annotations: + argocd.argoproj.io/sync-wave: "5" +spec: + template: + spec: + serviceAccountName: backstage + restartPolicy: OnFailure + containers: + - name: secret-generator + image: ghcr.io/cnoe-io/kubectl:1.34 + command: ["/bin/bash", "-c"] + args: + - | + # Generate random secrets + AUTH_API_TOKEN=$(openssl rand -base64 32) + BACKEND_AUTH_SECRET=$(openssl rand -base64 32) + + # Create the secret + kubectl create secret generic backstage-auth-secrets \ + --from-literal=AUTH_API_TOKEN_TEST="$AUTH_API_TOKEN" \ + --from-literal=BACKEND_AUTH_SECRET="$BACKEND_AUTH_SECRET" \ + --dry-run=client -o yaml | kubectl apply -f - + + echo "Auth secrets generated successfully" +--- apiVersion: v1 kind: Service metadata: @@ -262,6 +314,8 @@ spec: name: gitea-credentials - secretRef: name: argocd-credentials + - secretRef: + name: backstage-auth-secrets image: ghcr.io/suwhang-cisco/backstage-app:66ef030f9b7282f16df76f348898e3712dff1f8b name: backstage ports: diff --git a/caipe/workshop/replace.sh b/caipe/workshop/replace.sh new file mode 100755 index 00000000..ee9dde10 --- /dev/null +++ b/caipe/workshop/replace.sh @@ -0,0 +1,36 @@ +# this script replaces hostname and port used by this implementation. +# intended for use in environments such as Codespaces where external host and port need to be updated to access in-cluster resources. + +#!/bin/bash +set -e +# Check if the new port number is provided as an argument +if [ "$#" -ne 2 ]; then + echo "Usage: NEW_HOST NEW_PORT" + exit 1 +fi + +# Assign the first script argument to NEW_PORT +NEW_HOST="$1" +NEW_PORT="$2" + +# Base directory to start from, "." means the current directory +CURRENT_DIR=$(echo "${PWD##*/}") +if [[ ${CURRENT_DIR} != "ai-platform-engineering" ]]; then + echo "please run this script from the ai-platform-engineering directory" + exit 10 +fi +BASE_DIRECTORY="." + +# Find all .yaml files recursively starting from the base directory +# and perform an in-place search and replace from 8443 to the new port +find "$BASE_DIRECTORY" -type f -name "*.yaml" -exec sed -i "s/8443/${NEW_PORT}/g" {} + +find "$BASE_DIRECTORY" -type f -name "*.yaml" -exec sed -i "s/cnoe\.localtest\.me/${NEW_HOST}/g" {} + + +# Remove hostname-port configuration if the new port is 443. Browsers strip 443 but keycloak still expects 443 in url. +if [[ ${NEW_PORT} == "443" ]]; then + sed -i "/hostname-port/d" keycloak/manifests/install.yaml + sed -i "/hostname-admin/d" keycloak/manifests/install.yaml + sed -i '0,/:443/{s/:443//}' argo-workflows/manifests/dev/patches/cm-argo-workflows.yaml +fi + +echo "Replacement complete." diff --git a/caipe/workshop/scripts/refresh-secrets.sh b/caipe/workshop/scripts/refresh-secrets.sh new file mode 100644 index 00000000..71ca2f00 --- /dev/null +++ b/caipe/workshop/scripts/refresh-secrets.sh @@ -0,0 +1,29 @@ +#!/bin/bash + +set -euo pipefail + +echo "๐Ÿš€ Starting AI Platform Engineering cleanup process..." +echo "" + +echo "๐Ÿ”‘ Deleting all secrets in ai-platform-engineering namespace..." +kubectl delete secret --all -n ai-platform-engineering + +echo "" +echo "โฑ๏ธ Waiting 1 second for cleanup to complete..." +sleep 1 + +echo "" +echo "๐Ÿ“ Command executed: kubectl delete secret --all -n ai-platform-engineering" +echo "" + +echo "๐Ÿ—‘๏ธ Deleting all pods in ai-platform-engineering namespace..." +kubectl delete pod --all -n ai-platform-engineering + +echo "" +echo "๐Ÿ“Š Current pods in ai-platform-engineering namespace:" +echo "==================================================" +kubectl get pods -n ai-platform-engineering + +echo "" +echo "โœ… Cleanup process completed successfully!" +echo "๐ŸŽฏ All secrets and pods have been refreshed in the ai-platform-engineering namespace" diff --git a/caipe/workshop/scripts/setup-agent-secrets.sh b/caipe/workshop/scripts/setup-agent-secrets.sh new file mode 100644 index 00000000..687824b4 --- /dev/null +++ b/caipe/workshop/scripts/setup-agent-secrets.sh @@ -0,0 +1,555 @@ +#!/bin/bash + +set -e + +log() { + echo "[$(date '+%Y-%m-%d %H:%M:%S')] $1" +} + +# Check dependencies +for cmd in kubectl vault jq; do + if ! command -v $cmd &> /dev/null; then + log "โŒ $cmd is required but not installed" + exit 1 + fi +done + +log "๐Ÿ”ง Setting up agent secrets based on active agents" + +# Setup Vault connection +VAULT_TOKEN=$(kubectl get secret vault-root-token -n vault -o jsonpath='{.data.token}' | base64 -d) +export VAULT_ADDR="http://localhost:8200" +export VAULT_TOKEN + +# Start port forward +log "๐Ÿ”— Starting Vault port forward..." +kubectl port-forward -n vault svc/vault 8200:8200 & +VAULT_PID=$! +sleep 3 + +# Single-line, exact-byte prompt helper (no newline added, no stripping) +# Usage: prompt_with_env "" VAR_NAME is_secret +prompt_with_env() { + local prompt="$1" var_name="$2" is_secret="$3" + local env_value="${!var_name}" result + + if [[ -n "$env_value" ]]; then + if [[ "$is_secret" == "true" ]]; then + local hint="${env_value:0:5}..." + printf "%s (env: %s) [Enter to use, type new]: " "$prompt" "$hint" > /dev/tty + IFS= read -r choice < /dev/tty + if [[ -z "$choice" ]]; then + result="$env_value" + else + IFS= read -rs -p "$prompt: " result < /dev/tty + printf "\n" > /dev/tty + fi + else + IFS= read -r -p "$prompt (env: $env_value) [Enter to use, type new]: " choice < /dev/tty + if [[ -z "$choice" ]]; then + result="$env_value" + else + IFS= read -r -p "$prompt: " result < /dev/tty + fi + fi + else + if [[ "$is_secret" == "true" ]]; then + IFS= read -rs -p "$prompt: " result < /dev/tty + printf "\n" > /dev/tty + else + IFS= read -r -p "$prompt: " result < /dev/tty + fi + fi + + # Normalize only a trailing CR (some terminals send \r) + result=${result%$'\r'} + + # Output EXACTLY the bytes, no newline + printf '%s' "$result" +} + +# Check which agents are active +log "๐Ÿ” Checking active agents..." +active_agents=() + +# Check for GitHub agent (look for GitHub-related deployments or configs) +if kubectl get deployment -n ai-platform-engineering ai-platform-engineering-agent-github 2>/dev/null || \ + kubectl get configmap -n ai-platform-engineering | grep -i github >/dev/null 2>&1; then + active_agents+=("github") + log "โœ… GitHub agent detected" +fi + +# Check for GitLab agent +if kubectl get deployment -n ai-platform-engineering ai-platform-engineering-agent-gitlab 2>/dev/null || \ + kubectl get configmap -n ai-platform-engineering | grep -i gitlab >/dev/null 2>&1; then + active_agents+=("gitlab") + log "โœ… GitLab agent detected" +fi + +# Check for Jira agent +if kubectl get deployment -n ai-platform-engineering ai-platform-engineering-agent-jira 2>/dev/null || \ + kubectl get configmap -n ai-platform-engineering | grep -i jira >/dev/null 2>&1; then + active_agents+=("jira") + log "โœ… Jira agent detected" +fi + +# Check for Slack agent +if kubectl get deployment -n ai-platform-engineering ai-platform-engineering-agent-slack 2>/dev/null || \ + kubectl get configmap -n ai-platform-engineering | grep -i slack >/dev/null 2>&1; then + active_agents+=("slack") + log "โœ… Slack agent detected" +fi + +# Check for AWS agent +if kubectl get deployment -n ai-platform-engineering ai-platform-engineering-agent-aws 2>/dev/null || \ + kubectl get configmap -n ai-platform-engineering | grep -i aws >/dev/null 2>&1; then + active_agents+=("aws") + log "โœ… AWS agent detected" +fi + +# Check for ArgoCD agent +if kubectl get deployment -n ai-platform-engineering ai-platform-engineering-agent-argocd 2>/dev/null || \ + kubectl get configmap -n ai-platform-engineering | grep -i argocd >/dev/null 2>&1; then + active_agents+=("argocd") + log "โœ… ArgoCD agent detected" +fi + +# Check for Backstage agent +if kubectl get deployment -n ai-platform-engineering ai-platform-engineering-agent-backstage 2>/dev/null || \ + kubectl get configmap -n ai-platform-engineering | grep -i backstage >/dev/null 2>&1; then + active_agents+=("backstage") + log "โœ… Backstage agent detected" +fi + +# Check for PagerDuty agent +if kubectl get deployment -n ai-platform-engineering ai-platform-engineering-agent-pagerduty 2>/dev/null || \ + kubectl get configmap -n ai-platform-engineering | grep -i pagerduty >/dev/null 2>&1; then + active_agents+=("pagerduty") + log "โœ… PagerDuty agent detected" +fi + +# Check for Confluence agent +if kubectl get deployment -n ai-platform-engineering ai-platform-engineering-agent-confluence 2>/dev/null || \ + kubectl get configmap -n ai-platform-engineering | grep -i confluence >/dev/null 2>&1; then + active_agents+=("confluence") + log "โœ… Confluence agent detected" +fi + +# Check for Splunk agent +if kubectl get deployment -n ai-platform-engineering ai-platform-engineering-agent-splunk 2>/dev/null || \ + kubectl get configmap -n ai-platform-engineering | grep -i splunk >/dev/null 2>&1; then + active_agents+=("splunk") + log "โœ… Splunk agent detected" +fi + +# Check for Webex agent +if kubectl get deployment -n ai-platform-engineering ai-platform-engineering-agent-webex 2>/dev/null || \ + kubectl get configmap -n ai-platform-engineering | grep -i webex >/dev/null 2>&1; then + active_agents+=("webex") + log "โœ… Webex agent detected" +fi + +# Check for Komodor agent +if kubectl get deployment -n ai-platform-engineering ai-platform-engineering-agent-komodor 2>/dev/null || \ + kubectl get configmap -n ai-platform-engineering | grep -i komodor >/dev/null 2>&1; then + active_agents+=("komodor") + log "โœ… Komodor agent detected" +fi + +# If no agents detected, ask user to select +if [[ ${#active_agents[@]} -eq 0 ]]; then + log "๐Ÿค” No active agents detected. Please select which agents to configure:" + echo "" + echo "Available agents:" + echo "1) GitHub" + echo "2) Jira" + echo "3) Slack" + echo "4) AWS" + echo "5) ArgoCD" + echo "6) Backstage" + echo "7) PagerDuty" + echo "8) Confluence" + echo "9) Splunk" + echo "10) Webex" + echo "11) Komodor" + echo "12) All of the above" + echo "" + read -p "Select agents (comma-separated numbers, e.g., 1,3,4): " agent_selection + + IFS=',' read -ra selected <<< "$agent_selection" + for choice in "${selected[@]}"; do + case $choice in + 1) active_agents+=("github") ;; + 2) active_agents+=("jira") ;; + 3) active_agents+=("slack") ;; + 4) active_agents+=("aws") ;; + 5) active_agents+=("argocd") ;; + 6) active_agents+=("backstage") ;; + 7) active_agents+=("pagerduty") ;; + 8) active_agents+=("confluence") ;; + 9) active_agents+=("splunk") ;; + 10) active_agents+=("webex") ;; + 11) active_agents+=("komodor") ;; + 12) active_agents=("github" "jira" "slack" "aws" "argocd" "backstage" "pagerduty" "confluence" "splunk" "webex" "komodor") ;; + esac + done +fi + +log "๐Ÿ“ Configuring secrets for agents: ${active_agents[*]}" +echo "" +log "๐Ÿ”’ Note: Sensitive credentials will not be displayed on screen" + +# Initialize all fields as empty +GITHUB_PERSONAL_ACCESS_TOKEN="" +ATLASSIAN_TOKEN="" +ATLASSIAN_API_URL="" +ATLASSIAN_EMAIL="" +ATLASSIAN_VERIFY_SSL="" +SLACK_BOT_TOKEN="" +SLACK_TOKEN="" +SLACK_APP_TOKEN="" +SLACK_SIGNING_SECRET="" +SLACK_CLIENT_SECRET="" +SLACK_TEAM_ID="" +AWS_ACCESS_KEY_ID="" +AWS_SECRET_ACCESS_KEY="" +AWS_REGION="" +ARGOCD_TOKEN="" +ARGOCD_API_URL="" +ARGOCD_VERIFY_SSL="" +BACKSTAGE_API_TOKEN="" +BACKSTAGE_URL="" +PAGERDUTY_API_URL="" +PAGERDUTY_API_KEY="" +CONFLUENCE_API_URL="" +SPLUNK_API_URL="" +SPLUNK_TOKEN="" +WEBEX_TOKEN="" +KOMODOR_TOKEN="" +KOMODOR_API_URL="" +AZURE_OPENAI_API_KEY="" +AZURE_OPENAI_ENDPOINT="" +AZURE_OPENAI_DEPLOYMENT="" +AZURE_OPENAI_API_VERSION="" + +# Collect credentials based on active agents +for agent in "${active_agents[@]}"; do + case $agent in + "github") + echo "" + log "๐Ÿ™ Configuring GitHub agent secrets..." + GITHUB_PERSONAL_ACCESS_TOKEN="$(prompt_with_env 'GitHub Personal Access Token' 'GITHUB_PERSONAL_ACCESS_TOKEN' 'true')" + ;; + "jira") + echo "" + log "๐ŸŽซ Configuring Jira agent secrets..." + ATLASSIAN_TOKEN=$(prompt_with_env "Atlassian API Token" "ATLASSIAN_TOKEN" "true") + ATLASSIAN_API_URL=$(prompt_with_env "Atlassian API URL (e.g., https://company.atlassian.net)" "ATLASSIAN_API_URL" "false") + ATLASSIAN_EMAIL=$(prompt_with_env "Atlassian Email" "ATLASSIAN_EMAIL" "false") + ATLASSIAN_VERIFY_SSL=$(prompt_with_env "Verify SSL (true/false)" "ATLASSIAN_VERIFY_SSL" "false" "true") + ;; + "slack") + echo "" + log "๐Ÿ’ฌ Configuring Slack agent secrets..." + SLACK_BOT_TOKEN=$(prompt_with_env "Slack Bot Token (xoxb-...)" "SLACK_BOT_TOKEN" "true") + SLACK_TOKEN=$(prompt_with_env "Slack Token" "SLACK_TOKEN" "true") + SLACK_APP_TOKEN=$(prompt_with_env "Slack App Token (xapp-...)" "SLACK_APP_TOKEN" "true") + SLACK_SIGNING_SECRET=$(prompt_with_env "Slack Signing Secret" "SLACK_SIGNING_SECRET" "true") + SLACK_CLIENT_SECRET=$(prompt_with_env "Slack Client Secret" "SLACK_CLIENT_SECRET" "true") + SLACK_TEAM_ID=$(prompt_with_env "Slack Team ID" "SLACK_TEAM_ID" "false") + ;; + "aws") + echo "" + log "โ˜๏ธ Configuring AWS agent secrets..." + AWS_ACCESS_KEY_ID=$(prompt_with_env "AWS Access Key ID" "AWS_ACCESS_KEY_ID" "false") + AWS_SECRET_ACCESS_KEY=$(prompt_with_env "AWS Secret Access Key" "AWS_SECRET_ACCESS_KEY" "true") + AWS_REGION=$(prompt_with_env "AWS Region" "AWS_REGION" "false" "us-east-1") + ;; + "argocd") + echo "" + log "๐Ÿš€ Populating ArgoCD secrets with local ArgoCD set up and grab following values:" + log "1. ARGOCD_TOKEN will be from k8s secret argocd-admin-token in namespace vault, key: token" + log "2. ARGOCD_API_URL will be from the same k8s secret but key: apiUrl" + log "3. ARGOCD_VERIFY_SSL set to 'false'" + + # Get ArgoCD token from Kubernetes secret + ARGOCD_TOKEN=$(kubectl get secret -n vault argocd-admin-token -o jsonpath='{.data.token}' 2>/dev/null | base64 -d 2>/dev/null || echo "") + if [[ -z "$ARGOCD_TOKEN" ]]; then + log "โš ๏ธ Could not retrieve ARGOCD_TOKEN from secret argocd-admin-token in vault namespace" + else + log "โœ… ARGOCD_TOKEN retrieved from Kubernetes secret" + fi + + # Get ArgoCD API URL from Kubernetes secret + ARGOCD_API_URL=$(kubectl get secret -n vault argocd-admin-token -o jsonpath='{.data.apiUrl}' 2>/dev/null | base64 -d 2>/dev/null || echo "") + if [[ -z "$ARGOCD_API_URL" ]]; then + log "โš ๏ธ Could not retrieve ARGOCD_API_URL from secret argocd-admin-token in vault namespace" + ARGOCD_API_URL="http://argocd-server.argocd.svc.cluster.local" + log "๐Ÿ“ Using default ARGOCD_API_URL: $ARGOCD_API_URL" + else + log "โœ… ARGOCD_API_URL retrieved from Kubernetes secret: $ARGOCD_API_URL" + fi + + # Set ArgoCD SSL verification to false + ARGOCD_VERIFY_SSL="false" + log "โœ… ARGOCD_VERIFY_SSL set to: $ARGOCD_VERIFY_SSL" + ;; + "backstage") + echo "" + log "๐ŸŽญ Populating Backstage secrets with local Backstage set up and grab following values:" + log "1. BACKSTAGE_API_TOKEN from k8s secret backstage-auth-secrets in namespace backstage, key: AUTH_API_TOKEN_TEST" + log "2. BACKSTAGE_URL set to http://backstage.backstage.svc.cluster.local:7007" + + # Get Backstage API token from Kubernetes secret + BACKSTAGE_API_TOKEN=$(kubectl get secret -n backstage backstage-auth-secrets -o jsonpath='{.data.AUTH_API_TOKEN_TEST}' 2>/dev/null | base64 -d 2>/dev/null || echo "") + if [[ -z "$BACKSTAGE_API_TOKEN" ]]; then + log "โš ๏ธ Could not retrieve BACKSTAGE_API_TOKEN from secret backstage-auth-secrets in backstage namespace" + else + log "โœ… BACKSTAGE_API_TOKEN retrieved from Kubernetes secret" + fi + + # Set Backstage URL + BACKSTAGE_URL="http://backstage.backstage.svc.cluster.local:7007" + log "โœ… BACKSTAGE_URL set to: $BACKSTAGE_URL" + ;; + "pagerduty") + echo "" + log "๐Ÿ“Ÿ Configuring PagerDuty agent secrets..." + PAGERDUTY_API_KEY=$(prompt_with_env "PagerDuty API Key" "PAGERDUTY_API_KEY" "true") + PAGERDUTY_API_URL=$(prompt_with_env "PagerDuty API URL" "PAGERDUTY_API_URL" "false" "https://api.pagerduty.com") + ;; + "confluence") + echo "" + log "๐Ÿ“š Configuring Confluence agent secrets..." + CONFLUENCE_API_URL=$(prompt_with_env "Confluence API URL (e.g., https://company.atlassian.net/wiki)" "CONFLUENCE_API_URL" "false") + if [[ -z "$ATLASSIAN_TOKEN" ]]; then + ATLASSIAN_TOKEN=$(prompt_with_env "Atlassian API Token" "ATLASSIAN_TOKEN" "true") + ATLASSIAN_EMAIL=$(prompt_with_env "Atlassian Email" "ATLASSIAN_EMAIL" "false") + ATLASSIAN_VERIFY_SSL=$(prompt_with_env "Verify SSL (true/false)" "ATLASSIAN_VERIFY_SSL" "false" "true") + fi + ;; + "splunk") + echo "" + log "๐Ÿ” Configuring Splunk agent secrets..." + SPLUNK_TOKEN=$(prompt_with_env "Splunk Token" "SPLUNK_TOKEN" "true") + SPLUNK_API_URL=$(prompt_with_env "Splunk API URL (e.g., https://splunk.company.com)" "SPLUNK_API_URL" "false") + ;; + "webex") + echo "" + log "๐Ÿ“น Configuring Webex agent secrets..." + WEBEX_TOKEN=$(prompt_with_env "Webex Token" "WEBEX_TOKEN" "true") + ;; + "komodor") + echo "" + log "๐Ÿ”ง Configuring Komodor agent secrets..." + KOMODOR_TOKEN=$(prompt_with_env "Komodor Token" "KOMODOR_TOKEN" "true") + KOMODOR_API_URL=$(prompt_with_env "Komodor API URL" "KOMODOR_API_URL" "false" "https://api.komodor.com") + ;; + esac +done + +# Store all secrets in Vault +log "๐Ÿ’พ Storing agent secrets in Vault..." + +# Store secrets individually for each active agent +for agent in "${active_agents[@]}"; do + case $agent in + "github") + if [[ -n "$GITHUB_PERSONAL_ACCESS_TOKEN" ]]; then + vault kv put secret/ai-platform-engineering/github-secret \ + GITHUB_PERSONAL_ACCESS_TOKEN="$GITHUB_PERSONAL_ACCESS_TOKEN" >/dev/null + log "โœ… GitHub secrets stored" + fi + ;; + "jira") + if [[ -n "$ATLASSIAN_TOKEN" ]]; then + vault kv put secret/ai-platform-engineering/jira-secret \ + ATLASSIAN_TOKEN="$ATLASSIAN_TOKEN" \ + ATLASSIAN_API_URL="$ATLASSIAN_API_URL" \ + ATLASSIAN_EMAIL="$ATLASSIAN_EMAIL" \ + ATLASSIAN_VERIFY_SSL="$ATLASSIAN_VERIFY_SSL" >/dev/null + log "โœ… Jira secrets stored" + fi + ;; + "slack") + if [[ -n "$SLACK_BOT_TOKEN" ]]; then + vault kv put secret/ai-platform-engineering/slack-secret \ + SLACK_BOT_TOKEN="$SLACK_BOT_TOKEN" \ + SLACK_TOKEN="$SLACK_TOKEN" \ + SLACK_APP_TOKEN="$SLACK_APP_TOKEN" \ + SLACK_SIGNING_SECRET="$SLACK_SIGNING_SECRET" \ + SLACK_CLIENT_SECRET="$SLACK_CLIENT_SECRET" \ + SLACK_TEAM_ID="$SLACK_TEAM_ID" >/dev/null + log "โœ… Slack secrets stored" + fi + ;; + "aws") + if [[ -n "$AWS_ACCESS_KEY_ID" ]]; then + vault kv put secret/ai-platform-engineering/aws-secret \ + AWS_ACCESS_KEY_ID="$AWS_ACCESS_KEY_ID" \ + AWS_SECRET_ACCESS_KEY="$AWS_SECRET_ACCESS_KEY" \ + AWS_REGION="$AWS_REGION" >/dev/null + log "โœ… AWS secrets stored" + fi + ;; + "argocd") + if [[ -n "$ARGOCD_TOKEN" ]]; then + vault kv put secret/ai-platform-engineering/argocd-secret \ + ARGOCD_TOKEN="$ARGOCD_TOKEN" \ + ARGOCD_API_URL="$ARGOCD_API_URL" \ + ARGOCD_VERIFY_SSL="$ARGOCD_VERIFY_SSL" >/dev/null + log "โœ… ArgoCD secrets stored" + fi + ;; + "backstage") + if [[ -n "$BACKSTAGE_API_TOKEN" ]]; then + vault kv put secret/ai-platform-engineering/backstage-secret \ + BACKSTAGE_API_TOKEN="$BACKSTAGE_API_TOKEN" \ + BACKSTAGE_URL="$BACKSTAGE_URL" >/dev/null + log "โœ… Backstage secrets stored" + fi + ;; + "pagerduty") + if [[ -n "$PAGERDUTY_API_KEY" ]]; then + vault kv put secret/ai-platform-engineering/pagerduty-secret \ + PAGERDUTY_API_KEY="$PAGERDUTY_API_KEY" \ + PAGERDUTY_API_URL="$PAGERDUTY_API_URL" >/dev/null + log "โœ… PagerDuty secrets stored" + fi + ;; + "confluence") + if [[ -n "$CONFLUENCE_API_URL" ]]; then + vault kv put secret/ai-platform-engineering/confluence-secret \ + CONFLUENCE_API_URL="$CONFLUENCE_API_URL" \ + ATLASSIAN_TOKEN="$ATLASSIAN_TOKEN" \ + ATLASSIAN_EMAIL="$ATLASSIAN_EMAIL" \ + ATLASSIAN_VERIFY_SSL="$ATLASSIAN_VERIFY_SSL" >/dev/null + log "โœ… Confluence secrets stored" + fi + ;; + "splunk") + if [[ -n "$SPLUNK_TOKEN" ]]; then + vault kv put secret/ai-platform-engineering/splunk-secret \ + SPLUNK_TOKEN="$SPLUNK_TOKEN" \ + SPLUNK_API_URL="$SPLUNK_API_URL" >/dev/null + log "โœ… Splunk secrets stored" + fi + ;; + "webex") + if [[ -n "$WEBEX_TOKEN" ]]; then + vault kv put secret/ai-platform-engineering/webex-secret \ + WEBEX_TOKEN="$WEBEX_TOKEN" >/dev/null + log "โœ… Webex secrets stored" + fi + ;; + "komodor") + if [[ -n "$KOMODOR_TOKEN" ]]; then + vault kv put secret/ai-platform-engineering/komodor-secret \ + KOMODOR_TOKEN="$KOMODOR_TOKEN" \ + KOMODOR_API_URL="$KOMODOR_API_URL" >/dev/null + log "โœ… Komodor secrets stored" + fi + ;; + esac +done + +# Store global LLM credentials +log "๐Ÿค– Storing global LLM credentials in Vault..." +source $HOME/.env_vars +vault kv put secret/ai-platform-engineering/global \ + LLM_PROVIDER=azure-openai \ + AZURE_OPENAI_API_KEY="${AZURE_OPENAI_API_KEY}" \ + AZURE_OPENAI_ENDPOINT="${AZURE_OPENAI_ENDPOINT}" \ + AZURE_OPENAI_DEPLOYMENT="${AZURE_OPENAI_DEPLOYMENT}" \ + AZURE_OPENAI_API_VERSION="${AZURE_OPENAI_API_VERSION}" \ + AWS_ACCESS_KEY_ID="" \ + AWS_SECRET_ACCESS_KEY="" \ + AWS_REGION="" \ + AWS_BEDROCK_MODEL_ID="" \ + AWS_BEDROCK_PROVIDER="" \ + OPENAI_API_KEY="" \ + OPENAI_ENDPOINT="" \ + OPENAI_MODEL_NAME="" >/dev/null +log "โœ… Global LLM credentials stored" + +log "โœ… Agent secrets successfully stored in Vault" +echo "" +log "๐Ÿ” You can verify individual agent secrets at:" +for agent in "${active_agents[@]}"; do + case $agent in + "github") log " ๐Ÿ™ GitHub: https://vault.cnoe.localtest.me:8443/ui/vault/secrets/secret/kv/ai-platform-engineering%2Fgithub-secret" ;; + "jira") log " ๐ŸŽซ Jira: https://vault.cnoe.localtest.me:8443/ui/vault/secrets/secret/kv/ai-platform-engineering%2Fjira-secret" ;; + "slack") log " ๐Ÿ’ฌ Slack: https://vault.cnoe.localtest.me:8443/ui/vault/secrets/secret/kv/ai-platform-engineering%2Fslack-secret" ;; + "aws") log " โ˜๏ธ AWS: https://vault.cnoe.localtest.me:8443/ui/vault/secrets/secret/kv/ai-platform-engineering%2Faws-secret" ;; + "argocd") log " ๐Ÿš€ ArgoCD: https://vault.cnoe.localtest.me:8443/ui/vault/secrets/secret/kv/ai-platform-engineering%2Fargocd-secret" ;; + "backstage") log " ๐ŸŽญ Backstage: https://vault.cnoe.localtest.me:8443/ui/vault/secrets/secret/kv/ai-platform-engineering%2Fbackstage-secret" ;; + "pagerduty") log " ๐Ÿ“Ÿ PagerDuty: https://vault.cnoe.localtest.me:8443/ui/vault/secrets/secret/kv/ai-platform-engineering%2Fpagerduty-secret" ;; + "confluence") log " ๐Ÿ“š Confluence: https://vault.cnoe.localtest.me:8443/ui/vault/secrets/secret/kv/ai-platform-engineering%2Fconfluence-secret" ;; + "splunk") log " ๐Ÿ” Splunk: https://vault.cnoe.localtest.me:8443/ui/vault/secrets/secret/kv/ai-platform-engineering%2Fsplunk-secret" ;; + "webex") log " ๐Ÿ“น Webex: https://vault.cnoe.localtest.me:8443/ui/vault/secrets/secret/kv/ai-platform-engineering%2Fwebex-secret" ;; + "komodor") log " ๐Ÿ”ง Komodor: https://vault.cnoe.localtest.me:8443/ui/vault/secrets/secret/kv/ai-platform-engineering%2Fkomodor-secret" ;; + esac +done +log " ๐Ÿค– Global LLM: https://vault.cnoe.localtest.me:8443/ui/vault/secrets/secret/kv/ai-platform-engineering%2Fglobal" + +# Create Kubernetes secret for agents +log "๐Ÿ”„ Creating Kubernetes secret for agents..." +kubectl create secret generic agent-secrets -n ai-platform-engineering \ + --from-literal=GITHUB_PERSONAL_ACCESS_TOKEN="$GITHUB_PERSONAL_ACCESS_TOKEN" \ + --from-literal=ATLASSIAN_TOKEN="$ATLASSIAN_TOKEN" \ + --from-literal=ATLASSIAN_API_URL="$ATLASSIAN_API_URL" \ + --from-literal=ATLASSIAN_EMAIL="$ATLASSIAN_EMAIL" \ + --from-literal=ATLASSIAN_VERIFY_SSL="$ATLASSIAN_VERIFY_SSL" \ + --from-literal=SLACK_BOT_TOKEN="$SLACK_BOT_TOKEN" \ + --from-literal=SLACK_TOKEN="$SLACK_TOKEN" \ + --from-literal=SLACK_APP_TOKEN="$SLACK_APP_TOKEN" \ + --from-literal=SLACK_SIGNING_SECRET="$SLACK_SIGNING_SECRET" \ + --from-literal=SLACK_CLIENT_SECRET="$SLACK_CLIENT_SECRET" \ + --from-literal=SLACK_TEAM_ID="$SLACK_TEAM_ID" \ + --from-literal=AWS_ACCESS_KEY_ID="$AWS_ACCESS_KEY_ID" \ + --from-literal=AWS_SECRET_ACCESS_KEY="$AWS_SECRET_ACCESS_KEY" \ + --from-literal=AWS_REGION="$AWS_REGION" \ + --from-literal=ARGOCD_TOKEN="$ARGOCD_TOKEN" \ + --from-literal=ARGOCD_API_URL="$ARGOCD_API_URL" \ + --from-literal=ARGOCD_VERIFY_SSL="$ARGOCD_VERIFY_SSL" \ + --from-literal=BACKSTAGE_API_TOKEN="$BACKSTAGE_API_TOKEN" \ + --from-literal=BACKSTAGE_URL="$BACKSTAGE_URL" \ + --from-literal=PAGERDUTY_API_KEY="$PAGERDUTY_API_KEY" \ + --from-literal=PAGERDUTY_API_URL="$PAGERDUTY_API_URL" \ + --from-literal=CONFLUENCE_API_URL="$CONFLUENCE_API_URL" \ + --from-literal=SPLUNK_TOKEN="$SPLUNK_TOKEN" \ + --from-literal=SPLUNK_API_URL="$SPLUNK_API_URL" \ + --from-literal=WEBEX_TOKEN="$WEBEX_TOKEN" \ + --from-literal=KOMODOR_TOKEN="$KOMODOR_TOKEN" \ + --from-literal=KOMODOR_API_URL="$KOMODOR_API_URL" \ + --from-literal=AZURE_OPENAI_API_KEY="$AZURE_OPENAI_API_KEY" \ + --from-literal=AZURE_OPENAI_ENDPOINT="$AZURE_OPENAI_ENDPOINT" \ + --from-literal=AZURE_OPENAI_DEPLOYMENT="$AZURE_OPENAI_DEPLOYMENT" \ + --from-literal=AZURE_OPENAI_API_VERSION="$AZURE_OPENAI_API_VERSION" \ + --dry-run=client -o yaml | kubectl apply -f - + +log "โœ… Kubernetes secret created/updated" + +# Summary +echo "" +log "๐Ÿ“Š Configuration Summary:" +log " ๐Ÿค– Global LLM: Azure OpenAI credentials configured" +for agent in "${active_agents[@]}"; do + case $agent in + "github") log " ๐Ÿ™ GitHub: Personal Access Token configured" ;; + "jira") log " ๐ŸŽซ Jira: Atlassian Token and API URL configured" ;; + "slack") log " ๐Ÿ’ฌ Slack: Bot Token, App Token, and additional tokens configured" ;; + "aws") log " โ˜๏ธ AWS: Access Keys and Region configured" ;; + "argocd") log " ๐Ÿš€ ArgoCD: Token and API URL configured" ;; + "backstage") log " ๐ŸŽญ Backstage: API Token and URL configured" ;; + "pagerduty") log " ๐Ÿ“Ÿ PagerDuty: API Key and URL configured" ;; + "confluence") log " ๐Ÿ“š Confluence: API URL and Atlassian credentials configured" ;; + "splunk") log " ๐Ÿ” Splunk: Token and API URL configured" ;; + "webex") log " ๐Ÿ“น Webex: Token configured" ;; + "komodor") log " ๐Ÿ”ง Komodor: Token and API URL configured" ;; + esac +done + +# Cleanup +kill $VAULT_PID 2>/dev/null +log "๐ŸŽ‰ Agent secrets setup complete!" diff --git a/caipe/workshop/vault.yaml b/caipe/workshop/vault.yaml new file mode 100644 index 00000000..3f5b0ccb --- /dev/null +++ b/caipe/workshop/vault.yaml @@ -0,0 +1,35 @@ +apiVersion: argoproj.io/v1alpha1 +kind: Application +metadata: + name: vault + namespace: argocd + finalizers: + - resources-finalizer.argocd.argoproj.io +spec: + project: default + sources: + - repoURL: https://helm.releases.hashicorp.com + chart: vault + targetRevision: "0.27.0" + helm: + valueFiles: + - $values/values.yaml + - repoURL: cnoe://vault + targetRevision: HEAD + path: manifests + ref: values + destination: + server: https://kubernetes.default.svc + namespace: vault + syncPolicy: + automated: + prune: true + selfHeal: true + syncOptions: + - CreateNamespace=true + ignoreDifferences: + # mutating webhook configuration has caBundle injected so ignore diff + - group: admissionregistration.k8s.io + kind: MutatingWebhookConfiguration + jsonPointers: + - /webhooks diff --git a/caipe/workshop/vault/manifests/argocd-token-job.yaml b/caipe/workshop/vault/manifests/argocd-token-job.yaml new file mode 100644 index 00000000..a8babbb0 --- /dev/null +++ b/caipe/workshop/vault/manifests/argocd-token-job.yaml @@ -0,0 +1,190 @@ +apiVersion: v1 +kind: ServiceAccount +metadata: + name: argocd-token-writer + namespace: vault + annotations: + argocd.argoproj.io/sync-wave: "1" +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: Role +metadata: + name: argocd-token-secret-manager + namespace: vault + annotations: + argocd.argoproj.io/sync-wave: "1" +rules: +- apiGroups: [""] + resources: ["secrets"] + verbs: ["get", "create", "update", "patch", "list"] +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: RoleBinding +metadata: + name: argocd-token-secret-manager-binding + namespace: vault + annotations: + argocd.argoproj.io/sync-wave: "1" +subjects: +- kind: ServiceAccount + name: argocd-token-writer + namespace: vault +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: Role + name: argocd-token-secret-manager +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: Role +metadata: + name: argocd-read-admin-and-deploy + namespace: argocd + annotations: + argocd.argoproj.io/sync-wave: "1" +rules: +- apiGroups: [""] + resources: ["secrets"] + resourceNames: ["argocd-initial-admin-secret"] + verbs: ["get"] +- apiGroups: ["apps"] + resources: ["deployments"] + resourceNames: ["argocd-server"] + verbs: ["get", "list", "watch"] +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: RoleBinding +metadata: + name: argocd-read-admin-and-deploy-binding + namespace: argocd + annotations: + argocd.argoproj.io/sync-wave: "1" +subjects: +- kind: ServiceAccount + name: argocd-token-writer + namespace: vault +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: Role + name: argocd-read-admin-and-deploy +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + name: argocd-namespace-reader + annotations: + argocd.argoproj.io/sync-wave: "1" +rules: +- apiGroups: [""] + resources: ["namespaces"] + resourceNames: ["argocd"] + verbs: ["get"] +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRoleBinding +metadata: + name: argocd-namespace-reader-binding + annotations: + argocd.argoproj.io/sync-wave: "1" +subjects: +- kind: ServiceAccount + name: argocd-token-writer + namespace: vault +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: argocd-namespace-reader +--- +apiVersion: batch/v1 +kind: Job +metadata: + name: argocd-token + namespace: vault + annotations: + argocd.argoproj.io/sync-wave: "2" + argocd.argoproj.io/hook-delete-policy: HookSucceeded +spec: + backoffLimit: 1 + template: + metadata: + labels: + app: argocd-token + spec: + serviceAccountName: argocd-token-writer + restartPolicy: OnFailure + containers: + - name: generate-token + image: dtzar/helm-kubectl:3.14.4 + imagePullPolicy: IfNotPresent + command: + - /bin/sh + - -c + args: + - | + set -eu + echo "Waiting for namespace 'argocd' to exist..." + until kubectl get ns argocd >/dev/null 2>&1; do + echo "Waiting for namespace 'argocd' to exist... WTF" + echo "printing kubectl get ns argocd output" + kubectl get ns argocd + sleep 5 + done + + echo "Waiting for Argo CD server deployment to exist..." + until kubectl -n argocd get deploy argocd-server >/dev/null 2>&1; do + sleep 5 + done + + echo "Waiting for Argo CD server to be available..." + kubectl -n argocd rollout status deploy/argocd-server --timeout=600s + + echo "Fetching initial admin password..." + ADMIN_PWD=$(kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath='{.data.password}' | base64 -d || true) + if [ -z "$ADMIN_PWD" ]; then + echo "Admin password not found. Exiting." + exit 1 + fi + + echo "Requesting Argo CD auth token..." + API_URL="https://argocd-server.argocd.svc/api/v1" + if command -v curl >/dev/null 2>&1; then + RESPONSE=$(curl -ksS -X POST "$API_URL/session" \ + -H "Content-Type: application/json" \ + -d "{\"username\":\"admin\",\"password\":\"${ADMIN_PWD}\"}" || true) + TOKEN=$(echo "$RESPONSE" | sed -n 's/.*"token":"\([^"]*\)".*/\1/p') + if [ -z "$TOKEN" ]; then + echo "HTTPS request failed or no token found, falling back to HTTP..." + API_URL="http://argocd-server.argocd.svc/api/v1" + RESPONSE=$(curl -sS -X POST "$API_URL/session" \ + -H "Content-Type: application/json" \ + -d "{\"username\":\"admin\",\"password\":\"${ADMIN_PWD}\"}" || true) + TOKEN=$(echo "$RESPONSE" | sed -n 's/.*"token":"\([^"]*\)".*/\1/p') + fi + elif command -v wget >/dev/null 2>&1; then + RESPONSE=$(wget -qO- --no-check-certificate --header="Content-Type: application/json" \ + --post-data="{\"username\":\"admin\",\"password\":\"${ADMIN_PWD}\"}" \ + "$API_URL/session" || true) + TOKEN=$(echo "$RESPONSE" | sed -n 's/.*"token":"\([^"]*\)".*/\1/p') + if [ -z "$TOKEN" ]; then + echo "HTTPS request failed or no token found, falling back to HTTP..." + API_URL="http://argocd-server.argocd.svc/api/v1" + RESPONSE=$(wget -qO- --header="Content-Type: application/json" \ + --post-data="{\"username\":\"admin\",\"password\":\"${ADMIN_PWD}\"}" \ + "$API_URL/session" || true) + TOKEN=$(echo "$RESPONSE" | sed -n 's/.*"token":"\([^"]*\)".*/\1/p') + fi + else + echo "Neither curl nor wget found in the container." + exit 1 + fi + if [ -z "$TOKEN" ]; then + echo "Failed to obtain token over HTTPS and HTTP. Last response: $RESPONSE" + exit 1 + fi + + echo "Writing token to secret 'argocd-admin-token' in namespace 'vault'..." + kubectl -n vault create secret generic argocd-admin-token \ + --from-literal=token="$TOKEN" \ + --from-literal=apiUrl="http://argocd-server.argocd.svc:443" \ + --dry-run=client -o yaml | kubectl apply -f - + echo "Done." + diff --git a/caipe/workshop/vault/manifests/ingress.yaml b/caipe/workshop/vault/manifests/ingress.yaml new file mode 100644 index 00000000..c2ae63d9 --- /dev/null +++ b/caipe/workshop/vault/manifests/ingress.yaml @@ -0,0 +1,39 @@ +apiVersion: networking.k8s.io/v1 +kind: Ingress +metadata: + name: vault-ingress + namespace: vault + annotations: + nginx.ingress.kubernetes.io/backend-protocol: "HTTP" +spec: + ingressClassName: "nginx" + rules: + - http: + paths: + - path: / + pathType: Prefix + backend: + service: + name: vault + port: + number: 8200 + - host: vault.cnoe.localtest.me + http: + paths: + - path: / + pathType: Prefix + backend: + service: + name: vault + port: + number: 8200 + - host: localhost + http: + paths: + - path: /vault(/|$)(.*) + pathType: ImplementationSpecific + backend: + service: + name: vault + port: + number: 8200 diff --git a/caipe/workshop/vault/manifests/secret-store.yaml b/caipe/workshop/vault/manifests/secret-store.yaml new file mode 100644 index 00000000..f78a163a --- /dev/null +++ b/caipe/workshop/vault/manifests/secret-store.yaml @@ -0,0 +1,17 @@ +apiVersion: external-secrets.io/v1beta1 +kind: ClusterSecretStore +metadata: + name: vault-secret-store + annotations: + argocd.argoproj.io/sync-wave: "2" +spec: + provider: + vault: + server: "http://vault.vault.svc.cluster.local:8200" + path: "secret" + version: "v2" + auth: + tokenSecretRef: + name: "vault-root-token" + key: "token" + namespace: "vault" diff --git a/ai-platform-engineering/vault/manifests/vault-config-job.yaml b/caipe/workshop/vault/manifests/vault-config-job.yaml similarity index 98% rename from ai-platform-engineering/vault/manifests/vault-config-job.yaml rename to caipe/workshop/vault/manifests/vault-config-job.yaml index 95a6bdd3..2b51a177 100644 --- a/ai-platform-engineering/vault/manifests/vault-config-job.yaml +++ b/caipe/workshop/vault/manifests/vault-config-job.yaml @@ -108,7 +108,8 @@ spec: # Backstage agent secrets (empty data as per example) echo "Creating backstage-secret..." vault kv put secret/ai-platform-engineering/backstage-secret \ - placeholder="empty" + BACKSTAGE_API_TOKEN="" \ + BACKSTAGE_URL="" # Slack agent secrets echo "Creating slack-secret..." diff --git a/caipe/workshop/vault/manifests/vault-init-job.yaml b/caipe/workshop/vault/manifests/vault-init-job.yaml new file mode 100644 index 00000000..a0e5fef3 --- /dev/null +++ b/caipe/workshop/vault/manifests/vault-init-job.yaml @@ -0,0 +1,230 @@ +apiVersion: batch/v1 +kind: Job +metadata: + name: vault-init + namespace: vault + annotations: + argocd.argoproj.io/sync-wave: "0" + argocd.argoproj.io/hook-delete-policy: HookSucceeded +spec: + template: + spec: + serviceAccountName: vault-init + restartPolicy: OnFailure + containers: + - name: vault-init + image: docker.io/library/ubuntu:22.04 + env: + - name: VAULT_ADDR + value: "http://vault:8200" + - name: VAULT_SKIP_VERIFY + value: "true" + + command: ["/bin/bash", "-c"] + args: + - | + #! /bin/bash + + set -ex -o pipefail + + # Install required tools FIRST + echo "Installing required tools..." + apt-get update -qq || { echo "ERROR: apt-get update failed"; exit 1; } + apt-get install -y -qq curl netcat-openbsd software-properties-common wget jq libcap2-bin || { echo "ERROR: package installation failed"; exit 1; } + + # Install Vault CLI + echo "Installing Vault CLI..." + wget -O- https://apt.releases.hashicorp.com/gpg | gpg --dearmor > /usr/share/keyrings/hashicorp-archive-keyring.gpg + echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" > /etc/apt/sources.list.d/hashicorp.list + apt-get update + apt-get install vault -y + setcap cap_ipc_lock= /usr/bin/vault + + # Install kubectl + echo "Installing kubectl..." + curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl" || { echo "ERROR: kubectl download failed"; exit 1; } + chmod +x kubectl + mv kubectl /usr/local/bin/ + + echo "Tools installed successfully. Checking connectivity..." + + echo "Waiting for Vault to be ready for initialization or unsealing..." + ATTEMPTS=0 + MAX_ATTEMPTS=24 # 24 * 10s = 120s = 4 minutes + while true; do + HTTP_CODE=$(curl -s -o /dev/null -w '%{http_code}' http://vault:8200/v1/sys/health) + # 501 = not initialized (ready for init), 503 = sealed (ready for unseal), 200 = ready + if [ "$HTTP_CODE" = "501" ] || [ "$HTTP_CODE" = "503" ] || [ "$HTTP_CODE" = "200" ]; then + echo "Vault is ready (HTTP $HTTP_CODE)" + break + fi + + ATTEMPTS=$((ATTEMPTS + 1)) + if [ "$ATTEMPTS" -gt "$MAX_ATTEMPTS" ]; then + echo "Error: Timed out waiting for Vault to become ready. Last HTTP code: $HTTP_CODE" + exit 1 + fi + echo "Vault not ready yet (HTTP $HTTP_CODE, attempt $ATTEMPTS/$MAX_ATTEMPTS). Retrying in 10 seconds..." + sleep 10 + done + + echo "Checking Vault initialization status..." + + # Check vault status via API + STATUS=$(curl -s http://vault:8200/v1/sys/health || echo '{"initialized":false,"sealed":true}') + INITIALIZED=$(echo $STATUS | grep -o '"initialized":[^,}]*' | cut -d: -f2 | tr -d '"') + SEALED=$(echo $STATUS | grep -o '"sealed":[^,}]*' | cut -d: -f2 | tr -d '"') + + echo "Vault status: initialized=$INITIALIZED, sealed=$SEALED" + + # If vault is unsealed, we're done + if [ "$SEALED" = "false" ]; then + echo "Vault is already unsealed and ready" + exit 0 + fi + + # If vault is initialized but sealed, try to unseal + if [ "$INITIALIZED" = "true" ]; then + echo "Vault is initialized but sealed. Checking for existing unseal key..." + + if kubectl get secret vault-unseal-key -n vault >/dev/null 2>&1; then + echo "Found existing unseal key, unsealing vault..." + UNSEAL_KEY=$(kubectl get secret vault-unseal-key -n vault -o jsonpath='{.data.key}' | base64 -d) + + # Unseal via API + UNSEAL_RESPONSE=$(curl -s -X POST -H "Content-Type: application/json" -d "{\"key\":\"$UNSEAL_KEY\"}" http://vault:8200/v1/sys/unseal 2>/dev/null) + if echo $UNSEAL_RESPONSE | grep -q '"sealed":false'; then + echo "Vault unsealed successfully!" + exit 0 + else + echo "Failed to unseal vault with stored key" + exit 1 + fi + else + echo "No unseal key found. Vault was initialized externally." + echo "For development environment, clearing vault data to allow re-initialization..." + + # Find and delete the PVC associated with the vault-0 pod + PVC_NAME=$(kubectl get pod vault-0 -n vault -o jsonpath='{.spec.volumes[?(@.persistentVolumeClaim)].persistentVolumeClaim.claimName}' 2>/dev/null) + + if [ -n "$PVC_NAME" ]; then + echo "Found PVC '$PVC_NAME', deleting it to clear Vault state..." + kubectl delete pvc "$PVC_NAME" -n vault + else + echo "No PVC found attached to pod vault-0. Skipping PVC deletion." + fi + + # Now, force-delete the pod to trigger a restart with a fresh volume + echo "Deleting pod vault-0 to reset its state..." + kubectl delete pod vault-0 -n vault --force --grace-period=0 + + # Wait for the pod to be recreated + echo "Waiting for vault pod to be recreated..." + sleep 10 + kubectl wait --for=jsonpath='{.status.phase}'=Running pod/vault-0 -n vault --timeout=120s + + # Wait for vault service to be available again + echo "Waiting for vault service to be available after restart..." + sleep 5 + until nc -z vault 8200 2>/dev/null; do + echo "Waiting for vault service..." + sleep 5 + done + + echo "Vault restarted. Checking if it's now uninitialized..." + # Re-check vault status after restart + STATUS=$(curl -s http://vault:8200/v1/sys/health 2>/dev/null || echo '{"initialized":false,"sealed":true}') + INITIALIZED=$(echo $STATUS | grep -o '"initialized":[^,}]*' | cut -d: -f2 | tr -d '"') + + if [ "$INITIALIZED" = "false" ]; then + echo "Vault is now uninitialized. Proceeding with initialization..." + else + echo "Vault is still initialized after restart. Manual intervention required." + exit 1 + fi + fi + fi + + # Vault is not initialized, initialize it + echo "Vault is not initialized. Initializing now..." + set +x # Disable logging for the next command + INIT_RESPONSE=$(vault operator init -format=json -key-shares=1 -key-threshold=1) + set -x # Re-enable logging + + if [ -z "$INIT_RESPONSE" ]; then + echo "Failed to initialize vault. The init command returned no output." + exit 1 + fi + + # Extract keys from response using jq, with logging disabled + set +x + UNSEAL_KEY=$(echo "$INIT_RESPONSE" | jq -r .unseal_keys_b64[0]) + ROOT_TOKEN=$(echo "$INIT_RESPONSE" | jq -r .root_token) + set -x + + if [ -z "$UNSEAL_KEY" ] || [ "$UNSEAL_KEY" = "null" ]; then + echo "Failed to extract unseal key from init response." + exit 1 + fi + + echo "Unsealing Vault..." + set +x # Disable logging for the unseal command + vault operator unseal "$UNSEAL_KEY" + UNSEAL_EXIT_CODE=$? + set -x # Re-enable logging + + if [ $UNSEAL_EXIT_CODE -ne 0 ]; then + echo "Failed to unseal vault. Manual intervention required." + exit 1 + fi + echo "Vault unsealed successfully!" + + echo "Storing credentials in Kubernetes secrets..." + # The following commands are safe because the secret is piped, not part of the command args + echo -n "$UNSEAL_KEY" | kubectl create secret generic vault-unseal-key \ + --from-file=key=/dev/stdin \ + --namespace=vault \ + --dry-run=client -o yaml | kubectl apply -f - + + echo -n "$ROOT_TOKEN" | kubectl create secret generic vault-root-token \ + --from-file=token=/dev/stdin \ + --namespace=vault \ + --dry-run=client -o yaml | kubectl apply -f - + + echo "Vault initialization completed successfully!" +--- +apiVersion: v1 +kind: ServiceAccount +metadata: + name: vault-init + namespace: vault + +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: Role +metadata: + name: vault-init + namespace: vault + +rules: +- apiGroups: [""] + resources: ["secrets"] + verbs: ["create", "get", "patch", "update"] +- apiGroups: [""] + resources: ["pods"] + verbs: ["delete", "get", "list", "watch"] +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: RoleBinding +metadata: + name: vault-init + namespace: vault + +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: Role + name: vault-init +subjects: +- kind: ServiceAccount + name: vault-init + namespace: vault diff --git a/caipe/workshop/vault/manifests/vault-unsealer.yaml b/caipe/workshop/vault/manifests/vault-unsealer.yaml new file mode 100644 index 00000000..f8479f0a --- /dev/null +++ b/caipe/workshop/vault/manifests/vault-unsealer.yaml @@ -0,0 +1,132 @@ +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + name: vault-unsealer + namespace: vault + annotations: + argocd.argoproj.io/sync-wave: "2" + labels: + app: vault-unsealer +spec: + replicas: 1 + selector: + matchLabels: + app: vault-unsealer + template: + metadata: + labels: + app: vault-unsealer + spec: + serviceAccountName: vault-init + restartPolicy: Always + containers: + - name: vault-unsealer + image: docker.io/library/ubuntu:22.04 + env: + - name: VAULT_ADDR + value: "http://vault:8200" + - name: VAULT_SKIP_VERIFY + value: "true" + command: ["/bin/bash", "-c"] + args: + - | + #! /bin/bash + + set -e -o pipefail + + # Install required tools + echo "Installing required tools..." + apt-get update -qq + apt-get install -y -qq curl jq + + # Install kubectl + echo "Installing kubectl..." + curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl" + chmod +x kubectl + mv kubectl /usr/local/bin/ + + echo "Vault unsealer started. Monitoring Vault status..." + + # Function to check and unseal vault + check_and_unseal() { + local http_code=$(curl -s -o /dev/null -w '%{http_code}' http://vault:8200/v1/sys/health 2>/dev/null || echo "000") + + case $http_code in + 200|429) + echo "$(date): Vault is unsealed and ready (HTTP $http_code)" + return 0 + ;; + 503) + echo "$(date): Vault is sealed (HTTP $http_code), attempting to unseal..." + + # Check if unseal key exists + if kubectl get secret vault-unseal-key -n vault >/dev/null 2>&1; then + UNSEAL_KEY=$(kubectl get secret vault-unseal-key -n vault -o jsonpath='{.data.key}' | base64 -d) + + # Attempt to unseal + UNSEAL_RESPONSE=$(curl -s -X POST -H "Content-Type: application/json" \ + -d "{\"key\":\"$UNSEAL_KEY\"}" \ + http://vault:8200/v1/sys/unseal 2>/dev/null) + + if echo "$UNSEAL_RESPONSE" | jq -r '.sealed' 2>/dev/null | grep -q "false"; then + echo "$(date): Successfully unsealed Vault!" + return 0 + else + echo "$(date): Failed to unseal Vault. Response: $UNSEAL_RESPONSE" + return 1 + fi + else + echo "$(date): No unseal key found in vault-unseal-key secret" + return 1 + fi + ;; + 501) + echo "$(date): Vault is not initialized (HTTP $http_code)" + return 1 + ;; + 000) + echo "$(date): Vault is not reachable" + return 1 + ;; + *) + echo "$(date): Vault returned unexpected status (HTTP $http_code)" + return 1 + ;; + esac + } + + # Main monitoring loop + while true; do + if ! check_and_unseal; then + echo "$(date): Vault check failed, will retry in 30 seconds..." + fi + sleep 30 + done + resources: + requests: + memory: "64Mi" + cpu: "50m" + limits: + memory: "128Mi" + cpu: "100m" + livenessProbe: + exec: + command: + - /bin/bash + - -c + - "curl -s http://vault:8200/v1/sys/health >/dev/null" + initialDelaySeconds: 60 + periodSeconds: 60 + timeoutSeconds: 10 + failureThreshold: 3 + readinessProbe: + exec: + command: + - /bin/bash + - -c + - "curl -s http://vault:8200/v1/sys/health >/dev/null" + initialDelaySeconds: 30 + periodSeconds: 30 + timeoutSeconds: 5 + failureThreshold: 2 diff --git a/caipe/workshop/vault/values.yaml b/caipe/workshop/vault/values.yaml new file mode 100644 index 00000000..b30669b7 --- /dev/null +++ b/caipe/workshop/vault/values.yaml @@ -0,0 +1,19 @@ +# This values file configures the official Vault Helm chart. + +# Vault UI/API does not support domain subpathing - must use subdomain approach +# Using vault.cnoe.localtest.me instead of cnoe.localtest.me/vault +server: + extraEnvironmentVars: + VAULT_ADDR: 'https://vault.cnoe.localtest.me:8443' + + # Configure readiness probe to use HTTP endpoint instead of vault status command + # Setting 'path' triggers httpGet mode in Helm template instead of exec vault status + readinessProbe: + enabled: true + path: "/v1/sys/health?standbyok=true&sealedcode=204&uninitcode=204" + port: 8200 + initialDelaySeconds: 5 + periodSeconds: 5 + timeoutSeconds: 3 + failureThreshold: 2 + successThreshold: 1 \ No newline at end of file