Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
61 changes: 50 additions & 11 deletions .agents/skills/helm-dev-environment/SKILL.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
name: helm-dev-environment
description: Start up, tear down, and configure the local Kubernetes development environment for OpenShell. Uses k3d (Docker-backed k3s) + Skaffold + Helm. Covers cluster lifecycle, optional add-ons (Keycloak OIDC, Envoy Gateway), HA testing, and port mappings. Trigger keywords - local k8s, local cluster, k3d, skaffold, helm dev, start cluster, stop cluster, tear down cluster, delete cluster, create cluster, helm:k3s, helm:skaffold, local dev environment, dev cluster, k8s dev, envoy gateway local, keycloak local, high availability, HA.
description: Start up, tear down, and configure the local Kubernetes development environment for OpenShell. Uses k3d (Docker-backed k3s) + Skaffold + Helm. Covers cluster lifecycle, optional add-ons (Keycloak OIDC, Envoy Gateway, agentgateway), HA testing, and port mappings. Trigger keywords - local k8s, local cluster, k3d, skaffold, helm dev, start cluster, stop cluster, tear down cluster, delete cluster, create cluster, helm:k3s, helm:skaffold, local dev environment, dev cluster, k8s dev, envoy gateway local, agentgateway local, keycloak local, high availability, HA.
metadata:
internal: true
---
Expand Down Expand Up @@ -47,10 +47,12 @@ Port mappings created at cluster time (cannot be changed without recreating):

| Host port | Target | Used by |
|-----------|--------|---------|
| `8080` | Port `80` via k3d load balancer | Envoy Gateway LoadBalancer service (`values-gateway.yaml`) |
| `8080` | Port `80` via k3d load balancer | Gateway API controller LoadBalancer service (for example, Envoy Gateway or agentgateway) |
| Optional | Port `443` via k3d load balancer | Gateway API controller HTTPS listener diagnostics |

Override with env vars before running `helm:k3s:create`:
- `HELM_K3S_LB_HOST_PORT` (default: `8080`)
- `HELM_K3S_LB_TLS_HOST_PORT` (unset by default)
- `HELM_K3S_PRELOAD_SANDBOX_IMAGE` (default:
`nvcr.io/nvidia/base/ubuntu:24.04`; set to an empty value to skip)
- `HELM_K3S_COLLECTOR_IMAGE` (default:
Expand Down Expand Up @@ -271,15 +273,49 @@ The kube e2e wrapper creates only one port-forward, to `svc/openshell`; it no
longer forwards the unauthenticated health listener or runs a `/readyz` e2e
target. `/readyz` remains covered by server unit/integration tests.

Use `mise run e2e:kubernetes:ha-rebalancing` for full-suite HA coverage. The
task creates an external PostgreSQL fixture, installs Envoy Gateway, applies
`deploy/kube/manifests/envoy-gateway-openshell.yaml`, enables the chart
`GRPCRoute`, and runs the full Kubernetes e2e suite, including
`kubernetes_ha_rebalancing`. That coverage validates sandbox create/watch and
exec through the Envoy proxy while gateway replicas scale up, scale down, and
rotate. It also keeps a long-running sandbox alive and runs upload/download
operations while gateway pods roll, so file sync exercises the same relay retry
path as interactive sessions.
Use `mise run e2e:kubernetes:ha-rebalancing` for HA coverage through Envoy
Gateway, the default controller. Select agentgateway with:

```bash
OPENSHELL_E2E_KUBE_GATEWAY_CONTROLLER=agentgateway \
mise run e2e:kubernetes:ha-rebalancing
```

The task creates an external PostgreSQL fixture, enables the chart `GRPCRoute`,
and runs the full Kubernetes e2e suite, including
`kubernetes_ha_rebalancing`. The Envoy selection installs Envoy Gateway and
applies `deploy/kube/manifests/envoy-gateway-openshell.yaml`; the agentgateway
selection installs the pinned agentgateway release while the OpenShell chart
creates its dedicated `Gateway`.
This coverage validates sandbox create/watch and exec through the selected
proxy while gateway replicas scale up, scale down, and rotate. It also keeps a
long-running sandbox alive and runs upload/download operations while gateway
pods roll, so file sync exercises the same relay retry path as interactive
sessions.

Use `mise run e2e:kubernetes:agentgateway-tls` to configure HTTPS directly on
the chart-created agentgateway `Gateway`, provision a short-lived certificate,
and run CLI conformance plus the focused Rust `port_forward` test through the
verified HTTPS listener. This covers status and sandbox lifecycle operations as
well as SSH relay setup and TCP data transfer without the broad suite's
unrelated host fixtures.

Use `mise run e2e:kubernetes:agentgateway-backend-tls` to exercise the dedicated
Gateway's HTTPS listener while keeping TLS enabled on the OpenShell pod. The
task waits for the standard `BackendTLSPolicy` to be accepted and its references
resolved, then runs CLI conformance through agentgateway's re-encrypted backend
connection.

For transport diagnostics on an ephemeral k3d cluster, set
`OPENSHELL_E2E_KUBE_DIRECT_GATEWAY_PORT` to reach agentgateway TLS through the
k3d load balancer instead of `kubectl port-forward`. The wrapper maps that host
port to load-balancer port 443 and otherwise preserves the selected agentgateway
topology and test command.

To test an alternate agentgateway build, set `OPENSHELL_AGENTGATEWAY_CHART` and
`OPENSHELL_AGENTGATEWAY_CRDS_CHART` to Helm OCI chart references and set
`OPENSHELL_AGENTGATEWAY_VERSION` to their shared version. Defaults remain the
pinned release from `cr.agentgateway.dev`.

If you reuse an existing Skaffold cluster for the full kube suite, make sure the
chart has `server.hostGatewayIP` set so sandbox pods can resolve
Expand Down Expand Up @@ -445,6 +481,9 @@ for dependencies still declared in `Chart.yaml`.
| `deploy/helm/openshell/ci/values-skaffold.yaml` | Dev overrides (image pull policy, TLS disabled for local Skaffold) |
| `deploy/helm/openshell/ci/values-cert-manager.yaml` | cert-manager PKI overlay (opt-in; disables pkiInitJob) |
| `deploy/helm/openshell/ci/values-gateway.yaml` | Envoy Gateway GRPCRoute + Gateway overlay |
| `deploy/helm/openshell/ci/values-gateway-agentgateway.yaml` | dedicated agentgateway Gateway + GRPCRoute overlay |
| `deploy/helm/openshell/ci/values-gateway-agentgateway-tls.yaml` | dedicated agentgateway HTTPS Gateway overlay |
| `deploy/helm/openshell/ci/values-gateway-agentgateway-backend-tls.yaml` | dedicated agentgateway frontend and backend TLS overlay |
| `deploy/helm/openshell/ci/values-high-availability.yaml` | HA test overlay (`replicaCount: 2` with external PostgreSQL Secret) |
| `deploy/helm/openshell/ci/values-keycloak.yaml` | Keycloak OIDC overlay |
| `deploy/helm/openshell/ci/values-spire.yaml` | SPIFFE/SPIRE provider token grant overlay |
Expand Down
69 changes: 67 additions & 2 deletions .github/workflows/branch-e2e.yml
Original file line number Diff line number Diff line change
Expand Up @@ -443,7 +443,72 @@ jobs:
external-postgres-secret: openshell-ha-pg
test-name: kubernetes_ha_rebalancing
kubernetes-features: e2e,e2e-host-gateway,e2e-kubernetes,e2e-kubernetes-ha
use-envoy-gateway: true
gateway-controller: envoy
conformance-artifact-prefix: openshell-conformance

kubernetes-agentgateway-e2e:
needs: [pr_metadata, build-binaries, build-images]
if: needs.pr_metadata.outputs.should_run == 'true' && needs.pr_metadata.outputs.run_kubernetes_ha_e2e == 'true'
permissions:
actions: read
contents: read
packages: read
uses: ./.github/workflows/e2e-kubernetes-test.yml
with:
image-tag: ${{ github.sha }}
job-name: Kubernetes agentgateway E2E
e2e-task: e2e:kubernetes:agentgateway
kubernetes-features: e2e,e2e-host-gateway,e2e-kubernetes
gateway-controller: agentgateway
conformance-artifact-prefix: openshell-conformance

kubernetes-agentgateway-ha-e2e:
needs: [pr_metadata, build-binaries, build-images]
if: needs.pr_metadata.outputs.should_run == 'true' && needs.pr_metadata.outputs.run_kubernetes_ha_e2e == 'true'
permissions:
actions: read
contents: read
packages: read
uses: ./.github/workflows/e2e-kubernetes-test.yml
with:
image-tag: ${{ github.sha }}
job-name: Kubernetes agentgateway HA E2E
e2e-task: e2e:kubernetes:ha-rebalancing
extra-helm-values: deploy/helm/openshell/ci/values-high-availability.yaml
external-postgres-secret: openshell-ha-pg
test-name: kubernetes_ha_rebalancing
kubernetes-features: e2e,e2e-host-gateway,e2e-kubernetes,e2e-kubernetes-ha
gateway-controller: agentgateway
conformance-artifact-prefix: openshell-conformance

kubernetes-agentgateway-tls-e2e:
needs: [pr_metadata, build-binaries, build-images]
if: needs.pr_metadata.outputs.should_run == 'true' && needs.pr_metadata.outputs.run_kubernetes_ha_e2e == 'true'
permissions:
actions: read
contents: read
packages: read
uses: ./.github/workflows/e2e-kubernetes-test.yml
with:
image-tag: ${{ github.sha }}
job-name: Kubernetes agentgateway TLS E2E
e2e-task: e2e:kubernetes:agentgateway-tls
gateway-controller: agentgateway
conformance-artifact-prefix: openshell-conformance

kubernetes-agentgateway-backend-tls-e2e:
needs: [pr_metadata, build-binaries, build-images]
if: needs.pr_metadata.outputs.should_run == 'true' && needs.pr_metadata.outputs.run_kubernetes_ha_e2e == 'true'
permissions:
actions: read
contents: read
packages: read
uses: ./.github/workflows/e2e-kubernetes-test.yml
with:
image-tag: ${{ github.sha }}
job-name: Kubernetes agentgateway backend TLS E2E
e2e-task: e2e:kubernetes:agentgateway-backend-tls
gateway-controller: agentgateway
conformance-artifact-prefix: openshell-conformance

kubernetes-credential-drivers-e2e:
Expand Down Expand Up @@ -502,7 +567,7 @@ jobs:

kubernetes-ha-e2e-result:
name: Kubernetes HA E2E result
needs: [pr_metadata, kubernetes-ha-e2e]
needs: [pr_metadata, kubernetes-ha-e2e, kubernetes-agentgateway-e2e, kubernetes-agentgateway-ha-e2e, kubernetes-agentgateway-tls-e2e, kubernetes-agentgateway-backend-tls-e2e]
if: always() && needs.pr_metadata.outputs.should_run == 'true' && needs.pr_metadata.outputs.run_kubernetes_ha_e2e == 'true'
runs-on: ubuntu-latest
steps:
Expand Down
10 changes: 5 additions & 5 deletions .github/workflows/e2e-kubernetes-test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -52,11 +52,11 @@ on:
required: false
type: string
default: ""
use-envoy-gateway:
description: "Route the test through Envoy Gateway instead of a direct gateway Service port-forward"
gateway-controller:
description: "Gateway API controller to route through: none, envoy, or agentgateway"
required: false
type: boolean
default: false
type: string
default: "none"
mise-version:
description: "mise version to install on the bare Kubernetes e2e runner"
required: false
Expand Down Expand Up @@ -145,7 +145,7 @@ jobs:
OPENSHELL_E2E_KUBE_EXTERNAL_POSTGRES_SECRET: ${{ inputs.external-postgres-secret }}
OPENSHELL_E2E_KUBE_TEST: ${{ inputs.test-name }}
OPENSHELL_E2E_KUBERNETES_FEATURES: ${{ inputs.kubernetes-features }}
OPENSHELL_E2E_KUBE_USE_ENVOY: ${{ inputs.use-envoy-gateway && '1' || '0' }}
OPENSHELL_E2E_KUBE_GATEWAY_CONTROLLER: ${{ inputs.gateway-controller }}
IMAGE_TAG: ${{ inputs.image-tag }}
OPENSHELL_REGISTRY: ghcr.io/nvidia/openshell
E2E_TASK: ${{ inputs.e2e-task }}
Expand Down
10 changes: 7 additions & 3 deletions crates/openshell-cli/src/ssh.rs
Original file line number Diff line number Diff line change
Expand Up @@ -162,9 +162,13 @@ async fn ssh_session_config(
// to 127.0.0.1 but the cluster is actually running on a remote host.
#[allow(clippy::cast_possible_truncation)]
let gateway_port_u16 = session.gateway_port as u16;
let (gateway_host, gateway_port) =
resolve_ssh_gateway(&session.gateway_host, gateway_port_u16, server);
format_gateway_url(&session.gateway_scheme, &gateway_host, gateway_port)
let (gateway_scheme, gateway_host, gateway_port) = resolve_ssh_gateway(
&session.gateway_scheme,
&session.gateway_host,
gateway_port_u16,
server,
);
format_gateway_url(&gateway_scheme, &gateway_host, gateway_port)
};
let gateway_name = tls
.gateway_name()
Expand Down
Loading
Loading