From e876b74eb9a83ecda9d1b1e349981cfea84c6cef Mon Sep 17 00:00:00 2001 From: ansjindal Date: Thu, 17 Sep 2026 13:36:12 +0200 Subject: [PATCH] feat(helm): make cluster-scoped RBAC optional The gateway chart always rendered the ClusterRole and ClusterRoleBinding, so every install and upgrade required cluster-admin even when only namespaced objects were needed. Installers that are namespace-admin GitOps or platform controllers could not run the release at all, and clusters where cluster-scoped RBAC is owned by a separate team had no supported way to split the install. Add an rbac values block so a cluster-admin can apply the cluster-scoped objects once and a namespace-admin can install and upgrade the release without cluster-scoped permissions: rbac: create: true clusterScoped: create: true clusterRoleName: "" clusterRoleBindingName: "" rbac.clusterScoped.create gates the ClusterRole and ClusterRoleBinding, and is independent of the workspace mode. rbac.create additionally gates the namespaced sandbox Role and RoleBinding, which matters because Kubernetes escalation prevention stops an installer holding only the built-in admin role from creating a Role that grants agents.x-k8s.io verbs it does not itself hold. The certgen hook and credential driver RBAC keep their existing flags. Both flags default to true, so current installs are unchanged. The helpers treat a missing rbac block as enabled so upgrades with --reuse-values do not drop RBAC, matching the existing workspaceResources pattern. The ClusterRoleBinding roleRef follows clusterRoleName so a separately applied ClusterRole can carry a name the cluster-admin chooses. Document the migration for a release that already owns the cluster-scoped objects: Helm deletes objects that leave the manifest, so annotate them with helm.sh/resource-policy=keep before setting the flag, otherwise the gateway loses TokenReview until a cluster-admin re-applies them. Co-Authored-By: Claude Opus 5 (1M context) Signed-off-by: ansjindal --- deploy/helm/openshell/README.md | 75 ++++++++++++++++++ deploy/helm/openshell/README.md.gotmpl | 71 +++++++++++++++++ .../openshell/ci/values-namespace-admin.yaml | 24 ++++++ deploy/helm/openshell/templates/_helpers.tpl | 60 +++++++++++++++ .../helm/openshell/templates/clusterrole.yaml | 4 +- .../templates/clusterrolebinding.yaml | 6 +- deploy/helm/openshell/templates/role.yaml | 2 +- .../helm/openshell/templates/rolebinding.yaml | 2 +- .../openshell/tests/clusterrole_test.yaml | 40 ++++++++++ .../tests/clusterrolebinding_test.yaml | 45 +++++++++++ deploy/helm/openshell/tests/rbac_test.yaml | 43 +++++++++++ deploy/helm/openshell/values.yaml | 20 +++++ deploy/helm/test-split-ownership.sh | 26 +++++++ docs/kubernetes/setup.mdx | 77 ++++++++++++++++++- 14 files changed, 489 insertions(+), 6 deletions(-) create mode 100644 deploy/helm/openshell/ci/values-namespace-admin.yaml create mode 100644 deploy/helm/openshell/tests/rbac_test.yaml diff --git a/deploy/helm/openshell/README.md b/deploy/helm/openshell/README.md index 003475df96..a20dd9ab8c 100644 --- a/deploy/helm/openshell/README.md +++ b/deploy/helm/openshell/README.md @@ -17,6 +17,77 @@ namespace. The gateway and workspace releases can then be upgraded and removed independently. Use Kubernetes `operator` workspace mode when one gateway serves multiple pre-provisioned workspace namespaces. +## Cluster-scoped vs namespaced objects + +Most objects in this chart are namespaced and land in the release namespace. +Only two are cluster-scoped: + +| Object | Default name | +| --- | --- | +| `ClusterRole` | `-node-reader-` | +| `ClusterRoleBinding` | `-node-reader-` | + +By default the release creates both, so an install by a cluster-admin is +unchanged. On clusters where cluster-scoped RBAC is owned by a different team, +split the install in two. + +A cluster-admin applies the cluster-scoped objects once per gateway +ServiceAccount, rendered from the same values the release uses: + +```shell +helm template openshell oci://ghcr.io/nvidia/openshell/helm-chart --version \ + --namespace openshell -f my-values.yaml \ + --set rbac.clusterScoped.create=true \ + --set agentSandbox.preflight.enabled=false \ + --show-only templates/clusterrole.yaml \ + --show-only templates/clusterrolebinding.yaml | kubectl apply -f - +``` + +A namespace-admin then installs and upgrades the release with cluster-scoped +objects omitted, using [`ci/values-namespace-admin.yaml`](ci/values-namespace-admin.yaml) +or the equivalent `--set`: + +```shell +helm upgrade --install openshell oci://ghcr.io/nvidia/openshell/helm-chart --version \ + --namespace openshell -f my-values.yaml \ + --set supervisor.sandboxRuntime.networkPolicyEnforced=true \ + --set rbac.clusterScoped.create=false +``` + +The gateway ServiceAccount name and namespace do not change, so the +pre-created `ClusterRoleBinding` keeps matching the release. This works with +`serviceAccount.create=false` too: the `ClusterRoleBinding` subject follows +`serviceAccount.name`, so render the admin step with the same values. + +### Migrating an existing release + +Helm deletes objects that leave a release manifest, so setting +`rbac.clusterScoped.create=false` on a release that already owns the +`ClusterRole` and `ClusterRoleBinding` deletes them. The gateway then loses +TokenReview until a cluster-admin re-applies them. Hand ownership over first, as +cluster-admin, so nothing is deleted: + +```shell +kubectl annotate clusterrole "openshell-node-reader-" \ + helm.sh/resource-policy=keep --overwrite +kubectl annotate clusterrolebinding "openshell-node-reader-" \ + helm.sh/resource-policy=keep --overwrite +``` + +The objects then survive the upgrade that sets the flag, and the cluster-admin +owns them from that point on. Fresh installs need no such step. + +`rbac.clusterScoped.create` is independent of +`server.drivers.kubernetes.workspaceMode`. Managed and operator modes change +what the `ClusterRole` contains, but they never force the namespaced release to +apply it. Re-run the cluster-admin step after changing values that affect the +`ClusterRole` rules. + +Set `rbac.create=false` to also omit the namespaced sandbox `Role` and +`RoleBinding`. The certgen hook and credential driver RBAC keep their own flags +(`pkiInitJob.enabled` and +`server.credentialDrivers.kubernetesSecrets.rbac.create`). + ## Prerequisites The Kubernetes Agent Sandbox CRDs and controller must be installed on the cluster before deploying OpenShell. Install them with: @@ -230,6 +301,10 @@ discovery endpoint or its TLS CA. | probes.startup.failureThreshold | int | `30` | Startup probe failure threshold before the container is killed. | | probes.startup.periodSeconds | int | `2` | Startup probe period, in seconds. | | probes.startup.timeoutSeconds | int | `1` | Startup probe timeout, in seconds. | +| rbac.clusterScoped.clusterRoleBindingName | string | `""` | Name for the ClusterRoleBinding. Empty uses the `-node-reader-` default. | +| rbac.clusterScoped.clusterRoleName | string | `""` | Name for the ClusterRole. Empty uses the `-node-reader-` default. | +| rbac.clusterScoped.create | bool | `true` | Create the cluster-scoped ClusterRole and ClusterRoleBinding. Disable for a namespace-admin install where a cluster-admin applies them separately; the gateway ServiceAccount name and namespace are unchanged, so a pre-created ClusterRoleBinding still matches. | +| rbac.create | bool | `true` | Create the RBAC objects that grant the gateway ServiceAccount access. Disable to supply the namespaced sandbox Role/RoleBinding and the cluster-scoped ClusterRole/ClusterRoleBinding out of band. The certgen hook and credential driver RBAC keep their own flags. | | replicaCount | int | `1` | Number of OpenShell gateway replicas. Values greater than 1 require server.externalDbSecret because the default SQLite backend is per pod. | | resources | object | `{}` | Gateway pod resource requests and limits. | | sandboxRuntime.image.pullPolicy | string | `""` | Sandbox runtime image pull policy. Defaults to the gateway image pull policy when empty. | diff --git a/deploy/helm/openshell/README.md.gotmpl b/deploy/helm/openshell/README.md.gotmpl index 75e651e214..94969da4ae 100644 --- a/deploy/helm/openshell/README.md.gotmpl +++ b/deploy/helm/openshell/README.md.gotmpl @@ -17,6 +17,77 @@ namespace. The gateway and workspace releases can then be upgraded and removed independently. Use Kubernetes `operator` workspace mode when one gateway serves multiple pre-provisioned workspace namespaces. +## Cluster-scoped vs namespaced objects + +Most objects in this chart are namespaced and land in the release namespace. +Only two are cluster-scoped: + +| Object | Default name | +| --- | --- | +| `ClusterRole` | `-node-reader-` | +| `ClusterRoleBinding` | `-node-reader-` | + +By default the release creates both, so an install by a cluster-admin is +unchanged. On clusters where cluster-scoped RBAC is owned by a different team, +split the install in two. + +A cluster-admin applies the cluster-scoped objects once per gateway +ServiceAccount, rendered from the same values the release uses: + +```shell +helm template openshell oci://ghcr.io/nvidia/openshell/helm-chart --version \ + --namespace openshell -f my-values.yaml \ + --set rbac.clusterScoped.create=true \ + --set agentSandbox.preflight.enabled=false \ + --show-only templates/clusterrole.yaml \ + --show-only templates/clusterrolebinding.yaml | kubectl apply -f - +``` + +A namespace-admin then installs and upgrades the release with cluster-scoped +objects omitted, using [`ci/values-namespace-admin.yaml`](ci/values-namespace-admin.yaml) +or the equivalent `--set`: + +```shell +helm upgrade --install openshell oci://ghcr.io/nvidia/openshell/helm-chart --version \ + --namespace openshell -f my-values.yaml \ + --set supervisor.sandboxRuntime.networkPolicyEnforced=true \ + --set rbac.clusterScoped.create=false +``` + +The gateway ServiceAccount name and namespace do not change, so the +pre-created `ClusterRoleBinding` keeps matching the release. This works with +`serviceAccount.create=false` too: the `ClusterRoleBinding` subject follows +`serviceAccount.name`, so render the admin step with the same values. + +### Migrating an existing release + +Helm deletes objects that leave a release manifest, so setting +`rbac.clusterScoped.create=false` on a release that already owns the +`ClusterRole` and `ClusterRoleBinding` deletes them. The gateway then loses +TokenReview until a cluster-admin re-applies them. Hand ownership over first, as +cluster-admin, so nothing is deleted: + +```shell +kubectl annotate clusterrole "openshell-node-reader-" \ + helm.sh/resource-policy=keep --overwrite +kubectl annotate clusterrolebinding "openshell-node-reader-" \ + helm.sh/resource-policy=keep --overwrite +``` + +The objects then survive the upgrade that sets the flag, and the cluster-admin +owns them from that point on. Fresh installs need no such step. + +`rbac.clusterScoped.create` is independent of +`server.drivers.kubernetes.workspaceMode`. Managed and operator modes change +what the `ClusterRole` contains, but they never force the namespaced release to +apply it. Re-run the cluster-admin step after changing values that affect the +`ClusterRole` rules. + +Set `rbac.create=false` to also omit the namespaced sandbox `Role` and +`RoleBinding`. The certgen hook and credential driver RBAC keep their own flags +(`pkiInitJob.enabled` and +`server.credentialDrivers.kubernetesSecrets.rbac.create`). + ## Prerequisites The Kubernetes Agent Sandbox CRDs and controller must be installed on the cluster before deploying OpenShell. Install them with: diff --git a/deploy/helm/openshell/ci/values-namespace-admin.yaml b/deploy/helm/openshell/ci/values-namespace-admin.yaml new file mode 100644 index 0000000000..425123417e --- /dev/null +++ b/deploy/helm/openshell/ci/values-namespace-admin.yaml @@ -0,0 +1,24 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +# Namespace-admin overlay — renders only namespaced objects. +# +# Use this when a cluster-admin applies the gateway ClusterRole and +# ClusterRoleBinding once, out of band, and the OpenShell release is installed +# and upgraded by an installer that holds no cluster-scoped permissions. +# +# Generate the cluster-scoped objects for the cluster-admin step from the same +# release values, then apply them as cluster-admin: +# helm template openshell oci://ghcr.io/nvidia/openshell/helm-chart \ +# --version --namespace openshell -f my-values.yaml \ +# --set rbac.clusterScoped.create=true \ +# --set agentSandbox.preflight.enabled=false \ +# --show-only templates/clusterrole.yaml \ +# --show-only templates/clusterrolebinding.yaml | kubectl apply -f - +# +# The gateway ServiceAccount name and release namespace are unchanged, so the +# pre-created ClusterRoleBinding still matches this release. + +rbac: + clusterScoped: + create: false diff --git a/deploy/helm/openshell/templates/_helpers.tpl b/deploy/helm/openshell/templates/_helpers.tpl index 8ce4c73656..1a6a6e018e 100644 --- a/deploy/helm/openshell/templates/_helpers.tpl +++ b/deploy/helm/openshell/templates/_helpers.tpl @@ -83,6 +83,66 @@ default to enabled so upgrades with --reuse-values preserve the old topology. {{- if $enabled -}}true{{- end -}} {{- end }} +{{/* +Whether this chart owns gateway RBAC objects. Missing legacy values default to +enabled so upgrades with --reuse-values preserve the old topology. +*/}} +{{- define "openshell.rbacCreate" -}} +{{- $rbac := .Values.rbac | default dict -}} +{{- $create := true -}} +{{- if hasKey $rbac "create" -}} +{{- $create = get $rbac "create" -}} +{{- end -}} +{{- if $create -}}true{{- end -}} +{{- end }} + +{{/* +The rbac.clusterScoped values map, tolerating missing legacy values. +*/}} +{{- define "openshell.clusterScopedRbacValues" -}} +{{- $rbac := .Values.rbac | default dict -}} +{{- $clusterScoped := dict -}} +{{- if hasKey $rbac "clusterScoped" -}} +{{- $clusterScoped = get $rbac "clusterScoped" | default dict -}} +{{- end -}} +{{- toYaml $clusterScoped -}} +{{- end }} + +{{/* +Whether this chart owns the cluster-scoped ClusterRole and ClusterRoleBinding. +Disable for a namespace-admin install where a cluster-admin applies them +separately. Missing legacy values default to enabled. +*/}} +{{- define "openshell.clusterRbacCreate" -}} +{{- if include "openshell.rbacCreate" . -}} +{{- $clusterScoped := include "openshell.clusterScopedRbacValues" . | fromYaml -}} +{{- $create := true -}} +{{- if hasKey $clusterScoped "create" -}} +{{- $create = get $clusterScoped "create" -}} +{{- end -}} +{{- if $create -}}true{{- end -}} +{{- end -}} +{{- end }} + +{{/* +Name of the gateway ClusterRole. The release namespace is part of the default +name so multiple releases on one cluster do not collide. +*/}} +{{- define "openshell.clusterRoleName" -}} +{{- $clusterScoped := include "openshell.clusterScopedRbacValues" . | fromYaml -}} +{{- $default := printf "%s-node-reader-%s" (include "openshell.fullname" .) .Release.Namespace -}} +{{- default $default (get $clusterScoped "clusterRoleName") -}} +{{- end }} + +{{/* +Name of the gateway ClusterRoleBinding. +*/}} +{{- define "openshell.clusterRoleBindingName" -}} +{{- $clusterScoped := include "openshell.clusterScopedRbacValues" . | fromYaml -}} +{{- $default := printf "%s-node-reader-%s" (include "openshell.fullname" .) .Release.Namespace -}} +{{- default $default (get $clusterScoped "clusterRoleBindingName") -}} +{{- end }} + {{/* Gateway image reference. Uses image.tag when set; falls back to .Chart.AppVersion so a released chart automatically pulls the matching image without extra overrides. diff --git a/deploy/helm/openshell/templates/clusterrole.yaml b/deploy/helm/openshell/templates/clusterrole.yaml index e48c27cdd2..ce13298b57 100644 --- a/deploy/helm/openshell/templates/clusterrole.yaml +++ b/deploy/helm/openshell/templates/clusterrole.yaml @@ -1,3 +1,4 @@ +{{- if include "openshell.clusterRbacCreate" . }} # SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. # SPDX-License-Identifier: Apache-2.0 @@ -5,7 +6,7 @@ apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: - name: {{ include "openshell.fullname" . }}-node-reader-{{ .Release.Namespace }} + name: {{ include "openshell.clusterRoleName" . }} labels: {{- include "openshell.labels" . | nindent 4 }} rules: @@ -165,3 +166,4 @@ rules: - update {{- end }} {{- end }} +{{- end }} diff --git a/deploy/helm/openshell/templates/clusterrolebinding.yaml b/deploy/helm/openshell/templates/clusterrolebinding.yaml index 9b3b254586..8de246730b 100644 --- a/deploy/helm/openshell/templates/clusterrolebinding.yaml +++ b/deploy/helm/openshell/templates/clusterrolebinding.yaml @@ -1,17 +1,19 @@ +{{- if include "openshell.clusterRbacCreate" . }} # SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. # SPDX-License-Identifier: Apache-2.0 apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: - name: {{ include "openshell.fullname" . }}-node-reader-{{ .Release.Namespace }} + name: {{ include "openshell.clusterRoleBindingName" . }} labels: {{- include "openshell.labels" . | nindent 4 }} roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole - name: {{ include "openshell.fullname" . }}-node-reader-{{ .Release.Namespace }} + name: {{ include "openshell.clusterRoleName" . }} subjects: - kind: ServiceAccount name: {{ include "openshell.serviceAccountName" . }} namespace: {{ .Release.Namespace }} +{{- end }} diff --git a/deploy/helm/openshell/templates/role.yaml b/deploy/helm/openshell/templates/role.yaml index c781b46e3d..807e173632 100644 --- a/deploy/helm/openshell/templates/role.yaml +++ b/deploy/helm/openshell/templates/role.yaml @@ -1,5 +1,5 @@ {{- $workspaceMode := .Values.server.drivers.kubernetes.workspaceMode | default "shared" -}} -{{- if and (eq $workspaceMode "shared") (include "openshell.workspaceResourcesEnabled" .) }} +{{- if and (eq $workspaceMode "shared") (include "openshell.workspaceResourcesEnabled" .) (include "openshell.rbacCreate" .) }} # SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. # SPDX-License-Identifier: Apache-2.0 apiVersion: rbac.authorization.k8s.io/v1 diff --git a/deploy/helm/openshell/templates/rolebinding.yaml b/deploy/helm/openshell/templates/rolebinding.yaml index 32f11644bf..49cc6f7fba 100644 --- a/deploy/helm/openshell/templates/rolebinding.yaml +++ b/deploy/helm/openshell/templates/rolebinding.yaml @@ -1,5 +1,5 @@ {{- $workspaceMode := .Values.server.drivers.kubernetes.workspaceMode | default "shared" -}} -{{- if and (eq $workspaceMode "shared") (include "openshell.workspaceResourcesEnabled" .) }} +{{- if and (eq $workspaceMode "shared") (include "openshell.workspaceResourcesEnabled" .) (include "openshell.rbacCreate" .) }} # SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. # SPDX-License-Identifier: Apache-2.0 apiVersion: rbac.authorization.k8s.io/v1 diff --git a/deploy/helm/openshell/tests/clusterrole_test.yaml b/deploy/helm/openshell/tests/clusterrole_test.yaml index 9639232c32..73205acedf 100644 --- a/deploy/helm/openshell/tests/clusterrole_test.yaml +++ b/deploy/helm/openshell/tests/clusterrole_test.yaml @@ -186,3 +186,43 @@ tests: apiGroups: [""] resources: ["secrets"] verbs: ["create", "list", "delete"] + + - it: omits the ClusterRole when cluster-scoped RBAC is disabled + set: + rbac.clusterScoped.create: false + asserts: + - hasDocuments: + count: 0 + + - it: omits the ClusterRole when all chart-managed RBAC is disabled + set: + rbac.create: false + asserts: + - hasDocuments: + count: 0 + + - it: omits the ClusterRole independently of the workspace mode + set: + rbac.clusterScoped.create: false + server.drivers.kubernetes.workspaceMode: managed + asserts: + - hasDocuments: + count: 0 + + - it: uses the configured ClusterRole name + set: + rbac.clusterScoped.clusterRoleName: platform-openshell-node-reader + asserts: + - equal: + path: metadata.name + value: platform-openshell-node-reader + + - it: creates the ClusterRole when legacy values omit the rbac block + set: + rbac: null + asserts: + - hasDocuments: + count: 1 + - equal: + path: metadata.name + value: openshell-node-reader-my-namespace diff --git a/deploy/helm/openshell/tests/clusterrolebinding_test.yaml b/deploy/helm/openshell/tests/clusterrolebinding_test.yaml index 5b9a83c04c..4cbf6d7d91 100644 --- a/deploy/helm/openshell/tests/clusterrolebinding_test.yaml +++ b/deploy/helm/openshell/tests/clusterrolebinding_test.yaml @@ -29,3 +29,48 @@ tests: kind: ServiceAccount name: openshell namespace: my-namespace + + - it: omits the ClusterRoleBinding when cluster-scoped RBAC is disabled + set: + rbac.clusterScoped.create: false + asserts: + - hasDocuments: + count: 0 + + - it: omits the ClusterRoleBinding when all chart-managed RBAC is disabled + set: + rbac.create: false + asserts: + - hasDocuments: + count: 0 + + - it: uses the configured ClusterRoleBinding and ClusterRole names + set: + rbac.clusterScoped.clusterRoleName: platform-openshell-node-reader + rbac.clusterScoped.clusterRoleBindingName: platform-openshell-node-reader-binding + asserts: + - equal: + path: metadata.name + value: platform-openshell-node-reader-binding + - equal: + path: roleRef.name + value: platform-openshell-node-reader + + - it: binds the custom gateway service account created by the namespaced release + set: + serviceAccount.create: false + serviceAccount.name: my-existing-sa + asserts: + - contains: + path: subjects + content: + kind: ServiceAccount + name: my-existing-sa + namespace: my-namespace + + - it: creates the ClusterRoleBinding when legacy values omit the rbac block + set: + rbac: null + asserts: + - hasDocuments: + count: 1 diff --git a/deploy/helm/openshell/tests/rbac_test.yaml b/deploy/helm/openshell/tests/rbac_test.yaml new file mode 100644 index 0000000000..d76968a317 --- /dev/null +++ b/deploy/helm/openshell/tests/rbac_test.yaml @@ -0,0 +1,43 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +suite: Gateway RBAC creation +templates: + - templates/role.yaml + - templates/rolebinding.yaml +release: + name: openshell + namespace: my-namespace + +tests: + - it: creates the namespaced sandbox Role and RoleBinding by default + asserts: + - hasDocuments: + count: 1 + - equal: + path: metadata.name + value: openshell-sandbox + + - it: keeps the namespaced Role and RoleBinding when only cluster-scoped RBAC is disabled + set: + rbac.clusterScoped.create: false + asserts: + - hasDocuments: + count: 1 + - equal: + path: metadata.name + value: openshell-sandbox + + - it: omits the namespaced Role and RoleBinding when chart-managed RBAC is disabled + set: + rbac.create: false + asserts: + - hasDocuments: + count: 0 + + - it: creates the namespaced Role and RoleBinding when legacy values omit the rbac block + set: + rbac: null + asserts: + - hasDocuments: + count: 1 diff --git a/deploy/helm/openshell/values.yaml b/deploy/helm/openshell/values.yaml index 67d5587ec9..a39085be8b 100644 --- a/deploy/helm/openshell/values.yaml +++ b/deploy/helm/openshell/values.yaml @@ -93,6 +93,26 @@ sandboxServiceAccount: # -- Existing service account name for sandbox pods when sandboxServiceAccount.create is false. name: "" +# RBAC objects created for the gateway ServiceAccount. Cluster-scoped objects +# can be omitted so a cluster-admin applies them once and a namespace-admin +# installs and upgrades the gateway release without cluster-scoped permissions. +rbac: + # -- Create the RBAC objects that grant the gateway ServiceAccount access. + # Disable to supply the namespaced sandbox Role/RoleBinding and the + # cluster-scoped ClusterRole/ClusterRoleBinding out of band. The certgen hook + # and credential driver RBAC keep their own flags. + create: true + clusterScoped: + # -- Create the cluster-scoped ClusterRole and ClusterRoleBinding. Disable + # for a namespace-admin install where a cluster-admin applies them + # separately; the gateway ServiceAccount name and namespace are unchanged, + # so a pre-created ClusterRoleBinding still matches. + create: true + # -- Name for the ClusterRole. Empty uses the `-node-reader-` default. + clusterRoleName: "" + # -- Name for the ClusterRoleBinding. Empty uses the `-node-reader-` default. + clusterRoleBindingName: "" + # Namespace-scoped resources needed to run sandboxes. Disable this when the # gateway and workspace prerequisites are managed as separate Helm releases # using the openshell-workspace chart. diff --git a/deploy/helm/test-split-ownership.sh b/deploy/helm/test-split-ownership.sh index a7a865363b..249c061e24 100755 --- a/deploy/helm/test-split-ownership.sh +++ b/deploy/helm/test-split-ownership.sh @@ -22,6 +22,32 @@ if yq ea -e \ exit 1 fi +helm template openshell "${repo_root}/deploy/helm/openshell" \ + --namespace openshell \ + --set agentSandbox.preflight.enabled=false \ + --set supervisor.sandboxRuntime.networkPolicyEnforced=true \ + --set rbac.clusterScoped.create=false \ + >"${work_dir}/namespace-admin.yaml" + +cluster_scoped_kinds="$( + yq ea -N -r \ + 'select(.kind == "ClusterRole" or .kind == "ClusterRoleBinding") | + [.kind, .metadata.name] | join(" ")' \ + "${work_dir}/namespace-admin.yaml" +)" +if [[ -n "${cluster_scoped_kinds}" ]]; then + echo "gateway chart rendered cluster-scoped objects despite rbac.clusterScoped.create=false:" >&2 + echo "${cluster_scoped_kinds}" >&2 + exit 1 +fi + +if ! yq ea -e \ + 'select(.kind == "ServiceAccount" and .metadata.name == "openshell")' \ + "${work_dir}/namespace-admin.yaml" >/dev/null 2>&1; then + echo "gateway chart dropped the gateway ServiceAccount with rbac.clusterScoped.create=false" >&2 + exit 1 +fi + helm template openshell-workspace "${repo_root}/deploy/helm/openshell-workspace" \ --namespace app-a \ --set gateway.serviceAccount.name=openshell \ diff --git a/docs/kubernetes/setup.mdx b/docs/kubernetes/setup.mdx index c3f4e2416e..57d4963fc5 100644 --- a/docs/kubernetes/setup.mdx +++ b/docs/kubernetes/setup.mdx @@ -276,7 +276,10 @@ The chart creates the following RBAC resources in the release namespace: | ServiceAccount | Namespace | `openshell` | | ServiceAccount | Namespace | `openshell-sandbox` (for sandbox pods) | | Role + RoleBinding | Namespace | `openshell-sandbox` | -| ClusterRole + ClusterRoleBinding | Cluster | `openshell-node-reader` | +| ClusterRole + ClusterRoleBinding | Cluster | `openshell-node-reader-` | + +Every other object the chart creates is namespaced. The `ClusterRole` and +`ClusterRoleBinding` in the last row are the only cluster-scoped objects. The namespaced Role covers sandbox lifecycle and identity: @@ -307,6 +310,78 @@ helm upgrade --install openshell \ The ServiceAccount must already have the Role and ClusterRole bindings described above. +### Installing without cluster-admin + +By default the release creates the `ClusterRole` and `ClusterRoleBinding`, so +the installer needs cluster-scoped permissions. When cluster-scoped RBAC is +owned by a different team, or the installer is a namespace-admin GitOps +controller, split the install into a cluster-admin step and a namespace-admin +step. + +A cluster-admin applies the cluster-scoped objects once per gateway +ServiceAccount. Render them from the same values the release uses, so the rules +match the configured workspace mode and credential driver: + +```shell +helm template openshell \ + oci://ghcr.io/nvidia/openshell/helm-chart \ + --version \ + --namespace openshell \ + -f my-values.yaml \ + --set rbac.clusterScoped.create=true \ + --set agentSandbox.preflight.enabled=false \ + --show-only templates/clusterrole.yaml \ + --show-only templates/clusterrolebinding.yaml | kubectl apply -f - +``` + +A namespace-admin then installs and upgrades the chart with cluster-scoped +objects omitted. The release renders only namespaced objects and needs no +permission on `clusterroles` or `clusterrolebindings`: + +```shell +helm upgrade --install openshell \ + oci://ghcr.io/nvidia/openshell/helm-chart \ + --version \ + --namespace openshell \ + -f my-values.yaml \ + --set supervisor.sandboxRuntime.networkPolicyEnforced=true \ + --set rbac.clusterScoped.create=false +``` + +The gateway ServiceAccount name and namespace are unchanged by this flag, so +the pre-created `ClusterRoleBinding` still binds the ServiceAccount the +namespaced release creates. The same holds with `serviceAccount.create=false`: +the binding subject follows `serviceAccount.name`, so pass the same values to +both steps. + +#### Migrating an existing release + +Helm deletes objects that leave a release manifest, so setting +`rbac.clusterScoped.create=false` on a release that already owns the +`ClusterRole` and `ClusterRoleBinding` deletes them. The gateway then loses +TokenReview until a cluster-admin re-applies them. Hand ownership over first, as +cluster-admin, so nothing is deleted: + +```shell +kubectl annotate clusterrole "openshell-node-reader-" \ + helm.sh/resource-policy=keep --overwrite +kubectl annotate clusterrolebinding "openshell-node-reader-" \ + helm.sh/resource-policy=keep --overwrite +``` + +The objects then survive the upgrade that sets the flag, and the cluster-admin +owns them from that point on. Fresh installs need no such step. + +`rbac.clusterScoped.create` is independent of the workspace mode. `managed` and +`operator` modes change what the `ClusterRole` grants, but they never require a +namespace-admin to apply it. Re-run the cluster-admin step after changing any +value that affects the `ClusterRole` rules. + +Set `rbac.clusterScoped.clusterRoleName` and +`rbac.clusterScoped.clusterRoleBindingName` when the cluster-admin owns the +naming, and `rbac.create=false` to also leave the namespaced sandbox `Role` and +`RoleBinding` to a separate step. + ## Probes The gateway exposes `/healthz` for process liveness and `/readyz` for dependency-aware readiness on the health port. The Helm chart wires both into Kubernetes probes: