Update Istio configuration to use CNI node agents instead of istio-init containers (#474)

MaciejKaras · web-flow · commit 2b99a7f36844 · 2025-10-22T11:01:27.000+02:00
# Summary We are using `Istio` as a service mesh provider for our Multi Cluster tests. The way it works by default is `Istio` adds privileged `init-istio` container to every Pod that configures network accordingly. >By default Istio injects an init container, istio-init, in pods deployed in the mesh. The istio-init container sets up the pod network traffic redirection to/from the Istio sidecar proxy. This requires the user or service-account deploying pods to the mesh to have sufficient Kubernetes RBAC permissions to deploy [containers with the NET_ADMIN and NET_RAW capabilities](https://kubernetes.io/docs/tasks/configure-pod-container/security-context/#set-capabilities-for-a-container). While this works fine it is not meeting the [PSS](https://v1-32.docs.kubernetes.io/docs/concepts/security/pod-security-standards/) restricted level, thus making it less secure. Related [HELP-81729](https://jira.mongodb.org/browse/HELP-81729) and #473 that enables `restricted` level in `warn` mode. Additionally we provide Istio sidecar configuration as an example in our code snippets thus not following the best practice. There is another way to configure Istio mesh that does not require `istio-init` init-container - using [Istio CNI node agent](https://istio.io/latest/docs/setup/additional-setup/cni/#using-the-istio-cni-node-agent). This PR configures our e2e tests and code snippets that way. Great blog entry about difference between `istio-init` and Istio CNI node agent architecture -> https://www.solo.io/blog/traffic-ambient-mesh-istio-cni-node-configuration. With `istio-init`: <img width="810" height="820" alt="image" src="https://github.com/user-attachments/assets/026350af-3b51-4fe9-9cb8-c8911e661eca" /> With `Istio CNI node agent`: <img width="942" height="1084" alt="image" src="https://github.com/user-attachments/assets/37733169-7737-4063-90a0-de3d116402a9" /> ⚠️ Init containers execute before the sidecar proxy starts, which can result in traffic loss during their execution. This can be avoided by setting `runAsUser: 1337`. More info -> https://istio.io/latest/docs/setup/additional-setup/cni/#compatibility-with-application-init-containers ## Proof of Work Passing CI is enough. Since `private_gke_code_snippets` are not run automatically in CI I've triggered manual patch to test this -> https://spruce.mongodb.com/version/68d50e694baed3000742566d/tasks?sorts=STATUS%3AASC%3BBASE_STATUS%3ADESC ## Checklist - [ ] Have you linked a jira ticket and/or is the ticket in the title? - [x] Have you checked whether your jira ticket required DOCSP changes? - [x] Have you added changelog file? - use `skip-changelog` label if not needed - refer to [Changelog files and Release Notes](https://github.com/mongodb/mongodb-kubernetes/blob/master/CONTRIBUTING.md#changelog-files-and-release-notes) section in CONTRIBUTING.md for more details
diff --git a/docker/mongodb-kubernetes-tests/tests/opsmanager/fixtures/om_https_enabled.yaml b/docker/mongodb-kubernetes-tests/tests/opsmanager/fixtures/om_https_enabled.yaml
@@ -22,7 +22,7 @@ spec:
         spec:
           volumes:
             - name: mongodb-versions
-              emptyDir: {}
+              emptyDir: { }
           containers:
             - name: mongodb-ops-manager
               volumeMounts:
@@ -37,6 +37,8 @@ spec:
           initContainers:
             - name: setting-up-rhel-mongodb
               image: curlimages/curl:latest
+              securityContext:
+                runAsUser: 1337 # workaround for init-container istio issue -> https://istio.io/latest/docs/setup/additional-setup/cni/#compatibility-with-application-init-containers
               command:
                 - curl
                 - -L
@@ -48,6 +50,8 @@ spec:
                   mountPath: /mongodb-ops-manager/mongodb-releases
             - name: setting-up-rhel-mongodb-4-4
               image: curlimages/curl:latest
+              securityContext:
+                runAsUser: 1337 # workaround for init-container istio issue -> https://istio.io/latest/docs/setup/additional-setup/cni/#compatibility-with-application-init-containers
               command:
                 - curl
                 - -L
@@ -59,6 +63,8 @@ spec:
                   mountPath: /mongodb-ops-manager/mongodb-releases
             - name: setting-up-rhel-mongodb-5-0
               image: curlimages/curl:latest
+              securityContext:
+                runAsUser: 1337 # workaround for init-container istio issue -> https://istio.io/latest/docs/setup/additional-setup/cni/#compatibility-with-application-init-containers
               command:
                 - curl
                 - -L
@@ -70,6 +76,8 @@ spec:
                   mountPath: /mongodb-ops-manager/mongodb-releases
             - name: setting-up-rhel-mongodb-6-0
               image: curlimages/curl:latest
+              securityContext:
+                runAsUser: 1337 # workaround for init-container istio issue -> https://istio.io/latest/docs/setup/additional-setup/cni/#compatibility-with-application-init-containers
               command:
                 - curl
                 - -L
@@ -81,6 +89,8 @@ spec:
                   mountPath: /mongodb-ops-manager/mongodb-releases
             - name: setting-up-rhel-mongodb-6-0-sig
               image: curlimages/curl:latest
+              securityContext:
+                runAsUser: 1337 # workaround for init-container istio issue -> https://istio.io/latest/docs/setup/additional-setup/cni/#compatibility-with-application-init-containers
               command:
                 - curl
                 - -L
@@ -92,6 +102,8 @@ spec:
                   mountPath: /mongodb-ops-manager/mongodb-releases
             - name: setting-up-rhel-mongodb-6-0-21
               image: curlimages/curl:latest
+              securityContext:
+                runAsUser: 1337 # workaround for init-container istio issue -> https://istio.io/latest/docs/setup/additional-setup/cni/#compatibility-with-application-init-containers
               command:
                 - curl
                 - -L
@@ -103,6 +115,8 @@ spec:
                   mountPath: /mongodb-ops-manager/mongodb-releases
             - name: setting-up-rhel-mongodb-6-0-21-sig
               image: curlimages/curl:latest
+              securityContext:
+                runAsUser: 1337 # workaround for init-container istio issue -> https://istio.io/latest/docs/setup/additional-setup/cni/#compatibility-with-application-init-containers
               command:
                 - curl
                 - -L
@@ -112,9 +126,10 @@ spec:
               volumeMounts:
                 - name: mongodb-versions
                   mountPath: /mongodb-ops-manager/mongodb-releases
-
             - name: setting-up-rhel-mongodb-7-0
               image: curlimages/curl:latest
+              securityContext:
+                runAsUser: 1337 # workaround for init-container istio issue -> https://istio.io/latest/docs/setup/additional-setup/cni/#compatibility-with-application-init-containers
               command:
                 - curl
                 - -L
@@ -126,6 +141,8 @@ spec:
                   mountPath: /mongodb-ops-manager/mongodb-releases
             - name: setting-up-rhel-mongodb-7-0-sig
               image: curlimages/curl:latest
+              securityContext:
+                runAsUser: 1337 # workaround for init-container istio issue -> https://istio.io/latest/docs/setup/additional-setup/cni/#compatibility-with-application-init-containers
               command:
                 - curl
                 - -L
@@ -137,6 +154,8 @@ spec:
                   mountPath: /mongodb-ops-manager/mongodb-releases
             - name: setting-up-rhel-mongodb-8-0
               image: curlimages/curl:latest
+              securityContext:
+                runAsUser: 1337 # workaround for init-container istio issue -> https://istio.io/latest/docs/setup/additional-setup/cni/#compatibility-with-application-init-containers
               command:
                 - curl
                 - -L
@@ -148,6 +167,8 @@ spec:
                   mountPath: /mongodb-ops-manager/mongodb-releases
             - name: setting-up-rhel-mongodb-8-0-sig
               image: curlimages/curl:latest
+              securityContext:
+                runAsUser: 1337 # workaround for init-container istio issue -> https://istio.io/latest/docs/setup/additional-setup/cni/#compatibility-with-application-init-containers
               command:
                 - curl
                 - -L
diff --git a/docker/mongodb-kubernetes-tests/tests/opsmanager/fixtures/om_localmode-single-pv.yaml b/docker/mongodb-kubernetes-tests/tests/opsmanager/fixtures/om_localmode-single-pv.yaml
@@ -35,6 +35,8 @@ spec:
           initContainers:
             - name: setting-up-rhel-mongodb-4-2-8
               image: curlimages/curl:latest
+              securityContext:
+                runAsUser: 1337 # workaround for init-container istio issue -> https://istio.io/latest/docs/setup/additional-setup/cni/#compatibility-with-application-init-containers
               command:
                 - curl
                 - -L
@@ -46,6 +48,8 @@ spec:
                   mountPath: /mongodb-ops-manager/mongodb-releases
             - name: setting-up-rhel-mongodb-6-0-21
               image: curlimages/curl:latest
+              securityContext:
+                runAsUser: 1337 # workaround for init-container istio issue -> https://istio.io/latest/docs/setup/additional-setup/cni/#compatibility-with-application-init-containers
               command:
                 - curl
                 - -L
@@ -57,6 +61,8 @@ spec:
                   mountPath: /mongodb-ops-manager/mongodb-releases
             - name: setting-up-rhel-mongodb-7-0
               image: curlimages/curl:latest
+              securityContext:
+                runAsUser: 1337 # workaround for init-container istio issue -> https://istio.io/latest/docs/setup/additional-setup/cni/#compatibility-with-application-init-containers
               command:
                 - curl
                 - -L
@@ -68,6 +74,8 @@ spec:
                   mountPath: /mongodb-ops-manager/mongodb-releases
             - name: setting-up-rhel-mongodb-8-0
               image: curlimages/curl:latest
+              securityContext:
+                runAsUser: 1337 # workaround for init-container istio issue -> https://istio.io/latest/docs/setup/additional-setup/cni/#compatibility-with-application-init-containers
               command:
                 - curl
                 - -L
diff --git a/docker/mongodb-kubernetes-tests/tests/opsmanager/fixtures/remote_fixtures/nginx.yaml b/docker/mongodb-kubernetes-tests/tests/opsmanager/fixtures/remote_fixtures/nginx.yaml
@@ -29,6 +29,8 @@ spec:
       initContainers:
         - name: setting-up-mongosh-1-4-1
           image: curlimages/curl:latest
+          securityContext:
+            runAsUser: 1337 # workaround for init-container istio issue -> https://istio.io/latest/docs/setup/additional-setup/cni/#compatibility-with-application-init-containers
           command:
             - sh
             - -c
@@ -38,6 +40,8 @@ spec:
               mountPath: /mongodb-ops-manager/mongodb-releases/compass
         - name: setting-up-mongosh-1-9-1
           image: curlimages/curl:latest
+          securityContext:
+            runAsUser: 1337 # workaround for init-container istio issue -> https://istio.io/latest/docs/setup/additional-setup/cni/#compatibility-with-application-init-containers
           command:
             - sh
             - -c
@@ -47,6 +51,8 @@ spec:
               mountPath: /mongodb-ops-manager/mongodb-releases/compass
         - name: setting-up-mongosh-1-10-4
           image: curlimages/curl:latest
+          securityContext:
+            runAsUser: 1337 # workaround for init-container istio issue -> https://istio.io/latest/docs/setup/additional-setup/cni/#compatibility-with-application-init-containers
           command:
             - sh
             - -c
@@ -56,6 +62,8 @@ spec:
               mountPath: /mongodb-ops-manager/mongodb-releases/compass
         - name: setting-up-mongosh-2-0-0
           image: curlimages/curl:latest
+          securityContext:
+            runAsUser: 1337 # workaround for init-container istio issue -> https://istio.io/latest/docs/setup/additional-setup/cni/#compatibility-with-application-init-containers
           command:
             - sh
             - -c
@@ -65,6 +73,8 @@ spec:
               mountPath: /mongodb-ops-manager/mongodb-releases/compass
         - name: setting-up-mongosh-2-0-2
           image: curlimages/curl:latest
+          securityContext:
+            runAsUser: 1337 # workaround for init-container istio issue -> https://istio.io/latest/docs/setup/additional-setup/cni/#compatibility-with-application-init-containers
           command:
             - sh
             - -c
@@ -74,6 +84,8 @@ spec:
               mountPath: /mongodb-ops-manager/mongodb-releases/compass
         - name: setting-up-mongosh-2-0-2-om7
           image: curlimages/curl:latest
+          securityContext:
+            runAsUser: 1337 # workaround for init-container istio issue -> https://istio.io/latest/docs/setup/additional-setup/cni/#compatibility-with-application-init-containers
           command:
             - sh
             - -c
@@ -83,6 +95,8 @@ spec:
               mountPath: /mongodb-ops-manager/mongodb-releases/compass
         - name: setting-up-mongosh-2-1-5-om7
           image: curlimages/curl:latest
+          securityContext:
+            runAsUser: 1337 # workaround for init-container istio issue -> https://istio.io/latest/docs/setup/additional-setup/cni/#compatibility-with-application-init-containers
           command:
             - sh
             - -c
@@ -92,6 +106,8 @@ spec:
               mountPath: /mongodb-ops-manager/mongodb-releases/compass
         - name: setting-up-mongosh-2-2-3-om7
           image: curlimages/curl:latest
+          securityContext:
+            runAsUser: 1337 # workaround for init-container istio issue -> https://istio.io/latest/docs/setup/additional-setup/cni/#compatibility-with-application-init-containers
           command:
             - sh
             - -c
@@ -101,6 +117,8 @@ spec:
               mountPath: /mongodb-ops-manager/mongodb-releases/compass
         - name: setting-up-mongosh-2-2-4-om7
           image: curlimages/curl:latest
+          securityContext:
+            runAsUser: 1337 # workaround for init-container istio issue -> https://istio.io/latest/docs/setup/additional-setup/cni/#compatibility-with-application-init-containers
           command:
             - sh
             - -c
@@ -110,6 +128,8 @@ spec:
               mountPath: /mongodb-ops-manager/mongodb-releases/compass
         - name: setting-up-mongosh-2-4-0
           image: curlimages/curl:latest
+          securityContext:
+            runAsUser: 1337 # workaround for init-container istio issue -> https://istio.io/latest/docs/setup/additional-setup/cni/#compatibility-with-application-init-containers
           command:
             - sh
             - -c
@@ -119,22 +139,23 @@ spec:
               mountPath: /mongodb-ops-manager/mongodb-releases/compass
         - name: setting-up-mongosh-2-5-6
           image: curlimages/curl:latest
+          securityContext:
+            runAsUser: 1337 # workaround for init-container istio issue -> https://istio.io/latest/docs/setup/additional-setup/cni/#compatibility-with-application-init-containers
           command:
             - sh
             - -c
             - curl -LO https://downloads.mongodb.com/compass/mongosh-2.5.6-linux-x64-openssl11.tgz --output-dir /mongodb-ops-manager/mongodb-releases/compass && true
           volumeMounts:
             - name: mongosh-versions
               mountPath: /mongodb-ops-manager/mongodb-releases/compass
-
       restartPolicy: Always
-      securityContext: {}
+      securityContext: { }
       terminationGracePeriodSeconds: 30
       volumes:
         - name: mongodb-versions
-          emptyDir: {}
+          emptyDir: { }
         - name: mongosh-versions
-          emptyDir: {}
+          emptyDir: { }
         - configMap:
             name: nginx-conf
           name: nginx-conf
diff --git a/docker/mongodb-kubernetes-tests/tests/opsmanager/om_remotemode.py b/docker/mongodb-kubernetes-tests/tests/opsmanager/om_remotemode.py
@@ -14,6 +14,7 @@
 
 VERSION_NOT_IN_WEB_SERVER = "4.2.1"
 
+
 # If this test is failing after an OM Bump, ensure that the nginx deployment fixture contains the associated mongosh
 # version. More details in this ticket: https://jira.mongodb.org/browse/CLOUDP-332640
 
@@ -47,6 +48,10 @@ def add_mdb_version_to_deployment(deployment: Dict[str, Any], version: str):
             "name": KubernetesTester.random_k8s_name(prefix="mdb-download"),
             "image": "curlimages/curl:latest",
             "command": ["sh", "-c", f"{curl_command} && true"],
+            "securityContext": {
+                # workaround for init-container istio issue -> https://istio.io/latest/docs/setup/additional-setup/cni/#compatibility-with-application-init-containers
+                "runAsUser": 1337,
+            },
             "volumeMounts": [
                 {
                     "name": "mongodb-versions",
diff --git a/multi_cluster/tools/install_istio.sh b/multi_cluster/tools/install_istio.sh
@@ -38,6 +38,7 @@ make -f ../tools/certs/Makefile.selfsigned.mk "${CTX_CLUSTER3}-cacerts" || make
 # create cluster secret objects with the certs and keys
 kubectl --context="${CTX_CLUSTER1}" delete ns istio-system || true
 kubectl --context="${CTX_CLUSTER1}" create ns istio-system
+kubectl --context="${CTX_CLUSTER1}" label --overwrite ns istio-system pod-security.kubernetes.io/enforce=privileged
 kubectl --context="${CTX_CLUSTER1}" create secret generic cacerts -n istio-system \
   --from-file=${CTX_CLUSTER1}/ca-cert.pem \
   --from-file=${CTX_CLUSTER1}/ca-key.pem \
@@ -46,6 +47,7 @@ kubectl --context="${CTX_CLUSTER1}" create secret generic cacerts -n istio-syste
 
 kubectl --context="${CTX_CLUSTER2}" delete ns istio-system || true
 kubectl --context="${CTX_CLUSTER2}" create ns istio-system
+kubectl --context="${CTX_CLUSTER2}" label --overwrite ns istio-system pod-security.kubernetes.io/enforce=privileged
 kubectl --context="${CTX_CLUSTER2}" create secret generic cacerts -n istio-system \
   --from-file=${CTX_CLUSTER2}/ca-cert.pem \
   --from-file=${CTX_CLUSTER2}/ca-key.pem \
@@ -54,6 +56,7 @@ kubectl --context="${CTX_CLUSTER2}" create secret generic cacerts -n istio-syste
 
 kubectl --context="${CTX_CLUSTER3}" delete ns istio-system || true
 kubectl --context="${CTX_CLUSTER3}" create ns istio-system
+kubectl --context="${CTX_CLUSTER3}" label --overwrite ns istio-system pod-security.kubernetes.io/enforce=privileged
 kubectl --context="${CTX_CLUSTER3}" create secret generic cacerts -n istio-system \
   --from-file=${CTX_CLUSTER3}/ca-cert.pem \
   --from-file=${CTX_CLUSTER3}/ca-key.pem \
@@ -67,6 +70,10 @@ apiVersion: install.istio.io/v1alpha1
 kind: IstioOperator
 spec:
   tag: ${VERSION}
+  components:
+    cni:
+      namespace: istio-system
+      enabled: true
   meshConfig:
     defaultConfig:
       terminationDrainDuration: 30s
@@ -81,13 +88,17 @@ spec:
       network: network1
 EOF
 
-bin/istioctl install --context="${CTX_CLUSTER1}" -f cluster1.yaml -y &
+bin/istioctl install --context="${CTX_CLUSTER1}" --set components.cni.enabled=true -f cluster1.yaml -y &
 
 cat <<EOF >cluster2.yaml
 apiVersion: install.istio.io/v1alpha1
 kind: IstioOperator
 spec:
   tag: ${VERSION}
+  components:
+    cni:
+      namespace: istio-system
+      enabled: true
   meshConfig:
     defaultConfig:
       terminationDrainDuration: 30s
@@ -102,13 +113,17 @@ spec:
       network: network1
 EOF
 
-bin/istioctl install --context="${CTX_CLUSTER2}" -f cluster2.yaml -y &
+bin/istioctl install --context="${CTX_CLUSTER2}" --set components.cni.enabled=true -f cluster2.yaml -y &
 
 cat <<EOF >cluster3.yaml
 apiVersion: install.istio.io/v1alpha1
 kind: IstioOperator
 spec:
   tag: ${VERSION}
+  components:
+    cni:
+      namespace: istio-system
+      enabled: true
   meshConfig:
     defaultConfig:
       terminationDrainDuration: 30s
@@ -123,7 +138,7 @@ spec:
       network: network1
 EOF
 
-bin/istioctl install --context="${CTX_CLUSTER3}" -f cluster3.yaml -y &
+bin/istioctl install --context="${CTX_CLUSTER3}" --set components.cni.enabled=true -f cluster3.yaml -y &
 
 wait
 
diff --git a/multi_cluster/tools/install_istio_central.sh b/multi_cluster/tools/install_istio_central.sh
@@ -10,4 +10,4 @@ source multi_cluster/tools/download_istio.sh
 cd istio-${VERSION}
 
 bin/istioctl x uninstall --context="${CTX_CLUSTER}" --purge --skip-confirmation
-bin/istioctl install --context="${CTX_CLUSTER}" --set profile=default --set meshConfig.outboundTrafficPolicy.mode=REGISTRY_ONLY --skip-confirmation
+bin/istioctl install --context="${CTX_CLUSTER}" --set components.cni.enabled=true --set profile=default --set meshConfig.outboundTrafficPolicy.mode=REGISTRY_ONLY --skip-confirmation
diff --git a/public/architectures/setup-multi-cluster/ra-03-setup-istio/install_istio_separate_network.sh b/public/architectures/setup-multi-cluster/ra-03-setup-istio/install_istio_separate_network.sh
@@ -94,6 +94,7 @@ spec:
       network: network1
 EOF
 bin/istioctl install --context="${CTX_CLUSTER1}" -f cluster1.yaml -y
+
 samples/multicluster/gen-eastwest-gateway.sh \
     --mesh mesh1 --cluster cluster1 --network network1 | \
     bin/istioctl --context="${CTX_CLUSTER1}" install -y -f -
diff --git a/public/samples/ops-manager/ops-manager-remote-mode.yaml b/public/samples/ops-manager/ops-manager-remote-mode.yaml
@@ -73,7 +73,6 @@ spec:
           volumeMounts:
             - name: mongodb-versions
               mountPath: /mongodb-ops-manager/mongodb-releases/linux
-
         - name: setting-up-rhel-mongodb-4-4-ent
           image: curlimages/curl:latest
           command:
diff --git a/scripts/release/kubectl-mongodb/install_istio_separate_network.sh b/scripts/release/kubectl-mongodb/install_istio_separate_network.sh
