Limit Istio Sidecar Scope to reduce memory and make cluster more scalable

[han-steve]

Pull Request Template for Kubeflow Manifests

✏️ Summary of Changes

Describe the changes you have made, including any refactoring or feature additions.

Adding a new istio sidecar resource to limit the sidecar's egress visibility to unnecessary services.

We (Roblox) have been running Kubeflow in production for a long time, and we are noticing that the istio sidecar memory is almost 1GB now due to the amount of services in the cluster that has to be cached in each sidecar. This adds up to over 2 TB of memory in total. This change limits the caching of cluster services in each sidecar, thus helping the scalability of the cluster.

This change can save TBs of memory and spare our DNS services. But I want to ask the community to see if there are any istio-enabled egress communication from kubeflow pods that we haven't considered. As far as we know
Communications to Notebook and Pipeline backends go through the ingress gateway instead of directly inside the cluster, so that won't matter
Communications to kserve models go through cluster ingress gateway
All other CRD-based workloads don't need any egress communication

🐛 Related Issues

Link any issues that are resolved or affected by this PR.

knative/serving#12917 We are facing this issue where each sidecar is pinging DNS to resolve the cluster ingressgateway ip, essentially DDOSing our DNS. Removing the ExternalName service for cluster ingress gateway from the sidecars would resolve this problem.

✅ Contributor Checklist

• I have tested these changes with kustomize. See Installation Prerequisites.
• All commits are signed-off to satisfy the DCO check.

---

You can join the CNCF Slack and access our meetings at the Kubeflow Community website. Our channel on the CNCF Slack is here #kubeflow-platform.

Slack message link: https://cloud-native.slack.com/archives/C073W572LA2/p1741893411623659

[google-oss-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign juliusvonkohout for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[juliusvonkohout]

Thank you for the PR.

/ok-to-test

[juliusvonkohout]

Does that block stuff that is not on the servicemesh by default ? E.g. talking from your jupyterlab to a deployment in the same namespace with istio-injection disabled. Or egress to a non-istio native kubernetes service in another namespaces or just the internet in general. Because that must be still allowed. Maybe we could just block inter kubeflow-profile namespace communication by default.