feat: Add scripts for kubernetes dev env using vLLM and vLLM-p2p #60

kfirtoledo · 2025-04-25T05:21:13Z

Add support for Kubernetes environment development using GIE with KGateway and vLLM
This PR introduces support for the vllm mode, enabling integration testing of GIE with vLLM.
It also adds support for the vllm-p2p mode, which includes:

Deployment of Redis and LMCache alongside the vLLM image
Peer-to-peer (P2P) communication between vLLM instances
Use of the EPP image to enable kv-cache-aware routing

…tup for kvcache-aware) Signed-off-by: Kfir Toledo <[email protected]>

shaneutt

Looking great. Most of my comments are smaller, but I do have some questions for other folks as to what effect this will have.

Also, cc @elevran @shmuelk who I think should take a look.

shaneutt · 2025-04-25T15:55:26Z

deploy/components/inference-gateway/inference-models.yaml

+apiVersion: inference.networking.x-k8s.io/v1alpha2
+kind: InferenceModel
+metadata:
+  name: mistarli


Suggested change

name: mistarli

name: mistral

Actually, I removed it and use just a base model

shaneutt · 2025-04-25T15:56:49Z

deploy/components/vllm-p2p/deployments/redis-deployment.yaml

+          # securityContext:
+          #   allowPrivilegeEscalation: false
+          #   capabilities:
+          #     drop:
+          #       - ALL


If it's helpful for future readers to see this commented out bit, we should add a comment explaining why. Otherwise we should probably just remove it.

shaneutt · 2025-04-25T15:57:48Z

deploy/components/vllm-p2p/deployments/vllm-deployment.yaml

+      # securityContext:
+      #   runAsUser: ${PROXY_UID}
+        # runAsNonRoot: true
+        # seccompProfile:
+        #   type: RuntimeDefault


Similar to above: if this is an important breadcrumb for future readers, let's add a comment explaining why, otherwise if its just leftovers let's remove it.

shaneutt · 2025-04-25T16:00:47Z

deploy/components/vllm/deployments.yaml

+        - name: lora-adapter-syncer
+          tty: true
+          stdin: true
+          image: us-central1-docker.pkg.dev/k8s-staging-images/gateway-api-inference-extension/lora-syncer:main


I noticed this is a hardcoded image on the main tag, and not something we patch from the kustomization.yaml. Do we want this to be customizable? Or at the very least, do we want to pin to a specific SHA so that updates are more deliberate?

shaneutt · 2025-04-25T16:01:06Z

deploy/components/vllm/kustomization.yaml

+# ------------------------------------------------------------------------------
+# vLLM Deployment
+#
+# This deploys the full vLLM model server, capable of serving real models such
+# as Llama 3.1-8B-Instruct via the OpenAI-compatible API. It is intended for
+# environments with GPU resources and where full inference capabilities are
+# required.
+#
+# The deployment can be customized using environment variables to set:
+#   - The container image and tag (VLLM_IMAGE, VLLM_TAG)
+#   - The model to load (MODEL_NAME)
+#
+# This setup is suitable for testing and production with Kubernetes (including
+# GPU-enabled nodes or clusters with scheduling for `nvidia.com/gpu`).
+# -----------------------------------------------------------------------------


shaneutt · 2025-04-25T16:02:05Z

deploy/components/vllm-p2p/kustomization.yaml

@@ -0,0 +1,29 @@
+apiVersion: kustomize.config.k8s.io/v1beta1


I noticed we added some documentation on the kustomization.yaml for the vllm component, but not for this one. Perhaps we should add a little comment here explaining how this one differs from the standard one.

shaneutt · 2025-04-25T16:03:44Z

deploy/environments/dev/kubernetes-kgateway/gateway-parameters.yaml

-      type: NodePort
+      type: LoadBalancer


The reason NodePort was chosen originally is because it's more universal and in theory this deployment is meant to work generally on Kubernetes clusters. Perhaps we can make this variable, defaulting to NodePort and allowing folks to opt-in to LoadBalancer? LMKWYT? 🤔

We can, I agree that on Kind we should use nodeport, but this is not for Kind, in a development environment, I think we should use LB. This is also how the GIE uses that in their example.

Is there an LB in the test clusters?

shaneutt · 2025-04-25T16:06:21Z

deploy/environments/dev/kubernetes-kgateway/kustomization.yaml

- ../../../components/vllm-sim/
 - ../../../components/inference-gateway/
 - gateway-parameters.yaml

 images:
- name: quay.io/vllm-d/vllm-sim
-  newName: ${VLLM_SIM_IMAGE}
-  newTag: ${VLLM_SIM_TAG}


So the idea with the deployments under deploy/environments is that these use the deploy/components to provide some level of a working environment. With this change, this environment wont really have anything that works OOTB, so maybe what we should do is leave the simulator in this one, and rename it deploy/environments/dev/kubernetes-kgateway-vllm-sim?

The main reason I separated vLLM from the gateway is that I believe it will be similar in Istio. I didn’t want to duplicate the files because the vLLM YAMLs shouldn’t depend on which gateway we use.
Another option we could add a third level of abstraction: components -> vllm -> gateway.
It would be cleaner, but possibly also more complicated to understand.

shaneutt · 2025-04-25T16:11:59Z

deploy/environments/dev/kubernetes-vllm/vllm-p2p/kustomization.yaml

@@ -0,0 +1,11 @@
+apiVersion: kustomize.config.k8s.io/v1beta1


Oh ok, I see what you're doing with the naming now. The difference now is that any one of these deployments is deploying only a working VLLM stack, and then you have to deploy your inference-gateway stack separately.

cc @tumido @Gregory-Pereira @vMaroon just wanting to check with you on how this will work with your Helm chart?

shaneutt · 2025-04-25T16:12:35Z

scripts/kubernetes-dev-env.sh

+  echo "ERROR: NAMESPACE environment variable is not set."
+  exit 1
+fi
+if [[ -z "${VLLM_MODE:-}" ]]; then


Consider whether we should just default this to vllm-sim given that we know it's functional on practically any cluster.

We can, but I want people to set it up to be sure on which environment they testing and running

+1 for using vLLM simulator. If you think you want to steer people towards using real vLLM, you can add a print message when set to simulator (e.g., ** INFO: running simulated vLLM instances **

vMaroon · 2025-04-25T21:20:10Z

deploy/components/vllm-p2p/deployments/redis-deployment.yaml

+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: ${REDIS_SVC_NAME}


I think this should be turned to REDIS_DEPLOYMENT_NAME or something like that, and in the Service CR add a -service extension. Otherwise the deployment's name will be weird.

Signed-off-by: Kfir Toledo <[email protected]>

Gregory-Pereira · 2025-04-26T16:55:58Z

deploy/components/vllm-p2p/deployments/vllm-deployment.yaml

+            - "-c"
+          args:
+            - |
+              export LMCACHE_DISTRIBUTED_URL=$${${POD_IP}}:80 && \


Is this syntax correct? Shouldnt it be:

Suggested change

export LMCACHE_DISTRIBUTED_URL=$${${POD_IP}}:80 && \

export LMCACHE_DISTRIBUTED_URL=${POD_IP}:80 && \

In theory, you are right, the end goal is that it will be ${POD_IP},
But the issue is I use envsubst, so I need to change it to this =$${${POD_IP}} + export POD_IP="POD_IP",
That after I trigger envsubst, I will get ${POD_IP} (took me a lot of time to figure it out)

kfirtoledo · 2025-04-26T23:53:59Z

@shaneutt , PTOL.

elevran · 2025-04-27T12:10:35Z

DEVELOPMENT.md

-
-The same thing will need to be done for the EPP:
+- `vllm-sim`: Lightweight simulator for simple environments
+- `vllm`: Full vLLM model server for real inference


Suggested change

- `vllm`: Full vLLM model server for real inference

- `vllm`: Full vLLM model server, using GPU/CPU for inferencing

elevran · 2025-04-27T12:11:17Z

DEVELOPMENT.md

 ```

+- Set hugging face token variable:


Suggested change

- Set hugging face token variable:

- Set Hugging Face token variable:

elevran · 2025-04-27T12:12:19Z

DEVELOPMENT.md

@@ -152,6 +152,13 @@ Create the namespace:
 ```console
 kubectl create namespace ${NAMESPACE}
 ```
+Set the default namespace for kubectl commands
+
+```console


nit: Please be consistent with using bash or console for command line/shell snippets

elevran · 2025-04-27T12:13:08Z

DEVELOPMENT.md


-```console
-export VLLM_SIM_IMAGE="<YOUR_REGISTRY>/<YOUR_IMAGE>"


are the images being set elsewhere, to match simulator/vLLM/vLLM-p2p?

elevran · 2025-04-27T12:14:00Z

DEVELOPMENT.md

@@ -202,16 +208,72 @@ make environment.dev.kubernetes
 This will deploy the entire stack to whatever namespace you chose. You can test
 by exposing the inference `Gateway` via port-forward:

-```console
-kubectl -n ${NAMESPACE} port-forward service/inference-gateway-istio 8080:80
+```bash


nit: console vs bash?

elevran · 2025-04-27T12:33:19Z

deploy/components/vllm/kustomization.yaml

+#   - The container image and tag (VLLM_IMAGE, VLLM_TAG)
+#   - The model to load (MODEL_NAME)
+#
+# This setup is suitable for testing and production with Kubernetes (including


do you want to stand behind the production qualification? As opposed to testing in real hardware.
I imagine there are ways to configure vLLM optimally for hardware that goes beyond what's done in this PR.

elevran · 2025-04-27T12:33:39Z

deploy/components/vllm/kustomization.yaml

+configMapGenerator:
+- name: vllm-model-config
+  literals:
+    - MODEL_NAME=${MODEL_NAME}


nit: missing newline at end of file

elevran · 2025-04-27T12:34:23Z

deploy/environments/dev/kubernetes-kgateway/gateway-parameters.yaml

-      type: NodePort
+      type: LoadBalancer


Is there an LB in the test clusters?

elevran · 2025-04-27T12:34:55Z

deploy/environments/dev/kubernetes-kgateway/kustomization.yaml

 - name: quay.io/vllm-d/gateway-api-inference-extension/epp
  newName: ${EPP_IMAGE}
  newTag: ${EPP_TAG}

 patches:
 - path: patch-deployments.yaml
- path: patch-gateways.yaml
+- path: patch-gateways.yaml


nit: missing newline at end of file

elevran · 2025-04-27T12:37:43Z

scripts/kubernetes-dev-env.sh

+  echo "ERROR: NAMESPACE environment variable is not set."
+  exit 1
+fi
+if [[ -z "${VLLM_MODE:-}" ]]; then


+1 for using vLLM simulator. If you think you want to steer people towards using real vLLM, you can add a print message when set to simulator (e.g., ** INFO: running simulated vLLM instances **

Signed-off-by: Kfir Toledo <[email protected]>

feat: add scripts for kubernetes dev env using vLLM and vLLM-p2p (se…

a80c163

…tup for kvcache-aware) Signed-off-by: Kfir Toledo <[email protected]>

kfirtoledo requested review from shaneutt, vMaroon and elevran April 25, 2025 05:21

kfirtoledo added help wanted Extra attention is needed WIP labels Apr 25, 2025

kfirtoledo changed the title ~~feat: add scripts for kubernetes dev env using vLLM and vLLM-p2p~~ feat: Add scripts for kubernetes dev env using vLLM and vLLM-p2p Apr 25, 2025

kfirtoledo force-pushed the dev branch from 832da6e to 1b6ab70 Compare April 25, 2025 12:03

shaneutt requested changes Apr 25, 2025

View reviewed changes

shaneutt requested a review from shmuelk April 25, 2025 16:15

kfirtoledo force-pushed the dev branch from 1b6ab70 to f95936f Compare April 25, 2025 19:38

vMaroon reviewed Apr 25, 2025

View reviewed changes

kfirtoledo force-pushed the dev branch from f95936f to fc98576 Compare April 25, 2025 23:56

[fix]: Small fixes for development YAMLs

1a7fa8e

Signed-off-by: Kfir Toledo <[email protected]>

kfirtoledo force-pushed the dev branch from fc98576 to 1a7fa8e Compare April 26, 2025 00:05

Gregory-Pereira reviewed Apr 26, 2025

View reviewed changes

kfirtoledo force-pushed the dev branch 2 times, most recently from 390c50a to b189362 Compare April 26, 2025 23:52

kfirtoledo removed help wanted Extra attention is needed WIP labels Apr 27, 2025

elevran reviewed Apr 27, 2025

View reviewed changes

[fix]: Small fixes for deployment and fix comments

f5bc959

Signed-off-by: Kfir Toledo <[email protected]>

kfirtoledo force-pushed the dev branch from b189362 to f5bc959 Compare April 27, 2025 13:47

	export LMCACHE_DISTRIBUTED_URL=$${${POD_IP}}:80 && \
	export LMCACHE_DISTRIBUTED_URL=${POD_IP}:80 && \

	- `vllm`: Full vLLM model server for real inference
	- `vllm`: Full vLLM model server, using GPU/CPU for inferencing

	- Set hugging face token variable:
	- Set Hugging Face token variable:


		```console
		export VLLM_SIM_IMAGE="<YOUR_REGISTRY>/<YOUR_IMAGE>"

feat: Add scripts for kubernetes dev env using vLLM and vLLM-p2p #60

Are you sure you want to change the base?

feat: Add scripts for kubernetes dev env using vLLM and vLLM-p2p #60

Conversation

kfirtoledo commented Apr 25, 2025

shaneutt left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kfirtoledo commented Apr 26, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment