add kvcache aware plugin e2e test

[FAUST-BENCHOU]

What type of PR is this?

What this PR does / why we need it:

After #1199 and #1178 only kvcache-aware plugin left now

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

[volcano-sh-bot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign yaozengzeng for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• test/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[Copilot]

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

Adds an end-to-end KV-cache-aware scheduler plugin test chain by introducing a ZMQ bridge and wiring the mock deployment to publish/consume KV events through runtime + Redis.

Changes:

• Added a Python ZMQ bridge helper and created it as a ConfigMap during E2E setup.
• Updated the mock deployment to include zmq-bridge and runtime containers and enabled the sim’s native ZMQ publishing.
• Added a new KV-cache-aware E2E test plus Redis/rollout helper functions.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 11 comments.

Show a summary per file

File	Description
test/e2e/router/router-plugins/testdata/zmq-bridge.py	New helper script to forward sim ZMQ events into the runtime topic layout
test/e2e/router/router-plugins/testdata/LLM-Mock-plugins.yaml	Adds bridge + runtime sidecars and config/ports to enable full KV-cache-aware chain
test/e2e/router/router-plugins/context/context.go	Creates ConfigMap containing the bridge script before deploying mocks
test/e2e/router/plugins_test.go	Adds TestSchedulerPluginKVCacheAware full-chain routing preference test
test/e2e/router/plugins_helpers.go	Adds Redis port-forward client setup and Redis-wait helpers for KV mappings

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[gemini-code-assist]

Code Review

This pull request introduces end-to-end testing for the kvcache-aware scheduling plugin, adding a ZMQ bridge helper script, configuring a runtime container in the mock LLM deployment, and implementing the TestSchedulerPluginKVCacheAware test. The review feedback focuses on improving test robustness and preventing flakiness. Key recommendations include refactoring setupRedisClient to return errors instead of asserting inside Eventually, resolving the zmq-bridge.py path dynamically using runtime.Caller to support running tests from package subdirectories, avoiding redundant rolling updates in patchMockRedisHost when the host is already set, and adding a buffer to the log scraping start time to account for potential clock skew.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[gemini-code-assist]

Using require assertions (which call t.FailNow()) inside a helper function that is executed within require.Eventually defeats the purpose of retrying. If any transient error occurs (such as a temporary port-forwarding failure or a Redis ping timeout while the container is starting up), the test will fail immediately instead of waiting and retrying. Refactor setupRedisClient to return an error so that the caller can handle it gracefully inside Eventually.

func setupRedisClient(t *testing.T, kube kubernetes.Interface, namespace string) (*redis.Client, func(), error) {
	t.Helper()
	pods := utils.ListReadyPodsByLabel(t, kube, namespace, redisServerAppLabel)
	if len(pods) == 0 {
		return nil, nil, fmt.Errorf("no ready redis pods in namespace %s", namespace)
	}

	localPort := utils.AllocateLocalPort(t)
	pf, err := utils.SetupPortForwardToPod(namespace, pods[0].Name, localPort, "6379")
	if err != nil {
		return nil, nil, fmt.Errorf("port-forward to redis pod %s: %w", pods[0].Name, err)
	}

	addr := fmt.Sprintf("127.0.0.1:%s", localPort)
	client := redis.NewClient(&redis.Options{Addr: addr})
	ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
	defer cancel()
	if err := client.Ping(ctx).Err(); err != nil {
		_ = client.Close()
		pf.Close()
		return nil, nil, fmt.Errorf("redis ping via port-forward: %w", err)
	}

	return client, func() {
		_ = client.Close()
		pf.Close()
	}, nil
}

[Copilot]

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 8 comments.

[Copilot]

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 9 comments.

[Copilot]

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 8 comments.

[Copilot]

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 7 comments.

[Copilot]

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 8 comments.

[FAUST-BENCHOU]

Just a compatibility. The ZMQ topic sent by the sim (sim instance) is inconsistent with the topic listened to by the runtime: the sim uses a per-pod topic like kv@<podIP>@<model>, while the runtime only subscribes to kv-events. Without conversion, it won't receive any kv-cache events, and Redis won't be written.

Example: Before processing, the sim sends a message with the topic kv@10.0.1.23@llm-mock, and the runtime, subscribing to kv-events, doesn't consume it at all. After processing, zmq_bridge changes the same message to kv-events and forwards it to the runtime (payload unchanged), allowing the runtime to write to Redis.

[FAUST-BENCHOU]

Write REDIS_HOST into the ConfigMap before deploying SetupPluginComponents . I tried patching mid-deployment, but the patch triggered Pod restarts, interrupted warmup and kv cache writes.

[FAUST-BENCHOU]

In the kvcache-aware chain, the kv-cache capacity/block count of sim is very small. During testing, I found that if chat requests generate too many tokens, it can easily fill the kv cache and trigger a 500 error.so I simply added a token control.

[FAUST-BENCHOU]

@hzxuzhonghu ptal

[YaoZengzeng]

Isn't it >= 8?

[FAUST-BENCHOU]

no , <= 8 refers to the maximum kv-cache size of the sim (kv-cache-size=8), not a requirement of at least 8 blocks. We keep the prompt and max_tokens very small to avoid the prompt blocks + output blocks exceeding 8, which would result in a 500 error.

[FAUST-BENCHOU]

maybe need to update comment

[Copilot]

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 6 comments.

[Copilot]

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 3 comments.

[FAUST-BENCHOU]

This is not in a hurry

[aeron-gh]

went through the draft while preparing for the lfx project. one thing none of the reviews mention: when the bridge republishes it swaps the sim's sequence frame for an empty one (send_multipart([topic, b"", payload])). the subscriber ignores frame 1 today so nothing breaks, but the seq is the only restart and ordering signal on the wire, and with #1527 and #1534 touching topic handling and the #1541 discussion pointing at agent side epoch work, keeping the wire faithful through the bridge seems worth one line: forward parts[1] as is.

separately, happy to help get this moving again if useful. it is currently conflicting with main; i could rebase it and split the fixture, bridge, and scenario pieces into smaller prs, keeping your structure.

[aeron-gh]

quick direction question: should the kvcache e2e suite assume the bridge long term, or the bridgeless path now that #1534 made the subscriber honor VLLM_ZMQ_TOPIC_FILTER? bridgeless would mean composing the pod ip into that value in the fixture, downward api covers it since the sidecar shares the pod with the sim. the scenarios work against either transport, i just want to build on the one you prefer.

[HARSHRAJ2789]

Ran this branch against current main to see whether the July failure still stands. It does, and the rebase takes some work, so writing up what I found.

It no longer applies to main. plugins_test.go conflicts, entirely because main refactored TestSchedulerPluginRandom into batched sampling after this branched. The PR's own commit touches no Random lines, so keeping main's side resolves it and the KVCache test is untouched. After that, go build ./test/... and gofmt are clean. I tested against 55b9bcf; the conflict is still there on 75a3153.

Before rebasing, the chart RBAC is stale. kthena-router crashloops on failed to list *v1alpha1.ExternalModelProvider: ... is forbidden and then Failed to sync informer caches, so nothing runs at all. That resolves once the chart comes from main.

Rebased, TestSchedulerPluginKVCacheAware still fails the same way. Four consecutive runs: 98.25s, 100.20s, 100.16s, 94.19s, all Model router-plugin-model did not become ready within 1m30s. The CI run on 2026-07-08 failed at 94.12s, so this is the same failure, not a new one.

Four explanations I checked and dropped:

1. A sidecar never becoming ready. It does. Sampling every 3s, llm-engine, runtime and zmq-bridge are all ready=true by t=48s with 0 restarts, and the pod sits at 3/3 while requests keep failing for another minute.
2. A port mismatch. The sim runs --port=8000, the container declares containerPort: 8000, and the ModelServer has workloadPort: {port: 8000, protocol: http}.
3. The sim not serving. From inside the pod, GET http://127.0.0.1:8000/v1/models returns 200 with router-plugin-model and max_model_len: 1024. Its log shows http.go:120 Server starting protocol=HTTP port=8000.
4. A selector problem. The pod carries app=router-plugin-mock and the ModelServer selects matchLabels: {app: router-plugin-mock}.

What the router reports once the backends are healthy:

router.go:624 request failed reqID: ...: request to all pods failed
"POST /v1/chat/completions HTTP/1.1" 503 error=proxy:request processing failed
  model_name=router-plugin-model model_route=.../router-plugin-route
  model_server=.../router-plugin-mock tokens=9/0 timings=1ms(0+0+0)

The route and the ModelServer both resolve, and then every attempt fails in 1 to 3ms with all three timing components at zero. That reads as the router having no usable endpoint to try, as opposed to a backend that is slow or refusing. That is where I stopped: I have not proven what the router's view of the pool is at that moment, so treat this as a narrowing, not a diagnosis.

Happy to push the rebased branch somewhere if that is useful, or to keep digging on the endpoint question with schedulerOnlyKVCacheAware in place.

Environment: kind, one node, kthena images built from the rebased branch, cert-manager and Volcano per test/e2e/setup.sh, macOS arm64 host.

[aeron-gh]

nice narrowing. one discriminator you already have in the logs: the proxy retry loop logs every failed attempt at error level ("pod request error: ..." in router.go, right before that 503). if those lines are present, the router had endpoints and the dial failed, and the check missing from your list is router to pod ip rather than localhost: hit http://:8000/v1/models from outside the pod (a throwaway curl pod works, the router image is distroless so there is no shell to exec into). the sim answering on 127.0.0.1 does not prove it binds 0.0.0.0. if the lines are absent, BestPods was empty and it never dialed anything, which points at the scheduler or datastore side, and i can dig there, that pool view is the code i have been living in (#1463, #1513, #1542).

[HARSHRAJ2789]

That discriminator answers it, and the logs I already had are decisive: 73 request to all pods failed lines, 0 pod request error.

Went to the source to confirm the description, and it lines up. The loop is for i := 0; i < len(ctx.BestPods); i++ (router/router.go:797), the only klog.Errorf(" pod request error: %v", err) is at :819 inside it after proxyRequest returns, and the fallthrough is :831. So an empty BestPods reaches the 503 without emitting a single line from the body, which is exactly the shape of what I captured. Sampling caveat: I polled kubectl logs --since=6s every 3s, so windows overlap and there is no gap, but 0 out of 73 is a strong number, short of a proof.

So BestPods is empty and nothing is ever dialled. That also explains timings=1ms(0+0+0), since the phases only accumulate inside the loop.

Your other point stands regardless and I will not claim otherwise: the sim answering on 127.0.0.1 from inside the pod says nothing about 0.0.0.0. I have not tested router-to-pod-IP, so that check remains open even though this path did not reach it.

Worth adding that it is schedulerOnlyKVCacheAware here, so the KVCache plugin is the only scorer and an empty result has nowhere to fall back to. Please do dig on the pool view if you have the context for it. I have a rebased branch of this PR that applies to main and reproduces in about 2m30s per run, so I can test anything you want to try quickly.