Enable EPP to support endpoint discovery using pod selector #1833

capri-xiyue · 2025-11-07T19:47:40Z

What type of PR is this?

/kind feature

What this PR does / why we need it:
See #1779
Which issue(s) this PR fixes:

Part of #1779

Does this PR introduce a user-facing change?:

NONE to existing features with inferencepool. 
But users can use epp without inference pool via args 
`
        - --selector
        - "app:vllm-llama3-8b-instruct"
        - --target-ports
        - "8000"
`

Signed-off-by: Xiyue Yu <[email protected]>

k8s-ci-robot · 2025-11-07T19:47:46Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: capri-xiyue
Once this PR has been reviewed and has the lgtm label, please assign kfswain for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

netlify · 2025-11-07T19:47:47Z

✅ Deploy Preview for gateway-api-inference-extension ready!

Name	Link
🔨 Latest commit	`9d1514a`
🔍 Latest deploy log	https://app.netlify.com/projects/gateway-api-inference-extension/deploys/691b6cd0e81ab50008a7a9b4
😎 Deploy Preview	https://deploy-preview-1833--gateway-api-inference-extension.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

capri-xiyue · 2025-11-07T19:54:04Z

No need to review it right now. I just made the CUJ of standalone epp work without inferencepool. Still need to fix the e2e and ut

elevran

cursory review to provide initial feedback (realizing this is work in progress)
The main question I have (and might be worth mentioning in the PR description) is the need for a new abstraction/type (EndPointsPool). A naive/simple solution (which perhaps does not work...) would be to copy the selector and port array into a Go InferencePool object and use datastore.PoolSet() along with disabling the Pool notification/reconciliation so it does not overwrite with nil.
Hopefully the rest of the code should not care or depend on the pool's origin (from command line or the API server)

cmd/epp/runner/runner.go

pkg/epp/controller/pod_reconciler.go

pkg/epp/controller/inferenceobjective_reconciler_test.go

capri-xiyue · 2025-11-14T23:39:02Z

assign @ahg-g for early review.

capri-xiyue · 2025-11-14T23:39:19Z

assign @kfswain for early review

cmd/epp/runner/runner.go

pkg/epp/controller/pod_reconciler.go

pkg/epp/datalayer/endpointsPools.go

pkg/epp/datastore/datastore.go

Signed-off-by: Xiyue Yu <[email protected]>

ahg-g · 2025-11-17T21:34:41Z

cmd/epp/runner/runner.go

+	selector            = flag.String("selector", "", "selector to filter pods on, only key value paris is supported. Format: a comma-separated list of key value paris,  e.g., 'app:vllm-llama3-8b-instruct,env=prod'.")
+	targetPorts         = flag.String("target-ports", "", "target ports of model server pods. Format: a comma-separated list of numbers, e.g., '3000,3001,3002'")


too generic, perhaps prefix each flag with "endpoint"? endpoint-selector and endpoint-target-ports

ahg-g · 2025-11-17T21:43:23Z

cmd/epp/runner/runner.go

 	return nil
 }

+func strToUniqueIntSlice(s string) ([]int, error) {


there is no existing library for that in k8s?

ahg-g · 2025-11-17T21:43:58Z

cmd/epp/runner/runner.go

+	return intList, nil
+}
+
+func strToMap(s string) (map[string]string, error) {


ditto, no existing library for this tokenization?

ahg-g · 2025-11-17T21:44:29Z

config/charts/inferencepool/templates/epp-deployment.yaml

          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
+        - name: EPP_NAME


why do we need this?

ahg-g · 2025-11-17T22:13:30Z

pkg/epp/controller/inferencepool_reconciler.go

-	// 4. Convert the fetched object to the canonical v1.InferencePool.
-	v1infPool := &v1.InferencePool{}
-
+	var endPointsPool *datalayer.EndpointPool


endpointPool

ahg-g · 2025-11-17T22:13:51Z

pkg/epp/controller/inferencepool_reconciler_test.go

 		t.Errorf("Unexpected InferencePool reconcile error: %v", err)
 	}
-	if diff := diffStore(ds, diffStoreParams{wantPool: pool1, wantPods: []string{"pod1-rank-0", "pod2-rank-0"}}); diff != "" {
+	endPointsPool1 := pool.InferencePoolToEndPointsPool(pool1)


ditto, endpointPool1, and all other similar instances in this file.

ahg-g · 2025-11-17T22:14:40Z

pkg/epp/datalayer/endpointPool.go

+limitations under the License.
+*/
+
+package datalayer


file name should be endpoint_pool.go

ahg-g · 2025-11-17T22:15:24Z

pkg/epp/datastore/datastore.go

-// /// InferencePool APIs ///
-func (ds *datastore) PoolSet(ctx context.Context, reader client.Reader, pool *v1.InferencePool) error {
-	if pool == nil {
+// /// EndPoints APIs ///


EndpointPool

ahg-g · 2025-11-17T22:17:29Z

pkg/epp/util/pool/pool.go

+	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/datalayer"
+)
+
+func InferencePoolToEndPointsPool(inferencePool *v1.InferencePool) *datalayer.EndpointPool {


InferencePoolToEndpointPool

ahg-g · 2025-11-17T22:17:38Z

pkg/epp/util/pool/pool.go

+	return endPointsPool
+}
+
+func AlphaInferencePoolToEndPointsPool(inferencePool *v1alpha2.InferencePool) *datalayer.EndpointPool {


ahg-g · 2025-11-17T22:17:47Z

pkg/epp/util/pool/pool.go

+	return endPointsPool
+}
+
+func EndPointsPoolToInferencePool(endPointsPool *datalayer.EndpointPool) *v1.InferencePool {


ahg-g · 2025-11-17T22:18:18Z

pkg/epp/server/runserver.go

 	GrpcPort                         int
-	PoolNamespacedName               types.NamespacedName
-	PoolGKNN                         common.GKNN
+	EndPointsPool                    *datalayer.EndpointPool


EndpointPool

ahg-g · 2025-11-17T22:19:00Z

pkg/epp/server/controller_manager.go


 // defaultManagerOptions returns the default options used to create the manager.
-func defaultManagerOptions(gknn common.GKNN, metricsServerOptions metricsserver.Options) (ctrl.Options, error) {
+func defaultManagerOptions(endPointsPool *datalayer.EndpointPool, metricsServerOptions metricsserver.Options) (ctrl.Options, error) {


ditto, endpointPool

capri-xiyue added 7 commits October 31, 2025 16:18

partial draft

5945cfb

refactor

370f1a3

Signed-off-by: Xiyue Yu <[email protected]>

fixed some ut

85e4622

make epp controller ut pass

579b17e

make ut pass

e9d704d

fixed build

42cb218

fixed build

943c53d

k8s-ci-robot added the kind/feature Categorizes issue or PR as related to a new feature. label Nov 7, 2025

k8s-ci-robot requested a review from elevran November 7, 2025 19:47

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Nov 7, 2025

k8s-ci-robot requested a review from nirrozenbaum November 7, 2025 19:47

capri-xiyue changed the title ~~Enable EPP to support endpoint discovery using pod selector~~ [WIP] Enable EPP to support endpoint discovery using pod selector Nov 7, 2025

k8s-ci-robot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Nov 7, 2025

capri-xiyue added 3 commits November 10, 2025 12:54

fixed build failure

1e38821

fixed lint

30fd667

fix format

9e85377

elevran mentioned this pull request Nov 11, 2025

Standalone EPP #1838

Open

elevran reviewed Nov 13, 2025

View reviewed changes

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Nov 14, 2025

merge conflicts

43c87fa

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Nov 14, 2025

fixed import format

0267569

ahg-g reviewed Nov 14, 2025

View reviewed changes

rename and refactor

200dbf4

Signed-off-by: Xiyue Yu <[email protected]>

added epp name in env

9d1514a

capri-xiyue requested a review from ahg-g November 17, 2025 18:43

capri-xiyue changed the title ~~[WIP] Enable EPP to support endpoint discovery using pod selector~~ Enable EPP to support endpoint discovery using pod selector Nov 17, 2025

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Nov 17, 2025

ahg-g reviewed Nov 17, 2025

View reviewed changes

		selector = flag.String("selector", "", "selector to filter pods on, only key value paris is supported. Format: a comma-separated list of key value paris, e.g., 'app:vllm-llama3-8b-instruct,env=prod'.")
		targetPorts = flag.String("target-ports", "", "target ports of model server pods. Format: a comma-separated list of numbers, e.g., '3000,3001,3002'")

Enable EPP to support endpoint discovery using pod selector #1833

Are you sure you want to change the base?

Enable EPP to support endpoint discovery using pod selector #1833

Conversation

capri-xiyue commented Nov 7, 2025

Uh oh!

k8s-ci-robot commented Nov 7, 2025

Uh oh!

netlify bot commented Nov 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for gateway-api-inference-extension ready!

Uh oh!

capri-xiyue commented Nov 7, 2025

Uh oh!

elevran left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

capri-xiyue commented Nov 14, 2025

Uh oh!

capri-xiyue commented Nov 14, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

netlify bot commented Nov 7, 2025 •

edited

Loading