feat(flowcontrol): Add Foundational Types and Architecture #997

LukeAVanDrie · 2025-06-16T21:09:02Z

Description

This PR introduces the foundational building blocks for the new Flow Control system, designed to manage priority, fairness, and queuing for inference workloads.

The goal is to provide a sophisticated mechanism for managing diverse SLOs and preventing issues like Head-of-Line blocking and system overload, which are not addressed by simple FCFS temportal scheduling. This is a relatively sophisticated and complex system that will need to be split across several PRs. This initial submission lays the groundwork for the entire module by establishing its core concepts and data contracts (basically all types with no cross-package dependencies).

This work tracks issue #674

For historical context, the initial KEP-like proposal can be found here. Please note that the design has evolved significantly, and this PR represents the most current architecture (or at least a small slice of it). I will do some documentation cleanup once more of these PRs with the canonical architectural decisions are out for review.

Contribution

This PR includes two main pieces:

Top-Level README.md: Provides a high-level overview of the Flow Controller, including:
- A clear Motivation for why this component is necessary.
- A high-level Architectural Diagram showing the component interactions and request flow.
- A summary of the core Architectural Pillars (Controller, Framework, Registry, etc.).
The types Package: This new package establishes the core "vocabulary" for the entire Flow Controller module. It includes:
- Request Lifecycle Interfaces (FlowControlRequest, QueueItemAccessor, QueueItemHandle).
- Final Outcome Reporting via the QueueOutcome enum and a structured set of error types (ErrRejected, ErrEvicted).
- The FlowSpecification interface for defining workload identity and priority.
- A README.md detailing the concepts within the types package.

Review Focus

As this is a foundational PR, I'm particularly looking for feedback on:

Does the high-level architecture and motivation make sense?
Are the data models in the types package clear, logical, and sufficient for the tasks ahead?
Is the vocabulary (e.g., QueueOutcome, the error hierarchy) well-defined and intuitive?

Future PRs will build upon these types to implement the framework, registry, and controller packages.

netlify · 2025-06-16T21:09:08Z

✅ Deploy Preview for gateway-api-inference-extension ready!

Name	Link
🔨 Latest commit	`25a1e08`
🔍 Latest deploy log	https://app.netlify.com/projects/gateway-api-inference-extension/deploys/6851efe911955d0008622782
😎 Deploy Preview	https://deploy-preview-997--gateway-api-inference-extension.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

k8s-ci-robot · 2025-06-16T21:09:09Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: LukeAVanDrie
Once this PR has been reviewed and has the lgtm label, please assign kfswain for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot · 2025-06-16T21:09:13Z

Hi @LukeAVanDrie. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

LukeAVanDrie · 2025-06-16T21:12:02Z

I have the followup PRs for the rest of the type system that contextualize this queued up, but I am currently on a 🚂 and my connection is spotty. Will send those out soon.

The next PR will detail the framework package (all the extension points) which should be a greater source of discussion compared to this PR. I will then have followups for the ports (saturation detector and flow registry) and the core execution engine (flow controller). These will have much more detailed architecture diagrams as well. The top-level README is intended to have the high level view only.

danehans · 2025-06-16T21:19:07Z

/ok-to-test

Introduces the foundational packages for the new Flow Controller component. This change includes: - The top-level README outlining the motivation, high-level architecture, and component pillars. - The `types` package, which defines the core data contracts, request lifecycle interfaces, error-handling vocabulary, and final outcome enums for the entire module. This foundational PR establishes the core concepts and data models upon which the rest of the Flow Controller implementation will be built.

ahg-g

Do you have a followup PR that uses those types, it is easier to reason about those interfaces and types when they are actually used.

ahg-g · 2025-06-20T17:06:44Z

pkg/epp/flowcontrol/README.md

+The Flow Controller is a sophisticated library designed to solve these problems. It acts as a crucial gatekeeper that
+decides *if* and *when* a request should proceed to be scheduled. Its primary mission is to enable predictable, fair,
+and efficient utilization of shared backend resources by enforcing prioritization, applying fairness policies, managing
+request queuing under saturation, and orchestrating displacement (the eviction of lower-priority queued items to make


I would use the word shedding instead of eviction

I named the specific extension point "displacement". I documented the justification in: https://github.com/LukeAVanDrie/gateway-api-inference-extension/blob/flow-control/pkg/epp/flowcontrol/framework/README.md#terminology-dispatch-vs-displacement.

Eviction, preemption, and shedding were also considered.

ahg-g · 2025-06-20T17:21:09Z

pkg/epp/flowcontrol/README.md

+It is designed for extensibility, allowing custom logic for policies and queuing mechanisms to be plugged into a robust,
+high-performance orchestration engine.
+
+### Role in the Gateway API Inference Extension


I recommend to define goals and non-goals, here is my rough list:

Goals:

Enable priority and fairness across workloads.

Optimize request scheduling by shifting a portion of request queueing from the model server to a centralized queue.

O(seconds) average and O(single digit minutes) tail queueing time

Non-goals:

Persistence: Queueing is handled in the endpoint picker's memory.

Scale: Scale is limited by available memory and number of ext-proc connections of the endpoint picker and L7LB. While they can horizontally scale, maintaining connections for many minutes to hours is not a goal.

A substitute for a message queue: The queue manages requests with open connections via the L7LB and is not intended for asynchronous request handling.

ahg-g · 2025-06-20T17:54:26Z

pkg/epp/flowcontrol/README.md

+    class A2 downstream_err;
+    class D,E,F,P,SD default;
+```
+


We should clearly distinguish between request flow (the top one?) and configuration flow (the bottom one?).

Ack, I will add some controller (e.g., a k8s operator) that configures the Flow Registry.

We don't want to completely detach from epp, so I don't think we want to describe a generic k8s operator that does this.

I am just asking to distinguish between the arrows that are describing a request flow (traversed for each request) and the ones that are configuration (registering plugins and whatnot). I would split this actually into two diagrams if necessary. I would like to see the data flow separately from the configuration flow

ahg-g · 2025-06-20T17:57:04Z

pkg/epp/flowcontrol/README.md

+    concurrent-safe request storage. It uses a `QueueCapability` system that allows for diverse and extensible queue
+    implementations (e.g., FIFO, Priority Heap) while maintaining a stable interface.
+
+4.  **The `FlowRegistry` (`./registry`, `./ports`)**: This is the stateful control plane of the system. It manages the


what is a port in this context?

I am using a "Ports and Adapters" (or "Hexagonal") architecture pattern here. The ports in this flow control system are documented here: https://github.com/LukeAVanDrie/gateway-api-inference-extension/blob/flow-control/pkg/epp/flowcontrol/ports/README.md.

ahg-g · 2025-06-20T18:01:35Z

pkg/epp/flowcontrol/types/README.md

@@ -0,0 +1,33 @@
+# Flow Control Core Types


I recommend having this documentation in the code as comments, I worry that as the code evolves, such detailed documentation will quickly go out of sync.

ahg-g · 2025-06-20T18:07:37Z

pkg/epp/flowcontrol/types/flow.go

+	// Priority returns the numerical priority level currently associated with this flow within the Flow Registry.
+	//
+	// Convention: Lower numerical values indicate higher priority.
+	Priority() uint


This is great, so we are not limited to discrete values, correct?

Yep; however, the priority bands (defining the possible values of Priority()) are startup-time configured in the Flow Registry. Most of the system supports dynamic updates, just not this.

Flows can migrate between priorities at runtime, you just cannot change what the priority options are.

ahg-g · 2025-06-20T18:09:31Z

pkg/epp/flowcontrol/types/request.go

+// An object implementing this interface is the primary input to `FlowController.EnqueueAndWait()`. The controller then
+// wraps this object with its own internal structures (which implement `QueueItemAccessor`) to manage the request's
+// lifecycle without modifying the original.
+type FlowControlRequest interface {


Do we expect to have more than one implementation of this interface? if so, is it really necessary to have this as an interface vs a struct?

I left it this way so this can be imported outside EPP. All of this is functional as a portable library. Also, I have an escape hatch with OriginalRequest that allows back casting to get richer request metadata not commonly shared between policies. This is useful for prototyping for advanced use cases. I think there is some value in allowing any type implementing this interface to be enqueued.

This also makes testing a bit easier.

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Jun 16, 2025

k8s-ci-robot requested review from Jeffwan and kfswain June 16, 2025 21:09

k8s-ci-robot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Jun 16, 2025

k8s-ci-robot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Jun 16, 2025

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Jun 16, 2025

LukeAVanDrie force-pushed the flow-control-system branch from c86d3d8 to 0645012 Compare June 17, 2025 22:11

LukeAVanDrie force-pushed the flow-control-system branch from 2c6952e to 25a1e08 Compare June 17, 2025 22:44

ahg-g reviewed Jun 20, 2025

View reviewed changes

feat(flowcontrol): Add Foundational Types and Architecture #997

Are you sure you want to change the base?

feat(flowcontrol): Add Foundational Types and Architecture #997

Conversation

LukeAVanDrie commented Jun 16, 2025

Description

Contribution

Review Focus

Uh oh!

netlify bot commented Jun 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for gateway-api-inference-extension ready!

Uh oh!

k8s-ci-robot commented Jun 16, 2025

Uh oh!

k8s-ci-robot commented Jun 16, 2025

Uh oh!

LukeAVanDrie commented Jun 16, 2025

Uh oh!

danehans commented Jun 16, 2025

Uh oh!

ahg-g left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

LukeAVanDrie Jun 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

LukeAVanDrie Jun 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

LukeAVanDrie Jun 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

netlify bot commented Jun 16, 2025 •

edited

Loading

LukeAVanDrie Jun 20, 2025 •

edited

Loading

LukeAVanDrie Jun 20, 2025 •

edited

Loading

LukeAVanDrie Jun 20, 2025 •

edited

Loading