Skip to content

Add preferred prefill hint routing#4269

Draft
ztorlakTT wants to merge 1 commit into
mainfrom
ztorlak/preferred-prefill-hint
Draft

Add preferred prefill hint routing#4269
ztorlakTT wants to merge 1 commit into
mainfrom
ztorlak/preferred-prefill-hint

Conversation

@ztorlakTT

Copy link
Copy Markdown
Collaborator
  • Add an optional preferred_prefill_id hint that Dynamo/raw requests can pass through decode to PrefillGateway.
  • Make PrefillGateway honor the preferred prefill only when it is healthy, accepting tasks, and under capacity; otherwise it falls back to prefix/load/round-robin routing.
  • Add selector and dispatcher coverage for preferred-prefill routing and fallback behavior.

@ztorlakTT ztorlakTT self-assigned this Jun 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant