feat(server): resize pools over the admin API, and two things it needed to work

[weilei0120]

Where this started

Three questions about scaling a PD deployment, and what the answers turned out
to be:

Can the whole thing scale at once? Partly. Each service has its own
replicas, and for a multi-node service that count is groups rather than Pods.
There is no total, and no P:D ratio — 2P4D means setting two numbers.

Can prefill and decode scale independently? Yes, fully. Separate pools,
selected per request. Neither can go to zero: PD dispatch fails closed.

Is there an API? There was not. kubectl patch on the CR was the only way
in. This adds one.

The API

GET/POST on /v1/admin/scale, off unless --enable-scaling-api.

curl -X POST http://router:8000/v1/admin/scale \
  -H 'Content-Type: application/json' \
  -d '{"services": {"prefill": 4, "decode": 8}}'

It writes the InferaDeployment, which is the only write that survives: the
generated Deployment and LeaderWorkerSet are derived state, rewritten every
reconcile pass, so scaling those succeeds, reports no error, and is undone
seconds later.

Reads report three counts per pool — replicas (asked for), current_replicas
(Pods that exist), ready_replicas (workers serving). They converge left to
right, and the gap is the point: a Pod appears in seconds and serves in minutes.

Three decisions worth reviewing

Every pool named in one request moves in one write. Rebalancing prefill
against decode cannot half-apply and leave a deployment lopsided until someone
notices.

Zero is refused. An empty pool stops serving, and in PD an empty side fails
every request rather than only the ones that would have landed there.
Retiring a pool stays a kubectl edit, where the intent is unambiguous. Unknown
service names and negative counts are refused the same way, and a refused
request writes nothing.

The deployment comes from the server's own Pod labels, not a flag. The two
cannot disagree that way; a stale name would resize a fleet nobody asked about.
The operator's discovery Role gains get/patch on inferadeployments,
restricted by resourceNames to the one it belongs to — every Pod in a
deployment shares that identity, and a worker has no business resizing anything.

Two operator fixes this uncovered

Both predate this branch and both broke Kubernetes-native discovery on the
deployments the PD example tells people to write, since every service there uses
extraPodSpec.

extraPodSpec pods had no identity. POD_NAME and POD_NAMESPACE were
injected on the path where the operator renders the template itself; a template
supplied by the user took a different path and got only the watch selector. A
worker cannot patch its own annotation to register without them.

Nothing supplied POD_IP. A worker binds 0.0.0.0 and advertises something
else, resolving it from POD_IP — logic that has been there all along, reading
a variable nobody set. So it registered its bind address and looked healthy
doing it: up, weights loaded, listed in /v1/workers, and the first inference
request came back {"error":"worker 0.0.0.0:8080 unreachable"}.

Not in scope

No autoscaler. This is somewhere to push a decision that has already been made,
not a control loop — nothing here watches load. A worker takes about two minutes
to become useful, which outlasts most bursts, and Kubernetes offers no way to
choose which replica a scale-down removes, so the one holding the warmest cache
is as likely to go as any other.

An InferaDeployment also cannot carry a Kubernetes /scale subresource:
spec.services is a map with user-chosen keys, while specReplicasPath must be
a static JSONPath and a CRD may declare only one. Naming the service in the
request sidesteps that.

Test plan

• 31 unit tests (tests/unit/server/test_scaling.py) and 5 operator tests;
  full suites pass. The 6 remaining Python failures are a pre-existing
  baseline (sglang/gaie not installed), confirmed unchanged against main.
• Cluster, PD deployment. 1 prefill / 2 decode resized to 3 / 4 by one
  call: CR updated immediately, Pods within 35 s, and the read distinguished
  requested from observed during the window. Scaling to zero, an unknown
  service and a negative count were each refused with the CR unchanged; a
  server without the flag answered 403. The generated Role was read back and
  carried get/patch scoped to the single deployment.
• Cluster, real engine. One vLLM worker (Qwen3-8B, MI300X) scaled to two
  and back over the API. The added worker loaded weights, registered, and
  served 4 of the 8 requests issued after it joined — a Pod existing is
  not the same as a worker answering.
• Both operator fixes verified on a manifest carrying neither POD_IP nor
  --advertise-host: the worker logged advertising Pod IP 172.16.1.57,
  registered that, and served inference. The same deployment previously
  needed the variable written by hand.

Made with Cursor

[Copilot]

Pull request overview

This PR introduces an opt-in admin API for resizing InferaDeployment service pools via the router/server, and makes operator changes needed for Kubernetes-native discovery/registration to work reliably (especially when using extraPodSpec).

Changes:

• Add GET/POST /v1/admin/scale (guarded by --enable-scaling-api) to read and update per-service replica counts by patching the InferaDeployment.
• Fix operator-rendered Pod templates to always provide required downward-API identity (POD_NAME, POD_NAMESPACE) and routable address (POD_IP), including the extraPodSpec path.
• Expand docs and tests covering scaling semantics, RBAC scoping, and the new HTTP surface.

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
tests/unit/server/test_scaling.py	Adds unit tests for scaler behavior (atomic multi-pool patching, validation) and FastAPI endpoints.
manual/getting_started/quickstart.md	Adds a “Scale it” section and links to scaling documentation.
manual/features/scaling.md	Documents scaling over the admin API, response shape, and constraints (e.g., refusing zero).
infera/server/scaling.py	Implements DeploymentScaler and validation for patching InferaDeployment replicas via the in-cluster API.
infera/server/args.py	Adds --enable-scaling-api flag (and env var default) to gate the new endpoints.
infera/server/app.py	Wires new /v1/admin/scale routes and integrates a DeploymentScaler instance into app initialization.
infera/server/main.py	Instantiates and injects the scaler when scaling API is enabled.
deploy/operator/internal/controller/scale_paths_test.go	Adds tests asserting discovery Role can patch only its own InferaDeployment (ResourceNames-scoped).
deploy/operator/internal/controller/builders.go	Injects POD_IP and ensures identity env vars are present for extraPodSpec; adds inferadeployments RBAC rule.
deploy/operator/internal/controller/builders_test.go	Adds tests for extraPodSpec identity injection and POD_IP downward-API wiring.

Suppressed comments (1)

infera/server/scaling.py:148

• Same as snapshot(): make_client() can raise OSError when in-cluster credentials are missing, which currently escapes as a 500 because only ScalingError is translated by the FastAPI route. Catch OSError around the client creation and return a ScalingError with a clear message/status.

        async with make_client() as client:

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.