fix: unblock long-context requests in the rust router, and kvd bug - #120
Merged
Conversation
The rust router 413s any request whose body exceeds 2 MiB, so a long-context
prompt never reaches the engine. In the 8-14 GLM-5.2 agentic run this was 26 of
the 27 dropped requests, reported by the aggregation as
"error_categories": {"InvalidInferenceResultError": 1, "Payload Too Large": 26}
and at concurrency 64 it aborted warmup outright.
handlers.rs takes the body with the `Bytes` extractor, which inherits axum's
DefaultBodyLimit of 2 MiB. Nothing in the router configures it -- there is no
DefaultBodyLimit, body_limit or RequestBodyLimitLayer anywhere in rust/router.
A web-form default is the wrong ceiling for an LLM proxy, where a prompt near
the engine's context length is several MiB of JSON.
Measured against a live 1P1D deployment (GLM-5.2-MXFP4, TP8 per leg):
| body | rust :8100 | python :8101 | engine :30000 |
|----------|------------|--------------|---------------|
| 2.00 MiB | 200 | | |
| 2.05 MiB | 413 | | |
| 3.00 MiB | 413 | 400 | 400 |
The 400 is the engine rejecting the body's contents, which means the body
arrived. So the engine accepts 3 MiB and so does this router's own python
backend: Starlette imposes no body limit and nothing in infera/server/ or
infera/api/ adds one. Only the rust backend refused it, and it is the default
backend, so the two backends of the same router disagreed on what is a legal
request.
Disable the limit rather than raising it. Any number here is arbitrary and will
be wrong for someone; the ceiling on prompt size is the engine's
--context-length, and a proxy has no business imposing a second, smaller one.
This also makes the two backends agree.
functional.rs gets a regression case that posts a 4 MiB prompt through the
assembled app and asserts the mock worker received it whole. With the layer
commented out it reproduces the production failure exactly (left: 413,
right: 200). The mock worker needs the same disable, since it stands in for an
engine that caps a prompt by context length rather than by request bytes.
cargo fmt, clippy --all-targets -- -D warnings and the full
cargo test -p infera-router suite (134 unit + 15 functional) are clean.
Verified on the deployment afterwards: a 3.00 MiB body uploads all 3,145,765
bytes and comes back 400 from the engine's field validation instead of 413.
Signed-off-by: xiaobochen-amd <xiaobo.chen@amd.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
… write A kvd prefill leg died at startup on lmsysorg/sglang:v0.5.17-rocm720-mi35x, on the first assignment _finish_wiring makes: sa.enable_hierarchical_cache = True ServerArgs.__setattr__ raises AttributeError for any public field assigned after resolution -- "server_args.<name> assigned after resolution; server_args is read-only -- use get_context().override(source, ...)". The guard is not new; on this base it is unconditional where it used to be gated on SGLANG_STRICT_CONFIG_MUTATION, so wiring that worked before now raises. ServerArgs is not a frozen dataclass (__dataclass_params__.frozen is False), so the failure is that custom __setattr__ rather than dataclass immutability. Every write in _finish_wiring is metadata-only. The engine runs in a subprocess that re-parses argv, and _append_sglang_hicache_argv is what actually selects the backend, so these assignments only keep the in-process record tidy. Losing one must not cost a leg. Make each best-effort, log the refusal at debug, and emit the "implies ..." info line only when the value landed -- otherwise the log claims a sync that did not happen. Not exercised by the 1P1D agentic deployments in use: they run without --infera-kvd-socket and so never enter this function. The change is confined to that path. Signed-off-by: xiaobochen-amd <xiaobo.chen@amd.com> Co-authored-by: Cursor <cursoragent@cursor.com>
xiaobochen-amd
requested review from
JohnQinAMD,
jiejingzhangamd and
limou102
as code owners
August 15, 2026 13:25
Contributor
There was a problem hiding this comment.
Pull request overview
This PR contains two independent bug fixes: (1) removing an unintended 2 MiB request-body cap in the Rust router that was 413’ing long-context prompts before they reached the engine, and (2) making SGLang KVD wiring metadata updates best-effort to avoid startup failure on the v0.5.17 base where ServerArgs becomes read-only after resolution.
Changes:
- Disable axum’s default request body limit on the Rust router so long-context requests are forwarded to the engine instead of being rejected at the proxy.
- Add a functional regression test ensuring the router forwards a request body larger than axum’s default limit.
- Make KVD wiring’s
ServerArgsmetadata mutations resilient toAttributeError(read-only ServerArgs) so KVD legs don’t crash at startup.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| rust/router/src/handlers.rs | Disables axum default body limit for Bytes request bodies so long-context prompts aren’t rejected by the router. |
| rust/router/tests/functional.rs | Adds a regression test that sends an over-2 MiB request body and asserts it reaches the upstream worker. |
| infera/engine/sglang/kvd_wiring.py | Wraps ServerArgs metadata writes in a best-effort helper to avoid startup aborts on read-only configurations. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Two independent fixes found while running GLM-5.2 1P1D agentic on MI355X.
Measurements are in the per-commit messages.
Type of change
Changes
The rust router 413s any body over 2 MiB. The
Bytesextractors inhandlers.rsinherit axum's default limit, so a long-context prompt neverreaches the engine: 26 of the 27 dropped requests in a 3600s agentic run, and
an outright warmup abort at concurrency 64. The engine accepts 3 MiB and so
does this router's own python backend, so the two backends disagreed on what a
legal request is. Disabled rather than raised — prompt size is the engine's
--context-lengthto enforce, not a proxy's. Regression test included.A kvd leg dies at startup on the v0.5.17 base.
ServerArgs.__setattr__now refuses public writes after resolution unconditionally, where it used to
be gated on
SGLANG_STRICT_CONFIG_MUTATION, and_finish_wiring's firstassignment raises. Those writes are metadata-only — the engine subprocess
re-parses argv — so they are best-effort now.
Checklist:
cargo fmt,clippy --all-targets -- -D warningsandcargo test -p infera-routerare clean; the regression test reproduces the 413 with the layerremoved. Verified on a live deployment: a 3.00 MiB body now comes back as the
engine's 400 instead of 413.
The kvd path is not exercised by deployments that run without
--infera-kvb-socket, so that fix is reasoned rather than measured.