fix(tonic-xds): reduce hot-path allocs and config tearing in the routing layer - #2794
Merged
YutaoMa merged 4 commits intoAug 11, 2026
Merged
Conversation
## Motivation The routing layer loaded the config twice per request with the header-mutation hook in between, so the hook and the matching that consumes its output could see different versions and the request would silently take the default route. The same split also skipped the hook until the first config arrived. Matching was async only to host that wait, so every request paid a nested boxed future and the owned copies a `'static` future forces; the second load handed the hook an owned metadata clone, and the config `Arc` stayed alive for the whole request rather than just the decision. ## Solution One required trait method returning either an available config or a future for the first one. Matching borrows it and turns synchronous, dropping a nested boxed future, an authority `String`, a `HeaderMap` clone and a metadata deep clone; block-scoped so the `Arc` releases before the inner call.
YutaoMa
changed the title
YutaoMa
marked this pull request as ready for review
August 6, 2026 22:21
mingley
reviewed
Aug 6, 2026
mingley
reviewed
Aug 6, 2026
ankurmittal
approved these changes
Aug 11, 2026
LYZJU2019
added a commit
to LYZJU2019/tonic
that referenced
this pull request
Aug 11, 2026
…olicy Wire the gRPC channel's retry configuration to the control plane so retry behavior tracks RDS (RouteConfiguration) updates without rebuilding the channel. The retry settings come from the standard Envoy `RouteAction.retry_policy` (gRFC A44), so OSS parses them natively — no caller-supplied extractor is needed. - Parse `RouteAction.retry_policy` in the RDS resource: add `RouteRetryConfig` (raw `retry_on`, `num_retries`, backoff intervals) and attach it to `RouteConfigResource` as the effective retry policy, taken from the first route (document order, across virtual hosts) that carries one. This models today's single blanket-route RDS emitted for gRPC services; per-route (per-method) retry is future work once the retry layer can be keyed by the matched route. - Derive each request's retry policy from the same route-config snapshot the routing layer routes with, so routing and retry never tear across an RDS update. The routing service stamps the matched route's `RouteRetryConfig` into the request extensions; the retry layer reads it and builds a per-request `GrpcRetryPolicy` via `PerRequestRetryPolicy::for_request`, falling back to the layer's default policy when no retry config is present (non-xDS callers, or an RDS update with no `RouteAction.retry_policy`). - Map Envoy `retry_on` conditions to gRPC status codes (gRFC A44) with `grpc_retry_on_codes`; non-gRPC tokens are ignored (connection-level retries are handled separately). Envoy `numRetries` maps directly to `RetryConfig.num_retries` (retries, not attempts); unset Envoy fields fall back to RetryConfig defaults. This builds on the routing layer's config snapshot (grpc#2794): the retry layer consumes the snapshot routing already acquired instead of watching RDS independently, so no background task or atomically-swapped shared state is needed. The resource layer stays transport-neutral: `RouteRetryConfig` holds raw Envoy values, and the client retry layer applies gRPC semantics and defaults.
LYZJU2019
added a commit
to LYZJU2019/tonic
that referenced
this pull request
Aug 12, 2026
…olicy Wire the gRPC channel's retry configuration to the control plane so retry behavior tracks RDS (RouteConfiguration) updates without rebuilding the channel. The retry settings come from the standard Envoy `RouteAction.retry_policy` (gRFC A44), so OSS parses them natively — no caller-supplied extractor is needed. - Parse `RouteAction.retry_policy` in the RDS resource: add `RouteRetryConfig` (raw `retry_on`, `num_retries`, backoff intervals) and attach it to `RouteConfigResource` as the effective retry policy, taken from the first route (document order, across virtual hosts) that carries one. This models today's single blanket-route RDS emitted for gRPC services; per-route (per-method) retry is future work once the retry layer can be keyed by the matched route. - Derive each request's retry policy from the same route-config snapshot the routing layer routes with, so routing and retry never tear across an RDS update. The routing service stamps the matched route's pointer-stable `Arc<RouteRetryConfig>` into the request extensions; the retry layer builds a `GrpcRetryPolicy` from it via `RetryPolicyFromRoute` and memoizes it, keyed by the config `Arc`'s identity. Since RDS updates are rare, the hot path reuses the cached policy (a pointer bump) and only rebuilds — parsing `retry_on` and allocating the code set — when the config version changes. Requests without retry config (non-xDS callers, or an RDS update with no `RouteAction.retry_policy`) fall back to the layer's default policy. - Map Envoy `retry_on` conditions to gRPC status codes (gRFC A44) with `grpc_retry_on_codes`; non-gRPC tokens are ignored (connection-level retries are handled separately). Envoy `numRetries` maps directly to `RetryConfig.num_retries` (retries, not attempts); unset Envoy fields fall back to RetryConfig defaults. This builds on the routing layer's config snapshot (grpc#2794): the retry layer consumes the snapshot routing already acquired instead of watching RDS independently, so no background task or atomically-swapped shared state is needed. The resource layer stays transport-neutral: `RouteRetryConfig` holds raw Envoy values, and the client retry layer applies gRPC semantics and defaults.
LYZJU2019
added a commit
to LYZJU2019/tonic
that referenced
this pull request
Aug 12, 2026
…olicy Wire the gRPC channel's retry configuration to the control plane so retry behavior tracks RDS (RouteConfiguration) updates without rebuilding the channel. The retry settings come from the standard Envoy `RouteAction.retry_policy` (gRFC A44), so OSS parses them natively — no caller-supplied extractor is needed. - Parse `RouteAction.retry_policy` in the RDS resource: add `RouteRetryConfig` (raw `retry_on`, `num_retries`, backoff intervals) and attach it to `RouteConfigResource` as the effective retry policy, taken from the first route (document order, across virtual hosts) that carries one. This models today's single blanket-route RDS emitted for gRPC services; per-route (per-method) retry is future work once the retry layer can be keyed by the matched route. - Derive each request's retry policy from the same route-config snapshot the routing layer routes with, so routing and retry never tear across an RDS update. The routing service stamps the matched route's pointer-stable `Arc<RouteRetryConfig>` into the request extensions; the retry layer builds a `GrpcRetryPolicy` from it via `RetryPolicyFromRoute` and memoizes it, keyed by the config `Arc`'s identity. Since RDS updates are rare, the hot path reuses the cached policy (a pointer bump) and only rebuilds — parsing `retry_on` and allocating the code set — when the config version changes. Requests without retry config (non-xDS callers, or an RDS update with no `RouteAction.retry_policy`) fall back to the layer's default policy. - Map Envoy `retry_on` conditions to gRPC status codes (gRFC A44) with `grpc_retry_on_codes`; non-gRPC tokens are ignored (connection-level retries are handled separately). Envoy `numRetries` maps directly to `RetryConfig.num_retries` (retries, not attempts); unset Envoy fields fall back to RetryConfig defaults. This builds on the routing layer's config snapshot (grpc#2794): the retry layer consumes the snapshot routing already acquired instead of watching RDS independently, so no background task or atomically-swapped shared state is needed. The resource layer stays transport-neutral: `RouteRetryConfig` holds raw Envoy values, and the client retry layer applies gRPC semantics and defaults.
YutaoMa
pushed a commit
that referenced
this pull request
Aug 13, 2026
…olicy (#2786) ## Summary Drives the gRPC channel's retry configuration from the control plane so retry behavior tracks **RDS** (`RouteConfiguration`) updates without rebuilding the channel. The retry settings come from the standard Envoy `RouteAction.retry_policy` (gRFC A44), so `tonic-xds` parses them **natively** — no caller-supplied extractor closure is required. This is additive and non-breaking; there is no public API change. Built on top of #2794. Retry is **per route**: each request retries according to the exact route it matched. The Envoy retry policy is validated when the `RouteConfiguration` is validated; the gRPC retry config is compiled **once per RDS update** and shared behind an `Arc`, so the request hot path just looks it up and clones that pointer, doing no parsing or allocation. ## What changed - **Validate the retry policy at RDS-validation time; keep gRPC types out of the resource layer.** `RouteRetryConfig::from_proto` validates the Envoy `RetryPolicy` (gRFC A44 — NACK on `numRetries == 0`, non-positive `baseInterval` / `maxInterval`, or out-of-range `google.protobuf.Duration` values) and stores the validated, transport-neutral values on the matched route as `RouteConfig.retry_config: Option<Arc<RouteRetryConfig>>`. A route uses its own `RouteAction.retry_policy` when set, otherwise it inherits the enclosing `VirtualHost.retry_policy` (gRFC A44: a route-level policy completely overrides the virtual host's — values are not merged); `None` when neither specifies retry, and routes inheriting the vhost policy share one `Arc`. The xDS resource types hold no gRPC-specific types. - **Compile and select the gRPC config per route via the routing decision.** A `RoutingSnapshot` (client layer) bundles the validated `RouteConfigResource` with a map of `GrpcRetrySharedConfig`s compiled from it **once per RDS update** (the `XdsRouter` watch task), keyed by route-config identity. The routing layer resolves the matched route, looks up its compiled retry config in that same snapshot, and stamps it into the request's `RouteDecision`. Because routing and retry read one snapshot, they always act on the same RDS version (no cross-layer skew), and retry runs inside routing so the decision is fixed across a request's retry attempts. Bundling both in one `Arc` keeps the request hot path to a map lookup plus a pointer clone. - **Separate the shared config from per-request state.** `RetrySharedConfig<C>` holds the immutable config (attempt cap, backoff, retryable code set) behind an `Arc`; `RetryPolicy<C>` holds a pointer to it plus the per-request retry state (backoff cursor, attempt count). Instantiating a policy for a request (`RetryPolicy::from_shared`) is an `Arc` pointer clone plus a zero-field state init — the request hot path does no parsing or allocation. Requests with no `RouteDecision` (non-xDS callers), whose route carries no retry policy, or whose `retry_on` maps to no gRPC status code use the layer's fallback config. - **Map `retry_on` to gRPC status codes.** `grpc_retry_on_codes` maps Envoy `retry_on` conditions to gRPC status codes (gRFC A44); non-gRPC tokens are ignored (connection-level retries are handled separately). When the mapped set is empty, the route installs no status-code policy — it falls back to the layer default rather than masking connection retries. Envoy `numRetries` maps directly to `RetryConfig.num_retries` (retries, not attempts); unset Envoy fields fall back to `RetryConfig` defaults. The retry engine (`RetryPolicy`, `RetrySharedConfig`, and the `RetryClassifier` seam) stays transport-agnostic; only the retry layer is gRPC-specific, because it reads the concrete `RouteDecision` extension. The xDS resource types stay free of gRPC business logic: the gRPC retry config is compiled in the client/routing layer, once per RDS update, which is what keeps the request path allocation-free. ## Example RDS consumed ```json { "@type": "type.googleapis.com/envoy.config.route.v3.RouteConfiguration", "name": "AccessControlApi", "virtualHosts": [{ "name": "AccessControlApi", "domains": ["*"], "routes": [{ "match": { "prefix": "" }, "route": { "cluster": "AgentLifecycleGrpc|0", "timeout": "60s", "retryPolicy": { "retryOn": "unavailable", "numRetries": 2, "retryBackOff": { "baseInterval": "0.100s", "maxInterval": "1s" } } } }] }] } ``` The `retryPolicy` may equivalently be set at the `virtualHosts[*]` level, in which case every route in the virtual host that doesn't set its own inherits it. ## Testing UTs passed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
This PR includes a few optimizations I identified while trying to use the routing layer interceptor APIs.
The routing layer loaded the config twice per request with the header-mutation hook in between, so the hook and the matching that consumes its output could see different versions and the request would silently take the default route. The same split also skipped the hook until the first config arrived.
Matching was async only to host that wait, so every request paid a nested boxed future and the owned copies a
'staticfuture forces; the second load handed the hook an owned metadata clone, and the configArcstayed alive for the whole request rather than just the decision.Solution
One required trait method returning either an available config or a future for the first one. Matching borrows it and turns synchronous, dropping a nested boxed future, an authority
String, aHeaderMapclone and a metadata deep clone; block-scoped so theArcreleases before the inner call.