diff --git a/charts/kthena/charts/networking/README.md b/charts/kthena/charts/networking/README.md
index 30aa3cd63..7a5c25b98 100644
--- a/charts/kthena/charts/networking/README.md
+++ b/charts/kthena/charts/networking/README.md
@@ -20,6 +20,22 @@ kthenaRouter:
pullPolicy: IfNotPresent
```
+#### Access Log Configuration
+
+Access logging is enabled in text format by default for compatibility. Select
+JSON explicitly when a structured log collector expects JSON records.
+
+```yaml
+kthenaRouter:
+ accessLog:
+ enabled: true
+ format: json
+ output: stdout
+```
+
+Prefer `stdout` in Kubernetes. A file path writes inside the router container
+and requires a mounted volume and separate collection strategy.
+
#### Request Scheduling Configuration
The router schedules requests through a per-model queue using one of two mutually
@@ -65,6 +81,9 @@ kthenaRouter:
| `kthenaRouter.sessionBoost.inflightPerPod` | int | `16` | Inflight requests per backend pod; total = perPod x pod count |
| `kthenaRouter.sessionBoost.gracePeriod` | string | `"0s"` | Wait time for a same-session follow-up (disabled by default) |
| `kthenaRouter.sessionBoost.timeout` | string | `"30s"` | Max queue wait before 504; set a non-positive duration (e.g. `0s`) to disable |
+| `kthenaRouter.accessLog.enabled` | boolean | `true` | Enable request access logging |
+| `kthenaRouter.accessLog.format` | string | `"text"` | Access-log format: `text` or `json` |
+| `kthenaRouter.accessLog.output` | string | `"stdout"` | Access-log destination: `stdout`, `stderr`, or a file path |
#### Session Boost Configuration
diff --git a/charts/kthena/values.yaml b/charts/kthena/values.yaml
index 56a368330..e09405684 100644
--- a/charts/kthena/values.yaml
+++ b/charts/kthena/values.yaml
@@ -54,6 +54,13 @@ networking:
tag: latest
# -- Image pull policy for Kthena Router.
pullPolicy: IfNotPresent
+ accessLog:
+ # -- Enable router access logging.
+ enabled: true
+ # -- Access-log format. Supported values are `text` and `json`.
+ format: "text"
+ # -- Access-log destination: `stdout`, `stderr`, or a file path.
+ output: "stdout"
tls:
# -- Enable TLS for Kthena Router server.
enabled: false
diff --git a/docs/kthena/docs/reference/helm-chart-values.md b/docs/kthena/docs/reference/helm-chart-values.md
index 39dc53cc5..9b2308616 100644
--- a/docs/kthena/docs/reference/helm-chart-values.md
+++ b/docs/kthena/docs/reference/helm-chart-values.md
@@ -18,6 +18,9 @@ A Helm chart for deploying Kthena
| global.certManagementMode | string | `"auto"` | Certificate Management Mode.
Three mutually exclusive options for managing TLS certificates:
- `auto`: Webhook servers generate self-signed certificates automatically.
- `cert-manager`: Use cert-manager to generate and manage certificates (requires cert-manager installation).
- `manual`: Provide your own certificates via caBundle. |
| global.webhook.caBundle | string | `""` | CA bundle for webhook server certificates (base64-encoded).
This is ONLY required when `certManagementMode` is set to "manual".
You can generate it with: `cat /path/to/your/ca.crt | base64 | tr -d '\n'`
|
| networking.enabled | bool | `true` | Enable the networking subchart. |
+| networking.kthenaRouter.accessLog.enabled | bool | `true` | Enable router access logging. |
+| networking.kthenaRouter.accessLog.format | string | `"text"` | Access-log format. Supported values are `text` and `json`. |
+| networking.kthenaRouter.accessLog.output | string | `"stdout"` | Access-log destination: `stdout`, `stderr`, or a file path. |
| networking.kthenaRouter.debugPort | int | `15000` | Debug server port for Kthena Router (localhost only). |
| networking.kthenaRouter.drainTimeout | string | `"5m"` | This should be less than terminationGracePeriodSeconds. |
| networking.kthenaRouter.enabled | bool | `true` | Enable Kthena Router. |
diff --git a/docs/kthena/docs/reference/router-access-log-fields.md b/docs/kthena/docs/reference/router-access-log-fields.md
index ceae01c4c..7bbe7f10f 100644
--- a/docs/kthena/docs/reference/router-access-log-fields.md
+++ b/docs/kthena/docs/reference/router-access-log-fields.md
@@ -1,109 +1,114 @@
# Router Access Log Fields Reference
-This document provides a comprehensive reference for all fields available in Kthena Router access logs.
+Kthena emits one access-log entry after each `/v1/*` inference request.
+Requests routed through Gateway API on other paths currently do not emit an
+access-log entry. Each recorded entry contains the HTTP result, routing
+selection, token counts, timing checkpoints, and any router error captured
+during processing.
-## Overview
+## Formats and defaults
-Kthena Router generates structured access logs for all AI inference requests. These logs provide detailed information about request processing, including timing breakdowns, routing decisions, token usage, and error information.
+Access logs support two formats:
-## Log Format
+- `text` is the current Helm default and preserves existing installations.
+- `json` is recommended for log aggregation and field-based queries, but must
+ be enabled explicitly.
-Access logs are available in two formats:
-- **JSON**: Structured JSON format suitable for log aggregation and analysis
-- **Text**: Human-readable format for development and debugging
+The text representation is a single line with this shape:
-### Text Format Structure
-
-The text format follows this structure:
-```
-[timestamp] "METHOD /path PROTOCOL" status_code [error=type:message] model_name=name model_route=route model_server=server selected_pod=pod request_id=id tokens=input/output timings=total(req+upstream+resp)ms
+```text
+[timestamp] "METHOD /path PROTOCOL" status_code [error=type:message] [model_name=name] [model_route=route] [model_server=server] [selected_pod=pod] [request_id=id] [gateway=gateway] [http_route=route] [inference_pool=pool] [tokens=input/output] timings=totalms(request+upstream+response)
```
-Key features of the text format:
-- **Error placement**: Error information appears immediately after the status code when present
-- **Timing format**: Shows total time with breakdown in parentheses: `timings=2350ms(45+2180+5)`
-- **Compact representation**: All information on a single line for easy parsing
-
-## Field Reference
-
-### Standard HTTP Fields
+Fields in square brackets are omitted when no value is available. JSON uses
+the names in the tables below.
-These fields follow the Envoy access log format for compatibility with existing log processing tools.
+## HTTP fields
-| Field | Type | Description | Example |
-| ------------- | ------------------ | ------------------------------------------------ | -------------------------- |
-| `timestamp` | `string` (RFC3339) | ISO 8601 timestamp when the request was received | `2024-01-15T10:30:45.123Z` |
-| `method` | `string` | HTTP method used for the request | `POST`, `GET` |
-| `path` | `string` | Request path including query parameters | `/v1/chat/completions` |
-| `protocol` | `string` | HTTP protocol version | `HTTP/1.1`, `HTTP/2` |
-| `status_code` | `integer` | HTTP response status code | `200`, `400`, `500` |
+| Field | Type | Presence | Description | Example |
+| --- | --- | --- | --- | --- |
+| `timestamp` | RFC3339 timestamp | Always | Time the router received the request | `2026-01-09T14:35:22.147Z` |
+| `method` | String | Always | HTTP method | `POST` |
+| `path` | String | Always | URL path only; the query string is not included | `/v1/chat/completions` |
+| `protocol` | String | Always | HTTP protocol reported by Go | `HTTP/1.1` |
+| `status_code` | Integer | Always | Final HTTP response status | `200` |
-### Error Information
+## Error fields
-Error information is included when a request fails and appears immediately after the status code.
+The `error` object is omitted from JSON when the router did not record an
+error. In text logs it appears immediately after the status code.
-| Field | Type | Description | Example |
-| --------------- | -------- | ---------------------- | ------------------------------------------ |
-| `error.type` | `string` | Error category or type | `timeout`, `rate_limit`, `model_not_found` |
-| `error.message` | `string` | Detailed error message | `Model inference timeout after 30s` |
+| Field | Type | Description | Example |
+| --- | --- | --- | --- |
+| `error.type` | String | Stable error category assigned by the router | `pod_discovery` |
+| `error.message` | String | Diagnostic message for this failure | `no available pods for model server: default/llama2-server` |
-#### Common Error Types
+Current router error categories are:
-| Error Type | Description | Typical Status Code |
-| ----------------------- | ------------------------------------ | ------------------- |
-| `timeout` | Request exceeded configured timeout | `504` |
-| `rate_limit` | Request was rate limited | `429` |
-| `model_not_found` | Requested model is not available | `404` |
-| `authentication_failed` | Authentication credentials invalid | `401` |
-| `authorization_failed` | User lacks required permissions | `403` |
-| `upstream_error` | Error from model inference backend | `502`, `503` |
-| `invalid_request` | Malformed request body or parameters | `400` |
+| Error type | Failure area |
+| --- | --- |
+| `request_parsing` | Request body or required model field could not be parsed |
+| `prompt_parsing` | Prompt extraction failed |
+| `input_rate_limit` | Input-token limit was exceeded |
+| `output_rate_limit` | Output-token limit was exceeded |
+| `rate_limit` | General request/token rate limit was exceeded |
+| `model_server_matching` | No matching ModelServer could be selected |
+| `pod_discovery` | Backend pod lookup failed or returned no available pods |
+| `inference_pool_selection` | A matched HTTPRoute had no eligible InferencePool backend |
+| `inference_pool_discovery` | The selected InferencePool could not be found |
+| `port_discovery` | An InferencePool had no usable target port |
+| `route_not_found` | Neither ModelRoute nor HTTPRoute routing matched |
+| `scheduling` | Queue admission or backend scheduling failed |
+| `proxy` | Forwarding to the selected backend failed |
-### AI-Specific Routing Information
+An HTTP status is not uniquely determined by `error.type`; consult
+`status_code` for the actual result.
-These fields provide information about how the request was routed through the AI router.
+## Model and routing fields
-| Field | Type | Description | Example |
-| -------------- | -------- | ----------------------------------------- | -------------------------------------- |
-| `model_name` | `string` | Name of the AI model requested | `llama2-7b`, `gpt-3.5-turbo` |
-| `model_route` | `string` | Name of the ModelRoute resource used | `default/llama2-route-v1` |
-| `model_server` | `string` | ModelServer that handled the request | `default/llama2-server` |
-| `selected_pod` | `string` | Specific pod that processed the inference | `llama2-deployment-5f7b8c9d-xk2p4` |
-| `request_id` | `string` | Unique identifier for request tracing | `550e8400-e29b-41d4-a716-446655440000` |
+| Field | Type | Presence | Description | Example |
+| --- | --- | --- | --- | --- |
+| `model_name` | String | Always in JSON; may be empty when parsing fails | Model requested by the client | `llama2-7b` |
+| `model_route` | String | When selected | Namespaced ModelRoute | `default/llama2-route-v1` |
+| `model_server` | String | When selected | Namespaced ModelServer | `default/llama2-server` |
+| `selected_pod` | String | When selected | Backend pod name | `llama2-deployment-5f7b8c9d-xk2p4` |
+| `request_id` | String | Normally present | Incoming `x-request-id`, or an ID generated by the access-log middleware | `550e8400-e29b-41d4-a716-446655440000` |
+| `gateway` | String | Gateway API requests | Namespaced Gateway | `default/inference-gateway` |
+| `http_route` | String | HTTPRoute requests | Namespaced HTTPRoute | `default/inference-route` |
+| `inference_pool` | String | Inference Extension requests | Namespaced InferencePool | `default/llama2-pool` |
-### Token Information
+## Token fields
-Token usage metrics for the inference request.
+| Field | Type | Presence | Description | Example |
+| --- | --- | --- | --- | --- |
+| `input_tokens` | Integer | JSON omits zero | Prompt-token count calculated by the router | `150` |
+| `output_tokens` | Integer | JSON omits zero | Completion-token count parsed from the backend response when available | `75` |
-| Field | Type | Description | Example |
-| --------------- | --------- | -------------------------------------- | ------- |
-| `input_tokens` | `integer` | Number of tokens in the request prompt | `150` |
-| `output_tokens` | `integer` | Number of tokens generated in response | `75` |
+Text output includes `tokens=input/output` when either count is nonzero.
-### Timing Breakdown
+## Timing fields
-All timing values are in milliseconds and provide detailed performance metrics.
+All timing values are integer milliseconds.
-| Field | Type | Description | Example |
-| ------------------------------ | --------- | ----------------------------------------------- | ------- |
-| `duration_total` | `integer` | Total end-to-end request processing time (ms) | `2350` |
-| `duration_request_processing` | `integer` | Router request processing overhead (ms) | `45` |
-| `duration_upstream_processing` | `integer` | Model inference time on backend pod (ms) | `2180` |
-| `duration_response_processing` | `integer` | Response processing and serialization time (ms) | `5` |
+| Field | Type | Description | Example |
+| --- | --- | --- | --- |
+| `duration_total` | Integer | End-to-end time from receipt until access-log finalization | `2350` |
+| `duration_request_processing` | Integer | Initial router parsing and tokenization phase | `45` |
+| `duration_upstream_processing` | Integer | Marked backend-proxy phase | `2180` |
+| `duration_response_processing` | Integer | Marked response-processing phase | `5` |
-#### Timing Phases
+There is no `duration_queue` field and no access-log field uses an `_ms`
+suffix. `duration_total` is authoritative. The phase fields are independently
+rounded checkpoints and may not sum to the total, including when rate limiting,
+routing, scheduling, or queue time falls between marked phases.
-1. **Request Processing**: Time spent parsing the request, authentication, rate limiting, and routing decisions
-2. **Upstream Processing**: Time spent on actual model inference in the backend pod
-3. **Response Processing**: Time spent formatting and serializing the response
+## Examples
-## Example Access Logs
-
-### Successful Request (JSON Format)
+### Successful JSON entry
```json
{
- "timestamp": "2024-01-15T10:30:45.123Z",
+ "timestamp": "2026-01-09T14:35:22.147Z",
"method": "POST",
"path": "/v1/chat/completions",
"protocol": "HTTP/1.1",
@@ -122,53 +127,70 @@ All timing values are in milliseconds and provide detailed performance metrics.
}
```
-### Failed Request with Error (JSON Format)
+### Failed JSON entry
```json
{
- "timestamp": "2024-01-15T10:35:22.456Z",
+ "timestamp": "2026-01-09T14:35:28.456Z",
"method": "POST",
"path": "/v1/chat/completions",
"protocol": "HTTP/1.1",
- "status_code": 504,
+ "status_code": 503,
"error": {
- "type": "timeout",
- "message": "Model inference timeout after 30s"
+ "type": "pod_discovery",
+ "message": "no available pods for model server: default/llama2-server"
},
"model_name": "llama2-7b",
- "model_route": "default/llama2-route-v1",
- "model_server": "default/llama2-server",
- "selected_pod": "llama2-deployment-5f7b8c9d-xk2p4",
"request_id": "660e8400-e29b-41d4-a716-446655440001",
"input_tokens": 200,
- "output_tokens": 0,
- "duration_total": 30050,
- "duration_request_processing": 50,
- "duration_upstream_processing": 30000,
+ "duration_total": 51,
+ "duration_request_processing": 4,
+ "duration_upstream_processing": 47,
"duration_response_processing": 0
}
```
-### Text Format Example
+### Text entry
+```text
+[2026-01-09T14:35:22.147Z] "POST /v1/chat/completions HTTP/1.1" 200 model_name=llama2-7b model_route=default/llama2-route-v1 model_server=default/llama2-server selected_pod=llama2-deployment-5f7b8c9d-xk2p4 request_id=550e8400-e29b-41d4-a716-446655440000 tokens=150/75 timings=2350ms(45+2180+5)
```
-[2024-01-15T10:30:45.123Z] "POST /v1/chat/completions HTTP/1.1" 200 model_name=llama2-7b model_route=default/llama2-route-v1 model_server=default/llama2-server selected_pod=llama2-deployment-5f7b8c9d-xk2p4 request_id=550e8400-e29b-41d4-a716-446655440000 tokens=150/75 timings=2350ms(45+2180+5)
-```
-### Text Format with Error
+## Helm configuration
+
+The root chart uses the following values. The defaults shown here preserve the
+current text-format behavior:
+```yaml
+networking:
+ kthenaRouter:
+ accessLog:
+ enabled: true
+ format: text
+ output: stdout
```
-[2024-01-15T10:35:22.456Z] "POST /v1/chat/completions HTTP/1.1" 504 error=timeout:Model inference timeout after 30s model_name=llama2-7b model_route=default/llama2-route-v1 model_server=default/llama2-server selected_pod=llama2-deployment-5f7b8c9d-xk2p4 request_id=660e8400-e29b-41d4-a716-446655440001 tokens=200/0 timings=30050ms(50+30000+0)
+
+Set `format: json` explicitly when a structured log pipeline requires JSON.
+
+For the standalone networking subchart, omit the `networking` prefix:
+
+```yaml
+kthenaRouter:
+ accessLog:
+ enabled: true
+ format: json
+ output: stdout
```
-## Configuration
+## Environment variables
-Access logging is configured through environment variables in the kthena router deployment:
+The deployment maps Helm values to these router environment variables:
-### Environment Variables
+| Variable | Description | Helm default | Accepted values |
+| --- | --- | --- | --- |
+| `ACCESS_LOG_ENABLED` | Enable access logging | `true` | Values accepted by Go's boolean parser |
+| `ACCESS_LOG_FORMAT` | Output format | `text` | `text`, `json` |
+| `ACCESS_LOG_OUTPUT` | Destination | `stdout` | `stdout`, `stderr`, or a file path |
-| Variable | Description | Default | Valid Values |
-| -------------------- | -------------------------------- | -------- | -------------------------------- |
-| `ACCESS_LOG_ENABLED` | Enable or disable access logging | `true` | `true`, `false` |
-| `ACCESS_LOG_FORMAT` | Log output format | `text` | `json`, `text` |
-| `ACCESS_LOG_OUTPUT` | Where to write logs | `stdout` | `stdout`, `stderr`, or file path |
+Prefer `stdout` for Kubernetes workloads. A file path writes inside the router
+container and requires a mounted volume plus a separate collection strategy.
diff --git a/docs/kthena/docs/user-guide/kvcache-aware.md b/docs/kthena/docs/user-guide/kvcache-aware.md
index 3409c91b8..2fb942bbb 100644
--- a/docs/kthena/docs/user-guide/kvcache-aware.md
+++ b/docs/kthena/docs/user-guide/kvcache-aware.md
@@ -363,7 +363,7 @@ The output shows pod identifiers (e.g., `pod-name.namespace`) as field names and
The router exposes scheduler plugin metrics at the `/metrics` endpoint. You can check for score plugin activity:
```bash
-kubectl port-forward svc/kthena-router 8080:8080 -n
+kubectl port-forward service/kthena-router 8080:80 -n
curl -s http://localhost:8080/metrics | grep -i kvcache
```
diff --git a/docs/kthena/docs/user-guide/router-observability.md b/docs/kthena/docs/user-guide/router-observability.md
index 6836ed83f..1577cdc74 100644
--- a/docs/kthena/docs/user-guide/router-observability.md
+++ b/docs/kthena/docs/user-guide/router-observability.md
@@ -1,73 +1,147 @@
# Router Observability
-## Overview & Purpose
+The Kthena router exposes three complementary observability surfaces:
-The **kthena-router** serves as the central data-plane gateway for all inference traffic in the Kthena LLM inference platform.
-It is responsible for request routing, load balancing, scheduling, fairness queuing, rate limiting, token accounting, and (when applicable) disaggregated prefill/decode forwarding.
+- Prometheus metrics for alerting, dashboards, and capacity analysis
+- One access-log record for each inference request
+- Loopback-only configuration-dump and pprof endpoints for live diagnosis
-Without strong observability, diagnosing issues such as:
+## Endpoint map
-- Why is this model slow?
-- Which users are being unfairly delayed?
-- Where are the 5xx errors coming from?
-- Is the scheduler making good pod selections?
-- Are we hitting rate limits or resource exhaustion?
+The current Helm chart serves health and metrics routes on the inference
+listener. The debug listener is a separate loopback-only server inside the
+router pod.
-becomes extremely difficult and time-consuming.
+| Listener | Default | Endpoints | Exposure |
+| --- | --- | --- | --- |
+| Router | Container port `8080`; Service port `80` | `/healthz`, `/readyz`, `/metrics`, inference APIs | The `kthena-router` LoadBalancer Service |
+| Debug | `localhost:15000` | `/debug/config_dump/*`, `/debug/pprof/*` | Pod loopback only; no Service port |
-This observability framework provides production-grade visibility through three main channels:
+The chart does not currently provide an `observability.metrics` values block.
+The metrics path is fixed at `/metrics`, and its port follows
+`networking.kthenaRouter.port`. Do not set the previously documented
+`observability.metrics` keys; Helm ignores them.
-1. **Prometheus metrics** — quantitative signals for dashboards, alerting, and trending
-2. **Structured access logs** — rich per-request forensic details
-3. **Debug endpoints** — instant insight into routing configuration and system state
+To inspect these endpoints without relying on their external exposure:
-Together they enable fast root-cause analysis, performance tuning, capacity planning, cost monitoring, and abuse detection.
-
-## Metrics
-
-### Endpoint
-
-- **Metrics Port**: `8080` (default)
-- **Metrics Path**: `/metrics`
+```bash
+# The Service listens on 80 and forwards to the router's default port, 8080.
+kubectl port-forward -n kthena-system service/kthena-router 8080:80
-**Note:** The Prometheus metrics are exposed on port **8080** by default.
-The debug endpoints (`/debug/config_dump/*`) are served on port **15000**.
+# Run separately when debug access is needed. This selects a router pod and
+# reaches the process's loopback-only listener from inside its network namespace.
+kubectl port-forward -n kthena-system deployment/kthena-router 15000:15000
+```
-### Core Request & Latency Metrics
+If the release namespace is not `kthena-system`, replace it in the commands.
-| Metric Name | Type | Description | Labels | Buckets |
-|------------------------------------------------------|-----------|--------------------------------------------------------------|---------------------------------------------|-------------------------------------------------------------------------|
-| `kthena_router_requests_total` | Counter | Total requests processed | `model`, `path`, `status_code`, `error_type` | — |
-| `kthena_router_request_duration_seconds` | Histogram | End-to-end latency (client → response) | `model`, `path`, `status_code` | 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10, 30, 60 |
-| `kthena_router_request_prefill_duration_seconds` | Histogram | Prefill (prompt processing) phase duration | `model`, `path`, `status_code` | same as above |
-| `kthena_router_request_decode_duration_seconds` | Histogram | Decode (token generation) phase duration | `model`, `path`, `status_code` | same as above |
-| `kthena_router_active_requests` | Gauge | Currently active requests handled by the router | — | — |
-| `kthena_router_active_downstream_requests` | Gauge | Currently active client requests | `model` | — |
-| `kthena_router_active_upstream_requests` | Gauge | Currently active requests to inference pods | `model_route`, `model_server` | — |
+## Metrics
-### Token & Usage Metrics
+The tables below list every Kthena-owned router metric family registered by the
+current binary. Histograms additionally expose Prometheus `_bucket`, `_sum`, and
+`_count` series. Standard Go runtime, process, and Prometheus handler metrics are
+also present; those collector-provided families can vary with dependency
+versions.
+
+Some metrics appear only after the corresponding feature or request path has
+been exercised.
+
+### Requests, traffic, and rate limiting
+
+| Metric | Type | Labels | Description |
+| --- | --- | --- | --- |
+| `kthena_router_requests_total` | Counter | `model`, `path`, `status_code`, `error_type` | Completed HTTP requests |
+| `kthena_router_request_duration_seconds` | Histogram | `model`, `path`, `status_code` | End-to-end request latency |
+| `kthena_router_request_prefill_duration_seconds` | Histogram | `model`, `path`, `status_code` | Prefill latency for prefill/decode-disaggregated requests |
+| `kthena_router_request_decode_duration_seconds` | Histogram | `model`, `path`, `status_code` | Decode latency for prefill/decode-disaggregated requests |
+| `kthena_router_tokens_total` | Counter | `model`, `path`, `token_type` | Input or output tokens; `token_type` is `input` or `output` |
+| `kthena_router_rate_limit_exceeded_total` | Counter | `model`, `limit_type`, `path` | Requests rejected by input-token, output-token, or request rate limits |
+| `kthena_router_active_requests` | Gauge | none | All requests currently handled by this router process |
+| `kthena_router_active_downstream_requests` | Gauge | `model` | Active client-to-router requests |
+| `kthena_router_active_upstream_requests` | Gauge | `model_server`, `model_route` | Active router-to-backend requests |
+
+### Scheduler and user-fairness queue
+
+| Metric | Type | Labels | Description |
+| --- | --- | --- | --- |
+| `kthena_router_scheduler_plugin_duration_seconds` | Histogram | `model`, `plugin`, `type` | Scheduler plugin execution time; `type` is `filter` or `score` |
+| `kthena_router_fairness_queue_size` | Gauge | `model`, `user_id` | Pending requests in the fairness queue |
+| `kthena_router_fairness_queue_duration_seconds` | Histogram | `model`, `user_id` | Time spent waiting in the fairness queue |
+| `kthena_router_fairness_queue_cancelled_total` | Counter | `model`, `user_id` | Requests cancelled or timed out while queued |
+| `kthena_router_fairness_queue_dequeue_total` | Counter | `model`, `user_id` | Requests successfully dequeued |
+| `kthena_router_fairness_queue_inflight` | Gauge | `model` | Requests admitted through the fairness semaphore |
+| `kthena_router_fairness_queue_priority_refresh_total` | Counter | `model` | Dequeue-time priority refresh and reinsert operations |
+| `kthena_router_fairness_queue_heap_rebuild_total` | Counter | `model` | Full heap rebuilds caused by priority drift |
+
+`user_id` values originate from authenticated request identity. Treat them as
+sensitive, and account for their cardinality when retaining or federating these
+series.
+
+### Tokenizer and cache-aware scheduling
+
+| Metric | Type | Labels | Description |
+| --- | --- | --- | --- |
+| `kthena_router_tokenizer_unsupported_engine_total` | Counter | `model`, `engine` | Tokenizer lookups for which no pod used a supported engine |
+| `kthena_router_prefix_cache_match_ratio` | Histogram | `model` | Best prefix-cache match ratio for each scheduling attempt |
+| `kthena_router_prefix_cache_evictions_total` | Counter | `model` | Per-pod prefix-cache entries evicted at capacity |
+| `kthena_router_prefix_cache_entries` | Gauge | none | Current `(prefix block, pod)` entries across all local caches |
+| `kthena_router_kvcache_aware_match_ratio` | Histogram | `model` | Best external KV-cache match ratio for each attempt |
+| `kthena_router_kvcache_aware_redis_duration_seconds` | Histogram | `model` | Batched Redis lookup latency |
+| `kthena_router_kvcache_aware_tokenize_duration_seconds` | Histogram | `model` | Prompt tokenization latency for KV-cache-aware matching |
+| `kthena_router_kvcache_aware_errors_total` | Counter | `model`, `stage` | Aborted KV-cache-aware attempts; `stage` is `tokenize` or `redis` |
+
+### Session-boost queue
+
+| Metric | Type | Labels | Description |
+| --- | --- | --- | --- |
+| `kthena_router_session_boost_queue_size` | Gauge | `model` | Pending requests in the session-boost queue |
+| `kthena_router_session_boost_queue_duration_seconds` | Histogram | `model` | Time spent waiting in the session-boost queue |
+| `kthena_router_session_boost_queue_cancelled_total` | Counter | `model` | Requests cancelled or timed out while queued |
+| `kthena_router_session_boost_queue_dequeue_total` | Counter | `model` | Requests successfully dequeued |
+| `kthena_router_session_boost_queue_inflight` | Gauge | `model` | Requests admitted through the session-boost queue |
+
+### Histogram buckets
+
+| Metrics | Buckets |
+| --- | --- |
+| Request, prefill, and decode durations | `0.005`, `0.01`, `0.025`, `0.05`, `0.1`, `0.25`, `0.5`, `1`, `2.5`, `5`, `10`, `30`, `60` seconds |
+| Scheduler plugin duration | `0.001`, `0.005`, `0.01`, `0.05`, `0.1`, `0.5` seconds |
+| Fairness and session-boost queue durations | `0.001`, `0.005`, `0.01`, `0.025`, `0.05`, `0.1`, `0.25`, `0.5`, `1`, `2.5`, `5` seconds |
+| Prefix-cache and KV-cache match ratios | `0`, `0.1`, `0.25`, `0.5`, `0.75`, `0.9`, `0.95`, `0.99`, `1.0` |
+| KV-cache Redis and tokenization durations | `0.0005`, `0.001`, `0.0025`, `0.005`, `0.01`, `0.025`, `0.05`, `0.1`, `0.25`, `0.5`, `1`, `2.5` seconds |
-| Metric Name | Type | Description | Labels |
-|----------------------------------------|---------|--------------------------------------------------|-------------------------------------|
-| `kthena_router_tokens_total` | Counter | Total tokens processed (input + output) | `model`, `path`, `token_type` (input/output) |
+## Access logs
-### Scheduler & Fairness Metrics
+Helm enables access logging in `text` format by default. This is the
+compatibility default and is not changed by this documentation update. JSON is
+recommended for structured ingestion, but operators must opt in explicitly.
-| Metric Name | Type | Description | Labels | Buckets |
-|-------------------------------------------------------|-----------|--------------------------------------------------------|-------------------------------|------------------------------------------------------------------------|
-| `kthena_router_scheduler_plugin_duration_seconds` | Histogram | Execution time per scheduler plugin | `model`, `plugin`, `type` | 0.001, 0.005, 0.01, 0.05, 0.1, 0.5 |
-| `kthena_router_fairness_queue_size` | Gauge | Current queued requests per model/user | `model`, `user_id` | — |
-| `kthena_router_fairness_queue_duration_seconds` | Histogram | Time spent waiting in fairness/priority queue | `model`, `user_id` | 0.001, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5 |
+```yaml
+networking:
+ kthenaRouter:
+ accessLog:
+ enabled: true
+ format: json
+ output: stdout
+```
-### Rate Limiting & Protection
+For a standalone deployment, the equivalent environment variables are:
-| Metric Name | Type | Description | Labels |
-|--------------------------------------------------|---------|------------------------------------------------------|-------------------------------|
-| `kthena_router_rate_limit_exceeded_total` | Counter | Requests rejected due to rate limiting | `model`, `limit_type`, `path` |
+```yaml
+env:
+ - name: ACCESS_LOG_ENABLED
+ value: "true"
+ - name: ACCESS_LOG_FORMAT
+ value: "json"
+ - name: ACCESS_LOG_OUTPUT
+ value: "stdout"
+```
-## Access Logs
+The valid formats are `text` and `json`. Output can be `stdout`, `stderr`, or a
+file path. Prefer `stdout` in Kubernetes so the container runtime and log agent
+can collect the records; a file path is local to the container filesystem.
-### Recommended Format: Structured JSON
+The JSON field names below match the emitted contract:
```json
{
@@ -83,212 +157,93 @@ The debug endpoints (`/debug/config_dump/*`) are served on port **15000**.
"request_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
"input_tokens": 412,
"output_tokens": 189,
- "duration_total_ms": 3840,
- "duration_queue_ms": 180,
- "duration_request_processing_ms": 65,
- "duration_upstream_ms": 3480,
- "duration_response_processing_ms": 115,
- "error": null
+ "duration_total": 3840,
+ "duration_request_processing": 65,
+ "duration_upstream_processing": 3480,
+ "duration_response_processing": 115
}
```
-## Configuration
+All duration fields are integer milliseconds. There is no `duration_queue`
+field. `duration_total` is authoritative and can exceed the sum of the marked
+processing phases, including when routing or queue time falls between phase
+checkpoints. See the [access-log field reference](../reference/router-access-log-fields.md)
+for optional routing fields and current error types.
-Observability features are configured via the Kthena Router's deployment or ConfigMap.
-Most settings are controlled through environment variables or the router's configuration file (depending on your deployment method).
-
-### Access Log Configuration
-
-```yaml
-accessLogger:
- enabled: true
- format: "json" # "json" (strongly recommended) or "text"
- output: "stdout" # "stdout", "stderr", or file path
-```
-
-#### Equivalent environment variables (recommended for most deployments)
-
-```yaml
-env:
-- name: ACCESS_LOG_ENABLED
- value: "true"
-- name: ACCESS_LOG_FORMAT
- value: "json"
-- name: ACCESS_LOG_OUTPUT
- value: "stdout"
-```
-
-#### Metrics Configuration
-
-```yaml
-observability:
- metrics:
- enabled: true
- port: 8080 # Default metrics port
- path: /metrics
-```
-
-## Debug Endpoints
-
-All available on the same `:15000` port
-
-| Endpoint | Description |
-| --- | --- |
-| `/debug/config_dump/modelroutes` | All ModelRoute resources |
-| `/debug/config_dump/modelservers` | All ModelServer resources |
-| `/debug/config_dump/pods` | Current view of healthy/ready inference pods |
-| `/debug/config_dump/namespaces/{ns}/modelroutes/{name}` | Detailed single ModelRoute |
-| `/debug/config_dump/namespaces/{ns}/modelservers/{name}` | Detailed single ModelServer |
-
-## Quick Start – Observability in Action
+When `ACCESS_LOG_OUTPUT` is `stdout` or `stderr`, `kubectl logs` can contain
+access-log records alongside other process logs. The following examples
+therefore filter JSON lines before invoking `jq`. If access logs are written to
+a file path, they do not appear in `kubectl logs`; inspect the mounted file or
+the configured log collector instead.
```bash
-# Forward metrics port (8080)
-kubectl port-forward -n kthena-system svc/kthena-router 8080:8080 &
-
-# Forward debug port (15000) when needed
-kubectl port-forward -n kthena-system svc/kthena-router 15000:15000 &
-
-# Watch real-time request rate by model
-watch -n 2 'curl -s http://localhost:8080/metrics | grep kthena_router_requests_total | sort'
-
-# Tail logs and filter access log entries (JSON lines only)
-kubectl logs -n kthena-system deployment/kthena-router -f \
- | grep -E '^{.*}$' | jq . # Only process valid JSON lines
-
-# Alternative: look for model-related entries in all logs
kubectl logs -n kthena-system deployment/kthena-router -f \
- | grep -E "model_name|request_id|duration_total"
+ | grep -E '^\{.*\}$' \
+ | jq .
```
-**Important note about logs**
-
-- Router logs usually contain both regular application logs (plain text) and structured access logs (JSON).
-- Piping everything directly to `jq` will cause errors on non-JSON lines.
-- Use `grep` to filter JSON lines first, or use a log processor (like fluentd, vector, or loki) in production.
+## Health endpoints
-## Troubleshooting Guide
+| Endpoint | Successful response | Purpose |
+| --- | --- | --- |
+| `/healthz` | HTTP `200`, `{"message":"ok"}` | Process liveness |
+| `/readyz` | HTTP `200`, `{"message":"router is ready"}` | Controller and datastore readiness; returns HTTP `503` until ready |
-### Preparation
+## Debug and pprof endpoints
-```bash
-# Forward metrics port (8080) for metrics queries
-kubectl port-forward -n kthena-system svc/kthena-router 8080:8080 &
+The debug server binds to `localhost:15000` by default. It is intentionally not
+published by the Helm Service. A port-forward grants access to sensitive routing
+state and runtime profiles; keep it open only while diagnosing a trusted cluster.
-# Forward debug port (15000) for debug endpoints
-kubectl port-forward -n kthena-system svc/kthena-router 15000:15000 &
+### Configuration dump
-# Live structured logs (recommended, with filtering)
-kubectl logs -n kthena-system deployment/kthena-router -f \
- | grep -E '^{.*}$' | jq .
-```
-
-#### 1. High Error Rate (5xx, timeouts, internal server errors)
+| Endpoint | Description |
+| --- | --- |
+| `/debug/config_dump/modelroutes` | All ModelRoute resources known to the router |
+| `/debug/config_dump/modelservers` | All ModelServer resources known to the router |
+| `/debug/config_dump/pods` | All inference pods known to the router |
+| `/debug/config_dump/gateways` | All Gateway resources known to the router |
+| `/debug/config_dump/httproutes` | All HTTPRoute resources known to the router |
+| `/debug/config_dump/inferencepools` | All InferencePool resources known to the router |
+| `/debug/config_dump/namespaces/{namespace}/modelroutes/{name}` | One namespaced ModelRoute |
+| `/debug/config_dump/namespaces/{namespace}/modelservers/{name}` | One namespaced ModelServer |
+| `/debug/config_dump/namespaces/{namespace}/pods/{name}` | One namespaced pod |
+| `/debug/config_dump/namespaces/{namespace}/gateways/{name}` | One namespaced Gateway |
+| `/debug/config_dump/namespaces/{namespace}/httproutes/{name}` | One namespaced HTTPRoute |
+| `/debug/config_dump/namespaces/{namespace}/inferencepools/{name}` | One namespaced InferencePool |
+
+### Runtime profiling
-Count errors by status & model:
+| Endpoint | Description |
+| --- | --- |
+| `/debug/pprof/` | pprof index |
+| `/debug/pprof/profile` | CPU profile |
+| `/debug/pprof/goroutine` | Goroutine profile |
+| `/debug/pprof/heap` | Heap profile |
+| `/debug/pprof/allocs` | Allocation profile |
+| `/debug/pprof/block` | Blocking profile |
+| `/debug/pprof/mutex` | Mutex-contention profile |
-```bash
-curl -s http://localhost:8080/metrics | grep 'status_code=5' | sort -nr
-```
+## Troubleshooting examples
-Top affected models:
+After starting both port-forwards from the [endpoint map](#endpoint-map):
```bash
+# Request counters by model and result.
curl -s http://localhost:8080/metrics \
- | grep 'status_code=5' \
- | grep -o 'model="[^"]*"' | sort | uniq -c | sort -nr
-```
+ | grep '^kthena_router_requests_total'
-Inspect recent failed requests:
+# Router configuration and currently known pods.
+curl -s http://localhost:15000/debug/config_dump/modelservers | jq .
+curl -s http://localhost:15000/debug/config_dump/pods | jq .
-```bash
+# Recent 5xx access records (requires JSON access-log format).
kubectl logs -n kthena-system deployment/kthena-router --since=30m \
- | jq 'select(.status_code >= 500) | {ts: .timestamp, model: .model_name, err: .error, pod: .selected_pod, dur: .duration_total_ms}'
-```
+ | grep -E '^\{.*\}$' \
+ | jq 'select(.status_code >= 500) | {timestamp, model: .model_name, error, pod: .selected_pod, duration: .duration_total}'
-Check upstream health:
-
-```bash
-curl http://localhost:15000/debug/config_dump/modelservers | jq .
-curl http://localhost:15000/debug/config_dump/pods | jq .
-```
-
-#### 2. High Latency / Slow TTFT or Generation Speed
-
-Latency percentiles (p50/p95/p99):
-
-```bash
-curl -s http://localhost:8080/metrics \
- | grep -E 'kthena_router_request_duration_seconds_(bucket|sum|count)'
-```
-
-Find slowest requests:
-
-```bash
+# Requests slower than four seconds (requires JSON access-log format).
kubectl logs -n kthena-system deployment/kthena-router --since=20m \
- | jq 'select(.duration_total_ms > 4000) | {model: .model_name, total: .duration_total_ms, upstream: .duration_upstream_ms, pod: .selected_pod}'
-```
-
-Check queue pressure:
-
-```bash
-watch -n 2 'curl -s http://localhost:8080/metrics | grep -E "(active_downstream|fairness_queue_size)"'
-```
-
-#### 3. Queue Buildup / Fairness / Throttling
-
-Live queue monitoring:
-
-```bash
-watch -n 3 'curl -s http://localhost:8080/metrics | grep fairness_queue_size'
-```
-
-Queue wait time distribution:
-
-```bash
-curl -s http://localhost:8080/metrics | grep fairness_queue_duration_seconds
-```
-
-Find throttled/rejected requests:
-
-```bash
-kubectl logs -n kthena-system deployment/kthena-router --since=1h \
- | jq 'select(.error? | .type? == ("rate_limit","throttled","queue_full"))'
-```
-
-#### 4. Wrong Routing / 404 / Pod Selection Issues
-
-Validate full routing table:
-
-```bash
-curl http://localhost:15000/debug/config_dump/modelroutes | jq .
-```
-
-Check pod readiness:
-
-```bash
-curl http://localhost:15000/debug/config_dump/pods | jq .
-```
-
-Trace a specific request:
-
-```bash
-# Use request_id from client or error message
-kubectl logs -n kthena-system deployment/kthena-router \
- | jq 'select(.request_id == "a1b2c3d4-...")'
-```
-
-#### 5. Token Usage / Cost / Abuse Monitoring
-
-Current token consumption rate:
-
-```bash
-curl -s http://localhost:8080/metrics | grep kthena_router_tokens_total
-```
-
-High-token requests:
-
-```bash
-kubectl logs -n kthena-system deployment/kthena-router --since=2h \
- | jq 'select(.input_tokens > 3000 or .output_tokens > 1500)'
+ | grep -E '^\{.*\}$' \
+ | jq 'select(.duration_total > 4000) | {model: .model_name, total: .duration_total, upstream: .duration_upstream_processing, pod: .selected_pod}'
```