diff --git a/charts/kthena/charts/networking/README.md b/charts/kthena/charts/networking/README.md index 30aa3cd63..7a5c25b98 100644 --- a/charts/kthena/charts/networking/README.md +++ b/charts/kthena/charts/networking/README.md @@ -20,6 +20,22 @@ kthenaRouter: pullPolicy: IfNotPresent ``` +#### Access Log Configuration + +Access logging is enabled in text format by default for compatibility. Select +JSON explicitly when a structured log collector expects JSON records. + +```yaml +kthenaRouter: + accessLog: + enabled: true + format: json + output: stdout +``` + +Prefer `stdout` in Kubernetes. A file path writes inside the router container +and requires a mounted volume and separate collection strategy. + #### Request Scheduling Configuration The router schedules requests through a per-model queue using one of two mutually @@ -65,6 +81,9 @@ kthenaRouter: | `kthenaRouter.sessionBoost.inflightPerPod` | int | `16` | Inflight requests per backend pod; total = perPod x pod count | | `kthenaRouter.sessionBoost.gracePeriod` | string | `"0s"` | Wait time for a same-session follow-up (disabled by default) | | `kthenaRouter.sessionBoost.timeout` | string | `"30s"` | Max queue wait before 504; set a non-positive duration (e.g. `0s`) to disable | +| `kthenaRouter.accessLog.enabled` | boolean | `true` | Enable request access logging | +| `kthenaRouter.accessLog.format` | string | `"text"` | Access-log format: `text` or `json` | +| `kthenaRouter.accessLog.output` | string | `"stdout"` | Access-log destination: `stdout`, `stderr`, or a file path | #### Session Boost Configuration diff --git a/charts/kthena/values.yaml b/charts/kthena/values.yaml index 56a368330..e09405684 100644 --- a/charts/kthena/values.yaml +++ b/charts/kthena/values.yaml @@ -54,6 +54,13 @@ networking: tag: latest # -- Image pull policy for Kthena Router. pullPolicy: IfNotPresent + accessLog: + # -- Enable router access logging. + enabled: true + # -- Access-log format. Supported values are `text` and `json`. + format: "text" + # -- Access-log destination: `stdout`, `stderr`, or a file path. + output: "stdout" tls: # -- Enable TLS for Kthena Router server. enabled: false diff --git a/docs/kthena/docs/reference/helm-chart-values.md b/docs/kthena/docs/reference/helm-chart-values.md index 39dc53cc5..9b2308616 100644 --- a/docs/kthena/docs/reference/helm-chart-values.md +++ b/docs/kthena/docs/reference/helm-chart-values.md @@ -18,6 +18,9 @@ A Helm chart for deploying Kthena | global.certManagementMode | string | `"auto"` | Certificate Management Mode.
Three mutually exclusive options for managing TLS certificates:
- `auto`: Webhook servers generate self-signed certificates automatically.
- `cert-manager`: Use cert-manager to generate and manage certificates (requires cert-manager installation).
- `manual`: Provide your own certificates via caBundle. | | global.webhook.caBundle | string | `""` | CA bundle for webhook server certificates (base64-encoded).
This is ONLY required when `certManagementMode` is set to "manual".
You can generate it with: `cat /path/to/your/ca.crt | base64 | tr -d '\n'`
| | networking.enabled | bool | `true` | Enable the networking subchart. | +| networking.kthenaRouter.accessLog.enabled | bool | `true` | Enable router access logging. | +| networking.kthenaRouter.accessLog.format | string | `"text"` | Access-log format. Supported values are `text` and `json`. | +| networking.kthenaRouter.accessLog.output | string | `"stdout"` | Access-log destination: `stdout`, `stderr`, or a file path. | | networking.kthenaRouter.debugPort | int | `15000` | Debug server port for Kthena Router (localhost only). | | networking.kthenaRouter.drainTimeout | string | `"5m"` | This should be less than terminationGracePeriodSeconds. | | networking.kthenaRouter.enabled | bool | `true` | Enable Kthena Router. | diff --git a/docs/kthena/docs/reference/router-access-log-fields.md b/docs/kthena/docs/reference/router-access-log-fields.md index ceae01c4c..7bbe7f10f 100644 --- a/docs/kthena/docs/reference/router-access-log-fields.md +++ b/docs/kthena/docs/reference/router-access-log-fields.md @@ -1,109 +1,114 @@ # Router Access Log Fields Reference -This document provides a comprehensive reference for all fields available in Kthena Router access logs. +Kthena emits one access-log entry after each `/v1/*` inference request. +Requests routed through Gateway API on other paths currently do not emit an +access-log entry. Each recorded entry contains the HTTP result, routing +selection, token counts, timing checkpoints, and any router error captured +during processing. -## Overview +## Formats and defaults -Kthena Router generates structured access logs for all AI inference requests. These logs provide detailed information about request processing, including timing breakdowns, routing decisions, token usage, and error information. +Access logs support two formats: -## Log Format +- `text` is the current Helm default and preserves existing installations. +- `json` is recommended for log aggregation and field-based queries, but must + be enabled explicitly. -Access logs are available in two formats: -- **JSON**: Structured JSON format suitable for log aggregation and analysis -- **Text**: Human-readable format for development and debugging +The text representation is a single line with this shape: -### Text Format Structure - -The text format follows this structure: -``` -[timestamp] "METHOD /path PROTOCOL" status_code [error=type:message] model_name=name model_route=route model_server=server selected_pod=pod request_id=id tokens=input/output timings=total(req+upstream+resp)ms +```text +[timestamp] "METHOD /path PROTOCOL" status_code [error=type:message] [model_name=name] [model_route=route] [model_server=server] [selected_pod=pod] [request_id=id] [gateway=gateway] [http_route=route] [inference_pool=pool] [tokens=input/output] timings=totalms(request+upstream+response) ``` -Key features of the text format: -- **Error placement**: Error information appears immediately after the status code when present -- **Timing format**: Shows total time with breakdown in parentheses: `timings=2350ms(45+2180+5)` -- **Compact representation**: All information on a single line for easy parsing - -## Field Reference - -### Standard HTTP Fields +Fields in square brackets are omitted when no value is available. JSON uses +the names in the tables below. -These fields follow the Envoy access log format for compatibility with existing log processing tools. +## HTTP fields -| Field | Type | Description | Example | -| ------------- | ------------------ | ------------------------------------------------ | -------------------------- | -| `timestamp` | `string` (RFC3339) | ISO 8601 timestamp when the request was received | `2024-01-15T10:30:45.123Z` | -| `method` | `string` | HTTP method used for the request | `POST`, `GET` | -| `path` | `string` | Request path including query parameters | `/v1/chat/completions` | -| `protocol` | `string` | HTTP protocol version | `HTTP/1.1`, `HTTP/2` | -| `status_code` | `integer` | HTTP response status code | `200`, `400`, `500` | +| Field | Type | Presence | Description | Example | +| --- | --- | --- | --- | --- | +| `timestamp` | RFC3339 timestamp | Always | Time the router received the request | `2026-01-09T14:35:22.147Z` | +| `method` | String | Always | HTTP method | `POST` | +| `path` | String | Always | URL path only; the query string is not included | `/v1/chat/completions` | +| `protocol` | String | Always | HTTP protocol reported by Go | `HTTP/1.1` | +| `status_code` | Integer | Always | Final HTTP response status | `200` | -### Error Information +## Error fields -Error information is included when a request fails and appears immediately after the status code. +The `error` object is omitted from JSON when the router did not record an +error. In text logs it appears immediately after the status code. -| Field | Type | Description | Example | -| --------------- | -------- | ---------------------- | ------------------------------------------ | -| `error.type` | `string` | Error category or type | `timeout`, `rate_limit`, `model_not_found` | -| `error.message` | `string` | Detailed error message | `Model inference timeout after 30s` | +| Field | Type | Description | Example | +| --- | --- | --- | --- | +| `error.type` | String | Stable error category assigned by the router | `pod_discovery` | +| `error.message` | String | Diagnostic message for this failure | `no available pods for model server: default/llama2-server` | -#### Common Error Types +Current router error categories are: -| Error Type | Description | Typical Status Code | -| ----------------------- | ------------------------------------ | ------------------- | -| `timeout` | Request exceeded configured timeout | `504` | -| `rate_limit` | Request was rate limited | `429` | -| `model_not_found` | Requested model is not available | `404` | -| `authentication_failed` | Authentication credentials invalid | `401` | -| `authorization_failed` | User lacks required permissions | `403` | -| `upstream_error` | Error from model inference backend | `502`, `503` | -| `invalid_request` | Malformed request body or parameters | `400` | +| Error type | Failure area | +| --- | --- | +| `request_parsing` | Request body or required model field could not be parsed | +| `prompt_parsing` | Prompt extraction failed | +| `input_rate_limit` | Input-token limit was exceeded | +| `output_rate_limit` | Output-token limit was exceeded | +| `rate_limit` | General request/token rate limit was exceeded | +| `model_server_matching` | No matching ModelServer could be selected | +| `pod_discovery` | Backend pod lookup failed or returned no available pods | +| `inference_pool_selection` | A matched HTTPRoute had no eligible InferencePool backend | +| `inference_pool_discovery` | The selected InferencePool could not be found | +| `port_discovery` | An InferencePool had no usable target port | +| `route_not_found` | Neither ModelRoute nor HTTPRoute routing matched | +| `scheduling` | Queue admission or backend scheduling failed | +| `proxy` | Forwarding to the selected backend failed | -### AI-Specific Routing Information +An HTTP status is not uniquely determined by `error.type`; consult +`status_code` for the actual result. -These fields provide information about how the request was routed through the AI router. +## Model and routing fields -| Field | Type | Description | Example | -| -------------- | -------- | ----------------------------------------- | -------------------------------------- | -| `model_name` | `string` | Name of the AI model requested | `llama2-7b`, `gpt-3.5-turbo` | -| `model_route` | `string` | Name of the ModelRoute resource used | `default/llama2-route-v1` | -| `model_server` | `string` | ModelServer that handled the request | `default/llama2-server` | -| `selected_pod` | `string` | Specific pod that processed the inference | `llama2-deployment-5f7b8c9d-xk2p4` | -| `request_id` | `string` | Unique identifier for request tracing | `550e8400-e29b-41d4-a716-446655440000` | +| Field | Type | Presence | Description | Example | +| --- | --- | --- | --- | --- | +| `model_name` | String | Always in JSON; may be empty when parsing fails | Model requested by the client | `llama2-7b` | +| `model_route` | String | When selected | Namespaced ModelRoute | `default/llama2-route-v1` | +| `model_server` | String | When selected | Namespaced ModelServer | `default/llama2-server` | +| `selected_pod` | String | When selected | Backend pod name | `llama2-deployment-5f7b8c9d-xk2p4` | +| `request_id` | String | Normally present | Incoming `x-request-id`, or an ID generated by the access-log middleware | `550e8400-e29b-41d4-a716-446655440000` | +| `gateway` | String | Gateway API requests | Namespaced Gateway | `default/inference-gateway` | +| `http_route` | String | HTTPRoute requests | Namespaced HTTPRoute | `default/inference-route` | +| `inference_pool` | String | Inference Extension requests | Namespaced InferencePool | `default/llama2-pool` | -### Token Information +## Token fields -Token usage metrics for the inference request. +| Field | Type | Presence | Description | Example | +| --- | --- | --- | --- | --- | +| `input_tokens` | Integer | JSON omits zero | Prompt-token count calculated by the router | `150` | +| `output_tokens` | Integer | JSON omits zero | Completion-token count parsed from the backend response when available | `75` | -| Field | Type | Description | Example | -| --------------- | --------- | -------------------------------------- | ------- | -| `input_tokens` | `integer` | Number of tokens in the request prompt | `150` | -| `output_tokens` | `integer` | Number of tokens generated in response | `75` | +Text output includes `tokens=input/output` when either count is nonzero. -### Timing Breakdown +## Timing fields -All timing values are in milliseconds and provide detailed performance metrics. +All timing values are integer milliseconds. -| Field | Type | Description | Example | -| ------------------------------ | --------- | ----------------------------------------------- | ------- | -| `duration_total` | `integer` | Total end-to-end request processing time (ms) | `2350` | -| `duration_request_processing` | `integer` | Router request processing overhead (ms) | `45` | -| `duration_upstream_processing` | `integer` | Model inference time on backend pod (ms) | `2180` | -| `duration_response_processing` | `integer` | Response processing and serialization time (ms) | `5` | +| Field | Type | Description | Example | +| --- | --- | --- | --- | +| `duration_total` | Integer | End-to-end time from receipt until access-log finalization | `2350` | +| `duration_request_processing` | Integer | Initial router parsing and tokenization phase | `45` | +| `duration_upstream_processing` | Integer | Marked backend-proxy phase | `2180` | +| `duration_response_processing` | Integer | Marked response-processing phase | `5` | -#### Timing Phases +There is no `duration_queue` field and no access-log field uses an `_ms` +suffix. `duration_total` is authoritative. The phase fields are independently +rounded checkpoints and may not sum to the total, including when rate limiting, +routing, scheduling, or queue time falls between marked phases. -1. **Request Processing**: Time spent parsing the request, authentication, rate limiting, and routing decisions -2. **Upstream Processing**: Time spent on actual model inference in the backend pod -3. **Response Processing**: Time spent formatting and serializing the response +## Examples -## Example Access Logs - -### Successful Request (JSON Format) +### Successful JSON entry ```json { - "timestamp": "2024-01-15T10:30:45.123Z", + "timestamp": "2026-01-09T14:35:22.147Z", "method": "POST", "path": "/v1/chat/completions", "protocol": "HTTP/1.1", @@ -122,53 +127,70 @@ All timing values are in milliseconds and provide detailed performance metrics. } ``` -### Failed Request with Error (JSON Format) +### Failed JSON entry ```json { - "timestamp": "2024-01-15T10:35:22.456Z", + "timestamp": "2026-01-09T14:35:28.456Z", "method": "POST", "path": "/v1/chat/completions", "protocol": "HTTP/1.1", - "status_code": 504, + "status_code": 503, "error": { - "type": "timeout", - "message": "Model inference timeout after 30s" + "type": "pod_discovery", + "message": "no available pods for model server: default/llama2-server" }, "model_name": "llama2-7b", - "model_route": "default/llama2-route-v1", - "model_server": "default/llama2-server", - "selected_pod": "llama2-deployment-5f7b8c9d-xk2p4", "request_id": "660e8400-e29b-41d4-a716-446655440001", "input_tokens": 200, - "output_tokens": 0, - "duration_total": 30050, - "duration_request_processing": 50, - "duration_upstream_processing": 30000, + "duration_total": 51, + "duration_request_processing": 4, + "duration_upstream_processing": 47, "duration_response_processing": 0 } ``` -### Text Format Example +### Text entry +```text +[2026-01-09T14:35:22.147Z] "POST /v1/chat/completions HTTP/1.1" 200 model_name=llama2-7b model_route=default/llama2-route-v1 model_server=default/llama2-server selected_pod=llama2-deployment-5f7b8c9d-xk2p4 request_id=550e8400-e29b-41d4-a716-446655440000 tokens=150/75 timings=2350ms(45+2180+5) ``` -[2024-01-15T10:30:45.123Z] "POST /v1/chat/completions HTTP/1.1" 200 model_name=llama2-7b model_route=default/llama2-route-v1 model_server=default/llama2-server selected_pod=llama2-deployment-5f7b8c9d-xk2p4 request_id=550e8400-e29b-41d4-a716-446655440000 tokens=150/75 timings=2350ms(45+2180+5) -``` -### Text Format with Error +## Helm configuration + +The root chart uses the following values. The defaults shown here preserve the +current text-format behavior: +```yaml +networking: + kthenaRouter: + accessLog: + enabled: true + format: text + output: stdout ``` -[2024-01-15T10:35:22.456Z] "POST /v1/chat/completions HTTP/1.1" 504 error=timeout:Model inference timeout after 30s model_name=llama2-7b model_route=default/llama2-route-v1 model_server=default/llama2-server selected_pod=llama2-deployment-5f7b8c9d-xk2p4 request_id=660e8400-e29b-41d4-a716-446655440001 tokens=200/0 timings=30050ms(50+30000+0) + +Set `format: json` explicitly when a structured log pipeline requires JSON. + +For the standalone networking subchart, omit the `networking` prefix: + +```yaml +kthenaRouter: + accessLog: + enabled: true + format: json + output: stdout ``` -## Configuration +## Environment variables -Access logging is configured through environment variables in the kthena router deployment: +The deployment maps Helm values to these router environment variables: -### Environment Variables +| Variable | Description | Helm default | Accepted values | +| --- | --- | --- | --- | +| `ACCESS_LOG_ENABLED` | Enable access logging | `true` | Values accepted by Go's boolean parser | +| `ACCESS_LOG_FORMAT` | Output format | `text` | `text`, `json` | +| `ACCESS_LOG_OUTPUT` | Destination | `stdout` | `stdout`, `stderr`, or a file path | -| Variable | Description | Default | Valid Values | -| -------------------- | -------------------------------- | -------- | -------------------------------- | -| `ACCESS_LOG_ENABLED` | Enable or disable access logging | `true` | `true`, `false` | -| `ACCESS_LOG_FORMAT` | Log output format | `text` | `json`, `text` | -| `ACCESS_LOG_OUTPUT` | Where to write logs | `stdout` | `stdout`, `stderr`, or file path | +Prefer `stdout` for Kubernetes workloads. A file path writes inside the router +container and requires a mounted volume plus a separate collection strategy. diff --git a/docs/kthena/docs/user-guide/kvcache-aware.md b/docs/kthena/docs/user-guide/kvcache-aware.md index 3409c91b8..2fb942bbb 100644 --- a/docs/kthena/docs/user-guide/kvcache-aware.md +++ b/docs/kthena/docs/user-guide/kvcache-aware.md @@ -363,7 +363,7 @@ The output shows pod identifiers (e.g., `pod-name.namespace`) as field names and The router exposes scheduler plugin metrics at the `/metrics` endpoint. You can check for score plugin activity: ```bash -kubectl port-forward svc/kthena-router 8080:8080 -n +kubectl port-forward service/kthena-router 8080:80 -n curl -s http://localhost:8080/metrics | grep -i kvcache ``` diff --git a/docs/kthena/docs/user-guide/router-observability.md b/docs/kthena/docs/user-guide/router-observability.md index 6836ed83f..1577cdc74 100644 --- a/docs/kthena/docs/user-guide/router-observability.md +++ b/docs/kthena/docs/user-guide/router-observability.md @@ -1,73 +1,147 @@ # Router Observability -## Overview & Purpose +The Kthena router exposes three complementary observability surfaces: -The **kthena-router** serves as the central data-plane gateway for all inference traffic in the Kthena LLM inference platform. -It is responsible for request routing, load balancing, scheduling, fairness queuing, rate limiting, token accounting, and (when applicable) disaggregated prefill/decode forwarding. +- Prometheus metrics for alerting, dashboards, and capacity analysis +- One access-log record for each inference request +- Loopback-only configuration-dump and pprof endpoints for live diagnosis -Without strong observability, diagnosing issues such as: +## Endpoint map -- Why is this model slow? -- Which users are being unfairly delayed? -- Where are the 5xx errors coming from? -- Is the scheduler making good pod selections? -- Are we hitting rate limits or resource exhaustion? +The current Helm chart serves health and metrics routes on the inference +listener. The debug listener is a separate loopback-only server inside the +router pod. -becomes extremely difficult and time-consuming. +| Listener | Default | Endpoints | Exposure | +| --- | --- | --- | --- | +| Router | Container port `8080`; Service port `80` | `/healthz`, `/readyz`, `/metrics`, inference APIs | The `kthena-router` LoadBalancer Service | +| Debug | `localhost:15000` | `/debug/config_dump/*`, `/debug/pprof/*` | Pod loopback only; no Service port | -This observability framework provides production-grade visibility through three main channels: +The chart does not currently provide an `observability.metrics` values block. +The metrics path is fixed at `/metrics`, and its port follows +`networking.kthenaRouter.port`. Do not set the previously documented +`observability.metrics` keys; Helm ignores them. -1. **Prometheus metrics** — quantitative signals for dashboards, alerting, and trending -2. **Structured access logs** — rich per-request forensic details -3. **Debug endpoints** — instant insight into routing configuration and system state +To inspect these endpoints without relying on their external exposure: -Together they enable fast root-cause analysis, performance tuning, capacity planning, cost monitoring, and abuse detection. - -## Metrics - -### Endpoint - -- **Metrics Port**: `8080` (default) -- **Metrics Path**: `/metrics` +```bash +# The Service listens on 80 and forwards to the router's default port, 8080. +kubectl port-forward -n kthena-system service/kthena-router 8080:80 -**Note:** The Prometheus metrics are exposed on port **8080** by default. -The debug endpoints (`/debug/config_dump/*`) are served on port **15000**. +# Run separately when debug access is needed. This selects a router pod and +# reaches the process's loopback-only listener from inside its network namespace. +kubectl port-forward -n kthena-system deployment/kthena-router 15000:15000 +``` -### Core Request & Latency Metrics +If the release namespace is not `kthena-system`, replace it in the commands. -| Metric Name | Type | Description | Labels | Buckets | -|------------------------------------------------------|-----------|--------------------------------------------------------------|---------------------------------------------|-------------------------------------------------------------------------| -| `kthena_router_requests_total` | Counter | Total requests processed | `model`, `path`, `status_code`, `error_type` | — | -| `kthena_router_request_duration_seconds` | Histogram | End-to-end latency (client → response) | `model`, `path`, `status_code` | 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10, 30, 60 | -| `kthena_router_request_prefill_duration_seconds` | Histogram | Prefill (prompt processing) phase duration | `model`, `path`, `status_code` | same as above | -| `kthena_router_request_decode_duration_seconds` | Histogram | Decode (token generation) phase duration | `model`, `path`, `status_code` | same as above | -| `kthena_router_active_requests` | Gauge | Currently active requests handled by the router | — | — | -| `kthena_router_active_downstream_requests` | Gauge | Currently active client requests | `model` | — | -| `kthena_router_active_upstream_requests` | Gauge | Currently active requests to inference pods | `model_route`, `model_server` | — | +## Metrics -### Token & Usage Metrics +The tables below list every Kthena-owned router metric family registered by the +current binary. Histograms additionally expose Prometheus `_bucket`, `_sum`, and +`_count` series. Standard Go runtime, process, and Prometheus handler metrics are +also present; those collector-provided families can vary with dependency +versions. + +Some metrics appear only after the corresponding feature or request path has +been exercised. + +### Requests, traffic, and rate limiting + +| Metric | Type | Labels | Description | +| --- | --- | --- | --- | +| `kthena_router_requests_total` | Counter | `model`, `path`, `status_code`, `error_type` | Completed HTTP requests | +| `kthena_router_request_duration_seconds` | Histogram | `model`, `path`, `status_code` | End-to-end request latency | +| `kthena_router_request_prefill_duration_seconds` | Histogram | `model`, `path`, `status_code` | Prefill latency for prefill/decode-disaggregated requests | +| `kthena_router_request_decode_duration_seconds` | Histogram | `model`, `path`, `status_code` | Decode latency for prefill/decode-disaggregated requests | +| `kthena_router_tokens_total` | Counter | `model`, `path`, `token_type` | Input or output tokens; `token_type` is `input` or `output` | +| `kthena_router_rate_limit_exceeded_total` | Counter | `model`, `limit_type`, `path` | Requests rejected by input-token, output-token, or request rate limits | +| `kthena_router_active_requests` | Gauge | none | All requests currently handled by this router process | +| `kthena_router_active_downstream_requests` | Gauge | `model` | Active client-to-router requests | +| `kthena_router_active_upstream_requests` | Gauge | `model_server`, `model_route` | Active router-to-backend requests | + +### Scheduler and user-fairness queue + +| Metric | Type | Labels | Description | +| --- | --- | --- | --- | +| `kthena_router_scheduler_plugin_duration_seconds` | Histogram | `model`, `plugin`, `type` | Scheduler plugin execution time; `type` is `filter` or `score` | +| `kthena_router_fairness_queue_size` | Gauge | `model`, `user_id` | Pending requests in the fairness queue | +| `kthena_router_fairness_queue_duration_seconds` | Histogram | `model`, `user_id` | Time spent waiting in the fairness queue | +| `kthena_router_fairness_queue_cancelled_total` | Counter | `model`, `user_id` | Requests cancelled or timed out while queued | +| `kthena_router_fairness_queue_dequeue_total` | Counter | `model`, `user_id` | Requests successfully dequeued | +| `kthena_router_fairness_queue_inflight` | Gauge | `model` | Requests admitted through the fairness semaphore | +| `kthena_router_fairness_queue_priority_refresh_total` | Counter | `model` | Dequeue-time priority refresh and reinsert operations | +| `kthena_router_fairness_queue_heap_rebuild_total` | Counter | `model` | Full heap rebuilds caused by priority drift | + +`user_id` values originate from authenticated request identity. Treat them as +sensitive, and account for their cardinality when retaining or federating these +series. + +### Tokenizer and cache-aware scheduling + +| Metric | Type | Labels | Description | +| --- | --- | --- | --- | +| `kthena_router_tokenizer_unsupported_engine_total` | Counter | `model`, `engine` | Tokenizer lookups for which no pod used a supported engine | +| `kthena_router_prefix_cache_match_ratio` | Histogram | `model` | Best prefix-cache match ratio for each scheduling attempt | +| `kthena_router_prefix_cache_evictions_total` | Counter | `model` | Per-pod prefix-cache entries evicted at capacity | +| `kthena_router_prefix_cache_entries` | Gauge | none | Current `(prefix block, pod)` entries across all local caches | +| `kthena_router_kvcache_aware_match_ratio` | Histogram | `model` | Best external KV-cache match ratio for each attempt | +| `kthena_router_kvcache_aware_redis_duration_seconds` | Histogram | `model` | Batched Redis lookup latency | +| `kthena_router_kvcache_aware_tokenize_duration_seconds` | Histogram | `model` | Prompt tokenization latency for KV-cache-aware matching | +| `kthena_router_kvcache_aware_errors_total` | Counter | `model`, `stage` | Aborted KV-cache-aware attempts; `stage` is `tokenize` or `redis` | + +### Session-boost queue + +| Metric | Type | Labels | Description | +| --- | --- | --- | --- | +| `kthena_router_session_boost_queue_size` | Gauge | `model` | Pending requests in the session-boost queue | +| `kthena_router_session_boost_queue_duration_seconds` | Histogram | `model` | Time spent waiting in the session-boost queue | +| `kthena_router_session_boost_queue_cancelled_total` | Counter | `model` | Requests cancelled or timed out while queued | +| `kthena_router_session_boost_queue_dequeue_total` | Counter | `model` | Requests successfully dequeued | +| `kthena_router_session_boost_queue_inflight` | Gauge | `model` | Requests admitted through the session-boost queue | + +### Histogram buckets + +| Metrics | Buckets | +| --- | --- | +| Request, prefill, and decode durations | `0.005`, `0.01`, `0.025`, `0.05`, `0.1`, `0.25`, `0.5`, `1`, `2.5`, `5`, `10`, `30`, `60` seconds | +| Scheduler plugin duration | `0.001`, `0.005`, `0.01`, `0.05`, `0.1`, `0.5` seconds | +| Fairness and session-boost queue durations | `0.001`, `0.005`, `0.01`, `0.025`, `0.05`, `0.1`, `0.25`, `0.5`, `1`, `2.5`, `5` seconds | +| Prefix-cache and KV-cache match ratios | `0`, `0.1`, `0.25`, `0.5`, `0.75`, `0.9`, `0.95`, `0.99`, `1.0` | +| KV-cache Redis and tokenization durations | `0.0005`, `0.001`, `0.0025`, `0.005`, `0.01`, `0.025`, `0.05`, `0.1`, `0.25`, `0.5`, `1`, `2.5` seconds | -| Metric Name | Type | Description | Labels | -|----------------------------------------|---------|--------------------------------------------------|-------------------------------------| -| `kthena_router_tokens_total` | Counter | Total tokens processed (input + output) | `model`, `path`, `token_type` (input/output) | +## Access logs -### Scheduler & Fairness Metrics +Helm enables access logging in `text` format by default. This is the +compatibility default and is not changed by this documentation update. JSON is +recommended for structured ingestion, but operators must opt in explicitly. -| Metric Name | Type | Description | Labels | Buckets | -|-------------------------------------------------------|-----------|--------------------------------------------------------|-------------------------------|------------------------------------------------------------------------| -| `kthena_router_scheduler_plugin_duration_seconds` | Histogram | Execution time per scheduler plugin | `model`, `plugin`, `type` | 0.001, 0.005, 0.01, 0.05, 0.1, 0.5 | -| `kthena_router_fairness_queue_size` | Gauge | Current queued requests per model/user | `model`, `user_id` | — | -| `kthena_router_fairness_queue_duration_seconds` | Histogram | Time spent waiting in fairness/priority queue | `model`, `user_id` | 0.001, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5 | +```yaml +networking: + kthenaRouter: + accessLog: + enabled: true + format: json + output: stdout +``` -### Rate Limiting & Protection +For a standalone deployment, the equivalent environment variables are: -| Metric Name | Type | Description | Labels | -|--------------------------------------------------|---------|------------------------------------------------------|-------------------------------| -| `kthena_router_rate_limit_exceeded_total` | Counter | Requests rejected due to rate limiting | `model`, `limit_type`, `path` | +```yaml +env: + - name: ACCESS_LOG_ENABLED + value: "true" + - name: ACCESS_LOG_FORMAT + value: "json" + - name: ACCESS_LOG_OUTPUT + value: "stdout" +``` -## Access Logs +The valid formats are `text` and `json`. Output can be `stdout`, `stderr`, or a +file path. Prefer `stdout` in Kubernetes so the container runtime and log agent +can collect the records; a file path is local to the container filesystem. -### Recommended Format: Structured JSON +The JSON field names below match the emitted contract: ```json { @@ -83,212 +157,93 @@ The debug endpoints (`/debug/config_dump/*`) are served on port **15000**. "request_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890", "input_tokens": 412, "output_tokens": 189, - "duration_total_ms": 3840, - "duration_queue_ms": 180, - "duration_request_processing_ms": 65, - "duration_upstream_ms": 3480, - "duration_response_processing_ms": 115, - "error": null + "duration_total": 3840, + "duration_request_processing": 65, + "duration_upstream_processing": 3480, + "duration_response_processing": 115 } ``` -## Configuration +All duration fields are integer milliseconds. There is no `duration_queue` +field. `duration_total` is authoritative and can exceed the sum of the marked +processing phases, including when routing or queue time falls between phase +checkpoints. See the [access-log field reference](../reference/router-access-log-fields.md) +for optional routing fields and current error types. -Observability features are configured via the Kthena Router's deployment or ConfigMap. -Most settings are controlled through environment variables or the router's configuration file (depending on your deployment method). - -### Access Log Configuration - -```yaml -accessLogger: - enabled: true - format: "json" # "json" (strongly recommended) or "text" - output: "stdout" # "stdout", "stderr", or file path -``` - -#### Equivalent environment variables (recommended for most deployments) - -```yaml -env: -- name: ACCESS_LOG_ENABLED - value: "true" -- name: ACCESS_LOG_FORMAT - value: "json" -- name: ACCESS_LOG_OUTPUT - value: "stdout" -``` - -#### Metrics Configuration - -```yaml -observability: - metrics: - enabled: true - port: 8080 # Default metrics port - path: /metrics -``` - -## Debug Endpoints - -All available on the same `:15000` port - -| Endpoint | Description | -| --- | --- | -| `/debug/config_dump/modelroutes` | All ModelRoute resources | -| `/debug/config_dump/modelservers` | All ModelServer resources | -| `/debug/config_dump/pods` | Current view of healthy/ready inference pods | -| `/debug/config_dump/namespaces/{ns}/modelroutes/{name}` | Detailed single ModelRoute | -| `/debug/config_dump/namespaces/{ns}/modelservers/{name}` | Detailed single ModelServer | - -## Quick Start – Observability in Action +When `ACCESS_LOG_OUTPUT` is `stdout` or `stderr`, `kubectl logs` can contain +access-log records alongside other process logs. The following examples +therefore filter JSON lines before invoking `jq`. If access logs are written to +a file path, they do not appear in `kubectl logs`; inspect the mounted file or +the configured log collector instead. ```bash -# Forward metrics port (8080) -kubectl port-forward -n kthena-system svc/kthena-router 8080:8080 & - -# Forward debug port (15000) when needed -kubectl port-forward -n kthena-system svc/kthena-router 15000:15000 & - -# Watch real-time request rate by model -watch -n 2 'curl -s http://localhost:8080/metrics | grep kthena_router_requests_total | sort' - -# Tail logs and filter access log entries (JSON lines only) -kubectl logs -n kthena-system deployment/kthena-router -f \ - | grep -E '^{.*}$' | jq . # Only process valid JSON lines - -# Alternative: look for model-related entries in all logs kubectl logs -n kthena-system deployment/kthena-router -f \ - | grep -E "model_name|request_id|duration_total" + | grep -E '^\{.*\}$' \ + | jq . ``` -**Important note about logs** - -- Router logs usually contain both regular application logs (plain text) and structured access logs (JSON). -- Piping everything directly to `jq` will cause errors on non-JSON lines. -- Use `grep` to filter JSON lines first, or use a log processor (like fluentd, vector, or loki) in production. +## Health endpoints -## Troubleshooting Guide +| Endpoint | Successful response | Purpose | +| --- | --- | --- | +| `/healthz` | HTTP `200`, `{"message":"ok"}` | Process liveness | +| `/readyz` | HTTP `200`, `{"message":"router is ready"}` | Controller and datastore readiness; returns HTTP `503` until ready | -### Preparation +## Debug and pprof endpoints -```bash -# Forward metrics port (8080) for metrics queries -kubectl port-forward -n kthena-system svc/kthena-router 8080:8080 & +The debug server binds to `localhost:15000` by default. It is intentionally not +published by the Helm Service. A port-forward grants access to sensitive routing +state and runtime profiles; keep it open only while diagnosing a trusted cluster. -# Forward debug port (15000) for debug endpoints -kubectl port-forward -n kthena-system svc/kthena-router 15000:15000 & +### Configuration dump -# Live structured logs (recommended, with filtering) -kubectl logs -n kthena-system deployment/kthena-router -f \ - | grep -E '^{.*}$' | jq . -``` - -#### 1. High Error Rate (5xx, timeouts, internal server errors) +| Endpoint | Description | +| --- | --- | +| `/debug/config_dump/modelroutes` | All ModelRoute resources known to the router | +| `/debug/config_dump/modelservers` | All ModelServer resources known to the router | +| `/debug/config_dump/pods` | All inference pods known to the router | +| `/debug/config_dump/gateways` | All Gateway resources known to the router | +| `/debug/config_dump/httproutes` | All HTTPRoute resources known to the router | +| `/debug/config_dump/inferencepools` | All InferencePool resources known to the router | +| `/debug/config_dump/namespaces/{namespace}/modelroutes/{name}` | One namespaced ModelRoute | +| `/debug/config_dump/namespaces/{namespace}/modelservers/{name}` | One namespaced ModelServer | +| `/debug/config_dump/namespaces/{namespace}/pods/{name}` | One namespaced pod | +| `/debug/config_dump/namespaces/{namespace}/gateways/{name}` | One namespaced Gateway | +| `/debug/config_dump/namespaces/{namespace}/httproutes/{name}` | One namespaced HTTPRoute | +| `/debug/config_dump/namespaces/{namespace}/inferencepools/{name}` | One namespaced InferencePool | + +### Runtime profiling -Count errors by status & model: +| Endpoint | Description | +| --- | --- | +| `/debug/pprof/` | pprof index | +| `/debug/pprof/profile` | CPU profile | +| `/debug/pprof/goroutine` | Goroutine profile | +| `/debug/pprof/heap` | Heap profile | +| `/debug/pprof/allocs` | Allocation profile | +| `/debug/pprof/block` | Blocking profile | +| `/debug/pprof/mutex` | Mutex-contention profile | -```bash -curl -s http://localhost:8080/metrics | grep 'status_code=5' | sort -nr -``` +## Troubleshooting examples -Top affected models: +After starting both port-forwards from the [endpoint map](#endpoint-map): ```bash +# Request counters by model and result. curl -s http://localhost:8080/metrics \ - | grep 'status_code=5' \ - | grep -o 'model="[^"]*"' | sort | uniq -c | sort -nr -``` + | grep '^kthena_router_requests_total' -Inspect recent failed requests: +# Router configuration and currently known pods. +curl -s http://localhost:15000/debug/config_dump/modelservers | jq . +curl -s http://localhost:15000/debug/config_dump/pods | jq . -```bash +# Recent 5xx access records (requires JSON access-log format). kubectl logs -n kthena-system deployment/kthena-router --since=30m \ - | jq 'select(.status_code >= 500) | {ts: .timestamp, model: .model_name, err: .error, pod: .selected_pod, dur: .duration_total_ms}' -``` + | grep -E '^\{.*\}$' \ + | jq 'select(.status_code >= 500) | {timestamp, model: .model_name, error, pod: .selected_pod, duration: .duration_total}' -Check upstream health: - -```bash -curl http://localhost:15000/debug/config_dump/modelservers | jq . -curl http://localhost:15000/debug/config_dump/pods | jq . -``` - -#### 2. High Latency / Slow TTFT or Generation Speed - -Latency percentiles (p50/p95/p99): - -```bash -curl -s http://localhost:8080/metrics \ - | grep -E 'kthena_router_request_duration_seconds_(bucket|sum|count)' -``` - -Find slowest requests: - -```bash +# Requests slower than four seconds (requires JSON access-log format). kubectl logs -n kthena-system deployment/kthena-router --since=20m \ - | jq 'select(.duration_total_ms > 4000) | {model: .model_name, total: .duration_total_ms, upstream: .duration_upstream_ms, pod: .selected_pod}' -``` - -Check queue pressure: - -```bash -watch -n 2 'curl -s http://localhost:8080/metrics | grep -E "(active_downstream|fairness_queue_size)"' -``` - -#### 3. Queue Buildup / Fairness / Throttling - -Live queue monitoring: - -```bash -watch -n 3 'curl -s http://localhost:8080/metrics | grep fairness_queue_size' -``` - -Queue wait time distribution: - -```bash -curl -s http://localhost:8080/metrics | grep fairness_queue_duration_seconds -``` - -Find throttled/rejected requests: - -```bash -kubectl logs -n kthena-system deployment/kthena-router --since=1h \ - | jq 'select(.error? | .type? == ("rate_limit","throttled","queue_full"))' -``` - -#### 4. Wrong Routing / 404 / Pod Selection Issues - -Validate full routing table: - -```bash -curl http://localhost:15000/debug/config_dump/modelroutes | jq . -``` - -Check pod readiness: - -```bash -curl http://localhost:15000/debug/config_dump/pods | jq . -``` - -Trace a specific request: - -```bash -# Use request_id from client or error message -kubectl logs -n kthena-system deployment/kthena-router \ - | jq 'select(.request_id == "a1b2c3d4-...")' -``` - -#### 5. Token Usage / Cost / Abuse Monitoring - -Current token consumption rate: - -```bash -curl -s http://localhost:8080/metrics | grep kthena_router_tokens_total -``` - -High-token requests: - -```bash -kubectl logs -n kthena-system deployment/kthena-router --since=2h \ - | jq 'select(.input_tokens > 3000 or .output_tokens > 1500)' + | grep -E '^\{.*\}$' \ + | jq 'select(.duration_total > 4000) | {model: .model_name, total: .duration_total, upstream: .duration_upstream_processing, pod: .selected_pod}' ```