cachebackend-api.md

Design: CacheBackend API

Status: implemented · Tracks: InferenceCache tech spec §4.1 · API group: inferencecache.io/v1alpha1

CacheBackend is the namespaced CRD that describes an engine-side cache implementation, an optional remote-storage tier, and the engine integration policy that should use them. Provider lifecycle belongs to storage-provider adapters; runtime adapters own engine Pod wiring only.

Identity

	Value
group	inferencecache.io
version	v1alpha1
kind	CacheBackend
plural	cachebackends
short name	cb

The v1alpha1 contract is pre-launch and explicitly unstable (see the carve-out paragraph below for the precise terms); after the v1beta1 promotion, new fields must be additive and tightening validation on existing fields requires a versioned migration path.

Pre-launch carve-out (active until v1beta1). The project is pre-launch and v1alpha1 is explicitly unstable: where keeping an inert, unidiomatic, or operator-confusing field through to v1beta1 would compound the cleanup work, a per-change waiver allows in-place removal during alpha. Each such removal is gated on (1) a locked design decision naming the field and the reason, (2) zero current consumers (no external operator manifests, no cross-component code), and (3) replacement of the operator-facing surface where one existed. Closed precedent: CacheTenant.spec.quota.maxMemoryBytes and status.memoryUsed removed (we cannot enforce per-tenant byte budgets on shared engines, and the underlying observation would be double-counted across tenants). The cluster-aggregate sibling CacheIndex.status.tenants[].memoryUsed has the same honesty problem (summing per-tenant memory across replicas on a shared engine double-counts the same bytes once per tenant), but because it is a published v1alpha1 status field it is deprecated and zeroed in place rather than removed: the controller stops populating it (always 0) and operators are redirected to the per-replica CacheIndex.status.replicas[].cacheMemoryBytes (engine total per replica, honest at that altitude), while the field stays in the schema for wire/shape compatibility until its removal at v1beta1. Current applied removals: CacheBackend.status.health and the CacheBackendHealth enum removed in favour of the standard status.conditions[Ready|Degraded|Progressing] surface (the old Degraded health value is replaced by Conditions[Degraded]), which the new Ready printer column displays; and CacheBackend.spec.storage{,.pvc} + status.capacity removed — the lm:// LMCache server we provision is in-memory, so a local PVC could not honestly back it, and durability is expressed as a backend choice (the Mooncake backend, now implemented — see Mooncake provider configuration) rather than a generic volume knob (locked decision: docs/design/lmcache-server-persistence.md; replacement surface: backend-type selection). Once v1beta1 is promoted, this carve-out is closed: subsequent breaking changes require a versioned migration.

Cache hierarchy and ownership

The canonical API assigns one architectural dimension to each field:

spec:
  runtime: SGLang
  type: LMCache
  lmCache:
    chunkSizeTokens: 256
    hostMemory:
      capacity: 32Gi
  remoteStorage:
    provider: Redis
    ownership: Managed
    redis:
      image: docker.io/library/redis:7.4-alpine
      resources:
        limits:
          memory: 8Gi
  observation:
    modelID: Qwen/Qwen3

• runtime selects the inference runtime.
• type selects only the engine-side cache implementation.
• lmCache and hiCache configure local/host cache behavior.
• remoteStorage.provider selects the optional remote technology.
• remoteStorage.ownership selects controller-managed or external lifecycle.
• Provider-specific workload settings live below their provider object.
• observation owns event-observation identity and timing.

Omitting remoteStorage is meaningful and never selects infrastructure:

spec:
  runtime: SGLang
  type: LMCache
  lmCache:
    hostMemory:
      capacity: 32Gi

This requests SGLang -> LMCache host memory only. The controller creates no provider Deployment or Service, and the engine adapter injects the node-local LMCache MP worker without an L2 adapter.

Capability resolution is deliberately two-dimensional:

runtime + type                 provider + ownership
      |                                  |
      v                                  v
engine-wire adapter              storage-provider adapter
      |                                  |
      +--------- optional Binding -------+

The provider adapter owns workload and Service rendering and emits a structured binding (lm, resp, or mooncakestore). The engine adapter declares which bindings it accepts. Admission rejects unsupported combinations before an engine Pod is created.

Cache type validation

spec.type is a closed CRD enum containing LMCache and SGLangHiCache. Remote-provider technology and lifecycle ownership are not cache types: Mooncake is selected through remoteStorage.provider, and externally managed infrastructure through remoteStorage.ownership. The API server rejects the old type: Mooncake and type: External spellings before admission.

The canonical External and Mooncake examples are available in config/samples/cachebackend-external.yaml and config/samples/cachebackend-mooncake.yaml.

Spec

Field	Type	Purpose
runtime	enum	Required inference runtime: VLLM or SGLang. Values are case-sensitive.
type	enum	Engine-side cache implementation: LMCache or SGLangHiCache. Defaults to LMCache.
lmCache	object	Typed LMCache engine configuration: chunk size, host-memory capacity, MP-worker image/port, and remote serde.
remoteStorage	object	Optional remote tier. Omitting it means host-only and provisions no provider workload.
remoteStorage.provider	enum	Redis, LMCacheServer, or Mooncake.
remoteStorage.ownership	enum	Managed or External.
remoteStorage.endpoint	string	Required for External, rejected for Managed; managed endpoints are controller-observed in status. Bare host:port is portable across all providers. LMCacheServer also accepts lm://host:port, Mooncake also accepts mooncakestore://host:port, and Redis accepts only bare host:port. Every provider requires a numeric port in 1-65535; admission rejects schemes belonging to another provider.
remoteStorage.redis	object	Redis-owned image and resource configuration.
remoteStorage.lmCacheServer	object	Standalone LMCache-server-owned image, command, and resource configuration.
remoteStorage.mooncake	object	Mooncake-owned image, command, and resource configuration.
observation	object	Observation-owned modelID and firstEventTimeout.
deploymentKind	enum	Managed workload kind: Deployment or StatefulSet. Defaults to Deployment.
replicas	integer	Desired managed backend replicas. Defaults to 1. Minimum 0. See Defaulting for the interaction with spec.autoscaling.minReplicas (first-apply-only).
autoscaling.minReplicas	integer	Lower bound for HPA replica count. Auto-defaulted to spec.replicas on FIRST APPLY ONLY by the admission defaulter when spec.autoscaling is set and minReplicas is left unset (see Defaulting for the first-apply-only semantics); subsequent edits to spec.replicas do NOT move this floor. Minimum 1.
autoscaling.maxReplicas	integer	Upper bound for HPA replica count. Required when autoscaling is set. Minimum 1. Cross-field validation: minReplicas <= maxReplicas.
autoscaling.targetCPUUtilizationPercent	integer	Target average per-pod CPU utilization for the HPA. Defaults to 80 when unset. Range [1, 100].
integration.mode	enum	Which cache tiers the engine is wired for: Offload (default) or EventsOnly. Offload is full participation — cache-aware routing (tier-1) plus the KV-offload connector (tier-2). It may remain host-only, connect to externally owned remote storage, or provision a provider workload when remoteStorage.ownership is Managed. EventsOnly wires routing only: the kvevent-subscriber sidecar is injected when the controller runs with --kvevent-subscriber-image set and observation.modelID is present; otherwise the append is skipped fail-open. No KV connector or backend server is created. See Events-only mode.
integration.role	enum	Engine participation mode: ReadOnly, WriteOnly, or ReadWrite. Defaults to ReadWrite.
integration.failOpen	boolean	Default true. When true, engine pods fall back to local prefill on cache unreachability — the cache is an optimization, never a serving dependency. Setting it to false is an advanced opt-in to fail-closed serving (the cache becomes a serving dependency); the controller surfaces this as a Warning Kubernetes Event on the owning CacheBackend. Pair-specific exception — (sglang, LMCache): SGLang has no cacheless code path while --enable-lmcache is on, so its co-scheduled MP worker is a serving prerequisite (a worker that never starts wedges the engine), not a remote dependency that degrades to local prefill. failOpen is still honored at the tier that can actually be "unavailable" — the shared L2 (the worker comes up L1-only when Redis is unreachable). This is a documented, accepted boundary; see the fail-open semantics in sglang-lmcache-mp-mode.md and SGLang engine support.
integration.engineOverrides	object	Optional engine-injection overrides applied to the args/env the pod-mutating webhook would otherwise inject into the engine container. See Engine-injection overrides.
engineSelector.matchLabels	map	Equality-based label selector matched against engine pod labels (the pod template's metadata.labels, not Deployment, DaemonSet, or any other workload-level labels). Every key/value here must appear on the pod for it to match. matchExpressions is intentionally not exposed in v1alpha1 — the surface is matchLabels only.
hiCache	object	Typed SGLang native HiCache configuration. Required only for type: SGLangHiCache; see SGLang native HiCache.
template	object	Optional pod-level overrides for managed backend pods. This is a narrow override surface, not a full PodSpec; backend containers come from controller defaults.
allowCrossNamespace	boolean	Opt-in flag that allows spec.remoteStorage.endpoint to resolve to a Kubernetes Service in a different namespace from the CacheBackend itself. Without it, admission rejects cross-namespace Service-DNS endpoints. External hostnames and IPs are unaffected. Defaults to false.

Per-namespace lookup tuning lives on CachePolicy, not CacheBackend. The lookup latency budget and the minimum-prefix-token gate are configured via CachePolicy.spec.lookupTimeoutMs and CachePolicy.spec.minimumPrefixTokens, which are the surfaces actually wired into the server's ResolvedPolicy and the LookupRoute path.

Template Overrides

spec.template supports partial pod-level overrides that can be merged with managed backend defaults:

• nodeSelector
• affinity
• tolerations
• topologySpreadConstraints
• imagePullSecrets
• serviceAccountName
• securityContext
• priorityClassName
• schedulerName
• runtimeClassName
• terminationGracePeriodSeconds

It intentionally does not expose containers; requiring users to provide containers would conflict with managed backend defaults and would make simple scheduling overrides unnecessarily large.

Resources

Canonical resources place corev1.ResourceRequirements under the provider that owns the workload: remoteStorage.redis.resources, remoteStorage.lmCacheServer.resources, or remoteStorage.mooncake.resources. The provider renderer deep-copies that block onto its managed container. If the typed block is omitted, the provider uses a bounded 4Gi request / 8Gi limit without persisting a default into the CR.

Pass-through to the rendered container. The provider adapter DeepCopy's the selected typed resource block onto Container.Resources. The deep copy is load-bearing: the reconciler reads from an informer cache, and writing through the spec pointer would corrupt the cached object for every subsequent reader. An explicit empty provider resources: {} suppresses the provider default.

redis-l2: the memory limit also sizes the L2 keyspace. The rendered Redis provider derives --maxmemory from remoteStorage.redis.resources.limits.memory at roughly 80%, with allkeys-lru.

Autoscaling CPU-request fallback. A targetCPUUtilizationPercent HPA needs a positive CPU request as the denominator for its utilization math, so when spec.autoscaling is set the adapter fills in cpu: 250m whenever the selected provider resource block's requests.cpu is absent OR non-positive. The non-positive case matters because the admission validator admits requests.cpu: "0" as a valid kubelet shape (an explicit "no guaranteed minimum" for non-autoscaled pods); without the autoscaling-side replacement, the HPA would dial against a 0 denominator. A positive operator-supplied value (e.g. requests.cpu: "1") survives untouched. The fallback is CPU-only — it never synthesises a memory request — and the operator-supplied memory block (or the legacy webhook/provider default) flows through unchanged.

resources.claims is rejected at admission. corev1.ResourceRequirements also exposes a Claims slice for Dynamic Resource Allocation (DRA), but the renderer does not plumb the matching pod-level spec.resourceClaims — a claim-bound container.resources.claims would render a pod the apiserver rejects (claim name doesn't resolve at the pod level). The validating webhook (rejectResourceClaims) hard-rejects non-empty claims until DRA is wired end-to-end; a nil/empty claims slice admits unchanged.

Request/limit relationship is resource-aware. The validating webhook (rejectResourceLimitsBelowRequests) enforces K8s' two-regime contract:

• Overcommittable resources (cpu, memory, ephemeral-storage): limits[X] must be >= requests[X] when both are set. Memory is the motivating case (an inverted memory limit deepens the OOM-kill cliff this field exists to close), and the rule catches CPU typos with the same diagnostic.
• Non-overcommittable resources (hugepages-* and vendor-prefixed extended resources like "nvidia.com/gpu"): limits[X] must EQUAL requests[X] when both are set. K8s does not allow overcommitting these — every page or device is dedicated — so request and limit must agree.

Limits-only shapes admit unchanged for any resource — K8s auto-populates requests from limits. Requests-only is admitted only for overcommittable resources (cpu, memory, ephemeral-storage); a non-overcommittable resource declared in requests without a matching limits entry is rejected (rejectRequestsOnlyForNonOvercommittableResources), because K8s requires hugepages and extended resources to declare both halves together.

Extended-resource quantities must be integers. Vendor-prefixed extended resources (e.g. nvidia.com/gpu) are allocated by whole units — K8s rejects a fractional shape like nvidia.com/gpu: 500m on the rendered Pod. The validating webhook (rejectFractionalExtendedResources) mirrors that rule at admission, so the operator sees a field-scoped error at kubectl apply. Standard overcommittable resources (cpu, memory, ephemeral-storage) admit fractional values — 250m is the canonical kubelet CPU shape and is unaffected.

Hugepage quantities must align to the page size. The Linux kernel allocates hugepages in whole-page chunks, so K8s rejects a misaligned shape like hugepages-2Mi: 3Mi (3Mi isn't a multiple of 2Mi). The validating webhook (rejectMisalignedHugepageQuantities) parses the page size from the resource name's suffix and rejects any positive quantity that is not a whole multiple of that page size. Zero quantities admit trivially (no allocation) and negative quantities are caught upstream by rejectNegativeResourceQuantities.

Quantities must be non-negative. The CRD-schema layer treats each requests/limits entry as a resource.Quantity string and admits a leading - without complaint. The kubelet only flags the negative quantity once the child pod tries to schedule — by which time the operator is chasing it through Deployment events. The validating webhook (rejectNegativeResourceQuantities) rejects any strictly-negative entry at admission so the regression surfaces at kubectl apply instead. Zero is admitted (an operator who writes requests.memory: "0" is explicitly opting into "no guaranteed minimum", which matches the kubelet's >= 0 contract).

Resource names must match K8s container-resource rules. ResourceList keys are opaque map keys at the CRD-schema layer; an invalid name like "foo" or "" persists in etcd and only fails when the apiserver later rejects the child pod. The validating webhook (rejectInvalidResourceNames) applies the same rules the apiserver applies to a Container.Resources map: standard names (cpu, memory, ephemeral-storage) admit unconditionally; a hugepages-<size> name admits only when the size suffix parses as a strictly-positive resource.Quantity (e.g. "hugepages-2Mi", "hugepages-1Gi" — a bare "hugepages-" or non-numeric "hugepages-nope" is rejected because the apiserver requires the size token); any other name must be third-party vendor-prefixed (e.g. "nvidia.com/gpu") and pass IsQualifiedName. A bare unqualified "foo" is rejected even though IsQualifiedName alone admits it, because the apiserver's container-resource layer requires extended resources to carry a vendor identity. Names under the K8s-reserved prefixes kubernetes.io/ and requests.kubernetes.io/ are also rejected — those prefixes are reserved for native resources, so extended resources may not use them. The rejection names the offending key so multi-key errors surface together.

Inert without a controller-managed workload. Host-only, externally owned, and SGLangHiCache configurations provision no cache-server workload of their own. HiCache host memory belongs to the user-owned engine container and must be sized on that workload instead.

SGLang engine support

SGLang supports two peer cache integrations:

Runtime/backend pair	Data plane	Controller-managed workload
(SGLang, LMCache) without remoteStorage	Node-local LMCache MP worker, host-only	None
(SGLang, LMCache) with Managed Redis	Node-local LMCache MP worker with a shared Redis remote tier	Redis Deployment and Service
(sglang, SGLangHiCache)	Native engine-local host cache	None

SGLang LMCache MP mode

SGLang drives LMCache in multiprocess (MP) mode (implemented, GPU-validated end to end). Unlike vLLM, SGLang reads LMCache config from a --lmcache-config-file (carrying mp_host/mp_port), attaches to a node-local MP worker over ZMQ + a shared-memory data path, and offloads to a shared L2 store (the worker's --l2-adapter) — it does NOT use a cluster-reachable lm:// server (lm:// is not even a valid MP --l2-adapter type). So the (sglang, LMCache) data plane differs from vLLM's on both halves, and the sections below reflect that. Authoritative design + validation evidence: sglang-lmcache-mp-mode.md.

SGLang is the second runtime the cache plane supports (spec.runtime: SGLang, spec.type: LMCache; adapter at internal/adapters/builtin/runtime). Its engine adapter configures the node-local MP worker and accepts either no binding (host-only) or a RESP binding. The independent Redis provider adapter creates a Redis workload only when spec.remoteStorage explicitly selects provider: Redis, ownership: Managed.

Cluster prerequisite — Kubernetes ≥ 1.29 (REQUIRED for the SGLang MP wire). The MP worker is injected as a native sidecar — an initContainers entry with restartPolicy: Always, which K8s only understands from 1.29 (beta, on by default; stable 1.33). On an older cluster the apiserver does not recognize that field, so a (sglang, LMCache) engine pod fails admission (or the worker degrades to a plain init container that exits before the engine starts) rather than failing open — the one place this pair has a hard cluster-version floor. vLLM+LMCache and the routing-only path have no such floor. There is no in-webhook version gate today; operators on the SGLang pair must run 1.29+.

Two more caveats on the SGLang support surface (details below): (1) server-derived LookupRoute with raw token_ids/prompt_text only hits when the server's single global --engine-block-size matches SGLang's page size (see the "Block-size alignment" note later in this section); gateways that send pre-computed prefix_hash/block_hashes are unaffected. (2) The lmcache-kernel-check init container is vLLM-only today (the SGLang adapter does not implement InitContainerProvider), so EngineKernelsHealthy is not published for SGLang pods.

The webhook renders the MP data plane on the SGLang engine pod. Alongside the engine container it adds a node-local MP-worker native sidecar (an init container with restartPolicy: Always) that writes the --lmcache-config-file then runs the LMCache MP server on 127.0.0.1. With a RESP binding it appends --l2-adapter and offloads to Redis; without a binding it runs host-only. NVIDIA_VISIBLE_DEVICES=all lets the GPU-less sidecar CUDA-IPC the engine's GPU with no device-plugin allocation, an exec startup-probe on the loopback ZMQ port gates the engine's start, and a shared emptyDir carries the config file. For /dev/shm (the L1 tier) it reuses the engine's own volume when the engine already mounts one (a duplicate mountPath is an invalid Pod), else adds a sized emptyDir{medium: Memory} — see the reserved-names note below for the reuse/reject rules. On the engine container (name sglang) it injects:

• --enable-lmcache — SGLang's boolean flag (an argparse store_true) that activates its LMCache connector. This replaces vLLM's --kv-transfer-config JSON.
• --lmcache-config-file <path> — points the engine at the MP config file the worker writes (mp_host/mp_port); MP mode aborts at startup without it.
• LMCACHE_USE_EXPERIMENTAL=True — gates SGLang's experimental LMCache integration; without it --enable-lmcache does not engage the connector.
• INFERENCECACHE_FAIL_OPEN=<true|false> — the spec.integration.failOpen mirror.

GPU visibility on the MP worker — an isolation trade-off to know about before running this on a shared node. The worker sidecar carries NVIDIA_VISIBLE_DEVICES=all, so with the NVIDIA container runtime it can see every GPU on the node, not only the one its engine was allocated. It holds no device-plugin allocation (no nvidia.com/gpu request), so it consumes no GPU from the node's allocatable — but the visibility is real, and on a node shared with other tenants' GPU workloads the worker is not confined to its own device.

Why it cannot be narrowed. The worker moves KV by CUDA-IPC: the engine hands it a device UUID, which LMCache resolves to a local device index — and that resolution fails unless the device is visible to the worker process. Revoking visibility is GPU-validated as fatal, not degraded: the worker dies with RuntimeError: Device UUID <uuid> not found in the discovered devices and the engine never reaches ready. Scoping to just the engine's device is not available to a mutating webhook: the device plugin assigns the UUID at kubelet time, after admission runs, so there is nothing to narrow to yet. Giving the worker its own nvidia.com/gpu request is worse — it burns a second GPU and the scheduler would hand it a different device than the engine's.

Scope of what the adapter adds. This is the engine image's own posture rather than something the adapter introduces: sglang images ship NVIDIA_VISIBLE_DEVICES=all in their ENV, and the device plugin overrides it only for containers that request a GPU (the engine gets a specific UUID; a request-less sidecar keeps the image default). The adapter sets it explicitly so the wire also works on a workerImage that lacks that default, instead of depending on an image side effect. Operators who need hard GPU isolation between tenants should not co-schedule those tenants on one node — the same guidance that applies to any CUDA-IPC sidecar.

Names the MP wire reserves on the engine pod. The init container lmcache-mp-worker, the volumes lmcache-config + lmcache-dshm, and the mount path /etc/lmcache are adapter-owned. If the pod already carries one of them and the adapter did not render it, admission rejects the injection — which the pod webhook turns into a fail-open admit, so the pod starts un-wired (no cache) rather than with its own container silently overwritten. The same applies when the engine mounts /dev/shm read-only or from a configMap/secret/downwardAPI/projected volume: the MP data path writes there, so it is rejected at admission instead of failing deep inside LMCache at runtime. Rename the colliding object (or drop the readOnly) to get the pod wired. Re-injecting an already-wired pod is not a collision — the adapter recognises its own worker and converges it on the current render.

The old lm:// LMCACHE_REMOTE_URL / serde / chunk-size / local-CPU env is NOT injected — SGLang MP mode ignores it. New manifests use typed spec.lmCache fields:

Field	Default	Bounds	Purpose
lmCache.chunkSizeTokens	256	>=1	The worker's --chunk-size and config-file chunk_size.
lmCache.hostMemory.capacity	4Gi	positive quantity	Host-memory budget; rendered to the worker's whole-GiB L1 allocation.
lmCache.workerPort	5555	1–65535	Loopback ZMQ port used by the engine and worker.
lmCache.workerImage	engine image	—	Optional MP-worker image override.

Deliberately not injected for SGLang (a real engine difference, not an omission): VLLM_USE_V1 (a vLLM-internal codepath with no SGLang analogue) and PYTHONHASHSEED (vLLM pins it to stabilise its builtin-hash()-seeded block-hash chain across TP workers; SGLang derives its prefix hash with hashlib.sha256 over the token-id bytes, independent of PYTHONHASHSEED).

spec.integration.role support. vLLM maps the role onto its LMCache connector's kv_role (ReadOnly→kv_consumer, WriteOnly→kv_producer, ReadWrite→kv_both). SGLang's --enable-lmcache integration has no kv_role split — it always both stores and retrieves — so a (sglang, LMCache) backend supports only ReadWrite (the default). Admission rejects ReadOnly / WriteOnly for SGLang (rejectUnsupportedSGLangRole) rather than silently treating them as ReadWrite; the rule lifts if SGLang's LMCache integration gains a producer/consumer split.

Reserved set (internal/adapters/builtin/runtime): ReservedArgs() = --enable-lmcache, --lmcache-config-file; ReservedEnv() = LMCACHE_USE_EXPERIMENTAL, INFERENCECACHE_FAIL_OPEN. In MP mode the old lm:// LMCACHE_REMOTE_URL is neither injected nor reserved. VLLM_USE_V1 / PYTHONHASHSEED are not reserved because they are never injected.

The two override surfaces are separate: spec.lmCache shapes the worker sidecar, while spec.integration.engineOverrides edits the engine container's args/env only.

KV-event source & hash_scheme: "sglang". SGLang adopted vLLM's KV-event wire wholesale — --kv-events-config drives a ZMQ ZmqEventPublisher emitting the same msgspec array-like BlockStored / BlockRemoved / AllBlocksCleared tuples (BlockStored carries token_ids, so the subscriber derives the same in-pod content fingerprint it does for vLLM). The shipped kvevent-subscriber binary therefore decodes SGLang's stream unchanged; the only difference is the adapter pins the sidecar's --hash-scheme=sglang. The index keys on (tenant, model, hash_scheme, adapter, prefix_hash), so SGLang prefixes occupy a domain disjoint from vLLM's: a request hashed under one scheme never false-hits a bytewise-identical entry recorded under the other, even when both engines tokenize the same text to the same token ids and the content fingerprints collide. As with vLLM, the operator must launch the engine with --kv-events-config '{"publisher":"zmq","endpoint":"tcp://*:5557","topic":"kv-events"}' for the publisher to be active — the adapter wires the cache offload, not the event publisher.

Block-size alignment for server-derived lookups (operational note). The subscriber-ingested path is block-size-safe by construction: the subscriber derives each prefix fingerprint in-pod using the block_size carried on the engine's own BlockStored event (SGLang's --page-size, often 64), so SGLang entries land in the index at SGLang's block size with no server involvement. The server-derived lookup path is the catch: when a gateway calls LookupRoute with raw token_ids / prompt_text (rather than a pre-computed prefix_hash / block_hashes chain), the server fingerprints them with its single global --engine-block-size (default 16, vLLM's). For SGLang those server-side hashes only line up with SGLang-ingested entries when --engine-block-size is set to SGLang's page size. Because the flag is one global value, a single server cannot serve raw-token_ids lookups for both a vLLM (16) and an SGLang (64) deployment at once — in a mixed-engine cluster, have gateways send the pre-computed prefix_hash / block_hashes (the subscriber/fingerprint path, which is block-size-correct per engine) for the raw-token path, or run a server per block size. Making --engine-block-size per-hash_scheme is a server change tracked as a follow-up; it is the first concrete latently-vLLM-centric assumption this second engine surfaced.

MP mode implemented (GPU-validated 2026-07). The once-open wire-test question — does SGLang honour LMCACHE_REMOTE_URL from the env, or require a config file? — resolved to "neither the old way": SGLang ignores the LMCACHE_* env and reads config only from --lmcache-config-file, driving LMCache in MP mode, not the lm:// remote-server model. The adapter now renders exactly that — config-file + node-local MP-worker sidecar + shared Redis L2 — validated end to end (store→flush→retrieve reuses KV via the worker). Full design + evidence: sglang-lmcache-mp-mode.md.

SGLang native HiCache

The (sglang, SGLangHiCache) pair configures the selected SGLang engine Pods directly. It does not create a cache-server Deployment, Service, HPA, or endpoint. The first implementation intentionally publishes no Ready condition: Kubernetes Pod readiness proves that SGLang is serving, but does not prove a HiCache host-tier write/read round trip. A dedicated readiness contract is a separate follow-up.

The required integration shape is:

spec:
  runtime: SGLang
  type: SGLangHiCache
  engineSelector:
    matchLabels:
      app: sglang
  hiCache:
    # Exactly one:
    ratio: "2.0"
    # sizeGB: 64
    # Optional typed pass-through:
    writePolicy: write_through
    ioBackend: kernel
    memoryLayout: layer_first

ratio is a string containing a finite number greater than zero; sizeGB is a positive integer. The optional fields are injected only when present, leaving defaults to the SGLang version in the engine image. Their accepted values match the SGLang CLI:

• writePolicy: write_back, write_through, write_through_selective
• ioBackend: direct, kernel, kernel_ascend
• memoryLayout: layer_first, page_first, page_first_direct, page_first_kv_split, page_head

The webhook injects --enable-hierarchical-cache plus the corresponding --hicache-* flags at Pod CREATE time. A one-container Pod may use any container name; a multi-container Pod must name its engine container sglang. Existing matching arguments are left byte-for-byte unchanged. A different value, the opposite capacity mode, a malformed/duplicate argument, or existing SGLang LMCache flags causes the whole injection to fail open without partial HiCache wiring.

Changing or deleting the CacheBackend does not mutate live Pods. Roll the SGLang workload to apply a new configuration or switch between LMCache and HiCache. The webhook does not inspect image tags: the chosen image must support these SGLang arguments.

HiCache host memory is charged to the engine container's cgroup. The operator must size the engine's memory request/limit and node capacity accordingly. Inference-cache does not derive resource changes from sizeGB or ratio, and does not add /dev/shm, hugepages, memlock, hostIPC, or privileged settings. The KV-event subscriber reads its model identity from spec.observation.modelID, independently of the HiCache configuration.

Events-only mode (spec.integration.mode = EventsOnly)

spec.integration.mode selects which cache tiers an engine is wired for. The default, Offload, is full participation: cache-aware routing (tier-1) PLUS KV offload (tier-2). Server-backed managed adapters include a controller-provisioned backend server; engine-local adapters such as SGLang HiCache provide tier-2 inside the engine pod and create no server workload or endpoint. EventsOnly wires the routing tier only.

What events-only does and does not provision. An events-only backend is the lighter, routing-only deployment:

• No provisioned server. The reconciler creates no Deployment and no Service for an events-only backend, and status.endpoint stays empty (there is no server address to publish). Flipping an existing Offload backend to EventsOnly sheds the previously-provisioned Deployment + Service on the next reconcile.
• No KV connector. The pod webhook does NOT inject the --kv-transfer-config arg or the LMCACHE_* env into the engine container — the engine container is left otherwise untouched. Because nothing dials a cache server, no endpoint is required, and the webhook injects an events-only engine pod even though status.endpoint is empty (the usual empty-endpoint fail-open is bypassed for this mode).
• Mode wins over host-tier configuration. If spec.lmCache is present, EventsOnly still injects no LMCache connector or host-tier settings; the block is ignored for engine wiring. Operators should omit spec.lmCache on routing-only resources so the manifest does not imply an active host tier. spec.remoteStorage is rejected rather than ignored because it declares a provider that nothing would dial.
• The kvevent-subscriber sidecar is injected — when wired. That is the whole point of routing: once the sidecar is appended, LookupRoute and the per-backend status.indexParticipation slice behave identically to a managed backend; only the offload tier (server + connector) is absent. The append is gated exactly as for a managed backend and is skipped fail-open when either gate is unmet: the controller must run with --kvevent-subscriber-image set (unset by default, so a default install injects no subscriber) AND spec.observation.modelID must be present to supply --model-id. When skipped, the webhook leaves the engine pod untouched and stamps no injected-by annotation.
• Evictions are tier-aware. The subscriber tags each prefix with a cache tier from the block lifecycle: BlockStored → T1 (resident in HBM). On a BlockRemoved, the two modes diverge. In Offload mode the paired LMCache L2 tier still holds the block after the engine evicts it from HBM, so the subscriber (--ignore-block-removed=true) re-reports the evicted prefix at tier T2 (reload-able from host RAM), anchored at the eviction timestamp — the entry is kept, not dropped, and honestly tagged colder than HBM; a later BlockStored of the same content re-reports it back at T1. In EventsOnly mode there is no L2 retaining the block, so a BlockRemoved genuinely means the prefix is gone and the hint MUST be pruned — the subscriber omits the flag and forwards the eviction as PREFIX_EVICTED. Either way a stale/mis-tagged hint is soft state (a cache miss at worst, never a wrong answer). See docs/design/kvevent-subscriber-wiring.md "L2 cache tier semantics".

Readiness is gated on the first KV event, same as managed. An events-only backend has no workload to wait on, so it is "up" the moment it exists — the firstEventTimeout clock starts immediately (status.firstAvailableAt is latched on the first reconcile). It then runs the same KV-event readiness gate as a managed backend: Ready=False/AwaitingFirstKVEvent until the first event, Ready=True/KVEventsObserved once status.indexParticipation.lastEventAt is observed, and Ready=False/NoKVEventsObserved, Degraded=True if the window elapses with no event. The base Ready reason is EventsOnlyActive. The managed-only advisory conditions FunctionalProbeOK, EngineKernelsHealthy, T2Degraded, and EngineCompatibility are never published on an events-only backend — there is no server to functionally probe, no LMCache native-kernel check (events-only loads no connector, so the lmcache-kernel-check init container is never injected), no tier-2 to mark degraded, and no injected KV connector that could be incompatible (events-only injects none, and an Offload→EventsOnly flip clears any prior verdict).

Why it exists. EventsOnly is the supported integration for hybrid-attention models that cannot take a vLLM KV connector — Qwen3.6/Next gated-DeltaNet, Mamba/Jamba, KDA, Falcon-H, Granite-hybrid, and similar. vLLM disables its hybrid KV-cache manager the moment any KV connector is loaded (KV-spec unification then fails at init), so these models cannot take the tier-2 connector; but their KV events coexist fine with the hybrid manager, so cache-aware routing still works. It is also simply a lighter deployment for routing-only users who do not want an offload tier at all.

Admission constraints. Because an events-only backend provisions no server, server-shaped configuration is structurally meaningless and is rejected at admission:

• spec.remoteStorage is forbidden — any Managed or External declaration requests an offload provider that events-only deliberately does not wire.
• spec.autoscaling is forbidden — there is no workload to scale. The rejection is field-scoped to spec.autoscaling.

LMCache server / client version alignment

The standalone lmcache-server image (spec.remoteStorage.lmCacheServer.image, default lmcache/standalone:v0.4.7) and the lmcache client compiled into the engine image (operator-supplied, or pip-installed into the engine at runtime) communicate over a versioned wire protocol. They must be wire-compatible. A mismatch does not fail loudly: remote KV stores fail (e.g. [Errno 32] Broken pipe / connection resets), the backend records 0 reload hits, and tier-2 (remote KV offload) is silently disabled with no surfaced error. The cache plane keeps serving and routing; it simply never gets a tier-2 hit, which is hard to distinguish from a cold cache.

The same silent store-failure signature can also come from an under-provisioned server that is OOMKilled under load — the standalone server keeps KV in memory, and a default memory request far below a large model's working-set KV (e.g. a 32B model's KV is tens of GB) will OOM the server the moment stores begin, dropping every connection. Size the server's memory to the expected working set. (Surfacing tier-2 store-failure / hit-rate health so neither failure mode stays silent is a separate follow-up.)

Because of this:

• The default serverImage is pinned to a specific, non-floating version, never :latest. A floating tag can drift to a server build whose wire protocol no longer matches the client, reintroducing the silent-disable failure mode on an unrelated pull. (The default tag v0.4.7 is version-aligned with the validated lmcache 0.4.7 client, but the standalone server image was not independently wire-tested; confirm against a tested build — ideally an @sha256: digest — before release. See the TODO on defaultLMCacheServerImage in internal/adapters/builtin/storage/lmcache_server.go.)
• Pin both sides. When an operator overrides remoteStorage.lmCacheServer.image, they must choose an lmcache-server version that is wire-compatible with the lmcache client version their engine image carries, and pin the engine's client too (a pip install lmcache at engine startup is itself a floating reference). For non-local runs, prefer an @sha256: digest.
• IC cannot auto-match these versions: it has no source of truth for the engine's client version (the engine image is operator-supplied and the client may be pip-installed at runtime), so it cannot detect or warn on a skew today. The mitigation is this alignment contract plus the pinned default; runtime detection / a tier-2 health signal is a separate follow-up.

LMCache client kernels ↔ engine-image CUDA / vLLM alignment

The lmcache client compiled into the engine image ships native CUDA kernels (lmcache.c_ops). They must match the engine image's CUDA runtime. A mismatch — e.g. a pip install lmcache that pulls a CUDA-13-built wheel onto a CUDA-12.9 image — fails to load (Failed to import backend lmcache.c_ops: libcudart.so.13) and lmcache silently falls back to a single-stream torch path that does not parallelize. T2 reload still works in isolation but serializes under concurrency (measured ~10× slower), and the failure surfaces only as a log WARNING. This is the same silent-degradation class as the wire-protocol skew above, in its local-kernel variant.

Because import lmcache.c_ops is overridden to a fallback shim on load failure (so it always succeeds and cannot be used as a health check), the control plane detects this at deploy time with an injected lmcache-kernel-check init container that force-loads the native extension from disk in the engine's own image. It reports onto the CacheBackend EngineKernelsHealthy condition (see Conditions) and is configured per-CacheBackend via the inferencecache.io/lmcache-kernel-check annotation:

Annotation value	Behavior
auto (default / unset)	Inject in report-only mode only when the engine container requests a GPU (the kernels are GPU-only; a CPU build legitimately has none).
report-only	Always inject; a c_ops load failure makes the detector exit 0, so it does not block the engine pod (best-effort fail-open — see the residual cases in Boundaries). The condition surfaces the result.
strict	Always inject; on failure the engine pod stays in Init and never serves (fail-closed), and the managed CacheBackend Ready is downgraded with reason EngineKernelDegraded.
off	Never inject.

The annotation value is validated at admission — an unrecognized value (e.g. a strcit typo) is rejected, so a typo cannot silently relax strict enforcement back to report-only. Changing the annotation affects only newly-admitted engine pods; the EngineKernelsHealthy condition and any strict Ready downgrade reflect each pod's actual admitted mode (read from the pod, not the CacheBackend's current annotation), so flipping the annotation on a live backend takes effect as its pods roll.

Engine scope — vLLM only today. The kernel-check init container is provided by the runtime adapter via the private internal InitContainerProvider capability, which only the vLLM+LMCache adapter implements. The SGLang+LMCache adapter does not implement it yet, so inferencecache.io/lmcache-kernel-check has no effect on SGLang engine pods and EngineKernelsHealthy is not published for them — even though SGLang loads the same lmcache client and would benefit from the same check. The annotation is still shape-validated for any CacheBackend (the value rule is engine-agnostic); it simply injects nothing on the SGLang path. Extending the check to SGLang is a follow-up.

Boundaries (what the check does and does not prove):

• EngineKernelsHealthy=True means the native kernels loaded — it does not mean T2 reload is fast under concurrency. The reload-serialization symptom is validated separately by a real-GPU concurrency canary.
• The check proves load-time linkage (it catches a missing/mismatched libcudart). It does not prove runtime executability: a libcudart present but paired with a too-old driver loads cleanly and fails only at kernel launch. That residual is caught only at runtime.
• Strict-mode GPU cost: a pod stuck in Init (failing the check in strict mode) still holds its nvidia.com/gpu reservation while serving nothing. Reclaim it by fixing the engine image's lmcache/CUDA alignment or switching the annotation to report-only.
• The check runs import torch (the native extension links libtorch), adding a few seconds to GPU engine-pod startup. The engine imports torch anyway.
• Report-only fail-open is best-effort. The init container runs the engine image's own python3; in report-only mode the detector always exits 0, so a c_ops failure never blocks the pod. The init container declares small CPU/ memory requests and no limits — the most broadly-compatible shape, but note that no resource shape is fail-open under every namespace policy: a ResourceQuota/LimitRange that requires per-container requests rejects a container with none, while a LimitRange per-container max can reject large ones. Small requests stay below any max the GBs-needing engine already satisfies and are subsumed by the engine's in the pod's effective request (so no scheduling/quota footprint increase); omitting limits avoids the max-limit trip. The other residual ways it could block are python3 failing to start at all (which means the Python engine is itself broken — not a false outage caused by this check) or an OOM during import torch (the check sets no memory limit, so the import is bounded by the pod/node unless a namespace LimitRange defaults a memory limit onto it — in which case a too-small default could OOM the import, the same way it would constrain any unlimited container). The check is deliberately not wrapped in a shell to force exit 0, because a minimal/distroless image could lack /bin/sh and reintroduce the very block the wrapper aimed to avoid.

EngineKernelsHealthy complements FunctionalProbeOK (which round-trips the server-side cache path): the kernel check catches the engine-side load cause that the round-trip probe cannot see.

Mooncake provider configuration

spec.remoteStorage.mooncake selects the Mooncake provider adapter (internal/adapters/builtin/storage/mooncake.go) to reconcile the standalone Mooncake master workload. The vLLM runtime adapter separately wires engine pods to it through the LMCache remote-binding contract. Mooncake is the durable / shared cache path — the backend-type expression of the persistence decision in docs/design/lmcache-server-persistence.md (the in-memory lm:// lmcache-server is the simple default; Mooncake is the scalable one — durability is a backend choice, not a generic volume knob).

Operator requirement — the Mooncake master runs on the host network. Unlike LMCache's lm:// (one server, one port, one connection — a virtual ClusterIP suffices), Mooncake is a peer-to-peer transfer-engine mesh: the master on :50051 returns only a directory pointer ("this block lives on node B"), and the engine then dials that node's real IP on a dynamically negotiated port to move the KV bytes. A ClusterIP Service forwards only the ports declared on it, and CNI overlay pod IPs are not reachable for the mesh — so the adapter renders the master with hostNetwork: true behind a headless Service (clusterIP: None), whose DNS name (published as status.endpoint) therefore resolves straight to the master's node IP with every port reachable. Consequences you must plan for:

• The namespace must permit hostNetwork — a Pod Security restricted namespace will reject the master pod.

• The master reserves its ports (50051 / 8080 / 9003) on its node (the API server defaults hostPort=containerPort for hostNetwork pods), and its Deployment uses the Recreate rollout strategy — a rolling surge would collide on those ports.

• The master is a singleton. spec.replicas > 1 and spec.autoscaling are rejected at admission when remoteStorage.provider: Mooncake: a second replica either fails to schedule because its node ports are already bound or comes up as an independent master and silently splits the store. spec.replicas: 0 (disabled) and 1 remain valid.

• Network exposure — plan for it. Host networking publishes the master's RPC (50051), metadata (8080) and metrics (9003) ports, plus the transfer engine's dynamically negotiated data ports, directly on the node's interfaces, outside the pod network. NetworkPolicy selects pods by pod IP and therefore does not constrain a hostNetwork pod's listeners — the isolation you get from pod-network policy is simply absent here. Restrict access with node-level controls instead: security-group / firewall rules on the node interfaces, and by constraining which nodes the master and its engines may schedule onto. Treat all of these ports as cluster-internal only; none of them authenticate callers.

• Engine pods need host networking too — opt in with spec.integration.engineHostNetwork: true. Mooncake's mesh is dialed from the engine, so an overlay engine pod cannot participate. With the flag set, the Pod webhook moves matched engine pods onto the host network (hostNetwork + dnsPolicy: ClusterFirstWithHostNet) alongside the usual LMCACHE_* wiring. Until it is set, admission warns on every Mooncake remoteStorage apply and the backend reports Ready while transferring zero KV.

  It is opt-in, never injected by default, because it rewrites the networking of a pod you own. hostNetwork is a privilege, and mutating webhooks run before Pod Security validation — so silently adding it would turn a working engine pod into one a restricted namespace rejects, with an error naming Pod Security rather than this controller. The flag is rejected on backend types that do not need it, so it can never sit inert.

  Setting the flag does not move pods that already exist. Injection happens at pod admission, and a Pod's hostNetwork is immutable — so enabling it changes only pods admitted afterwards. Roll your engine workload (kubectl rollout restart deployment/<engine>) after enabling it, or the running engines stay on the overlay and keep transferring zero KV.

  Host networking is applied together with the LMCACHE_* connector, behind the same gate: if the backend has not yet published status.endpoint, a matched engine pod admits un-wired — no connector and no hostNetwork. It is never granted to a pod that has nothing to use it for. Such a pod needs a roll once the backend reports Ready in any case, since it is missing the connector env too.

• Engine scheduling and rollout, under host networking. These constraints land on your engine Deployment, which this controller does not own and therefore cannot clamp the way it clamps the master:

  • The API server defaults hostPort to containerPort for hostNetwork pods, so each engine replica reserves its serving port (e.g. 8000) on its node — at most one engine replica per node per port. Size the engine's replica count against schedulable nodes, not just GPUs.
  • A RollingUpdate engine Deployment can deadlock: the surge pod cannot bind a port the outgoing pod still holds, so it stays Pending forever and the rollout never completes. Use strategy: Recreate (or maxSurge: 0) on a hostNetwork engine Deployment.
  • Pod-network isolation is absent for engine pods too — the same NetworkPolicy caveat above applies to them.

This is inherent to Mooncake, not a choice the adapter can avoid. Host-only LMCache and the standalone LMCacheServer provider are unaffected and stay on the pod network.

Mooncake is wired as an LMCache remote backend, not vLLM's native MooncakeStoreConnector. The engine runs the same LMCache connector the LMCache backend uses (kv_connector=LMCacheConnectorV1) pointed at a mooncakestore://host:port remote store — the Mooncake analog of lm://. So the engine-side injected wire follows the same pod-webhook engine-wiring contract except that LMCACHE_REMOTE_URL carries the mooncakestore:// scheme. The native MooncakeStoreConnector is configured exclusively through a MOONCAKE_CONFIG_PATH JSON file (it has no env-var surface for the master address), and the pod-mutating webhook can only inject env + args — it cannot write a file into a user-owned engine container — so routing the controller-resolved master endpoint through LMCACHE_REMOTE_URL=mooncakestore://… is the only path that lets status.endpoint reach the engine via injection alone. Operators who prefer the native connector pre-bake their own config file; this adapter targets the auto-wired path.

Provider-side fields consumed by provider.ResolveMooncakeServer:

Field	Default	Purpose
remoteStorage.mooncake.image	docker.io/kvcacheai/mooncake:0.3.11.post1 (pinned, non-floating; fully qualified)	Container image for the standalone Mooncake master. Fully qualified (docker.io/…) so CRI-O nodes without short-name resolution configured do not reject it; it is version-aligned with the mooncake-transfer-engine 0.3.11.post1 release on PyPI. Pin to an @sha256: digest for non-local runs.
remoteStorage.mooncake.command	mooncake_master --rpc_port=50051 --metrics_port=9003 --enable_http_metadata_server=true --http_metadata_server_host=0.0.0.0 --http_metadata_server_port=8080	Master command and arguments. The default launches RPC, Prometheus metrics, and the embedded HTTP metadata server. Do not change the RPC (50051) or HTTP metadata (8080) ports through this override: the rendered Service, readiness probe, status endpoint, and engine binding use those fixed values and are not derived from free-form command text.
remoteStorage.mooncake.resources	memory request 4Gi, limit 8Gi	Resources for the managed master container. An explicit typed block replaces the defaults; autoscaling also supplies a 250m CPU request when no positive CPU request is present.

The Service exposes the master's RPC port (50051) first so the reconciler's engine-agnostic serviceEndpoint helper publishes it into status.endpoint, plus the HTTP metadata port (8080).

Engine-side: the adapter injects the same --kv-transfer-config '{"kv_connector":"LMCacheConnectorV1","kv_role":"<role>"}' arg and the same LMCACHE_* / VLLM_USE_V1 / INFERENCECACHE_FAIL_OPEN / PYTHONHASHSEED env as an LMCacheServer binding, with LMCACHE_REMOTE_URL=mooncakestore://<status.endpoint>. Chunk size, serializer, and host-memory settings come from spec.lmCache. The reserved args/env are therefore identical. The kvevent-subscriber sidecar is also identical (the KV-event stream comes from vLLM, not the L2 store; --hash-scheme=vllm, --ignore-block-removed=true).

Transfer-engine tuning is operator-supplied, not env-injected. Mooncake's static transfer-engine config (metadata_server, protocol tcp/rdma, device_name, segment sizes) lives in LMCache's extra_config, which is read from an engine-side config file (LMCACHE_CONFIG_FILE / MOONCAKE_CONFIG_PATH) — not from env vars, so the webhook cannot inject it. The adapter wires the controller-resolved master address + the connector; the transfer-engine defaults (P2P-handshake metadata) cover the simplest deployment, and operators provide a config file for a real RDMA / HTTP-metadata setup. A kind reference stack that validates the end-to-end Mooncake deployment shape (the A2-equivalent of the LMCache reference stack) is a tracked follow-up. The master image entrypoint + RPC/metadata/metrics ports are now confirmed on a live cluster; until that stack lands, treat the extra_config transfer-engine defaults and the full end-to-end deployment shape as not-yet-cluster-validated.

Status

Field	Type	Purpose
endpoint	string	Observed endpoint clients should use. For External ownership this mirrors spec.remoteStorage.endpoint; for Managed remote storage it is populated from the controller-rendered Service. It stays empty for host-only and events-only backends because neither has a remote provider address to publish.
matchedEnginePods	integer	Snapshot count, at the last reconcile, of pods in the CacheBackend's namespace whose labels satisfy spec.engineSelector. Pointer in Go so nil ("not yet computed") is distinguishable from an observed 0 ("computed and zero pods match"). Refreshed at reconcile cadence — not a real-time per-pod counter. The steady cadence is 30s; during known churn the reconciler uses a conditional 5s cadence when the observed matching Pod count differs from the desired replica sum of Deployments whose pod-template labels match the selector. This keeps the no-Pod-watch design while reducing stale operator output during rolling restarts. The field stays nil when no claim-capable selector is configured — both when spec.engineSelector is absent AND when spec.engineSelector.matchLabels is present but empty (the webhook treats an empty match map as no-claim by design, so the count is no-claim too). A CR that previously had a non-empty selector and just lost it gets its prior value cleared back to nil so the printer column does not advertise a stale match.
engineSelectorMessage	string	Operator-facing diagnosis for selector drift. Set when spec.engineSelector.matchLabels is configured and matchedEnginePods is observed as 0 while engine pods are expected; the message echoes the selector (spec.engineSelector.matchLabels={...}) and states that no Pods in the namespace match. If the selector matches a Deployment that is intentionally scaled to zero, matchedEnginePods still reports the observed 0, but this message stays empty because no engine pods are expected. Cleared once at least one pod matches, the matching Deployment is scaled to zero, or the selector is removed. The controller also emits a Normal EngineSelectorUnmatched Event when the initial observation is zero, when a previously non-zero match count transitions to zero, or when upgrading an existing zero-count status that did not yet have the diagnostic message; steady-state zero with an unchanged message does not re-emit.
failOpen	boolean	Observed echo of the effective spec.integration.failOpen. Represented as a pointer in Go so an explicit false is serialized and operators can read the current mode from status alone.
indexParticipation	object	Per-backend slice of the cluster-wide cache index, projected from the server's /snapshot by grouping replicas by owning CacheBackend. Populated by the CacheIndex poller (status-only). Object is unset until the poller has observed a successful scrape that names the backend's replicas (see Index Participation).
firstKVEventObservedAt	time	Write-once latch: the first time the KV-event readiness gate observed indexParticipation.lastEventAt populated. This is the durable "have we EVER seen a KV event" signal — lastEventAt itself is a current-view projection the poller legitimately clears when a backend's replicas drain, so reading it alone would let a backend that already passed the gate regress. Set write-once by the controller and never cleared (a monotonic marker). It is inert while the backend is not managed (External / unsupported runtime) and is intentionally left in place there — clearing it would be ineffective anyway, since the preserved poller-owned lastEventAt would immediately re-satisfy the gate on a return to the managed path — so a return to managed stays Ready without re-gating, consistent with the "ever observed" contract.
firstAvailableAt	time	Write-once latch: the stable anchor for the firstEventTimeout clock. For a managed (Offload) backend it latches the first time the managed cache-backend workload was observed Available — used instead of the live Deployment Available condition's LastTransitionTime precisely because that resets on an availability flap. For an EventsOnly backend there is no workload to wait on, so it latches at the first reconcile (the clock starts immediately). Anchoring on this monotonic value keeps the elapsed window growing WITHIN a serving mode, so once a backend breaches the timeout (Degraded/NoKVEventsObserved) a later flap cannot bounce it back to AwaitingFirstKVEvent — it stays Degraded until an event arrives. It is stable across flaps and a recreated managed Deployment, but NOT across a mode change: a server-bearing→EventsOnly flip re-anchors it to the flip moment (and bypasses the sticky NoKVEventsObserved reason) so the flip gets a fresh first-event window rather than inheriting the old mode's availability time or timed-out verdict, and an unmanaged transition clears it so a later re-entry starts fresh.
observedServerInstance	string	The controller's cascade-decision baseline — a stable identifier for the Ready cache-server pod set the controller last anchored against. NOT a live "current pod set" view: it is intentionally pinned through transient rolling-update midpoints and through no-Ready windows so the cascade does not fire on rollbacks or transient outages. For the current matched-pod inventory operators should consult status.matchedEnginePods (engine side) and kubectl get pod (cache-server side). Shape: <pod-uid>:<restart-sum> per Ready pod, comma-joined and lex-sorted by pod name; <restart-sum> sums pod.status.containerStatuses[].RestartCount filtered to the cache-server's own containers (the container names declared on the owned Deployment's pod template). Sidecars injected by other admission webhooks (service-mesh proxies, Datadog, etc.) appear in containerStatuses but are absent from the template and are intentionally excluded — a sidecar crash-loop must not advance the identifier and roll the engine fleet. An in-place restart of a cache-server container (kubelet respawning a crashed container — OOM with restartPolicy=Always reuses pod.UID) DOES advance the identifier and is observable. On a transition that reflects an actual replacement (a prior pod is gone, or a persisting pod's restart-sum advanced — NOT a rolling-update strict-superset midpoint) the reconciler cascade-restarts every engine Deployment that owns pods carrying this backend's inferencecache.io/injected-by + matching injected-by-uid, by patching inferencecache.io/cache-server-restart-trigger onto each Deployment's pod template — the same mechanism kubectl rollout restart uses. Rate-limited to once per ~30s per backend. Empty until the first Ready pod; empty→set never cascades (there is no prior server-instance to invalidate, so any engines that connected during the empty window are connecting to the very pod now being baselined). Strict-superset transitions are persisted as the new baseline ONLY when the owning Deployment is converged (spec.replicas == status.readyReplicas == status.updatedReplicas == len(live Ready pods) AND observedGeneration >= metadata.generation — the live-count clause cross-checks the Deployment's reported state against the pod list this reconcile actually saw, so a stale status.readyReplicas=1 while two pods are mid-rollout cannot fake convergence) — a converged steady-state widening is an operator-driven scale-up, so the added pods must enter the baseline or a later replacement of just an added pod would still look like a strict superset and miss the cascade. Strict-superset transitions where the Deployment is NOT converged are rolling-update midpoints; persisting them would let a rolled-back rollout (new pod briefly Ready, then killed, leaving the original pod alone) look like "the new pod was replaced" and false-cascade. Stale-while-unavailable: when no Ready cache-server pod exists at all (Deployment scaled to 0, mid-rollout, image-pull stuck), this field intentionally retains its prior value rather than clearing — clearing would turn the eventual recovery's "" → new-uid:0 transition back into a first-observation baseline and silently skip the cascade, defeating the controller's purpose. Inert and cleared on every transition out of the managed-provider path: External ownership, host-only caching, events-only mode, and unsupported runtimes. The in-process cascade shadow is wiped alongside the field on each of these paths so a later return to a managed provider starts from a clean baseline rather than the prior-period UID. Operator-side recovery for the upstream LMCache LMServerConnector EPIPE-on-restart bug — see LMCache/LMCache#3565.
observedGeneration	integer	The .metadata.generation last reconciled by the controller. Lets clients tell whether the observed status reflects the current spec.
conditions	array	Kubernetes conditions keyed by type. See Conditions.

Conditions

The set of published condition types depends on the backend's integration mode and type:

• Offload-managed backends (spec.integration.mode=Offload on a managed type, where the controller renders a Deployment + Service) publish up to seven: Ready, Degraded, Progressing, FunctionalProbeOK, EngineKernelsHealthy (when a matched engine pod runs the lmcache kernel-check), T2Degraded (once a tier-2/LMCache backend has been exercised), and EngineCompatibility (when an injected engine pod is observed crash-looping after connector injection).
• Host-only backends (resources with no spec.remoteStorage) publish Ready, Degraded, and Progressing, plus the engine-side advisory conditions when applicable. Their endpoint stays empty. HostOnlyActive is the base Ready=True reason before the KV-event gate overlays AwaitingFirstKVEvent, KVEventsObserved, or NoKVEventsObserved.
• Events-only backends (spec.integration.mode=EventsOnly) publish exactly three: Ready, Degraded, Progressing. FunctionalProbeOK, T2Degraded, EngineKernelsHealthy, and EngineCompatibility are Offload-managed-only and are never published on an events-only backend — there is no provisioned server to functionally probe, no tier-2 offload to mark degraded, no LMCache native kernels to check, and no injected KV connector that could be incompatible (events-only injects none); an Offload→EventsOnly flip clears all four (see Events-only mode).
• Externally owned remote storage publishes Ready + Progressing only (there is no rollout to degrade and no probe to drive; the operator manages the provider out-of-band and the controller only validates and mirrors the endpoint).

The Ready / Degraded / Progressing semantics below apply to both Offload-managed and events-only backends (an events-only backend has no workload to roll out, so it is "up" the moment it exists and the KV-event gate starts immediately — see Events-only mode); the FunctionalProbeOK, T2Degraded, EngineKernelsHealthy, and EngineCompatibility rows are Offload-managed-only.

Managed backends (Offload-managed; the FunctionalProbeOK / T2Degraded / EngineKernelsHealthy / EngineCompatibility rows do not apply to events-only):

Type	Meaning
Ready	True once the backend Deployment has rolled out its current generation, has enough updated + available replicas to serve traffic, and — when the KV-event readiness gate applies — at least one KV event has been observed for the backend (reason KVEventsObserved), and — when the functional-probe gate applies — the most recent probe call succeeded across every stage the backend runs. Workload Available but no event yet is Ready=False, reason AwaitingFirstKVEvent. Workload Available and KV-event observed but the probe reported a stage failure is Ready=False with reason ProbeIngestFailed / ProbeRoutingFailed / ProbeT2Failed. Deployment-level reasons: BackendReady (both gates disabled and Available), RolloutInProgress, ScaledToZero, ReplicasUnavailable. And — when a matched engine pod admitted in strict kernel-check mode reports a kernel load failure — Ready=False with reason EngineKernelDegraded (see EngineKernelsHealthy); report-only mode never downgrades Ready. The BackendDegraded / BackendRecovered Events narrate the ReplicasUnavailable → BackendReady / KVEventsObserved transitions.
Degraded	True when the backend is in a terminal unhealthy state: rolled out but replicas unavailable (reason ReplicasUnavailable), or the managed workload is Available but no KV event observed within firstEventTimeout (reason NoKVEventsObserved). False (NotDegraded) otherwise. The functional-probe gate does NOT participate in Degraded — a probe failure is reflected only in Ready and FunctionalProbeOK, leaving Degraded reserved for managed-Deployment health (so an operator can tell "the probe says the cache plane is broken" apart from "the workload itself is in a terminal state").
Progressing	True while the controller is still driving the live state toward the desired state (rollout in flight, first apply, awaiting first KV event). False once converged (Synced), stuck (Degraded), or scaled to zero (ScaledToZero). The pair (Ready=False, Progressing=True) means "still converging"; (Ready=False, Progressing=False) means "stuck/degraded" (or scaled to zero).
T2Degraded	Advisory tier-2 (external offload, e.g. LMCache) health, derived from status.indexParticipation.t2HitRate (written by the CacheIndex poller). Published only once the tier has been exercised (external lookups observed): True/T2ZeroHitRate when it was queried but served zero reloads (wired but useless — a store/connection failure, an under-sized remote server, or a scheduler/worker hash mismatch); False/T2Serving when it is serving reloads (hit-rate > 0). Absent entirely until the tier is exercised (distinct from False). It never gates Ready — tier-2 is an optimization, not a serving dependency (fail-open). For Prometheus alerting use the inferencecache_backend_t2_hit_rate{backend} gauge (CR .status is not scraped). The signal is lifetime-cumulative — it flags a tier that has never served a reload (the silent-from-start failures: scheduler/worker hash mismatch, server OOM, version skew); a mid-life regression (served reloads before, now zero) keeps hit-rate > 0 and so does not trip T2Degraded. That windowed case is caught instead by the per-pod LMCacheT2NoHits alert.
FunctionalProbeOK	The most recent functional-probe outcome. True/ProbeOK when every enabled stage (ingest, routing, and — for LMCache — tier-2 put/get) round-tripped; True/ProbeBypassed when the operator opted this CR out via the inferencecache.io/skip-functional-probe: "true" annotation; False/ProbeIngestFailed, False/ProbeRoutingFailed, or False/ProbeT2Failed when the named stage failed, with the server's diagnostic in .message; Unknown/ProbeError when the controller could not reach the server's /probe endpoint at all (transport error, 5xx) AND no prior stage failure existed. Sticky-False: an HTTP error while a False/Probe*Failed is already published preserves the prior failure and keeps Ready downgraded, so a transient server outage cannot fade a known per-stage failure back to Unknown and then to Ready=True. See functional-probe gate.
EngineKernelsHealthy	Engine-side native CUDA-kernel (lmcache c_ops) load health, read from the lmcache-kernel-check init container on matched engine pods. True/KernelsHealthy when the native kernels loaded on every reporting pod; False/KernelLoadFailed when one or more failed to load — a libcudart/CUDA-runtime mismatch (the root cause), a CPU/pure-python build with no compiled extension, or lmcache not importable; the specific cause is in the condition .message. In strict mode a False also downgrades Ready (reason EngineKernelDegraded); Unknown/KernelCheckError when a check terminated without a recognized result; Unknown/KernelCheckPending while a check is still running. Absent when no matched engine pod runs the check (CPU backends, annotation off). Default mode is fail-open observability — it does NOT gate Ready unless strict. See client kernels ↔ image CUDA alignment.
EngineCompatibility	Advisory engine↔connector observation, derived from the live container state of the engine pods this backend injected cache config into. Published False/InjectedEngineCrashLooping only when an injected engine container is in CrashLoopBackOff after the cache plane wired a KV connector — the live observation, not a confirmed root cause. A structural connector incompatibility is a common cause, canonically a hybrid-attention model (Qwen3.6/Next gated-DeltaNet, Mamba/Jamba, Falcon-H, Granite-hybrid, …): vLLM disables its hybrid KV-cache manager the moment any KV connector (LMCache, Mooncake, NIXL) is wired, then fails KV-spec unification at init. But a crash-loop is generic — it can equally be a bad image, command, missing dependency/secret, or OOM — so verify the cause via the engine logs. Absent when no injected engine is stuck. It never gates Ready (the engine is operator-owned; Ready is driven by the managed Deployment + the KV-event gate) — it names an otherwise-silent crash-loop that often sits behind a NoKVEventsObserved Degraded and points at the likely fix. If it is the connector (e.g. a hybrid model), the routing-preserving fix is the connector-less events-only integration — set spec.integration.mode: EventsOnly (kv-events, no offload), the supported remedy for hybrid-attention models. (Do not reach for inferencecache.io/skip-inject: it opts the pod out of cache wiring entirely, the kvevent-subscriber included, so it stops routing rather than preserving it.) An InjectedEngineCrashLooping Warning Event narrates the transition. See supported-model matrix.

When the desired replica count is owned by an HPA (spec.autoscaling set) the controller compares the Ready condition against the HPA-written Deployment spec.replicas rather than the user-set spec.replicas.

Externally owned remote storage:

Resources express this shape with spec.remoteStorage.ownership: External, the selected provider, and spec.remoteStorage.endpoint. There is no Deployment to roll out, so provider-specific endpoint validation is the only readiness signal the controller has. The controller mirrors the trimmed endpoint to status.endpoint and publishes both conditions immediately on every reconcile (the KV-event gate never applies to external ownership):

Type	Status	Reason	Meaning
Ready	True	ExternalEndpointAccepted	The active endpoint field is non-empty and valid for the selected provider. LMCacheServer accepts host:port or lm://host:port; Mooncake accepts host:port or mooncakestore://host:port; Redis accepts bare host:port. A numeric port in 1-65535 is always required, embedded whitespace and URL path/query/fragment components are rejected, and IPv6 must be bracketed. The controller provisions no provider pod for External ownership, so admission acceptance is the readiness signal.
Ready	False	ExternalEndpointMissing	The active endpoint field is empty or whitespace-only. Current admission rejects this, so the state is reachable only for a CR already stored before the webhook was installed. Status reflects the gap loudly rather than dropping the condition.
Ready	False	ExternalEndpointInvalid	The active endpoint is non-empty but fails the selected provider's shape check. Current admission rejects these values; the reason is reachable only for a CR stored before the relevant rule shipped. The message names spec.remoteStorage.endpoint and carries the shape error. The pod webhook applies the same validation and admits the engine pod unwired on failure.
Progressing	False	mirrors Ready's reason	External ownership completes admission immediately — there is no rollout the controller is still driving. Always False; the reason matches Ready (ExternalEndpointAccepted / ExternalEndpointMissing / ExternalEndpointInvalid) so kubectl describe shows a coherent pair.

Reachability of an externally owned endpoint is not probed by the controller; trusting the operator is part of External ownership. A future enhancement could degrade Ready on a probe failure, but that is deliberately out of scope today (fail-soft, never a serving dependency).

kubectl get cachebackend displays a Ready column sourced from status.conditions[?(@.type=="Ready")].status (the standard K8s pattern — operators read readiness through conditions, not through a custom enum field), the observed status.endpoint, a Matched column sourced from status.matchedEnginePods, plus status.indexParticipation.prefixCount (as PREFIXES) and status.indexParticipation.lastEventAt (as LASTEVENT). Managed provider backends therefore show readiness, the endpoint, the operator-actionable engine-fleet count, and live index participation once reconciliation has populated them and the poller has observed a /snapshot tick. An empty Matched cell means the count has not yet been computed (e.g. cold start before the first reconcile) or the CR has no spec.engineSelector configured. Externally owned bindings display the operator-supplied endpoint immediately. Their indexParticipation is typically unset — the operator-managed provider itself has no observation sidecar — but it is not special-cased by ownership: the poller attributes replicas by engine-pod selector/annotation. An externally owned binding whose engine pods run the subscriber therefore projects indexParticipation the same way as a managed binding. The readiness gate still never applies to External ownership (readiness comes from endpoint acceptance), so a populated lastEventAt affects the displayed columns but not readiness.

Supported-model matrix

The cache plane has two levers — routing (driven by kv-events; works on any engine) and the tier-2 offload (driven by a vLLM KV connector). Standard-attention models get both; hybrid-attention models get routing only, because vLLM's KV-connector interface is mutually exclusive with its hybrid KV-cache manager — wiring any connector triggers a fatal KV-spec-unification error at engine init. This is a vLLM-level constraint that applies to any connector (LMCache, Mooncake, NIXL); it is not an LMCache gap, and swapping the backend does not lift it.

Model family	Attention	Routing (T1)	Offload (T2)	Integration
Llama, Qwen3 dense, Mixtral, …	standard	✅	✅	full (connector + subscriber)
Qwen3.6 / Qwen3-Next (gated-DeltaNet)	hybrid	✅ (events-only)	❌	events-only — supported (spec.integration.mode: EventsOnly)
Mamba / Jamba, Falcon-H, Granite-hybrid, KDA	hybrid	✅ (events-only)	❌	events-only — supported (as above)

A hybrid model wired with the full (connector) integration crash-loops at engine init; the controller surfaces the crash-loop as EngineCompatibility=False/InjectedEngineCrashLooping (above) instead of leaving it a silent CrashLoopBackOff. The condition reports the observation, not a proven cause — confirm via the engine logs (a crash-loop can also be a bad image/command/secret/OOM). When it is the connector incompatibility, set spec.integration.mode: EventsOnly — the supported connector-less integration that preserves routing (kv-events, no offload; the controller provisions no server and injects no connector, but still wires the kvevent-subscriber). Avoid inferencecache.io/skip-inject, which opts the pod out of cache wiring entirely (subscriber included) and so drops routing rather than preserving it.

KV-event readiness gate

Ready (for managed backends) means "the managed backend is up and the cache plane is actually receiving engine state" — not merely "the workload rolled out". The reconciler observes the managed cache-backend Deployment it owns (its rollout + Available condition); that proves the backend workload is up but says nothing about whether engine pods are attached and their ZMQ KV-event publishers are publishing. An engine can be serving HTTP while its publisher is silent (mis-configured --kv-events-config, a ZMQ bind failure, or an in-process publisher crash), or no engine pods may be wired to the backend at all — in either case LookupRoute keeps returning NO_HINT for that backend's prefixes while the CR claims everything is fine. The gate makes that silent degradation loud.

The signal source is status.indexParticipation.lastEventAt (written by the CacheIndex poller from engine-pod reports); the reconciler only reads it. Once the managed cache-backend Deployment is Available:

• no event yet, within firstEventTimeout → Ready=False/AwaitingFirstKVEvent, Degraded=False. The timeout clock starts when the backend becomes "up" — for a managed (Offload) backend when its Deployment first reports Available, for an EventsOnly backend (which has no workload) on the first reconcile it is wired — captured in status.firstAvailableAt (a stable anchor, so a later availability flap does not restart the window; see that field's row for the mode-transition re-anchor).
• at least one event ever observed → Ready=True/KVEventsObserved, Degraded=False. An event already present on the first reconcile counts — there is no required transition through AwaitingFirstKVEvent. "Ever observed" is durable: the first observation is latched into status.firstKVEventObservedAt, because the poller's lastEventAt is a current-view value it clears when the backend's replicas drain — without the latch a drained-but-healthy backend would wrongly fall back to AwaitingFirstKVEvent. The gate is a first-event startup probe, not an ongoing liveness check.
• no event by firstEventTimeout → Ready=False/NoKVEventsObserved, Degraded=True/NoKVEventsObserved. Once Degraded it stays Degraded until an event arrives, then transitions to Ready.

The gate is on by default and opt-out per CR with the annotation inferencecache.io/require-kv-events: "false" (alpha soft-rollout knob; an annotation rather than a spec field so it can be retired once the gate is trusted). External ownership is always exempt — the control plane does not own a provider workload to gate, so readiness is determined by accepting the provider-specific spec.remoteStorage.endpoint as described above.

Operator note. If a backend is stuck at Ready=False/AwaitingFirstKVEvent (and then flips to Degraded=True/NoKVEventsObserved after firstEventTimeout), either no engine pods are attached to the backend or the engine's KV-event publisher is mis-configured — check that engine pods are wired to the backend, then the engine's --kv-events-config and that its ZMQ socket bound. In kubectl get cachebackend the Ready column shows False and the LASTEVENT column shows <none>; the specific reason (AwaitingFirstKVEvent / NoKVEventsObserved) and the remediation hint live in the Ready / Degraded conditions, which kubectl describe surfaces along with the NoKVEventsObserved Warning Event.

Functional-probe gate

The KV-event gate confirms that "at least one engine event has flowed into the index"; the functional-probe gate confirms that the cache plane's round-trip is actually working. Past phases have seen silent-failure modes — a tenant_id mismatch between subscriber and lookup that returns NO_HINT for every routed request; a proxy↔server hash-encoding skew that produces 0% PREFIX_MATCH despite well-formed events; a tier-2 client/server version skew that 0-hits across millions of queries — where each gate above said Ready=True while the cache plane was effectively a no-op. The functional probe drives a deterministic synthetic round-trip per CacheBackend and reflects the per-stage result on the CR.

Composition order on a managed CacheBackend's Reconcile is managed-readiness → KV-event gate → functional-probe gate. The probe gate only fires when the upstream KV-event gate would otherwise say Ready=True (a broken upstream can't be diagnosed by a downstream probe, and probing every not-yet-ready backend on every reconcile is pure noise). The signal source is the server-side /probe endpoint the controller POSTs to once per backend per rate-limit window (~30s); the reconciler reads the per-stage outcome (ingest, routing, t2) and translates it into FunctionalProbeOK:

• all stages ok (or skipped) → FunctionalProbeOK=True/ProbeOK, no Ready change. t2 is intentionally skipped on every install today — no T2Prober is registered into the server binary yet, so Stage C reports skipped for every managed backend that runs the gate (the future state where a T2Prober is wired flips this to ok/failed). External backends never reach this evaluation at all — they are wholly exempt from the functional-probe gate per the per-CR exemption noted below — so the True/ProbeOK outcome is only ever published on managed backends.
• stage failed → FunctionalProbeOK=False with reason ProbeIngestFailed / ProbeRoutingFailed / ProbeT2Failed and the server's stage diagnostic in .message; Ready is downgraded to False with the same reason so the operator-visible Ready signal points at the broken layer.
• transport / HTTP error → behavior depends on whether a prior failure exists on this backend:
  • If FunctionalProbeOK is absent OR True, write FunctionalProbeOK=Unknown/ProbeError and leave Ready alone. A brief server outage should not flap every backend Ready=False — the snapshot poller / policy pusher use the same noise-avoidance posture, and Unknown is the operator's signal to investigate the probe wiring itself rather than the backends.
  • If FunctionalProbeOK is already False/Probe*Failed (a prior stage failure), preserve the existing condition AND keep Ready=False with the prior failure's reason (sticky-False). Letting an HTTP error fade a known per-stage failure to Unknown (and therefore back to Ready=True) would mask a real regression every time the server happens to be transiently unreachable. The False condition is sticky until a successful probe explicitly resolves it.
• rate-limited reconcile → no probe call. If the existing FunctionalProbeOK is already False, the Ready downgrade is re-applied so the status patch doesn't silently overwrite the prior failure with the upstream KV gate's Ready=True. The reconciler also schedules a requeue at the rate-limit window expiry so a quiet stuck backend re-probes even without external watch events.

The gate is on by default when the controller is wired with a --server-probe-url. The operator escape hatch is the annotation inferencecache.io/skip-functional-probe: "true" (alpha soft-rollout knob; annotation not spec field so it can be retired once the gate is trusted): when set, the probe call is skipped entirely and FunctionalProbeOK=True/ProbeBypassed is published — the Ready gate does not downgrade. Disabling the controller-side gate entirely is achieved by passing --server-probe-url="" to the controller binary; the CacheBackend reconciler then never calls the endpoint and clears any stale FunctionalProbeOK condition the next time it processes the CR. External ownership is always exempt — there is no managed Deployment for the gate to compose with, and the controller does not drive a cache-plane round trip for an operator-managed endpoint.

Operator note. If a backend is stuck at Ready=False/Probe*Failed, read the condition's .message — the server populates a stage-specific diagnostic (e.g. "synthesized event not in index — in-process index ingest path is broken"). For ProbeIngestFailed the cache-server's ingest path is broken; for ProbeRoutingFailed the index routing / key-derivation layer is broken; for ProbeT2Failed the configured tier-2 backend rejected the put or returned nothing on the get. FunctionalProbeOK=Unknown/ProbeError means the controller could not reach /probe at all — check that --server-probe-url is reachable from the controller pod (intra-cluster inference-cache-server:8081 by default), that the projected SA token is mounted (/var/run/secrets/inferencecache.io/controller-token/token), and that the audience-bound TokenReview accepts the controller SA.

Index Participation

Field	Type	Purpose
indexParticipation.prefixCount	integer	Sum of distinct prefix entries currently attributed to this backend's replicas. 0 is a valid observed value (the backend is up but holds no warm prefixes yet); always serialized.
indexParticipation.lastEventAt	time	Most recent KV-event timestamp observed for any of this backend's replicas. Unset until the first event arrives; readiness gates must treat the absent value as "not yet observed" rather than epoch.
indexParticipation.hitRate	string	Prefix-count-weighted cache hit rate across this backend's replicas, formatted as a decimal in [0,1]. Always unset today — a missing value MUST NOT be interpreted as 0. The per-replica snapshot now carries a stats-reported presence bit, but the per-backend view aggregates many replicas onto one backend with no defined backend-level hit-rate reduction (mean? token-weighted? over which replicas?), so backend hit-rate aggregation is deliberately deferred to a follow-up; that presence bit is consumed only by the cluster-aggregate CacheIndex.status. Do not expect this field to begin populating from the presence-bit change alone.
indexParticipation.t2HitRate	string	Query-weighted reload hit-rate of the tier-2 (external offload, e.g. LMCache) cache across this backend's replicas, as a decimal in [0,1]. Sourced from the engines' vllm:external_prefix_cache_{hits,queries}_total. Presence is load-bearing: unset means tier-2 has not been exercised (no external lookups across any replica) — distinct from "0", which means the tier WAS queried but served zero reloads, i.e. a silently-degraded offload tier (store/connection failure, under-sized remote server, or scheduler/worker hash mismatch). A healthy reusing workload reads well above 0.

The poller attributes each /snapshot.replicas[] entry to a single owning CacheBackend by resolving the engine pod it came from. The subscriber sidecar runs inside the engine pod and reports replica_id = <pod-name>, tenant_id = <pod-namespace>. For each replica the poller:

1. Looks up the engine pod by (tenant, replicaID).
2. If the pod carries the webhook's inferencecache.io/injected-by annotation (stamped as <namespace>/<name>), resolves the owning CacheBackend directly. This is the authoritative wiring signal — the engine container was wired to exactly that backend's endpoint.
3. Otherwise, iterates that namespace's CacheBackends sorted by metadata.name and picks the first whose spec.engineSelector.matchLabels is non-empty and is a subset of the pod's labels. This mirrors the pod webhook's first-match rule for pods that bypassed the webhook (manual sidecar attachment, opt-out).

Only ONE CacheBackend ever claims a given replica — overlapping selectors must agree on which backend owns the pod, otherwise status would disagree with what the engine was actually wired to. A CacheBackend without an EngineSelector (or with empty MatchLabels) is excluded from the selector fallback — otherwise a misconfigured backend would silently claim every replica in its namespace by vacuous truth — but a pod can still be attributed to it via the injected-by annotation. A replica whose pod can no longer be found (drained between events and now) is skipped; its data still appears in the cluster-wide CacheIndex. A failing scrape preserves existing state (soft-state); a successful scrape that finds no matching replicas resets prefixCount to 0 so stale positive values do not survive a drain.

Contract Notes

• Lookup paths fail open by default. spec.integration.failOpen defaults to true and the engine adapter MUST fall back to local prefill on unreachability of a remote/shared cache tier — that tier is an optimization, never a serving dependency. One pair-specific exception applies to the co-scheduled component of the SGLang MP wire: (sglang, LMCache) has no cacheless engine path while --enable-lmcache is on, so its in-pod MP worker is a serving prerequisite (fail-open is still honored at the tier that can be unavailable — the shared Redis L2, which degrades to L1-only). See the integration.failOpen row above and the fail-open semantics in sglang-lmcache-mp-mode.md. Operators may opt into fail-closed serving by setting failOpen: false, which is loud and visible: the controller emits a Warning FailClosedEnabled Event on the CacheBackend to make it explicit that the cache has been promoted to a serving dependency.
• The controller emits Events on the CacheBackend only on meaningful state changes, never on steady-state reconciles. Condition-transition-keyed Events: BackendDegraded (Warning) on entering Conditions[Degraded]=True with reason ReplicasUnavailable (the KV-event-gate NoKVEventsObserved flavor is suppressed — it carries its own event), BackendRecovered (Normal) on the transition back to Ready=True (similarly suppressed when recovering from NoKVEventsObserved, which carries its own KVEventsObserved event); the FailClosedEnabled / FailOpenRestored pair above; the KV-event readiness gate's AwaitingFirstKVEvent (Normal), KVEventsObserved (Normal), and NoKVEventsObserved (Warning); EngineSelectorUnmatched (Normal) when a configured selector first observes zero matching pods while engine pods are expected, transitions from matched to zero, or gains the diagnostic message during an upgrade from an older zero-count status. One advisory Event is recorded on the CacheBackend but triggered by engine-pod state rather than a CacheBackend condition transition: InjectedEngineCrashLooping (Warning) is emitted once when an injected engine pod's engine container is first observed in CrashLoopBackOff after connector injection — commonly a connector incompatibility (esp. a hybrid-attention model), surfaced as EngineCompatibility=False/InjectedEngineCrashLooping, but a crash-loop can also be a bad image/command/secret/OOM, so the cause is verified via the engine logs, not asserted by the Event. The controller does not watch engine pod status — it detects this on the next CacheBackend reconcile that lists the pods, so the Event reflects observation time, not the instant the container entered CrashLoopBackOff; a transient pod-list failure preserves the prior condition rather than re-firing it.
• A Normal InjectedByCacheBackend Event is emitted on engine pods the mutating webhook stamps with both inferencecache.io/injected-by AND inferencecache.io/injected-by-uid, where the UID annotation matches the live CacheBackend's metadata.uid at reconcile time. The controller deliberately skips emission when (a) the named CR cannot be looked up (NotFound), (b) the UID annotation is absent (failurePolicy=Ignore forgery shape), or (c) the UID does not match the live CR (forgery or CR was recreated under the same name). Non-NotFound lookup errors surface as reconcile errors so controller-runtime retries with backoff. A pod explicitly opted out with a truthy inferencecache.io/skip-inject is instead stamped with inferencecache.io/inject-skipped: skip-inject-annotation; the same post-create controller emits a Normal SkippedByOperator Event only when both the truthy opt-out annotation and the webhook's skipped marker are present. The Events are recorded by a Pod-watching controller, not by the webhook itself: at mutating-admission time the apiserver hasn't assigned metadata.uid to the pod yet, so an event recorded from the webhook would carry involvedObject.uid="" and be invisible to describe (which filters events by UID). Routing the emission through a post-create controller is what guarantees the event reaches the user-visible surface. There is no NoMatchingCacheBackend Event; the no-match signals are status.matchedEnginePods == 0, status.engineSelectorMessage, and EngineSelectorUnmatched on the CacheBackend.
• Optional nested specs are pointer fields in Go so omitted objects stay absent in JSON and server-side apply does not claim empty nested objects. spec.integration is the deliberate exception — the defaulting webhook materialises it on admission, derives engine from canonical spec.runtime (or uses legacy vllm), and gives the nested schema-level defaults a parent object to apply to. The apiserver then applies the +kubebuilder:default= markers on mode (Offload), role (ReadWrite), failOpen (true), and firstEventTimeout (5m) before persisting the CR. Operators reading the persisted CR therefore see the effective compatibility fields explicitly. The IntegrationFailOpen reader helper still exists (nil spec or nil field ⇒ true) as defence-in-depth for callers that bypass admission. Other optional nested specs (spec.autoscaling, spec.template, spec.engineSelector) are NOT materialised — omitted means absent. Webhook-stamped and apiserver-stamped fields are owned by their respective field managers, not the operator's SSA apply, so SSA semantics for operator-set fields are unaffected.

Admission

The controller serves two webhooks for CacheBackend, both registered as failurePolicy: Fail with sideEffects: None on CREATE and UPDATE. Webhook serving requires cert-manager (see README "Cluster Prerequisites").

Defaulting (mutating)

Most Phase-1 literal defaults ride on +kubebuilder:default= markers stamped by the apiserver before the webhook runs (spec.type=LMCache, spec.deploymentKind=Deployment, spec.replicas=1, spec.integration.mode=Offload, spec.integration.role=ReadWrite, spec.integration.failOpen=true, spec.observation.firstEventTimeout=5m). The webhook handles context-dependent defaults; operator-set values are never clobbered.

Field	Default	Layer
spec.type, spec.deploymentKind, spec.replicas, spec.integration.{mode,role,failOpen}, spec.observation.firstEventTimeout	per-field literals (see field godoc)	+kubebuilder:default= markers — apiserver
spec.observation.firstEventTimeout (when spec.observation is omitted entirely)	5m	webhook materialises spec.observation so the nested marker has a parent object to apply to
spec.autoscaling.minReplicas (FIRST APPLY ONLY, when spec.autoscaling != nil and spec.autoscaling.minReplicas == nil)	= spec.replicas (post-marker-default; skipped when spec.replicas is 0 to avoid violating the schema's Minimum=1)	webhook

The spec.autoscaling.minReplicas default is first-apply only. The defaulter refuses to overwrite a non-nil value, AND once stamped the field is owned by the apiserver field manager, so a subsequent edit to spec.replicas does NOT recompute or move minReplicas. This matches the standard Kubernetes HPA convention that scaling intent flows through HPA fields once an HPA owns the workload — to widen or narrow the autoscaling band post-apply, edit spec.autoscaling.minReplicas directly. (The replicas=0 + autoscaling + nil minReplicas case is rejected at admission rather than defaulted; see the validator table below.)

Validating

Rejects structurally-broken specs that the reconciler cannot do anything useful with, with field-scoped error messages. Multiple violations on a single spec are aggregated into one Invalid status so kubectl prints them together.

Rule	Rejects
Cache hierarchy must be internally consistent	A provider-specific typed block does not match remoteStorage.provider/ownership, lmCache is used with a non-LMCache type, or host-only configuration requests workload autoscaling.
External remote storage requires an endpoint	remoteStorage.ownership=External without remoteStorage.endpoint; managed ownership rejects a user-supplied endpoint.
Engine wire must accept the provider binding	Every (runtime, type) adapter must explicitly implement the remote-binding contract, and admission rejects a binding it does not accept (lm, resp, mooncakestore, or host-only). Native SGLang HiCache accepts only the nil host-only binding; attaching any remoteStorage is rejected.
Provider resources must be valid	Typed provider resource blocks are checked for request/limit relationships, claims, quantities, resource names, extended resources, and hugepage alignment, with errors reported at the selected provider path.
Endpoint ownership is explicit	spec.remoteStorage.endpoint is required for External ownership and rejected for Managed ownership. A managed endpoint always comes from the live Service the controller provisions, so a user-supplied value would be misleading. Whitespace-only values are treated as empty.
Cross-namespace endpoint requires opt-in	spec.remoteStorage.endpoint resolves to a Service in a namespace other than the CacheBackend's, while spec.allowCrossNamespace is false. Crossing the namespace is a tenancy boundary the operator must acknowledge. Bare hostnames, IPs, and unqualified names pass through because no namespace can be inferred.
spec.replicas=0 + autoscaling requires explicit minReplicas	spec.replicas=0 with spec.autoscaling != nil and spec.autoscaling.minReplicas == nil. The defaulter declines to compute minReplicas from a 0 replicas value (it would violate the schema's Minimum=1), so without this rule the apiserver accepts the CR and the reconciler's HPA fallback silently picks 1 — overriding the operator's "scale to zero" intent with no notification. The rejection tells the operator to either set minReplicas explicitly or remove spec.autoscaling to scale to zero unconditionally.
spec.integration.engineOverrides cannot touch reserved args/env	An entry in engineOverrides.args / engineOverrides.suppressArgs matches a leading flag token the adapter declares as ReservedArgs(), or an entry in engineOverrides.env / engineOverrides.suppressEnv matches a name in ReservedEnv(). The rejection names both the offending flag/env and the adapter so the operator can fix the spec rather than wait for the engine to crash. The reserved set is per-adapter (the vLLM+LMCache adapter reserves --kv-transfer-config, VLLM_USE_V1, LMCACHE_REMOTE_URL, INFERENCECACHE_FAIL_OPEN, PYTHONHASHSEED).
Provider resource limits and requests must agree	Under spec.remoteStorage.<provider>.resources, overcommittable resource limits must be ≥ requests; hugepages and extended resources must use equal request/limit values.
Requests-only is rejected for non-overcommittable resources	A hugepage or vendor-prefixed extended resource is present in a provider resources.requests map without a matching limit.
Provider resources.claims is not supported	A selected provider resource block contains Dynamic Resource Allocation claim names, but the renderer does not yet create matching pod-level spec.resourceClaims.
Extended-resource quantities must be integers	A selected provider resource block gives a vendor-prefixed extended resource a fractional value.
Hugepage quantities must align to the page size	A selected provider resource block contains a positive hugepages-<size> quantity that is not a whole multiple of its page size.
Provider resource quantities must be non-negative	A selected provider resources.requests or resources.limits entry is negative.
Provider resource names must be valid	A selected provider resource key is not a valid standard, hugepage, or vendor-prefixed container resource name.
Runtime/cache pair must be supported by an installed adapter	The (runtime, engine-cache type) pair has no registered runtime adapter, so the reconciler cannot observe engine compatibility and the pod webhook would fail open without injecting engine config. The shipping pairs are VLLM/LMCache, SGLang/LMCache, and SGLang/SGLangHiCache; remote provider selection is validated independently through remoteStorage. The registry's SupportedPairs list is included in the field-scoped rejection.
Events-only requires spec.type=LMCache	spec.integration.mode=EventsOnly with any spec.type other than LMCache (the default). Events-only wires no KV connector, so declaring an offload-oriented cache type is contradictory. LMCache supplies the kvevent-subscriber that the routing tier needs. See Events-only mode.
Events-only forbids spec.autoscaling	spec.integration.mode=EventsOnly with spec.autoscaling set. An events-only backend provisions no server workload, so there is nothing to autoscale. Field-scoped to spec.autoscaling.

The structural rules are an ordered, pluggable list (CacheBackendValidator.Rules); the runtime/backend compatibility check runs separately because it needs to consult the shared adapterruntime.Registry rather than just the spec.

ValidateUpdate only rejects violations the update introduces: errors that already existed on the previous object are filtered out so an unrelated edit (a label tweak, an annotation) on a CR admitted under a laxer rule set is not suddenly un-updatable. A kubectl edit that flips a previously-valid field into an invalid one is still rejected, because the violation is then new to the diff. Errors are compared by (Type, Field, BadValue, Detail), so an operator changing one bad endpoint to a different bad endpoint on the same field counts as a fresh violation — the rule still bites when the operator actively edits the bad field.

Breaking API cleanup

Inference-cache has not been formally deployed, so this version does not ship a resource conversion or compatibility reader. Manifests must use spec.runtime, typed spec.lmCache, spec.remoteStorage.<provider>, spec.observation, and provider-owned resources. The removed spec.integration.engine, spec.integration.firstEventTimeout, spec.backendConfig, and top-level spec.resources fields are not part of the served CRD schema.

Engine-injection overrides (spec.integration.engineOverrides)

spec.integration.engineOverrides lets the operator amend the non-reserved args/env the pod-mutating webhook injects into the engine container — without forking an adapter. It is the user-facing seam that today's CPU-vLLM-with-LMCache use case and the SGLang+LMCache adapter reach to tune adapter-injected knobs (chunk size, max model length, serdes) that the canonical injection would otherwise hard-code. The reserved set (per locked decision #5/#6 below) makes this surface unsuitable for turning the integration off: operators who need to skip injection entirely on a pod should use the inferencecache.io/skip-inject annotation instead.

Shape, in corev1 vocabulary:

Field	Type	Behavior
args	[]string	Args added to the engine container, scoped to adapter-owned flags. An entry whose leading flag token matches an adapter-owned canonical arg replaces it; an entry whose token is in neither the adapter-owned set nor the user pod-template is appended; an entry colliding with a user-template flag the adapter did not touch is a silent no-op. Order preserved.
suppressArgs	[]string	Leading flag names the adapter MUST NOT inject. Restricted to the adapter-owned set: a suppress entry that names a user-template flag the adapter did not inject is a silent no-op.
env	[]corev1.EnvVar	Env upserted by Name, scoped to adapter-owned canonical entries. An override of an adapter-owned name wins; a new name (not on the user template) is appended; a name colliding with a user-owned env the adapter did not touch is a silent no-op.
suppressEnv	[]string	Env var Names the adapter MUST NOT inject. Restricted to adapter-owned entries; user-owned env is protected.

The "adapter-owned" set is derived by the webhook at admission time by diffing the engine container's args/env immediately before and after InjectEngineConfig runs. A flag/env is adapter-owned if the adapter added it OR modified an existing value. User pod-template entries the adapter does not touch are protected from CR-driven mutation — the CR can amend the engine integration, but not silently rewrite the engine pod owner's own template.

No command override (the entrypoint stays user-owned). No resources override on the engine container here — engine-pod resources are user-owned via the engine's own pod template, not this CR. Managed provider resources are configured under spec.remoteStorage.<provider>.resources.

The CRD field default is byte-identical to the prior behavior: a CacheBackend with no engineOverrides block renders the same injected patch as before.

Reserved declarations and admission hard-reject

Each KVCacheRuntimeAdapter declares two methods:

• ReservedArgs() []string — leading flag tokens the user MUST NOT override or suppress.
• ReservedEnv() []string — env var names the user MUST NOT override or suppress.

The validating webhook selects the adapter from spec.runtime, then iterates its reserved lists and hard-rejects any engineOverrides.{args,suppressArgs} entry that overlaps ReservedArgs() and any engineOverrides.{env,suppressEnv} entry that overlaps ReservedEnv(). The rejection names the offending flag/env and the adapter. Warning-only would let a user silently un-wire the integration and discover it via a crashed engine; the hard-reject keeps the breadcrumb at admission time.

The vLLM+LMCache adapter (internal/adapters/builtin/runtime/vllm_lmcache.go) reserves the args/env the integration cannot function without:

• ReservedArgs(): --kv-transfer-config (the LMCache connector wiring).
• ReservedEnv(): VLLM_USE_V1 (selects the engine codepath the connector targets), LMCACHE_REMOTE_URL (the resolved cache endpoint), INFERENCECACHE_FAIL_OPEN (mirror of spec.integration.failOpen — overriding it would silently desync the pod from the CR contract), PYTHONHASHSEED (pins the deterministic NONE_HASH so LMCache reload matches under TP>1 — overriding or suppressing it silently 0-hits reload).

The same reserved set applies when the canonical vLLM/LMCache engine cache has an External LMCacheServer binding or a Mooncake binding: the selected runtime adapter still runs the LMCache connector and varies only the structured binding's protocol and endpoint. Admission therefore rejects an override that would remove connector wiring regardless of provider ownership. See Mooncake provider configuration.

The SGLang+LMCache adapter (internal/adapters/builtin/runtime) reserves a different set, because SGLang's engine-side wire is the LMCache MP wire, not the lm:// one (see SGLang engine support): ReservedArgs() = --enable-lmcache, --lmcache-config-file; ReservedEnv() = LMCACHE_USE_EXPERIMENTAL, INFERENCECACHE_FAIL_OPEN. Suppressing --lmcache-config-file un-wires MP mode (the engine aborts at startup without it), hence its reservation. In MP mode the lm:// LMCACHE_REMOTE_URL is neither injected nor reserved, and VLLM_USE_V1 / PYTHONHASHSEED are never injected for SGLang. Reservation is per-adapter precisely so each engine guards only the flags/env its own integration cannot function without.

LMCACHE_CHUNK_SIZE, LMCACHE_REMOTE_SERDE, LMCACHE_LOCAL_CPU, LMCACHE_MAX_LOCAL_CPU_SIZE are deliberately NOT reserved — they are perf/mode tunables the operator may legitimately want to change. Canonical chunk size, serializer, and host-memory capacity use spec.lmCache; engineOverrides.env remains the engine-agnostic seam for explicit environment-level tuning.

Shape rationale (A vs. B)

Two shapes were on the table:

• A — typed K8s vocabulary ([]string args, []corev1.EnvVar env, plus suppression). Chosen.
• B — free-form magic keys (cpuMode: "true", gpuLimit: "0", extraArgs: "..."). Rejected.

A is more general: Mooncake remote bindings, the SGLang adapter, and further engine/backend pairs plug in with no per-adapter free-form schema churn. It keeps the CRD disciplined. B is faster to ship but bakes engine-specific knobs into the CRD, which is the trap an "engine-agnostic backend" surface is meant to avoid.

Residual risk

A user can still set non-reserved values that break the engine in subtle ways the validator can't catch — e.g. --max-model-len 999999999 OOMing the engine, or env that subtly changes vLLM behavior. Mitigations shipped with this surface:

• Field godoc carries a "known-fragile" callout.
• ReservedEnv() mirrors ReservedArgs() for the worst offenders, so the canonical wiring can't be silently un-wired.
• Default samples in config/samples/ exercise the no-override path so a future drift in the adapter's canonical injection breaks them loudly.

Mutating Pod webhook (engine wiring)

A separate mutating admission webhook on corev1/v1.Pod (name: mpod.inferencecache.io) auto-wires user-supplied inference engine pods to the matching CacheBackend across all three lifecycle shapes: controller-managed server backends, operator-managed External endpoints, and engine-local backends such as SGLang HiCache. Operators do not have to hand-edit the adapter-specific args, env, sidecars, volumes, or mounts onto their pod templates. The handler lives in internal/webhook/pod and runs on every Pod CREATE.

Aspect	Behavior
Selection	Lists CacheBackends in the pod's namespace via the manager's APIReader (uncached live client; an informer-cache miss on a freshly-Ready backend would leave the pod permanently unwired since pod CREATE is a one-shot), then matches pod.Labels against each Spec.EngineSelector.MatchLabels. The first matching CacheBackend wins; one with a nil or empty EngineSelector is skipped (a "match-everything" selector would silently claim every pod in the namespace).
Injection	Resolves the runtime adapter via runtime.Registry.Select(runtimeID, cache), resolves spec.remoteStorage independently, and constructs a structured provider Binding{Protocol, Endpoint}. Managed ownership uses status.endpoint from the live Service; External ownership uses the trimmed, provider-validated spec.remoteStorage.endpoint with no fallback to stale status; omitted remoteStorage produces a nil host-only binding. SupportsBinding is part of the required runtime adapter interface, and the webhook passes the binding directly to adapter.InjectEngineConfig, so the adapter selects the LMCache, RESP, or Mooncake engine wire from the binding protocol instead of inferring storage from spec.type. A non-nil binding with a missing endpoint fails open. Events-only skips engine injection because it wires no KV connector and appends only the kvevent-subscriber sidecar. Adapters preserve existing user args/env and make repeat injection idempotent.
Annotations	Stamps TWO annotations on every successfully mutated pod: inferencecache.io/injected-by: <namespace>/<name> (operator-readable identity, shows in kubectl describe pod) AND inferencecache.io/injected-by-uid: <cache.UID> (the matched CR's metadata.uid). Successful injection also clears any stale inferencecache.io/inject-skipped marker. Reads inferencecache.io/skip-inject: <truthy> as an opt-out: the webhook returns Allowed, skips engine wiring, clears any stale injected-by/injected-by-uid pair, and stamps inferencecache.io/inject-skipped: skip-inject-annotation so explicit operator opt-out is distinguishable from selector drift. On all other fail-open returns after the pod is decoded (list/no match/missing endpoint/adapter errors), the webhook strips stale injected-by/injected-by-uid and inject-skipped annotations so a user cannot trick the events controller by pre-stamping a pod template. Decode failures fail open before a Pod exists to patch, so stale annotations cannot be cleared on that path.
Events	The webhook itself does NOT record events (the apiserver assigns metadata.uid after mutating admission, so a webhook-recorded event would carry involvedObject.uid="" and be invisible to kubectl describe pod). Instead, the pod-watching engine-pod-events controller reads the persisted decision annotations after CREATE. For injected pods, it validates inferencecache.io/injected-by-uid against the live CR's metadata.uid and records a Normal InjectedByCacheBackend event on the now-persisted pod. For explicitly skipped pods carrying both a truthy inferencecache.io/skip-inject and inferencecache.io/inject-skipped: skip-inject-annotation, it records a Normal SkippedByOperator event on that pod. The skip marker is not authenticated, and skipInjection treats a pre-existing correct marker as already converged; SkippedByOperator therefore means the persisted pod carries the explicit opt-out plus skipped marker, not proof that the webhook authored the marker. The UID match REDUCES — but does NOT eliminate — the failurePolicy=Ignore forgery surface for injected pods: a casual copy-paste of an injected pod's annotations into a fresh template won't match the live CR's UID, but metadata.uid is not secret, so a pod creator with get RBAC on CacheBackends can read it and stamp the pair correctly. The injected Event signals "the webhook claims this pod was injected and the claim is consistent with the live CR," not "the webhook was cryptographically authenticated." The controller skips the injected event when the CR is missing, the UID annotation is absent, or the UID does not match — see the controller godoc for the full skip table. controller-runtime's EventBroadcaster aggregates duplicates on the apiserver side, so a re-enqueue across controller restarts upserts the existing event rather than spamming.
Idempotency	The handler calls the adapter unconditionally on every admission and trusts the adapter to converge the full injected contract. For LMCache this is env plus the engine-specific required surface — --kv-transfer-config for vLLM; for SGLang --enable-lmcache + --lmcache-config-file plus the MP-worker native sidecar and the shared config / /dev/shm volumes + mounts. Its merge primitives (upsertEnv / upsertArgPair / upsertFlag, and for SGLang adoptContainer / adoptVolume / upsertMountByName) converge on the desired value rather than appending a duplicate. The SGLang adopt* pair additionally distinguishes the adapter's own prior injection (converge) from an operator's object squatting a reserved name (reject → fail-open admit) — see Names the MP wire reserves. Native HiCache validates all reserved arguments against the original pod before mutation, preserves one matching or well-formed operator-supplied value, appends each missing canonical argument once, and rejects conflicts, malformed values, or duplicates without partially changing the pod. Re-admission of a fully-injected pod therefore produces an empty JSON-patch set. Trusting the adapter rather than a handler-side env-presence shortcut avoids the trap where a partially-injected pod is admitted permanently missing the rest of the contract.
Fail-open	Every error path (decode failure, list error, no matching backend, missing status.endpoint, no registered adapter, adapter rejection, re-encode failure) returns admission.Allowed(...) with a reason — webhook errors MUST NOT block engine admission. MutatingWebhookConfiguration.failurePolicy is also pinned to Ignore as a belt-and-suspenders second layer.
Verbs	CREATE only. UPDATE re-admissions to a running pod don't re-inject (and the engine container can't pick up env changes without a restart anyway); UPDATEs to engine pods are rare in this fleet.