feat(index): scope base-model evictions via a CacheEvent adapter_scoped flag

docs/design/grpc-contract.md

@@ -62,9 +62,9 @@ service InferenceCache {
 - **ReplicaStats** `{ string replica_id = 1; int64 cache_memory_bytes = 2; float hit_rate = 3; float pressure = 4; string client_version = 5; int64 t2_hit_tokens = 6; int64 t2_query_tokens = 7; }`
   `client_version` is an opaque version string identifying the client-side cache library the reporting replica is linked against (e.g. an LMCache client release). It carries no semver semantics on the wire — producers populate it, the server accepts it, no semver parsing or ordering is performed at this layer. Empty / unset is allowed and means "unknown"; older producers that don't fill the field MUST keep being accepted (additive, v1alpha1-compatible). The field reserves wire space for the producer half of an end-to-end client/server version-skew detection surface; the consumer half (server-side storage, per-CacheBackend exposure, the operator-visible status condition that closes the silent client/server mismatch class of bug) lands in a follow-up change and is intentionally out of scope for this contract update.
   `t2_hit_tokens` / `t2_query_tokens` are cumulative token counters for the **tier-2 (external offload, e.g. LMCache) cache**, sourced from the engine's `vllm:external_prefix_cache_{hits,queries}_total`. The server derives a presence-aware tier-2 hit-rate per CacheBackend (`status.indexParticipation.t2HitRate`): nil until the tier is exercised (`t2_query_tokens > 0`), then the query-weighted `hits/queries` ratio — so a value of `0` means tier-2 is configured but serving no reloads (a silently-degraded offload tier), distinct from "not yet used". Both default to 0 for producers that don't populate them (additive, v1alpha1-compatible).
-- **CacheEvent** `{ Type type; string replica_id; string model_id; string tenant_id; bytes prefix_hash; int64 timestamp_us; string adapter_id = 7; }`
+- **CacheEvent** `{ Type type; string replica_id; string model_id; string tenant_id; bytes prefix_hash; int64 timestamp_us; string adapter_id = 7; bool adapter_scoped = 8; }`
   Type ∈ `PREFIX_ADDED | PREFIX_EVICTED | REPLICA_UPDATED | ALL_CLEARED`
-  `adapter_id` narrows a `PREFIX_EVICTED` to one adapter partition; empty removes the prefix across **every** partition (the conservative legacy behavior — removal is soft state). Ignored by `ALL_CLEARED` (a flush clears the replica across adapters) and `REPLICA_UPDATED` (liveness is adapter-independent).
+  `adapter_id` narrows a `PREFIX_EVICTED` to one adapter partition **when `adapter_scoped` is set** (so `""` targets the base-model partition, not a sweep) **or when `adapter_id` is non-empty** (a pre-`adapter_scoped` producer naming a LoRA adapter — honored so a mixed-version fleet doesn't regress its narrowing). Only an empty `adapter_id` with `adapter_scoped` unset removes the prefix across **every** partition (the conservative legacy sweep — removal is soft state). Ignored by `ALL_CLEARED` (a flush clears the replica across adapters) and `REPLICA_UPDATED` (liveness is adapter-independent).
 - **Metric** `{ string name; string type; map<string,string> labels; double value; int64 timestamp_us; }` (Prometheus `inferencecache_*`, tech spec §4.3)
 - **StreamEventsRequest** `{ string model_id; string tenant_id; repeated CacheEvent.Type types; }`
 - **StreamMetricsRequest** `{ repeated string names; }`
@@ -268,7 +268,7 @@ The fix partitions the **index**, not the hash:
 - **The fingerprint construction is unchanged.** Adapter identity is never mixed into the hash, so `pkg/fingerprint`, its golden vectors, and every cross-language implementation (Go / Python / Rust) stay byte-for-byte as they were. The hash still answers "what content is this?"; the partition answers "whose KV is it?".
 - **Matching inside a partition is unchanged.** Exact-match and the block-chain longest-leading-run rule behave identically; the partition only bounds which entries are candidates.
 - `LookupRouteResponse.adapter_id` (5) echoes the partition the index was consulted in, so a caller can confirm the round trip. It is empty on the fail-open envelopes returned *before* the index is consulted (`TIMEOUT`, tokenizer fail-open, `POLICY_REQUIRES_CHAIN`).
-- `CacheEvent.adapter_id` (7) narrows a `PREFIX_EVICTED` to one partition. Without it, one adapter's GPU eviction would drop the identical hash under *every* adapter — discarding hints that are still valid. Empty keeps the original cross-partition sweep.
+- `CacheEvent.adapter_id` (7) narrows a `PREFIX_EVICTED` to one partition, with the new `adapter_scoped` (8) flag distinguishing a base-model eviction from a legacy sweep. The removal scopes to `adapter_id` when `adapter_scoped` is set (so `adapter_id=""` drops only the base partition, no longer discarding another adapter's still-valid hint for the identical hash) **or** when `adapter_id` is non-empty (a pre-`adapter_scoped` producer that named a LoRA adapter — honored so a mixed-version fleet doesn't regress a `sql-lora` eviction into a cross-partition sweep). Only an empty `adapter_id` with `adapter_scoped` unset keeps the conservative cross-partition sweep. A separate bool rather than making `adapter_id` `optional`, which the wire-compat gate (`buf breaking`) rejects as a cardinality change — the bool is purely additive.
 
 **Producer / consumer must agree, exactly as they already do for `hash_scheme`.** A lookup that sets `adapter_id` will not match entries ingested without one, and vice versa. Empty/unset is the **default partition**, and this is exactly the pre-adapter behavior **for base-model / non-LoRA traffic**: an engine serving no LoRA emits a nil `lora_id`, so ingest and lookup both land under `""` and are completely unaffected — no regression, no configuration required. **This is not a "single LoRA adapter just works" claim.** Only a *nil* `lora_id` uses `""`; *any* non-nil `lora_id` — including a deployment with exactly one adapter — must resolve to a **configured** `--lora-adapter-names` identity, otherwise it is dropped at ingest (**fail-closed**) rather than landing under `""` or a replica-local alias. So even one LoRA adapter requires the ingest identifier (the subscriber's resolved `adapter_id`) to match the gateway's query `adapter_id` end-to-end, or every lookup silently misses — the same agreement `hash_scheme` already demands.
 

docs/reference-stack/scripts/default_install_smoke.sh

@@ -1196,6 +1196,25 @@ grpcurl_report_cache_state() {
     2>>"$err_file"
 }
 
+grpcurl_publish_event() {
+  local payload="$1"
+  local err_file="${2:-$LOG_DIR/grpcurl-publish-event.err}"
+  # Default install is plaintext (see grpcurl_lookup_route).
+  if printf '%s\n' "$payload" | grpcurl -plaintext -max-time 5 -d @ \
+       "localhost:$GRPC_LOCAL_PORT" \
+       inferencecache.v1alpha1.InferenceCache/PublishEvent \
+       2>"$err_file"; then
+    return 0
+  fi
+  log "reflection PublishEvent probe failed; falling back to proto-file probe" >&2
+  printf '%s\n' "$payload" | grpcurl -plaintext -max-time 5 \
+    -import-path proto -proto inferencecache/v1alpha1/inferencecache.proto \
+    -d @ \
+    "localhost:$GRPC_LOCAL_PORT" \
+    inferencecache.v1alpha1.InferenceCache/PublishEvent \
+    2>>"$err_file"
+}
+
 has_reason_code() {
   local resp="$1"
   local want="$2"
@@ -1472,6 +1491,86 @@ assert_adapter_no_alias "$adapter_none_payload" "$LOG_DIR/grpcurl-adapter-none.e
   "LookupRoute without adapter_id matched an adapter-scoped prefix — adapter-scoped ingest must stay out of the default partition"
 log "adapter partition enforced end-to-end: same adapter_id → PREFIX_MATCH; a different adapter_id and an absent adapter_id each returned the fail-open AFFINITY_HINT (RPC succeeded, off the prefix-match path) on the identical prefix hash"
 
+# --- adapter-scoped eviction probe (adapter_scoped) ------------------------
+#
+# A base-model PREFIX_EVICTED (adapter_id="" WITH adapter_scoped) must drop ONLY
+# the base partition and leave a co-resident LoRA hint for the SAME token hash
+# intact — the over-sweep the adapter_scoped flag fixes. Driven end-to-end with no
+# engine traffic: seed the identical prefix hash under BOTH the base ("") and an
+# "evict-lora" partition on one replica, PublishEvent a base-scoped eviction, then
+# assert via LookupRoute that the base partition stopped matching while the LoRA
+# partition still returns PREFIX_MATCH.
+log "seeding base + LoRA partitions and asserting a base-scoped PREFIX_EVICTED spares the LoRA hint"
+evict_replica="evict-smoke-replica"
+evict_hash_b64="ZXZpY3QtcHJlZml4" # base64("evict-prefix"); tokenCount=64 clears both policy gates
+
+evict_base_seed="{\"replicaId\":\"$evict_replica\",\"modelId\":\"$adapter_model\",\"tenantId\":\"$POLICY_SMOKE_NS\",\"hashScheme\":\"vllm\",\"prefixes\":[{\"prefixHash\":\"$evict_hash_b64\",\"tokenCount\":64}]}"
+evict_lora_seed="{\"replicaId\":\"$evict_replica\",\"modelId\":\"$adapter_model\",\"tenantId\":\"$POLICY_SMOKE_NS\",\"hashScheme\":\"vllm\",\"prefixes\":[{\"prefixHash\":\"$evict_hash_b64\",\"tokenCount\":64,\"adapterId\":\"evict-lora\"}]}"
+for seed in "$evict_base_seed" "$evict_lora_seed"; do
+  grpcurl_report_cache_state "$seed" "$LOG_DIR/grpcurl-evict-seed.err" >/dev/null || {
+    cat "$LOG_DIR/grpcurl-evict-seed.err" >&2 || true
+    fail "grpcurl ReportCacheState did not accept an eviction-probe seed"
+  }
+done
+
+evict_base_lookup="{\"modelId\":\"$adapter_model\",\"tenantId\":\"$POLICY_SMOKE_NS\",\"hashScheme\":\"vllm\",\"prefixHash\":\"$evict_hash_b64\",\"prefixTokenCount\":64}"
+evict_lora_lookup="{\"modelId\":\"$adapter_model\",\"tenantId\":\"$POLICY_SMOKE_NS\",\"hashScheme\":\"vllm\",\"prefixHash\":\"$evict_hash_b64\",\"prefixTokenCount\":64,\"adapterId\":\"evict-lora\"}"
+
+# Precondition: both partitions match before the eviction. Wait for the seeds to
+# apply (evict-lora), then confirm base is present too — else "base gone" below
+# would pass vacuously.
+deadline=$(($(date +%s) + POLICY_PUSH_TIMEOUT))
+evict_lora_pre=""
+until has_reason_code "$evict_lora_pre" "PREFIX_MATCH"; do
+  evict_lora_pre="$(grpcurl_lookup_route "$evict_lora_lookup" "$LOG_DIR/grpcurl-evict-pre-lora.err")" || true
+  has_reason_code "$evict_lora_pre" "PREFIX_MATCH" && break
+  if [ "$(date +%s)" -ge "$deadline" ]; then
+    echo "evict-lora precondition LookupRoute (want PREFIX_MATCH):" >&2; echo "$evict_lora_pre" >&2
+    fail "eviction-probe precondition: the evict-lora partition never matched within ${POLICY_PUSH_TIMEOUT}s"
+  fi
+  sleep 2
+done
+evict_base_pre="$(grpcurl_lookup_route "$evict_base_lookup" "$LOG_DIR/grpcurl-evict-pre-base.err")" || true
+has_reason_code "$evict_base_pre" "PREFIX_MATCH" || {
+  echo "base precondition LookupRoute (want PREFIX_MATCH):" >&2; echo "$evict_base_pre" >&2
+  fail "eviction-probe precondition: the base partition did not match before the eviction"
+}
+
+# Base-scoped eviction: adapter_scoped=true with an empty adapter_id targets the
+# base partition ONLY (without the flag an empty adapter_id sweeps every partition).
+evict_event="{\"type\":\"PREFIX_EVICTED\",\"replicaId\":\"$evict_replica\",\"modelId\":\"$adapter_model\",\"tenantId\":\"$POLICY_SMOKE_NS\",\"prefixHash\":\"$evict_hash_b64\",\"adapterScoped\":true}"
+evict_ack="$(grpcurl_publish_event "$evict_event" "$LOG_DIR/grpcurl-evict-publish.err")" || {
+  cat "$LOG_DIR/grpcurl-evict-publish.err" >&2 || true
+  fail "grpcurl PublishEvent did not accept the base-scoped PREFIX_EVICTED"
+}
+log "PublishEvent (base-scoped eviction) response: $evict_ack"
+
+# The base partition must stop matching (proves the eviction applied); once it
+# has, the LoRA partition must STILL match (proves it was not over-swept).
+deadline=$(($(date +%s) + POLICY_PUSH_TIMEOUT))
+evict_base_post=""
+while :; do
+  evict_base_post="$(grpcurl_lookup_route "$evict_base_lookup" "$LOG_DIR/grpcurl-evict-post-base.err")" || true
+  if [ -n "$evict_base_post" ] && ! has_reason_code "$evict_base_post" "PREFIX_MATCH"; then
+    break
+  fi
+  if [ "$(date +%s)" -ge "$deadline" ]; then
+    echo "base LookupRoute after eviction (want NOT PREFIX_MATCH):" >&2; echo "$evict_base_post" >&2
+    fail "the base partition was still PREFIX_MATCH ${POLICY_PUSH_TIMEOUT}s after a base-scoped PREFIX_EVICTED — the eviction did not apply"
+  fi
+  sleep 2
+done
+
+evict_lora_post="$(grpcurl_lookup_route "$evict_lora_lookup" "$LOG_DIR/grpcurl-evict-post-lora.err")" || {
+  cat "$LOG_DIR/grpcurl-evict-post-lora.err" >&2 || true
+  fail "grpcurl LookupRoute (evict-lora, post-eviction) failed — a transport error must not mask the over-sweep check"
+}
+has_reason_code "$evict_lora_post" "PREFIX_MATCH" || {
+  echo "evict-lora LookupRoute after the base eviction (want PREFIX_MATCH):" >&2; echo "$evict_lora_post" >&2
+  fail "a base-scoped PREFIX_EVICTED also swept the evict-lora partition — the adapter_scoped over-sweep fix regressed"
+}
+log "adapter_scoped eviction enforced end-to-end: a base-scoped PREFIX_EVICTED dropped the base partition while the co-resident LoRA hint for the identical hash survived"
+
 # --- routingFloorScore end-to-end probe ------------------------------------
 # Proves the new field flows CR → controller flatten → /policy push → server
 # resolver → buildLookupResponse downgrade. The same 64-token prefix that

pkg/adapters/engine/mapper.go

@@ -89,16 +89,20 @@ func (c Config) ClearedEvent(tsSeconds float64) *icpb.CacheEvent {
 // EvictedEvent builds one PREFIX_EVICTED CacheEvent for an already-derived prefix
 // hash (our content fingerprint, the index key to drop) in a specific adapter
 // partition. The subscriber maps an evicted engine block hash to both via
-// positionalIndex.Removed. An empty adapterID is the default partition; the
-// server then falls back to its cross-partition (legacy) removal.
+// positionalIndex.Removed. adapter_scoped is always set, so the server drops the
+// prefix from exactly the adapterID partition (even "" for the base model) — a
+// base-model eviction never sweeps live LoRA hints for the same token hash.
+// (Only a producer that leaves adapter_scoped false AND adapter_id empty gets the
+// conservative cross-partition legacy sweep; a non-empty adapter_id still narrows.)
 func (c Config) EvictedEvent(prefixHash []byte, adapterID string, tsSeconds float64) *icpb.CacheEvent {
 	return &icpb.CacheEvent{
-		Type:        icpb.CacheEvent_PREFIX_EVICTED,
-		ReplicaId:   c.ReplicaID,
-		ModelId:     c.ModelID,
-		TenantId:    c.TenantID,
-		PrefixHash:  prefixHash,
-		AdapterId:   adapterID,
-		TimestampUs: microsFromSeconds(tsSeconds),
+		Type:          icpb.CacheEvent_PREFIX_EVICTED,
+		ReplicaId:     c.ReplicaID,
+		ModelId:       c.ModelID,
+		TenantId:      c.TenantID,
+		PrefixHash:    prefixHash,
+		AdapterId:     adapterID,
+		AdapterScoped: true,
+		TimestampUs:   microsFromSeconds(tsSeconds),
 	}
 }

pkg/adapters/engine/mapper_test.go

@@ -22,17 +22,23 @@ func TestEvictedEvent(t *testing.T) {
 	if !bytes.Equal(e.PrefixHash, []byte{0x01, 0x02}) {
 		t.Errorf("prefix hash = %x, want 0102", e.PrefixHash)
 	}
-	if e.AdapterId != "" {
-		t.Errorf("adapter id = %q, want \"\" (no adapter)", e.AdapterId)
+	// A base-model eviction carries adapter_id="" but marks it authoritative
+	// (adapter_scoped), so the server drops only the base ("") partition instead
+	// of sweeping every adapter.
+	if got := e.GetAdapterId(); got != "" {
+		t.Errorf("adapter id = %q, want \"\" (base partition)", got)
+	}
+	if !e.GetAdapterScoped() {
+		t.Error("adapter_scoped = false; a base eviction must mark adapter_id authoritative to scope removal to the base partition")
 	}
 }
 
 // An adapter-scoped eviction must name its partition, so the server drops the
 // prefix only there — the same hash can be live under another adapter.
 func TestEvictedEventCarriesAdapter(t *testing.T) {
 	e := testConfig().EvictedEvent([]byte{0x01}, "sql-lora", 1.0)
-	if e.AdapterId != "sql-lora" {
-		t.Errorf("adapter id = %q, want sql-lora", e.AdapterId)
+	if e.GetAdapterId() != "sql-lora" || !e.GetAdapterScoped() {
+		t.Errorf("adapter id = %q scoped = %v, want sql-lora / true", e.GetAdapterId(), e.GetAdapterScoped())
 	}
 }
 

pkg/index/adapter_partition_test.go

@@ -188,7 +188,7 @@ func TestAdapterPartitionEvictionIsScopedToItsAdapter(t *testing.T) {
 	idx.ApplyEvent(Event{
 		Type: EventPrefixEvicted, ReplicaID: "replica-0",
 		Model: adapterModel, Tenant: adapterTenant,
-		PrefixHash: hash("same-tokens"), Adapter: "sql-lora",
+		PrefixHash: hash("same-tokens"), Adapter: "sql-lora", AdapterScoped: true,
 	})
 
 	if got := lookupUnder(idx, "sql-lora", "same-tokens"); len(got) != 0 {
@@ -199,6 +199,51 @@ func TestAdapterPartitionEvictionIsScopedToItsAdapter(t *testing.T) {
 	}
 }
 
+// A base-model eviction (Adapter "" WITH AdapterScoped) drops ONLY the base
+// partition and must not sweep a live LoRA hint for the same token hash — the
+// over-sweep that the adapter_scoped flag exists to fix.
+func TestAdapterPartitionBaseEvictionScopedSparesLoRA(t *testing.T) {
+	idx := New()
+	ingestUnder(idx, "replica-0", "", "same-tokens")
+	ingestUnder(idx, "replica-0", "sql-lora", "same-tokens")
+
+	idx.ApplyEvent(Event{
+		Type: EventPrefixEvicted, ReplicaID: "replica-0",
+		Model: adapterModel, Tenant: adapterTenant,
+		PrefixHash: hash("same-tokens"), Adapter: "", AdapterScoped: true,
+	})
+
+	if got := lookupUnder(idx, "", "same-tokens"); len(got) != 0 {
+		t.Errorf("base entry survived its own eviction: %+v", got)
+	}
+	if got := lookupUnder(idx, "sql-lora", "same-tokens"); len(got) != 1 {
+		t.Errorf("sql-lora entry = %+v, want it untouched by the base-model eviction", got)
+	}
+}
+
+// Upgrade compatibility: a pre-adapter_scoped producer (new server, not-yet-
+// upgraded subscriber) sends a non-empty adapter_id but leaves adapter_scoped
+// false. Its LoRA eviction must still narrow to that adapter, not regress into a
+// cross-partition sweep that wipes another adapter's live hint.
+func TestAdapterPartitionEvictionNonEmptyAdapterScopesWithoutFlag(t *testing.T) {
+	idx := New()
+	ingestUnder(idx, "replica-0", "sql-lora", "same-tokens")
+	ingestUnder(idx, "replica-0", "chat-lora", "same-tokens")
+
+	idx.ApplyEvent(Event{
+		Type: EventPrefixEvicted, ReplicaID: "replica-0",
+		Model: adapterModel, Tenant: adapterTenant,
+		PrefixHash: hash("same-tokens"), Adapter: "sql-lora", // AdapterScoped left false (pre-upgrade producer)
+	})
+
+	if got := lookupUnder(idx, "sql-lora", "same-tokens"); len(got) != 0 {
+		t.Errorf("sql-lora entry survived its own eviction: %+v", got)
+	}
+	if got := lookupUnder(idx, "chat-lora", "same-tokens"); len(got) != 1 {
+		t.Errorf("chat-lora entry = %+v, want it untouched — a non-empty adapter_id must narrow even without adapter_scoped", got)
+	}
+}
+
 // An eviction with NO adapter keeps the original conservative behavior: it
 // sweeps every partition. That is what a pre-adapter producer emits, and for
 // such a producer all entries live in the "" partition anyway — so the