Add detection-layer eval fixtures replayed from real cluster states - #12
Open
ericjohanson-langchain wants to merge 1 commit into
Open
ericjohanson-langchain wants to merge 1 commit into
ericjohanson-langchain wants to merge 1 commit into
Conversation
The 31-example dataset in evals/ describes incidents in prose and checks whether the interactive agent reasons about them correctly. It cannot reach the collector, and the collector is where every production bug so far has lived: restart_count treated as a fault signal, stale events about deleted pods, missing utilization and PVC data, an HPA read as saturated because min equals max. None of those are reachable from a scenario string, because none of them involve the agent reasoning. They are detection failures. This adds the missing layer. A fixture is a recorded cluster state replayed through the production _classify_pod, event_is_current, and _format_snapshot, with a hand-written expectation of what the right answer is. Two design choices worth keeping. Relative time, not absolute. Every timestamp is stored as an offset in minutes from capture and reconstructed against a fixed `now`. The classifier is recency-based, so a fixture holding absolute timestamps would silently change verdict as it aged, and a regression test that drifts is worse than none. There is a test asserting two replays at different `now` values agree. Allowlist projection, not redaction. Only the fields the classifier and renderer read are copied out of the Kubernetes objects, so annotations, env values, mounted secret names, and image references cannot reach a fixture by construction. Event messages are dropped wholesale rather than sanitized, since they are free text from arbitrary controllers and can embed hostnames, IPs, and connection strings. This repository is public, and capture_snapshot.py redacts namespace, pod, and node names to stable pseudonyms by default; --no-redact prints a warning. Also extracts event_is_current() out of _collect_cluster_data. The age and deleted-pod checks were inline, which meant a fixture could not exercise them at all: replay copied post-filter events straight through and the first version of this suite passed a stale event. Both the collector and the replay harness now call the same predicate. First fixture reconstructs the 2026-08-11 incident: 10 pods Running and Ready with 18-43 lifetime restarts, every last termination Completed exit 0, plus a 51-minute-old InvalidImageName event for an already-deleted pod. Verified it does what it is for by monkeypatching the pre-fix classifier back in: 0 unhealthy with the current code, 10 with the old, so the fixture fails and names the bug. 7 fixture tests plus the parametrised suite. 171 passing overall. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The 31-example dataset in
evals/describes incidents in prose and checks whether the interactive agent reasons about them correctly. It cannot reach the collector — and the collector is where every production bug so far has lived.restarts >= 5flagging healthy podsOverallSeverityrejectinginfoNone of those involve the agent reasoning. They are detection failures, and detecting them needs real cluster state, replayed.
What a fixture is
A recorded cluster state plus a hand-written expectation, replayed through the production
_classify_pod,event_is_current, and_format_snapshot.Relative time, not absolute. Timestamps are stored as minute offsets from capture and reconstructed against a fixed
now. The classifier is recency-based, so absolute timestamps would make a fixture silently change verdict as it aged. There is a test asserting two replays at differentnowvalues agree.Allowlist projection, not redaction. Only fields the classifier and renderer read are copied out of the Kubernetes objects, so annotations, env values, mounted secret names, and image references cannot reach a fixture by construction. Event messages are dropped wholesale rather than sanitized, since they are free text from arbitrary controllers and can carry hostnames, IPs, and connection strings. This repo is public, so
capture_snapshot.pyredacts namespace, pod, and node names to stable pseudonyms by default and--no-redactprints a warning.A refactor this forced
event_is_current()is now extracted from_collect_cluster_data. The age and deleted-pod checks were inline, which meant a fixture could not exercise them: replay copied post-filter events straight through, and the first version of this suite passed a stale event. Both the collector and the replay harness now call the same predicate.Does it actually catch anything
The first fixture reconstructs the 2026-08-11 incident: 10 pods
RunningandReadywith 18-43 lifetime restarts, every last terminationCompletedexit 0, plus a 51-minute-oldInvalidImageNameevent for an already-deleted pod.Verified by monkeypatching the pre-fix classifier back in:
171 tests passing.
Not in this PR
Capturing further fixtures from the live cluster needs cluster credentials. Auto-recording critical findings to a LangSmith dataset is blocked separately:
LANGSMITH_API_KEYis in neither the cluster secret nor the deployment env, which is why the pod logsLangSmithMissingAPIKeyWarning.