Repository navigation
alerting: derive the infra expected-hosts roster from reporters, not Buildkite - #138
Merged
Merged
Conversation
added 2 commits
September 3, 2026 15:52
The infra scan never completed a single run at production data scale: gpu_temperatures and the gpu side of latest_reports run unbounded DISTINCT ON over gpu_snapshots (8.5M rows), and gpu_temperatures exceeds the 2-minute statement timeout every scan. - gpu_temperatures: only readings from the last 2 days count. Stale readings should not sustain a temperature breach anyway — a host that stops reporting is the unreporting alert's job. - latest_reports (gpu side): bound at 14 days, twice the 7-day retirement age, so retirement aging still sees the last reports it needs while skipping the rest of history. Signed-off-by: Thang Nguyen <thang.nguyen@inferact.ai>
…Buildkite The infra scan's first live minutes paged the channel for subjects that can never report: the Buildkite gpu-queue agents list is dominated by ephemeral k8s pod agents (buildkite-<uuid>-<rand>) and autoscaled EC2 CI instances (bk-cpu-queue-*, ip-10-*.ec2.internal), none of which run host reporters. As 'expected but never reported' subjects they opened unreporting episodes on the first scans after enablement. The roster is now reporter-derived only: optional kubectl node lists plus every hostname seen in gpu_snapshots within the 7-day retirement window (the union already existed in InfraScanHandler). A host that has been reporting and goes silent still pages within ~15-20 minutes; the never-reported detection is dropped because in this fleet it was pure noise. Removes the now-unused BuildkiteGpuQueueAgentsSource, its port, and BuildkiteRestClient.list_agents; the infra consumer no longer requires BUILDKITE_TOKEN. Signed-off-by: Thang Nguyen <thang.nguyen@inferact.ai>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Minutes after enabling infra alerting, the Slack channel was paged for
buildkite-01a068de-…-9bnrwandvllm-ci-cpu-vllm-us-central1-b-*— subjects that can never report.Inspected the live gpu-queue roster from the worker: the first 100 agents are ephemeral k8s pod agents (
buildkite-<uuid>-<rand>) and autoscaled EC2 CI instances (bk-cpu-queue-premerge-…,ip-10-*.ec2.internal) — none of them run host reporters, and the real reporter hosts (dgxb200-*, …) aren't even on the page. As "expected but never reported" subjects they all opened unreporting episodes.(Infra is currently set to
disabledto stop the spam while this lands.)Fix
The roster is now reporter-derived: optional kubectl node lists + every hostname seen in
gpu_snapshotswithin the 7-day retirement window (that union already existed inInfraScanHandler). Alerting semantics are unchanged for real hosts — a reporter that goes silent still pages within ~15–20 min. What goes away is "never reported" detection, which in this fleet was 100% noise.Removed as dead code:
BuildkiteGpuQueueAgentsSource,BuildkiteAgentPort,BuildkiteRestClient.list_agents(theread_agentstoken scope is no longer needed by the worker), and the infra consumer no longer requiresBUILDKITE_TOKEN. Docs updated (alerting/docs/infra.md).Tests
pytest alerting197 passed (2 deleted Buildkite-source tests), ruff clean, mypy shows only the pre-existing local-venv errors (identical without this change).Rollout
After merge: redeploy worker (
install.shfrommain), delete the bogus open pod episodes fromalerting_infra_alerts/alerting_infra_host_states, then setcontrol/infra.modeback tolive.Signed-off-by: Thang Nguyen thang.nguyen@inferact.ai