Repository navigation
gpu report: host metrics ingestion, temp/power exposure, retention decoupling - #113
Merged
Merged
Conversation
added 3 commits
September 2, 2026 19:15
Add the host-metrics half of the GPU report contract: an optional host object (cpu_util/cpu_count, RAM byte triple, per-mount disks with role classification), reporter_status/last_error, and Kubernetes node_conditions. Legacy GPU-only payloads remain valid. - migration 0018: host_snapshots + host_history_5m (raw + 5-minute rollup, mirroring the GPU tables) and alert_thresholds seeded with the unreporting/disk/gpu-temperature defaults; all RLS-protected - storage: one transaction writes GPU rows, host snapshot, and both rollups; hostnames normalized to lowercase on ingest - route: fail-closed auth (unset GPU_REPORT_SECRET is a 503 configuration error, not open access), strict payload validation with per-field ranges, and a streaming body cap (GPU_REPORT_MAX_BYTES, 256 KiB default, 4 MiB ceiling) - contract tests at the route seam; migrations tests assert the new tables, seed rows, and RLS grants Signed-off-by: Thang Nguyen <thang.nguyen@inferact.ai>
Surface the already-stored GPU temperature and power on the /gpu host table (own columns, threshold-colored temp, per-GPU values in the existing memory-bar tooltips) and replace the binary offline badge with the fleet vocabulary: Stale after 5 minutes without a report, Unreporting after 10, matching the alerting condition. queryGpuLatest now selects temperature_c/power_draw_w/power_limit_w and the host row model carries maxTemperatureC and aggregate power fields. Signed-off-by: Thang Nguyen <thang.nguyen@inferact.ai>
GPU raw-snapshot cleanup previously ran inline in poll-metrics, so a Buildkite queue-poll failure would silently stop GPU retention too. Move it to a dedicated /api/cron/retention Vercel cron (daily) that also covers the new host_snapshots table. Raw rows keep 30 days on both families; the 5-minute rollups are kept forever as the long-range history source. queue_snapshots cleanup stays with its own poll. Signed-off-by: Thang Nguyen <thang.nguyen@inferact.ai>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
This was referenced Sep 3, 2026
Merged
…t env docs Signed-off-by: Thang Nguyen <thang.nguyen@inferact.ai>
Thangnguyenvn98
force-pushed
the
agent/gpu-temp-power-badges
branch
from
September 3, 2026 02:07
4655f66 to
c461c9e
Compare
added 3 commits
September 2, 2026 22:11
Signed-off-by: Thang Nguyen <thang.nguyen@inferact.ai> Co-authored-by: Thang Nguyen <thang.nguyen@inferact.ai>
…nfra tab + per-alert docs (#115) Signed-off-by: Thang Nguyen <thang.nguyen@inferact.ai> Co-authored-by: Thang Nguyen <thang.nguyen@inferact.ai>
…fix, and pytest coverage (#116) Signed-off-by: Thang Nguyen <thang.nguyen@inferact.ai> Co-authored-by: Thang Nguyen <thang.nguyen@inferact.ai>
Thangnguyenvn98
force-pushed
the
agent/gpu-temp-power-badges
branch
from
September 3, 2026 02:12
c461c9e to
3f22258
Compare
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Foundation PR of the GPU infra-health stack (spec: wayfinding trail in
~/.scratch/vllm-gpu-infra-health/). Three commits:hostobject (CPU util/count, RAM bytes, per-mount disks with role classification),reporter_status/last_error, and k8snode_conditions. New tableshost_snapshots+host_history_5m(raw + 5-min rollup, mirroring the GPU pattern) andalert_thresholdsseeded with unreporting=10min / disk=90% / temp=85°C — thresholds are tuned by editing rows in Supabase, no redeploy. Legacy GPU-only payloads stay valid so old reporters keep working during rollout./api/cron/retentionthat also covershost_snapshots(raw 30 days; rollups kept forever).Hardening included
Ingestion auth is fail-closed (unset
GPU_REPORT_SECRET→ 503, not open access), payloads are strictly validated with a streaming body cap (GPU_REPORT_MAX_BYTES, 256 KiB default), and hostnames are lowercased on ingest.Test plan
npm test74/74Deploy notes
Migration
0018_gpu_host_ingest.sqlis already applied to production. Merge order: after #112 (independent), before the stacked host-UI / infra-alerting / reporters PRs.