Repository navigation
feat(gpu): link GPU hosts to Buildkite queues and running jobs - #132
Merged
Merged
Conversation
Sample connected Buildkite agents in the 5-minute poll-metrics cron and persist hostname, queue tags, and the current job to a new buildkite_agent_snapshots table (migration 0018). A new GET /api/gpu/agents endpoint joins GPU hosts to agents by normalized hostname (lowercased, domain suffix stripped), and the /gpu Host Summary renders the queue and current job with a Buildkite link under each host name.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Resolve conflicts with the GPU host-health restructure (#135): - gpu-dashboard.tsx: keep upstream's GpuHostTable/buildHostRows and pass the /api/gpu/agents map through as an optional prop instead of the old inline Host Summary table. - gpu-host-table.tsx: optional agents prop renders 'queue . job' with a Buildkite link under the host name. - poll-metrics: keep agent sampling; move buildkite_agent_snapshots retention into the dedicated /api/cron/retention task per upstream's decoupling of raw-snapshot cleanup. - Renumber the agent-snapshots migration to 0020 (upstream took 0018 and 0019).
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
/gpushows hosts likedgxb200-01at high memory/utilization with no way to see which queue the host serves or which job is running on it. This wires GPU hosts to their Buildkite agents:poll-metricscron now also fetches connected agents from the Buildkite GraphQL API and persists one row per agent (hostname,queue=meta-data tags, current job id/label/build number/url) into a newbuildkite_agent_snapshotstable, with 30-day retention matchingqueue_snapshots.migrations/sql/0020_buildkite_agent_snapshots.sql. Like other dashboard tables it is intentionally left out of the RLS lockdown in0009.GET /api/gpu/agentsjoins recent GPU hosts to the latest agent sample per agent. Hostnames are normalized (lowercased, domain suffix stripped; IPv4 addresses kept whole) — GPU reporters send short names while Buildkite may record FQDNs or different casing. Hosts with no matching agent are returned with null agent fields so the mismatch is visible rather than silently dropped; when several agents share a normalized hostname, one running a job wins./gpuHost Summary rendersqueue · job nameas plain text with a Buildkite link under the host name. No styling changes beyond reusing existing classes.Verification
npm run lint— cleannpm test— 66/66 pass, including new unit tests fornormalizeHostname,queueTagsFromMetaData, andjoinGpuHostsToAgentsinsrc/lib/buildkite-agent-query.test.tsnpx tsc --noEmit— cleanmigrations:uv run pytest(25/25),ruff check,mypy— cleanNot verified: the GraphQL agents query against the live Buildkite API (needs
BUILDKITE_API_TOKEN); the response mapping is defensive (all fields optional, invalid payloads raise a clear 502).Deployment note
Apply migration
0020before deploying, per the migrations module conventions:DATABASE_URL=postgres://... uv run vllm-dashboard-migrate # read-only plan DATABASE_URL=postgres://... uv run vllm-dashboard-migrate --apply --confirm-target db.example.supabase.co:5432/postgresAgent rows only start accumulating on the first
poll-metricsrun after deploy, so queue/job links appear within ~5 minutes.