Skip to content

builds: fix OTel blocked status and restore missing group columns - #111

Merged
khluu merged 1 commit into
mainfrom
otel-builds-state-fix
Sep 3, 2026
Merged

khluu merged 1 commit into
mainfrom
otel-builds-state-fix

Conversation

@khluu

@khluu khluu commented Sep 2, 2026

Copy link
Copy Markdown
Member

Summary

Follow-up to #105 fixing two regressions in the OTel-backed Builds tab (live now that CI_DATA_SOURCE=otel):

1. Statuses showed "blocked" instead of "passed"

The vLLM CI pipeline ends at a manual gate. When the automatic jobs finish, the build sits in blocked state awaiting an unblock. The OTel buildkite.build span is emitted once with blocked and is not updated to the resolved state. Fivetran/Databricks re-syncs the record from the REST API, so it shows the final passed.

Fix: map blocked to passed for display in the builds query and duration chart.

Verified against the production otel_spans table: every OTel blocked build (including ones with failed-but-retried or gated-matrix failures, e.g. #86349 with 18) is passed in Databricks. Sample of the fixed query:

#86847 raw=blocked -> display=passed
#86839 raw=failed  -> display=failed
#86838 raw=blocked -> display=passed

2. Group columns went missing

OTel only emits spans for jobs that execute. The ~250-400 extended-matrix jobs gated behind the manual unblock step never run, so they produce no spans and their group columns vanished. The complete job roster only exists in the Buildkite API.

Fix: the groups/jobs endpoints now read the full roster from the Buildkite REST API (Get a build), which includes blocked jobs. Falls back to OTel spans if the roster returns empty (e.g. token misconfigured), so the table degrades gracefully instead of going blank.

Not a duplicate

Checked open PRs (gh pr list --search "otel", --search "blocked") — only #98 (OpenAPI docs) and #73 (Docker traces), both unrelated.

Test plan

  • npx tsc --noEmit — clean
  • npx eslint on touched files — clean
  • npm test — 56/56 pass
  • npm run build — succeeds
  • Status mapping verified against the production otel_spans table (query output above).

Not yet verified

The REST roster call needs BUILDKITE_API_TOKEN, which is a hidden Vercel secret and not available locally, so I could not exercise that path end-to-end before opening this. The endpoint follows the documented Get a build shape, and there is a graceful fallback to OTel spans. I want to verify the columns render on a live deployment before relying on this — preview deployments are SSO-protected, so I will confirm on production right after merge (the fallback makes this low-risk).

AI assistance

Developed with AI assistance (Kimi Code). I reviewed the diff and ran the checks above.

@vercel

vercel Bot commented Sep 2, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
vllm-ci-dashboard Ready Ready Preview Sep 3, 2026 4:41am UTC

Request Review

Two fixes for the OTel-backed Builds tab (live when CI_DATA_SOURCE=otel):

1. Statuses showed "blocked" instead of "passed". The vLLM CI pipeline ends
   at a manual gate, and the OTel build span's terminal state stays
   "blocked" even after the build is unblocked and passes. Map blocked to
   passed for display so the status column matches the warehouse (verified:
   every blocked OTel build is "passed" in Databricks).

2. Group columns went missing. OTel only emits spans for jobs that execute,
   so the extended-matrix jobs gated behind the manual unblock step never
   produce spans and their group columns vanished. Read the full job roster
   (including blocked jobs) from the Buildkite REST API Get-a-build endpoint
   instead of otel_spans, falling back to OTel spans if the roster is empty
   so the table degrades gracefully instead of going blank.

Co-authored-by: Kimi Code <noreply@anthropic.com>
Signed-off-by: khluu <khluu000@gmail.com>
@khluu
khluu force-pushed the otel-builds-state-fix branch from 5224069 to 44f52ec Compare September 3, 2026 04:41
@khluu
khluu merged commit 7202e78 into main Sep 3, 2026
3 of 4 checks passed
Thangnguyenvn98 added a commit that referenced this pull request Sep 3, 2026
…bbered by #111's stale merge (#135)

Signed-off-by: Thang Nguyen <thang.nguyen@inferact.ai>
Co-authored-by: Thang Nguyen <thang.nguyen@inferact.ai>

This branch was successfully deployed

1 active deployment
Preview — 44f52ecb Deployed Sep 3, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant