CollectiveX explorer backed by a lazy-ingest Neon database#497
Open
Oseltamivir wants to merge 19 commits into
Open
CollectiveX explorer backed by a lazy-ingest Neon database#497Oseltamivir wants to merge 19 commits into
Oseltamivir wants to merge 19 commits into
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Replace the synthetic preview with server-side JIT discovery and validation of the promoted CollectiveX V1 GitHub Actions artifact. Scope channel, immutable dataset, query cache, and UI selection by benchmark version while preserving publisher-owned decisions. 中文：将合成预览替换为服务端即时发现并校验 CollectiveX V1 GitHub Actions 发布产物。Channel、不可变数据集、查询缓存与 UI 选择器均按基准版本隔离，同时继续由 publisher 独占资格判定、排名与推荐逻辑。
The backend now publishes a v1 dataset from a single qualifying run (qualification index 1) and no longer measures cross-allocation stability. Align the frontend consumer to the new emitted shape: - Drop the per-point `stability` block and the series/cohort eligibility stability fields (`stable_p50`, `stable_p99`, `p50_max_min_ratio`, `p99_max_min_ratio`) from the zod schemas, and pin `promotion.required_allocations` to 1 and `promotion.qualification_indices` to [1]. - Remove the reader's per-point qualification-evidence cross-check (the removed `stability.qualification_indices`); keep the evidence and coverage/selected-attempt invariants. - Remove stability rendering from the inventory, decision tables, display cohorts, and chart tooltip; relabel "3-run evidence" -> "Evidence" and "3/3 qualification runs" -> "1/1 qualification run". - Drop the dead `unstable-p50`, `unstable-p99`, and `awaiting-repeat-allocations` reason labels (no longer emitted). - Regenerate full-catalog.v1.json for the frozen 322-case matrix (167 runnable / 155 unsupported, 1314 points) and refresh the pinned file/matrix digests and counts in data.test.ts. - Update the fixture and reader/e2e tests to the single-allocation, no-stability shape.
…ntity
The CollectiveX release version becomes a first-class numeric identity
(1, 2, 3, ...) instead of the hardcoded string "v1", matching the backend
release marker's "version": N and the cxpublication-<N>-* artifact name.
Future releases increment the number; a version dropdown selects which one.
- types.ts: COLLECTIVEX_VERSIONS is [1] (numeric); add COLLECTIVEX_DEFAULT_VERSION
(max), collectiveXVersionLabel (`V${n}`), and parseCollectiveXVersion (strict
positive-int guard). Frozen data-format literals (collectivex.public.v1,
cx*-v1-*, collectivex_public_v1_*.ndjson) are schema-version and stay unchanged.
- SelectControl/SelectOption generalized to `string | number`; Radix Select is
string-native, so numeric option values round-trip via String() and are
recovered from the option list on change.
- route.ts validates the URL version segment through parseCollectiveXVersion
(404 on unknown) instead of a blind cast.
- reader/api/use-collectivex default to COLLECTIVEX_DEFAULT_VERSION.
- collectivex-github PUBLICATION_POLICY keyed by numeric version.
- promotion badge is version-agnostic ("Promoted"/"已发布").
Add a run picker that lists eligible (tag + success) publication runs for a release version on demand and lets the operator pin the view to any one run's promoted dataset, alongside the numeric version dropdown. - types: collectivex.runs.v1 listing schema (frozen data-format literal) and optional partial-coverage promotion fields (coverage_scope, covered_skus) - reader: parseCollectiveXRuns, fetchCollectiveXRuns, and fetchCollectiveXByDigest (content-hash-verified, promoted-only, synthesizes the channel pointer) - backend: /collectivex-data/<version>/runs.json enumerates candidates via listCollectiveXPublications with a 60s edge cache - hooks/api: useCollectiveXRuns (gated behind Load runs) + useCollectiveXRun - UI: Run control (Load runs then select), version dropdown, and an error-card fallback to the latest published channel Full and partial coverage both parse; pre-partial datasets remain valid.
…ic reasons Align the frontend consumer with the single-run publisher: - types.ts: trial_count 192 -> 64 (single allocation, 64 trials); admit 0.0 data-rate percentiles for host-staging components that spend real latency moving zero logical bytes, while keeping latency strictly positive. - test-fixture.ts: trial_count 192 -> 64 across trial diagnostics. - reader.test.ts: cover the zero-byte / zero-rate host-staging component and assert negative rate and zero latency are still rejected. - CollectiveXDisplay.tsx: in free-form diagnostics mode (no cohort pinned), surface the distinct eligibility reasons across the revealed series so a flagged series explains itself without pinning a cohort. - collectivex.cy.ts: new e2e for the per-series reason line; realign six stale assertions to the single-run shape (64 trials, 1 allocation, 8 attempts, single source bundle, no cross-allocation stability).
A promoted public dataset can legitimately carry rankings and sensitivities but zero recommendations: recommendations are unique-p99-latency-winner callouts, and when an official cohort's candidates fall within the bootstrap equivalence band the winners tie, so no recommendation is emitted. This is a routine, honest outcome for single-allocation v1 publications. The reader's recommendation-coverage check already derives 0 as the expected count in that case, but the decision-graph gate then rejected the same dataset for having an empty recommendations array — an internal contradiction. Require rankings (the decision graph) but not recommendations. Verified end-to-end against the real published v1 artifact (49 series, 66 rankings, 0 recommendations) and with a new regression test.
…lacks decision-grade coverage The landing view combined the Controlled (decision-grade only) evidence scope with the decode phase. On a partial-coverage publication — e.g. a single-run v1 where the strict within-run trial gate leaves whole SKUs diagnostic-tier — the only decision-grade decode cohorts are H100/H200, so Controlled hid every other working SKU (B200, MI300X) by default. The page appeared to have only two SKUs. Choose the landing scope per dataset: keep Controlled when the decision-grade set already covers every SKU measured at the landing phase (decode); otherwise fall back to the full-evidence (diagnostic) scope so no working SKU is hidden. The scope is re-evaluated per content digest on a version/run switch, while an in-session user toggle is preserved. Make the diagnostic scope a true full-evidence superset (all measured series, both tiers, badged by status) so decision-grade series stay visible alongside the rest and the SKU/backend/routing dropdowns run off the complete dataset. Reword the diagnostic warning accordingly. Tests: unit discriminator in data.test.ts; e2e updated for the superset line count and warning text, plus a new default-to-full-evidence case.
…facts Rework the CollectiveX frontend around the trimmed return-code-only artifact contract (collectivex.matrix.v1 / .ep.v1 / .terminal.v1). The GitHub loader assembles a run dataset at request time from the sweep's matrix + shard artifacts, newest-completed-run wins regardless of conclusion, and runs are selected by the matrix's numeric `version` field: latest/list skip runs tagged for another version and by-id 404s on a mismatch. The format tag stays collectivex.matrix.v1 as the schema id; the numeric version is the content axis the UI selects on. Rewrites the reader, data model, display/chart/inventory/tables components, API surface, route, and the vitest + cypress suites for the neutral model.
The backend dropped the promotion-era shape: no format/schema_version
strings, no eplb/activation_profile/implementation.provenance/
resource_profile, and terminal-outcome docs are gone (failed correctness
now rides in-band as a case-attempt with outcome.status: invalid).
Reader/ingest updated to the current artifacts:
- discriminate docs by record_type ('case-attempt'), not doc.format;
ignore 'samples'; identify the matrix structurally (requested_cases +
include + version)
- relax the 3 raw ingest schemas (drop format literals; make removed
fields optional) and remove the dead terminal schema
- surface non-success attempts as invalid coverage/attempts instead of
mislabeling them 'pending'
- defensive derefs + valid-safeId defaults for the dropped nested objects
De-slop: remove dead exports, an unused param, and a stale i18n key.
Frontend-internal collectivex.view.v1 / collectivex.runs.v1 unchanged.
Align the frontend with the current neutral artifact contract, remove retired diagnostics and selector systems, and keep partial sweeps and reruns visible.\n\nFix run-list cache expiry so rerun attempts refresh after the configured TTL.
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
…nsion CollectiveX case_ids now carry a precision factor (bf16|fp8). The reader keyed series/coverage on case_id so bf16 and fp8 were already distinct, but the label, color key, and selection omitted precision, so the two collided in the UI (same label, same chart color, not filterable). - types: CollectiveXPrecision; precision on CollectiveXSeries + CollectiveXCoverage. - reader: parse case.precision (toPrecision defaults legacy/no-field artifacts to bf16); thread into series, coverage, and the coverage label. - data: precision in series label + colorKey (distinct legend colors) and in the series selection (seriesMatchesSelection requires it). - display: precision toggle mirroring the phase idiom; Precision column in the inventory and tables; en/zh strings. - fixtures + tests updated (bf16/fp8 split series, distinct color keys, legacy default-to-bf16). COLLECTIVEX_VERSIONS unchanged (benchmark sweep.json still v1).
…on DB
The CollectiveX tab previously assembled its dataset from GitHub Actions
artifacts on every request: data vanished with the 14-day artifact expiry,
cold requests paid GitHub paging + ZIP downloads, and the run picker was
capped at 8 runs.
The tab is now backed by a dedicated Neon database acting as a durable
cache of GitHub, populated lazily on read:
- packages/db gains the shared neutral contract (reader/types/fixtures
moved from the app), a second migration set (cx_runs + cx_run_docs
storing RAW sweep documents so the evolving sweep JSON contract only
ever touches the reader), tombstone-aware queries, and a CLI ingest for
pre-warming runs before artifact expiry, backfills, and un-deletes.
- /api/v1/collectivex/{latest,runs,runs/[runId]} discover and persist
sweep runs from any harness branch in-request: race-safe
single-statement CTE inserts, FOR UPDATE-guarded refresh when GitHub
reports a newer run attempt, assembled datasets served through the
shared reader, and stored-data fallback when GitHub is down. DELETE
tombstones a run (discovery never resurrects it) behind a dedicated
COLLECTIVEX_ADMIN_SECRET bearer token and purges only the collectivex
CDN tag scope.
- The frontend keeps the run picker (now DB-backed, capped at 50) and
gains a delete-run flow with a remembered admin token.
- Removed: the JIT layer (collectivex-github.ts, /collectivex-data route).
No CI ingest plumbing or GitHub secrets are required; runtime needs
GITHUB_TOKEN, DATABASE_COLLECTIVEX_READONLY_URL/WRITE_URL, and
COLLECTIVEX_ADMIN_SECRET (already provisioned on Vercel), plus the
applied migrations-collectivex/ migration.
Design rationale and invariants are documented in docs/collectivex.md.
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| runId: string, | ||
| runAttempt: number, | ||
| ): string[] { | ||
| const pattern = new RegExp(`^${SHARD_PREFIX}(?<cell>.+)-${runId}-(?<attempt>[1-9][0-9]*)$`, 'u'); |
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 59340f3. Configure here.
…ped docs Two concurrency holes found in post-review sweep of the lazy-ingest writes: - Concurrent attempt-refreshes targeting DIFFERENT attempts (2->3 blocked on 1->2) could leave mixed doc sets: the loser's single-snapshot CTE DELETE cannot see docs the winner committed while it waited on the row lock. cx_run_docs rows are now stamped with their run_attempt (migration 002) and readers fetch row + docs in ONE query filtered to the row's current attempt, so superseded or racing doc sets are invisible; the next refresh's DELETE garbage-collects them. This also removes the separate docs roundtrip and the reader-side race where the row and doc reads could straddle a committing refresh. - deleteCollectiveXRun tombstoned the run and freed its docs in two statements; on the neon HTTP driver each autocommits, so a failure between them orphaned the docs forever (re-delete short-circuits to 404 and never retries cleanup). Tombstone + doc removal is now one atomic CTE statement. Migration 002 applied to the provisioned Neon instance.
… dimension The sweep now fans decode cases out over a kernel-mode axis (harness commit bb77c328: DeepEP legacy-Buffer LL + MoRI IntraNodeLL, decode-only), so case_ids carry mode (normal|low-latency) as a varying factor. The reader keyed series/coverage on case_id so the modes were already distinct rows, but labels, color keys, and the selection omitted mode — normal and LL kernels of one cell collided in the UI. - types: CollectiveXMode; mode on CollectiveXSeries + CollectiveXCoverage. - reader: parse case.mode (toMode defaults artifacts without the field to normal); thread into series, coverage, and the coverage label. - data: mode in the series label + color key (distinct legend colors) and in the series selection (seriesMatchesSelection requires it). - display: kernel-mode toggle mirroring the phase/precision idiom, rendered only when a run measured both modes (LL is decode-only, so single-mode runs keep the current controls); precision availability now cascades through the selected mode; Mode column in the inventory and tables (reusing the existing zh header strings). - fixtures + tests: mode override in the shard builder (null models a pre-mode artifact), normal/LL split series with distinct color keys and labels, mode-toggle e2e; checked-in fixtures regenerated. COLLECTIVEX_VERSIONS unchanged (sweep.json stays v1; normal-mode case_ids are identical). Verified against the stored production runs — their case dicts already carry mode: normal and serve unchanged.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Replaces the
/collectivextab's just-in-time GitHub artifact serving with a dedicated Neon database that acts as a durable, lazily-populated cache of GitHub Actions. Sweep runs now outlive their 14-day artifact retention once viewed, the run picker lists up to 50 persisted runs (was 8, rebuilt per request), and runs are deletable from the dashboard.Architecture
/api/v1/collectivex/{latest,runs,runs/[runId]}check the DB first; on a miss they discover completedCollectiveX Sweepruns (any harness branch — workflow identity only), download thecxsweep-matrix-*/cxshard-*artifact ZIPs in-request, and persist the raw documents. The shared reader (packages/db/src/collectivex/) assembles datasets at read time, so the evolving sweep JSON contract only ever touches the reader and applies retroactively to stored runs. No ingest workflow, cross-repo dispatch, or GitHub secrets.ON CONFLICT DO NOTHING(concurrent first-viewers race safely; a partial run can never become visible); aFOR UPDATE-guarded refresh replaces a run's contents when GitHub reports a newerrun_attempt(re-run of failed shards).cx_runs.deleted_at, docs freed) so re-discovery never resurrects a deleted run; the CLI (pnpm admin:db:ingest:collectivex <run>) pre-warms runs before artifact expiry, backfills, and un-deletes.collectivexCDN tag with a 60s freshness window;POST /api/v1/invalidate?scope=collectivexand run deletion purge only that scope.DELETE /api/v1/collectivex/runs/[runId]uses a dedicatedCOLLECTIVEX_ADMIN_SECRETBearer token (browser-held, independently rotatable — deliberately notINVALIDATE_SECRET), prompted once in the UI and remembered in localStorage.Design rationale and invariants:
docs/collectivex.md.Deploy prerequisites (already provisioned)
Neon marketplace store
inferencex-collectivexon this project; env varsDATABASE_COLLECTIVEX_READONLY_URL(readonly role @ pooled),DATABASE_COLLECTIVEX_WRITE_URL(owner @ direct),COLLECTIVEX_ADMIN_SECRETset for production + preview;migrations-collectivex/001applied; 8 real sweep runs already ingested.Validation
collectivex.cy.tsspecs (fixtures mode), including the delete flow (token prompt, 401 → token cleared, declined confirm).Authorizationheaders returning 401.pnpm typecheck && pnpm lint && pnpm fmtclean; production build succeeds.Note: unofficial-run overlay support does not apply — CollectiveX is not inference/evaluation chart data.
Note
Medium Risk
New production data path (lazy GitHub ingest, separate DB credentials, admin delete + cache scoping) and CDN behavior; mistakes could affect CollectiveX availability or resurrect deleted runs if tombstone logic fails.
Overview
Adds a CollectiveX dashboard tab backed by a separate Neon Postgres store that lazy-ingests GitHub Actions sweep artifacts on API read (raw docs in DB, shared reader assembles datasets at serve time). New routes:
GET /api/v1/collectivex/latest,runs, andruns/[runId]; adminDELETEon a run usesCOLLECTIVEX_ADMIN_SECRET(notINVALIDATE_SECRET), tombstones rows, and purges only thecollectivexCDN cache tag.POST /api/v1/invalidate?scope=collectivexscopes invalidation the same way; Bearer checks use byte-length comparison for multibyte safety.Env / ops:
DATABASE_COLLECTIVEX_*URLs,GITHUB_TOKENrequired for ingest,admin:db:migrate:collectivexandadmin:db:ingest:collectivexscripts;docs/collectivex.mddocuments the pipeline. UI:/collectivexand/zh/collectivex, tab nav + sitemap, D3 explorer chart and e2e coverage.Also vendors Neon agent skills under
.agents/skills/(docs for agents, no runtime app change) and documents a temporary English-only override at the top ofAGENTS.md.Reviewed by Cursor Bugbot for commit 439a160. Bugbot is set up for automated code reviews on this repo. Configure here.