comfyci-runpod-worker

GPU regression testing for ComfyUI on RunPod serverless, feeding ci.comfy.org.

For every new commit on ComfyUI master, the GPU regression workflow runs the curated workflows in manifest/workflows.json on a RunPod serverless endpoint at that exact commit (the worker checks the commit out at request time), then compares the fixed-seed outputs against two baselines:

• golden — blessed reference outputs generated from a known-good release tag
• previous — the last completed run on the same branch

Metrics (per frame): MSE, PSNR mean/min, max abs channel diff, % pixels changed, plus diff heatmaps and side-by-side strips. Everything is published under regression/ where the dashboard reads it.

Results storage

Two interchangeable backends behind one path layout (scripts/storage.py, pick with --storage / REGRESSION_STORAGE):

• github (current default): results live on this repo's orphan results branch; the dashboard reads them from https://raw.githubusercontent.com/Comfy-Org/comfyci-runpod-worker/results/regression/. Zero extra secrets — the scheduled workflow pushes with its own GITHUB_TOKEN. Fine at current volume; migrate before the run rate or the workflow set grows much (git history only accumulates).
• gcs: gs://comfy-ci-results/regression/ — the long-term home once a GCS_SERVICE_ACCOUNT_JSON with write access exists. Switching is an env change here plus NEXT_PUBLIC_REGRESSION_BASE on the dashboard.

Layout

Path	What
Dockerfile, docker/	Worker image: CUDA + torch + pre-cloned ComfyUI. Commit under test is checked out per request.
src/	RunPod handler (handler.py), commit checkout, workflow runner, volume model sync
manifest/workflows.json	Single source of truth: workflows, models, seeds, thresholds (staged workflow JSONs not yet in the manifest live in manifest/workflows/)
manifest/lanes.json	Environment lanes (python / torch / cuda / GPU, cadence, storage prefix) published to the dashboard through the run index
manifest/custom_nodes.json	Custom-node packs (repo, pinned commit) the pack workflows install on the worker, baked into the images CI builds (see Custom-node packs)
.github/workflows/build-index.yml	Backfills or repairs the run index for a branch/lane (workflow_dispatch)
scripts/	Orchestration run on the (CPU) CI runner: submit/poll, compare, publish, golden generation
scripts/template_api_export.py, scripts/import_template.py	Turn a workflow template into a corpus entry (see Adding a workflow)
.github/workflows/regression.yml	Scheduled poller that tests each new master commit (interim, until the core CI job below)
action.yml	Composite action for ComfyUI's test-ci.yml (GCS phase)

Adding a workflow

Templates are the corpus's main source: a template that stops producing its image is a release blocker, and the comparison here is what notices. One PR to this repo per template.

1. Dump /object_info from the ComfyUI version the corpus runs against, with the template's models present so their combo values resolve, and import the template from a workflow_templates checkout:

   curl -s http://127.0.0.1:8188/object_info > object_info.json
   python scripts/import_template.py image_pixeldit_t2i \
       --templates-dir ../workflow_templates/templates \
       --object-info object_info.json --id pixeldit_t2i
   

   That writes manifest/workflows/<id>_api.json and prints the manifest entry. It converts the template the way the frontend's Export (API) does, picks the branch a constant switch selects (so the other branch's models never have to be synced), drops preview nodes, normalises the output to a SaveImage PNG prefixed with the workflow id, and fills models from the download links in the template's own model-links note, with the sha256 ComfyUI publishes.

   It refuses rather than guess. A template whose widget layout lives on the node instance, or one that needs custom nodes, has to be exported by hand from ComfyUI (Workflow → Export (API)) instead; scripts/tests/test_template_api_export.py documents which conversion rules are covered. A loader whose models/ subdirectory it does not know is also a refusal, not a guess: add the (class_type, input name) pair to DIRECTORY_BY_LOADER_INPUT.

2. Paste the entry under workflows in manifest/workflows.json and choose a timeout_s the model can finish in. Leave "enabled": false.

   For a hand-exported workflow, the entry needs the same shape: workflow_path, seed_overrides (node id → input patch, belt-and-braces for the seed), models (name + download url + ComfyUI models subdirectory, plus sha256 when the host publishes one), and optionally gpu_type, thresholds, comfy_flags. Input images for LoadImage nodes (and clips for VHS_LoadVideo) are listed under models too, with "directory": "input" (pin the url to a commit, not a branch), and the entry sets "comfy_flags": ["--input-directory", "/runpod-volume/models/input"] so ComfyUI reads them from the volume. A model that lives in a subfolder of a models directory keeps the subfolder in its name, the way the loader's combo lists it ("name": "bbox/face_yolov8m.pt", "directory": "ultralytics").

   A workflow that uses custom nodes lists its packs under custom_nodes (names from manifest/custom_nodes.json) and restricts itself with lanes to the lanes whose image bakes them; see Custom-node packs.

3. Check it. python -m pytest scripts/tests -q lints the manifest against every workflow it references (loaded files and models match both ways, every node feeds the SaveImage, seed override targets, model folders, SaveImage prefix = workflow id). Then confirm a real server accepts the prompt, which is what catches a conversion that produced a valid-looking file with the wrong input names:

   curl -sS -X POST http://127.0.0.1:8188/prompt -H 'Content-Type: application/json' \
       -d "{\"prompt\": $(cat manifest/workflows/pixeldit_t2i_api.json)}"
   

   A 200 means every node, input and link survived; the run can then fail on missing weights without telling you anything new.

4. Merge. sync-models.yml downloads the new models onto the network volume automatically.

5. Generate + bless a golden for it: run the Golden baselines workflow with the current release tag and workflows: <id>, inspect, bless.

6. Flip "enabled": true in a follow-up PR.

Verdicts

Verdict	Meaning	CI effect
pass	metrics within thresholds vs golden	green
fail	metrics exceed thresholds vs golden	red
execution_error	workflow errored on the commit under test, or a custom-node pack it requested did not import (worker status custom_node_error)	red
no_baseline	no blessed golden yet	warning, green
infra_error	RunPod/queue/checkout/model-volume/pack-install problem (after 1 retry)	warning, green

Thresholds come from the manifest entry if set, otherwise from the manifest defaults loosened to 3× the golden's own re-run noise floor (noise_floor.json), so nondeterministic workflows don't false-alarm.

A failure vs golden is also classified against the previous tested run (detail in summary.json): new_drift when this commit's outputs differ from the previous run (the change happened here), inherited when they are bit-identical to a previous run that already failed against the same golden, and null when there is nothing to compare against or the outputs match a previous run that did not fail (the golden changed, not the code). Inherited failures publish no figures (figures: false): the figures of the first failing commit already show the diff. summary.json carries schema_version: 2 and has_infra_error (infra errors never turn overall red). Each workflow also records prior_verdict / prior_commit: its most recent red or green verdict before this commit, read from the run index, which (unlike the previous-run pointer) includes runs that had execution errors.

Accepting an intentional change (re-bless)

When a commit changes numerics on purpose (a kernel optimisation that moves every pixel by a rounding error, say), the golden is stale, not the code. Re-bless with a reason so the history explains itself:

• From a release tag on master, with a fresh two-run noise floor: run Golden baselines with ref=<tag>, workflows=<id>, bless=false, inspect golden/<id>/<tag>/outputs/ and noise_floor.json, then re-run with bless_only=true and a reason.
• Without GPU time, from a run that already exists on the results branch: from_run=master/<full sha>, workflows=<id>, reason=..., bless=true. The golden is named after the commit (12-char sha; pass ref=<tag> to name it after a tag that resolves to that same commit), has no run_r2.json, and its noise floor is inherited from the previous golden and marked measured: false.

An existing golden/<id>/<tag>/ is never overwritten unless force is set. golden/<id>/current.json records reason, supersedes and the golden's output_sha256; golden/<id>/history.json keeps every bless; the run index is refreshed in the same publish so accepted drift stops being reported as an open regression.

Run index

Every publish maintains a compact index under regression/index/: a per-lane head file with the newest 500 runs and the first bad commit of each open failing chain, complete monthly shards, and lanes.json with the lane registry from manifest/lanes.json, per-branch latest pointers and the current golden tag and output sha per workflow. Entries carry the commit subject, author and PR (looked up with the Actions token; the PR title, author and avatar too when the commit landed through a PR), the tested range to the previous run (HEAD polling can skip commits), per-workflow verdicts with the drift class, output sha, metrics and timings. Entries are ordered by commit time, so re-testing an older commit never reorders history. The dashboard's history and matrix views read only these files.

Build run index (workflow_dispatch) backfills or repairs a branch/lane from the committed summaries; dispatch it once after the first deploy and whenever an index file needs regenerating.

PR comments (PULSE)

After each run, scripts/notify_pr.py can leave one comment on the ComfyUI pull request that landed the tested commit. Its first line starts with (PULSE); below it, a row per workflow (verdict, new or inherited, PSNR, mean MSE, % pixels changed, exec time, peak VRAM), the lane and the run's page on ci.comfy.org.

• When: a workflow's outputs changed at this commit (detail: new_drift), or it hit an execution error while its prior run (prior_verdict) produced outputs. Never for pass, inherited drift, infra errors, missing goldens, a workflow with no prior run, a re-test of a commit older than the previous run, or a run that covered several new commits (the change cannot be pinned on one PR).
• One comment per PR: found again by the hidden marker <!-- pulse-regression:v1 --> and the token owner's login, and edited in place, so reruns never add a second one. A later clean run of the same commit turns it into a short resolved note.
• Whose PRs: only PRs authored by a ComfyUI code owner, i.e. the users on the * rule of ComfyUI's CODEOWNERS (team entries are ignored). The repo variable PULSE_ALLOWED_AUTHORS (comma-separated logins) replaces that list. If CODEOWNERS cannot be read and no override is set, nothing is posted. Only the primary lane on master comments.
• Modes (repo variable PULSE_PR_COMMENTS): dry-run (default) renders the comment into the job summary and posts nothing; on posts; off skips the step's work entirely. on without the token falls back to dry-run.
• Token (repo secret PULSE_GH_TOKEN): a personal access token of the account the comments should come from; they appear as that user. Prefer a fine-grained token with resource owner Comfy-Org, repository Comfy-Org/ComfyUI only, and Issues and Pull requests set to read and write. Fine-grained tokens for an organisation's repositories may need approval by an org owner, depending on the org's token policy. A classic token with the public_repo scope also works, but it can push to every public repository the account can write to, and anyone who can edit this repo's workflows can use it. Reads (PR lookup, CODEOWNERS) use the Actions token.
• Non-blocking: the step runs even when the suite fails, has continue-on-error, and turns every API or network problem into a warning.

To turn it on: add PULSE_GH_TOKEN (Settings → Secrets and variables → Actions), check the dry-run output in a few job summaries, then set the variable PULSE_PR_COMMENTS to on. Setting it to off stops it at once.

Lanes

A lane is one environment the corpus runs in: a worker image (Python, torch build, CUDA channel), the RunPod endpoint running that image, a cadence and a storage prefix. manifest/lanes.json declares the images once under images and the lanes under lanes; scripts/lanes.py derives the build matrix and the per-commit matrix from it, and scripts/lanes.py validate (run by the tests) keeps the file consistent. Goldens are per lane: cu130+ images enable comfy-kitchen kernels that cu128 cannot, so a new lane always starts with its own baselines.

A workflow can restrict itself to lanes with lanes in manifest/workflows.json (a list of lane ids; null, the default, means every lane). run_regression.py, golden_baseline.py and the per-commit plan (lanes.py per-commit lists each lane's workflows) all honour it, even when the workflow is named explicitly, and the dashboard shows a skipped cell as not run. Workflows that need custom-node packs must use it (see below).

Adding a lane:

1. Add the image under images (python, cuda, torch_index, torch_spec with exact torch and torchvision pins; no torchaudio, ComfyUI dropped it) and the lane under lanes with "endpoint_id": null, layout: 2 and prefix: regression/lanes/<lane id>. Merge.

2. build-image.yml builds every buildable image on push to main (or dispatch it from the branch with the image id) and tags it :<image> and :<image>-<sha>.

3. Create the serverless endpoint from the :<image>-<sha> tag, once, by hand (it is the one billable step; idle endpoints cost nothing at 0 minimum workers):

   RUNPOD_API_KEY=... python scripts/runpod_endpoints.py create \
       --lane <lane id> --tag <image>-<sha> --write
   

   The request comes from endpoint_defaults in lanes.json (pool, workers, volume, data center, registry credential, timeouts) plus the lane's CUDA floor; --dry-run prints it, --write records the new id as the lane's endpoint_id. Commit that. From then on build-image.yml repoints the lane at each new sha tag (runpod_endpoints.py set-image); the legacy lane is never moved. A repoint is a rolling release: workers already running the old image keep serving jobs until RunPod replaces them, which can take a long time when the new image is large and the new workers sit throttled behind them, so a run right after a repoint may silently use the old image (its env.image_sha tells). Check isStale in the endpoint's worker list before generating goldens; setting the endpoint's max workers to 0 and back to its normal value flushes the stale workers within seconds.

4. Generate and bless goldens for the lane: Golden baselines with the current release tag, workflows: all, lane: <lane id>, inspect, then bless_only.

5. The per-commit cron picks the lane up on its next tick (a lane with a null endpoint_id is skipped), and ci.comfy.org lists it in the lane selector after the first publish.

Custom-node packs

Four workflows exercise custom-node packs on real GPUs, so a core commit or a torch lane that breaks a pack is caught within the hour and bit-compared against goldens like every other workflow: kjnodes_sdxl_t2i (KJNodes image nodes on an SDXL render), vhs_video_roundtrip (VideoHelperSuite decode of a clip, PNG frames, gif encode), impact_sdxl_detailer (Impact-Pack face detection and detailing with the Impact-Subpack ultralytics detector) and manager_import_smoke (ComfyUI-Manager import, no pack nodes).

manifest/custom_nodes.json is the registry: packs keyed by their custom_nodes directory name, with repo, sha (a full commit), label, kind (git, the default, or pip for a Python package such as Manager 4.x, which ComfyUI loads with --enable-manager and must not be cloned into custom_nodes), requirements (install the pack's requirements.txt), and optionally pip_args, comfy_flags the pack needs, files written into the checkout before the server starts (Manager's config, which keeps its startup offline), and for a pip pack the log line that proves it loaded (log_ok) or that ComfyUI prints when it is absent (log_missing). constraints are extra version pins for pack installs on top of the lane's torch pins (both opencv variants in lockstep, so Manager's startup fixer finds nothing to reinstall). Pins follow the frontend's custom-node gate (browser_tests/fixtures/data/customNodeManifest.core.json in ComfyUI_frontend), so both CI systems test the same pack commits; bump them by PR.

A workflow entry names its packs under custom_nodes and restricts itself with lanes to lanes whose image bakes them. Pack requirements are baked into the images CI builds (build not false in lanes.json); the legacy cu128 image is never rebuilt, so pack workflows never run on its lane and their cells show as not run there. scripts/lanes.py validate and the manifest lint enforce both rules.

How a job runs a pack (src/custom_nodes.py):

• The image build fetches every pack at its sha (git init, fetch --depth 1 <sha>, checkout FETCH_HEAD: the gate's own recipe) into /comfy-cache/custom_nodes/<name>/<sha>, installs its requirements under the torch pins plus the registry's, and records under /comfy-config/custom_nodes/ the requirements hash (or the pip pin) it baked. The image's env block carries custom_nodes_sha256, the registry it was built from. A pack's install.py is never run: dependencies come from its requirements.txt, models from the workflow manifest.
• Per job, after the commit checkout, the worker removes the packs an earlier job left behind, copies each requested pack from the cache into ComfyUI/custom_nodes/<name> (fetching it first when that pin is not cached) and pip-installs its requirements only when their hash differs from the baked one, so a pin bump works before the rebuild lands. A job that requests no packs runs the bare custom_nodes/ the image shipped. ComfyUI ignores custom_nodes/ and user/ in git, so the checkout's git clean never touches them.
• Once the server is up, ComfyUI's own import summary (Import times for custom nodes:, read with the console formatter's colour codes and [INFO] tags stripped) is read from the server log. A requested pack that failed to import, or has no line there, fails the run with worker status custom_node_error (verdict execution_error) before the workflow is validated, whether or not the workflow uses its nodes. A server that dies or never answers is reported the same way only when the log proves a pack is at fault (the summary was printed and the pack is not ok in it, or ComfyUI said a pip pack is absent); a server that fell over before loading packs (CUDA init, OOM, a core import error) keeps server_died / server_start_timeout (infra, retried once) with the packs' status as detail. A pack that could not be fetched or pip-installed is custom_node_install_error (infra, retried once).
• run.json carries custom_nodes (per pack: sha, where it came from, pip time, import status and time), summary.json copies it per workflow, the run index keeps the import status per pack (cn), and blessed.json records the packs a golden was generated with.

Adding a pack: declare it in the registry, add the workflow with custom_nodes and lanes, merge (the baking images rebuild and their lanes are repointed), let sync-models.yml fetch any models it needs, generate and bless the golden, then enable the workflow. Bumping a pin: edit the sha, merge, re-bless the workflows that use the pack.

Frontend CI index

frontend-index.yml (cron 13,43 * * * *, offset from the regression cron) runs scripts/frontend_index.py, which turns three public data sources of Comfy-Org/ComfyUI_frontend into compact static JSON under frontend/ on the results branch, next to regression/. ci.comfy.org's Frontend tab reads only these files (NEXT_PUBLIC_FRONTEND_BASE, default https://raw.githubusercontent.com/Comfy-Org/comfyci-runpod-worker/results/frontend).

frontend/
  perf/head.json             metric registry, rule, scenarios, newest 100 entries, latest comparison
  perf/<YYYY-MM>.json        complete monthly shards (newest first)
  ci/main.json               column registry, newest 300 main commits, first_bad, coverage
  ci/main/<YYYY-MM>.json     complete monthly shards
  ci/custom-nodes.json       newest 90 custom-nodes gate runs with shard rows, core pin, streak

All times are UTC epoch seconds, every file has schema_version: 1 and every head carries updated_ts. A tick is a fetch phase, budgeted by the API's X-RateLimit-Remaining with a 150-request reserve, then a write phase: the three writers are derived writers (re-run on the freshest tree when a push retries behind a concurrent publisher), merge by key and only write a file whose content changed, so a tick with nothing new pushes nothing.

Perf (perf-data branch, baselines/perf-<ts>-<sha8>.json, of which the frontend keeps only the newest 20): one entry per baseline, kept forever.

key	meaning
id, c, t	<ts>-<sha8> (the file is baselines/perf-<id>.json), commit, baseline time
n, nx	samples per scenario; nx lists the scenarios with a different count
s	per scenario, the mean of each sample metric in metrics order (8 significant digits, bytes as integers, null when absent)
cls	per scenario, one letter per metric: r regression, i improvement, n neutral, x noisy (CV > 50 %), - fewer than two history values
head metrics, targets, window, rule, stats	the frontend's 12 reported metrics with label, unit and min_abs_delta; p5_fps: 52; the 20-run window; the z > 2 / CV > 50 % rule; current = median, history = mean, sample stddev, delta against the history mean
head latest.cmp	per scenario, [current median, history mean, history stddev, history n, z, cls] per metric for the newest baseline

cls is computed once, at ingest, with the frontend's own rule (scripts/perf-stats.ts, scripts/perf-report.ts): the baseline's medians against the means of the up to 20 entries preceding it by time, so every historical point carries the badge the frontend would have shown.

CI on main (push runs of every CI: workflow): one entry per commit.

key	meaning
c, ct, o	commit, commit time, rollup of the core columns (failure, pending, cancelled, success; skipped never blocks)
m, mi	subject, author, PR, PR title/author/avatar; mi: true while m is provisional (from the run's head_commit, upgraded with the commits API at 25 commits per tick, newest first, monthly shards included)
w	per column id: {id: run id, c: conclusion, s: status while not completed, t: start, d: duration s, a: attempt when > 1}
head columns	ordered registry {id, label, paths, name, group}, groups core / website / billing / validation; lint matches both lint files; any other CI: workflow seen on a main push is appended as group other
head first_bad	per column, the start of the open failing chain (red opens, success closes, cancelled/pending/missing neither)
head coverage	from_ts, to_ts: the index is complete for runs created between them (to_ts is the tick time of the last complete listing, as of the last write: a tick that learns nothing else is not written); partial: the last backfill stopped short (budget)

Incremental ticks list runs created since the oldest unsettled commit (a cell still running, or a core column missing on a commit younger than 3 h), at least 6 h back and never later than coverage.to_ts, so a stalled cron or a broken tick leaves no hole. When that listing overflows its five pages (days away), the gap above to_ts is closed in day windows while the budget lasts and the next tick continues from where it stopped. Reruns of commits older than 6 h are only caught by a backfill (the API cannot sort by updated_at). Dispatch the workflow with backfill_days to walk day windows back from coverage.from_ts (a 90-day backfill is about 500 requests plus metadata upgrades spread over the following ticks); only (csv of perf, ci, gate) limits a run.

Custom-nodes gate (ci-tests-custom-nodes.yaml, scheduled daily against a pinned ComfyUI core, plus manual dispatches; PR runs are excluded):

key	meaning
id, t, ev, c, d, s, a	run id, start, event, conclusion, duration, status while running, attempt when > 1
sha, pin, ref, rs	frontend commit, COMFYUI_PIN read from the workflow file at that commit (null until read; a read skipped by the budget or failed is repaired on a later tick, one request per sha), the core ref the run used (pin for scheduled runs; unknown for dispatches until phase 2), pin or dispatch
rows, fs, final, jobs	rows[proof_row][shard] job conclusions ("0" are the clean shards, other rows are detection proofs), failed step per non-green job, the verdict job's conclusion, [matrix jobs, red jobs]
head pin, streak	the newest pin; {state, since, days, runs, capped} over the clean rows (capped when the streak reaches the oldest stored run)

Steady state costs 5-9 requests per tick (3 when nothing is new) against the Actions token's 1,000 per hour. FRONTEND_GH_TOKEN (a fine-grained PAT with Actions read on the frontend repo) is optional and reserved for phase 2 (dispatching the gate, reading run logs and artifacts); without it the GITHUB_TOKEN is used. Local dry run, nothing pushed: python scripts/frontend_index.py --skip-publish --results-workdir /tmp/results.

One-time infrastructure setup

1. Docker images: build-image.yml pushes one image per entry in manifest/lanes.json images to ghcr.io/comfy-org/comfyci-runpod-worker using the built-in GITHUB_TOKEN (no registry secrets). After the first build, make the package public (repo → Packages → package settings → Change visibility) so RunPod can pull it without credentials, or give the endpoint a registry credential.
2. RunPod: create a Network Volume (≥200 GB, datacenter with 4090 availability), then one serverless endpoint per lane from its image with the volume attached (see Lanes). Optionally set HF_TOKEN as an endpoint env var for gated models.
3. Repo secrets (this repo): RUNPOD_API_KEY, RUNPOD_ENDPOINT_ID, and optionally HF_TOKEN (gated models) and PULSE_GH_TOKEN (PR comments, see above). For the GCS phase, additionally GCS_SERVICE_ACCOUNT_JSON (write access to the CI bucket) and GCS_BUCKET (comfy-ci-results).
4. Seed the volume: run sync-models.yml (workflow_dispatch).
5. First goldens: run Golden baselines with the latest release tag and bless: true.
6. Done — regression.yml now tests each new master commit on its own.
7. (GCS phase) Wire into core CI: add the job below to ComfyUI's .github/workflows/test-ci.yml plus the RUNPOD_API_KEY / RUNPOD_ENDPOINT_ID secrets there, and retire the scheduled poller:

  gpu-regression:
    runs-on: ubuntu-latest
    steps:
      - name: RunPod regression suite
        uses: comfy-org/comfyci-runpod-worker@main
        with:
          comfy-commit: ${{ github.sha }}
          branch: ${{ github.ref_name }}
          gcs-bucket: comfy-ci-results
        env:
          RUNPOD_API_KEY: ${{ secrets.RUNPOD_API_KEY }}
          RUNPOD_ENDPOINT_ID: ${{ secrets.RUNPOD_ENDPOINT_ID }}
          GCS_SERVICE_ACCOUNT_JSON: ${{ secrets.GCS_SERVICE_ACCOUNT_JSON }}

Results layout

Identical on both backends (branch root or bucket root):

regression/
  runs/<branch>/<commit>/<workflow_id>/{outputs/, run.json, comparison.json, figures/}
                                           # figures/ only when the frames differ and the
                                           # failure is not inherited
  runs/<branch>/<commit>/summary.json      # per-commit rollup (dashboard entrypoint)
  latest/<branch>.json                     # previous-run pointer, written last
  manifest-snapshot/<commit>.json
  golden/<workflow_id>/<tag>/{outputs/, run_r1.json, run_r2.json, noise_floor.json, blessed.json}
  golden/<workflow_id>/current.json        # active blessed tag (+ reason, supersedes, output_sha256)
  golden/<workflow_id>/history.json        # every bless
  index/lanes.json                         # lane registry + per-branch latest + golden shas
  index/<branch>/<lane>.json               # newest 500 runs + first bad commit per workflow
  index/<branch>/<lane>/<YYYY-MM>.json     # complete monthly shards
frontend/                                  # ComfyUI_frontend CI index (see above)
  perf/head.json                           # perf metric registry, newest 100 baselines, latest comparison
  perf/<YYYY-MM>.json                      # complete monthly shards
  ci/main.json                             # column registry, newest 300 main commits, first_bad, coverage
  ci/main/<YYYY-MM>.json                   # complete monthly shards
  ci/custom-nodes.json                     # newest 90 custom-nodes gate runs, core pin, streak

Run outputs also carry 256-px WebP previews under <workflow_id>/outputs/thumbs/ for the dashboard's list views.

Notes

• Goldens are valid per lane (GPU type and worker image torch/CUDA): after changing either, regenerate and re-bless (blessed.json records both, and every run record carries an env block with the image id, build sha, python, torch, CUDA, host driver and the kernel backends ComfyUI logged).
• Custom-node pins are part of the image too: a bump in manifest/custom_nodes.json rebuilds the baking images and moves the pack workflows' outputs, so re-bless them (blessed.json records the pins).
• Local dry run without publishing anything: python scripts/run_regression.py --commit <sha> --branch test --skip-publish
• Script tests (run on every PR by the Script tests workflow): pip install -r scripts/requirements.txt pytest && python -m pytest scripts/tests
• The results checkout is a blobless sparse clone: only the aggregate directories are materialised, run directories on first use.
• run.json delay_s is in seconds from summary schema_version: 2 on; earlier records hold RunPod's raw millisecond value under the same key.