Matrix P2 (part 1): data-driven upgrade baselines from live telemetry - #1987
Matrix P2 (part 1): data-driven upgrade baselines from live telemetry#1987dev-punia-altimate wants to merge 43 commits into
Conversation
…-split update matrix, widen release-gate poll + error state, capture CLI stderr
…rsion distribution
…sights + fallback)
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
VSIX Install + Update Matrix❌ No result files found — treating as a blocking failure |
… upgrade-from version - pick_by_coverage: add most-used versions until target+selected users >= 90% - build-matrix: choose versions DYNAMICALLY via az (local) or APPINSIGHTS_API_KEY (CI), not just REST - every chosen upgrade-from version tested on linux+macos+windows (22 cells) - shared query_rows/dist_from_rows helpers; fallback only when no telemetry backend
|
✅ Green on CI (run 26685728525) — versions now picked dynamically from live telemetry, ≥90% user coverage, tested on all 3 OSes. The Update matrix
Fresh-install ✅ on all 3 OSes + Insiders + code-server. |
…e upgrade, Linux/xvfb)
… backstop so it can't hang the job Electron forks a process tree; killing only AppRun left children that kept xvfb-run (and the job) alive until the 40min GHA timeout. Now: launch detached (own pgid), SIGTERM+SIGKILL the whole group on teardown, and a node-level hard deadline that writes a timeout RESULT_JSON and force-exits. Job timeout 40->20m.
…imeout -s KILL + guaranteed RESULT_JSON Root cause (verified from log): execFileSync on Cursor's CLI shim has no timeout; the shim can launch the GUI and never return, blocking the event loop so the async backstop never fires -> 20min job timeout. Fix: 90s timeout+SIGKILL on every CLI call; a 12m 'timeout -s KILL' around the whole node run; pkill stragglers; and the shell synthesizes a fail RESULT_JSON if node never wrote one.
…CLI shim hangs) Cursor's bundled --install-extension hangs headless. Replace it: download each .vsix (deps + baseline from Open VSX by id/linux-x64, target from local build) and unzip into <ext-dir>/<pub>.<name>-<ver>/. Verified locally. Also corrects the earlier wrong claim that ms-python.python isn't on Open VSX — it is (2026.4.0).
… marker, upload as artifact, and emit activation diagnostics on miss
…t landed A single transient --install-extension hiccup (empty version resolution for older platform-specific builds) failed a blocking cell and the whole gate. Retry with backoff and verify via --list-extensions so flakes self-heal. Same macos 0.60.7 cell passed in a prior run with no code change -> confirmed flake, not a regression.
|
✅ Cursor fork lane is green (run 26691386045) — fresh install and upgrade pass; the extension activates headless inside Cursor 3.6.21.
Update matrix: all 6 baselines × {linux, windows, macos} ✅, cursor upgrade ✅, code-server ✅. What it took (each diagnosed from real CI logs):
Correction: an earlier note claimed Cursor stays non-blocking ( |
…ntime) provision/windsurf.sh downloads the pinned tarball from windsurf-stable.codeium.com manifest API, verifies sha256, extracts (launcher Windsurf/windsurf). cursor-cell.mjs generalized with --runtime so the same proven install(Open VSX download+unzip) + detached-launch + stdout-scan logic serves Windsurf. Non-blocking, Linux. Install path verified locally; CI validates headless launch+activation.
|
✅ Windsurf lane added and green (run 26692184471) — fresh install and upgrade pass on the first try; the extension activates headless in Windsurf 2.3.15.
Update matrix: 6 baselines × 3 OS for VSCode ✅, cursor/windsurf/code-server upgrade ✅. Windsurf worked immediately because Editor coverage now ~93% of the install base (VSCode 79% + Cursor 13.5% + Windsurf 0.4%, plus code-server + Insiders). |
…e kiro) provision/kiro.sh resolves the real .tar.gz from prod.download.desktop.kiro.dev metadata (the metadata lists cert/tar.gz/sig under one version — pick the tar.gz, not max-version, which was grabbing certificate.pem). Launcher Kiro/kiro verified. Non-blocking Linux lane wired into aggregate. Editor coverage now ~94.4%.
- cursor-cell.mjs: retry the fork launch ONCE when it looks like a launch crash (no ext host + tiny log, e.g. the Cursor DBus instant-death flake); never retry when the host loaded but the dbt marker wasn't reached (real signal). - test-matrix/run-local.sh: run the matrix locally — VSCode lane (fresh+upgrade) natively, code-server via Docker, aggregator+board; forks are Linux/CI-only.
…already prints it)
…rks) locally == CI - test-matrix/Dockerfile: Ubuntu 24.04 + Node 20 + Python + xvfb + Electron libs, mirrors ubuntu-latest; runs the same run-local.sh. - run-local.sh now RUNS the Cursor/Windsurf/Kiro lanes when on Linux (i.e. inside the container), skips them with a pointer on macOS. - run-in-docker.sh: build + run the whole matrix in the container (linux/amd64), surfacing the board to .matrix-docker-out/. 'green here == green in CI'.
…stall needs it) npm ci runs 'node postInstall.js' which downloadZMQ()s + reads @jupyterlab files from the repo; a package.json-only COPY made it exit 1. Copy the whole repo first (host node_modules/.vscode-test/venv excluded via scoped .dockerignore), exactly mirroring CI's checkout -> npm ci order.
…ane in xvfb on Linux
…ed prod channel without approval The Slack step fell back to the repo's SLACK_WEBHOOK_URL secret (= prod channel altimate-eng-alerts-prod), which was never approved for matrix notifications. Remove it outright until the matrix is thoroughly tested and a dedicated channel is approved. No SLACK_WEBHOOK_URL reference remains; re-enable instructions in a comment (dedicated MATRIX_SLACK_WEBHOOK only, never a fallback).
… QEMU
Local CI-parity run (test-matrix/run-in-docker.sh) now passes 8/8 cells:
vscode/cursor/windsurf/kiro × {fresh install, upgrade from baseline}.
Fixes found by running the full matrix in the parity container on Apple
Silicon (linux/amd64 under Docker Desktop QEMU):
- run-in-docker.sh: guard empty arrays (${arr[@]+...}) so bash 3.2 under
set -u doesn't abort before launching the container; forward
MATRIX_MAX_VERSIONS into the container.
- run-local.sh: don't let an indirect-var expansion (${!BINVAR}) abort the
whole fork stage under set -u; capture provisioner stderr and skip cleanly
on failure instead of crashing the run.
- provision/cursor.sh: extract Cursor's AppImage without executing it —
the x86_64 self-extract runtime returns 126 under QEMU. Read the squashfs
offset from the ELF header and unsquashfs it. No-op on native CI runners.
- Dockerfile: add squashfs-tools + binutils for that no-exec extraction.
- cursor-cell.mjs: resolve --target latest from Open VSX (linux-x64) like
the VSCode lane instead of treating 'latest' as a file path; harden the
downloader with --retry-all-errors (Open VSX /file/ 302s to a CDN that
--retry alone won't re-attempt after the redirect).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…r repro) Verified jest reproduction of an incomplete-redaction bug in TelemetryService.removeGenericSecretsFromStackTrace (src/telemetry/index.ts): the regex lacks /g and masks only the keyword (not the value), so only the first secret in a multi-line stack is touched and later ones reach App Insights. 3 tests, all green, eslint clean: two pin current behaviour, one test.failing tripwire for the fix. No source change applied. Local only. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…d jest tests Overnight bug-hunt (local only). 6 repro suites / 45 tests, all green, eslint clean. Each claim re-verified against real source; agent overclaims corrected in test-matrix/findings/OVERNIGHT_RESULTS.md. Confirmed real bugs (repro pins behaviour; no source fix applied): - Formatter EOF marker leaks into formatted SQL — dead isDeleteChange guard in processDiffOutput (real TS). formatDbtModelApplyDiffError 31k/30d. - dbt Fusion executeSQL per-line JSON.parse throws 'Unexpected end of JSON input' on interior blank lines (no try/catch, unlike sibling paths) — @altimateai/dbt-integration. catchAllError 131k/30d. - Manifest warnings parse silently drops malformed lines (parseJSON returns undefined, filtered out) — @altimateai/dbt-integration. 74k/30d. Working-as-intended guards: sqlfmt-not-found (35k), cross-IDE helper invariance. Downgraded: extendErrorWithSupportLinks is lossy coercion only (agent's 'throws on nullish via error[-1]' root cause was hallucinated; real source has no such code). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… (my downgrade was wrong) Re-verified src/utils.ts:96-101 + ran the exact expression: error[-1] throws TypeError on undefined/null (the catchAll paths pass (err as Error).message = undefined for non-Error throws) and the dead negative-index dedup doubles spaces. The agent's is_real_bug=true was correct; my earlier 'downgrade' hallucinated that the source lacked error[-1]. Net: 5 real bugs (A formatter EOF leak, B Fusion JSON.parse, C manifest silent-drop, D this, + telemetry scrubber). Tests unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Consolidated write-up of the 5 telemetry-reproduced bugs with root cause, field volume, fix, and repro-test path each. Filed as umbrella Jira AI-6873 (Medium, assigned Ralph). 3 in-repo (telemetry scrubber, formatter EOF, extendErrorWithSupportLinks), 2 upstream @altimateai/dbt-integration (Fusion JSON.parse, manifest drop). Bugs 3 and 4 match GitHub issues #1887 and #1579. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…H issues AI-6873 covers only the newly-discovered bugs (telemetry scrubber, formatter EOF marker, extendErrorWithSupportLinks throw — all in-repo). Bugs #3 and #4 already have GitHub issues #1887 and #1579, so no new ticket; the repro tests serve as regression coverage instead. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
| GitGuardian id | GitGuardian status | Secret | Commit | Filename | |
|---|---|---|---|---|---|
| 33671392 | Triggered | Generic Password | c161f4f | src/test/suite/telemetryRedaction.repro.test.ts | View secret |
🛠 Guidelines to remediate hardcoded secrets
- Understand the implications of revoking this secret by investigating where it is used in your code.
- Replace and store your secret safely. Learn here the best practices.
- Revoke and rotate this secret.
- If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.
To avoid such incidents in the future consider
- following these best practices for managing and storing secrets including API keys and other credentials
- install secret detection on pre-commit to catch secret before it leaves your machine and ease remediation.
🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.
…enario labels
The board now answers 'how many users does each cell affect': each upgrade cell
shows pass/fail + the share of the running base on that version×OS, with a
ranked 'Blocking upgrade failures (by user impact)' callout. Install/update
tables carry explicit scenario sentences (fresh = clean install; update =
direct-to-latest, one hop — VS Code does not upgrade sequentially).
- active-versions.py: add per-(version,os) install-distribution query (common.os
-> macos/windows/linux) + impact_by_version_os() share map.
- build-matrix.py: compute the impact map, emit as impact= output ({} on
fallback so CI never breaks).
- workflow: plan job writes impact.json + uploads as 'impact-map' artifact;
aggregate downloads it (continue-on-error) and passes --impact-file.
- aggregate.py: render per-cell %, per-OS fresh-install shares, impact-ranked
failures; degrades to no-% if the map is absent. No Slack.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- aggregate.py: split the dense VSCode version×OS grid (newest baseline first, centered cells) from the single-baseline fork/code-server rows (compact list), add a status+counts headline and a legend, rank blocking failures by user impact. Fixes the cramped 7-col wall-of-dashes layout. - workflow: write agg/matrix.md to $GITHUB_STEP_SUMMARY so the board renders as formatted tables on the run Summary page (no artifact download), pass or fail. No Slack. Display-only — does not change what is tested. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- aggregate.py: split update grid by runtime CLASS (blocking vs fork), not baseline count — a single-baseline vscode run no longer mislabels its rows under 'Forks & code-server'. - aggregate.py: resolve the install-board target to the newest semver in the impact map when cells label as 'latest'/'pr-build' (CI case), so the per-OS target share actually renders instead of silently dropping. - aggregate.py: non-semver baselines now sort to the END of the newest-first grid (reverse=True alone pushed them to the front). - active-versions.py: impact_by_version_os(published=) filters junk/fork version strings (e.g. 1.2.16) out of the emitted map so they can't be picked as the 'newest target'; denominator stays the full running base. build-matrix passes the published set. KQL_BY_OS cap 200->2000 so the impact denominator isn't truncated. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…esolved board title #1 Windsurf provisioning hardcoded Windsurf/windsurf, which broke silently when Windsurf was repackaged under the Cognition 'Devin' name (Devin/devin-desktop). Discover the launcher dynamically (top-level ELF, fall back to bin/ CLI) in both windsurf.sh and kiro.sh so a future repackage can't break the lane unnoticed. #2 Fork cells wrote a hardcoded reason 'cell wedged — killed by outer timeout' + duration_s 720 on ANY failure — masking a fast provisioning error as a 12-min hang. set -e never caught the failure because provisioning ran inside $(...). Now cursor/windsurf/kiro steps check provisioning explicitly, capture the real stderr reason, and record measured duration. #3 Board title showed 'target latest' while the body resolved to a real version; the count said 'N failed' for non-blocking fork⚠️ . Title now resolves to the semver (e.g. '0.61.6 (latest)') and the count separates blocking ❌ from non-blocking⚠️ . Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
npm run docker:deploy failed on macOS Docker Desktop with 'mounts denied': the dbt-project volume defaulted its HOST side to /home/coder/jaffle-shop-duckdb (a CONTAINER path), which isn't shared from the host. Default it to the compose dir as a harmless placeholder so the bind always resolves; start-code-server.sh falls back to the seeded built-in project (placeholder has no dbt_project.yml). Also fix the fallback project path in start-code-server.sh: it used jaffle_shop_duckdb (underscores) while the seeding loop creates jaffle-shop-duckdb (hyphens), so the fallback opened a non-existent folder. Gate on dbt_project.yml presence instead of a bare non-empty check. Verified: 'docker compose up --build -d' with no .env now serves localhost:3001, opens the jaffle project, and registers innoverio.vscode-dbt-power-user. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
✅ Tests — All Passed |
What
Makes the install/update matrix's upgrade baselines data-driven from the extension's live Azure App Insights version distribution, instead of a hardcoded list.
test-matrix/active-versions.py— queries App Insights for the per-install latest version (which versions are still running), filters to real Open VSX releases (drops junk telemetry strings like1.2.16), and picks baselines = most-run older versions + the oldest still-meaningful one.test-matrix/build-matrix.py— emits the GitHub Actions matrixincludefrom those baselines; a newplanjob feeds it tovscode-cellsviafromJSON.APPINSIGHTS_API_KEYsecret is set; hardcoded fallback otherwise (fork PRs / telemetry outage), so CI never breaks on missing telemetry.Current live pick (2026-05-30): target
0.61.5, baselines[0.55.5, 0.60.7, 0.61.0, 0.61.2, 0.61.3, 0.61.4]— tracks the real population (0.61.5 41% / 0.61.4 21% / 0.61.2 10% / 0.60.7 7.5%).Why
The upgrade matrix should test the versions customers actually run today, and follow them as the population shifts — not a list that goes stale.
Tests
22 unit tests (aggregate + active-versions + build-matrix) pass; workflow is actionlint-clean. This is part 1 of P2; the Cursor fork lane follows in a separate PR (blueprint researched: Playwright
electron.launchon an extracted Cursor AppImage).Pending
APPINSIGHTS_API_KEYrepo secret (App Insights read key) to enable live mode; until then the fallback list is used.🤖 Generated with Claude Code