Skip to content

Matrix P2 (part 1): data-driven upgrade baselines from live telemetry - #1987

Open
dev-punia-altimate wants to merge 43 commits into
feat/vsix-install-smokefrom
matrix/cursor-lane-p2
Open

Matrix P2 (part 1): data-driven upgrade baselines from live telemetry#1987
dev-punia-altimate wants to merge 43 commits into
feat/vsix-install-smokefrom
matrix/cursor-lane-p2

Conversation

@dev-punia-altimate

Copy link
Copy Markdown
Contributor

What

Makes the install/update matrix's upgrade baselines data-driven from the extension's live Azure App Insights version distribution, instead of a hardcoded list.

  • test-matrix/active-versions.py — queries App Insights for the per-install latest version (which versions are still running), filters to real Open VSX releases (drops junk telemetry strings like 1.2.16), and picks baselines = most-run older versions + the oldest still-meaningful one.
  • test-matrix/build-matrix.py — emits the GitHub Actions matrix include from those baselines; a new plan job feeds it to vscode-cells via fromJSON.
  • Live when APPINSIGHTS_API_KEY secret is set; hardcoded fallback otherwise (fork PRs / telemetry outage), so CI never breaks on missing telemetry.

Current live pick (2026-05-30): target 0.61.5, baselines [0.55.5, 0.60.7, 0.61.0, 0.61.2, 0.61.3, 0.61.4] — tracks the real population (0.61.5 41% / 0.61.4 21% / 0.61.2 10% / 0.60.7 7.5%).

Why

The upgrade matrix should test the versions customers actually run today, and follow them as the population shifts — not a list that goes stale.

Tests

22 unit tests (aggregate + active-versions + build-matrix) pass; workflow is actionlint-clean. This is part 1 of P2; the Cursor fork lane follows in a separate PR (blueprint researched: Playwright electron.launch on an extracted Cursor AppImage).

Pending

  • APPINSIGHTS_API_KEY repo secret (App Insights read key) to enable live mode; until then the fallback list is used.

🤖 Generated with Claude Code

@coderabbitai

coderabbitai Bot commented May 30, 2026

Copy link
Copy Markdown
Contributor

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 2f38df43-6577-48d5-925f-51de40f71267

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch matrix/cursor-lane-p2

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@github-actions

github-actions Bot commented May 30, 2026

Copy link
Copy Markdown

VSIX Install + Update Matrix

❌ No result files found — treating as a blocking failure

… upgrade-from version

- pick_by_coverage: add most-used versions until target+selected users >= 90%
- build-matrix: choose versions DYNAMICALLY via az (local) or APPINSIGHTS_API_KEY (CI), not just REST
- every chosen upgrade-from version tested on linux+macos+windows (22 cells)
- shared query_rows/dist_from_rows helpers; fallback only when no telemetry backend
@dev-punia-altimate

Copy link
Copy Markdown
Contributor Author

Green on CI (run 26685728525) — versions now picked dynamically from live telemetry, ≥90% user coverage, tested on all 3 OSes.

The plan job summed each version's user count (target 0.61.5 → +0.61.4 → +0.61.2 → +0.60.7 → +0.61.3 → +0.61.0 → +0.61.1) until the running total of users crossed 90.1% of the ~47k active installs, then ran every chosen upgrade-from version on linux/macOS/windows.

Update matrix

Runtime / OS 0.60.7 0.61.0 0.61.1 0.61.2 0.61.3 0.61.4
vscode (linux)
vscode (windows)
vscode (macos)

Fresh-install ✅ on all 3 OSes + Insiders + code-server. source=live via az locally / APPINSIGHTS_API_KEY in CI; hardcoded fallback only when no telemetry backend (fork PRs). 26 unit tests pass.

… backstop so it can't hang the job

Electron forks a process tree; killing only AppRun left children that kept
xvfb-run (and the job) alive until the 40min GHA timeout. Now: launch detached
(own pgid), SIGTERM+SIGKILL the whole group on teardown, and a node-level hard
deadline that writes a timeout RESULT_JSON and force-exits. Job timeout 40->20m.
…imeout -s KILL + guaranteed RESULT_JSON

Root cause (verified from log): execFileSync on Cursor's CLI shim has no timeout;
the shim can launch the GUI and never return, blocking the event loop so the async
backstop never fires -> 20min job timeout. Fix: 90s timeout+SIGKILL on every CLI
call; a 12m 'timeout -s KILL' around the whole node run; pkill stragglers; and the
shell synthesizes a fail RESULT_JSON if node never wrote one.
…CLI shim hangs)

Cursor's bundled --install-extension hangs headless. Replace it: download each
.vsix (deps + baseline from Open VSX by id/linux-x64, target from local build)
and unzip into <ext-dir>/<pub>.<name>-<ver>/. Verified locally. Also corrects the
earlier wrong claim that ms-python.python isn't on Open VSX — it is (2026.4.0).
… marker, upload as artifact, and emit activation diagnostics on miss
…t landed

A single transient --install-extension hiccup (empty version resolution for older
platform-specific builds) failed a blocking cell and the whole gate. Retry with
backoff and verify via --list-extensions so flakes self-heal. Same macos 0.60.7
cell passed in a prior run with no code change -> confirmed flake, not a regression.
@dev-punia-altimate

Copy link
Copy Markdown
Contributor Author

Cursor fork lane is green (run 26691386045) — fresh install and upgrade pass; the extension activates headless inside Cursor 3.6.21.

Runtime linux windows macos
vscode
vscode-insiders
cursor
code-server

Update matrix: all 6 baselines × {linux, windows, macos} ✅, cursor upgrade ✅, code-server ✅.

What it took (each diagnosed from real CI logs):

  1. Cursor's bundled CLI --install-extension hangs headless → bypassed it: download each .vsix from Open VSX (deps by id + baseline linux-x64) and unzip into the extensions-dir.
  2. Electron forks a process tree that kept xvfb-run alive → launch detached in its own process group + group-kill + outer timeout -s KILL.
  3. The "Initialized dbt project" marker goes to Cursor's stdout, not <user-data-dir>/logs/*.log → capture stdout and scan it (this is what turned Cursor ⚠️→✅).
  4. A transient marketplace flake (empty-version resolution) failed the macOS 0.60.7 cell → retry baseline installs 3× and verify via --list-extensions.

Correction: an earlier note claimed ms-python.python isn't on Open VSX — it is (2026.4.0), as are the other two deps. No fork dependency-portability gap.

Cursor stays non-blocking (⚠️ on failure, never blocks the gate); only stock VSCode gates.

@dev-punia-altimate
dev-punia-altimate marked this pull request as ready for review May 30, 2026 18:52
…ntime)

provision/windsurf.sh downloads the pinned tarball from windsurf-stable.codeium.com
manifest API, verifies sha256, extracts (launcher Windsurf/windsurf). cursor-cell.mjs
generalized with --runtime so the same proven install(Open VSX download+unzip) +
detached-launch + stdout-scan logic serves Windsurf. Non-blocking, Linux. Install
path verified locally; CI validates headless launch+activation.
@dev-punia-altimate

Copy link
Copy Markdown
Contributor Author

Windsurf lane added and green (run 26692184471) — fresh install and upgrade pass on the first try; the extension activates headless in Windsurf 2.3.15.

Runtime linux windows macos
vscode
vscode-insiders
cursor
windsurf
code-server

Update matrix: 6 baselines × 3 OS for VSCode ✅, cursor/windsurf/code-server upgrade ✅.

Windsurf worked immediately because cursor-cell.mjs was generalized with --runtime — the hard parts (CLI-hang bypass via Open VSX download+unzip, detached-launch process-group kill, stdout marker-scan) are shared. provision/windsurf.sh downloads the sha256-verified tarball from the windsurf-stable.codeium.com manifest API. Non-blocking, like Cursor.

Editor coverage now ~93% of the install base (VSCode 79% + Cursor 13.5% + Windsurf 0.4%, plus code-server + Insiders).

dev-punia-altimate and others added 13 commits May 31, 2026 17:28
…e kiro)

provision/kiro.sh resolves the real .tar.gz from prod.download.desktop.kiro.dev
metadata (the metadata lists cert/tar.gz/sig under one version — pick the tar.gz,
not max-version, which was grabbing certificate.pem). Launcher Kiro/kiro verified.
Non-blocking Linux lane wired into aggregate. Editor coverage now ~94.4%.
- cursor-cell.mjs: retry the fork launch ONCE when it looks like a launch crash
  (no ext host + tiny log, e.g. the Cursor DBus instant-death flake); never retry
  when the host loaded but the dbt marker wasn't reached (real signal).
- test-matrix/run-local.sh: run the matrix locally — VSCode lane (fresh+upgrade)
  natively, code-server via Docker, aggregator+board; forks are Linux/CI-only.
…rks) locally == CI

- test-matrix/Dockerfile: Ubuntu 24.04 + Node 20 + Python + xvfb + Electron libs,
  mirrors ubuntu-latest; runs the same run-local.sh.
- run-local.sh now RUNS the Cursor/Windsurf/Kiro lanes when on Linux (i.e. inside
  the container), skips them with a pointer on macOS.
- run-in-docker.sh: build + run the whole matrix in the container (linux/amd64),
  surfacing the board to .matrix-docker-out/. 'green here == green in CI'.
…stall needs it)

npm ci runs 'node postInstall.js' which downloadZMQ()s + reads @jupyterlab files
from the repo; a package.json-only COPY made it exit 1. Copy the whole repo first
(host node_modules/.vscode-test/venv excluded via scoped .dockerignore), exactly
mirroring CI's checkout -> npm ci order.
…ed prod channel without approval

The Slack step fell back to the repo's SLACK_WEBHOOK_URL secret (= prod channel
altimate-eng-alerts-prod), which was never approved for matrix notifications.
Remove it outright until the matrix is thoroughly tested and a dedicated channel
is approved. No SLACK_WEBHOOK_URL reference remains; re-enable instructions in a
comment (dedicated MATRIX_SLACK_WEBHOOK only, never a fallback).
… QEMU

Local CI-parity run (test-matrix/run-in-docker.sh) now passes 8/8 cells:
vscode/cursor/windsurf/kiro × {fresh install, upgrade from baseline}.

Fixes found by running the full matrix in the parity container on Apple
Silicon (linux/amd64 under Docker Desktop QEMU):

- run-in-docker.sh: guard empty arrays (${arr[@]+...}) so bash 3.2 under
  set -u doesn't abort before launching the container; forward
  MATRIX_MAX_VERSIONS into the container.
- run-local.sh: don't let an indirect-var expansion (${!BINVAR}) abort the
  whole fork stage under set -u; capture provisioner stderr and skip cleanly
  on failure instead of crashing the run.
- provision/cursor.sh: extract Cursor's AppImage without executing it —
  the x86_64 self-extract runtime returns 126 under QEMU. Read the squashfs
  offset from the ELF header and unsquashfs it. No-op on native CI runners.
- Dockerfile: add squashfs-tools + binutils for that no-exec extraction.
- cursor-cell.mjs: resolve --target latest from Open VSX (linux-x64) like
  the VSCode lane instead of treating 'latest' as a file path; harden the
  downloader with --retry-all-errors (Open VSX /file/ 302s to a CDN that
  --retry alone won't re-attempt after the redirect).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…r repro)

Verified jest reproduction of an incomplete-redaction bug in
TelemetryService.removeGenericSecretsFromStackTrace (src/telemetry/index.ts):
the regex lacks /g and masks only the keyword (not the value), so only the first
secret in a multi-line stack is touched and later ones reach App Insights.
3 tests, all green, eslint clean: two pin current behaviour, one test.failing
tripwire for the fix. No source change applied. Local only.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…d jest tests

Overnight bug-hunt (local only). 6 repro suites / 45 tests, all green, eslint clean.
Each claim re-verified against real source; agent overclaims corrected in
test-matrix/findings/OVERNIGHT_RESULTS.md.

Confirmed real bugs (repro pins behaviour; no source fix applied):
- Formatter EOF marker leaks into formatted SQL — dead isDeleteChange guard in
  processDiffOutput (real TS). formatDbtModelApplyDiffError 31k/30d.
- dbt Fusion executeSQL per-line JSON.parse throws 'Unexpected end of JSON input'
  on interior blank lines (no try/catch, unlike sibling paths) — @altimateai/dbt-integration.
  catchAllError 131k/30d.
- Manifest warnings parse silently drops malformed lines (parseJSON returns undefined,
  filtered out) — @altimateai/dbt-integration. 74k/30d.

Working-as-intended guards: sqlfmt-not-found (35k), cross-IDE helper invariance.
Downgraded: extendErrorWithSupportLinks is lossy coercion only (agent's 'throws on
nullish via error[-1]' root cause was hallucinated; real source has no such code).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… (my downgrade was wrong)

Re-verified src/utils.ts:96-101 + ran the exact expression: error[-1] throws
TypeError on undefined/null (the catchAll paths pass (err as Error).message =
undefined for non-Error throws) and the dead negative-index dedup doubles spaces.
The agent's is_real_bug=true was correct; my earlier 'downgrade' hallucinated that
the source lacked error[-1]. Net: 5 real bugs (A formatter EOF leak, B Fusion
JSON.parse, C manifest silent-drop, D this, + telemetry scrubber). Tests unchanged.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Consolidated write-up of the 5 telemetry-reproduced bugs with root cause,
field volume, fix, and repro-test path each. Filed as umbrella Jira AI-6873
(Medium, assigned Ralph). 3 in-repo (telemetry scrubber, formatter EOF,
extendErrorWithSupportLinks), 2 upstream @altimateai/dbt-integration (Fusion
JSON.parse, manifest drop). Bugs 3 and 4 match GitHub issues #1887 and #1579.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…H issues

AI-6873 covers only the newly-discovered bugs (telemetry scrubber, formatter
EOF marker, extendErrorWithSupportLinks throw — all in-repo). Bugs #3 and #4
already have GitHub issues #1887 and #1579, so no new ticket; the repro tests
serve as regression coverage instead.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@gitguardian

gitguardian Bot commented Jun 5, 2026

Copy link
Copy Markdown

⚠️ GitGuardian has uncovered 1 secret following the scan of your pull request.

Please consider investigating the findings and remediating the incidents. Failure to do so may lead to compromising the associated services or software components.

🔎 Detected hardcoded secret in your pull request
GitGuardian id GitGuardian status Secret Commit Filename
33671392 Triggered Generic Password c161f4f src/test/suite/telemetryRedaction.repro.test.ts View secret
🛠 Guidelines to remediate hardcoded secrets
  1. Understand the implications of revoking this secret by investigating where it is used in your code.
  2. Replace and store your secret safely. Learn here the best practices.
  3. Revoke and rotate this secret.
  4. If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.

To avoid such incidents in the future consider


🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.

dev-punia-altimate and others added 5 commits June 5, 2026 22:41
…enario labels

The board now answers 'how many users does each cell affect': each upgrade cell
shows pass/fail + the share of the running base on that version×OS, with a
ranked 'Blocking upgrade failures (by user impact)' callout. Install/update
tables carry explicit scenario sentences (fresh = clean install; update =
direct-to-latest, one hop — VS Code does not upgrade sequentially).

- active-versions.py: add per-(version,os) install-distribution query (common.os
  -> macos/windows/linux) + impact_by_version_os() share map.
- build-matrix.py: compute the impact map, emit as impact= output ({} on
  fallback so CI never breaks).
- workflow: plan job writes impact.json + uploads as 'impact-map' artifact;
  aggregate downloads it (continue-on-error) and passes --impact-file.
- aggregate.py: render per-cell %, per-OS fresh-install shares, impact-ranked
  failures; degrades to no-% if the map is absent. No Slack.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- aggregate.py: split the dense VSCode version×OS grid (newest baseline first,
  centered cells) from the single-baseline fork/code-server rows (compact list),
  add a status+counts headline and a legend, rank blocking failures by user
  impact. Fixes the cramped 7-col wall-of-dashes layout.
- workflow: write agg/matrix.md to $GITHUB_STEP_SUMMARY so the board renders as
  formatted tables on the run Summary page (no artifact download), pass or fail.
No Slack. Display-only — does not change what is tested.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- aggregate.py: split update grid by runtime CLASS (blocking vs fork), not
  baseline count — a single-baseline vscode run no longer mislabels its rows
  under 'Forks & code-server'.
- aggregate.py: resolve the install-board target to the newest semver in the
  impact map when cells label  as 'latest'/'pr-build' (CI case), so the
  per-OS target share actually renders instead of silently dropping.
- aggregate.py: non-semver baselines now sort to the END of the newest-first
  grid (reverse=True alone pushed them to the front).
- active-versions.py: impact_by_version_os(published=) filters junk/fork
  version strings (e.g. 1.2.16) out of the emitted map so they can't be picked
  as the 'newest target'; denominator stays the full running base. build-matrix
  passes the published set. KQL_BY_OS cap 200->2000 so the impact denominator
  isn't truncated.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…esolved board title

#1 Windsurf provisioning hardcoded Windsurf/windsurf, which broke silently when
   Windsurf was repackaged under the Cognition 'Devin' name (Devin/devin-desktop).
   Discover the launcher dynamically (top-level ELF, fall back to bin/ CLI) in both
   windsurf.sh and kiro.sh so a future repackage can't break the lane unnoticed.
#2 Fork cells wrote a hardcoded reason 'cell wedged — killed by outer timeout' +
   duration_s 720 on ANY failure — masking a fast provisioning error as a 12-min
   hang. set -e never caught the failure because provisioning ran inside $(...).
   Now cursor/windsurf/kiro steps check provisioning explicitly, capture the real
   stderr reason, and record measured duration.
#3 Board title showed 'target latest' while the body resolved to a real version;
   the count said 'N failed' for non-blocking fork ⚠️. Title now resolves to the
   semver (e.g. '0.61.6 (latest)') and the count separates blocking ❌ from
   non-blocking ⚠️.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
npm run docker:deploy failed on macOS Docker Desktop with 'mounts denied':
the dbt-project volume defaulted its HOST side to /home/coder/jaffle-shop-duckdb
(a CONTAINER path), which isn't shared from the host. Default it to the compose
dir as a harmless placeholder so the bind always resolves; start-code-server.sh
falls back to the seeded built-in project (placeholder has no dbt_project.yml).

Also fix the fallback project path in start-code-server.sh: it used
jaffle_shop_duckdb (underscores) while the seeding loop creates
jaffle-shop-duckdb (hyphens), so the fallback opened a non-existent folder. Gate
on dbt_project.yml presence instead of a bare non-empty check.

Verified: 'docker compose up --build -d' with no .env now serves localhost:3001,
opens the jaffle project, and registers innoverio.vscode-dbt-power-user.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@dev-punia-altimate

Copy link
Copy Markdown
Contributor Author

✅ Tests — All Passed

cc @dev-punia-altimate

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant