Skip to content

Stabilize version, profile, and package cache tests - #8669

Merged
gonzaloriestra merged 3 commits into
mainfrom
gonzalo/fix-monthly-test-flakes
Oct 2, 2026
Merged

gonzaloriestra merged 3 commits into
mainfrom
gonzalo/fix-monthly-test-flakes

Conversation

@gonzaloriestra

@gonzaloriestra gonzaloriestra commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

WHY are these changes introduced?

The Main tests workflow had 16 failed runs out of 112 from August 25 through September 25, 2026 (105 on main, seven on stable branches). Repeated failures came from shared cache state, an unintended network request, and version subprocess timeouts on Windows.

WHAT is this pull request doing?

  • Give each package-version cache test a real temporary store and use a fixed clock for expiry assertions.
  • Mock crawler-signature requests in theme profile tests so session setup cannot contact the Admin API or continue into later tests after a timeout.
  • Run the real version command with the existing standard-stream capture helper, retaining exact stdout, empty stderr, and launcher schema assertions without starting TypeScript subprocesses.

Audit findings (counts overlap where a run had multiple failures):

Failure Runs Resolution
Version subprocess timeouts 10 In-process output and schema assertions in this PR
Profile timeout / consumed response body 2 Network isolation in this PR
Package-version cache call counts 2 Cache and clock isolation in this PR
Theme watcher EPERM 1 Already fixed in 7ee6fe2419
Real npm audit timeout on stable/4.8 1 Already fixed on main in 6b70f18ffd
Brotli file-filter integration timeout 1 Unresolved: passed locally; logs do not establish the cause

The version subprocess tests were introduced in #8542, merged September 21. Earlier main-workflow logs show the old command test taking 16–25 ms; the new file takes 54–60 seconds on failing Windows runs. Local Node 22 profiling measured roughly 2.3 seconds importing the command through ts-node versus 33 ms executing it. A diagnostic invocation performed 1,638 compiler file reads and 5,295 file-existence checks. The replacement command assertions took 4–13 ms locally and use the default test timeout.

Exact output and schema checks remain in the required unit-test matrix. The version unit tests no longer assert child-process exit codes.

Checklist

  • I've considered possible cross-platform impacts (Mac, Linux, Windows)
  • I've considered possible documentation changes
  • I've considered analytics changes to measure impact
  • The change is user-facing — I've identified the correct bump type (patch for bug fixes · minor for new features · major for breaking changes) and added a changeset with pnpm changeset add

@github-actions github-actions Bot added the no-changelog This PR doesn't include a changeset entry. Is an internal only change not relevant to end users. label Sep 25, 2026
@gonzaloriestra
gonzaloriestra marked this pull request as ready for review September 25, 2026 10:48
@gonzaloriestra
gonzaloriestra requested review from a team as code owners September 25, 2026 10:48
@gonzaloriestra

Copy link
Copy Markdown
Contributor Author

@Shopify/developer-platforms can I get a review here, please? 🙏

@karreiro karreiro left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you, @gonzaloriestra!

@gonzaloriestra
gonzaloriestra added this pull request to the merge queue Oct 2, 2026
Merged via the queue into main with commit 135e12d Oct 2, 2026
30 checks passed
@gonzaloriestra
gonzaloriestra deleted the gonzalo/fix-monthly-test-flakes branch October 2, 2026 09:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

no-changelog This PR doesn't include a changeset entry. Is an internal only change not relevant to end users.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants