Skip to content

Prioritize recent flaky tests in the automated testing task - #8670

Merged
gonzaloriestra merged 1 commit into
mainfrom
tests-prioritize-flaky-tests
Sep 28, 2026
Merged

gonzaloriestra merged 1 commit into
mainfrom
tests-prioritize-flaky-tests

Conversation

@gonzaloriestra

@gonzaloriestra gonzaloriestra commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

WHY are these changes introduced?

The automated testing task starts with coverage gaps, leaving recurring CI failures competing with new test coverage for attention.

WHAT is this pull request doing?

Prioritize flaky-test fixes by reviewing every Main tests workflow run from the last seven days, including failed attempts and matrix jobs. Require evidence, check for existing fixes, investigate the cause, and repeat focused tests. Fall back to coverage improvements only when no actionable, non-duplicate flake remains.

Keep fixes focused and preserve useful assertions instead of hiding failures with retries or larger timeouts. Include representative failed-job links and validation limits in the PR description.

Label maintenance PR notifications as "CLI maintenance PR" so their repository context is clear.

Validated the instruction flow, workflow reference, notification text, and diff formatting. Instruction and message-only changes; no test execution needed.

Checklist

  • I've considered possible cross-platform impacts (Mac, Linux, Windows)
  • I've considered possible documentation changes
  • I've considered analytics changes to measure impact
  • The change is user-facing — I've identified the correct bump type (patch for bug fixes · minor for new features · major for breaking changes) and added a changeset with pnpm changeset add

@github-actions github-actions Bot added the no-changelog This PR doesn't include a changeset entry. Is an internal only change not relevant to end users. label Sep 25, 2026
@gonzaloriestra
gonzaloriestra force-pushed the tests-prioritize-flaky-tests branch from 32b78cb to ff055da Compare September 25, 2026 10:48
@gonzaloriestra
gonzaloriestra marked this pull request as ready for review September 25, 2026 10:50
@gonzaloriestra
gonzaloriestra requested a review from a team as a code owner September 25, 2026 10:50

@Suleimanlatrsh Suleimanlatrsh left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice, this makes the task way more useful. Did you get a chance to try it with a workflow_dispatch run? The permissions block doesn't list actions, so I wasn't sure the bot can read run logs.

@gonzaloriestra

Copy link
Copy Markdown
Contributor Author

@Suleimanlatrsh We can't try it until it gets merged. The repo is public, so I think you don't need permission to read the action logs. But let's merge and adjust if needed!

@gonzaloriestra
gonzaloriestra added this pull request to the merge queue Sep 28, 2026
Merged via the queue into main with commit 3a15440 Sep 28, 2026
30 checks passed
@gonzaloriestra
gonzaloriestra deleted the tests-prioritize-flaky-tests branch September 28, 2026 08:19
@gonzaloriestra

gonzaloriestra commented Sep 29, 2026 •

Copy link
Copy Markdown
Contributor Author

@Suleimanlatrsh it worked, this PR mentions the flaky tests found.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

no-changelog This PR doesn't include a changeset entry. Is an internal only change not relevant to end users.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants