Skip to content

DO NOT MERGE: detect-test-pollution - #108281

Closed
joshuarli wants to merge 17 commits into
masterfrom
detect-test-pollution
Closed

joshuarli wants to merge 17 commits into
masterfrom
detect-test-pollution

Conversation

@joshuarli

@joshuarli joshuarli commented Feb 13, 2026 •

Copy link
Copy Markdown
Member

this adds a workflow_dispatch that when given a shuffle tests run url:

  • it extracts TEST_GROUP, TOTAL_TEST_GROUPS, SENTRY_SHUFFLE_TESTS_SEED, SENTRY_REVISION, and the flaky testid from the raw run logs
  • uses that to invoke an exact reproduction of the shuffle test's pytest collection run which is used to collect the exact set and order of testids
  • feeds those testids into detect-test-pollution which automatically bisects down to the polluting tests

this is really useful for parallelizing the bisection of these polluting tests since the alternative would be to have these things going in separate sentry devenvironment VMs which do not exist at the moment

@github-actions github-actions Bot added the Scope: Backend Automatically applied to PRs that change backend components label Feb 13, 2026
joshuarli added a commit that referenced this pull request Feb 14, 2026
seen in this run of shuffle-tests:
https://github.com/getsentry/sentry/actions/runs/21654747518/job/62427093320

i'm working on some tools to use detect-test-pollution to automatically
bisect test sets from shuffle-test runs here:
#108281


https://github.com/getsentry/sentry/actions/runs/22007196153/job/63593472299

that identified that running these two tests in this order:

```
pytest tests/sentry/integrations/github/test_installation.py::InstallationEndpointTest::test_installation_endpoint tests/sentry/utils/test_sdk.py::CheckScopeTransactionTest::test_custom_transaction_name
```

fails consistently without this fix

---

- `test_custom_transaction_name` was mocking the wrong scope — it used
`patch_isolation_scope()` (which patches `Scope.get_isolation_scope()`),
but `check_current_scope_transaction()` reads from
`sentry_sdk.get_current_scope()`. The test passed in isolation only
because the current scope had `_transaction=None`, short-circuiting the
check without ever validating the `source="custom"` logic.
- When preceded by a test like `test_installation_endpoint` that sets a
real transaction on the current scope, the leaked state caused the
assertion to fail.
- Fixed by mocking `sentry_sdk.get_current_scope` instead, consistent
with the other two tests in the same class.
@joshuarli

Copy link
Copy Markdown
Member Author

closing; will be combining with #108286

@joshuarli joshuarli closed this Feb 17, 2026
@github-actions github-actions Bot locked and limited conversation to collaborators Mar 5, 2026
@joshuarli
joshuarli deleted the detect-test-pollution branch March 24, 2026 00:25

This branch was successfully deployed

1 active deployment
Preview — 42159881 Deployed Feb 14, 2026 by vercel[bot]
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

Scope: Backend Automatically applied to PRs that change backend components

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant