Skip to content

fix(retro): count pytest/minitest test_*.py in the test-file census - #2812

Open
Jino00 wants to merge 1 commit into
garrytan:mainfrom
Jino00:fix/retro-pytest-test-file-census
Open

fix(retro): count pytest/minitest test_*.py in the test-file census#2812
Jino00 wants to merge 1 commit into
garrytan:mainfrom
Jino00:fix/retro-pytest-test-file-census

Conversation

@Jino00

@Jino00 Jino00 commented Sep 6, 2026

Copy link
Copy Markdown

The problem

bin/gstack-retro-metrics counts the repo-wide test census with suffix forms only:

_TF_TOTAL=$(git ls-files | grep -cE '(\.test\.|\.spec\.|_test\.|_spec\.)')

That covers JS/TS (.test. / .spec.), Go (_test.) and RSpec (_spec.), but misses pytest's default discovery name. pytest ships with python_files = test_*.py *_test.py, and the test_*.py prefix form is the dominant one in the wild — unittest and minitest use the same shape.

So a Python repo whose entire suite is tests/test_*.py gets a near-zero TEST_FILES_TOTAL while TEST_RATIO on the same run is healthy. Two lines of one retro disagree, and the wrong one is the low one.

That is not cosmetic: retro/sections/report-format.md feeds this count into a judgment —

If test ratio < 20%: flag as growth area — "100% test coverage is the goal."

— and the Test Health section reports Total test files: N straight to the user.

Measured on a Python monorepo (FastAPI + React, ~20k tests):

before after
TEST_FILES_TOTAL 6 1,389
TEST_RATIO (same run) 38% 38%

The 6 were the handful of *_test.py / .test.js files that happened to match. Everything under backend/tests/test_*.py was invisible.

The fix

One clause added to the census pattern:

-_TF_TOTAL=$(git ls-files | grep -cE '(\.test\.|\.spec\.|_test\.|_spec\.)')
+_TF_TOTAL=$(git ls-files | grep -cE '(\.test\.|\.spec\.|_test\.|_spec\.|(^|/)test_[^/]*\.(py|rb)$)')

The existing comment said "Deliberately suffix-only (narrower than is_test's dir-based patterns)". I kept that intent and updated the wording, because the reason for the narrowness is not counting dir-homed helpers — and a filename prefix does not readmit them:

  • tests/test_login.py counts (a test file by convention)
  • tests/utils.py still does not count (dir-homed helper — is_test handles those for the insertion ratio)
  • app/testing.py still does not count (test prefix, but not test_)

Verified against the same monorepo: of the 1,389 matches, 1,373 are under a tests/ directory and the remaining 16 are real test files living elsewhere (e.g. frontend/src/lib/*.test.js). No helper files were readmitted.

Test

New hermetic case in test/gstack-retro-metrics.test.ts, in the existing edges describe. It builds a throwaway repo with pytest- and minitest-named files plus the two files that must stay out, and pins TEST_FILES_TOTAL: 4.

The existing TEST_FILES_TOTAL: 1 assertion and its fixture are untouched.

bun test test/gstack-retro-metrics.test.ts
 12 pass
 0 fail

Also verified on a fresh clone of the fork with no bun install — the case only uses bun:test plus node builtins.

Mutation-checked: removing the new clause from the pattern fails the new case (11 pass / 1 fail) and nothing else. Restoring it returns 12/12. The test bites the line it is meant to protect.

Not in scope

  • is_test (the dir-based insertion-ratio matcher) is unchanged.
  • REGRESSION_TEST_COMMITS is unchanged — it greps test(qa): / test(design): / test: coverage commit subjects, and reporting 0 for a repo that doesn't use those prefixes is correct behaviour, not a bug.
  • No VERSION bump — leaving release mechanics to the maintainers.

🤖 Generated with Claude Code

The repo-wide census in gstack-retro-metrics matched suffix forms only
(.test. .spec. _test. _spec.), so a Python repo following pytest's default
discovery (`python_files = test_*.py *_test.py`) reported a near-zero
TEST_FILES_TOTAL while TEST_RATIO on the same run was healthy. Two lines of
the same retro disagreed, and the low one was silently wrong -- the skill
feeds that count into its "test ratio < 20% => flag as growth area" nudge.

Measured on a Python monorepo: 6 -> 1,389 test files. Of the 1,389, only 16
sit outside a tests/ directory, and none of those are dir-homed helpers.

The added clause is a filename convention, not a directory one, so it does
not readmit the helpers the census deliberately excludes (tests/utils.py
still does not count; app/testing.py still does not count). Comment updated
from "suffix-only" to state the actual rule.

Test: new hermetic case builds a pytest/minitest-style repo and pins
TEST_FILES_TOTAL: 4 while asserting the two non-test files stay out.
Removing the new clause fails it (mutation-checked). 12/12 pass.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@trunk-io

trunk-io Bot commented Sep 6, 2026

Copy link
Copy Markdown

Merging to main in this repository is managed by Trunk.

  • To merge this pull request, check the box to the left or comment /trunk merge below.

After your PR is submitted to the merge queue, this comment will be automatically updated with its status. If the PR fails, failure details will also be posted here

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant