fix(retro): count pytest/minitest test_*.py in the test-file census - #2812
Open
Jino00 wants to merge 1 commit into
Open
fix(retro): count pytest/minitest test_*.py in the test-file census#2812Jino00 wants to merge 1 commit into
Jino00 wants to merge 1 commit into
Conversation
The repo-wide census in gstack-retro-metrics matched suffix forms only (.test. .spec. _test. _spec.), so a Python repo following pytest's default discovery (`python_files = test_*.py *_test.py`) reported a near-zero TEST_FILES_TOTAL while TEST_RATIO on the same run was healthy. Two lines of the same retro disagreed, and the low one was silently wrong -- the skill feeds that count into its "test ratio < 20% => flag as growth area" nudge. Measured on a Python monorepo: 6 -> 1,389 test files. Of the 1,389, only 16 sit outside a tests/ directory, and none of those are dir-homed helpers. The added clause is a filename convention, not a directory one, so it does not readmit the helpers the census deliberately excludes (tests/utils.py still does not count; app/testing.py still does not count). Comment updated from "suffix-only" to state the actual rule. Test: new hermetic case builds a pytest/minitest-style repo and pins TEST_FILES_TOTAL: 4 while asserting the two non-test files stay out. Removing the new clause fails it (mutation-checked). 12/12 pass. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Merging to
After your PR is submitted to the merge queue, this comment will be automatically updated with its status. If the PR fails, failure details will also be posted here |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The problem
bin/gstack-retro-metricscounts the repo-wide test census with suffix forms only:_TF_TOTAL=$(git ls-files | grep -cE '(\.test\.|\.spec\.|_test\.|_spec\.)')That covers JS/TS (
.test./.spec.), Go (_test.) and RSpec (_spec.), but misses pytest's default discovery name. pytest ships withpython_files = test_*.py *_test.py, and thetest_*.pyprefix form is the dominant one in the wild — unittest and minitest use the same shape.So a Python repo whose entire suite is
tests/test_*.pygets a near-zeroTEST_FILES_TOTALwhileTEST_RATIOon the same run is healthy. Two lines of one retro disagree, and the wrong one is the low one.That is not cosmetic:
retro/sections/report-format.mdfeeds this count into a judgment —— and the Test Health section reports
Total test files: Nstraight to the user.Measured on a Python monorepo (FastAPI + React, ~20k tests):
TEST_FILES_TOTALTEST_RATIO(same run)The
6were the handful of*_test.py/.test.jsfiles that happened to match. Everything underbackend/tests/test_*.pywas invisible.The fix
One clause added to the census pattern:
The existing comment said "Deliberately suffix-only (narrower than
is_test's dir-based patterns)". I kept that intent and updated the wording, because the reason for the narrowness is not counting dir-homed helpers — and a filename prefix does not readmit them:tests/test_login.pycounts (a test file by convention)tests/utils.pystill does not count (dir-homed helper —is_testhandles those for the insertion ratio)app/testing.pystill does not count (testprefix, but nottest_)Verified against the same monorepo: of the 1,389 matches, 1,373 are under a
tests/directory and the remaining 16 are real test files living elsewhere (e.g.frontend/src/lib/*.test.js). No helper files were readmitted.Test
New hermetic case in
test/gstack-retro-metrics.test.ts, in the existingedgesdescribe. It builds a throwaway repo with pytest- and minitest-named files plus the two files that must stay out, and pinsTEST_FILES_TOTAL: 4.The existing
TEST_FILES_TOTAL: 1assertion and its fixture are untouched.Also verified on a fresh clone of the fork with no
bun install— the case only usesbun:testplus node builtins.Mutation-checked: removing the new clause from the pattern fails the new case (11 pass / 1 fail) and nothing else. Restoring it returns 12/12. The test bites the line it is meant to protect.
Not in scope
is_test(the dir-based insertion-ratio matcher) is unchanged.REGRESSION_TEST_COMMITSis unchanged — it grepstest(qa):/test(design):/test: coveragecommit subjects, and reporting0for a repo that doesn't use those prefixes is correct behaviour, not a bug.VERSIONbump — leaving release mechanics to the maintainers.🤖 Generated with Claude Code