fix(retro-metrics): count Swift/XCTest tests, exclude generated output and prose specs - #2809
Open
snig-17 wants to merge 1 commit into
Open
fix(retro-metrics): count Swift/XCTest tests, exclude generated output and prose specs#2809snig-17 wants to merge 1 commit into
snig-17 wants to merge 1 commit into
Conversation
…t and prose Three related blind spots in test detection, all found running /retro on a SwiftUI project with 243 tests that reported a 0% test ratio. 1. XCTest naming. is_test matched lowercase test/|tests?/|spec/ dirs and .test./_test. suffixes. Swift uses AppTests/StoreTests.swift, which matches neither. Even SwiftPM's standard Tests/ directory missed, because the pattern is case-sensitive. Same blind spot hits C# (Foo.Tests/FooTests.cs). 2. Generated output counted as authored code. A committed build directory put FOCUS_SCORE at 97% (.build/) instead of the real 60% (Nourish/), and .build/Logs/Test/ counted as test LOC. is_generated now drops build output before any counter is touched. 3. "spec" is overloaded. docs/specs/*.md are written specifications, not RSpec tests, and were adding thousands of prose lines to TEST_INSERTIONS. Prose is never test code, whatever directory it sits in. TEST_FILES_CHANGED also used the narrow suffix-only pattern rather than is_test, so it read 0 while 21 test files had changed. On the project that surfaced this, TEST_INSERTIONS and TEST_FILES_CHANGED now match a hand count exactly (2423 and 21). Adds a regression test with its own fixture repo, leaving the shared fixture's deterministic aggregates untouched. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Merging to
After your PR is submitted to the merge queue, this comment will be automatically updated with its status. If the PR fails, failure details will also be posted here |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Running
/retroon a SwiftUI project with 243 tests reported a 0% test ratio and a focus score of 97% (.build/). Three related blind spots ingstack-retro-metrics, each fixable independently.1. XCTest naming is invisible
is_testmatched lowercasetest/|tests?/|spec/|__tests__/directories and.test./.spec./_test./_spec.suffixes.Swift uses
AppTests/StoreTests.swift— capital T, no separator beforeTests— which matches neither. Worth noting that even SwiftPM's standardTests/directory missed, because the directory pattern is case-sensitive. The same gap hits C# (Foo.Tests/FooTests.cs).Added: capitalized
[Tt]ests?//[Ss]pecs?/directories, and a CamelCase filename formFooTests.swift/FooTest.cs. The capital letter is load-bearing, soLatest.swiftandContests.swiftdo not match.2. Generated output counted as authored code
A committed build directory dominated
FOCUS_SCORE(97%.build/rather than the real 60%Nourish/), and.build/Logs/Test/counted as test LOC — build output inflating the test ratio.is_generatednow drops.build/,build/,dist/,out/,target/,vendor/,node_modules/,DerivedData/,.next/,.nuxt/,__pycache__/,coverage/before any counter is touched.3. "spec" is overloaded
docs/specs/*.mdare written specifications, not RSpec tests. On this project they were adding ~1,800 lines of prose toTEST_INSERTIONS. Prose is never test code, whatever directory it sits in, sois_testnow returns 0 for markdown/text and anything underdocs/.Also
TEST_FILES_CHANGEDused the narrow suffix-only pattern instead ofis_test, so it read 0 while 21 test files had changed in the window.Verification
TEST_INSERTIONSTEST_FILES_CHANGEDTEST_FILES_TOTALFOCUS_SCORE.build/)Nourish/)All 11 existing contract tests still pass. Adds a 12th with its own fixture repo covering all three cases, so the shared fixture's deterministic aggregates are untouched.
🤖 Generated with Claude Code