docs(testing): tests and benches must be drawn from proven-tests-and-benches first - #564
Conversation
…benches first The taxonomy defines 16 test categories, 14 aspect dimensions and a benchmark taxonomy, but said nothing about WHERE an implementation should come from -- and referenced hyperpolymath/proven-tests-and-benches zero times, despite that repo holding 38 .idr files and 4 .ipkg of Idris2 tests and benchmarks. Adds an explicit provenance rule before Part I, with an ordered preference: 1. proven-tests-and-benches -- the Idris2 suite. Adapt it. 2. The Appendix C reference implementation. 3. A sibling repo already assessed at CRG C or better. 4. Write a new one -- and contribute it BACK, so the next repo inherits it. Why the Idris2 suite ranks first: a test that type-checks has discharged its obligation in the type system, so it cannot silently degrade into a test that passes without asserting anything. And copying a shared test is how a shared expectation STAYS shared -- writing a fresh one per repo is how sixteen subtly different definitions of 'the same' test appear, which is the one-template-copied-N-times failure this estate keeps paying for. Carries an explicit caution: a test written to make a gate go green rather than to establish a fact is WORSE than no test, because it is read as evidence. Where no honest test exists yet, mark the category N/A with justification as Scope already requires. That is a truthful state; a passing-but-vacuous test is not. Verified: asciidoctor renders clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Note Automatic reviews are paused because your trial's included automatic processing has been used for this period. Upgrade now, or comment "Gitar review" to run a review anytime. Code Review ✅ ApprovedAdds a mandatory provenance rule to the testing documentation requiring tests and benches to be drawn from proven-tests-and-benches first. Clean AsciiDoc update with no issues found.
OptionsDisplay: compact → Showing less information. Comment with these commands to change the behavior for this request:
Was this helpful? React with 👍 / 👎 | Gitar |
|
![gitar-bot[bot]](https://avatars.githubusercontent.com/in/827041?s=60&v=4){kind=small}
1 participant
The taxonomy defines 16 test categories, 14 aspect dimensions and a benchmark taxonomy — but said nothing about where an implementation should come from, and referenced
proven-tests-and-bencheszero times, despite that repo holding 38.idrfiles and 4.ipkgof Idris2 tests and benchmarks.Adds an explicit provenance rule before Part I:
proven-tests-and-benches— the Idris2 suite. Adapt it.Why Idris2 ranks first: a test that type-checks has discharged its obligation in the type system, so it cannot silently degrade into a test that passes without asserting anything. And copying a shared test is how a shared expectation stays shared — writing a fresh one per repo is how sixteen subtly different definitions of "the same" test appear.
Carries an explicit caution: a test written to make a gate go green, rather than to establish a fact, is worse than no test — it is read as evidence. Where no honest test exists, mark the category N/A with justification, as Scope already requires.
Verified: asciidoctor renders clean.
🤖 Generated with Claude Code