docs: harvest review lessons from PRs reviewed 2026-07-31 to 2026-08-06 - #10145
docs: harvest review lessons from PRs reviewed 2026-07-31 to 2026-08-06#10145saltas888 wants to merge 1 commit into
Conversation
There was a problem hiding this comment.
2 issues found across 10 files
Confidence score: 4/5
- In
dev/guidelines/backend/testing.md, the new statement about keeping case data inside@parametrizeconflicts with the existing module-level*_TEST_CASESpattern, which can cause inconsistent tests and reviewer churn as teams follow different styles — align the guidance to one canonical pattern and update the example to match. - In
dev/guidelines/backend/python.md, recommending a top-level import in tasks can interact badly withtasks/__init__.pyeager submodule imports, potentially reloading much of the backend on eachinvokeand slowing local tooling — narrow the recommendation to lazy/local imports (or document the task-package exception explicitly).
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="dev/guidelines/backend/testing.md">
<violation number="1" location="dev/guidelines/backend/testing.md:246">
P3: The new guidance "the case data lives in the `parametrize` decorator" contradicts the documented pattern just above it, which places case data in a module-level constant (`MY_FUNCTION_TEST_CASES`) and has the decorator reference that list. Taken literally it would push developers to inline case data into the decorator, violating guideline #4 (define test cases as module-level constants). Rephrase to say the decorator passes the full case object (so the pytest id identifies it), while the data stays in the module-level list.</violation>
</file>
<file name="dev/guidelines/backend/python.md">
<violation number="1" location="dev/guidelines/backend/python.md:84">
P3: This recommendation can undo the exception it extends: `tasks/__init__.py` eagerly imports every task submodule, so a top-level import of the new module in a task would reload the full backend on every `invoke` — the exact eager-load the exception avoids. The new module should keep heavy imports deferred (or the task should import it lazily), otherwise the 'deferred imports mostly disappear' advice is misleading when followed literally. Consider clarifying where the single import lives so the guidance stays consistent with the stated rationale.</violation>
</file>
Shadow auto-approve: would not auto-approve because issues were found.
Re-trigger cubic
| - Test cases share a common structure | ||
| - You want readable test IDs in pytest output | ||
| - The test logic is the same but data varies | ||
| Whichever you pick, the case data lives in the `parametrize` decorator. Don't parametrize over the keys of a module-level dict and look the values up inside the test — the reader has to hold two places in their head to see what a case actually asserts, and the pytest id no longer tells them. |
There was a problem hiding this comment.
P3: The new guidance "the case data lives in the parametrize decorator" contradicts the documented pattern just above it, which places case data in a module-level constant (MY_FUNCTION_TEST_CASES) and has the decorator reference that list. Taken literally it would push developers to inline case data into the decorator, violating guideline #4 (define test cases as module-level constants). Rephrase to say the decorator passes the full case object (so the pytest id identifies it), while the data stays in the module-level list.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At dev/guidelines/backend/testing.md, line 246:
<comment>The new guidance "the case data lives in the `parametrize` decorator" contradicts the documented pattern just above it, which places case data in a module-level constant (`MY_FUNCTION_TEST_CASES`) and has the decorator reference that list. Taken literally it would push developers to inline case data into the decorator, violating guideline #4 (define test cases as module-level constants). Rephrase to say the decorator passes the full case object (so the pytest id identifies it), while the data stays in the module-level list.</comment>
<file context>
@@ -240,12 +240,10 @@ SCHEMA_VALIDATION_TEST_CASES: list[SchemaValidationTestCase] = [
-- Test cases share a common structure
-- You want readable test IDs in pytest output
-- The test logic is the same but data varies
+Whichever you pick, the case data lives in the `parametrize` decorator. Don't parametrize over the keys of a module-level dict and look the values up inside the test — the reader has to hold two places in their head to see what a case actually asserts, and the pytest id no longer tells them.
For simpler cases with only 2-3 scenarios, standard `@pytest.mark.parametrize` with tuples may be sufficient:
</file context>
| Whichever you pick, the case data lives in the `parametrize` decorator. Don't parametrize over the keys of a module-level dict and look the values up inside the test — the reader has to hold two places in their head to see what a case actually asserts, and the pytest id no longer tells them. | |
| Whichever you pick, the parametrize decorator passes each case object whole — the pytest id is the case's `name` — rather than parametrizing over the keys of a module-level dict and looking the values up inside the test. The latter makes the reader hold two places in their head to see what a case actually asserts, and the pytest id no longer tells them. |
| imports (stdlib, `invoke`, sibling `.shared`/`.utils` modules) at the top; defer the rest into the | ||
| function that needs them. | ||
|
|
||
| The exception covers a thin task wrapper, not logic that happens to live in `tasks/`. A task body |
There was a problem hiding this comment.
P3: This recommendation can undo the exception it extends: tasks/__init__.py eagerly imports every task submodule, so a top-level import of the new module in a task would reload the full backend on every invoke — the exact eager-load the exception avoids. The new module should keep heavy imports deferred (or the task should import it lazily), otherwise the 'deferred imports mostly disappear' advice is misleading when followed literally. Consider clarifying where the single import lives so the guidance stays consistent with the stated rationale.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At dev/guidelines/backend/python.md, line 84:
<comment>This recommendation can undo the exception it extends: `tasks/__init__.py` eagerly imports every task submodule, so a top-level import of the new module in a task would reload the full backend on every `invoke` — the exact eager-load the exception avoids. The new module should keep heavy imports deferred (or the task should import it lazily), otherwise the 'deferred imports mostly disappear' advice is misleading when followed literally. Consider clarifying where the single import lives so the guidance stays consistent with the stated rationale.</comment>
<file context>
@@ -81,6 +81,10 @@ deliberate, documented exception: `pyproject.toml`'s `"tasks/**.py"` per-file-ig
imports (stdlib, `invoke`, sibling `.shared`/`.utils` modules) at the top; defer the rest into the
function that needs them.
+The exception covers a thin task wrapper, not logic that happens to live in `tasks/`. A task body
+needing a dozen deferred imports is telling you the logic belongs in a module of its own, which the
+task then imports once — put it there and the deferred imports mostly disappear with it.
</file context>
6d0cb74 to
facc5d3
Compare
Fifteen merged PRs audited across develop and release-1.11. Three of the lessons were already written down and a reviewer still had to raise them, so those are strengthened in place rather than duplicated. Strengthened: - backend-component-design: the whole object graph is built at the entry point in one pass, not part-way through a run, and invalid wiring raises while the graph is built instead of degrading to a runtime fallback. One reviewer made this point four times in two PRs against a rule that is auto-injected on every turn; it only forbade construction inside __init__. - backend/testing: names the parametrize shape that actually got written — dict keys in the decorator, values looked up inside the test. - frontend/page-architecture: the no-mirroring rule was stated for forms and useState only, so URL state copied into a Jotai atom slipped past it. Added: - query-pattern: pass limit/offset to the base Query constructor, since execute() reads self.limit/self.offset to choose how to run and pages the query in chunks when both are None; and read a collect() column with get_as_list_of_type rather than unpacking rows. - code-doc-style: spec vocabulary is as unreadable as a spec ID in a test name or docstring. - documentation: measurements go in as relative comparisons, not absolute figures that depend on the environment they were taken in. - frontend/writing-component-tests: a test has to fail when the behavior breaks — assert the value and not the static text around it, and check the component reads the flag the setup varies. - python: the tasks/*.py function-local import exception covers a thin wrapper, not logic hosted in tasks/. Pruned: - typescript.md taught useMemo/useCallback dependency arrays while knowledge/frontend/react.md says not to use them at all under the React Compiler. A reviewer had to ask which was right. - schema-definitions.md pointed at an issue for planned improvements that closed on 2026-07-20. - query-pattern.md was 520 lines against a 200-400 range and this run adds to it, so its five near-identical result-dataclass walkthroughs collapse into one example plus an accessor table. The file ends at 459. Net -40 lines. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
9bd7dbd to
4c4a19d
Compare
A harvesting-review run over the week's review threads, restricted to the PRs the in-flight harvests (#10145, #10098) did not read: open and recently-merged PRs across develop, stable, and release-1.11. New rules and knowledge: - mutations.md: retry_db_transaction placement — wrap only rollback-able transaction scopes, skip under a caller-supplied transaction (#10121) - query-pattern.md: READ queries with insert_limit=False need their own LIMIT; auto-paginated reads need a total-order ORDER BY (#10132) - creating-migrations.md: fix data bugs at the migration layer, not in runtime save paths (#10105); per-item error collection (#10132); batch by the memory-bounding unit (#10132) - database-schema.md: generic kinds exist only as labels — n.kind never matches a generic; type concrete-only inputs as list[NodeSchema] (#9805) - events.md: changelog models mask secrets only at construction — post-hoc assignment leaks them into events (#10105) - async-tasks.md: InfrahubBatch is concurrent, not ordered (#10113) - testing.md + testing-python.md rules: wiring tests parse source with ast/inspect instead of instrumenting production code (#10121); poll-don't-sleep for async effects (#10133); branch-attributable removal assertions (#10132); pure-function extraction before skipping the cheap test tier (#10137) - checklist.md: new Settings fields must reach both compose entry points — one generated and CI-checked, one hand-maintained (#10122) - backend/AGENTS.md: new REST endpoints are ask-first — prefer existing GraphQL for SDK needs (#8594); route creating-migrations.md in Guides - pre-ci.md: invoke lint tasks run ruff check --diff, which exits 0 on unfixable violations — added the plain CI-parity ruff check (#10146) - development/grafana/AGENTS.md (new): only defined datasource variables, sweep drill-down links, regenerate the standalone compose (#10153) - docs/AGENTS.md: Excalidraw exports use an opaque white background (#9953) Strengthened in place: - creating-changelog-entries skill: removed the 'most maintenance still gets a fragment' push that contradicted the no-user-facing-effect boundary; added lint/typing config cleanups as a named example (#10146) Paid for by compressing testing.md's nested-dataclass example and query-pattern.md's duplicated accessor/return-properties sections. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LYisRTn3sPygP55cxbVqzG
A harvesting-review run over the week's review threads, restricted to the PRs the in-flight harvests (#10145, #10098) did not read: open and recently-merged PRs across develop, stable, and release-1.11. New rules and knowledge: - mutations.md: retry_db_transaction placement — wrap only rollback-able transaction scopes, skip under a caller-supplied transaction (#10121) - query-pattern.md: READ queries with insert_limit=False need their own LIMIT; auto-paginated reads need a total-order ORDER BY (#10132) - creating-migrations.md: fix data bugs at the migration layer, not in runtime save paths (#10105); per-item error collection (#10132); batch by the memory-bounding unit (#10132) - database-schema.md: generic kinds exist only as labels — n.kind never matches a generic; type concrete-only inputs as list[NodeSchema] (#9805) - events.md: changelog models mask secrets only at construction — post-hoc assignment leaks them into events (#10105) - async-tasks.md: InfrahubBatch is concurrent, not ordered (#10113) - testing.md + testing-python.md rules: wiring tests parse source with ast/inspect instead of instrumenting production code (#10121); poll-don't-sleep for async effects (#10133); branch-attributable removal assertions (#10132); pure-function extraction before skipping the cheap test tier (#10137) - checklist.md: new Settings fields must reach both compose entry points — one generated and CI-checked, one hand-maintained (#10122) - backend/AGENTS.md: new REST endpoints are ask-first — prefer existing GraphQL for SDK needs (#8594); route creating-migrations.md in Guides - pre-ci.md: invoke lint tasks run ruff check --diff, which exits 0 on unfixable violations — added the plain CI-parity ruff check (#10146) - development/grafana/AGENTS.md (new): only defined datasource variables, sweep drill-down links, regenerate the standalone compose (#10153) - docs/AGENTS.md: Excalidraw exports use an opaque white background (#9953) Strengthened in place: - creating-changelog-entries skill: removed the 'most maintenance still gets a fragment' push that contradicted the no-user-facing-effect boundary; added lint/typing config cleanups as a named example (#10146) Paid for by compressing testing.md's nested-dataclass example and query-pattern.md's duplicated accessor/return-properties sections. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LYisRTn3sPygP55cxbVqzG (cherry picked from commit ca2190d)
Review follow-ups from #10030's threads, applied on top of the three cherry-picked harvest runs (#10098, #10145, #10157): - backend/AGENTS.md: keep the coding-standards list contiguous and let the exceptions.md pointer follow it, dropping the duplicated exception-handling bullet (cubic, polmichel) - speckit.opsmill.extract command + skill: error-handling conventions now route to dev/guidelines/backend/exceptions.md, and the target list is marked as routing examples to re-verify before writing (cubic, polmichel) - dev/guidelines/backend/python.md: replace the str-satisfies-Sequence survival section with a rule against one-or-many unions - prefer Sequence[T]/list[T] parameters and let callers wrap (polmichel) - AGENTS.md: compress the ruff --diff caveat and point at /pre-ci, which carries the CI-parity check (polmichel) - dev/guidelines/repository-organization.md: point the Git Workflow cross reference at its #pull-requests anchor (cubic) - harvesting-review skill 0.8.0: a split or move repoints every inbound reference including .agents/skills and .agents/commands routes; a lesson whose root cause is a fragile pattern becomes a rule steering away from the pattern, not a survival guide; lessons owned by an existing skill (creating-changelog-entries, pre-ci, pruning-residues) are routed into that skill instead of a parallel dev/ rule. Paid for by compressing the sweep's fix-every-hit prose and the report template comments. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Why
A
/harvesting-reviewrun over the PRs reviewed 2026-07-31 → 2026-08-06, the window since the previous run (#10098, which covered 2026-07-24 → 2026-07-31). Reviewers kept spending their time on the same things; each lesson lived in a thread and died there.Stacked on #10098 → #10030 →
develop. This run depends on both: it strengthens a rule #10098 added and follows the size discipline #10030 introduced.The headline is that three of the eight lessons were already documented and a reviewer still had to raise them. Those are strengthened where they already live — no duplicate rules.
Source PRs
Fifteen merged PRs read, resolved and unresolved threads both, across
developandrelease-1.11. Dependabot bumps and branch-sync merges skipped.ajtmccarty×3: a component building its collaborator mid-run; wiring errors raised at runtime instead of construction.cubic: a cascade source accepted with no output capturer; both sides of the contract erased toAnyajtmccarty: injectSchemaManagerinstead of reaching forregistry; a predicate living on the modelajtmccarty: parametrize over dict keys with an in-test lookup; task logic that should live in a module, which would also retire its deferred imports; a closed value set typed asstrgmazoyer×2: aQuerysubclass keeping its ownlimit/offset;get_as_list_of_typeexisting and being missedgmazoyer: spec vocabulary in a test name that means nothing without the specajtmccarty: an absolute performance figure in an ADR.cubic: an ADR claim the code contradictssaltas888: URL state mirrored into a Jotai atom.cubic: a toast assertion that passes with the wrong branch namesaltas888: is auseMemostill needed under the React Compiler?cubic: anisFetchingflag the provider never reads, making its test inertcubic×3:"main"→is_defaultbehavior changes with no test exercising a non-maindefaultajtmccarty×3: doc claims about merged-branch behavior the code contradicts#10079 and #10002 also merged in this window but were already harvested into #10030.
Strengthened — the rule existed and was missed anyway
.agents/rules/backend-component-design.md— one reviewer made the same point four times across two PRs, against a rule that is auto-injected into every agent turn. It only forbade constructing a sub-component inside a parent's__init__, so building one part-way through a run slipped past it, and it said nothing about when bad wiring must fail. Now: the graph is built at the entry point in one pass, and a mismatch raises there rather than degrading to a runtime fallback. Paid for by merging two overlapping paragraphs on constructor injection — the file grows by 4 lines for two new rules.dev/guidelines/backend/testing.md— said "use dataclasses for parametrized cases" but never named the hybrid that got written. Now names it. Paid for by compressing the four-bullet "when to use" list that said one thing.dev/guidelines/frontend/page-architecture.md— the no-mirroring rule was written for forms anduseState, so URL state copied into a Jotai atom wasn't covered. Generalized to any state with an owner in the table above it.Added
dev/knowledge/backend/query-pattern.md— passlimit/offsetto the baseQueryconstructor. Verified incore/query/__init__.py:execute()branches onself.limit or self.offset, so a subclass holding private copies leaves bothNoneand getsquery_with_size_limit()'s chunked re-execution on top of its own paging. Plus: read acollect()column withget_as_list_of_type..agents/rules/code-doc-style.md— spec vocabulary is as unreadable as a spec ID in a test name or docstring.dev/guidelines/documentation.md— measurements go in as relative comparisons; the absolute figure depends on the environment it was taken in.dev/guides/frontend/writing-component-tests.md— a test must fail when the behavior breaks: assert the value rather than the static text beside it, and check the component actually reads the flag the setup varies.dev/guidelines/backend/python.md— thetasks/*.pyfunction-local import exception (added in docs: harvest review lessons from PRs merged 2026-07-24 to 2026-07-31 #10098) covers a thin wrapper, not logic hosted intasks/. A task body needing a dozen deferred imports is saying the logic belongs elsewhere.Pruned
dev/guidelines/frontend/typescript.mdtaughtuseMemo/useCallbackdependency arrays whiledev/knowledge/frontend/react.mdsays not to use them at all under the React Compiler. A reviewer had to ask which was right — the contradiction is the finding.dev/knowledge/backend/schema-definitions.mdpointed at an issue for "planned improvements" that closed on 2026-07-20.dev/knowledge/backend/query-pattern.mdwas 520 lines against a 200–400 range, and this run adds to it, so its five near-identical result-dataclass walkthroughs collapse into one example plus an accessor table — which is where theget_as_list_of_typelesson naturally lands. The file ends at 459.Sweep otherwise clean: no defect-snapshot notes survive, and the remaining
#NNNNgrep hits aregit-workflow.md's issue-syntax examples anddocumentation.md's own rule text.Context budget
+56 / −92 lines — net −36. Every file inside its range except
query-pattern.md, which is 59 over after dropping 61 lines; splitting its## Internalsblock is the next candidate and wants its own change.writing-component-tests.mdis at 398 of 400, so the next lesson landing there compresses first.dev/knowledge/backend/query-pattern.mddev/guides/frontend/writing-component-tests.mddev/guidelines/backend/testing.mddev/guidelines/backend/python.md.agents/rules/backend-component-design.mdNot lessons
#10106's keyset-pagination pair —
ajtmccartyrejected it with reasoning and nothing landed. #10090's "SDK lacksIPAddress" and #10104's "missing author filter" — both bots retracted after being corrected. #10055's artifact-hard-coded failure fallback — real, but confined to that module. #10114's doc corrections and #10108's contradicted ADR claim — already covered by the verify-claims-against-the-diff rule rootAGENTS.mdgained last run. A "TIL", a screenshot request, and a jotai-naming aside — no rule.How to test
Docs only.
markdown-lint,validate-dev-guideline-links, andvalidate-documentation-stylecover these paths in CI;uv run invoke docs.lintglobsdocs/docs/**, so it doesn't reachdev/or.agents/. Relative links and code-fence balance checked locally.🤖 Generated with Claude Code