Skip to content

test_recall real-query floor leg couples to self-index corpus stats — large refactors false-red it (2 sightings) #420

Closed

Description

@robbe1912

Defect (test-stability, systemic — 2 sightings today)

test_recall's "real query: top rows clear the floor" leg (tests/test_recall.py:500-502) asserts that recall.search("graph signal wiring edges")[:3] keeps at least one un-weak row against the suite's SELF-INDEX of the checkout. The weak mark is floor-gated (cos < 0.48 AND bm25 < 6.0, issue #297), so the leg silently couples to the repository's own BM25F corpus statistics: any large textual refactor shifts IDF/length norms until all top-3 rows fall under the floor — with ZERO behavior change in the engine.

Sightings (2026-09-20/21): PR #416's merge-commit CI red (decode: corpus-delta into pre-#407 main; rebase + fresh CI green); PR #419's deterministic red (+146 common.py / −86 c / −97 cpp shifts stats; passes on pristine main, fails with diff, fails with the tooth test removed — pure corpus effect). Both required lane-hours to decode.

Fix shape

De-couple the leg from the mutable self-index, one of:

Teeth

Mutation: corpus-only textual churn (comment moves) must NOT flip the leg; a genuine floor regression (floor raised above real-query evidence) must still fail it.

Context

Filed by GK during the #419 gate (ruling: query recalibrated in-PR as explicit pinned delta; floors are the pinned contract — bench/golden + the constants check). This issue owns the systemic de-coupling so the NEXT big refactor doesn't pay the decode again.

Activity

  1. robbe1912 commented on Sep 20, 2026

    @robbe1912
    OwnerAuthor

    Scope addendum (from the #419 gate): the 'garbage fn query' leg couples to STORE POPULATION history, not just corpus stats — graph.find_functions has no score cutoff ([] iff the fn collection is unpopulated), so its bool() precondition pinned rig/env state (split observed: one rig [] on pristine+diff, another rows-across-diff, CI rows). De-coupling fix for that leg rides this issue: assert the population precondition explicitly (populate-or-skip-loud) rather than via bool(fng).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions