fix(citation): repair search-inside against the live IA contract

[tieguy]

Two defects in the Layer-2 book-grounding path (citation/search_inside.rs), both caused by the Internet Archive's full-text contract having moved away from what this module — and its test fixtures — encode. Found while porting this logic to another codebase and exercising it against the live endpoints.

1. Zero matches on a wide query produced a false not_supported

The endpoint returns zero matches for the entire query when any single term is absent from that item's OCR index. Because an empty result is what NoMatches reports, and the caller turns that into not_supported (PRD-0009 resolved Q4), one unindexed word could turn a well-supported claim into a verdict against the citation.

Reproducible today against onoriginofspeci00darw — the 1871 D. Appleton printing of On the Origin of Species, which uses the American spelling "favored":

OCAID=onoriginofspeci00darw
read SRV DIR <<< $(curl -s "https://archive.org/metadata/$OCAID" \
  | python3 -c "import sys,json;d=json.load(sys.stdin);print(d['server'],d['dir'])")
hits() { curl -s "https://$SRV/fulltext/inside.php?item_id=$OCAID&doc=$OCAID&path=$DIR&q=$1" \
  | python3 -c "import sys,json;print(len(json.load(sys.stdin).get('matches',[])))"; }

hits "natural+selection+preservation+slight"              # 493
hits "natural+selection+preservation+slight+favourable"   #   0
hits "favourable"                                         #   0
hits "preservation"                                       #  25

Five terms appearing on hundreds of pages are zeroed by one sixth term the index does not hold. Spelling variants, OCR damage (this scan contains sufBciently, indi\4duals) and hyphenation breaks all collapse the same way.

Fix. search_query_ladder yields progressively narrower queries — the capped term list, then the three most distinctive terms, then the single most distinctive — and run_search_ladder stops at the first rung returning matches. NoMatches is reached only when every rung is empty, so not_supported now means the scan genuinely lacks the claim's rarest word. At most three requests, and only when earlier ones come back empty.

The prior comment on search_query held that term selection "only affects recall, never grounding". That was true while snippets were merely evidence; it stopped being true once an empty result became a verdict.

2. Highlight markers leaked into supposedly verbatim OCR

The live endpoint emits <IA_FTS_MATCH>…</IA_FTS_MATCH>; only the historical {{{…}}} form was stripped. A live response:

{"text": "\"The only distinct moaning of the word '<IA_FTS_MATCH>natural</IA_FTS_MATCH>' is..."}

SearchInsideMatch::text is documented as "exactly what the OCR holds", so this put markup into the bytes we hand the verifier. Two consequences: the tags reach the prompt, and page_of_passage locates quotes via locate_quote/locate_quote_fuzzy against text containing them — a quote spanning a match boundary can fail to locate and get attributed to the wrong page.

Fix. Both marker forms are stripped.

Why these survived

The fixtures encoded the dead shapes: SEARCH_HITS used {{{…}}}, so the marker assertion passed against a form the API no longer emits. The fixtures now carry the live form, with the historical form asserted separately, plus a check that no marker survives into text documented as verbatim.

Worth noting the 2026-07-12 contract smoke did catch the itemURL → /borrow drift; it just didn't cover marker syntax or query semantics.

Also in this commit

The module header claimed snippet search "typically works even for lending-restricted scans, so grounding needs no borrow". The second half holds; the first does not — the endpoint returns 403 for print-disabled/lendable items, so in practice only full access scans can be grounded.

Spot-checked on 2026-07-23 by walking the book citations of seven en.wikipedia articles (Charles Darwin, Jane Austen, Isaac Newton, Battle of Waterloo, On the Origin of Species, Herodotus, Charles Dickens) through catalog resolution and scan availability: of 104 citations resolving to a record, 72 had no archive.org scan at all, 10 had only a different-edition scan, and 11 had an exact groundable scan — every one of those 11 lendable, all 403. This is a convenience sample, not a survey of the encyclopedia, and the header states it as a dated measurement.

This is a reach limit rather than a correctness problem: a 403 is Unreachable and degrades to SourceUnavailable, never to not_supported.

Testing

• cargo test -p sp42-citation — 343 passing.
• cargo test --workspace --exclude sp42-desktop — 1101 passing, 0 failures.
• cargo fmt --all -- --check and pedantic clippy on the crate — clean.
• scripts/ci-all.sh could not be run to completion locally: glib-sys fails its build script for want of system GTK/glib headers on this machine, which only affects sp42-desktop. CI should cover it.

run_search_ladder was extracted from prepare_book_grounding because inlining the loop tripped clippy::too_many_lines; an extraction seemed better than an #[allow], but it is easy to reshape if you'd prefer it otherwise.

One existing test needed its stub queue extended — page.rs::indexed_scan_with_no_snippets_is_not_supported_never_unavailable queued a single search response and the ladder issues three for that claim. Its assertion is unchanged.

Not addressed

extract_ocaid rejects the /stream/ path form that live full-access items actually carry (http://www.archive.org/stream/onoriginofspeci00darw). The fromRecord fallback in openlibrary.rs recovers the ocaid in the case observed, so it only bites when a /stream/ item lacks fromRecord and multiple records make the sole-record fallback ambiguous. Left out to keep this focused; happy to add it here or separately.

Draft

Draft because fix 1 changes verdicts the module has been emitting. CONTRIBUTING asks for a PRD on operator-facing behavior changes — my reading is that this restores PRD-0009 Q4's stated intent rather than altering it, but that is a maintainer call and I would rather ask than assume.

🤖 Generated with Claude Code

https://claude.ai/code/session_01CQDLheiqLry2xc8gYSAEG8