Book-citation grounding + Open Library enrichment mechanism (PRD-0009, ADR-0018/0019)#152
Closed
tieguy wants to merge 10 commits into
Closed
Book-citation grounding + Open Library enrichment mechanism (PRD-0009, ADR-0018/0019)#152tieguy wants to merge 10 commits into
tieguy wants to merge 10 commits into
Conversation
…ce 1)
Draft ADR-0018, the Open Library / Internet Archive read contract spawned
by PRD-0009 (Q6a): identifier-gated resolution, a strictly side-effect-free
Books API catalog lookup (never the import-on-miss /isbn/{isbn}.json),
Read API exact-vs-similar scan availability, and the search-inside
grounding-source rules including the scoped ShortBody bypass.
Implement the first read-only slice:
- sp42-platform: BookIdentifier (ISBN checksum-validated, OCLC, LCCN
LC-normalized, OLID canonicalized with author ids rejected) and
BookSource; BlockRef carries book_sources alongside cited sources.
- sp42-parsoid: cite-template data-mw extraction now also reads
isbn/oclc/lccn/ol params (and the cited page), independent of the
url= gate, so a URL-less {{cite book}} finally yields something.
- sp42-citation extract: a url-less ref with a validated identifier is
skipped as BookSource (with its identifiers) instead of blending into
NonUrlSource; the page report prints those identifiers.
- sp42-citation openlibrary: pure build/parse pairs for the Books API
catalog lookup and the Read API scan availability, partitioned so only
exact-match scans can ever feed grounding.
Wiring resolution into the verify path and report is the next slice.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012u5MVp3vAN44KhsVg3vroY
The page orchestrator now resolves every skipped book ref through the
side-effect-free Open Library lane (ADR-0018) and the report shows what
the catalog said, so a book citation prints a resolved record instead of
a bare identifier.
- sp42-citation openlibrary: resolve_book_source executor — tries a
ref's identifiers in template order against the Books API, then checks
Read API scan availability on the first hit. A clean miss is NotFound
(never a create), a transport failure is LookupFailed (existence
unknown), and an availability failure degrades to scan-unknown without
losing the catalog hit. A recording-client test asserts the whole lane
addresses only /api/books and /api/volumes/brief, never the
import-on-miss /isbn/{isbn}.json path.
- verify_page: a bounded (<=3, REST politeness) resolution pass over the
skipped refs' book sources; failures stay inside the books lane and
never fail the page. Report gains book_resolutions plus
books_resolved/books_not_found/book_lookups_failed stats, all
serde-default so older reports still deserialize.
- Renderer: a Books section (resolved record with title/authors/record
URL and an exact/similar_only/none/unknown scan label; distinct
not_found vs lookup_failed lines) and a books lead tally.
- SkippedRef now carries its BookSource structures (identifiers + cited
page) instead of a flattened identifier list, so resolution keeps
per-template identifier order and the cited page for Layer 2.
Search-inside grounding (Layer 2) is the next slice.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012u5MVp3vAN44KhsVg3vroY
Layer 2 of the book lane (ADR-0018 Decision 4): a resolved book citation with an exact-edition scan now gets a real verdict instead of a skip. - sp42-citation search_inside: the Internet Archive grounding source — item-metadata read for the designated search server (texts-only gate), BookReader full-text search build/parse (verbatim OCR snippets with highlight markers stripped, page numbers kept), conservative claim→query term selection, page-anchored deep links, and prepare_book_grounding mapping every degraded outcome onto the ADR-0007 split: no usable body (not a text item / no index) -> SourceUnavailable; an indexed scan searched with zero snippets -> not_supported, never SourceUnavailable. Cited page first, whole-book fallback, per PRD-0009 Q5. - verify: FetchedSource gains a book_snippet provenance flag whose only effect is bypassing the generic ShortBody floor at the usability gate — arbitrary short web pages still short-circuit, an empty snippet stays unusable (covered by a three-way test). CitationFinding gains optional BookScanProvenance (ocaid, scanned vs cited page, refined note) and two no-model book outcomes for the unavailable/searched-empty cases. - extract: a url-less ref with validated identifiers is now a BookUseSite (same sentence attach as URL refs, shared ordinal sequence) instead of an unconditional skip; only identifier-less refs skip at extract time. - verify_page: the book pass resolves each site against Open Library, grounds resolved books through the snippet body and the model panel (resolved record as context-only metadata sidecar), degrades similar- only/no-scan books to an honest SourceUnavailable pointing at the OL record, and re-skips catalog misses with the refined BookSource reason. The scanned page the passage actually located on is recorded so pagination mismatches surface instead of causing false negatives. - Renderer: finding lines show scan/scanned_page/cited_page/note; the Books section and stats are unchanged from slice 2. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012u5MVp3vAN44KhsVg3vroY
Answers the open question blocking the apply-contract ADR: what write mechanism an ordinary operator account can actually use in production. Key findings (primary sources: openlibrary-client, infogami's API plugin, the addbook.py form handler, the Writing Bots policy, upstream issues): - The PRD's "REST save PUT is internal/localhost-only" premise was wrong in its reason, right in its conclusion: the PUT and /api/save_many DO run against production (the official client and the whole bot ecosystem use them), but infogami's can_write() gates every API write on membership in /usergroup/api or /usergroup/admin — an ordinary account gets 403 even for a single-record save. - Ordinary accounts write through the website form path (POST /books/OL...M/edit, SaveBookHelper): logged-in + per-record permission only, edition./work. field prefixes, _comment preserved, no CSRF token today, a v revision parameter but no server-side refuse-on-conflict (drift protection must be client-side). - The API usergroup is obtainable by a sanctioned human process (ask the admins; bot naming conventions for bulk accounts), and membership is publicly readable at /usergroup/api.json — a write-free capability probe. - Auth for either lane: POST /account/login with credentials or per-operator IA S3 keys -> session cookie. No published rate limits; the policy is an open upstream discussion. Implication recorded for the ADR: a two-lane apply contract — the authenticated form POST as the default lane every operator has, and the REST PUT as a privileged lane for API-usergroup members — both per-operator, both proposal-only until the ADR and a one-time live form spike land. PRD-0009's three localhost-only mentions corrected to the verified usergroup-gating mechanism. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012u5MVp3vAN44KhsVg3vroY
The apply-contract ADR the enrichment lane was waiting on, built on the 2026-07-08 apply-path research: - Two apply lanes selected by derived capability, never configuration: the authenticated edit-form POST every ordinary operator account has, and the REST PUT for operators in the API usergroup (membership read from the public /usergroup/api.json — a write-free probe). - Per-operator session via IA S3-key login, held under the ADR-0002 local-session posture; no shared identity. - ADR-0010's propose/confirm/refuse-on-drift discipline re-implemented client-side around the record revision, since upstream refuses nothing on conflict; one field per apply; full audit of every attempt. - The form lane is a versioned adapter that GETs the form first, replays hidden fields, and fails closed on any contract drift (proposal-only rather than a best-guess POST to a public catalog). - The write lane ships disabled behind an explicit enablement gate: a one-time live form spike, upstream courtesy contact, and the operator capability report. Cross-references updated: PRD-0009 spawned-ADRs header, the research note's status, and the STATUS.md book-lane bullet. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012u5MVp3vAN44KhsVg3vroY
Review outcome: the usergroup-list derivation guarded against a failure that is already benign (a pre-mutation 403) while adding a second unofficial contract that mirrors server permission internals. Replaced with the server's own answer: attempt the REST lane, fall back to the form lane on 403 within the same apply, cache the discovered lane per operator session. Members never pay an extra request; non-members pay one refused request per session; the 403 handler is the permanent correctness mechanism and the cache only an optimization, so no state can be wrong. The fallback re-runs the drift re-read before the form submit. Also strengthened the fail-closed form adapter with a post-apply read-back (verify the confirmed field holds the proposed value; surface anything else loudly), updated the capability-report wording for discovery-on-first-apply, and recorded both rejected selection mechanisms (membership-list derivation, operator config knob) in Alternatives. Research note's implication section updated to match. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012u5MVp3vAN44KhsVg3vroY
…bled) The enrichment lane exists as mechanism + fixture tests only: no server route or CLI surface performs a write, and ENRICHMENT_WRITE_LANE_ENABLED is the constant (false, with a tripwire test) any future surface must check per the ADR's enablement gate. - sp42-citation enrich: pure proposal computation. Deterministic ISBN-completion candidates (ISBN-13 from ISBN-10 and back, checksum arithmetic on values the record or citation already carries — never a verbatim relay of wiki content); raw-record parsing that pins the revision; EnrichmentProposal with the replacement value, provenance, comment, and the content hash a confirmation binds to. A gap that is closed on the raw record yields no proposal. - sp42-citation openlibrary_apply: the two-lane apply. S3-key login -> session cookie; REST PUT carrying the full updated record + _comment; the form-lane adapter that GETs the edit form, parses it fail-closed (unknown widgets, ambiguous selects, a missing target or _comment slot, or an occupied gap all refuse as contract drift), replays every field with one changed, and checks the redirect success shape; the executor with drift re-read, 403 fallback that restarts the drift window and caches the discovered lane per session, and a post-apply read-back that verifies the exact proposed value landed before reporting success. - Books report: resolved entries now list the read-only enrichment candidates (propose=isbn_13:...), PRD-0009's proposal-listing surface. Covered by tests: lane discovery and caching (no PUT after a cached form lane), refusal-before-write on a moved revision, contract-drift refusal with nothing posted, read-back mismatch surfacing as failure, and the complete-record-yields-no-proposal rule. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012u5MVp3vAN44KhsVg3vroY
Manual, network-touching, never CI (ADR-0009 keeps tests on fixtures).
Walks the exact read chain verify-page uses — Books API catalog lookup,
Read API scan availability, archive.org item metadata, BookReader
search-inside — with read-only GETs and asserts precisely the response
fields the PRD-0009 parsers read. A pass means production still matches
the fixtures; a failure names the drifted contract and keeps the real
payload for fixture refresh. By construction it never touches the
import-on-miss /isbn/{isbn}.json path or any write endpoint.
Header notes that Open Library 403s datacenter IPs (observed from this
environment), so the smoke runs from a developer connection — the same
caveat applies to any future cloud-hosted SP42 deployment.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012u5MVp3vAN44KhsVg3vroY
Review outcome: no local Open Library instance will be set up. The enablement spike's form-contract validation moves to a zero-write production capture — the edit-form GET is read-only, so the real HTML can be fetched under a session cookie and committed as the adapter's fixture, putting parse/fill against real markup in the normal test suite. POST semantics (success shape, _comment in history, stale-v behavior) stay in the one-write phase of the spike as before. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012u5MVp3vAN44KhsVg3vroY
Brings in the Wikidata entity read path (#119), the citation repair/insertion browser surface (#109), the architecture map + CI freshness gate (#118/#139), coordination fail-closed (#146), the as-built ADRs 0020-0023, and the docs attic move. Conflicts were the book-citation lane (this branch) and the bare-URL repair lane (main) touching the same seams; resolved by union: - BlockRef carries both book_sources (branch) and is_bare_url_ref (main); parsoid threads RefSources and the ref_bare map side by side. - CitationFinding carries both book_scan and is_bare_url_ref. - verify_page keeps the books pass; main's re-verify surface (select_reverify_use_site / apply_reverified_finding) now tallies through the five-argument tally_stats with book resolutions intact. - STATUS.md keeps both progress bullets. Regenerated docs/platform/architecture.md for the new citation modules (openlibrary, search_inside, enrich, openlibrary_apply). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012u5MVp3vAN44KhsVg3vroY
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 47d7ef78d0
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
Comment on lines
+58
to
+59
| if segments.next() != Some("details") { | ||
| return None; |
There was a problem hiding this comment.
When the Open Library Read API returns an exact scan whose itemURL uses /stream/<ocaid> (the official Read API example uses this shape: https://openlibrary.org/dev/docs/api/read), this check rejects it because only /details/ URLs are accepted. In that case verify_book_use_site sees no groundable_ocaid and reports SourceUnavailable instead of grounding a readable exact scan, so exact book citations can be skipped despite having usable IA OCR.
Useful? React with 👍 / 👎.
Collaborator
Author
|
This branch ( Two things make merging this branch instead actively wrong:
Closing in favor of #147 (review in progress) and #148 (stacked follow-up). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements PRD-0009 end to end, governed by two new ADRs written first:
/isbn/{isbn}import-on-miss path), exact-vs-similar scan availability via the Read API, and search-inside grounding with a provenance-scoped short-body bypass.PUT, usergroup-gated server-side, with pre-mutation 403 fallback to a fail-closed edit-form adapter), per-session lane cache, client-side refuse-on-drift on record revision, post-apply read-back — and the enablement gate that keeps the whole write lane disabled and unwired in this PR.What lands, layer by layer:
BookIdentifier(ISBN-10/13 with checksums, OCLC, LC-normalized LCCN, edition-only OLID) parsed structurally from cite-templatedata-mw;BlockRefgainsbook_sources.verify-pageresolves each book ref through side-effect-free Open Library lookups: Books API catalog record, then Read API exact-vs-similar scan availability. Clean miss →NotFound; transport/shape failure →LookupFailed; a lookup failure never fails the page.not_supported, no usable body →SourceUnavailable. Never verifies against a merely similar edition.ENRICHMENT_WRITE_LANE_ENABLED = falsewith a tripwire test; nothing calls the apply path.propose=lines) plusbook_scanprovenance on findings.scripts/openlibrary-contract-smoke.shexercises the live read contract (Books API → Read API → metadata → search-inside) with field assertions matching the parsers; it must run from an operator machine (Open Library blocks datacenter egress).Branch is merged with current main; the book lane and the bare-URL repair lane (#109) coexist (
BlockRefcarries bothbook_sourcesandis_bare_url_ref;CitationFindingcarries bothbook_scanandis_bare_url_ref).Type
Validation Required
Manual (operator machine, not CI):
scripts/openlibrary-contract-smoke.shagainst the live services — the sandbox cannot reach openlibrary.org/archive.org.Risk
Auth surface: the ADR-0019 apply machinery includes Open Library S3-key login and session-cookie handling — but it is dead code behind
ENRICHMENT_WRITE_LANE_ENABLED = false(tripwire-tested), reachable only from its own unit tests. No Wikimedia action behavior changes. Both governing ADRs (0018, 0019) are in-tree in this PR.Notes For Reviewers
citation/openlibrary.rs(resolve),citation/search_inside.rs(grounding), and their wiring incitation/page.rs. The ADR-0018 invariants worth checking: no request ever hits an import-on-miss endpoint, and a similar-edition scan is never used for verification.citation/openlibrary_apply.rsis mechanism-ahead-of-enablement by design (ADR-0019 Decision 6): review it as a contract implementation, not a live feature. The form adapter's fail-closed rules (unknown input kinds, non-empty target slot, missing_commentslot → refuse) are the load-bearing part.base_revisionso drift refuses client-side before any lane is chosen.BlockRef/CitationFindingwithout interacting.🤖 Generated with Claude Code
https://claude.ai/code/session_012u5MVp3vAN44KhsVg3vroY
Generated by Claude Code