Crawler split issue 30 - #33
Draft
nikitakarpei wants to merge 15 commits into
Draft
Conversation
Phase A of the crawler split (#30): contracts only, no behavior change for existing consumers. Move fetching, extraction, the content format graph, and page derivations out of yacycrawler into libraries/pagescrape, with their tests and ADRs 0005, 0008, 0009 and 0010, renumbered 0001-0004 in the new module. The crawler consumes the library; crawl/fetchedpage and the pagevisit fetcher port become pagescrape/pagefetch. Add ReachedPage, CRAWL_REACHED_PAGES and crawl.reachedpage to the crawl contract. The crawler creates the stream and publishes every page it fetched and did not dispose, alongside today's representation streams. Nothing downstream reads it yet. ReachedPage carries only CanonicalURL: the OrderID the plan named has no consumer and the subject is unrouted. Two units come out of pagevisit, which had grown six collaborators: fetchtiming, a pagefetch.Fetcher decorator that times the fetch, shaped like redirectrecording; and reachedpagepublication, which canonicalizes and publishes a reached page, shaped like disposal.Disposer. Unlike Disposer it returns a failed publish instead of warning, because the reached page is the crawler's primary output from here on. The crawler e2e suite gains a case asserting a reached page arrives on the new stream for a crawled fixture site. Its two stream-wait helpers, which used different mechanisms and different timeouts, collapse into one. Fix make arch-diagram: for_each_go moved to xargs -I{} but the recipe still named $m, so every module overwrote arch-diagrams/.svg. Arch-lint rules dropped or widened, in services/yacycrawler: - new components fetchtiming, reachedpagepublication, reachedpagesjetstream - new vendors pagescrapepagefetch, pagescrapecontentextraction and pagescrapecontentformatgraph, replacing readability, htmltomarkdown and xnet, which leave with the code - pagevisit mayDependOn reachedpagepublication, canUse pagescrapepagefetch - reachedpagepublication canUse canonicalurl; pagevisit no longer does - reachedpagesjetstream canUse nats - fetchtiming, recrawlrulesalwaysdue, recrawlrulesdueaftergrace and redirectrecording canUse pagescrapepagefetch - pageabsorption canUse pagescrapecontentextraction, pagescrapepagefetch - pagepublication, pagerepresentations{rwi,text,markdown} and representationpublishersjetstream canUse pagescrapecontentformatgraph - progressobserversprometheus mayDependOn fetchtiming and reachedpagepublication - commonComponents drops contentformatgraph; redirectrecording drops mayDependOn pagevisit libraries/pagescrape gets its own ruleset, carrying the component edges and the readability, htmltomarkdown and xnet vendors from the crawler. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XxeJbGijz5jUp9fRaLJpqQ
corpusmarkdown consumes reached pages and derives markdown for itself through pagescrape, so the crawler no longer publishes a markdown representation. Plan phase B step 3. Container extraction goes with it, in the same commit because the two changes share pagescrape's scraper and the markdown intake loop; no intermediate state of the two apart is green. A reached page names one URL, but a container held many documents under synthetic member URLs that no client can fetch. Keeping it would make every consumer re-expand the same archive to rebuild the same unaddressable corpus keys. Only HTML has a registered extractor, so an archive had to hold HTML to yield anything at all. One fetch now yields at most one document. contentextraction keeps its media-type router and returns that document; the ContainerExpander port, ContainerMember, the archive expander, and the nesting, document, and member bounds are gone, as are the nesting-too-deep and document-budget-exhausted disposal reasons that nothing can raise any more. ExtractedContent and ExtractedDocument merge, pagescrape.Scrape returns one page and whether it scraped one, and pageabsorption loses its per-document fan-in. pagescrape ADR 5 records the removal and supersedes the container half of ADR 3; the routing half stands. Arch-lint allows more in two places: pagescrape declares a root component that may depend on every extraction and derivation component and use canonicalurl, and corpusmarkdown adds the pagescrape, pagescrape/contentformatgraph, and pagescrape/pagefetchers/http vendors with markdownintake allowed to use pagescrape. It allows less by dropping the containerexpandersarchive and pagerepresentationsmarkdown components. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XxeJbGijz5jUp9fRaLJpqQ
corpustext consumes the crawler's reached pages, fetches each one through the egress proxy, and derives readable text with pagescrape. The crawler's text representation, PageTextRepresentation, and PageReference are deleted. Both search indexes now take a searchdocument.Document, so they no longer depend on the crawl contract; the intake maps the scraped page into the document and stamps the time it fetched. corpusrecall and firecrawlshim e2e still configured corpusmarkdown for the retired crawled-page subject and never set the now-required proxy URL. Both are fixed here. Arch-lint widenings: corpustext gains the pagescrape, contentformatgraph, and pagefetchers/http vendors, the pageintake -> scrapedpagedocument edge, and pageintake's use of pagescrape and searchdocument. Narrowings: the crawledpagedocument component becomes scrapedpagedocument, the search-index components drop yacycrawlcontract, and yacycrawler drops pagerepresentationstext. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XxeJbGijz5jUp9fRaLJpqQ
yacynode consumes the crawler's reached pages, fetches each one through the egress proxy, and derives its reverse word index with pagescrape. The chunk reassembly in crawlresults is deleted: with one message per reached page there is nothing to reassemble, and the tokenizer and index builder move in from the crawler as pagerwi. The intake splits a page's postings by the same cap that configures posting admission, so batching moves off the wire and into the consumer. With RWI gone, the crawler publishes only the URL of every page it reached. Deleted: pagepublication, pagerepresentations, representationpublishers, the representation catalog and its YACYCRAWLER_PUBLISH_* / NATS_PAGE_* config, the publication metrics, and PageRepresentationKind, PageRWIRepresentation, PageRWIChunk, and CrawledPageStreamName/Subject from the contract. pageabsorption now extracts, decides disposal, and returns discovered links. The crawler's spec kept all-or-nothing publication requirements that the last representation leaving made false, so they are rewritten here rather than at the planned later step. ADR 0012 records its supersession in its Status section and is otherwise left historical. Arch-lint widenings: yacynode gains the pagescrape vendor, the poisonhalt and pullintake serviceruntime paths, the pagerwi and reachedpageintake components, and the reachedpageintake -> pagerwi edge. Narrowings: crawlresults is gone, crawlbroker drops its project dependency, and yacycrawler drops the pagepublication, pagerepresentationsrwi, and representationpublishersjetstream components and the yacymodel vendor. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XxeJbGijz5jUp9fRaLJpqQ
The node stored a reached page's postings in batches sized by the
admission cap, and derived URLMetadata.ByteSize from the text it
tokenized rather than from the document the origin served.
ScrapedPage now carries DocumentByteSize, taken from the fetched body,
so pagerwi reports the served document size while still building the
index from derived full text. corpustext and corpusmarkdown gain the
same fact.
reachedpageintake makes one urls.Receive and one postings.Receive per
page; postingBatchesOf, storeBatch, and PostingBatchCap are deleted.
That moves the too-large refusal. It lived in rwiadmission.Receive, the
receiver both the local intake and remote DHT peers pass through, so
leaving it there would refuse any page with more than 1000 distinct
words. It now sits in rwiingress, the endpoint an untrusted peer request
arrives at, which is the only caller it ever bounded. rwiadmission loses
Config.BatchCap, Receipt.TooLarge, and the branch; rwiingress.Mount
takes Config{PostingCap, Pause, Refusals} and answers ResultBusy with
the pause. The request_too_large metric label is unchanged, reported
through the same rwiadmission.RefusalObserver. node.go renames
postingAdmissionBatchCapacity to peerPostingTransferCapacity, its one
remaining reader.
Receipt.TooLarge had no reader left once the intake stopped batching, so
it is deleted rather than left as a dead field.
No arch-lint rule changed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XxeJbGijz5jUp9fRaLJpqQ
Structure comes from this branch: page representations, chunking, and the crawled page document stay deleted, and pagescrape stays a library. Canonical URL typing comes from main: a URL is a value type again, not a string. The type moves to libraries/canonicalurl. Main declared it in the crawl contract, but phase A moved link discovery into pagescrape, which three services share, so a library cannot depend on one service's contract. The crawler contract now consumes the library. pagerwi.Of no longer returns an error. A CanonicalURL always parses, so the failure it reported cannot happen; the tests that built an invalid URL go with it. Arch-lint allows more: canonicalurl becomes a common vendor in yacycrawler, yacynode, corpusmarkdown, corpusrecall, and visitcrawl, and a single edge in the two contract modules and in pagescrape/contentextraction. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XxeJbGijz5jUp9fRaLJpqQ
corpusmarkdown gains the read API that corpusrecall reaches into its object store for today. MarkdownCorpus/RecallPage answers the markdown held for one URL, or NOT_FOUND; it neither fetches the page nor orders a crawl. The contract lives in the module that already owns the markdown vocabulary, services/corpusmarkdown/contract (pagemarkdownstore), rather than in a module of its own: the proto names the same corpus the bucket and object names do. The server canonicalizes the requested URL at the edge, so MarkdownOf takes a canonicalurl.CanonicalURL and the response's canonical_url reports what the corpus was actually asked for instead of echoing the request. A URL that cannot be canonicalized — the empty one included — is InvalidArgument. CORPUSMARKDOWN_LISTEN_ADDR defaults to :8094; firecrawlshim already owns :8093. Arch-lint: the contract module gains the grpc and protobuf vendors and mayDependOn itself, matching corpusrecall/contract; in the service, markdownrecallreceiversgrpc uses the pagemarkdownstore vendor, which is wider than the generated-package-only vendor it replaces. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XxeJbGijz5jUp9fRaLJpqQ
The crawler answers what crawling last did with a URL: where it resolved to, and whether it is disposed. Callers read the two key-value buckets directly today, which ties every reader to the crawler's storage layout and to broker credentials. CrawlOutcomes/ReadPage replaces both reads, and the buckets become private to the crawler again. The contract lives in the module that already owns the crawl vocabulary, services/yacycrawler/contract, rather than in a module of its own. ReadPageRequest carries url, not the canonical_url the plan named, and the server canonicalizes at the edge as the markdown corpus does. A caller no longer needs the canonicalization library to ask a question, and canonical_url in the answer reports what was asked for instead of echoing it. A URL that cannot be canonicalized is InvalidArgument. resolved_url always carries a URL, the canonical one itself when no redirect was recorded, so no caller needs a branch for an empty field. disposal.Disposer passes the reason to DisposedPages.Record, which stores it as the entry value; the reason was discarded after the metric until now. The disposal mark is the bucket revision, so a caller that captures it before it places an order can tell a new disposal from an earlier one. Both bucket-owning units gained a read side, so both took the name of what they own: Recorder became DisposedPages and RedirectResolutions, and their tests now write and read through those surfaces instead of reaching into the bucket. YACYCRAWLER_LISTEN_ADDR defaults to :8095, the next address free of renderproxy, visitcrawl, corpusrecall, firecrawlshim, and corpusmarkdown. ADR 13 records the gRPC decision against core-NATS request-reply, and ADR 14 the traffic arithmetic behind one message per reached page. Arch-lint rules this widens: the contract module gains the grpc and protobuf vendors; the service gains a grpc vendor and the crawloutcomereceiversgrpc component, and both it and disposedpagesjetstream may now depend on disposal, whose reason and mark they carry. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XxeJbGijz5jUp9fRaLJpqQ
firecrawlshim now owns the order-and-wait loop: it reads what crawling last did with a URL over CrawlOutcomes/ReadPage, places a crawl order on NATS, then polls MarkdownCorpus/RecallPage for the resolved URL until the corpus answers, crawling disposes of the page, or the recall limit runs out. It no longer depends on corpusrecallapi. The scrape response loses title and language with the text representation, which no read API serves. Arch-lint gains the markdownrecall and crawlorderplacersjetstream components with the canonicalurl, corpusmarkdownapi, crawlerapi, jetstream, uuid, and yacycrawlcontract vendors; the corpusrecallapi vendor is dropped. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XxeJbGijz5jUp9fRaLJpqQ
firecrawlshim now owns the recall loop, and corpusmarkdown serves the markdown read API, so nothing is left for corpusrecall to do. The service, its contract module, its e2e suite, its image build workflow, and its proto and e2e entries in the Makefile go with it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XxeJbGijz5jUp9fRaLJpqQ
Absorption had one caller, always immediately after a successful fetch, so the two were stages of one procedure rather than two units. What is left of absorption since derivation moved out of the crawler is extract, read off links and refusal, name a disposal — private detail of a visit. pagevisit now owns IndexingRefusal and PageExtractor, and its visitor absorbs the fetched page itself. AbsorberSource, Absorber, and AbsorptionOutcome are gone; the extraction failure that Absorb declared but never returned goes with them. Arch-lint drops the pageabsorption component; pagevisit gains its pagescrapecontentextraction vendor. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XxeJbGijz5jUp9fRaLJpqQ
A reached page was the crawler's report of what it did. What travels on that stream is a request that any interested corpus scrape a URL, so the crawler's vocabulary must not stay stamped on it. libraries/scraperequestcontract now holds ScrapeRequestsStreamName, ScrapeRequestSubject, ScrapeRequest and its marshallers. yacycrawlcontract keeps only OrdersStreamName and the order marshallers. The vocabulary went the whole path: packages scraperequests, scraperequestpublication and scraperequestintake, the metric yacycrawler_scrape_requests_published_total, the env vars NATS_SCRAPE_REQUEST_SUBJECT and NATS_SCRAPE_REQUEST_DURABLE, the subject scrape.request, and the service docs. The stream is LimitsPolicy with DiscardOld now. It was a work queue, which permits only non-overlapping consumers and removes each message on first ack, so with three corpora attached NATS rejected the overlapping durables and a page reached one corpus rather than all three. A crawler e2e case reads one scrape request through two durables. DiscardNew went with the work queue: nothing frees space under LimitsPolicy, so the stream would fill and stall every publisher for good. Arch-lint widening: a new scraperequestcontract vendor in four modules, granted to markdownintake, pageintake, crawlbroker, nodeconfiguration, scraperequestintake, and the crawler's commonVendors. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
No service creates SCRAPE_REQUESTS any more. The crawler and the shim publish to it, three corpora read it, and none of them owns its retention. The crawler gains SCRAPE_REQUEST_NATS_URL beside CRAWL_NATS_URL, which it keeps for the crawl orders it does own; the corpora and the node rename their CRAWL_NATS_URL to the same name. The deployment example provisions the stream from a nats-box init service that every publisher and consumer waits for, and its README carries the retention window an operator now sizes. test/e2eharness/scraperequeststream stands in for that operator in every suite that drives a scrape request, and replaces the divergent copy the searxng plugin suite carried. test/scraperequestfanout is a new suite that publishes one scrape request to two corpora and asserts both absorb it: corpusmarkdown stores the markdown object, corpustext indexes the text. It lives beside the harness because no single service owns the claim. The Makefile's e2e module glob now covers test/*/go.mod, and a suite may name its own directory through E2E_SUITE_DIR_<name>. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The service placed a crawl order with MaxDepth 0 and one seed, then polled CrawlOutcomes/ReadPage and RecallPage until the markdown appeared. That order was never a crawl, and the waiting machine around it observed the result of a fetch the service could have made itself. Making it scrape for itself was tried first: derive the markdown in-process through pagescrape, answer from that, and publish a scrape request so the corpora still gain the page. The publish is a fire-and-forget side effect of serving an HTTP call, owned by no unit in the service. The Firecrawl API is worth having, so it comes back with a design of its own, and a unit that owns announcing a page. Nothing else in the repo depended on it. The module, its Dockerfile, its e2e suite, its docs, its build workflow, and its go.work, CI, and Makefile entries all go with it. It was the only external reader of CrawlOutcomes/ReadPage, so the crawler can now shed the outcome API. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Nothing outside the crawler read a crawl outcome once firecrawlshim went, and both KV buckets became write-only: the gRPC receiver was the sole reader of redirectresolvers and disposedpages. Deleted: internal/crawloutcomereceivers/grpc, contract/crawler/v1, contract/crawloutcomesclienttest, internal/redirectresolvers, internal/disposedpages, the two bucket contracts, the ensure and open calls for both buckets, the gRPC listener, and YACYCRAWLER_LISTEN_ADDR with its port. grpc and protobuf leave both modules, and make proto no longer generates crawler/v1. internal/crawl/redirectrecording goes whole: it was confirmed to have no caller inside crawl logic, so with its bucket gone it recorded for nobody. disposal.Disposer goes the same way. With the persistence removed it only forwarded a reason to the metric, so ordertraversal counts a disposal through its own progress observer and disposal holds only Reason. TraversalProgress gains PageDisposed; DisposalProgress goes. The spec loses the resolve-and-disposal-answering requirement. Its known limitation about the consumer outage window is already stated in the deployment example's README, which is where an operator sizes the retention it describes. ADR 0013 is deleted rather than superseded, at the user's request. Arch-lint drops four components (redirectrecording, redirectresolversjetstream, disposedpagesjetstream, crawloutcomereceiversgrpc) and the grpc vendor in the service, and the grpc and protobuf vendors in the contract. No rule widens. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.