How do you assess a crawler's recall without building a better crawler yourself?

[sohamsinha2018]

I'm running a challenge where people build agents to research companies. For each company, the agent has to find the people, locations, jobs, financial results and other public information, and give a source for each fact.

The part I'm trying to get right is recall: how do I know how much each agent missed?
If I compare them with my own crawler, I'm limited by whatever my crawler found. If mine already finds everything, I've solved the problem I'm asking other people to solve.

The approach I'm considering is to combine the facts found by every submission and our own crawlers, verify them, remove duplicates, and then score each agent against that shared set. A fact found by one agent would count even if all the others missed it.

The obvious hole is that every agent might miss the same fact. I could manually research a random sample of companies to estimate that gap, but I'm not sure whether that is enough.

Has anyone here evaluated something similar? How would you test this without first building a better crawler yourself?

[liubrain39]

Your shared verified set resembles pooling in information-retrieval evaluation, where judgments are collected for results contributed by multiple systems and sometimes expert searches. Applying that idea to company facts, I would separate coverage of the pool from an estimate of everything that exists.

I would use two reports:

1. Coverage of a versioned, verified pool. Freeze the company list, fact types, permitted source types and date cutoff. Define a fact's identity before deduplicating: a current role and an old role, or financial results for different periods, need not be the same fact. Verify submitted claims against their cited sources, and report their precision separately from how many reference facts they cover. Keep unsupported or inaccessible claims unconfirmed. If a later submission contributes verified new facts, version the pool and rescore every submission against the same version.
2. An independent audit of sampled companies. Select companies from the original company list, rather than only from companies with rich pooled results. Research them with a fixed source checklist before showing auditors the agents' outputs, then compare the inventories. Report the audited companies, fact types, audit effort and remaining uncertainties. This can expose omissions shared by all agents; the human inventory can still miss facts, so I would label the result coverage of the audited reference set rather than absolute recall of the open web.

For a reproducible score, use a bounded snapshot of selected source pages and a fixed, manually verified fact inventory under the agreed schema. The denominator is then known for that reference inventory. Report recall against the inventory, rather than asserting that it captures every possible fact on those pages; discoveries outside the inventory can remain a separately verified result.

Disclosure: I maintain BrowseSprint's manual browser-task diagnostic guide. Its related completion-evidence lesson is to keep verified results, failures and unresolved outcomes separate before comparing runs. This reply is a proposed protocol prepared with AI assistance; I have not run your company-research challenge or a Crawl4AI evaluation, and I am not claiming measured recall or speed improvements.