Proposal: isolated academia-registry-v0 suite for evidence-grounded registry research

[reacher-z]

Proposal: isolated academia-registry-v0 suite for evidence-grounded registry research

This follows the maintainer guidance on closed PR #304: do not change the released V1 denominator or published statistics; if the tasks add a genuinely new capability axis, propose a new v3 or special academia suite.

Proposed scope

Create an independently versioned, read-only suite tentatively named academia-registry-v0. It would evaluate whether browser agents can retrieve, reconcile, and cite small factual records from authoritative public research and policy registries.

The current local inventory contains 60 distinct task definitions across six source families (10 each):

• Crossref bibliographic metadata
• PubMed publication records
• ClinicalTrials.gov registration fields
• GitHub public repository metadata
• World Bank indicator metadata and values
• SEC EDGAR filing metadata

This is intended as a separate evaluation axis—not an addition to V1 or V2.

Isolation guarantees

• New namespace/path and CLI selector.
• Separate task count, version, dataset card, results table, and changelog.
• No change to V1, V1-lite, or V2 files, denominators, scores, or paper analysis.
• No reuse of V1/V2 numeric IDs.
• Read-only tasks only; no credentials, purchases, submissions, or private data.

Evaluation contract to agree before a PR

These tasks ask for a factual final answer plus browser evidence, so the existing interception-only contract is not sufficient. The current main branch already contains judge_answer(...) and judge_context foundations, while docs/answer-mode-tasks.md notes that runner wiring remains incomplete.

I propose a 12-task design slice (two tasks per source family) using an opt-in answer-mode contract that:

1. captures the final answer and visited-source evidence;
2. normalizes task-specific fields deterministically;
3. rejects the wrong record, wrong field, unsupported answer, missing browser evidence, and stale-source-date cases;
4. keeps reference answers and rubrics hidden from the agent;
5. records a source snapshot date and refresh policy for mutable fields.

The remaining 48 tasks would be proposed only after this contract and the 12-task slice are accepted.

Current evidence and explicit limits

• 60 task JSON files exist locally and pass the repository's task-schema validation.
• Source-family diversity and exact duplicate checks pass locally.
• The previous V1-targeted PR was closed unmerged; none of those tasks is represented as accepted.
• No official human-mode or model-panel run is claimed yet.
• No acceptance, merge, leaderboard inclusion, or paper credit is claimed.

Maintainer decisions requested

1. Does an independently versioned academia registry suite fit ClawBench's scope?
2. Should this use the proposed answer-mode path, or another evaluator maintained by the project?
3. Is a 12-task design slice the right first PR size?
4. Which namespace and version label should be used so released-suite statistics remain immutable?

Contributor: Yuxuan Zhang (reacher-z)
Affiliation: University of British Columbia; Vector Institute
Profile: https://yuxuan.world/

[reacher-z]

Prototype update (kept isolated from released V1/V2): I built the proposed first 12-task slice locally, balanced as 2 tasks each across Crossref, PubMed, ClinicalTrials.gov, GitHub, World Bank, and SEC EDGAR.

Executed checks:

• 12/12 task documents validate against the proposed answer-mode schema;
• 12/12 frozen reference answers pass a deterministic labeled-field verifier;
• 60/60 semantic negative controls are rejected (missing, extra, wrong, malformed, and duplicate fields);
• all tasks are read-only and explicitly disallow write actions.

The old placeholder-interceptor approach is not being presented as quality evidence. This prototype uses a proposed answer_exact_fields contract because current main documents that final-answer runner wiring is still incomplete.

I have not opened a task PR yet and am not treating this as acceptance. Before mapping the full 60 into an official branch, could a maintainer confirm whether (1) the isolated academia suite fits scope and (2) you prefer this deterministic exact-field mode or the emerging judge_answer path?