Skip to content

Add 60 public-registry verification tasks - #304

Closed
reacher-z wants to merge 1 commit into
mainfrom
contributor/public-registry-verification-60
Closed

Add 60 public-registry verification tasks#304
reacher-z wants to merge 1 commit into
mainfrom
contributor/public-registry-verification-60

Conversation

@reacher-z

Copy link
Copy Markdown
Collaborator

Add 60 public-registry verification tasks

Category and sites

This proposed V1 category adds 60 read-only academia-research / public-registry-verification tasks across six authoritative public registries:

  • Crossref (10)
  • PubMed (10)
  • ClinicalTrials.gov (10)
  • GitHub (10)
  • World Bank Data (10)
  • SEC EDGAR (10)

The tasks use IDs 887–946. Every task asks the agent to open one explicit authoritative record and return a small labeled fact set. No task asks the agent to purchase, submit, send, delete, edit, or otherwise mutate external state.

Interceptor behavior

There is no Submit action. Every task is read-only and uses the repository's non-matching placeholder interceptor:

{"url_pattern":"__PLACEHOLDER_WILL_NOT_MATCH__","method":"POST"}

Validation performed

  • 60/60 conform to test-cases/task.schema.json.
  • IDs are unique and exactly cover 887–946.
  • Frozen reference solutions match all 60 source records.
  • All 60 prompts explicitly prohibit write actions.
  • Maximum pairwise instruction-token Jaccard is 0.835616, below the 0.84 review threshold.
  • Focused repository tests: 9 passed.
  • Full repository tests: 180 passed.

Draft status and remaining review gates

  • Run each task in official --human mode and retain the completion receipt.
  • Confirm each source remains navigable in the benchmark browser image.
  • Refresh mutable public fields immediately before the final commit and record any drift.
  • Confirm whether 10 minutes is appropriate after observed human completion times.

The branch has been pushed and this text is intended for a Draft PR. It deliberately does not claim human-mode completion, acceptance, merge, or paper credit. The Draft PR is the external submission receipt; it must remain a draft until the four gates above are resolved.

@reacher-z
reacher-z force-pushed the contributor/public-registry-verification-60 branch from 7fd82f1 to b978609 Compare August 18, 2026 11:56
@Perry2004

Copy link
Copy Markdown
Collaborator

The V1 suite has been released for months already and no new tasks should be added since that will change the suite statistics and all existing scores/analysis, including those in the paper.

If these tasks are truly needed or brings in something new, add them as a new suite (v3, or maybe a special academia suite).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

invalid This doesn't seem right

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants