80 of the 240 scenarios are a private test set that only we run (held-out test set). Email us to have a model scored on all 240, released or not, and we send you a private report.
- Email seanwu@stanford.edu the model and the name to show on the leaderboard, how we can call it (a public API, access for our API account, a key made for this evaluation, or open weights), whether it is released (if not, when it will be), and whether your API trains on our requests.
- We run it once, with the same pipeline as every model on the leaderboard (evaluation policy).
- You get a private report: the score, the released and held-out scores, every dimension, where the model would rank, and the transcripts of the released scenarios. The held-out scenarios are never shared.
The model needs tool calling and a context window of about 128,000 tokens. We pay for the teammates and the judges; the model's own calls go to your account. Never send a key by email: if we need one, send a key made for this evaluation only, through a one-time secret link.
The result stays private until the model is released, and until then we tell no one about it (we can sign a confidentiality agreement). After the release we check that the released model is the one we scored (release check). A model that is never released is not published.
Version 2026-10-22. By submitting a model you agree that:
- We evaluate each configuration once, at most three per developer at a time, and publish the result: a released model's with the next leaderboard update, an unreleased model's no later than 30 days after its release. It is not withdrawn or changed on request.
- If the released model is not the one we evaluated, we publish the released model's result instead.
- Your API does not train on our requests.
- We publish the result card and the released scenarios' transcripts, and keep all transcripts privately to rescore them under later versions of the benchmark.
- We may decline a submission.