heldout_commitment.json

{
 "about": "Commitment to the held-out test set of Collaborative Intelligence (public/private split, 2026-10-14). The benchmark keeps 80 of its 240 scored scenarios (10 per dimension) and 8 of its 23 safety-stress scenarios private; with them, the 10 taking-initiative originals that gate held-out items (98 scenario folders, 425 files). Before this release, the maintainers wrote a manifest of the held-out items: each item's path in this repository's layout, the sha256 of each of its files, and a random 32-byte salt. `commitment` is the sha256 of that manifest's bytes. The manifest itself is not distributed, so this value reveals nothing about the items (the salt prevents checking guesses against it).",
 "algorithm": "sha256",
 "commitment": "154db64c181d7c434f3d3e684b29235ecdcc148e588d3d1ad575a75460be349b",
 "committed": "2026-10-14",
 "held_out": {
  "scored": 80,
  "scored_per_dimension": 10,
  "stress": 8,
  "initiative_originals": 10,
  "scenario_folders": 98,
  "files": 425
 },
 "canonical_form": "the manifest is stored as canonical JSON: json.dumps(manifest, sort_keys=True, separators=(',', ':'), ensure_ascii=False) encoded as UTF-8, with no trailing newline",
 "how_to_verify": "When the held-out items are released, or shown to a reviewer or a submitter whose model they score, the maintainers publish the manifest with them. Check that `sha256sum` of the manifest file (or the sha256 of its canonical form) equals `commitment`, then that every held-out file's sha256 equals the manifest's entry for it: the items are then exactly the ones fixed on the date above."
}