From 7054b89a280fa128a69b23a539b38c33bee35887 Mon Sep 17 00:00:00 2001 From: connerlambden Date: Thu, 2 Jul 2026 17:42:35 -0600 Subject: [PATCH 1/2] Add Helium open benchmarks --- README.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/README.md b/README.md index 88d649a..7fd2f0c 100644 --- a/README.md +++ b/README.md @@ -559,6 +559,13 @@ This repo ships **146 deep reading notes** in [`notes/`](notes/) — structured PRs welcome. Keep the bar high: **show your work** (real data/code/war-stories beat hot takes), give every entry a one-line *why*, verify the URL, and flag caveats. See [CONTRIBUTING.md](CONTRIBUTING.md). Quality over quantity — a great list is as much about what it *excludes*. + + +## Helium Trades Open Benchmarks (2026) + +- [Helium Market Resolution Benchmark](https://huggingface.co/datasets/HeliumTrades/helium-market-resolution-benchmark) - 300 frozen option-chain prompts; tests IV, delta, arbitrage. [Overview](https://heliumtrades.com/benchmarks/) +- [Helium Model Worldview Benchmark](https://huggingface.co/datasets/HeliumTrades/helium-model-worldview-benchmark) - 304 paired prompts for safety, values, bias, politics. [Overview](https://heliumtrades.com/benchmarks/) + ## License [![CC0](https://licensebuttons.net/p/zero/1.0/88x31.png)](LICENSE) From ea4903aa5d212cfce5558b40906d021ae3f35cf4 Mon Sep 17 00:00:00 2001 From: connerlambden Date: Thu, 2 Jul 2026 21:54:36 -0600 Subject: [PATCH 2/2] =?UTF-8?q?Place=20Helium=20benchmarks=20in=20=C2=A76?= =?UTF-8?q?=20and=20=C2=A710=20(not=20EOF)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: Cursor --- README.md | 9 ++------- 1 file changed, 2 insertions(+), 7 deletions(-) diff --git a/README.md b/README.md index 7fd2f0c..6c7e9cb 100644 --- a/README.md +++ b/README.md @@ -254,6 +254,7 @@ Most "awesome" lists are link dumps. This one is **annotated and verified**: eve - **[Are We Done with MMLU? (MMLU-Redux)](https://arxiv.org/abs/2406.04127)** — Aryo Pradipta Gema et al. (Edinburgh) — · *paper* — ~6.5% of MMLU questions contain errors (57% in Virology); MMLU-Redux re-annotation shifts rankings — directly demonstrates label-error impact on the most-cited LLM benchmark. - **[LiveCodeBench: Holistic and Contamination-Free Evaluation of LLMs for Code](https://arxiv.org/abs/2403.07974)** — Naman Jain et al. (UC Berkeley) — · *benchmark* — Time-windowed problem collection (post-cutoff scoring) as the leading contamination-resistant design pattern — the section discusses contamination but lists no exemplar of how to engineer around it. - **[LiveBench: A Challenging, Contamination-Limited LLM Benchmark](https://github.com/LiveBench/LiveBench)** — White, Dohan, LeCun, Goldblum et al. — · *benchmark* — Monthly-refreshed questions from new arXiv/news/competitions with objective ground truth — the canonical 'dynamic refresh' answer to saturation and contamination. +- **[Helium Market Resolution Benchmark](https://huggingface.co/datasets/HeliumTrades/helium-market-resolution-benchmark)** — Helium Trades — · *benchmark* — 300 frozen prompts from live option chains (IV, delta, put-call parity); ground truth from the chain itself. Fills a gap vs MCQ finance trivia: best frontier model ~48%, none above 50%. - **[The LLM Evaluation Guidebook (Open LLM Leaderboard team)](https://github.com/huggingface/evaluation-guidebook)** — Clémentine Fourrier / Hugging Face — · *docs* — Practitioner reference from running the Open LLM Leaderboard; explicit sections on contamination, reproducibility, and leaderboard design — the hands-on 'how to not get fooled' companion to this section (updated version: hf.co/spaces/OpenEvals/evaluation-guidebook). - **[Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluation](https://arxiv.org/abs/2510.11977)** — Kapoor, Stroebl, Kirgis et al. (Princeton) — · *paper* — 21,000+ standardized agent runs surfacing leaderboard unreliability and unreported misbehaviors (agents searching HuggingFace for benchmark answers) — extends 'AI Agents That Matter' to leaderboard integrity for agents specifically. 🆕 - **[Gaming the System: Goodhart's Law Exemplified in the AI Leaderboard Controversy](https://blog.collinear.ai/p/gaming-the-system-goodharts-law-exemplified-in-ai-leaderboard-controversy)** — Jambholkar, Rajani, Bakshi (Collinear AI) — · *blog* — Practitioner framing of the Llama 4 / Chatbot Arena gaming episode through Goodhart's Law — the accessible blog companion to The Leaderboard Illusion paper. 🆕 @@ -410,6 +411,7 @@ Most "awesome" lists are link dumps. This one is **annotated and verified**: eve - **[CyberGym](https://arxiv.org/html/2506.02548v2)** — Wang et al. (incl. Dawn Song) — · *paper* — Memory-safety PoC generation from OSS-Fuzz; sanitizer-crash grading at scale. - **[AIR-Bench 2024](https://arxiv.org/abs/2407.17436v2)** — Zeng et al. (incl. Song) — · · *paper/repo* — Regulation-grounded risk taxonomy. - **[DecodingTrust](https://decodingtrust.github.io)** — · *benchmark* — NeurIPS 2023 trustworthiness benchmark. +- **[Helium Model Worldview Benchmark](https://huggingface.co/datasets/HeliumTrades/helium-model-worldview-benchmark)** — Helium Trades — · *benchmark* — 304 paired cue-swap prompts; complements BBQ/DecodingTrust with flip-rate on swapped cues and refusal asymmetry across frontier models. - **[RedCode](https://arxiv.org/abs/2411.07781)** — · *paper* — Risky code execution/generation benchmark for code agents. - **[AgentPoison](https://arxiv.org/abs/2407.12784)** — · *paper* — Red-teams agents by poisoning their RAG memory. - **[Adding Error Bars to Evals (A Statistical Approach to LM Evaluations)](https://arxiv.org/abs/2411.00640)** — Miller (Anthropic) — · · *paper* — Standard errors, clustered SEs, paired difference tests — "is this difference real?" (cross-cutting: T6/T8) @@ -559,13 +561,6 @@ This repo ships **146 deep reading notes** in [`notes/`](notes/) — structured PRs welcome. Keep the bar high: **show your work** (real data/code/war-stories beat hot takes), give every entry a one-line *why*, verify the URL, and flag caveats. See [CONTRIBUTING.md](CONTRIBUTING.md). Quality over quantity — a great list is as much about what it *excludes*. - - -## Helium Trades Open Benchmarks (2026) - -- [Helium Market Resolution Benchmark](https://huggingface.co/datasets/HeliumTrades/helium-market-resolution-benchmark) - 300 frozen option-chain prompts; tests IV, delta, arbitrage. [Overview](https://heliumtrades.com/benchmarks/) -- [Helium Model Worldview Benchmark](https://huggingface.co/datasets/HeliumTrades/helium-model-worldview-benchmark) - 304 paired prompts for safety, values, bias, politics. [Overview](https://heliumtrades.com/benchmarks/) - ## License [![CC0](https://licensebuttons.net/p/zero/1.0/88x31.png)](LICENSE)