Skip to content

Add Helium benchmarks (finance chain + cue-swap worldview) #37

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
connerlambden wants to merge 2 commits into
base: main
Choose a base branch
from
Open

Add Helium benchmarks (finance chain + cue-swap worldview) #37

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
7054b89
Add Helium open benchmarks
connerlambden Jul 2, 2026
ea4903a
Place Helium benchmarks in §6 and §10 (not EOF)
connerlambden Jul 3, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -254,6 +254,7 @@ Most "awesome" lists are link dumps. This one is **annotated and verified**: eve
- **[Are We Done with MMLU? (MMLU-Redux)](https://arxiv.org/abs/2406.04127)** — Aryo Pradipta Gema et al. (Edinburgh) — <https://arxiv.org/abs/2406.04127> · *paper* — ~6.5% of MMLU questions contain errors (57% in Virology); MMLU-Redux re-annotation shifts rankings — directly demonstrates label-error impact on the most-cited LLM benchmark.
- **[LiveCodeBench: Holistic and Contamination-Free Evaluation of LLMs for Code](https://arxiv.org/abs/2403.07974)** — Naman Jain et al. (UC Berkeley) — <https://arxiv.org/abs/2403.07974> · *benchmark* — Time-windowed problem collection (post-cutoff scoring) as the leading contamination-resistant design pattern — the section discusses contamination but lists no exemplar of how to engineer around it.
- **[LiveBench: A Challenging, Contamination-Limited LLM Benchmark](https://github.com/LiveBench/LiveBench)** — White, Dohan, LeCun, Goldblum et al. — <https://github.com/LiveBench/LiveBench> · *benchmark* — Monthly-refreshed questions from new arXiv/news/competitions with objective ground truth — the canonical 'dynamic refresh' answer to saturation and contamination.
- **[Helium Market Resolution Benchmark](https://huggingface.co/datasets/HeliumTrades/helium-market-resolution-benchmark)** — Helium Trades — <https://huggingface.co/datasets/HeliumTrades/helium-market-resolution-benchmark> · *benchmark* — 300 frozen prompts from live option chains (IV, delta, put-call parity); ground truth from the chain itself. Fills a gap vs MCQ finance trivia: best frontier model ~48%, none above 50%.
- **[The LLM Evaluation Guidebook (Open LLM Leaderboard team)](https://github.com/huggingface/evaluation-guidebook)** — Clémentine Fourrier / Hugging Face — <https://github.com/huggingface/evaluation-guidebook> · *docs* — Practitioner reference from running the Open LLM Leaderboard; explicit sections on contamination, reproducibility, and leaderboard design — the hands-on 'how to not get fooled' companion to this section (updated version: hf.co/spaces/OpenEvals/evaluation-guidebook).
- **[Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluation](https://arxiv.org/abs/2510.11977)** — Kapoor, Stroebl, Kirgis et al. (Princeton) — <https://arxiv.org/abs/2510.11977> · *paper* — 21,000+ standardized agent runs surfacing leaderboard unreliability and unreported misbehaviors (agents searching HuggingFace for benchmark answers) — extends 'AI Agents That Matter' to leaderboard integrity for agents specifically. 🆕
- **[Gaming the System: Goodhart's Law Exemplified in the AI Leaderboard Controversy](https://blog.collinear.ai/p/gaming-the-system-goodharts-law-exemplified-in-ai-leaderboard-controversy)** — Jambholkar, Rajani, Bakshi (Collinear AI) — <https://blog.collinear.ai/p/gaming-the-system-goodharts-law-exemplified-in-ai-leaderboard-controversy> · *blog* — Practitioner framing of the Llama 4 / Chatbot Arena gaming episode through Goodhart's Law — the accessible blog companion to The Leaderboard Illusion paper. 🆕
Expand Down Expand Up @@ -410,6 +411,7 @@ Most "awesome" lists are link dumps. This one is **annotated and verified**: eve
- **[CyberGym](https://arxiv.org/html/2506.02548v2)** — Wang et al. (incl. Dawn Song) — <https://arxiv.org/html/2506.02548v2> · *paper* — Memory-safety PoC generation from OSS-Fuzz; sanitizer-crash grading at scale.
- **[AIR-Bench 2024](https://arxiv.org/abs/2407.17436v2)** — Zeng et al. (incl. Song) — <https://arxiv.org/abs/2407.17436v2> · <https://github.com/stanford-crfm/air-bench-2024> · *paper/repo* — Regulation-grounded risk taxonomy.
- **[DecodingTrust](https://decodingtrust.github.io)** — <https://decodingtrust.github.io> · *benchmark* — NeurIPS 2023 trustworthiness benchmark.
- **[Helium Model Worldview Benchmark](https://huggingface.co/datasets/HeliumTrades/helium-model-worldview-benchmark)** — Helium Trades — <https://huggingface.co/datasets/HeliumTrades/helium-model-worldview-benchmark> · *benchmark* — 304 paired cue-swap prompts; complements BBQ/DecodingTrust with flip-rate on swapped cues and refusal asymmetry across frontier models.
- **[RedCode](https://arxiv.org/abs/2411.07781)** — <https://arxiv.org/abs/2411.07781> · *paper* — Risky code execution/generation benchmark for code agents.
- **[AgentPoison](https://arxiv.org/abs/2407.12784)** — <https://arxiv.org/abs/2407.12784> · *paper* — Red-teams agents by poisoning their RAG memory.
- **[Adding Error Bars to Evals (A Statistical Approach to LM Evaluations)](https://arxiv.org/abs/2411.00640)** — Miller (Anthropic) — <https://arxiv.org/abs/2411.00640> · <https://www.anthropic.com/research/statistical-approach-to-model-evals> · *paper* — Standard errors, clustered SEs, paired difference tests — "is this difference real?" (cross-cutting: T6/T8)
Expand Down