Skip to content
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -394,6 +394,7 @@ Most "awesome" lists are link dumps. This one is **annotated and verified**: eve
- **[Open-Sourcing Harvey's Long Horizon Legal Agent Benchmark](https://www.harvey.ai/blog/introducing-harveys-legal-agent-benchmark)** — Harvey AI — <https://www.harvey.ai/blog/introducing-harveys-legal-agent-benchmark> · *benchmark* — 1,200+ long-horizon legal agent tasks across 24 practice areas, graded by 75,000+ expert-written rubric criteria with all-pass grading mirroring real law-firm merge standards. Open-source eval framework; mirrored on Vals AI and Artificial Analysis leaderboards. The domain-expert-graded benchmark for legal agents that legal-AI teams benchmark against. 🆕
- **[CodeScaleBench: Testing coding agents on large codebases](https://sourcegraph.com/blog/codescalebench-testing-coding-agents-on-large-codebases-and-multi-repo-software-engineering-tasks)** — Sourcegraph — <https://sourcegraph.com/blog/codescalebench-testing-coding-agents-on-large-codebases-and-multi-repo-software-engineering-tasks> · *benchmark* — 370 tasks across 40+ large repos (Kubernetes, Django, Linux, VSCode) and 9 languages in two suites: SDLC (150 patch-based tasks across 9 phases) and Org (220 cross-repo tasks). Agents with only local tools fail systematically above ~400k LOC; MCP-augmented agents are 30% cheaper, 38% faster, and 2–3× better retrieval precision. 🆕
- **[WorkBench Revisited: Workplace Agents Two Years On](https://arxiv.org/abs/2606.13715)** — Olly Styles — <https://arxiv.org/abs/2606.13715> · *paper* — Longitudinal re-run of the WorkBench workplace-agent benchmark across 21 models (March 2023–May 2026): GPT-4 completed 43% of tasks with 26% unintended harmful actions in 2024; Claude Opus 4.8 completes 89% with 2.5% harmful actions in 2026 — capability and safety improvements trend together rather than trading off. First two-year longitudinal dataset for workplace-agent benchmarking. 🆕
- **[TAC (Travel Agent Compassion): Your AI Travel Agent Would Book You a Bullfight](https://arxiv.org/abs/2606.18142)** — Brazilek et al. (CaML) — <https://arxiv.org/abs/2606.18142> · *benchmark* — Agentic values-under-ambiguity eval: the model books tickets via real tool calls where the strongest topical match is always an animal-exploitation option and the user never mentions welfare, so it isolates whether the agent brings the value unprompted. Fully programmatic scorer (last `purchase_tickets` call; no LLM judge), canary-protected gated dataset, runs via `inspect_evals/tac`; frontier base welfare rates span 17–65% (live leaderboard: compassionbench.com). 🆕
- **[Closing the loop: Evaluating and improving Replit Agent at scale](https://replit.com/blog/evaluating-and-improving-agent-at-scale)** — James Austin et al. (Replit) — <https://replit.com/blog/evaluating-and-improving-agent-at-scale> · *good* — **Three-layer eval system: (1) ViBench — offline benchmark where each task pairs a PRD with natural-language test plans and Playwright + LLM judges verify built apps actually work; (2) A/B testing in production for most agent-affecting changes; (3) Telescope — trace clustering using embeddings + DBSCAN to surface emergent failure patterns, feeding a self-improvement loop where agents propose and test their own fixes.** _(excerpt: "It summarizes failure trajectories, embeds them, clusters similar cases, and classifies new sessions as the distribution changes.")_ 🆕
- **[A practical guide to hill climbing](https://cline.bot/blog/a-practical-guide-to-hill-climbing)** — Ara Khan (Cline) — <https://cline.bot/blog/a-practical-guide-to-hill-climbing> · *good* — **A worked coding-agent eval loop: run Cline CLI across all 89 Terminal-Bench tasks with Harbor, summarize and bucket failed rollouts, A/B test prompt/config/code changes, and keep only changes that raise the aggregate pass rate (47% to 57%). Useful because it turns "hill climbing" into a repeatable eval workflow instead of a leaderboard anecdote.** _(excerpt: "change one thing (a prompt tweak, a bug fix, a config flag), run again, and keep the change if the score goes up.")_ 🆕 (synced from repo)
- **[A New Framework for Evaluating Voice Agents (EVA)](https://huggingface.co/blog/ServiceNow-AI/eva)** — Tara Bogavelli, Gabrielle Gauthier Melançon, Katrina Stankiewicz, Oluwanifemi Bamgbose, Hoang Nguyen, Raghav Mehndiratta, Hari Subramani (ServiceNow AI) — <https://huggingface.co/blog/ServiceNow-AI/eva> · *article* (excellent) — EVA is an end-to-end voice-agent eval framework using a bot-to-bot audio harness (user simulator + Pipecat agent + deterministic tool executor + validators) that jointly scores task accuracy (EVA-A: completion, faithfulness via LLM-judge, speech fidelity via LALM-judge) and conversational… 🆕
Expand Down
Loading