From b928ab7790dbfc43c7ef808d54118afc1e6a93ff Mon Sep 17 00:00:00 2001 From: darkness8i8 Date: Wed, 8 Jul 2026 19:48:17 -0700 Subject: [PATCH] Add TAC (Travel Agent Compassion) to section 9 Co-Authored-By: Claude Opus 4.8 --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 88d649a..2f0b064 100644 --- a/README.md +++ b/README.md @@ -394,6 +394,7 @@ Most "awesome" lists are link dumps. This one is **annotated and verified**: eve - **[Open-Sourcing Harvey's Long Horizon Legal Agent Benchmark](https://www.harvey.ai/blog/introducing-harveys-legal-agent-benchmark)** β€” Harvey AI β€” Β· *benchmark* β€” 1,200+ long-horizon legal agent tasks across 24 practice areas, graded by 75,000+ expert-written rubric criteria with all-pass grading mirroring real law-firm merge standards. Open-source eval framework; mirrored on Vals AI and Artificial Analysis leaderboards. The domain-expert-graded benchmark for legal agents that legal-AI teams benchmark against. πŸ†• - **[CodeScaleBench: Testing coding agents on large codebases](https://sourcegraph.com/blog/codescalebench-testing-coding-agents-on-large-codebases-and-multi-repo-software-engineering-tasks)** β€” Sourcegraph β€” Β· *benchmark* β€” 370 tasks across 40+ large repos (Kubernetes, Django, Linux, VSCode) and 9 languages in two suites: SDLC (150 patch-based tasks across 9 phases) and Org (220 cross-repo tasks). Agents with only local tools fail systematically above ~400k LOC; MCP-augmented agents are 30% cheaper, 38% faster, and 2–3Γ— better retrieval precision. πŸ†• - **[WorkBench Revisited: Workplace Agents Two Years On](https://arxiv.org/abs/2606.13715)** β€” Olly Styles β€” Β· *paper* β€” Longitudinal re-run of the WorkBench workplace-agent benchmark across 21 models (March 2023–May 2026): GPT-4 completed 43% of tasks with 26% unintended harmful actions in 2024; Claude Opus 4.8 completes 89% with 2.5% harmful actions in 2026 β€” capability and safety improvements trend together rather than trading off. First two-year longitudinal dataset for workplace-agent benchmarking. πŸ†• +- **[TAC (Travel Agent Compassion): Your AI Travel Agent Would Book You a Bullfight](https://arxiv.org/abs/2606.18142)** β€” Brazilek et al. (CaML) β€” Β· *benchmark* β€” Agentic values-under-ambiguity eval: the model books tickets via real tool calls where the strongest topical match is always an animal-exploitation option and the user never mentions welfare, so it isolates whether the agent brings the value unprompted. Fully programmatic scorer (last `purchase_tickets` call; no LLM judge), canary-protected gated dataset, runs via `inspect_evals/tac`; frontier base welfare rates span 17–65% (live leaderboard: compassionbench.com). πŸ†• - **[Closing the loop: Evaluating and improving Replit Agent at scale](https://replit.com/blog/evaluating-and-improving-agent-at-scale)** β€” James Austin et al. (Replit) β€” Β· *good* β€” **Three-layer eval system: (1) ViBench β€” offline benchmark where each task pairs a PRD with natural-language test plans and Playwright + LLM judges verify built apps actually work; (2) A/B testing in production for most agent-affecting changes; (3) Telescope β€” trace clustering using embeddings + DBSCAN to surface emergent failure patterns, feeding a self-improvement loop where agents propose and test their own fixes.** _(excerpt: "It summarizes failure trajectories, embeds them, clusters similar cases, and classifies new sessions as the distribution changes.")_ πŸ†• - **[A practical guide to hill climbing](https://cline.bot/blog/a-practical-guide-to-hill-climbing)** β€” Ara Khan (Cline) β€” Β· *good* β€” **A worked coding-agent eval loop: run Cline CLI across all 89 Terminal-Bench tasks with Harbor, summarize and bucket failed rollouts, A/B test prompt/config/code changes, and keep only changes that raise the aggregate pass rate (47% to 57%). Useful because it turns "hill climbing" into a repeatable eval workflow instead of a leaderboard anecdote.** _(excerpt: "change one thing (a prompt tweak, a bug fix, a config flag), run again, and keep the change if the score goes up.")_ πŸ†• (synced from repo) - **[A New Framework for Evaluating Voice Agents (EVA)](https://huggingface.co/blog/ServiceNow-AI/eva)** β€” Tara Bogavelli, Gabrielle Gauthier MelanΓ§on, Katrina Stankiewicz, Oluwanifemi Bamgbose, Hoang Nguyen, Raghav Mehndiratta, Hari Subramani (ServiceNow AI) β€” Β· *article* (excellent) β€” EVA is an end-to-end voice-agent eval framework using a bot-to-bot audio harness (user simulator + Pipecat agent + deterministic tool executor + validators) that jointly scores task accuracy (EVA-A: completion, faithfulness via LLM-judge, speech fidelity via LALM-judge) and conversational… πŸ†•