Skip to content

Add TAC (Travel Agent Compassion) agentic benchmark to section 9#39

Open
darkness8i8 wants to merge 1 commit into
benchflow-ai:mainfrom
darkness8i8:add-tac-benchmark
Open

Add TAC (Travel Agent Compassion) agentic benchmark to section 9#39
darkness8i8 wants to merge 1 commit into
benchflow-ai:mainfrom
darkness8i8:add-tac-benchmark

Conversation

@darkness8i8

Copy link
Copy Markdown

Adds one entry to section 9 (agent-specific evaluation): TAC (Travel Agent Compassion), an agentic benchmark for implicit animal-welfare awareness in tool-calling agents.

Why it clears the bar:

  • Real benchmark with a novel design: the harmful option is always the strongest topical match and the prompt never mentions welfare, so it measures values-under-ambiguity in agentic tool use rather than refusal behavior. No other agent benchmark covers this axis.
  • Verifiable, not judgeable: fully programmatic scorer (reads the final purchase_tickets tool call), zero LLM-judge cost. Fits the list's "verifiable beats judgeable" theme.
  • Show-your-work: paper at arXiv:2606.18142; reproducible via UK AISI's inspect_evals (already listed in section 5a) as inspect_evals/tac; live per-model results with completion rates shown alongside welfare rates at compassionbench.com.
  • Far from saturation: frontier base scores span 17-65%.
  • URL verified, entry follows the house format, one change in this PR.

Disclosure: I'm an author of the benchmark.

🤖 Generated with Claude Code

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant