Skip to content

feat: add LiteLLM judge api_type - #288

Open
prodmanpd wants to merge 1 commit into
TIGER-AI-Lab:mainfrom
prodmanpd:feat/add-litellm-provider
Open

feat: add LiteLLM judge api_type#288
prodmanpd wants to merge 1 commit into
TIGER-AI-Lab:mainfrom
prodmanpd:feat/add-litellm-provider

Conversation

@prodmanpd

@prodmanpd prodmanpd commented Aug 17, 2026

Copy link
Copy Markdown

What does this PR do?

Adds a litellm judge api_type, so the ClawBench judge can reach any of 100+ providers (OpenAI, Anthropic, Gemini, Bedrock, Vertex, Azure, Groq, ...) through a single call instead of a per-wire-format raw-HTTP function. litellm.completion natively speaks the OpenAI / Anthropic / Gemini / Bedrock protocols, so it collapses the existing per-api_type handlers into one path, and drop_params=True keeps a single call portable across providers.

Changes:

  • src/clawbench/runner/judge.py and judge_llm.py - new _call_litellm(...) + a litellm branch in each judge's dispatch (mirrors the sibling _call_openai_chat / _call_anthropic_messages).
  • src/clawbench/runner/run_support/api_preflight.py - litellm preflight validation (a tiny call before containers run).
  • src/clawbench/tui.py - litellm added to API_TYPES + a LiteLLM provider preset.
  • pyproject.toml - optional extra litellm = ["litellm>=1.85.0,<2.0"] (lazy-imported, so the base install is unaffected).
  • tests/test_judge_litellm.py - dispatch / kwargs / preflight / import-error coverage.

Additive and opt-in: existing api_type handlers are untouched; litellm is an optional dependency imported lazily inside _call_litellm, with a clear ImportError pointing to pip install clawbench[litellm] if the api_type is used without it.

Corpus

  • v2
  • v1
  • both
  • not applicable

Test plan

  • Verified against the repo's frozen uv environment:
    • uv run --frozen pytest tests/test_judge_litellm.py -> 6 passed (dispatch in both judges, drop_params=True, credentials forwarded only when set, preflight, and the ImportError path).
    • uv run --frozen ruff check . -> All checks passed!; uv run --frozen ruff format --check . -> clean; uv run --frozen pyright <changed files> -> 0 errors.
    • Live E2E: ran the real judge_llm.judge_request with api_type: "litellm" routed through a LiteLLM proxy (gpt-4o-mini). Aligned instruction -> {"match": true, ...}; contradicting instruction -> {"match": false, "reason": "Instruction specifies 'BLUE shirt', request body shows 'item: red shirt'"}. This exercises the full chain: judge_request -> api_type dispatch -> _call_litellm -> litellm.completion -> provider -> parsed verdict.
    • Pin resolves: litellm>=1.85.0,<2.0 installs against the current latest stable (1.97.0).

@Perry2004

Copy link
Copy Markdown
Collaborator

Thanks for the contribution.

Could you clarify the use case for adding a LiteLLM API type? ClawBench already have support for OpenAI, Anthropic, and Google GenAI API types and custom URLs, which should cover almost all cases calling LLM APIs I believe.

Can you explain the motivation why a LiteLLM is needed for calling LLM APIs for the judge.

@Perry2004 Perry2004 added the question Further information is requested label Aug 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

question Further information is requested

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants