Stick to OpenAI Client + ChatCompletions?

[alt-glitch]

Was going through the repo and noticed the slowly growing list of LLM clients and was wondering if it makes sense to stick to only OpenAI Client and switching base URLs and model names to different providers as is done in verifiers?

Happy to put in a refactor for this!

[lakshyaag]

Related question here: Does it make sense to support Responses API from OpenAI? It comes with its own code_executor tool, so that might be a conflict

[alt-glitch]

ChatCompletions is nice because it reduces the need to maintain different LLM APIs. Adopting Responses API would bring in the same issues of maintenance burden.

code_executor tool wouldn't be used since in this project we control the REPLs and their execution

[lakshyaag]

Yeah, that's fair.

[alexzhang13]

@alt-glitch Yeah looking at it now, I don't love the way we route things currently. What you're proposing is probably better, and I'm sure people who use the other clients (e.g. Vercel, etc.) are used to this.

The only problem there's some weird discrepancies I've noticed (e.g. for Prime Intellect inference, it expects an extra kwarg) so we might need to just hard-code them. If you have time, feel free to re-factor. I'll check back on this thread later.

[alt-glitch]

Cool. Let me work on this.

[kmad]

Are folks opposed to using e.g., LiteLLM so we don't need to have provider-specific implementations (and the inevitable API upgrades that come with it)? It should live at the right level of abstraction to give us what we need (completions, tool calling) without going to the API level. Thoughts?

[alexzhang13]

@kmad I think I do still want to support the major routers in case that's what people are used to (e.g. OpenRouter, LiteLLM, etc.).