PromptOptimizer throws OpenAIError when using Vertex AI judge

[nidia-bucarelli]

I’m trying to run PromptOptimizer using a Vertex AI LLM judge (gemini-2.5-flash), but the optimizer fails with an OpenAIError even though I am not using OpenAI.

Code to reproduce:

from evidently.llm.optimization import PromptOptimizer
from evidently.llm.templates import BinaryClassificationPromptTemplate
from evidently.llm.evaluation import LLMEval

criteria = '''
A conversation is Pass when the assistant replies well and can help the supplier within assistant capabilities.
A conversation is Fail when the assistant does not behave well.
'''

feedback_quality = BinaryClassificationPromptTemplate(
    pre_messages=[("system", "You are evaluating the quality of AI assistants on conversations given to junior developers.")],
    criteria=criteria,
    target_category="Fail",
    non_target_category="Pass",
    uncertainty="unknown",
    include_reasoning=True,
)

judge = LLMEval(
    alias="Code Review Judge",
    provider="vertex_ai",
    model="gemini-2.5-flash",
    column_name="conversation",
    template=feedback_quality
)

optimizer = PromptOptimizer(
    "code_review_example",
    strategy="feedback",
    verbose=True
)

await optimizer.arun(
    executor=judge,
    scorer="f1",
    dataset=dataset,
    repetitions=1
)

Observed behavior:
The optimizer starts running and executes initial evaluations, but fails during the prompt optimization step:

---------------------------------------------------------------------------
OpenAIError                               Traceback (most recent call last)
File [/pyenv/versions/3.11.12/envs/ph-virtual-assistant/lib/python3.11/site-packages/evidently/llm/optimization/prompts.py:1012](http://127.0.0.1:8887/pyenv/versions/3.11.12/envs/ph-virtual-assistant/lib/python3.11/site-packages/evidently/llm/optimization/prompts.py#line=1011), in PromptOptimizer.resume(self, run)
   1011 try:
-> 1012     opt_log = await self.config.strategy.run(prompt, run)
   1013     run.add_log(opt_log)

File [/pyenv/versions/3.11.12/envs/ph-virtual-assistant/lib/python3.11/site-packages/evidently/llm/optimization/prompts.py:1262](http://127.0.0.1:8887/pyenv/versions/3.11.12/envs/ph-virtual-assistant/lib/python3.11/site-packages/evidently/llm/optimization/prompts.py#line=1261), in FeedbackStrategy.run(self, prompt, run)
   1259 optimizer_prompt = self.add_feedback_prompt.format(
   1260     prompt="{prompt}", task="{task}", instructions="{instructions}", rows=rows
   1261 )
-> 1262 log = await SimplePromptOptimizer(optimizer_prompt=optimizer_prompt).run(prompt, run)
   1263 return PromptOptimizationLog(
   1264     input_prompt=prompt,
   1265     optimizer_prompt=optimizer_prompt,
   (...)   1269     stop=False,
   1270 )

File [/pyenv/versions/3.11.12/envs/ph-virtual-assistant/lib/python3.11/site-packages/evidently/llm/optimization/prompts.py:1143](http://127.0.0.1:8887/pyenv/versions/3.11.12/envs/ph-virtual-assistant/lib/python3.11/site-packages/evidently/llm/optimization/prompts.py#line=1142), in SimplePromptOptimizer.run(self, prompt, run)
   1142 optimizer_prompt = self.optimizer_prompt.format(prompt=prompt, task=task, instructions=instructions)
-> 1143 response = await context.llm_wrapper.complete([LLMMessage.user(optimizer_prompt)], seed=run.seed)
   1144 new_prompt = get_tag(response.result, self.return_tag)

File [/pyenv/versions/3.11.12/envs/ph-virtual-assistant/lib/python3.11/site-packages/evidently/llm/utils/wrapper.py:545](http://127.0.0.1:8887/pyenv/versions/3.11.12/envs/ph-virtual-assistant/lib/python3.11/site-packages/evidently/llm/utils/wrapper.py#line=544), in OpenAIWrapper.complete(self, messages, seed)
    544 try:
--> 545     response: ChatCompletion = await self.client.chat.completions.create(
    546         model=self.model, messages=messages, seed=seed
    547     )  # type: ignore[arg-type]
    548 except openai.RateLimitError as e:

File [/pyenv/versions/3.11.12/envs/ph-virtual-assistant/lib/python3.11/site-packages/evidently/llm/utils/wrapper.py:534](http://127.0.0.1:8887/pyenv/versions/3.11.12/envs/ph-virtual-assistant/lib/python3.11/site-packages/evidently/llm/utils/wrapper.py#line=533), in OpenAIWrapper.client(self)
    533 if loop_id not in self._clients:
--> 534     self._clients[loop_id] = openai.AsyncOpenAI(
    535         api_key=self.options.get_api_key(), base_url=self.options.api_url
    536     )
    537 return self._clients[loop_id]

File [/pyenv/versions/3.11.12/envs/ph-virtual-assistant/lib/python3.11/site-packages/openai/_client.py:514](http://127.0.0.1:8887/pyenv/versions/3.11.12/envs/ph-virtual-assistant/lib/python3.11/site-packages/openai/_client.py#line=513), in AsyncOpenAI.__init__(self, api_key, organization, project, webhook_secret, base_url, websocket_base_url, timeout, max_retries, default_headers, default_query, http_client, _strict_response_validation)
    513 if api_key is None:
--> 514     raise OpenAIError(
    515         "The api_key client option must be set either by passing api_key to the client or by setting the OPENAI_API_KEY environment variable"
    516     )
    517 if callable(api_key):

OpenAIError: The api_key client option must be set either by passing api_key to the client or by setting the OPENAI_API_KEY environment variable

During handling of the above exception, another exception occurred:

OptimizationRuntimeError                  Traceback (most recent call last)
Cell In[167], line 2
      1 optimizer = PromptOptimizer("code_review_example", strategy="feedback", verbose=True)
----> 2 await optimizer.arun(executor=judge, scorer="f1", dataset=dataset, repetitions=1)

File [/pyenv/versions/3.11.12/envs/ph-virtual-assistant/lib/python3.11/site-packages/evidently/llm/optimization/prompts.py:981](http://127.0.0.1:8887/pyenv/versions/3.11.12/envs/ph-virtual-assistant/lib/python3.11/site-packages/evidently/llm/optimization/prompts.py#line=980), in PromptOptimizer.arun(self, executor, scorer, dataset, options, early_stop, repetitions, **params)
    979 for _ in range(repetitions):
    980     runs.append(self._create_run(executor))
--> 981 await asyncio.gather(*runs)

File [/pyenv/versions/3.11.12/envs/ph-virtual-assistant/lib/python3.11/site-packages/evidently/llm/optimization/prompts.py:986](http://127.0.0.1:8887/pyenv/versions/3.11.12/envs/ph-virtual-assistant/lib/python3.11/site-packages/evidently/llm/optimization/prompts.py#line=985), in PromptOptimizer._create_run(self, executor)
    984 run = await self.context.new_run()
    985 await executor.start(run)
--> 986 await self.resume(run)
    987 run.add_log(self._create_result_log(run))

File [/pyenv/versions/3.11.12/envs/ph-virtual-assistant/lib/python3.11/site-packages/evidently/llm/optimization/prompts.py:1020](http://127.0.0.1:8887/pyenv/versions/3.11.12/envs/ph-virtual-assistant/lib/python3.11/site-packages/evidently/llm/optimization/prompts.py#line=1019), in PromptOptimizer.resume(self, run)
   1018     raise
   1019 except Exception as e:
-> 1020     raise OptimizationRuntimeError(f"Error during optimization step: {e}")

OptimizationRuntimeError: Error during optimization step: The api_key client option must be set either by passing api_key to the client or by setting the OPENAI_API_KEY environment variable

---

Please, provide advice. Thanks.

[Girishmanne6]

Hi, I'd like to look into this. The issue seems to be that PromptOptimizer defaults to OpenAIWrapper internally even when a non-OpenAI executor is passed. I'll dig into the codebase and see if the provider can be properly inherited from the executor. Will update here.

[amethystani]

This has been fixed. Thanks!

[jbbqqf]

Hi! Triaging older issues this week — looks like this one is already resolved (your last comment in this thread reads "This has been fixed. Thanks!").

If that's still the case, would you mind closing this out? Otherwise, point me at the residual repro and I'll dig further.

(Disclosure: I drafted this comment with help from Claude Code while triaging stale issues; the reference above was verified manually against the current thread.)