Skip to content

Commit 2d7d1cd

Browse files
committed
fix(skill): drop parallel_tool_calls=False from tool-less eval agents
The eval-set generator, trigger grader, and coverage grader have no tools — they only produce text. OpenAI's API rejects `parallel_tool_calls` when `tools` is unset: Invalid value for 'parallel_tool_calls': 'parallel_tool_calls' is only allowed when 'tools' are specified. `skill eval` would die at the very first LLM call, before any prompt was graded. Tests masked it because they patch `Runner.run` and never hit the LiteLLM layer. Fix: stop passing `model_settings=ModelSettings(parallel_tool_calls=False)` on those three agents — the flag is both invalid and meaningless when the agent has zero tools to parallelise. `ModelSettings` is no longer referenced anywhere in this module, so the import is dropped too. Pre-existing bug carried over from the original evaluator design; this is the first time anyone has run `skill eval` end-to-end against a real LLM (everything else mocks Runner.run).
1 parent ea63a33 commit 2d7d1cd

1 file changed

Lines changed: 0 additions & 4 deletions

File tree

openkb/skill/evaluator.py

Lines changed: 0 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -39,7 +39,6 @@
3939

4040
from agents import Agent, Runner
4141
from agents.exceptions import MaxTurnsExceeded
42-
from agents.model_settings import ModelSettings
4342

4443
from openkb.skill import extract_body, extract_frontmatter
4544

@@ -207,7 +206,6 @@ async def generate_eval_set(
207206
name="eval-set-generator",
208207
instructions=instructions,
209208
model=f"litellm/{model}",
210-
model_settings=ModelSettings(parallel_tool_calls=False),
211209
)
212210
try:
213211
result = await Runner.run(agent, "Generate the eval set now.", max_turns=3)
@@ -266,7 +264,6 @@ async def grade_one(
266264
name="trigger-grader",
267265
instructions=instructions,
268266
model=f"litellm/{model}",
269-
model_settings=ModelSettings(parallel_tool_calls=False),
270267
)
271268
try:
272269
result = await Runner.run(agent, f"Question: {question}", max_turns=2)
@@ -315,7 +312,6 @@ async def grade_coverage(
315312
name="coverage-grader",
316313
instructions=instructions,
317314
model=f"litellm/{model}",
318-
model_settings=ModelSettings(parallel_tool_calls=False),
319315
)
320316
try:
321317
result = await Runner.run(agent, f"Question: {question}", max_turns=2)

0 commit comments

Comments
 (0)