perf(skill): parallel eval, parallel compile reads, query_wiki cap

Three findings from the performance review on b6607ec:

1. `run_eval` was awaiting ~30 LLM calls sequentially (20 trigger
   graders + 10 coverage graders). Each prompt is independent — same
   `desc`/`content` inputs, results accumulated in eval_set order — so
   `asyncio.gather` is correctness-preserving. Wrap in a
   `Semaphore(EVAL_CONCURRENCY=8)` to bound simultaneous requests under
   provider rate limits. Expected wall-clock cut for `openkb skill eval`
   is roughly 4-15x depending on provider latency, with the floor set
   by the semaphore.

2. The compile agent had `parallel_tool_calls=False`, forcing every
   read tool (`list_wiki_dir`, `read_wiki_file`, `get_page_content`)
   into its own turn. The early phase of compile is naturally a
   read-fan-out (survey directories, read N summaries, follow
   `full_text` pointers to source page-ranges). Allowing parallel tool
   calls lets the model batch independent reads, saving roughly 5-10
   outer turns per compile (~20-40s at Opus-class latencies). Writes
   serialise naturally because each `write_skill_file` depends on
   accumulated reads.

3. `query_wiki` is a nested `Runner.run(max_turns=50)` inside the outer
   compile (`max_turns=80`). The docstring's "narrow follow-ups only"
   was the only enforcement; a pathological run could spawn many nested
   calls. Added a per-compile counter via closure: after
   `QUERY_WIKI_MAX_CALLS=3` invocations, the tool returns an error
   string steering the agent back to direct file reads. Bounds tail
   latency without breaking the common case (legitimate cross-document
   sub-questions still get answered).

Prompt-caching for `grade_coverage` (the fourth finding in the review)
was deferred: the openai-agents SDK takes `instructions` as a plain
string with no hook for emitting Anthropic `cache_control` markers.
OpenAI's automatic prefix caching already applies because the system
prompt is byte-stable across the 10 coverage calls; closing the
Anthropic gap is SDK-level work, not application-level.

Tests pass (463; unrelated trafilatura env failure in test_url_ingest
not touched).

Author: KylinMountain

openkb/skill/creator.py

@@ -31,6 +31,7 @@
 from openkb.schema import get_agents_md
 
 MAX_TURNS = 80  # higher than query (50) because compile can write multiple files
+QUERY_WIKI_MAX_CALLS = 3  # bound nested-LLM tail latency; agents should prefer direct reads
 
 
 def build_skill_create_agent(
@@ -103,6 +104,12 @@ def get_image(image_path: str) -> ToolOutputImage | ToolOutputText:
             return ToolOutputImage(image_url=result["image_url"])
         return ToolOutputText(text=result["text"])
 
+    # Per-compile counter for query_wiki. Each invocation spawns a nested
+    # Runner.run(max_turns=50), so an agent that leans on this tool can
+    # blow up wall-clock and token cost. The docstring already nudges
+    # toward direct reads; this hard cap is the structural backstop.
+    query_wiki_calls = 0
+
     @function_tool
     async def query_wiki(question: str) -> str:
         """Semantic search over the wiki — narrow follow-ups only.
@@ -112,7 +119,18 @@ async def query_wiki(question: str) -> str:
         does the book say about X across multiple chapters?"). For primary
         traversal, use list/read/get_page_content instead — they are
         cheaper and give you the raw text, not another LLM's summary.
+
+        Capped at ``QUERY_WIKI_MAX_CALLS`` invocations per compile.
         """
+        nonlocal query_wiki_calls
+        query_wiki_calls += 1
+        if query_wiki_calls > QUERY_WIKI_MAX_CALLS:
+            return (
+                f"query_wiki call cap reached "
+                f"({QUERY_WIKI_MAX_CALLS} per compile). Use direct file "
+                f"reads (read_wiki_file / get_page_content) for further "
+                f"investigation."
+            )
         # Lazy import to avoid a circular dependency at module load time.
         from openkb.agent.query import run_query
         kb_dir = Path(wiki_root).parent
@@ -141,7 +159,14 @@ def done(summary: str) -> str:
             done,
         ],
         model=f"litellm/{model}",
-        model_settings=ModelSettings(parallel_tool_calls=False),
+        # Allow the model to issue multiple read tool calls in one turn —
+        # the compile's early phase is a fan-out (list dir -> read N
+        # summaries -> read N source page-ranges), and serialising each
+        # read into its own turn costs roughly 5-10 extra round-trips per
+        # compile. Writes serialise naturally because each
+        # `write_skill_file` depends on accumulated reads; the model has
+        # no reason to issue parallel writes to the same path.
+        model_settings=ModelSettings(parallel_tool_calls=True),
     )
 
 

openkb/skill/evaluator.py

@@ -30,6 +30,7 @@
 """
 from __future__ import annotations
 
+import asyncio
 import json
 from dataclasses import dataclass, field
 from pathlib import Path
@@ -45,6 +46,11 @@
 
 EVAL_DEFAULT_COUNT = 10  # 10 trigger + 10 no-trigger = 20 prompts
 REFERENCES_PREVIEW_BYTES = 4000  # cap reference content fed to the eval LLM
+# Bound on concurrent grader LLM calls in run_eval. Without this the
+# default count=10 would fire ~30 simultaneous requests, which most
+# providers rate-limit. 8 is a conservative starting point — runs ~4x
+# the sequential baseline while staying well under typical RPM caps.
+EVAL_CONCURRENCY = 8
 
 
 @dataclass
@@ -366,23 +372,52 @@ async def run_eval(
     desc = _read_description(skill_dir)
     content = _skill_content_block(skill_dir)
     result = EvalResult(prompts=eval_set)
-    for prompt in eval_set:
-        graded = await grade_one(desc, prompt.question, model=model)
+
+    # Run grading concurrently. Each prompt is independent — graders read
+    # the same `desc`/`content` strings and produce results that are then
+    # appended to `result` in eval_set order below, so concurrent
+    # execution is correctness-preserving. A semaphore caps simultaneous
+    # LLM calls to avoid hitting provider rate limits.
+    sem = asyncio.Semaphore(EVAL_CONCURRENCY)
+
+    async def _trigger(p: EvalPrompt) -> Literal["trigger", "no-trigger"]:
+        async with sem:
+            return await grade_one(desc, p.question, model=model)
+
+    async def _coverage(p: EvalPrompt) -> tuple[
+        Literal["supported", "unsupported", "ambiguous"], str
+    ]:
+        async with sem:
+            return await grade_coverage(content, p.question, model=model)
+
+    trigger_tasks = [_trigger(p) for p in eval_set]
+    # Body alignment only meaningful on questions the skill claims to
+    # handle — for should-not questions the body is correctly empty of
+    # relevant material.
+    coverage_prompts = [p for p in eval_set if p.expected == "trigger"]
+    coverage_tasks = [_coverage(p) for p in coverage_prompts]
+
+    trigger_results, coverage_results = await asyncio.gather(
+        asyncio.gather(*trigger_tasks),
+        asyncio.gather(*coverage_tasks),
+    )
+
+    # Walk inputs in original order so `result.*` lists are deterministic
+    # even though the gather() above completed out of order.
+    for prompt, graded in zip(eval_set, trigger_results):
         if graded != prompt.expected:
             result.misses.append(EvalMiss(prompt=prompt, graded=graded))
-        # Body alignment only meaningful on questions the skill claims to
-        # handle — for should-not questions the body is correctly empty
-        # of relevant material.
-        if prompt.expected == "trigger":
-            verdict, reason = await grade_coverage(content, prompt.question, model=model)
-            if verdict == "ambiguous":
-                result.coverage_ambiguous.append(
-                    CoverageMiss(prompt=prompt, reason=reason)
-                )
-            elif verdict == "unsupported":
-                result.coverage_misses.append(
-                    CoverageMiss(prompt=prompt, reason=reason)
-                )
+
+    for prompt, (verdict, reason) in zip(coverage_prompts, coverage_results):
+        if verdict == "ambiguous":
+            result.coverage_ambiguous.append(
+                CoverageMiss(prompt=prompt, reason=reason)
+            )
+        elif verdict == "unsupported":
+            result.coverage_misses.append(
+                CoverageMiss(prompt=prompt, reason=reason)
+            )
+
     return result