Skip to content

fix(eval): harden GEPA discovery and reuse - #88

Open
LukasParke wants to merge 5 commits into
mattapperson:mainfrom
LukasParke:lukeparke/openrouter-gepa-hardening
Open

fix(eval): harden GEPA discovery and reuse#88
LukasParke wants to merge 5 commits into
mattapperson:mainfrom
LukasParke:lukeparke/openrouter-gepa-hardening

Conversation

@LukasParke

Copy link
Copy Markdown
Collaborator

What

  • thread optimization scope into runtime field discovery
  • use stable tool-index paths in discovery and mutation
  • traverse current composite step kinds without restoring pre-feat!: unify Noetic public API naming #68 names
  • bound per-field teacher proposals with the shared run pool
  • reuse LLM-judge harnesses for identical provider/model configuration
  • add timeout and kill grace to the Pi coding-agent RPC path

Why

GEPA discovery ignored the requested scope, mutable tool names made candidate paths unstable, composite traversal missed current step shapes, and judge harnesses were rebuilt for every score call.

This ports item 5, the remainder of fork/port/openrouter-fixes commit aedf712f, on top of item 4 #72.

Test plan

  • 49 focused discovery/GEPA tests pass
  • eval typecheck
  • root lint
  • sentrux check .

Dependency

Depends on #72 for the shared run pool and removed dead budget option. The branch currently includes that prerequisite and must be rebased through main after merge.

Port the eval safety and case-concurrency portion of fork/port/openrouter-fixes commit aedf712 onto current main. The dead budget option is removed; GEPA discovery and traversal remain a separate follow-up.

Signed-off-by: Luke Parke <5702154+LukasParke@users.noreply.github.com>
Port item 5 from fork/port/openrouter-fixes commit aedf712: scoped discovery, stable tool paths, bounded proposals, judge reuse, and coding-agent timeout.

Signed-off-by: Luke Parke <5702154+LukasParke@users.noreply.github.com>
@vercel

vercel Bot commented Aug 13, 2026

Copy link
Copy Markdown

@LukasParke is attempting to deploy a commit to the Matt Apperson's projects Team on Vercel.

A member of the Team first needs to authorize it.

@LukasParke

Copy link
Copy Markdown
Collaborator Author

Agent: Code CI/structural/DCO are green where configured; remaining external red checks are compat with an empty upstream OPENROUTER_API_KEY and Vercel authorization. Where core CI is red, the only failure is the unrelated timing-sensitive executeRunCode retry-backoff test; affected local/full suites passed. I cannot rerun upstream Actions without admin rights.

Run executeRunCode tests serially so their temporary global setTimeout proxy cannot capture delays from sibling tests under CI concurrency.

Signed-off-by: Luke Parke <5702154+LukasParke@users.noreply.github.com>
Expose the pure retry-delay calculation for deterministic assertions instead of proxying global setTimeout, which captured sibling test timers under CI concurrency.

Signed-off-by: Luke Parke <5702154+LukasParke@users.noreply.github.com>
@LukasParke

Copy link
Copy Markdown
Collaborator Author

Agent: Root-caused the repeated core CI failure to retry tests proxying global setTimeout and capturing sibling-test timers. Pushed a deterministic fix that tests the pure retry-delay function without mutating globals. DCO and Structural gate pass; CI rerun is in progress.

Signed-off-by: Luke Parke <5702154+LukasParke@users.noreply.github.com>
@LukasParke

Copy link
Copy Markdown
Collaborator Author

Agent: Root cause was Bun mock.module() leakage from optimizer.test.ts: mock.restore() does not evict the mocked dynamic-import module, so later GEPA environment-resolution tests could receive the stale fake bridge. Commit 112cde45 adds an explicit bridge-loader seam for optimizer tests, removes module mocking, and adds a full-suite regression. Verification: eval 284/284, eval test:ci, eval typecheck, root lint, and root test 1,446 pass / 5 skip. DCO and Structural gate are green; CI rerun is in progress.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant