Skip to content

fix(eval): guard writes and bound case concurrency - #72

Open
LukasParke wants to merge 2 commits into
mattapperson:mainfrom
LukasParke:lukeparke/openrouter-eval-safety
Open

fix(eval): guard writes and bound case concurrency#72
LukasParke wants to merge 2 commits into
mattapperson:mainfrom
LukasParke:lukeparke/openrouter-eval-safety

Conversation

@LukasParke

Copy link
Copy Markdown
Collaborator

What

  • refuse optimizer write-back unless every target is tracked and clean in Git
  • add --force-dirty for deliberate dirty/untracked overrides while still requiring a repository
  • run the guard before coding-agent apply and skip stale literal locations in files the agent rewrites
  • run eval cases through an order-preserving bounded pool (default 4, configurable with --concurrency)
  • remove the accepted-but-ignored --budget/budget option

Why

Optimization write-back replaces source literals. Without a version-control guard it can destroy uncommitted work, including changes made by a structural coding-agent pass. Eval suites also ran cases serially even though each case owns an independent context.

The removed budget option had no downstream reader and therefore provided false safety. maxMetricCalls remains the actual GEPA evaluation bound.

This semantically ports only the eval safety/concurrency portion of fork/port/openrouter-fixes commit aedf712f. GEPA discovery, traversal, and judge reuse remain item 5 and are not included here.

Test plan

  • eval suite — 2,066 pass, 6 integration/benchmark skips in the root test command
  • focused write-guard/optimizer/concurrency tests — 29 pass
  • eval typecheck
  • root lint
  • sentrux check .
  • sentrux gate .
  • clean-context Pi review; fixed absolute/worktree path handling, no-repo override, API validation, and docs

Breaking behavior

  • --budget and the corresponding programmatic budget option are removed because they were silently ignored.
  • eval cases now run with concurrency 4 by default; use --concurrency 1 for suites sharing external mutable state.

Port the eval safety and case-concurrency portion of fork/port/openrouter-fixes commit aedf712 onto current main. The dead budget option is removed; GEPA discovery and traversal remain a separate follow-up.

Signed-off-by: Luke Parke <5702154+LukasParke@users.noreply.github.com>
@vercel

vercel Bot commented Aug 13, 2026

Copy link
Copy Markdown

@LukasParke is attempting to deploy a commit to the Matt Apperson's projects Team on Vercel.

A member of the Team first needs to authorize it.

@LukasParke

Copy link
Copy Markdown
Collaborator Author

Agent: CI reached the full core suite and failed only the unrelated timing-sensitive executeRunCode retry backoff test (fixed backoff or linear backoff depending on the run). The affected package tests/typechecks and local full suite passed. The separate compat failure is the repository workflow receiving an empty OPENROUTER_API_KEY, and Vercel requires upstream authorization. I cannot rerun upstream Actions without repository admin rights.

Run executeRunCode tests serially so their temporary global setTimeout proxy cannot capture delays from sibling tests under CI concurrency.

Signed-off-by: Luke Parke <5702154+LukasParke@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant