fix(eval): guard writes and bound case concurrency - #72
Open
LukasParke wants to merge 2 commits into
Open
Conversation
Port the eval safety and case-concurrency portion of fork/port/openrouter-fixes commit aedf712 onto current main. The dead budget option is removed; GEPA discovery and traversal remain a separate follow-up. Signed-off-by: Luke Parke <5702154+LukasParke@users.noreply.github.com>
|
@LukasParke is attempting to deploy a commit to the Matt Apperson's projects Team on Vercel. A member of the Team first needs to authorize it. |
Collaborator
Author
|
Agent: CI reached the full core suite and failed only the unrelated timing-sensitive |
4 tasks
Run executeRunCode tests serially so their temporary global setTimeout proxy cannot capture delays from sibling tests under CI concurrency. Signed-off-by: Luke Parke <5702154+LukasParke@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
--force-dirtyfor deliberate dirty/untracked overrides while still requiring a repository--concurrency)--budget/budgetoptionWhy
Optimization write-back replaces source literals. Without a version-control guard it can destroy uncommitted work, including changes made by a structural coding-agent pass. Eval suites also ran cases serially even though each case owns an independent context.
The removed budget option had no downstream reader and therefore provided false safety.
maxMetricCallsremains the actual GEPA evaluation bound.This semantically ports only the eval safety/concurrency portion of
fork/port/openrouter-fixescommitaedf712f. GEPA discovery, traversal, and judge reuse remain item 5 and are not included here.Test plan
sentrux check .sentrux gate .Breaking behavior
--budgetand the corresponding programmaticbudgetoption are removed because they were silently ignored.--concurrency 1for suites sharing external mutable state.