Skip to content

fix(core): make evaluation errors visible in results and reports#7

Merged
dpaluy merged 1 commit into
masterfrom
fix/issue-4-evaluation-error-visibility
Jul 11, 2026
Merged

fix(core): make evaluation errors visible in results and reports#7
dpaluy merged 1 commit into
masterfrom
fix/issue-4-evaluation-error-visibility

Conversation

@dpaluy

@dpaluy dpaluy commented Jul 11, 2026

Copy link
Copy Markdown
Owner

Closes #4

Problem

Evaluator#call rescues judge errors into nil scores, and downstream everything silently drops them: Result#overall compacts nils, Result#pass? can return true on partial data, Report#metric_stats/summary hide errored metrics, and Report#worst sorts nil-overall results as if they were the best. A judge outage that kills 5 of 6 metrics still shows green in CI.

Changes

  • Result#errors returns per-metric error messages (derived from details entries with an :error key); Result#valid? is true when there are none
  • Result#pass? now fails closed: returns false when any metric errored, regardless of the surviving metrics' scores
  • Result#to_h includes errors
  • Report#summary appends an error line when errors occurred, e.g. Errors: 3 metric errors across 2 samples
  • JSON export (serializable_hash) includes per-metric error counts under errors
  • Report#worst now ranks results with no valid scores as worst instead of best
  • Version bumped to 0.3.0 with CHANGELOG entry (pass? and worst are behavior changes)

overall intentionally stays as the mean of valid scores, a partial score is still informative; only the gate requires completeness.

Verification

  • bundle exec rake test: 136 runs, 301 assertions, 0 failures (9 new regression tests: all-nil scores, partial failure with high overall, error-free results, non-Hash details, report summary/worst/JSON with errors)
  • bundle exec rubocop: no offenses
  • End-to-end repro through Evaluator with a metric raising JudgeError: {failing: nil, passing: 1.0} now yields pass? == false, appears in Report#failures, and the summary shows the error count line

Implementation delegated to Codex, reviewed and verified with Claude Code.

CodeFactory builder

Result#pass? now fails closed when any metric errored, Result exposes
errors/valid?, report summaries and JSON exports include metric error
counts, and Report#worst ranks results with no valid scores as worst.

Closes #4

CodeFactory builder
@dpaluy
dpaluy merged commit 477b2ff into master Jul 11, 2026
2 checks passed
@dpaluy
dpaluy deleted the fix/issue-4-evaluation-error-visibility branch July 11, 2026 21:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Make evaluation errors visible in results and reports

1 participant