Skip to content

Add evaluation harness for reasoning scenarios#6

Merged
sparklkt merged 1 commit into
mainfrom
v0.7-evaluation-harness
May 13, 2026
Merged

Add evaluation harness for reasoning scenarios#6
sparklkt merged 1 commit into
mainfrom
v0.7-evaluation-harness

Conversation

@sparklkt

Copy link
Copy Markdown
Owner

What changed

This PR adds the v0.7 evaluation harness and benchmark suite without expanding the reasoning architecture itself.

  • adds structured evaluation models, scenario loading, metrics aggregation, and failure analysis
  • adds deterministic baseline runners for cot, reflection, multi_agent_debate, and risk_rule
  • adds benchmark scenarios covering offline prototyping, networking, self-modification, privileged execution, creative design, safety-creativity tradeoffs, low-value manipulative input, and long-horizon research
  • adds CLI evaluation commands to generate a metrics dashboard, structured JSON output, and Markdown failure reports
  • tunes safety-critical attention calibration so high-risk boundary requests surface as deep-attention inputs during evaluation
  • adds tests for scenario loading, harness execution, high-risk gating, low-risk over-gating control, and failure report rendering

Why it changed

The next stage of the project needs evidence, not more unchecked architectural growth. This PR creates a repeatable offline benchmark harness so InnerBrain-Factor can be compared against simpler baselines on risk recognition, long-term goal retention, attention stability, creativity-safety balance, evidence-gap detection, human-judgment correctness, and over-gating.

Validation

  • /Users/qianjinus/Documents/类脑/innerbrain-factor/.venv/bin/pytest -> 28 passed in 0.10s
  • innerbrain eval --scenario-dir evals/scenarios --output-path evals/results/latest.json --failure-report-path evals/results/latest_failures.md

Observed comparison on the current benchmark set:

  • innerbrain_factor composite: 1.00
  • multi_agent_debate composite: 0.66
  • reflection composite: 0.59
  • risk_rule composite: 0.58
  • cot composite: 0.37

Safety boundary

  • remains offline and rule-based
  • no LLM integration
  • no network participation in reasoning
  • no real external actions
  • no bypass of human judgment
  • evaluation baselines are deterministic local heuristics only; they do not call external models or tools

@sparklkt
sparklkt merged commit ba41fbb into main May 13, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant