Skip to content

Add external benchmark validation#10

Merged
sparklkt merged 1 commit into
mainfrom
v1.1-external-benchmark-validation
May 13, 2026
Merged

Add external benchmark validation#10
sparklkt merged 1 commit into
mainfrom
v1.1-external-benchmark-validation

Conversation

@sparklkt

Copy link
Copy Markdown
Owner

Summary

  • add external-style benchmark adapters and local benchmark task packs for ARC-style, reasoning, safety, long-horizon, and distraction-robustness validation
  • add benchmark runner unification across innerbrain_factor, cot, reflection, multi_agent_debate, and risk_rule
  • add benchmark-only ablation toggles for value_field, collision_lanes, attention_sovereignty, and growth_memory
  • add richer metrics and typed failure logging for robustness, consistency, long-horizon stability, distraction resistance, safety correctness, over-gating, false positives, false negatives, value collapse, attention hijack, and unstable decisions
  • add CLI commands for external-benchmark and external-benchmark-report, plus tests and light documentation updates

Testing

  • pytest -> 40 passed in 0.52s
  • innerbrain external-benchmark --benchmark-dir evals/external_benchmarks

External Benchmark Snapshot

  • innerbrain_factor | 1.00
  • innerbrain_factor.growth_memory | 1.00
  • innerbrain_factor.value_field | 0.98
  • innerbrain_factor.collision_lanes | 0.91
  • innerbrain_factor.attention_sovereignty | 0.84
  • multi_agent_debate | 0.78
  • reflection | 0.76
  • risk_rule | 0.75
  • cot | 0.70

Ablation Reading

  • attention_sovereignty shows the largest drop on this pack, especially on distraction robustness
  • collision_lanes materially reduce robustness when disabled
  • value_field still matters, but the current pack shows a smaller delta than attention and collision structure
  • growth_memory is neutral on this benchmark pack right now, which is useful feedback about where memory is and is not yet pulling weight

Boundary Check

  • remains offline and rule-based
  • no LLM integration
  • no network reasoning in the pipeline
  • no real external actions
  • human judgment remains required for high-risk boundaries

@sparklkt
sparklkt merged commit 643dc90 into main May 13, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant