Skip to content

Add anti-self-bias evaluation suite#11

Closed
sparklkt wants to merge 9 commits into
mainfrom
v1.2-anti-self-bias-evaluation-suite
Closed

Add anti-self-bias evaluation suite#11
sparklkt wants to merge 9 commits into
mainfrom
v1.2-anti-self-bias-evaluation-suite

Conversation

@sparklkt

Copy link
Copy Markdown
Owner

What changed

This PR adds the v1.2 anti-self-bias evaluation suite as an independent black-box benchmark path for innerbrain.

Key additions:

  • evals/anti_self_bias_dataset.json with 35 base tasks across 7 categories:
    • raw reasoning
    • adversarial instruction
    • distractor-rich
    • long-horizon planning
    • hidden rule
    • cross-domain transfer
    • calibration and uncertainty
  • automatic clean, misleading, and overloaded variants for every task
  • external baseline runners in innerbrain/evals/baseline_runner.py
    • simple_prompt
    • cot_prompt
    • reflection_prompt
    • debate_prompt
    • rule_heuristic_solver
  • independent evaluator in innerbrain/evals/evaluator.py
  • anti-bias orchestration and rendering in innerbrain/evals/anti_self_bias_suite.py
  • CLI commands:
    • innerbrain anti-bias run
    • innerbrain anti-bias report
    • innerbrain anti-bias compare
  • extended anti-bias metrics in innerbrain/evals/metrics.py, including ASBI
  • explicit external failure typing for:
    • false positive
    • false negative
    • value collapse
    • attention hijack
    • unstable decision

Why it changed

The goal is to test whether InnerBrain-Factor is improving on independent tasks, or only improving on benchmarks that already align with its own internal structure.

This suite keeps scoring external to the core pipeline:

  • benchmark input is plain text
  • benchmark output is a plain text answer
  • scoring is done by an external evaluator
  • baselines do not reuse InnerBrain scoring logic

Validation

  • pytest -> 58 passed in 0.59s
  • innerbrain anti-bias run --dataset-path evals/anti_self_bias_dataset.json --output-path /tmp/anti_bias_full.json --report-path /tmp/anti_bias_full.md --internal-benchmark-reference 1.0

Bundled anti-bias snapshot from the local deterministic run:

  • rule_heuristic_solver external performance: 0.97
  • reflection_prompt external performance: 0.96
  • cot_prompt external performance: 0.96
  • debate_prompt external performance: 0.79
  • innerbrain_black_box external performance: 0.64
  • simple_prompt external performance: 0.35
  • ASBI: 0.27

Safety / scope

This PR remains within the project boundaries:

  • offline
  • deterministic and rule-based
  • no LLM
  • no network reasoning
  • no real external actions
  • no bypass of human judgment boundaries

Copy link
Copy Markdown
Owner Author

Superseded by #12. This draft branch was opened from an older linear development branch and pulled in previously merged history, so the diff was much larger than the actual v1.2 scope. Please review #12 instead for the clean main-based v1.2 change set.

@sparklkt sparklkt closed this May 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant