Tutorsim

Tutorsim is a benchmark for measuring how well language models tutor. It runs a model-under-test against a simulated student on frozen tutoring moments (expert teacher-annotated key moments cut from real tutoring transcripts), then scores the generated continuation.

The implemented metrics are:

• Appropriate Scaffolding: How often does the LM tutor introduce scaffolds that make content more accessible in moments when expert teachers think that scaffolding should occur, without over-scaffolding
• Appropriate Rigor: How often does the LM tutor push for rigor by increasing the cognitive demands of the task in moments when expert teachers think that pushes for rigor should occur (without over-scaffolding)
• Avoids Over-Scaffolding: How often does the LM avoid introducing more supports than needed given the student's state?

Installation

Tutorsim requires Python 3.11 or newer. Install from a checkout:

python -m pip install --upgrade pip
python -m pip install -e .

This installs the tutorsim command and the tutorsim Python package — the benchmark runtime. Everything a run depends on ships inside the package: src/tutorsim/default_config.yaml and the tutor/student/scorer prompts under src/tutorsim/prompts/.

Running

TODO: Update the dataset path Point the runner at the released Hugging Face dataset and provide an API key to run the benchmark.

export ANTHROPIC_API_KEY=...   # or the provider's key

tutorsim run \
  --tutors claude-opus-4-8 \
  --modes plain scaffolding_rigor \
  --sample 10 \
  --dataset <org>/tutorsim-transcripts-preview

Useful run options:

Option	Purpose
--tutors MODEL_ID ...	Tutor model IDs or registered custom tutor names to evaluate.
--modes MODE ...	Tutor prompt modes. Defaults to plain scaffolding_rigor.
--dataset HF_ID	Hugging Face dataset id (default: dataset.id from config).
--data_path DIR	Run from a local data dir; wins over --dataset.
--dataset-revision REV	Pin a HF dataset revision for reproducibility.
--sample N	Use the first N moments from the dataset.
--trials N	Run each tutor/mode cell multiple times and summarize mean/spread.
--max-turns N	Maximum generated tutor/student turns per conversation.
--log-level LEVEL	Log verbosity: DEBUG, INFO (default), WARNING, ERROR.
--log-file FILE	Optional combined log for the whole invocation (per-run logs are always written).

Supported tutor prompt modes are:

Mode	Description
plain	Baseline tutor prompt.
scaffolding_rigor	Tutor prompt that explicitly targets scaffolding and rigor.
oracle	Reference-aware tutor prompt that requires a moment's reference transcript.

Each run writes under results/. Re-running skips moments whose transcript and score are already present. Partial failures are recorded in summary.json; a run where zero moments complete raises an error and does not produce a leaderboard-ready empty summary.

Run output layout:

results/<run_id>/
  config.json
  run.log
  transcripts/<scenario_id>.json
  scores/<scenario_id>.json
  summary.json

Each run's config.json records the dataset source (id, revision or local path, config name, record count, and the record-level content hash), the resolved config, package version, git commit, config hash, prompt hashes, and the dataset manifest when running from a local release dir. A run is reproducible from: pinned dataset revision + content hash and the resolved config (up to LLM sampling nondeterminism).

Logging

Both CLIs (tutorsim and tutorsim-build) log progress to the console at INFO level, and every run automatically writes its own log file — nothing to enable:

• tutorsim run writes results/<run_id>/run.log per cell, next to config.json and summary.json. The run directory is the complete record of the run: what ran, what was resumed, and which moments failed.
• tutorsim-build dataset build / build-from-run / build-ground-truth write a build.log into their output directory (skipped on --dry-run).

Log files append, so a resumed run continues the same log. In a multi-tutor sweep, parallel lanes each write only to their own run's log, and every console line is tagged with its cell (e.g. [gpt-5-4/plain]) so interleaved lanes stay readable. Logs narrate the two phases of each run: Replay (generating tutor/student continuations) and Classification (the 3-pass batch scorer).

Two optional knobs: --log-level DEBUG|INFO|WARNING|ERROR controls verbosity, and --log-file FILE additionally writes one combined log for the whole invocation — useful for watching a sweep across lanes, or for the subcommands without an output directory (report, view, validate). TUTORSIM_LOG_LEVEL / TUTORSIM_LOG_FILE environment variables set defaults for the flags.

Running new tutor models

No code changes are needed for hosted models: add a roster entry in your config and export the provider API key. The provider is inferred from the model id (claude-* → anthropic, gpt-* → openai, gemini-* → gemini, deepseek-ai/* → together).

models:
  my-new-model-id: { thinking: true, effort: high }

tutorsim run --tutors my-new-model-id --data_path <release dir>

Tutors are selectable per-run via the CLI. The Student model is set in the config; changing the student model would change the shape of the evaluation. Modify your config to swap the student model with another API-callable model:

student: { model: claude-opus-4-6, mode: oracle, thinking: false } # default

Custom Tutor / Student Models

You can register a different approach to the LM Tutor / Synthetic student, for example a synthetic student without student-traits.

from tutorsim import register_tutor, register_student

@register_tutor("my-tutor")
def my_tutor(conversation):
    return "next tutor turn"

@register_student("my-student")
def my_student(conversation):
    return "next student turn"

For CLI use, import the registration module before calling tutorsim.cli.main() from a small wrapper script. There is not yet a plugin discovery mechanism.

Example wrapper:

from tutorsim import register_tutor, register_student
from tutorsim.cli import main


@register_tutor("my-tutor")
def my_tutor(conversation):
    return "next tutor turn"


@register_student("my-student")
def my_student(conversation):
    return "next student turn"


if __name__ == "__main__":
    main()

Then run the wrapper with normal CLI arguments, for example:

python my_tutorsim_wrapper.py run --tutors my-tutor --sample 1 --data_path <release dir>

The dataset

The released dataset carries two representations of the same underlying data:

• Source configs for data consumers — transcripts (full sessions), annotations (human SAR annotations), ground_truth (labeled key moments).
• moments for eval runners — the frozen, self-contained runnable set (520 moments: 260 scaffolding / 260 rigor). Each record carries the pre-cut transcript context, gold dimension, the real human continuation (for the oracle student), the frozen student trait (the exact persona the paper's simulated student ran with — generated from the pre-cut prefix only, so every benchmark run everywhere evaluates against the same students), and full provenance back to the source transcripts. It is built once by maintainers and published; the runtime never constructs or regenerates any of it.

Validate a local release directory:

tutorsim-build dataset validate --data_path data/balanced_520_release

The release ships moments.jsonl with moments.manifest.json and moments.schema.json beside it (flat, like the other release files — so a full release download is itself a valid --data_path dir). The manifest records the set name, version, schema version, record count, a record-level content_hash (comparable across the HF and local load paths), a file-level file_sha256, creation date, and provenance.

Reports And Viewers

Aggregate completed run summaries into leaderboard files:

tutorsim report --results-root results --out leaderboard

This writes leaderboard.md and leaderboard.csv.

Build a self-contained HTML viewer:

tutorsim view --results-root results --out viewer.html

The report and viewer commands read summary.json files from run directories.

Scoring

Tutorsim scores generated continuations with model-backed annotation passes and then aggregates per-moment scores into leaderboard metrics:

Metric	summary.json source	Interpretation
Appropriate Scaffolding	scaffold_calibrated.score	Of scaffolding-appropriate moments, fraction where the tutor scaffolded without over-scaffolding.
Appropriate Rigor	rigor_calibrated.score	Of rigor-appropriate moments, fraction where the tutor pushed for rigor without over-scaffolding.
Avoids Over-Scaffolding	1 - overscaffold.rate	Fraction of all moments free of over-scaffolding. Computed at report time.

These are the three metrics the paper reports; all are higher-is-better and appear in the leaderboard as appropriate_scaffolding, appropriate_rigor, and avoids_overscaffold.

Analysis

Paper notebooks, plots, and the action-taxonomy classifier live under analysis/ — interpretation layers over results, not part of the runtime. Install the taxonomy extra (pandas/matplotlib), then:

python -m analysis.taxonomy classify --kind key_moments --input key_moments.jsonl --output ./human
python -m analysis.taxonomy classify --kind tutorsim --input results/<run_id>/ \
  --scenarios data/balanced_520_release/moments.jsonl --output ./lm
python -m analysis.taxonomy headline --human ./human/classified.csv --lm ./lm/classified.csv --output ./headline

python -m analysis.taxonomy run chains all three.

Configuration

Tutorsim loads config in this order:

1. tutorsim run --config path/to/config.yaml
2. TUTORSIM_CONFIG
3. ./config.yaml
4. packaged src/tutorsim/default_config.yaml

Use configs/local.example.yaml as a starting point for local overrides.

Config files have these top-level sections:

Key	Purpose
dataset	Released benchmark source: id (HF dataset), revision (pin), config (which config holds the runnable moments; default moments).
providers	Maps provider names to the environment variable that contains the API key.
models	Model roster for hosted tutor models. The provider is inferred from the model ID.
student	Hosted model and mode used for the simulated student.
scorer	Hosted model used for the scoring passes.
defaults	Default trials and max_turns values for tutorsim run.
retry	Retry behavior for direct model calls.
batch	Batch API polling timeout for scorer/provider paths that use batch APIs.

Repository Layout

.
├── pyproject.toml
├── configs/
│   └── local.example.yaml
├── src/tutorsim/            installable benchmark runtime
├── tutorsim_build/           maintainer-only dataset construction + release tooling
├── analysis/                paper notebooks, plots, taxonomy
├── data/                    local datasets and release dirs, gitignored
├── results/                 run outputs, gitignored
└── tests/                   tutorsim/ (runtime), tutorsim_build/, analysis/

The core rule: code needed to run or score the benchmark lives in src/tutorsim/; code that creates the dataset lives in tutorsim_build/; code that explains or visualizes results lives in analysis/. Build and analysis code may import the runtime; the runtime never imports them.

Development

Maintainer extras (not needed to run the benchmark):

pip install -e ".[dev]"                    # + pytest (runtime tests)
pip install -e ".[build,build-dev]"        # + dataset construction tooling
pip install -e ".[taxonomy]"               # + pandas/matplotlib for analysis

Run from a local directory

tutorsim run \
  --tutors claude-opus-4-8 \
  --modes plain scaffolding_rigor \
  --sample 10 \
  --data_path data/balanced_520_release

Run the runtime test suite without real API calls:

pytest tests/tutorsim -q          # runtime only (needs [dev])
pytest tests -q                   # full suite (needs [dev,build-dev,taxonomy]; missing extras skip)

Dataset construction (maintainers)

Dataset construction lives in tutorsim_build/ — outside the runtime package, because changing it can change the benchmark itself. Install with the build extra and use the tutorsim-build CLI:

# Raw human annotations -> per-conversation ground-truth JSON (LLM batch pipeline)
tutorsim-build dataset build-ground-truth --input <annotations.jsonl> --labeller hybrid

# Ground truth + transcripts + id list -> release dir (moments.jsonl + manifest)
tutorsim-build dataset build \
  --set balanced_520 \
  --ids tutorsim_build/balanced_520_ids.json \
  --ground-truth <ground-truth dir> \
  --transcripts <transcripts dir> \
  --tutoring-provider-a-jsonl <transcripts.jsonl> \
  --out data/balanced_520_release \
  --created 2026-07-01

# Rebuild the paper's exact 520 from the published reference run
# (the canonical record of the benchmark-time detections)
tutorsim-build dataset build-from-run \
  --set balanced_520 \
  --reference-run <benchmark_520_full_run.jsonl> \
  --tutoring-provider-a-jsonl <transcripts.jsonl> \
  --ids tutorsim_build/balanced_520_ids.json \
  --out data/balanced_520_release \
  --created 2026-07-01

tutorsim_build/balanced_520_ids.json is the canonical, frozen selection of the paper's 520 moments (committed; deidentified UUID surrogates only). The selection is not derivable — the sampler that chose it predates this repo — so the id list plus the published reference run are the reproduction path.