cli.md

CLI Reference

Every command is available two ways:

agent-routing-eval-lab <subcommand> [flags]      # installed console script
python -m agent_routing_eval_lab.cli <subcommand> [flags]

Each subcommand also accepts -h/--help for inline usage.

Global flags

These apply to every subcommand and must be placed before the subcommand:

Flag	Description
--version	Print the package version and exit.
-v, --verbose	Show all diagnostics (full warning lists, debug info) on stderr.
-q, --quiet	Suppress warning diagnostics.

Diagnostics (warnings, progress) go to stderr; result data goes to stdout, so JSON output stays machine-consumable when piped.

Exit codes

Code	Meaning
0	Success, or gate/validate passed.
1	A gate threshold was violated, or validate found problems. The run worked; the result is "no-go".
2	Usage or data error: bad input, missing file, unknown policy, malformed gate config.

Unexpected errors (genuine bugs) still surface a Python traceback.

generate-data

Generate a synthetic logged-decisions CSV.

Flag	Default	Description
--output	required	Destination CSV path.
--rows	300	Number of rows (must be a positive integer).
--seed	7	Random seed for deterministic output.

agent-routing-eval-lab generate-data --output examples/logged_decisions.sample.csv --rows 300 --seed 7

evaluate

Evaluate the built-in candidate policies against logged decisions.

Flag	Default	Description
--input	required	Logged-decisions CSV (see input-schema.md).
--format	text	text for the ASCII chart, json for machine-readable results (see json-schema.md).
--dump-decisions	off	Directory to write one <policy>_decisions.csv per policy for drill-down analysis.
--policies	built-in set	Directory of policy-candidate YAML files to evaluate instead of the built-in set. Requires the optional config extra (pip install -e .[config]).
--weights	built-in weights	JSON file overriding the composite-score weights (keys: success, correct_tool, safety, unresolved, cost, latency). See evaluation_methodology.md.

agent-routing-eval-lab evaluate --input examples/logged_decisions.sample.csv
agent-routing-eval-lab evaluate --input examples/logged_decisions.sample.csv --format json
agent-routing-eval-lab evaluate --input examples/logged_decisions.sample.csv --dump-decisions out/
agent-routing-eval-lab evaluate --input examples/logged_decisions.sample.csv --policies examples/policy_candidates/
agent-routing-eval-lab evaluate --input examples/logged_decisions.sample.csv --weights weights.json

The built-in candidate set (baseline, cost_aware, strict_policy, contextweaver_v1, contextweaver_v2) mirrors examples/policy_candidates/; a drift-guard test keeps them in sync.

report

Write the Markdown evaluation report (optionally also JSON).

Flag	Default	Description
--input	required	Logged-decisions CSV.
--output	required	Markdown report path (written atomically).
--json-output	off	Also write machine-readable JSON results to this path.
--policies	built-in set	Directory of policy-candidate YAML files to evaluate instead of the built-in set. Requires the optional config extra.
--weights	built-in weights	JSON file overriding the composite-score weights. See evaluation_methodology.md.

agent-routing-eval-lab report --input examples/logged_decisions.sample.csv --output reports/example_report.md

demo

Generate data, evaluate, and write a report end-to-end.

Flag	Default	Description
--output-dir	current directory	Where to write examples/ and reports/ artifacts. Files are written atomically and resolved paths are printed.

agent-routing-eval-lab demo
agent-routing-eval-lab demo --output-dir /tmp/demo-run

unsafe-demo

Run the ungoverned unsafe baseline agent over a fixed synthetic scenario set and show what breaks — full-catalog distraction, prompt-only "safety", raw tool output trusted as instruction, and unapproved sensitive writes. Emits evaluator-ready decision logs. See governed_path.md.

Flag	Default	Description
--output-dir	current directory	Where to write the examples/unsafe_baseline_decisions.sample.csv log.

agent-routing-eval-lab unsafe-demo
agent-routing-eval-lab unsafe-demo --output-dir /tmp/demo-run

governed-demo

Run the governed agent (bounded tool choices, context firewall, approval-aware action guard) over the same scenarios and print a before/after comparison against the unsafe baseline. Emits auditable governed decision logs and a comparison report. See governed_path.md.

Flag	Default	Description
--output-dir	current directory	Where to write examples/governed_path_decisions.sample.csv and reports/governed_comparison.md.

agent-routing-eval-lab governed-demo
agent-routing-eval-lab governed-demo --output-dir /tmp/demo-run

gate

Evaluate policies and exit non-zero when thresholds are violated — the CI pre-deployment gate.

Flag	Default	Description
--input	required	Logged-decisions CSV.
--max-unsafe-rate	off	Fail if any gated policy's unsafe-action rate exceeds this.
--min-success-rate	off	Fail if success rate is below this.
--max-low-support-share	off	Fail if the low-support share exceeds this.
--max-avg-cost	off	Fail if average cost exceeds this.
--policy-name	all	Gate only this policy instead of every evaluated one.
--config	off	Load thresholds from a committed JSON file (keys match the flags above plus policy_name).
--format	text	text or json violation output.

At least one threshold (via a flag or --config) is required: a gate with no thresholds can only ever pass, so it exits 2 (usage error) rather than reporting a misleading "PASSED".

agent-routing-eval-lab gate --input examples/logged_decisions.sample.csv --max-unsafe-rate 0.05 --min-success-rate 0.6

Use as a CI step:

- name: Routing gate
  run: agent-routing-eval-lab gate --input logs.csv --max-unsafe-rate 0.05

compare

Show request-level decision diffs between two policies (incumbent vs candidate).

Flag	Default	Description
--input	required	Logged-decisions CSV.
--policy-a	required	First policy name (the baseline/incumbent).
--policy-b	required	Second policy name (the candidate).
--limit	20	Maximum number of differing requests to show.
--only-regressions	off	Show only requests where policy B regressed.
--format	text	text or json.

Valid policy names: baseline, cost_aware, strict_policy, contextweaver_v1. An unknown name exits 2.

agent-routing-eval-lab compare --input examples/logged_decisions.sample.csv --policy-a baseline --policy-b strict_policy --only-regressions

validate

Check a bring-your-own logged-decisions CSV against the schema before evaluating.

Flag	Default	Description
--input	required	Logged-decisions CSV to validate.
--max-errors	all	Stop after reporting this many errors.

Exits 0 when the file is valid, 1 (with errors on stderr) otherwise. See input-schema.md for the column contract.

agent-routing-eval-lab validate --input my_logs.csv

lint

Static safety/leakage lint of a schema-valid logged-decisions CSV, without running the full replay. Distinct from validate (which checks the schema): lint checks the meaning of the rows.

Flag	Default	Description
--input	required	Logged-decisions CSV to lint.
--format	text	text or json findings output.
--ignore	none	Comma-separated lint codes to suppress (e.g. lint.oracle_unavailable).

Checks: lint.unavailable_chosen (error — a logged action that was not available), lint.oracle_unavailable (warning — no candidate can match the oracle), lint.bad_propensity (warning — a propensity outside (0, 1]), and lint.possible_oracle_leak (warning — the logging policy is never wrong). Exits 1 if any error-severity finding is present; warnings alone still pass.

agent-routing-eval-lab lint --input examples/logged_decisions.sample.csv
agent-routing-eval-lab lint --input my_logs.csv --ignore lint.oracle_unavailable