Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .claude-plugin/plugin.json
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"name": "agent-estimate",
"description": "Effort estimation for AI coding agents — PERT three-point estimation with METR reliability thresholds and wave planning",
"version": "0.7.5",
"version": "0.8.0",
"skills": ["./skills/estimate/claude"],
"author": {
"name": "Kiloloop"
Expand Down
2 changes: 1 addition & 1 deletion .github/ISSUE_TEMPLATE/bug_report.yml
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,7 @@ body:
id: version
attributes:
label: agent-estimate version
placeholder: "0.7.5"
placeholder: "0.8.0"
validations:
required: true

Expand Down
28 changes: 27 additions & 1 deletion .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -85,6 +85,22 @@ jobs:
fi
done
exit $status
- name: Workflows in the public export must only import modules it ships
run: |
cd exported
status=0
# A workflow naming an export-ignore'd module passes here and fails only
# after promotion, where the module is absent. Resolve every module an
# exported workflow imports against the export itself.
for mod in $(grep -rhoE 'import agent_estimate[A-Za-z0-9_.]*' .github/workflows/ \
| sed 's/^import //' | sort -u); do
rel=$(echo "$mod" | tr '.' '/')
if [ ! -e "src/${rel}.py" ] && [ ! -e "src/${rel}/__init__.py" ]; then
echo "::error::exported workflows import ${mod}, which the public export does not ship"
status=1
fi
done
exit $status
- name: Every test the public export ships must pass from inside it
run: |
cd exported
Expand All @@ -105,9 +121,19 @@ jobs:
run: |
# Click 8.2 exercises the compatibility boundary and the
# no_args_is_help fix that establishes the Typer floor.
pip install typer==0.16.1 click==8.2.0 pyyaml==6.0 pydantic==2.4.0 networkx==3.0 typing-extensions==4.6.1
pip install typer==0.16.1 click==8.2.0 pyyaml==6.0 pydantic==2.5.0 networkx==3.0 typing-extensions==4.6.1
pip install --no-deps -e .
- name: CLI constructs and answers at the floors
run: |
agent-estimate --help
agent-estimate --version
- name: Import and test the package at the floors
run: |
pip install 'pytest>=9.0.3,<10.0' 'tomli>=2.0'
# Import a module that SHIPS: tracing/ is export-ignore'd, so naming it
# here passes upstream and fails in the public repo, where it is absent.
# contract.schema exercises the same Pydantic floor (AwareDatetime,
# StringConstraints, AfterValidator); trace schemas keep their own test,
# which the pytest run below picks up wherever the module exists.
python -c "import agent_estimate.contract.schema"
python -m pytest tests/unit/ -x -q
7 changes: 6 additions & 1 deletion .github/workflows/publish.yml
Original file line number Diff line number Diff line change
Expand Up @@ -8,11 +8,12 @@ permissions: {}

jobs:
publish:
if: ${{ github.repository == 'kiloloop/agent-estimate' }}
runs-on: ubuntu-latest
environment: pypi
permissions:
id-token: write
contents: read
contents: write
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
Expand All @@ -21,3 +22,7 @@ jobs:
- run: pip install build==1.5.0
- run: python -m build
- uses: pypa/gh-action-pypi-publish@dc37677b2e1c63e2034f94d8a5b11f265b73ba33 # v1.14.2
- name: Attach distributions to the release
run: gh release upload "${{ github.event.release.tag_name }}" dist/* --clobber
env:
GH_TOKEN: ${{ github.token }}
1 change: 1 addition & 0 deletions .github/workflows/testpypi.yml
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ permissions:

jobs:
publish-testpypi:
if: ${{ github.repository == 'kiloloop/agent-estimate' }}
runs-on: ubuntu-latest
environment: testpypi
permissions:
Expand Down
31 changes: 30 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,11 +2,39 @@

All notable changes to this project will be documented in this file.

The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
Entries from v0.8 onward follow [Common Changelog](https://common-changelog.org/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [Unreleased]

## [0.8.0] - 2026-09-06

### Changed

- **Breaking:** Score matching work or wall-time actuals against expected values and reject cap-based scoring ([scoring rules](https://github.com/kiloloop/agent-estimate#v08-forecast-contract)).
- **Breaking:** Require an explicit expected-work attestation for legacy calibration ([calibration guidance](https://github.com/kiloloop/agent-estimate#v08-forecast-contract)).
- Require Pydantic 2.5 for trace-schema imports ([minimum dependencies](https://github.com/kiloloop/agent-estimate/blob/main/pyproject.toml)).
- Scale assigned work once through agent profile adjustments without changing task assignments ([profile adjustments](https://github.com/kiloloop/agent-estimate#agent-fleet)).
- Show applied profile factors and work minutes before and after adjustment ([adjustment reports](https://github.com/kiloloop/agent-estimate#agent-fleet)).

### Added

- Add typed records separating task facts, execution profiles, admission caps, forecasts, and outcomes ([forecast contract](https://github.com/kiloloop/agent-estimate#v08-forecast-contract)).
- Accept validated, versioned YAML requests through `estimate --spec` ([request example](https://github.com/kiloloop/agent-estimate/blob/main/examples/estimate-request.yaml)).
- Enforce required capabilities on the named spec executor ([agent selection](https://github.com/kiloloop/agent-estimate#v08-forecast-contract)).
- Identify expected wall time and its provenance independently of admission caps in reports ([duration forecasts](https://github.com/kiloloop/agent-estimate#v08-forecast-contract)).
- Accept caller-supplied token forecasts with explicit local-policy provenance ([token priors](https://github.com/kiloloop/agent-estimate/blob/main/docs/token-forecast-priors.md)).
- Document the bot-only comment update filter and PAT identity limitation ([comment identity](https://github.com/kiloloop/agent-estimate#permissions-and-comment-identity)).

### Removed

- **Breaking:** Reject `settings.review_overhead` with exit code 2 and review-mode migration guidance ([R1 migration](https://github.com/kiloloop/agent-estimate/blob/main/docs/migration-v0.8.md#r1-remove-settingsreview_overhead)).
- **Breaking:** Remove the JSON `estimated_cost` alias in favor of `heuristic_cost` ([R2 migration](https://github.com/kiloloop/agent-estimate/blob/main/docs/migration-v0.8.md#r2-replace-the-json-estimated_cost-alias)).

### Fixed

- Restrict PyPI and TestPyPI publishing to the public repository ([publishing workflow](https://github.com/kiloloop/agent-estimate/blob/main/.github/workflows/publish.yml)).

## [0.7.5] - 2026-08-26

### Added
Expand Down Expand Up @@ -152,6 +180,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- Modifier flags: `--warm-context`, `--spec-clarity`, `--issues`
- PyPI package: `pip install agent-estimate`

[0.8.0]: https://github.com/kiloloop/agent-estimate/releases/tag/v0.8.0
[0.7.5]: https://github.com/kiloloop/agent-estimate/releases/tag/v0.7.5
[0.7.4]: https://github.com/kiloloop/agent-estimate/releases/tag/v0.7.4
[0.7.3]: https://github.com/kiloloop/agent-estimate/releases/tag/v0.7.3
Expand Down
119 changes: 114 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,85 @@ agent-estimate estimate --repo myorg/myrepo --issues 11,12,14
agent-estimate session --agents 3 --rounds 2 --type review
```

### v0.8 forecast contract

The v0.8 source introduces a versioned request for callers that already know the
task and executor. These features are not in the published v0.7.5 package; use an
installation from this checkout until v0.8 is released. Bare task descriptions,
text files, and GitHub issue input remain available.

| Contract | Owns |
|----------|------|
| `TaskSpec` | Task facts, required capabilities, dependencies, and independent scope estimates |
| `ExecutionProfile` | Runtime and model identity, configuration provenance, context, modifiers, and review plan |
| `AdmissionEnvelope` | Declared minute/file caps and optional replayable cap calculations |
| `ForecastRecord` | Expected wall minutes, expected files, additive review minutes, and forecast provenance |

The [typed contracts](src/agent_estimate/contract/schema.py) are frozen, reject
unknown fields and non-finite numbers, and require explicit schema versions.
Model identity must be either a supplied identifier or an `unknown_reason`;
selecting an agent does not establish its serving model. IDs are caller supplied.
The library also exposes `OutcomeObservation` with nullable future actuals slots;
it does not ingest actuals or persist binding receipts.

From the repository root, with the matching package installed, run the complete
checked-in [request example](examples/estimate-request.yaml):

```bash
agent-estimate estimate --spec examples/estimate-request.yaml --format json
```

The JSON includes `"schema_version": "agent-estimate/report/v1"`, assigns
`Add input validation` to `Codex`, and reports `forecast.basis: expected-wall`.
The example's 90-minute admission cap does not determine its expected minutes.
`--spec` emits the versioned report; library callers can construct the separate
`ForecastRecord` with `forecast_from_report` in `agent_estimate.contract.duration`.

`--spec` accepts one full `EstimateRequest` containing `task_spec`,
`execution_profile`, and `admission`, plus an optional `token_prior`. It can be
combined with `--config`, `--format`, `--compact`, and `--title`. Put task facts,
scope, and modifiers in the request; alternate input sources and history flags
cannot be combined with `--spec`. Conflicts and invalid fields exit 2. It does
not read ambient `data.json` history.
The named agent must exist in the selected fleet and satisfy every required
capability. `config_profile` records provenance; `--config` selects the actual
fleet file. This single-task CLI path requires empty `dependency_task_ids` and
`execution_profile.estimate_multiplier: 1.0`; use the configured agent's
`estimate_multiplier` for [profile adjustments](#agent-fleet).

Context identity belongs to `execution_profile.context.context_key`. An explicit
`modifiers.warm_context` takes precedence: pairing it with
`context.implicit_co_dispatch: true` is rejected to prevent double counting.
Review intent is additive: supported plans are no review, one or two standard or
complex rounds, or three standard rounds. Unsupported plans exit 2.

**Expected values and caps.** Reports label duration `expected-wall`, with
`source` and `as_of` provenance (an unknown date stays null). Expected wall minutes
include work and additive review. Score matching actual wall minutes against
independent expected minutes, never `declared_cap_minutes` or a cap divided by
guessed headroom. File expectations likewise come from task scope, not file caps.
The `validate` command uses `actual_total_minutes` for `expected-wall` observations
and `actual_work_minutes` for `expected-work`; cap-only and cap-derived inputs are
rejected. Wall scoring is report-only in the current calibration store. Legacy
`estimated_minutes` means expected work only, and `calibrate --basis expected-work`
is an explicit attestation that every stored estimate is expected work rather
than a cap; inspect the rows before using it. Without that attestation,
`calibrate` exits 2 with guidance on stderr.

**Token honesty.** Typed forecasts default to `tokens.basis: unavailable`, with
null `expected_tokens_total` and `expected_tokens_output`. Total means processed
tokens including cache carry; output is a separate count included in total.
A caller-supplied prior uses `basis: local-policy`, a source, date, population,
and a mandatory population mismatch warning. An absent count stays unavailable;
zero is a supplied count. These are not calibrated forecasts. JSON includes
`forecast.tokens` only when a prior is supplied; Markdown then shows both slots
and their provenance. There are **no packaged token priors or numeric rates**.
See the [explicitly uncalibrated rate-shape example](docs/token-forecast-priors.md#rate-shape-example-only--not-calibrated)
for caller-owned policy inputs.

Upgrading configuration or JSON consumers? Read the
[v0.8 migration notes](docs/migration-v0.8.md) for the two removed surfaces.

## How It Works

agent-estimate produces three-point [PERT](https://en.wikipedia.org/wiki/Program_evaluation_and_review_technique) estimates from agent-work priors, not human-duration estimates:
Expand Down Expand Up @@ -156,6 +235,13 @@ Available on the [GitHub Marketplace](https://github.com/marketplace/actions/age

The report goes wherever `output-mode` points: the job summary (`summary`, the default), a PR comment (`pr-comment`), an issue comment (`issue-comment`), or a step output for downstream steps (`step-output`) — combinable with `+` (e.g. `summary+pr-comment`).

The Action accepts GitHub issue input; it has no `spec` or token-prior input.
The [forecast contract](#v08-forecast-contract) describes the CLI's
`--spec` path, expected-versus-cap scoring, and token provenance. The Action's
`expected-minutes` output is expected wall time, not an admission cap.

#### Permissions and comment identity

Grant only the permissions required by the selected output modes:

| Output mode | Required `permissions:` |
Expand All @@ -167,6 +253,14 @@ Grant only the permissions required by the selected output modes:

Add `contents: read` only when the calling workflow uses `actions/checkout`; the Action itself does not require a checkout. Combined modes need the union of their rows.

Use the default `${{ github.token }}` or a GitHub App installation token for
comment updates. The upsert filter selects only bot-authored comments with the
Action's marker. A personal access token (PAT) posts as its human owner, so its
comments are never selected for later updates. Repeated PAT runs create new
comments when there is no matching bot-authored comment to update. Existing
PAT-authored comments are never adopted, even by a later run using a bot token.
Granting write permissions does not change the comment author's identity.

By default, the Action installs `agent-estimate` from its own checked-out
`GITHUB_ACTION_PATH`, so the Python implementation stays coupled to the
`uses:` ref. Set `version` only when you deliberately want a published package
Expand Down Expand Up @@ -270,8 +364,8 @@ jobs:

The full JSON report is available as `steps.estimate.outputs.report` for custom processing.
Its footer records `engine_version` and `registry_version`. Agent-load rows expose
the five-minute-turn estimate as `heuristic_cost`; `estimated_cost` remains as a
compatibility alias until the v0.8 report schema.
the five-minute-turn estimate as `heuristic_cost`; consumers of the removed
`estimated_cost` alias must use `heuristic_cost`.

</details>

Expand Down Expand Up @@ -337,15 +431,30 @@ agents:
parallelism: 3
cost_per_turn: 0.08
model_tier: production
estimate_multiplier: 1.0
settings:
friction_multiplier: 1.15
inter_wave_overhead: 0.25
metr_fallback_threshold: 45.0
```

Legacy configs that set `settings.review_overhead` emit a deprecation warning;
the field is optional and ignored, and it will be removed in v0.8. Remove it and
select additive review overhead with `--review-mode` instead.
Configs containing `settings.review_overhead` now exit 2 before model validation,
even if its value is zero or null. Delete the key and select additive review
overhead with `--review-mode` (or the request's review plan with `--spec`).

Each agent's optional `estimate_multiplier` is a finite positive number, default
`1.0`. The pipeline assigns tasks first, calls the assigned profile's
`adjust_estimate` once per task, then scales work and its PERT range by the resulting
factor. It updates wave timing, reliability warnings and heuristic cost without
reassigning tasks or scaling review overhead, inter-wave gaps, or human-equivalent
work. Structural profile plugins retain their own hook and replace same-name YAML
profiles; their result must be finite positive work (zero only for zero work).
When an adjustment changes work, reports show the applied factor and the work
minutes before and after adjustment.

With `--spec`, `task_spec.required_capabilities` must all be available on the named
configured agent. An unsatisfiable requirement exits 2 instead of selecting another
agent. Library callers can continue setting `TaskNode.required_capabilities` directly.

```bash
agent-estimate estimate "Ship packaging flow" --config ./my_agents.yaml
Expand Down
19 changes: 16 additions & 3 deletions action.yml
Original file line number Diff line number Diff line change
@@ -1,5 +1,18 @@
name: 'agent-estimate'
description: 'Run AI agent effort estimation in CI — PERT + METR + wave planning'
description: |-
Run AI agent effort estimation in CI — PERT + METR + wave planning.

Forecast contract: TaskSpec holds task facts; ExecutionProfile describes execution;
AdmissionEnvelope carries caps; ForecastRecord records expected wall forecasts.
This Action accepts GitHub issue input, with no spec or token-prior input.
The CLI accepts versioned requests: from the repository root with the matching
package installed, run agent-estimate estimate --spec examples/estimate-request.yaml --format json.
Contract, expected-versus-cap scoring, token honesty labels and caller-only priors:
https://github.com/kiloloop/agent-estimate#v08-forecast-contract

Comment updates match bot-authored markers. Use github.token or a GitHub App
installation token; PAT-authored comments are never adopted. Repeated PAT runs
create new comments when no matching bot-authored comment exists.
author: 'Kiloloop'

branding:
Expand Down Expand Up @@ -64,10 +77,10 @@ inputs:
required: false
default: '3.12'
version:
description: 'agent-estimate version to install (e.g. "0.7.5"). Omit to install from this Action ref.'
description: 'agent-estimate version to install (e.g. "0.8.0"). Omit to install from this Action ref.'
required: false
token:
description: 'GitHub token for issue fetching and PR comments'
description: 'GitHub token for issue fetching and comment output. Use github.token or an App installation token for update-in-place comments; PAT-authored comments are never adopted, so repeated PAT runs create new comments when no matching bot comment exists.'
required: false
default: ${{ github.token }}

Expand Down
55 changes: 55 additions & 0 deletions docs/migration-v0.8.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
# Migrating to v0.8

These changes ship in v0.8.0. Update configuration and JSON consumers before
upgrading from v0.7.5.
The [forecast contract and runnable example](../README.md#v08-forecast-contract)
describe the new request format separately.

## R1: remove `settings.review_overhead`

v0.7.5 warned about this ignored configuration key. v0.8 rejects its presence
before model validation and exits with code 2, including when its value is `0`
or `null`. Delete the key from the fleet configuration:

```yaml
# Before
settings:
friction_multiplier: 1.15
review_overhead: 15
```

```yaml
# After
settings:
friction_multiplier: 1.15
```

Select additive review with `--review-mode none`, `standard`, `complex`, or
`3-round` on ordinary CLI estimates. In a `--spec` request, set
`execution_profile.review` instead; `--review-mode` cannot be combined with
`--spec`. For example, one standard round adds 15 minutes:

```yaml
# Under execution_profile in a complete --spec request
review:
mode: single_round
expected_rounds: 1
intensity: standard
```

This removes only the configuration key. The report field
`review_overhead_minutes` and additive review modeling remain available.

## R2: replace the JSON `estimated_cost` alias

v0.7.5 emitted both `estimated_cost` and `heuristic_cost` on each JSON
`agent_load` row. v0.8 emits only `heuristic_cost`. Update consumers, including
those reading the Action's JSON `report` output:

```text
Before: report["agent_load"][0]["estimated_cost"]
After: report["agent_load"][0]["heuristic_cost"]
```

The value still uses the five-minute-turn cost approximation. It is not a
token-metered charge, and this removal does not introduce a token pricing model.
Loading
Loading