Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -442,14 +442,16 @@ jobs:
tests/test_upstream_comparison_generator.py
python -m agentrunproof check-upstream-bundle \
evidence/upstream-comparison/v1/bundle.json
python -m agentrunproof check-upstream-bundle \
evidence/upstream-comparison/v2/bundle.json
- name: Reproduce committed comparison evidence when present
id: reproduce
shell: bash
env:
INPUT_UPSTREAM_TARGET_COMMIT: ${{ inputs.upstream_target_commit }}
run: |
set -euo pipefail
committed_bundle="evidence/upstream-comparison/v2/bundle.json"
committed_bundle="evidence/upstream-comparison/v3/bundle.json"
if [[ ! -f "$committed_bundle" ]]; then
target_commit="$INPUT_UPSTREAM_TARGET_COMMIT"
if [[ -z "$target_commit" ]]; then
Expand Down Expand Up @@ -528,7 +530,7 @@ jobs:
)"
test "${#evidence_parent_fields[@]}" -eq 2
evidence_parent="${evidence_parent_fields[1]}"
git merge-base --is-ancestor "$source_commit" "$evidence_parent"
test "$evidence_parent" = "$source_commit"
test "$(git rev-parse "$source_commit^{tree}")" = "$source_tree"

python - "$committed_bundle" > "$RUNNER_TEMP/expected-upstream-evidence-files" <<'PY'
Expand Down
14 changes: 12 additions & 2 deletions .github/workflows/publish.yml
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ on:
tag:
description: Existing final GitHub Release tag to publish
required: true
default: v0.2.0
default: v0.3.0
type: string
publish:
description: Upload the verified distributions to PyPI
Expand Down Expand Up @@ -77,6 +77,12 @@ jobs:
evidence_checker=check-upstream-bundle
evidence_kind=upstream-comparison
;;
v0.3.0)
evidence_directory=evidence/upstream-comparison/v3
evidence_member=
evidence_checker=check-upstream-bundle
evidence_kind=upstream-comparison
;;
*)
echo "Unsupported release evidence profile: $RELEASE_TAG" >&2
exit 1
Expand Down Expand Up @@ -229,7 +235,7 @@ jobs:
test "${#evidence_commit_fields[@]}" -eq 2
evidence_parent="${evidence_commit_fields[1]}"
if [[ "$EVIDENCE_KIND" = upstream-comparison ]]; then
git merge-base --is-ancestor "$source_commit" "$evidence_parent"
test "$evidence_parent" = "$source_commit"
python - "$committed_bundle" > "$RUNNER_TEMP/expected-evidence-files" <<'PY'
import json
import sys
Expand Down Expand Up @@ -278,6 +284,10 @@ jobs:
test "$(git rev-parse HEAD)" = "$evidence_commit"
: > "$RUNNER_TEMP/expected-post-evidence-files"
;;
v0.3.0)
test "$(git rev-parse HEAD)" = "$evidence_commit"
: > "$RUNNER_TEMP/expected-post-evidence-files"
;;
esac
git diff --name-only "$evidence_commit..HEAD" | sort \
> "$RUNNER_TEMP/actual-post-evidence-files"
Expand Down
6 changes: 3 additions & 3 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,15 +2,15 @@

All notable changes to AgentRunProof will be documented here.

## 0.3.0 - Unreleased
## 0.3.0 - 2026-08-21

- Extend the declared dependency window to `openai-agents>=0.20.0,<0.23` and add exact 0.22.0
packaged-wheel coverage to the Python 3.10–3.14 CI matrix alongside 0.20.0 and 0.21.0.
- Treat SDK 0.22's output-guardrail replacement semantically: preserve call/output linkage and
require the rejected raw tool result to be absent without pinning the SDK's replacement text;
use scenario/case revision 2 while older history keeps revision 1.
- Preserve certificate v1 plus all existing historical evidence and lock files. Publication of
v0.3.0 remains blocked on a fresh clean-commit comparison bundle and complete release gates.
- Preserve certificate v1 plus all existing historical evidence and lock files, and require a
fresh v3 comparison bundle bound to the 0.3.0 source and release wheel.
- Add a provider-free real-`Runner` tool example, contribution and security guidance, structured
issue and pull-request templates, and a five-minute upstream `RunState` case study.
- Rework the project landing page around a 30-second PyPI check, supported SDK boundaries, honest
Expand Down
3 changes: 1 addition & 2 deletions CITATION.cff
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,7 @@ type: software
authors:
- family-names: Fu
given-names: Xiaonan
version: 0.2.0
date-released: 2026-08-15
version: 0.3.0
repository-code: "https://github.com/FU-max-boop/agentrunproof"
url: "https://github.com/FU-max-boop/agentrunproof"
license: MIT
Expand Down
11 changes: 9 additions & 2 deletions PLAN.md
Original file line number Diff line number Diff line change
Expand Up @@ -56,8 +56,9 @@ Deliver a standalone, installable tool that deterministically checks OpenAI Agen
- [x] (2026-08-21) Prepared the v0.3.0 source release candidate for SDK 0.22: widened metadata to
`<0.23`, added the exact 0.22.0 packaged-wheel cells, and made the output-guardrail history check
require linked redacted replay without matching replacement prose. One built wheel passed the
full suite locally on CPython 3.12 against exact SDK 0.20.0, 0.21.0, and 0.22.0. The full
Python 3.10–3.14 CI matrix and a fresh release-bound v3 evidence bundle remain publication gates.
full suite locally on CPython 3.12 against exact SDK 0.20.0, 0.21.0, and 0.22.0. The full hosted
Python 3.10–3.14 matrix passed on the release-candidate PR and main; a fresh release-bound v3
evidence bundle remains the publication gate.

## Milestones

Expand Down Expand Up @@ -158,6 +159,12 @@ The milestone is complete when the charter's Gate 4 is satisfied.
signed-off evidence and historical lock files remain immutable. A v0.3 release remains blocked
until fresh clean-commit evidence and all release gates exist; documentation and workflow
preparation must not imply that 0.3.0 has been published.
- **2026-08-21 — v0.3 evidence publication sequence:** Freeze all package inputs, final release
metadata, release-profile logic, and evidence-generation workflow in one clean source commit C.
Generate the versioned v3 comparison on canonical Linux from C, then add only the six JSON
evidence members in a direct child commit E. The v0.3.0 tag must point directly to E; v1 and v2
evidence remain immutable and continue to be checked. This preserves the two-commit provenance
design and prevents a release artifact from claiming evidence generated by a different source.

## Outcomes & Retrospective

Expand Down
7 changes: 3 additions & 4 deletions PROJECT_CHARTER.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,10 +25,9 @@ The first public contract covers the standard text `Runner` and these released p

For one logical scenario, AgentRunProof may execute several variants. A variant is a real SDK run, not a simulation of the runner. The harness controls only model responses, local test tools, and the test session.

The v0.3 release candidate declares the compatibility window `openai-agents>=0.20,<0.23` on
Python 3.10 through 3.14. Its acceptance matrix installs the packaged artifact against exactly SDK
0.20.0, 0.21.0, and 0.22.0; a Python/SDK cell is claimed as tested only after that exact CI cell
passes. The latest published release remains v0.2.0 until the v0.3 release gates complete.
The v0.3 source contract declares the compatibility window `openai-agents>=0.20,<0.23` on Python 3.10
through 3.14. Its acceptance matrix installs the packaged artifact against exactly SDK 0.20.0,
0.21.0, and 0.22.0; a Python/SDK cell is claimed as tested only after that exact CI cell passes.

v0.3 retains certificate v1, the v0.1 runtime scenarios, and v0.2's explicit
`DeterministicModel(..., emit_traces=True)` interoperability path: supported SDKs 0.21 and 0.22
Expand Down
28 changes: 13 additions & 15 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,11 +14,10 @@ across `run` and `run_streamed`, and writes content-addressed JSON records for s
tool-linkage, and `RunState` resume invariants. A failing record carries the normalized
counterexample observations.

The v0.3.0 release candidate in this source tree declares
`openai-agents>=0.20,<0.23` on Python 3.10–3.14. Its packaged-wheel CI matrix requires the exact
0.20.0, 0.21.0, and 0.22.0 release baselines. The latest published package remains v0.2.0, whose
declared window ends before SDK 0.22. Built-in scenarios make no model API call and require no API
key.
The 0.3.0 source contract declares `openai-agents>=0.20,<0.23` on Python 3.10–3.14. Its
packaged-wheel CI matrix requires the exact 0.20.0, 0.21.0, and 0.22.0 release baselines. Published
availability is shown by the PyPI badge and immutable GitHub Releases. Built-in scenarios make no
model API call and require no API key.

> AgentRunProof-backed reports are referenced by two merged maintainer fixes,
> [#4413](https://github.com/openai/openai-agents-python/pull/4413) and
Expand All @@ -36,7 +35,7 @@ or [CLI reference](https://github.com/FU-max-boop/agentrunproof/blob/main/docs/c
## 30-second local check

```bash
python -m pip install "agentrunproof==0.2.0"
python -m pip install agentrunproof
agentrunproof probe basic-tool-session-parity --certificate proof.json
agentrunproof check-certificate proof.json
```
Expand All @@ -63,17 +62,16 @@ and the

Maintaining a downstream library? The
[isolated, test-only CI guide](https://github.com/FU-max-boop/agentrunproof/blob/main/docs/ci-adoption.md)
provides a copyable real-`Runner` contract test and an ephemeral `uv` matrix for exact SDK 0.20.0
and 0.21.0 using the published v0.2.0 package, plus the pending v0.3.0 expansion to SDK 0.22.0. It
keeps AgentRunProof out of runtime metadata and the project lockfile. Provider-free means the
built-in deterministic model sends no model-provider request; it does not make arbitrary downstream
code or dependency installation network-isolated.

For artifact review, pin `agentrunproof==0.2.0` and use the immutable
[v0.2.0 release](https://github.com/FU-max-boop/agentrunproof/releases/tag/v0.2.0), whose
provides a copyable real-`Runner` contract test and an ephemeral `uv` matrix. Its published v0.2.0
example covers exact SDK 0.20.0 and 0.21.0; the 0.3.0 source contract adds SDK 0.22.0 and may be
pinned downstream only after its immutable release and PyPI pages exist. The isolated pattern keeps
AgentRunProof out of runtime metadata and the project lockfile.

For artifact review, pin an exact published version and use its matching
[immutable GitHub Release](https://github.com/FU-max-boop/agentrunproof/releases). Each
`SHA256SUMS` binds the wheel and sdist. The release workflow rebuilds and byte-compares those
artifacts, smoke-tests the wheel, and publishes the same verified files to PyPI through OIDC trusted
publishing. The CI guide records the exact hashes and the remaining third-party-code trust boundary.
publishing. The CI guide explains the remaining third-party-code trust boundary.

## Where it fits

Expand Down
23 changes: 11 additions & 12 deletions docs/ci-adoption.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,10 +39,9 @@ uv run --isolated --no-project --python 3.12 \
-- python -m pytest -q examples/ci_adoption/test_runner_contract.py
```

These commands intentionally remain on the published v0.2.0 contract. The 0.3.0 source candidate
adds exact SDK 0.22.0 coverage, but `agentrunproof==0.3.0` must not be used in a downstream PyPI job
until that release exists and publishes verified artifact hashes. From this repository checkout,
the candidate can be tested without changing project metadata:
The 0.3.0 source contract adds exact SDK 0.22.0 coverage. Pin `agentrunproof==0.3.0` downstream
only after its immutable release and PyPI pages exist. Before then, the source contract can be
tested without changing project metadata:

```bash
uv run --isolated --no-project --python 3.12 \
Expand All @@ -62,7 +61,7 @@ the adapter, name the applicable extra, for example `--with-editable ".[openai]"

This job is deliberately separate from the downstream project's normal dependency installation.
It grants read-only repository permission, pins the action and tool revisions, and tests the two
exact SDK releases covered by AgentRunProof 0.2.0.
exact SDK releases covered by the published AgentRunProof 0.2.0 contract.

```yaml
name: OpenAI Agents Runner contract
Expand All @@ -85,10 +84,10 @@ jobs:
matrix:
sdk-version: ["0.20.0", "0.21.0"]
steps:
- uses: actions/checkout@fbc6f3992d24b796d5a048ff273f7fcc4a7b6c09 # v5.1.0
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6.3.0
- uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: "3.12.13"
- uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1
Expand Down Expand Up @@ -126,17 +125,17 @@ the test intentionally covers an integration boundary.

## Dependency and release trust

- Pin `agentrunproof==0.2.0` and an exact `openai-agents` release in the isolated job. The supported
- Pin `agentrunproof==0.2.0` and an exact `openai-agents` release in the isolated job. Its supported
range is `openai-agents>=0.20.0,<0.22`; 0.20.0 and 0.21.0 are the exact packaged CI baselines.
- The 0.3.0 source candidate expands that declaration to `openai-agents>=0.20.0,<0.23` and adds an
- The 0.3.0 source contract expands that declaration to `openai-agents>=0.20.0,<0.23` and adds an
exact 0.22.0 packaged-wheel cell on every supported Python. Switch downstream pins and matrices
only after the final release, checksums, and release-bound evidence are public.
after the final release, checksums, and release-bound evidence are public.
- The immutable [v0.2.0 GitHub release](https://github.com/FU-max-boop/agentrunproof/releases/tag/v0.2.0)
publishes wheel and sdist SHA-256 values in `SHA256SUMS`. Its release workflow re-downloads those
assets, compares them byte-for-byte with a rebuild from the tagged source, and smoke-tests the
wheel before the same verified files are sent to PyPI through OIDC trusted publishing.
- The released wheel SHA-256 is
`e393e98bf797cc10f07ea151ec6fffd3b1ebbb21307256309e08f11e97a27d51`; the sdist SHA-256 is
- The v0.2.0 wheel SHA-256 is
`e393e98bf797cc10f07ea151ec6fffd3b1ebbb21307256309e08f11e97a27d51`; its sdist SHA-256 is
`c0ad9c2aeb425cfcaff81b6daeee9c9ded069397dfc7a3319fda28f9842f2ace`.
- A repository with a lock or hash policy should resolve this isolated closure with its normal
dependency-review process and commit the resulting test-only lock data. The short `uv run`
Expand Down
14 changes: 6 additions & 8 deletions docs/openai-agents.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,27 +23,25 @@ SDK regression test for the fix.

## Compatibility

The AgentRunProof 0.3.0 release candidate declares `openai-agents>=0.20.0,<0.23` on Python 3.10
through 3.14. Its packaged-wheel acceptance matrix covers exact SDK 0.20.0, 0.21.0, and 0.22.0
installations. Other patch, prerelease, or source revisions are not individually verified support
cells. The latest published AgentRunProof release is still 0.2.0 and does not declare SDK 0.22
support.
The AgentRunProof 0.3.0 source contract declares `openai-agents>=0.20.0,<0.23` on Python 3.10 through 3.14. Its
packaged-wheel acceptance matrix covers exact SDK 0.20.0, 0.21.0, and 0.22.0 installations. Other
patch, prerelease, or source revisions are not individually verified support cells.

Install the released package in a fresh environment:

```bash
python -m venv .venv
source .venv/bin/activate
python -m pip install "agentrunproof==0.2.0"
python -m pip install agentrunproof
mkdir -p build
```

Built-in scenarios use deterministic local responses and make zero provider requests. They require
no API key. This guarantee covers the built-in scenarios, not arbitrary user-supplied Python code;
package installation may still contact the configured package index.

To exercise the 0.3.0 candidate against its new exact baseline from a source checkout, install the
checkout and SDK together instead of naming a nonexistent PyPI release:
To exercise the 0.3.0 source contract against SDK 0.22 before its immutable release and PyPI pages
exist, install the checkout and exact SDK together:

```bash
python -m pip install -e . "openai-agents==0.22.0"
Expand Down
9 changes: 7 additions & 2 deletions tests/test_publish_workflow.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,13 +9,15 @@
def test_upstream_comparison_release_profiles_are_versioned() -> None:
workflow = WORKFLOW.read_text(encoding="utf-8")

assert "default: v0.2.0" in workflow
assert "default: v0.3.0" in workflow
assert "v0.1.0)" in workflow
assert "v0.1.1)" in workflow
assert "v0.1.2)" in workflow
assert "v0.2.0)" in workflow
assert "v0.3.0)" in workflow
assert "evidence_directory=evidence/upstream-comparison/v1" in workflow
assert "evidence_directory=evidence/upstream-comparison/v2" in workflow
assert "evidence_directory=evidence/upstream-comparison/v3" in workflow
assert "evidence_checker=check-upstream-bundle" in workflow
assert "evidence_kind=upstream-comparison" in workflow

Expand All @@ -38,12 +40,15 @@ def test_comparison_evidence_commit_is_minimal_and_is_the_tag_target() -> None:
tag_policy = workflow.rsplit('case "$RELEASE_TAG" in', 1)[1]
v012_policy = tag_policy.split("v0.1.2)", 1)[1].split(";;", 1)[0]
v020_policy = tag_policy.split("v0.2.0)", 1)[1].split(";;", 1)[0]
v030_policy = tag_policy.split("v0.3.0)", 1)[1].split(";;", 1)[0]

assert 'git merge-base --is-ancestor "$source_commit" "$evidence_parent"' in workflow
assert 'git merge-base --is-ancestor "$source_commit" "$evidence_parent"' not in workflow
assert 'git diff --name-only "$evidence_parent..$evidence_commit"' in workflow
assert 'git diff --exit-code "$evidence_commit" HEAD -- "$EVIDENCE_DIRECTORY"' in workflow
assert 'test "$(git rev-parse HEAD)" = "$evidence_commit"' in v012_policy
assert 'test "$(git rev-parse HEAD)" = "$evidence_commit"' in v020_policy
assert 'test "$(git rev-parse HEAD)" = "$evidence_commit"' in v030_policy
assert 'test "$evidence_parent" = "$source_commit"' in workflow
assert 'bundle_path.parent / run["certificate"]["path"]' in workflow


Expand Down
7 changes: 4 additions & 3 deletions tests/test_upstream_comparison_generator.py
Original file line number Diff line number Diff line change
Expand Up @@ -184,12 +184,13 @@ def test_ci_can_generate_the_first_canonical_candidate_from_an_exact_target() ->
assert "runstate-recursive-agent-tool-approval-serialization,PASS" in workflow
assert "candidate-top-level" not in workflow
assert "python -m agentrunproof check-upstream-bundle" in workflow
assert 'committed_bundle="evidence/upstream-comparison/v2/bundle.json"' in workflow
assert 'committed_bundle="evidence/upstream-comparison/v3/bundle.json"' in workflow
assert "evidence/upstream-comparison/v1/bundle.json" in workflow
assert "evidence/upstream-comparison/v2/bundle.json" in workflow
assert "find evidence/upstream " not in workflow
assert 'git merge-base --is-ancestor "$source_commit" "$evidence_parent"' in workflow
assert 'git merge-base --is-ancestor "$source_commit" "$evidence_parent"' not in workflow
assert 'git diff --name-only "$evidence_parent..$evidence_commit"' in workflow
assert 'test "${evidence_parent_fields[1]}" = "$source_commit"' not in workflow
assert 'test "$evidence_parent" = "$source_commit"' in workflow


def test_replaceable_source_target_binds_clean_git_provenance(tmp_path: Path) -> None:
Expand Down