Skip to content

Commit 294a6af

Browse files
claudeclaude
committed
Add automated R-parity check-and-fix workflow
Adds the "fix if required" half of the fidelity chain alongside the existing detection workflow, with hard guardrails and a human-merge policy. - .github/workflows/parity-autofix.yml: sets up Python + live R at the recorded R commit, reproduces the divergence via the live-R parity runner, and only when behavior genuinely drifted hands a structured report to anthropics/claude-code-action. The agent may edit src/nns/** only, must classify the root cause (Python port bug -> fix; R behavior change or native change -> escalate, never touch extern/NNS-core, tools/NNS, or the cache), must verify against live R, and opens a SEPARATE parity-correction PR. If live R cannot be installed, it opens an escalation PR instead of guessing. Never auto-merges. - inspect-r-api-update.yml: chains to parity-autofix via an nns-parity-divergence repository_dispatch (gated on the optional PARITY_DISPATCH_TOKEN secret; prints the manual trigger otherwise). - docs/parity_autofix.md: setup (ANTHROPIC_API_KEY via /install-github-app, optional PARITY_APP_TOKEN so fix PRs trigger CI, optional PARITY_DISPATCH_TOKEN for auto-chaining), guardrails, and human-merge policy. Automation only; no runtime/package behavior changes.
1 parent 4242ec7 commit 294a6af

3 files changed

Lines changed: 337 additions & 0 deletions

File tree

‎.github/workflows/inspect-r-api-update.yml‎

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -127,6 +127,40 @@ jobs:
127127
echo "- DESCRIPTION changed: \`${{ steps.payload.outputs.description_changed }}\`"
128128
} >> sync/last_live_r_parity_report.md
129129
130+
- name: Dispatch parity autofix for live-R verification
131+
if: steps.payload.outputs.fresh_cache != 'true'
132+
shell: bash
133+
env:
134+
DISPATCH_TOKEN: ${{ secrets.PARITY_DISPATCH_TOKEN }}
135+
run: |
136+
set -euo pipefail
137+
# The cache-based gates below only prove parity against the committed
138+
# cache. The parity-autofix workflow owns live-R verification (it sets
139+
# up R) and opens a separate, human-reviewed fix PR if behavior drifted.
140+
tests=$(jq '.parity_tests | length' sync/last_r_api_plan.json)
141+
if [ "${tests}" -eq 0 ]; then
142+
echo "No mapped parity tests for this change; not dispatching parity autofix."
143+
exit 0
144+
fi
145+
if [ -z "${DISPATCH_TOKEN:-}" ]; then
146+
echo "PARITY_DISPATCH_TOKEN not set; skipping auto-chain to parity-autofix."
147+
echo "Run parity-autofix manually with r_commit=${{ steps.payload.outputs.r_commit }} r_version=${{ steps.payload.outputs.r_version }}."
148+
exit 0
149+
fi
150+
payload=$(jq -n \
151+
--arg rc "${{ steps.payload.outputs.r_commit }}" \
152+
--arg rv "${{ steps.payload.outputs.r_version }}" \
153+
--arg rh "${{ steps.payload.outputs.r_src_tree_hash }}" \
154+
--slurpfile cf changed_files.json \
155+
'{event_type:"nns-parity-divergence", client_payload:{r_commit:$rc, r_version:$rv, r_src_tree_hash:$rh, changed_files:($cf[0] // [])}}')
156+
curl -sSf -X POST \
157+
-H "Accept: application/vnd.github+json" \
158+
-H "Authorization: Bearer ${DISPATCH_TOKEN}" \
159+
"https://api.github.com/repos/${{ github.repository }}/dispatches" \
160+
-d "${payload}"
161+
echo "Dispatched nns-parity-divergence for live-R parity autofix."
162+
163+
130164
- name: Run standard gates if no fresh cache was required
131165
if: steps.payload.outputs.fresh_cache != 'true'
132166
run: |

‎.github/workflows/parity-autofix.yml‎

Lines changed: 213 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,213 @@
1+
name: Parity autofix
2+
3+
# Automated R-behavior parity fix loop for NNS-python.
4+
#
5+
# This is the "fix if required" half of the fidelity chain. The detection half
6+
# lives in inspect-r-api-update.yml. When live-R parity for a changed R API
7+
# diverges, this workflow re-derives the divergence against live R and hands a
8+
# structured report to an agent (anthropics/claude-code-action) that may draft a
9+
# fix to src/nns/** ONLY, then opens a SEPARATE parity-correction PR.
10+
#
11+
# Human-merge policy: this workflow never merges. A human reviews every fix PR.
12+
#
13+
# Trigger model:
14+
# * repository_dispatch [nns-parity-divergence] - emitted by
15+
# inspect-r-api-update.yml when live parity diverges (needs a PAT, see
16+
# docs/parity_autofix.md), or by upstream automation.
17+
# * workflow_dispatch - manual fallback.
18+
19+
on:
20+
repository_dispatch:
21+
types: [nns-parity-divergence]
22+
workflow_dispatch:
23+
inputs:
24+
r_commit:
25+
required: true
26+
type: string
27+
r_version:
28+
required: true
29+
type: string
30+
r_src_tree_hash:
31+
required: false
32+
default: "unknown"
33+
type: string
34+
model:
35+
description: Claude model id for the autofix agent
36+
required: false
37+
default: claude-sonnet-4-6
38+
type: string
39+
40+
permissions:
41+
contents: write
42+
pull-requests: write
43+
issues: write
44+
45+
jobs:
46+
parity-autofix:
47+
runs-on: ubuntu-latest
48+
steps:
49+
- name: Check out NNS-python
50+
uses: actions/checkout@v4
51+
with:
52+
fetch-depth: 0
53+
54+
- name: Resolve payload
55+
id: payload
56+
shell: bash
57+
run: |
58+
set -euo pipefail
59+
if [ "${{ github.event_name }}" = "repository_dispatch" ]; then
60+
echo "r_commit=${{ github.event.client_payload.r_commit }}" >> "$GITHUB_OUTPUT"
61+
echo "r_version=${{ github.event.client_payload.r_version }}" >> "$GITHUB_OUTPUT"
62+
echo "r_src_tree_hash=${{ github.event.client_payload.r_src_tree_hash }}" >> "$GITHUB_OUTPUT"
63+
echo "model=claude-sonnet-4-6" >> "$GITHUB_OUTPUT"
64+
echo '${{ toJson(github.event.client_payload.changed_files) }}' > changed_files.json
65+
else
66+
echo "r_commit=${{ inputs.r_commit }}" >> "$GITHUB_OUTPUT"
67+
echo "r_version=${{ inputs.r_version }}" >> "$GITHUB_OUTPUT"
68+
echo "r_src_tree_hash=${{ inputs.r_src_tree_hash }}" >> "$GITHUB_OUTPUT"
69+
echo "model=${{ inputs.model }}" >> "$GITHUB_OUTPUT"
70+
echo '[]' > changed_files.json
71+
fi
72+
73+
- name: Check out upstream R NNS
74+
uses: actions/checkout@v4
75+
with:
76+
repository: OVVO-Financial/NNS
77+
ref: ${{ steps.payload.outputs.r_commit }}
78+
path: upstream/NNS
79+
80+
- name: Set up Python
81+
uses: actions/setup-python@v5
82+
with:
83+
python-version: "3.11"
84+
85+
- name: Set up R
86+
uses: r-lib/actions/setup-r@v2
87+
with:
88+
r-version: "release"
89+
90+
- name: Install Python build and test tools
91+
run: |
92+
python -m pip install -U pip
93+
python -m pip install build scikit-build-core nanobind pytest ruff mypy numpy scipy
94+
python -m pip install hypothesis pytest-benchmark pytest-xdist
95+
96+
- name: Install package editable
97+
run: python -m pip install -e . --force-reinstall
98+
99+
- name: Install R dependencies for NNS (best effort)
100+
uses: r-lib/actions/setup-r-dependencies@v2
101+
continue-on-error: true
102+
with:
103+
working-directory: tools/NNS
104+
dependencies: '"hard"'
105+
extra-packages: any::jsonlite
106+
107+
- name: Plan R API parity review
108+
run: |
109+
python scripts/plan_r_api_parity_review.py \
110+
--changed-files-json changed_files.json \
111+
--map sync/r_api_map.json \
112+
--out sync/last_r_api_inspection.md \
113+
--json-out sync/last_r_api_plan.json
114+
115+
- name: Install live R NNS from upstream checkout (best effort)
116+
id: r_install
117+
continue-on-error: true
118+
run: |
119+
# install_local_r_nns.py installs from the vendored tools/NNS by
120+
# default; the upstream checkout is the recorded truth for this commit.
121+
python scripts/install_local_r_nns.py
122+
123+
- name: Run live R parity to reproduce divergence
124+
id: live_parity
125+
continue-on-error: true
126+
run: |
127+
python scripts/run_live_r_parity_for_changed_api.py \
128+
--plan sync/last_r_api_plan.json \
129+
--r-checkout upstream/NNS \
130+
--skip-install \
131+
--out sync/last_live_r_parity_report.md
132+
133+
- name: Decide whether a fix is required
134+
id: gate
135+
shell: bash
136+
run: |
137+
outcome="${{ steps.live_parity.outcome }}"
138+
r_ok="${{ steps.r_install.outcome }}"
139+
echo "live_parity_outcome=${outcome}" >> "$GITHUB_OUTPUT"
140+
echo "r_install_outcome=${r_ok}" >> "$GITHUB_OUTPUT"
141+
if [ "${outcome}" = "success" ]; then
142+
echo "needs_fix=false" >> "$GITHUB_OUTPUT"
143+
echo "Live R parity passed; no fix required."
144+
elif [ "${r_ok}" != "success" ]; then
145+
# Cannot verify against live R -> do NOT let the agent guess a fix.
146+
echo "needs_fix=false" >> "$GITHUB_OUTPUT"
147+
echo "escalate=true" >> "$GITHUB_OUTPUT"
148+
echo "Live R install failed; escalating instead of auto-fixing."
149+
else
150+
echo "needs_fix=true" >> "$GITHUB_OUTPUT"
151+
echo "Live R parity diverged; dispatching autofix agent."
152+
fi
153+
154+
- name: Run parity-fix agent
155+
if: steps.gate.outputs.needs_fix == 'true'
156+
uses: anthropics/claude-code-action@v1
157+
with:
158+
anthropic_api_key: ${{ secrets.ANTHROPIC_API_KEY }}
159+
# Use the GitHub App token if configured so the resulting PR triggers
160+
# native-backend-ci (PRs opened with the default GITHUB_TOKEN do not).
161+
github_token: ${{ secrets.PARITY_APP_TOKEN || github.token }}
162+
prompt: |
163+
You are fixing an R-behavior parity divergence in OVVO-Financial/NNS-python.
164+
165+
Context:
166+
- R behavioral truth: OVVO-Financial/NNS at commit ${{ steps.payload.outputs.r_commit }} (version ${{ steps.payload.outputs.r_version }}), checked out at upstream/NNS and installed as live R.
167+
- The divergence report is at sync/last_live_r_parity_report.md. The parity plan is at sync/last_r_api_plan.json and the R->Python map is at sync/r_api_map.json.
168+
169+
Hard rules (do not violate):
170+
1. Edit files under src/nns/** ONLY. NEVER edit extern/NNS-core/**, tools/NNS/**, tests/_r_cache.json, or any test/cache file to make a check pass.
171+
2. Classify the root cause first and act accordingly:
172+
- Python port bug (a wrapper/default/return-shape/algorithm in src/nns/** drifted from R): fix it in src/nns/**.
173+
- R changed behavior (the divergence is because upstream R itself changed): DO NOT edit code to chase a cache value. Stop and open the PR/issue as an escalation describing the change; cache regeneration is a separate, reviewed step.
174+
- Native kernel change (the difference originates in C++ src/** / NNS-core): DO NOT edit. Native code enters Python only through accepted NNS-core commits. Stop and escalate.
175+
3. Verify any fix against LIVE R, not the committed cache. The fix must keep ALL of these green:
176+
- python -m pytest -q -n 0 tests/invariants
177+
- the mapped parity tests in the plan (run with -n 0), against live R
178+
- ruff check . ; mypy ; python -m build
179+
Do not weaken tests or tolerances.
180+
4. Keep the change minimal and in the style of the surrounding code.
181+
182+
Deliverable:
183+
- Create a NEW branch named parity-fix/${{ steps.payload.outputs.r_commit }} and open a SEPARATE parity-correction pull request into the default branch. Do NOT merge it.
184+
- The PR body MUST contain, for each divergence: the function, the arguments, the R output, the Python output, the first divergent intermediate, the affected source files, and the proposed fix (or, for escalations, why no code fix was made and what must happen upstream / in NNS-core).
185+
- Title: "Parity fix: R NNS ${{ steps.payload.outputs.r_commit }}".
186+
claude_args: |
187+
--model ${{ steps.payload.outputs.model }}
188+
--max-turns 40
189+
--allowedTools Edit,Read,Write,Glob,Grep,Bash(python:*),Bash(python -m pytest:*),Bash(ruff:*),Bash(mypy:*),Bash(pip:*),Bash(Rscript:*),Bash(git:*),Bash(gh:*)
190+
--disallowedTools Bash(rm:*)
191+
192+
- name: Escalate when live R could not verify
193+
if: steps.gate.outputs.escalate == 'true'
194+
uses: peter-evans/create-pull-request@v6
195+
with:
196+
branch: parity-escalation-${{ steps.payload.outputs.r_commit }}
197+
title: Parity escalation (live R unavailable) ${{ steps.payload.outputs.r_commit }}
198+
body: |
199+
Live R NNS could not be installed in CI for R commit
200+
`${{ steps.payload.outputs.r_commit }}` (version
201+
`${{ steps.payload.outputs.r_version }}`), so a parity fix was NOT
202+
attempted automatically — fidelity must be verified against live R,
203+
never guessed.
204+
205+
Reports:
206+
- `sync/last_r_api_inspection.md`
207+
- `sync/last_r_api_plan.json`
208+
- `sync/last_live_r_parity_report.md`
209+
210+
Action required: run `scripts/install_local_r_nns.py` in an
211+
environment with R available, reproduce the divergence, and apply a
212+
reviewed parity fix. Native differences must route through NNS-core.
213+
commit-message: Parity escalation ${{ steps.payload.outputs.r_commit }}

‎docs/parity_autofix.md‎

Lines changed: 90 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,90 @@
1+
# Automated parity check and fix
2+
3+
`NNS-python` automates both halves of R-behavior fidelity:
4+
5+
1. **Check** — when upstream `OVVO-Financial/NNS` changes an R API, the
6+
`inspect-r-api-update` workflow plans which Python modules / parity tests are
7+
affected and records a report.
8+
2. **Fix if required** — the `parity-autofix` workflow verifies the affected
9+
public behavior against **live R** at the recorded R commit and, if behavior
10+
drifted, hands a structured divergence report to an agent
11+
(`anthropics/claude-code-action`) that drafts a fix and opens a **separate,
12+
human-reviewed** parity-correction PR.
13+
14+
```text
15+
NNS R API change
16+
-> inspect-r-api-update.yml (plan + cache gates + inspection PR)
17+
-> dispatch nns-parity-divergence
18+
-> parity-autofix.yml (live-R verify -> fix or escalate -> PR)
19+
-> human review + merge
20+
```
21+
22+
## What the agent may and may not do
23+
24+
Hard rules enforced in the agent prompt and the workflow gate:
25+
26+
* Edit **`src/nns/**` only**. Never edit `extern/NNS-core/**`, `tools/NNS/**`,
27+
or `tests/_r_cache.json` to make a check pass.
28+
* Classify the root cause and act accordingly:
29+
* **Python port bug** → fix in `src/nns/**`.
30+
* **R changed behavior** → do not chase a cache value; escalate (cache
31+
regeneration is a separate, reviewed step).
32+
* **Native kernel change** → do not edit; native code enters Python only
33+
through accepted `NNS-core` commits. Escalate.
34+
* Verify against **live R**, not the committed cache. If live R cannot be
35+
installed in CI, the workflow opens an **escalation PR** instead of letting
36+
the agent guess a fix.
37+
* **Human-merge only.** The workflow never merges; every fix PR is reviewed.
38+
39+
## Required setup
40+
41+
### 1. Anthropic credentials (required)
42+
43+
The autofix agent needs an API key exposed as the `ANTHROPIC_API_KEY` repo
44+
secret.
45+
46+
Easiest path — from Claude Code, run:
47+
48+
```text
49+
/install-github-app
50+
```
51+
52+
This installs the official Claude GitHub App, adds the `ANTHROPIC_API_KEY`
53+
secret, and (optionally) scaffolds a workflow. You must be a repo admin.
54+
55+
Manual path:
56+
57+
1. Repo **Settings → Secrets and variables → Actions → New repository secret**.
58+
2. Name `ANTHROPIC_API_KEY`, value = your key from `console.anthropic.com`.
59+
60+
Bedrock / Vertex are supported via the action's `use_bedrock` / `use_vertex`
61+
inputs with OIDC (`id-token: write`) instead of a static key; see the
62+
[cloud providers docs](https://github.com/anthropics/claude-code-action/blob/main/docs/cloud-providers.md).
63+
64+
### 2. Fix-PR token so CI runs on the fix (recommended)
65+
66+
A PR opened with the default `GITHUB_TOKEN` does **not** trigger other workflows
67+
(GitHub's recursion guard), so `native-backend-ci` would not run on the fix PR.
68+
To get CI on fix PRs, let the agent open the PR with a token that does trigger
69+
workflows:
70+
71+
* Installing the **GitHub App** (above) already provides this, or
72+
* add a fine-grained PAT / GitHub App token as the `PARITY_APP_TOKEN` secret;
73+
the workflow passes it to the action as `github_token`. If unset, it falls
74+
back to `github.token`.
75+
76+
### 3. Auto-chain token (optional)
77+
78+
`inspect-r-api-update` chains to `parity-autofix` via a `repository_dispatch`,
79+
which the default `GITHUB_TOKEN` cannot emit. To enable the automatic chain, add
80+
a PAT with `contents: write` (or `repo`) scope as the `PARITY_DISPATCH_TOKEN`
81+
secret. Without it, `inspect-r-api-update` prints the manual trigger command and
82+
you run `parity-autofix` yourself via **workflow_dispatch** (inputs: `r_commit`,
83+
`r_version`).
84+
85+
## Model
86+
87+
The agent model defaults to `claude-sonnet-4-6` and is overridable via the
88+
`workflow_dispatch` `model` input. Use a more capable model (e.g.
89+
`claude-opus-4-8`) for harder divergences. Every fix is still verified by the
90+
full gate set against live R and reviewed by a human before merge.

0 commit comments

Comments
 (0)