cli: substitute safer sequences by default, gated --force, stable executor codes

CHANGELOG.md

@@ -0,0 +1,41 @@
+# Changelog
+
+Notable changes to pg-sprite, with emphasis on anything that changes what
+automation observes: exit codes, verdict fields, and outcome vocabulary.
+The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
+
+## [Unreleased]
+
+### Changed — observable outcomes for automation callers
+
+- **A plain (blocking) `CREATE INDEX` now succeeds instead of refusing.**
+  The engine substitutes `CREATE INDEX CONCURRENTLY` and drives it to a
+  verified valid index: exit 0, with the substitution disclosed in the
+  verdict's `executed_sql`. Previously this refused with exit 2 and reason
+  `index-statement`. Anything branching on the old refusal will now see
+  success for the same input.
+- **Rewrite-requiring changes (e.g. `ALTER COLUMN ... TYPE`) refuse up
+  front with reason `backend-unavailable`** instead of running a blind
+  bounded attempt that budget-cancels. No lock acquisition is attempted;
+  the refusal is decided from the classification alone. Previously the same
+  input ended in reason `not-native-safe-budget-exceeded` after a cancelled
+  attempt.
+- **Blocking `ALTER TABLE` forms with a safer native sequence (e.g.
+  `SET NOT NULL`) run as that sequence by default**, disclosed in
+  `executed_sql`. The submitted form can still be forced with
+  `--force <schema.table>`, which is audited and recorded in the verdict's
+  `forced` field.
+
+### Added
+
+- **A third verdict outcome, `failed`,** for execution failures (still exit
+  1 — refusals remain exit 2). The verdict carries the executor's stable
+  outcome code in `code`, and for a mid-sequence failure the 1-based
+  `failed_step`, its `failed_step_sql`, and the committed prefix in
+  `executed_sql` — an empty prefix means nothing committed, so automation
+  can distinguish "nothing happened" from "partial state left behind"
+  without parsing stderr.
+- `--force <schema.table>`: an audited override that runs the submitted
+  form as one bounded attempt, unlocking only the substitution-override,
+  rewrite-required, and backend-unavailable paths. Planner refusals stay
+  unforceable.

README.md

@@ -27,9 +27,11 @@ verdict when it can't prove one (see
 
 **Status: Phases 1 and 2.1–2.5.** The parse boundary, declarative diff,
 classifier, router seam, versioned dry-run plan report, offline linter, and
-advisory `suggest` command are implemented. `pg-sprite migrate --alter '…'` runs a bounded optimistic
-native attempt; routed execution beyond that attempt lands in Phase 3.
-Changes without an available backend get a structured refusal (exit code 2).
+advisory `suggest` command are implemented. `pg-sprite migrate --alter '…'` classifies and
+routes the statement, then executes the routed SQL — the planner's safer native sequence by
+default when the submitted form blocks (reported in the verdict's `executed_sql`), a bounded
+optimistic native attempt otherwise. A gated `--force` runs the submitted form as-is under the
+same budgets. Changes without an available backend get a structured refusal (exit code 2).
 The design docs and the phased
 build plan live in [docs/](docs/) — start with
 [docs/README.md](docs/README.md); the vision — what pg-sprite is and is not —

docs/architecture.md

@@ -128,14 +128,14 @@ different levels of commitment:
 | `pkg/dbconn` | Pool with bounded session timeouts, retries, RDS/Aurora auto-TLS (embedded CA bundle), terminate-blockers; advisory-lock mutual exclusion lands here | exists |
 | `pkg/statement` | `go-pgquery` (Wasm `libpg_query`) parse boundary, typed per-operation descriptors, and advisory rewrites (never hand-parse SQL); migration-time shadow DDL + fingerprints are derived by `pkg/schemadiff` via scratch-DB execute-and-introspect | exists |
 | `pkg/preflight` | Precondition verification and refusals before any write | exists (Phase 1: table-size guard); grows through Phase 2 |
-| `pkg/verdict` | Structured outcome contract (executed / refused + reason + safer idiom), rendering, exit codes | exists (Phase 1) |
+| `pkg/verdict` | Structured outcome contract (executed / refused / failed + reason, stable executor code, and safer idiom), rendering, exit codes | exists (Phase 1) |
 | `pkg/schemadiff` | Execute-and-introspect desired state, introspect the live catalog, and produce an ordered declarative diff | exists |
 | `pkg/planner` | Classify typed operations and emit safer native SQL | exists |
 | `pkg/lint` | Offline lint findings with typed codes: unsupported operations are errors; blocking idioms, rewrites, and destructive drops are warnings | exists (Phase 2.5) |
 | `pkg/plan` | Versioned machine-readable dry-run plan report — the one JSON contract both front doors emit and an orchestrator consumes | exists (Phase 2.5) |
 | `pkg/diffplan` | The declarative front door as a library: desired schema in, routed `plan.Report` out — the CLI `diff` and embedding orchestrators share this one pipeline | exists |
 | `pkg/router` | Route classified statements to native / copy-and-swap / refuse dispositions; copy-and-swap reports unavailable until that backend lands | exists (Phase 2.4) |
-| `pkg/executor` | Bounded optimistic native attempt; the `Executor` contract (`Plan`/`Execute`/`Status`/`Abort`) lands in Phase 3 | bounded optimistic attempt exists |
+| `pkg/executor` | Bounded optimistic native attempt, the concurrent index build, and the autocommit safer-sequence runner, with stable outcome codes; the full `Executor` contract (`Plan`/`Execute`/`Status`/`Abort`) arrives with the copy-and-swap backend | native execution exists |
 | `pkg/table` | PK-range chunkers (single-column fast path, composite), dynamic time-based sizing | Phase 4 |
 | `pkg/copier` | Parallel chunked copy into the shadow table (never overwrites) | Phase 4 |
 | `pkg/checksum` | The mandatory correctness gate; continuous checker; repair primitive | Phase 5 |

docs/high-level-design.md

@@ -102,8 +102,9 @@ the planner's classifier:
   almost no parsing logic. This path lives at the `migrate` front door.
 - **Classified planning path (Phases 2.1–2.4).** Parse the statement and introspect the live schema to
   **predict the path up front** — native-safe, needs-rewrite, or refuse — without trial
-  execution. The planner drives `diff` and `migrate --dry-run`; Phase 3 will make its classified
-  route drive execution and remove the wasted/aborted attempts that the optimistic path can incur.
+  execution. The planner drives `diff` and `migrate --dry-run`, and its classified route drives
+  `migrate`'s execution: a blocking submitted form is substituted with the planner's safer
+  native sequence instead of incurring a wasted/aborted blind attempt.
 
 > **PostgreSQL caveat.** Unlike MySQL's `ALGORITHM=INSTANT`, PostgreSQL has **no assertion** that
 > forces a change to be instant-or-error — a rewrite attempt acquires `ACCESS EXCLUSIVE` and does
@@ -171,7 +172,7 @@ The engine accepts a change two ways, both feeding the **same** planner pipeline
 
 - **Declarative** — the user supplies the **desired end-state** (a checked-in `CREATE TABLE`
   `.sql` file); the engine **derives** the `ALTER` by diffing desired vs live, then runs it
-  through classify → route. Phase 3 adds execution of the classified route.
+  through classify → route — the same classified route `migrate` executes.
 - **Imperative** — the user supplies the `ALTER` directly. It is the **same** pipeline with the
   diff step skipped.
 
@@ -200,16 +201,18 @@ literal statement:
                           │ yes
                           ▼
    ┌──────────────────────────────────────────────────────────-┐
-   │  RECOMMENDATION (does NOT execute):                       │
+   │  RECOMMENDATION (dry-run does NOT execute):               │
    │   you asked:   CREATE INDEX idx ON orders (customer_id)   │
    │   safer form:  CREATE INDEX CONCURRENTLY idx ON orders …  │
    │   why: a plain CREATE INDEX takes SHARE and blocks writes │
    │        for the whole build; CONCURRENTLY does not.        │
    └──────────────────────────────────────────────────────────-┘
-        │ Phase 3: apply recommendation       │ Phase 3: insist on literal
-        ▼                                      ▼
-   execute the safe idiom                 --force  ⇒  DANGER prompt +
-   (classified sequence)                 explicit approval, then run as-is
+        │ migrate (default):                  │ migrate --force <table>:
+        │ apply recommendation                │ insist on literal
+        ▼                                     ▼
+   execute the safe idiom                typed acknowledgement of the
+   (classified sequence)                 resolved table, then run as-is
+                                         under the same budgets
 ```
 
 Examples of what it suggests (the same idioms the classifier already knows):
@@ -232,17 +235,17 @@ Two principles govern this:
   that must be detected and rebuilt — see the
   [online DDL reference](postgres-online-ddl-reference.md)), which is why the engine owns
   executing it rather than handing it to the user to run manually.
-- **The planned force route is loud and explicit.** Phase 3 adds a `--force`
-  (run-as-submitted) flag for the rare case where the operator genuinely wants the literal
-  statement. It will be gated behind
-  prominent **DANGER / CAUTION** output explaining exactly what will block and for how long, and
-  an **explicit confirmation** (typed acknowledgement, not a bare `-y`), and the override is
-  logged. Force is an escape hatch, not a convenience.
-
-Today the classifier constructs safer sequences, `diff` / `migrate --dry-run` render them, and
-the library's sequence executor runs them under the autocommit-each-step contract; default
-`migrate` still uses the bounded optimistic Phase 1 path. Phase 3's remaining work is the
-substitution wiring that routes the classified sequences into `migrate`.
+- **The force route is loud and explicit.** `--force` (run-as-submitted) exists for the rare
+  case where the operator genuinely wants the literal statement. It is gated behind an
+  **explicit typed acknowledgement** — the flag's value must name the resolved schema-qualified
+  target table, not a bare `-y` — the override is logged unconditionally and recorded in the
+  verdict's `forced` field, and the statement still runs under the executor's budgets and the
+  size guard. Force is an escape hatch, not a convenience (mechanics in the
+  [low-level design](low-level-design.md#the---force-gate)).
+
+The classifier constructs safer sequences, `diff` / `migrate --dry-run` render them, and
+default `migrate` substitutes and executes them; the submitted form runs as-is only when it is
+already the safe idiom or under an acknowledged `--force`.
 
 In non-interactive contexts (CI), advisory mode is a natural gate: the engine prints the
 recommended rewrites and exits non-zero if a submitted statement would need a riskier path than

docs/low-level-design.md

@@ -337,35 +337,38 @@ Classification belongs to `pkg/planner`; `pkg/statement` supplies typed operatio
 | --- | --- |
 | `lint` | Offline (no database): classify every statement with zero live facts and report typed findings — unsupported operations are errors (non-zero exit), blocking idioms (with the safer SQL), conservative rewrites, and destructive drops are warnings. |
 | `diff` / `migrate --dry-run` | Print the classified, routed plan and safer SQL where applicable. **Never writes the live table.** `diff` has no `--dry-run` flag because it never executes the plan (its desired-state diff does run the desired DDL in an always-rolled-back scratch transaction — see the scratch-schema note above). |
-| `migrate` (default) | Run the Phase 1 statement gate, preflight, and bounded optimistic native attempt. It does not yet execute classifier-produced safer SQL. |
-| `migrate --force` (planned Phase 3) | Run each statement **exactly as submitted**, bypassing the safe rewrite. Gated — see below. |
-
-The classifier constructs `CREATE INDEX CONCURRENTLY` and other safer sequences today, and the
-library executes the multi-step idiom families — `pkg/executor`'s sequence executor runs a safer
-sequence under the autocommit-each-step contract (brief steps bounded like an optimistic
-attempt, the validation scan and concurrent builds under their own budgets). The one-step
-`CONCURRENTLY` rewrites the classifier also emits (`DROP INDEX`, `REINDEX`,
-`DETACH PARTITION`) are not yet driven: the sequence executor refuses them typed, because a
-cancelled wait leaves recovery states it does not yet own. The CLI front door does not yet
-route to it: `diff` and `migrate --dry-run` render the sequences, and Phase 3's substitution
-work wires the classified route into execution.
+| `migrate` (default) | Run the statement gate, classify and route exactly as dry-run would, then execute the routed SQL: the planner's **safer native sequence by default** when the submitted form blocks, the submitted form as a bounded optimistic attempt otherwise. The verdict's `executed_sql` reports the substitution. |
+| `migrate --force` | Run the statement **exactly as submitted**, overriding a safer-sequence substitution or a rewrite-required / backend-unavailable refusal. Gated — see below. |
+
+The classifier constructs `CREATE INDEX CONCURRENTLY` and other safer sequences, and `migrate`
+executes them through `pkg/executor`'s sequence executor under the autocommit-each-step
+contract (brief steps bounded like an optimistic attempt, the validation scan and concurrent
+builds under their own budgets). The one-step `CONCURRENTLY` rewrites the classifier also
+emits (`DROP INDEX`, `REINDEX`, `DETACH PARTITION`) are not driven: `migrate`'s statement gate
+refuses `DROP INDEX` and `REINDEX` with the safer-idiom pointer, and the sequence executor's
+whole-sequence admission refuses a substituted `DETACH PARTITION CONCURRENTLY` typed before
+anything executes — a cancelled wait leaves recovery states it does not yet own — which
+`migrate` reports as a refusal verdict, not an operational error. When the
+planner says a safer sequence is required but cannot construct one, `migrate` refuses
+(`not-native-safe-rewrite-required`) rather than run the blocking form the plan itself
+flagged.
 
 ### The `--force` gate
 
-This entire gate is planned Phase 3 behavior; no `--force` flag exists today. It is deliberately
-high-friction:
+`--force` is deliberately high-friction:
 
-1. Print a prominent **DANGER / CAUTION** block: the exact statement, the lock it will take, what
-   it blocks (reads? writes?), and the expected/worst-case duration and lock-queue impact
-   (cross-link to 12-mysql-vs-postgresql.md § the lock queue).
-2. Require an **explicit typed acknowledgement** (e.g. type the table name, or
-   `--i-understand-the-risk`), not a bare `-y`/`--yes`.
-3. Still wrap the statement in `lock_timeout` + bounded retry unless the user *also* opts out of
-   that explicitly (a second, separate flag) — force means "run my statement", not "remove every
-   guardrail".
-4. **Log the override** (who, when, what statement, what the recommendation was) for audit.
+1. Require an **explicit typed acknowledgement**: the flag's value must name the resolved
+   schema-qualified target table exactly — the operator names the relation whose lock they are
+   accepting. A mismatch is a usage error; nothing executes.
+2. Still run under `lock_timeout` / `statement_timeout` budgets and the table-size guard —
+   force overrides the *routing*, never the executor's protections. There is no opt-out:
+   force means "run my statement", not "remove every guardrail".
+3. **Log the override** unconditionally (warn-level, not gated by `--debug`) and record it
+   machine-readably in the verdict's `forced` field for audit.
+4. Planner refusals (no known safe path) and unsupported statement kinds are **not** forceable —
+   there is nothing bounded to acknowledge.
 
-Force is planned for the rare legitimate case (e.g. a maintenance window where the table is known
+Force exists for the rare legitimate case (e.g. a maintenance window where the table is known
 idle and a plain rewrite is acceptable); it is an escape hatch, not a shortcut, consistent with
 *decisions, not options*.
 
@@ -796,21 +799,23 @@ path exists in `pkg/executor` — session-scoped, outside any transaction, under
 wait policy (no per-lock timeout, one overall deadline), with invalid-index detection that
 fails closed into a typed, state-specific outcome (the executor never drops an index: a
 name-based drop cannot prove ownership until the LK-1 lease exists; the operator runbook is
-[invalid-index-recovery.md](invalid-index-recovery.md)). The executor is deliberately not yet
-reachable from the CLI — the engine lands first, the front door next. Remaining Phase 3 work,
-roughly in order:
-
-- the CLI front door for the native path: `migrate` routing an admitted statement to the
-  executor, resolving an unqualified table name once against the session's `search_path` and
-  re-emitting the qualified statement (the library-level `ErrUnqualifiedTable` refusal stays;
-  the CLI moves the qualification burden off the user), and rendering the typed outcomes —
-  each executor outcome gaining a stable string code in the report contracts, the same
-  treatment `pkg/lint` gave its findings, so orchestrators branch on one vocabulary,
-- execute classifier-produced safer sequences through the routed native path,
+[invalid-index-recovery.md](invalid-index-recovery.md)). The CLI front door for the native
+path is wired: `migrate` routes an admitted statement through classify → route, resolves an
+unqualified table name once against the session's `search_path` and re-emits the qualified
+statement (the library-level `ErrUnqualifiedTable` refusal stays; the CLI moves the
+qualification burden off the user), substitutes and executes classifier-produced safer
+sequences by default with the guarded `--force` escape hatch, and renders the typed outcomes.
+At the library seam, each executor outcome maps to a stable string code
+(`executor.OutcomeCode`), the same treatment `pkg/lint` gave its findings, so an orchestrator
+embedding `pkg/executor` branches on one vocabulary; the CLI's verdict JSON carries the same
+codes — an execution failure ends in a `failed` verdict (exit 1, distinct from the refusal
+exit 2) with the code, the failed step, and the committed prefix in `executed_sql`, so
+automation can distinguish nothing-committed from partial state left behind. Remaining
+Phase 3 work, roughly in order:
+
 - bound lock acquisition with timeout and retry for the blocking idioms,
-- substitution by default, the guarded `--force` escape hatch, and progress reporting
-  (`pg_stat_progress_create_index` by the build's backend PID, which the executor already
-  captures for its ownership proof).
+- progress reporting (`pg_stat_progress_create_index` by the build's backend PID, which the
+  executor already captures for its ownership proof).
 
 The copy-and-swap backend, including change capture, copying, applying, checksumming, and
 cutover, follows Phase 3.

docs/postgres-online-ddl-reference.md

@@ -216,8 +216,8 @@ cutover) are the **table-rewrite** ones, because PostgreSQL cannot do them onlin
 **Everything else can be done natively-safe** with PostgreSQL's own
 `CONCURRENTLY` / `NOT VALID`+`VALIDATE` / fast-default / `USING INDEX` patterns. By the
 [*classify-before-copy* principle](design-principles.md#classify-first-leverage-native-postgresql),
-the plan **detects those and routes them to native DDL** instead of a copy; executing the routed
-backend lands in Phase 3 — the same bypass Spirit applies when it attempts `INSTANT`/`INPLACE`
+the plan **detects those and routes them to native DDL** instead of a copy, and `migrate`
+executes the routed sequence — the same bypass Spirit applies when it attempts `INSTANT`/`INPLACE`
 before falling back to a table copy. A shadow-table copy is the last resort, used only when no
 native online path exists.