Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 17 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,23 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).

### Changed — observable outcomes for automation callers

- **Index builds on partitioned parents now refuse with exit 2 and reason
`unsupported-partitioned-parent`** instead of reaching a mid-change exit-1
PostgreSQL failure. Concurrent and blocking builds carry distinct typed
causes; blocking builds are refused by policy even though PostgreSQL supports
them. Leaf partitions retain the blocking-to-`CONCURRENTLY` substitution
described below.
- **Partitioned-parent `ADD CONSTRAINT ... USING INDEX` now refuses before
execution** on every supported PostgreSQL version, with typed cause
`parent-index-adoption`.
- **Partitioned-parent foreign keys added `NOT VALID` refuse on PostgreSQL
before 18** with reason `unsupported-partitioned-parent`; PostgreSQL 18 and
later retain the supported in-place path.
- **Dry-run and declarative diff reports now account for partitioned-parent
admission**, reporting disposition `refuse` and reason
`unsupported-partitioned-parent` instead of advertising execution. Refused
statements now carry the reason directly and omit impossible safer-SQL advice.

- **A plain (blocking) `CREATE INDEX` now succeeds instead of refusing.**
The engine substitutes `CREATE INDEX CONCURRENTLY` and drives it to a
verified valid index: exit 0, with the substitution disclosed in the
Expand Down
1 change: 1 addition & 0 deletions docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@ combination is the product is [vision.md](vision.md); start there.
| [invariants.md](invariants.md) | The canonical **invariant registry** — testable runtime MUST-statements (correctness, locking, state/resume, refusals, orchestration), each with its enforcement point and source. Mined from this doc set plus [Spirit](https://github.com/block/spirit)'s stated safety invariants and [SchemaBot](https://github.com/block/schemabot)'s control-plane discipline; the build plan's phases carry per-invariant test obligations. |
| [tcb-model.md](tcb-model.md) | The **TCB model** — the trusted-computing-base partition of the engine: which components are the small trusted core that enforces the invariant registry vs the untrusted periphery, the never-trust-callers rule, domain types that make illegal states unrepresentable, the in-TCB engineering rules (from TigerBeetle TIGER_STYLE, s2n-tls, qmail, bitcoin-core), the verification ladder, and the per-side AI-assisted development policy. |
| [plan-report.md](plan-report.md) | The **plan report contract** — the versioned JSON shape both front doors emit for dry-run plans: fields, closed vocabularies, the fingerprint identity, required consumer behavior for unknown versions/values, and one generated example per source (pinned by test). |
| [limitations.md](limitations.md) | The **current limitations** — schema changes pg-sprite refuses today, why they are unsafe or unsupported, and where an operator must act outside the engine. |
| [lint-report.md](lint-report.md) | The **lint report contract** — the versioned JSON shape `pg-sprite lint` emits for offline CI gating: finding fields (verbatim SQL, line/column), the codes table, severities and exit behavior, the offline-conservatism rules, and how the contract versions relative to the plan report. |
| [suggest-report.md](suggest-report.md) | The **suggest report contract** — the versioned JSON shape `pg-sprite suggest` emits for offline advice: the typed caveat vocabulary (what changes about how you must run a safer form, and what a failed step leaves behind), the typed guidance codes for rewrites the planner cannot construct, and the operation → safer form → caveats table (pinned by test). |
| [engine-role.md](engine-role.md) | The **engine-role provisioning contract** — the tiered minimum access a PostgreSQL user needs to run schema changes against tables it does not own: role membership for owner-gated DDL, schema `CREATE` for index builds and shadow objects, `SET ROLE` for owner-correct shadow creation, replication access for CDC, and the explicit list of powers the engine role must *not* have. Preflight refusals name the missing `GRANT` and point here. |
Expand Down
5 changes: 5 additions & 0 deletions docs/engine-role.md
Original file line number Diff line number Diff line change
Expand Up @@ -103,3 +103,8 @@ evaluates the tier checks for the change's planned strategy. A missing requireme
typed refusal naming the exact `GRANT` statement that would satisfy it — the same
fail-closed posture as every other refusal in the engine, and the reason this page exists:
the refusal points here, and this page says what to provision and why it is safe.

Preflight also records the target's relation kind. Leaf partitions are ordinary tables and
use the same paths as any other table. A partitioned parent may run supported in-place changes;
the parent-specific refusal boundaries and their rationale are documented in
[current limitations](limitations.md).
7 changes: 6 additions & 1 deletion docs/invariants.md
Original file line number Diff line number Diff line change
Expand Up @@ -267,6 +267,11 @@ Each refusal is a preflight **error with a stated reason** — never a warning,
- **RF-5** — The dangerous literal never runs silently: risky statements with a safer native
idiom get the idiom (reported) or a recommendation; running as-submitted requires the loud,
typed, audited `--force`. *Source:* [high-level-design § advisory mode](high-level-design.md#advisory-mode-suggest-the-safe-rewrite-dont-silently-run-the-risky-one).
- **RF-6** — A partitioned-parent sequence is refused before its first step when it would build
an index, or on PostgreSQL before 18 when it would add a foreign key `NOT VALID`. pg-sprite
does not substitute a blocking parent build for the missing partition-aware online flow.
*Enforced:* preflight and sequence-executor admission. *Source:* PostgreSQL relation-kind and
version capabilities.

## Orchestration / control-plane (OC)

Expand Down Expand Up @@ -330,7 +335,7 @@ about **how we write and review the code**.

| Invariant | Landed by phase | Test obligation |
| --- | --- | --- |
| CO-7, RF-1..RF-5 | 1–2 (gate/linter), full at 2 | golden refusal/parse tests |
| CO-7, RF-1..RF-6 | 1–3 (gate/linter/executor) | golden refusal/parse and executor-admission tests |
| LK-1 | 0–1 (before any executing mode ships) | two-instance mutual-exclusion + keepalive-loss test |
| LK-2 | 3 (native), 7 (cutover) | lock-bounding + CIC-exception tests |
| CO-1, CO-2, CO-3 | 5 (gate), 8 (watermark/divergence policy) | inject-divergence, repair-invalidates-watermark |
Expand Down
11 changes: 11 additions & 0 deletions docs/limitations.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
# Current limitations

pg-sprite refuses a schema change when it cannot provide its online-safety
guarantees. These are current capability boundaries, not escape hatches:

| Change | Current behavior |
| --- | --- |
| Index builds on a partitioned parent | PostgreSQL cannot build an index concurrently at the parent level. PostgreSQL supports a plain blocking build, but pg-sprite refuses it by policy because it takes `ACCESS EXCLUSIVE`; `--force` does not bypass this decision. An operator who chooses a maintenance-window blocking build must run it outside pg-sprite. The partition-aware `CREATE INDEX ON ONLY` → per-partition `CREATE INDEX CONCURRENTLY` → `ATTACH PARTITION` flow is planned but not yet implemented. |
| `ADD CONSTRAINT ... USING INDEX` on a partitioned parent | PostgreSQL does not support adopting an existing index on a partitioned parent in any supported version. pg-sprite refuses before execution. |
| `ADD FOREIGN KEY ... NOT VALID` on a partitioned parent | PostgreSQL does not support this before version 18, so pg-sprite refuses it on versions 14–17. It is supported on version 18 and later. |
| Copy-and-swap | The copy-and-swap backend is not yet available. Statements that require it route to `refuse`; pg-sprite never falls through to a blocking rewrite. |
28 changes: 22 additions & 6 deletions docs/plan-report.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ the examples at the end of this page.

Every report carries `format_version`. A consumer that does not recognize the version must
**reject the report** — never guess at field semantics. The version covers more than the
field shape: the closed vocabularies below (sources, routes, reasons, backends, dispositions,
field shape: the closed vocabularies below (sources, routes, backends, dispositions,
kinds) and the fingerprint serialization are all pinned to it. Adding a vocabulary value or
changing the fingerprint definition is a contract change and bumps `format_version`, even if
no field is added or renamed.
Expand All @@ -23,10 +23,11 @@ two version independently. Lint findings embed this contract's Reasons vocabular

## Consumer behavior for unknown values

Every enum field in the report draws from a closed vocabulary listed here. A consumer that
meets a value it does not recognize must **treat the statement as unknown and refuse it** —
never ignore it and proceed. This is the same fail-closed posture the engine itself takes
with SQL it does not fully understand.
Except for target-dependent refusal `reason` fields, every enum field in the report draws
from a closed vocabulary listed here. Refusal reasons are open and owned by `pkg/verdict`.
A consumer that meets any value it does not recognize must **treat the statement as unknown
and refuse it** — never ignore it and proceed. This is the same fail-closed posture the
engine itself takes with SQL it does not fully understand.

## Report fields

Expand All @@ -39,6 +40,7 @@ with SQL it does not fully understand.
| `server_version` | string | when connected | The PostgreSQL `server_version` the plan was derived against. Classification is version-sensitive; a stored or forwarded report names the server whose rules produced it. |
| `table_exists` | bool | diff source only | Whether the live table was found. Absent means "not introspected" (alter source); `false` means the plan is the full desired schema. |
| `disposition` | string | always | Aggregate disposition across all statements (see Dispositions). |
| `reason` | string | target-dependent refusal only | Aggregate typed refusal cause when target facts override an otherwise executable route. |
| `fingerprint` | string | always | The plan's stable identity (see Fingerprint). |
| `statements` | array | always | The ordered plan; `[]` (never `null`) means nothing to do. |

Expand All @@ -52,6 +54,7 @@ with SQL it does not fully understand.
| `route` | string | always | The planner's aggregate route for the statement (see Routes). |
| `backend` | string | except refusals | The assigned execution strategy (see Backends); absent for refusals. |
| `disposition` | string | always | What execution would do with this statement now (see Dispositions). |
| `reason` | string | target-dependent refusal only | Typed refusal cause for this statement; currently `unsupported-partitioned-parent`. An unknown value must be treated as refused. |
| `decisions` | array | always | The planner's per-operation classifications (below). |
| `exec_sql` | array | native route | The ordered SQL the native backend would run — the safer sequence when the planner constructed one. Absent for non-native routes. |
| `execution` | string | with `exec_sql` | The typed execution contract for `exec_sql` (see Execution contracts). A consumer that runs the statements itself branches on this — it is what says the steps must not be wrapped in a transaction block. Present exactly when `exec_sql` is. |
Expand Down Expand Up @@ -85,7 +88,7 @@ with SQL it does not fully understand.
| `copy-and-swap` | Needs a table rewrite; only the engine's shadow copy + cutover can do it online. |
| `refuse` | No known safe path; not executed. |

### Reasons (`reason`)
### Planner decision reasons (`decisions[].reason`)

| Value | Meaning |
|---|---|
Expand All @@ -102,6 +105,19 @@ with SQL it does not fully understand.
| `partition-parent-lock` | Partition attach/detach in its lock-taking form. |
| `unsupported-operation` | The planner does not recognize the operation or knows no safe path for it. |

### Target-dependent refusal reasons (`reason`, `statements[].reason`)

This vocabulary is owned by `pkg/verdict` and is open: new values may appear without changing
the meaning of existing values. A consumer that sees an unknown reason must fail closed and
treat the statement and report as refused.

| Value | Meaning |
|---|---|
| `unsupported-partitioned-parent` | Target facts show that the statement cannot run safely on a partitioned parent. |

On the apply path, refusal checks have deterministic precedence: table size, then partition
support, then privileges.

### Backends (`backend`)

| Value | Meaning |
Expand Down
29 changes: 23 additions & 6 deletions internal/cli/dryrun.go
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ import (
"github.com/block/pg-sprite/pkg/dbconn"
"github.com/block/pg-sprite/pkg/plan"
"github.com/block/pg-sprite/pkg/planner"
"github.com/block/pg-sprite/pkg/preflight"
"github.com/block/pg-sprite/pkg/router"
"github.com/block/pg-sprite/pkg/schemadiff"
"github.com/block/pg-sprite/pkg/statement"
Expand All @@ -35,7 +36,7 @@ func (c *MigrateCmd) runDryRun(ctx context.Context, out io.Writer) error {
}
defer pool.Close()

facts, err := dryRunFacts(ctx, pool, st)
facts, targetFacts, err := dryRunFacts(ctx, pool, st)
if err != nil {
return err
}
Expand Down Expand Up @@ -64,6 +65,18 @@ func (c *MigrateCmd) runDryRun(ctx context.Context, out io.Writer) error {
for _, rs := range routed.Statements {
report.Statements = append(report.Statements, plan.FromRouted(rs))
}
if targetFacts.Partitioned() {
refused := make([]bool, len(report.Statements))
for i := range report.Statements {
var cause preflight.PartitionRefusalCause
cause, err = preflight.RefusesPartitionedParent(targetFacts.ServerMajor(), report.Statements[i].ExecSQL)
if err != nil {
return err
}
refused[i] = cause != ""
}
plan.RefuseUnsupportedPartitionedParent(&report, refused)
}
report.Fingerprint = plan.Fingerprint(report.Statements)

if c.JSON {
Expand Down Expand Up @@ -92,16 +105,20 @@ func resolvedSchema(st statement.Statement) string {
// dryRunFacts introspects the statement's target table for classifier
// facts. Statements without a single table target (index drops, REINDEX)
// and missing tables classify with zero facts.
func dryRunFacts(ctx context.Context, pool *pgxpool.Pool, st statement.Statement) (planner.Facts, error) {
func dryRunFacts(ctx context.Context, pool *pgxpool.Pool, st statement.Statement) (planner.Facts, preflight.TargetFacts, error) {
if st.Table() == "" {
return planner.Facts{}, nil
return planner.Facts{}, preflight.TargetFacts{}, nil
}
live, err := schemadiff.Introspect(ctx, pool, resolvedSchema(st), st.Table())
switch {
case errors.Is(err, schemadiff.ErrTableNotFound):
return planner.Facts{}, nil
return planner.Facts{}, preflight.TargetFacts{}, nil
case err != nil:
return planner.Facts{}, err
return planner.Facts{}, preflight.TargetFacts{}, err
}
targetFacts, err := preflight.LookupTargetFacts(ctx, pool, resolvedSchema(st), st.Table())
if err != nil {
return planner.Facts{}, preflight.TargetFacts{}, err
}
return planner.FactsFrom(live), nil
return planner.FactsFrom(live), targetFacts, nil
}
20 changes: 20 additions & 0 deletions internal/cli/dryrun_integration_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ import (
"github.com/block/pg-sprite/pkg/plan"
"github.com/block/pg-sprite/pkg/planner"
"github.com/block/pg-sprite/pkg/router"
"github.com/block/pg-sprite/pkg/verdict"
)

// dryRunPlan runs migrate --dry-run --json and decodes the plan report.
Expand Down Expand Up @@ -57,6 +58,25 @@ func TestMigrateDryRunRoutesRewriteWithoutExecuting(t *testing.T) {
assert.Equal(t, "integer", typ, "dry-run must not execute the change")
}

func TestMigrateDryRunRefusesPartitionedParentIndexPlan(t *testing.T) {
url := testutil.StartPostgres(t)
pool, err := dbconn.NewPool(t.Context(), dbconn.Config{URL: url})
require.NoError(t, err)
defer pool.Close()
schema := createPartitionFixture(t, pool)
report := dryRunPlan(t, url, fmt.Sprintf("CREATE INDEX p_v_idx ON %s.p (v)", schema))
assert.Equal(t, router.DispositionRefuse, report.Disposition)
assert.Equal(t, verdict.ReasonUnsupportedPartitionedParent, report.Reason)
require.Len(t, report.Statements, 1)
assert.Equal(t, router.DispositionRefuse, report.Statements[0].Disposition)
assert.Equal(t, verdict.ReasonUnsupportedPartitionedParent, report.Statements[0].Reason)
assert.Empty(t, report.Statements[0].ExecSQL)
for _, decision := range report.Statements[0].Decisions {
assert.Empty(t, decision.SaferSQL)
assert.Empty(t, decision.SaferSQLExecution)
}
}

// Live facts feed the imperative dry-run: a widen the classifier can only
// prove with the live column type routes native, and still executes nothing.
func TestMigrateDryRunUsesLiveFacts(t *testing.T) {
Expand Down
52 changes: 38 additions & 14 deletions internal/cli/migrate.go
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,7 @@ func (c *MigrateCmd) run(ctx context.Context, out io.Writer) error {
}
}

facts, err := dryRunFacts(ctx, pool, st)
facts, _, err := dryRunFacts(ctx, pool, st)
if err != nil {
return err
}
Expand Down Expand Up @@ -188,6 +188,29 @@ func (c *MigrateCmd) auditForce(st statement.Statement, rs router.Statement) {
func (c *MigrateCmd) execute(ctx context.Context, out io.Writer, pool *pgxpool.Pool,
st statement.Statement, execSQL []string, plan planner.Plan,
substituted, forced bool, logger *slog.Logger) error {
limit := int64(c.MaxTableSize)
if !sizeGuardApplies(plan, substituted) {
limit = preflight.NoSizeLimit
}
pt, err := preflight.CheckTable(ctx, pool, st.Schema(), st.Table(), limit)
var sizeErr *preflight.SizeError
if errors.As(err, &sizeErr) {
return c.emit(out, sizeGuardVerdict(st, sizeErr, forced))
}
if err != nil {
return err
}
serverMajor, err := dbconn.ServerMajor(ctx, pool)
if err != nil {
return err
}
if err := preflight.CheckPartitionSupport(pt, serverMajor, execSQL); err != nil {
var partitionErr *preflight.UnsupportedPartitionedParentError
if errors.As(err, &partitionErr) {
return c.emit(out, partitionedParentVerdict(st, partitionErr, forced))
}
return err
}
tier, err := preflight.RequiredTier(execSQL)
if err != nil {
return err
Expand All @@ -203,19 +226,6 @@ func (c *MigrateCmd) execute(ctx context.Context, out io.Writer, pool *pgxpool.P
}
logger.Debug("privilege preflight passed",
"role", priv.Role(), "owner", priv.Owner(), "tier", tier.String())

limit := int64(c.MaxTableSize)
if !sizeGuardApplies(plan, substituted) {
limit = preflight.NoSizeLimit
}
pt, err := preflight.CheckTable(ctx, pool, st.Schema(), st.Table(), limit)
var sizeErr *preflight.SizeError
if errors.As(err, &sizeErr) {
return c.emit(out, sizeGuardVerdict(st, sizeErr, forced))
}
if err != nil {
return err
}
logger.Debug("preflight passed",
"table", qualified(st), "total_bytes", pt.TotalBytes(), "limit_bytes", limit)
if substituted {
Expand Down Expand Up @@ -471,6 +481,20 @@ func privilegeVerdict(st statement.Statement, privErr *preflight.PrivilegeError,
}
}

// partitionedParentVerdict refuses unsupported execution steps on a
// partitioned parent before the sequence executor runs anything.
func partitionedParentVerdict(st statement.Statement, partitionErr *preflight.UnsupportedPartitionedParentError,
forced bool) verdict.Verdict {
return verdict.Verdict{
Outcome: verdict.OutcomeRefused,
Reason: verdict.ReasonUnsupportedPartitionedParent,
Statement: st.SQL(),
Table: qualified(st),
Forced: forced,
Detail: partitionErr.Error(),
}
}

// rewriteRequiredVerdict is the refusal for a statement whose submitted
// form blocks but for which the planner could not construct the safer
// native sequence — a multi-operation statement, or a pattern it cannot
Expand Down
Loading
Loading