schema: remove a constraint from a property that stays #1064

Activity

1. ragnorc

   added

   needs-triageAwaiting maintainer triageAwaiting maintainer triage

   featureFeature proposalFeature proposal

   on Oct 18, 2026
2. ragnorc commented on Oct 20, 2026

   

   ragnorc

   on Oct 20, 2026 – with Cursor

   ContributorAuthor

   More actions

   Summary

   schema plan refuses every removal of a constraint from a property that stays. The only supported way to lose the constraint is to drop the property, or to rebuild the graph. That refusal is still present on current main. No pull request implements the removal.

   The accepted schema is the source of truth for constraints and for indexes. A removal has to make the published snapshot match that schema. Keeping the Lance index after the declaration is gone leaves plans, optimize, and full-text search on physical state the schema no longer describes.

   Reproduction

   Checked again immediately before this comment. Issue #1064 is open. It has no comments. The cross-references are PR #1062 (merged, and it leaves this case refused) and issue #1048 (change feeds across schema changes). Neither one implements constraint removal.

   • Main SHA: b692610c4aa31066cb6a9fd34830f56977288943 (b692610c, 2026-10-20). origin/main matches this SHA.
   • Issue evidence SHA: 764fb422db1d8f7f64b99379dd5d01797449fff1. At that commit the Lance pin was =11.0.0. Current main pins =13.0.0 (PR build(storage): upgrade lance to 13.0.0 #1072).
   • Host: Linux 6.12.94+, x86_64.
   • Toolchain in the checkout: rustc 1.97.1, from rust-toolchain.toml.
   • CLI: target/debug/omnigraph from this checkout, omnigraph 0.14.0, internal-schema 16.
   • Graphs were empty local directories. schema plan does not need rows. Index build does, so this run did not create a Lance index.

   Commands, for each case:

   omnigraph init --schema base.pg <graph>
   omnigraph schema plan --schema next.pg --target <graph>
   omnigraph schema plan --schema next.pg --target <graph> --json
   omnigraph schema apply --schema next.pg --target <graph> --json
   omnigraph schema show --target <graph>
   

   schema plan exits 0 when supported is false. schema apply exits 1, writes nothing, and schema show still prints the base schema. Apply reports only the first unsupported step. The second step is visible in the plan, not in the apply error.

   Removal	Plan steps	Apply error	Verdict
   email: String? @unique kept, @unique removed	2: property constraints on Doc.email, then constraints on Doc	first message only	Reproduced. Also two refusals, same shape as bare @index.
   @range(age, 0..200) removed, age kept	1: constraints on Doc	that message	Reproduced. One refusal.
   @check(code, "[A-Z]+") removed, code kept	1: constraints on Doc	that message	Reproduced. One refusal.
   Edge w: I64? @index removed, w kept	2: property constraints on E.w, then constraints on E	first message	Reproduced. Two refusals.
   Composite @index(a, b) removed, a and b kept	1: constraints on Doc	that message	Reproduced. One refusal.
   score: I64 @index removed, score kept	2: Doc.score, then Doc	first message	Reproduced. This is the double refusal in the issue.
   v: Vector(3)? @index removed, v kept	2: Doc.v, then Doc	first message	Reproduced.
   body: String? @index removed, body kept	2: Doc.body, then Doc	first message	Reproduced.
   slug: String @key kept, @key removed	2: Doc.slug, then Doc	first message	Reproduced. @key stays refused.
   status: enum(...) @index, when: Date? @index, flag: Bool? @index	2 each	first message	Same planner refusal as a node scalar.
   Type-level @index(score) with no property annotation	1: constraints on Doc	that message	Extra case. One refusal. Same physical BTREE as property @index once the table has rows.

   Exact plan text for score: I64 @index:

   schema plan for <graph>
   supported: no
   - unsupported change on node:Doc.score: removing property constraints from 'Doc.score' is not supported in schema migration v1
   - unsupported change on node:Doc: removing constraints from 'Doc' is not supported in schema migration v1
   

   JSON kind is unsupported_change. code is absent. Apply returns code: bad_request and only removing property constraints from 'Doc.score' is not supported in schema migration v1.

   The double refusal is the property annotation plus the type constraint that the parser copies from it (desugar_property_constraints). @range and @check are type constraints only, so they produce one step. Composite @index(a, b) is also type-level only.

   A .gqt case cannot express schema plan. The cheapest existing Rust test is plan_drops_a_property_with_the_constraints_that_name_only_it_issue_1014 in crates/omnigraph-compiler/src/catalog/schema_plan.rs. On this SHA it passes. Its row @index removed from a property that stays expects supported == false and a reason that starts with removing constraints from. It does not assert the property-level step.

   A temporary compiler test expected one supported step for removing @unique from a kept property. It failed, then the edit was removed. Nothing was pushed.

   plan_removes_a_metadata_constraint_from_a_property_that_stays_issue_1064 FAILED
   SchemaMigrationPlan { supported: false, steps: [
     UnsupportedChange { entity: "node:Doc.email",
       reason: "removing property constraints from 'Doc.email' is not supported in schema migration v1", code: None },
     UnsupportedChange { entity: "node:Doc",
       reason: "removing constraints from 'Doc' is not supported in schema migration v1", code: None }] }
   plan_drops_a_property_with_the_constraints_that_name_only_it_issue_1014 ... ok
   

   Command: cargo test -p omnigraph-compiler --lib issue_10.

   What changed since 764fb42

   The refusal text and the two functions are unchanged. Line numbers moved.

   • plan_property_constraints and plan_constraints are still the two refusal sites (schema_plan.rs lines 787-795 and 857-865).
   • PR fix(schema): drop a property together with the constraints that name only it #1062 (merged 2026-10-18) drops a constraint when every property it names is dropped. Its tests still expect this issue's case to be unsupported. The is_key comment in validate.rs now says the planner refuses every key change. The skip itself is unchanged (validate.rs lines 329-334).
   • PR feat(query)!: refuse full-text calls without a full-text index #1066 (merged 2026-10-18) added T27 and closed Unindexed full-text search is case-sensitive #747. At 764fb42, T27 did not exist. A full-text call on a free-text String with no @index is now a compile error. RFC 0047 step 3 is on main for that refusal.
   • PR build(storage): upgrade lance to 13.0.0 #1072 (merged 2026-10-20) moved the Lance pin from 11.0.0 to 13.0.0.
   • engine/plan_source/indexes.rs gather is unchanged from 764fb42.
   • can_fold_index has the same predicate. It moved from line 2694 to line 2709.
   • The composite skip moved from table_ops.rs line 799 to line 751. The edge comment moved from line 855 to line 805.
   • The merge fast path still starts at exec/merge.rs line 3451. The range, check, and unique checks are lines 3475-3477. The function body matches 764fb42.
   • FullTextCoverage::of_facts has the same rule. of_table is new: a table with zero fragments is Full. The trait default changed from Absent to Partial. Neither change reads the schema declaration.

   Root cause

   plan_property_constraints treats any removed property constraint shape (@key, @unique, or @index) as unsupported. plan_constraints treats any removed ConstraintIR that does not end with a dropped property as unsupported. A property annotation is stored in both places, so one edit produces two steps.

   There is no DropConstraint variant. SchemaMigrationStep::AddConstraint exists. Apply handles AddConstraint as catalog-only (schema_apply.rs lines 412-425). Apply has no path that removes an index from a column that stays. A property drop removes that column's index because Lance retain_relevant_indices drops an index on a dropped field. That path does not run when the field stays.

   Physical-state table

   Issue row	Verdict on this SHA	Evidence
   @unique, @range, @check are catalog rules. A removal makes later writes less strict. No stored row becomes invalid. The pure-insert fast path only widens.	Reproduced in code. Not run as a merge.	constraints_for emits non-key @unique with is_key: false, so the committed lookup runs (validate.rs lines 313-322 and 1186-1199). Range and check are catalog constraints. plan_index_work_node does not build an index for them. proven_fast_forward_needs_no_validation returns true only when range, check, and unique lists are empty (merge.rs lines 3475-3477).
   Edge @index has no user index.	Reproduced in code.	plan_index_work_edge_on_dataset builds only the id, src, and dst BTREEs (table_ops.rs lines 805-836).
   Composite @index(a, b) has no physical index.	Reproduced in code.	index_cols.len() != 1 skips the declaration (table_ops.rs lines 750-753).
   Node scalar, enum, date, or bool @index is a Lance BTREE.	Partially reproduced.	PropType::index_kind returns Btree for those scalars, and for DateTime (types.rs lines 242-258). The builder runs only when the table has rows (table_ops.rs line 732). These graphs were empty, so no BTREE was created.
   Node Vector @index is a Lance vector index. nearest then uses the exact scan.	Partially reproduced.	index_kind returns Vector. Empty graphs built no index. The planner always attaches NearestPolicy::DEFAULT. Execution calls Lance nearest and sets use_index(false) on the exact pass (scan.rs). This run did not execute nearest.
   Free-text String @index is an inverted index whose segment carries the analyzer.	Partially reproduced.	index_kind returns FullText for a non-enum String. ensure_indices and the untrained-segment declaration read declared indexes. This run did not build a segment. docs/dev/lance.md still says a segment applies its analyzer to rows it does not cover, and a column with no segment uses Lance's bare tokenizer.
   @key must stay refused. The is_key skip depends on a key that exists from type creation.	Reproduced.	Plan refuses. validate.rs lines 329-334 document the skip.

   Why a kept index is the wrong result

   1. Plans read physical indexes. gather loads dataset.load_indices() and does not read the catalog (indexes.rs lines 13-29). Scan access and validate-access use those facts. FullTextCoverage::of_facts counts an Inverted fact on the column (source.rs lines 318-336). A BTREE the schema no longer declares still narrows scans. An inverted index the schema no longer declares still counts as coverage.
   2. optimize keeps the cost. can_fold_index excludes system indexes, indexes with no details, and full-text indexes (table_store.rs lines 2709-2710). It does not exclude an undeclared BTREE or vector index. A lagging one is rebuilt on optimize.
   3. Full-text becomes a second source of truth. ensure_indices and the full-text rebuild walk declared indexes (table_ops.rs lines 750-767 and the declared-column filter near line 693). can_fold_index does not fold full-text. A leftover segment keeps the analyzer, and nothing in the schema maintains it.

   Draft RFC 0046 still reports declared: false for an index dropped by a schema change, and it says the fold pass maintains undeclared foldable indexes. That report shows the drift. It does not remove the drift.

   Lance

   Lance does not fix this issue. The refusal is in the OmniGraph planner. The substrate can already express the index removal.

   Pinned Lance is crates.io 13.0.0, the newest stable release (published 2026-10-07, tag v13.0.0). There is no stable 13.0.1, 13.1, or 14. No open OmniGraph pull request upgrades Lance. v14.0.0-beta.21 exists. It is not a stable release.

   Dataset::drop_index in Lance 13.0.0 builds Operation::CreateIndex { new_indices: [], removed_indices } and commits it with apply_commit, which is a linear commit (index.rs on v13.0.0). The same function body is in v14.0.0-beta.21. The transaction spec says CreateIndex adds, replaces, or removes secondary indexes. Both new_indices and removed_indices are repeated fields.

   The manifest builder removes indexes by UUID and accepts an empty new_indices list (lance-table manifest_build.rs, CreateIndex arm). A Lance test commits that shape on the linear history (dataset_transactions.rs, the covering-fence case). Detached commit uses that same manifest build. do_commit_detached_transaction does not reject CreateIndex. OmniGraph commit_detached (table_store.rs line 826) already commits a transaction with with_detached(true).

   This is support in the commit path. It is not a test that a detached index-only removal opens and keeps the rows. Graph tables must not call drop_index, because that moves linear HEAD.

   Open Lance PR lance-format/lance#6806 adds an uncommitted transaction for existing index segments. It does not drop an index, and it does not change drop_index. Lance PR #9481 (in the v14 betas, not in 13.0.0) changes the BTREE row domain. It does not change the removal transaction. Later index commits on index.rs after v13.0.0 are full-text finalize, distributed optimize, coverage tracing, and remap. None of them remove the need for OmniGraph to plan the constraint change.

   Design

   Keep the three-step split. Change one classification. Step 1 must follow physical backing, not the annotation syntax.

   A type-level @index(score) on a node scalar is the same BTREE as score: I64 @index. If step 1 drops only the declaration, the BTREE stays, and the failure mode in the issue returns. Edge @index, composite @index, @unique, @range, and @check have no physical index. Those are step 1. A one-column node BTREE or vector index is step 2, in either syntax. A one-column free-text String @index stays refused. @key stays refused in both directions.

   Emit one DropConstraint step. Plan it from ConstraintIR, which is the stored constraint. The property-constraint shape is a copy of @key, @unique, and @index. If both passes still run, the double refusal remains. AddConstraint is the mirror, and apply already records it in the catalog with no table write.

   Re-adding @unique, @range, or @check stays refused. That work is #721, because a new rule has to be proved against existing rows. The plan line and the schema guide should say the removal cannot be undone by apply. Re-adding @index stays allowed. AddConstraint already plans it. An index is not a one-way door. A rule is, until #721.

   This one-way door is the same class as a property drop. Older commits keep the old contract. The new contract accepts rows the old rule rejected. That is the user need. An empty-table-only removal is safer and does not meet that need.

   Step 2 stages one detached CreateIndex with an empty new_indices list and every segment of that index name in removed_indices. Pin that version in the one __manifest publication. If the same apply also changes columns, chain the index removal after the Project and Merge commits, as schema apply already chains those commits (schema_apply.rs lines 821-841). Do not call Dataset::drop_index. cleanup reclaims the index directory when no retained version names it. Older commits keep their index.

   The crash window is the existing schema-apply window. A detached commit before publication is unreachable. A crash after publication leaves the pin without the index. This is a new effect inside the schema-apply writer, so the detached-commit matrix, the failpoint, and the DST cost golden need a row. The current matrix has a schema-apply writer. It does not have an index-removal transaction.

   Vector removal changes cost. nearest no longer has that index, so the scan is exact. Recall can rise. The plan line should say that.

   Step 3 stays refused until the analyzer is part of the schema (#792). T27 is already on main. A query that calls search, match_text, fuzzy, or bm25 on a property with no @index fails at compile time. That removes the silent tokenizer change for a new query. Two gaps remain.

   • Cluster deploy typechecks stored queries against the desired catalog (validate_query_source). A removed full-text @index that a stored query still searches fails query_typecheck_error before publish. Standalone schema apply has no stored-query registry. A later server start recompiles stored queries and quarantines the graph when one fails (RFC 0047, Operators). When a registry is present, the plan should list those queries before apply. Cluster plan already has the typecheck. Standalone apply should say that it cannot see stored queries.
   • rfc: analyzed lexical search on the shared expression model #792 binds the analyzer on the field. After that, removal changes cost only, and it can join step 2. Until then, the segment is the only record of the analyzer.

   #963 is an open RFC for schema apply with live branches. It records today's fact that removing an @index is refused, and that a non-full-text index has no table effect. Its stored signature includes full-text @index and excludes other indexes. A branch keeps its own contract and pin. That stays consistent.

   Step 2 breaks the "no table effect" sentence. The index removal does not change rows, so #963's live-branch check would not wait for a branch. A later merge treats the table as uniform, because the signature ignores the BTREE or vector index. The merge can then publish a version that still has the index under a contract that does not declare it. optimize would fold that index again.

   Amend #963 before step 2, or in the same change. The stored signature should include declared node BTREE and vector indexes, as it already includes full-text indexes. A removal then makes the table migrated, and the merge takes the newer side's version, which is the version apply published without the index. Step 1 does not need that signature change. A removed rule changes the contract sequence, and validation uses the newer contract. The rows were valid under the stricter rule.

   #1015 plans after files are composed into one schema. DropConstraint should use the canonical type name. It does not need a second constraint store.

   #884 keeps one table per concrete node. Interface properties are not planned through plan_constraints today. Do not add an interface-level physical index. If a later change copies an interface @index onto each implementer, the drop has to name each concrete table.

   RFC 0046 should say that schema apply does not publish a contract that drops an index while the pinned version still has that index. declared: false remains the report for an index that was created outside the engine, and for a crash window that has not been published.

   render_schema_plan_step is an exhaustive match (output.rs). A new variant does not compile until that arm exists. The human line should name the constraint, and for a rule it should say that apply cannot add it back. For a vector index it should say that nearest becomes an exact scan. cluster plan prints each step as JSON (write_cluster_plan_human), so the new kind appears there without a new renderer. Operators will read raw JSON until that path uses the same line.

   SchemaApplyOutput.steps is annotated as Vec<Value> (omnigraph-api-types). openapi.json does not gain a variant. A client that matches on kind needs the new variant. A client that ignores an unknown kind keeps working. Rust deserializes the enum, so an old Rust client fails on the new kind.

   Alternatives

   • Keep the refusal. PR fix(schema): drop a property together with the constraints that name only it #1062 already documents it. Users keep a wrong rule and an unused index for the life of the type.
   • Remove the declaration and keep the index. Plans still use it. optimize still rebuilds a BTREE or vector index. Full-text search keeps an analyzer the schema does not name.
   • One rule that always drops whatever index exists, including full-text. Search results change, and invariant 7 says missing coverage may change cost and must not change correctness. T27 now makes the query fail instead of changing rows, but the analyzer is still lost.
   • Ship steps 1 and 2 together. Step 2 adds a crash window to schema apply. Step 1 is planner-only.

   The split is the lower-liability sequence. Step 1 removes the stuck rule. Step 2 removes the stuck index, with the recovery tests the write path already requires. Doing the physical drop in the same change as the catalog rule adds storage liability before the planner behavior is settled.

   What this deprioritizes: the speed of dropping the index. Cleanup deletes the index files later, when no retained version names them. That is the same retention rule as a dropped column. Correctness of one publication comes first. Efficiency of immediate file deletion does not.

   Test plan

   • Compiler: one supported DropConstraint for @unique, @range, @check, an edge @index, and a composite @index. Assert one step. Assert @key removal stays unsupported. Assert re-adding @unique, @range, and @check stays unsupported. Assert a node scalar @index, a type-level @index(score), a vector @index, and a free-text @index stay unsupported until their own step.
   • CLI: the human line for DropConstraint, including the one-way sentence for a rule.
   • Cluster: cluster plan JSON contains the new kind. A stored search() on a property whose @index is removed fails typecheck before apply.
   • Step 2: after apply, the pinned dataset has no user BTREE or vector index for that column, and the id BTREE remains. The previous snapshot still has the index. ensure_indices does not rebuild it. optimize does not fold it. A failpoint before publication leaves the graph pin and the index unchanged. Add the schema-apply index-removal cell to detached_commit_matrix.rs and the DST storage-cost golden row.
   • Vector: a search test that nearest is exact after the index is gone.
   • Full-text: the plan stays unsupported. Do not add a case that drops the segment.

   Related work

   • bug: schema plan refuses to drop a property that carries @index #1014, closed by PR fix(schema): drop a property together with the constraints that name only it #1062. A drop takes constraints that name only dropped properties. This case stays refused on purpose.
   • schema: support adding required, unique properties to existing types #721, open. Adding a required or @unique property. Re-adding a rule belongs there.
   • Unindexed full-text search is case-sensitive #747, closed by PR feat(query)!: refuse full-text calls without a full-text index #1066 as completed. T27 refuses unindexed full-text calls.
   • Index state changes which rows a text query matches #749, open. Index state still changes which rows fuzzy matches. rfc: analyzed lexical search on the shared expression model #792 is the fix. It is another reason to keep free-text removal refused.
   • PR rfc: analyzed lexical search on the shared expression model #792, open RFC. The analyzer becomes a field property. After that, free-text removal can join step 2.
   • PR rfc: schema apply with live branches #963, open RFC. Schema apply with live branches. It assumes this removal stays refused. Step 2 needs the signature change above.
   • PR rfc: schema namespaces and multi-file schemas #1015, open RFC. Multi-file schemas and namespaces. Planning stays on the composed schema.
   • PR rfc: polymorphic types with interfaces, unions and polymorphic edges #884, open RFC. Polymorphic types. Index removal stays on the concrete node table.
   • RFC 0046, index status, including declared: false.
   • RFC 0047, search plan validation. Step 3 (T27) is on main. The rollout table in the RFC still lists step 3 as the step that closes Unindexed full-text search is case-sensitive #747.
   • Issue changes: continue change feeds across additive schema changes and outside the request's type filter #1048 cross-references schema: remove a constraint from a property that stays #1064. It is about change feeds across additive schema changes. It does not replace this issue.

   No merged or open pull request removes a constraint from a property that stays.

   Supersession

   No open issue has the same need with a wider or more accurate scope. #1014 was the property-drop case and is closed. This comment does not close #1064.

   Limitations

   • The graphs were empty. No BTREE, vector index, or full-text segment was built, and no nearest or search query was run.
   • The detached index-only commit was read in Lance 13.0.0 source. It was not executed.
   • The temporary failing test was deleted after the run. The tree was not changed.
   • Lance pages used for this note are the transaction spec's CreateIndex section and the pinned 13.0.0 sources. The v14 comparison is the drop_index function body, not a full diff of Lance.