Repository navigation
TML-3387: db verify and the migration planner compare column types as data types, not type names - #30777
Open
wmadden-electric wants to merge 69 commits into
Open
TML-3387: db verify and the migration planner compare column types as data types, not type names#30777wmadden-electric wants to merge 69 commits into
wmadden-electric wants to merge 69 commits into
Conversation
Signed-off-by: willbot <w.a.madden+machine@gmail.com> Signed-off-by: Will Madden <madden@prisma.io>
Signed-off-by: willbot <w.a.madden+machine@gmail.com> Signed-off-by: Will Madden <madden@prisma.io>
numeric(10) and numeric(10,0) values are now one value. DataTypeSpec gains an optional normalize, and sqlDataType passes its own. Signed-off-by: willbot <w.a.madden+machine@gmail.com> Signed-off-by: Will Madden <madden@prisma.io>
SqlColumnIR gains typeParams beside dataType, outside JSON and equality, and withDataType to copy a column with both. The contract side sets typeParams from the column's codec parameters. Signed-off-by: willbot <w.a.madden+machine@gmail.com> Signed-off-by: Will Madden <madden@prisma.io>
The SQL family's introspect resolves each column's reported type over the stack's declared data types and sets dataType and normal-form typeParams. A column no data type claims keeps no data type. Every caller of introspect gets resolved columns: db verify, db init, db update, migration plan and both migration runners. Signed-off-by: willbot <w.a.madden+machine@gmail.com> Signed-off-by: Will Madden <madden@prisma.io>
Signed-off-by: willbot <w.a.madden+machine@gmail.com> Signed-off-by: Will Madden <madden@prisma.io>
Signed-off-by: willbot <w.a.madden+machine@gmail.com> Signed-off-by: Will Madden <madden@prisma.io>
…stored types Signed-off-by: willbot <w.a.madden+machine@gmail.com> Signed-off-by: Will Madden <madden@prisma.io>
…written sql.normalize now applies the same check as the normal form a value stores, so contractToSchemaIR and values treat refused parameters one way. Signed-off-by: willbot <w.a.madden+machine@gmail.com> Signed-off-by: Will Madden <madden@prisma.io>
Rebuilding frozen schema IR nodes after introspection copied them without their constructors. Instead the family exports resolveIntrospectedColumnType, and the Postgres and SQLite control adapters call it for each column with the stack's declared data types, which their descriptors now pass in. The tree walk and SqlColumnIR.withDataType are deleted. Signed-off-by: willbot <w.a.madden+machine@gmail.com> Signed-off-by: Will Madden <madden@prisma.io>
SqlColumnIR gains storedType, and typeParams are that type's parameters in normal form. The contract side sets the type its codec's data type is stored as; the database side sets the type that claims the reported type. Signed-off-by: willbot <w.a.madden+machine@gmail.com> Signed-off-by: Will Madden <madden@prisma.io>
An adapter built without them would give no column a data type, which shows up later as false type mismatches. The constructor argument is now required. Tests build adapters through one helper per adapter package that passes the target's data types; the runtime adapters pass the target's data types for the marker reads they make. Signed-off-by: willbot <w.a.madden+machine@gmail.com> Signed-off-by: Will Madden <madden@prisma.io>
Signed-off-by: willbot <w.a.madden+machine@gmail.com> Signed-off-by: Will Madden <madden@prisma.io>
…apters, and assert storedType sqlDataType hands its checked normal form to the framework directly. The runtime adapters build their control adapter with the stack's declared data types. Both adapters' introspection tests assert storedType. Signed-off-by: willbot <w.a.madden+machine@gmail.com> Signed-off-by: Will Madden <madden@prisma.io>
sqlColumnTypesEqual says two columns have the same type when both have a stored type, the two are one data type, their parameters are equal as canonical JSON, and both or neither are lists. It replaces every comparison of type texts: SqlColumnIR.isEqualTo, both issue planners, the SQLite recreate postchecks and strategy, the Postgres convert and backfill strategies, safe widenings, and the alterColumnType refusal of an unchanged type. A column no data type claims never has the same type as another. A not-equal column explains itself: the two types as written and reported, or that no data type in the stack declares the reported type. The planners' column copies keep the type fields, and the SQLite plan keeps the column instances it is given. Tests that build introspected columns by hand resolve their type as introspection would. Signed-off-by: willbot <w.a.madden+machine@gmail.com> Signed-off-by: Will Madden <madden@prisma.io>
…s type text readPostgresDefault, readDefaultWithCodec and the Postgres adapter's default reading take the column's list flag instead of a type name with a [] suffix. A DefaultResolver gets the column's written type and list flag. The box check goes: no data type declares box, so a box column's default is never read with a codec. Signed-off-by: willbot <w.a.madden+machine@gmail.com> Signed-off-by: Will Madden <madden@prisma.io>
…column A PGlite journey creates columns with other spellings of the contract's types (character varying(20), timestamp(3) with time zone, numeric(10,0), character(1), a list, an enum outside public) and verifies them clean, with the enum's schema on and off the search path. A column changed by raw SQL, and one of a type no data type declares, are named with the type the database reports. An enum outside public against a column of the unbound namespace stays a known mismatch. A mismatch names a reported type of a kind with its schema, so that known mismatch reads app.status rather than status. Signed-off-by: willbot <w.a.madden+machine@gmail.com> Signed-off-by: Will Madden <madden@prisma.io>
…und 1 Signed-off-by: willbot <w.a.madden+machine@gmail.com> Signed-off-by: Will Madden <madden@prisma.io>
SqlColumnIR.withName copies every own property of the column, enumerable or not, through the constructor. Both working-schema renames and the SQLite plan's rebuild of a column stored under another key use it, so a field added later is not dropped. Signed-off-by: willbot <w.a.madden+machine@gmail.com> Signed-off-by: Will Madden <madden@prisma.io>
Signed-off-by: willbot <w.a.madden+machine@gmail.com> Signed-off-by: Will Madden <madden@prisma.io>
The Postgres and SQLite target test helpers only default the data types and call the sql-contract test helper. Their typecheck rootDir widens to packages/, as the adapters' already does, so their tests may import a test helper of another package. Signed-off-by: willbot <w.a.madden+machine@gmail.com> Signed-off-by: Will Madden <madden@prisma.io>
CatalogColumnType.dataType held information_schema.columns.data_type, text, where dataType elsewhere names a data type. Signed-off-by: willbot <w.a.madden+machine@gmail.com> Signed-off-by: Will Madden <madden@prisma.io>
renderColumnDdl and renderColumnAlterType on Postgres, and columnSpecFromNode and ddlColumnFromNode on SQLite, write a column node's type from its storedType and its codec's parameters as the contract writes them, without the codec and data type lookups. The DDL is unchanged. Signed-off-by: willbot <w.a.madden+machine@gmail.com> Signed-off-by: Will Madden <madden@prisma.io>
…alisers Nothing compares type names any more, so the normalised type text both sides carried for comparison goes: SqlColumnIR.resolvedNativeType and codecBaseNativeType, the column default node's codecBaseNativeType, normalizeSchemaNativeType, normalizeFormattedType, TYPE_NAME_ALIASES and WITH_TIME_ZONE on Postgres, and normalizeSqliteNativeType with its package export. introspectedNativeType keeps the reported text and whether the column is a list. An introspected column's type text is now the text the database reports, so db schema prints integer where it printed int4. Signed-off-by: willbot <w.a.madden+machine@gmail.com> Signed-off-by: Will Madden <madden@prisma.io>
A column's typeText is the text the database reports, on the introspected side, or the name the contract writes, on the contract side, for display and DDL only; nothing compares it. introspectedNativeType becomes introspectedTypeText. The db schema JSON output names the field typeText. Stale nativeType keys on codec descriptor objects in tests are removed. Signed-off-by: willbot <w.a.madden+machine@gmail.com> Signed-off-by: Will Madden <madden@prisma.io>
TypeLookups in @internal/framework-components/codec replaces SqlTypeLookups, MongoTypeLookups, contract-ts's local TypeLookups and AuthoringTypeLookups, which were the same pair. The stored value reader, readStoredValue and the value-object default input take it instead of two loose fields. Signed-off-by: willbot <w.a.madden+machine@gmail.com> Signed-off-by: Will Madden <madden@prisma.io>
…de tests leave Signed-off-by: willbot <w.a.madden+machine@gmail.com> Signed-off-by: Will Madden <madden@prisma.io>
…und 1 Signed-off-by: willbot <w.a.madden+machine@gmail.com> Signed-off-by: Will Madden <madden@prisma.io>
…renamed An extension fragment covers TypeLookups replacing SqlTypeLookups and MongoTypeLookups, the removed type-name normalisers, introspectedTypeText and informationSchemaType, SqlColumnIR's typeText and its removed fields, and the DefaultResolver and readPostgresDefault signatures. An app fragment covers the typeText key in db schema --json and the reported types in its tree output. Each change's detection is tested on the old and the new spelling. Signed-off-by: willbot <w.a.madden+machine@gmail.com> Signed-off-by: Will Madden <madden@prisma.io>
The DDL cast is described as a cast to the column's type, the uuid test says it depends on the codec's data type, and the schema-ir README says columns compare by their stored data type and parameters. Signed-off-by: willbot <w.a.madden+machine@gmail.com> Signed-off-by: Will Madden <madden@prisma.io>
…ernal import Signed-off-by: willbot <w.a.madden+machine@gmail.com> Signed-off-by: Will Madden <madden@prisma.io>
Signed-off-by: willbot <w.a.madden+machine@gmail.com> Signed-off-by: Will Madden <madden@prisma.io>
Signed-off-by: willbot <w.a.madden+machine@gmail.com> Signed-off-by: Will Madden <madden@prisma.io>
…smatch names both sides alike Signed-off-by: willbot <w.a.madden+machine@gmail.com> Signed-off-by: Will Madden <madden@prisma.io>
…nges any column Signed-off-by: willbot <w.a.madden+machine@gmail.com> Signed-off-by: Will Madden <madden@prisma.io>
…de detection test Signed-off-by: willbot <w.a.madden+machine@gmail.com> Signed-off-by: Will Madden <madden@prisma.io>
…side storage types Signed-off-by: willbot <w.a.madden+machine@gmail.com> Signed-off-by: Will Madden <madden@prisma.io>
…dataType is gone Signed-off-by: willbot <w.a.madden+machine@gmail.com> Signed-off-by: Will Madden <madden@prisma.io>
…rt from the codec's typeParams Signed-off-by: willbot <w.a.madden+machine@gmail.com> Signed-off-by: Will Madden <madden@prisma.io>
Signed-off-by: willbot <w.a.madden+machine@gmail.com> Signed-off-by: Will Madden <madden@prisma.io>
…ct columns' parameters in normal form Signed-off-by: willbot <w.a.madden+machine@gmail.com> Signed-off-by: Will Madden <madden@prisma.io>
…ds it Signed-off-by: willbot <w.a.madden+machine@gmail.com> Signed-off-by: Will Madden <madden@prisma.io>
Signed-off-by: willbot <w.a.madden+machine@gmail.com> Signed-off-by: Will Madden <madden@prisma.io>
Signed-off-by: willbot <w.a.madden+machine@gmail.com> Signed-off-by: Will Madden <madden@prisma.io>
Signed-off-by: willbot <w.a.madden+machine@gmail.com> Signed-off-by: Will Madden <madden@prisma.io>
…mn with no declared type Signed-off-by: willbot <w.a.madden+machine@gmail.com> Signed-off-by: Will Madden <madden@prisma.io>
…ans nothing, on PGlite and SQLite Signed-off-by: willbot <w.a.madden+machine@gmail.com> Signed-off-by: Will Madden <madden@prisma.io>
…red data types Signed-off-by: willbot <w.a.madden+machine@gmail.com> Signed-off-by: Will Madden <madden@prisma.io>
Signed-off-by: willbot <w.a.madden+machine@gmail.com> Signed-off-by: Will Madden <madden@prisma.io>
…SQLite type Signed-off-by: willbot <w.a.madden+machine@gmail.com> Signed-off-by: Will Madden <madden@prisma.io>
Signed-off-by: willbot <w.a.madden+machine@gmail.com> Signed-off-by: Will Madden <madden@prisma.io>
…87-verify-through-data-types Signed-off-by: willbot <w.a.madden+machine@gmail.com> Signed-off-by: Will Madden <madden@prisma.io> # Conflicts: # docs/reference/error-reference.md # packages/1-framework/1-core/framework-components/test/temporal-presets.test-d.ts # packages/2-mongo-family/1-foundation/mongo-contract/src/mongo-data-type.ts # packages/2-sql/1-core/schema-ir/src/ir/sql-column-default-ir.ts # packages/2-sql/1-core/schema-ir/src/ir/sql-column-ir.ts # test/integration/test/planner-golden/manifest.json
…with no declared type Signed-off-by: willbot <w.a.madden+machine@gmail.com> Signed-off-by: Will Madden <madden@prisma.io>
Signed-off-by: willbot <w.a.madden+machine@gmail.com> Signed-off-by: Will Madden <madden@prisma.io>
@prisma/orm-extension-arktype-json
@prisma/orm-extension-middleware-cache
@prisma/orm-extension-paradedb
@prisma/orm-extension-pgvector
@prisma/orm-extension-postgis
@prisma/orm-extension-supabase
@prisma/orm-family-mongo
@prisma/orm-family-sql
@prisma/orm-framework
@prisma/orm-mongo
@prisma/orm-postgres
@prisma/orm-sqlite
@prisma/orm-target-mongo
@prisma/orm-target-postgres
@prisma/orm-target-sqlite
@prisma/orm-toolchain
commit: |
Contributor
size-limit report 📦
|
Signed-off-by: willbot <w.a.madden+machine@gmail.com> Signed-off-by: Will Madden <madden@prisma.io>
The adapter control entries and the ./target aggregate loaded it only because the bundler placed code they share with the target control entry inside that entry's file. The slice's new imports moved the shared code into a common chunk, so they no longer import the control entry. No application entry loads the polyfill, and every control stack still loads the target control entry, which sets the fallback Temporal. Signed-off-by: willbot <w.a.madden+machine@gmail.com> Signed-off-by: Will Madden <madden@prisma.io>
Contributor
|
Important Review skippedToo many files! This PR contains 328 files, which is 28 over the limit of 300. To get a review, reduce the PR to 300 files or fewer by splitting it into smaller PRs or changing its base branch. Usage-priced reviews support at most 300 files. ⚙️ Run configuration
⛔ Files ignored due to path filters: 15
📒 Files selected for processing: 328
You can disable this status message by setting the
Comment |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A contract column and the table the database already has:
db verifyon this database:varchar(20)) and the database's (character varying(20)) are pushed through hand-written tables of aliases until the strings matchpg/varcharwith{ length: 20 }, and comparednumeric(10)againstnumeric(10,0){ precision: 10, scale: 0 }point)The database column is "point", which no data type in the configured stack declares.The decision
db verifyand the migration planner decide whether two columns have the same type by comparing their data types, not their type names. A column's data type is the database type Prisma declares for it (pg/varchar,pg/numeric,pgvector/vector), with its parameters. The database side gets its data type by matching what the database reports against the names each data type declares. This finishes the "Data types own column types" project's work on reading a database: TML-3582 did it forcontract infer, and this does it for verify and the planner.How it works
The database side gets a data type. Each data type already declares the names a database reports for it:
pg/int4claimsintegerandint4,pg/varcharclaimscharacter varying(n), andpg/enumclaims every type the database lists as an enum. When either adapter introspects a column, it hands what the database reports to one function in the SQL family, which finds the data type that claims it and reads its parameters. That function contains no target code; the adapters pass it the data types of the configured stack, so a pack's types (pgvector, PostGIS) are found the same way as the target's.Both sides carry the same three things. A schema column now carries the data type it is stored as, its parameters in that type's normal form, and whether it is a list. The contract side takes these from the column's codec; the database side from the resolution above. "Stored as" matters on SQLite, where a
DateTimecolumn's data type is stored as SQLitetext, andtextis what the database reports.One rule decides "same type". Two columns have the same type when they are stored as the same data type, with equal normal-form parameters, and are both lists or both not. That rule replaces seven separate string comparisons in verify, both planners, the SQLite table-rebuild checks and the Postgres safe-widening check. A column whose type no data type claims never equals another, and its mismatch names the type as the database reported it.
Enums no longer depend on the search path. An enum's name now comes from the catalog's schema and name, not from
format_type, which qualifies the name only when its schema is off the search path. An enum in schemaappcompares equal whether or notappis on the path.The check after a type change uses the same declarations. After
ALTER COLUMN TYPE, the Postgres planner checks the new type: an enum by identity (atttypid = to_regtype(...)), every other type by comparingformat_typewith the name its data type says the catalog prints.The old string machinery is deleted. The alias tables (
normalizeSchemaNativeType,TYPE_NAME_ALIASES,WITH_TIME_ZONE,normalizeFormattedType,normalizeSqliteNativeType), the fields that held normalised names (resolvedNativeType,codecBaseNativeType) and the unuseddataTypefield are gone. A column's display name is nowtypeText, which nothing compares. Four identical types that paired a codec lookup with a data type lookup are now one,TypeLookups.The Prisma 7 schema reader names data types too. Its table of Prisma 7 types now names the data type and parameters Prisma 7 creates, and takes the constructor from the data type. Its output is unchanged.
What users see
db verifyis clean for any spelling the database uses for a type, and a type no configured pack declares is reported by name.db schemaprints the type names the database reports (integerwhere it printedint4), and its--jsonoutput names the column fieldtypeTextinstead ofnativeType.CONTRACT.DEFAULT_INVALIDistypeTextinstead ofnativeType.Upgrade notes are in
upgrade-instructions/pending/column-types-through-data-types/. Applications: the two output changes above. Extension authors: the renamed and removed exports, the two control adapters' constructors, which now take the stack's data types, and how to build a schema column by hand.No committed contract, migration or snapshot changes, and no DDL changes.
How it was checked
int,character varying(20),timestamp(3) with time zone,numeric(10,0),character(1),double precision, a two-dimensional array, an enum in another schema off the search path, pgvector installed in anextensionsschema) verifies clean, and the planner plans nothing. A committed test does this on PGlite and SQLite.ALTER COLUMN TYPEcheck is tested by changing a column to every Postgres data type, as a column and a list, and enums in and outsidepublic.projects/data-types-completion/slices/4/manual-qa.md): the above against PostgreSQL 17, SQLite and PostGIS; every committed example verified against its own database.Not in this pull request
mainalso has: TML-3586 (adopting a database whose PostGIS extension already exists), TML-3637 (db schemashows onlypublicand drops[]), TML-3638 (migration planre-planning applied changes, not yet checked onmain), TML-3639 (db verifycreating an empty SQLite file).publicstill does not compare equal; this is a documented limit.Tickets
Linear TML-3387, the fourth part of the "Data types own column types" project.
Alternatives considered
DateTimecolumn's codec type issqlite/datetime, but the database stores and reportstext, so the comparison must use the type the column is stored as.contract inferdoes. A bare@db.Charwould then store no length and change the contract hash of every Prisma 7 project that uses it. The reader builds a contract, so it keeps the parameters Prisma 7 created.Agent: frigg-97
🤖 Generated with Claude Code