Skip to content

polis-api: the database tests also run on the shape of an installation whose database predates the migrations, and four reads no longer depend on the migrations' shape - #3393

Merged
colinmegill merged 1 commit into
edgefrom
polis-api/tests-on-older-database-shape
Nov 10, 2026
Merged

colinmegill merged 1 commit into
edgefrom
polis-api/tests-on-older-database-shape

Conversation

@colinmegill

Copy link
Copy Markdown
Member

Observable changes

None for any response served today, on any installation.

  • In a database built from the migrations every changed read returns the bytes it returned before. The statements select the same rows; the two reads whose key order now follows the stored column order produce the migrations' order there, and one of them (GET /api/v3/participants) still goes through the contract's own encoding in that case.
  • In an installation whose database predates the migrations, the four reads below could not be served by polis-api at all: two statements were refused by PostgreSQL and two handlers handed every non-empty answer back, so Node answered, as it does today. With this change polis-api can answer them, with the bytes Node gives on that installation. None of the four routes is switched on by this change; switching one on remains a separate decision.

Why

An installation that was in service before the migrations existed has the same tables in a slightly different shape: its math documents are json where the migrations build jsonb, some tables keep their columns in another order, a few columns exist on one side only. Every database test of this crate builds its tables the migrations' way, so a statement that works only on that shape passes every test and fails on the older installation. One such statement in the vote writer was fixed in PR #3390. This change makes the test suite able to see the rest.

What changes

A second pass of the database tests. make check-polis-api (the entry point hosted CI calls, ci/local/polis-api.sh) now runs cargo test --locked -p polis-api --features db-tests twice: once as before, and once with queue-rs/polis-api/tests/older_database_shape.sql installed on the test server. That file records, as ordinary SQL, the columns of the thirteen tables that differ and that the API reads (users, conversations, participants, participants_extended, comments, votes, xids, reports, site_domain_whitelist, math_main, math_ticks, math_bidtopid, math_ptptstats), and an event trigger that reshapes a table created under one of those names. In every such table a jsonb math document becomes json. A table declared with every column the first migration gives it (the databases the vote writer and treevite tests build from the migration files, and two read fixtures) also takes the older column order, the columns only an older installation has, and its nullability. Most tables the tests declare are narrower and keep the order the test gave them. The tests themselves are unchanged and unaware of it. The file's header and docs/local-checks.md say exactly what is reproduced and what is not. A small program (examples/older_database_shape.rs) installs and removes it; the entry point removes it before the first pass and after the second, also when the tests fail, and one test in each pass checks that its databases really have the shape the pass names.

Four reads fixed (each found failing by the second pass, each with a test that fails on the old code):

Read What depended on the migrations' shape Now
Next-comment priorities (next_priorities.rs) jsonb_typeof(data): no such function for a json column The type is taken from the column in the statement itself, as PR #3390 did for the vote writer; nothing is converted; any other column type is still refused
Whole math generation, used by the math membership routes (math_bundle.rs) data <> 'null'::jsonb: no such comparison for a json column Asks the column's own type whether the document is a JSON null; any other column type is refused
GET /api/v3/votes/me (votes_read.rs) Required the columns of votes in the migrations' order, and handed back otherwise The answer's keys follow the stored column order, as Node's SELECT * rows do; another set of columns is still handed back
GET /api/v3/participants (participants_read.rs) The same, for participants The same

For the last two, Node's own bytes differ between the two shapes, because Node answers with its SELECT * row and JSON keys keep the columns' order. Where votes stores created before weight_x_32767, Node answers {"pid":…,"tid":…,"vote":…,"created":…,"weight_x_32767":…,"high_priority":…,"weight":null,"conversation_id":…}; where the migrations built the table, weight_x_32767 comes before created. polis-api now does the same on each.

Two existing tests loaded a full-width table by position; their INSERT statements now name their columns. No test was weakened, skipped or removed, and none had to be marked for one shape only.

What was proved

On PostgreSQL 17.11, pinned Rust, RUST_TEST_THREADS=2:

  • The unchanged suite on the older shape, before any fix: 518 passed, 14 failed, 1 ignored. Eleven failures were the next-comment statement, one the math generation statement, and two were the position-loaded fixtures in front of the two column-order reads.
  • With this change, cargo test --locked -p polis-api --features db-tests: 538 passed, 0 failed, 1 ignored on the migrations' shape, and the same 538, 0, 1 on the older shape (tests/key_boot: 3 passed in each). 532 of them are the tests that were there before; six are new.
  • cargo test --locked -p polis-api: 326 passed, 0 failed (tests/key_boot: 3 passed).
  • The new tests and the two repaired ones against the old statements: on the older shape all five fail, each with the statement's own error (function jsonb_typeof(json) does not exist, operator does not exist: json <> jsonb, a hand-back); on the migrations' shape the three tests that set the type or the order themselves fail and the two existing ones pass.
  • The shape file against the migrations: every migration applies with it installed; the thirteen tables come out with the recorded columns, order, types and nullability (two documented exceptions in the last columns of conversations and participants_extended); constraints, indexes, defaults, owned sequences, triggers, rules and functions are otherwise identical to a database built without it. This is a test in the suite and runs in both passes.
  • The route conformance runners, which compare complete HTTP bytes from the unchanged Node server and from polis-api on one generated stack: votes-read-run.sh: 334 requests, 169 answered by polis-api with Node's exact bytes, 165 handed to Node as the runner expects, 0 differences, on a database built by the migrations and again on one built with the older shape installed before the migration runner ran. participants-read-run.sh: 138 cases, 112 exact, 26 handed back, 0 failed, on both. math-membership-run.sh: 192 cases, 129 exact, 63 handed back, 0 failed on the migrations' shape; on the older shape the runner's own fixture inserts math_ticks.caching_tick, a column that shape lacks (math-membership-replay.cjs:142), so it was run there with that one column kept, with the same 192, 129, 63 and 0. In the recordings Node's body differs between the two shapes in 31 of the 334 vote-read requests and in 69 of the 138 participant requests, by key order only; polis-api answers 15 and 56 of those, with Node's bytes on each shape.
  • cargo clippy --locked -p polis-api --all-targets -- -D warnings, with and without --features db-tests: clean. cargo fmt -p polis-api --check: clean. make check-vote-sign-lint: 32 tests pass, 581 sites, 0 new.

The second pass adds about 1 minute 50 seconds to the check on a 14-core machine (the first pass takes the same); it compiles nothing but the small install program.

What is not verified

  • Hosted CI: nothing here has run on a hosted runner, so the second pass's duration there, and installing an event trigger in the service container's template1, are untested (the container's postgres user is a superuser).
  • next-comment-run.sh (it needs the math engine's seed image), the nginx routing and shadow tests, the image builds and the statement-importance run of the entry point: not run. No route, nginx file or image definition changes. The entry point's two database test blocks were run as written, the release-mode queue test included.
  • The route conformance runners were not run against the old code on the older shape; the database tests show the old statements failing there.
  • The shape file reproduces column names, order, types and nullability for thirteen tables. It does not reproduce the defaults of existing columns, constraint and index names, triggers or function bodies, and two tables differ from an older installation in their last columns (the file's header names them). Tables the API does not read are left alone. It records one older installation's shape; another may differ in other ways.
  • A table that a test declares with fewer columns than the migrations' table keeps the order the test gave it, so a read whose only tests declare such a table is covered for column types, not for column order. The four SELECT * reads of comments go by column name and pass on both shapes.
  • Large json math documents on the next-comment read. On a json column the priorities read costs about twice what it costs on jsonb (measured: about 4.6 ms against 2 ms per megabyte of document text). A document too large to read within that pool's 50 ms statement limit is handed back to Node on every next-comment request of its conversation, after the route's 75 ms deadline, and each request repeats the read; PostgreSQL does not stop such a read at the limit, so the connection is held until the document has been read once. Requests for other conversations are answered when one of the pool's two connections is free and handed back within a few milliseconds when both are busy. A jsonb column behaves the same way at about twice the size. The vote writer remembers a read that failed (PR polis-api: the vote writer admits the jQuery client's Content-Type and reads comment priorities from a json math column #3390); this read does not yet. The route is off unless switched on.
  • Data is not modelled: what rows an older installation holds (a NULL where the migrations forbid one, a document only json can store) is outside a shape.
  • One boundary differs between the two types and is unchanged: the math generation read hands back a document whose text is over 65,536 bytes, and a json column keeps the text as written while jsonb re-renders it, so a document near that size can be answered on one and handed back on the other. The bytes of an answer are the same on both.

No schema change, no migration, no Node change.

🤖 Generated with Claude Code

…n whose database predates the migrations, and four reads no longer depend on the migrations' shape

A second pass of the database tests runs with the older shape installed
(json where the migrations build jsonb, another column order). Four reads
that failed or handed back there are fixed: the next-comment priorities
read and the math generation read take the column's type from the column;
the votes and participants reads follow the stored column order, as Node's
rows do. Bytes on a database built from the migrations are unchanged.

No schema change, no migration, no Node change.

Co-Authored-By: Claude <noreply@anthropic.com>
@colinmegill
colinmegill merged commit 09d6f84 into edge Nov 10, 2026
52 checks passed
@colinmegill
colinmegill deleted the polis-api/tests-on-older-database-shape branch November 10, 2026 23:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant