Repository navigation
polis-api: the database tests also run on the shape of an installation whose database predates the migrations, and four reads no longer depend on the migrations' shape - #3393
Merged
Conversation
…n whose database predates the migrations, and four reads no longer depend on the migrations' shape A second pass of the database tests runs with the older shape installed (json where the migrations build jsonb, another column order). Four reads that failed or handed back there are fixed: the next-comment priorities read and the math generation read take the column's type from the column; the votes and participants reads follow the stored column order, as Node's rows do. Bytes on a database built from the migrations are unchanged. No schema change, no migration, no Node change. Co-Authored-By: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Observable changes
None for any response served today, on any installation.
GET /api/v3/participants) still goes through the contract's own encoding in that case.polis-apiat all: two statements were refused by PostgreSQL and two handlers handed every non-empty answer back, so Node answered, as it does today. With this changepolis-apican answer them, with the bytes Node gives on that installation. None of the four routes is switched on by this change; switching one on remains a separate decision.Why
An installation that was in service before the migrations existed has the same tables in a slightly different shape: its math documents are
jsonwhere the migrations buildjsonb, some tables keep their columns in another order, a few columns exist on one side only. Every database test of this crate builds its tables the migrations' way, so a statement that works only on that shape passes every test and fails on the older installation. One such statement in the vote writer was fixed in PR #3390. This change makes the test suite able to see the rest.What changes
A second pass of the database tests.
make check-polis-api(the entry point hosted CI calls,ci/local/polis-api.sh) now runscargo test --locked -p polis-api --features db-teststwice: once as before, and once withqueue-rs/polis-api/tests/older_database_shape.sqlinstalled on the test server. That file records, as ordinary SQL, the columns of the thirteen tables that differ and that the API reads (users,conversations,participants,participants_extended,comments,votes,xids,reports,site_domain_whitelist,math_main,math_ticks,math_bidtopid,math_ptptstats), and an event trigger that reshapes a table created under one of those names. In every such table ajsonbmath document becomesjson. A table declared with every column the first migration gives it (the databases the vote writer and treevite tests build from the migration files, and two read fixtures) also takes the older column order, the columns only an older installation has, and its nullability. Most tables the tests declare are narrower and keep the order the test gave them. The tests themselves are unchanged and unaware of it. The file's header anddocs/local-checks.mdsay exactly what is reproduced and what is not. A small program (examples/older_database_shape.rs) installs and removes it; the entry point removes it before the first pass and after the second, also when the tests fail, and one test in each pass checks that its databases really have the shape the pass names.Four reads fixed (each found failing by the second pass, each with a test that fails on the old code):
next_priorities.rs)jsonb_typeof(data): no such function for ajsoncolumnmath_bundle.rs)data <> 'null'::jsonb: no such comparison for ajsoncolumnGET /api/v3/votes/me(votes_read.rs)votesin the migrations' order, and handed back otherwiseSELECT *rows do; another set of columns is still handed backGET /api/v3/participants(participants_read.rs)participantsFor the last two, Node's own bytes differ between the two shapes, because Node answers with its
SELECT *row and JSON keys keep the columns' order. Wherevotesstorescreatedbeforeweight_x_32767, Node answers{"pid":…,"tid":…,"vote":…,"created":…,"weight_x_32767":…,"high_priority":…,"weight":null,"conversation_id":…}; where the migrations built the table,weight_x_32767comes beforecreated.polis-apinow does the same on each.Two existing tests loaded a full-width table by position; their
INSERTstatements now name their columns. No test was weakened, skipped or removed, and none had to be marked for one shape only.What was proved
On PostgreSQL 17.11, pinned Rust,
RUST_TEST_THREADS=2:cargo test --locked -p polis-api --features db-tests: 538 passed, 0 failed, 1 ignored on the migrations' shape, and the same 538, 0, 1 on the older shape (tests/key_boot: 3 passed in each). 532 of them are the tests that were there before; six are new.cargo test --locked -p polis-api: 326 passed, 0 failed (tests/key_boot: 3 passed).function jsonb_typeof(json) does not exist,operator does not exist: json <> jsonb, a hand-back); on the migrations' shape the three tests that set the type or the order themselves fail and the two existing ones pass.conversationsandparticipants_extended); constraints, indexes, defaults, owned sequences, triggers, rules and functions are otherwise identical to a database built without it. This is a test in the suite and runs in both passes.polis-apion one generated stack:votes-read-run.sh: 334 requests, 169 answered bypolis-apiwith Node's exact bytes, 165 handed to Node as the runner expects, 0 differences, on a database built by the migrations and again on one built with the older shape installed before the migration runner ran.participants-read-run.sh: 138 cases, 112 exact, 26 handed back, 0 failed, on both.math-membership-run.sh: 192 cases, 129 exact, 63 handed back, 0 failed on the migrations' shape; on the older shape the runner's own fixture insertsmath_ticks.caching_tick, a column that shape lacks (math-membership-replay.cjs:142), so it was run there with that one column kept, with the same 192, 129, 63 and 0. In the recordings Node's body differs between the two shapes in 31 of the 334 vote-read requests and in 69 of the 138 participant requests, by key order only;polis-apianswers 15 and 56 of those, with Node's bytes on each shape.cargo clippy --locked -p polis-api --all-targets -- -D warnings, with and without--features db-tests: clean.cargo fmt -p polis-api --check: clean.make check-vote-sign-lint: 32 tests pass, 581 sites, 0 new.The second pass adds about 1 minute 50 seconds to the check on a 14-core machine (the first pass takes the same); it compiles nothing but the small install program.
What is not verified
template1, are untested (the container'spostgresuser is a superuser).next-comment-run.sh(it needs the math engine's seed image), the nginx routing and shadow tests, the image builds and the statement-importance run of the entry point: not run. No route, nginx file or image definition changes. The entry point's two database test blocks were run as written, the release-mode queue test included.SELECT *reads ofcommentsgo by column name and pass on both shapes.jsonmath documents on the next-comment read. On ajsoncolumn the priorities read costs about twice what it costs onjsonb(measured: about 4.6 ms against 2 ms per megabyte of document text). A document too large to read within that pool's 50 ms statement limit is handed back to Node on every next-comment request of its conversation, after the route's 75 ms deadline, and each request repeats the read; PostgreSQL does not stop such a read at the limit, so the connection is held until the document has been read once. Requests for other conversations are answered when one of the pool's two connections is free and handed back within a few milliseconds when both are busy. Ajsonbcolumn behaves the same way at about twice the size. The vote writer remembers a read that failed (PR polis-api: the vote writer admits the jQuery client's Content-Type and reads comment priorities from a json math column #3390); this read does not yet. The route is off unless switched on.jsoncan store) is outside a shape.jsoncolumn keeps the text as written whilejsonbre-renders it, so a document near that size can be answered on one and handed back on the other. The bytes of an answer are the same on both.No schema change, no migration, no Node change.
🤖 Generated with Claude Code