Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 6 additions & 2 deletions coordinator-rs/src/schema/catalog.json
Original file line number Diff line number Diff line change
Expand Up @@ -5998,7 +5998,7 @@
"nullable": "NO",
"default": null,
"ordinal": 1,
"definition_type": "character varying(100)",
"definition_type": "character varying(2000)",
"introduced": "server/postgres/migrations/000000_initial.sql:943"
},
{
Expand All @@ -6009,7 +6009,7 @@
"nullable": "NO",
"default": null,
"ordinal": 2,
"definition_type": "character varying(100)",
"definition_type": "character varying(2000)",
"introduced": "server/postgres/migrations/000000_initial.sql:944"
},
{
Expand Down Expand Up @@ -12818,6 +12818,10 @@
{
"path": "server/postgres/migrations/000044_pid_auto_transaction_lock.sql",
"sha256": "8eaffb393aca2b89ffe41264a1a9676f83a74685866a095befd3bb648784903f"
},
{
"path": "server/postgres/migrations/000045_page_ids_width.sql",
"sha256": "eab730473dda1fdd7b0e3b6fbbae5eb6201fbb579670ebcf43557aeb23f8c4a2"
}
]
}
2 changes: 1 addition & 1 deletion coordinator-rs/src/schema/mod.rs
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
//! Descriptive storage types only. No runtime caller adopts these rows.
//!
//! Source inventory: server/postgres/migrations/000000_initial.sql through
//! 000044_pid_auto_transaction_lock.sql. The adjacent catalog was measured on
//! 000045_page_ids_width.sql. The adjacent catalog was measured on
//! PostgreSQL 17.11; tests bind it to every migration's bytes. A changed or new
//! migration requires a fresh isolated census and review, even for a change
//! that does not alter columns. This intentionally avoids parsing PL/pgSQL DDL.
Expand Down
5 changes: 3 additions & 2 deletions docs/migrations.md
Original file line number Diff line number Diff line change
Expand Up @@ -233,7 +233,7 @@ before. `apply` never runs this release's pending SQL after a tolerated newer
migration; that order is refused before any change. The rule only helps
releases that contain it: a release built before it refuses every newer row.

Declarations for 000019–000044 (the older code is the release just before each
Declarations for 000019–000045 (the older code is the release just before each
migration, as recorded in `release.txt` history):

| Migration | Declared | Evidence |
Expand All @@ -255,9 +255,10 @@ migration, as recorded in `release.txt` history):
| 000042 statement-number lock to commit | additive | `tid_auto` (BEFORE INSERT on `comments`) takes `pg_advisory_xact_lock` instead of the session lock `pg_advisory_lock`, on the same key, so a conversation's statement-number lock is held until the inserting transaction ends rather than released before it commits; `tid_auto_unlock` and its AFTER INSERT trigger are dropped. Numbering is unchanged (MAX(tid) + 1 per conversation, from 0, no gaps); no table, column, index, constraint or row changes. No code in the earlier release calls `tid_auto_unlock` or takes the lock; every writer lets the trigger assign `tid` and reads it back, so an earlier release inserts and numbers statements as before. Locks: `DROP TRIGGER` takes ACCESS EXCLUSIVE on `comments` until the file commits, waiting at most the 5 s `lock_timeout` for transactions already using `comments`; comment reads and writes queue behind it for that wait, once; on timeout nothing changes and the deploy can be retried. Reversal: `down/000042_restore_tid_auto_session_lock.sql` restores both functions and the trigger exactly as `000000_initial.sql` defines them (SHARE ROW EXCLUSIVE on `comments`, bounded at 5 s), and brings back the race this migration removes. The runner's `forward.py` tolerates it as additive. |
| 000043 activity indexes | additive | Three ordinary btree indexes only; no table, function, row or application SQL changes. Concurrent builds preserve ordinary participation writes. |
| 000044 participant-number lock to commit | additive | `pid_auto` (BEFORE INSERT on `participants`) takes `pg_advisory_xact_lock` instead of the session lock `pg_advisory_lock`, on the same key, so a conversation's participant-number lock is held until the inserting transaction ends rather than released before it commits; `pid_auto_unlock` and its AFTER INSERT trigger are dropped. Numbering is unchanged (MAX(pid) + 1 per conversation, from 0, no gaps), as is the `conversations.participant_count` update, whose row lock was already held to commit, so no join waits longer; no table, column, index, constraint or row changes. No code in the earlier release calls `pid_auto_unlock` or takes the lock; every writer lets the trigger assign `pid` and reads it back, so an earlier release joins and numbers participants as before. Locks: `DROP TRIGGER` takes ACCESS EXCLUSIVE on `participants` until the file commits, waiting at most the 5 s `lock_timeout` for transactions already using `participants`; participant reads and writes (votes, comments, nextComment, participationInit) queue behind it for that wait, once; on timeout nothing changes and the deploy can be retried. Reversal: `down/000044_restore_pid_auto_session_lock.sql` restores both functions and the trigger exactly as `000000_initial.sql` defines them (SHARE ROW EXCLUSIVE on `participants`, bounded at 5 s), and brings back the race and the held lock this migration removes. The runner's `forward.py` tolerates it as additive. |
| 000045 embed identity width | additive | Widens `page_ids.page_id` and `page_ids.site_id` from `varchar(100)` to `varchar(2000)`, retaining `UNIQUE(site_id, page_id)`. Increasing these limits is metadata-only in Postgres; a populated 10,000-row test retained both heap and index relfilenodes and identical contents. Earlier servers can read and write the same values. ACCESS EXCLUSIVE is held until commit, with a 5 s lock acquisition timeout. The down migration refuses any value over 100 characters before narrowing; it never truncates identities. |

So a rollback to a release with this rule that lists 000027 starts beside
000028–000031, 000038, 000039, 000041–000044; a rollback past 000027 (or 000024) refuses, as it must: the
000028–000031, 000038, 000039, 000041–000045; a rollback past 000027 (or 000024) refuses, as it must: the
earlier queue daemon would exit on the newer contract.

## CI and local test stacks
Expand Down
2 changes: 1 addition & 1 deletion queue-rs/polis-migrate/tests/forward.py
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ def test_01_ledger_declares_every_migration(self):
'000038_create_delphi_comment_embeddings.sql', '000039_clear_zinvites_uuid.sql',
'000041_create_delphi_run_record.sql', '000042_tid_auto_transaction_lock.sql',
'000043_add_conversation_activity_indexes.sql',
'000044_pid_auto_transaction_lock.sql'})
'000044_pid_auto_transaction_lock.sql', '000045_page_ids_width.sql'})

def test_02_deploy_records_declarations(self):
db = self.new('records'); p.runner(db, 'deploy')
Expand Down
148 changes: 148 additions & 0 deletions server/__tests__/integration/implicit-conversation.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -154,4 +154,152 @@ describe("implicit embed conversations", () => {
});
});

async function counts() {
const { rows } = await pool.query(
`SELECT
(SELECT count(*)::int FROM conversations WHERE owner = $1) AS conversations,
(SELECT count(*)::int FROM page_ids WHERE site_id = $2) AS pages,
(SELECT count(*)::int FROM zinvites z JOIN conversations c ON c.zid = z.zid
WHERE c.owner = $1) AS invites`,
[owner, siteId]
);
return rows[0];
}

test.each([150, 2000])(
"a %i-character page creates once and the next visit reuses it",
async (length) => {
const page = "p".repeat(length);
const before = await counts();
const first = await visit(siteId, page);
const second = await visit(siteId, page);
expect(first.status).toBe(302);
expect(second.status).toBe(302);
expect(second.headers.location).toBe(first.headers.location);
expect(await pageRows(page)).toHaveLength(1);
expect(await counts()).toEqual({
conversations: before.conversations + 1,
pages: before.pages + 1,
invites: before.invites + 1,
});
}
);

test.each(["page_id", "site_id"])(
"a 2,001-character %s answers 400 before any database access",
async (name) => {
const before = await counts();
const transaction = jest.spyOn(pg, "withTransaction");
const read = jest.spyOn(pg, "queryP_readOnly");
const response = await visit(
name === "site_id" ? "polis_site_id" + "s".repeat(1988) : siteId,
name === "page_id" ? "p".repeat(2001) : "oversized-site"
);
expect(response.status).toBe(400);
expect(response.body.error).toBe(`polis_err_${name}_length_1_to_2000`);
expect(transaction).not.toHaveBeenCalled();
expect(read).not.toHaveBeenCalled();
expect(await counts()).toEqual(before);
}
);

test("a missing or empty page answers 400 without writes", async () => {
const before = await counts();
for (const path of [`/${siteId}`, `/${siteId}/`]) {
const response = await request(app)
.get(path)
.query({ parent_url: parentUrl });
expect(response.status).toBe(400);
expect(response.body.error).toBe("polis_err_page_id_length_1_to_2000");
}
expect(await counts()).toEqual(before);
});

test.each(["page_ids", "zinvites"])(
"a failing %s insert rolls back every row",
async (table) => {
const before = await counts();
const transaction = pg.withTransaction.bind(pg);
let attemptedZid: number | undefined;
jest.spyOn(pg, "withTransaction").mockImplementation((callback) =>
transaction(async (client) => {
const query = client.query.bind(client);
jest.spyOn(client, "query").mockImplementation((async (
sql: any,
params: any
) => {
if (new RegExp(`insert into ${table}`, "i").test(String(sql))) {
attemptedZid = table === "page_ids" ? params[2] : params[0];
return query("SELECT 1 / 0");
}
return query(sql, params);
}) as any);
return callback(client);
})
);
const response = await visit(siteId, `failed-${table}`);
expect(response.status).toBe(500);
expect(attemptedZid).toEqual(expect.any(Number));
expect(await counts()).toEqual(before);
for (const tableName of ["conversations", "page_ids", "zinvites"]) {
const { rows } = await pool.query(
`SELECT zid FROM ${tableName} WHERE zid = $1`,
[attemptedZid]
);
expect(rows).toHaveLength(0);
}
}
);

test("two simultaneous first visits roll back the loser and redirect to one conversation", async () => {
const page = "concurrent-first-visit";
const before = await counts();
const read = pg.queryP_readOnly.bind(pg);
let arrivals = 0;
let release!: () => void;
const bothLookedUp = new Promise<void>((resolve) => {
release = resolve;
});
jest
.spyOn(pg, "queryP_readOnly")
.mockImplementation(async (sql, params) => {
const rows = await read(sql, params);
if (
String(sql).startsWith("select * from page_ids") &&
params[1] === page
) {
expect(rows).toEqual([]);
if (++arrivals === 2) release();
await bothLookedUp;
}
return rows;
});
const transaction = pg.withTransaction.bind(pg);
let collisions = 0;
jest.spyOn(pg, "withTransaction").mockImplementation(async (callback) => {
try {
return await transaction(callback);
} catch (error: any) {
if (
error.code === "23505" &&
error.constraint === "page_ids_site_id_page_id_key"
)
collisions++;
throw error;
}
});
const responses = await Promise.all([
visit(siteId, page),
visit(siteId, page),
]);
expect(responses.map((response) => response.status)).toEqual([302, 302]);
expect(responses[0].headers.location).toBe(responses[1].headers.location);
expect(collisions).toBe(1);
expect(await pageRows(page)).toHaveLength(1);
expect(await counts()).toEqual({
conversations: before.conversations + 1,
pages: before.pages + 1,
invites: before.invites + 1,
});
});
});
30 changes: 30 additions & 0 deletions server/__tests__/unit/embed-id-parameter.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
import { getEmbedId } from "../../src/utils/parameter";

jest.mock("../../src/db/pg-query", () => ({ __esModule: true, default: {} }));
jest.mock("../../src/user", () => ({}));
jest.mock("../../src/conversation", () => ({}));

describe.each(["site_id", "page_id"] as const)("%s length", (name) => {
test.each([1, 100, 150, 2000])(
"accepts %i characters unchanged",
async (length) => {
const id = "x".repeat(length);
await expect(getEmbedId(id, name)).resolves.toBe(id);
}
);

test.each([undefined, null, 1, [], "", "x".repeat(2001)])(
"rejects invalid input %#",
async (id) => {
await expect(getEmbedId(id, name)).rejects.toThrow(
`polis_err_${name}_length_1_to_2000`
);
}
);

test("preserves spaces and encoding and counts Unicode characters", async () => {
for (const id of [" a%2Fb ", "😀".repeat(2000)]) {
await expect(getEmbedId(id, name)).resolves.toBe(id);
}
});
});
2 changes: 1 addition & 1 deletion server/bin/build-migration-report.py
Original file line number Diff line number Diff line change
Expand Up @@ -86,7 +86,7 @@ def manifest(path):
'000031_scope_delphi_result_reads.sql', '000038_create_delphi_comment_embeddings.sql',
'000039_clear_zinvites_uuid.sql', '000041_create_delphi_run_record.sql',
'000042_tid_auto_transaction_lock.sql', '000043_add_conversation_activity_indexes.sql',
'000044_pid_auto_transaction_lock.sql'}
'000044_pid_auto_transaction_lock.sql', '000045_page_ids_width.sql'}
assert selected == {p.name for p in files} | pending, 'report scope differs from release selection'
assert held == {'000021_create_polis_coordinator.sql'}, 'review changed hold policy'
numbered = sorted(MIG.glob('*.sql'))
Expand Down
15 changes: 15 additions & 0 deletions server/postgres/migrations/000045_page_ids_width.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
-- 000045_page_ids_width.sql
--
-- Embed identities can contain up to 2,000 characters. Widening varchar(100)
-- to varchar(2000) is a metadata-only change in Postgres: no table rewrite or
-- stored-value conversion. The UNIQUE(site_id, page_id) constraint is retained.
-- Compatibility: additive. Old servers can keep reading and writing while this
-- schema is installed, including during a rolling application replacement.
-- ALTER TABLE takes ACCESS EXCLUSIVE until commit; reads and writes briefly
-- wait. The 5 s lock timeout bounds acquisition; failure rolls everything back.
BEGIN;
SET LOCAL lock_timeout='5s';
ALTER TABLE public.page_ids
ALTER COLUMN page_id TYPE varchar(2000),
ALTER COLUMN site_id TYPE varchar(2000);
COMMIT;
3 changes: 2 additions & 1 deletion server/postgres/migrations/compat.txt
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
# this file. An undeclared migration counts as breaking. Recorded declarations
# only ever tighten (additive -> breaking). Every release carries every line
# forward; a new migration adds its own line in number order, with its evidence
# in the PR. Evidence for 000019-000044: docs/migrations.md "Forward compatibility".
# in the PR. Evidence for 000019-000045: docs/migrations.md "Forward compatibility".
# 000000-000018 predate the runner and were not evaluated: fail closed.
000000_initial.sql breaking
000001_update_pwreset_table.sql breaking
Expand Down Expand Up @@ -44,3 +44,4 @@
000042_tid_auto_transaction_lock.sql additive
000043_add_conversation_activity_indexes.sql additive
000044_pid_auto_transaction_lock.sql additive
000045_page_ids_width.sql additive
19 changes: 19 additions & 0 deletions server/postgres/migrations/down/000045_restore_page_ids_width.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
-- Reversal of 000045_page_ids_width.sql. Never run automatically.
-- Narrowing may rewrite the table and requires ACCESS EXCLUSIVE. Refuse if
-- any identity exceeds the old limit: never truncate or change page identity.
-- The operator reconciles runner history only after this succeeds.
BEGIN;
SET LOCAL lock_timeout='5s';
LOCK TABLE public.page_ids IN ACCESS EXCLUSIVE MODE;
DO $$
BEGIN
IF EXISTS (SELECT 1 FROM public.page_ids
WHERE char_length(page_id) > 100 OR char_length(site_id) > 100) THEN
RAISE EXCEPTION 'page_ids contains identities longer than 100 characters';
END IF;
END;
$$;
ALTER TABLE public.page_ids
ALTER COLUMN page_id TYPE varchar(100),
ALTER COLUMN site_id TYPE varchar(100);
COMMIT;
1 change: 1 addition & 0 deletions server/postgres/migrations/release.txt
Original file line number Diff line number Diff line change
Expand Up @@ -34,3 +34,4 @@
000042_tid_auto_transaction_lock.sql
000043_add_conversation_activity_indexes.sql
000044_pid_auto_transaction_lock.sql
000045_page_ids_width.sql
5 changes: 3 additions & 2 deletions server/postgres/migrations/report/first-deploy.sql
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
-- First deployment only: no existing ledger or queue. Other states refuse.
-- This is a catalog forecast, not DDL success or deployment-health proof.
-- sha256 8fa1066d9fc29c1df528b300199c619a2f9edecf1432926ad6a8146ebc650593 adoption/helpers.sql
-- sha256 4959eb616a51a172d7f7e5d3d06e92782fa24a67b099022fff366af7d5455c97 release.txt
-- sha256 7cf3c7d13306ca2ed5f9cb2e04a5ad3b8e0fff7c6c44e9c6473875a8cb7d8a63 release.txt
-- sha256 8a8b24fa47a25c613a8329aa79f413ac7461a93b12134045ae1cf2457e2ee3b7 held.txt
-- sha256 c6b5c71247d129964dd89ba560a4522caef1abbb9379ad1b3445d6ed96be805b adoption/000000_initial.sql
-- sha256 28ef33976beb9c7c9248ac800f29aab048f4410c2170ad12e3ade71d72e5ecb5 adoption/000001_update_pwreset_table.sql
Expand Down Expand Up @@ -59,6 +59,7 @@
-- sha256 ae12bd2dcce3a8e055adb54b241c090f36861ad4185771e173a87abd278b53f6 000042_tid_auto_transaction_lock.sql
-- sha256 1c85ef8a67bb0add2bcf6dd7e38fabb977c26fe004994cf78e06fed8eda9b906 000043_add_conversation_activity_indexes.sql
-- sha256 8eaffb393aca2b89ffe41264a1a9676f83a74685866a095befd3bb648784903f 000044_pid_auto_transaction_lock.sql
-- sha256 eab730473dda1fdd7b0e3b6fbbae5eb6201fbb579670ebcf43557aeb23f8c4a2 000045_page_ids_width.sql
BEGIN ISOLATION LEVEL REPEATABLE READ READ ONLY;
SET LOCAL search_path=pg_catalog,public;
SET LOCAL statement_timeout='30s';
Expand Down Expand Up @@ -1400,7 +1401,7 @@ SELECT (SELECT EXISTS(SELECT 1 FROM pg_index WHERE indexrelid=to_regclass('publi
CASE WHEN NOT can_reconcile THEN 'BLOCKED_RECONCILIATION'
WHEN NOT reconcile_authority OR NOT can_provision THEN 'REVIEW_MIGRATION_SESSION_AUTHORITY'
ELSE 'WOULD_APPLY' END
FROM (VALUES ('000019_create_polis_queue.sql'),('000023_create_delphi_foundation.sql'),('000024_create_polis_queue_large_class.sql'),('000027_create_sealed_job_graphs.sql'),('000028_create_delphi_results.sql'),('000029_extend_delphi_graph_stages.sql'),('000030_create_delphi_writers.sql'),('000031_scope_delphi_result_reads.sql'),('000038_create_delphi_comment_embeddings.sql'),('000039_clear_zinvites_uuid.sql'),('000041_create_delphi_run_record.sql'),('000042_tid_auto_transaction_lock.sql'),('000043_add_conversation_activity_indexes.sql'),('000044_pid_auto_transaction_lock.sql')) p(migration) CROSS JOIN readiness
FROM (VALUES ('000019_create_polis_queue.sql'),('000023_create_delphi_foundation.sql'),('000024_create_polis_queue_large_class.sql'),('000027_create_sealed_job_graphs.sql'),('000028_create_delphi_results.sql'),('000029_extend_delphi_graph_stages.sql'),('000030_create_delphi_writers.sql'),('000031_scope_delphi_result_reads.sql'),('000038_create_delphi_comment_embeddings.sql'),('000039_clear_zinvites_uuid.sql'),('000041_create_delphi_run_record.sql'),('000042_tid_auto_transaction_lock.sql'),('000043_add_conversation_activity_indexes.sql'),('000044_pid_auto_transaction_lock.sql'),('000045_page_ids_width.sql')) p(migration) CROSS JOIN readiness
UNION ALL
SELECT migration, NULL::boolean, 'OUTSIDE_RELEASE'
FROM (VALUES ('000020'),('000021'),('000025'),('000026')) p(migration)
Expand Down
Loading
Loading