Repository navigation
Promote edge to stable: Polis 2.44.0 - #3426
Open
github-actions[bot] wants to merge 7 commits into
Open
github-actions[bot] wants to merge 7 commits into
github-actions[bot] wants to merge 7 commits into
Conversation
…listed conversations, switched off (#3422) * delphi: a run can leave out its visualization pages and remove its local files (switched off) Two switches for run_delphi.py, both off unless set to exactly "1"; off, a run starts the same stages and writes the same files as before. DELPHI_SKIP_DATAMAPPLOT=1 does not start the per-layer datamapplot step (700_datamapplot_for_layer.py uploads each page, which holds comment text, to the S3 bucket), and the UMAP stage writes none of its local layer pages. Every result family is still written; a failed UMAP stage still fails the run. DELPHI_LOCAL_OUTPUT_CLEANUP=1 removes polis_data/<zid> and visualizations/<zid> below the app directory and the working directory when the run ends, however it ends, so repeated runs leave nothing on the container's disk. The paths are built from the conversation id as an integer; another conversation's files are never touched. Tests: tests/test_run_delphi_shadow_switches.py, on the stub-stage fixture of test_run_delphi_exit_codes.py. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * delphi: the image carries all-MiniLM-L6-v2 at a pinned revision, read only when asked (switched off) The Delphi image's build fetches the embedding model's eleven loading files from the Hugging Face hub once, at revision 1110a243fdf4706b3f48f1d95db1a4f5529b4d41 (the hub's main for sentence-transformers/all-MiniLM-L6-v2 today), and refuses the build when any file's SHA-256 differs from delphi/embedding-model.lock.json. The files (about 92 MB, weights as model.safetensors only: no ONNX, OpenVINO, TensorFlow or PyTorch-bin copies) and the lock are copied into the final stage under /opt/polis/embedding. At run time the UMAP stage reads that directory, with local_files_only and so no network, only when DELPHI_EMBED_MODEL_PATH names it; with DELPHI_EMBED_MODEL_LOCK also set, the directory is checked file by file against the lock first and a difference refuses the run. Unset (every service today, and the image sets neither), the stage loads the model by name from the hub exactly as before. polismath/embedding_model.py has three commands: fetch (the build's), verify (a directory against the lock) and compare, which encodes the same texts with the model loaded by name (today's run) and loaded from the image's directory and prints both SHA-256 digests of the float32 vectors; it exits non-zero unless they are byte for byte equal. .dockerignore lets the lock into the build context; the deploy-hook test inputs carry the lock and .dockerignore so the Dockerfile checks run in the test container. Tests: tests/test_embedding_model.py. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * delphi: every step of a run can report its seconds and peak memory (switched off) With DELPHI_STEP_LOG=1 a full-pipeline run prints one line per step on standard output: delphi_step/1 {"step":"evoc","status":"ok","seconds":4.218,"peak_rss_mb":1208.4,...} Under the job daemon a child's standard output is stored as its attempt's log lines (polis_queue_logs, keyed by the attempt id its run record names) and goes to the container's log stream, so each run reports what each step cost beside its run record. Steps: census and manifest (the daemon's input and output records), pipeline (the UMAP stage as a child process, with its exact peak from wait4 and its CPU seconds), inputs, embed_load (and whether the model came from the image or the hub), embed_encode, umap, evoc (layers, clusters per layer, whether it fell back to KMeans, and Numba's threading layer), store_base, keywords, layer_pages, topic_names (one per layer), datamapplot (one per layer, or one line saying it was skipped), total and run. A step's peak memory is sampled from /proc every DELPHI_STEP_SAMPLE_MS (default 20) milliseconds while it runs; each line also carries the process's exact peak so far. Lines hold counts, seconds, megabytes and closed labels: no comment, text or conversation id. Off (unset, or anything but "1"), nothing is printed and every stage runs as before: children run through subprocess.run as they did. The UMAP stage's conversation processing is split into three functions (inputs, the first writes, the rest) with no change to what it reads or writes. Tests: tests/test_step_log.py. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * queue-rs: polis-delphi-shadow admits free-only Delphi runs one at a time, paced (switched off) A new binary beside polis-jobs, built into the Delphi image, off unless DELPHI_SHADOW_ENABLED=1. It admits full-pipeline runs, through the queue's own pd_writer_admit, of live conversations (a comment created or moderated in the last DELPHI_SHADOW_LIVE_DAYS, default 30) whose comment set changed since their last shadow run, under a result env no reader serves: shadow-delphi-1 by default, and any env that does not start with "shadow-" is refused at start, as is a QUEUE_ENV that differs from it. A polis-jobs daemon claiming that env runs them; nothing here starts one. What it admits. The comment set's digest is computed with the input bundle's own function from the same columns, so it equals the comment_set_sha256 the run's census records; an unchanged set is never run again (and is counted: a rerun there would recompute what is stored). Sets outside 20 to 5,000 comments are skipped, smallest first, at most 200 runs a day. The idempotency key is the comment set itself, so a restart admits nothing twice; the configuration names the set, its size class and the scheduler, and is in the job's run record (config_effective.shadow). How it is paced, as the math backfill is. One run at a time: nothing is admitted until the last admission is terminal (or past DELPHI_SHADOW_JOB_DEADLINE_SECONDS). After each run it rests for the duty cycle (DELPHI_SHADOW_DUTY_CYCLE, default 0.5: as long as the run took). Before each admission it pauses, and logs why, for the operator's pause file, a disk below DELPHI_SHADOW_DISK_FREE_MIN_MB (10 GiB) or above DELPHI_SHADOW_DISK_USED_MAX_PCT (85%), a database whose median SELECT 1 round trip is over DELPHI_SHADOW_PROBE_MAX_MS (250, resuming below half), a math engine whose newest write trails the newest vote by more than DELPHI_SHADOW_MATH_LAG_MAX_SECONDS (180), and the daily cap. The participation path. The scheduler holds no pool and shares no connection with the server or the Rust API: it opens one connection per login for one call and closes it. Its reads (comments, math_main, the newest vote's time; never a vote's value) are each one READ ONLY transaction with its own statement and lock timeouts; on the queue login it calls only the queue's functions, whose admission takes FOR KEY SHARE on the conversation's row. tests/delphi_shadow.rs (jobs-integration, run by ci/local/queue-rs.sh) holds the read transaction and an admission open while a comment, the conversation update votes and comments cause, a vote (a pass), the next comment's candidate read and the notification update run under a 50 ms lock timeout on the same conversation, and samples pg_stat_activity through whole cycles: at most one scheduler connection at a time, none left open. It writes one delphi_shadow/1 line per event (admitted, finished with the run's attempt and whether the census digest matched, unchanged, paused, resumed, idle) with job ids, counts, digests and closed words, and keeps a small state file (written whole and renamed). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * compose, deploy: the Delphi shadow's two services on the Delphi box, started only by their switch (off) docker-compose.yml gains delphi-shadow (a polis-jobs daemon) and delphi-shadow-scheduler (polis-delphi-shadow), both in the profile delphi-shadow and both off: each process exits 0 at once unless DELPHI_SHADOW_ENABLED=1, and the deploy hook starts them on the Delphi box only when the environment document carries the line DELPHI_SHADOW_ENABLED=1 exactly, after delphi and math-python, from the Delphi image pulled by digest. A failure to start them is logged and never fails the deploy. Without the line the hook's commands and its image override are the same as before (the existing tests' UNCHANGED lines). The daemon claims only DELPHI_SHADOW_ENV (default shadow-delphi-1), the full pipeline only, one job at a time, with the input bundle and the embeddings table on. It is free by construction: no LLM provider key, no Anthropic model, no AWS key and no S3 setting is listed, so none reaches the container whatever the environment document holds; topic naming falls back to keyword labels under the model "none". It writes no visualization page, reads the embedding model from the image offline, and prints its step lines. Its scratch, Hugging Face and Numba caches and the pipeline's local outputs are tmpfs (2 GiB, 512 MiB and 64 MiB, inside its memory limit), cleaned per run; only the daemon's journal and the scheduler's state file are on the volume delphi-shadow-state. Limits: 2 CPUs and 6 GiB for the daemon, a quarter CPU and 256 MiB for the scheduler. Both use the queue login and transport settings the delphi service already has; every scheduler bound is forwarded with its default. Tests: tests/test_compose_delphi_shadow.py (profiles, defaults off, no key reaches either container from a production-like environment document, the daemon's settings and limits, both name the same shadow env, the delphi service unchanged) and three cases in tests/test_after_install_hook.py (not started without the exact line, started after the engine with the Delphi image when it is there, never on another role). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * queue-rs: twin runs repeat a shadow run once on the same comment set and measure the difference (off) With DELPHI_SHADOW_TWIN_EVERY=N (default 0, off), every Nth first shadow run that succeeds is repeated once, on the same comment set, before any new set is admitted. Before the twin is admitted the scheduler keeps the first run's cluster assignments (from the shadow env's current rows) and reads the comment set again; a set that changed in between drops the twin. The twin's admission names the run it repeats, so its run record carries it (config_effective.shadow.twin_of, beside the comment set's digest), and both records carry every family's content_sha256: whether each family repeated byte for byte is read from the records themselves. When the twin succeeds the scheduler adds what digests cannot say, for the cluster assignments, layer by layer: the adjusted Rand index between the two runs (1.0 for the same partition, whatever the cluster numbers), the cluster counts of each, and how many comments moved under the best one-to-one matching of the two labelings. It logs one delphi_shadow/1 "twin" line with both job ids, the families' equality and those measures, keeps the newest 500 in its state file, and counts compared and identical twins. This is the measurement the clustering's run-to-run difference on today's image needs before any change to the image or the seeding. Tests: twin.rs (the textbook adjusted Rand value, relabelled partitions, moved counts, family comparison) and a jobs-integration case (the twin goes first, carries twin_of and the same digest in its run record, and is dropped when the set changed). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * queue-rs: a twin is identical when its clustering is, not when every family is Families whose rows carry the run's own job id or time (the topic names' keys, the UMAP configuration's dates) differ between any two runs, so the first rehearsal of twin runs reported every twin as different, including twins whose assignments were byte for byte the same. A twin is now identical when it has the same comments and layers, no comment moved in any layer, and the assignments family's digest is not different; the per-family equality is still reported as it was. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * delphi: a rehearsal of the shadow on public fixtures, with the network sealed delphi/tests/shadow/rehearsal.sh runs the repository's own delphi-shadow and delphi-shadow-scheduler services (docker-compose.yml unchanged, with an override for the network and the image) against a throwaway Postgres with the release migration chain and public fixtures only: conversation 1424 (the biodiversity export with its votes, through the vote convention module) and seed_public.py's comment-only conversations of 25 to 441 comments made from the two public exports. It also publishes a collective statement under the env readers serve, before anything starts. In order: the byte-for-byte check of the embedding model (the public comments encoded by name from the hub and from the image's files); the seal (Postgres moves to an internal network with no route out; proofs that no internet host is reachable, that the model cannot be downloaded, and that the image's model encodes offline, again from inside the running daemon at its start and its end); both services, with twin runs on; then one added comment and one moderation, and their reruns; then report.py: every job's run record facts and step lines, which envs hold rows in each queue and result table, whether the published pointer moved, and the tables' sizes. The stack is removed on exit, volumes included. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * deploy: the Delphi shadow starts after the role chain, so the Delphi role keeps its one start line The deploy hook's Delphi role must have exactly one compose `up` line, the one that starts delphi and math-python (tests/test_compose_math_env.py pins it); the shadow's switch-gated start was a second one inside that branch. It now follows the role chain as its own block. DELPHI_SHADOW_SELECTED is 1 only on the Delphi box and only for the exact switch line, so what runs where is unchanged; the start is still non-fatal. The deploy-hook test's constant for the shadow's start line is renamed: it had replaced the existing SHADOW_UP of the web role's vote shadow tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * delphi: the shadow rehearsal checks its published port by the shared host-port rule Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * delphi: the image test looks for an ENV instruction setting the model switch, not the word in a comment Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * deploy: switching the Delphi shadow off removes its containers by name With its -f files named, Compose v2.40.0 (the version the boxes run) leaves a profile-gated service's containers running through a `down` that does not name the profile. So the Delphi box's `down` never removed delphi-shadow or delphi-shadow-scheduler: with the switch line taken out of the environment document, a shadow a previous deploy started kept running, on that deploy's image, through every later deploy. On the Delphi box with the switch off, the hook now names both services in `rm -s -f` after the engine's start; with none there it does nothing, and a failure is reported without failing the deploy. Switched on, the start's --force-recreate replaces them as before; other roles never touch them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * queue-rs: a twin across a deploy is dropped, not compared A deploy recreates the scheduler and the daemon on the new image, and the scheduler's state file keeps a pending twin through it. A first run and a twin either side of a deploy (or a twin queued before one and run after it) were compared as if they repeated one build, so their difference would have been read as the clustering's nondeterminism. A twin is now admitted only when its first run's record carries this scheduler's code version, and compared only when both records do; otherwise the scheduler writes a twin_dropped line (code_version_changed) and moves on. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * queue-rs, delphi: the shadow's tests name no real env and pass the vote-sign lint The participation test's next-comment read wrote the rejected-comment value as a literal beside a vote, which the vote-sign lint counts as a new sign site, so the lint exits non-zero. It now names the moderation value as a constant, as the Rust next-comment read does; the statement is unchanged. The configuration test and the rehearsal's default named the result env a production reader serves; they now use a neutral example name. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * queue-rs, deploy: the Delphi shadow's daemon and scheduler start only on a shadow env The shadow's daemon is polis-jobs, which claims and publishes under any env it is given, and its env came from one line of the environment document (DELPHI_SHADOW_ENV) that nothing checked: that line naming the env a reader serves would have made the shadow a second production daemon, claiming that env's full-pipeline jobs and publishing under it. The daemon's container now runs `polis-delphi-shadow daemon`, which refuses to start (exit 2, one line, no value printed) unless QUEUE_ENV and DELPHI_RESULT_ENV are the same env beginning shadow-, and only then replaces itself with polis-jobs (exec, so it is the same process and the daemon's signal handling and drain are unchanged). The scheduler already refused such an env in its configuration; a test now runs both binaries as the container starts them: a stand-in polis-jobs on PATH runs, as the same process, only for a shadow env, and the scheduler exits 2 for a served one. An argument the binary does not know is refused rather than taken as the scheduler. Before it starts the shadow, the deploy hook reads back the composed configuration of both services (the image override included, with the delphi-shadow profile): the daemon's queue env must be a shadow env, its result env the same, its command the checked start, and the scheduler's two envs that same env. jq prints the names of what is not so, never a value. Refused, or unreadable, the shadow is not started and its containers are removed by name; the Delphi box's own deploy goes on, as it does when the shadow's start fails. Tests in the hook suite's style: the read-back runs before the Delphi box replaces anything, each wrong setting is refused, nothing is read back when the shadow is not selected, and the real jq names exactly the wrong settings; the compose suite runs the hook's program on docker-compose.yml as it is (it passes) and with the environment document giving the shadow a served env (it is refused). The production delphi service is unchanged: it never runs the new subcommand, and the switch-off path is the same. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * queue-rs, compose: the Delphi shadow stops admitting at a storage budget read from the database Shadow rows cannot be deleted (result rows, families and embeddings are immutable, and nothing prunes a shadow env), and nothing bounded how many the shadow could add. The scheduler now holds them under a budget: DELPHI_SHADOW_STORAGE_BUDGET_MB in all (1024, 1 GiB, by default) and DELPHI_SHADOW_STORAGE_DAILY_MB per UTC day (100 by default; never more than the budget). Before each admission it pauses, with a `paused` line naming storage_budget or storage_daily and the bytes against each bound, once either is reached, or when the next run's estimate (its comments times the bytes a comment has measured per run, 8 KiB before any is measured, plus 16 KiB for the queue's own rows of a job) would pass either. A waiting twin is held the same way and kept. The bytes are read from the database, never by scanning a table on a poll. Each run is read once, when it is terminal and before the next admission, by one read-only statement on the read login whose every read is an index range on that run's own keys: its attempts, and per attempt (a result batch is an attempt) its result rows, families, log lines, and the embeddings it first wrote. The bytes are modelled row by row from the stored values (pg_column_size): the heap row's share of a page of rows its size, or a small row and TOAST chunks when its values are moved out, and each index entry at half a leaf, since every index of these tables is led by the run's own random id. Against what the four tables really grew by for a run of 12,000 result rows, 2,000 embeddings and 6,000 log lines, the measure reads 4 to 7% over (per table 1.00 to 1.07); a test holds it within 10% under and 20% over, holds the indexes it models to the ones the migrations create, and holds the plan to index ranges. The total is kept in the state file; the file is a cache of the database, not the record. A scheduler that starts without one lists every job of every shadow- env from delphi_jobs once and measures each again, twenty per poll, admitting nothing until it has (storage_unmeasured); a run whose bytes cannot be read holds admission the same way. Tests: the ledger and the pause order as pure functions; a fresh scheduler finding a 28 MB shadow run already stored pauses at an 8 MiB budget, then at an 8 MiB daily cap under a larger budget, keeps its count across a restart, and with room admits, and reads the run's bytes before admitting another. Compose passes both settings to the scheduler with these defaults. The production delphi service and every switched-off path are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * deploy: tests that a Delphi shadow failing to start never fails the deploy, and never holds up the math engine's stop The Delphi role names its services in DELPHI_SERVICES and ValidateService requires exactly those; the shadow's two services are optional work and are never named there. These tests pin that: - a shadow whose start fails leaves the deploy exit 0, with the engine's start unchanged and the previous deploy's shadow containers removed before it; - the real hook's DELPHI_SERVICES line names delphi and math-python only, and neither shadow service has a depends_on (starting them never starts anything else); - switched on with math-python taken off the list (its durable stop): the hook removes math-python and the shadow's containers, starts delphi, then the shadow, and never starts math-python; - ValidateService on the Delphi box with the shadow's containers absent passes, and never polls them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * queue-rs, compose: the Delphi shadow's storage budget is 3 GiB in all and 300 MiB a day by default DELPHI_SHADOW_STORAGE_BUDGET_MB defaults to 3072 (was 1024) and DELPHI_SHADOW_STORAGE_DAILY_MB to 300 (was 100), in the scheduler and in the Compose service, so the first pass can cover the live conversations and a generous set of others. Either is still set by the environment document; a day may still not hold more than the whole budget. The default tests (the scheduler's config and the Compose environment) say the new numbers. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * queue-rs, compose: the Delphi shadow also runs a listed set of conversations and every large one, whatever their activity The scheduler picks from three sources, in priority order, each still held to the comment bounds, the rerun window, the pacing and the storage budget: 1. An important-conversations list, DELPHI_SHADOW_IMPORTANT: conversation codes (zinvites.zinvite, the code in a conversation's URL) separated by commas, highest priority first, empty by default. It is set only in the environment document; nothing names a conversation in the repository. An entry that is not a code (1-64 letters and digits) refuses the start by its position, never its text; a repeated code keeps its first place; at most 10,000. 2. Every conversation with at least DELPHI_SHADOW_LARGE_PARTICIPANTS participants (200 by default; 0 turns it off), most participants first, from conversations.participant_count (kept by the participant trigger). 3. The live conversations whose comment set changed, smallest first, as before. The first two are read as one roster, in one read-only transaction, at the start and once each UTC day (a failed read keeps the roster there is and is retried after DELPHI_SHADOW_ROSTER_RETRY_SECONDS, an hour by default): the codes through zinvites' unique index, the large conversations by one pass over conversations (that column has no index), each with its comment count by the comments' zid index. No poll reads either again: between reads each roster entry takes its fresh count and newest change from the live read the poll already makes, so a listed conversation that changes is seen at the next poll. A conversation in two sources is tried once, at its first. The first candidate whose run does not fit the budget holds admission, so a listed or large conversation keeps its place rather than letting smaller ones go first. One whose run alone would pass a whole day's cap is never tried (too_costly), so it cannot hold back the others every day. The log carries counts only (roster, idle) and each admission's source (selected: important, large, live or twin); the state file counts the admissions from the first two. The participation path: the new reads are ACCESS SHARE in the scheduler's read-only transaction, on a connection opened for it and closed after, which neither waits on nor makes wait the conversations updates that votes and comments cause, nor the zinvites read a participant's request makes. The isolation test now holds both reads open across the vote, comment and next-comment statements, and reads the roster again in each of its whole cycles. Its connection count is taken in the test's own database: with two test threads another test's scheduler was counted beside it. Tests: the list's parsing and refusals; the three sources' order, once each, with an unchanged set skipped and a changed one first again; the roster's overlay; a run too costly for a day skipped; against Postgres, the list, then large conversations, then live ones admitted in that order, a listed conversation that changes after the roster read admitted at the next poll, and a conversation that grows past the threshold waiting for the next day's roster. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * queue-rs: the important list's refusal test names what was accepted instead of unwrapping The workspace's clippy denies unwrap_err() in tests; the two refusals now read the error with let-else and say which entry was accepted if one ever is. The assertions are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * delphi: the hook test's fake compose keeps the failed Rust start as its last line The fake docker-compose exits with its last line's status, which is how a failed polis-api start reaches the hook. The Delphi shadow's start-failure line had been appended after it, so a failed Rust start exited 0 and three of the Rust start tests failed. The shadow's line now comes before it, and a comment says why the order matters. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * deploy: the Delphi shadow's start block names the Delphi role, so the web upstreams check reads it as Delphi-only The check of the web boxes' startup (file-server/nginx/production_upstreams.py) reads every block of the deploy hook that a role condition does not rule out as a possible web start. The shadow's start was guarded only by DELPHI_SHADOW_SELECTED, which is 1 only on the Delphi box, so the check read its `up ... || echo` line as a web start and refused the `||`. Rewriting the line as `if ! ...; then ...; fi` is not enough: the check then counts the shadow's two services as started on a web box and refuses the mismatch with the web images. The block's condition now names the role as well: `[ "$SERVICE_FROM_FILE" == "delphi" ] && [ "$DELPHI_SHADOW_SELECTED" = 1 ]`. DELPHI_SHADOW_SELECTED is already 0 on every other box, so the hook does exactly what it did; the check sees a Delphi-only block, and the line keeps its `|| echo`, so a failed shadow start still never fails the deploy. The check itself is unchanged: it goes on refusing anything on a web path it cannot read. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * delphi shadow: no daily storage limit by default, and 1 GiB of the budget kept for live conversations The scheduler's daily storage limit (DELPHI_SHADOW_STORAGE_DAILY_MB) is now off unless set: unset, empty or 0 means no daily limit, and the log lines say "no daily limit" where they gave the bytes. Set above 0 it holds as before, for every source. Compose passes 0. Of the total (DELPHI_SHADOW_STORAGE_BUDGET_MB, 3072 by default), DELPHI_SHADOW_LIVE_RESERVE_MB (1024 by default, 0 keeps none, never more than the total) is kept for the live conversations. The important list, the large conversations and twin runs stop at the total less the reserve (2048 MB by default), pausing as storage_reserve; live runs go on and stop only at the whole total. A listed or large conversation held by the reserve alone holds the rest of those two sources, and the live ones are still tried. A run that alone could never fit its share is never tried, as before with the daily limit. The log lines carry the reserve in bytes. The total, the run count a day, the pacing (database latency, math lag, disk floor, pause file) and the selection are unchanged. Tests: no daily limit by default and a set one still holding (config, plan and the scheduler against the database's own bytes); the first two sources stopping at the total less the reserve while live runs continue, and live runs stopping at the total (plan and the scheduler); Compose's defaults and pass-through. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * delphi shadow: three named sets, benchmark, featured (open data) and current, each run naming its set The scheduler now selects from three named sets, in priority order: - benchmark: the configured list of conversation codes, renamed from DELPHI_SHADOW_IMPORTANT to DELPHI_SHADOW_BENCHMARK (same rules: codes only, in priority order, empty by default, set only in the environment document); - featured: every open-data conversation (conversations.is_data_open, the owner's "anyone can export a data dump" setting), most participants first; - current: the live conversations whose comment set changed, as before. The participant-count source is removed with its setting. The scheduler refuses to start, naming the setting and never its value, if either DELPHI_SHADOW_LARGE_PARTICIPANTS or the list's old name is set, so a stale line is never silently ignored; Compose passes neither. The open-data read replaces the participant-count read in the roster: one pass over conversations (no index on the column) in the same read-only transaction as the benchmark codes, at the start and once each UTC day, never per poll, ACCESS SHARE only. The live reserve now protects the current set: benchmark and featured together (and twins) stop at the total less the reserve, current runs at the whole total. A conversation in two sets runs once, in the first. Every log line about a run (admitted, refused, unchanged, finished, storage, storage_unread and the twin lines) carries "set": benchmark, featured or current (a twin, its first run's), and so does the run record (config_effective.shadow.set). The roster and idle lines count each set. The set is not in the admission key: one comment set runs once whichever set selects it. A state file an earlier build wrote still loads. Tests: the order, each set once, a benchmark that is also open data counted as a benchmark, the reserve by set, the set words and the admission naming its set (plan); the renamed list and the retired settings (config); an earlier state file (state); against Postgres 17 with the release chain, the order and each run record's set, a conversation of 900 participants that is not open data never running until its data is opened and then only at the next day's roster read, the reserve holding benchmark and featured while current runs go on, a twin carrying its first run's set, a job listed again from the database keeping its set, and the participation-path test with the open-data read held open; Compose. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
colinmegill
marked this pull request as ready for review
November 15, 2026 22:32
* cdk: the Delphi box's root volume is 200 GiB The Delphi small launch template's gp3 root volume goes from 50 to 200 GiB. That disk holds every image the box pulls (the deploy hook pulls the new release's images before it removes the previous release's, so two sets are on disk while a deploy runs), and the Delphi shadow's scheduler stops admitting work under 10 GiB free. Between deploys the disk is about 70 % used; it has filled once. The synthesized template differs from edge in that one VolumeSize. The group has no rolling or replacing update policy and takes the template's latest version number, so a cdk deploy writes a new launch template version and changes no running instance: the box running now keeps its 50 GiB volume until it is replaced or its volume is grown in place. cdk/README.md says how to grow it in place. No schema change. The stack template snapshot also takes the log group's 365-day retention, which the stack has declared since the change that made the server log group keep a year; the snapshot was not updated then and failed on edge. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * cdk: the README's grow-in-place note names no volume and checks the root first The note on growing the Delphi box's root volume in place now gives the volume as a placeholder to fill with the verified root volume's id, says the host commands run only once the modification is optimizing or completed and the root is verified to be XFS directly on partition 1, the last partition on the disk, and says precisely which update policy the group lacks: one that replaces instances. It claims no restart or planned outage, nothing more. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * cdk: the stack template snapshot keeps edge's log retention line The volume-size commit also set the stack template snapshot's log group retention from 30 to 365 days, the value the stack has declared since the server log group was made to keep a year. That stale line is a separate defect that predates this branch, and is handled on its own. This restores it to edge's value, so the branch changes only the Delphi small launch template's volume size in the snapshot. As on edge, the stack template snapshot test fails on that one retention line until it is repaired. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…ce (used by nothing yet) (#3430) * topic map: the generator crate joins the queue-rs workspace, with its datamapplot parity tests and two public fixtures queue-rs/polis-topic-map is the generation-time half of a first-party topic map: from a Delphi run's stored UMAP positions, cluster layers and topic names it computes everything about the map that depends only on the run (the standard size, the first view, the point radius, each topic's label anchor and weight, its parent one level up, and both datamapplot palettes) and writes it as one JSON document, polis-topic-map/1. A library (build) and a command (polis-topic-map data | page). Nothing calls it yet: no route, no pipeline stage, no stored data. The source is the prototype's, formatted by rustfmt, with lint allowances on six lines (three index loops over parallel arrays, two unwraps on values that cannot be empty, one expect on the crate's constant colour table) so the crate can deny unwrap and expect as the other workspace members do. serde_json is asked for float_roundtrip, so numbers are read the same way whichever crate it is built with (polis-api turns it on for a whole workspace build). Tests: tests/datamapplot_parity.rs (the port against the values datamapplot 0.6.4 and colorspacious computed for the same runs) and tests/document_recording.rs (the whole document's length and SHA-256 for each fixture, from the library and from the command; the recorded values are what the prototype wrote for the same inputs). The public fixtures are a Delphi run over two compdemocracy/openData conversations (CC BY 4.0), bg2050-volunteers and scoop-hivemind.biodiversity: comment ids, positions, clusters, topic names and datamapplot's own values, 1,759,374 bytes in six files, no comment text and no vote. NOTICE carries datamapplot's MIT licence, the CET-C2s colour map's CC BY 4.0 attribution and the fixtures'. No new external package: Cargo.lock gains only the crate's own entry (serde, serde_json and, for the tests, sha2 are already locked). Co-Authored-By: Claude <noreply@anthropic.com> * ci: make check-polis-topic-map runs the topic map generator's formatting, clippy and tests, and hosted CI calls it ci/local/polis-topic-map.sh runs, in queue-rs, cargo fmt --check, cargo clippy --all-targets -D warnings and cargo test for polis-topic-map, the same three steps the other single-crate entry points run for theirs. It needs the pinned toolchain and nothing else: no database, no Docker, no network beyond fetching the locked crates. make check-polis-topic-map calls it, docs/local-checks.md lists it, and the queue-rs workflow has a new job, polis-topic-map, whose only step is that make target. The workflow's path filters already cover every file the check reads (queue-rs/**, ci/local/**, Makefile). Co-Authored-By: Claude <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com>
…ent package (no page uses it yet) (#3431) * client-topic-map: the topic map, the topic list and their neutral theme become one package the clients can share; no client uses it yet A folder beside the clients, client-topic-map/, holds the first-party topic map as plain browser scripts with no dependencies and no build step: topic-map.js (PolisTopicMap, one canvas, no library), topic-list.js (PolisTopicList, the same topics as a tree) and theme-neutral.js (every colour, type size, stroke, spacing and duration they use). The three files are unchanged from the prototype they come from. Each works as a classic script and as a CommonJS module. The map measures and draws topic names on its canvas in the theme's face, and which names show is decided from the shapes of their letters, so the package carries the face: Roboto, variable weight, latin subset (37,520 bytes, SIL Open Font License 1.1, with its licence), and fonts/roboto.css, the @font-face rule that declares it. Nothing is observable: no page, client or build imports the package. Tests, with Node's own runner: - geometry.test.js: the prototype's ten tests of the pure rules, unchanged except that the wrapping test reads its public fixture from test/fixtures/ (the topic names and datamapplot's label strings for two public conversations, 20,388 bytes); - api.test.js: the methods and the three events (topicselect, topichover, levelchange) with their details and origins, on a public fixture document (29,111 bytes); - self-contained.test.js: no script or style sheet of the package names a URL with a scheme or opens a request, and the style sheet's one URL is the font file beside it. NOTICE names what the code is derived from or carries: datamapplot, deck.gl and mjolnir.js (MIT), Python's textwrap (PSF License), Roboto (OFL 1.1) and the public fixtures (CC BY 4.0, compdemocracy/openData). README.md says how each client's build would take the package in. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ci: make check-client-topic-map runs the topic map package's tests, locally and in hosted CI ci/local/client-topic-map.sh checks for Node 24.22.1, as the other client checks do, and runs node --test over client-topic-map/test. The Makefile target calls it, a new workflow calls the Makefile target on pull requests that touch the package, ci/local or the Makefile and on pushes to edge, and docs/local-checks.md lists it. The package has no dependencies, so nothing is installed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * client-topic-map: the notice for the port of Python's textwrap names its source, Python 3.12, and keeps that source's copyright notices The port of textwrap in topic-map.js (WORDSEP and wrapLabel) was made from Lib/textwrap.py of Python 3.12, the Python that runs datamapplot in the Delphi image; that file is the same in every 3.12 release from v3.12.0. NOTICE now names that revision, keeps the module's own notices (Gregory P. Ward 1999-2001, the Python Software Foundation 2002 and 2003), and carries the Python Software Foundation's notice and licence exactly as Python 3.12's LICENSE gives them; before, it gave a shortened notice taken from a later LICENSE. The summary of changes also names one more: in the second look-behind for a hyphenated word, the character after the first hyphen is matched by the class of word characters and punctuation, where Python matches a letter. Only NOTICE changes; the scripts are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
… dimensions); one row in delphi_embedding_models (migration 000046) (#3428) * migrations: granite's row in delphi_embedding_models, so Delphi can store its embeddings beside MiniLM's (migration 000046) One data row in an existing table: model_id 2, 'ibm-granite/granite-embedding-311m-multilingual-r2@44399559930365213510b1ee2eb15ded83374f0e:384', 384 dimensions. The name is the one a Delphi run records for that model at that revision, kept to its first 384 dimensions. No table, column, index, constraint, function, trigger or grant changes; model_id 1 and every stored embedding and set stay as they are. Without the row, a run with the per-row embeddings table on (the Delphi shadow) cannot store granite's vectors: pd_embedding_put and pd_embedding_close look the model up by name and raise "unknown embedding model". Locks: the INSERT takes ROW EXCLUSIVE on delphi_embedding_models until commit, milliseconds. That conflicts only with SHARE and stronger locks, which no reader or writer of the table takes. Nothing on the vote, comment or next-comment path reads the table. Declared additive in compat.txt and docs/migrations.md. Reversal: down/000046_remove_granite_embedding_model.sql deletes the row only while nothing is stored under model_id 2, with the row's immutability trigger disabled inside that one transaction and enabled again before commit. The first-deploy report, the forward proof's additive ledger and the coordinator's schema census include it; the census, measured on PostgreSQL 17.11, finds no table or column change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * delphi: comments are embedded with granite-embedding-311m-multilingual-r2, kept to 384 dimensions all-MiniLM-L6-v2 is English-only. On other languages its nearest neighbours mostly do not share a comment's topic, and a conversation that mixes languages clusters by language. granite-embedding-311m-multilingual-r2 (Apache-2.0, revision 44399559930365213510b1ee2eb15ded83374f0e) clusters such conversations by topic. It is 768 wide and Matryoshka-trained; Delphi keeps the first 384 dimensions, the width it has always stored, and normalizes them. - The image fetches granite by delphi/embedding-model.lock.json (ten files, each SHA-256 checked) into /opt/polis/embedding/granite-embedding-311m-multilingual-r2: about 627 MB, where MiniLM was 92 MB. - polismath/embedding_model.py names a run's model once: repo@revision:384 from the lock for the image's directory, and the same for the default model by name, which is pinned to the lock's revision. A directory without its lock, or a SENTENCE_TRANSFORMER_MODEL that names another model than the directory holds, refuses, so vectors are never stored under another model's name. Every load truncates to 384 and every encode normalizes. - The pipeline stores under that name: in the per-row embeddings table (granite is model_id 2) and in the DynamoDB rows' embedding and conversation labels. The visualisation's served-set read asks for the deployment's model instead of MiniLM. The importer maps granite's name to its row, and still refuses a conversation whose rows come from two models. - The Delphi shadow reads granite from the image, offline; the delphi service's default name is granite. - The job graph's embedding stage is unchanged: it declares, checks and stores MiniLM, and refuses vectors that are not 384 wide, which granite's untruncated output is not. Vectors from the two models are not comparable. Nothing stored moves; a conversation run again under granite gets a new map, new clusters and new topic names. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * delphi: the run record names the embedding model by repository, revision and width The full pipeline's run record wrote models.embed from SENTENCE_TRANSFORMER_MODEL: no revision, no truncation, and nothing checked it against the directory the pipeline loaded. It now writes the name polismath/embedding_model.py resolves from the same environment and lock the pipeline loads with, as the stores record it: ibm-granite/granite-embedding-311m-multilingual-r2@44399559930365213510b1ee2eb15ded83374f0e:384 so two runs can be compared by the exact model they used. A model that cannot be named fails the manifest instead of recording a guess. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * delphi: the default embedding model loaded by name is checked against the shipped lock first With DELPHI_EMBED_MODEL_PATH unset, the default model was handed to SentenceTransformer by its hub name and revision, and nothing compared the files it read with the lock's SHA-256s. The run record still named repo@revision:384, a requested revision rather than checked bytes. Now, by name, the default model's files are fetched once, at the lock's revision, into a directory under the Hugging Face cache. A first fetch goes to a private directory and takes the shared name only after it verifies. Every load checks that directory against the lock and builds the model from it with no network, as the image's directory is loaded. A changed byte, a missing or extra file, or another revision's files refuse the run before any model is built or any embedding written. The shipped lock was looked up beside the package (/app in the image), where the image does not carry it; in the image the module may also be imported from site-packages. It is now found beside the package in a checkout, else where the image installs it (/opt/polis/embedding/...lock.json). A model other than the default, named on purpose, is still loaded from the hub as given, unchecked, and recorded under the name as given. The tests drive load() with the directory variables absent, replacing only the hub download and the model constructor: a valid snapshot loads; a one-byte change as served, or in the stored copy, refuses before construction; another revision's files refuse. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * compose: the delphi service reads the embedding model baked into its image The delphi service named the model only, so every new container downloaded it from the model hub on its first run (about 658 MB for granite). It now has the directory and lock the Delphi shadow reads, the copy baked into the image: each load checks the files against the lock and reads them with no network, and nothing is downloaded. Both are written as the service's other model settings are, with a default an environment document may override: DELPHI_EMBED_MODEL_PATH=${DELPHI_EMBED_MODEL_PATH:-/opt/polis/embedding/granite-embedding-311m-multilingual-r2} and its .lock.json beside it. A SENTENCE_TRANSFORMER_MODEL that names another model now refuses the run unless both are overridden to that model's directory and lock. The test that pinned the directory's absence from this service keeps the shadow's own run settings out of it; a new test pins the baked directory and lock (equal to the shadow's) and their override. Configuration docs and example.env say how the service gets its model. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * delphi: the stored-copy test seeds the cached model itself before changing a byte The test that a stored copy of the default model, changed by one byte, is refused took its directory from a first load(). Before the check existed that load never made a directory, so the test failed in its own setup with FileNotFoundError rather than on the refusal it is there to pin. It now writes the locked files where a by-name fetch keeps them (<HF_HOME>/polis-embedding/<name>@<revision>) without going through the code under test, changes one byte, and expects load() to refuse with nothing fetched and no model built. Without the check it fails with DID NOT RAISE ModelMismatch; with it, it passes. No product code changes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * delphi: tests that run run_delphi.py name the default model whatever the image sets The writers check runs its Python tests in the job graph's test image (delphi/tests/dynamo_removal/Dockerfile), which points DELPHI_EMBED_MODEL_PATH at a MiniLM directory, with no lock, for the graph's embedding stage. Three tests there run run_delphi.py, whose run record now names the model from those settings and refuses a directory without its lock, so each exited 5: test_actual_pipeline_orchestration_uses_private_reset_and_v2_manifest, test_daemon_mode_leaves_stdout_exactly_as_the_legacy_path_prints_it and test_the_retired_stages_never_run_on_either_path. The Delphi suite's own image sets none of these, so it passed there. These runs mean the default model by name. The fixtures behind them (app_dir in test_job_child_protocol.py, writer in test_delphi_writer.py) now clear SENTENCE_TRANSFORMER_MODEL, DELPHI_EMBED_MODEL_PATH and DELPHI_EMBED_MODEL_LOCK, as one test already did. The refusal and every assertion are unchanged, and the graph stage keeps its MiniLM directory; the test image's Dockerfile says why it sets it. Comments and docs that still described MiniLM as the model a run downloads: the two run-record conformance harnesses, the polis-contracts README and the job queue schema's example. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs: the model setting's census rows and example.env say what each Delphi service does docs/configuration-env-reference.md: the SENTENCE_TRANSFORMER_MODEL rows pointed at lines this change had moved (converter.py :149 and :179 are now :152 and :185; embedding_model.py reads it at :130 and :168, not :158), and one row named 500_generate_embedding_umap_cluster.py, removed in September. example.env: naming another model alone is refused by the delphi service, whose directory holds the default model, but a queue worker box's daemon, which has no directory set, loads it by name, unchecked. The comment said only the first. Documentation only; no test or check reads either file. Co-Authored-By: Claude <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Contributor
Author
Delphi Coverage Report
|
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Release plan
Version: 2.44.0
Reason: new selected migrations
Previous stable:
bce88d88bba0f25cdfc1e0b4d38f0091a042af95Candidate:
1559ab192dce597928372e2bf111f593f72409b7release.txt additions:
000046_add_granite_embedding_model.sqlMerge the reviewed version preparation into edge, then refresh this plan for the resulting edge SHA before merging the promotion with a merge commit.