Skip to content

offline profile 5: the conversation data export streams from the server when no S3 bucket is configured (stacked on #2932) - #2940

Open
colinmegill wants to merge 16 commits into
edgefrom
offline-5-export-without-s3
Open

colinmegill wants to merge 16 commits into
edgefrom
offline-5-export-without-s3

Conversation

@colinmegill

@colinmegill colinmegill commented Oct 3, 2026 •

Copy link
Copy Markdown
Member

Stacked on #2932 (p078-e-export-import-declared-sign). The first five-plus commits in this PR's diff against edge are #2932's. This PR adds four commits on top: 56ab1abe7, 66f2ae876, 8454ba205, f4e9269a6 (review follow-up). Merge #2932 first, or review only those three.

Offline profile item 10: the conversation data export works without S3.

Decision

GET /api/v3/dataExport has no producer, so when no bucket is configured the server streams the zip itself.

Evidence:

  • The route only inserts a generate_export_data row into worker_tasks (server/src/utils/common.ts doAddDataExportTask) and answers {}. The consumer was the Clojure math worker's export task. Production has not consumed worker_tasks since 2022-04-18: the task system was commented out of the worker's full system in 7d6db8eb6 ("fixes math worker crashes when run in dev mode #1279"). client-admin has no caller.
  • /api/v3/dataExport/results signs a URL into the hard-coded AWS bucket polis-datadump, with no endpoint override (server/src/routes/dataExport.ts:6-10 on edge).
  • The report exports (/api/v3/reportExport/<report_id>/<file>, server/src/routes/export.ts) read Postgres and stream their CSVs directly, so they need no S3. They are the producer that already works offline.

What changes:

  • AWS_S3_BUCKET_NAME unset, or OFFLINE set (1/true): after the existing ownership check, GET /api/v3/dataExport streams polis-export-<conversation_id>-<millis>.zip in its response. The zip holds format.json, summary.csv, comments.csv, votes.csv and participant-votes.csv. Each one is produced by the same report-export function and is byte-identical to that route's output. The writer holds at most one 256 KiB run of input: entries are deflated in sync-flushed runs as the rows stream out of Postgres, and data descriptors carry the CRC and sizes. The Postgres streams pause while the response needs draining, so a slow client slows the reads instead of growing Node's socket buffer. A client that disconnects aborts the export: the running query's cursor is closed, its client is discarded and the handler returns. This works through an AsyncLocalStorage context (src/db/streamControl.ts) that stream_queryP_readOnly reads, and it changes nothing outside that context. comments.csv and summary.csv are each still built as one string, as in the report export. There are no zip64 records. GET /api/v3/dataExport/results answers 404 polis_err_data_export_results_not_stored, because no dump is stored and a signed URL would point the browser at AWS.
  • Size limit: before the first byte, the route counts the vote rows, the distinct voters and the comments. If max(votes, voters × comments) exceeds DATA_EXPORT_MAX_CELLS, it answers 413 polis_err_data_export_too_large with cells and limit.
  • Auth: the dataExport route keeps its own gate (hybridAuth plus isModerator). The report export is keyed by an unguessable report_id, whereas this route takes a conversation_id, which is public. Copying the report export's auth here would therefore remove the owner check, so this PR does not.
  • Bucket set and OFFLINE unset: unchanged. The route still queues the worker task and answers {}, and the results route still signs the S3 URL.

The offline key in server/src/config.ts is #2937's definition, copied verbatim, so the two PRs merge cleanly in either order.

Knobs

Setting Default Effect
AWS_S3_BUCKET_NAME unset in test.env; set in example.env and CI unset → the server streams the export
OFFLINE unset 1/true → the server streams the export even with a bucket
DATA_EXPORT_MAX_CELLS 20000000 the largest conversation streamed (max of vote rows and voters × comments). Over it → 413. At the default, votes.csv is about 2 GB at most, under the 4 GiB zip32 limit

Tests

  • Unit (conversationZip.test.ts, 20 tests; streamControl.test.ts, 4 tests).
    • Zip correctness: the writer is checked with an independent reader that verifies the central directory, local headers, data descriptors, CRCs and the UTC timestamp field. The assembled zip holds the five generated-fixture files in order, byte for byte.
    • Route modes: no bucket; OFFLINE with a bucket; bucket set, which enqueues exactly as before.
    • Refusals and failures: 413 above the size limit and a stream at it; the ownership refusal; a mid-stream failure cuts the connection; the results route returns 404.
    • Disconnect: a destroyed output makes writes throw and finish() reject rather than hang. The abort reaches the producer, and the handler returns when the client goes away.
    • Stream control: an abort mid-stream destroys the stream and discards the client; an abort before start runs no query; the stream pauses and resumes on drain; with no context, behaviour is unchanged.
  • Integration (data-export-stream.test.ts, 6 tests, Postgres only).
    • Byte identity: a generated-fixture conversation is written straight to the database, and every zip entry equals the bytes /api/v3/reportExport serves. The file also covers OFFLINE with a bucket, 413 and the non-owner 403.
    • Slow and vanished clients, on a 600,000-vote generated fixture over a real socket. A client that stops reading holds the votes.csv query paused, with no bytes sent for 1.5 s. On disconnect, the handler returns and that backend is gone from pg_stat_activity. With the abort hook removed, this test fails. A reading client gets every row, and nothing is left running.
  • Also checked: unzip -t, Python zipfile and bsdtar accept the writer's output. tsc is clean, ESLint reports 0 errors and Prettier is clean.

Docs

Known limits

  • Each 256 KiB run is deflated synchronously on the main thread (a few ms per run).
  • sendCommentSummary only logs a Postgres stream error and never answers, so a failure there leaves the request hanging. This is true of the report export today as well, and this PR does not change that function.

No behaviour change when AWS_S3_BUCKET_NAME is set and OFFLINE unset. No schema change.

🤖 Generated with Claude Code

https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s

colinmegill and others added 16 commits October 3, 2026 00:46
One file names the vote sign. It owns the wire convention (frozen at
agree = -1; named WIRE_AGREE/WIRE_DISAGREE/WIRE_PASS), the export
convention (fixed at agree = +1), and the storage convention, read from
an injectable ConventionSource whose only implementation today is the
constant STORAGE_AGREE_VALUE = -1. The pure conversions map between the
three numeric forms and a typed Vote ('agree' | 'disagree' | 'pass').

NULL in is NULL out for every conversion; the CSV exports' NULL cell is
the named EXPORT_NULL_VALUE (0, as today). Out-of-range integers throw
unless the caller opts into "skip" (semantic reads) or "keep" (keep
today's bytes), so a legacy tolerance is visible at the call site.

Unit tests cover every function, NULL, out-of-range values, round trips
under both storage conventions, and an injected source.

No call site changes in this commit. No schema change. No API change.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s
report.ts had three hand-written flips (votes.csv String(-row.vote),
participant-votes.csv and participant-importance.csv -row.vote) and two
opposite literal tables in one file: comments.csv counted -1 as agree,
participant-votes.csv counted 1 as agree, each right only because of
where the flip sat. Both now classify the stored value once through
storageToSemantic and count by name; the cells are written by
storageToExport (agree = +1). The importance export never used the
flipped value, so its map is gone and n-votes counts the voted tids.

The bytes do not change: a new test pins every vote column of the four
exports for agree, disagree, pass, NULL (written 0, as before) and an
out-of-range stored value (written negated, as before), and passes on
both the old and the new report.ts.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s
The import swapped exact +1/-1 by hand ("INTENTIONAL VOTE FLIPPING,
REMOVE AFTER VOTES REFACTOR"). vote_value is in the export convention
(agree = +1, as the admin import screen documents), so the reader now
converts it with exportToStorage. Values other than -1/0/1 are still
stored as given ({ onInvalid: "keep" }); a new test pins that, and it
passes on the old and new code. Refusing them belongs to the declared
import format change, not here.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s
POST /votes and POST /comments parse the vote with
getIntInRange(WIRE_VOTE_MIN, WIRE_VOTE_MAX) instead of a bare (-1, 1).
doVotesPost, the single INSERT reached from votes, comments and the bulk
seed upload, converts the wire value with wireToStorage before writing;
today wire and storage agree (-1), so the stored value is unchanged, and
when the storage convention changes only the source changes. The seed
comments' default author vote is WIRE_PASS rather than a bare 0. The
(unreached) modOptions branch reads the vote by name rather than through
the push/pull reaction literals.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s
- GET /votes, GET /votes/me and participationInit's votes return each row
  through storageRowToWire (wire = storage today, so the value and the
  JSON key order are unchanged; an out-of-range stored value is returned
  as stored, as before).
- The moderation list counts (comment.ts) and the famous/ptptoi letter
  vectors (server-helpers.ts) classify the stored value by name through
  storageToSemantic instead of the push/pull reaction literals; NULL and
  unknown values keep their old treatment (counted nowhere, left 'u').
- The collective statement query builds its CASE predicates from
  storageSqlValue; the SQL text is the same as before.
- polisTypes.reactions (push/pull/see/pass), which never said agree or
  disagree, is retired; nothing reads it any more.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s
A unit test reads every server source file (src/, app.ts, index.ts, bin/,
scripts/) except src/votes/convention.ts and fails on a line that names
a vote and carries a sign literal or a negation: -1, === 1, * -1,
-row.vote, vote < 0, vote = 1 in SQL, or the retired push/pull reaction
names. It also checks that the patterns catch the shapes the audit found
(and do not catch group-1 or an index sentinel), and that every
allowlist entry still matches, so the list can only shrink.

After the routing commits the scan finds nothing and the allowlist is
empty.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s
The witness executes the real votesGet / handle_GET_votes_me source in a
vm context. Those routes now call storageRowToWire, so the witness
transpiles src/votes/convention.ts as-is and exposes its exports in the
same context (nothing is copied). A source tree that predates the module
gets none, since its routes do not reference it.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s
…ion module

The wire witness now runs votes.ts together with the real
src/votes/convention.ts, so the test's copied source tree includes it,
and mutation A targets the routed read (storageRowToWire) instead of the
old bare resolve(results.rows). With a local passfile DSN all eight wire
tests pass; in CI they fail the same way before and after this change
(the fixture's password DSN is refused by the DSN admission check).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s
…format.json)

The CSV exports use agree = 1, disagree = -1, pass = 0, but no file said
so. The convention module now builds one declaration from its export
constants,

  agree=+1;disagree=-1;pass=0;format=polis-export/1

and the exports carry it without changing a CSV header or cell:

- summary.csv gains one last key/value row, vote-convention,<declaration>.
  Every row above it keeps its position and bytes.
- /api/v3/reportExport/:report_id/format.json is a new sidecar beside the
  CSVs: the format id, the declaration, the vote values, and per file
  which columns carry a vote value, a count, or an importance flag.

The module also reads a declaration back (the declaration string or a
format.json document) and refuses one that is malformed or that names
another convention, with two closed codes; the votes-bulk import uses
that next.

The byte pins gain the summary.csv and format.json goldens (fixture
files under __tests__/fixtures/vote-declaration), and assert that
summary.csv minus its last row is the pre-declaration bytes. The vote
CSVs' pins are untouched.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s
A votes-bulk CSV may now declare its sign, on a first line before the
header,

  # vote-convention: agree=+1;disagree=-1;pass=0;format=polis-export/1

or as a format field in the request body with format.json's shape.

- Absent: the file is read in the export convention, exactly as before
  (the route stores the same bytes; the worker reads the same rows).
- Present and the export convention: accepted. The worker drops the
  declaration line before the CSV header is parsed and converts every
  value through exportToStorage, as for an undeclared file.
- Present and another sign or format: refused with
  polis_err_vote_convention_declaration_mismatch; unreadable:
  polis_err_vote_convention_declaration_malformed. The route answers
  400 before anything is stored or enqueued; the worker checks the
  stored file's first line again and fails the job with the same code
  before opening a transaction.

Any other first line, including another # line, still reaches the CSV
parser as the header.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s
docs/export-format.md is the in-repo statement of the sign of every
export file and column (until now only the off-repo export page and the
admin import screen said it): the export convention, the summary.csv row
and format.json sidecar that declare it, each file's vote-valued and
count columns, and the votes-bulk import rule with its two closed error
codes.

Linked from docs/configuration.md, delphi/docs/QUICK_START.md, the
report page's raw data export section (which also links format.json and
lists its curl line) and the admin vote import screen (which mentions
the optional first-line declaration).

A unit test reads the convention module's declaration cases from a
fixture file and checks that the page states the declaration, the
closed codes, and every file and column format.json lists.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s
…lumns

The empty-math integration test compares loadConversationSummary with
rows recorded on edge. It now asserts that the last row is the
vote-convention declaration and compares the rows before it with the
golden, which is not re-recorded on this branch.

format.json and docs/export-format.md now also list the two importance
columns, neither of them a vote value: votes.csv important (a flag) and
comments.csv importance (a count). Fixes the test file name in
exportFormat.ts's header.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s
…en no S3 bucket is configured

GET /api/v3/dataExport queued a generate_export_data task for the Clojure
export worker, which has not consumed worker_tasks since 2022, and its
results route signs a URL into the AWS polis-datadump bucket. With
AWS_S3_BUCKET_NAME unset, or OFFLINE set, the route now builds the zip on
request instead: format.json, summary.csv, comments.csv, votes.csv and
participant-votes.csv from the report export functions, streamed entry by
entry (deflate in 256 KiB sync-flushed runs, data descriptors, no zip64).
A conversation over DATA_EXPORT_MAX_CELLS (max of vote rows and voters x
comments; default 20,000,000) is refused with 413
polis_err_data_export_too_large before any byte. The results route answers
404 polis_err_data_export_results_not_stored in that mode. The ownership
gate is unchanged.

The OFFLINE config key is #2937's definition, verbatim, so the two merge
cleanly. With a bucket set and OFFLINE unset nothing changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s
…o the report export

Generated fixture conversation written straight to Postgres; both handlers
mounted on a bare Express app; every zip entry compared with the bytes
/api/v3/reportExport serves. Also the OFFLINE-with-bucket mode, the 413
refusal and the unchanged ownership refusal.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s
… its reads to the client

Review of #2940 (F-1): a client that disconnected left the handler pending
forever (a destroyed response never emits finish), and the producers read
and compressed the rest of the conversation into a dead socket.

- src/db/streamControl.ts: an AsyncLocalStorage context for the Postgres
  row streams. stream_queryP_readOnly, inside it, destroys the running
  stream on abort (cursor closed, client discarded), fails a stream that
  has not started, and pauses while the output needs draining. Outside a
  context nothing changes.
- writeConversationZip aborts on the response's close and runs each
  producer inside the context with the response as the output.
- ZipStreamWriter: writes to a destroyed or ended output throw
  polis_err_zip_output_closed; finish() resolves on finish and rejects on
  close. Each entry also carries the UTC extended timestamp (0x5455).
- The handler logs a client close at info and returns.
- The size count reads votes once (COUNT and COUNT DISTINCT together).
- F-2: positiveIntOr no longer sits under isTrueOrBlank's comment.

Tests: unit for the destroyed output, the abort reaching the producer and
the handler returning; unit for the stream control (abort mid-stream,
abort before start, pause and resume). Integration on a 600,000-vote
generated fixture: a client that stops reading holds the votes.csv
query paused, and on disconnect the handler returns and the backend is
gone; a reading client gets every row.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s
@github-actions

github-actions Bot commented Oct 3, 2026

Copy link
Copy Markdown
Contributor

Delphi Coverage Report

File Stmts Miss Cover
init.py 2 0 100%
benchmarks/bench_pca.py 128 107 16%
benchmarks/bench_repness.py 81 65 20%
benchmarks/bench_update_votes.py 38 28 26%
benchmarks/benchmark_utils.py 34 0 100%
components/init.py 1 0 100%
components/config.py 165 70 58%
conversation/init.py 2 0 100%
conversation/conversation.py 1290 264 80%
conversation/group_vote_counts.py 38 2 95%
conversation/manager.py 131 42 68%
database/init.py 1 0 100%
database/dynamodb.py 395 178 55%
database/postgres.py 399 101 75%
empty_output.py 29 2 93%
engine_adapter.py 316 264 16%
pca_kmeans_rep/init.py 5 0 100%
pca_kmeans_rep/clusters.py 257 38 85%
pca_kmeans_rep/corr.py 98 17 83%
pca_kmeans_rep/group_k_smoother.py 28 0 100%
pca_kmeans_rep/legacy_kmeans.py 197 7 96%
pca_kmeans_rep/pca.py 162 19 88%
pca_kmeans_rep/repness.py 226 5 98%
poller/init.py 4 0 100%
poller/admission.py 318 15 95%
poller/backfill.py 936 32 97%
poller/capacity.py 444 17 96%
poller/capacity_manifest.py 260 13 95%
poller/coordinator_bridge.py 316 316 0%
poller/large_class.py 149 2 99%
poller/math_writer.py 113 3 97%
poller/promotion.py 127 2 98%
poller/readiness.py 368 12 97%
poller/service.py 641 39 94%
poller/worker_pool.py 140 0 100%
queue/executor.py 223 111 50%
regression/init.py 4 0 100%
regression/clojure_comparer.py 188 20 89%
regression/comparer.py 891 640 28%
regression/datasets.py 135 27 80%
regression/recorder.py 36 27 25%
regression/utils.py 130 85 35%
replay/init.py 2 0 100%
replay/attribution_capture.py 49 0 100%
replay/axis_continuity.py 347 24 93%
replay/certify.py 999 59 94%
replay/crosslang.py 128 2 98%
replay/decision_ties.py 55 8 85%
replay/diagnostics.py 50 7 86%
replay/driver.py 73 1 99%
replay/dsn_admission.py 36 1 97%
replay/equiv_query.py 10 0 100%
replay/event_ingress.py 95 1 99%
replay/fixture_bundle.py 815 62 92%
replay/fixture_config.py 266 11 96%
replay/fixture_extract.py 283 12 96%
replay/fixture_generate.py 103 1 99%
replay/fixture_samples.py 157 14 91%
replay/fixture_selection.py 119 0 100%
replay/fixture_survey.py 147 11 93%
replay/legacy_pca.py 29 17 41%
replay/polarity.py 319 25 92%
replay/poller_equiv.py 777 99 87%
replay/prodclone.py 193 2 99%
replay/real_data.py 146 10 93%
replay/schedule.py 205 10 95%
replay/served_math.py 159 4 97%
replay/shard_bench.py 189 65 66%
replay/stagecompare.py 890 161 82%
replay/stages.py 256 40 84%
replay/stepcompare.py 86 1 99%
replay/store.py 120 11 91%
replay/tie_capture.py 107 19 82%
replay/types.py 71 6 92%
run_math_pipeline.py 264 114 57%
types.py 22 0 100%
umap_narrative/501_calculate_comment_extremity.py 112 53 53%
umap_narrative/502_calculate_priorities.py 135 135 0%
umap_narrative/700_datamapplot_for_layer.py 502 502 0%
umap_narrative/701_static_datamapplot_for_layer.py 310 310 0%
umap_narrative/702_consensus_divisive_datamapplot.py 434 275 37%
umap_narrative/801_narrative_report_batch.py 790 726 8%
umap_narrative/803_check_batch_status.py 185 185 0%
umap_narrative/llm_factory_constructor/init.py 2 2 0%
umap_narrative/llm_factory_constructor/model_provider.py 234 234 0%
umap_narrative/polismath_commentgraph/init.py 1 0 100%
umap_narrative/polismath_commentgraph/cli.py 270 270 0%
umap_narrative/polismath_commentgraph/core/init.py 3 3 0%
umap_narrative/polismath_commentgraph/core/clustering.py 108 108 0%
umap_narrative/polismath_commentgraph/core/embedding.py 104 104 0%
umap_narrative/polismath_commentgraph/lambda_handler.py 219 219 0%
umap_narrative/polismath_commentgraph/schemas/init.py 2 0 100%
umap_narrative/polismath_commentgraph/schemas/dynamo_models.py 160 9 94%
umap_narrative/polismath_commentgraph/tests/conftest.py 17 17 0%
umap_narrative/polismath_commentgraph/tests/test_clustering.py 74 74 0%
umap_narrative/polismath_commentgraph/tests/test_embedding.py 55 55 0%
umap_narrative/polismath_commentgraph/tests/test_storage.py 87 87 0%
umap_narrative/polismath_commentgraph/utils/init.py 3 0 100%
umap_narrative/polismath_commentgraph/utils/converter.py 283 237 16%
umap_narrative/polismath_commentgraph/utils/group_data.py 331 289 13%
umap_narrative/polismath_commentgraph/utils/storage.py 584 470 20%
umap_narrative/reset_conversation.py 159 50 69%
umap_narrative/run_pipeline.py 374 230 38%
umap_narrative/topic_naming.py 148 148 0%
utils/clj_hash.py 44 0 100%
utils/env_flags.py 13 0 100%
utils/general.py 63 39 38%
utils/output_profile.py 66 2 97%
utils/serialization.py 10 5 50%
utils/vote_convention.py 29 2 93%
Total 22924 8208 64%

colinmegill added a commit that referenced this pull request Oct 3, 2026
…ocal narratives

example.offline.env sets OLLAMA_HOST=http://ollama:11434: the server
reads it for the narrative and collective statement and would otherwise
use localhost. docs/offline.md takes the lines the data export PR
(#2940) and the Ollama narrative PR (#2945) left for it: checklist item 6
adds the conversation data export zip, item 7 has the narrative and
collective statement from the local model, and the known gaps drop the
Anthropic-only line and describe the streamed export.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

math worker crashes when run in dev mode

1 participant