Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 24 additions & 0 deletions .github/workflows/client-topic-map.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
name: Topic map package
permissions:
contents: read
on:
pull_request:
paths:
- 'client-topic-map/**'
- 'ci/local/**'
- 'Makefile'
- '.github/workflows/client-topic-map.yml'
push:
branches: [edge]
concurrency:
group: client-topic-map-${{ github.ref }}
cancel-in-progress: true
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: '24.22.1'
- run: make check-client-topic-map
15 changes: 13 additions & 2 deletions .github/workflows/queue-rs-ci.yml
Original file line number Diff line number Diff line change
@@ -1,8 +1,9 @@
name: queue-rs

# Build, lint and test the queue-rs workspace: the polis-queue/1 adapter and
# the polis-jobs daemon (job `queue-rs`), and the polis-api route server with
# its nginx routing switch (job `polis-api`). The integration tests run against a throwaway
# the polis-jobs daemon (job `queue-rs`), the polis-api route server with
# its nginx routing switch (job `polis-api`), and the topic map generator
# (job `polis-topic-map`). The integration tests run against a throwaway
# PostgreSQL 17 service container with the repository's migration chain,
# 000023 (the Delphi job table) included.
on:
Expand Down Expand Up @@ -230,6 +231,16 @@ jobs:
env:
GITHUB_EVENT_NAME: ${{ github.event_name }}
CHECK_BASE_REF: origin/${{ github.base_ref }}
polis-topic-map:
name: topic map generator build, clippy and tests
runs-on: ubuntu-24.04
timeout-minutes: 15
steps:
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
with:
persist-credentials: false
- name: Run polis-topic-map checks
run: make check-polis-topic-map
job-graphs:
name: Per-step job graph core
runs-on: ubuntu-24.04
Expand Down
8 changes: 8 additions & 0 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -328,6 +328,10 @@ check-client-admin: ## Run the client-admin verification commands
check-client-report: ## Run the client-report verification commands
@bash ci/local/client-report.sh

.PHONY: check-client-topic-map
check-client-topic-map: ## Run the topic map package's tests
@bash ci/local/client-topic-map.sh

.PHONY: check-lint
check-lint: ## Run the lint verification commands
@bash ci/local/lint.sh
Expand Down Expand Up @@ -364,6 +368,10 @@ check-queue-rs: ## Run queue-rs checks
check-polis-migrate: ## Run polis-migrate checks
@bash ci/local/polis-migrate.sh

.PHONY: check-polis-topic-map
check-polis-topic-map: ## Run the topic map generator's checks
@bash ci/local/polis-topic-map.sh

.PHONY: check-job-graphs
check-job-graphs: ## Run job-graphs checks
@bash ci/local/job-graphs.sh
Expand Down
2 changes: 1 addition & 1 deletion VERSION
Original file line number Diff line number Diff line change
@@ -1 +1 @@
2.43.10
2.44.0
1 change: 1 addition & 0 deletions cdk/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,7 @@ The deployment process is orchestrated via GitHub Actions:
2. Start an instance refresh on `AsgDelphiSmall` and wait until the `r7i.2xlarge` instance is InService.
3. Only then raise `DELPHI_POLLER_CONTAINER_MEMORY` in the `polis-web-app-env-vars` secret from `6g` to `48g`. With Delphi's `DELPHI_CONTAINER_MEMORY=8g` beside it, that leaves about 8 GiB of the 64 GiB for the host and the file cache; keep at least that much free. Raising it before the new box is up would let the poller outgrow a 16 GiB host.
4. Redeploy the application so the hooks rewrite `.env` and recreate the `math-python` container with the new limit.
* **Delphi small box: 200 GiB root volume:** `DelphiSmallLaunchTemplate` ([launchTemplates.ts](launchTemplates.ts)) gives the box a 200 GiB gp3 root volume (it was 50 GiB). The root volume holds every image the box pulls, two sets of them while a deploy runs, and the Delphi shadow's scheduler stops admitting under 10 GiB free. As with the instance type, a `cdk deploy` writes only a new launch-template version (the group has no update policy that replaces instances); the running box keeps its volume until it is replaced or its volume is grown in place. Growing it in place needs no restart or planned outage: `aws ec2 modify-volume --volume-id <verified-root-volume-id> --size 200`; then, only once that modification is `optimizing` or `completed` and the box's root has been verified to be XFS directly on partition 1 of that volume (`/dev/nvme0n1p1` on Amazon Linux 2023, with partition 1 the last partition on the disk), on the box `sudo growpart /dev/nvme0n1 1` and then `sudo xfs_growfs -d /`. A volume cannot be shrunk.
* **Health Checks:** Health checks are configured on the Application Load Balancer's target group, pointing to `/api/v3/testConnection`. These health checks determine the health of the *web server* instances. The CDK code does *not* define custom health checks for the math worker instances, relying instead on the default EC2 instance status checks.
* **Race Condition:** A potential race condition exists during CDK updates that modify the EC2 instance configuration (e.g., changes to the launch template, security groups, or UserData). If a CDK update causes the EC2 instances to fail during launch, the ASG will continuously attempt to launch new instances, while CodeDeploy will be waiting for these instances to become healthy. This can lead to a deployment deadlock. *Thoroughly review and test any changes to the EC2 instance configuration before deploying to production.*
* **Environment Variables:** Environment variables and secrets for the application should be managed through AWS Secrets Manager (specifically, the `polis-web-app-env-vars` secret).
Expand Down
6 changes: 5 additions & 1 deletion cdk/launchTemplates.ts
Original file line number Diff line number Diff line change
Expand Up @@ -280,10 +280,14 @@ if (enableOllama) {
securityGroup: delphiSecurityGroup,
keyPair: delphiSmallKeyPair,
role: instanceRole,
// 200 GiB: the root volume holds every image this box pulls (two sets of
// them during a deploy) and the Delphi shadow's scheduler stops admitting
// under 10 GiB free. Only boxes launched from this template get it; a
// running box keeps its volume (cdk/README.md).
blockDevices: [
{
deviceName: '/dev/xvda',
volume: ec2.BlockDeviceVolume.ebs(50, {
volume: ec2.BlockDeviceVolume.ebs(200, {
volumeType: ec2.EbsDeviceVolumeType.GP3,
deleteOnTermination: true,
}),
Expand Down
2 changes: 1 addition & 1 deletion cdk/test/__snapshots__/workerClasses.test.ts.snap
Original file line number Diff line number Diff line change
Expand Up @@ -2295,7 +2295,7 @@ rm -fr $TMP_DIR",
"DeviceName": "/dev/xvda",
"Ebs": {
"DeleteOnTermination": true,
"VolumeSize": 50,
"VolumeSize": 200,
"VolumeType": "gp3",
},
},
Expand Down
6 changes: 6 additions & 0 deletions ci/local/client-topic-map.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
#!/usr/bin/env bash
. "$(dirname "$0")/runtime.sh"
check_node 24.22.1

# The package has no dependencies: Node's own test runner runs its tests.
node --test client-topic-map/test/*.test.js
12 changes: 12 additions & 0 deletions ci/local/polis-topic-map.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
#!/usr/bin/env bash
. "$(dirname "$0")/runtime.sh"

(
cd "$CHECK_ROOT/queue-rs"
rustup show active-toolchain
cargo fmt -p polis-topic-map --check
cargo clippy --locked -p polis-topic-map --all-targets -- -D warnings
# The datamapplot parity tests and the recorded document, on the two public
# fixtures in polis-topic-map/tests/fixtures.
cargo test --locked -p polis-topic-map
)
3 changes: 3 additions & 0 deletions ci/local/queue-rs.sh
Original file line number Diff line number Diff line change
Expand Up @@ -25,4 +25,7 @@ python3 "$CHECK_ROOT/ci/host_ports.py" env POLIS_JOBS_TEST_DATABASE_URL
export POLIS_JOBS_TEST_DATABASE_URL
export POLIS_JOBS_TEST_PYTHON="${POLIS_JOBS_TEST_PYTHON:-python3}"
cargo test --locked --features jobs-integration --test jobs_integration -- --test-threads 4
# The Delphi shadow's scheduler: what it admits, and that its sessions never
# make a vote, a comment or the next comment wait.
cargo test --locked --features jobs-integration --test delphi_shadow -- --test-threads 2
)
189 changes: 189 additions & 0 deletions client-topic-map/NOTICE
Original file line number Diff line number Diff line change
@@ -0,0 +1,189 @@
Notices for work this package is derived from or carries

1. datamapplot 0.6.4 (https://github.com/TutteInstitute/datamapplot)

The label rules in topic-map.js (how a name wraps, its type size, which names
give way to which), the first view, the selection figures, and the values in
theme-neutral.js marked "datamapplot" are ported from or reproduce
datamapplot's create_plots.py, interactive_rendering.py and static/js/datamap.js.

MIT License

Copyright (c) 2023 Tutte Institute for Mathematics and Computing

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

2. deck.gl (https://github.com/visgl/deck.gl), which draws datamapplot's maps

topic-map.js reproduces deck.gl's drawing of those maps: the Web Mercator world
512 units wide, the zoom step of one wheel event (wheelScale), and the test of
25 pixels around a label's anchor with the label faded by the share it owns,
raised to 2.2 (resolveCollisions, _claimsRead). theme-neutral.js reproduces
deck.gl's zoom durations and tooltip padding.

Copyright Vis.gl contributors.

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.

3. mjolnir.js (https://github.com/visgl/mjolnir.js), deck.gl's input layer

wheelDelta in topic-map.js ports how mjolnir.js reports a wheel event: lines
counted as 40 pixels, the 4.000244140625 step of some mouse wheels, and a
quarter of the step with Shift held.

MIT License

Copyright (c) 2017 Uber Technologies, Inc.

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

4. Python's textwrap module (https://github.com/python/cpython, Lib/textwrap.py)

The port was made from Lib/textwrap.py of Python 3.12, the Python that runs
datamapplot in the Delphi image; that file is the same in every 3.12 release
from v3.12.0. Its copyright notices:

Copyright (C) 1999-2001 Gregory P. Ward.
Copyright (C) 2002, 2003 Python Software Foundation.
Written by Greg Ward <gward@python.net>

datamapplot wraps a label with Python's textwrap.fill(text, width,
break_long_words=False). wrapLabel in topic-map.js ports it: WORDSEP is
TextWrapper.wordsep_re written as a JavaScript regular expression, and the
loop that fills lines is TextWrapper._wrap_chunks for that one setting. Summary
of the changes: translated from Python to JavaScript; only the path for
break_long_words=False, with TextWrapper's other defaults and no indent,
placeholder or line limit, is kept; Python's word and letter classes are
written as Unicode property classes; a tab becomes one space rather than
spaces to the next tab stop; in the second look-behind for a hyphenated word
(letter, hyphen, letter, hyphen in Python), the character after the first
hyphen is matched by the class of word characters and punctuation.

Copyright (c) 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010,
2011, 2012, 2013, 2014, 2015, 2016, 2017, 2018, 2019, 2020, 2021, 2022, 2023 Python Software Foundation;
All Rights Reserved

The licence of the Python Software Foundation, as in the LICENSE file of
Python 3.12's source release:

PYTHON SOFTWARE FOUNDATION LICENSE VERSION 2
--------------------------------------------

1. This LICENSE AGREEMENT is between the Python Software Foundation
("PSF"), and the Individual or Organization ("Licensee") accessing and
otherwise using this software ("Python") in source or binary form and
its associated documentation.

2. Subject to the terms and conditions of this License Agreement, PSF hereby
grants Licensee a nonexclusive, royalty-free, world-wide license to reproduce,
analyze, test, perform and/or display publicly, prepare derivative works,
distribute, and otherwise use Python alone or in any derivative version,
provided, however, that PSF's License Agreement and PSF's notice of copyright,
i.e., "Copyright (c) 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010,
2011, 2012, 2013, 2014, 2015, 2016, 2017, 2018, 2019, 2020, 2021, 2022, 2023 Python Software Foundation;
All Rights Reserved" are retained in Python alone or in any derivative version
prepared by Licensee.

3. In the event Licensee prepares a derivative work that is based on
or incorporates Python or any part thereof, and wants to make
the derivative work available to others as provided herein, then
Licensee hereby agrees to include in any such work a brief summary of
the changes made to Python.

4. PSF is making Python available to Licensee on an "AS IS"
basis. PSF MAKES NO REPRESENTATIONS OR WARRANTIES, EXPRESS OR
IMPLIED. BY WAY OF EXAMPLE, BUT NOT LIMITATION, PSF MAKES NO AND
DISCLAIMS ANY REPRESENTATION OR WARRANTY OF MERCHANTABILITY OR FITNESS
FOR ANY PARTICULAR PURPOSE OR THAT THE USE OF PYTHON WILL NOT
INFRINGE ANY THIRD PARTY RIGHTS.

5. PSF SHALL NOT BE LIABLE TO LICENSEE OR ANY OTHER USERS OF PYTHON
FOR ANY INCIDENTAL, SPECIAL, OR CONSEQUENTIAL DAMAGES OR LOSS AS
A RESULT OF MODIFYING, DISTRIBUTING, OR OTHERWISE USING PYTHON,
OR ANY DERIVATIVE THEREOF, EVEN IF ADVISED OF THE POSSIBILITY THEREOF.

6. This License Agreement will automatically terminate upon a material
breach of its terms and conditions.

7. Nothing in this License Agreement shall be deemed to create any
relationship of agency, partnership, or joint venture between PSF and
Licensee. This License Agreement does not grant permission to use PSF
trademarks or trade name in a trademark sense to endorse or promote
products or services of Licensee, or any third party.

8. By copying, installing or otherwise using Python, Licensee
agrees to be bound by the terms and conditions of this License
Agreement.

5. Roboto (fonts/Roboto-latin-wght.woff2)

Roboto, variable weight, latin subset. Copyright 2011 The Roboto Project
Authors. SIL Open Font License, Version 1.1; the full text is fonts/OFL.txt.

6. The public fixtures (test/fixtures/)

Two public conversations from https://github.com/compdemocracy/openData,
licensed CC BY 4.0: bg2050-volunteers and scoop-hivemind.biodiversity. Data was
gathered using the Polis software (see: compdemocracy.org/polis and
github.com/compdemocracy/polis) and is sub-licensed under CC BY 4.0 with
Attribution to The Computational Democracy Project.

- biodiversity.topic-map.json is the polis-topic-map/1 document of one Delphi
run of scoop-hivemind.biodiversity (154 comments moderated in, of 316): the
comments' positions (embeddings all-MiniLM-L6-v2, UMAP), their topics at two
levels (EVoC clustering) and the comments' text. Its topic names were written
by a small local model (qwen2.5:1.5b) and are not the names pol.is shows. Its
topic colours were computed from the CET-C2s colour map (colorcet 3.1.0;
Peter Kovesi, "Good Colour Maps: How to Design Them", arXiv:1509.03700,
Creative Commons Attribution 4.0 International License).
- datamapplot-labels.json holds, for one Delphi run of each of the two
conversations, the topic names by level and the label strings datamapplot
0.6.4 made of them; nothing else.
Loading
Loading