Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
2 changes: 1 addition & 1 deletion VERSION
Original file line number Diff line number Diff line change
@@ -1 +1 @@
2.43.5
2.43.6
19 changes: 19 additions & 0 deletions ci/local/polis-api.sh
Original file line number Diff line number Diff line change
Expand Up @@ -41,12 +41,31 @@ check_project "$IMPORTANCE_PROJECT"
# Keep aggregate test pools below PostgreSQL's default connection limit.
# Pin this here so local and hosted checks use the same database budget.
export RUST_TEST_THREADS=2
# The first pass runs on test databases as PostgreSQL creates them. An
# interrupted earlier run may have left the second pass's shape installed.
cargo run --locked -p polis-api --example older_database_shape -- remove
cargo test --locked -p polis-api --features db-tests
# Compare timely replies with full selections from the same run. Execute
# the optimized queue test in both local and hosted CI.
cargo test --release --locked -p polis-api --features db-tests sustained_votes_keep_delivering -- --nocapture
)

(
cd "$CHECK_ROOT/queue-rs"
export POLIS_API_TEST_DATABASE_URL
export POLIS_AUTH_TEST_DATABASE_URL
export RUST_TEST_THREADS=2
# The second pass: the same tests, unchanged, on test databases in the shape
# of an installation whose database predates the migrations (json math
# documents, another column order, a few other columns). While
# polis-api/tests/older_database_shape.sql is installed in the test
# database and in template1, the tables it lists take that shape as the
# tests create them. It is removed again whatever the tests answer.
trap 'cargo run --locked -p polis-api --example older_database_shape -- remove' EXIT
cargo run --locked -p polis-api --example older_database_shape -- install
POLIS_API_TEST_DATABASE_SHAPE=older cargo test --locked -p polis-api --features db-tests
)

(
cd "$CHECK_ROOT/queue-rs"
cargo build --locked --release -p polis-api
Expand Down
2 changes: 1 addition & 1 deletion client-report/src/components/app.jsx
Original file line number Diff line number Diff line change
Expand Up @@ -264,7 +264,7 @@ const App = (props) => {
searchParamsCache ? `&noCache=${searchParamsCache}` : ``
}`,
{
credentials: "include",
credentials: "omit",
method: "get",
headers: {
Accept: "application/json, text/plain, */*",
Expand Down
67 changes: 67 additions & 0 deletions client-report/src/components/narrativeFetch.test.jsx
Original file line number Diff line number Diff line change
@@ -0,0 +1,67 @@
import React from 'react';
import { render, waitFor } from '@testing-library/react';

// The narrative fetch must send no credentials: the report uses no cookies, and
// a credentialed request is refused by the browser when the API answers
// `Access-Control-Allow-Origin: *` to a cross-origin caller.

const mockUser = { access_token: 'test-token' };
jest.mock('react-oidc-context', () => ({
useAuth: () => ({
isAuthenticated: true,
isLoading: false,
user: mockUser,
signinRedirect: jest.fn(),
signoutRedirect: jest.fn(),
removeUser: jest.fn(),
}),
}));

jest.mock('../util/net.js', () => ({
__esModule: true,
default: { polisGet: jest.fn(() => Promise.resolve([])), polisPost: jest.fn(() => Promise.resolve({})) },
}));

jest.mock('./globals.js', () => ({
brandColors: { agree: 'green', disagree: 'red', pass: 'yellow' },
}));

describe('narrative fetch', () => {
const originalClientId = process.env.AUTH_CLIENT_ID;

beforeAll(() => {
jest.spyOn(console, 'log').mockImplementation(() => {});
jest.spyOn(console, 'error').mockImplementation(() => {});
jest.spyOn(console, 'warn').mockImplementation(() => {});
process.env.AUTH_CLIENT_ID = 'test-client';
window.history.pushState({}, '', '/narrativeReport/r1abc');
});

afterAll(() => {
if (originalClientId === undefined) delete process.env.AUTH_CLIENT_ID;
else process.env.AUTH_CLIENT_ID = originalClientId;
jest.restoreAllMocks();
});

test('passes credentials "omit"', async () => {
const net = require('../util/net.js').default;
net.polisGet.mockImplementation((path) =>
path === '/api/v3/reports'
? Promise.resolve([{ report_id: 'r1abc', conversation_id: 'c1' }])
: Promise.resolve({})
);
fetch.mockResponse('');
const App = require('./app').default;
render(<App />);

await waitFor(() => {
expect(
fetch.mock.calls.some(([url]) => String(url).includes('api/v3/reportNarrative'))
).toBe(true);
});
const [, options] = fetch.mock.calls.find(([url]) =>
String(url).includes('api/v3/reportNarrative')
);
expect(options.credentials).toBe('omit');
});
});
490 changes: 245 additions & 245 deletions client-report/src/data/__tests__/baseline-view-source.json

Large diffs are not rendered by default.

8 changes: 0 additions & 8 deletions docs/configuration-env-reference.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,14 +57,6 @@ Use [configuration guidance](configuration.md) and [deployment/service inventory
| `DD_ENV` | [server/src/config.ts:111](../server/src/config.ts#L111) | undefined at lookup; no fallback in this expression | No (configuration/identifier; may still be private) | application / script / infrastructure; TypeScript AST |
| `DEV_MODE` | [server/src/config.ts:6](../server/src/config.ts#L6) | false when absent (isTrue conversion) | No (configuration/identifier; may still be private) | application / script / infrastructure; TypeScript AST |
| `DOMAIN_OVERRIDE` | [server/src/config.ts:7](../server/src/config.ts#L7) | fallback expression: process.env.DOMAIN_OVERRIDE &#124;&#124; null | No (configuration/identifier; may still be private) | application / script / infrastructure; TypeScript AST |
| `DOMAIN_WHITELIST_ITEM_01` | [server/src/config.ts:182](../server/src/config.ts#L182) | fallback expression: process.env.DOMAIN_WHITELIST_ITEM_01 &#124;&#124; null | No (configuration/identifier; may still be private) | application / script / infrastructure; TypeScript AST |
| `DOMAIN_WHITELIST_ITEM_02` | [server/src/config.ts:183](../server/src/config.ts#L183) | fallback expression: process.env.DOMAIN_WHITELIST_ITEM_02 &#124;&#124; null | No (configuration/identifier; may still be private) | application / script / infrastructure; TypeScript AST |
| `DOMAIN_WHITELIST_ITEM_03` | [server/src/config.ts:184](../server/src/config.ts#L184) | fallback expression: process.env.DOMAIN_WHITELIST_ITEM_03 &#124;&#124; null | No (configuration/identifier; may still be private) | application / script / infrastructure; TypeScript AST |
| `DOMAIN_WHITELIST_ITEM_04` | [server/src/config.ts:185](../server/src/config.ts#L185) | fallback expression: process.env.DOMAIN_WHITELIST_ITEM_04 &#124;&#124; null | No (configuration/identifier; may still be private) | application / script / infrastructure; TypeScript AST |
| `DOMAIN_WHITELIST_ITEM_05` | [server/src/config.ts:186](../server/src/config.ts#L186) | fallback expression: process.env.DOMAIN_WHITELIST_ITEM_05 &#124;&#124; null | No (configuration/identifier; may still be private) | application / script / infrastructure; TypeScript AST |
| `DOMAIN_WHITELIST_ITEM_06` | [server/src/config.ts:187](../server/src/config.ts#L187) | fallback expression: process.env.DOMAIN_WHITELIST_ITEM_06 &#124;&#124; null | No (configuration/identifier; may still be private) | application / script / infrastructure; TypeScript AST |
| `DOMAIN_WHITELIST_ITEM_07` | [server/src/config.ts:188](../server/src/config.ts#L188) | fallback expression: process.env.DOMAIN_WHITELIST_ITEM_07 &#124;&#124; null | No (configuration/identifier; may still be private) | application / script / infrastructure; TypeScript AST |
| `DOMAIN_WHITELIST_ITEM_08` | [server/src/config.ts:189](../server/src/config.ts#L189) | fallback expression: process.env.DOMAIN_WHITELIST_ITEM_08 &#124;&#124; null | No (configuration/identifier; may still be private) | application / script / infrastructure; TypeScript AST |
| `DYNAMODB_ENDPOINT` | [server/src/config.ts:112](../server/src/config.ts#L112) | fallback expression: process.env.DYNAMODB_ENDPOINT &#124;&#124; null | No (configuration/identifier; may still be private) | application / script / infrastructure; TypeScript AST |
| `DYNAMODB_ENDPOINT` | [server/src/config.ts:160](../server/src/config.ts#L160) | undefined at lookup; no fallback in this expression | No (configuration/identifier; may still be private) | application / script / infrastructure; TypeScript AST |
| `DYNAMODB_ENDPOINT` | [server/src/routes/delphi/topics.ts:17](../server/src/routes/delphi/topics.ts#L17) | undefined at lookup; no fallback in this expression | No (configuration/identifier; may still be private) | application / script / infrastructure; TypeScript AST |
Expand Down
1 change: 0 additions & 1 deletion docs/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -201,7 +201,6 @@ the caps above.
- **`API_DEV_HOSTNAME`** defaults to `localhost:5000` in [config:5](../server/src/config.ts#L5); set it to the development hostname and port you actually use.
- **`API_PROD_HOSTNAME`** the hostname of your site (e.g. `pol.is`, or `example.com`). Should match **`DOMAIN_OVERRIDE`**. (In the future these two options may be combined into one.)
- **`DOMAIN_OVERRIDE`** the hostname of your site. Should match **`API_PROD_HOSTNAME`**.
- **`DOMAIN_WHITELIST_ITEM_01`** - **`08`** up to 8 possible additional whitelisted domains for client applications to make API requests from. Typical setups that use the same URL for the API service as for the public-facing web sites do not need to configure these.
- **`EMBED_SERVICE_HOSTNAME`** is a legacy participation build input with fallback `pol.is` ([webpack:86](../client-participation/webpack.config.js#L86)); set it for the target environment. Alpha uses separate `PUBLIC_SERVICE_URL` / `INTERNAL_SERVICE_URL` inputs ([net.ts:13](../client-participation-alpha/src/lib/net.ts#L13)).
- **`SERVICE_URL`** used by client-report to make API calls. Only necessary if client-report is hosted separately from the API service. Can be left blank.
- **`STATIC_FILES_HOST`** Used by the API service to fetch static assets (the compiled client applications) from a static file server. Within the docker compose setup this is `file-server`, but could be an external hostname, such as a CDN or S3 bucket.
Expand Down
2 changes: 1 addition & 1 deletion docs/deployment-configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -151,7 +151,7 @@ The last column lists direct named environment reads in the same source file, if
| 083 | Google Translate | [server/src/comment.ts:54](../server/src/comment.ts#L54) | in scope; application / script / infrastructure | None in this file; inspect imports/arguments/SDK |
| 084 | Google Translate | [server/src/comment.ts:511](../server/src/comment.ts#L511) | in scope; application / script / infrastructure | None in this file; inspect imports/arguments/SDK |
| 085 | Google Translate | [server/src/comment.ts:534](../server/src/comment.ts#L534) | in scope; application / script / infrastructure | None in this file; inspect imports/arguments/SDK |
| 086 | OIDC | [server/src/config.ts:115](../server/src/config.ts#L115) | in scope; application / script / infrastructure | `ADMIN_EMAILS`, `ADMIN_EMAIL_DATA_EXPORT`, `ADMIN_EMAIL_EMAIL_TEST`, `ADMIN_UIDS`, `ANTHROPIC_API_KEY`, `API_DEV_HOSTNAME`, `API_PROD_HOSTNAME`, `API_SERVER_PORT`, `APPLICATION_NAME`, `AUTH0_CLIENT_ID`, `AUTH0_CLIENT_SECRET`, `AUTH0_DOMAIN`, `AUTH_AUDIENCE`, `AUTH_CLIENT_ID`, `AUTH_CLIENT_SECRET`, `AUTH_DOMAIN`, `AUTH_ISSUER`, `AUTH_NAMESPACE`, `AWS_ACCESS_KEY_ID`, `AWS_LOG_GROUP_NAME`, `AWS_REGION`, `AWS_S3_BUCKET_NAME`, `AWS_S3_ENDPOINT`, `AWS_S3_JOB_BUCKET_NAME`, `AWS_S3_PUBLIC_ENDPOINT`, `AWS_SECRET_ACCESS_KEY`, `CACHE_MATH_RESULTS`, `DATABASE_SSL`, `DATABASE_URL`, `DD_ENV`, `DEV_MODE`, `DOMAIN_OVERRIDE`, `DOMAIN_WHITELIST_ITEM_01`, `DOMAIN_WHITELIST_ITEM_02`, `DOMAIN_WHITELIST_ITEM_03`, `DOMAIN_WHITELIST_ITEM_04`, `DOMAIN_WHITELIST_ITEM_05`, `DOMAIN_WHITELIST_ITEM_06`, `DOMAIN_WHITELIST_ITEM_07`, `DOMAIN_WHITELIST_ITEM_08`, `DYNAMODB_ENDPOINT`, `EMAIL_TRANSPORT_TYPES`, `ENCRYPTION_PASSWORD_00001`, `GEMINI_API_KEY`, `GOOGLE_CREDENTIALS_BASE64`, `GOOGLE_CREDS_STRINGIFIED`, `JWKS_URI`, `JWT_PRIVATE_KEY`, `JWT_PRIVATE_KEY_PATH`, `JWT_PUBLIC_KEY`, `JWT_PUBLIC_KEY_PATH`, `LOGIN_CODE_PEPPER`, `MATH_ENV`, `MAX_REPORT_CACHE_DURATION`, `NODE_ENV`, `OPENAI_API_KEY`, `POLIS_FROM_ADDRESS`, `POLIS_JWT_AUDIENCE`, `POLIS_JWT_ISSUER`, `POLIS_QUEUE_SUBSTRATE_ENABLED`, `POLIS_REACHABLE_ERROR_HANDLER`, `PORT`, `READ_ONLY_DATABASE_URL`, `SERVER_LOG_LEVEL`, `SERVER_LOG_TO_FILE`, `SES_ENDPOINT`, `SHOULD_USE_TRANSLATION_API`, `SQS_LOCAL_ENDPOINT`, `SQS_QUEUE_URL`, `STATIC_FILES_ADMIN_PORT`, `STATIC_FILES_HOST`, `STATIC_FILES_PARTICIPATION_PORT`, `STATIC_FILES_PORT`, `TESTING`, `TOPICAL_COMMENT_RATIO`, `USE_NETWORK_HOST` |
| 086 | OIDC | [server/src/config.ts:115](../server/src/config.ts#L115) | in scope; application / script / infrastructure | `ADMIN_EMAILS`, `ADMIN_EMAIL_DATA_EXPORT`, `ADMIN_EMAIL_EMAIL_TEST`, `ADMIN_UIDS`, `ANTHROPIC_API_KEY`, `API_DEV_HOSTNAME`, `API_PROD_HOSTNAME`, `API_SERVER_PORT`, `APPLICATION_NAME`, `AUTH0_CLIENT_ID`, `AUTH0_CLIENT_SECRET`, `AUTH0_DOMAIN`, `AUTH_AUDIENCE`, `AUTH_CLIENT_ID`, `AUTH_CLIENT_SECRET`, `AUTH_DOMAIN`, `AUTH_ISSUER`, `AUTH_NAMESPACE`, `AWS_ACCESS_KEY_ID`, `AWS_LOG_GROUP_NAME`, `AWS_REGION`, `AWS_S3_BUCKET_NAME`, `AWS_S3_ENDPOINT`, `AWS_S3_JOB_BUCKET_NAME`, `AWS_S3_PUBLIC_ENDPOINT`, `AWS_SECRET_ACCESS_KEY`, `CACHE_MATH_RESULTS`, `DATABASE_SSL`, `DATABASE_URL`, `DD_ENV`, `DEV_MODE`, `DOMAIN_OVERRIDE`, `DYNAMODB_ENDPOINT`, `EMAIL_TRANSPORT_TYPES`, `ENCRYPTION_PASSWORD_00001`, `GEMINI_API_KEY`, `GOOGLE_CREDENTIALS_BASE64`, `GOOGLE_CREDS_STRINGIFIED`, `JWKS_URI`, `JWT_PRIVATE_KEY`, `JWT_PRIVATE_KEY_PATH`, `JWT_PUBLIC_KEY`, `JWT_PUBLIC_KEY_PATH`, `LOGIN_CODE_PEPPER`, `MATH_ENV`, `MAX_REPORT_CACHE_DURATION`, `NODE_ENV`, `OPENAI_API_KEY`, `POLIS_FROM_ADDRESS`, `POLIS_JWT_AUDIENCE`, `POLIS_JWT_ISSUER`, `POLIS_QUEUE_SUBSTRATE_ENABLED`, `POLIS_REACHABLE_ERROR_HANDLER`, `PORT`, `READ_ONLY_DATABASE_URL`, `SERVER_LOG_LEVEL`, `SERVER_LOG_TO_FILE`, `SES_ENDPOINT`, `SHOULD_USE_TRANSLATION_API`, `SQS_LOCAL_ENDPOINT`, `SQS_QUEUE_URL`, `STATIC_FILES_ADMIN_PORT`, `STATIC_FILES_HOST`, `STATIC_FILES_PARTICIPATION_PORT`, `STATIC_FILES_PORT`, `TESTING`, `TOPICAL_COMMENT_RATIO`, `USE_NETWORK_HOST` |
| 087 | SES | [server/src/email/senders.ts:11](../server/src/email/senders.ts#L11) | in scope; application / script / infrastructure | None in this file; inspect imports/arguments/SDK |
| 088 | SES | [server/src/email/senders.ts:12](../server/src/email/senders.ts#L12) | in scope; application / script / infrastructure | None in this file; inspect imports/arguments/SDK |
| 089 | SES | [server/src/email/senders.ts:17](../server/src/email/senders.ts#L17) | in scope; application / script / infrastructure | None in this file; inspect imports/arguments/SDK |
Expand Down
44 changes: 44 additions & 0 deletions docs/local-checks.md
Original file line number Diff line number Diff line change
Expand Up @@ -115,6 +115,50 @@ These are sampled peaks, not upper bounds. At a shell open-file limit of 256,
default parallelism also fails tests with "Too many open files"; two threads pass. Two threads leave headroom without requiring a larger test
database; test selection, assertions and concurrency within each test are unchanged.

The API entry point runs its full Rust database test suite twice. The first
pass uses test databases as PostgreSQL creates them, so every table a test
builds has the shape the migrations give it. The second pass runs the same
tests, unchanged, on the shape of an installation whose database predates the
migrations: such a database stores the math documents as `json` where the
migrations build `jsonb`, keeps some columns in another order, has a few
columns the migrations never create and lacks two they do. A statement can
depend on any of that without saying so (a function that exists for `jsonb`
only, a `SELECT *` read by position), and only a database in that shape shows
it. `queue-rs/polis-api/tests/older_database_shape.sql` records the shape of
the thirteen tables that differ and that the API reads, and reshapes a table
with one of those names when it is created; its header says exactly what is
and is not reproduced. Between the passes the entry point installs it in the
test database and in that server's `template1` (every database a test creates
is a copy of `template1`), sets `POLIS_API_TEST_DATABASE_SHAPE=older` for the
second pass, and removes it afterwards, also when the tests fail. Installing
needs a superuser, and `template1` is shared by the whole server, so
`POLIS_API_TEST_DATABASE_URL` must name a server that belongs to this run. One
test in each pass checks that its databases have the shape the pass names, so
a shape left installed by an interrupted run fails the first pass instead of
changing it silently; the first pass also begins by removing it. To run one
pass by hand:

```sh
cd queue-rs
cargo run --locked -p polis-api --example older_database_shape -- install
POLIS_API_TEST_DATABASE_SHAPE=older RUST_TEST_THREADS=2 \
cargo test --locked -p polis-api --features db-tests
cargo run --locked -p polis-api --example older_database_shape -- remove
```

A test that needs a table in one particular column order, in both passes,
creates it under another name and renames it; a test that loads a table by
position declares fewer columns than the migrations' table or names the
columns it inserts. The release-mode queue test runs in the first pass only.

A migration that adds a column to one of the thirteen tables also adds it to
`older_database_shape.sql`, as the last row of that table and marked `later`
(the file's header says why). Until it does,
`every_migration_applies_and_the_listed_tables_take_the_older_shape` fails in
both passes and says so. Most tables the tests declare have fewer columns than
the migrations' tables: in the second pass those keep the order the test gave
them, and only their `jsonb` math documents become `json`.

CLI differential needs
its `CLI_DIFF_*_IMAGE` variables and an independent `SETUP_ADMISSION_PROJECT`.
Mixed identity uses all six `MIXED_*_PORT` / `POLIS_MIXED_WORKER_PORT` values.
Expand Down
8 changes: 0 additions & 8 deletions example.env
Original file line number Diff line number Diff line change
Expand Up @@ -190,14 +190,6 @@ API_DEV_HOSTNAME=localhost
# to the same value. In the future these will be combined into one setting.
API_PROD_HOSTNAME=pol.is
DOMAIN_OVERRIDE=
DOMAIN_WHITELIST_ITEM_01=
DOMAIN_WHITELIST_ITEM_02=
DOMAIN_WHITELIST_ITEM_03=
DOMAIN_WHITELIST_ITEM_04=
DOMAIN_WHITELIST_ITEM_05=
DOMAIN_WHITELIST_ITEM_06=
DOMAIN_WHITELIST_ITEM_07=
DOMAIN_WHITELIST_ITEM_08=
# In production, should match API_PROD_HOSTNAME or DOMAIN_OVERRIDE, if set:
EMBED_SERVICE_HOSTNAME=localhost
# Full hostname with protocol of your API server:
Expand Down
18 changes: 17 additions & 1 deletion file-server/nginx/ROUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -93,12 +93,28 @@ and observe mode"). The shadow answers each copy 204 and writes nothing.
20 ms send, 30 ms read, 30 ms resolve) when it is slow or silent. The body
is read before Node is contacted (`mirror_request_body`); votes are small.
- **What it logs.** Two JSON lines per shadowed vote, both with
`listener: "shadow"` and nginx's `request_id`, beside the usual access line:
`listener: "shadow"` and nginx's `request_id`, beside the access line:
`shadow_edge` (Node's status and time as nginx saw them, the participant's
total time) and `shadow_mirror` (the copy's status and time). A copy skipped
by the body bound logs `mirror_status` 204 with an empty
`mirror_upstream_status`. Neither line carries an address, a header value or
a body.
- **The access line of a shadowed vote records no address.** It is written in
`polis_shadow_access`: the image's `main` format with its two address
fields written `-`, the first (`$remote_addr`, the peer) and the last
(`X-Forwarded-For`; `"-"` is what `main` writes for a request without that
header). Both fields keep their place. A participant's address is in one of
the two: in `X-Forwarded-For` behind a load balancer, in the peer address
where nginx faces participants directly. `shadow_edge`, written beside the
access line, carries the request id that joins Node's and the shadow's
lines; with an address on the access line, one vote's lines would tie an
address to them. Every other request, the other methods on the same path
included, keeps `main`. Node and the shadow receive `X-Forwarded-For`
exactly as before. Anything that counts votes or voters from an address on
nginx's access lines reads `-` for a shadowed vote. Not changed: the
request line, the `Referer` and the user agent on that line; and nginx's
error log, where a line written when the copy or Node fails names the peer
as `client:` (it carries no request id).
- **With `votesRead`.** The vote read route's own location answers the path,
with the POST diversion inserted first (POST goes to the shadow's named
location; every other request keeps the read route's handling).
Expand Down
Loading
Loading