OpenBesluitvorming

• Zoek door meer dan 5 miljoen documenten van meer dan 300 organisaties.
• PDF's worden weergegeven als afbeeldingen (laad zeer snel) of als markdown text (toegankelijk)
• Vergaderingen zijn gekoppeld aan documenten voor context, losse agendapunten zichtbaar.
• Moties en amendementen met hun uitslag, en waar de raad digitaal stemt de stem van elk raadslid met fractie.
• Gesproken woord: video- en audioregistraties met hoofdstukken per agendapunt en, waar de leverancier die publiceert, het transcript — doorzoekbaar op wat er gezegd is.
• Importeert dagelijks uit verschillende vergadersoftware (Notubiz, Ibabs, GemeenteOplossingen, Parlaeus)

OpenBesluitvorming is de vervanger van OpenRaadsinformatie.

Use the API

See API.md, rendered at openbesluitvorming.nl/docs/api.

Running locally

docker compose up -d
# visit http://0.0.0.0:8787

For frontend iteration with HMR, use one command:

pnpm run dev

That does two things:

• starts the Docker backend services needed for app development: quickwit and openbesluitvorming
• starts the Vite HMR frontend on the host

That gives you:

• the Docker backend on http://127.0.0.1:8787
• the Vite dev server with HMR on http://127.0.0.1:4317

Vite proxies /api/* calls to the Docker backend, so reruns and extraction use the same environment as the containerized app, including the installed transmutation binary. Open http://127.0.0.1:4317 while iterating on the frontend.

Object storage is taken from your environment configuration. If your .env points to external S3-compatible storage such as Hetzner, pnpm run dev uses that directly and does not start local MinIO.

Important:

• pnpm run dev is the intended development entrypoint
• pnpm run dev starts Docker-backed services and the API, then runs Vite with HMR
• pnpm run dev clears stale listeners on the HMR port before starting, so leftover Vite processes do not usually require manual cleanup

If you only want the infra:

pnpm run dev:infra

Run:

pnpm run dev

To stop the Docker side again:

pnpm run dev:down

If you want a different dev port:

WOOZI_WEB_PORT=4401 pnpm run dev:web

If you want a different HMR web port:

WOOZI_WEB_PORT=4401 pnpm run dev

Production with Caddy

The repo includes a production-oriented compose file and Caddy config:

• docker-compose.production.yml
• Caddyfile

That setup is intended for:

• openbesluitvorming
• quickwit
• caddy

with external S3-compatible object storage from .env.

Preferred production deploy flow:

git push origin main

That triggers the GitHub Actions workflow in .github/workflows/publish-openbesluitvorming.yml, which builds and publishes:

• ghcr.io/ontola/openbesluitvorming:main
• ghcr.io/ontola/openbesluitvorming:sha-<git-sha>
• ghcr.io/ontola/openbesluitvorming:latest

A successful publish then triggers .github/workflows/deploy-production.yml automatically, which deploys that image. Pushing to main is the deploy — there is no separate manual step in the normal flow.

Run it by hand only to deploy a specific revision, or when the automatic deploy did not run:

pnpm run deploy:production

deploy:production does two things: it syncs the runtime config (compose, Caddyfile, quickwit.yaml) to the server, and over SSH tells it to pull ghcr.io/ontola/openbesluitvorming:sha-<short-git-sha> and restart the app and worker containers.

It refuses to run with uncommitted changes, because CI only publishes an image for a committed revision. It does not wait for imports to finish: the worker hands claimed runs back to the queue on SIGTERM, so a deploy during an import is safe by design.

By default, deploy:production derives the GHCR owner from your origin remote. If you need to override it explicitly:

IMAGE_REPOSITORY=ghcr.io/your-org/openbesluitvorming pnpm run deploy:production

When production infra files change, sync those separately:

pnpm run deploy:production:infra

That is only for runtime config such as:

• docker-compose.production.yml
• Caddyfile
• quickwit/quickwit.yaml

Required production env includes:

• DOMAIN
• ADMIN_PASSWORD_HASH
• S3_ACCESS_KEY
• S3_SECRET_KEY
• S3_STORAGE_BUCKET_NAME
• S3_STORAGE_ENDPOINT
• S3_STORAGE_REGION
• QUICKWIT_INDEX_ID
• QUICKWIT_CLUSTER_ID
• QUICKWIT_NODE_ID
• QUICKWIT_INDEX_ROOT_PREFIX

The server should run production with:

docker compose -f docker-compose.production.yml up -d

Point your domain to the server first so Caddy can obtain Let's Encrypt certificates.

Important:

• the GHCR package must be public, or the server must be logged in to GHCR
• code deploys should update container images, not rsync source files

Quickwit defaults are intentionally different between local and production so both environments do not accidentally share the same S3-backed metastore and index:

• local/dev defaults:
  • QUICKWIT_CLUSTER_ID=woozi-dev
  • QUICKWIT_NODE_ID=quickwit-dev
  • QUICKWIT_INDEX_ROOT_PREFIX=indexes-dev
  • QUICKWIT_INDEX_ID=woozi-events-dev
• production defaults:
  • QUICKWIT_CLUSTER_ID=woozi-prod
  • QUICKWIT_NODE_ID=quickwit-prod
  • QUICKWIT_INDEX_ROOT_PREFIX=indexes-prod
  • QUICKWIT_INDEX_ID=woozi-events-prod

Important:

• if production previously used indexes + woozi-events, switching to indexes-prod + woozi-events-prod creates a fresh search projection
• after that change, production search needs a reindex/reimport before results appear again

Architecture

Woozi is designed as an event-driven indexing system.

The core split is:

• extract data from source systems
• normalize it into canonical entities
• emit events only for changes
• build read projections from those events

The main flow is:

source system
  -> extractor / poller
  -> canonical entity
  -> event broker
  -> projection services
  -> search index / resolver / other read models

Main Components

1. Extractors

Extractors talk to source systems such as Notubiz, iBabs, GO, and Parlaeus.

Responsibilities:

• poll for changes in a date range or from a source cursor
• fetch raw payloads and documents
• store retrieved files in S3-compatible object storage
• extract markdown-ready document text from PDFs and office files
• normalize source-specific structures
• produce canonical entities

The current ORI extractor logic is the starting point for this layer.

2. Canonical Entities

The system does not index raw source payloads directly.

Instead, each source payload is transformed into a canonical entity such as:

• Meeting
• Document
• Committee
• Motion — motie/amendement, with its outcome and, where the council votes digitally, the vote of every individual member
• Recording — the video or audio registration of a meeting, with chapters per agenda item and the transcript where the supplier publishes one
• Party and Person — the fractie and the council member a vote points at

The schemas in schemas/ are the first version of those contracts.

3. Change Detection

Quickwit is not used to determine whether something changed.

Change detection happens before indexing:

• each canonical entity gets a stable ID
• each canonical payload gets a content hash
• the latest known hash is compared against metadata storage
• if nothing changed, no event is emitted
• if something changed, a new commit event is emitted

Edits are modeled as new versions, not in-place mutations. Deletes are modeled as tombstones or delete commits.

4. Event Broker

When an entity changes, Woozi emits an event into an event broker.

The expected event model is:

• CloudEvents envelope
• entity.commit payload

This broker decouples extraction from indexing and makes replay possible.

5. Canonical Storage

The canonical representation should not live in Quickwit.

Canonical JSON and original files should be stored in object storage, such as:

• S3
• MinIO

Example object classes:

• raw source payloads
• canonical JSON snapshots
• original files
• derived markdown and search text

6. Metadata Store

Woozi still needs a small metadata store.

PostgreSQL remains useful for:

• source cursors
• latest entity head per ID
• content hashes
• commit metadata
• projector checkpoints
• resolver mappings

PostgreSQL should be small and boring in Woozi. It is not the document store and not the search index.

Current prototype note:

• extraction/admin run state is currently stored in a local SQLite file
• that is an implementation shortcut for the prototype, not the intended final metadata design

7. Projection Services

Projection services consume events and build read models.

Initial projections:

• search projection
• document resolver projection
• admin/reporting projection

Each projection can be rebuilt by replaying events and canonical snapshots.

8. Quickwit

Quickwit is the search projection, not the source of truth.

Responsibilities:

• store search-ready projected documents
• support search and filtering at large scale
• stay cheap by relying on object storage-backed indexing

Quickwit should receive projection documents derived from canonical entities. It should not receive raw source payloads and should not be the place where updates are detected.

Design Rules

• Raw source payloads are not the public contract.
• Canonical entities are the internal contract.
• Events describe changes to canonical entities.
• Quickwit is a projection.
• Object storage holds canonical payloads and files.
• PostgreSQL holds metadata and coordination state.

Current State

This folder currently contains:

• minimal entity schemas
• a first Deno-based Notubiz extractor slice
• a Vite + TypeScript frontend with HMR for the public UI and admin UI
• shared frontend/backend TypeScript API types
• entity.commit events for canonical meetings, documents, motions and recordings
• attachment download into S3-compatible object storage
• markdown extraction for PDF and Word-style documents
• motions and per-member votes from iBabs list entries and Notubiz module items
• video/audio registrations with an agenda-item timeline and ASR transcripts
• a local Quickwit setup and projection client
• a small admin UI for reruns and extraction run inspection
• live e2e coverage that ingests Haarlem meetings and attached files into Quickwit and the GUI

Commands

From woozi/:

• pnpm run dev
• pnpm run dev:infra
• pnpm run dev:docker
• pnpm run dev:down
• pnpm run dev:web
• pnpm run web
• pnpm run serve:web
• pnpm run build:web
• pnpm test
• pnpm test:live
• pnpm test:e2e
• pnpm test:gui
• pnpm test:quickwit
• pnpm run extract:haarlem
• pnpm run ingest:haarlem
• pnpm run lint
• pnpm run format
• pnpm run check-format

Tests

pnpm test is hermetic and runs in about ten seconds. It is what CI gates on.

The integration tests are opt-in behind WOOZI_RUN_LIVE_INTEGRATION=1, which is what pnpm test:live (or the individual test:e2e, test:gui, test:quickwit) sets. They need Docker and minio, the live Notubiz API, and the PDF extraction fleet — and that fleet only accepts connections from the production host. Off production they do not fail quickly: each document waits out its retries first, so a run takes minutes and ends in extraction_service_failed. Run them from a machine that can reach the fleet, or not at all.

Quickwit helpers live in quickwit/.

For real S3-compatible storage, put these values in .env.example copied to .env:

• S3_ACCESS_KEY
• S3_SECRET_KEY
• S3_STORAGE_BUCKET_NAME
• S3_STORAGE_ENDPOINT
• S3_STORAGE_REGION

To run against external S3-compatible storage:

docker compose up -d

To run the local stack with MinIO for development and tests:

docker compose --profile local-s3 up -d

To extract one Haarlem day and ingest the resulting commit events into Quickwit:

pnpm run ingest:haarlem

That command now:

• extracts public Haarlem meetings from Notubiz
• downloads attached source files
• stores the originals in S3-compatible object storage
• extracts markdown from those files and serves it lazily in the detail view
• emits entity.commit events for Meeting and Document
• projects both entity types into Quickwit

To start the frontend prototype:

docker compose up -d --build openbesluitvorming

For local production-style serving without Docker:

pnpm run web

Local extractor requirements

The host-side ingest command currently uses:

• the Rust transmutation CLI for PDFs
• direct text decoding for .txt, .md, .json, and HTML-like documents
• office formats like .doc, .docx, .rtf, and .odt are not supported yet and are logged as extraction warnings

In Docker and compose, the transmutation CLI is installed in the image automatically. On the host, Woozi will try to call transmutation from PATH or from WOOZI_TRANSMUTATION_BIN.

For Apple Silicon macOS, you can install it like this:

cargo install --locked transmutation

If the CLI is missing, PDF extraction fails explicitly and the import records an extraction issue.

There is no remaining host-only Office extractor in the runtime anymore. Unsupported office formats are currently skipped with an explicit extraction warning.