Split query log capture into a log source and a log type

CHANGELOG.md

@@ -11,6 +11,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 - **Common values of declared columns.** The workload bundle now has `distributions.json`. It holds the most common values from `pg_stats` for each column under `database.workload.distributions`. Values ship verbatim unless `hash_values: true`. The capture role needs `SELECT` on each declared column. See [Recording the common values of chosen columns](docs/workload_capture.md#recording-the-common-values-of-chosen-columns).
 
+### Changed
+
+- **Query log capture settings.** `capture_log_type` is required when `capture_log` is on, and is `statement` or `pgaudit`. `capture_log_source` is now `file` or `log_fdw`, and defaults to `file`. The tool reads whether an exported file is plain text, csvlog or a pgAudit JSON export. The old `capture_log_source` values `pgaudit`, `pgaudit-json`, `stderr` and `auto` are rejected, and the error names the settings that replace them. See [Capturing transaction shapes and values](docs/workload_capture.md#capturing-transaction-shapes-and-values).
+
 ## [2.1.0] - 2026-10-05
 
 ### Added

README.md

@@ -301,9 +301,12 @@ more accurate sharding plan.
 | 2 | Statement log or pgAudit records, read from the standard server log | Which tables each transaction writes together. How unevenly the values of a candidate shard key are accessed. | No, but recommended |
 
 Level 1 alone gives a complete bundle, and the bundle names what level 2 would
-add. To add level 2, set `capture_log: true` under `database.workload`. pgAudit
-is the default source. Platforms without pgAudit can use `log_fdw` (RDS and
-Aurora) or `stderr` (self-managed hosts). See
+add. To add level 2, set `capture_log: true` under `database.workload`, and set
+`capture_log_type` to `statement` or `pgaudit`. `statement` needs no extension.
+When a log holds both record types, the capture reads only the configured type.
+`capture_log_source` is `file` by default, which reads a log that you export and
+name in `capture_log_file`. On RDS and Aurora, `log_fdw` reads the log over the
+tool's connection. See
 [Capturing transaction shapes and values](docs/workload_capture.md#capturing-transaction-shapes-and-values).
 
 Level 2 records literal values from your queries in `burst.csv`. Review that

docs/providers/aws.md

@@ -457,45 +457,54 @@ is a static parameter. For Aurora, set it on the cluster parameter group.
 so run it before you change anything. See
 [Workload Capture](../workload_capture.md#3-enable-pg_stat_statements).
 
-### Capturing transaction shapes with pgAudit
+### Capturing transaction shapes
 
 Beyond the query counts above, the capture can also read the server's own
 query log, to see which statements ran in the same transaction and the literal
 values they carried. See
 [Capturing transaction shapes and values](../workload_capture.md#capturing-transaction-shapes-and-values)
 for what it collects and why you might want it.
 
-RDS and Aurora can do this two ways. **pgAudit is the default.** Read the
-comparison below before you choose `log_fdw` instead.
+Logging is configured using two settings. `capture_log_type` is required: 
+`statement` or `pgaudit`. There is no default. `statement` needs no extension
+but requires additional configuration to emit statements into the log. `pgaudit` 
+requires an extension to be enabled and installed.
+`capture_log_source` is where that log is read from. It defaults to `file`.
+`log_fdw` reads the csv log over this connection instead.
 
-#### log_fdw or pgAudit
+#### Log type
 
-| | pgAudit (`capture_log_source: pgaudit`) | log_fdw (`capture_log_source: log_fdw`) |
+| | `statement` | `pgaudit` |
+| --- | --- | --- |
+| Extension | None | `shared_preload_libraries`, one reboot, `CREATE EXTENSION pgaudit` |
+| Per-capture setup | `log_min_duration_statement = 0` for the window, then reset it | None if logging is preconfigured using server parameters, runtime per-role configuration possible |
+| Scope of what is logged | The whole server, every role | Configurable at server level and per-role |
+| What it records | Statement, session, transaction framing, per-statement duration, error text | Statement, session, transaction framing, bind values |
+| Log volume | Every statement, at `log_min_duration_statement = 0` | Depends on configured logging scope, up to as much as "every statement" |
+| Superuser traffic | Logged like any other | Depends on configured logging scope |
+
+Use `statement` unless you want granular configuration and can accept a reboot.
+The reboot is once per instance when enabling the extension, not once per capture.
+Either log type can be read from a file or over `log_fdw`.
+
+#### Log source
+
+| | `file` (the default) | `log_fdw` |
 | --- | --- | --- |
 | Writes to your database | Nothing | A work schema, a foreign server, and the extension if absent, per `collect` |
-| One-time setup | `shared_preload_libraries` in the parameter group, one reboot, `CREATE EXTENSION pgaudit` | `CREATE EXTENSION log_fdw`, no reboot |
-| Per-capture setup | Two `ALTER ROLE` statements and a reset | None |
-| Privilege the capture role needs | `pg_monitor` | `rds_superuser` |
-| Scope of what is logged | One role, the application's | The whole server, every role |
 | Hands the log over by | An export you download and name in `capture_log_file` | This tool's own connection, no export |
-| What it records | Statement, session, transaction framing, bind values | The same, plus per-statement duration and error text |
-| Log volume it produces | The audited role's traffic | Every statement on the instance, at `log_min_duration_statement = 0` |
-| Superuser traffic | Not reliably audited | Logged like any other |
-| Cleanup afterwards | Reset the role, delete the exported files | The same, plus `workload init --cleanup` if a `collect` was killed |
+| Privilege the capture role needs | `pg_monitor` (for stats, not the log itself) | `rds_superuser` |
+| What it can read | The exported log (plain text or csvlog) for either type | csvlog only, for either type |
+| Cleanup afterwards | Delete the exported files | The same, plus `workload init --cleanup` if a `collect` was killed, disable extension if no longer needed |
 
-- **Choose pgAudit** unless you cannot reboot. It logs one role and writes
-  nothing to your database, and the reboot is once per instance however many
-  captures you run.
-- **Choose `log_fdw`** when a reboot on a production primary is not something
-  you can schedule, or when you need the durations and error text that only the
-  server log carries. It costs `rds_superuser` for the capture role, objects
-  created and dropped inside each `collect`, and a log holding every role's
-  statements rather than one.
+Use `file` unless you cannot export files. `log_fdw` needs `csvlog` in
+`log_destination`. It requires `rds_superuser` for the capture role, and 
+creates/drops objects inside each `collect`. It does not change what is logged.
 
-Both produce the same bundle.
+Both types and both sources produce the same bundle.
 
-`ps-discovery workload init --check` reports which of the two this server can
-supply today, and names any objects a killed `collect` left behind.
+`ps-discovery workload init --check` reports what this server can supply, and
+names any objects a killed `collect` left behind.
 
 #### Permissions for log capture
 
@@ -572,7 +581,7 @@ Name the file in `config.yaml`, then run the usual `collect`.
 database:
   workload:
     capture_log: true
-    capture_log_source: pgaudit
+    capture_log_type: pgaudit
     capture_log_file: pgaudit-capture.log
 ```
 
@@ -592,30 +601,33 @@ Then delete the log files you downloaded. They hold literal values from your
 queries. Watch `FreeStorageSpace` in CloudWatch level off to confirm the
 logging stopped.
 
-### Capturing over log_fdw instead
+### Reading the log over log_fdw
 
-Use this only after reading
-[log_fdw or pgAudit](#log_fdw-or-pgaudit) above.
+Use this only after reading [Log source](#log-source) above. `log_fdw` reads
+the csv log. It only defines how the log is read, not what log type the server
+generates (statement log or pgAudit). Set `capture_log_type` for that.
 
 **Set up:** `CREATE EXTENSION log_fdw;` as a member of `rds_superuser`, and
 grant `rds_superuser` to the capture role, which needs it to create the foreign
 server and read the log files. Reconnect after the grant, since it does not
 reach an open session. Set `log_destination` to include `csvlog` in the
-parameter group, and `log_min_duration_statement = 0` for the window. Neither
-needs a reboot, and both need the parameter-group permissions listed under
+parameter group. That needs no reboot, and it needs the parameter-group
+permissions listed under
 [Permissions for log capture](#permissions-for-log-capture).
 
 ```yaml
 database:
   workload:
     capture_log: true
+    capture_log_type: statement
     capture_log_source: log_fdw
     capture_log_seconds: 600
 ```
 
 Each `collect` then watches the log for `capture_log_seconds` and reads that
 window over its own connection. Keep the schedule interval longer than
-`capture_log_seconds`.
+`capture_log_seconds`. The same connection reads a pgAudit csv log when
+`capture_log_type` is `pgaudit` instead of `statement`.
 
 **Turn it off afterwards:** return `log_min_duration_statement` to the value it
 had. RDS re-adds `stderr` alongside `csvlog`, so every event is written twice

docs/providers/gcp.md

@@ -558,7 +558,7 @@ yourself.
 database:
   workload:
     capture_log: true
-    capture_log_source: pgaudit-json
+    capture_log_type: pgaudit
     capture_log_file: pgaudit-capture.jsonl
 ```
 
@@ -584,8 +584,8 @@ timestamp>="..." timestamp<="..."
 The payload is the same `PgAuditEntry`, and chunking is reassembled by the
 tool the same way.
 
-**Read it:** the same `capture_log_source: pgaudit-json` config block as
-Cloud SQL, above.
+**Read it:** the same `capture_log_type: pgaudit` config block as Cloud SQL,
+above. The JSON encoding is read from the file.
 
 ## Additional Resources
 

docs/providers/supabase.md

@@ -422,7 +422,7 @@ Name the file in `config.yaml`, then run the usual `collect`.
 database:
   workload:
     capture_log: true
-    capture_log_source: pgaudit
+    capture_log_type: pgaudit
     capture_log_file: exported.log
 ```
 

docs/workload_capture.md

@@ -392,8 +392,8 @@ The directory is mode `0700`. Delete it when you have the bundle. To stop
 collection, remove the cron entry and delete the directory.
 
 Nothing persists on the database server with `capture_log` off, and nothing
-persists with the default `pgaudit` source either, which reads a file you
-exported. Only `capture_log_source: log_fdw` writes to the server: each collect
+persists when `capture_log_source` is `file`, which reads a log you exported.
+Only `capture_log_source: log_fdw` writes to the server: each collect
 creates a work schema, a foreign server and the `log_fdw` extension to read the
 log, and drops all three when it finishes. If a collect is killed between the
 two, the next one clears what was left behind, and
@@ -687,13 +687,16 @@ This is a setting, not a command. Nothing about the flow changes: the same
 database:
   workload:
     capture_log: true
-    capture_log_source: pgaudit
-    capture_log_file: pgaudit-capture.log
+    capture_log_type: statement
+    capture_log_file: exported.log
 ```
 
-`pgaudit` is the default source. You turn statement logging on for one role,
-export the window through your provider's own tooling, and name the file. Each
-`collect` takes its snapshot as before and then reads the file.
+`capture_log_type` is required. `statement` reads the server's statement log
+and needs no extension. `pgaudit` reads pgAudit. `capture_log_source` defaults
+to `file`: you export the window and name it in `capture_log_file`. Each
+`collect` takes its snapshot as before and then reads the file. The tool reads
+the file's encoding itself: plain text, csvlog, or a Cloud SQL or AlloyDB
+pgAudit JSON export.
 
 With `capture_log_source: log_fdw`, the tool reads the server's log over its own
 connection instead, and each `collect` watches the log for
@@ -712,42 +715,42 @@ reason as a warning and exits 0. A capture never fails because of this.
 
 ### Where the log comes from
 
-`capture_log_source` selects how the log is read.
+`capture_log_type` selects what was logged. `capture_log_source` selects where
+the log is read. `capture_log_type` is required when `capture_log` is on.
+`statement` needs no extension. `pgaudit` logs one role's traffic, it needs no
+write of any kind to your database, and every setting it uses can be changed
+without a restart after the extension is enabled.
 
-**pgAudit is the default.** It logs one role's traffic, it needs no write of any
-kind to your database, and every setting it uses can be changed without a
-restart. Use another source only where pgAudit cannot run.
+`capture_log_source` defaults to `file`. `log_fdw` reads the csv log the server
+is already writing, over this connection, on RDS and Aurora. That read is
+always csvlog. pgAudit over it is `capture_log_type: pgaudit` plus
+`capture_log_source: log_fdw`.
 
-| Platform | `capture_log_source` | Reads the log by |
-| --- | --- | --- |
-| RDS, Aurora | `pgaudit` | a file you export |
-| RDS, Aurora, no pgAudit available | `log_fdw` | SQL, over `log_fdw` |
-| Cloud SQL, AlloyDB | `pgaudit` or `pgaudit-json` | a file you export |
-| Supabase | `pgaudit` | a file you export |
-| Self-managed | `pgaudit`, or `stderr` with no extension | the log file on the host |
-| Neon, Heroku Postgres, PlanetScale | cannot supply a log | see the provider's guide |
+| `capture_log_type` | Supported `capture_log_source` values |
+| --- | --- |
+| `statement` | `file` (plain text or csvlog). `log_fdw` on RDS and Aurora, which reads csvlog over this connection |
+| `pgaudit` | `file` (plain text, csvlog, or a Cloud SQL or AlloyDB JSON export). `log_fdw` on RDS and Aurora, which reads csvlog over this connection |
 
-`ps-discovery workload init --check` reports which row your server is on. It
-opens a connection, prints what the server can supply, creates no session and
-changes nothing. Add `--json` to sweep an estate.
+`ps-discovery workload init --check` reports what this server can supply. It
+opens a connection, creates no session and changes nothing. Add `--json` to
+sweep an estate.
 
 `log_fdw` reads the log the server is already writing, over this tool's own
 connection, so it needs no export and no restart. It needs the `log_fdw`
 extension, csvlog output and a role holding `rds_superuser`, and it is the only
 source that creates objects in your database.
-[The AWS guide](providers/aws.md#log_fdw-or-pgaudit) compares it with pgAudit in
-full. To get statements into the log in the first place, set
+[The AWS guide](providers/aws.md#log-source) compares `file` and `log_fdw`. To get statements into the log in the first place, set
 `log_min_duration_statement = 0` for the window; see
 [Set the logging up yourself](#set-the-logging-up-yourself).
 
-For the three file sources, export the log through the provider's own tooling
-and name the file:
+For a file source, export the log through the provider's own tooling and name
+the file:
 
 ```yaml
 database:
   workload:
     capture_log: true
-    capture_log_source: pgaudit
+    capture_log_type: pgaudit
     capture_log_file: exported.log
 ```
 
@@ -762,18 +765,20 @@ read and the run says that it took no snapshot. A session of nothing but
 imports has no window to measure the log against, so run at least two `collect`s
 that can connect.
 
-`pgaudit-json` reads Cloud SQL's and AlloyDB's JSON export, either one
-`PgAuditEntry` payload per line or a `gcloud logging read --format=json` array.
-`stderr` reads a plain server log, for hosts where pgAudit cannot be installed
-at all.
+A file is classified before it is parsed. A leading `{` or `[` is the Cloud
+SQL and AlloyDB `PgAuditEntry` export, either one payload per line or a
+`gcloud logging read --format=json` array, and it is read only when
+`capture_log_type` is `pgaudit`. A csvlog row is csv. Anything else with a
+severity marker (`LOG:`, `ERROR:`, and the rest) is plain text.
+`capture_log_type: statement` pointed at a JSON export records no log window.
 
 Per-provider export steps:
-[RDS and Aurora](providers/aws.md#capturing-transaction-shapes-with-pgaudit),
+[RDS and Aurora](providers/aws.md#capturing-transaction-shapes),
 [Cloud SQL and AlloyDB](providers/gcp.md#capturing-transaction-shapes-with-pgaudit),
 [Supabase](providers/supabase.md#capturing-transaction-shapes-with-pgaudit).
 [Neon](providers/neon.md#capturing-transaction-shapes-with-pgaudit) and
 [Heroku](providers/heroku.md#capturing-transaction-shapes-with-pgaudit) cannot
-produce this source; their guides say why.
+produce a log this capture can read; their guides say why.
 
 ### Permissions for log capture
 
@@ -856,8 +861,8 @@ statement and corrupts the record pairing. Keep `log_statement` at `'none'`
 during the window, or every statement is logged twice.
 
 Without pgAudit, set `log_min_duration_statement = 0` for the window instead,
-leave `log_line_prefix` as it is (the tool reads it from the file), and use
-`capture_log_source: stderr`. No extension is needed. The trade-off: the plain
+leave `log_line_prefix` as it is (the tool reads it from the file), and set
+`capture_log_type: statement`. No extension is needed. The trade-off: the plain
 log carries durations and pgAudit does not, but a busy server logs a great deal
 at `log_min_duration_statement = 0`.
 
@@ -874,10 +879,10 @@ needs another.
 | AlloyDB | the `alloydb.enable_pgaudit` flag, restart, `CREATE EXTENSION pgaudit;` |
 | Supabase | enable pgAudit in the dashboard, or `CREATE EXTENSION pgaudit;`; no preload step |
 
-Do this on RDS and Aurora too. A pgAudit capture writes nothing to your
-database and logs one role rather than the whole server. Use `log_fdw` where
-the reboot cannot be scheduled; see
-[log_fdw or pgAudit](providers/aws.md#log_fdw-or-pgaudit).
+Do this on RDS and Aurora too. A pgAudit capture logs one role. With
+`capture_log_source: file` the tool writes nothing to the database. Without
+the reboot, set `capture_log_type: statement`. See
+[Log type](providers/aws.md#log-type).
 
 Some limits carry into every pgAudit capture. None of them stops a capture
 being useful, but each one qualifies what a transaction shape proves.