diff --git a/src/current/v25.4/configure-logs.md b/src/current/v25.4/configure-logs.md index 0dcc3a313c2..3dc0266bac2 100644 --- a/src/current/v25.4/configure-logs.md +++ b/src/current/v25.4/configure-logs.md @@ -15,6 +15,12 @@ This page describes how to configure CockroachDB logs with the [`--log` or `log- For examples of how these settings can be used in practice, see [Logging Use Cases]({% link {{ page.version.version }}/logging-use-cases.md %}). +{{site.data.alerts.callout_info}} +In a future release, certain events will be directed to new [logging channels]({% link {{ page.version.version }}/logging-overview.md %}#logging-channels). For more details, refer to [`log.channel_compatibility_mode.enabled`]({% link {{ page.version.version }}/logging-overview.md %}#log-channel_compatibility_mode-enabled). + +To assess potential downstream impacts on your logging setup and pipelines, set the `log.channel_compatibility_mode.enabled` cluster setting to `false` in a non-production environment. +{{site.data.alerts.end}} + ## Flag To configure the logging behavior of a `cockroach` command, include one of these flags with the command: @@ -865,6 +871,19 @@ sinks: redactable: true exit-on-error: true buffering: NONE + kv-exec: + channels: {INFO: [KV_EXEC]} + dir: /cockroach-data/logs + max-file-size: 10MiB + max-group-size: 100MiB + file-permissions: "0640" + buffered-writes: true + filter: INFO + format: crdb-v2 + redact: false + redactable: true + exit-on-error: true + buffering: NONE pebble: channels: {INFO: [STORAGE]} dir: /cockroach-data/logs diff --git a/src/current/v25.4/log-sql-activity-to-datadog.md b/src/current/v25.4/log-sql-activity-to-datadog.md index d6369b3a0da..b65d2d42687 100644 --- a/src/current/v25.4/log-sql-activity-to-datadog.md +++ b/src/current/v25.4/log-sql-activity-to-datadog.md @@ -7,6 +7,12 @@ docs_area: manage This tutorial describes how to configure logging of telemetry events, including [`sampled_query`]({% link {{ page.version.version }}/eventlog.md %}#sampled_query) and [`sampled_transaction`]({% link {{ page.version.version }}/eventlog.md %}#sampled_query), to [Datadog](https://www.datadoghq.com/) for finer granularity and long-term retention of SQL activity. The `sampled_query` and `sampled_transaction` events contain common SQL event and execution details for [statements]({% link {{ page.version.version }}/sql-statements.md %}) and [transactions]({% link {{ page.version.version }}/transactions.md %}). +{{site.data.alerts.callout_info}} +`sampled_query` and `sampled_transaction` events currently sent to the [`TELEMETRY`]({% link {{ page.version.version }}/logging-overview.md %}#telemetry) channel will be routed to the [`SQL_EXEC`]({% link {{ page.version.version }}/logging-overview.md %}#sql_exec) channel in a future release. + +To assess potential downstream impacts on your logging setup and pipelines, set the [`log.channel_compatibility_mode.enabled`]({% link {{ page.version.version }}/logging-overview.md %}#log-channel_compatibility_mode-enabled) cluster setting to `false` in a non-production environment. This will log these events to the [`SQL_EXEC`]({% link {{ page.version.version }}/logging-overview.md %}#sql_exec) channel. +{{site.data.alerts.end}} + CockroachDB supports a built-in integration with Datadog which sends these events as logs via the [Datadog HTTP API](https://docs.datadoghq.com/api/latest/logs/). This integration is the recommended path to achieve high throughput data ingestion, which will in turn provide more query and transaction events for greater workload observability. {{site.data.alerts.callout_info}} diff --git a/src/current/v25.4/logging-overview.md b/src/current/v25.4/logging-overview.md index 6ff5436246a..2f8dfc43014 100644 --- a/src/current/v25.4/logging-overview.md +++ b/src/current/v25.4/logging-overview.md @@ -24,28 +24,50 @@ All [`cockroach` commands]({% link {{ page.version.version }}/cockroach-commands {% include {{ page.version.version }}/log-channel-note.md %} -Log messages in CockroachDB are directed into logging channels, which can in turn be assigned to output to one or more [log sinks]({% link {{ page.version.version }}/configure-logs.md %}#configure-log-sinks). +Log messages in CockroachDB are directed to logging channels, which can in turn be assigned to output to one or more [log sinks]({% link {{ page.version.version }}/configure-logs.md %}#configure-log-sinks). This allows you to group channels that log related information (e.g., operational, security, or SQL events) into their own sinks. Each sink can output to a predetermined destination where the logs can be collected and parsed. For usage examples, see [Logging Use Cases]({% link {{ page.version.version }}/logging-use-cases.md %}). | Channel | Description | |-----------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| New in v25.4: [`CHANGEFEED`]({% link {{ page.version.version }}/logging.md %}) | Changefeed events.

**Note**: [Changefeed telemetry events]({% link {{ page.version.version }}/eventlog.md %}#changefeed-telemetry-events) currently sent to the [`TELEMETRY`](#telemetry) channel will be routed to this channel in a future release.
To assess potential downstream impacts on your logging setup and pipelines, set the [`log.channel_compatibility_mode.enabled`](#log-channel_compatibility_mode-enabled) cluster setting to `false` in a non-production environment. This will log these events to this `CHANGEFEED` channel. | | [`DEV`]({% link {{ page.version.version }}/logging.md %}#dev) | Uncategorized and debug messages. | -| [`OPS`]({% link {{ page.version.version }}/logging.md %}#ops) | Process starts, stops, shutdowns, and crashes (if they can be logged); changes to cluster topology, such as node additions, removals, and decommissions. | +| [`KV_DISTRIBUTION`]({% link {{ page.version.version }}/logging.md %}#kv_distribution) | Data distribution events, such as moving [replicas]({% link {{ page.version.version }}/architecture/overview.md %}#architecture-replica) between [stores]({% link {{ page.version.version }}/cockroach-start.md %}#store) in a cluster, adding replicas to [ranges]({% link {{ page.version.version }}/architecture/overview.md %}#range), and removing replicas from ranges. | +| New in v25.4: [`KV_EXEC`]({% link {{ page.version.version }}/logging.md %}) | KV execution events that are not related to distribution.| | [`HEALTH`]({% link {{ page.version.version }}/logging.md %}#health) | Resource usage; node-node connection events, including connection errors; up- and down-replication and range unavailability. | -| [`STORAGE`]({% link {{ page.version.version }}/logging.md %}#storage) | Low-level storage logs from the [Pebble storage engine]({% link {{ page.version.version }}/architecture/storage-layer.md %}#pebble). | +| [`OPS`]({% link {{ page.version.version }}/logging.md %}#ops) | Process starts, stops, shutdowns, and crashes (if they can be logged); changes to cluster topology, such as node additions, removals, and decommissions. | +| [`PRIVILEGES`]({% link {{ page.version.version }}/logging.md %}#privileges) | Changes to privileges and object ownership. | +| [`SENSITIVE_ACCESS`]({% link {{ page.version.version }}/logging.md %}#sensitive_access) | SQL audit events (when enabled via [`ALTER TABLE ... EXPERIMENTAL_AUDIT`]({% link {{ page.version.version }}/alter-table.md %}#experimental_audit)). | | [`SESSIONS`]({% link {{ page.version.version }}/logging.md %}#sessions) | Client connections and disconnections (when enabled via the `server.auth_log.sql_connections.enabled` [cluster setting]({% link {{ page.version.version }}/cluster-settings.md %})); SQL authentication logins/attempts and session/query terminations (when enabled via the `server.auth_log.sql_sessions.enabled` [cluster setting]({% link {{ page.version.version }}/cluster-settings.md %})). | +| [`SQL_EXEC`]({% link {{ page.version.version }}/logging.md %}#sql_exec) | SQL statement executions (when enabled via the `sql.trace.log_statement_execute` [cluster setting]({% link {{ page.version.version }}/cluster-settings.md %})) and uncaught Go panic errors during SQL statement execution.

**Warning**: Logging cluster-wide executions by enabling the `sql.trace.log_statement_execute` cluster setting will incur considerable overhead and may have a negative performance impact.

**Note**: In a future release, the following events will be routed to this channel:To assess potential downstream impacts on your logging setup and pipelines, set the [`log.channel_compatibility_mode.enabled`](#log-channel_compatibility_mode-enabled) cluster setting to `false` in a non-production environment. This will log these events to this `SQL_EXEC` channel. | +| [`SQL_PERF`]({% link {{ page.version.version }}/logging.md %}#sql_perf) | SQL executions that impact performance, such as slow queries (when enabled via the `sql.log.slow_query.latency_threshold` and/or `sql.log.slow_query.experimental_full_table_scans.enabled` [cluster settings]({% link {{ page.version.version }}/cluster-settings.md %})).

**Note**: As of v25.4, the `SQL_PERF` and the `SQL_INTERNAL_PERF` channels are **deprecated** and will be removed in a future release. [SQL Slow Query Log]({% link {{ page.version.version }}/eventlog.md %}#sql-slow-query-log) and [SQL Slow Query Log (Internal)]({% link {{ page.version.version }}/eventlog.md %}#sql-slow-query-log-internal) events currently sent to the `SQL_PERF` and [`SQL_INTERNAL_PERF`]({% link {{ page.version.version }}/logging.md %}#sql_internal_perf) channels, respectively, will be routed to the [`SQL_EXEC`](#sql_exec) channel in a future release.
To assess potential downstream impacts on your logging setup and pipelines, set the [`log.channel_compatibility_mode.enabled`](#log-channel_compatibility_mode-enabled) cluster setting to `false` in a non-production environment. This will log these events to the [`SQL_EXEC`](#sql_exec) channel. | | [`SQL_SCHEMA`]({% link {{ page.version.version }}/logging.md %}#sql_schema) | Database, schema, table, sequence, view, and type creation; changes to table columns and sequence parameters. | +| [`STORAGE`]({% link {{ page.version.version }}/logging.md %}#storage) | Low-level storage logs from the [Pebble storage engine]({% link {{ page.version.version }}/architecture/storage-layer.md %}#pebble). | +| [`TELEMETRY`]({% link {{ page.version.version }}/logging.md %}#telemetry) | Telemetry events for internal usage, except for [`sampled_query`]({% link {{ page.version.version }}/eventlog.md %}#sampled_query) and [`sampled_transaction`]({% link {{ page.version.version }}/eventlog.md %}#sampled_transaction) events which may be [logged externally to Datadog]({% link {{ page.version.version }}/log-sql-activity-to-datadog.md %}).

**Note**: `sampled_query` and `sampled_transaction` events currently sent to this channel will be routed to the [`SQL_EXEC`](#sql_exec) channel in a future release. [Changefeed telemetry events]({% link {{ page.version.version }}/eventlog.md %}#changefeed-telemetry-events) currently sent to this channel will be routed to the [`CHANGEFEED`](#changefeed) channel in a future release.
To assess potential downstream impacts on your logging setup and pipelines, set the [`log.channel_compatibility_mode.enabled`](#log-channel_compatibility_mode-enabled) cluster setting to `false` in a non-production environment. This will log these events to the [`SQL_EXEC`](#sql_exec) channel. | | [`USER_ADMIN`]({% link {{ page.version.version }}/logging.md %}#user_admin) | Changes to users, roles, and authentication credentials. | -| [`PRIVILEGES`]({% link {{ page.version.version }}/logging.md %}#privileges) | Changes to privileges and object ownership. | -| [`SENSITIVE_ACCESS`]({% link {{ page.version.version }}/logging.md %}#sensitive_access) | SQL audit events (when enabled via [`ALTER TABLE ... EXPERIMENTAL_AUDIT`]({% link {{ page.version.version }}/alter-table.md %}#experimental_audit)). | -| [`SQL_EXEC`]({% link {{ page.version.version }}/logging.md %}#sql_exec) | SQL statement executions (when enabled via the `sql.trace.log_statement_execute` [cluster setting]({% link {{ page.version.version }}/cluster-settings.md %})) and uncaught Go panic errors during SQL statement execution.

**Warning**: Logging cluster-wide executions by enabling the `sql.trace.log_statement_execute` cluster setting will incur considerable overhead and may have a negative performance impact. | -| [`SQL_PERF`]({% link {{ page.version.version }}/logging.md %}#sql_perf) | SQL executions that impact performance, such as slow queries (when enabled via the `sql.log.slow_query.latency_threshold` and/or `sql.log.slow_query.experimental_full_table_scans.enabled` [cluster settings]({% link {{ page.version.version }}/cluster-settings.md %})). | -| [`TELEMETRY`]({% link {{ page.version.version }}/logging.md %}#telemetry) | Telemetry events for internal usage, except for [`sampled_query` events]({% link {{ page.version.version }}/eventlog.md %}#sampled_query) which may be [logged externally to Datadog]({% link {{ page.version.version }}/log-sql-activity-to-datadog.md %}). | -| [`KV_DISTRIBUTION`]({% link {{ page.version.version }}/logging.md %}#kv_distribution) | Data distribution events, such as moving [replicas]({% link {{ page.version.version }}/architecture/overview.md %}#architecture-replica) between [stores]({% link {{ page.version.version }}/cockroach-start.md %}#store) in a cluster, adding replicas to [ranges]({% link {{ page.version.version }}/architecture/overview.md %}#range), and removing replicas from ranges. | Logging channels are analogous to [logging facilities in Syslog](https://wikipedia.org/wiki/Syslog) or [logging services in Datadog](https://docs.datadoghq.com/logs/log_collection/?tab=http#reserved-attributes). For more details on the contents of each logging channel, see the [Logging reference]({% link {{ page.version.version }}/logging.md %}#logging-channels). +### `log.channel_compatibility_mode.enabled` + +{% include_cached new-in.html version="v25.4" %} The `log.channel_compatibility_mode.enabled` [cluster setting]({% link {{ page.version.version }}/cluster-settings.md %}) when set to: + +- `true` (default): Logs are sent to the expected [logging channels](#logging-channels). Specifically: + + - [Changefeed telemetry events]({% link {{ page.version.version }}/eventlog.md %}#changefeed-telemetry-events) are sent to the [`TELEMETRY`](#telemetry) channel. + - [SQL Slow Query Log]({% link {{ page.version.version }}/eventlog.md %}#sql-slow-query-log) and [SQL Slow Query Log (Internal)]({% link {{ page.version.version }}/eventlog.md %}#sql-slow-query-log-internal) events are sent to the [`SQL_PERF`](#sql_perf) and [`SQL_INTERNAL_PERF`]({% link {{ page.version.version }}/logging.md %}#sql_internal_perf) channels, respectively. + - [`sampled_query`]({% link {{ page.version.version }}/eventlog.md %}#sampled_query) and [`sampled_transaction`]({% link {{ page.version.version }}/eventlog.md %}#sampled_transaction) events are sent to the [`TELEMETRY`](#telemetry) channel. + +- `false`: Logs are sent to different logging channels as part of the logging channel consolidation effort. Specifically: + + - [Changefeed telemetry events]({% link {{ page.version.version }}/eventlog.md %}#changefeed-telemetry-events) are sent to the [`CHANGEFEED`](#changefeed) channel. + - [SQL Slow Query Log]({% link {{ page.version.version }}/eventlog.md %}#sql-slow-query-log) and [SQL Slow Query Log (Internal)]({% link {{ page.version.version }}/eventlog.md %}#sql-slow-query-log-internal) events are sent to the [`SQL_EXEC`](#sql_exec) channel. + - [`sampled_query`]({% link {{ page.version.version }}/eventlog.md %}#sampled_query) and [`sampled_transaction`]({% link {{ page.version.version }}/eventlog.md %}#sampled_transaction) events are sent to the [`SQL_EXEC`](#sql_exec) channel. + +{{site.data.alerts.callout_danger}} +In future releases, `log.channel_compatibility_mode.enabled` will default to `false` and then this cluster setting will be removed. To assess potential downstream impacts on your logging setup and pipelines, set the `log.channel_compatibility_mode.enabled` cluster setting to `false` in a non-production environment. +{{site.data.alerts.end}} + ## Logging destinations When using the [default logging configuration]({% link {{ page.version.version }}/configure-logs.md %}#default-logging-configuration), the events collected on each [logging channel](#logging-channels) are split into log files as follows: @@ -53,8 +75,10 @@ When using the [default logging configuration]({% link {{ page.version.version } | Filename | Description | Channels | |----------------------------+-------------------------+-----------------------------| | `cockroach.log` | General CockroachDB log | `DEV` | +| `cockroach-changefeed.log` | Changefeed log | `CHANGEFEED` | | `cockroach-health.log` | Health log | `HEALTH` | | `cockroach-kv-distribution.log` | Data distribution log | `KV_DISTRIBUTION` | +| `cockroach-kv-exec.log` | KV execution log | `KV_EXEC` | | `cockroach-security.log` | SQL security log | `PRIVILEGES` + `USER_ADMIN` | | `cockroach-sql-audit.log` | SQL access audit log | `SENSITIVE_ACCESS` | | `cockroach-sql-auth.log` | SQL authentication log | `SESSIONS` | diff --git a/src/current/v25.4/logging-use-cases.md b/src/current/v25.4/logging-use-cases.md index 17addd64f40..23991a051a0 100644 --- a/src/current/v25.4/logging-use-cases.md +++ b/src/current/v25.4/logging-use-cases.md @@ -16,6 +16,12 @@ We provide an example [file sink configuration]({% link {{ page.version.version Your deployment may use an external service (e.g., [Elasticsearch](https://www.elastic.co/elastic-stack), [Splunk](https://www.splunk.com/)) to collect and programmatically read logging data. +{{site.data.alerts.callout_info}} +In a future release, certain events will be directed to new [logging channels]({% link {{ page.version.version }}/logging-overview.md %}#logging-channels). For more details, refer to [`log.channel_compatibility_mode.enabled`]({% link {{ page.version.version }}/logging-overview.md %}#log-channel_compatibility_mode-enabled). + +To assess potential downstream impacts on your logging setup and pipelines, set the `log.channel_compatibility_mode.enabled` cluster setting to `false` in a non-production environment. +{{site.data.alerts.end}} + {{site.data.alerts.callout_info}} All log examples on this page use the default `crdb-v2` format, except for the [network logging](#network-logging) configuration, which uses the default `json-fluent-compact` format for network output. Most log entries for non-`DEV` channels record *structured* events, which use a standardized format that can be reliably parsed by an external collector. All structured event types and their fields are detailed in the [Notable events reference]({% link {{ page.version.version }}/eventlog.md %}).