From 252d3ed322bb1acc6a69fc653fd1f8dc73724598 Mon Sep 17 00:00:00 2001 From: katmayb Date: Fri, 23 May 2025 10:17:32 -0400 Subject: [PATCH 1/5] Add CHECK EXTERNAL CONNECTION SQL syntax to docs 25.1 25.2 --- .../backups/external-storage-check-tip.md | 1 + .../_includes/v25.1/sidebar-data/sql.json | 6 ++ .../backups/external-storage-check-tip.md | 1 + .../_includes/v25.2/sidebar-data/sql.json | 6 ++ src/current/v25.1/backup-validation.md | 4 + src/current/v25.1/backup.md | 4 + .../v25.1/check-external-connection.md | 93 +++++++++++++++++++ .../v25.1/cloud-storage-authentication.md | 4 + src/current/v25.1/use-cloud-storage.md | 4 + src/current/v25.2/backup-validation.md | 4 + src/current/v25.2/backup.md | 4 + .../v25.2/check-external-connection.md | 93 +++++++++++++++++++ .../v25.2/cloud-storage-authentication.md | 4 + src/current/v25.2/use-cloud-storage.md | 4 + 14 files changed, 232 insertions(+) create mode 100644 src/current/_includes/v25.1/backups/external-storage-check-tip.md create mode 100644 src/current/_includes/v25.2/backups/external-storage-check-tip.md create mode 100644 src/current/v25.1/check-external-connection.md create mode 100644 src/current/v25.2/check-external-connection.md diff --git a/src/current/_includes/v25.1/backups/external-storage-check-tip.md b/src/current/_includes/v25.1/backups/external-storage-check-tip.md new file mode 100644 index 00000000000..7337ed153e8 --- /dev/null +++ b/src/current/_includes/v25.1/backups/external-storage-check-tip.md @@ -0,0 +1 @@ +You can test the connection to your external storage from each node in the cluster with [`CHECK EXTERNAL CONNECTION`]({% link {{ page.version.version }}/check-external-connection.md %}). \ No newline at end of file diff --git a/src/current/_includes/v25.1/sidebar-data/sql.json b/src/current/_includes/v25.1/sidebar-data/sql.json index 148d7a6a3a7..2d2ea16d846 100644 --- a/src/current/_includes/v25.1/sidebar-data/sql.json +++ b/src/current/_includes/v25.1/sidebar-data/sql.json @@ -160,6 +160,12 @@ "/${VERSION}/cancel-session.html" ] }, + { + "title": "CHECK EXTERNAL CONNECTION", + "urls": [ + "/${VERSION}/check-external-connection.html" + ] + }, { "title": "COMMENT ON", "urls": [ diff --git a/src/current/_includes/v25.2/backups/external-storage-check-tip.md b/src/current/_includes/v25.2/backups/external-storage-check-tip.md new file mode 100644 index 00000000000..7337ed153e8 --- /dev/null +++ b/src/current/_includes/v25.2/backups/external-storage-check-tip.md @@ -0,0 +1 @@ +You can test the connection to your external storage from each node in the cluster with [`CHECK EXTERNAL CONNECTION`]({% link {{ page.version.version }}/check-external-connection.md %}). \ No newline at end of file diff --git a/src/current/_includes/v25.2/sidebar-data/sql.json b/src/current/_includes/v25.2/sidebar-data/sql.json index b4b9e2a71b6..8759eae97e6 100644 --- a/src/current/_includes/v25.2/sidebar-data/sql.json +++ b/src/current/_includes/v25.2/sidebar-data/sql.json @@ -166,6 +166,12 @@ "/${VERSION}/cancel-session.html" ] }, + { + "title": "CHECK EXTERNAL CONNECTION", + "urls": [ + "/${VERSION}/check-external-connection.html" + ] + }, { "title": "COMMENT ON", "urls": [ diff --git a/src/current/v25.1/backup-validation.md b/src/current/v25.1/backup-validation.md index a4f00b7754a..44fde6cfdb4 100644 --- a/src/current/v25.1/backup-validation.md +++ b/src/current/v25.1/backup-validation.md @@ -15,6 +15,10 @@ You can validate a backup of a [cluster]({% link {{ page.version.version }}/back The options that give the most validation coverage will increase the runtime of the check. That is, `verify_backup_table_data` will take a longer time to validate a backup compared to `check_files` or `schema_only` alone. Despite that, each of these validation options provide a quicker way to validate a backup over running a "regular" restore. +{{site.data.alerts.callout_success}} +{% include {{ page.version.version }}/backups/external-storage-check-tip.md %} +{{site.data.alerts.end}} + ## Recommendations Cockroach Labs recommends implementing the following validation plan to test your backups: diff --git a/src/current/v25.1/backup.md b/src/current/v25.1/backup.md index e45b569f058..b93d5e3e8f6 100644 --- a/src/current/v25.1/backup.md +++ b/src/current/v25.1/backup.md @@ -44,6 +44,10 @@ To view the contents of an backup created with the `BACKUP` statement, use [`SHO - Modifying backup files in the storage location could invalidate a backup, and therefore, prevent a restore. In v22.1 and later, **we recommend enabling [object locking]({% link {{ page.version.version }}/use-cloud-storage.md %}#immutable-storage) in your cloud storage bucket.** - While Cockroach Labs actively tests Amazon S3, Google Cloud Storage, and Azure Storage, we **do not** test [S3-compatible services]({% link {{ page.version.version }}/use-cloud-storage.md %}) (e.g., [MinIO](https://min.io/), [Red Hat Ceph](https://docs.ceph.com/en/pacific/radosgw/s3/)). +{{site.data.alerts.callout_success}} +{% include {{ page.version.version }}/backups/external-storage-check-tip.md %} +{{site.data.alerts.end}} + ## Required privileges {% include {{ page.version.version }}/backups/updated-backup-privileges.md %} diff --git a/src/current/v25.1/check-external-connection.md b/src/current/v25.1/check-external-connection.md new file mode 100644 index 00000000000..a4205fe4297 --- /dev/null +++ b/src/current/v25.1/check-external-connection.md @@ -0,0 +1,93 @@ +--- +title: CHECK EXTERNAL CONNECTION +summary: Test the connection of each node to your cloud storage location. +toc: true +--- + +{% include_cached new-in.html version="v25.1" %} The `CHECK EXTERNAL CONNECTION` tests the connection from each node in the cluster to an external cloud storage location. `CHECK EXTERNAL CONNECTION` will measure the time it takes each node to write a file, read it, and delete it from the specified storage location. You can adjust the number and concurrency of the tests run as well as the size of the file to write and read for each test. + +{{site.data.alerts.callout_info}} +`CHECK EXTERNAL CONNECTION` supports testing the connection to [**cloud storage**]({% link {{ page.version.version }}/use-cloud-storage.md %}) locations. +{{site.data.alerts.end}} + +## Synopsis + +
+{% remote_include https://raw.githubusercontent.com/cockroachdb/generated-diagrams/{{ page.release_info.crdb_branch_name }}/grammar_svg/check_external_connection.html %} +
+ +## Parameters + +Parameter | Description +----------+------------ +`connection_uri` | The URI to the external storage. Specify the [provider's URI]({% link {{ page.version.version }}/use-cloud-storage.md %}) (e.g., `gs://bucket_name?AUTH...'`) or a user-defined [external connection]({% link {{ page.version.version }}/create-external-connection.md %}) (e.g., `external://gcs`). + +## Options + +Option | Value | Description +--------+-------+------------ +`concurrently` | `INT` | Run multiple connection tests concurrently. If you also set the `time` option, it will run the specified number of concurrent tests until the time has elapsed. By default, only `1` connection test will run. +`time` | `STRING` | Run the test repeatedly until the duration has elapsed. +`transfer` | `STRING` | The size of the file that is written and read during each iteration of the connection test. By default, this will transfer a `32MiB` file. + +## Responses + +Field | Value | Description +------|-------|------------ +`node` | `INT` | The node ID. +`locality` | `STRING` | The [locality]({% link {{ page.version.version }}/cockroach-start.md %}#locality) of the node. +`ok` | `BOOL` | The success of the test run. +`error` | `STRING` | Errors encountered during the test run. +`transferred` | `STRING` | The size of the file transferred during the test. +`read_speed` | `STRING` | The speed at which the node read the test file. +`write_speed` | `STRING` | The speed at which the node wrote the test file. +`can_delete` | `BOOL` | The success of file deletion. + +## Examples + +Specify the connection URI to the [external storage location]({% link {{ page.version.version }}/use-cloud-storage.md %}), or a created [external connection]({% link {{ page.version.version }}/create-external-connection.md %}): + +{% include_cached copy-clipboard.html %} +~~~ sql +CHECK EXTERNAL CONNECTION 'external://cloud-storage'; +~~~ + +~~~ + node | locality | ok | error | transferred | read_speed | write_speed | can_delete +-------+-------------------------------------------+----+-------+-------------+-------------+-------------+------------- + 1 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 32 MiB | 66.17 MiB/s | 37.52 MiB/s | t + 3 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 32 MiB | 41.77 MiB/s | 33.55 MiB/s | t + 2 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 32 MiB | 14.23 MiB/s | 37.12 MiB/s | t +~~~ + +To modify the testing parameters, use one or a combination of the options: `concurrently`, `time`, `transfer`. For details on each, refer to [Options](#options). + +{% include_cached copy-clipboard.html %} +~~~ sql +CHECK EXTERNAL CONNECTION 'external://cloud-storage' WITH transfer = '50MiB', concurrently = 5, time = '1ms'; +~~~ +~~~ + node | locality | ok | error | transferred | read_speed | write_speed | can_delete +-------+-------------------------------------------+----+-------+-------------+-------------+-------------+------------- + 2 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 50 MiB | 59.85 MiB/s | 34.99 MiB/s | t + 1 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 50 MiB | 58.26 MiB/s | 34.91 MiB/s | t + 1 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 50 MiB | 57.69 MiB/s | 32.30 MiB/s | t + 2 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 50 MiB | 55.51 MiB/s | 33.02 MiB/s | t + 3 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 50 MiB | 59.29 MiB/s | 31.45 MiB/s | t + 3 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 50 MiB | 55.61 MiB/s | 32.58 MiB/s | t + 3 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 50 MiB | 61.04 MiB/s | 29.63 MiB/s | t + 1 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 50 MiB | 47.69 MiB/s | 34.04 MiB/s | t + 1 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 50 MiB | 55.66 MiB/s | 30.39 MiB/s | t + 1 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 50 MiB | 57.77 MiB/s | 29.64 MiB/s | t + 2 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 50 MiB | 44.95 MiB/s | 34.41 MiB/s | t + 2 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 50 MiB | 46.77 MiB/s | 33.31 MiB/s | t + 2 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 50 MiB | 57.64 MiB/s | 28.96 MiB/s | t + 3 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 50 MiB | 58.99 MiB/s | 26.65 MiB/s | t + 3 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 50 MiB | 15.14 MiB/s | 33.45 MiB/s | t +~~~ + +## See also + +- [Use Cloud Storage]({% link {{ page.version.version }}/use-cloud-storage.md %}) +- [Backup and Restore Overview]({% link {{ page.version.version }}/backup-and-restore-overview.md %}) +- [`CREATE EXTERNAL CONNECTION`]({% link {{ page.version.version }}/create-external-connection.md %}) \ No newline at end of file diff --git a/src/current/v25.1/cloud-storage-authentication.md b/src/current/v25.1/cloud-storage-authentication.md index 1357e00a422..8d225a6e2fb 100644 --- a/src/current/v25.1/cloud-storage-authentication.md +++ b/src/current/v25.1/cloud-storage-authentication.md @@ -11,6 +11,10 @@ Cockroach Labs supports different levels of authentication to cloud storage. Whe We recommend using IAM roles for users to authenticate to cloud storage resources. For more detail, see the assume role and workload identity sections for [Amazon S3]({% link {{ page.version.version }}/cloud-storage-authentication.md %}#set-up-amazon-s3-assume-role) and [Google Cloud Storage](cloud-storage-authentication.html?filters=gcs#set-up-google-cloud-storage-assume-role). {{site.data.alerts.end}} +{{site.data.alerts.callout_success}} +{% include {{ page.version.version }}/backups/external-storage-check-tip.md %} +{{site.data.alerts.end}} +
diff --git a/src/current/v25.1/use-cloud-storage.md b/src/current/v25.1/use-cloud-storage.md index b7acaac9b10..07c942e4a75 100644 --- a/src/current/v25.1/use-cloud-storage.md +++ b/src/current/v25.1/use-cloud-storage.md @@ -59,6 +59,10 @@ You can disable the use of implicit credentials when accessing external cloud st ### Example file URLs +{{site.data.alerts.callout_success}} +{% include {{ page.version.version }}/backups/external-storage-check-tip.md %} +{{site.data.alerts.end}} + Example URLs for [`BACKUP`]({% link {{ page.version.version }}/backup.md %}), [`RESTORE`]({% link {{ page.version.version }}/restore.md %}), or [`EXPORT`]({% link {{ page.version.version }}/export.md %}) given a bucket or container name of `acme-co` and an `employees` subdirectory: Location | Example diff --git a/src/current/v25.2/backup-validation.md b/src/current/v25.2/backup-validation.md index a4f00b7754a..44fde6cfdb4 100644 --- a/src/current/v25.2/backup-validation.md +++ b/src/current/v25.2/backup-validation.md @@ -15,6 +15,10 @@ You can validate a backup of a [cluster]({% link {{ page.version.version }}/back The options that give the most validation coverage will increase the runtime of the check. That is, `verify_backup_table_data` will take a longer time to validate a backup compared to `check_files` or `schema_only` alone. Despite that, each of these validation options provide a quicker way to validate a backup over running a "regular" restore. +{{site.data.alerts.callout_success}} +{% include {{ page.version.version }}/backups/external-storage-check-tip.md %} +{{site.data.alerts.end}} + ## Recommendations Cockroach Labs recommends implementing the following validation plan to test your backups: diff --git a/src/current/v25.2/backup.md b/src/current/v25.2/backup.md index e45b569f058..b93d5e3e8f6 100644 --- a/src/current/v25.2/backup.md +++ b/src/current/v25.2/backup.md @@ -44,6 +44,10 @@ To view the contents of an backup created with the `BACKUP` statement, use [`SHO - Modifying backup files in the storage location could invalidate a backup, and therefore, prevent a restore. In v22.1 and later, **we recommend enabling [object locking]({% link {{ page.version.version }}/use-cloud-storage.md %}#immutable-storage) in your cloud storage bucket.** - While Cockroach Labs actively tests Amazon S3, Google Cloud Storage, and Azure Storage, we **do not** test [S3-compatible services]({% link {{ page.version.version }}/use-cloud-storage.md %}) (e.g., [MinIO](https://min.io/), [Red Hat Ceph](https://docs.ceph.com/en/pacific/radosgw/s3/)). +{{site.data.alerts.callout_success}} +{% include {{ page.version.version }}/backups/external-storage-check-tip.md %} +{{site.data.alerts.end}} + ## Required privileges {% include {{ page.version.version }}/backups/updated-backup-privileges.md %} diff --git a/src/current/v25.2/check-external-connection.md b/src/current/v25.2/check-external-connection.md new file mode 100644 index 00000000000..a741b261a81 --- /dev/null +++ b/src/current/v25.2/check-external-connection.md @@ -0,0 +1,93 @@ +--- +title: CHECK EXTERNAL CONNECTION +summary: Test the connection of each node to your cloud storage location. +toc: true +--- + +The `CHECK EXTERNAL CONNECTION` tests the connection from each node in the cluster to an external cloud storage location. `CHECK EXTERNAL CONNECTION` will measure the time it takes each node to write a file, read it, and delete it from the specified storage location. You can adjust the number and concurrency of the tests run as well as the size of the file to write and read for each test. + +{{site.data.alerts.callout_info}} +`CHECK EXTERNAL CONNECTION` supports testing the connection to [**cloud storage**]({% link {{ page.version.version }}/use-cloud-storage.md %}) locations. +{{site.data.alerts.end}} + +## Synopsis + +
+{% remote_include https://raw.githubusercontent.com/cockroachdb/generated-diagrams/{{ page.release_info.crdb_branch_name }}/grammar_svg/check_external_connection.html %} +
+ +## Parameters + +Parameter | Description +----------+------------ +`connection_uri` | The URI to the external storage. Specify the [provider's URI]({% link {{ page.version.version }}/use-cloud-storage.md %}) (e.g., `gs://bucket_name?AUTH...'`) or a user-defined [external connection]({% link {{ page.version.version }}/create-external-connection.md %}) (e.g., `external://gcs`). + +## Options + +Option | Value | Description +--------+-------+------------ +`concurrently` | `INT` | Run multiple connection tests concurrently. If you also set the `time` option, it will run the specified number of concurrent tests until the time has elapsed. By default, only `1` connection test will run. +`time` | `STRING` | Run the test repeatedly until the duration has elapsed. +`transfer` | `STRING` | The size of the file that is written and read during each iteration of the connection test. By default, this will transfer a `32MiB` file. + +## Responses + +Field | Value | Description +------|-------|------------ +`node` | `INT` | The node ID. +`locality` | `STRING` | The [locality]({% link {{ page.version.version }}/cockroach-start.md %}#locality) of the node. +`ok` | `BOOL` | The success of the test run. +`error` | `STRING` | Errors encountered during the test run. +`transferred` | `STRING` | The size of the file transferred during the test. +`read_speed` | `STRING` | The speed at which the node read the test file. +`write_speed` | `STRING` | The speed at which the node wrote the test file. +`can_delete` | `BOOL` | The success of file deletion. + +## Examples + +Specify the connection URI to the [external storage location]({% link {{ page.version.version }}/use-cloud-storage.md %}), or a created [external connection]({% link {{ page.version.version }}/create-external-connection.md %}): + +{% include_cached copy-clipboard.html %} +~~~ sql +CHECK EXTERNAL CONNECTION 'external://cloud-storage'; +~~~ + +~~~ + node | locality | ok | error | transferred | read_speed | write_speed | can_delete +-------+-------------------------------------------+----+-------+-------------+-------------+-------------+------------- + 1 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 32 MiB | 66.17 MiB/s | 37.52 MiB/s | t + 3 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 32 MiB | 41.77 MiB/s | 33.55 MiB/s | t + 2 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 32 MiB | 14.23 MiB/s | 37.12 MiB/s | t +~~~ + +To modify the testing parameters, use one or a combination of the options: `concurrently`, `time`, `transfer`. For details on each, refer to [Options](#options). + +{% include_cached copy-clipboard.html %} +~~~ sql +CHECK EXTERNAL CONNECTION 'external://cloud-storage' WITH transfer = '50MiB', concurrently = 5, time = '1ms'; +~~~ +~~~ + node | locality | ok | error | transferred | read_speed | write_speed | can_delete +-------+-------------------------------------------+----+-------+-------------+-------------+-------------+------------- + 2 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 50 MiB | 59.85 MiB/s | 34.99 MiB/s | t + 1 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 50 MiB | 58.26 MiB/s | 34.91 MiB/s | t + 1 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 50 MiB | 57.69 MiB/s | 32.30 MiB/s | t + 2 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 50 MiB | 55.51 MiB/s | 33.02 MiB/s | t + 3 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 50 MiB | 59.29 MiB/s | 31.45 MiB/s | t + 3 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 50 MiB | 55.61 MiB/s | 32.58 MiB/s | t + 3 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 50 MiB | 61.04 MiB/s | 29.63 MiB/s | t + 1 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 50 MiB | 47.69 MiB/s | 34.04 MiB/s | t + 1 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 50 MiB | 55.66 MiB/s | 30.39 MiB/s | t + 1 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 50 MiB | 57.77 MiB/s | 29.64 MiB/s | t + 2 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 50 MiB | 44.95 MiB/s | 34.41 MiB/s | t + 2 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 50 MiB | 46.77 MiB/s | 33.31 MiB/s | t + 2 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 50 MiB | 57.64 MiB/s | 28.96 MiB/s | t + 3 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 50 MiB | 58.99 MiB/s | 26.65 MiB/s | t + 3 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 50 MiB | 15.14 MiB/s | 33.45 MiB/s | t +~~~ + +## See also + +- [Use Cloud Storage]({% link {{ page.version.version }}/use-cloud-storage.md %}) +- [Backup and Restore Overview]({% link {{ page.version.version }}/backup-and-restore-overview.md %}) +- [`CREATE EXTERNAL CONNECTION`]({% link {{ page.version.version }}/create-external-connection.md %}) \ No newline at end of file diff --git a/src/current/v25.2/cloud-storage-authentication.md b/src/current/v25.2/cloud-storage-authentication.md index 32f07e11c26..6a81d6ec648 100644 --- a/src/current/v25.2/cloud-storage-authentication.md +++ b/src/current/v25.2/cloud-storage-authentication.md @@ -11,6 +11,10 @@ Cockroach Labs supports different levels of authentication to cloud storage. Whe We recommend using IAM roles for users to authenticate to cloud storage resources. For more detail, see the assume role and workload identity sections for [Amazon S3]({% link {{ page.version.version }}/cloud-storage-authentication.md %}#set-up-amazon-s3-assume-role) and [Google Cloud Storage](cloud-storage-authentication.html?filters=gcs#set-up-google-cloud-storage-assume-role). {{site.data.alerts.end}} +{{site.data.alerts.callout_success}} +{% include {{ page.version.version }}/backups/external-storage-check-tip.md %} +{{site.data.alerts.end}} +
diff --git a/src/current/v25.2/use-cloud-storage.md b/src/current/v25.2/use-cloud-storage.md index b7acaac9b10..07c942e4a75 100644 --- a/src/current/v25.2/use-cloud-storage.md +++ b/src/current/v25.2/use-cloud-storage.md @@ -59,6 +59,10 @@ You can disable the use of implicit credentials when accessing external cloud st ### Example file URLs +{{site.data.alerts.callout_success}} +{% include {{ page.version.version }}/backups/external-storage-check-tip.md %} +{{site.data.alerts.end}} + Example URLs for [`BACKUP`]({% link {{ page.version.version }}/backup.md %}), [`RESTORE`]({% link {{ page.version.version }}/restore.md %}), or [`EXPORT`]({% link {{ page.version.version }}/export.md %}) given a bucket or container name of `acme-co` and an `employees` subdirectory: Location | Example From e9f4059e9046e7b34e16138ef0424df7bed3bfce Mon Sep 17 00:00:00 2001 From: katmayb Date: Thu, 5 Jun 2025 10:05:34 -0400 Subject: [PATCH 2/5] KC feedback --- .../_includes/v25.1/backups/external-storage-check-tip.md | 2 +- .../_includes/v25.2/backups/external-storage-check-tip.md | 2 +- src/current/v25.1/check-external-connection.md | 4 ++-- src/current/v25.2/check-external-connection.md | 4 ++-- 4 files changed, 6 insertions(+), 6 deletions(-) diff --git a/src/current/_includes/v25.1/backups/external-storage-check-tip.md b/src/current/_includes/v25.1/backups/external-storage-check-tip.md index 7337ed153e8..b8cb7561f4c 100644 --- a/src/current/_includes/v25.1/backups/external-storage-check-tip.md +++ b/src/current/_includes/v25.1/backups/external-storage-check-tip.md @@ -1 +1 @@ -You can test the connection to your external storage from each node in the cluster with [`CHECK EXTERNAL CONNECTION`]({% link {{ page.version.version }}/check-external-connection.md %}). \ No newline at end of file +You can test the connection from each node in the cluster to your external storage with the [`CHECK EXTERNAL CONNECTION`]({% link {{ page.version.version }}/check-external-connection.md %}) statement. \ No newline at end of file diff --git a/src/current/_includes/v25.2/backups/external-storage-check-tip.md b/src/current/_includes/v25.2/backups/external-storage-check-tip.md index 7337ed153e8..b8cb7561f4c 100644 --- a/src/current/_includes/v25.2/backups/external-storage-check-tip.md +++ b/src/current/_includes/v25.2/backups/external-storage-check-tip.md @@ -1 +1 @@ -You can test the connection to your external storage from each node in the cluster with [`CHECK EXTERNAL CONNECTION`]({% link {{ page.version.version }}/check-external-connection.md %}). \ No newline at end of file +You can test the connection from each node in the cluster to your external storage with the [`CHECK EXTERNAL CONNECTION`]({% link {{ page.version.version }}/check-external-connection.md %}) statement. \ No newline at end of file diff --git a/src/current/v25.1/check-external-connection.md b/src/current/v25.1/check-external-connection.md index a4205fe4297..5d90de67445 100644 --- a/src/current/v25.1/check-external-connection.md +++ b/src/current/v25.1/check-external-connection.md @@ -4,7 +4,7 @@ summary: Test the connection of each node to your cloud storage location. toc: true --- -{% include_cached new-in.html version="v25.1" %} The `CHECK EXTERNAL CONNECTION` tests the connection from each node in the cluster to an external cloud storage location. `CHECK EXTERNAL CONNECTION` will measure the time it takes each node to write a file, read it, and delete it from the specified storage location. You can adjust the number and concurrency of the tests run as well as the size of the file to write and read for each test. +{% include_cached new-in.html version="v25.1" %} The `CHECK EXTERNAL CONNECTION` tests the connection from each node in the cluster to an external cloud storage location. `CHECK EXTERNAL CONNECTION` will measure the time it takes each node to write a file, read it, and delete it from the specified storage location. You can adjust the number and concurrency of the test runs as well as the size of the file to write and read for each test. {{site.data.alerts.callout_info}} `CHECK EXTERNAL CONNECTION` supports testing the connection to [**cloud storage**]({% link {{ page.version.version }}/use-cloud-storage.md %}) locations. @@ -20,7 +20,7 @@ toc: true Parameter | Description ----------+------------ -`connection_uri` | The URI to the external storage. Specify the [provider's URI]({% link {{ page.version.version }}/use-cloud-storage.md %}) (e.g., `gs://bucket_name?AUTH...'`) or a user-defined [external connection]({% link {{ page.version.version }}/create-external-connection.md %}) (e.g., `external://gcs`). +`connection_uri` | The URI to the external storage. Specify the [provider's URI]({% link {{ page.version.version }}/use-cloud-storage.md %}) (e.g., `gs://bucket_name?AUTH...`) or a user-defined [external connection]({% link {{ page.version.version }}/create-external-connection.md %}) (e.g., `external://gcs`). ## Options diff --git a/src/current/v25.2/check-external-connection.md b/src/current/v25.2/check-external-connection.md index a741b261a81..5d0bca62cb1 100644 --- a/src/current/v25.2/check-external-connection.md +++ b/src/current/v25.2/check-external-connection.md @@ -4,7 +4,7 @@ summary: Test the connection of each node to your cloud storage location. toc: true --- -The `CHECK EXTERNAL CONNECTION` tests the connection from each node in the cluster to an external cloud storage location. `CHECK EXTERNAL CONNECTION` will measure the time it takes each node to write a file, read it, and delete it from the specified storage location. You can adjust the number and concurrency of the tests run as well as the size of the file to write and read for each test. +The `CHECK EXTERNAL CONNECTION` tests the connection from each node in the cluster to an external cloud storage location. `CHECK EXTERNAL CONNECTION` will measure the time it takes each node to write a file, read it, and delete it from the specified storage location. You can adjust the number and concurrency of the test runs as well as the size of the file to write and read for each test. {{site.data.alerts.callout_info}} `CHECK EXTERNAL CONNECTION` supports testing the connection to [**cloud storage**]({% link {{ page.version.version }}/use-cloud-storage.md %}) locations. @@ -20,7 +20,7 @@ The `CHECK EXTERNAL CONNECTION` tests the connection from each node in the clust Parameter | Description ----------+------------ -`connection_uri` | The URI to the external storage. Specify the [provider's URI]({% link {{ page.version.version }}/use-cloud-storage.md %}) (e.g., `gs://bucket_name?AUTH...'`) or a user-defined [external connection]({% link {{ page.version.version }}/create-external-connection.md %}) (e.g., `external://gcs`). +`connection_uri` | The URI to the external storage. Specify the [provider's URI]({% link {{ page.version.version }}/use-cloud-storage.md %}) (e.g., `gs://bucket_name?AUTH...`) or a user-defined [external connection]({% link {{ page.version.version }}/create-external-connection.md %}) (e.g., `external://gcs`). ## Options From 07fc12286c29ca18df2ec8aed734456241afa385 Mon Sep 17 00:00:00 2001 From: katmayb Date: Mon, 9 Jun 2025 10:14:25 -0400 Subject: [PATCH 3/5] AL feedback --- src/current/v25.1/check-external-connection.md | 2 +- src/current/v25.2/check-external-connection.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/src/current/v25.1/check-external-connection.md b/src/current/v25.1/check-external-connection.md index 5d90de67445..b629babe4c4 100644 --- a/src/current/v25.1/check-external-connection.md +++ b/src/current/v25.1/check-external-connection.md @@ -7,7 +7,7 @@ toc: true {% include_cached new-in.html version="v25.1" %} The `CHECK EXTERNAL CONNECTION` tests the connection from each node in the cluster to an external cloud storage location. `CHECK EXTERNAL CONNECTION` will measure the time it takes each node to write a file, read it, and delete it from the specified storage location. You can adjust the number and concurrency of the test runs as well as the size of the file to write and read for each test. {{site.data.alerts.callout_info}} -`CHECK EXTERNAL CONNECTION` supports testing the connection to [**cloud storage**]({% link {{ page.version.version }}/use-cloud-storage.md %}) locations. +You can use the `CHECK EXTERNAL CONNECTION` to test the connection to [**cloud storage**]({% link {{ page.version.version }}/use-cloud-storage.md %}) locations. {{site.data.alerts.end}} ## Synopsis diff --git a/src/current/v25.2/check-external-connection.md b/src/current/v25.2/check-external-connection.md index 5d0bca62cb1..73f0fb48b45 100644 --- a/src/current/v25.2/check-external-connection.md +++ b/src/current/v25.2/check-external-connection.md @@ -7,7 +7,7 @@ toc: true The `CHECK EXTERNAL CONNECTION` tests the connection from each node in the cluster to an external cloud storage location. `CHECK EXTERNAL CONNECTION` will measure the time it takes each node to write a file, read it, and delete it from the specified storage location. You can adjust the number and concurrency of the test runs as well as the size of the file to write and read for each test. {{site.data.alerts.callout_info}} -`CHECK EXTERNAL CONNECTION` supports testing the connection to [**cloud storage**]({% link {{ page.version.version }}/use-cloud-storage.md %}) locations. +You can use the `CHECK EXTERNAL CONNECTION` to test the connection to [**cloud storage**]({% link {{ page.version.version }}/use-cloud-storage.md %}) locations. {{site.data.alerts.end}} ## Synopsis From 928429641e520755505a958bfc8862770120fcc0 Mon Sep 17 00:00:00 2001 From: katmayb Date: Mon, 9 Jun 2025 14:20:11 -0400 Subject: [PATCH 4/5] RL feedback --- src/current/v25.2/check-external-connection.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/src/current/v25.2/check-external-connection.md b/src/current/v25.2/check-external-connection.md index 73f0fb48b45..3037bb413d4 100644 --- a/src/current/v25.2/check-external-connection.md +++ b/src/current/v25.2/check-external-connection.md @@ -36,14 +36,14 @@ Field | Value | Description ------|-------|------------ `node` | `INT` | The node ID. `locality` | `STRING` | The [locality]({% link {{ page.version.version }}/cockroach-start.md %}#locality) of the node. -`ok` | `BOOL` | The success of the test run. +`ok` | `BOOL` | Whether the test run succeeded. `error` | `STRING` | Errors encountered during the test run. `transferred` | `STRING` | The size of the file transferred during the test. `read_speed` | `STRING` | The speed at which the node read the test file. `write_speed` | `STRING` | The speed at which the node wrote the test file. -`can_delete` | `BOOL` | The success of file deletion. +`can_delete` | `BOOL` | Whether file deletion succeeded. -## Examples +## Test an external connection Specify the connection URI to the [external storage location]({% link {{ page.version.version }}/use-cloud-storage.md %}), or a created [external connection]({% link {{ page.version.version }}/create-external-connection.md %}): From 5427e8a99c351dd8b8419ac45d45e30093eb29ac Mon Sep 17 00:00:00 2001 From: katmayb Date: Mon, 9 Jun 2025 14:48:13 -0400 Subject: [PATCH 5/5] Add v25.3 version --- .../backups/external-storage-check-tip.md | 1 + .../_includes/v25.3/sidebar-data/sql.json | 6 ++ src/current/v25.3/backup-validation.md | 4 + src/current/v25.3/backup.md | 4 + .../v25.3/check-external-connection.md | 93 +++++++++++++++++++ .../v25.3/cloud-storage-authentication.md | 4 + src/current/v25.3/use-cloud-storage.md | 4 + 7 files changed, 116 insertions(+) create mode 100644 src/current/_includes/v25.3/backups/external-storage-check-tip.md create mode 100644 src/current/v25.3/check-external-connection.md diff --git a/src/current/_includes/v25.3/backups/external-storage-check-tip.md b/src/current/_includes/v25.3/backups/external-storage-check-tip.md new file mode 100644 index 00000000000..b8cb7561f4c --- /dev/null +++ b/src/current/_includes/v25.3/backups/external-storage-check-tip.md @@ -0,0 +1 @@ +You can test the connection from each node in the cluster to your external storage with the [`CHECK EXTERNAL CONNECTION`]({% link {{ page.version.version }}/check-external-connection.md %}) statement. \ No newline at end of file diff --git a/src/current/_includes/v25.3/sidebar-data/sql.json b/src/current/_includes/v25.3/sidebar-data/sql.json index b4b9e2a71b6..8759eae97e6 100644 --- a/src/current/_includes/v25.3/sidebar-data/sql.json +++ b/src/current/_includes/v25.3/sidebar-data/sql.json @@ -166,6 +166,12 @@ "/${VERSION}/cancel-session.html" ] }, + { + "title": "CHECK EXTERNAL CONNECTION", + "urls": [ + "/${VERSION}/check-external-connection.html" + ] + }, { "title": "COMMENT ON", "urls": [ diff --git a/src/current/v25.3/backup-validation.md b/src/current/v25.3/backup-validation.md index a4f00b7754a..44fde6cfdb4 100644 --- a/src/current/v25.3/backup-validation.md +++ b/src/current/v25.3/backup-validation.md @@ -15,6 +15,10 @@ You can validate a backup of a [cluster]({% link {{ page.version.version }}/back The options that give the most validation coverage will increase the runtime of the check. That is, `verify_backup_table_data` will take a longer time to validate a backup compared to `check_files` or `schema_only` alone. Despite that, each of these validation options provide a quicker way to validate a backup over running a "regular" restore. +{{site.data.alerts.callout_success}} +{% include {{ page.version.version }}/backups/external-storage-check-tip.md %} +{{site.data.alerts.end}} + ## Recommendations Cockroach Labs recommends implementing the following validation plan to test your backups: diff --git a/src/current/v25.3/backup.md b/src/current/v25.3/backup.md index e45b569f058..b93d5e3e8f6 100644 --- a/src/current/v25.3/backup.md +++ b/src/current/v25.3/backup.md @@ -44,6 +44,10 @@ To view the contents of an backup created with the `BACKUP` statement, use [`SHO - Modifying backup files in the storage location could invalidate a backup, and therefore, prevent a restore. In v22.1 and later, **we recommend enabling [object locking]({% link {{ page.version.version }}/use-cloud-storage.md %}#immutable-storage) in your cloud storage bucket.** - While Cockroach Labs actively tests Amazon S3, Google Cloud Storage, and Azure Storage, we **do not** test [S3-compatible services]({% link {{ page.version.version }}/use-cloud-storage.md %}) (e.g., [MinIO](https://min.io/), [Red Hat Ceph](https://docs.ceph.com/en/pacific/radosgw/s3/)). +{{site.data.alerts.callout_success}} +{% include {{ page.version.version }}/backups/external-storage-check-tip.md %} +{{site.data.alerts.end}} + ## Required privileges {% include {{ page.version.version }}/backups/updated-backup-privileges.md %} diff --git a/src/current/v25.3/check-external-connection.md b/src/current/v25.3/check-external-connection.md new file mode 100644 index 00000000000..3037bb413d4 --- /dev/null +++ b/src/current/v25.3/check-external-connection.md @@ -0,0 +1,93 @@ +--- +title: CHECK EXTERNAL CONNECTION +summary: Test the connection of each node to your cloud storage location. +toc: true +--- + +The `CHECK EXTERNAL CONNECTION` tests the connection from each node in the cluster to an external cloud storage location. `CHECK EXTERNAL CONNECTION` will measure the time it takes each node to write a file, read it, and delete it from the specified storage location. You can adjust the number and concurrency of the test runs as well as the size of the file to write and read for each test. + +{{site.data.alerts.callout_info}} +You can use the `CHECK EXTERNAL CONNECTION` to test the connection to [**cloud storage**]({% link {{ page.version.version }}/use-cloud-storage.md %}) locations. +{{site.data.alerts.end}} + +## Synopsis + +
+{% remote_include https://raw.githubusercontent.com/cockroachdb/generated-diagrams/{{ page.release_info.crdb_branch_name }}/grammar_svg/check_external_connection.html %} +
+ +## Parameters + +Parameter | Description +----------+------------ +`connection_uri` | The URI to the external storage. Specify the [provider's URI]({% link {{ page.version.version }}/use-cloud-storage.md %}) (e.g., `gs://bucket_name?AUTH...`) or a user-defined [external connection]({% link {{ page.version.version }}/create-external-connection.md %}) (e.g., `external://gcs`). + +## Options + +Option | Value | Description +--------+-------+------------ +`concurrently` | `INT` | Run multiple connection tests concurrently. If you also set the `time` option, it will run the specified number of concurrent tests until the time has elapsed. By default, only `1` connection test will run. +`time` | `STRING` | Run the test repeatedly until the duration has elapsed. +`transfer` | `STRING` | The size of the file that is written and read during each iteration of the connection test. By default, this will transfer a `32MiB` file. + +## Responses + +Field | Value | Description +------|-------|------------ +`node` | `INT` | The node ID. +`locality` | `STRING` | The [locality]({% link {{ page.version.version }}/cockroach-start.md %}#locality) of the node. +`ok` | `BOOL` | Whether the test run succeeded. +`error` | `STRING` | Errors encountered during the test run. +`transferred` | `STRING` | The size of the file transferred during the test. +`read_speed` | `STRING` | The speed at which the node read the test file. +`write_speed` | `STRING` | The speed at which the node wrote the test file. +`can_delete` | `BOOL` | Whether file deletion succeeded. + +## Test an external connection + +Specify the connection URI to the [external storage location]({% link {{ page.version.version }}/use-cloud-storage.md %}), or a created [external connection]({% link {{ page.version.version }}/create-external-connection.md %}): + +{% include_cached copy-clipboard.html %} +~~~ sql +CHECK EXTERNAL CONNECTION 'external://cloud-storage'; +~~~ + +~~~ + node | locality | ok | error | transferred | read_speed | write_speed | can_delete +-------+-------------------------------------------+----+-------+-------------+-------------+-------------+------------- + 1 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 32 MiB | 66.17 MiB/s | 37.52 MiB/s | t + 3 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 32 MiB | 41.77 MiB/s | 33.55 MiB/s | t + 2 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 32 MiB | 14.23 MiB/s | 37.12 MiB/s | t +~~~ + +To modify the testing parameters, use one or a combination of the options: `concurrently`, `time`, `transfer`. For details on each, refer to [Options](#options). + +{% include_cached copy-clipboard.html %} +~~~ sql +CHECK EXTERNAL CONNECTION 'external://cloud-storage' WITH transfer = '50MiB', concurrently = 5, time = '1ms'; +~~~ +~~~ + node | locality | ok | error | transferred | read_speed | write_speed | can_delete +-------+-------------------------------------------+----+-------+-------------+-------------+-------------+------------- + 2 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 50 MiB | 59.85 MiB/s | 34.99 MiB/s | t + 1 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 50 MiB | 58.26 MiB/s | 34.91 MiB/s | t + 1 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 50 MiB | 57.69 MiB/s | 32.30 MiB/s | t + 2 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 50 MiB | 55.51 MiB/s | 33.02 MiB/s | t + 3 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 50 MiB | 59.29 MiB/s | 31.45 MiB/s | t + 3 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 50 MiB | 55.61 MiB/s | 32.58 MiB/s | t + 3 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 50 MiB | 61.04 MiB/s | 29.63 MiB/s | t + 1 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 50 MiB | 47.69 MiB/s | 34.04 MiB/s | t + 1 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 50 MiB | 55.66 MiB/s | 30.39 MiB/s | t + 1 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 50 MiB | 57.77 MiB/s | 29.64 MiB/s | t + 2 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 50 MiB | 44.95 MiB/s | 34.41 MiB/s | t + 2 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 50 MiB | 46.77 MiB/s | 33.31 MiB/s | t + 2 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 50 MiB | 57.64 MiB/s | 28.96 MiB/s | t + 3 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 50 MiB | 58.99 MiB/s | 26.65 MiB/s | t + 3 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 50 MiB | 15.14 MiB/s | 33.45 MiB/s | t +~~~ + +## See also + +- [Use Cloud Storage]({% link {{ page.version.version }}/use-cloud-storage.md %}) +- [Backup and Restore Overview]({% link {{ page.version.version }}/backup-and-restore-overview.md %}) +- [`CREATE EXTERNAL CONNECTION`]({% link {{ page.version.version }}/create-external-connection.md %}) \ No newline at end of file diff --git a/src/current/v25.3/cloud-storage-authentication.md b/src/current/v25.3/cloud-storage-authentication.md index 32f07e11c26..6a81d6ec648 100644 --- a/src/current/v25.3/cloud-storage-authentication.md +++ b/src/current/v25.3/cloud-storage-authentication.md @@ -11,6 +11,10 @@ Cockroach Labs supports different levels of authentication to cloud storage. Whe We recommend using IAM roles for users to authenticate to cloud storage resources. For more detail, see the assume role and workload identity sections for [Amazon S3]({% link {{ page.version.version }}/cloud-storage-authentication.md %}#set-up-amazon-s3-assume-role) and [Google Cloud Storage](cloud-storage-authentication.html?filters=gcs#set-up-google-cloud-storage-assume-role). {{site.data.alerts.end}} +{{site.data.alerts.callout_success}} +{% include {{ page.version.version }}/backups/external-storage-check-tip.md %} +{{site.data.alerts.end}} +
diff --git a/src/current/v25.3/use-cloud-storage.md b/src/current/v25.3/use-cloud-storage.md index b7acaac9b10..07c942e4a75 100644 --- a/src/current/v25.3/use-cloud-storage.md +++ b/src/current/v25.3/use-cloud-storage.md @@ -59,6 +59,10 @@ You can disable the use of implicit credentials when accessing external cloud st ### Example file URLs +{{site.data.alerts.callout_success}} +{% include {{ page.version.version }}/backups/external-storage-check-tip.md %} +{{site.data.alerts.end}} + Example URLs for [`BACKUP`]({% link {{ page.version.version }}/backup.md %}), [`RESTORE`]({% link {{ page.version.version }}/restore.md %}), or [`EXPORT`]({% link {{ page.version.version }}/export.md %}) given a bucket or container name of `acme-co` and an `employees` subdirectory: Location | Example