Move OP destination shortcodes (#33278)

* move destination shortcodes

* fix datadog logs

* Apply suggestions from code review

Co-authored-by: Bryce Eadie <[email protected]>

* apply image and link fixes

* Fix link

---------

Co-authored-by: Bryce Eadie <[email protected]>

Author: maycmlee

content/en/logs/log_configuration/archives.md

@@ -427,7 +427,7 @@ This directory structure simplifies the process of querying your historical log
 [1]: /logs/indexes/#exclusion-filters
 [2]: /logs/archives/rehydrating/
 [3]: https://app.datadoghq.com/logs/pipelines/log-forwarding
-[4]: /observability_pipelines/archive_logs/
+[4]: /observability_pipelines/configuration/explore_templates/?tab=logs#archive-logs
 [5]: /account_management/rbac/permissions/?tab=ui#logs_write_archives
 [6]: https://app.datadoghq.com/logs/pipelines/archives
 [7]: /integrations/azure/

content/en/observability_pipelines/configuration/install_the_worker/advanced_worker_configurations.md

@@ -11,7 +11,7 @@ further_reading:
 - link: "/observability_pipelines/dual_ship_logs/"
   tag: "Documentation"
   text: "Dual ship logs with Observability Pipelines"
-- link: "/observability_pipelines/archive_logs/"
+- link: "/observability_pipelines/configuration/explore_templates/?tab=logs#archive-logs"
   tag: "Documentation"
   text: "Archive logs with Observability Pipelines"
 - link: "/observability_pipelines/split_logs/"

content/en/observability_pipelines/destinations/amazon_s3.md

@@ -44,7 +44,46 @@ You need to have Datadog's [AWS integration][3] installed to set up Datadog Log
 
 Set up the Amazon S3 destination and its environment variables when you [set up an Archive Logs pipeline][4]. The information below is configured in the pipelines UI.
 
-{{% observability_pipelines/destination_settings/datadog_archives_amazon_s3 %}}
+1. Enter your S3 bucket name. If you configured Log Archives, it's the name of the bucket you created earlier.
+1. Enter the AWS region the S3 bucket is in.
+1. Enter the key prefix.
+    - Prefixes are useful for partitioning objects. For example, you can use a prefix as an object key to store objects under a particular directory. If using a prefix for this purpose, it must end in `/` to act as a directory path; a trailing `/` is not automatically added.
+    - See [template syntax][8] if you want to route logs to different object keys based on specific fields in your logs.
+     - **Note**: Datadog recommends that you start your prefixes with the directory name and without a lead slash (`/`). For example, `app-logs/` or `service-logs/`.
+1. Select the storage class for your S3 bucket in the **Storage Class** dropdown menu. If you are going to archive and rehydrate your logs:
+    - **Note**: Rehydration only supports the following [storage classes][9]:
+        - Standard
+        - Intelligent-Tiering, only if [the optional asynchronous archive access tiers][10] are both disabled.
+        - Standard-IA
+        - One Zone-IA
+    - If you wish to rehydrate from archives in another storage class, you must first move them to one of the supported storage classes above.
+    - See the [Example destination and log archive setup](#example-destination-and-log-archive-setup) section of this page for how to configure your Log Archive based on your Amazon S3 destination setup.
+1. Optionally, select an AWS authentication option. If you are only using the [user or role you created earlier](#set-up-an-iam-policy-that-allows-workers-to-write-to-the-s3-bucket) for authentication, do not select **Assume role**. The **Assume role** option should only be used if the user or role you created earlier needs to assume a different role to access the specific AWS resource and that permission has to be explicitly defined.<br>If you select **Assume role**:
+    1. Enter the ARN of the IAM role you want to assume.
+    1. Optionally, enter the assumed role session name and external ID.
+    - **Note:** The [user or role you created earlier](#set-up-an-iam-policy-that-allows-workers-to-write-to-the-s3-bucket) must have permission to assume this role so that the Worker can authenticate with AWS.
+1. Optionally, toggle the switch to enable **Buffering Options**.<br>**Note**: Buffering options is in Preview. Contact your account manager to request access.
+	- If left disabled, the maximum size for buffering is 500 events.
+	- If enabled:
+		1. Select the buffer type you want to set (**Memory** or **Disk**).
+		1. Enter the buffer size and select the unit.
+
+#### Example destination and log archive setup
+
+If you enter the following values for your Amazon S3 destination:
+- S3 Bucket Name: `test-op-bucket`
+- Prefix to apply to all object keys: `op-logs`
+- Storage class for the created objects: `Standard`
+
+{{< img src="observability_pipelines/setup/amazon_s3_destination.png" alt="The Amazon S3 destination setup with the example values" style="width:40%;" >}}
+
+Then these are the values you enter for configuring the S3 bucket for Log Archives:
+
+- S3 bucket: `test-op-bucket`
+- Path: `op-logs`
+- Storage class: `Standard`
+
+{{< img src="observability_pipelines/setup/amazon_s3_archive.png" alt="The log archive configuration with the example values" style="width:70%;" >}}
 
 ### Set the environment variables
 
@@ -78,7 +117,10 @@ A batch of events is flushed when one of these parameters is met. See [event bat
 [1]: /logs/log_configuration/archives/
 [2]: /logs/log_configuration/rehydrating/
 [3]: /integrations/amazon_web_services/#setup
-[4]: /observability_pipelines/archive_logs/
+[4]: /observability_pipelines/configuration/explore_templates/?tab=logs#archive-logs
 [5]: /observability_pipelines/configuration/set_up_pipelines/
 [6]: https://docs.snowflake.com/en/user-guide/data-load-snowpipe-auto-s3
 [7]: /observability_pipelines/destinations/#event-batching
+[8]: /observability_pipelines/destinations/#template-syntax
+[9]: /logs/log_configuration/archives/?tab=awss3#storage-class
+[10]: https://aws.amazon.com/s3/storage-classes/intelligent-tiering/

content/en/observability_pipelines/destinations/amazon_security_lake.md

@@ -17,7 +17,25 @@ Set up the Amazon Security Lake destination and its environment variables when y
 
 ### Set up the destination
 
-{{% observability_pipelines/destination_settings/amazon_security_lake %}}
+1. Enter your S3 bucket name.
+1. Enter the AWS region.
+1. Enter the custom source name.
+1. Optionally, select an [AWS authentication][5] option.
+    1. Enter the ARN of the IAM role you want to assume.
+    1. Optionally, enter the assumed role session name and external ID.
+1. Optionally, toggle the switch to enable TLS. If you enable TLS, the following certificate and key files are required.<br>**Note**: All file paths are made relative to the configuration data directory, which is `/var/lib/observability-pipelines-worker/config/` by default. See [Advanced Worker Configurations][4] for more information. The file must be owned by the `observability-pipelines-worker group` and `observability-pipelines-worker` user, or at least readable by the group or user.
+    - `Server Certificate Path`: The path to the certificate file that has been signed by your Certificate Authority (CA) Root File in DER or PEM (X.509).
+    - `CA Certificate Path`: The path to the certificate file that is your Certificate Authority (CA) Root File in DER or PEM (X.509).
+    - `Private Key Path`: The path to the `.key` private key file that belongs to your Server Certificate Path in DER or PEM (PKCS#8) format.
+1. Optionally, toggle the switch to enable **Buffering Options**.<br>**Note**: Buffering options is in Preview. Contact your account manager to request access.
+	- If left disabled, the maximum size for buffering is 500 events.
+	- If enabled:
+		1. Select the buffer type you want to set (**Memory** or **Disk**).
+		1. Enter the buffer size and select the unit.
+
+**Notes**:
+- When you add the Amazon Security Lake destination, the OCSF processor is automatically added so that you can convert your logs to Parquet before they are sent to Amazon Security Lake. See [Remap to OCSF documentation][3] for setup instructions.
+- Only logs formatted by the OCSF processor are converted to Parquet.
 
 ### Set the environment variables
 
@@ -42,4 +60,7 @@ A batch of events is flushed when one of these parameters is met. See [event bat
 | None           | 256,000,000     | 300                 |
 
 [1]: https://app.datadoghq.com/observability-pipelines
-[2]: /observability_pipelines/destinations/#event-batching
+[2]: /observability_pipelines/destinations/#event-batching
+[3]: /observability_pipelines/processors/remap_ocsf
+[4]: /observability_pipelines/configuration/install_the_worker/advanced_worker_configurations/
+[5]: /observability_pipelines/destinations/amazon_security_lake/#aws-authentication

content/en/observability_pipelines/destinations/azure_storage.md

@@ -17,7 +17,16 @@ You need to have Datadog's [Azure integration][3] installed to set up Datadog Lo
 
 Set up the Azure Storage destination and its environment variables when you [set up an Archive Logs pipeline][4]. The information below is configured in the pipelines UI.
 
-{{% observability_pipelines/destination_settings/datadog_archives_azure_storage %}}
+1. Enter the name of the Azure container you created earlier.
+1. Optionally, enter a prefix.
+    - Prefixes are useful for partitioning objects. For example, you can use a prefix as an object key to store objects under a particular directory. If using a prefix for this purpose, it must end in `/` to act as a directory path; a trailing `/` is not automatically added.
+    - See [template syntax][6] if you want to route logs to different object keys based on specific fields in your logs.
+     - **Note**: Datadog recommends that you start your prefixes with the directory name and without a lead slash (`/`). For example, `app-logs/` or `service-logs/`.
+1. Optionally, toggle the switch to enable **Buffering Options**.<br>**Note**: Buffering options is in Preview. Contact your account manager to request access.
+	- If left disabled, the maximum size for buffering is 500 events.
+	- If enabled:
+		1. Select the buffer type you want to set (**Memory** or **Disk**).
+		1. Enter the buffer size and select the unit.
 
 ### Set the environment variables
 
@@ -36,5 +45,6 @@ A batch of events is flushed when one of these parameters is met. See [event bat
 [1]: /logs/log_configuration/archives/
 [2]: /logs/log_configuration/rehydrating/
 [3]: /integrations/azure/#setup
-[4]: /observability_pipelines/archive_logs/
-[5]: /observability_pipelines/destinations/#event-batching
+[4]: /observability_pipelines/configuration/explore_templates/?tab=logs#archive-logs
+[5]: /observability_pipelines/destinations/#event-batching
+[6]: /observability_pipelines/destinations/#template-syntax

content/en/observability_pipelines/destinations/crowdstrike_ng_siem.md

@@ -11,7 +11,19 @@ Set up the CrowdStrike NG-SIEM destination and its environment variables when yo
 
 ### Set up the destination
 
-{{% observability_pipelines/destination_settings/crowdstrike_ng_siem %}}
+To use the CrowdStrike NG-SIEM destination, you need to set up a CrowdStrike data connector using the HEC/HTTP Event Connector. See [Step 1: Set up the HEC/HTTP event data connector][3] for instructions. When you set up the data connector, you are given a HEC API key and URL, which you use when you configure the Observability Pipelines Worker later on.
+
+1. Select **JSON** or **Raw** encoding in the dropdown menu.
+1. Optionally, enable compressions and select an algorithm (**gzip** or **zlib**) in the dropdown menu.
+1. Optionally, toggle the switch to enable TLS. If you enable TLS, the following certificate and key files are required.<br>**Note**: All file paths are made relative to the configuration data directory, which is `/var/lib/observability-pipelines-worker/config/` by default. See [Advanced Worker Configurations][4] for more information. The file must be owned by the `observability-pipelines-worker group` and `observability-pipelines-worker` user, or at least readable by the group or user.
+    - `Server Certificate Path`: The path to the certificate file that has been signed by your Certificate Authority (CA) Root File in DER or PEM (X.509).
+    - `CA Certificate Path`: The path to the certificate file that is your Certificate Authority (CA) Root File in DER or PEM (X.509).
+    - `Private Key Path`: The path to the `.key` private key file that belongs to your Server Certificate Path in DER or PEM (PKCS#8) format.
+1. Optionally, toggle the switch to enable **Buffering Options**.<br>**Note**: Buffering options is in Preview. Contact your account manager to request access.
+	- If left disabled, the maximum size for buffering is 500 events.
+	- If enabled:
+		1. Select the buffer type you want to set (**Memory** or **Disk**).
+		1. Enter the buffer size and select the unit.
 
 ### Set the environment variables
 
@@ -28,4 +40,6 @@ A batch of events is flushed when one of these parameters is met. See [event bat
 | None           | 1,000,000       | 1                   |
 
 [1]: https://app.datadoghq.com/observability-pipelines
-[2]: /observability_pipelines/destinations/#event-batching
+[2]: /observability_pipelines/destinations/#event-batching
+[3]: https://falcon.us-2.crowdstrike.com/documentation/page/bdded008/hec-http-event-connector-guide
+[4]: /observability_pipelines/configuration/install_the_worker/advanced_worker_configurations/

content/en/observability_pipelines/destinations/datadog_logs.md

@@ -11,7 +11,11 @@ Set up the Datadog Logs destination and its environment variables when you [set
 
 ### Set up the destination
 
-{{% observability_pipelines/destination_settings/datadog %}}
+1. Optionally, toggle the switch to enable **Buffering Options**.<br>**Note**: Buffering options is in Preview. Contact your account manager to request access.
+	- If left disabled, the maximum size for buffering is 500 events.
+	- If enabled:
+		1. Select the buffer type you want to set (**Memory** or **Disk**).
+		1. Enter the buffer size and select the unit.
 
 ### Set the environment variables
 

content/en/observability_pipelines/destinations/google_chronicle.md

@@ -12,7 +12,21 @@ Set up the Google Chronicle destination and its environment variables when you [
 
 ### Set up the destination
 
-{{% observability_pipelines/destination_settings/chronicle %}}
+To set up the Worker's Google Chronicle destination:
+
+1. Enter the customer ID for your Google Chronicle instance.
+1. If you have a credentials JSON file, enter the path to your credentials JSON file. The credentials file must be placed under `DD_OP_DATA_DIR/config`. Alternatively, you can use the `GOOGLE_APPLICATION_CREDENTIALS` environment variable to provide the credential path.
+    - If you're using [workload identity][6] on Google Kubernetes Engine (GKE), the `GOOGLE_APPLICATION_CREDENTIALS` is provided for you.
+    - The Worker uses standard [Google authentication methods][7].
+1. Select **JSON** or **Raw** encoding in the dropdown menu.
+1. Enter the log type. See [template syntax][4] if you want to route logs to different log types based on specific fields in your logs.
+1. Optionally, toggle the switch to enable **Buffering Options**.<br>**Note**: Buffering options is in Preview. Contact your account manager to request access.
+	- If left disabled, the maximum size for buffering is 500 events.
+	- If enabled:
+		1. Select the buffer type you want to set (**Memory** or **Disk**).
+		1. Enter the buffer size and select the unit.
+
+**Note**: Logs sent to the Google Chronicle destination must have ingestion labels. For example, if the logs are from a A10 load balancer, it must have the ingestion label `A10_LOAD_BALANCER`. See Google Cloud's [Support log types with a default parser][5] for a list of available log types and their respective ingestion labels.
 
 ### Set the environment variables
 
@@ -30,4 +44,8 @@ A batch of events is flushed when one of these parameters is met. See [event bat
 
 [1]: https://app.datadoghq.com/observability-pipelines
 [2]: /observability_pipelines/destinations/#event-batching
-[3]: https://cloud.google.com/docs/authentication#auth-flowchart
+[3]: https://cloud.google.com/docs/authentication#auth-flowchart
+[4]: /observability_pipelines/destinations/#template-syntax
+[5]: https://cloud.google.com/chronicle/docs/ingestion/parser-list/supported-default-parsers#with-default-parser
+[6]:https://cloud.google.com/kubernetes-engine/docs/concepts/workload-identity
+[7]: https://cloud.google.com/docs/authentication#auth-flowchart

content/en/observability_pipelines/destinations/google_cloud_storage.md

@@ -23,7 +23,22 @@ You need to have Datadog's [Google Cloud Platform integration][3] installed to s
 
 Set up the Google Cloud Storage destination and its environment variables when you [set up an Archive Logs pipeline][4]. The information below is configured in the pipelines UI.
 
-{{% observability_pipelines/destination_settings/datadog_archives_google_cloud_storage %}}
+1. Enter the name of your Google Cloud storage bucket. If you configured Log Archives, it's the bucket you created earlier.
+1. If you have a credentials JSON file, enter the path to your credentials JSON file. If you configured Log Archives it's the credentials you downloaded [earlier](#create-a-service-account-to-allow-workers-to-write-to-the-bucket). The credentials file must be placed under `DD_OP_DATA_DIR/config`. Alternatively, you can use the `GOOGLE_APPLICATION_CREDENTIALS` environment variable to provide the credential path.
+    - If you're using [workload identity][9] on Google Kubernetes Engine (GKE), the `GOOGLE_APPLICATION_CREDENTIALS` is provided for you.
+    - The Worker uses standard [Google authentication methods][8].
+1. Select the storage class for the created objects.
+1. Select the access level of the created objects.
+1. Optionally, enter in the prefix.
+    - Prefixes are useful for partitioning objects. For example, you can use a prefix as an object key to store objects under a particular directory. If using a prefix for this purpose, it must end in `/` to act as a directory path; a trailing `/` is not automatically added.
+    - See [template syntax][7] if you want to route logs to different object keys based on specific fields in your logs.
+     - **Note**: Datadog recommends that you start your prefixes with the directory name and without a lead slash (`/`). For example, `app-logs/` or `service-logs/`.
+1. Optionally, click **Add Header** to add metadata.
+1. Optionally, toggle the switch to enable **Buffering Options**.<br>**Note**: Buffering options is in Preview. Contact your account manager to request access.
+	- If left disabled, the maximum size for buffering is 500 events.
+	- If enabled:
+		1. Select the buffer type you want to set (**Memory** or **Disk**).
+		1. Enter the buffer size and select the unit.
 
 ### Set the environment variables
 
@@ -42,6 +57,9 @@ A batch of events is flushed when one of these parameters is met. See [event bat
 [1]: /logs/log_configuration/archives/
 [2]: /logs/log_configuration/rehydrating/
 [3]: /integrations/google_cloud_platform/#setup
-[4]: /observability_pipelines/archive_logs/
+[4]: /observability_pipelines/configuration/explore_templates/?tab=logs#archive-logs
 [5]: /observability_pipelines/destinations/#event-batching
-[6]: https://cloud.google.com/docs/authentication#auth-flowchart
+[6]: https://cloud.google.com/docs/authentication#auth-flowchart
+[7]: /observability_pipelines/destinations/#template-syntax
+[8]: https://cloud.google.com/docs/authentication#auth-flowchart
+[9]: https://cloud.google.com/kubernetes-engine/docs/concepts/workload-identity