Move OP source shortcodes to main doc (#33275)

maycmlee · buraizu · web-flow · commit 348bcd129b4f · 2025-12-10T15:43:52.000-08:00
* move shortcodes

* Apply suggestions from code review

Co-authored-by: Bryce Eadie &lt;bryce.eadie@datadoghq.com&gt;

* Fix link

---------

Co-authored-by: Bryce Eadie &lt;bryce.eadie@datadoghq.com&gt;
diff --git a/content/en/observability_pipelines/sources/amazon_data_firehose.md b/content/en/observability_pipelines/sources/amazon_data_firehose.md
@@ -13,7 +13,13 @@ Use Observability Pipelines' Amazon Data Firehose source to receive logs from Am
 
 Select and set up this source when you [set up a pipeline][1]. The information below is for the source settings in the pipeline UI.
 
-{{% observability_pipelines/source_settings/amazon_data_firehose %}}
+1. Optionally, select an AWS authentication option. If you select **Assume role**:
+    1. Enter the ARN of the IAM role you want to assume.
+    1. Optionally, enter the assumed role session name and external ID.
+1. Optionally, toggle the switch to enable TLS. If you enable TLS, the following certificate and key files are required.<br>**Note**: All file paths are made relative to the configuration data directory, which is `/var/lib/observability-pipelines-worker/config/` by default. See [Advanced Worker Configurations][2] for more information. The file must be owned by the `observability-pipelines-worker group` and `observability-pipelines-worker` user, or at least readable by the group or user.
+    - `Server Certificate Path`: The path to the certificate file that has been signed by your Certificate Authority (CA) Root File in DER or PEM (X.509).
+    - `CA Certificate Path`: The path to the certificate file that is your Certificate Authority (CA) Root File in DER or PEM (X.509).
+    - `Private Key Path`: The path to the `.key` private key file that belongs to your Server Certificate Path in DER or PEM (PKCS#8) format.
 
 ## Set the environment variables
 
@@ -31,4 +37,5 @@ Select and set up this source when you [set up a pipeline][1]. The information b
 
 {{% observability_pipelines/aws_authentication/amazon_s3_source/permissions %}}
 
-[1]: /observability_pipelines/configuration/set_up_pipelines/
+[1]: /observability_pipelines/configuration/set_up_pipelines/
+[2]: /observability_pipelines/configuration/install_the_worker/advanced_worker_configurations/
diff --git a/content/en/observability_pipelines/sources/amazon_s3.md b/content/en/observability_pipelines/sources/amazon_s3.md
@@ -13,7 +13,14 @@ Use Observability Pipelines' Amazon S3 source to receive logs from Amazon S3. Se
 
 Select and set up this source when you [set up a pipeline][1]. The information below is for the source settings in the pipeline UI.
 
-{{% observability_pipelines/source_settings/amazon_s3 %}}
+1. Enter the AWS region.
+1. Optionally, select an AWS authentication option. If you select **Assume role**:
+    1. Enter the ARN of the IAM role you want to assume.
+    1. Optionally, enter the assumed role session name and external ID.
+1. Optionally, toggle the switch to enable TLS. If you enable TLS, the following certificate and key files are required.<br>**Note**: All file paths are made relative to the configuration data directory, which is `/var/lib/observability-pipelines-worker/config/` by default. See [Advanced Worker Configurations][2] for more information. The file must be owned by the `observability-pipelines-worker group` and `observability-pipelines-worker` user, or at least readable by the group or user.
+    - `Server Certificate Path`: The path to the certificate file that has been signed by your Certificate Authority (CA) Root File in DER or PEM (X.509).
+    - `CA Certificate Path`: The path to the certificate file that is your Certificate Authority (CA) Root File in DER or PEM (X.509).
+    - `Private Key Path`: The path to the `.key` private key file that belongs to your Server Certificate Path in DER or PEM (PKCS#8) format.
 
 ## Set the environment variables
 
@@ -28,4 +35,5 @@ Select and set up this source when you [set up a pipeline][1]. The information b
 {{% observability_pipelines/aws_authentication/amazon_s3_source/permissions %}}
 
 
-[1]: /observability_pipelines/configuration/set_up_pipelines/
+[1]: /observability_pipelines/configuration/set_up_pipelines/
+[2]: /observability_pipelines/configuration/install_the_worker/advanced_worker_configurations/
diff --git a/content/en/observability_pipelines/sources/fluent.md b/content/en/observability_pipelines/sources/fluent.md
@@ -13,7 +13,10 @@ Use Observability Pipelines' Fluentd or Fluent Bit source to receive logs from t
 
 Select and set up this source when you [set up a pipeline][1]. The information below are for the source settings in the pipeline UI.
 
-{{% observability_pipelines/source_settings/fluent %}}
+Optionally, toggle the switch to enable TLS. If you enable TLS, the following certificate and key files are required.<br>**Note**: All file paths are made relative to the configuration data directory, which is `/var/lib/observability-pipelines-worker/config/` by default. See [Advanced Worker Configurations][2] for more information. The file must be owned by the `observability-pipelines-worker group` and `observability-pipelines-worker` user, or at least readable by the group or user.
+- `Server Certificate Path`: The path to the certificate file that has been signed by your Certificate Authority (CA) Root File in DER or PEM (X.509) format.
+- `CA Certificate Path`: The path to the certificate file that is your Certificate Authority (CA) Root File in DER or PEM (X.509) format.
+- `Private Key Path`: The path to the `.key` private key file that belongs to your Server Certificate Path in DER or PEM (PKCS#8) format.
 
 ## Set the environment variables
 
@@ -23,4 +26,5 @@ Select and set up this source when you [set up a pipeline][1]. The information b
 
 {{% observability_pipelines/log_source_configuration/fluent %}}
 
-[1]: /observability_pipelines/configuration/set_up_pipelines/
+[1]: /observability_pipelines/configuration/set_up_pipelines/
+[2]: /observability_pipelines/configuration/install_the_worker/advanced_worker_configurations/
diff --git a/content/en/observability_pipelines/sources/google_pubsub.md b/content/en/observability_pipelines/sources/google_pubsub.md
@@ -13,10 +13,22 @@ Use Observability Pipelines' Google Pub/Sub source to pull logs from the Google
 
 Select and set up this source when you [set up a pipeline][1]. The information below is for the source settings in the pipeline UI.
 
-{{% observability_pipelines/source_settings/google_pubsub %}}
+1. Enter the name of the source project.
+1. If you have a credentials JSON file, enter the path to your credentials JSON file. The credentials file must be placed under `DD_OP_DATA_DIR/config`. Alternatively, you can use the `GOOGLE_APPLICATION_CREDENTIALS` environment variable to provide the credential path.
+    - If you're using [workload identity][2] on Google Kubernetes Engine (GKE), the `GOOGLE_APPLICATION_CREDENTIALS` is provided for you.
+    - The Worker uses standard [Google authentication methods][4].
+1. Enter the subscription name.
+1. Select the decoder you want to use (Bytes, GELF, JSON, syslog).
+1. Optionally, toggle the switch to enable TLS. If you enable TLS, the following certificate and key files are required.<br>**Note**: All file paths are made relative to the configuration data directory, which is `/var/lib/observability-pipelines-worker/config/` by default. See [Advanced Worker Configurations][3] for more information. The file must be owned by the `observability-pipelines-worker group` and `observability-pipelines-worker` user, or at least readable by the group or user.
+    - `Server Certificate Path`: The path to the certificate file that has been signed by your Certificate Authority (CA) Root File in DER or PEM (X.509).
+    - `CA Certificate Path`: The path to the certificate file that is your Certificate Authority (CA) Root File in DER or PEM (X.509).
+    - `Private Key Path`: The path to the `.key` private key file that belongs to your Server Certificate Path in DER or PEM (PKCS #8) format.
 
 ## Set the environment variables
 
 {{% observability_pipelines/configure_existing_pipelines/source_env_vars/google_pubsub %}}
 
-[1]: /observability_pipelines/configuration/set_up_pipelines/
+[1]: /observability_pipelines/configuration/set_up_pipelines/
+[2]: https://cloud.google.com/kubernetes-engine/docs/concepts/workload-identity
+[3]: /observability_pipelines/configuration/install_the_worker/advanced_worker_configurations/
+[4]: https://cloud.google.com/docs/authentication#auth-flowchart
diff --git a/content/en/observability_pipelines/sources/http_client.md b/content/en/observability_pipelines/sources/http_client.md
@@ -13,10 +13,22 @@ Use Observability Pipelines' HTTP/S Client source to pull logs from the upstream
 
 Select and set up this source when you [set up a pipeline][1]. The information below is for the source settings in the pipeline UI.
 
-{{% observability_pipelines/source_settings/http_client %}}
+To configure your HTTP/S Client source:
+
+1. Select your authorization strategy.
+2. Select the decoder you want to use on the HTTP messages. Logs pulled from the HTTP source must be in this format.
+3. Optionally, toggle the switch to enable TLS. If you enable TLS, the following certificate and key files are required.<br>**Note**: All file paths are made relative to the configuration data directory, which is `/var/lib/observability-pipelines-worker/config/` by default. See [Advanced Worker Configurations][2] for more information. The file must be owned by the `observability-pipelines-worker group` and `observability-pipelines-worker` user, or at least readable by the group or user.
+   - `Server Certificate Path`: The path to the certificate file that has been signed by your Certificate Authority (CA) Root File in DER or PEM (X.509) format.
+   - `CA Certificate Path`: The path to the certificate file that is your Certificate Authority (CA) Root File in DER or PEM (X.509) format.
+   - `Private Key Path`: The path to the `.key` private key file that belongs to your Server Certificate Path in DER or PEM (PKCS#8) format.
+4. Enter the interval between scrapes.
+   - Your HTTP Server must be able to handle GET requests at this interval.
+   - Since requests run concurrently, if a scrape takes longer than the interval given, a new scrape is started, which can consume extra resources. Set the timeout to a value lower than the scrape interval to prevent this from happening.
+5. Enter the timeout for each scrape request.
 
 ## Set the environment variables
 
 {{% observability_pipelines/configure_existing_pipelines/source_env_vars/http_client %}}
 
 [1]: /observability_pipelines/configuration/set_up_pipelines/
+[2]: /observability_pipelines/configuration/install_the_worker/advanced_worker_configurations/
diff --git a/content/en/observability_pipelines/sources/http_server.md b/content/en/observability_pipelines/sources/http_server.md
@@ -15,7 +15,14 @@ You can also [send AWS vended logs with Datadog Lambda Forwarder to Observabilit
 
 Select and set up this source when you [set up a pipeline][1]. The information below is for the source settings in the pipeline UI.
 
-{{% observability_pipelines/source_settings/http_server %}}
+To configure your HTTP/S Server source, enter the following:
+
+1. Select your authorization strategy.
+1. Select the decoder you want to use on the HTTP messages. Your HTTP client logs must be in this format. **Note**: If you select `bytes` decoding, the raw log is stored in the `message` field.
+1. Optionally, toggle the switch to enable TLS. If you enable TLS, the following certificate and key files are required.<br>**Note**: All file paths are made relative to the configuration data directory, which is `/var/lib/observability-pipelines-worker/config/` by default. See [Advanced Worker Configurations][2] for more information. The file must be owned by the `observability-pipelines-worker group` and `observability-pipelines-worker` user, or at least readable by the group or user.
+    - `Server Certificate Path`: The path to the certificate file that has been signed by your Certificate Authority (CA) Root File in DER or PEM (X.509).
+    - `CA Certificate Path`: The path to the certificate file that is your Certificate Authority (CA) Root File in DER or PEM (X.509).
+    - `Private Key Path`: The path to the `.key` private key file that belongs to your Server Certificate Path in DER or PEM (PKCS #8) format.
 
 ## Set the environment variables
 
@@ -39,3 +46,4 @@ To send AWS vended logs to Observability Pipelines with the HTTP/S Server source
 {{% observability_pipelines/lambda_forwarder/deploy_forwarder %}}
 
 [1]: https://app.datadoghq.com/observability-pipelines
+[2]: /observability_pipelines/configuration/install_the_worker/advanced_worker_configurations/
diff --git a/content/en/observability_pipelines/sources/kafka.md b/content/en/observability_pipelines/sources/kafka.md
@@ -15,7 +15,18 @@ You can also [send Azure Event Hub logs to Observability Pipelines using the Kaf
 
 Select and set up this source when you [set up a pipeline][1]. The information below is for the source settings in the pipeline UI.
 
-{{% observability_pipelines/source_settings/kafka %}}
+1. Enter the group ID.
+1. Enter the topic name. If there is more than one, click **Add Field** to add additional topics.
+1. Optionally, toggle the switch to enable SASL Authentication and select the mechanism (**PLAIN**, **SCHRAM-SHA-256**, or **SCHRAM-SHA-512**) in the dropdown menu.
+1. Optionally, toggle the switch to enable TLS. If you enable TLS, the following certificate and key files are required.<br>**Note**: All file paths are made relative to the configuration data directory, which is `/var/lib/observability-pipelines-worker/config/` by default. See [Advanced Worker Configurations][5] for more information. The file must be owned by the `observability-pipelines-worker group` and `observability-pipelines-worker` user, or at least readable by the group or user.
+    - `Server Certificate Path`: The path to the certificate file that has been signed by your Certificate Authority (CA) Root File in DER or PEM (X.509).
+    - `CA Certificate Path`: The path to the certificate file that is your Certificate Authority (CA) Root File in DER or PEM (X.509).
+    - `Private Key Path`: The path to the `.key` private key file that belongs to your Server Certificate Path in DER or PEM (PKCS#8) format.
+1. Optionally, click **Advanced** and click **Add Option** to add additional [librdkafka options](#librdkafka-options).
+    1. Select an option in the dropdown menu.
+    1. Enter a value for that option.
+    1. Check your values against the [librdkafka documentation][4] to make sure they have the correct type and are within the set range.
+    1. Click **Add Option** to add another librdkafka option.
 
 ## Set the environment variables
 
@@ -45,4 +56,6 @@ See the [librdkafka documentation][3] for more information and to ensure your va
 
 [1]: /observability_pipelines/configuration/set_up_pipelines/
 [2]: https://github.com/confluentinc/librdkafka/tree/master
-[3]: https://docs.confluent.io/platform/current/clients/librdkafka/html/md_CONFIGURATION.html
+[3]: https://docs.confluent.io/platform/current/clients/librdkafka/html/md_CONFIGURATION.html
+[4]: https://docs.confluent.io/platform/current/clients/librdkafka/html/md_CONFIGURATION.html
+[5]: /observability_pipelines/configuration/install_the_worker/advanced_worker_configurations/
diff --git a/content/en/observability_pipelines/sources/logstash.md b/content/en/observability_pipelines/sources/logstash.md
@@ -15,7 +15,10 @@ You can also use the Logstash source to [send logs to Observability Pipelines us
 
 Select and set up this source when you [set up a pipeline][1]. The information below is for the source settings in the pipeline UI.
 
-{{% observability_pipelines/source_settings/logstash %}}
+Optionally, toggle the switch to enable TLS. If you enable TLS, the following certificate and key files are required.<br>**Note**: All file paths are made relative to the configuration data directory, which is `/var/lib/observability-pipelines-worker/config/` by default. See [Advanced Worker Configurations][3] for more information. The file must be owned by the `observability-pipelines-worker group` and `observability-pipelines-worker` user, or at least readable by the group or user.
+- `Server Certificate Path`: The path to the certificate file that has been signed by your Certificate Authority (CA) Root File in DER or PEM (X.509).
+- `CA Certificate Path`: The path to the certificate file that is your Certificate Authority (CA) Root File in DER or PEM (X.509).
+- `Private Key Path`: The path to the `.key` private key file that belongs to your Server Certificate Path in DER or PEM (PKCS #8) format.
 
 ## Set the environment variables
 
@@ -26,4 +29,5 @@ Select and set up this source when you [set up a pipeline][1]. The information b
 {{% observability_pipelines/log_source_configuration/logstash %}}
 
 [1]: /observability_pipelines/configuration/set_up_pipelines/
-[2]: /observability_pipelines/sources/filebeat/
+[2]: /observability_pipelines/sources/filebeat/
+[3]: /observability_pipelines/configuration/install_the_worker/advanced_worker_configurations/
diff --git a/content/en/observability_pipelines/sources/socket.md b/content/en/observability_pipelines/sources/socket.md
@@ -13,7 +13,40 @@ Use Observability Pipelines' Socket source to send logs to the Worker over a soc
 
 Select and set up this source when you [set up a pipeline][1]. The information below is for the source settings in the pipeline UI.
 
-{{% observability_pipelines/source_settings/socket %}}
+1. In the **Mode** dropdown menu, select the socket type to use.
+1. In the **Framing** dropdown menu, select how to delimit the stream of events.
+    <table>
+        <colgroup>
+            <col style="width:40%">
+            <col style="width:60%">
+        </colgroup>
+        <thead>
+            <tr>
+                <th>FRAMING METHOD</th>
+                <th>DESCRIPTION</th>
+            </tr>
+        </thead>
+        <tr>
+            <td><code>newline_delimited</code></td>
+            <td>Byte frames are delimited by a newline character.</td>
+        </tr>
+        <tr>
+            <td><code>bytes</code></td>
+            <td>Byte frames are passed through as-is according to the underlying I/O boundaries (for example, split between messages or stream segments).</td>
+        </tr>
+        <tr>
+            <td><code>character_delimited</code></td>
+            <td>Byte frames are delimited by a chosen character.</td>
+        </tr>
+        <tr>
+            <td><code>chunked_gelf</code></td>
+            <td>Byte frames are chunked GELF messages.</td>
+        </tr>
+        <tr>
+            <td><code>octet_counting</code></td>
+            <td>Byte frames are delimited according to the octet counting format.</td>
+        </tr>
+    </table>
 
 ## Set the environment variables
 
diff --git a/content/en/observability_pipelines/sources/splunk_hec.md b/content/en/observability_pipelines/sources/splunk_hec.md
@@ -15,7 +15,10 @@ Use Observability Pipelines' Splunk HTTP Event Collector (HEC) source to receive
 
 Select and set up this source when you [set up a pipeline][1]. The information below is for the source settings in the pipeline UI.
 
-{{% observability_pipelines/source_settings/splunk_hec %}}
+Optionally, toggle the switch to enable TLS. If you enable TLS, the following certificate and key files are required.<br>**Note**: All file paths are made relative to the configuration data directory, which is `/var/lib/observability-pipelines-worker/config/` by default. See [Advanced Worker Configurations][5] for more information. The file must be owned by the `observability-pipelines-worker group` and `observability-pipelines-worker` user, or at least readable by the group or user.
+- `Server Certificate Path`: The path to the certificate file that has been signed by your Certificate Authority (CA) Root File in DER or PEM (X.509).
+- `CA Certificate Path`: The path to the certificate file that is your Certificate Authority (CA) Root File in DER or PEM (X.509).
+- `Private Key Path`: The path to the `.key` private key file that belongs to your Server Certificate Path in DER or PEM (PKCS#8) format.
 
 ## Set the environment variables
 
@@ -51,3 +54,4 @@ To send logs from the Splunk Distribution of the OpenTelemetry Collector:
 [2]: https://help.splunk.com/en/splunk-observability-cloud/manage-data/splunk-distribution-of-the-opentelemetry-collector/get-started-with-the-splunk-distribution-of-the-opentelemetry-collector/collector-for-kubernetes
 [3]: https://help.splunk.com/en/splunk-observability-cloud/manage-data/splunk-distribution-of-the-opentelemetry-collector/get-started-with-the-splunk-distribution-of-the-opentelemetry-collector/collector-for-linux
 [4]: /observability_pipelines/configuration/set_up_pipelines
+[5]: /observability_pipelines/configuration/install_the_worker/advanced_worker_configurations/
diff --git a/content/en/observability_pipelines/sources/splunk_tcp.md b/content/en/observability_pipelines/sources/splunk_tcp.md
diff --git a/content/en/observability_pipelines/sources/sumo_logic.md b/content/en/observability_pipelines/sources/sumo_logic.md
diff --git a/content/en/observability_pipelines/sources/syslog.md b/content/en/observability_pipelines/sources/syslog.md
