Skip to content

Commit cd4656f

Browse files
simranquirkysimranquirky
authored
added integration page for GCP Logs (#163)
* added integration page for GCP Logs * added Integration page for Google Cloud Run * added Integration page for Weblogic server * Integration page for NATS * add/integration page for Airflow * add/integration page for Cribl * add/integration page for Microsoft 365 * fixed 404s on user-guide section * add integration page for Mulesoft logs * removed some org_names --------- Co-authored-by: simranquirky <simranquirky>
1 parent cf96cc8 commit cd4656f

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

52 files changed

+1350
-3
lines changed

‎docs/integration/.pages

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,11 +5,16 @@ nav:
55
- AWS: aws
66
- Cloudflare: cloudflare.md
77
- Database: database
8+
- GCP: gcp
89
- Servers: servers
910
- DevOps : devops
1011
- Linux: linux.md
1112
- Windows: windows.md
1213
- Vercel : vercel.md
1314
- Heroku: heroku.md
1415
- Message Queues: message-brokers
16+
- Airflow: airflow.md
17+
- Cribl: cribl.md
18+
- Security: security
19+
- Mulesoft: mulesoft.md
1520

‎docs/integration/airflow.md

Lines changed: 158 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,158 @@
1+
---
2+
title: Apache Airflow Integration Guide
3+
description: Collect and monitor Apache Airflow logs and metrics with OpenTelemetry Collector and visualize them in OpenObserve.
4+
---
5+
6+
# Integration with Apache Airflow
7+
8+
This guide explains how to monitor **Apache Airflow** using the [OpenTelemetry Collector](https://opentelemetry.io/docs/collector/) (`otelcol`) and export logs, metrics, and traces to **OpenObserve** for visualization.
9+
10+
## Overview
11+
12+
Apache Airflow is a **workflow automation and orchestration tool** widely used for ETL pipelines, ML workflows, and data engineering tasks. Monitoring Airflow is critical for ensuring workflow reliability, debugging issues, and tracking system performance.
13+
</br>
14+
15+
With OpenTelemetry and OpenObserve, you gain **real-time observability** into Airflow DAG runs, task execution, scheduler activity, and worker performance.
16+
17+
![Airflow architechture](images/airflow-arch.png)
18+
19+
## Steps to Integrate
20+
21+
??? "Prerequisites"
22+
- OpenObserve account ([Cloud](https://cloud.openobserve.ai/web/) or [Self-Hosted](../../getting-started.md))
23+
- Apache Airflow installed and running
24+
- Basic understanding of Airflow configs (`airflow.cfg`)
25+
- OpenTelemetry Collector installed
26+
27+
??? "Step 1: Configure Airflow for OpenTelemetry"
28+
29+
Edit `airflow.cfg` to enable OTel metrics:
30+
31+
```ini
32+
[metrics]
33+
otel_on = True
34+
otel_host = localhost
35+
otel_port = 4318
36+
```
37+
38+
Restart Airflow services after updating config:
39+
40+
```bash
41+
airflow db migrate
42+
airflow scheduler -D
43+
airflow webserver -D
44+
```
45+
46+
??? "Step 2: Install OpenTelemetry Collector"
47+
48+
1. Download and install the OTel Collector:
49+
```bash
50+
wget https://github.com/open-telemetry/opentelemetry-collector-releases/releases/latest/download/otelcol-linux-amd64
51+
chmod +x otelcol-linux-amd64
52+
sudo mv otelcol-linux-amd64 /usr/local/bin/otelcol
53+
```
54+
55+
2. Verify installation:
56+
```bash
57+
otelcol --version
58+
```
59+
60+
??? "Step 3: Get OpenObserve Endpoint and Token"
61+
62+
1. In OpenObserve: go to **Data Sources → Otel Collector**
63+
2. Copy the **Ingestion URL** and **Access Token**
64+
![Get OpenObserve Ingestion URL and Token](../images/messagebroker/otel-metrics-source.png)
65+
66+
??? "Step 4: Configure OpenTelemetry Collector"
67+
68+
1. Create/edit config file:
69+
```bash
70+
sudo vi /etc/otel-config.yaml
71+
```
72+
73+
2. Add Airflow configuration:
74+
```yaml
75+
receivers:
76+
filelog/std:
77+
include:
78+
- /airflow/logs/*/*.log
79+
- /airflow/logs/scheduler/*/*/*/*.log
80+
start_at: beginning
81+
otlp:
82+
protocols:
83+
grpc:
84+
http:
85+
86+
processors:
87+
batch:
88+
89+
exporters:
90+
otlphttp/openobserve:
91+
endpoint: OPENOBSERVE_ENDPOINT
92+
headers:
93+
Authorization: "OPENOBSERVE_TOKEN"
94+
stream-name: airflow
95+
96+
service:
97+
pipelines:
98+
metrics:
99+
receivers: [otlp]
100+
processors: [batch]
101+
exporters: [otlphttp/openobserve]
102+
logs:
103+
receivers: [filelog/std, otlp]
104+
processors: [batch]
105+
exporters: [otlphttp/openobserve]
106+
traces:
107+
receivers: [otlp]
108+
processors: [batch]
109+
exporters: [otlphttp/openobserve]
110+
```
111+
112+
Replace placeholders with your OpenObserve details:
113+
114+
- `OPENOBSERVE_ENDPOINT` → API endpoint (e.g., `https://api.openobserve.ai`)
115+
- `OPENOBSERVE_TOKEN` → Access token
116+
117+
??? "Step 5: Start OpenTelemetry Collector"
118+
119+
```bash
120+
sudo systemctl start otel-collector
121+
sudo systemctl status otel-collector
122+
journalctl -u otel-collector -f
123+
```
124+
125+
> Check logs to confirm data is being sent to OpenObserve.
126+
127+
??? "Step 6: Visualize Logs in OpenObserve"
128+
129+
1. Go to **Streams → airflow** in OpenObserve to query logs.Airflow logs collected include: DAG execution logs, Scheduler logs, Worker logs and Task execution logs
130+
131+
![Visualize Logs in OpenObserve](images/airflow-logs.png)
132+
133+
134+
!!! tip "Prebuilt Dashboards"
135+
136+
</br>
137+
[Prebuilt Airflow dashboards](https://github.com/openobserve/dashboards/tree/main/Airflow) are available. You can download the JSON file and import it.
138+
139+
## Troubleshooting
140+
141+
- **No Logs in OpenObserve**
142+
143+
- Ensure `filelog` receiver paths match your Airflow log directory.
144+
- Verify Collector service is running.
145+
146+
- **Metrics Not Visible**
147+
148+
- Check `otel_on = True` in `airflow.cfg`.
149+
- Confirm Airflow is sending metrics to `localhost:4318`.
150+
151+
- **Collector Fails to Start**
152+
153+
- Run dry check:
154+
```bash
155+
otelcol --config /etc/otel-config.yaml --dry-run
156+
```
157+
- Fix syntax or missing receivers.
158+

‎docs/integration/cribl.md

Lines changed: 104 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,104 @@
1+
---
2+
title: Cribl Integration Guide
3+
description: Learn how to route, filter, and analyze logs and traces from Cribl into OpenObserve using a webhook destination.
4+
---
5+
6+
# Integration with Cribl
7+
8+
[Cribl Stream](https://cribl.io/) is a data engine designed to manage the flow of observability and security data. It allows you to route, filter, enrich, and transform data before forwarding it to your observability backend.
9+
10+
## Overview
11+
12+
**OpenObserve** is a high-performance, open-source observability platform built for real-time log and trace analytics. By integrating Cribl with OpenObserve, you can optimize data ingestion while reducing costs and improving monitoring.
13+
14+
## Steps to Integrate
15+
16+
??? "Prerequisites"
17+
18+
- Running **Cribl Stream** instance with access to the UI
19+
- OpenObserve account ([Cloud](https://cloud.openobserve.ai/web/) or [Self-Hosted](../../getting-started.md))
20+
21+
??? "Step 1: Configure an Internal Source in Cribl"
22+
23+
First configure an internal Cribl source to generate sample logs.
24+
25+
1. Open **Cribl Stream UI** → navigate to **Worker Group → Routing → QuickConnect**.
26+
![Cribl UI](images/cribl/quick-connect.png)
27+
28+
2. Select **Sources** from the left menu → **Add Source**.
29+
![Add Source](images/cribl/add-source.png)
30+
31+
3. Choose **System → Internal** as the source type, and select **Cribl Internal**.
32+
![Internal Source](images/cribl/cribl-internal.png)
33+
34+
4. Provide a name (e.g., `cribl`) → Save & Start.
35+
![Configured Source](images/cribl/configured-source.png)
36+
37+
At this point, Cribl will generate test logs.
38+
39+
??? "Step 2: Configure a Webhook Destination in Cribl"
40+
41+
1. In the Cribl UI, go to **Destinations** → **Add Destination**.Select **Webhook**.
42+
![Webhook Destination](images/cribl/select-webhook.png)
43+
44+
2. Configure the webhook:
45+
46+
- **Name:** `OpenObserve_Webhook`
47+
- **Webhook URL:**
48+
```
49+
http://<OPENOBSERVE_HOST>/api/default/cribl/_json
50+
```
51+
- **HTTP Method:** `POST`
52+
![Webhook Config](images/cribl/webhook-configuration.png)
53+
- **Authentication:**
54+
- Type: **Basic**
55+
- Username: `O2_USER`
56+
- Password: `O2_PASSWORD`
57+
58+
![Auth Config](images/cribl/webhook-auth.png)
59+
60+
3. Save and activate the destination.
61+
62+
??? "Step 3: Route Source Data to OpenObserve"
63+
64+
1. In Cribl, create a **Route**.
65+
2. Connect the **Internal Source (cribl)** → **Webhook Destination (OpenObserve_Webhook)**.
66+
3. Use **Passthru** for a simple route, then save.
67+
![Route](images/cribl/connection-configuration.png)
68+
69+
> You can test the setup by sending sample logs. A success message indicates that OpenObserve received the data.
70+
![Route Active](images/cribl/test-data.png)
71+
72+
??? "Step 4: Monitor Data in OpenObserve"
73+
74+
- Query Logs
75+
- Go to **Logs → Streams → cribl**. You should see logs ingested from Cribl.
76+
![Logs](images/cribl/query-logs.png)
77+
78+
- Query Traces
79+
- Go to **Traces → Streams → cribl**. You should see traces ingested from Cribl.
80+
![Traces](images/cribl/query-traces.png)
81+
![Traces View](images/cribl/traces-view.png)
82+
83+
## Troubleshooting
84+
85+
??? "No data appearing in OpenObserve"
86+
- Verify the **Route** in Cribl is active and connected to the Webhook destination.
87+
- Double-check the **Webhook URL** format:
88+
```
89+
http://<OPENOBSERVE_HOST>/api/default/cribl/_json
90+
```
91+
- Ensure the correct **stream name** (`cribl`) is set in the URL.
92+
93+
??? "Authentication failures (401 Unauthorized)"
94+
- Confirm the `O2_USER` and `O2_PASSWORD` are correct.
95+
- Check if the user has **ingest permissions** in OpenObserve.
96+
97+
??? "Connection errors / timeouts"
98+
- Make sure the OpenObserve host is reachable from Cribl.
99+
- If using OpenObserve Cloud, ensure your firewall/VPC rules allow outbound connections.
100+
101+
??? "Logs are ingested but not parsed correctly"
102+
- Confirm you’re using the `_json` endpoint in the Webhook URL.
103+
- Check if the incoming data structure matches what OpenObserve expects (JSON payload).
104+

‎docs/integration/gcp/.pages

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
nav:
2+
3+
- GCP Integrations Overview: index.md
4+
- GCP Logs: gcp-logs.md
5+
- Google Cloud Run: cloud-run.md
6+
7+

0 commit comments

Comments
 (0)