Skip to content

Commit 6e27d2f

Browse files
kforeverisbackkforeverisback
authored
out_azure_logs_ingestion: azure logs ingestion documentation and gitbook image asset (#1076)
* pipeline: outputs: azure_logs_ingestion: added documentation for Azure Logs Ingestion plugin Signed-off-by: Kushal Azim Ekram <[email protected]> * .gitbook: access: azure-logs-ingestion-overview: added supporting image for Azure Logs Ingestion document Signed-off-by: Kushal Azim Ekram <[email protected]> * pipeline: outputs: azure_logs_ingestion: modified doc Signed-off-by: Kushal Azim Ekram <[email protected]> * pipeline: outputs: azure_logs_ingestion: Fixed typos Signed-off-by: Kushal Azim Ekram <[email protected]> --------- Signed-off-by: Kushal Azim Ekram <[email protected]>
1 parent 214c564 commit 6e27d2f

File tree

2 files changed

+91
-0
lines changed

2 files changed

+91
-0
lines changed

‎.gitbook/assets/azure-logs-ingestion-overview.png

21.4 KB
Loading

‎pipeline/outputs/azure_logs_ingestion.md

Lines changed: 91 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,91 @@
1+
---
2+
description: 'Send logs to Azure Log Analytics using Logs Ingestion API with DCE and DCR'
3+
---
4+
5+
# Azure Logs Ingestion API
6+
7+
![](../../.gitbook/assets/image%20%287%29.png)
8+
9+
Azure Logs Ingestion plugin allows you ingest your records using [Logs Ingestion API in Azure Monitor](https://learn.microsoft.com/en-us/azure/azure-monitor/logs/logs-ingestion-api-overview) to supported [Azure tables](https://learn.microsoft.com/en-us/azure/azure-monitor/logs/logs-ingestion-api-overview#supported-tables) or to [custom tables](https://learn.microsoft.com/en-us/azure/azure-monitor/logs/create-custom-table#create-a-custom-table) that you create.
10+
11+
The Logs ingestion API requires the following components:
12+
13+
- A Data Collection Endpoint (DCE)
14+
- A Data Collection Rule (DCR) and
15+
- A Log Analytics Workspace
16+
17+
> Note: According to [this document](https://github.com/MicrosoftDocs/azure-docs/blob/main/articles/azure-monitor/logs/logs-ingestion-api-overview.md#components), all resources should be in the same region.
18+
19+
To visualize basic Logs Ingestion operation, see the following image:
20+
![](../../.gitbook/assets/azure-logs-ingestion-overview.png)
21+
22+
To get more details about how to setup these components, please refer to the following documentations:
23+
24+
- [Azure Logs Ingestion API](https://docs.microsoft.com/en-us/azure/log-analytics/)
25+
- [Send data to Azure Monitor Logs with Logs ingestion API (setup DCE, DCR and Log Analytics)](https://learn.microsoft.com/en-us/azure/azure-monitor/logs/tutorial-logs-ingestion-portal)
26+
27+
## Configuration Parameters
28+
29+
| Key | Description | Default |
30+
| :------------ | :------------------------- | :------ |
31+
| tenant\_id | _Required_ - The tenant ID of the AAD application. ||
32+
| client\_id | _Required_ - The client ID of the AAD application. ||
33+
| client\_secret| _Required_ - The client secret of the AAD application ([App Secret](https://docs.microsoft.com/en-us/azure/active-directory/develop/howto-create-service-principal-portal#option-2-create-a-new-application-secret)). ||
34+
| dce\_url | _Required_ - Data Collection Endpoint(DCE) URL. ||
35+
| dcr\_id | _Required_ - Data Collection Rule (DCR) immutable ID (see [this document](https://learn.microsoft.com/en-us/azure/azure-monitor/logs/tutorial-logs-ingestion-portal#collect-information-from-the-dcr) to collect the immutable id) ||
36+
| table\_name | _Required_ - The name of the custom log table (include the `_CL` suffix as well if applicable) ||
37+
| time\_key | _Optional_ - Specify the key name where the timestamp will be stored. | `@timestamp` |
38+
| time\_generated | _Optional_ - If enabled, will generate a timestamp and append it to JSON. The key name is set by the 'time_key' parameter. | `true` |
39+
| compress | _Optional_ - Enable HTTP payload gzip compression. | `true` |
40+
41+
## Getting Started
42+
43+
To send records into an Azure Log Analytics using Logs Ingestion API the following resources needs to be created:
44+
45+
- A Data Collection Endpoint (DCE) for ingestion
46+
- A Data Collection Rule (DCR) for data transformation
47+
- Either an [Azure tables](https://learn.microsoft.com/en-us/azure/azure-monitor/logs/logs-ingestion-api-overview#supported-tables) or [custom tables](https://learn.microsoft.com/en-us/azure/azure-monitor/logs/create-custom-table#create-a-custom-table)
48+
- An app registration with client secrets (for DCR access).
49+
50+
You can follow [this guideline](https://learn.microsoft.com/en-us/azure/azure-monitor/logs/tutorial-logs-ingestion-portal) to setup the DCE, DCR, app registration and a custom table.
51+
52+
### Configuration File
53+
54+
Use this configuration to quickly get started:
55+
56+
```ini
57+
[INPUT]
58+
Name tail
59+
Path /path/to/your/sample.log
60+
Tag sample
61+
Key RawData
62+
# Or use other plugins Plugin
63+
# [INPUT]
64+
# Name cpu
65+
# Tag sample
66+
67+
[FILTER]
68+
Name modify
69+
Match sample
70+
# Add a json key named "Application":"fb_log"
71+
Add Application fb_log
72+
73+
# Enable this section to see your json-log format
74+
#[OUTPUT]
75+
# Name stdout
76+
# Match *
77+
[OUTPUT]
78+
Name azure_logs_ingestion
79+
Match sample
80+
client_id XXXXXXXX-xxxx-yyyy-zzzz-xxxxyyyyzzzzxyzz
81+
client_secret some.secret.xxxzzz
82+
tenant_id XXXXXXXX-xxxx-yyyy-zzzz-xxxxyyyyzzzzxyzz
83+
dce_url https://log-analytics-dce-XXXX.region-code.ingest.monitor.azure.com
84+
dcr_id dcr-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
85+
table_name ladcr_CL
86+
time_generated true
87+
time_key Time
88+
Compress true
89+
```
90+
91+
Setup your DCR transformation accordingly based on the json output from fluent-bit's pipeline (input, parser, filter, output).

0 commit comments

Comments
 (0)