Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

processor_labels: add support for record accessor in values #9841

Merged
merged 3 commits into from
Mar 28, 2025

Conversation

edsiper
Copy link
Member

@edsiper edsiper commented Jan 16, 2025

Label processors allow to manipulate the labels from metrics, this PR extends the actions insert, update and upsert to support a record accessor pattern that helps to use the $TAG variable that can be populated dinamically, e.g:

service:
  flush: 1

pipeline:
  inputs:
    - name: event_type
      type: metrics
      tag: my_test

      processors:
        metrics:
          - name: labels
            upsert: tag $TAG

  outputs:
    - name: stdout
      match: '*'

Fluent Bit output:

______ _                  _    ______ _ _             ___  _____ 
|  ___| |                | |   | ___ (_) |           /   ||  _  |
| |_  | |_   _  ___ _ __ | |_  | |_/ /_| |_  __   __/ /| || |/' |
|  _| | | | | |/ _ \ '_ \| __| | ___ \ | __| \ \ / / /_| ||  /| |
| |   | | |_| |  __/ | | | |_  | |_/ / | |_   \ V /\___  |\ |_/ /
\_|   |_|\__,_|\___|_| |_|\__| \____/|_|\__|   \_/     |_(_)___/ 


[2025/01/15 22:16:42] [ info] [fluent bit] version=4.0.0, commit=27734ee3e6, pid=1586419
[2025/01/15 22:16:42] [ info] [storage] ver=1.5.2, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2025/01/15 22:16:42] [ info] [simd    ] disabled
[2025/01/15 22:16:42] [ info] [cmetrics] version=0.9.9
[2025/01/15 22:16:42] [ info] [ctraces ] version=0.5.7
[2025/01/15 22:16:42] [ info] [input:event_type:event_type.0] initializing
[2025/01/15 22:16:42] [ info] [input:event_type:event_type.0] storage_strategy='memory' (memory only)
[2025/01/15 22:16:42] [ info] [output:stdout:stdout.0] worker #0 started
[2025/01/15 22:16:42] [ info] [input:event_type:event_type.0] thread instance initialized
[2025/01/15 22:16:42] [ info] [sp] stream processor started
[2025/01/15 22:16:43] [ info] [input:event_type:event_type.0] [OK] collector_time
2025-01-16T04:16:43.837152867Z kubernetes_network_load_counter{tag="my_test"} = 3
2025-01-16T04:16:43.837152867Z kubernetes_network_load_counter{tag="my_test",hostname="localhost",app="cmetrics"} = 1
2025-01-16T04:16:43.837152867Z kubernetes_network_load_counter{tag="my_test",hostname="localhost",app="test"} = 12.15
2025-01-16T04:16:43.837152867Z kubernetes_network_load_gauge{tag="my_test"} = 1
2025-01-16T04:16:43.837152867Z k8s_disk_load_summary{tag="my_test"} = { quantiles = { 0.1=1.1, 0.2=2.2, 0.3=3.3, 0.4=4.4, 0.5=5.5 }, sum=51.6129, count=10 }
2025-01-16T04:16:43.837152867Z k8s_disk_load_summary{tag="my_test",my_label="my_label"} = { quantiles = { 0.1=0.1, 0.2=0.2, 0.3=0.3, 0.4=0.4, 0.5=0.5 }, sum=51.6129, count=5 }
2025-01-16T04:16:43.837152867Z k8s_disk_load_summary{tag="my_test",my_label="my_val"} = { quantiles = { 0.1=11.11, 0.2=0, 0.3=33.33, 0.4=44.44, 0.5=55.55 }, sum=51.6129, count=10 }
2025-01-16T04:16:43.837152867Z k8s_network_load_histogram{tag="my_test"} = { buckets = { 0.05=2, 5=3, 10=4, +Inf=0 }, sum=1013.02, count=5 }
2025-01-16T04:16:43.837152867Z k8s_network_load_histogram{tag="my_test",my_label="my_val"} = { buckets = { 0.05=2, 5=3, 10=4, +Inf=0 }, sum=1013.02, count=5 }

This PR also adds a new unit test for the processor.


Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

@edsiper edsiper force-pushed the labels-record-accessor branch from ebc1d3a to daad13e Compare January 17, 2025 03:39
@edsiper edsiper changed the title processor_labels: add support for record accessor in upsert action processor_labels: add support for record accessor in values Jan 17, 2025
}

TEST_LIST = {
#ifdef FLB_HAVE_METRICS
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is just my curiosity but do we need to use this if def for test cases? This is because when disabling FLB_HAVE_METRICS, the defined test cases are not effective under this condition.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hmm maybe we don't need it... I don't think metrics are optional anymore

@edsiper edsiper merged commit b0f3a63 into master Mar 28, 2025
88 checks passed
@edsiper edsiper deleted the labels-record-accessor branch March 28, 2025 23:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
docs-required ok-package-test Run PR packaging tests
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants