Getting incorrect output from go output plugin when multiple filters match the same tag

[rohit-bandlamudi-nr]

Bug Report

Describe the bug
When multiple filters of type record_modifier are used, all of them matching the same tag, any key value pairs that are added using "Record" and have the same key across multiple filters , are interpreted incorrectly by any Go output plugin.

For example: If the records a: abc and a: xy, are added through the first and second filter respectively (as seen in config below),
the go output plugin is receiving the record a: xyc, instead of receiving both a: abc and a: xy.

To Reproduce
Use the following config in fluent-bit.conf

[SERVICE]
    Plugins_File plugins.conf
[INPUT]
    Name  random
    Samples  -1
    Interval_Sec  1
    Interval_NSec 0
    Tag test
[FILTER]
    Name  record_modifier
    Match test
    Record a abc
    Record fb.input tail
    Record logtype applicationLogs
    Record filter 1
    Record hello world
[FILTER]
    Name  record_modifier
    Match test
    Record a xy
    Record fb.input tail
    Record logtype linux_syslog
    Record filter 2
    Record test nmm
# Output plugin written in C to print to the standard output
[OUTPUT]
    Name  stdout
    Match  *
# Output plugin written in go to print to the standard output
[OUTPUT]
    Name  gstdout
    Match  *

Expected behavior
The final output should contain all the 10 records that are added by both the filters. The plugin stdout (out-of-the-box C plugin) is behaving in the expected manner.

From the screenshot below, we can see the outputs from both stdout and gstdout (golang output plugin)

Decoded log message: {"rand_value"=>12458609389348480718, "a"=>"abc", "fb.input"=>"tail", "logtype"=>"applicationLogs", "filter"=>"1", "hello"=>"world", "a"=>"xy", "fb.input"=>"tail", "logtype"=>"linux_syslog", "filter"=>"2", "test"=>"nmm"}

[0] test: [2024-09-16 16:51:13.914998 +0530 IST, {"rand_value": 12458609389348480718, "fb.input": [116 97 105 108], "hello": [119 111 114 108 100], "a": [120 121 99], "logtype": [108 105 110 117 120 95 115 121 115 108 111 103 111 103 115], "filter": [50], "test": [110 109 109], }

The first line in the snippet above shows the expected output (emitted by stdout plugin). The second line shows the output emitted by the golang output plugin.

Notice that the first log has all the records from both the filters. The second log also do have all the records, but the records with the same key are malformed. We got a: [120 121 99] which translates to a: "xyc" ( first two letters of "abc" got replaced with "xy").Also, notice that the record with key "filter" has been overwritten. This behaviour is observed by any output plugin that is written in go.

Link to output plugin that is used to reproduce the issue:
https://github.com/fluent/fluent-bit-go/tree/master/examples/out_gstdout

Screenshots

Your Environment

• Version used: 3.1

[patrick-stephens]

Can you explain how you've made the plugin? It's going to be super hard to debug unknown code particularly if it is a dependency or other issue.

Is it this one? https://github.com/chronosphereio/calyptia-plugin/tree/main/examples/out_gstdout

[rohit-bandlamudi-nr]

@patrick-stephens , It is this one:
https://github.com/fluent/fluent-bit-go/tree/master/examples/out_gstdout

Also, I have validated another example plugin from the same repo that I attached above by adding a debug statement and found out that the decoded payload to the plugin is itself malformed.

[patrick-stephens]

I bet the dependencies are wildly out of date: are you updating them before build?

[rohit-bandlamudi-nr]

@patrick-stephens , I directly built the plugin using

go build -buildmode=c-shared -o out_gstdout.so .

Did not try updating any dependency

[patrick-stephens]

I would try that then, it looks like the code has not changed for 2 years...

[rohit-bandlamudi-nr]

@patrick-stephens , I tried updating the dependencies, still facing the same issue.