OTLP: Error when outputting OTLP in 3.2.7 (and 3.2.8)

[nortenorte]

Bug Report

Describe the bug
Error when outputing OTLP version 3.2.7 (and 3.2.8). Error happen on metrics, logs and traces.

Error have never happen on older versions, for example 3.2.6 works with exactly same test setup.

[error] [engine] chunk '422061-1741761815.979790809.flb' cannot be retried: task_id=0, input=opentelemetry.0 > output=opentelemetry.0

Se more details below.

To Reproduce

Build fluent-bit with the following defines:
cmake .. -DFLB_SIGNV4=Off -DFLB_AWS=Off -DFLB_FILTER_AWS=Off -DFLB_OUT_S3=Off -DFLB_OUT_KINESIS_FIREHOSE=Off -DFLB_OUT_KINESIS_STREAMS=Off -DFLB_OUT_CLOUDWATCH_LOGS=Off -DFLB_OUT_BIGQUERY=Off

1. Start opentelemetry-collector. Official https://github.com/open-telemetry/opentelemetry-collector
   Installation according to: https://opentelemetry.io/docs/collector/installation/

Start:
bin> ./otelcorecol_linux_amd64 --config otelconfig.yaml

otelconfig.yaml

receivers:
  otlp:
    protocols:
      http:
        endpoint: 127.0.0.1:4319
 
processors:
  batch:
  memory_limiter:
    # 75% of maximum memory up to 2G
    limit_mib: 1536
    # 25% of limit up to 2G
    spike_limit_mib: 512
    check_interval: 5s
 
exporters:
  debug:
    verbosity: detailed
 
service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [debug]
    metrics:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [debug]
    logs:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [debug]

2. Start Fluent-bit.
   ./fluent-bit -c ~/fluent-bit.yaml

fluent-bit.yaml

service:
  log_level: trace 

pipeline:
  inputs:
    - name: opentelemetry
      tag: input_open
      listen: 127.0.0.1
      port: 4318
      successful_response_code: 200
  outputs:
    - name: opentelemetry
      host: 127.0.0.1
      port: 4319
      match: '*' 

3. Send a otlp message.

Install protocurl: https://github.com/qaware/protocurl

Use protocurl to send a standard otlp metric message. (https://github.com/open-telemetry/opentelemetry-proto)

opentelemetry-proto> protocurl \
-i opentelemetry.proto.collector.logs.v1.ExportLogsServiceRequest \
-o opentelemetry.proto.collector.logs.v1.ExportLogsServiceResponse \
-u http://localhost:4318/v1/logs \
-d @examples/logs.json \
-I .

4. Check result from otel-collector:

[2025/03/12 07:43:36] [debug] [task] created task=0x7e5244033dd0 id=0 OK
[2025/03/12 07:43:36] [debug] [upstream] KA connection #36 to 127.0.0.1:4319 is connected
[2025/03/12 07:43:36] [ warn] [output:opentelemetry:opentelemetry.0] error performing HTTP request, remote host=127.0.0.1:4319 connection error
[2025/03/12 07:43:36] [debug] [upstream] KA connection #36 to 127.0.0.1:4319 is now available
[2025/03/12 07:43:36] [debug] [upstream] KA connection #36 to 127.0.0.1:4319 has been disconnected by the remote service
[2025/03/12 07:43:36] [debug] [out flush] cb_destroy coro_id=0
[2025/03/12 07:43:36] [debug] [retry] new retry created for task_id=0 attempts=1
[2025/03/12 07:43:36] [ warn] [engine] failed to flush chunk '422061-1741761815.979790809.flb', retry in 9 seconds: task_id=0, input=opentelemetry.0 > output=opentelemetry.0 (out_id=0)
[2025/03/12 07:43:45] [ warn] [output:opentelemetry:opentelemetry.0] error performing HTTP request, remote host=127.0.0.1:4319 connection error
[2025/03/12 07:43:45] [debug] [out flush] cb_destroy coro_id=1
[2025/03/12 07:43:45] [debug] [task] task_id=0 reached retry-attempts limit 1/1
[2025/03/12 07:43:45] [error] [engine] chunk '422061-1741761815.979790809.flb' cannot be retried: task_id=0, input=opentelemetry.0 > output=opentelemetry.0
[2025/03/12 07:43:45] [debug] [task] destroy task=0x7e5244033dd0 (task_id=0)

Expected behavior

Result from v3.2.6:

[2025/03/12 07:47:58] [debug] [task] created task=0x756318033e90 id=0 OK
[2025/03/12 07:47:58] [debug] [upstream] KA connection #36 to 127.0.0.1:4319 is connected
[2025/03/12 07:47:58] [ info] [output:opentelemetry:opentelemetry.0] 127.0.0.1:4319, HTTP status=200
[2025/03/12 07:47:58] [debug] [upstream] KA connection #36 to 127.0.0.1:4319 is now available
[2025/03/12 07:47:58] [debug] [out flush] cb_destroy coro_id=0
[2025/03/12 07:47:58] [debug] [task] destroy task=0x756318033e90 (task_id=0)

Screenshots

Your Environment

• Version used: 3.2.7 (and 3.2.8)
• Configuration: See above configuration
• Environment name and version (e.g. Kubernetes? What version?):
• Server type and version: N/A
• Operating System and version: Ubuntu 24.04 (x86), target system is embedded Linux on arm64. Same problem on both OS and hardware.
• Filters and plugins: See above configuration

Additional context
Can not use v3.2.7 and v3.2.8 as we can not get out any logs.

[patrick-stephens]

Does it work with the official builds/containers?

[nortenorte]

@patrick-stephens Does not work using the containers (works on 3.2.6).

Recreate using container:

fluent-bit_stdout_otlp.yaml

service:
    flush: 1
    log_level: trace
 
pipeline:
    inputs:
        - name: opentelemetry
          listen: 0.0.0.0
          port: 4318
          successful_response_code: 200
    outputs:
        - name: stdout
          match: '*'
        - name: opentelemetry
          match: '*'
          host: 127.0.0.1
          port: 4319

Create a directory and put fluent-bit_stdout_otlp.yaml in that directory then run the following command.
docker run -v $(pwd):$(pwd) -w $(pwd) --rm -it -p 4318:4318 --network host fluent/fluent-bit:3.2.8 /fluent-bit/bin/fluent-bit -c fluent-bit_stdout_otlp.yaml

Start opentelemetry-collector. Official https://github.com/open-telemetry/opentelemetry-collector
Installation according to: https://opentelemetry.io/docs/collector/installation/

Start:
bin> ./otelcorecol_linux_amd64 --config otelconfig.yaml

otelconfig.yaml

receivers:
  otlp:
    protocols:
      http:
        endpoint: 127.0.0.1:4319
  
processors:
  batch:
  memory_limiter:
    # 75% of maximum memory up to 2G
    limit_mib: 1536
    # 25% of limit up to 2G
    spike_limit_mib: 512
    check_interval: 5s
  
exporters:
  debug:
    verbosity: detailed
  
service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [debug]
    metrics:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [debug]
    logs:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [debug] 

Send a simple curl command:
curl --header "Content-Type: application/json" --request POST --data '{"resourceLogs":[{"resource":{},"scopeLogs":[{"scope":{},"logRecords":[{"timeUnixNano":"1660296023390371588","body":{"stringValue":"{\"message\":\"dummy\"}"},"traceId":"","spanId":""}]}]}]}' http://0.0.0.0:4318/v1/logs

Output:

[2025/03/12 15:01:05] [debug] [input:opentelemetry:opentelemetry.0] attributes missing
[2025/03/12 15:01:06] [debug] [task] created task=0x7bde54a36640 id=0 OK
[2025/03/12 15:01:06] [debug] [output:stdout:stdout.0] task_id=0 assigned to thread #0
[0] v1_logs: [[-1.000000000, {"schema"=>"otlp", "resource_id"=>0, "scope_id"=>0}], {"resource"=>{}, "scope"=>{}}]
[1] v1_logs: [[1660296023.1698112429, {"otlp"=>{"trace_id"=>"", "span_id"=>""}}], {"log"=>"{"message":"dummy"}"}]
[2] v1_logs: [[-2.000000000, {}], {}]
[2025/03/12 15:01:06] [debug] [out flush] cb_destroy coro_id=0
[2025/03/12 15:01:06] [debug] [upstream] KA connection #50 to 127.0.0.1:4319 is connected
[2025/03/12 15:01:06] [ warn] [output:opentelemetry:opentelemetry.1] error performing HTTP request, remote host=127.0.0.1:4319 connection error
[2025/03/12 15:01:06] [debug] [out flush] cb_destroy coro_id=0
[2025/03/12 15:01:06] [debug] [retry] new retry created for task_id=0 attempts=1
[2025/03/12 15:01:06] [ warn] [engine] failed to flush chunk '1-1741791665.723364920.flb', retry in 6 seconds: task_id=0, input=opentelemetry.0 > output=opentelemetry.1 (out_id=1)
[2025/03/12 15:01:12] [ warn] [output:opentelemetry:opentelemetry.1] error performing HTTP request, remote host=127.0.0.1:4319 connection error
[2025/03/12 15:01:12] [debug] [out flush] cb_destroy coro_id=1
[2025/03/12 15:01:12] [debug] [task] task_id=0 reached retry-attempts limit 1/1
[2025/03/12 15:01:12] [error] [engine] chunk '1-1741791665.723364920.flb' cannot be retried: task_id=0, input=opentelemetry.0 > output=opentelemetry.1
[2025/03/12 15:01:12] [debug] [task] destroy task=0x7bde54a36640 (task_id=0)

It's easy to switch between 3.2.6 and 3.2.8 and see the different result (success vs failure).

[edsiper]

wondering if this PR is the fix: #10083

[nortenorte]

@edsiper Unfortunately no.

I checked out the commit below on branch 3.2.

commit 554c0cd (HEAD -> 3.2, origin/3.2)
Author: Leonardo Alminana leonardo.alminana@chronosphere.io
Date: Fri Mar 14 11:42:24 2025 +0100

out_opentelemetry: restored old group meta record processing mechanism

Signed-off-by: Leonardo Alminana <leonardo.alminana@chronosphere.io>

Still same issue:

[2025/03/17 08:38:27] [debug] [task] created task=0x700340035a70 id=0 OK
[2025/03/17 08:38:27] [debug] [upstream] KA connection #36 to 127.0.0.1:4319 is connected
[2025/03/17 08:38:27] [ warn] [output:opentelemetry:opentelemetry.0] error performing HTTP request, remote host=127.0.0.1:4319 connection error
[2025/03/17 08:38:27] [debug] [out flush] cb_destroy coro_id=0
[2025/03/17 08:38:27] [debug] [retry] new retry created for task_id=0 attempts=1
[2025/03/17 08:38:27] [ warn] [engine] failed to flush chunk '121723-1742197106.492297243.flb', retry in 7 seconds: task_id=0, input=opentelemetry.0 > output=opentelemetry.0 (out_id=0)
[2025/03/17 08:38:34] [ warn] [output:opentelemetry:opentelemetry.0] error performing HTTP request, remote host=127.0.0.1:4319 connection error
[2025/03/17 08:38:34] [debug] [out flush] cb_destroy coro_id=1
[2025/03/17 08:38:34] [debug] [task] task_id=0 reached retry-attempts limit 1/1
[2025/03/17 08:38:34] [error] [engine] chunk '121723-1742197106.492297243.flb' cannot be retried: task_id=0, input=opentelemetry.0 > output=opentelemetry.0
[2025/03/17 08:38:34] [debug] [task] destroy task=0x700340035a70 (task_id=0)

[edsiper]

it's not a bug, Fluent Bit OpenTelemetry output enables http2 by default, and it seems the collector drops the connection.

Setting http2: off will make it work.

Considering turning that as a new default

[edsiper]

#10089

[Sabreclaw]

it's not a bug, Fluent Bit OpenTelemetry output enables http2 by default, and it seems the collector drops the connection.

Setting http2: off will make it work.

Considering turning that as a new default

This worked for me, thanks