[kafka-extension] Tracing for Knative Eventing doesn't work

[sotatek-hungpham5]

Describe the bug
I tried to configure tracing for Kevent, include kafka-extension
The config is:

apiVersion: operator.knative.dev/v1beta1
kind: KnativeEventing
metadata:
  name: knative-eventing
  namespace: knative-eventing
spec:
  config-tracing:
          backend: "zipkin"
          # zipkin-endpoint: "http://zipkin.monitoring.svc.cluster.local:9411/api/v2/spans"
          zipkin-endpoint: "http://jaeger-collector.monitoring.svc.cluster.local:9411/api/v2/spans"
          sample-rate: "0.1"
          debug: "false"

Note that it's the same with Kserving
I've tested with imc broker and it worked well but cannot see anything related to kafka-extension in trace data, for both trace and metrics.

Expected behavior
Can see full trace of eventing application including Kafka extension.

Knative release version
1.15

[pierDipi]

@sotatek-hungpham5 what tracing headers format are you using B3 or W3C ?

did you restart the Kafka data plane components after updating the config-tracing in knative-eventing namespace ?

[sotatek-hungpham]

Hi @pierDipi ,
Not sure B3 or W3C, how can i check ?
btw I restart all of them(recreate pods), in addition I see the log like "configmap config-tracing registered" in the kafka-* pods

[remeq]

@pierDipi In the screenshot below you can see that the kafka broker pod was loaded with W3C format:

[sotatek-hungpham]

@pierDipi Could you look a bit here ?

[pierDipi]

it seems the configuration is registered correctly, note that sample rate of 0.1 means that only 10% of the requests will be recorded.

For debugging purposes, can you try increasing sample-rate to 1 and enable debugging debug: "true" ?

[sotatek-hungpham5]

Yeah, let me try

[sotatek-hungpham5]

Thank you for the suggestion, I tried to increase it but looks like it still doesn't work.
The only thing I see in the Jaeger is broker-ingress.knative-eventing, and the IMC dispatcher

[remeq]

@sotatek-hungpham5 can you please share screenshot similar to this: #4190 (comment) but with debug enabled ?

[sotatek-hungpham5]

@remeq Idk why the loggingDebugEnabled still false, but I checked all the config applied, the value is true