Monitor/Analyse Pulsar without external instrumentation (eBPF)

[hpvd]

For Kafka there are some tools which make monitoring and analyzing without external instrumentation and without code modification possible.
They all leverage the possibilities of the extended Berkley Paket Filter (eBPF) which collects data directly in the kernel.

Some tools and demonstrations what's possible:

• https://docs.px.dev/tutorials/pixie-101/kafka-monitoring/
• https://www.groundcover.com/blog/monitoring-kafka-metrics
• https://pt.slideshare.net/HostedbyConfluent/monitoring-kafka-without-instrumentation-using-ebpf-with-antn-rodrguez-kafka-summit-london-2022

It's the same technology uses by Cilium to secure e.g. Kafka see: #20650

It would be fantastic to make these things possible for Pulsar too (from dev ad operation perspective)
and it would rise Pulsar to a new dimension of enterprise grade.

The question is:
how could we make it possible to access these world of eBPF?
Are there already any efforts?

[hpvd]

just an overview of the growing number of tools utilizing eBPF:
https://ebpf.io/applications/

[hpvd]

Since eBPF is very low overhead, this may somehow be of interest/ related to / utilized by:
PIP-264: Enhanced OTel-based metric system #20197

edit:
since discussions, seem not to show up in issue threat (like issues with mentions do)
-> maybe fyi @asafm ?

[asafm]

I'm not an expert on eBPF.
From what I gather, it's a kernel feature that allows you to implement a filtering mechanism outside your software.

It makes perfect sense for security tools, as they wish to implement a policy at the lowest network level possible yet have access to parse the data correctly. They want to be agnostic of the Pulsar version or any software - install this once on every machine and let it apply the policy.

I don't see how it has anything to do with PIP-264.

[hpvd]

yes you are right, it's a great tool for security and also network routing etc.

But in addition, e.g. for kafka cluster it can also be a metrics source for:

• topology (which topic has which consumers, producers)
• topic throughput bytes produced/consumed
• latency (50/90/99)
• CPU/RAM usage
• network throughput
• disk write throuput
• ...

everything with very low overhead and no modifications

Demo eBPF via Pixie used for this:
https://www.youtube.com/watch?v=42o5fURGXqI

[hpvd]

just to further clarify and detail:

BPF started with the use case of filtering network packets, but has since extended beyond that to cover a range of system calls. This makes eBPF a very interesting technology for observability use cases, as it offers a path for extracting telemetry data without modifying the application code. With that, eBPF offers a new way of achieving auto-instrumentation.

eBPF can also work across different types of traffic, which serves the goal of unified observability. For example, you may use eBPF to collect full body request traces, database queries, HTTP requests or gRPC streams.

You can also use eBPF to collect system metrics about resource utilization such as CPU usage or bytes sent, which can serve to calculate statistics, as well as profiling data to understand how many resources each function consumes. This sort of hardware or system information is much harder to access when instrumenting with agents or service mesh, which gives eBPF a clear advantage for these use cases. Another advantage of running in the kernel is that eBPF can handle encrypted traffic.

source: https://logz.io/blog/ebpf-auto-instrumentation-pixie-kubernetes-observability/

[asafm]

Can you clarify what you are proposing to be done in Pulsar?

[hpvd]

this looks like a detailed guide how eBPF usage could be implemented from scratch:
https://www.datadoghq.com/blog/ebpf-guide/

[hpvd]

this shows what pixie does for kafka
https://docs.px.dev/tutorials/pixie-101/kafka-monitoring/

and this are the scripts used:
https://github.com/pixie-io/pixie/tree/main/src/pxl_scripts

[hpvd]

overview and resources for eBPF development:
https://ebpf.io/what-is-ebpf/#development-toolchains