Performance degradation when using Linkerd

[nfsp3k]

Hi,

An application I'm using shows a quite low throughput and bad latency when attaching Linkerd proxy as shown in the figure below. (The application can handle only 360qps even though the load generator sends 500qps.)

When running the application WITHOUT Linkerd, it can handle 500qps sufficiently. It can also handle 1,000qps.

I set up the Linkerd proxy by executing the following command.

The application is an open-sourced microservice benchmark called DeathStarBench that consists of 30 microservices.

Linkerd is implemented in the native language Rust and should deliver much better performance than I experienced in my opinion.
I think I'm doing something wrong but I have no sense of how to figure it out...
Could you help me?

• CPU: Intel Xeon E5-2660 v4 @2.0GHz x 2 ( 14 x 2 = 28 physical CPU cores in total. Hyperthreading enabled.)
• Memory: 128GB RAM
• Linkerd: v2.8.1
• kubernetes: v1.17 (single master and single worker node)

[olix0r]

Are you able to try Linkerd 2.9.1? There were a number of significant performance improvements made since the 2.8.1 release.

[nfsp3k]

Thank you for your response, @olix0r.

The application with Linkerd v2.9.1 seems to show better performance than that with v2.8.1.
It can successfully handle around 800qps.

Thank you, @olix0r.
You saved my life.
I think I can use Linkerd in my experiments.

Best,
Byungkwon

[roodkcab]

Hello all, I'd like to continue this discussion with test results for v2.10.0.

I set up a 3 nodes cluster with Kong as Ingress, and a nginx pod as application.

benchmark without linkerd

benchmark with linkerd

as u can see, almost 70% of qps lost when linkerd-proxy being injected, and here is the Flamegraph
https://github.com/roodkcab/linkerd-benchmark/blob/master/out.svg

linkerd-proxy takes up 63% of cpu time and it seems only 8% of cpu time is related with network write and read.

unfortunately there's an unknown shared library, I don't know what it is, I compiled linkerd-proxy with debuginfo=1 and opt-level = "s", for debuginfo=2 rustc crashed with LLVM abbr code error... so looking for help, any ideas ?

[roodkcab]

both kong pod and nginx pod are injected with linkerd-proxy sidecar

[olix0r]

@roodkcab if you're able to share the manifests you're using for tests, we'd be eager to help look into these issues with you.

[roodkcab]

@olix0r sure, cloud u tell me what manifests do u need?

CPU: Intel(R) Xeon(R) Platinum 8269CY CPU @ 2.50GHz
MEM: 16G
linkerd: stable-2.10.0
k8s: v1.20.5
cni: terway + cilium in ipvlan mode

[roodkcab]

I inject linkerd-proxy with --proxy-cpu-limit=4000m --proxy-cpu-request=100m --proxy-memory-limit=200M

[roodkcab]

here's some more information I find

0. with viz.linkerd.io/disable-tap: "true" annotation set, and inject linkerd-proxy with --disable-identity, qps increase from 3900 to 4300.
1. set --proxy-cpu-limit=2000m --proxy-cpu-request=100m --proxy-memory-limit=200M makes linkerd-proxy working in multithread mode, and qps increase to 5900.
2. linkerd-proxy is indeed fast, sending request to the clusterIP or podIP without using service name, only 5% performance get lost.
3. when using service name with debug log opened, I found most cpu time is token by outbound progress, in which linkerd-proxy did these job

• it has to determine which backend service to use and check whether it is ready for connection (3.3ms)
• connect to endpoint (1.5ms)
• send request and waiting for response (1.5ms)

I think it quite clearly to me where is the bottleneck now, I lost 80% performance using default configuration. 77.5% lost when close tap and tls, finally 70% lost when making linkerd-proxy working in multi-thread mode. There's about 50% cpu time is token for calculate suitable service.