Using Pulsar inside Istio Service Mesh?

[valkenburg-prevue-ch]

In brief: is there a guide on how to best use pulsar in combination with Istio? Pulsar inside the mesh, or outside but with its own tls enabled? Or another way? How?

I have a k3s cluster with istio enabled. I then deploy the pulsar helm chart:

name: pulsar
version: ~3.0.0
repository: https://pulsar.apache.org/charts

This gets deployed in a namespace with "istio-injection=enabled" together with an allow-all authorization policy (just to get things up and running first, adding strict policies later). So all pods get an envoy sidecar which hijacks all networking, but all traffic through that sidecar between all pulsar pods is allowed for.

Where I get stuck, and I found no way to work around this, is with the initcontainer pulsar-bookkeeper-verify-clusterid used by the bookkeeper pod, which needs networking. In istio, networking is not available for initcontainers. So I get these logsfrom pulsar-bookkeeper-verify-clusterid:

2023-03-03T09:42:42,491+0000 [main-SendThread(pulsar-zookeeper:2181)] INFO  org.apache.zookeeper.ClientCnxn - Opening socket connection to server pulsar-zookeeper/10.42.5.43:2181.
2023-03-03T09:42:42,491+0000 [main-SendThread(pulsar-zookeeper:2181)] INFO  org.apache.zookeeper.ClientCnxn - SASL config status: Will not attempt to authenticate using SASL (unknown error)
2023-03-03T09:42:42,493+0000 [main-SendThread(pulsar-zookeeper:2181)] INFO  org.apache.zookeeper.ClientCnxn - Socket connection established, initiating session, client: /10.42.1.47:49610, server: pulsar-zookeeper/10.42.5.43:2181
2023-03-03T09:42:42,494+0000 [main-SendThread(pulsar-zookeeper:2181)] WARN  org.apache.zookeeper.ClientCnxn - Session 0x0 for sever pulsar-zookeeper/10.42.5.43:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.io.IOException: Connection reset by peer
        at sun.nio.ch.FileDispatcherImpl.read0(Native Method) ~[?:?]
        at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) ~[?:?]
        at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:276) ~[?:?]
        at sun.nio.ch.IOUtil.read(IOUtil.java:233) ~[?:?]
        at sun.nio.ch.IOUtil.read(IOUtil.java:223) ~[?:?]
        at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:356) ~[?:?]
        at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:74) ~[org.apache.zookeeper-zookeeper-3.6.3.jar:3.6.3]
        at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350) ~[org.apache.zookeeper-zookeeper-3.6.3.jar:3.6.3]
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1290) ~[org.apache.zookeeper-zookeeper-3.6.3.jar:3.6.3]

On the other hand, I do see that there is extraInitCommand, with the comment that one can use this for example with istio (extraInitCommand: "curl -fsI -X POST http://localhost:15020/quitquitquit ;" ). So there is the hint that people use pulsar with istio, but I cannot get it to work and I do not find documentation on how to do this. Does anyone know of a solution?

[ericsyh]

@valkenburg-prevue-ch I think your issue is caused by the Istio limitation. Like the issue mentioned, there is no easy solution now.

In our use case, we removed init containers on our pulsar operators to support Istio. Also, we did some extra integration like creating the Istio gateway & VirtualService through our operator. If you are interested to have a try, you can follow this documentation to deploy a pulsar cluster with Istio injected and use Istio gateway to expose Pulsar traffic.

[valkenburg-prevue-ch]

Thanks! Yes, that limitation is absolutely what is going on.

I see, so what is the most sustainable way to go: use your operator, or fork this helm chart and simply drop the initcontainers which need networking? How bad is it not to run pulsar-bookkeeper-verify-clusterid ?

[ericsyh]

I guess for now there are some extra efforts to make pulsar-helm-chart running with Istio b/c several components have initContainers in the chart.
I didn't try hacking to disable bk's initContainers before, but I think not running pulsar-bookkeeper-verify-clusterid should be okay. Bk nodes start running has no much dependency.

[valkenburg-prevue-ch]

Thanks for your help. I now moved to using the streamnative Pulsar Operator as you suggested, and indeed it works under istio without any extra effort, simply provisioning a cluster with the quickstart.yaml as per your link!