Pulsar initialize-cluster-metadata failed.

[jihwong]

I would like to use an external Zookeeper cluster, but there was an exception when initializing the metadata for the pulsar cluster.

My zookeeper cluster should be running normally because the 'bookkeeper shell initnewcluster' can execute successfully.
Error log

2023-07-25T02:23:18,820+0000 [main-SendThread(zk-svc.default.svc.cluster.local:2181)] INFO  org.apache.zookeeper.ClientCnxn - Session establishment complete on server zk-svc.default.svc.cluster.local/10.254.177.149:2181, session id = 0x18985f2648a00c9, negotiated timeout = 40000
2023-07-25T02:23:18,823+0000 [main-EventThread] INFO  org.apache.bookkeeper.zookeeper.ZooKeeperWatcherBase - ZooKeeper client is connected now.
Exception in thread "main" org.apache.pulsar.metadata.api.MetadataStoreException: org.apache.zookeeper.KeeperException$UnimplementedException: KeeperErrorCode = Unimplemented for /
	at org.apache.pulsar.metadata.impl.ZKMetadataStore.<init>(ZKMetadataStore.java:112)
	at org.apache.pulsar.metadata.impl.MetadataStoreFactoryImpl.newInstance(MetadataStoreFactoryImpl.java:56)
	at org.apache.pulsar.metadata.impl.MetadataStoreFactoryImpl.createExtended(MetadataStoreFactoryImpl.java:36)
	at org.apache.pulsar.metadata.api.extended.MetadataStoreExtended.create(MetadataStoreExtended.java:40)
	at org.apache.pulsar.PulsarClusterMetadataSetup.initMetadataStore(PulsarClusterMetadataSetup.java:380)
	at org.apache.pulsar.PulsarClusterMetadataSetup.main(PulsarClusterMetadataSetup.java:238)
Caused by: org.apache.zookeeper.KeeperException$UnimplementedException: KeeperErrorCode = Unimplemented for /
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:106)
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
	at org.apache.zookeeper.ZooKeeper.addWatch(ZooKeeper.java:3192)
	at org.apache.pulsar.metadata.impl.PulsarZooKeeperClient.access$3301(PulsarZooKeeperClient.java:71)
	at org.apache.pulsar.metadata.impl.PulsarZooKeeperClient$23.call(PulsarZooKeeperClient.java:1127)
	at org.apache.pulsar.metadata.impl.PulsarZooKeeperClient$23.call(PulsarZooKeeperClient.java:1121)
	at org.apache.pulsar.metadata.impl.PulsarZooKeeperClient$ZooWorker.syncCallWithRetries(PulsarZooKeeperClient.java:1528)
	at org.apache.pulsar.metadata.impl.PulsarZooKeeperClient.addWatch(PulsarZooKeeperClient.java:1121)
	at org.apache.pulsar.metadata.impl.ZKMetadataStore.<init>(ZKMetadataStore.java:105)
	... 5 more
2023-07-25T02:23:18,837+0000 [main-SendThread(zk-svc.default.svc.cluster.local:2181)] WARN  org.apache.zookeeper.ClientCnxn - Session 0x18985f2648a00c9 for sever zk-svc.default.svc.cluster.local/10.254.177.149:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
org.apache.zookeeper.ClientCnxn$EndOfStreamException: Unable to read additional data from server sessionid 0x18985f2648a00c9, likely server has closed socket
	at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:77) ~[org.apache.zookeeper-zookeeper-3.6.3.jar:3.6.3]
	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350) ~[org.apache.zookeeper-zookeeper-3.6.3.jar:3.6.3]
	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1290) ~[org.apache.zookeeper-zookeeper-3.6.3.jar:3.6.3]
2023-07-25T02:23:18,972+0000 [main-EventThread] INFO  org.apache.bookkeeper.zookeeper.ZooKeeperWatcherBase - ZooKeeper client is disconnected from zookeeper now, but it is OK unless we received EXPIRED event.
2023-07-25T02:23:20,376+0000 [main-SendThread(zk-svc.default.svc.cluster.local:2181)] INFO  org.apache.zookeeper.ClientCnxn - Opening socket connection to server zk-svc.default.svc.cluster.local/10.254.177.149:2181.
2023-07-25T02:23:20,376+0000 [main-SendThread(zk-svc.default.svc.cluster.local:2181)] INFO  org.apache.zookeeper.ClientCnxn - SASL config status: Will not attempt to authenticate using SASL (unknown error)
2023-07-25T02:23:20,378+0000 [main-SendThread(zk-svc.default.svc.cluster.local:2181)] INFO  org.apache.zookeeper.ClientCnxn - Socket connection established, initiating session, client: /10.0.5.229:55300, server: zk-svc.default.svc.cluster.local/10.254.177.149:2181
2023-07-25T02:23:20,382+0000 [main-SendThread(zk-svc.default.svc.cluster.local:2181)] INFO  org.apache.zookeeper.ClientCnxn - Session establishment complete on server zk-svc.default.svc.cluster.local/10.254.177.149:2181, session id = 0x18985f2648a00c9, negotiated timeout = 40000
2023-07-25T02:23:20,383+0000 [main-EventThread] INFO  org.apache.bookkeeper.zookeeper.ZooKeeperWatcherBase - ZooKeeper client is connected now.

[jihwong]

It seems that there is a version mismatch. The original Zookeeper is 3.4. x, while the Pulsar client is 3.6. x