Bookkeeper startup error: NoNode for /ledgers/LAYOUT

[abhilashmandaliya]

Hi guys. I am trying to run pulsar 2.11 with bookkeeper 4.15.4 and zookeeper 3.8.0. Zookeeper is running normally. I see the crash on bookkeeper startup with this error:

12:18:47.856 [main-SendThread(172.17.0.6:2181)] INFO  org.apache.zookeeper.ClientCnxn - Session establishment complete on server 172.17.0.6/172.17.0.6:2181, session id = 0x10001a9eeb10001, negotiated timeout = 30000
12:18:47.858 [main-EventThread] INFO  org.apache.bookkeeper.zookeeper.ZooKeeperWatcherBase - ZooKeeper client is connected now.
12:18:47.900 [main] ERROR org.apache.bookkeeper.server.Main - Failed to build bookie server
org.apache.bookkeeper.meta.exceptions.MetadataException: Failed to initialized ledger manager factory
        at org.apache.bookkeeper.meta.zk.ZKMetadataDriverBase.getLedgerManagerFactory(ZKMetadataDriverBase.java:263) ~[org.apache.bookkeeper-bookkeeper-server-4.15.3.jar:4.15.3]
        at org.apache.bookkeeper.server.Main.buildBookieServer(Main.java:357) ~[org.apache.bookkeeper-bookkeeper-server-4.15.3.jar:4.15.3]
        at org.apache.bookkeeper.server.Main.doMain(Main.java:272) ~[org.apache.bookkeeper-bookkeeper-server-4.15.3.jar:4.15.3]
        at org.apache.bookkeeper.server.Main.main(Main.java:255) ~[org.apache.bookkeeper-bookkeeper-server-4.15.3.jar:4.15.3]
Caused by: java.io.IOException: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /ledgers/LAYOUT
        at org.apache.bookkeeper.meta.ZkLayoutManager.storeLedgerLayout(ZkLayoutManager.java:86) ~[org.apache.bookkeeper-bookkeeper-server-4.15.3.jar:4.15.3]
        at org.apache.bookkeeper.meta.AbstractZkLedgerManagerFactory.createNewLMFactory(AbstractZkLedgerManagerFactory.java:336) ~[org.apache.bookkeeper-bookkeeper-server-4.15.3.jar:4.15.3]
        at org.apache.bookkeeper.meta.AbstractZkLedgerManagerFactory.newLedgerManagerFactory(AbstractZkLedgerManagerFactory.java:179) ~[org.apache.bookkeeper-bookkeeper-server-4.15.3.jar:4.15.3]
        at org.apache.bookkeeper.meta.zk.ZKMetadataDriverBase.getLedgerManagerFactory(ZKMetadataDriverBase.java:259) ~[org.apache.bookkeeper-bookkeeper-server-4.15.3.jar:4.15.3]
        ... 3 more
Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /ledgers/LAYOUT
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:118) ~[org.apache.zookeeper-zookeeper-3.8.0.jar:3.8.0]
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:54) ~[org.apache.zookeeper-zookeeper-3.8.0.jar:3.8.0]
        at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:1347) ~[org.apache.zookeeper-zookeeper-3.8.0.jar:3.8.0]
        at org.apache.bookkeeper.zookeeper.ZooKeeperClient.access$1901(ZooKeeperClient.java:70) ~[org.apache.bookkeeper-bookkeeper-server-4.15.3.jar:4.15.3]
        at org.apache.bookkeeper.zookeeper.ZooKeeperClient$9.call(ZooKeeperClient.java:714) ~[org.apache.bookkeeper-bookkeeper-server-4.15.3.jar:4.15.3]
        at org.apache.bookkeeper.zookeeper.ZooKeeperClient$9.call(ZooKeeperClient.java:708) ~[org.apache.bookkeeper-bookkeeper-server-4.15.3.jar:4.15.3]
        at org.apache.bookkeeper.zookeeper.ZooWorker.syncCallWithRetries(ZooWorker.java:140) ~[org.apache.bookkeeper-bookkeeper-server-4.15.3.jar:4.15.3]
        at org.apache.bookkeeper.zookeeper.ZooKeeperClient.create(ZooKeeperClient.java:708) ~[org.apache.bookkeeper-bookkeeper-server-4.15.3.jar:4.15.3]
        at org.apache.bookkeeper.meta.ZkLayoutManager.storeLedgerLayout(ZkLayoutManager.java:81) ~[org.apache.bookkeeper-bookkeeper-server-4.15.3.jar:4.15.3]
        at org.apache.bookkeeper.meta.AbstractZkLedgerManagerFactory.createNewLMFactory(AbstractZkLedgerManagerFactory.java:336) ~[org.apache.bookkeeper-bookkeeper-server-4.15.3.jar:4.15.3]
        at org.apache.bookkeeper.meta.AbstractZkLedgerManagerFactory.newLedgerManagerFactory(AbstractZkLedgerManagerFactory.java:179) ~[org.apache.bookkeeper-bookkeeper-server-4.15.3.jar:4.15.3]
        at org.apache.bookkeeper.meta.zk.ZKMetadataDriverBase.getLedgerManagerFactory(ZKMetadataDriverBase.java:259) ~[org.apache.bookkeeper-bookkeeper-server-4.15.3.jar:4.15.3]
        ... 3 more

Zookeeper CLI output

[zk: localhost:2181(CONNECTED) 2] ls /
[zookeeper]
[zk: localhost:2181(CONNECTED) 3] 

Any help is greatly appreciated 🙂

[abhilashmandaliya]

Got some help from the community over slack. They pointed me to the cluster metadata initialization link. My script was running the metadata command but it was failing which I didn't observe. The reason was my java runtime version 1.8 and pulsar 2.11 required Java 17. I was upgrading my cluster and forgot to update that part in the script. Using Java 17 solved it for me.