when a etcd down, bookkeeper cannot boot.

[zhangguochaoZZ]

@tisonkun when I run bookie, and edit bookkeeper.conf metadataServiceUri=metadata-store:etcd:http://192.168.189.210:2379,http://192.168.189.211:2379,http://192.168.189.212:2379,if etcd down on 192.168.189.210,some error as follow

root@ubuntu-1:/data/pulsar/bookie# ./bin/pulsar bookie
2023-04-07T18:36:25,542+0800 [main] INFO  org.apache.bookkeeper.server.Main - Using configuration file /data/pulsar/bookie/conf/bookkeeper.conf
2023-04-07T18:36:25,564+0800 [main] INFO  org.apache.bookkeeper.server.Main - Hello, I'm your bookie, bookieId is <not-set>, listening on port 3181. Metadata service uri is metadata-store:etcd:http://192.168.189.210:2379. Journals are in [data/bookkeeper/journal]. Ledgers are stored in [data/bookkeeper/ledgers].
2023-04-07T18:36:25,603+0800 [main] INFO  org.apache.bookkeeper.server.Main - Load lifecycle component : org.apache.bookkeeper.server.service.StatsProviderService
2023-04-07T18:36:25,659+0800 [main] INFO  org.apache.bookkeeper.meta.MetadataDrivers - BookKeeper metadata driver manager initialized
2023-04-07T18:36:30,369+0800 [metadata-store-etcd-session-watcher-4-1] WARN  org.apache.pulsar.metadata.impl.EtcdSessionWatcher - Etcd client is disconnected. Waiting to reconnect, time remaining = 25.0 seconds
2023-04-07T18:36:32,373+0800 [metadata-store-etcd-session-watcher-4-1] WARN  org.apache.pulsar.metadata.impl.EtcdSessionWatcher - Etcd client is disconnected. Waiting to reconnect, time remaining = 22.995 seconds
2023-04-07T18:36:34,375+0800 [metadata-store-etcd-session-watcher-4-1] WARN  org.apache.pulsar.metadata.impl.EtcdSessionWatcher - Etcd client is disconnected. Waiting to reconnect, time remaining = 20.994 seconds
2023-04-07T18:36:36,377+0800 [metadata-store-etcd-session-watcher-4-1] WARN  org.apache.pulsar.metadata.impl.EtcdSessionWatcher - Etcd client is disconnected. Waiting to reconnect, time remaining = 18.992 seconds
2023-04-07T18:36:38,378+0800 [metadata-store-etcd-session-watcher-4-1] WARN  org.apache.pulsar.metadata.impl.EtcdSessionWatcher - Etcd client is disconnected. Waiting to reconnect, time remaining = 16.99 seconds
2023-04-07T18:36:40,379+0800 [metadata-store-etcd-session-watcher-4-1] WARN  org.apache.pulsar.metadata.impl.EtcdSessionWatcher - Etcd client is disconnected. Waiting to reconnect, time remaining = 14.989 seconds
2023-04-07T18:36:42,381+0800 [metadata-store-etcd-session-watcher-4-1] WARN  org.apache.pulsar.metadata.impl.EtcdSessionWatcher - Etcd client is disconnected. Waiting to reconnect, time remaining = 12.987 seconds
2023-04-07T18:36:44,382+0800 [metadata-store-etcd-session-watcher-4-1] WARN  org.apache.pulsar.metadata.impl.EtcdSessionWatcher - Etcd client is disconnected. Waiting to reconnect, time remaining = 10.986 seconds
2023-04-07T18:36:46,384+0800 [metadata-store-etcd-session-watcher-4-1] WARN  org.apache.pulsar.metadata.impl.EtcdSessionWatcher - Etcd client is disconnected. Waiting to reconnect, time remaining = 8.985 seconds
2023-04-07T18:36:48,385+0800 [metadata-store-etcd-session-watcher-4-1] WARN  org.apache.pulsar.metadata.impl.EtcdSessionWatcher - Etcd client is disconnected. Waiting to reconnect, time remaining = 6.983 seconds
2023-04-07T18:36:50,387+0800 [metadata-store-etcd-session-watcher-4-1] WARN  org.apache.pulsar.metadata.impl.EtcdSessionWatcher - Etcd client is disconnected. Waiting to reconnect, time remaining = 4.981 seconds
2023-04-07T18:36:52,389+0800 [metadata-store-etcd-session-watcher-4-1] WARN  org.apache.pulsar.metadata.impl.EtcdSessionWatcher - Etcd client is disconnected. Waiting to reconnect, time remaining = 2.979 seconds
2023-04-07T18:36:54,391+0800 [metadata-store-etcd-session-watcher-4-1] WARN  org.apache.pulsar.metadata.impl.EtcdSessionWatcher - Etcd client is disconnected. Waiting to reconnect, time remaining = 0.978 seconds
2023-04-07T18:36:56,393+0800 [metadata-store-etcd-session-watcher-4-1] ERROR org.apache.pulsar.metadata.impl.EtcdSessionWatcher - Etcd lease keep-alive timeout. Notifying session is lost.

then, the bookie keep stuck, cannot connect to other etcds

How to configure metadataServiceUri ？I can't find the reference

[eolivelli]

Hello,
Are you using the Etcd driver in production? Or are you only testing it?

Are you on k8s?

(Tagging @merlimat who tested etcd support)

[zhangguochaoZZ]

Hello, Are you using the Etcd driver in production? Or are you only testing it?

Are you on k8s?

(Tagging @merlimat who tested etcd support)

@eolivelli thank your reply, because other applications use etcd cluster, so pulsar intends to use etcd as well,Do you think it's a good idea?

not in k8s

[liangyuanpeng]

what's your pulsar version?

[zhangguochaoZZ]

2.10.3