bin/pulsar initialize-cluster-metadata gives an error

[azeng528]

Search before asking

• I searched in the issues and found nothing similar.

Version

pulsar: 2.11.0
etcd: 3.5.7

Minimal reproduce step

Failed to run Pulsar with etcd cluster:

bin/pulsar initialize-cluster-metadata --cluster pulsar-cluster --metadata-store etcd:http://1.1.1.1:2379,http://2.2.2.2:2379,http://3.3.3.3:2379 --configuration-metadata-store etcd:http://1.1.1.1:2379,http://2.2.2.2:2379,http://3.3.3.3:2379 --web-service-url http://slb:8080 --broker-service-url pulsar://slb:6650
2023-03-09T19:38:17,625+0800 [main] INFO  org.apache.pulsar.PulsarClusterMetadataSetup - Setting up cluster pulsar-cluster with metadata-store=etcd:http://1.1.1.1:2379,http://2.2.2.2:2379,http://3.3.3.3:2379 configuration-metadata-store=etcd:http://1.1.1.1:2379,http://2.2.2.2:2379,http://3.3.3.3:2379
2023-03-09T19:38:18,446+0800 [main] INFO  org.apache.pulsar.PulsarClusterMetadataSetup - Cluster metadata for 'pulsar-cluster' setup correctly
Mar 09, 2023 7:38:18 PM com.google.common.util.concurrent.AbstractFuture executeListener
SEVERE: RuntimeException while executing runnable io.etcd.jetcd.ClientConnectionManager$$Lambda$163/0x0000000800e73060@54f21c79 with executor java.util.concurrent.ThreadPoolExecutor@4cc4feba[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 43]
java.util.concurrent.RejectedExecutionException: Task io.etcd.jetcd.ClientConnectionManager$$Lambda$163/0x0000000800e73060@54f21c79 rejected from java.util.concurrent.ThreadPoolExecutor@4cc4feba[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 43]
	at java.base/java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2065)
	at java.base/java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:833)
	at java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1365)
	at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1277)
	at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:1038)
	at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:808)
	at io.grpc.stub.ClientCalls$GrpcFuture.setException(ClientCalls.java:563)
	at io.grpc.stub.ClientCalls$UnaryStreamToFuture.onClose(ClientCalls.java:533)
	at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
	at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
	at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
	at io.etcd.jetcd.auth.AuthInterceptor$1$1.onClose(AuthInterceptor.java:101)
	at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:562)
	at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:70)
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:743)
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:722)
	at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
	at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:833)

### What did you expect to see?

cluster is initialize ok

### What did you see instead?

bin/pulsar initialize-cluster-metadata --cluster pulsar-cluster --metadata-store etcd:http://1.1.1.1:2379,http://2.2.2.2:2379,http://3.3.3.3:2379 --configuration-metadata-store etcd:http://1.1.1.1:2379,http://2.2.2.2:2379,http://3.3.3.3:2379 --web-service-url http://slb:8080 --broker-service-url pulsar://slb:6650
2023-03-09T19:38:17,625+0800 [main] INFO  org.apache.pulsar.PulsarClusterMetadataSetup - Setting up cluster pulsar-cluster with metadata-store=etcd:http://1.1.1.1:2379,http://2.2.2.2:2379,http://3.3.3.3:2379 configuration-metadata-store=etcd:http://1.1.1.1:2379,http://2.2.2.2:2379,http://3.3.3.3:2379
2023-03-09T19:38:18,446+0800 [main] INFO  org.apache.pulsar.PulsarClusterMetadataSetup - Cluster metadata for 'pulsar-cluster' setup correctly
Mar 09, 2023 7:38:18 PM com.google.common.util.concurrent.AbstractFuture executeListener
SEVERE: RuntimeException while executing runnable io.etcd.jetcd.ClientConnectionManager$$Lambda$163/0x0000000800e73060@54f21c79 with executor java.util.concurrent.ThreadPoolExecutor@4cc4feba[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 43]
java.util.concurrent.RejectedExecutionException: Task io.etcd.jetcd.ClientConnectionManager$$Lambda$163/0x0000000800e73060@54f21c79 rejected from java.util.concurrent.ThreadPoolExecutor@4cc4feba[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 43]
	at java.base/java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2065)
	at java.base/java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:833)
	at java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1365)
	at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1277)
	at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:1038)
	at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:808)
	at io.grpc.stub.ClientCalls$GrpcFuture.setException(ClientCalls.java:563)
	at io.grpc.stub.ClientCalls$UnaryStreamToFuture.onClose(ClientCalls.java:533)
	at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
	at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
	at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
	at io.etcd.jetcd.auth.AuthInterceptor$1$1.onClose(AuthInterceptor.java:101)
	at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:562)
	at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:70)
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:743)
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:722)
	at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
	at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:833)

Anything else?

No response

Are you willing to submit a PR?

• I'm willing to submit a PR!

[liangyuanpeng]

Actually, it should working for you, the log say Cluster metadata for 'pulsar-cluster' setup correctly