[Bug] boot broker failed with etcd

[zhangguochaoZZ]

Search before asking

• I searched in the issues and found nothing similar.

Version

2.10.2

Minimal reproduce step

configure:
metadataStoreUrl=etcd:http://10.51.4.x:2379,http://10.51.3.x:2379,http://10.51.3.x:2379

What did you expect to see?

null

What did you see instead?

WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by io.netty.util.internal.ReflectionUtil (file:/opt/midd/apache-pulsar-broker/lib/io.netty-netty-common-4.1.77.Final.jar) to constructor java.nio.DirectByteBuffer(long,int)
WARNING: Please consider reporting this to the maintainers of io.netty.util.internal.ReflectionUtil
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
2022-12-15T09:39:49,667+0000 [main] INFO org.apache.pulsar.broker.PulsarService - Starting Pulsar Broker service; version: '2.10.2'
2022-12-15T09:39:49,669+0000 [main] INFO org.apache.pulsar.broker.PulsarService - Git Revision 11b5df7
2022-12-15T09:39:49,670+0000 [main] INFO org.apache.pulsar.broker.PulsarService - Git Branch branch-2.10
2022-12-15T09:39:49,670+0000 [main] INFO org.apache.pulsar.broker.PulsarService - Built by JiangHaiting jianghaiting@apache.org on localhost at 2020-01-22T23:10:15+0800
2022-12-15T09:39:49,952+0000 [grpc-default-executor-0] WARN io.etcd.jetcd.resolver.IPNameResolver - Error wile getting list of servers
java.lang.IllegalArgumentException: port out of range:-1
at java.net.InetSocketAddress.checkPort(InetSocketAddress.java:143) ~[?:?]
at java.net.InetSocketAddress.(InetSocketAddress.java:224) ~[?:?]
at io.etcd.jetcd.resolver.IPNameResolver$HostAndPort.toAddressGroup(IPNameResolver.java:162) ~[io.etcd-jetcd-core-0.5.11.jar:?]
at io.etcd.jetcd.resolver.IPNameResolver.lambda$doResolve$1(IPNameResolver.java:133) ~[io.etcd-jetcd-core-0.5.11.jar:?]
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195) ~[?:?]
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1655) ~[?:?]
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) ~[?:?]
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) ~[?:?]
at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913) ~[?:?]
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578) ~[?:?]
at io.etcd.jetcd.resolver.IPNameResolver.doResolve(IPNameResolver.java:134) ~[io.etcd-jetcd-core-0.5.11.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:834) ~[?:?]
2022-12-15T09:39:50,016+0000 [grpc-default-executor-0] WARN io.grpc.internal.ManagedChannelImpl - [Channel<1>: (ip:///null:-1)] Failed to resolve name. status=Status{code=NOT_FOUND, description=null, cause=null}

Anything else?

No response

Are you willing to submit a PR?

• I'm willing to submit a PR!

[tisonkun]

@elysian-gc do you still suffer from this issue? It reads like a usage issue than a bug.

The error log indicates that you configure the port as -1. I run a three processes etcd cluster and connect to it locally and cannot reproduce your issue.

[tisonkun]

It seems that the DNS cannot resolve your domain name Failed to resolve name. Is it possible to use the IP:port pattern?

[tisonkun]

Or can you try it 2.11.0? I run the same command on master and cannot reproduce.

[zhangguochaoZZ]

use IP:port

root@ubuntu2-virtual-machine:/data/pulsar/bookie# ./bin/pulsar initialize-cluster-metadata -c pulsar-cluster -cms etcd:http://192.168.189.210:2379,http://192.168.189.211:2379,http://192.168.189.212:2379 -md etcd:http://192.168.189.210:2379,http://192.168.189.211:2379,http://192.168.189.212:2379 -uw http://192.168.189.210:8080,192.168.189.211:8080,192.168.189.212:8080 -ub pulsar://192.168.189.210:6650,192.168.189.211:6650,192.168.189.212:6650
2023-04-07T15:24:33,479+0800 [main] INFO  org.apache.pulsar.PulsarClusterMetadataSetup - Setting up cluster pulsar-cluster with metadata-store=etcd:http://192.168.189.210:2379,http://192.168.189.211:2379,http://192.168.189.212:2379 configuration-metadata-store=etcd:http://192.168.189.210:2379,http://192.168.189.211:2379,http://192.168.189.212:2379
2023-04-07T15:24:33,857+0800 [grpc-default-executor-0] WARN  io.etcd.jetcd.resolver.IPNameResolver - Error wile getting list of servers
java.lang.IllegalArgumentException: port out of range:-1
        at java.net.InetSocketAddress.checkPort(InetSocketAddress.java:152) ~[?:?]
        at java.net.InetSocketAddress.<init>(InetSocketAddress.java:233) ~[?:?]
        at io.etcd.jetcd.resolver.IPNameResolver$HostAndPort.toAddressGroup(IPNameResolver.java:162) ~[io.etcd-jetcd-core-0.5.11.jar:?]
        at io.etcd.jetcd.resolver.IPNameResolver.lambda$doResolve$1(IPNameResolver.java:133) ~[io.etcd-jetcd-core-0.5.11.jar:?]
        at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197) ~[?:?]
        at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625) ~[?:?]
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) ~[?:?]
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) ~[?:?]
        at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921) ~[?:?]
        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
        at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682) ~[?:?]
        at io.etcd.jetcd.resolver.IPNameResolver.doResolve(IPNameResolver.java:134) ~[io.etcd-jetcd-core-0.5.11.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
        at java.lang.Thread.run(Thread.java:833) ~[?:?]
Apr 07, 2023 3:24:33 PM io.grpc.internal.ManagedChannelImpl$NameResolverListener handleErrorInSyncContext
WARNING: [Channel<1>: (ip:///null:-1)] Failed to resolve name. status=Status{code=NOT_FOUND, description=null, cause=null}
2023-04-07T15:24:35,060+0800 [grpc-default-executor-1] WARN  io.etcd.jetcd.resolver.IPNameResolver - Error wile getting list of servers
java.lang.IllegalArgumentException: port out of range:-1
        at java.net.InetSocketAddress.checkPort(InetSocketAddress.java:152) ~[?:?]
        at java.net.InetSocketAddress.<init>(InetSocketAddress.java:233) ~[?:?]
        at io.etcd.jetcd.resolver.IPNameResolver$HostAndPort.toAddressGroup(IPNameResolver.java:162) ~[io.etcd-jetcd-core-0.5.11.jar:?]
        at io.etcd.jetcd.resolver.IPNameResolver.lambda$doResolve$1(IPNameResolver.java:133) ~[io.etcd-jetcd-core-0.5.11.jar:?]
        at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197) ~[?:?]
        at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625) ~[?:?]
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) ~[?:?]
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) ~[?:?]
        at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921) ~[?:?]
        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
        at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682) ~[?:?]
        at io.etcd.jetcd.resolver.IPNameResolver.doResolve(IPNameResolver.java:134) ~[io.etcd-jetcd-core-0.5.11.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
        at java.lang.Thread.run(Thread.java:833) ~[?:?]
Apr 07, 2023 3:24:35 PM io.grpc.internal.ManagedChannelImpl$NameResolverListener handleErrorInSyncContext
WARNING: [Channel<1>: (ip:///null:-1)] Failed to resolve name. status=Status{code=NOT_FOUND, description=null, cause=null}

[zhangguochaoZZ]

Or can you try it 2.11.0? I run the same command on master and cannot reproduce.

thank you so much,2.11.0 work well.

[tisonkun]

Good to know!

[zhangguochaoZZ]

@tisonkun sorry,I want ask another question,when run bookie, and edit bookkeeper.conf metadataServiceUri=metadata-store:etcd:http://192.168.189.210:2379,http://192.168.189.211:2379,http://192.168.189.212:2379,if etcd down on 192.168.189.210,some error as follow

root@ubuntu-1:/data/pulsar/bookie# ./bin/pulsar bookie
2023-04-07T18:23:06,897+0800 [main] INFO  org.apache.bookkeeper.server.Main - Using configuration file /data/pulsar/bookie/conf/bookkeeper.conf
2023-04-07T18:23:06,911+0800 [main] INFO  org.apache.bookkeeper.server.Main - Hello, I'm your bookie, bookieId is <not-set>, listening on port 3181. Metadata service uri is metadata-store:etcd:http://192.168.189.210:2379. Journals are in [data/bookkeeper/journal]. Ledgers are stored in [data/bookkeeper/ledgers].
2023-04-07T18:23:06,937+0800 [main] INFO  org.apache.bookkeeper.server.Main - Load lifecycle component : org.apache.bookkeeper.server.service.StatsProviderService
2023-04-07T18:23:06,981+0800 [main] INFO  org.apache.bookkeeper.meta.MetadataDrivers - BookKeeper metadata driver manager initialized
2023-04-07T18:23:11,446+0800 [metadata-store-etcd-session-watcher-4-1] WARN  org.apache.pulsar.metadata.impl.EtcdSessionWatcher - Etcd client is disconnected. Waiting to reconnect, time remaining = 25.0 seconds
2023-04-07T18:23:13,449+0800 [metadata-store-etcd-session-watcher-4-1] WARN  org.apache.pulsar.metadata.impl.EtcdSessionWatcher - Etcd client is disconnected. Waiting to reconnect, time remaining = 22.997 seconds
2023-04-07T18:23:15,450+0800 [metadata-store-etcd-session-watcher-4-1] WARN  org.apache.pulsar.metadata.impl.EtcdSessionWatcher - Etcd client is disconnected. Waiting to reconnect, time remaining = 20.996 seconds
2023-04-07T18:23:17,452+0800 [metadata-store-etcd-session-watcher-4-1] WARN  org.apache.pulsar.metadata.impl.EtcdSessionWatcher - Etcd client is disconnected. Waiting to reconnect, time remaining = 18.994 seconds
2023-04-07T18:23:19,454+0800 [metadata-store-etcd-session-watcher-4-1] WARN  org.apache.pulsar.metadata.impl.EtcdSessionWatcher - Etcd client is disconnected. Waiting to reconnect, time remaining = 16.991 seconds

How to configure metadataServiceUri ？I can't find the reference

[tisonkun]

Please open a new thread so that we can focus one by one and don't mix up topics.

A quick answer can be: it's not errors but warnings. Generally, the client should reconnect to other server if the quorm has enough members.