Unnecessary DNS lookup errors in logs

[sindhushreem25]

"On a K8s deployment, when pulsar client is used from a namespace different from pulsar broker/proxy namespace we see unnecessary errors in logs which is misleading. Pulsar client is appending all the DNS search domains causing connection errors. This happens even post a valid name is identified for connecting to server."

When we upgraded the netty to 118 in pulsar 3.0.X we are seeing a different behavior.

in older version it used to resolve to one dns name and fail or pass the connection .
WARN org.apache.pulsar.client.impl.ConnectionPool - [[id: 0x7a0a4f2b, L:/172.16.3.21:38920 ! R:pulsar-proxy.xx.xxx.cluster.local/xxx.xx.xx.xxx:6651]] Connection handshake failed: org.apache.pulsar.client.api.PulsarClientException: Connection already closed

but with 118 its trying to resolve the next dns in some interval and printing the below warn messages :
org.apache.pulsar.client.impl.ConnectionPool - Failed to open connection to pulsar-proxy.op:6651 : org.apache.pulsar.shade.io.netty.resolver.dns.DefaultDnsCache$2: Failed to resolve

[lhotari]

To locate the possible source of the error logs, it would be helpful to provide some examples.

Appending a trailing dot to the FQDN is a standard way to prevent using configured search domains. This is a not-so-well-known detail of the original DNS RFC 1034 written in 1987. Absolute DNS names end with a dot, and relative names don't have a trailing dot.

I haven't tested whether this solution works with the Pulsar client. For example, using pulsar://myns-pulsar-broker.myns.svc.cluster.local.:6650/ instead of pulsar://somens-pulsar-broker.myns.svc.cluster.local:6650/ in the client. (Please notice the extra dot before :6650)

If this doesn't work, please file an issue at https://github.com/apache/pulsar/issues. Excessive error logging could be a separate issue.

[sindhushreem25]

We are on pulsar 3.0.6 and i had to cherry pick the netty fix because of security vulnerability .post that the behavior changed.

92527df

[pulsar@itomdipulsar-bastion-0 pulsar]$ cat /etc/resolv.conf
search op.svc.cluster.local svc.cluster.local cluster.local
nameserver 172.17.0.78
options ndots:5

[lhotari]

We are on pulsar 3.0.6 and i had to cherry pick the netty fix because of security vulnerability .post that the behavior changed.

Newer Pulsar clients are compatible with older broker versions. Would you be able to test with Pulsar client 4.0.3 to see if there's any difference in behavior compared to what you are seeing? This would help isolate the problem if you'd help doing this part on your side.

[pulsar@itomdipulsar-bastion-0 pulsar]$ cat /etc/resolv.conf
search op.svc.cluster.local svc.cluster.local cluster.local
nameserver 172.17.0.78
options ndots:5

Is this the DNS configuration on the client side?

Regarding the problem itself, it could be useful to share more logs (it's fine to sanitize before sharing) as a GitHub gist. The previously shared logs are too brief.

[sindhushreem25]

@lhotari with 4.0.3 client also we are seeing same behaviour .

i have added the logs and sample program in the gist .

[lhotari]

@lhotari with 4.0.3 client also we are seeing same behaviour .

i have added the logs and sample program in the gist .

@sindhushreem25 did you share the gist URL somewhere? You can also share privately by sending the URL by email to lhotari@apache.org or on Pulsar Slack in a DM if you prefer to limit the audience of the logs.

[sindhushreem25]

@lhotari i have shared the github gist in the slack channel with you .

[lhotari]

@sindhushreem25 You should use pulsar+ssl://pulsar-proxy.op.svc.cluster.local.:6651 as the serviceUrl in your client to avoid this issue. That will prevent all unnecessary DNS lookups on the client side. I hope that helps.
I haven't found any change on Netty side triggering this behavior. Using an absolute FQDN name will fix the issue. An absolute FQDN name is one that ends in the root domain, which is represented by the trailing dot. You can find more information about FQDN and the trailing dot here or here.

[lhotari]

Adding the trailing dot isn't compatible when tls hostname verification is enabled. One possible solution is to use a separate API in Pulsar client to configure the Netty DNS resolver with ndots(0) or setting the search domains to an empty list.
One example of using the API (not specific to this use case)

pulsar/pulsar-client/src/test/java/org/apache/pulsar/client/impl/PulsarClientSharedResourcesBuilderImplTest.java

Lines 115 to 135 in 8559b68

	public void testDnsResolverConfig() throws PulsarClientException {
	PulsarClientSharedResources sharedResources =
	PulsarClientSharedResources.builder()
	.shareConfigured()
	.configureDnsResolver(dnsResolverConfig -> {
	dnsResolverConfig
	.localAddress(new InetSocketAddress(0))
	.serverAddresses(List.of(new InetSocketAddress("8.8.8.8", 53)))
	.minTtl(0)
	.maxTtl(45)
	.negativeTtl(10)
	.ndots(1)
	.searchDomains(List.of("mycompany.com"))
	.queryTimeoutMillis(5000L)
	.tcpFallbackEnabled(true)
	.tcpFallbackOnTimeoutEnabled(false)
	.traceEnabled(false);
	}).build();
	runClientsWithSharedResources(sharedResources, 2, false);
	sharedResources.close();
	}