Issue with TLS after upgrading from 2.8.x #23408

duncaan · 2024-10-07T18:46:17Z

duncaan
Oct 7, 2024

Hello,
I recently upgraded our staging cluster from 2.8 to 3.0.6. The majority of components seem to work just fine, but the toolset and pulsar-manager do not seem to be working with TLS enabled. If I disable this in the configmap, I'm able to do whatever I would normally like to do, but upon enabling, I get errors like this:

pulsar@pulsar-staging-toolset-0:/pulsar$ bin/pulsar-admin namespaces list mi
2024-10-07T18:43:37,978+0000 [main] DEBUG org.apache.pulsar.client.admin.PulsarAdmin - created: serviceUrl=https://pulsar-staging-proxy:443/, authMethodName=token
2024-10-07T18:43:38,568+0000 [main] DEBUG org.apache.pulsar.common.util.SecurityUtility - Found and Instantiated Bouncy Castle provider in classpath BC
2024-10-07T18:43:38,625+0000 [main] DEBUG org.apache.pulsar.common.util.SecurityUtility - Added security provider 'Conscrypt' from class org.conscrypt.OpenSSLProvider
2024-10-07T18:43:38,635+0000 [main] DEBUG io.netty.util.internal.logging.InternalLoggerFactory - Using SLF4J as the default logging framework
2024-10-07T18:43:38,657+0000 [main] DEBUG io.netty.util.ResourceLeakDetector - -Dio.netty.leakDetection.level: disabled
2024-10-07T18:43:38,657+0000 [main] DEBUG io.netty.util.ResourceLeakDetector - -Dio.netty.leakDetection.targetRecords: 4
2024-10-07T18:43:38,677+0000 [main] DEBUG io.netty.util.internal.PlatformDependent0 - -Dio.netty.noUnsafe: false
2024-10-07T18:43:38,677+0000 [main] DEBUG io.netty.util.internal.PlatformDependent0 - Java version: 17
2024-10-07T18:43:38,680+0000 [main] DEBUG io.netty.util.internal.PlatformDependent0 - sun.misc.Unsafe.theUnsafe: available
2024-10-07T18:43:38,681+0000 [main] DEBUG io.netty.util.internal.PlatformDependent0 - sun.misc.Unsafe base methods: all available
2024-10-07T18:43:38,681+0000 [main] DEBUG io.netty.util.internal.PlatformDependent0 - sun.misc.Unsafe.storeFence: available
2024-10-07T18:43:38,682+0000 [main] DEBUG io.netty.util.internal.PlatformDependent0 - java.nio.Buffer.address: available
2024-10-07T18:43:38,683+0000 [main] DEBUG io.netty.util.internal.PlatformDependent0 - direct buffer constructor: unavailable: Reflective setAccessible(true) disabled
2024-10-07T18:43:38,684+0000 [main] DEBUG io.netty.util.internal.PlatformDependent0 - java.nio.Bits.unaligned: available, true
2024-10-07T18:43:38,685+0000 [main] DEBUG io.netty.util.internal.PlatformDependent0 - jdk.internal.misc.Unsafe.allocateUninitializedArray(int): unavailable: class io.netty.util.internal.PlatformDependent0$7 cannot access class jdk.internal.misc.Unsafe (in module java.base) because module java.base does not export jdk.internal.misc to unnamed module @323b36e0
2024-10-07T18:43:38,690+0000 [main] DEBUG io.netty.util.internal.PlatformDependent0 - java.nio.DirectByteBuffer.<init>(long, {int,long}): unavailable
2024-10-07T18:43:38,690+0000 [main] DEBUG io.netty.util.internal.PlatformDependent - sun.misc.Unsafe: available
2024-10-07T18:43:38,690+0000 [main] DEBUG io.netty.util.internal.PlatformDependent - -Dio.netty.tmpdir: /tmp (java.io.tmpdir)
2024-10-07T18:43:38,691+0000 [main] DEBUG io.netty.util.internal.PlatformDependent - -Dio.netty.bitMode: 64 (sun.arch.data.model)
2024-10-07T18:43:38,692+0000 [main] DEBUG io.netty.util.internal.PlatformDependent - -Dio.netty.maxDirectMemory: -1 bytes
2024-10-07T18:43:38,692+0000 [main] DEBUG io.netty.util.internal.PlatformDependent - -Dio.netty.uninitializedArrayAllocationThreshold: -1
2024-10-07T18:43:38,693+0000 [main] DEBUG io.netty.util.internal.CleanerJava9 - java.nio.ByteBuffer.cleaner(): available
2024-10-07T18:43:38,693+0000 [main] DEBUG io.netty.util.internal.PlatformDependent - -Dio.netty.noPreferDirect: false
2024-10-07T18:43:38,702+0000 [main] DEBUG io.netty.buffer.AbstractByteBuf - -Dio.netty.buffer.checkAccessible: true
2024-10-07T18:43:38,702+0000 [main] DEBUG io.netty.buffer.AbstractByteBuf - -Dio.netty.buffer.checkBounds: true
2024-10-07T18:43:38,703+0000 [main] DEBUG io.netty.util.ResourceLeakDetectorFactory - Loaded default ResourceLeakDetector: io.netty.util.ResourceLeakDetector@78010562
2024-10-07T18:43:38,708+0000 [main] DEBUG io.netty.util.internal.InternalThreadLocalMap - -Dio.netty.threadLocalMap.stringBuilder.initialSize: 1024
2024-10-07T18:43:38,708+0000 [main] DEBUG io.netty.util.internal.InternalThreadLocalMap - -Dio.netty.threadLocalMap.stringBuilder.maxSize: 4096
2024-10-07T18:43:38,712+0000 [main] DEBUG io.netty.buffer.PooledByteBufAllocator - -Dio.netty.allocator.numHeapArenas: 4
2024-10-07T18:43:38,713+0000 [main] DEBUG io.netty.buffer.PooledByteBufAllocator - -Dio.netty.allocator.numDirectArenas: 4
2024-10-07T18:43:38,713+0000 [main] DEBUG io.netty.buffer.PooledByteBufAllocator - -Dio.netty.allocator.pageSize: 8192
2024-10-07T18:43:38,714+0000 [main] DEBUG io.netty.buffer.PooledByteBufAllocator - -Dio.netty.allocator.maxOrder: 9
2024-10-07T18:43:38,714+0000 [main] DEBUG io.netty.buffer.PooledByteBufAllocator - -Dio.netty.allocator.chunkSize: 4194304
2024-10-07T18:43:38,715+0000 [main] DEBUG io.netty.buffer.PooledByteBufAllocator - -Dio.netty.allocator.smallCacheSize: 256
2024-10-07T18:43:38,715+0000 [main] DEBUG io.netty.buffer.PooledByteBufAllocator - -Dio.netty.allocator.normalCacheSize: 64
2024-10-07T18:43:38,716+0000 [main] DEBUG io.netty.buffer.PooledByteBufAllocator - -Dio.netty.allocator.maxCachedBufferCapacity: 32768
2024-10-07T18:43:38,716+0000 [main] DEBUG io.netty.buffer.PooledByteBufAllocator - -Dio.netty.allocator.cacheTrimInterval: 8192
2024-10-07T18:43:38,717+0000 [main] DEBUG io.netty.buffer.PooledByteBufAllocator - -Dio.netty.allocator.cacheTrimIntervalMillis: 0
2024-10-07T18:43:38,717+0000 [main] DEBUG io.netty.buffer.PooledByteBufAllocator - -Dio.netty.allocator.useCacheForAllThreads: false
2024-10-07T18:43:38,718+0000 [main] DEBUG io.netty.buffer.PooledByteBufAllocator - -Dio.netty.allocator.maxCachedByteBuffersPerChunk: 1023
2024-10-07T18:43:38,718+0000 [main] DEBUG io.netty.buffer.PooledByteBufAllocator - -Dio.netty.allocator.disableCacheFinalizersForFastThreadLocalThreads: false
2024-10-07T18:43:38,726+0000 [main] DEBUG io.netty.buffer.ByteBufUtil - -Dio.netty.allocator.type: pooled
2024-10-07T18:43:38,726+0000 [main] DEBUG io.netty.buffer.ByteBufUtil - -Dio.netty.threadLocalDirectBufferSize: 0
2024-10-07T18:43:38,727+0000 [main] DEBUG io.netty.buffer.ByteBufUtil - -Dio.netty.maxThreadLocalCharBufferSize: 16384
2024-10-07T18:43:38,737+0000 [main] DEBUG io.netty.util.internal.NativeLibraryLoader - -Dio.netty.native.workdir: /tmp (io.netty.tmpdir)
2024-10-07T18:43:38,738+0000 [main] DEBUG io.netty.util.internal.NativeLibraryLoader - -Dio.netty.native.deleteLibAfterLoading: true
2024-10-07T18:43:38,738+0000 [main] DEBUG io.netty.util.internal.NativeLibraryLoader - -Dio.netty.native.tryPatchShadedId: true
2024-10-07T18:43:38,739+0000 [main] DEBUG io.netty.util.internal.NativeLibraryLoader - -Dio.netty.native.detectNativeLibraryDuplicates: true
2024-10-07T18:43:38,767+0000 [main] DEBUG io.netty.util.internal.NativeLibraryLoader - Successfully loaded the library /tmp/libnetty_tcnative_linux_x86_648423914568691120844.so
2024-10-07T18:43:38,768+0000 [main] DEBUG io.netty.util.internal.NativeLibraryLoader - Loaded library with name 'netty_tcnative_linux_x86_64'
2024-10-07T18:43:38,768+0000 [main] DEBUG io.netty.handler.ssl.OpenSsl - Initialize netty-tcnative using engine: 'default'
2024-10-07T18:43:38,769+0000 [main] DEBUG io.netty.handler.ssl.OpenSsl - netty-tcnative using native library: BoringSSL
2024-10-07T18:43:38,904+0000 [main] DEBUG io.netty.util.ResourceLeakDetectorFactory - Loaded default ResourceLeakDetector: io.netty.util.ResourceLeakDetector@237f7970
2024-10-07T18:43:38,909+0000 [main] DEBUG io.netty.util.Recycler - -Dio.netty.recycler.maxCapacityPerThread: 4096
2024-10-07T18:43:38,909+0000 [main] DEBUG io.netty.util.Recycler - -Dio.netty.recycler.ratio: 8
2024-10-07T18:43:38,909+0000 [main] DEBUG io.netty.util.Recycler - -Dio.netty.recycler.chunkSize: 32
2024-10-07T18:43:38,909+0000 [main] DEBUG io.netty.util.Recycler - -Dio.netty.recycler.blocking: false
2024-10-07T18:43:38,910+0000 [main] DEBUG io.netty.util.Recycler - -Dio.netty.recycler.batchFastThreadLocalOnly: true
2024-10-07T18:43:38,916+0000 [main] DEBUG io.netty.util.internal.PlatformDependent - org.jctools-core.MpscChunkedArrayQueue: available
2024-10-07T18:43:38,926+0000 [main] DEBUG io.netty.handler.ssl.CipherSuiteConverter - Cipher suite mapping: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 => ECDHE-ECDSA-AES128-GCM-SHA256
2024-10-07T18:43:38,926+0000 [main] DEBUG io.netty.handler.ssl.CipherSuiteConverter - Cipher suite mapping: SSL_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 => ECDHE-ECDSA-AES128-GCM-SHA256
2024-10-07T18:43:38,927+0000 [main] DEBUG io.netty.handler.ssl.CipherSuiteConverter - Cipher suite mapping: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 => ECDHE-RSA-AES128-GCM-SHA256
2024-10-07T18:43:38,927+0000 [main] DEBUG io.netty.handler.ssl.CipherSuiteConverter - Cipher suite mapping: SSL_ECDHE_RSA_WITH_AES_128_GCM_SHA256 => ECDHE-RSA-AES128-GCM-SHA256
2024-10-07T18:43:38,927+0000 [main] DEBUG io.netty.handler.ssl.CipherSuiteConverter - Cipher suite mapping: TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 => ECDHE-ECDSA-AES256-GCM-SHA384
2024-10-07T18:43:38,927+0000 [main] DEBUG io.netty.handler.ssl.CipherSuiteConverter - Cipher suite mapping: SSL_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 => ECDHE-ECDSA-AES256-GCM-SHA384
2024-10-07T18:43:38,927+0000 [main] DEBUG io.netty.handler.ssl.CipherSuiteConverter - Cipher suite mapping: TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 => ECDHE-RSA-AES256-GCM-SHA384
2024-10-07T18:43:38,928+0000 [main] DEBUG io.netty.handler.ssl.CipherSuiteConverter - Cipher suite mapping: SSL_ECDHE_RSA_WITH_AES_256_GCM_SHA384 => ECDHE-RSA-AES256-GCM-SHA384
2024-10-07T18:43:38,928+0000 [main] DEBUG io.netty.handler.ssl.CipherSuiteConverter - Cipher suite mapping: TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256 => ECDHE-ECDSA-CHACHA20-POLY1305
2024-10-07T18:43:38,928+0000 [main] DEBUG io.netty.handler.ssl.CipherSuiteConverter - Cipher suite mapping: SSL_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256 => ECDHE-ECDSA-CHACHA20-POLY1305
2024-10-07T18:43:38,929+0000 [main] DEBUG io.netty.handler.ssl.CipherSuiteConverter - Cipher suite mapping: TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256 => ECDHE-RSA-CHACHA20-POLY1305
2024-10-07T18:43:38,929+0000 [main] DEBUG io.netty.handler.ssl.CipherSuiteConverter - Cipher suite mapping: SSL_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256 => ECDHE-RSA-CHACHA20-POLY1305
2024-10-07T18:43:38,929+0000 [main] DEBUG io.netty.handler.ssl.CipherSuiteConverter - Cipher suite mapping: TLS_ECDHE_PSK_WITH_CHACHA20_POLY1305_SHA256 => ECDHE-PSK-CHACHA20-POLY1305
2024-10-07T18:43:38,929+0000 [main] DEBUG io.netty.handler.ssl.CipherSuiteConverter - Cipher suite mapping: SSL_ECDHE_PSK_WITH_CHACHA20_POLY1305_SHA256 => ECDHE-PSK-CHACHA20-POLY1305
2024-10-07T18:43:38,930+0000 [main] DEBUG io.netty.handler.ssl.CipherSuiteConverter - Cipher suite mapping: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA => ECDHE-ECDSA-AES128-SHA
2024-10-07T18:43:38,930+0000 [main] DEBUG io.netty.handler.ssl.CipherSuiteConverter - Cipher suite mapping: SSL_ECDHE_ECDSA_WITH_AES_128_CBC_SHA => ECDHE-ECDSA-AES128-SHA
2024-10-07T18:43:38,930+0000 [main] DEBUG io.netty.handler.ssl.CipherSuiteConverter - Cipher suite mapping: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA => ECDHE-RSA-AES128-SHA
2024-10-07T18:43:38,930+0000 [main] DEBUG io.netty.handler.ssl.CipherSuiteConverter - Cipher suite mapping: SSL_ECDHE_RSA_WITH_AES_128_CBC_SHA => ECDHE-RSA-AES128-SHA
2024-10-07T18:43:38,931+0000 [main] DEBUG io.netty.handler.ssl.CipherSuiteConverter - Cipher suite mapping: TLS_ECDHE_PSK_WITH_AES_128_CBC_SHA => ECDHE-PSK-AES128-CBC-SHA
2024-10-07T18:43:38,931+0000 [main] DEBUG io.netty.handler.ssl.CipherSuiteConverter - Cipher suite mapping: SSL_ECDHE_PSK_WITH_AES_128_CBC_SHA => ECDHE-PSK-AES128-CBC-SHA
2024-10-07T18:43:38,931+0000 [main] DEBUG io.netty.handler.ssl.CipherSuiteConverter - Cipher suite mapping: TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA => ECDHE-ECDSA-AES256-SHA
2024-10-07T18:43:38,931+0000 [main] DEBUG io.netty.handler.ssl.CipherSuiteConverter - Cipher suite mapping: SSL_ECDHE_ECDSA_WITH_AES_256_CBC_SHA => ECDHE-ECDSA-AES256-SHA
2024-10-07T18:43:38,932+0000 [main] DEBUG io.netty.handler.ssl.CipherSuiteConverter - Cipher suite mapping: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA => ECDHE-RSA-AES256-SHA
2024-10-07T18:43:38,932+0000 [main] DEBUG io.netty.handler.ssl.CipherSuiteConverter - Cipher suite mapping: SSL_ECDHE_RSA_WITH_AES_256_CBC_SHA => ECDHE-RSA-AES256-SHA
2024-10-07T18:43:38,932+0000 [main] DEBUG io.netty.handler.ssl.CipherSuiteConverter - Cipher suite mapping: TLS_ECDHE_PSK_WITH_AES_256_CBC_SHA => ECDHE-PSK-AES256-CBC-SHA
2024-10-07T18:43:38,932+0000 [main] DEBUG io.netty.handler.ssl.CipherSuiteConverter - Cipher suite mapping: SSL_ECDHE_PSK_WITH_AES_256_CBC_SHA => ECDHE-PSK-AES256-CBC-SHA
2024-10-07T18:43:38,933+0000 [main] DEBUG io.netty.handler.ssl.CipherSuiteConverter - Cipher suite mapping: TLS_RSA_WITH_AES_128_GCM_SHA256 => AES128-GCM-SHA256
2024-10-07T18:43:38,933+0000 [main] DEBUG io.netty.handler.ssl.CipherSuiteConverter - Cipher suite mapping: SSL_RSA_WITH_AES_128_GCM_SHA256 => AES128-GCM-SHA256
2024-10-07T18:43:38,933+0000 [main] DEBUG io.netty.handler.ssl.CipherSuiteConverter - Cipher suite mapping: TLS_RSA_WITH_AES_256_GCM_SHA384 => AES256-GCM-SHA384
2024-10-07T18:43:38,933+0000 [main] DEBUG io.netty.handler.ssl.CipherSuiteConverter - Cipher suite mapping: SSL_RSA_WITH_AES_256_GCM_SHA384 => AES256-GCM-SHA384
2024-10-07T18:43:38,933+0000 [main] DEBUG io.netty.handler.ssl.CipherSuiteConverter - Cipher suite mapping: TLS_RSA_WITH_AES_128_CBC_SHA => AES128-SHA
2024-10-07T18:43:38,933+0000 [main] DEBUG io.netty.handler.ssl.CipherSuiteConverter - Cipher suite mapping: SSL_RSA_WITH_AES_128_CBC_SHA => AES128-SHA
2024-10-07T18:43:38,933+0000 [main] DEBUG io.netty.handler.ssl.CipherSuiteConverter - Cipher suite mapping: TLS_PSK_WITH_AES_128_CBC_SHA => PSK-AES128-CBC-SHA
2024-10-07T18:43:38,933+0000 [main] DEBUG io.netty.handler.ssl.CipherSuiteConverter - Cipher suite mapping: SSL_PSK_WITH_AES_128_CBC_SHA => PSK-AES128-CBC-SHA
2024-10-07T18:43:38,934+0000 [main] DEBUG io.netty.handler.ssl.CipherSuiteConverter - Cipher suite mapping: TLS_RSA_WITH_AES_256_CBC_SHA => AES256-SHA
2024-10-07T18:43:38,934+0000 [main] DEBUG io.netty.handler.ssl.CipherSuiteConverter - Cipher suite mapping: SSL_RSA_WITH_AES_256_CBC_SHA => AES256-SHA
2024-10-07T18:43:38,934+0000 [main] DEBUG io.netty.handler.ssl.CipherSuiteConverter - Cipher suite mapping: TLS_PSK_WITH_AES_256_CBC_SHA => PSK-AES256-CBC-SHA
2024-10-07T18:43:38,934+0000 [main] DEBUG io.netty.handler.ssl.CipherSuiteConverter - Cipher suite mapping: SSL_PSK_WITH_AES_256_CBC_SHA => PSK-AES256-CBC-SHA
2024-10-07T18:43:38,934+0000 [main] DEBUG io.netty.handler.ssl.OpenSsl - Supported protocols (OpenSSL): [SSLv2Hello, TLSv1, TLSv1.1, TLSv1.2, TLSv1.3]
2024-10-07T18:43:38,934+0000 [main] DEBUG io.netty.handler.ssl.OpenSsl - Default cipher suites (OpenSSL): [TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_AES_128_GCM_SHA256, TLS_AES_256_GCM_SHA384, TLS_AES_128_GCM_SHA256, TLS_AES_256_GCM_SHA384, TLS_AES_128_GCM_SHA256, TLS_AES_256_GCM_SHA384, TLS_CHACHA20_POLY1305_SHA256]
2024-10-07T18:43:38,957+0000 [main] DEBUG io.netty.util.ResourceLeakDetectorFactory - Loaded default ResourceLeakDetector: io.netty.util.ResourceLeakDetector@4993febc
2024-10-07T18:43:38,976+0000 [main] DEBUG io.netty.util.concurrent.GlobalEventExecutor - -Dio.netty.globalEventExecutor.quietPeriodSeconds: 1
2024-10-07T18:43:38,992+0000 [main] DEBUG io.netty.channel.MultithreadEventLoopGroup - -Dio.netty.eventLoopThreads: 4
2024-10-07T18:43:39,002+0000 [main] DEBUG io.netty.channel.nio.NioEventLoop - -Dio.netty.noKeySetOptimization: false
2024-10-07T18:43:39,002+0000 [main] DEBUG io.netty.channel.nio.NioEventLoop - -Dio.netty.selectorAutoRebuildThreshold: 512
2024-10-07T18:43:39,646+0000 [jersey-client-async-executor-0] DEBUG io.netty.channel.DefaultChannelId - -Dio.netty.processId: 325 (auto-detected)
2024-10-07T18:43:39,649+0000 [jersey-client-async-executor-0] DEBUG io.netty.util.NetUtil - -Djava.net.preferIPv4Stack: true
2024-10-07T18:43:39,649+0000 [jersey-client-async-executor-0] DEBUG io.netty.util.NetUtil - -Djava.net.preferIPv6Addresses: false
2024-10-07T18:43:39,650+0000 [jersey-client-async-executor-0] DEBUG io.netty.util.NetUtilInitializations - Loopback interface: lo (lo, 127.0.0.1)
2024-10-07T18:43:39,652+0000 [jersey-client-async-executor-0] DEBUG io.netty.util.NetUtil - /proc/sys/net/core/somaxconn: 4096
2024-10-07T18:43:39,653+0000 [jersey-client-async-executor-0] DEBUG io.netty.channel.DefaultChannelId - -Dio.netty.machineId: ca:57:f4:ff:fe:3c:51:a2 (auto-detected)
2024-10-07T18:43:39,676+0000 [jersey-client-async-executor-0] DEBUG io.netty.bootstrap.ChannelInitializerExtensions - -Dio.netty.bootstrap.extensions: null
2024-10-07T18:43:39,714+0000 [AsyncHttpClient-7-1] DEBUG io.netty.util.ResourceLeakDetectorFactory - Loaded default ResourceLeakDetector: io.netty.util.ResourceLeakDetector@7a2213ea
2024-10-07T18:43:39,770+0000 [AsyncHttpClient-7-1] DEBUG io.netty.handler.ssl.ReferenceCountedOpenSslContext - verification of certificate failed
sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:439) ~[?:?]
	at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:306) ~[?:?]
	at sun.security.validator.Validator.validate(Validator.java:264) ~[?:?]
	at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:285) ~[?:?]
	at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:144) ~[?:?]
	at io.netty.handler.ssl.EnhancingX509ExtendedTrustManager.checkServerTrusted(EnhancingX509ExtendedTrustManager.java:69) ~[io.netty-netty-handler-4.1.113.Final.jar:4.1.113.Final]
	at io.netty.handler.ssl.ReferenceCountedOpenSslClientContext$ExtendedTrustManagerVerifyCallback.verify(ReferenceCountedOpenSslClientContext.java:235) ~[io.netty-netty-handler-4.1.113.Final.jar:4.1.113.Final]
	at io.netty.handler.ssl.ReferenceCountedOpenSslContext$AbstractCertificateVerifier.verify(ReferenceCountedOpenSslContext.java:801) ~[io.netty-netty-handler-4.1.113.Final.jar:4.1.113.Final]
	at io.netty.internal.tcnative.CertificateVerifierTask.runTask(CertificateVerifierTask.java:36) ~[io.netty-netty-tcnative-classes-2.0.66.Final.jar:2.0.66.Final]
	at io.netty.internal.tcnative.SSLTask.run(SSLTask.java:48) ~[io.netty-netty-tcnative-classes-2.0.66.Final.jar:2.0.66.Final]
	at io.netty.internal.tcnative.SSLTask.run(SSLTask.java:42) ~[io.netty-netty-tcnative-classes-2.0.66.Final.jar:2.0.66.Final]
	at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.runAndResetNeedTask(ReferenceCountedOpenSslEngine.java:1538) ~[io.netty-netty-handler-4.1.113.Final.jar:4.1.113.Final]
	at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.access$700(ReferenceCountedOpenSslEngine.java:94) ~[io.netty-netty-handler-4.1.113.Final.jar:4.1.113.Final]
	at io.netty.handler.ssl.ReferenceCountedOpenSslEngine$TaskDecorator.run(ReferenceCountedOpenSslEngine.java:1510) ~[io.netty-netty-handler-4.1.113.Final.jar:4.1.113.Final]
	at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1671) ~[io.netty-netty-handler-4.1.113.Final.jar:4.1.113.Final]
	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1517) ~[io.netty-netty-handler-4.1.113.Final.jar:4.1.113.Final]
	at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1358) ~[io.netty-netty-handler-4.1.113.Final.jar:4.1.113.Final]
	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1407) ~[io.netty-netty-handler-4.1.113.Final.jar:4.1.113.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:530) ~[io.netty-netty-codec-4.1.113.Final.jar:4.1.113.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:469) ~[io.netty-netty-codec-4.1.113.Final.jar:4.1.113.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) ~[io.netty-netty-codec-4.1.113.Final.jar:4.1.113.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[io.netty-netty-transport-4.1.113.Final.jar:4.1.113.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[io.netty-netty-transport-4.1.113.Final.jar:4.1.113.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[io.netty-netty-transport-4.1.113.Final.jar:4.1.113.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1357) ~[io.netty-netty-transport-4.1.113.Final.jar:4.1.113.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) ~[io.netty-netty-transport-4.1.113.Final.jar:4.1.113.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[io.netty-netty-transport-4.1.113.Final.jar:4.1.113.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:868) ~[io.netty-netty-transport-4.1.113.Final.jar:4.1.113.Final]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[io.netty-netty-transport-4.1.113.Final.jar:4.1.113.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) ~[io.netty-netty-transport-4.1.113.Final.jar:4.1.113.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) ~[io.netty-netty-transport-4.1.113.Final.jar:4.1.113.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) ~[io.netty-netty-transport-4.1.113.Final.jar:4.1.113.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) ~[io.netty-netty-transport-4.1.113.Final.jar:4.1.113.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[io.netty-netty-common-4.1.113.Final.jar:4.1.113.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[io.netty-netty-common-4.1.113.Final.jar:4.1.113.Final]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty-netty-common-4.1.113.Final.jar:4.1.113.Final]
	at java.lang.Thread.run(Thread.java:840) ~[?:?]
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:148) ~[?:?]
	at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:129) ~[?:?]
	at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:297) ~[?:?]
	at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:434) ~[?:?]
	... 36 more

I do have a certificate on the proxy that I purchased from a registrar, so it's not the normal self-signed certificate. This worked just fine in previous versions of pulsar. Any suggestions of what might work to fix this?

lhotari · 2024-10-07T22:32:45Z

lhotari
Oct 7, 2024
Collaborator

Which 2.8 version are you upgrading from? Could this be a result of the changes in default configurations due to Pulsar security vulnerabilities detected in 2022? The fixes seem to be in Pulsar 2.8.4. Did you have that versio?

4 replies

duncaan Oct 8, 2024
Author

We were on version 2.8.2. Seems that we should enable some added security around tlsHostnameVerificationEnabled. Is there any documentation on using a commercially obtained SSL certificate with the helm chart?

duncaan Oct 8, 2024
Author

I'm seeing a similar error on the brokers too.

lhotari Oct 8, 2024
Collaborator

The Pulsar TLS docs for 3.0.x are at https://pulsar.apache.org/docs/3.0.x/security-tls-transport/. The error message hints that tlsTrustCertsFilePath isn't properly configured. For broker clients, it's brokerClientTrustCertsFilePath. The config is slightly different if you happen to use tlsEnabledWithKeyStore.

I do have a certificate on the proxy that I purchased from a registrar, so it's not the normal self-signed certificate. This worked just fine in previous versions of Pulsar. Any suggestions on what might work to fix this?

Which Helm chart version are you using? Apache Pulsar Helm chart? There's an open issue to add support for pre-existing certificates: apache/pulsar-helm-chart#527.

Regarding information security, please notice the "Important Security Disclaimer for Helm Chart Usage" on the Apache Pulsar Helm chart, as well as the related Pulsar Proxy security considerations. In addition to this, please review Apache Pulsar Security Advisories.

duncaan Oct 9, 2024
Author

I'm using the latest apache pulsar helm chart. I've tried setting brokerClientTrustCertsFilePath and tlsTrustCertsFilePath on the toolset pod, but I'm not sure it made much of a difference. I also set tlsTrustCertsFilePath to the CA provided by the SSL certificate provider. I'm really unsure where to go from here.
Here's my helm values:


namespace: ""
namespaceCreate: false
nameOverride: pulsar
## clusterDomain as defined for your k8s cluster
clusterDomain: cluster.local
###
### Global Settings
###

## Set to true on install
initialize: false
## Set cluster name
# clusterName:

## add custom labels to components of cluster
# labels:
#   environment: dev
#   customer: apache


metadataPrefix: ""

tcpPrefix: "" # For Istio this will be "tcp-"
tlsPrefix: "" # For Istio this will be "tls-"

## Deprecated in favor of using `volumes.persistence`
persistence: true
## Volume settings
volumes:
  persistence: true
  # configure the components to use local persistent volume
  # the local provisioner should be installed prior to enable local persistent volume
  local_storage: false
## RBAC
##
## Configure settings related to RBAC such as limiting broker access to single
## namespece or enabling PSP
rbac:
  enabled: false
  psp: false
  limit_to_namespace: false
affinity:
  anti_affinity: true
  # Set the anti affinity type. Valid values:
  # requiredDuringSchedulingIgnoredDuringExecution - rules must be met for pod to be scheduled (hard) requires at least one node per replica
  # preferredDuringSchedulingIgnoredDuringExecution - scheduler will try to enforce but not guranentee
  type: requiredDuringSchedulingIgnoredDuringExecution
kube-prometheus-stack:
  enabled: false
  prometheusOperator:
    enabled: false
  grafana:
    enabled: false
  alertmanager:
    enabled: false
  prometheus:
    enabled: false
## Components
##
## Control what components of Apache Pulsar to deploy for the cluster
components:
  # zookeeper
  zookeeper: true
  # bookkeeper
  bookkeeper: true
  # Bookkeeper Autorecovery
  autorecovery: true
  # broker
  broker: true
  # functions
  functions: true
  # proxy
  proxy: true
  # toolset
  toolset: true
  # pulsar manager
  pulsar_manager: true
# Disable in-cluster prometheus, grafana, alert-manager, node_exporter
monitoring:
  # monitoring - prometheus
  prometheus: false
  # monitoring - grafana
  grafana: false
  # monitoring - node_exporter
  node_exporter: false
  # alerting - alert-manager
  alert_manager: false

## Images
##
## Control what images to use for each component
images:
  zookeeper:
    tag: 3.0.7
  bookie:
    tag: 3.0.7
  autorecovery:
    tag: 3.0.7
  broker:
    tag: 3.0.7
  proxy:
    tag: 3.0.7
  functions:
    tag: 3.0.7
  pulsar_manager:
    tag: v0.4.0
    hasCommand: false
## TLS
## templates/tls-certs.yaml
##
## The chart is using cert-manager for provisioning TLS certs for
## brokers and proxies.
tls:
  enabled: true
  ca_suffix: ca-tls
  # common settings for generating certs
  common:
    # 365 days
    duration: 8760h0m0s
    # 30 days
    renewBefore: 720h0m0s
    organization:
      - pulsar
    keySize: 4096
    keyAlgorithm: rsa
    keyEncoding: pkcs8
  # settings for generating certs for proxy
  proxy:
    enabled: true
    cert_name: tls
  # settings for generating certs for broker
  broker:
    enabled: true
    cert_name: tls-broker
  # settings for generating certs for bookies
  bookie:
    enabled: true
    cert_name: tls-bookie
  # settings for generating certs for zookeeper
  zookeeper:
    enabled: true
    cert_name: tls-zookeeper
  # settings for generating certs for recovery
  autorecovery:
    cert_name: tls-recovery
  # settings for generating certs for toolset
  toolset:
    cert_name: tls-toolset
# Enable or disable broker authentication and authorization.
auth:
  authentication:
    enabled: true
    provider: "jwt"
    jwt:
      # Enable JWT authentication
      # If the token is generated by a secret key, set the usingSecretKey as true.
      # If the token is generated by a private key, set the usingSecretKey as false.
      usingSecretKey: false
  authorization:
    enabled: true
  superUsers:
    # broker to broker communication
    broker: "broker-admin"
    # proxy to broker communication
    proxy: "proxy-admin"
    # pulsar-admin client to broker/proxy communication
    client: "admin"
    manager: "admin"
######################################################################
# External dependencies
######################################################################

## cert-manager
## templates/tls-cert-issuer.yaml
##
## Cert manager is used for automatically provisioning TLS certificates
## for components within a Pulsar cluster
certs:
  internal_issuer:
    apiVersion: cert-manager.io/v1
    enabled: true
    component: internal-cert-issuer
    type: selfsigning
    # 365 days
    duration: 8760h0m0s
    # 30 days
    renewBefore: 720h0m0s
  issuers:
    selfsigning:
######################################################################
# Below are settings for each component
######################################################################

## Pulsar: Zookeeper cluster
## templates/zookeeper-statefulset.yaml
##
zookeeper:
  # use a component name that matches your grafana configuration
  # so the metrics are correctly rendered in grafana dashboard
  component: zookeeper
  # the number of zookeeper servers to run. it should be an odd number larger than or equal to 3.
  replicaCount: 3
  updateStrategy:
    type: RollingUpdate
  podManagementPolicy: OrderedReady
  # If using Prometheus-Operator enable this PodMonitor to discover zookeeper scrape targets
  # Prometheus-Operator does not add scrape targets based on k8s annotations
  podMonitor:
    enabled: false
    interval: 10s
    scrapeTimeout: 10s
  # True includes annotation for statefulset that contains hash of corresponding configmap, which will cause pods to restart on configmap change
  restartPodsOnConfigMapChange: false
  ports:
    http: 8000
    client: 2181
    clientTls: 2281
    follower: 2888
    leaderElection: 3888
  # nodeSelector:
  # cloud.google.com/gke-nodepool: default-pool
  probe:
    liveness:
      enabled: true
      failureThreshold: 10
      initialDelaySeconds: 20
      periodSeconds: 30
      timeoutSeconds: 30
    readiness:
      enabled: true
      failureThreshold: 10
      initialDelaySeconds: 20
      periodSeconds: 30
      timeoutSeconds: 30
    startup:
      enabled: false
      failureThreshold: 30
      initialDelaySeconds: 20
      periodSeconds: 30
      timeoutSeconds: 30
  affinity:
    anti_affinity: true
    # Set the anti affinity type. Valid values:
    # requiredDuringSchedulingIgnoredDuringExecution - rules must be met for pod to be scheduled (hard) requires at least one node per replica
    # preferredDuringSchedulingIgnoredDuringExecution - scheduler will try to enforce but not guranentee
    type: requiredDuringSchedulingIgnoredDuringExecution
  annotations:
    prometheus.io/scrape: "true"
    prometheus.io/port: "8000"
  tolerations:
    - key: "service"
      operator: "Equal"
      value: "pulsar"
      effect: "NoSchedule"
  nodeSelector:
    service: "pulsar"
  gracePeriod: 30
  resources:
    requests:
      memory: 256Mi
      cpu: 0.1
  # extraVolumes and extraVolumeMounts allows you to mount other volumes
  # Example Use Case: mount ssl certificates
  # extraVolumes:
  #   - name: ca-certs
  #     secret:
  #       defaultMode: 420
  #       secretName: ca-certs
  # extraVolumeMounts:
  #   - mountPath: /pulsar/custom/log4j2.yaml
  #     name: logger-config
  #     subPath: log4j2.yaml
  extraVolumes: []
  extraVolumeMounts: []
  # Ensures 2.10.0 non-root docker image works correctly.
  securityContext:
    fsGroup: 0
    fsGroupChangePolicy: "OnRootMismatch"
  volumes:
    # use a persistent volume or emptyDir
    persistence: true
    data:
      name: data
      size: 20Gi
      local_storage: false
      ## If you already have an existent storage class and want to reuse it, you can specify its name with the option below
      ##
      storageClassName: pulsar-standard
      #
      ## Instead if you want to create a new storage class define it below
      ## If left undefined no storage class will be defined along with PVC
      ##
      # storageClass:
      # type: pd-ssd
      # fsType: xfs
      # provisioner: kubernetes.io/gce-pd
    useSingleCommonVolume: true
  ## Zookeeper configmap
  ## templates/zookeeper-configmap.yaml
  ##
  configData:
    PULSAR_MEM: >
      -Xms64m -Xmx128m

    PULSAR_GC: >
      -XX:+UseG1GC -XX:MaxGCPauseMillis=10 -Dcom.sun.management.jmxremote -Djute.maxbuffer=10485760 -XX:+ParallelRefProcEnabled -XX:+UnlockExperimentalVMOptions -XX:+DoEscapeAnalysis -XX:+DisableExplicitGC -XX:+ExitOnOutOfMemoryError -XX:+PerfDisableSharedMem

  ## Add a custom command to the start up process of the zookeeper pods (e.g. update-ca-certificates, jvm commands, etc)
  additionalCommand:
  ## Zookeeper service
  ## templates/zookeeper-service.yaml
  ##
  service:
    annotations:
      service.alpha.kubernetes.io/tolerate-unready-endpoints: "true"
  ## Zookeeper PodDisruptionBudget
  ## templates/zookeeper-pdb.yaml
  ##
  pdb:
    usePolicy: true
    maxUnavailable: 1
  customLoggingEnabled: true
## Pulsar: Bookkeeper cluster
## templates/bookkeeper-statefulset.yaml
##
bookkeeper:
  # use a component name that matches your grafana configuration
  # so the metrics are correctly rendered in grafana dashboard
  component: bookie
  ## BookKeeper Cluster Initialize
  ## templates/bookkeeper-cluster-initialize.yaml
  metadata:
    ## Set the resources used for running `bin/bookkeeper shell initnewcluster`
    ##
    resources:
    # requests:
    # memory: 4Gi
    # cpu: 2
  replicaCount: 4
  updateStrategy:
    type: RollingUpdate
  podManagementPolicy: Parallel
  # If using Prometheus-Operator enable this PodMonitor to discover bookie scrape targets
  # Prometheus-Operator does not add scrape targets based on k8s annotations
  podMonitor:
    enabled: false
    interval: 10s
    scrapeTimeout: 10s
  # True includes annotation for statefulset that contains hash of corresponding configmap, which will cause pods to restart on configmap change
  restartPodsOnConfigMapChange: false
  ports:
    http: 8000
    bookie: 3181
  # nodeSelector:
  # cloud.google.com/gke-nodepool: default-pool
  probe:
    liveness:
      enabled: true
      failureThreshold: 60
      initialDelaySeconds: 10
      periodSeconds: 30
      timeoutSeconds: 5
    readiness:
      enabled: true
      failureThreshold: 60
      initialDelaySeconds: 10
      periodSeconds: 30
      timeoutSeconds: 5
    startup:
      enabled: false
      failureThreshold: 30
      initialDelaySeconds: 60
      periodSeconds: 30
      timeoutSeconds: 5
  affinity:
    anti_affinity: true
    # Set the anti affinity type. Valid values:
    # requiredDuringSchedulingIgnoredDuringExecution - rules must be met for pod to be scheduled (hard) requires at least one node per replica
    # preferredDuringSchedulingIgnoredDuringExecution - scheduler will try to enforce but not guranentee
    type: requiredDuringSchedulingIgnoredDuringExecution
  annotations: {}
  tolerations:
    - key: "service"
      operator: "Equal"
      value: "pulsar"
      effect: "NoSchedule"
  nodeSelector:
    service: "pulsar"
  gracePeriod: 30
  resources:
    requests:
      memory: 512Mi
      cpu: 0.2
  # extraVolumes and extraVolumeMounts allows you to mount other volumes
  # Example Use Case: mount ssl certificates
  # extraVolumes:
  #   - name: ca-certs
  #     secret:
  #       defaultMode: 420
  #       secretName: ca-certs
  # extraVolumeMounts:
  #   - name: ca-certs
  #     mountPath: /certs
  #     readOnly: true
  extraVolumes: []
  extraVolumeMounts: []
  # Ensures 2.10.0 non-root docker image works correctly.
  securityContext:
    fsGroup: 0
    fsGroupChangePolicy: "OnRootMismatch"
  volumes:
    # use a persistent volume or emptyDir
    persistence: true
    journal:
      name: journal
      size: 10Gi
      local_storage: false
      ## If you already have an existent storage class and want to reuse it, you can specify its name with the option below
      ##
      # storageClassName: existent-storage-class
      #
      ## Instead if you want to create a new storage class define it below
      ## If left undefined no storage class will be defined along with PVC
      ##
      # storageClass:
      # type: pd-ssd
      # fsType: xfs
      # provisioner: kubernetes.io/gce-pd
      useMultiVolumes: false
      multiVolumes:
        - name: journal0
          size: 10Gi
          # storageClassName: existent-storage-class
          mountPath: /pulsar/data/bookkeeper/journal0
        - name: journal1
          size: 10Gi
          # storageClassName: existent-storage-class
          mountPath: /pulsar/data/bookkeeper/journal1
      storageClassName: pulsar-standard
    ledgers:
      name: ledgers
      size: 50Gi
      local_storage: false
      # storageClassName:
      # storageClass:
      # ...
      useMultiVolumes: false
      multiVolumes:
        - name: ledgers0
          size: 10Gi
          # storageClassName: existent-storage-class
          mountPath: /pulsar/data/bookkeeper/ledgers0
        - name: ledgers1
          size: 10Gi
          # storageClassName: existent-storage-class
          mountPath: /pulsar/data/bookkeeper/ledgers1
      storageClassName: pulsar-standard
    ## use a single common volume for both journal and ledgers
    useSingleCommonVolume: false
    common:
      name: common
      size: 60Gi
      local_storage: true
      # storageClassName:
      # storageClass: ## this is common too
      # ...
  ## Bookkeeper configmap
  ## templates/bookkeeper-configmap.yaml
  ##
  configData:
    # we use `bin/pulsar` for starting bookie daemons
    PULSAR_MEM: >
      -Xms128m -Xmx256m -XX:MaxDirectMemorySize=256m

    PULSAR_GC: >
      -XX:+UseG1GC -XX:MaxGCPauseMillis=10 -XX:+ParallelRefProcEnabled -XX:+UnlockExperimentalVMOptions -XX:+DoEscapeAnalysis -XX:ParallelGCThreads=4 -XX:ConcGCThreads=4 -XX:G1NewSizePercent=50 -XX:+DisableExplicitGC -XX:-ResizePLAB -XX:+ExitOnOutOfMemoryError -XX:+PerfDisableSharedMem -Xlog:gc* -Xlog:gc::utctime -Xlog:safepoint -Xlog:gc+heap=trace -verbosegc

    # configure the memory settings based on jvm memory settings
    dbStorage_writeCacheMaxSizeMb: "32"
    dbStorage_readAheadCacheMaxSizeMb: "32"
    dbStorage_rocksDB_writeBufferSizeMB: "8"
    dbStorage_rocksDB_blockCacheSize: "8388608"
    PULSAR_LOG_CONF: "/pulsar/custom/log4j2.yaml"
    numAddWorkerThreads: "8" # Number of threads that should handle write requests. if zero, the writes wouldbe handled by netty threads directly.
  ## Add a custom command to the start up process of the bookie pods (e.g. update-ca-certificates, jvm commands, etc)
  additionalCommand:
  ## Bookkeeper Service
  ## templates/bookkeeper-service.yaml
  ##
  service:
    spec:
      publishNotReadyAddresses: true
  ## Bookkeeper PodDisruptionBudget
  ## templates/bookkeeper-pdb.yaml
  ##
  pdb:
    usePolicy: true
    maxUnavailable: 1
  customLoggingEnabled: true
## Pulsar: Bookkeeper AutoRecovery
## templates/autorecovery-statefulset.yaml
##
autorecovery:
  # use a component name that matches your grafana configuration
  # so the metrics are correctly rendered in grafana dashboard
  component: recovery
  replicaCount: 1
  # If using Prometheus-Operator enable this PodMonitor to discover autorecovery scrape targets
  # # Prometheus-Operator does not add scrape targets based on k8s annotations
  podMonitor:
    enabled: false
    interval: 10s
    scrapeTimeout: 10s
  # True includes annotation for statefulset that contains hash of corresponding configmap, which will cause pods to restart on configmap change
  restartPodsOnConfigMapChange: false
  ports:
    http: 8000
  # nodeSelector:
  # cloud.google.com/gke-nodepool: default-pool
  affinity:
    anti_affinity: true
    # Set the anti affinity type. Valid values:
    # requiredDuringSchedulingIgnoredDuringExecution - rules must be met for pod to be scheduled (hard) requires at least one node per replica
    # preferredDuringSchedulingIgnoredDuringExecution - scheduler will try to enforce but not guranentee
    type: requiredDuringSchedulingIgnoredDuringExecution
  annotations: {}
  # tolerations: []
  gracePeriod: 30
  resources:
    requests:
      memory: 1Gi
      cpu: 0.05
    limits:
      memory: 2Gi
  ## Bookkeeper auto-recovery configmap
  ## templates/autorecovery-configmap.yaml
  ##
  configData:
    BOOKIE_MEM: >-
      -Xms1G -Xmx1G
    PULSAR_PREFIX_useV2WireProtocol: "true"
  tolerations:
    - key: "service"
      operator: "Equal"
      value: "pulsar"
      effect: "NoSchedule"
  nodeSelector:
    service: "pulsar"
## Pulsar Zookeeper metadata. The metadata will be deployed as
## soon as the last zookeeper node is reachable. The deployment
## of other components that depends on zookeeper, such as the
## bookkeeper nodes, broker nodes, etc will only start to be
## deployed when the zookeeper cluster is ready and with the
## metadata deployed
pulsar_metadata:
  component: pulsar-init
  image:
    # the image used for running `pulsar-cluster-initialize` job
    tag: 3.0.7
    pullPolicy: IfNotPresent
  ## set an existing configuration store
  # configurationStore:
  configurationStoreMetadataPrefix: ""
  configurationStorePort: 2181
  ## optional, you can provide your own zookeeper metadata store for other components
  # to use this, you should explicit set components.zookeeper to false
  #
  # userProvidedZookeepers: "zk01.example.com:2181,zk02.example.com:2181"
# Can be used to run extra commands in the initialization jobs e.g. to quit istio sidecars etc.
extraInitCommand: ""
## Pulsar: Broker cluster
## templates/broker-statefulset.yaml
##
broker:
  # use a component name that matches your grafana configuration
  # so the metrics are correctly rendered in grafana dashboard
  component: broker
  replicaCount: 3
  autoscaling:
    enabled: false
    minReplicas: 1
    maxReplicas: 3
    metrics: ~
  # If using Prometheus-Operator enable this PodMonitor to discover broker scrape targets
  # Prometheus-Operator does not add scrape targets based on k8s annotations
  podMonitor:
    enabled: false
    interval: 10s
    scrapeTimeout: 10s
  # True includes annotation for statefulset that contains hash of corresponding configmap, which will cause pods to restart on configmap change
  restartPodsOnConfigMapChange: false
  ports:
    http: 8080
    https: 8443
    pulsar: 6650
    pulsarssl: 6651
  # nodeSelector:
  # cloud.google.com/gke-nodepool: default-pool
  probe:
    liveness:
      enabled: true
      failureThreshold: 10
      initialDelaySeconds: 30
      periodSeconds: 10
      timeoutSeconds: 5
    readiness:
      enabled: true
      failureThreshold: 10
      initialDelaySeconds: 30
      periodSeconds: 10
      timeoutSeconds: 5
    startup:
      enabled: false
      failureThreshold: 30
      initialDelaySeconds: 60
      periodSeconds: 10
      timeoutSeconds: 5
  affinity:
    anti_affinity: true
    # Set the anti affinity type. Valid values:
    # requiredDuringSchedulingIgnoredDuringExecution - rules must be met for pod to be scheduled (hard) requires at least one node per replica
    # preferredDuringSchedulingIgnoredDuringExecution - scheduler will try to enforce but not guranentee
    type: preferredDuringSchedulingIgnoredDuringExecution
  annotations: {}
  tolerations:
    - key: "service"
      operator: "Equal"
      value: "pulsar"
      effect: "NoSchedule"
  nodeSelector:
    service: "pulsar"
  gracePeriod: 30
  resources:
    requests:
      memory: "3Gi"
      cpu: 0.2
    limits:
      memory: "3Gi"
  # extraVolumes and extraVolumeMounts allows you to mount other volumes
  # Example Use Case: mount ssl certificates
  # extraVolumes:
  #   - name: ca-certs
  #     secret:
  #       defaultMode: 420
  #       secretName: ca-certs
  # extraVolumeMounts:
  #   - name: ca-certs
  #     mountPath: /certs
  #     readOnly: true
  extraVolumes: []
  extraVolumeMounts: []
  extreEnvs: []
  #    - name: POD_NAME
  #      valueFrom:
  #        fieldRef:
  #          apiVersion: v1
  #          fieldPath: metadata.name
  ## Broker configmap
  ## templates/broker-configmap.yaml
  ##
  configData:
    PULSAR_MEM: >
      -Xms1G -Xmx1G -XX:MaxDirectMemorySize=1G

    PULSAR_GC: >
      -XX:+UseG1GC -XX:MaxGCPauseMillis=10 -Dio.netty.leakDetectionLevel=disabled -Dio.netty.recycler.linkCapacity=1024 -XX:+ParallelRefProcEnabled -XX:+UnlockExperimentalVMOptions -XX:+DoEscapeAnalysis -XX:ParallelGCThreads=4 -XX:ConcGCThreads=4 -XX:G1NewSizePercent=50 -XX:+DisableExplicitGC -XX:+ExitOnOutOfMemoryError -XX:+PerfDisableSharedMem

    managedLedgerDefaultEnsembleSize: "2"
    managedLedgerDefaultWriteQuorum: "2"
    managedLedgerDefaultAckQuorum: "2"
    defaultNumberOfNamespaceBundles: "9" #should be a multiple of the number of brokers we are running
    PULSAR_PREFIX_s3ManagedLedgerOffloadRole: #redacted
    PULSAR_PREFIX_s3ManagedLedgerOffloadRoleSessionName: #redacted
    PULSAR_PREFIX_s3ManagedLedgerOffloadBucket: #redacted
    allowAutoTopicCreation: "false"
    brokerClientTlsEnabled: "true"
    brokerDeleteInactiveTopicsEnabled: "false"
    loadBalancerAutoBundleSplitEnabled: "false" # Expected default is false, however the Docker image has this set to true and it can impact stability; Need to configure number of bundles for a namespace via pulsar-admin
    loadBalancerOverrideBrokerNicSpeedGbps: "2.5"
    managedLedgerCacheSizeMB: "1024" # This is the size of the on-broker managed ledger cache - 1GB
    managedLedgerCacheEvictionTimeThresholdMillis: "60000" # default is 1000
    managedLedgerNumWorkerThreads: "6"
    maxUnackedMessagesPerConsumer: "0" # Zero here means no limit
    maxUnackedMessagesPerSubscription: "0" # Zero here means no limit
    PULSAR_LOG_CONF: "/pulsar/custom/log4j2.yaml"
    replicationConnectionsPerBroker: "6"
    replicationTlsEnabled: "true"
  ## Add a custom command to the start up process of the broker pods (e.g. update-ca-certificates, jvm commands, etc)
  additionalCommand:
  ## Broker service
  ## templates/broker-service.yaml
  ##
  service:
    annotations: {}
  ## Broker PodDisruptionBudget
  ## templates/broker-pdb.yaml
  ##
  pdb:
    usePolicy: true
    maxUnavailable: 1
  ### Broker service account
  ## templates/broker-service-account.yaml
  service_account:
    annotations: {}
  hpa:
    enabled: false
  customLoggingEnabled: true

functions:
  component: functions-worker

proxy:
  component: proxy
  replicaCount: 3
  podMonitor:
    enabled: false
    interval: 10s
    scrapeTimeout: 10s
  restartPodsOnConfigMapChange: false
  probe:
    liveness:
      enabled: true
      failureThreshold: 10
      initialDelaySeconds: 30
      periodSeconds: 10
      timeoutSeconds: 5
    readiness:
      enabled: true
      failureThreshold: 10
      initialDelaySeconds: 30
      periodSeconds: 10
      timeoutSeconds: 5
    startup:
      enabled: false
      failureThreshold: 30
      initialDelaySeconds: 60
      periodSeconds: 10
      timeoutSeconds: 5
  affinity:
    anti_affinity: true
    type: requiredDuringSchedulingIgnoredDuringExecution
  annotations: {}
  tolerations:
    - key: "service"
      operator: "Equal"
      value: "pulsar"
      effect: "NoSchedule"
  nodeSelector:
    service: "pulsar"
  gracePeriod: 30
  resources:
    requests:
      memory: 128Mi
      cpu: 0.2
  configData:
    PULSAR_MEM: >
      -Xms64m -Xmx64m -XX:MaxDirectMemorySize=64m

    PULSAR_GC: >
      -XX:+UseG1GC -XX:MaxGCPauseMillis=10 -Dio.netty.leakDetectionLevel=disabled -Dio.netty.recycler.linkCapacity=1024 -XX:+ParallelRefProcEnabled -XX:+UnlockExperimentalVMOptions -XX:+DoEscapeAnalysis -XX:ParallelGCThreads=4 -XX:ConcGCThreads=4 -XX:G1NewSizePercent=50 -XX:+DisableExplicitGC -XX:-ResizePLAB -XX:+ExitOnOutOfMemoryError -XX:+PerfDisableSharedMem

    httpNumThreads: "8"
    PULSAR_PREFIX_authenticateMetricsEndpoint: "false"
  ## Add a custom command to the start up process of the proxy pods (e.g. update-ca-certificates, jvm commands, etc)
  additionalCommand:
  ## Proxy service
  ## templates/proxy-service.yaml
  ##
  ports:
    http: 80
    https: 443
    pulsar: 6650
    pulsarssl: 6651
  service:
    annotations:
      external-dns-enabled: "true"
      service.beta.kubernetes.io/aws-load-balancer-scheme: "internal"
      service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
    type: LoadBalancer
  proxyTrustedCertSecret: "pulsar-staging-tls"

toolset:
  component: toolset
  useProxy: true
  replicaCount: 1
  restartPodsOnConfigMapChange: false
  annotations: {}
  tolerations: []
  gracePeriod: 30
  resources:
    requests:
      memory: 256Mi
      cpu: 0.1
  configData:
    PULSAR_MEM: >
      -Xms64M -Xmx128M -XX:MaxDirectMemorySize=128M

pulsar_manager:
  component: pulsar-manager
  replicaCount: 1
  restartPodsOnConfigMapChange: false
  gracePeriod: 30
  resources:
    requests:
      memory: 250Mi
      cpu: 0.1
  configData:
    REDIRECT_HOST: "http://127.0.0.1"
    REDIRECT_PORT: "9527"
    DRIVER_CLASS_NAME: org.postgresql.Driver
    URL: # redacted
    LOG_LEVEL: ERROR
  service:
    type: ClusterIP
    port: 9527
    targetPort: 9527
    annotations: {}
  ingress:
    enabled: false
  existingSecretName:
  admin:
    existingSecret: pulsar-manager-secret
  dbSecretName: "pulsar-postgresql-app"
  databaseCredentials:
    user: username
    password: password
job:
  ttl:
    enabled: false
    secondsAfterFinished: 3600
fullnameOverride: pulsar-staging

Navigation Menu

Uh oh!

Issue with TLS after upgrading from 2.8.x #23408

Uh oh!

{{title}}

Uh oh!

Replies: 1 comment 4 replies

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Uh oh!

Issue with TLS after upgrading from 2.8.x #23408

Uh oh!

duncaan Oct 7, 2024

Replies: 1 comment · 4 replies

Uh oh!

lhotari Oct 7, 2024 Collaborator

Uh oh!

duncaan Oct 8, 2024 Author

Uh oh!

duncaan Oct 8, 2024 Author

Uh oh!

Uh oh!

lhotari Oct 8, 2024 Collaborator

Uh oh!

duncaan Oct 9, 2024 Author

duncaan
Oct 7, 2024

Replies: 1 comment 4 replies

lhotari
Oct 7, 2024
Collaborator

duncaan Oct 8, 2024
Author

duncaan Oct 8, 2024
Author

lhotari Oct 8, 2024
Collaborator

duncaan Oct 9, 2024
Author