clusters connected to kafbat-ui keeps going offline

[harshashenoy98]

Issue submitter TODO list

• I've looked up my issue in FAQ
• I've searched for an already existing issues here
• I've tried running main-labeled docker image and the issue still persists there
• I'm running a supported version of the application which is listed here

Describe the bug (actual behavior)

The clusters set/connected from kafbat-ui keep going offline every now and then.

upon checking logs we saw the error -

2025-03-17 13:10:52,161 ERROR [parallel-15] o.s.b.a.w.r.e.AbstractErrorWebExceptionHandler: [2b99cf3c-7117]  500 Server Error for HTTP GET "/api/clusters/BMA/topics?page=1&perPage=25&showInternal=true"
org.apache.kafka.common.errors.TimeoutException: The AdminClient thread has exited.
	Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException:
Error has been observed at the following site(s):
	*__checkpoint ⇢ Handler io.kafbat.ui.controller.TopicsController#getTopics(String, Integer, Integer, Boolean, String, TopicColumnsToSortDTO, SortOrderDTO, ServerWebExchange) [DispatcherHandler]
	*__checkpoint ⇢ io.kafbat.ui.config.CorsGlobalConfiguration$$Lambda/0x00007f50be6dfc00 [DefaultWebFilterChain]
	*__checkpoint ⇢ io.kafbat.ui.config.ReadOnlyModeFilter [DefaultWebFilterChain]
	*__checkpoint ⇢ io.kafbat.ui.config.CustomWebFilter [DefaultWebFilterChain]
	*__checkpoint ⇢ AuthorizationWebFilter [DefaultWebFilterChain]
	*__checkpoint ⇢ ExceptionTranslationWebFilter [DefaultWebFilterChain]
	*__checkpoint ⇢ LogoutWebFilter [DefaultWebFilterChain]
	*__checkpoint ⇢ ServerRequestCacheWebFilter [DefaultWebFilterChain]
	*__checkpoint ⇢ SecurityContextServerWebExchangeWebFilter [DefaultWebFilterChain]
	*__checkpoint ⇢ LogoutPageGeneratingWebFilter [DefaultWebFilterChain]
	*__checkpoint ⇢ LoginPageGeneratingWebFilter [DefaultWebFilterChain]
	*__checkpoint ⇢ StaticFileWebFilter [DefaultWebFilterChain]
	*__checkpoint ⇢ DefaultResourcesWebFilter [DefaultWebFilterChain]
	*__checkpoint ⇢ OAuth2LoginAuthenticationWebFilter [DefaultWebFilterChain]
	*__checkpoint ⇢ OAuth2AuthorizationRequestRedirectWebFilter [DefaultWebFilterChain]
	*__checkpoint ⇢ ReactorContextWebFilter [DefaultWebFilterChain]
	*__checkpoint ⇢ HttpHeaderWriterWebFilter [DefaultWebFilterChain]
	*__checkpoint ⇢ ServerWebExchangeReactorContextWebFilter [DefaultWebFilterChain]
	*__checkpoint ⇢ org.springframework.security.web.server.WebFilterChainProxy [DefaultWebFilterChain]
	*__checkpoint ⇢ HTTP GET "/api/clusters/BMA/topics?page=1&perPage=25&showInternal=true" [ExceptionHandlingWebHandler]
Original Stack Trace:
2025-03-17 13:10:53,224 ERROR [parallel-16] o.s.b.a.w.r.e.AbstractErrorWebExceptionHandler: [2b99cf3c-7121]  500 Server Error for HTTP GET "/api/clusters/BMA/topics?page=1&perPage=25&showInternal=true"
org.apache.kafka.common.errors.TimeoutException: The AdminClient thread has exited.
	Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException:
Error has been observed at the following site(s):
	*__checkpoint ⇢ Handler io.kafbat.ui.controller.TopicsController#getTopics(String, Integer, Integer, Boolean, String, TopicColumnsToSortDTO, SortOrderDTO, ServerWebExchange) [DispatcherHandler]
	*__checkpoint ⇢ io.kafbat.ui.config.CorsGlobalConfiguration$$Lambda/0x00007f50be6dfc00 [DefaultWebFilterChain]
	*__checkpoint ⇢ io.kafbat.ui.config.ReadOnlyModeFilter [DefaultWebFilterChain]
	*__checkpoint ⇢ io.kafbat.ui.config.CustomWebFilter [DefaultWebFilterChain]
	*__checkpoint ⇢ AuthorizationWebFilter [DefaultWebFilterChain]
	*__checkpoint ⇢ ExceptionTranslationWebFilter [DefaultWebFilterChain]
	*__checkpoint ⇢ LogoutWebFilter [DefaultWebFilterChain]
	*__checkpoint ⇢ ServerRequestCacheWebFilter [DefaultWebFilterChain]
	*__checkpoint ⇢ SecurityContextServerWebExchangeWebFilter [DefaultWebFilterChain]
	*__checkpoint ⇢ LogoutPageGeneratingWebFilter [DefaultWebFilterChain]
	*__checkpoint ⇢ LoginPageGeneratingWebFilter [DefaultWebFilterChain]
	*__checkpoint ⇢ StaticFileWebFilter [DefaultWebFilterChain]
	*__checkpoint ⇢ DefaultResourcesWebFilter [DefaultWebFilterChain]
	*__checkpoint ⇢ OAuth2LoginAuthenticationWebFilter [DefaultWebFilterChain]
	*__checkpoint ⇢ OAuth2AuthorizationRequestRedirectWebFilter [DefaultWebFilterChain]
	*__checkpoint ⇢ ReactorContextWebFilter [DefaultWebFilterChain]
	*__checkpoint ⇢ HttpHeaderWriterWebFilter [DefaultWebFilterChain]
	*__checkpoint ⇢ ServerWebExchangeReactorContextWebFilter [DefaultWebFilterChain]
	*__checkpoint ⇢ org.springframework.security.web.server.WebFilterChainProxy [DefaultWebFilterChain]
	*__checkpoint ⇢ HTTP GET "/api/clusters/BMA/topics?page=1&perPage=25&showInternal=true" [ExceptionHandlingWebHandler]
Original Stack Trace:
2025-03-17 13:10:55,284 ERROR [parallel-17] o.s.b.a.w.r.e.AbstractErrorWebExceptionHandler: [2b99cf3c-7122]  500 Server Error for HTTP GET "/api/clusters/BMA/topics?page=1&perPage=25&showInternal=true"
org.apache.kafka.common.errors.TimeoutException: The AdminClient thread has exited.
	Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException:
Error has been observed at the following site(s):
	*__checkpoint ⇢ Handler io.kafbat.ui.controller.TopicsController#getTopics(String, Integer, Integer, Boolean, String, TopicColumnsToSortDTO, SortOrderDTO, ServerWebExchange) [DispatcherHandler]
	*__checkpoint ⇢ io.kafbat.ui.config.CorsGlobalConfiguration$$Lambda/0x00007f50be6dfc00 [DefaultWebFilterChain]
	*__checkpoint ⇢ io.kafbat.ui.config.ReadOnlyModeFilter [DefaultWebFilterChain]
	*__checkpoint ⇢ io.kafbat.ui.config.CustomWebFilter [DefaultWebFilterChain]
	*__checkpoint ⇢ AuthorizationWebFilter [DefaultWebFilterChain]
	*__checkpoint ⇢ ExceptionTranslationWebFilter [DefaultWebFilterChain]
	*__checkpoint ⇢ LogoutWebFilter [DefaultWebFilterChain]
	*__checkpoint ⇢ ServerRequestCacheWebFilter [DefaultWebFilterChain]
	*__checkpoint ⇢ SecurityContextServerWebExchangeWebFilter [DefaultWebFilterChain]
	*__checkpoint ⇢ LogoutPageGeneratingWebFilter [DefaultWebFilterChain]
	*__checkpoint ⇢ LoginPageGeneratingWebFilter [DefaultWebFilterChain]
	*__checkpoint ⇢ StaticFileWebFilter [DefaultWebFilterChain]
	*__checkpoint ⇢ DefaultResourcesWebFilter [DefaultWebFilterChain]
	*__checkpoint ⇢ OAuth2LoginAuthenticationWebFilter [DefaultWebFilterChain]
	*__checkpoint ⇢ OAuth2AuthorizationRequestRedirectWebFilter [DefaultWebFilterChain]
	*__checkpoint ⇢ ReactorContextWebFilter [DefaultWebFilterChain]
	*__checkpoint ⇢ HttpHeaderWriterWebFilter [DefaultWebFilterChain]
	*__checkpoint ⇢ ServerWebExchangeReactorContextWebFilter [DefaultWebFilterChain]
	*__checkpoint ⇢ org.springframework.security.web.server.WebFilterChainProxy [DefaultWebFilterChain]
	*__checkpoint ⇢ HTTP GET "/api/clusters/BMA/topics?page=1&perPage=25&showInternal=true" [ExceptionHandlingWebHandler]
Original Stack Trace:
2025-03-17 13:11:15,232 ERROR [parallel-26] i.k.u.s.StatisticsService: Failed to collect cluster BMA info
org.apache.kafka.common.errors.TimeoutException: The AdminClient thread has exited.

Expected behavior

No response

Your installation details

deployed kafbat-ui via helm and provided all kafka clusters details from value.yaml

Steps to reproduce

deployed kafbat-ui via helm and provided all kafka clusters details from value.yaml

Screenshots

No response

Logs

No response

Additional context

No response

[github-actions]

Hi harshashenoy98! 👋

Welcome, and thank you for opening your first issue in the repo!

Please wait for triaging by our maintainers.

As development is carried out in our spare time, you can support us by sponsoring our activities or even funding the development of specific issues.
Sponsorship link

If you plan to raise a PR for this issue, please take a look at our contributing guide.

[germanosin]

Hi @harshashenoy98 !

Based on the logs you shared, Kafka UI is unable to retrieve topic information from the cluster due to a timeout exception. To troubleshoot the issue, you can:

• Verify that the instance running Kafka UI can connect to the brokers on the specified port.
• Ensure the correct port is configured.
• Check if your Kafka cluster uses advertised hostnames; if so, make sure they are accessible from the instance.

Overall, this appears to be a network-related issue.

[fantastywusheng]

The JDK version of my Kafbat-UI is "java version '21.0.6' 2025-01-21 LTS", and the Kafbat version is 4cf17a0 v1.1.0, the JDK version of my kafka broker is jdk11.
My configuration is logging:
level:
root: INFO
io.kafbat.ui: INFO
#org.springframework.http.codec.json.Jackson2JsonEncoder: DEBUG
#org.springframework.http.codec.json.Jackson2JsonDecoder: DEBUG
reactor.netty.http.server.AccessLog: INFO
org.springframework.security: INFO
server:
port: 8187
spring:
jmx:
enabled: true
#ldap:

urls: ldap://localhost:10389

base: "cn={0},ou=people,dc=planetexpress,dc=com"

admin-user: "cn=admin,dc=planetexpress,dc=com"

admin-password: "GoodNewsEveryone"

user-filter-search-base: "dc=planetexpress,dc=com"

user-filter-search-filter: "(&(uid={0})(objectClass=inetOrgPerson))"

group-filter-search-base: "ou=people,dc=planetexpress,dc=com"

kafka:
clusters:
- name: local
bootstrapServers: 192.168.10.116:9013
#schemaRegistry: http://localhost:8085
#ksqldbServer: http://localhost:8088
#kafkaConnect:
# - name: first
# address: http://localhost:8083
metrics:
port: 9793
type: JMX
dynamic.config.enabled: true
oauth2:
ldap:
activeDirectory: false
activeDirectory.domain: domain.com
auth:
type: DISABLED

type: OAUTH2

type: LDAP

oauth2:
client:
cognito:
clientId: # CLIENT ID
clientSecret: # CLIENT SECRET
scope: openid
client-name: cognito
provider: cognito
redirect-uri: http://localhost:8080/login/oauth2/code/cognito
authorization-grant-type: authorization_code
issuer-uri: https://cognito-idp.eu-central-1.amazonaws.com/eu-central-1_M7cIUn1nj
jwk-set-uri: https://cognito-idp.eu-central-1.amazonaws.com/eu-central-1_M7cIUn1nj/.well-known/jwks.json
user-name-attribute: cognito:username
custom-params:
type: cognito
logoutUrl: https://kek.net/logout
google:
provider: google
clientId: # CLIENT ID
clientSecret: # CLIENT SECRET
user-name-attribute: email
custom-params:
type: google
allowedDomain: kafbat.dev
github:
provider: github
clientId: # CLIENT ID
clientSecret: # CLIENT SECRET
scope:
- read:org
user-name-attribute: login
custom-params:
type: github
rbac:
roles:
- name: "memelords"
clusters:
- local
subjects:
- provider: oauth_google
type: domain
value: "kafbat.dev"
- provider: oauth_google
type: user
value: "name@kafbat.dev"
- provider: oauth_github
type: organization
value: "kafbat"
- provider: oauth_github
type: user
value: "memelord"
- provider: oauth_cognito
type: user
value: "username"
- provider: oauth_cognito
type: group
value: "memelords"
- provider: ldap
type: group
value: "admin_staff"
# NOT IMPLEMENTED YET
# - provider: ldap_ad
# type: group
# value: "admin_staff"
permissions:
- resource: applicationconfig
actions: all
- resource: clusterconfig
actions: all
- resource: topic
value: "."
actions: all
- resource: consumer
value: "."
actions: all
- resource: schema
value: "."
actions: all
- resource: connect
value: "."
actions: all
- resource: ksql
actions: all
- resource: acl
actions: all
- resource: audit
actions: all

nohup.log

[fantastywusheng]

The cluster was added through the URL /ui/clusters/create-new-cluster, and the log shows that the corresponding cluster already exists, and the metrics have also been collected. Why isn't the corresponding cluster displayed on the Kafka UI?

[fantastywusheng]

DEBUG [boundedElastic-1] i.k.u.s.m.JmxMetricsRetriever: 1211 metrics collected for service:jmx:rmi:///jndi/rmi://192.168.10.118:9793/jmxrmi
2025-03-18 13:55:53,927 DEBUG [boundedElastic-17] i.k.u.s.m.JmxMetricsRetriever: 1167 metrics collected for service:jmx:rmi:///jndi/rmi://192.168.10.116:9793/jmxrmi
2025-03-18 13:55:53,932 DEBUG [boundedElastic-4] i.k.u.s.m.JmxMetricsRetriever: 1211 metrics collected for service:jmx:rmi:///jndi/rmi://192.168.10.118:9793/jmxrmi
2025-03-18 13:55:53,934 DEBUG [boundedElastic-6] i.k.u.s.m.JmxMetricsRetriever: 1261 metrics collected for service:jmx:rmi:///jndi/rmi://192.168.10.117:9793/jmxrmi
2025-03-18 13:55:53,940 DEBUG [boundedElastic-13] i.k.u.s.m.JmxMetricsRetriever: 1211 metrics collected for service:jmx:rmi:///jndi/rmi://192.168.10.118:9793/jmxrmi
2025-03-18 13:55:53,941 DEBUG [boundedElastic-3] i.k.u.s.m.JmxMetricsRetriever: 1261 metrics collected for service:jmx:rmi:///jndi/rmi://192.168.10.117:9793/jmxrmi
2025-03-18 13:55:53,945 DEBUG [boundedElastic-13] i.k.u.s.ClustersStatisticsScheduler: Metrics updated for cluster: kafka-3.5.1
2025-03-18 13:55:53,952 DEBUG [boundedElastic-15] i.k.u.s.m.JmxMetricsRetriever: 1261 metrics collected for service:jmx:rmi:///jndi/rmi://192.168.10.117:9793/jmxrmi
2025-03-18 13:55:53,953 DEBUG [boundedElastic-7] i.k.u.s.m.JmxMetricsRetriever: 1211 metrics collected for service:jmx:rmi:///jndi/rmi://192.168.10.118:9793/jmxrmi
2025-03-18 13:55:53,956 DEBUG [boundedElastic-9] i.k.u.s.m.JmxMetricsRetriever: 1167 metrics collected for service:jmx:rmi:///jndi/rmi://192.168.10.116:9793/jmxrmi
2025-03-18 13:55:53,986 DEBUG [boundedElastic-18] i.k.u.s.m.JmxMetricsRetriever: 1211 metrics collected for service:jmx:rmi:///jndi/rmi://192.168.10.118:9793/jmxrmi
2025-03-18 13:55:53,990 DEBUG [boundedElastic-18] i.k.u.s.ClustersStatisticsScheduler: Metrics updated for cluster: local
2025-03-18 13:55:54,860 DEBUG [reactor-http-epoll-7] i.k.u.s.ClustersStatisticsScheduler: Metrics updated for cluster: kafka
2025-03-18 13:55:56,305 DEBUG [reactor-http-epoll-8] i.k.u.s.ClustersStatisticsScheduler: Metrics updated for cluster: kafka-kraft

[germanosin]

@fantastywusheng, your message is a bit unclear. Would you mind organizing it more clearly? Also, you mentioned both "config" and "dynamic config"—could you clarify it? It might be helpful to create a separate issue for this.

[fantastywusheng]

@fantastywusheng, your message is a bit unclear. Would you mind organizing it more clearly? Also, you mentioned both "config" and "dynamic config"—could you clarify it? It might be helpful to create a separate issue for this.

OK, I'll raise a new one. Thank you so much for your reply.This is the address of the issue that I create,#944

[harshashenoy98]

Hi @harshashenoy98 !

Based on the logs you shared, Kafka UI is unable to retrieve topic information from the cluster due to a timeout exception. To troubleshoot the issue, you can:

• Verify that the instance running Kafka UI can connect to the brokers on the specified port.
• Ensure the correct port is configured.
• Check if your Kafka cluster uses advertised hostnames; if so, make sure they are accessible from the instance.

Overall, this appears to be a network-related issue.

Hello @germanosin
I don't see any issue with the network checked the network connectivity as well.
cause restarting the kafka-ui pods gets the clusters all online.

we have been seeing this issue for a while now. and everytime just needs a kafka-ui pods to be restarted.

[germanosin]

@harshashenoy98 ,
That seems unusual. Based on the source code, this exception should only occur if the AdminClient is in a closing state. We'll investigate how this might be happening.