kafka-ui: Failing to connect to Confluent Cloud Kafka

Sorry for the typo in https://github.com/provectus/kafka-ui/issues/920. After I fix this and use these settings

KAFKA_CLUSTERS_0_NAME=azureDev
KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS=<server-name>.azure.confluent.cloud:9092
KAFKA_CLUSTERS_0_PROPERTIES_SECURITY_PROTOCOL=SASL_SSL
KAFKA_CLUSTERS_0_PROPERTIES_SASL_MECHANISM=PLAIN
KAFKA_CLUSTERS_0_PROPERTIES_CLIENT_DNS_LOOKUP=use_all_dns_ips
KAFKA_CLUSTERS_0_PROPERTIES_SASL_JAAS_CONFIG='org.apache.kafka.common.security.plain.PlainLoginModule required username=“*******” password=“***********”;’
KAFKA_CLUSTERS_0_DISABLELOGDIRSCOLLECTION=true

I get org.apache.kafka.common.KafkaException: Failed to create new KafkaAdminClient

In detail:

kafka-ui  | 16:04:57.390 [main] INFO  org.springframework.core.KotlinDetector - Kotlin reflection implementation not found at runtime, related features won't be available.
kafka-ui  |
kafka-ui  |   .   ____          _            __ _ _
kafka-ui  |  /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
kafka-ui  | ( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
kafka-ui  |  \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
kafka-ui  |   '  |____| .__|_| |_|_| |_\__, | / / / /
kafka-ui  |  =========|_|==============|___/=/_/_/_/
kafka-ui  |  :: Spring Boot ::        (v2.2.4.RELEASE)
kafka-ui  |
kafka-ui  | 16:04:57.981 [main] INFO  com.provectus.kafka.ui.KafkaUiApplication - Starting KafkaUiApplication on b64d97c898ac with PID 1 (/kafka-ui-api.jar started by root in /)
kafka-ui  | 16:04:57.982 [main] DEBUG com.provectus.kafka.ui.KafkaUiApplication - Running with Spring Boot v2.2.4.RELEASE, Spring v5.2.3.RELEASE
kafka-ui  | 16:04:57.982 [main] INFO  com.provectus.kafka.ui.KafkaUiApplication - No active profile set, falling back to default profiles: default
kafka-ui  | 16:04:58.253 [background-preinit] WARN  org.springframework.http.converter.json.Jackson2ObjectMapperBuilder - For Jackson Kotlin classes support please add "com.fasterxml.jackson.module:jackson-module-kotlin" to the classpath
kafka-ui  | SLF4J: Class path contains multiple SLF4J bindings.
kafka-ui  | SLF4J: Found binding in [jar:file:/kafka-ui-api.jar!/BOOT-INF/lib/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class]
kafka-ui  | SLF4J: Found binding in [jar:file:/kafka-ui-api.jar!/BOOT-INF/lib/log4j-slf4j-impl-2.12.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
kafka-ui  | SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
kafka-ui  | SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
kafka-ui  | log4j:WARN No appenders could be found for logger (reactor.util.Loggers$LoggerFactory).
kafka-ui  | log4j:WARN Please initialize the log4j system properly.
kafka-ui  | log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
kafka-ui  | 16:04:59.357 [main] WARN  org.springframework.boot.autoconfigure.jackson.JacksonAutoConfiguration$JodaDateTimeJacksonConfiguration - Auto-configuration of Jackson's Joda-Time integration is deprecated in favor of using java.time (JSR-310).
kafka-ui  | 16:04:59.388 [main] INFO  com.provectus.kafka.ui.serde.DeserializationService - Using SchemaRegistryAwareRecordSerDe for cluster 'azureDev'
kafka-ui  | 16:04:59.820 [main] INFO  org.springframework.boot.autoconfigure.security.reactive.ReactiveUserDetailsServiceAutoConfiguration -
kafka-ui  |
kafka-ui  | Using generated security password: *************
kafka-ui  |
kafka-ui  | 16:05:00.341 [main] INFO  org.springframework.scheduling.concurrent.ThreadPoolTaskScheduler - Initializing ExecutorService 'taskScheduler'
kafka-ui  | 16:05:00.447 [parallel-1] DEBUG com.provectus.kafka.ui.service.MetricsUpdateService - Start getting metrics for kafkaCluster: azureDev
kafka-ui  | 16:05:00.620 [main] INFO  org.springframework.boot.web.embedded.netty.NettyWebServer - Netty started on port(s): 8080
kafka-ui  | 16:05:00.624 [main] INFO  com.provectus.kafka.ui.KafkaUiApplication - Started KafkaUiApplication in 3.173 seconds (JVM running for 4.416)
kafka-ui  | 16:05:00.810 [parallel-1] ERROR com.provectus.kafka.ui.service.KafkaService - Failed to collect cluster azureDev info
kafka-ui  | org.apache.kafka.common.KafkaException: Failed to create new KafkaAdminClient
kafka-ui  | 	at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:540) ~[kafka-clients-2.8.0.jar!/:?]
kafka-ui  | 	at org.apache.kafka.clients.admin.Admin.create(Admin.java:134) ~[kafka-clients-2.8.0.jar!/:?]
kafka-ui  | 	at org.apache.kafka.clients.admin.AdminClient.create(AdminClient.java:39) ~[kafka-clients-2.8.0.jar!/:?]
kafka-ui  | 	at com.provectus.kafka.ui.service.AdminClientServiceImpl.lambda$createAdminClient$2(AdminClientServiceImpl.java:41) ~[classes!/:?]
kafka-ui  | 	at reactor.core.publisher.MonoSupplier.call(MonoSupplier.java:85) ~[reactor-core-3.3.2.RELEASE.jar!/:3.3.2.RELEASE]
kafka-ui  | 	at reactor.core.publisher.FluxFlatMap.trySubscribeScalarMap(FluxFlatMap.java:126) [reactor-core-3.3.2.RELEASE.jar!/:3.3.2.RELEASE]
kafka-ui  | 	at reactor.core.publisher.MonoFlatMap.subscribeOrReturn(MonoFlatMap.java:53) [reactor-core-3.3.2.RELEASE.jar!/:3.3.2.RELEASE]
kafka-ui  | 	at reactor.core.publisher.Mono.subscribe(Mono.java:4090) [reactor-core-3.3.2.RELEASE.jar!/:3.3.2.RELEASE]
kafka-ui  | 	at reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onComplete(FluxSwitchIfEmpty.java:75) [reactor-core-3.3.2.RELEASE.jar!/:3.3.2.RELEASE]
kafka-ui  | 	at reactor.core.publisher.Operators.complete(Operators.java:132) [reactor-core-3.3.2.RELEASE.jar!/:3.3.2.RELEASE]
kafka-ui  | 	at reactor.core.publisher.MonoEmpty.subscribe(MonoEmpty.java:45) [reactor-core-3.3.2.RELEASE.jar!/:3.3.2.RELEASE]
kafka-ui  | 	at reactor.core.publisher.Mono.subscribe(Mono.java:4105) [reactor-core-3.3.2.RELEASE.jar!/:3.3.2.RELEASE]
kafka-ui  | 	at reactor.core.publisher.FluxFlatMap$FlatMapMain.onNext(FluxFlatMap.java:418) [reactor-core-3.3.2.RELEASE.jar!/:3.3.2.RELEASE]
kafka-ui  | 	at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:114) [reactor-core-3.3.2.RELEASE.jar!/:3.3.2.RELEASE]
kafka-ui  | 	at reactor.core.publisher.FluxPublishOn$PublishOnSubscriber.runAsync(FluxPublishOn.java:398) [reactor-core-3.3.2.RELEASE.jar!/:3.3.2.RELEASE]
kafka-ui  | 	at reactor.core.publisher.FluxPublishOn$PublishOnSubscriber.run(FluxPublishOn.java:484) [reactor-core-3.3.2.RELEASE.jar!/:3.3.2.RELEASE]
kafka-ui  | 	at reactor.core.scheduler.WorkerTask.call(WorkerTask.java:84) [reactor-core-3.3.2.RELEASE.jar!/:3.3.2.RELEASE]
kafka-ui  | 	at reactor.core.scheduler.WorkerTask.call(WorkerTask.java:37) [reactor-core-3.3.2.RELEASE.jar!/:3.3.2.RELEASE]
kafka-ui  | 	at java.util.concurrent.FutureTask.run(Unknown Source) [?:?]
kafka-ui  | 	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) [?:?]
kafka-ui  | 	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
kafka-ui  | 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
kafka-ui  | 	at java.lang.Thread.run(Unknown Source) [?:?]
kafka-ui  | Caused by: java.lang.IllegalArgumentException: Login module control flag not specified in JAAS config
kafka-ui  | 	at org.apache.kafka.common.security.JaasConfig.parseAppConfigurationEntry(JaasConfig.java:110) ~[kafka-clients-2.8.0.jar!/:?]
kafka-ui  | 	at org.apache.kafka.common.security.JaasConfig.<init>(JaasConfig.java:63) ~[kafka-clients-2.8.0.jar!/:?]
kafka-ui  | 	at org.apache.kafka.common.security.JaasContext.load(JaasContext.java:88) ~[kafka-clients-2.8.0.jar!/:?]
kafka-ui  | 	at org.apache.kafka.common.security.JaasContext.loadClientContext(JaasContext.java:82) ~[kafka-clients-2.8.0.jar!/:?]
kafka-ui  | 	at org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:167) ~[kafka-clients-2.8.0.jar!/:?]
kafka-ui  | 	at org.apache.kafka.common.network.ChannelBuilders.clientChannelBuilder(ChannelBuilders.java:81) ~[kafka-clients-2.8.0.jar!/:?]
kafka-ui  | 	at org.apache.kafka.clients.ClientUtils.createChannelBuilder(ClientUtils.java:105) ~[kafka-clients-2.8.0.jar!/:?]
kafka-ui  | 	at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:513) ~[kafka-clients-2.8.0.jar!/:?]
kafka-ui  | 	... 22 more

About this issue

Original URL
State: closed
Created 3 years ago
Comments: 25 (12 by maintainers)

Most upvoted comments

Hello,

I’ve found this post because I was getting the same error: org.apache.kafka.common.errors.ClusterAuthorizationException. For me the problem was the ACLs of the user, you need to enable both describe and describe config for the Cluster in order to remove this error.

That’s a problem because there are different tenants accessing the cluster with segregated ACLs permission on topics and group ids based on different prefix strings per each tenant. Allowing this cluster visibility is a potential risk, since each tenant can then see the other topic names and ACLs.

I wonder if there is the possibility of making the access to the extra cluster information optional so we can still use kafka-ui without allowing these permissions.

Thanks

dmcarba on Mar 31, 2022

Don’t give up yet. Since it works for me, I’ll provide you a test build with extra debugging to check it out, deal?

On 11 Oct 2021, at 21:32, Ron Harlev @.***> wrote:

Thanks for testing it!!!

I copied your configuration as is, only replacing the bootstrap server name, username and password. I still get org.apache.kafka.common.errors.ClusterAuthorizationException: Cluster authorization failed.

I changed the username to a wrong one, just to see the difference. I get, as expected org.apache.kafka.common.errors.SaslAuthenticationException: Authentication failed

May be something specific to our cluster. Willing to give up at this point 😦

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android.

Haarolean on Oct 11, 2021

Hi, sorry for the delay. I got a confluent kafka with azure and my setup works fine for me. That’s how my docker-compose looks:
    environment:
      KAFKA_CLUSTERS_0_NAME: azureDev
      KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: pkc-lz6r3.northeurope.azure.confluent.cloud:9092
      KAFKA_CLUSTERS_0_PROPERTIES_SECURITY_PROTOCOL: SASL_SSL
      KAFKA_CLUSTERS_0_PROPERTIES_SASL_MECHANISM: PLAIN
      KAFKA_CLUSTERS_0_PROPERTIES_CLIENT_DNS_LOOKUP: use_all_dns_ips
      KAFKA_CLUSTERS_0_PROPERTIES_SASL_JAAS_CONFIG: "org.apache.kafka.common.security.plain.PlainLoginModule required username='xxx' password='yyy';"
      KAFKA_CLUSTERS_0_DISABLELOGDIRSCOLLECTION: 'true'
Please notice that quotes around username and passord are single and the ones around the whole string are double quotes. You could try to copy paste this and use your credentials. Let me know how it goes!

Thank you for sharing your solution. I resolved the same problem by following your instructions. Thanks!

dianadujing on Mar 24, 2023

Leaving this here for posterity. If you are running against confluent cloud and you have specified correctly the jass config and still continue getting these errors look to to see if you are passing confluent.license in the connector, absence of a license returns a number of bogus errors like “Login module control flag not specified in JAAS config”.

https://docs.confluent.io/platform/current/connect/license.html

Good resource for what properties are needed here: https://gist.github.com/rmoff/49526672990f1b4f7935b62609f6f567

nuria on May 6, 2022

Hi, sorry for the delay. I got a confluent kafka with azure and my setup works fine for me. That’s how my docker-compose looks:

    environment:
      KAFKA_CLUSTERS_0_NAME: azureDev
      KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: pkc-lz6r3.northeurope.azure.confluent.cloud:9092
      KAFKA_CLUSTERS_0_PROPERTIES_SECURITY_PROTOCOL: SASL_SSL
      KAFKA_CLUSTERS_0_PROPERTIES_SASL_MECHANISM: PLAIN
      KAFKA_CLUSTERS_0_PROPERTIES_CLIENT_DNS_LOOKUP: use_all_dns_ips
      KAFKA_CLUSTERS_0_PROPERTIES_SASL_JAAS_CONFIG: "org.apache.kafka.common.security.plain.PlainLoginModule required username='xxx' password='yyy';"
      KAFKA_CLUSTERS_0_DISABLELOGDIRSCOLLECTION: 'true'

Please notice that quotes around username and passord are single and the ones around the whole string are double quotes. You could try to copy paste this and use your credentials. Let me know how it goes!

Haarolean on Oct 11, 2021

Eh, this looks unfortunate. I’ll try to reproduce with my azure instance tomorrow.

On 7 Oct 2021, at 23:00, Ron Harlev @.***> wrote:

Thanks for the tips. After finding few places where " was replaced with “ and similar, and also converting from = to colons format, I got to the point I did get “Authentication Failed” when the password was just alpha characters.

Then when I put the correct user/password I get org.apache.kafka.common.errors.ClusterAuthorizationException: Cluster authorization failed… For the record I manage to use the same credential to read and write to Kafka with other tools.

kafka-ui | 19:33:24.268 [main] INFO org.springframework.core.KotlinDetector - Kotlin reflection implementation not found at runtime, related features won’t be available. kafka-ui | kafka-ui | _ __ __ _ _ _ ___ kafka-ui | | |/ /__ _ / | | ____ _ | | | | | kafka-ui | | ’ // | |_| |/ / _ |_____| | | || | kafka-ui | | . \ (| | | < (| |_____| || || | kafka-ui | ||__,|| ||__,| _/|| kafka-ui | kafka-ui | kafka-ui | 19:33:24.853 [main] INFO com.provectus.kafka.ui.KafkaUiApplication - Starting KafkaUiApplication on f3bff025220f with PID 1 (/kafka-ui-api.jar started by root in /) kafka-ui | 19:33:24.853 [main] DEBUG com.provectus.kafka.ui.KafkaUiApplication - Running with Spring Boot v2.2.4.RELEASE, Spring v5.2.3.RELEASE kafka-ui | 19:33:24.854 [main] INFO com.provectus.kafka.ui.KafkaUiApplication - No active profile set, falling back to default profiles: default kafka-ui | 19:33:25.121 [background-preinit] WARN org.springframework.http.converter.json.Jackson2ObjectMapperBuilder - For Jackson Kotlin classes support please add “com.fasterxml.jackson.module:jackson-module-kotlin” to the classpath kafka-ui | SLF4J: Class path contains multiple SLF4J bindings. kafka-ui | SLF4J: Found binding in [jar:file:/kafka-ui-api.jar!/BOOT-INF/lib/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class] kafka-ui | SLF4J: Found binding in [jar:file:/kafka-ui-api.jar!/BOOT-INF/lib/log4j-slf4j-impl-2.12.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] kafka-ui | SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. kafka-ui | SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] kafka-ui | log4j:WARN No appenders could be found for logger (reactor.util.Loggers$LoggerFactory). kafka-ui | log4j:WARN Please initialize the log4j system properly. kafka-ui | log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info. kafka-ui | 19:33:26.300 [main] INFO org.springframework.boot.autoconfigure.security.reactive.ReactiveUserDetailsServiceAutoConfiguration - kafka-ui | kafka-ui | Using generated security password: 22516340-c209-4e09-8a5b-0bc5b8565133 kafka-ui | kafka-ui | 19:33:26.357 [main] WARN org.springframework.boot.autoconfigure.jackson.JacksonAutoConfiguration$JodaDateTimeJacksonConfiguration - Auto-configuration of Jackson’s Joda-Time integration is deprecated in favor of using java.time (JSR-310). kafka-ui | 19:33:26.516 [main] INFO com.provectus.kafka.ui.serde.DeserializationService - Using SchemaRegistryAwareRecordSerDe for cluster ‘azureDev’ kafka-ui | 19:33:27.116 [main] INFO org.springframework.scheduling.concurrent.ThreadPoolTaskScheduler - Initializing ExecutorService ‘taskScheduler’ kafka-ui | 19:33:27.206 [parallel-1] DEBUG com.provectus.kafka.ui.service.MetricsUpdateService - Start getting metrics for kafkaCluster: azureDev kafka-ui | 19:33:27.372 [main] INFO org.springframework.boot.web.embedded.netty.NettyWebServer - Netty started on port(s): 8080 kafka-ui | 19:33:27.377 [main] INFO com.provectus.kafka.ui.KafkaUiApplication - Started KafkaUiApplication in 3.053 seconds (JVM running for 4.237) kafka-ui | 19:33:29.510 [kafka-admin-client-thread | adminclient-1] ERROR com.provectus.kafka.ui.service.KafkaService - Failed to collect cluster azureDev info kafka-ui | org.apache.kafka.common.errors.ClusterAuthorizationException: Cluster authorization failed. — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android.

Haarolean on Oct 7, 2021

I’m running in docker-compose locally against the Kafka service managed by Confluent on Azure

harlev on Oct 6, 2021