datahub: When I try logging in to datahub its says invalid username, I am unable to bootstrap datahub
When I run python mce_cli.py produce -d bootstrap_mce.dat the following docker logs show
schema-registry | [2019-11-07 21:32:21,330] INFO Wait to catch up until the offset of the last message at 1 (io.confluent.kafka.schemaregistry.storage.KafkaStore)
schema-registry | [2019-11-07 21:32:21,500] INFO 192.168.128.1 - - [07/Nov/2019:21:32:20 +0000] "POST /subjects/MetadataChangeEvent-value/versions HTTP/1.1" 200 8 942 (io.confluent.rest-utils.requests)
broker | [2019-11-07 21:32:22,445] INFO Creating topic MetadataChangeEvent with configuration {} and initial partition assignment Map(0 -> ArrayBuffer(1)) (kafka.zk.AdminZkClient)
zookeeper | [2019-11-07 21:32:22,448] INFO Got user-level KeeperException when processing sessionid:0x100037ed7540004 type:setData cxid:0x2f9 zxid:0xa2 txntype:-1 reqpath:n/a Error Path:/config/topics/MetadataChangeEvent Error:KeeperErrorCode = NoNode for /config/topics/MetadataChangeEvent (org.apache.zookeeper.server.PrepRequestProcessor)
broker | [2019-11-07 21:32:22,455] INFO [KafkaApi-1] Auto creation of topic MetadataChangeEvent with 1 partitions and replication factor 1 is successful (kafka.server.KafkaApis)
broker | [2019-11-07 21:32:22,461] INFO [Controller id=1] New topics: [Set(MetadataChangeEvent)], deleted topics: [Set()], new partition replica assignment [Map(MetadataChangeEvent-0 -> Vector(1))] (kafka.controller.KafkaController)
broker | [2019-11-07 21:32:22,461] INFO [Controller id=1] New partition creation callback for MetadataChangeEvent-0 (kafka.controller.KafkaController)
broker | [2019-11-07 21:32:22,461] TRACE [Controller id=1 epoch=1] Changed partition MetadataChangeEvent-0 state from NonExistentPartition to NewPartition with assigned replicas 1 (state.change.logger)
broker | [2019-11-07 21:32:22,462] TRACE [Controller id=1 epoch=1] Changed state of replica 1 for partition MetadataChangeEvent-0 from NonExistentReplica to NewReplica (state.change.logger)
broker | [2019-11-07 21:32:22,474] TRACE [Controller id=1 epoch=1] Changed partition MetadataChangeEvent-0 from NewPartition to OnlinePartition with state LeaderAndIsr(leader=1, leaderEpoch=0, isr=List(1), zkVersion=0) (state.change.logger)
broker | [2019-11-07 21:32:22,475] TRACE [Controller id=1 epoch=1] Sending become-leader LeaderAndIsr request PartitionState(controllerEpoch=1, leader=1, leaderEpoch=0, isr=1, zkVersion=0, replicas=1, isNew=true) to broker 1 for partition MetadataChangeEvent-0 (state.change.logger)
broker | [2019-11-07 21:32:22,476] TRACE [Controller id=1 epoch=1] Sending UpdateMetadata request PartitionState(controllerEpoch=1, leader=1, leaderEpoch=0, isr=[1], zkVersion=0, replicas=[1], offlineReplicas=[]) to brokers Set(1) for partition MetadataChangeEvent-0 (state.change.logger)
broker | [2019-11-07 21:32:22,476] TRACE [Controller id=1 epoch=1] Changed state of replica 1 for partition MetadataChangeEvent-0 from NewReplica to OnlineReplica (state.change.logger)
broker | [2019-11-07 21:32:22,479] TRACE [Broker id=1] Received LeaderAndIsr request PartitionState(controllerEpoch=1, leader=1, leaderEpoch=0, isr=1, zkVersion=0, replicas=1, isNew=true) correlation id 7 from controller 1 epoch 1 for partition MetadataChangeEvent-0 (state.change.logger)
broker | [2019-11-07 21:32:22,481] TRACE [Broker id=1] Handling LeaderAndIsr request correlationId 7 from controller 1 epoch 1 starting the become-leader transition for partition MetadataChangeEvent-0 (state.change.logger)
broker | [2019-11-07 21:32:22,482] INFO [ReplicaFetcherManager on broker 1] Removed fetcher for partitions Set(MetadataChangeEvent-0) (kafka.server.ReplicaFetcherManager)
broker | [2019-11-07 21:32:22,497] INFO [Log partition=MetadataChangeEvent-0, dir=/var/lib/kafka/data] Loading producer state till offset 0 with message format version 2 (kafka.log.Log)
broker | [2019-11-07 21:32:22,498] INFO [Log partition=MetadataChangeEvent-0, dir=/var/lib/kafka/data] Completed load of log with 1 segments, log start offset 0 and log end offset 0 in 3 ms (kafka.log.Log)
broker | [2019-11-07 21:32:22,500] INFO Created log for partition MetadataChangeEvent-0 in /var/lib/kafka/data with properties {compression.type -> producer, message.format.version -> 2.2-IV1, file.delete.delay.ms -> 60000, max.message.bytes -> 1000012, min.compaction.lag.ms -> 0, message.timestamp.type -> CreateTime, message.downconversion.enable -> true, min.insync.replicas -> 1, segment.jitter.ms -> 0, preallocate -> false, min.cleanable.dirty.ratio -> 0.5, index.interval.bytes -> 4096, unclean.leader.election.enable -> false, retention.bytes -> -1, delete.retention.ms -> 86400000, cleanup.policy -> [delete], flush.ms -> 9223372036854775807, segment.ms -> 604800000, segment.bytes -> 1073741824, retention.ms -> 604800000, message.timestamp.difference.max.ms -> 9223372036854775807, segment.index.bytes -> 10485760, flush.messages -> 9223372036854775807}. (kafka.log.LogManager)
broker | [2019-11-07 21:32:22,502] INFO [Partition MetadataChangeEvent-0 broker=1] No checkpointed highwatermark is found for partition MetadataChangeEvent-0 (kafka.cluster.Partition)
broker | [2019-11-07 21:32:22,502] INFO Replica loaded for partition MetadataChangeEvent-0 with initial high watermark 0 (kafka.cluster.Replica)
broker | [2019-11-07 21:32:22,503] INFO [Partition MetadataChangeEvent-0 broker=1] MetadataChangeEvent-0 starts at Leader Epoch 0 from offset 0. Previous Leader Epoch was: -1 (kafka.cluster.Partition)
broker | [2019-11-07 21:32:22,506] TRACE [Broker id=1] Stopped fetchers as part of become-leader request from controller 1 epoch 1 with correlation id 7 for partition MetadataChangeEvent-0 (last update controller epoch 1) (state.change.logger)
broker | [2019-11-07 21:32:22,506] TRACE [Broker id=1] Completed LeaderAndIsr request correlationId 7 from controller 1 epoch 1 for the become-leader transition for partition MetadataChangeEvent-0 (state.change.logger)
broker | [2019-11-07 21:32:22,510] TRACE [Controller id=1 epoch=1] Received response {error_code=0,partitions=[{topic=MetadataChangeEvent,partition=0,error_code=0}]} for request LEADER_AND_ISR with correlation id 7 sent to broker broker:29092 (id: 1 rack: null) (state.change.logger)
broker | [2019-11-07 21:32:22,516] TRACE [Broker id=1] Cached leader info PartitionState(controllerEpoch=1, leader=1, leaderEpoch=0, isr=[1], zkVersion=0, replicas=[1], offlineReplicas=[]) for partition MetadataChangeEvent-0 in response to UpdateMetadata request sent by controller 1 epoch 1 with correlation id 8 (state.change.logger)
broker | [2019-11-07 21:32:22,518] TRACE [Controller id=1 epoch=1] Received response {error_code=0} for request UPDATE_METADATA with correlation id 8 sent to broker broker:29092 (id: 1 rack: null) (state.change.logger)`
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Reactions: 1
- Comments: 18 (9 by maintainers)
Hi @NishitaNarvekar ,
Sorry for late response. Seems like Docker containers which are responsible for initializing/configuring Kafka and Elasticsearch fail because of timeout. These containers are
kafka-setupandelasticsearch-setup, respectively.datahub-gmsalso fails to initialize because of timeout. Below are the relevant lines showing these failures:[35melasticsearch-setup |[0m 2019/11/12 21:41:50 Timeout after 1m0s waiting on dependencies to become available: [http://elasticsearch:9200][32;1mdatahub-gms |[0m 2019/11/12 21:41:56 Timeout after 1m0s waiting on dependencies to become available: [tcp://mysql:3306 tcp://broker:29092 http://elasticsearch:9200][35;1mkafka-setup |[0m [main] ERROR io.confluent.admin.utils.ClusterStatus - Error while getting broker list. [35;1mkafka-setup |[0m java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. [35;1mkafka-setup |[0m at org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45) [35;1mkafka-setup |[0m at org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32) [35;1mkafka-setup |[0m at org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:89) [35;1mkafka-setup |[0m at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:260) [35;1mkafka-setup |[0m at io.confluent.admin.utils.ClusterStatus.isKafkaReady(ClusterStatus.java:149) [35;1mkafka-setup |[0m at io.confluent.admin.utils.cli.KafkaReadyCommand.main(KafkaReadyCommand.java:150) [35;1mkafka-setup |[0m Caused by: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment.[35melasticsearch-setup exited with code 7[35;1mkafka-setup exited with code 1Recently, I pushed a couple of fixes which increases these timeouts to compensate for all environments with different CPU and memory resources. Please pull latest changes and try again if it solves these issues.