kubernetes-kafka: Test failures for console producer/consumer with min.insync.replicas=2

Running the Kafka tests this one do not complete

NAME                    READY     STATUS    RESTARTS   AGE
kafkacat-4kvj4          3/3       Running   0          1h
produce-consume-kdd92   2/3       Running   0          1h

Logs from Testcase

Nothing consumed yet
Nothing consumed yet
Nothing consumed yet
Nothing consumed yet
Nothing consumed yet
Nothing consumed yet
Nothing consumed yet
....

Is it probably the producer does not generate enough data? No errors in producer or consumer.

Using kubernetes 1.8.5 on GKE

About this issue

Original URL
State: closed
Created 6 years ago
Comments: 40 (32 by maintainers)

Commits related to this issue

Removes explicit topic creation from standard tests which means we get the default number of replicas from broker config for #116 — committed to Yolean/kubernetes-kafka by solsson 6 years ago

Most upvoted comments

Looks bad. I’m sorry about leaving master relatively untested. Could be the cause of https://github.com/Yolean/kubernetes-kafka/issues/114 too. I’ve been caught up in not only holiday season but also https://github.com/Yolean/kubernetes-kafka/issues/101#issuecomment-355940612.

I wrote these tests while working on new features, but now that they run constantly I find the readiness indication very useful. I will try to reproduce the above issues.

solsson on Jan 8, 2018

@solsson I appreciate the goal of making something that works for developers and works in production.

We’ve used Kafka in production for 5 years at Ithaka. In production in just November, our 5 node kafka cluster had a topic that handled 30,419,294 messages per day, and another that was was about half that. Lots of other busy topics too. We don’t specify min.insync.replicas (so it defaults to 1) and it hasn’t been a problem so far as we know. We do set num.partitions=15 and default.replication.factor=2.

I’m not sure min.insync.replicas=2 is necessary for production level durability.

With that said, it sounds very nice in theory, and I’m very interested in understanding how you don’t get the same results I do. If there’s something else that needs to be turned on to make it work, then I’m all for it.

StevenACoffman on Jan 11, 2018