istio: Istio doesn't work with Cassandra

Describe the bug Cassandra node cannot communicate with each other (handshaking / gossip), with or without mTLS.

Note: this is different from issue https://github.com/istio/istio/issues/1277 which partly due to incorrect containerPort. That issue get this same problem once the containerPort was fixed.

Expected behavior istio-dump.tar.gz

Steps to reproduce the bug

Install Istio 1.0.3 withouth mTLS
Deploy cassandra (e.g following this tutorial). Make sure to declare containerPort for the pod in the statefulset.
Notice handshaking requests keep failing. Log on cassandra-0 shows issue connect to local backend

[2018-11-19T17:39:51.936Z] UF 0 0 0 "127.0.0.1:7000" inbound|7000||cassandra.default.svc.cluster.local 127.0.0.1:35262 10.44.4.11:7000 10.44.5.5:36344

Version Istio 1.0.3 Kube: 1.9

Installation {{ Please describe how Istio was installed }}

Environment GKE

Cluster state

About this issue

Original URL
State: closed
Created 6 years ago
Comments: 31 (16 by maintainers)

Commits related to this issue

Add FAQ entry for running Cassandra Addresses issue istio/istio#10053 — committed to mbanikazemi/istio.io by deleted user 5 years ago
Add FAQ entry for running Cassandra (#4867) * Add FAQ entry for running Cassandra Addresses issue istio/istio#10053 * Making clear why default config does not work * Removing numbering of su... — committed to istio/istio.io by deleted user 5 years ago

Most upvoted comments

@diemtvu & @costinm Istio is great for a lot of things but is not ready for production use if we can not run Cassandra/Kafka/Zookeeper/NameYourDistributedDataStuff. We have been struggling with istio transition for a while, and could not be able to deploy Cassandra in any way. I will try taking cassandra out of mesh and see if I can access it within mesh, while diminishing the benefits of istio and adding the complexity of it.

Should we back away for now?

yaman on Jan 24, 2019

I think a FAQ entry for this would be good, e.g., “Can I run Casandra inside an Istio mesh?”

frankbu on Aug 20, 2019

@mabushey

Thanks for the info. I am already running multiple namespaces for multiple environments, and istio already brings tons of complexity anyway, the last thing I would like to add more pseudo-namespaces just for istio-cassandra sake. I am seriously thinking to move back to good old days of nginx ingress world.

I am using incubator/cassandra helm chart. By default clusterIP: None and service.type is ClusterIP is set within the service.yaml. Removed clusterIP and cassandra nodes started to successfully handshake. Though still cannot reach to cassandra nodes in istio-mesh from my golang service.

I have another service couple with haskell+mongodb in istio-mesh. It works fine though.

yaman on Jan 24, 2019