nats-operator: Nats-Operator incompatible with istio?

When I follow the instructions in the project readme to create a nats cluster with 3 members on a gke cluster using istio, all three members immediately show unhealthy and quickly go to crashloopbackoff. Is there something additional I need to do to get nats-operator to play nice with a service mesh?

My Nats Cluster:

echo '
apiVersion: "nats.io/v1alpha2"
kind: "NatsCluster"
metadata:
  name: "example-nats-cluster"
spec:
  size: 3
  version: "1.3.0"
' | kubectl apply -f -

Log from one member:

[1] 2018/10/30 20:27:15.907885 [INF] Starting nats-server version 1.3.0
[1] 2018/10/30 20:27:15.907943 [INF] Git commit [eed4fbc]
[1] 2018/10/30 20:27:15.908133 [INF] Starting http monitor on 0.0.0.0:8222
[1] 2018/10/30 20:27:15.908194 [INF] Listening for client connections on 0.0.0.0:4222
[1] 2018/10/30 20:27:15.908208 [INF] Server is ready
[1] 2018/10/30 20:27:15.908541 [INF] Listening for route connections on 0.0.0.0:6222
[1] 2018/10/30 20:27:15.914868 [ERR] Error trying to connect to route: dial tcp 10.12.12.4:6222: connect: connection refused
[1] 2018/10/30 20:27:16.930604 [ERR] Error trying to connect to route: dial tcp 10.12.12.4:6222: connect: connection refused
[1] 2018/10/30 20:27:17.935214 [INF] 10.12.12.4:6222 - rid:1 - Route connection created
[1] 2018/10/30 20:27:17.940613 [INF] 127.0.0.1:41486 - rid:2 - Route connection created
[1] 2018/10/30 20:27:18.962862 [INF] 10.12.12.4:6222 - rid:3 - Route connection created

(and the Route connection messages continue 290 times before the container is shut down as unhealthy)

My Istio deployment is the default Isitio App from the GCP marketplace, with three nodes in it.
K8S version info:

Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.7", GitCommit:"0c38c362511b20a098d7cd855f1314dad92c2780", GitTreeState:"clean", BuildDate:"2018-08-20T10:09:03Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"9+", GitVersion:"v1.9.7-gke.6", GitCommit:"9b635efce81582e1da13b35a7aa539c0ccb32987", GitTreeState:"clean", BuildDate:"2018-08-16T21:33:47Z", GoVersion:"go1.9.3b4", Compiler:"gc", Platform:"linux/amd64"}

istio-pilot version is 1.3

I’d be happy to add more detail if there are follow up questions. I can also cross-post this issue to Istio if the problem appears to be on their side…

About this issue

  • Original URL
  • State: closed
  • Created 6 years ago
  • Reactions: 2
  • Comments: 19 (1 by maintainers)

Commits related to this issue

Most upvoted comments

From a first glance I was noticing that I wasn’t getting a response over telnet (with istio-sidecar)

Little telnet debug no response

> telnet nats.nats-system.svc.cluster.local 4222

Telnet with a PING -> instant response

>telnet nats.nats-system.svc.cluster.local 4222
PING
INFO {"server_id":"NBB2A2ML5APZWCXAX6SPEQBINHC4B5J2DPYHYRITHXXLQEDW64KVKWMM","server_name":"NBB2A2ML5APZWCXAX6SPEQBINHC4B5J2DPYHYRITHXXLQEDW64KVKWMM","version":"2.1.7","proto":1,"git_commit":"bf0930e","go":"go1.13.10","host":"0.0.0.0","port":4222,"max_payload":1048576,"client_id":1,"client_ip":"127.0.0.1"}
PONG

Interesting istio thinks it’s HTTP raw_buffer:

> istioctl pc listeners <pod-name> --port 4222
ADDRESS       PORT MATCH                        DESTINATION
10.96.249.161 4222 Trans: raw_buffer; App: HTTP Route: nats.nats-system.svc.cluster.local:4222

Istio Explicit Port Selection helped me.

Here the service for nats doesn’t declare tcp or tls. If they added appProtocol explicity for k8s 1.18+ or named the port tcp-client for example for tcp that would resolve it for Istio.

After renaming the port in service and on the pod spec:

istioctl pc listeners <pod-name> --port 4222
ADDRESS     PORT MATCH DESTINATION
10.96.0.174 4222 ALL   Cluster: outbound|4222||nats.nats-system.svc.cluster.local

Seems to have resolved my connectivity issues, but should be noted the same would need to be done for the other tcp ports.

+28
lukeweber on Jan 23, 2021

If anybody ends up coming here after a Google search, here is my TL;DR

Your kubernetes service should have this value set as its port name: tcp or tcp- as prefix

Thanks!

+2
rhummelmose on Sep 7, 2021

If anybody ends up coming here after a Google search, here is my TL;DR

Your kubernetes service should have this value set as its port name: tcp or tcp- as prefix

+2
tariq1890 on Mar 30, 2021

Hey all, just wanted to put this here for the record. I just spun up a nats cluster using this operator. I created the istio VirtualServices as you would normally do, and everything appears to be working as expected. 3 node cluster is live and seems to be properly clustered. I checked the cluster routes by hitting the /routez endpoint on the management network.

Here are the virtual service definitions:

---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: nats
spec:
  hosts:
  - nats-cluster.nats.svc.cluster.local
  tcp:
  - match:
    - port: 4222
    route:
    - destination:
        host: nats-cluster.nats.svc.cluster.local
        port:
          number: 4222
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: nats-management
spec:
  hosts:
  - nats-cluster-mgmt.nats.svc.cluster.local
  tcp:
  - match:
    - port: 8222
    route:
    - destination:
        host: nats-cluster-mgmt.nats.svc.cluster.local
        port:
          number: 8222
  - match:
    - port: 6222
    route:
    - destination:
        host: nats-cluster-mgmt.nats.svc.cluster.local
        port:
          number: 6222
+2
thedodd on Jan 13, 2019

sure 😃 , will do it before end of this week. @wallyqs

+1
narenarjun on Mar 29, 2021

thanks @narenarjun will see if can add a page on this, or feel free to make a PR to the docs too that can be found here: https://github.com/nats-io/nats.docs/tree/master/nats-on-kubernetes

+1
wallyqs on Mar 29, 2021
Read more comments on GitHub
← nats.net: Unity clients fail with IL2CPP builds
nats.rs: all async operations block forever at 500 concurrent subscriptions →