istio: rabbitmq fails to start with sidecar proxy

With this deployment file:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  labels:
    app: develop-rabbitmq
  name: develop-rabbitmq
  namespace: develop
spec:
  replicas: 1
  selector:
    matchLabels:
      app: develop-rabbitmq
  template:
    metadata:
      labels:
        app: develop-rabbitmq
      name: develop-rabbitmq
    spec:
      containers:
        - image: rabbitmq:alpine
          name: develop-rabbitmq

fails with:

 Starting RabbitMQ 3.7.4 on Erlang 20.1.7
 Copyright (C) 2007-2018 Pivotal Software, Inc.
 Licensed under the MPL.  See http://www.rabbitmq.com/
   ##  ##
  ##  ##      RabbitMQ 3.7.4. Copyright (C) 2007-2018 Pivotal Software, Inc.
  ##########  Licensed under the MPL.  See http://www.rabbitmq.com/
  ######  ##
  ##########  Logs: <stdout>
               Starting broker...
2018-06-02 18:45:15.951 [info] <0.183.0> 
 node           : rabbit@develop-rabbitmq-5f49ffb5cb-qjcnz
 home dir       : /var/lib/rabbitmq
 config file(s) : /etc/rabbitmq/rabbitmq.conf
 cookie hash    : l/AGo43Mz6TU8itvC2PZqQ==
 log(s)         : <stdout>
 database dir   : /var/lib/rabbitmq/mnesia/rabbit@develop-rabbitmq-5f49ffb5cb-qjcnz
2018-06-02 18:45:18.332 [info] <0.191.0> Memory high watermark set to 103196 MiB (108209038950 bytes) of 257990 MiB (270522597376 bytes) total
2018-06-02 18:45:18.337 [info] <0.193.0> Enabling free disk space monitoring
2018-06-02 18:45:18.337 [info] <0.193.0> Disk free limit set to 50MB
2018-06-02 18:45:18.341 [info] <0.195.0> Limiting to approx 1048476 file handles (943626 sockets)
2018-06-02 18:45:18.341 [info] <0.196.0> FHC read buffering:  OFF
2018-06-02 18:45:18.341 [info] <0.196.0> FHC write buffering: ON
2018-06-02 18:45:18.345 [info] <0.183.0> Node database directory at /var/lib/rabbitmq/mnesia/rabbit@develop-rabbitmq-5f49ffb5cb-qjcnz is empty. Assuming we need to join an existing cluster or initialise from scratch...
2018-06-02 18:45:18.345 [info] <0.183.0> Configured peer discovery backend: rabbit_peer_discovery_classic_config
2018-06-02 18:45:18.345 [info] <0.183.0> Will try to lock with peer discovery backend rabbit_peer_discovery_classic_config
2018-06-02 18:45:18.345 [info] <0.183.0> Peer discovery backend does not support locking, falling back to randomized delay
2018-06-02 18:45:18.345 [info] <0.183.0> Peer discovery backend rabbit_peer_discovery_classic_config does not support registration, skipping randomized startup delay.
2018-06-02 18:45:18.346 [info] <0.183.0> All discovered existing cluster peers: 
2018-06-02 18:45:18.346 [info] <0.183.0> Discovered no peer nodes to cluster with
2018-06-02 18:45:18.352 [info] <0.33.0> Application mnesia exited with reason: stopped
2018-06-02 18:45:18.390 [info] <0.33.0> Application mnesia started on node 'rabbit@develop-rabbitmq-5f49ffb5cb-qjcnz'
2018-06-02 18:45:18.663 [info] <0.183.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2018-06-02 18:45:18.691 [info] <0.183.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2018-06-02 18:45:18.744 [info] <0.183.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2018-06-02 18:45:18.744 [info] <0.183.0> Peer discovery backend rabbit_peer_discovery_classic_config does not support registration, skipping registration.
2018-06-02 18:45:18.746 [info] <0.183.0> Priority queues enabled, real BQ is rabbit_variable_queue
2018-06-02 18:45:18.750 [error] <0.380.0> CRASH REPORT Process <0.380.0> with 0 neighbours crashed with reason: no match of right hand value closed in rabbit_epmd_monitor:init/1 line 56
2018-06-02 18:45:18.750 [error] <0.379.0> Supervisor rabbit_epmd_monitor_sup had child rabbit_epmd_monitor started with rabbit_epmd_monitor:start_link() at undefined exit with reason no match of right hand value closed in rabbit_epmd_monitor:init/1 line 56 in context start_error
2018-06-02 18:45:18.751 [error] <0.182.0> CRASH REPORT Process <0.182.0> with 0 neighbours exited with reason: {error,{{shutdown,{failed_to_start_child,rabbit_epmd_monitor,{{badmatch,closed},[{rabbit_epmd_monitor,init,1,[{file,"src/rabbit_epmd_monitor.erl"},{line,56}]},{gen_server,init_it,2,[{file,"gen_server.erl"},{line,365}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,333}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,247}]}]}}},{child,undefined,rabbit_epmd_monitor_sup,{rabbit_restartable_sup,start_link,[rabbit_epmd_monitor_sup,{rabbit_epmd_monitor,start_link,[]},false]},transient,...}}} in application_master:init/4 line 134
2018-06-02 18:45:18.751 [info] <0.33.0> Application rabbit exited with reason: {error,{{shutdown,{failed_to_start_child,rabbit_epmd_monitor,{{badmatch,closed},[{rabbit_epmd_monitor,init,1,[{file,"src/rabbit_epmd_monitor.erl"},{line,56}]},{gen_server,init_it,2,[{file,"gen_server.erl"},{line,365}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,333}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,247}]}]}}},{child,undefined,rabbit_epmd_monitor_sup,{rabbit_restartable_sup,start_link,[rabbit_epmd_monitor_sup,{rabbit_epmd_monitor,start_link,[]},false]},transient,...}}}
{"Kernel pid terminated",application_controller,"{application_start_failure,rabbit,{bad_return,{{rabbit,start,[normal,[]]},{'EXIT',{error,{{shutdown,{failed_to_start_child,rabbit_epmd_monitor,{{badmatch,closed},[{rabbit_epmd_monitor,init,1,[{file,\"src/rabbit_epmd_monitor.erl\"},{line,56}]},{gen_server,init_it,2,[{file,\"gen_server.erl\"},{line,365}]},{gen_server,init_it,6,[{file,\"gen_server.erl\"},{line,333}]},{proc_lib,init_p_do_apply,3,[{file,\"proc_lib.erl\"},{line,247}]}]}}},{child,undefined,rabbit_epmd_monitor_sup,{rabbit_restartable_sup,start_link,[rabbit_epmd_monitor_sup,{rabbit_epmd_monitor,start_link,[]},false]},transient,infinity,supervisor,[rabbit_restartable_sup]}}}}}}}"}
 Kernel pid terminated (application_controller) ({application_start_failure,rabbit,{bad_return,{{rabbit,start,[normal,[]]},{'EXIT',{error,{{shutdown,{failed_to_start_child,rabbit_epmd_monitor,{{badmatc
 
 Crash dump is being written to: /var/log/rabbitmq/erl_crash.dump...done

with sidecar.istio.io/inject: "false" works, some else having a similar problem?

About this issue

  • Original URL
  • State: closed
  • Created 6 years ago
  • Comments: 17 (2 by maintainers)

Most upvoted comments

@lucj you need to create a service specifying all ports that you are using (dist, management, node and epmd). Without this, the sidecar container will not allow the pod to make connections to its own ip address.

For your deployment, the service should look like this:

apiVersion: v1
kind: Service
metadata:
  name: rabbitmq-service
  labels:
    app: mq
spec:
  type: ClusterIP
  ports:
  - name: epmd
    port: 4369
    targetPort: epmd
  - name: node
    port: 5672
    targetPort: node
  - name: dist
    port: 25672
    targetPort: dist
  - name: management
    port: 15672
    targetPort: management
  selector:
    app: mq
+10
eduardobaitello on Oct 17, 2018

Thanks @eduardobaitello that fixed the thing.

+6
devops-traxxs on Oct 17, 2018

I am trying to deploy stable/rabbitmq-ha helm chart with istio sidecar injection enabled. The helm chart deploys rabitmq as statefulset with headless rabbitmq-ha-devops-rabbitmq-ha-discovery service and rabbitmq-ha-devops-rabbitmq-ha clusterIP service.

RabbitMQ crashes with bellow error even though epmd port is defined in service definition:

2018-06-22 05:19:43.982 [error] <0.295.0> CRASH REPORT Process <0.295.0> with 0 neighbours crashed with reason: no function clause matching rabbit_epmd_monitor:init_handle_port_please(closed, erl_epmd, "rabbit", "rabbitmq-ha-devops-rabbitmq-ha-0.rabbitmq-ha-devops-rabbitmq-ha-discovery.devops.svc.cluster.lo...") line 59
2018-06-22 05:19:43.983 [error] <0.294.0> Supervisor rabbit_epmd_monitor_sup had child rabbit_epmd_monitor started with rabbit_epmd_monitor:start_link() at undefined exit with reason no function clause matching rabbit_epmd_monitor:init_handle_port_please(closed, erl_epmd, "rabbit", "rabbitmq-ha-devops-rabbitmq-ha-0.rabbitmq-ha-devops-rabbitmq-ha-discovery.devops.svc.cluster.lo...") line 59 in context start_error
2018-06-22 05:19:43.984 [error] <0.264.0> CRASH REPORT Process <0.264.0> with 0 neighbours exited with reason: {error,{{shutdown,{failed_to_start_child,rabbit_epmd_monitor,{function_clause,[{rabbit_epmd_monitor,init_handle_port_please,[closed,erl_epmd,"rabbit","rabbitmq-ha-devops-rabbitmq-ha-0.rabbitmq-ha-devops-rabbitmq-ha-discovery.devops.svc.cluster.local"],[{file,"src/rabbit_epmd_monitor.erl"},{line,59}]},{gen_server,init_it,2,[{file,"gen_server.erl"},{line,365}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,333}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,247}]}]}}},{child,...}}} in application_master:init/4 line 134
2018-06-22 05:19:43.984 [info] <0.33.0> Application rabbit exited with reason: {error,{{shutdown,{failed_to_start_child,rabbit_epmd_monitor,{function_clause,[{rabbit_epmd_monitor,init_handle_port_please,[closed,erl_epmd,"rabbit","rabbitmq-ha-devops-rabbitmq-ha-0.rabbitmq-ha-devops-rabbitmq-ha-discovery.devops.svc.cluster.local"],[{file,"src/rabbit_epmd_monitor.erl"},{line,59}]},{gen_server,init_it,2,[{file,"gen_server.erl"},{line,365}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,333}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,247}]}]}}},{child,...}}}
{"Kernel pid terminated",application_controller,"{application_start_failure,rabbit,{bad_return,{{rabbit,start,[normal,[]]},{'EXIT',{error,{{shutdown,{failed_to_start_child,rabbit_epmd_monitor,{function_clause,[{rabbit_epmd_monitor,init_handle_port_please,[closed,erl_epmd,\"rabbit\",\"rabbitmq-ha-devops-rabbitmq-ha-0.rabbitmq-ha-devops-rabbitmq-ha-discovery.devops.svc.cluster.local\"],[{file,\"src/rabbit_epmd_monitor.erl\"},{line,59}]},{gen_server,init_it,2,[{file,\"gen_server.erl\"},{line,365}]},{gen_server,init_it,6,[{file,\"gen_server.erl\"},{line,333}]},{proc_lib,init_p_do_apply,3,[{file,\"proc_lib.erl\"},{line,247}]}]}}},{child,undefined,rabbit_epmd_monitor_sup,{rabbit_restartable_sup,start_link,[rabbit_epmd_monitor_sup,{rabbit_epmd_monitor,start_link,[]},false]},transient,infinity,supervisor,[rabbit_restartable_sup]}}}}}}}"}
Kernel pid terminated (application_controller) ({application_start_failure,rabbit,{bad_return,{{rabbit,start,[normal,[]]},{'EXIT',{error,{{shutdown,{failed_to_start_child,rabbit_epmd_monitor,{function

Crash dump is being written to: /var/log/rabbitmq/erl_crash.dump...done

Service definition for rabbitmq-ha-devops-rabbitmq-ha :

apiVersion: v1
kind: Service
metadata:
  labels:
    app: rabbitmq-ha
    chart: rabbitmq-ha-1.3.2
    heritage: Tiller
    release: rabbitmq-ha-devops
  name: rabbitmq-ha-devops-rabbitmq-ha
  namespace: devops
spec:
  clusterIP: *****
  ports:
  - name: http
    port: 15672
    protocol: TCP
    targetPort: http
  - name: amqp
    port: 5672
    protocol: TCP
    targetPort: amqp
  - name: epmd
    port: 4369
    protocol: TCP
    targetPort: epmd
  selector:
    app: rabbitmq-ha
    release: rabbitmq-ha-devops
  sessionAffinity: None
  type: ClusterIP

Service definition for rabbitmq-ha-devops-rabbitmq-ha-discovery:

apiVersion: v1
kind: Service
metadata:
  labels:
    app: rabbitmq-ha
    chart: rabbitmq-ha-1.3.2
    heritage: Tiller
    release: rabbitmq-ha-devops
  name: rabbitmq-ha-devops-rabbitmq-ha-discovery
  namespace: devops
spec:
  clusterIP: None
  ports:
  - name: http
    port: 15672
    protocol: TCP
    targetPort: http
  - name: amqp
    port: 5672
    protocol: TCP
    targetPort: amqp
  - name: epmd
    port: 4369
    protocol: TCP
    targetPort: epmd
  selector:
    app: rabbitmq-ha
    release: rabbitmq-ha-devops
  sessionAffinity: None
  type: ClusterIP

Also as per #6029 , RabbitMQ is listening on 0.0.0.0 and not on external IP.

Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 0.0.0.0:4369            0.0.0.0:*               LISTEN      -
tcp        0      0 0.0.0.0:15672           0.0.0.0:*               LISTEN      -
tcp        0      0 0.0.0.0:25672           0.0.0.0:*               LISTEN      -
tcp        0      0 :::4369                 :::*                    LISTEN      -
tcp        0      0 :::5672                 :::*                    LISTEN      -
udp        0      0 0.0.0.0:49166           0.0.0.0:*                           -
+5
vinayakshnd on Jun 22, 2018

You are missing the Service. If you’ll also create something like the following the Pod should start without crashing:

apiVersion: v1
kind: Service
metadata:
  name: rabbitmq
  namespace: develop
  labels:
    app: develop-rabbitmq
spec:
  ports:
  - port: 5672
    name: client
  - port: 4369
    name: epmd
  selector:
    app: develop-rabbitmq
---
+3
ymesika on Jun 3, 2018

But you don’t have the epmd port specified there.

+2
ymesika on Jun 4, 2018

@arielb135 if all nodes have the sidecar mTLS should be seamless, as to the nodes themselves traffic appears as coming from localhost & unencrypted. IIRC.

Well, excluding epmd solved the issue, i’ve written a full description of this here: https://github.com/arielb135/RabbitMQ-with-istio-MTLS

+1
arielb135 on Apr 16, 2019
Read more comments on GitHub
← istio: proxy-postsubmit failed to build proxy
istio: Randomly get 503 UR upstream_reset_before_response_started{remote_reset} →