istio: rabbitmq fails to start with sidecar proxy
With this deployment file:
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
labels:
app: develop-rabbitmq
name: develop-rabbitmq
namespace: develop
spec:
replicas: 1
selector:
matchLabels:
app: develop-rabbitmq
template:
metadata:
labels:
app: develop-rabbitmq
name: develop-rabbitmq
spec:
containers:
- image: rabbitmq:alpine
name: develop-rabbitmq
fails with:
Starting RabbitMQ 3.7.4 on Erlang 20.1.7
Copyright (C) 2007-2018 Pivotal Software, Inc.
Licensed under the MPL. See http://www.rabbitmq.com/
## ##
## ## RabbitMQ 3.7.4. Copyright (C) 2007-2018 Pivotal Software, Inc.
########## Licensed under the MPL. See http://www.rabbitmq.com/
###### ##
########## Logs: <stdout>
Starting broker...
2018-06-02 18:45:15.951 [info] <0.183.0>
node : rabbit@develop-rabbitmq-5f49ffb5cb-qjcnz
home dir : /var/lib/rabbitmq
config file(s) : /etc/rabbitmq/rabbitmq.conf
cookie hash : l/AGo43Mz6TU8itvC2PZqQ==
log(s) : <stdout>
database dir : /var/lib/rabbitmq/mnesia/rabbit@develop-rabbitmq-5f49ffb5cb-qjcnz
2018-06-02 18:45:18.332 [info] <0.191.0> Memory high watermark set to 103196 MiB (108209038950 bytes) of 257990 MiB (270522597376 bytes) total
2018-06-02 18:45:18.337 [info] <0.193.0> Enabling free disk space monitoring
2018-06-02 18:45:18.337 [info] <0.193.0> Disk free limit set to 50MB
2018-06-02 18:45:18.341 [info] <0.195.0> Limiting to approx 1048476 file handles (943626 sockets)
2018-06-02 18:45:18.341 [info] <0.196.0> FHC read buffering: OFF
2018-06-02 18:45:18.341 [info] <0.196.0> FHC write buffering: ON
2018-06-02 18:45:18.345 [info] <0.183.0> Node database directory at /var/lib/rabbitmq/mnesia/rabbit@develop-rabbitmq-5f49ffb5cb-qjcnz is empty. Assuming we need to join an existing cluster or initialise from scratch...
2018-06-02 18:45:18.345 [info] <0.183.0> Configured peer discovery backend: rabbit_peer_discovery_classic_config
2018-06-02 18:45:18.345 [info] <0.183.0> Will try to lock with peer discovery backend rabbit_peer_discovery_classic_config
2018-06-02 18:45:18.345 [info] <0.183.0> Peer discovery backend does not support locking, falling back to randomized delay
2018-06-02 18:45:18.345 [info] <0.183.0> Peer discovery backend rabbit_peer_discovery_classic_config does not support registration, skipping randomized startup delay.
2018-06-02 18:45:18.346 [info] <0.183.0> All discovered existing cluster peers:
2018-06-02 18:45:18.346 [info] <0.183.0> Discovered no peer nodes to cluster with
2018-06-02 18:45:18.352 [info] <0.33.0> Application mnesia exited with reason: stopped
2018-06-02 18:45:18.390 [info] <0.33.0> Application mnesia started on node 'rabbit@develop-rabbitmq-5f49ffb5cb-qjcnz'
2018-06-02 18:45:18.663 [info] <0.183.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2018-06-02 18:45:18.691 [info] <0.183.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2018-06-02 18:45:18.744 [info] <0.183.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2018-06-02 18:45:18.744 [info] <0.183.0> Peer discovery backend rabbit_peer_discovery_classic_config does not support registration, skipping registration.
2018-06-02 18:45:18.746 [info] <0.183.0> Priority queues enabled, real BQ is rabbit_variable_queue
2018-06-02 18:45:18.750 [error] <0.380.0> CRASH REPORT Process <0.380.0> with 0 neighbours crashed with reason: no match of right hand value closed in rabbit_epmd_monitor:init/1 line 56
2018-06-02 18:45:18.750 [error] <0.379.0> Supervisor rabbit_epmd_monitor_sup had child rabbit_epmd_monitor started with rabbit_epmd_monitor:start_link() at undefined exit with reason no match of right hand value closed in rabbit_epmd_monitor:init/1 line 56 in context start_error
2018-06-02 18:45:18.751 [error] <0.182.0> CRASH REPORT Process <0.182.0> with 0 neighbours exited with reason: {error,{{shutdown,{failed_to_start_child,rabbit_epmd_monitor,{{badmatch,closed},[{rabbit_epmd_monitor,init,1,[{file,"src/rabbit_epmd_monitor.erl"},{line,56}]},{gen_server,init_it,2,[{file,"gen_server.erl"},{line,365}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,333}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,247}]}]}}},{child,undefined,rabbit_epmd_monitor_sup,{rabbit_restartable_sup,start_link,[rabbit_epmd_monitor_sup,{rabbit_epmd_monitor,start_link,[]},false]},transient,...}}} in application_master:init/4 line 134
2018-06-02 18:45:18.751 [info] <0.33.0> Application rabbit exited with reason: {error,{{shutdown,{failed_to_start_child,rabbit_epmd_monitor,{{badmatch,closed},[{rabbit_epmd_monitor,init,1,[{file,"src/rabbit_epmd_monitor.erl"},{line,56}]},{gen_server,init_it,2,[{file,"gen_server.erl"},{line,365}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,333}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,247}]}]}}},{child,undefined,rabbit_epmd_monitor_sup,{rabbit_restartable_sup,start_link,[rabbit_epmd_monitor_sup,{rabbit_epmd_monitor,start_link,[]},false]},transient,...}}}
{"Kernel pid terminated",application_controller,"{application_start_failure,rabbit,{bad_return,{{rabbit,start,[normal,[]]},{'EXIT',{error,{{shutdown,{failed_to_start_child,rabbit_epmd_monitor,{{badmatch,closed},[{rabbit_epmd_monitor,init,1,[{file,\"src/rabbit_epmd_monitor.erl\"},{line,56}]},{gen_server,init_it,2,[{file,\"gen_server.erl\"},{line,365}]},{gen_server,init_it,6,[{file,\"gen_server.erl\"},{line,333}]},{proc_lib,init_p_do_apply,3,[{file,\"proc_lib.erl\"},{line,247}]}]}}},{child,undefined,rabbit_epmd_monitor_sup,{rabbit_restartable_sup,start_link,[rabbit_epmd_monitor_sup,{rabbit_epmd_monitor,start_link,[]},false]},transient,infinity,supervisor,[rabbit_restartable_sup]}}}}}}}"}
Kernel pid terminated (application_controller) ({application_start_failure,rabbit,{bad_return,{{rabbit,start,[normal,[]]},{'EXIT',{error,{{shutdown,{failed_to_start_child,rabbit_epmd_monitor,{{badmatc
Crash dump is being written to: /var/log/rabbitmq/erl_crash.dump...done
with sidecar.istio.io/inject: "false"
works, some else having a similar problem?
About this issue
- Original URL
- State: closed
- Created 6 years ago
- Comments: 17 (2 by maintainers)
@lucj you need to create a service specifying all ports that you are using (
dist
,management
,node
andepmd
). Without this, the sidecar container will not allow the pod to make connections to its own ip address.For your deployment, the service should look like this:
Thanks @eduardobaitello that fixed the thing.
I am trying to deploy stable/rabbitmq-ha helm chart with istio sidecar injection enabled. The helm chart deploys rabitmq as statefulset with headless
rabbitmq-ha-devops-rabbitmq-ha-discovery
service andrabbitmq-ha-devops-rabbitmq-ha
clusterIP service.RabbitMQ crashes with bellow error even though
epmd
port is defined in service definition:Service definition for
rabbitmq-ha-devops-rabbitmq-ha
:Service definition for
rabbitmq-ha-devops-rabbitmq-ha-discovery
:Also as per #6029 , RabbitMQ is listening on 0.0.0.0 and not on external IP.
You are missing the
Service
. If you’ll also create something like the following the Pod should start without crashing:But you don’t have the
epmd
port specified there.Well, excluding epmd solved the issue, i’ve written a full description of this here: https://github.com/arielb135/RabbitMQ-with-istio-MTLS