firecamp: Kafka Zookeeper logs show connection refused

I used Cloudformation template firecamp-existingvpc to roll out the environment. Then created a zookeeper service according to the instruction (for Kafka). Here is what is in the Cloudwatch (firecamp-qa-zoo-qa) logs:

2017-12-01 08:20:31,031 [myid:3] - INFO [zoo-qa-2.firecamp-qa-firecamp.com/172.22.5.201:3888:QuorumPeer$QuorumServer@167] - Resolved hostname: zoo-qa-1.firecamp-qa-firecamp.com to address: zoo-qa-1.firecamp-qa-firecamp.com/172.22.2.62
2017-12-01 08:21:31,029 [myid:3] - INFO [zoo-qa-2.firecamp-qa-firecamp.com/172.22.5.201:3888:QuorumCnxManager$Listener@746] - Received connection request /172.22.2.62:59352
2017-12-01 08:21:31,030 [myid:3] - WARN [zoo-qa-2.firecamp-qa-firecamp.com/172.22.5.201:3888:QuorumCnxManager@588] - Cannot open channel to 2 at election address zoo-qa-1.firecamp-qa-firecamp.com/172.22.2.62:3888
java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:562)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.handleConnection(QuorumCnxManager.java:479)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.receiveConnection(QuorumCnxManager.java:379)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$Listener.run(QuorumCnxManager.java:757)
2017-12-01 08:21:31,033 [myid:3] - INFO [zoo-qa-2.firecamp-qa-firecamp.com/172.22.5.201:3888:QuorumPeer$QuorumServer@167] - Resolved hostname: zoo-qa-1.firecamp-qa-firecamp.com to address: zoo-qa-1.firecamp-qa-firecamp.com/172.22.2.62
2017-12-01 08:22:31,031 [myid:3] - INFO [zoo-qa-2.firecamp-qa-firecamp.com/172.22.5.201:3888:QuorumCnxManager$Listener@746] - Received connection request /172.22.2.62:59356
2017-12-01 08:22:31,032 [myid:3] - WARN [zoo-qa-2.firecamp-qa-firecamp.com/172.22.5.201:3888:QuorumCnxManager@588] - Cannot open channel to 2 at election address zoo-qa-1.firecamp-qa-firecamp.com/172.22.2.62:3888
java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:562)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.handleConnection(QuorumCnxManager.java:479)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.receiveConnection(QuorumCnxManager.java:379)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$Listener.run(QuorumCnxManager.java:757)

Is that OK?

I see that 172.22.2.62 has an established connection with 3rd zookeeper instance (1.119):

[root@ip-172-22-2-62 ec2-user]# netstat -anp|grep 3888
tcp        0      0 ::ffff:172.22.2.62:59908    ::ffff:172.22.5.201:3888    TIME_WAIT   -
tcp        0      0 ::ffff:172.22.2.62:52050    ::ffff:172.22.1.119:3888    ESTABLISHED 3637/java

About this issue

  • Original URL
  • State: closed
  • Created 7 years ago
  • Comments: 15 (15 by maintainers)

Most upvoted comments

Unfortunately, no. Changing the heap size on the running cluster is actually a service update operation. This is currently not supported yet. This is issue “Implement update-service operation in firecamp-service-cli”. Will support it when fixing that issue.

the fix for cassandra configurable heap size is committed to master. please let us know if you hit any issue!