rqlite: DNS not resolved in '-raft-adv-addr' after leader goes down

I’ve managed to create an rqlite (5.5.0) cluster on Kubernetes (as a StatefulSet) and it comes up perfectly fine. It can even handle followers going down and it re-registers them with the new pod IPs, which is great.

However, when killing all nodes, the leader comes back and tries to ping the old nodes which no longer exist as the IPs changed, and everything gets stuck. I was expecting the leader to resolve DNS again for the other raft nodes and try to re-establish the cluster, but it looks like it only resolves DNS the first time or when followers re-join. -raft-adv-addr should probably be resolved on-demand and the IP not saved to the raft DB.

Leader errors after killing all nodes and getting new IPs:

2020-11-26T09:05:49.321Z [INFO]  raft: Node at [::]:4002 [Candidate] entering Candidate state in term 29043
2020-11-26T09:05:49.322Z [ERROR] raft: Failed to make RequestVote RPC to {Voter node1 100.72.3.247:4002}: dial tcp 100.72.3.247:4002: connect: no route to host
2020-11-26T09:05:49.322Z [ERROR] raft: Failed to make RequestVote RPC to {Voter node1 100.72.3.247:4002}: dial tcp 100.72.3.247:4002: connect: no route to host
2020-11-26T09:05:49.323Z [ERROR] raft: Failed to make RequestVote RPC to {Voter node2 100.74.1.150:4002}: dial tcp 100.74.1.150:4002: connect: no route to host
2020-11-26T09:05:49.323Z [ERROR] raft: Failed to make RequestVote RPC to {Voter node2 100.74.1.150:4002}: dial tcp 100.74.1.150:4002: connect: no route to host
2020-11-26T09:05:50.589Z [WARN]  raft: Election timeout reached, restarting election

node0

rqlited \
  -node-id=node0 \
  -http-addr=0.0.0.0:4001 \
  -raft-addr=0.0.0.0:4002 \
  -http-adv-addr=rqlite-0.rqlite:4001 \
  -raft-adv-addr=rqlite-0.rqlite:4002 \
  /data

node1 and node2

rqlited \
  -node-id=node1 \
  -http-addr=0.0.0.0:4001 \
  -raft-addr=0.0.0.0:4002 \
  -http-adv-addr=rqlite-1.rqlite:4001 \
  -raft-adv-addr=rqlite-1.rqlite:4002 \
  -join=http://rqlite-0.rqlite:4001 \
  /data

rqlited \
  -node-id=node2 \
  -http-addr=0.0.0.0:4001 \
  -raft-addr=0.0.0.0:4002 \
  -http-adv-addr=rqlite-2.rqlite:4001 \
  -raft-adv-addr=rqlite-2.rqlite:4002 \
  -join=http://rqlite-0.rqlite:4001 \
  /data

/status?pretty

...
"leader": {
    "addr": "[::]:4002",
    "node_id": "node0"
},
"metadata": {
    "node0": {
        "api_addr": "rqlite-0.rqlite:4001",
        "api_proto": "http"
    },
    "node1": {
        "api_addr": "rqlite-1.rqlite:4001",
        "api_proto": "http"
    },
    "node2": {
        "api_addr": "rqlite-2.rqlite:4001",
        "api_proto": "http"
    }
},
"node_id": "node0",
"nodes": [
    {
        "id": "node0",
        "addr": "[::]:4002"
    },
    {
        "id": "node1",
        "addr": "100.72.3.247:4002"  <--- Should be DNS
    },
    {
        "id": "node2",
        "addr": "100.74.1.150:4002"  <--- Should be DNS
    }
],
...

Is this expected behavior and maybe I’m just using the flags wrong? Any advice would be much appreciated!

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Comments: 18 (15 by maintainers)

Most upvoted comments

Trying out removing the resolution in this PR: https://github.com/rqlite/rqlite/pull/697

Well, well, I forgot how my own code works:

https://github.com/rqlite/rqlite/blob/master/cluster/join.go#L58

The rqlite layer does resolve addresses before sending the details to the node it is joining. However I still believe Hashicorp Raft takes this address, resolves it, and stores that in its internal config.