autobase: How does the failover really works

Hello guys,

I’m doing some testing with my postgres cluster in which I am turning off (poweroff) one by one either the leader, the replica or the standby node.

During this testing I have seen that there could be some problems with either my config or my cluster…

More exactly, if I am left with the leader and the standby node, when I poweroff the leader there is no switchover of the standby node happening. Shouldn’t this node become a leader / takeover?

In the following case the leader is poweroff but somehow still listed in patronictl as running???

root@db2:~# patronictl list
2020-12-27 21:04:02,289 - ERROR - Failed to get list of machines from http://10.10.11.6:2379/v2: MaxRetryError("HTTPConnectionPool(host='10.10.11.6', port=2379): Max retries exceeded with url: /v2/machines (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7fd0aa8139a0>, 'Connection to 10.10.11.6 timed out. (connect timeout=1.6666666666666667)'))")
2020-12-27 21:04:03,961 - ERROR - Failed to get list of machines from http://10.10.11.4:2379/v2: MaxRetryError("HTTPConnectionPool(host='10.10.11.4', port=2379): Max retries exceeded with url: /v2/machines (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7fd0aa813bb0>, 'Connection to 10.10.11.4 timed out. (connect timeout=1.6666666666666667)'))")
+ Cluster: test (6883197882048089956) ---+----+-----------+--------------+
| Member | Host       | Role         | State   | TL | Lag in MB | Tags         |
+--------+------------+--------------+---------+----+-----------+--------------+
| db1    | 10.10.11.4 | Leader       | running |  5 |           | nosync: true |
| db2    | 10.10.11.5 | Sync Standby | running |  5 |         0 |              |

After I poweron the leader the roles switched and the other one becomes a replica…

root@db2:~# patronictl list
2020-12-27 21:11:25,254 - ERROR - Failed to get list of machines from http://10.10.11.6:2379/v2: MaxRetryError("HTTPConnectionPool(host='10.10.11.6', port=2379): Max retries exceeded with url: /v2/machines (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7fb4c47ab9a0>, 'Connection to 10.10.11.6 timed out. (connect timeout=1.6666666666666667)'))")
+ Cluster: test (6883197882048089956) ---+-----------+
| Member | Host       | Role    | State   | TL | Lag in MB |
+--------+------------+---------+---------+----+-----------+
| db1    | 10.10.11.4 | Replica | running |  5 |         0 |
| db2    | 10.10.11.5 | Leader  | running |  6 |           |
+--------+------------+---------+---------+----+-----------+

Another issue is after I poweron the other 2 servers and they join the cluster both of them are replicas. Shouldn’t there be a standby node anymore?

+ Cluster: test (6883197882048089956) ---+-----------+
| Member | Host       | Role    | State   | TL | Lag in MB |
+--------+------------+---------+---------+----+-----------+
| db0    | 10.10.11.6 | Leader  | running |  4 |           |
| db1    | 10.10.11.4 | Replica | running |  4 |         0 |
| db2    | 10.10.11.5 | Replica | running |  4 |         0 |
+--------+------------+---------+---------+----+-----------+

After a reboot of all 3 nodes at the same time the cluster is back to it’s initial state

root@db0:~# patronictl list
+ Cluster: test (6883197882048089956) ---+----+-----------+
| Member | Host       | Role         | State   | TL | Lag in MB |
+--------+------------+--------------+---------+----+-----------+
| db0    | 10.10.11.6 | Replica      | running |  5 |         0 |
| db1    | 10.10.11.4 | Sync Standby | running |  5 |         0 |
| db2    | 10.10.11.5 | Leader       | running |  5 |           |
+--------+------------+--------------+---------+----+-----------+

I would appreciate it very much if someone could explain to me how does the replica and sync standby nodes should behave when the leader is gone.

Thank you!!!

About this issue

  • Original URL
  • State: closed
  • Created 6 years ago
  • Comments: 18 (10 by maintainers)

Commits related to this issue

Most upvoted comments

Hi @vitabaks ,

Thank you for the fix & support. I was able to update & redeploy a cluster with the updated playbook and everything is configured as expected.

Thank you & Happy New Year!