orchestrator: Maybe a bug during recovery.

Hi @shlomi-noach , As mentioned in https://github.com/github/orchestrator/issues/419: I’ve used RecoveryIgnoreHostnameFilters and still it gets promoted. The topology looks like this:

[root@mysql-sredb02.xh ~]# orchestrator-client -c topology -i mysql-sredb03.yp
mysql-sredb03.yp:3306   [0s,ok,5.6.24-72.2-log,rw,ROW,>>,GTID]
+ mysql-sredb05.yp:3306 [0s,ok,5.6.24-72.2-log,rw,ROW,>>,GTID]
+ mysql-sredb06.yp:3306 [0s,ok,5.6.24-72.2-log,rw,ROW,>>,GTID]
+ mysql-sredb07.yp:3306 [0s,ok,5.6.24-72.2-log,rw,ROW,>>,GTID]

image

I’ve configure like this:

"RecoveryIgnoreHostnameFilters": [
    "mysql-sredb06.yp",
    "mysql-sredb05.yp"
  ],

When I stop the MySQL server on mysql-sredb03.yp, the mysql-sredb06.yp has been promoted.

2018-03-05 14:33:48 Detected DeadMaster on mysql-sredb03.yp:3306. Affected replicas: 3
2018-03-05 14:33:48 Will recover from DeadMaster on mysql-sredb03.yp:3306
2018-03-05 14:33:50 Detected DeadMaster on mysql-sredb03.yp:3306. Affected replicas: 1
2018-03-05 14:33:50 Recovered from DeadMaster on mysql-sredb03.yp:3306. Failed: mysql-sredb03.yp:3306; Promoted: mysql-sredb06.yp:3306
2018-03-05 14:33:50 (for all types) Recovered from DeadMaster on mysql-sredb03.yp:3306. Failed: mysql-sredb03.yp:3306; Successor: mysql-sredb06.yp:3306

Are there something wrong?

About this issue

Most upvoted comments

I do not reproduce the same problem now… When it happens again, I will tell you. Thanks.

Also can you please confirm you have restarted orchestrator after making configuration changes,

Yes, I have restarted orchestrator after making configuration changes.

Can you please clarify: does this behavior reproduce? Any time you run two successive failovers, one of the forbidden servers is promoted on the 2nd attempt?

I will test more times to confirm this. I have reproduced several times, but I don’t know whether its happens every 2nd times. Need I acknowledge the recoveries every time? I think its not necessary because the value of “RecoveryPeriodBlockSeconds” is 10.