orchestrator: Unable to promote slave

When promoting a slave with 3.0.6 we get:

[martini] Started GET /orchestrator/api/make-co-master/bc-db-integration-01/3306 for 127.0.0.1:42230
[mysql] 2018/02/27 21:37:10 packets.go:36: unexpected EOF
2018-02-27 21:37:10 ERROR invalid connection
2018-02-27 21:37:10 ERROR invalid connection
2018-02-27 21:37:10 ERROR invalid connection
[martini] Completed 500 Internal Server Error in 34.669359ms

Sometimes we are able to get past this error (manually doing change master to etc) and then when trying to fail back or do any operation we get:

2018-02-27 23:48:38 INFO starting orchestrator, version: 3.0.6, git commit: a77ee001dc95b8dc0476b929164f37b1141f54c1
2018-02-27 23:48:38 INFO Read config: /etc/orchestrator.conf.json
2018-02-27 23:48:38 DEBUG Hostname unresolved yet: bc-db-integration-01
2018-02-27 23:48:38 DEBUG Cache hostname resolve bc-db-integration-01 as bc-db-integration-01
2018-02-27 23:48:38 DEBUG Connected to orchestrator backend: orchestrator:?@tcp(127.0.0.1:3306)/orchestrator?timeout=1s
2018-02-27 23:48:38 DEBUG Orchestrator pool SetMaxOpenConns: 128
2018-02-27 23:48:38 DEBUG Initializing orchestrator
2018-02-27 23:48:38 INFO Connecting to backend: maxConnections: 128, maxIdleConns: 32
2018-02-27 23:48:38 DEBUG Hostname unresolved yet: bc-db-integration
2018-02-27 23:48:38 DEBUG Cache hostname resolve bc-db-integration as bc-db-integration
2018-02-27 23:48:38 DEBUG Hostname unresolved yet: bc-db-integration
2018-02-27 23:48:38 DEBUG Cache hostname resolve bc-db-integration as bc-db-integration
2018-02-27 23:48:38 INFO Will detach bc-db-integration-01:3306
2018-02-27 23:48:38 INFO auditType:begin-maintenance instance:bc-db-integration-01:3306 cluster:bc-db-integration:3306 message:maintenanceToken: 33, owner: root, reason: detach replica
2018-02-27 23:48:38 INFO Stopped slave on bc-db-integration-01:3306, Self:binlog.000002:679, Exec:binlog.000001:924
2018-02-27 23:48:38 ERROR Error 1776: Parameters MASTER_LOG_FILE, MASTER_LOG_POS, RELAY_LOG_FILE and RELAY_LOG_POS cannot be set when MASTER_AUTO_POSITION is active.
2018-02-27 23:48:38 ERROR Error 1776: Parameters MASTER_LOG_FILE, MASTER_LOG_POS, RELAY_LOG_FILE and RELAY_LOG_POS cannot be set when MASTER_AUTO_POSITION is active.
2018-02-27 23:48:38 INFO Started slave on bc-db-integration-01:3306
2018-02-27 23:48:38 ERROR Error 1776: Parameters MASTER_LOG_FILE, MASTER_LOG_POS, RELAY_LOG_FILE and RELAY_LOG_POS cannot be set when MASTER_AUTO_POSITION is active.
2018-02-27 23:48:38 INFO auditType:end-maintenance instance:bc-db-integration-01:3306 cluster:bc-db-integration:3306 message:maintenanceToken: 33
2018-02-27 23:48:38 FATAL Error 1776: Parameters MASTER_LOG_FILE, MASTER_LOG_POS, RELAY_LOG_FILE and RELAY_LOG_POS cannot be set when MASTER_AUTO_POSITION is active.

We looked at some previous tickets and changed grants as well:

+-----------------------------------------------------------------------------------------------------------------------------------------------------------+
| Grants for orchestrator@10.20.0.32                                                                                                                       |
+-----------------------------------------------------------------------------------------------------------------------------------------------------------+
| GRANT RELOAD, PROCESS, SUPER, REPLICATION SLAVE ON *.* TO 'orchestrator'@'10.20.0.32' IDENTIFIED BY PASSWORD '....' |
| GRANT ALL PRIVILEGES ON `mysql`.`slave_master_info` TO 'orchestrator'@'10.20.0.32'                                                                       |
+-----------------------------------------------------------------------------------------------------------------------------------------------------------+

cc @PsyonixEvan

About this issue

  • Original URL
  • State: open
  • Created 6 years ago
  • Comments: 15 (4 by maintainers)

Most upvoted comments

@shlomi-noach That’ll take a bit (it might be a few days), but yep I can do so. I’ll get back to you when I have done so.