gh-ost: gh-ost hang with statement based binlogs

Running gh-ost with these 3 options:

--assume-master-host=the-master-IP 
--tungsten 
--allow-on-master

on a setup where the master is using statement based binlogs, results on gh-ost hanging in this phase:

2016-10-03 15:06:05 INFO starting gh-ost 1.0.20
2016-10-03 15:06:05 INFO Migrating `test`.`test_table`
./gh-ost \
2016-10-03 15:06:05 INFO connection validated on 10.1.1.52:3306
2016-10-03 15:06:05 INFO User has SUPER, REPLICATION SLAVE privileges, and has ALL privileges on `test`.*
2016-10-03 15:06:05 INFO binary logs validated on 10.1.1.52:3306
2016-10-03 15:06:05 INFO Restarting replication on 10.1.1.52:3306 to make sure binlog settings apply to replication thread
2016-10-03 15:06:05 INFO Table found. Engine=InnoDB
2016-10-03 15:06:05 INFO Estimated number of rows via EXPLAIN: 1108
2016-10-03 15:06:05 INFO binary logs updates validated on 10.1.1.51:3306
2016-10-03 15:06:05 INFO Master found to be db1:3306
2016-10-03 15:06:05 INFO connection validated on 10.1.1.51:3306
2016-10-03 15:06:05 INFO Registering replica at 10.1.1.51:3306
2016-10-03 15:06:05 INFO Connecting binlog streamer at db0-bin.000002:1905053
2016-10-03 15:06:05 INFO connection validated on 10.1.1.51:3306
2016-10-03 15:06:05 INFO rotate to next log name: db0-bin.000002
2016-10-03 15:06:05 INFO connection validated on 10.1.1.51:3306
2016-10-03 15:06:05 INFO Droppping table `test`.`_test_table_gho`
2016-10-03 15:06:05 INFO Table dropped
2016-10-03 15:06:05 INFO Droppping table `test`.`_test_table_del`
2016-10-03 15:06:05 INFO Table dropped
2016-10-03 15:06:05 INFO Droppping table `test`.`_test_table_ghc`
2016-10-03 15:06:05 INFO Table dropped
2016-10-03 15:06:05 INFO Creating changelog table `test`.`_test_table_ghc`
2016-10-03 15:06:05 INFO Changelog table created
2016-10-03 15:06:05 INFO Creating ghost table `test`.`_test_table_gho`
2016-10-03 15:06:05 INFO Ghost table created
2016-10-03 15:06:05 INFO Altering ghost table `test`.`_test_table_gho`
2016-10-03 15:06:05 INFO Ghost table altered
2016-10-03 15:06:05 INFO Waiting for tables to be in place

About this issue

Original URL
State: closed
Created 8 years ago
Comments: 18 (11 by maintainers)

Most upvoted comments

Master is 10.1.1.51 and slave is 10.1.1.52:

./gh-ost \
--assume-master-host=10.1.1.51 \
--allow-on-master \
--tungsten \
--max-load=Threads_running=25 \
--critical-load=Threads_running=1000 \
--chunk-size=1000 \
--max-lag-millis=1500 \
--user="ghost" \
--password="ghost" \
--host=10.1.1.52 \
--database="test" \
--table="incoming_file_job" \
--verbose \
--alter="engine=innodb" \
--cut-over=default \
--default-retries=120 \
--panic-flag-file=/tmp/ghost.panic.flag \
--postpone-cut-over-flag-file=/tmp/ghost.postpone.flag \
--initially-drop-ghost-table \
--initially-drop-old-table \
--exact-rowcount \
--execute \
--debug

2016-10-06 11:52:58 INFO starting gh-ost 1.0.20
2016-10-06 11:52:58 INFO Migrating `test`.`incoming_file_job`
2016-10-06 11:52:58 INFO connection validated on 10.1.1.52:3306
2016-10-06 11:52:58 INFO User has SUPER, REPLICATION SLAVE privileges, and has ALL privileges on `test`.*
2016-10-06 11:52:58 INFO binary logs validated on 10.1.1.52:3306
2016-10-06 11:52:58 INFO Restarting replication on 10.1.1.52:3306 to make sure binlog settings apply to replication thread
2016-10-06 11:52:58 INFO Table found. Engine=InnoDB
2016-10-06 11:52:58 DEBUG Estimated number of rows via STATUS: 1123
2016-10-06 11:52:58 DEBUG Validated no foreign keys exist on table
2016-10-06 11:52:58 DEBUG Validated no triggers exist on table
2016-10-06 11:52:58 INFO Estimated number of rows via EXPLAIN: 1123
2016-10-06 11:52:58 DEBUG Potential unique keys in incoming_file_job: [PRIMARY (auto_increment): [job_id]; has nullable: false path: [path]; has nullable: true]
2016-10-06 11:52:58 DEBUG Looking for master on 10.1.1.52:3306
2016-10-06 11:52:58 INFO binary logs updates validated on 10.1.1.51:3306
2016-10-06 11:52:58 INFO Master found to be db1:3306
2016-10-06 11:52:58 INFO connection validated on 10.1.1.51:3306
2016-10-06 11:52:58 DEBUG Streamer binlog coordinates: db0-bin.000002:4547978
2016-10-06 11:52:58 INFO Registering replica at 10.1.1.51:3306
2016-10-06 11:52:58 INFO Connecting binlog streamer at db0-bin.000002:4547978
2016-10-06 11:52:58 DEBUG Beginning streaming
2016-10-06 11:52:58 INFO rotate to next log name: db0-bin.000002
2016-10-06 11:52:58 INFO connection validated on 10.1.1.51:3306
2016-10-06 11:52:58 INFO connection validated on 10.1.1.51:3306
2016-10-06 11:52:58 INFO Droppping table `test`.`_incoming_file_job_gho`
2016-10-06 11:52:58 INFO Table dropped
2016-10-06 11:52:58 INFO Droppping table `test`.`_incoming_file_job_del`
2016-10-06 11:52:58 INFO Table dropped
2016-10-06 11:52:58 INFO Droppping table `test`.`_incoming_file_job_ghc`
2016-10-06 11:52:58 INFO Table dropped
2016-10-06 11:52:58 INFO Creating changelog table `test`.`_incoming_file_job_ghc`
2016-10-06 11:52:58 INFO Changelog table created
2016-10-06 11:52:58 INFO Creating ghost table `test`.`_incoming_file_job_gho`
2016-10-06 11:52:58 INFO Ghost table created
2016-10-06 11:52:58 INFO Altering ghost table `test`.`_incoming_file_job_gho`
2016-10-06 11:52:58 DEBUG ALTER statement: alter /* gh-ost */ table `test`.`_incoming_file_job_gho` engine=innodb
2016-10-06 11:52:58 INFO Ghost table altered
2016-10-06 11:52:58 INFO Waiting for tables to be in place

igroene on Oct 6, 2016