gh-ost: StreamEvents encountered unexpected error: Err: runtime error: index out of range

I tried to migrate table with 200M records and I it crashed after 24 hours on 89% with exception runtime error: index out of range. I did the same migration on staging successfully.

I use MySQL 5.7.11 on AWS RDS Run gh-ost v1.0.20 on Ubuntu 14.04

./gh-ost \
  --assume-rbr \
  --max-load=Threads_running=25 \
  --critical-load=Threads_running=1000 \
  --chunk-size=1000 \
  --nice-ratio=2 \
  --throttle-control-replicas='sql02.xxx.us-east-1.rds.amazonaws.com' \
  --max-lag-millis=2000 \
  --host='sql01.xxx.us-east-1.rds.amazonaws.com' \
  --user='root' \
  --password='***' \
  --database='production' \
  --table='origins' \
  --verbose \
  --alter="CHANGE COLUMN deprecated_order_date deprecated_order_date datetime" \
  --allow-on-master \
  --cut-over=default \
  --exact-rowcount \
  --default-retries=120 \
  --panic-flag-file=/tmp/ghost.panic.flag \
  --postpone-cut-over-flag-file=/tmp/ghost.postpone.flag \
  --throttle-additional-flag-file=/tmp/ghost.throttle.flag \
  --initially-drop-ghost-table \
  --execute

# chunk-size: 3000; max-lag-millis: 2000ms; max-load: Threads_running=25; critical-load: Threads_running=1000; nice-ratio: 1.000000
# throttle-additional-flag-file: /tmp/ghost.throttle.flag
# postpone-cut-over-flag-file: /tmp/ghost.postpone.flag [set]
# panic-flag-file: /tmp/ghost.panic.flag
# Serving on unix socket: /tmp/gh-ost.production.origins.sock
Copy: 207141700/232068606 89.3%; Applied: 1374017; Backlog: 0/100; Time: 24h40m0s(total), 24h31m18s(copy); streamer: mysql-bin-changelog.117854:38741575; State: migrating; ETA: 2h57m3s
2016-10-12 20:43:45 INFO Reconnecting... Will resume at mysql-bin-changelog.117854:38741318
2016-10-12 20:43:45 INFO Registering replica at sql01.xxx.us-east-1.rds.amazonaws.com:3306
2016-10-12 20:43:45 INFO Connecting binlog streamer at mysql-bin-changelog.117854:4
2016-10-12 20:43:45 INFO rotate to next log name: mysql-bin-changelog.117854
Copy: 207153700/232068606 89.3%; Applied: 1374017; Backlog: 0/100; Time: 24h40m5s(total), 24h31m23s(copy); streamer: mysql-bin-changelog.117854:1876416; State: migrating; ETA: 2h56m58s
2016-10-12 20:43:48 INFO StreamEvents encountered unexpected error: Err: runtime error: index out of range
 Stack: goroutine 457822 [running]:
github.com/github/gh-ost/vendor/github.com/siddontang/go-mysql/mysql.Pstack(0x0, 0x0)
        /Users/shlomi-noach/dev/go/src/github.com/github/gh-ost/vendor/github.com/siddontang/go-mysql/mysql/util.go:18 +0x7b
github.com/github/gh-ost/vendor/github.com/siddontang/go-mysql/replication.(*BinlogSyncer).onStream.func1(0xc8204ffcc0, 0xc82045a900)
        /Users/shlomi-noach/dev/go/src/github.com/github/gh-ost/vendor/github.com/siddontang/go-mysql/replication/binlogsyncer.go:488 +0x66
panic(0x797b60, 0xc82000a090)
        /usr/local/Cellar/go/1.6.2/libexec/src/runtime/panic.go:443 +0x4e9
github.com/github/gh-ost/vendor/github.com/siddontang/go-mysql/replication.decodeDecimal(0xc82040129a, 0x7d, 0x566, 0x0, 0xa, 0x13, 0x5, 0x0, 0x0)
        /Users/shlomi-noach/dev/go/src/github.com/github/gh-ost/vendor/github.com/siddontang/go-mysql/replication/row_event.go:513 +0xcc7
github.com/github/gh-ost/vendor/github.com/siddontang/go-mysql/replication.(*RowsEvent).decodeValue(0xc8203f7400, 0xc82040129a, 0x7d, 0x566, 0xa00f6,
Copy: 207168700/232068606 89.3%; Applied: 1374017; Backlog: 0/100; Time: 24h40m10s(total), 24h31m28s(copy); streamer: mysql-bin-changelog.117854:38738560; State: migrating; ETA: 2h56m51s
2016-10-12 20:43:53 INFO Reconnecting... Will resume at mysql-bin-changelog.117854:38741318
2016-10-12 20:43:53 INFO Registering replica at sql01.xxx.us-east-1.rds.amazonaws.com:3306
2016-10-12 20:43:53 INFO Connecting binlog streamer at mysql-bin-changelog.117854:4
2016-10-12 20:43:53 INFO rotate to next log name: mysql-bin-changelog.117854
Copy: 207180700/232068606 89.3%; Applied: 1374017; Backlog: 0/100; Time: 24h40m15s(total), 24h31m33s(copy); streamer: mysql-bin-changelog.117854:32530931; State: migrating; ETA: 2h56m46s
2016-10-12 20:43:55 INFO StreamEvents encountered unexpected error: Err: runtime error: index out of range
 Stack: goroutine 457874 [running]:
github.com/github/gh-ost/vendor/github.com/siddontang/go-mysql/mysql.Pstack(0x0, 0x0)
        /Users/shlomi-noach/dev/go/src/github.com/github/gh-ost/vendor/github.com/siddontang/go-mysql/mysql/util.go:18 +0x7b
github.com/github/gh-ost/vendor/github.com/siddontang/go-mysql/replication.(*BinlogSyncer).onStream.func1(0xc820221b80, 0xc8204ca9c0)
        /Users/shlomi-noach/dev/go/src/github.com/github/gh-ost/vendor/github.com/siddontang/go-mysql/replication/binlogsyncer.go:488 +0x66
panic(0x797b60, 0xc82000a090)
        /usr/local/Cellar/go/1.6.2/libexec/src/runtime/panic.go:443 +0x4e9
github.com/github/gh-ost/vendor/github.com/siddontang/go-mysql/replication.decodeDecimal(0xc820360c9a, 0x7d, 0x566, 0x0, 0xa, 0x13, 0x5, 0x0, 0x0)
        /Users/shlomi-noach/dev/go/src/github.com/github/gh-ost/vendor/github.com/siddontang/go-mysql/replication/row_event.go:513 +0xcc7
github.com/github/gh-ost/vendor/github.com/siddontang/go-mysql/replication.(*RowsEvent).decodeValue(0xc8203f7b80, 0xc820360c9a, 0x7d, 0x566, 0xa00f6,
Copy: 207198700/232068606 89.3%; Applied: 1374017; Backlog: 0/100; Time: 24h40m20s(total), 24h31m38s(copy); streamer: mysql-bin-changelog.117854:38739132; State: migrating; ETA: 2h56m38s
2016-10-12 20:44:00 INFO Reconnecting... Will resume at mysql-bin-changelog.117854:38741318
2016-10-12 20:44:00 INFO Registering replica at sql01.xxx.us-east-1.rds.amazonaws.com:3306
2016-10-12 20:44:00 INFO Connecting binlog streamer at mysql-bin-changelog.117854:4
2016-10-12 20:44:00 INFO rotate to next log name: mysql-bin-changelog.117854
2016-10-12 20:44:03 INFO StreamEvents encountered unexpected error: Err: runtime error: index out of range
 Stack: goroutine 457902 [running]:
github.com/github/gh-ost/vendor/github.com/siddontang/go-mysql/mysql.Pstack(0x0, 0x0)
        /Users/shlomi-noach/dev/go/src/github.com/github/gh-ost/vendor/github.com/siddontang/go-mysql/mysql/util.go:18 +0x7b
github.com/github/gh-ost/vendor/github.com/siddontang/go-mysql/replication.(*BinlogSyncer).onStream.func1(0xc8200fc5c0, 0xc8204cad80)
        /Users/shlomi-noach/dev/go/src/github.com/github/gh-ost/vendor/github.com/siddontang/go-mysql/replication/binlogsyncer.go:488 +0x66
panic(0x797b60, 0xc82000a090)
        /usr/local/Cellar/go/1.6.2/libexec/src/runtime/panic.go:443 +0x4e9
github.com/github/gh-ost/vendor/github.com/siddontang/go-mysql/replication.decodeDecimal(0xc8201a6c9a, 0x7d, 0x566, 0x0, 0xa, 0x13, 0x5, 0x0, 0x0)
        /Users/shlomi-noach/dev/go/src/github.com/github/gh-ost/vendor/github.com/siddontang/go-mysql/replication/row_event.go:513 +0xcc7
github.com/github/gh-ost/vendor/github.com/siddontang/go-mysql/replication.(*RowsEvent).decodeValue(0xc8203f65a0, 0xc8201a6c9a, 0x7d, 0x566, 0xa00f6,
Copy: 207210700/232068606 89.3%; Applied: 1374017; Backlog: 0/100; Time: 24h40m25s(total), 24h31m43s(copy); streamer: mysql-bin-changelog.117854:38741575; State: migrating; ETA: 2h56m33s
2016-10-12 20:44:08 FATAL {%!!(MISSING)d(string=mysql-bin-changelog.117854) 38741318 0} successive failures in streamer reconnect at coordinates %!!(MISSING)v(MISSING)

About this issue

  • Original URL
  • State: closed
  • Created 8 years ago
  • Reactions: 1
  • Comments: 31 (13 by maintainers)

Most upvoted comments

@siddontang take your time and enjoy life