vitess: Bug Report: Vitess 16 online DDL makes xtrabackup to fails

Overview of the Issue

Similar to https://github.com/vitessio/vitess/issues/12300 but this time with an online DDL.

I runned:

SET @@ddl_strategy='vitess';

ALTER TABLE my_table
DROP COLUMN my_column;

alter vitess_migration '...' cleanup;

ALTER TABLE my_other_table
DROP COLUMN my_column;

alter vitess_migration '...' cleanup;

The next backup then failed:

"@timestamp",message
"Feb 28, 2023 @ 18:00:17.715","E0228 17:00:17.715768       9 main.go:105] remote error: rpc error: code = Unknown desc = TabletManager.Backup on d2-0000100011 error: xtrabackup failed with error. Output=: exit status 1: xtrabackup failed with error. Output=: exit status 1"
"Feb 28, 2023 @ 18:00:17.715","E0228 17:00:17.715327       9 main.go:96] E0228 17:00:17.714869 backup.go:108] E0228 17:00:17.714869 backup.go:143] failed to finish backup: [MalformedXML: The XML you provided was not well-formed or did not validate against our published schema."
"Feb 28, 2023 @ 18:00:17.715","	status code: 400, request id: txe2538300eabf4542a4043-0063fe3321, host id: txe2538300eabf4542a4043-0063fe3321]: %!v(MISSING)"
"Feb 28, 2023 @ 18:00:17.715","BackupShard Error: rpc error: code = Unknown desc = TabletManager.Backup on d2-0000100011 error: xtrabackup failed with error. Output=: exit status 1: xtrabackup failed with error. Output=: exit status 1"
"Feb 28, 2023 @ 18:00:17.688","xtrabackup failed with error. Output=]"
"Feb 28, 2023 @ 18:00:17.688","I0228 17:00:17.688641       9 main.go:96] I0228 17:00:17.688379 backup.go:108] I0228 17:00:17.688449 xtrabackupengine.go:146] Closing backup file backup.xbstream.gz"
"Feb 28, 2023 @ 18:00:17.688","E0228 17:00:17.688757       9 main.go:96] E0228 17:00:17.688492 backup.go:108] E0228 17:00:17.688557 backup.go:135] backup is not usable, aborting it: [exit status 1"
"Feb 28, 2023 @ 18:00:17.682","I0228 17:00:17.682735       9 main.go:96] I0228 17:00:17.682162 backup.go:108] I0228 17:00:17.682051 xtrabackupengine.go:369] xtrabackup stderr: 2023-02-28T17:00:17.681965-00:00 0 [ERROR] [MY-011825] [Xtrabackup] Please run OPTIMIZE TABLE or ALTER TABLE ALGORITHM=COPY on all listed tables to fix this issue."
"Feb 28, 2023 @ 18:00:17.682","I0228 17:00:17.682700       9 main.go:96] I0228 17:00:17.682126 backup.go:108] I0228 17:00:17.682017 xtrabackupengine.go:369] xtrabackup stderr: 2023-02-28T17:00:17.681943-00:00 0 [ERROR] [MY-011825] [Xtrabackup] vt_database/my_table"
"Feb 28, 2023 @ 18:00:17.682","I0228 17:00:17.682649       9 main.go:96] I0228 17:00:17.682085 backup.go:108] I0228 17:00:17.681985 xtrabackupengine.go:369] xtrabackup stderr: 2023-02-28T17:00:17.681920-00:00 0 [ERROR] [MY-011825] [Xtrabackup] Please check https://docs.percona.com/percona-xtrabackup/8.0/em/instant.html for more details."
"Feb 28, 2023 @ 18:00:17.682","I0228 17:00:17.682633       9 main.go:96] I0228 17:00:17.682063 backup.go:108] I0228 17:00:17.681969 xtrabackupengine.go:369] xtrabackup stderr: 2023-02-28T17:00:17.681904-00:00 0 [ERROR] [MY-011825] [Xtrabackup] This feature is not stable and will cause backup corruption."
"Feb 28, 2023 @ 18:00:17.682","I0228 17:00:17.682589       9 main.go:96] I0228 17:00:17.682011 backup.go:108] I0228 17:00:17.681940 xtrabackupengine.go:369] xtrabackup stderr: 2023-02-28T17:00:17.681854-00:00 0 [ERROR] [MY-011825] [Xtrabackup] Found tables with row versions due to INSTANT ADD/DROP columns"
"Feb 28, 2023 @ 18:00:17.682","I0228 17:00:17.682687       9 main.go:96] I0228 17:00:17.682108 backup.go:108] I0228 17:00:17.682002 xtrabackupengine.go:369] xtrabackup stderr: 2023-02-28T17:00:17.681932-00:00 0 [ERROR] [MY-011825] [Xtrabackup] Tables found:"
"Feb 28, 2023 @ 18:00:17.682","I0228 17:00:17.682711       9 main.go:96] I0228 17:00:17.682144 backup.go:108] I0228 17:00:17.682035 xtrabackupengine.go:369] xtrabackup stderr: 2023-02-28T17:00:17.681954-00:00 0 [ERROR] [MY-011825] [Xtrabackup] vt_database/my_other_table"
"Feb 28, 2023 @ 18:00:17.650","I0228 17:00:17.650506       9 main.go:96] I0228 17:00:17.650203 backup.go:108] I0228 17:00:17.650269 xtrabackupengine.go:369] xtrabackup stderr: 2023-02-28T17:00:17.650208-00:00 0 [Note] [MY-011825] [Xtrabackup] Executing LOCK INSTANCE FOR BACKUP ..."
"Feb 28, 2023 @ 18:00:17.639","I0228 17:00:17.639846       9 main.go:96] I0228 17:00:17.639590 backup.go:108] I0228 17:00:17.639670 xtrabackupengine.go:369] xtrabackup stderr: 2023-02-28T17:00:17.639594-00:00 0 [Note] [MY-011825] [Xtrabackup] Using server version 8.0.30"
"Feb 28, 2023 @ 18:00:17.629","I0228 17:00:17.628943       9 main.go:96] I0228 17:00:17.628479 backup.go:108] I0228 17:00:17.628536 xtrabackupengine.go:369] xtrabackup stderr: 2023-02-28T17:00:17.628431-00:00 0 [Note] [MY-011825] [Xtrabackup] Connecting to MySQL server host: localhost, user: vt_dba, password: not set, port: not set, socket: /vt/socket/mysql.sock"
"Feb 28, 2023 @ 18:00:17.619","I0228 17:00:17.619185       9 main.go:96] I0228 17:00:17.618611 backup.go:108] I0228 17:00:17.618562 xtrabackupengine.go:369] xtrabackup stderr: 230228 17:00:17  version_check Done."
"Feb 28, 2023 @ 18:00:17.160","I0228 17:00:17.160681       9 main.go:96] I0228 17:00:17.160340 backup.go:108] I0228 17:00:17.160353 xtrabackupengine.go:369] xtrabackup stderr: 230228 17:00:17  version_check Connected to MySQL server"
"Feb 28, 2023 @ 18:00:17.160","I0228 17:00:17.160778       9 main.go:96] I0228 17:00:17.160552 backup.go:108] I0228 17:00:17.160616 xtrabackupengine.go:369] xtrabackup stderr: 230228 17:00:17  version_check Executing a version check against the server..."
"Feb 28, 2023 @ 18:00:17.159","I0228 17:00:17.159330       9 main.go:96] I0228 17:00:17.158932 backup.go:108] I0228 17:00:17.158849 xtrabackupengine.go:369] xtrabackup stderr: 230228 17:00:17  version_check Connecting to MySQL server with DSN 'dbi:mysql:;mysql_read_default_group=xtrabackup;mysql_socket=/vt/socket/mysql.sock' as 'vt_dba'  (using password: NO)."
"Feb 28, 2023 @ 18:00:16.979","I0228 17:00:16.979713       9 main.go:96] I0228 17:00:16.979468 backup.go:108] I0228 17:00:16.979475 xtrabackupengine.go:369] xtrabackup stderr: 2023-02-28T17:00:16.979364-00:00 0 [Note] [MY-011825] [Xtrabackup] recognized client arguments: --backup=1 --socket=/vt/socket/mysql.sock --slave-info=1 --user=vt_dba --target-dir=/vt/vtdataroot/vt_0000100011/tmp --stream=xbstream "
"Feb 28, 2023 @ 18:00:16.979","I0228 17:00:16.979537       9 main.go:96] I0228 17:00:16.979280 backup.go:108] I0228 17:00:16.979322 xtrabackupengine.go:369] xtrabackup stderr: 2023-02-28T17:00:16.979129-00:00 0 [Note] [MY-011825] [Xtrabackup] recognized server arguments: --datadir=/vt/vtdataroot/vt_0000100011/data --innodb_data_home_dir=/vt/vtdataroot/vt_0000100011/innodb/data --innodb_log_group_home_dir=/vt/vtdataroot/vt_0000100011/innodb/logs --log_bin=/vt/vtdataroot/vt_0000100011/bin-logs/vt-0000100011-bin --server-id=2141437576 --tmpdir=/vt/vtdataroot/vt_0000100011/tmp --innodb_buffer_pool_size=20G --innodb_log_file_size=16M "
"Feb 28, 2023 @ 18:00:16.979","I0228 17:00:16.979737       9 main.go:96] I0228 17:00:16.979520 backup.go:108] I0228 17:00:16.979518 xtrabackupengine.go:369] xtrabackup stderr: xtrabackup version 8.0.32-25 based on MySQL server 8.0.32 Linux (x86_64) (revision id: 14f007fb)"
"Feb 28, 2023 @ 18:00:16.955","I0228 17:00:16.955697       9 main.go:96] I0228 17:00:16.955477 backup.go:108] I0228 17:00:16.954551 compression.go:257] Compressing backup using engine ""pargzip"""
"Feb 28, 2023 @ 18:00:16.951","I0228 17:00:16.951492       9 main.go:96] I0228 17:00:16.950760 backup.go:108] I0228 17:00:16.950648 xtrabackupengine.go:194] backup file name: backup.xbstream.gz"
"Feb 28, 2023 @ 18:00:16.951","I0228 17:00:16.951507       9 main.go:96] I0228 17:00:16.950785 backup.go:108] I0228 17:00:16.950668 xtrabackupengine.go:202] Starting backup with 0 stripe(s)"
"Feb 28, 2023 @ 18:00:16.718","OKI0228 17:00:16.951338       9 main.go:96] I0228 17:00:16.950713 backup.go:108] I0228 17:00:16.950594 xtrabackupengine.go:191] Detected MySQL flavor: MySQL56"
"Feb 28, 2023 @ 18:00:16.681","                                 Dload  Upload   Total   Spent    Left  Speed"
"Feb 28, 2023 @ 18:00:16.681","
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100     2  100     2    0     0     54      0 --:--:-- --:--:-- --:--:--    54"
"Feb 28, 2023 @ 18:00:16.681","  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current"

Reproduction Steps

Run an online DDL droping a column. Run xtrabackup and see it fails.

Binary Version

Version: 16.0.0 (Git revision bb768df0008fc09f7e6868a4fa571c32cc1cb526 branch 'heads/v16.0.0') built on Tue Feb 28 15:38:00 UTC 2023 by vitess@buildkitsandbox using go1.20.1 linux/amd64

Operating System and Environment details

docker

Log Fragments

No response

About this issue

Original URL
State: closed
Created a year ago
Comments: 29 (20 by maintainers)

Most upvoted comments

Is it planned to update MySQL >= 8.0.32 in Vitess 17? https://github.com/vitessio/vitess/blob/main/docker/lite/install_dependencies.sh#L86

@L3o-pold, AFAIK, we plan on keeping the MySQL version we use up-to-date with recent MySQL versions, the same way we did in v16 by upgrading from 8.0.23 to 8.0.30.

frouioui on Mar 1, 2023

Yes, correct 😅

shlomi-noach on Feb 28, 2023

From my understanding it needs MySQL >= 8.0.32

In my understanding it not only needs that version, it needs tables to have been altered in that version. So a table altered in a 8.0.31 is never going to work, even if later the server is upgraded to 8.0.32.

shlomi-noach on Feb 28, 2023

@shlomi-noach pointed out that this has been fixed in the version of xtrabackup that was released yesterday https://docs.percona.com/percona-xtrabackup/8.0/em/instant.html

Percona XtraBackup 8.0.32-25 allows instant columns in the backup for MySQL 8.0.32. This limitation remains for backing up MySQL 8.0.31 and lower versions.

deepthi on Feb 28, 2023

@deepthi do you still want me to try with downgraded version of xtrabackup ?

L3o-pold on Feb 28, 2023

@deepthi the one provided by the vitess/lite:16.0.0 docker image.

xtrabackup --version
2023-02-28T18:45:49.407191-00:00 0 [Note] [MY-011825] [Xtrabackup] recognized server arguments: --datadir=/var/lib/mysql 
xtrabackup version 8.0.32-25 based on MySQL server 8.0.32 Linux (x86_64) (revision id: 14f007fb)

L3o-pold on Feb 28, 2023