milvus-backup: [Bug]: Create backup successfully but cannot restore
Current Behavior
Milvus cluster is depolyed on k8s, and I try to run backup tool for it. It ends successfully, but there’re error logs when I restore the backup, and I find out that the backup file paths have one more level. For example,
- the original path is :
by-dev/delta_log/451699909127432302/451699909127432303/451699909127432419/451699909126213016 - while the backup path is :
backup/mybackup/binlogs/delta_log/451699909127432302/451699909127432303/451699909127432419/451699909127432419/451699909126213016
Seems that there’s one more 451699909127432419 in the backup path, the correct file cannot be find. so the restore log shows that:
[2024/08/26 14:44:13.600 +08:00] [INFO] [logutil/logutil.go:165] ["Log directory"] [configDir=]
[2024/08/26 14:44:13.601 +08:00] [INFO] [logutil/logutil.go:166] ["Set log file to "] [path=logs/backup.log]
[2024/08/26 14:44:13.601 +08:00] [INFO] [cmd/restore.go:46] ["restore cmd input args"] [args="[]"]
[2024/08/26 14:44:13.602 +08:00] [INFO] [core/backup_impl_restore_backup.go:26] ["receive RestoreBackupRequest"] [requestId=9b62f8ca-6376-11ef-bb74-8ea5ac62d0b8] [backupName=mybackup] [onlyMeta=false] [restoreIndex=true] [useAutoIndex=false] [dropExistCollection=false] [dropExistIndex=false] [skipCreateCollection=false] [collections="[]"] [CollectionSuffix=_recover2] [CollectionRenames={}] [async=false] [bucketName=] [path=] [databaseCollections=]
[2024/08/26 14:44:13.602 +08:00] [INFO] [core/backup_context.go:106] ["{Base:0x14000288280 MaxSegmentGroupSize:2147483648 BackupCollectionParallelism:4 BackupCopyDataParallelism:128 RestoreParallelism:2 KeepTempFiles:false GcPauseEnable:true GcPauseSeconds:7200 GcPauseAddress:http://localhost:9091}"]
[2024/08/26 14:44:13.602 +08:00] [INFO] [core/backup_context.go:107] ["{Base:0x14000288280 Enabled:true DebugMode:false SimpleResponse:true}"]
[2024/08/26 14:44:13.602 +08:00] [INFO] [core/backup_context.go:209] ["receive GetBackupRequest"] [requestId=9b630004-6376-11ef-bb74-8ea5ac62d0b8] [backupName=mybackup] [backupId=] [bucketName=] [path=]
[2024/08/26 14:44:14.214 +08:00] [INFO] [storage/minio_chunk_manager.go:151] ["minio chunk manager init success."] [bucketname=xjmz-ai-llm-milvus] [root=by-dev]
[2024/08/26 14:44:14.879 +08:00] [INFO] [core/backup_context.go:301] ["finish GetBackupRequest"] [requestId=9b630004-6376-11ef-bb74-8ea5ac62d0b8] [backupName=mybackup] [backupId=] [bucketName=] [path=]
[2024/08/26 14:44:14.879 +08:00] [INFO] [core/backup_impl_restore_backup.go:178] ["Collections to restore"] [collection_num=4]
[2024/08/26 14:44:15.571 +08:00] [INFO] [core/backup_impl_restore_backup.go:331] ["Start collection level restore pool"] [parallelism=2]
[2024/08/26 14:44:15.572 +08:00] [INFO] [core/backup_impl_restore_backup.go:337] ["executeRestoreBackupTask start"] [backup_name=mybackup] [backupBucketName=xjmz-ai-llm-milvus] [backupPath=backup/mybackup]
[2024/08/26 14:44:15.572 +08:00] [INFO] [core/backup_impl_restore_backup.go:389] ["start restore"] [db_name=test] [collection_name=testCollection3_recover2] [backupBucketName=xjmz-ai-llm-milvus] [backupPath=backup/mybackup]
[2024/08/26 14:44:15.572 +08:00] [INFO] [core/backup_impl_restore_backup.go:416] ["collection schema"] [fields="[{\"ID\":100,\"Name\":\"id\",\"PrimaryKey\":true,\"AutoID\":true,\"Description\":\"\",\"DataType\":5,\"TypeParams\":{},\"IndexParams\":{},\"IsDynamic\":false,\"IsPartitionKey\":false,\"ElementType\":0},{\"ID\":101,\"Name\":\"vector\",\"PrimaryKey\":false,\"AutoID\":false,\"Description\":\"\",\"DataType\":101,\"TypeParams\":{\"dim\":\"128\"},\"IndexParams\":{},\"IsDynamic\":false,\"IsPartitionKey\":false,\"ElementType\":0}]"]
[2024/08/26 14:44:15.573 +08:00] [INFO] [core/backup_impl_restore_backup.go:389] ["start restore"] [db_name=test] [collection_name=testCollection_recover2] [backupBucketName=xjmz-ai-llm-milvus] [backupPath=backup/mybackup]
[2024/08/26 14:44:15.573 +08:00] [INFO] [core/backup_impl_restore_backup.go:416] ["collection schema"] [fields="[{\"ID\":100,\"Name\":\"id\",\"PrimaryKey\":true,\"AutoID\":true,\"Description\":\"\",\"DataType\":5,\"TypeParams\":{},\"IndexParams\":{},\"IsDynamic\":false,\"IsPartitionKey\":false,\"ElementType\":0},{\"ID\":101,\"Name\":\"vector\",\"PrimaryKey\":false,\"AutoID\":false,\"Description\":\"\",\"DataType\":101,\"TypeParams\":{\"dim\":\"128\"},\"IndexParams\":{},\"IsDynamic\":false,\"IsPartitionKey\":false,\"ElementType\":0}]"]
[2024/08/26 14:44:16.358 +08:00] [INFO] [core/backup_impl_restore_backup.go:462] ["create collection"] [database=test] [collectionName=testCollection3_recover2] [hasPartitionKey=false]
[2024/08/26 14:44:16.359 +08:00] [INFO] [core/backup_impl_restore_backup.go:516] ["source index"] [indexName=vector] [indexType=AUTOINDEX] [params="{\"index_type\":\"AUTOINDEX\",\"metric_type\":\"L2\"}"]
[2024/08/26 14:44:16.359 +08:00] [INFO] [core/backup_impl_restore_backup.go:528] ["not auto index"]
[2024/08/26 14:44:18.009 +08:00] [INFO] [core/backup_impl_restore_backup.go:462] ["create collection"] [database=test] [collectionName=testCollection_recover2] [hasPartitionKey=false]
[2024/08/26 14:44:18.009 +08:00] [INFO] [core/backup_impl_restore_backup.go:516] ["source index"] [indexName=index] [indexType=HNSW] [params="{\"index_type\":\"HNSW\",\"metric_type\":\"L2\",\"params\":\"{\\\"M\\\":\\\"2\\\",\\\"efConstruction\\\":\\\"1\\\"}\"}"]
[2024/08/26 14:44:18.014 +08:00] [INFO] [core/backup_impl_restore_backup.go:528] ["not auto index"]
[2024/08/26 14:44:18.225 +08:00] [INFO] [core/backup_impl_restore_backup.go:564] ["start restore partition"] [backupCollectionName=testCollection3] [targetDBName=test] [targetCollectionName=testCollection3_recover2] [partition=_default]
[2024/08/26 14:44:18.462 +08:00] [INFO] [core/backup_impl_restore_backup.go:564] ["start restore partition"] [backupCollectionName=testCollection] [targetDBName=test] [targetCollectionName=testCollection_recover2] [partition=_default]
[2024/08/26 14:44:18.555 +08:00] [INFO] [core/backup_impl_restore_backup.go:610] ["create partition"] [collectionName=testCollection3_recover2] [partitionName=_default]
[2024/08/26 14:44:18.555 +08:00] [INFO] [core/backup_impl_restore_backup.go:822] [getBackupPartitionPaths] [bucketName=xjmz-ai-llm-milvus] [backupPath=backup/mybackup] [partitionID=451699909127432303] [groupId=451699909127432419]
[2024/08/26 14:44:18.611 +08:00] [INFO] [core/backup_impl_restore_backup.go:723] ["execute bulk insert"] [db=test] [collection=testCollection3_recover2] [partition=_default] [files="[backup/mybackup/binlogs/insert_log/451699909127432302/451699909127432303/451699909127432419/,backup/mybackup/binlogs/delta_log/451699909127432302/451699909127432303/451699909127432419/]"] [endTime=452107837374464]
[2024/08/26 14:44:18.651 +08:00] [INFO] [core/backup_impl_restore_backup.go:610] ["create partition"] [collectionName=testCollection_recover2] [partitionName=_default]
[2024/08/26 14:44:18.651 +08:00] [INFO] [core/backup_impl_restore_backup.go:822] [getBackupPartitionPaths] [bucketName=xjmz-ai-llm-milvus] [backupPath=backup/mybackup] [partitionID=451699909126010750] [groupId=451699909126211091]
[2024/08/26 14:44:18.709 +08:00] [INFO] [core/backup_impl_restore_backup.go:723] ["execute bulk insert"] [db=test] [collection=testCollection_recover2] [partition=_default] [files="[backup/mybackup/binlogs/insert_log/451699909126010749/451699909126010750/451699909126211091/,]"] [endTime=452107837374464]
[2024/08/26 14:44:18.778 +08:00] [ERROR] [core/backup_impl_restore_backup.go:737] ["fail to bulk insert"] [db=test] [collectionName=testCollection3_recover2] [partitionName=_default] [files="[backup/mybackup/binlogs/insert_log/451699909127432302/451699909127432303/451699909127432419/,backup/mybackup/binlogs/delta_log/451699909127432302/451699909127432303/451699909127432419/]"] [error="no binlog to import, input=[paths:\"backup/mybackup/binlogs/insert_log/451699909127432302/451699909127432303/451699909127432419/\" paths:\"backup/mybackup/binlogs/delta_log/451699909127432302/451699909127432303/451699909127432419/\" ]: invalid parameter"] [errorVerbose="no binlog to import, input=[paths:\"backup/mybackup/binlogs/insert_log/451699909127432302/451699909127432303/451699909127432419/\" paths:\"backup/mybackup/binlogs/delta_log/451699909127432302/451699909127432303/451699909127432419/\" ]: invalid parameter\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus-sdk-go/v2/client.handleRespStatus\n | \t/home/runner/go/pkg/mod/github.com/wayblink/milvus-sdk-go/v2@v2.3.6-beta3/client/collection.go:39\n | github.com/milvus-io/milvus-sdk-go/v2/client.(*GrpcClient).BulkInsert\n | \t/home/runner/go/pkg/mod/github.com/wayblink/milvus-sdk-go/v2@v2.3.6-beta3/client/insert.go:370\n | github.com/zilliztech/milvus-backup/core.(*MilvusClient).BulkInsert\n | \t/home/runner/work/milvus-backup/milvus-backup/core/milvus_sdk_wrapper.go:121\n | github.com/zilliztech/milvus-backup/core.(*BackupContext).executeBulkInsert\n | \t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:734\n | github.com/zilliztech/milvus-backup/core.(*BackupContext).restorePartition.func2\n | \t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:640\n | github.com/zilliztech/milvus-backup/core.(*BackupContext).restorePartition\n | \t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:689\n | github.com/zilliztech/milvus-backup/core.(*BackupContext).executeRestoreCollectionTask.func3\n | \t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:569\n | github.com/zilliztech/milvus-backup/internal/common.(*WorkerPool).work.func1\n | \t/home/runner/work/milvus-backup/milvus-backup/internal/common/workerpool.go:70\n | golang.org/x/sync/errgroup.(*Group).Go.func1\n | \t/home/runner/go/pkg/mod/golang.org/x/sync@v0.3.0/errgroup/errgroup.go:75\n | runtime.goexit\n | \t/opt/hostedtoolcache/go/1.18.10/x64/src/runtime/asm_arm64.s:1270\nWraps: (2) no binlog to import, input=[paths:\"backup/mybackup/binlogs/insert_log/451699909127432302/451699909127432303/451699909127432419/\" paths:\"backup/mybackup/binlogs/delta_log/451699909127432302/451699909127432303/451699909127432419/\" ]: invalid parameter\nError types: (1) *withstack.withStack (2) *errutil.leafError"] [stack="github.com/zilliztech/milvus-backup/core.(*BackupContext).executeBulkInsert\n\t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:737\ngithub.com/zilliztech/milvus-backup/core.(*BackupContext).restorePartition.func2\n\t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:640\ngithub.com/zilliztech/milvus-backup/core.(*BackupContext).restorePartition\n\t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:689\ngithub.com/zilliztech/milvus-backup/core.(*BackupContext).executeRestoreCollectionTask.func3\n\t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:569\ngithub.com/zilliztech/milvus-backup/internal/common.(*WorkerPool).work.func1\n\t/home/runner/work/milvus-backup/milvus-backup/internal/common/workerpool.go:70\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/sync@v0.3.0/errgroup/errgroup.go:75"]
[2024/08/26 14:44:18.779 +08:00] [ERROR] [core/backup_impl_restore_backup.go:642] ["fail to bulk insert to partition"] [backupCollectionName=testCollection3] [targetDBName=test] [targetCollectionName=testCollection3_recover2] [partition=_default] [error="no binlog to import, input=[paths:\"backup/mybackup/binlogs/insert_log/451699909127432302/451699909127432303/451699909127432419/\" paths:\"backup/mybackup/binlogs/delta_log/451699909127432302/451699909127432303/451699909127432419/\" ]: invalid parameter"] [errorVerbose="no binlog to import, input=[paths:\"backup/mybackup/binlogs/insert_log/451699909127432302/451699909127432303/451699909127432419/\" paths:\"backup/mybackup/binlogs/delta_log/451699909127432302/451699909127432303/451699909127432419/\" ]: invalid parameter\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus-sdk-go/v2/client.handleRespStatus\n | \t/home/runner/go/pkg/mod/github.com/wayblink/milvus-sdk-go/v2@v2.3.6-beta3/client/collection.go:39\n | github.com/milvus-io/milvus-sdk-go/v2/client.(*GrpcClient).BulkInsert\n | \t/home/runner/go/pkg/mod/github.com/wayblink/milvus-sdk-go/v2@v2.3.6-beta3/client/insert.go:370\n | github.com/zilliztech/milvus-backup/core.(*MilvusClient).BulkInsert\n | \t/home/runner/work/milvus-backup/milvus-backup/core/milvus_sdk_wrapper.go:121\n | github.com/zilliztech/milvus-backup/core.(*BackupContext).executeBulkInsert\n | \t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:734\n | github.com/zilliztech/milvus-backup/core.(*BackupContext).restorePartition.func2\n | \t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:640\n | github.com/zilliztech/milvus-backup/core.(*BackupContext).restorePartition\n | \t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:689\n | github.com/zilliztech/milvus-backup/core.(*BackupContext).executeRestoreCollectionTask.func3\n | \t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:569\n | github.com/zilliztech/milvus-backup/internal/common.(*WorkerPool).work.func1\n | \t/home/runner/work/milvus-backup/milvus-backup/internal/common/workerpool.go:70\n | golang.org/x/sync/errgroup.(*Group).Go.func1\n | \t/home/runner/go/pkg/mod/golang.org/x/sync@v0.3.0/errgroup/errgroup.go:75\n | runtime.goexit\n | \t/opt/hostedtoolcache/go/1.18.10/x64/src/runtime/asm_arm64.s:1270\nWraps: (2) no binlog to import, input=[paths:\"backup/mybackup/binlogs/insert_log/451699909127432302/451699909127432303/451699909127432419/\" paths:\"backup/mybackup/binlogs/delta_log/451699909127432302/451699909127432303/451699909127432419/\" ]: invalid parameter\nError types: (1) *withstack.withStack (2) *errutil.leafError"] [stack="github.com/zilliztech/milvus-backup/core.(*BackupContext).restorePartition.func2\n\t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:642\ngithub.com/zilliztech/milvus-backup/core.(*BackupContext).restorePartition\n\t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:689\ngithub.com/zilliztech/milvus-backup/core.(*BackupContext).executeRestoreCollectionTask.func3\n\t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:569\ngithub.com/zilliztech/milvus-backup/internal/common.(*WorkerPool).work.func1\n\t/home/runner/work/milvus-backup/milvus-backup/internal/common/workerpool.go:70\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/sync@v0.3.0/errgroup/errgroup.go:75"]
[2024/08/26 14:44:18.779 +08:00] [ERROR] [core/backup_impl_restore_backup.go:691] ["fail to (copy and) bulkinsert data"] [error="no binlog to import, input=[paths:\"backup/mybackup/binlogs/insert_log/451699909127432302/451699909127432303/451699909127432419/\" paths:\"backup/mybackup/binlogs/delta_log/451699909127432302/451699909127432303/451699909127432419/\" ]: invalid parameter"] [errorVerbose="no binlog to import, input=[paths:\"backup/mybackup/binlogs/insert_log/451699909127432302/451699909127432303/451699909127432419/\" paths:\"backup/mybackup/binlogs/delta_log/451699909127432302/451699909127432303/451699909127432419/\" ]: invalid parameter\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus-sdk-go/v2/client.handleRespStatus\n | \t/home/runner/go/pkg/mod/github.com/wayblink/milvus-sdk-go/v2@v2.3.6-beta3/client/collection.go:39\n | github.com/milvus-io/milvus-sdk-go/v2/client.(*GrpcClient).BulkInsert\n | \t/home/runner/go/pkg/mod/github.com/wayblink/milvus-sdk-go/v2@v2.3.6-beta3/client/insert.go:370\n | github.com/zilliztech/milvus-backup/core.(*MilvusClient).BulkInsert\n | \t/home/runner/work/milvus-backup/milvus-backup/core/milvus_sdk_wrapper.go:121\n | github.com/zilliztech/milvus-backup/core.(*BackupContext).executeBulkInsert\n | \t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:734\n | github.com/zilliztech/milvus-backup/core.(*BackupContext).restorePartition.func2\n | \t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:640\n | github.com/zilliztech/milvus-backup/core.(*BackupContext).restorePartition\n | \t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:689\n | github.com/zilliztech/milvus-backup/core.(*BackupContext).executeRestoreCollectionTask.func3\n | \t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:569\n | github.com/zilliztech/milvus-backup/internal/common.(*WorkerPool).work.func1\n | \t/home/runner/work/milvus-backup/milvus-backup/internal/common/workerpool.go:70\n | golang.org/x/sync/errgroup.(*Group).Go.func1\n | \t/home/runner/go/pkg/mod/golang.org/x/sync@v0.3.0/errgroup/errgroup.go:75\n | runtime.goexit\n | \t/opt/hostedtoolcache/go/1.18.10/x64/src/runtime/asm_arm64.s:1270\nWraps: (2) no binlog to import, input=[paths:\"backup/mybackup/binlogs/insert_log/451699909127432302/451699909127432303/451699909127432419/\" paths:\"backup/mybackup/binlogs/delta_log/451699909127432302/451699909127432303/451699909127432419/\" ]: invalid parameter\nError types: (1) *withstack.withStack (2) *errutil.leafError"] [backupCollectionName=testCollection3] [targetCollectionName=testCollection3_recover2] [partition=_default] [stack="github.com/zilliztech/milvus-backup/core.(*BackupContext).restorePartition\n\t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:691\ngithub.com/zilliztech/milvus-backup/core.(*BackupContext).executeRestoreCollectionTask.func3\n\t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:569\ngithub.com/zilliztech/milvus-backup/internal/common.(*WorkerPool).work.func1\n\t/home/runner/work/milvus-backup/milvus-backup/internal/common/workerpool.go:70\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/sync@v0.3.0/errgroup/errgroup.go:75"]
[2024/08/26 14:44:18.781 +08:00] [ERROR] [core/backup_impl_restore_backup.go:571] ["fail to restore partition"] [backupCollectionName=testCollection3] [targetDBName=test] [targetCollectionName=testCollection3_recover2] [partition=_default] [error="no binlog to import, input=[paths:\"backup/mybackup/binlogs/insert_log/451699909127432302/451699909127432303/451699909127432419/\" paths:\"backup/mybackup/binlogs/delta_log/451699909127432302/451699909127432303/451699909127432419/\" ]: invalid parameter"] [errorVerbose="no binlog to import, input=[paths:\"backup/mybackup/binlogs/insert_log/451699909127432302/451699909127432303/451699909127432419/\" paths:\"backup/mybackup/binlogs/delta_log/451699909127432302/451699909127432303/451699909127432419/\" ]: invalid parameter\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus-sdk-go/v2/client.handleRespStatus\n | \t/home/runner/go/pkg/mod/github.com/wayblink/milvus-sdk-go/v2@v2.3.6-beta3/client/collection.go:39\n | github.com/milvus-io/milvus-sdk-go/v2/client.(*GrpcClient).BulkInsert\n | \t/home/runner/go/pkg/mod/github.com/wayblink/milvus-sdk-go/v2@v2.3.6-beta3/client/insert.go:370\n | github.com/zilliztech/milvus-backup/core.(*MilvusClient).BulkInsert\n | \t/home/runner/work/milvus-backup/milvus-backup/core/milvus_sdk_wrapper.go:121\n | github.com/zilliztech/milvus-backup/core.(*BackupContext).executeBulkInsert\n | \t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:734\n | github.com/zilliztech/milvus-backup/core.(*BackupContext).restorePartition.func2\n | \t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:640\n | github.com/zilliztech/milvus-backup/core.(*BackupContext).restorePartition\n | \t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:689\n | github.com/zilliztech/milvus-backup/core.(*BackupContext).executeRestoreCollectionTask.func3\n | \t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:569\n | github.com/zilliztech/milvus-backup/internal/common.(*WorkerPool).work.func1\n | \t/home/runner/work/milvus-backup/milvus-backup/internal/common/workerpool.go:70\n | golang.org/x/sync/errgroup.(*Group).Go.func1\n | \t/home/runner/go/pkg/mod/golang.org/x/sync@v0.3.0/errgroup/errgroup.go:75\n | runtime.goexit\n | \t/opt/hostedtoolcache/go/1.18.10/x64/src/runtime/asm_arm64.s:1270\nWraps: (2) no binlog to import, input=[paths:\"backup/mybackup/binlogs/insert_log/451699909127432302/451699909127432303/451699909127432419/\" paths:\"backup/mybackup/binlogs/delta_log/451699909127432302/451699909127432303/451699909127432419/\" ]: invalid parameter\nError types: (1) *withstack.withStack (2) *errutil.leafError"] [stack="github.com/zilliztech/milvus-backup/core.(*BackupContext).executeRestoreCollectionTask.func3\n\t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:571\ngithub.com/zilliztech/milvus-backup/internal/common.(*WorkerPool).work.func1\n\t/home/runner/work/milvus-backup/milvus-backup/internal/common/workerpool.go:70\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/sync@v0.3.0/errgroup/errgroup.go:75"]
[2024/08/26 14:44:18.781 +08:00] [ERROR] [core/backup_impl_restore_backup.go:354] ["executeRestoreCollectionTask failed"] [TargetDBName=test] [TargetCollectionName=testCollection3_recover2] [error="no binlog to import, input=[paths:\"backup/mybackup/binlogs/insert_log/451699909127432302/451699909127432303/451699909127432419/\" paths:\"backup/mybackup/binlogs/delta_log/451699909127432302/451699909127432303/451699909127432419/\" ]: invalid parameter"] [errorVerbose="no binlog to import, input=[paths:\"backup/mybackup/binlogs/insert_log/451699909127432302/451699909127432303/451699909127432419/\" paths:\"backup/mybackup/binlogs/delta_log/451699909127432302/451699909127432303/451699909127432419/\" ]: invalid parameter\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus-sdk-go/v2/client.handleRespStatus\n | \t/home/runner/go/pkg/mod/github.com/wayblink/milvus-sdk-go/v2@v2.3.6-beta3/client/collection.go:39\n | github.com/milvus-io/milvus-sdk-go/v2/client.(*GrpcClient).BulkInsert\n | \t/home/runner/go/pkg/mod/github.com/wayblink/milvus-sdk-go/v2@v2.3.6-beta3/client/insert.go:370\n | github.com/zilliztech/milvus-backup/core.(*MilvusClient).BulkInsert\n | \t/home/runner/work/milvus-backup/milvus-backup/core/milvus_sdk_wrapper.go:121\n | github.com/zilliztech/milvus-backup/core.(*BackupContext).executeBulkInsert\n | \t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:734\n | github.com/zilliztech/milvus-backup/core.(*BackupContext).restorePartition.func2\n | \t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:640\n | github.com/zilliztech/milvus-backup/core.(*BackupContext).restorePartition\n | \t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:689\n | github.com/zilliztech/milvus-backup/core.(*BackupContext).executeRestoreCollectionTask.func3\n | \t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:569\n | github.com/zilliztech/milvus-backup/internal/common.(*WorkerPool).work.func1\n | \t/home/runner/work/milvus-backup/milvus-backup/internal/common/workerpool.go:70\n | golang.org/x/sync/errgroup.(*Group).Go.func1\n | \t/home/runner/go/pkg/mod/golang.org/x/sync@v0.3.0/errgroup/errgroup.go:75\n | runtime.goexit\n | \t/opt/hostedtoolcache/go/1.18.10/x64/src/runtime/asm_arm64.s:1270\nWraps: (2) no binlog to import, input=[paths:\"backup/mybackup/binlogs/insert_log/451699909127432302/451699909127432303/451699909127432419/\" paths:\"backup/mybackup/binlogs/delta_log/451699909127432302/451699909127432303/451699909127432419/\" ]: invalid parameter\nError types: (1) *withstack.withStack (2) *errutil.leafError"] [stack="github.com/zilliztech/milvus-backup/core.(*BackupContext).executeRestoreBackupTask.func2\n\t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:354\ngithub.com/zilliztech/milvus-backup/internal/common.(*WorkerPool).work.func1\n\t/home/runner/work/milvus-backup/milvus-backup/internal/common/workerpool.go:70\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/sync@v0.3.0/errgroup/errgroup.go:75"]
[2024/08/26 14:44:18.781 +08:00] [ERROR] [core/backup_impl_restore_backup.go:737] ["fail to bulk insert"] [db=test] [collectionName=testCollection_recover2] [partitionName=_default] [files="[backup/mybackup/binlogs/insert_log/451699909126010749/451699909126010750/451699909126211091/,]"] [error="rpc error: code = Canceled desc = context canceled"] [stack="github.com/zilliztech/milvus-backup/core.(*BackupContext).executeBulkInsert\n\t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:737\ngithub.com/zilliztech/milvus-backup/core.(*BackupContext).restorePartition.func2\n\t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:640\ngithub.com/zilliztech/milvus-backup/core.(*BackupContext).restorePartition\n\t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:689\ngithub.com/zilliztech/milvus-backup/core.(*BackupContext).executeRestoreCollectionTask.func3\n\t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:569\ngithub.com/zilliztech/milvus-backup/internal/common.(*WorkerPool).work.func1\n\t/home/runner/work/milvus-backup/milvus-backup/internal/common/workerpool.go:70\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/sync@v0.3.0/errgroup/errgroup.go:75"]
[2024/08/26 14:44:18.781 +08:00] [ERROR] [core/backup_impl_restore_backup.go:642] ["fail to bulk insert to partition"] [backupCollectionName=testCollection] [targetDBName=test] [targetCollectionName=testCollection_recover2] [partition=_default] [error="rpc error: code = Canceled desc = context canceled"] [stack="github.com/zilliztech/milvus-backup/core.(*BackupContext).restorePartition.func2\n\t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:642\ngithub.com/zilliztech/milvus-backup/core.(*BackupContext).restorePartition\n\t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:689\ngithub.com/zilliztech/milvus-backup/core.(*BackupContext).executeRestoreCollectionTask.func3\n\t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:569\ngithub.com/zilliztech/milvus-backup/internal/common.(*WorkerPool).work.func1\n\t/home/runner/work/milvus-backup/milvus-backup/internal/common/workerpool.go:70\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/sync@v0.3.0/errgroup/errgroup.go:75"]
[2024/08/26 14:44:18.781 +08:00] [ERROR] [core/backup_impl_restore_backup.go:691] ["fail to (copy and) bulkinsert data"] [error="rpc error: code = Canceled desc = context canceled"] [backupCollectionName=testCollection] [targetCollectionName=testCollection_recover2] [partition=_default] [stack="github.com/zilliztech/milvus-backup/core.(*BackupContext).restorePartition\n\t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:691\ngithub.com/zilliztech/milvus-backup/core.(*BackupContext).executeRestoreCollectionTask.func3\n\t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:569\ngithub.com/zilliztech/milvus-backup/internal/common.(*WorkerPool).work.func1\n\t/home/runner/work/milvus-backup/milvus-backup/internal/common/workerpool.go:70\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/sync@v0.3.0/errgroup/errgroup.go:75"]
[2024/08/26 14:44:18.781 +08:00] [ERROR] [core/backup_impl_restore_backup.go:571] ["fail to restore partition"] [backupCollectionName=testCollection] [targetDBName=test] [targetCollectionName=testCollection_recover2] [partition=_default] [error="rpc error: code = Canceled desc = context canceled"] [stack="github.com/zilliztech/milvus-backup/core.(*BackupContext).executeRestoreCollectionTask.func3\n\t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:571\ngithub.com/zilliztech/milvus-backup/internal/common.(*WorkerPool).work.func1\n\t/home/runner/work/milvus-backup/milvus-backup/internal/common/workerpool.go:70\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/sync@v0.3.0/errgroup/errgroup.go:75"]
[2024/08/26 14:44:18.781 +08:00] [ERROR] [core/backup_impl_restore_backup.go:354] ["executeRestoreCollectionTask failed"] [TargetDBName=test] [TargetCollectionName=testCollection_recover2] [error="rpc error: code = Canceled desc = context canceled"] [stack="github.com/zilliztech/milvus-backup/core.(*BackupContext).executeRestoreBackupTask.func2\n\t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:354\ngithub.com/zilliztech/milvus-backup/internal/common.(*WorkerPool).work.func1\n\t/home/runner/work/milvus-backup/milvus-backup/internal/common/workerpool.go:70\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/sync@v0.3.0/errgroup/errgroup.go:75"]
[2024/08/26 14:44:18.782 +08:00] [ERROR] [core/backup_impl_restore_backup.go:312] ["execute restore collection fail"] [backupId=4fce65b6-6376-11ef-862f-8ea5ac62d0b8] [error="workerpool: execute job no binlog to import, input=[paths:\"backup/mybackup/binlogs/insert_log/451699909127432302/451699909127432303/451699909127432419/\" paths:\"backup/mybackup/binlogs/delta_log/451699909127432302/451699909127432303/451699909127432419/\" ]: invalid parameter"] [stack="github.com/zilliztech/milvus-backup/core.(*BackupContext).RestoreBackup\n\t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:312\ngithub.com/zilliztech/milvus-backup/cmd.glob..func6\n\t/home/runner/work/milvus-backup/milvus-backup/cmd/restore.go:83\ngithub.com/spf13/cobra.(*Command).execute\n\t/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.5.0/command.go:876\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.5.0/command.go:990\ngithub.com/spf13/cobra.(*Command).Execute\n\t/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.5.0/command.go:918\ngithub.com/zilliztech/milvus-backup/cmd.Execute\n\t/home/runner/work/milvus-backup/milvus-backup/cmd/root.go:28\nmain.main\n\t/home/runner/work/milvus-backup/milvus-backup/main.go:24\nruntime.main\n\t/opt/hostedtoolcache/go/1.18.10/x64/src/runtime/proc.go:250"]
workerpool: execute job no binlog to import, input=[paths:"backup/mybackup/binlogs/insert_log/451699909127432302/451699909127432303/451699909127432419/" paths:"backup/mybackup/binlogs/delta_log/451699909127432302/451699909127432303/451699909127432419/" ]: invalid parameter
duration:5 s
Run backup check, the status is :
Succeed to connect to milvus and storage.
Milvus version: v2.4.5
Storage:
milvus-bucket: ##
milvus-rootpath: by-dev
backup-bucket: ##
backup-rootpath: backup
Expected Behavior
It should be restored and the path is correct.
Steps To Reproduce
1. deploy milvus on k8s.
2. run backup with config.
3. show the logs.
Environment
Milvus version:2.4.5 Milvus backup version:0.4.21
backup.yaml:
# Configures the system log output.
log:
level: info
console: true
file:
rootPath: "logs/backup.log"
http:
simpleResponse: true
milvus:
address: ##
port: 30278
authorizationEnabled: true
tlsMode: 0
user: "root"
password: "##"
minio:
storageType: "minio" # support storage type: local, minio, s3, aws, gcp, ali(aliyun), azure, tc(tencent)
address: ##
port: 443
accessKeyID: ##
secretAccessKey: ##
useSSL: true
useIAM: false
iamEndpoint: ""
bucketName: "##"
rootPath: "by-dev"
# Backup storage configs, the storage you want to put the backup data
backupStorageType: "minio" # support storage type: local, minio, s3, aws, gcp, ali(aliyun), azure, tc(tencent)
backupAddress: same as minio
backupPort: 443
backupAccessKeyID: same as minio
backupSecretAccessKey: same as minio
backupBucketName: same as minio
backupRootPath: "backup"
crossStorage: "false"
backup:
maxSegmentGroupSize: 2G
parallelism:
backupCollection: 4
copydata: 128
restoreCollection: 2
keepTempFiles: false
gcPause:
enable: true
seconds: 7200
address: milvus-address
Anything else?
Why the backup file path changed? Full log of backup creating: (I masked the bucket name to ‘bucket-name’) ./milvus-backup create -n mybackup
0.4.21 (Built on 2024-08-22T04:00:46Z from Git SHA 760ea2c7e05992e1f0f01c9b7a98aee2796b90cb)
config:backup.yaml
[2024/08/26 15:41:19.802 +08:00] [INFO] [logutil/logutil.go:165] ["Log directory"] [configDir=]
[2024/08/26 15:41:19.802 +08:00] [INFO] [logutil/logutil.go:166] ["Set log file to "] [path=logs/backup.log]
[2024/08/26 15:41:19.803 +08:00] [INFO] [core/backup_impl_create_backup.go:31] ["receive CreateBackupRequest"] [requestId=958fc2d6-637e-11ef-9b59-8ea5ac62d0b8] [backupName=mybackup] [collections="[]"] [databaseCollections=] [async=false] [force=false] [metaOnly=false]
[2024/08/26 15:41:19.803 +08:00] [INFO] [core/backup_context.go:117] ["{Base:0x1400047d340 MaxSegmentGroupSize:2147483648 BackupCollectionParallelism:4 BackupCopyDataParallelism:128 RestoreParallelism:2 KeepTempFiles:false GcPauseEnable:true GcPauseSeconds:7200 GcPauseAddress:http://##.com}"]
[2024/08/26 15:41:19.803 +08:00] [INFO] [core/backup_context.go:118] ["{Base:0x1400047d340 Enabled:true DebugMode:false SimpleResponse:true}"]
[2024/08/26 15:41:20.502 +08:00] [INFO] [storage/minio_chunk_manager.go:144] ["minio chunk manager init success."] [bucketname=bucket-name] [root=backup]
[2024/08/26 15:41:20.878 +08:00] [INFO] [core/backup_impl_create_backup.go:628] ["Pause Milvus GC response"] [response="{\"code\":1800,\"message\":\"user hasn't authenticated\"}"] [address=http://##.com] [pauseSeconds=7200]
[2024/08/26 15:41:21.224 +08:00] [INFO] [core/backup_impl_create_backup.go:684] ["collections to backup"] [collections="[test.testCollection,test.testCollection3]"]
[2024/08/26 15:41:21.225 +08:00] [INFO] [core/backup_impl_create_backup.go:230] ["start backup collection"] [db=test] [collection=testCollection]
[2024/08/26 15:41:21.226 +08:00] [INFO] [core/backup_impl_create_backup.go:230] ["start backup collection"] [db=test] [collection=testCollection3]
[2024/08/26 15:41:21.326 +08:00] [INFO] [core/backup_impl_create_backup.go:263] ["try to get index"] [collection_name=testCollection]
[2024/08/26 15:41:21.429 +08:00] [INFO] [core/backup_impl_create_backup.go:263] ["try to get index"] [collection_name=testCollection3]
[2024/08/26 15:41:21.625 +08:00] [INFO] [core/backup_impl_create_backup.go:283] ["field index"] [collection_name=testCollection] [field_name=id] ["index info"="[]"]
[2024/08/26 15:41:21.817 +08:00] [INFO] [core/backup_impl_create_backup.go:283] ["field index"] [collection_name=testCollection3] [field_name=id] ["index info"="[]"]
[2024/08/26 15:41:22.008 +08:00] [INFO] [core/backup_impl_create_backup.go:283] ["field index"] [collection_name=testCollection3] [field_name=vector] ["index info"="[{}]"]
[2024/08/26 15:41:22.197 +08:00] [INFO] [core/backup_impl_create_backup.go:283] ["field index"] [collection_name=testCollection] [field_name=vector] ["index info"="[{}]"]
[2024/08/26 15:41:22.774 +08:00] [INFO] [core/backup_impl_create_backup.go:369] ["GetPersistentSegmentInfo before flush from milvus"] [databaseName=test] [collectionName=testCollection3] [segmentNumBeforeFlush=1]
[2024/08/26 15:41:22.867 +08:00] [INFO] [core/backup_impl_create_backup.go:369] ["GetPersistentSegmentInfo before flush from milvus"] [databaseName=test] [collectionName=testCollection] [segmentNumBeforeFlush=2]
[2024/08/26 15:41:23.016 +08:00] [INFO] [core/backup_impl_create_backup.go:397] ["flush segments"] [databaseName=test] [collectionName=testCollection3] [newSealedSegmentIDs="[]"] [flushedSegmentIDs="[451699909127432419]"] [timeOfSeal=1724658083] [BackupTimestamp=0] [channelCPs="{\"by-dev-rootcoord-dml_2_451699909127432302v0\":{\"channel_name\":\"by-dev-rootcoord-dml_2_451699909127432302v0\",\"msgID\":\"Yu1aAAAAAAA=\",\"msgGroup\":\"datanode-38-by-dev-rootcoord-dml_2_451699909127432302v0-true\",\"timestamp\":452108758873014273}}"]
[2024/08/26 15:41:23.171 +08:00] [INFO] [core/backup_impl_create_backup.go:397] ["flush segments"] [databaseName=test] [collectionName=testCollection] [newSealedSegmentIDs="[]"] [flushedSegmentIDs="[451699909126211091,451699909126413316]"] [timeOfSeal=1724658083] [BackupTimestamp=0] [channelCPs="{\"by-dev-rootcoord-dml_0_451699909126010749v0\":{\"channel_name\":\"by-dev-rootcoord-dml_0_451699909126010749v0\",\"msgID\":\"QTZbAAAAAAA=\",\"msgGroup\":\"datanode-38-by-dev-rootcoord-dml_0_451699909126010749v0-true\",\"timestamp\":452108758873014273}}"]
[2024/08/26 15:41:23.265 +08:00] [INFO] [core/backup_impl_create_backup.go:419] ["GetPersistentSegmentInfo after flush from milvus"] [databaseName=test] [collectionName=testCollection3] [segmentIDsBeforeFlush="[451699909127432419]"] [segmentIDsAfterFlush="[451699909127432419]"] [newL0SegmentsIDs="[]"]
[2024/08/26 15:41:23.265 +08:00] [INFO] [core/backup_impl_create_backup.go:473] ["Finished fill segment"] [databaseName=test] [collectionName=testCollection3] [segments="[451699909127432419]"]
[2024/08/26 15:41:23.265 +08:00] [INFO] [core/backup_impl_create_backup.go:514] ["finish build partition info"] [collectionName=testCollection3] [partitionNum=1] [l0SegmentsNum=0]
[2024/08/26 15:41:23.367 +08:00] [INFO] [core/backup_impl_create_backup.go:419] ["GetPersistentSegmentInfo after flush from milvus"] [databaseName=test] [collectionName=testCollection] [segmentIDsBeforeFlush="[451699909126211091,451699909126413316]"] [segmentIDsAfterFlush="[451699909126413316,451699909126211091]"] [newL0SegmentsIDs="[]"]
[2024/08/26 15:41:23.367 +08:00] [INFO] [core/backup_impl_create_backup.go:473] ["Finished fill segment"] [databaseName=test] [collectionName=testCollection] [segments="[451699909126211091,451699909126413316]"]
[2024/08/26 15:41:23.367 +08:00] [INFO] [core/backup_impl_create_backup.go:514] ["finish build partition info"] [collectionName=testCollection] [partitionNum=1] [l0SegmentsNum=0]
[2024/08/26 15:41:23.367 +08:00] [INFO] [core/backup_impl_create_backup.go:703] ["Finish prepare all collections meta"]
[2024/08/26 15:41:23.367 +08:00] [INFO] [core/backup_impl_create_backup.go:708] ["before backupCollectionExecute"] [collectionID=451699909127432302] [collection=testCollection3]
[2024/08/26 15:41:23.367 +08:00] [INFO] [core/backup_impl_create_backup.go:708] ["before backupCollectionExecute"] [collectionID=451699909126010749] [collection=testCollection]
[2024/08/26 15:41:23.367 +08:00] [INFO] [core/backup_impl_create_backup.go:529] [backupCollectionExecute] [collectionMeta="id:\"958fc2d6-637e-11ef-9b59-8ea5ac62d0b8\" start_time:1724658082 collection_id:451699909126010749 db_name:\"test\" collection_name:\"testCollection\" schema:<name:\"testCollection\" fields:<fieldID:100 name:\"id\" is_primary_key:true data_type:Int64 autoID:true > fields:<fieldID:101 name:\"vector\" data_type:FloatVector type_params:<key:\"dim\" value:\"128\" > > enable_dynamic_field:true > shards_num:1 consistency_level:Bounded backup_timestamp:452108758873014273 has_index:true index_infos:<field_name:\"vector\" index_name:\"index\" index_type:\"HNSW\" params:<key:\"index_type\" value:\"HNSW\" > params:<key:\"metric_type\" value:\"L2\" > params:<key:\"params\" value:\"{\\\"M\\\":\\\"2\\\",\\\"efConstruction\\\":\\\"1\\\"}\" > > load_state:\"Loaded\" backup_physical_timestamp:1724658083 channel_checkpoints:<key:\"by-dev-rootcoord-dml_0_451699909126010749v0\" value:\"CitieS1kZXYtcm9vdGNvb3JkLWRtbF8wXzQ1MTY5OTkwOTEyNjAxMDc0OXYwEghBNlsAAAAAABo8ZGF0YW5vZGUtMzgtYnktZGV2LXJvb3Rjb29yZC1kbWxfMF80NTE2OTk5MDkxMjYwMTA3NDl2MC10cnVlIIGA4KPV0I2jBg==\" > "]
[2024/08/26 15:41:23.368 +08:00] [INFO] [core/backup_impl_create_backup.go:529] [backupCollectionExecute] [collectionMeta="id:\"958fc2d6-637e-11ef-9b59-8ea5ac62d0b8\" start_time:1724658082 collection_id:451699909127432302 db_name:\"test\" collection_name:\"testCollection3\" schema:<name:\"testCollection3\" fields:<fieldID:100 name:\"id\" is_primary_key:true data_type:Int64 autoID:true > fields:<fieldID:101 name:\"vector\" data_type:FloatVector type_params:<key:\"dim\" value:\"128\" > > > shards_num:1 consistency_level:Bounded backup_timestamp:452108758873014273 has_index:true index_infos:<field_name:\"vector\" index_name:\"vector\" index_type:\"AUTOINDEX\" params:<key:\"index_type\" value:\"AUTOINDEX\" > params:<key:\"metric_type\" value:\"L2\" > > load_state:\"NotLoad\" backup_physical_timestamp:1724658083 channel_checkpoints:<key:\"by-dev-rootcoord-dml_2_451699909127432302v0\" value:\"CitieS1kZXYtcm9vdGNvb3JkLWRtbF8yXzQ1MTY5OTkwOTEyNzQzMjMwMnYwEghi7VoAAAAAABo8ZGF0YW5vZGUtMzgtYnktZGV2LXJvb3Rjb29yZC1kbWxfMl80NTE2OTk5MDkxMjc0MzIzMDJ2MC10cnVlIIGA4KPV0I2jBg==\" > "]
[2024/08/26 15:41:24.102 +08:00] [INFO] [storage/minio_chunk_manager.go:144] ["minio chunk manager init success."] [bucketname=bucket-name] [root=by-dev]
[2024/08/26 15:41:24.266 +08:00] [INFO] [storage/minio_chunk_manager.go:144] ["minio chunk manager init success."] [bucketname=bucket-name] [root=by-dev]
[2024/08/26 15:41:25.910 +08:00] [INFO] [core/backup_impl_create_backup.go:564] ["Begin copy data"] [dbName=test] [collectionName=testCollection3] [collectionID=451699909127432302] [partitionID=451699909127432303]
[2024/08/26 15:41:25.910 +08:00] [INFO] [core/backup_impl_create_backup.go:800] ["copy segment"] [collection_id=451699909127432302] [partition_id=451699909127432303] [segment_id=451699909127432419] [group_id=1] [backupBinlogPath=backup/mybackup/binlogs]
[2024/08/26 15:41:28.451 +08:00] [INFO] [core/backup_impl_create_backup.go:564] ["Begin copy data"] [dbName=test] [collectionName=testCollection] [collectionID=451699909126010749] [partitionID=451699909126010750]
[2024/08/26 15:41:28.451 +08:00] [INFO] [core/backup_impl_create_backup.go:800] ["copy segment"] [collection_id=451699909126010749] [partition_id=451699909126010750] [segment_id=451699909126413316] [group_id=1] [backupBinlogPath=backup/mybackup/binlogs]
[2024/08/26 15:41:28.452 +08:00] [INFO] [core/backup_impl_create_backup.go:800] ["copy segment"] [collection_id=451699909126010749] [partition_id=451699909126010750] [segment_id=451699909126211091] [group_id=1] [backupBinlogPath=backup/mybackup/binlogs]
[2024/08/26 15:41:31.143 +08:00] [INFO] [core/backup_impl_create_backup.go:604] ["Finish copy data"] [dbName=test] [collectionName=testCollection3]
[2024/08/26 15:41:34.666 +08:00] [INFO] [core/backup_impl_create_backup.go:604] ["Finish copy data"] [dbName=test] [collectionName=testCollection]
[2024/08/26 15:41:34.666 +08:00] [INFO] [core/backup_meta_manager.go:501] ["Get backup"] [state=BACKUP_SUCCESS] [backupedSize=8135812] [totalSize=8135812] [progress=100]
[2024/08/26 15:41:34.667 +08:00] [INFO] [core/backup_impl_create_backup.go:754] ["Final backupInfo"] [backupInfo="id:\"958fc2d6-637e-11ef-9b59-8ea5ac62d0b8\" state_code:BACKUP_SUCCESS start_time:1724658080780 end_time:1724658094666 progress:100 name:\"mybackup\" collection_backups:<id:\"958fc2d6-637e-11ef-9b59-8ea5ac62d0b8\" start_time:1724658082 end_time:1724658091 collection_id:451699909127432302 db_name:\"test\" collection_name:\"testCollection3\" schema:<name:\"testCollection3\" fields:<fieldID:100 name:\"id\" is_primary_key:true data_type:Int64 autoID:true > fields:<fieldID:101 name:\"vector\" data_type:FloatVector type_params:<key:\"dim\" value:\"128\" > > > shards_num:1 consistency_level:Bounded partition_backups:<partition_id:451699909127432303 partition_name:\"_default\" collection_id:451699909127432302 segment_backups:<segment_id:451699909127432419 collection_id:451699909127432302 partition_id:451699909127432303 num_of_rows:100 binlogs:<binlogs:<log_path:\"by-dev/insert_log/451699909127432302/451699909127432303/451699909127432419/0/451699909126213011\" log_size:677 > > binlogs:<fieldID:1 binlogs:<log_path:\"by-dev/insert_log/451699909127432302/451699909127432303/451699909127432419/1/451699909126213012\" log_size:457 > > binlogs:<fieldID:100 binlogs:<log_path:\"by-dev/insert_log/451699909127432302/451699909127432303/451699909127432419/100/451699909126213009\" log_size:677 > > binlogs:<fieldID:101 binlogs:<log_path:\"by-dev/insert_log/451699909127432302/451699909127432303/451699909127432419/101/451699909126213010\" log_size:53729 > > deltalogs:<fieldID:451699909126213016 binlogs:<log_path:\"by-dev/delta_log/451699909127432302/451699909127432303/451699909127432419/451699909126213016\" log_size:675 > > size:56215 group_id:451699909127432419 backuped:true > size:56215 load_state:\"NotLoad\" > backup_timestamp:452108758873014273 size:56215 has_index:true index_infos:<field_name:\"vector\" index_name:\"vector\" index_type:\"AUTOINDEX\" params:<key:\"index_type\" value:\"AUTOINDEX\" > params:<key:\"metric_type\" value:\"L2\" > > load_state:\"NotLoad\" backup_physical_timestamp:1724658083 channel_checkpoints:<key:\"by-dev-rootcoord-dml_2_451699909127432302v0\" value:\"CitieS1kZXYtcm9vdGNvb3JkLWRtbF8yXzQ1MTY5OTkwOTEyNzQzMjMwMnYwEghi7VoAAAAAABo8ZGF0YW5vZGUtMzgtYnktZGV2LXJvb3Rjb29yZC1kbWxfMl80NTE2OTk5MDkxMjc0MzIzMDJ2MC10cnVlIIGA4KPV0I2jBg==\" > > collection_backups:<id:\"958fc2d6-637e-11ef-9b59-8ea5ac62d0b8\" start_time:1724658082 end_time:1724658094 collection_id:451699909126010749 db_name:\"test\" collection_name:\"testCollection\" schema:<name:\"testCollection\" fields:<fieldID:100 name:\"id\" is_primary_key:true data_type:Int64 autoID:true > fields:<fieldID:101 name:\"vector\" data_type:FloatVector type_params:<key:\"dim\" value:\"128\" > > enable_dynamic_field:true > shards_num:1 consistency_level:Bounded partition_backups:<partition_id:451699909126010750 partition_name:\"_default\" collection_id:451699909126010749 segment_backups:<segment_id:451699909126211091 collection_id:451699909126010749 partition_id:451699909126010750 num_of_rows:5000 binlogs:<binlogs:<log_path:\"by-dev/insert_log/451699909126010749/451699909126010750/451699909126211091/0/451699909126213001\" log_size:13709 > > binlogs:<fieldID:1 binlogs:<log_path:\"by-dev/insert_log/451699909126010749/451699909126010750/451699909126211091/1/451699909126212997\" log_size:459 > > binlogs:<fieldID:100 binlogs:<log_path:\"by-dev/insert_log/451699909126010749/451699909126010750/451699909126211091/100/451699909126212998\" log_size:13709 > > binlogs:<fieldID:101 binlogs:<log_path:\"by-dev/insert_log/451699909126010749/451699909126010750/451699909126211091/101/451699909126212999\" log_size:2567476 > > binlogs:<fieldID:102 binlogs:<log_path:\"by-dev/insert_log/451699909126010749/451699909126010750/451699909126211091/102/451699909126213000\" log_size:100788 > > deltalogs:<> size:2696141 group_id:451699909126211091 backuped:true > segment_backups:<segment_id:451699909126413316 collection_id:451699909126010749 partition_id:451699909126010750 num_of_rows:10000 binlogs:<binlogs:<log_path:\"by-dev/insert_log/451699909126010749/451699909126010750/451699909126413316/0/451699909126213006\" log_size:28174 > > binlogs:<fieldID:1 binlogs:<log_path:\"by-dev/insert_log/451699909126010749/451699909126010750/451699909126413316/1/451699909126213007\" log_size:464 > > binlogs:<fieldID:100 binlogs:<log_path:\"by-dev/insert_log/451699909126010749/451699909126010750/451699909126413316/100/451699909126213003\" log_size:28174 > > binlogs:<fieldID:101 binlogs:<log_path:\"by-dev/insert_log/451699909126010749/451699909126010750/451699909126413316/101/451699909126213004\" log_size:5129681 > > binlogs:<fieldID:102 binlogs:<log_path:\"by-dev/insert_log/451699909126010749/451699909126010750/451699909126413316/102/451699909126213005\" log_size:196963 > > deltalogs:<> size:5383456 group_id:451699909126413316 backuped:true > size:8079597 load_state:\"Loaded\" > backup_timestamp:452108758873014273 size:8079597 has_index:true index_infos:<field_name:\"vector\" index_name:\"index\" index_type:\"HNSW\" params:<key:\"index_type\" value:\"HNSW\" > params:<key:\"metric_type\" value:\"L2\" > params:<key:\"params\" value:\"{\\\"M\\\":\\\"2\\\",\\\"efConstruction\\\":\\\"1\\\"}\" > > load_state:\"Loaded\" backup_physical_timestamp:1724658083 channel_checkpoints:<key:\"by-dev-rootcoord-dml_0_451699909126010749v0\" value:\"CitieS1kZXYtcm9vdGNvb3JkLWRtbF8wXzQ1MTY5OTkwOTEyNjAxMDc0OXYwEghBNlsAAAAAABo8ZGF0YW5vZGUtMzgtYnktZGV2LXJvb3Rjb29yZC1kbWxfMF80NTE2OTk5MDkxMjYwMTA3NDl2MC10cnVlIIGA4KPV0I2jBg==\" > > size:8135812 milvus_version:\"v2.4.5\" "]
[2024/08/26 15:41:34.668 +08:00] [INFO] [core/backup_impl_create_backup.go:757] ["collection meta"] [value="{\"infos\":[{\"id\":\"958fc2d6-637e-11ef-9b59-8ea5ac62d0b8\",\"state_code\":0,\"start_time\":1724658082,\"end_time\":1724658091,\"progress\":0,\"collection_id\":451699909127432302,\"db_name\":\"test\",\"collection_name\":\"testCollection3\",\"schema\":{\"name\":\"testCollection3\",\"fields\":[{\"fieldID\":100,\"name\":\"id\",\"is_primary_key\":true,\"data_type\":5,\"autoID\":true},{\"fieldID\":101,\"name\":\"vector\",\"data_type\":101,\"type_params\":[{\"key\":\"dim\",\"value\":\"128\"}]}]},\"shards_num\":1,\"consistency_level\":2,\"partition_backups\":[{\"partition_id\":451699909127432303,\"partition_name\":\"_default\",\"collection_id\":451699909127432302,\"segment_backups\":[{\"segment_id\":451699909127432419,\"collection_id\":451699909127432302,\"partition_id\":451699909127432303,\"num_of_rows\":100,\"binlogs\":[{\"binlogs\":[{\"log_path\":\"by-dev/insert_log/451699909127432302/451699909127432303/451699909127432419/0/451699909126213011\",\"log_size\":677}]},{\"fieldID\":1,\"binlogs\":[{\"log_path\":\"by-dev/insert_log/451699909127432302/451699909127432303/451699909127432419/1/451699909126213012\",\"log_size\":457}]},{\"fieldID\":100,\"binlogs\":[{\"log_path\":\"by-dev/insert_log/451699909127432302/451699909127432303/451699909127432419/100/451699909126213009\",\"log_size\":677}]},{\"fieldID\":101,\"binlogs\":[{\"log_path\":\"by-dev/insert_log/451699909127432302/451699909127432303/451699909127432419/101/451699909126213010\",\"log_size\":53729}]}],\"deltalogs\":[{\"fieldID\":451699909126213016,\"binlogs\":[{\"log_path\":\"by-dev/delta_log/451699909127432302/451699909127432303/451699909127432419/451699909126213016\",\"log_size\":675}]}],\"size\":56215,\"group_id\":451699909127432419,\"backuped\":true}],\"size\":56215,\"load_state\":\"NotLoad\"}],\"backup_timestamp\":452108758873014273,\"size\":56215,\"has_index\":true,\"index_infos\":[{\"field_name\":\"vector\",\"index_name\":\"vector\",\"index_type\":\"AUTOINDEX\",\"params\":{\"index_type\":\"AUTOINDEX\",\"metric_type\":\"L2\"}}],\"load_state\":\"NotLoad\",\"backup_physical_timestamp\":1724658083,\"channel_checkpoints\":{\"by-dev-rootcoord-dml_2_451699909127432302v0\":\"CitieS1kZXYtcm9vdGNvb3JkLWRtbF8yXzQ1MTY5OTkwOTEyNzQzMjMwMnYwEghi7VoAAAAAABo8ZGF0YW5vZGUtMzgtYnktZGV2LXJvb3Rjb29yZC1kbWxfMl80NTE2OTk5MDkxMjc0MzIzMDJ2MC10cnVlIIGA4KPV0I2jBg==\"}},{\"id\":\"958fc2d6-637e-11ef-9b59-8ea5ac62d0b8\",\"state_code\":0,\"start_time\":1724658082,\"end_time\":1724658094,\"progress\":0,\"collection_id\":451699909126010749,\"db_name\":\"test\",\"collection_name\":\"testCollection\",\"schema\":{\"name\":\"testCollection\",\"fields\":[{\"fieldID\":100,\"name\":\"id\",\"is_primary_key\":true,\"data_type\":5,\"autoID\":true},{\"fieldID\":101,\"name\":\"vector\",\"data_type\":101,\"type_params\":[{\"key\":\"dim\",\"value\":\"128\"}]}],\"enable_dynamic_field\":true},\"shards_num\":1,\"consistency_level\":2,\"partition_backups\":[{\"partition_id\":451699909126010750,\"partition_name\":\"_default\",\"collection_id\":451699909126010749,\"segment_backups\":[{\"segment_id\":451699909126211091,\"collection_id\":451699909126010749,\"partition_id\":451699909126010750,\"num_of_rows\":5000,\"binlogs\":[{\"binlogs\":[{\"log_path\":\"by-dev/insert_log/451699909126010749/451699909126010750/451699909126211091/0/451699909126213001\",\"log_size\":13709}]},{\"fieldID\":1,\"binlogs\":[{\"log_path\":\"by-dev/insert_log/451699909126010749/451699909126010750/451699909126211091/1/451699909126212997\",\"log_size\":459}]},{\"fieldID\":100,\"binlogs\":[{\"log_path\":\"by-dev/insert_log/451699909126010749/451699909126010750/451699909126211091/100/451699909126212998\",\"log_size\":13709}]},{\"fieldID\":101,\"binlogs\":[{\"log_path\":\"by-dev/insert_log/451699909126010749/451699909126010750/451699909126211091/101/451699909126212999\",\"log_size\":2567476}]},{\"fieldID\":102,\"binlogs\":[{\"log_path\":\"by-dev/insert_log/451699909126010749/451699909126010750/451699909126211091/102/451699909126213000\",\"log_size\":100788}]}],\"deltalogs\":[{}],\"size\":2696141,\"group_id\":451699909126211091,\"backuped\":true},{\"segment_id\":451699909126413316,\"collection_id\":451699909126010749,\"partition_id\":451699909126010750,\"num_of_rows\":10000,\"binlogs\":[{\"binlogs\":[{\"log_path\":\"by-dev/insert_log/451699909126010749/451699909126010750/451699909126413316/0/451699909126213006\",\"log_size\":28174}]},{\"fieldID\":1,\"binlogs\":[{\"log_path\":\"by-dev/insert_log/451699909126010749/451699909126010750/451699909126413316/1/451699909126213007\",\"log_size\":464}]},{\"fieldID\":100,\"binlogs\":[{\"log_path\":\"by-dev/insert_log/451699909126010749/451699909126010750/451699909126413316/100/451699909126213003\",\"log_size\":28174}]},{\"fieldID\":101,\"binlogs\":[{\"log_path\":\"by-dev/insert_log/451699909126010749/451699909126010750/451699909126413316/101/451699909126213004\",\"log_size\":5129681}]},{\"fieldID\":102,\"binlogs\":[{\"log_path\":\"by-dev/insert_log/451699909126010749/451699909126010750/451699909126413316/102/451699909126213005\",\"log_size\":196963}]}],\"deltalogs\":[{}],\"size\":5383456,\"group_id\":451699909126413316,\"backuped\":true}],\"size\":8079597,\"load_state\":\"Loaded\"}],\"backup_timestamp\":452108758873014273,\"size\":8079597,\"has_index\":true,\"index_infos\":[{\"field_name\":\"vector\",\"index_name\":\"index\",\"index_type\":\"HNSW\",\"params\":{\"index_type\":\"HNSW\",\"metric_type\":\"L2\",\"params\":\"{\\\"M\\\":\\\"2\\\",\\\"efConstruction\\\":\\\"1\\\"}\"}}],\"load_state\":\"Loaded\",\"backup_physical_timestamp\":1724658083,\"channel_checkpoints\":{\"by-dev-rootcoord-dml_0_451699909126010749v0\":\"CitieS1kZXYtcm9vdGNvb3JkLWRtbF8wXzQ1MTY5OTkwOTEyNjAxMDc0OXYwEghBNlsAAAAAABo8ZGF0YW5vZGUtMzgtYnktZGV2LXJvb3Rjb29yZC1kbWxfMF80NTE2OTk5MDkxMjYwMTA3NDl2MC10cnVlIIGA4KPV0I2jBg==\"}}]}"]
[2024/08/26 15:41:35.608 +08:00] [INFO] [core/backup_impl_create_backup.go:787] ["finish writeBackupInfoMeta"] [path=backup/mybackup/] [backupName=mybackup] ["backup meta"="{\"id\":\"958fc2d6-637e-11ef-9b59-8ea5ac62d0b8\",\"state_code\":2,\"start_time\":1724658080780,\"end_time\":1724658094666,\"progress\":100,\"name\":\"mybackup\",\"size\":8135812,\"milvus_version\":\"v2.4.5\"}"]
[2024/08/26 15:41:35.608 +08:00] [INFO] [core/backup_impl_create_backup.go:744] ["finish executeCreateBackup"] [requestId=958fc2d6-637e-11ef-9b59-8ea5ac62d0b8] [backupName=mybackup] [collections="[]"] [async=false]
[2024/08/26 15:41:35.662 +08:00] [INFO] [core/backup_impl_create_backup.go:645] ["Resume Milvus GC response"] [response="{\"code\":1800,\"message\":\"user hasn't authenticated\"}"] [address=http://##.com]
success
duration:16 s
Full log of restoring: ./milvus-backup restore --restore_index -n mybackup -s _recover
0.4.21 (Built on 2024-08-22T04:00:46Z from Git SHA 760ea2c7e05992e1f0f01c9b7a98aee2796b90cb)
[2024/08/26 16:30:18.394 +08:00] [INFO] [logutil/logutil.go:165] ["Log directory"] [configDir=]
[2024/08/26 16:30:18.395 +08:00] [INFO] [logutil/logutil.go:166] ["Set log file to "] [path=logs/backup.log]
[2024/08/26 16:30:18.395 +08:00] [INFO] [cmd/restore.go:47] ["restore cmd input args"] [args="[]"]
[2024/08/26 16:30:18.395 +08:00] [INFO] [core/backup_impl_restore_backup.go:29] ["receive RestoreBackupRequest"] [requestId=6d196fda-6385-11ef-b4ff-8ea5ac62d0b8] [backupName=mybackup] [onlyMeta=false] [restoreIndex=true] [useAutoIndex=false] [dropExistCollection=false] [dropExistIndex=false] [skipCreateCollection=false] [collections="[]"] [CollectionSuffix=_recover] [CollectionRenames={}] [async=false] [bucketName=] [path=] [databaseCollections=] [skipDiskQuotaCheck=false]
[2024/08/26 16:30:18.396 +08:00] [INFO] [core/backup_context.go:117] ["{Base:0x140003f0b00 MaxSegmentGroupSize:2147483648 BackupCollectionParallelism:4 BackupCopyDataParallelism:128 RestoreParallelism:2 KeepTempFiles:false GcPauseEnable:true GcPauseSeconds:7200 GcPauseAddress:http://##.com}"]
[2024/08/26 16:30:18.396 +08:00] [INFO] [core/backup_context.go:118] ["{Base:0x140003f0b00 Enabled:true DebugMode:false SimpleResponse:true}"]
[2024/08/26 16:30:18.396 +08:00] [INFO] [core/backup_context.go:308] ["receive GetBackupRequest"] [requestId=6d1977c8-6385-11ef-b4ff-8ea5ac62d0b8] [backupName=mybackup] [backupId=] [bucketName=] [path=]
[2024/08/26 16:30:18.775 +08:00] [INFO] [storage/minio_chunk_manager.go:144] ["minio chunk manager init success."] [bucketname=bucket-name] [root=backup]
[2024/08/26 16:30:19.535 +08:00] [INFO] [core/backup_context.go:385] ["finish GetBackupRequest"] [requestId=6d1977c8-6385-11ef-b4ff-8ea5ac62d0b8] [backupName=mybackup] [backupId=] [bucketName=] [path=]
[2024/08/26 16:30:19.535 +08:00] [INFO] [core/backup_impl_restore_backup.go:199] ["Collections to restore"] [collection_num=2]
[2024/08/26 16:30:19.953 +08:00] [INFO] [core/backup_impl_restore_backup.go:354] ["Start collection level restore pool"] [parallelism=2]
[2024/08/26 16:30:19.953 +08:00] [INFO] [core/backup_impl_restore_backup.go:358] ["executeRestoreBackupTask start"] [backup_name=mybackup] [backupBucketName=bucket-name] [backupPath=backup/mybackup]
[2024/08/26 16:30:19.954 +08:00] [INFO] [core/backup_impl_restore_backup.go:369] ["start restore"] [backup_db_name=test] [backup_collection_name=test] [target_db_name=test] [target_collection_name=testCollection_recover] [skipDiskQuotaCheck=false] [backupBucketName=bucket-name] [backupPath=backup/mybackup]
[2024/08/26 16:30:19.954 +08:00] [INFO] [core/backup_impl_restore_backup.go:369] ["collection schema"] [backup_db_name=test] [backup_collection_name=test] [target_db_name=test] [target_collection_name=testCollection_recover] [skipDiskQuotaCheck=false] [fields="[{\"ID\":100,\"Name\":\"id\",\"PrimaryKey\":true,\"AutoID\":true,\"Description\":\"\",\"DataType\":5,\"TypeParams\":{},\"IndexParams\":{},\"IsDynamic\":false,\"IsPartitionKey\":false,\"IsClusteringKey\":false,\"ElementType\":0},{\"ID\":101,\"Name\":\"vector\",\"PrimaryKey\":false,\"AutoID\":false,\"Description\":\"\",\"DataType\":101,\"TypeParams\":{\"dim\":\"128\"},\"IndexParams\":{},\"IsDynamic\":false,\"IsPartitionKey\":false,\"IsClusteringKey\":false,\"ElementType\":0}]"]
[2024/08/26 16:30:19.955 +08:00] [INFO] [core/backup_impl_restore_backup.go:369] ["start restore"] [backup_db_name=test] [backup_collection_name=test] [target_db_name=test] [target_collection_name=testCollection3_recover] [skipDiskQuotaCheck=false] [backupBucketName=bucket-name] [backupPath=backup/mybackup]
[2024/08/26 16:30:19.955 +08:00] [INFO] [core/backup_impl_restore_backup.go:369] ["collection schema"] [backup_db_name=test] [backup_collection_name=test] [target_db_name=test] [target_collection_name=testCollection3_recover] [skipDiskQuotaCheck=false] [fields="[{\"ID\":100,\"Name\":\"id\",\"PrimaryKey\":true,\"AutoID\":true,\"Description\":\"\",\"DataType\":5,\"TypeParams\":{},\"IndexParams\":{},\"IsDynamic\":false,\"IsPartitionKey\":false,\"IsClusteringKey\":false,\"ElementType\":0},{\"ID\":101,\"Name\":\"vector\",\"PrimaryKey\":false,\"AutoID\":false,\"Description\":\"\",\"DataType\":101,\"TypeParams\":{\"dim\":\"128\"},\"IndexParams\":{},\"IsDynamic\":false,\"IsPartitionKey\":false,\"IsClusteringKey\":false,\"ElementType\":0}]"]
[2024/08/26 16:30:21.502 +08:00] [INFO] [core/backup_impl_restore_backup.go:369] ["create collection"] [backup_db_name=test] [backup_collection_name=test] [target_db_name=test] [target_collection_name=testCollection_recover] [skipDiskQuotaCheck=false] [hasPartitionKey=false]
[2024/08/26 16:30:21.503 +08:00] [INFO] [core/backup_impl_restore_backup.go:369] ["source index"] [backup_db_name=test] [backup_collection_name=test] [target_db_name=test] [target_collection_name=testCollection_recover] [skipDiskQuotaCheck=false] [indexName=index] [indexType=HNSW] [params="{\"index_type\":\"HNSW\",\"metric_type\":\"L2\",\"params\":\"{\\\"M\\\":\\\"2\\\",\\\"efConstruction\\\":\\\"1\\\"}\"}"]
[2024/08/26 16:30:21.503 +08:00] [INFO] [core/backup_impl_restore_backup.go:369] ["not auto index"] [backup_db_name=test] [backup_collection_name=test] [target_db_name=test] [target_collection_name=testCollection_recover] [skipDiskQuotaCheck=false]
[2024/08/26 16:30:22.493 +08:00] [INFO] [core/backup_impl_restore_backup.go:369] ["create collection"] [backup_db_name=test] [backup_collection_name=test] [target_db_name=test] [target_collection_name=testCollection3_recover] [skipDiskQuotaCheck=false] [hasPartitionKey=false]
[2024/08/26 16:30:22.493 +08:00] [INFO] [core/backup_impl_restore_backup.go:369] ["source index"] [backup_db_name=test] [backup_collection_name=test] [target_db_name=test] [target_collection_name=testCollection3_recover] [skipDiskQuotaCheck=false] [indexName=vector] [indexType=AUTOINDEX] [params="{\"index_type\":\"AUTOINDEX\",\"metric_type\":\"L2\"}"]
[2024/08/26 16:30:22.493 +08:00] [INFO] [core/backup_impl_restore_backup.go:369] ["not auto index"] [backup_db_name=test] [backup_collection_name=test] [target_db_name=test] [target_collection_name=testCollection3_recover] [skipDiskQuotaCheck=false]
[2024/08/26 16:30:23.935 +08:00] [INFO] [storage/minio_chunk_manager.go:144] ["minio chunk manager init success."] [bucketname=bucket-name] [root=by-dev]
[2024/08/26 16:30:23.940 +08:00] [INFO] [core/backup_impl_restore_backup.go:369] ["start restore partition"] [backup_db_name=test] [backup_collection_name=test] [target_db_name=test] [target_collection_name=testCollection_recover] [skipDiskQuotaCheck=false] [partition=_default]
[2024/08/26 16:30:24.155 +08:00] [INFO] [core/backup_impl_restore_backup.go:867] [getBackupPartitionPaths] [bucketName=bucket-name] [backupPath=backup/mybackup] [partitionID=451699909126010750] [groupId=451699909126211091]
[2024/08/26 16:30:24.775 +08:00] [INFO] [storage/minio_chunk_manager.go:144] ["minio chunk manager init success."] [bucketname=bucket-name] [root=by-dev]
[2024/08/26 16:30:24.775 +08:00] [INFO] [core/backup_impl_restore_backup.go:369] ["start restore partition"] [backup_db_name=test] [backup_collection_name=test] [target_db_name=test] [target_collection_name=testCollection3_recover] [skipDiskQuotaCheck=false] [partition=_default]
[2024/08/26 16:30:24.933 +08:00] [INFO] [core/backup_impl_restore_backup.go:867] [getBackupPartitionPaths] [bucketName=bucket-name] [backupPath=backup/mybackup] [partitionID=451699909126010750] [groupId=451699909126413316]
[2024/08/26 16:30:24.978 +08:00] [INFO] [core/backup_impl_restore_backup.go:867] [getBackupPartitionPaths] [bucketName=bucket-name] [backupPath=backup/mybackup] [partitionID=451699909127432303] [groupId=451699909127432419]
[2024/08/26 16:30:25.386 +08:00] [INFO] [core/backup_impl_restore_backup.go:760] ["execute bulk insert"] [db=test] [collection=testCollection_recover] [partition=_default] [files="[backup/mybackup/binlogs/insert_log/451699909126010749/451699909126010750/451699909126211091/,]"] [endTime=452108758873014273]
[2024/08/26 16:30:25.387 +08:00] [INFO] [core/backup_impl_restore_backup.go:760] ["execute bulk insert"] [db=test] [collection=testCollection_recover] [partition=_default] [files="[backup/mybackup/binlogs/insert_log/451699909126010749/451699909126010750/451699909126413316/,]"] [endTime=452108758873014273]
[2024/08/26 16:30:25.600 +08:00] [ERROR] [core/backup_impl_restore_backup.go:782] ["fail to bulk insert"] [db=test] [collectionName=testCollection_recover] [partitionName=_default] [files="[backup/mybackup/binlogs/insert_log/451699909126010749/451699909126010750/451699909126211091/,]"] [error="no binlog to import, input=[paths:\"backup/mybackup/binlogs/insert_log/451699909126010749/451699909126010750/451699909126211091/\" paths:\"\" ]: invalid parameter"] [errorVerbose="no binlog to import, input=[paths:\"backup/mybackup/binlogs/insert_log/451699909126010749/451699909126010750/451699909126211091/\" paths:\"\" ]: invalid parameter\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus-sdk-go/v2/client.handleRespStatus\n | \t/home/runner/go/pkg/mod/github.com/milvus-io/milvus-sdk-go/v2@v2.4.2-0.20240821030256-0c339b63d265/client/collection.go:39\n | github.com/milvus-io/milvus-sdk-go/v2/client.(*GrpcClient).BulkInsert\n | \t/home/runner/go/pkg/mod/github.com/milvus-io/milvus-sdk-go/v2@v2.4.2-0.20240821030256-0c339b63d265/client/insert.go:443\n | github.com/zilliztech/milvus-backup/core.(*MilvusClient).BulkInsert\n | \t/home/runner/work/milvus-backup/milvus-backup/core/milvus_sdk_wrapper.go:124\n | github.com/zilliztech/milvus-backup/core.(*BackupContext).executeBulkInsert\n | \t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:778\n | github.com/zilliztech/milvus-backup/core.(*BackupContext).executeRestoreCollectionTask.func3\n | \t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:605\n | github.com/zilliztech/milvus-backup/core.(*BackupContext).executeRestoreCollectionTask.func7\n | \t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:685\n | github.com/zilliztech/milvus-backup/internal/common.(*WorkerPool).work.func1\n | \t/home/runner/work/milvus-backup/milvus-backup/internal/common/workerpool.go:70\n | golang.org/x/sync/errgroup.(*Group).Go.func1\n | \t/home/runner/go/pkg/mod/golang.org/x/sync@v0.3.0/errgroup/errgroup.go:75\n | runtime.goexit\n | \t/opt/hostedtoolcache/go/1.18.10/x64/src/runtime/asm_arm64.s:1270\nWraps: (2) no binlog to import, input=[paths:\"backup/mybackup/binlogs/insert_log/451699909126010749/451699909126010750/451699909126211091/\" paths:\"\" ]: invalid parameter\nError types: (1) *withstack.withStack (2) *errutil.leafError"] [stack="github.com/zilliztech/milvus-backup/core.(*BackupContext).executeBulkInsert\n\t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:782\ngithub.com/zilliztech/milvus-backup/core.(*BackupContext).executeRestoreCollectionTask.func3\n\t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:605\ngithub.com/zilliztech/milvus-backup/core.(*BackupContext).executeRestoreCollectionTask.func7\n\t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:685\ngithub.com/zilliztech/milvus-backup/internal/common.(*WorkerPool).work.func1\n\t/home/runner/work/milvus-backup/milvus-backup/internal/common/workerpool.go:70\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/sync@v0.3.0/errgroup/errgroup.go:75"]
[2024/08/26 16:30:25.601 +08:00] [ERROR] [core/backup_impl_restore_backup.go:685] ["fail to bulk insert to partition"] [backup_db_name=test] [backup_collection_name=test] [target_db_name=test] [target_collection_name=testCollection_recover] [skipDiskQuotaCheck=false] [partition=_default] [error="no binlog to import, input=[paths:\"backup/mybackup/binlogs/insert_log/451699909126010749/451699909126010750/451699909126211091/\" paths:\"\" ]: invalid parameter"] [errorVerbose="no binlog to import, input=[paths:\"backup/mybackup/binlogs/insert_log/451699909126010749/451699909126010750/451699909126211091/\" paths:\"\" ]: invalid parameter\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus-sdk-go/v2/client.handleRespStatus\n | \t/home/runner/go/pkg/mod/github.com/milvus-io/milvus-sdk-go/v2@v2.4.2-0.20240821030256-0c339b63d265/client/collection.go:39\n | github.com/milvus-io/milvus-sdk-go/v2/client.(*GrpcClient).BulkInsert\n | \t/home/runner/go/pkg/mod/github.com/milvus-io/milvus-sdk-go/v2@v2.4.2-0.20240821030256-0c339b63d265/client/insert.go:443\n | github.com/zilliztech/milvus-backup/core.(*MilvusClient).BulkInsert\n | \t/home/runner/work/milvus-backup/milvus-backup/core/milvus_sdk_wrapper.go:124\n | github.com/zilliztech/milvus-backup/core.(*BackupContext).executeBulkInsert\n | \t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:778\n | github.com/zilliztech/milvus-backup/core.(*BackupContext).executeRestoreCollectionTask.func3\n | \t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:605\n | github.com/zilliztech/milvus-backup/core.(*BackupContext).executeRestoreCollectionTask.func7\n | \t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:685\n | github.com/zilliztech/milvus-backup/internal/common.(*WorkerPool).work.func1\n | \t/home/runner/work/milvus-backup/milvus-backup/internal/common/workerpool.go:70\n | golang.org/x/sync/errgroup.(*Group).Go.func1\n | \t/home/runner/go/pkg/mod/golang.org/x/sync@v0.3.0/errgroup/errgroup.go:75\n | runtime.goexit\n | \t/opt/hostedtoolcache/go/1.18.10/x64/src/runtime/asm_arm64.s:1270\nWraps: (2) no binlog to import, input=[paths:\"backup/mybackup/binlogs/insert_log/451699909126010749/451699909126010750/451699909126211091/\" paths:\"\" ]: invalid parameter\nError types: (1) *withstack.withStack (2) *errutil.leafError"] [stack="github.com/zilliztech/milvus-backup/core.(*BackupContext).executeRestoreCollectionTask.func7\n\t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:685\ngithub.com/zilliztech/milvus-backup/internal/common.(*WorkerPool).work.func1\n\t/home/runner/work/milvus-backup/milvus-backup/internal/common/workerpool.go:70\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/sync@v0.3.0/errgroup/errgroup.go:75"]
[2024/08/26 16:30:25.602 +08:00] [ERROR] [core/backup_impl_restore_backup.go:371] ["executeRestoreCollectionTask failed"] [TargetDBName=test] [TargetCollectionName=testCollection_recover] [error="no binlog to import, input=[paths:\"backup/mybackup/binlogs/insert_log/451699909126010749/451699909126010750/451699909126211091/\" paths:\"\" ]: invalid parameter"] [errorVerbose="no binlog to import, input=[paths:\"backup/mybackup/binlogs/insert_log/451699909126010749/451699909126010750/451699909126211091/\" paths:\"\" ]: invalid parameter\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus-sdk-go/v2/client.handleRespStatus\n | \t/home/runner/go/pkg/mod/github.com/milvus-io/milvus-sdk-go/v2@v2.4.2-0.20240821030256-0c339b63d265/client/collection.go:39\n | github.com/milvus-io/milvus-sdk-go/v2/client.(*GrpcClient).BulkInsert\n | \t/home/runner/go/pkg/mod/github.com/milvus-io/milvus-sdk-go/v2@v2.4.2-0.20240821030256-0c339b63d265/client/insert.go:443\n | github.com/zilliztech/milvus-backup/core.(*MilvusClient).BulkInsert\n | \t/home/runner/work/milvus-backup/milvus-backup/core/milvus_sdk_wrapper.go:124\n | github.com/zilliztech/milvus-backup/core.(*BackupContext).executeBulkInsert\n | \t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:778\n | github.com/zilliztech/milvus-backup/core.(*BackupContext).executeRestoreCollectionTask.func3\n | \t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:605\n | github.com/zilliztech/milvus-backup/core.(*BackupContext).executeRestoreCollectionTask.func7\n | \t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:685\n | github.com/zilliztech/milvus-backup/internal/common.(*WorkerPool).work.func1\n | \t/home/runner/work/milvus-backup/milvus-backup/internal/common/workerpool.go:70\n | golang.org/x/sync/errgroup.(*Group).Go.func1\n | \t/home/runner/go/pkg/mod/golang.org/x/sync@v0.3.0/errgroup/errgroup.go:75\n | runtime.goexit\n | \t/opt/hostedtoolcache/go/1.18.10/x64/src/runtime/asm_arm64.s:1270\nWraps: (2) no binlog to import, input=[paths:\"backup/mybackup/binlogs/insert_log/451699909126010749/451699909126010750/451699909126211091/\" paths:\"\" ]: invalid parameter\nError types: (1) *withstack.withStack (2) *errutil.leafError"] [stack="github.com/zilliztech/milvus-backup/core.(*BackupContext).executeRestoreBackupTask.func1\n\t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:371\ngithub.com/zilliztech/milvus-backup/internal/common.(*WorkerPool).work.func1\n\t/home/runner/work/milvus-backup/milvus-backup/internal/common/workerpool.go:70\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/sync@v0.3.0/errgroup/errgroup.go:75"]
[2024/08/26 16:30:25.603 +08:00] [ERROR] [core/backup_impl_restore_backup.go:782] ["fail to bulk insert"] [db=test] [collectionName=testCollection_recover] [partitionName=_default] [files="[backup/mybackup/binlogs/insert_log/451699909126010749/451699909126010750/451699909126413316/,]"] [error="rpc error: code = Canceled desc = context canceled"] [stack="github.com/zilliztech/milvus-backup/core.(*BackupContext).executeBulkInsert\n\t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:782\ngithub.com/zilliztech/milvus-backup/core.(*BackupContext).executeRestoreCollectionTask.func3\n\t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:605\ngithub.com/zilliztech/milvus-backup/core.(*BackupContext).executeRestoreCollectionTask.func7\n\t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:685\ngithub.com/zilliztech/milvus-backup/internal/common.(*WorkerPool).work.func1\n\t/home/runner/work/milvus-backup/milvus-backup/internal/common/workerpool.go:70\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/sync@v0.3.0/errgroup/errgroup.go:75"]
[2024/08/26 16:30:25.603 +08:00] [ERROR] [core/backup_impl_restore_backup.go:685] ["fail to bulk insert to partition"] [backup_db_name=test] [backup_collection_name=test] [target_db_name=test] [target_collection_name=testCollection_recover] [skipDiskQuotaCheck=false] [partition=_default] [error="rpc error: code = Canceled desc = context canceled"] [stack="github.com/zilliztech/milvus-backup/core.(*BackupContext).executeRestoreCollectionTask.func7\n\t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:685\ngithub.com/zilliztech/milvus-backup/internal/common.(*WorkerPool).work.func1\n\t/home/runner/work/milvus-backup/milvus-backup/internal/common/workerpool.go:70\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/sync@v0.3.0/errgroup/errgroup.go:75"]
[2024/08/26 16:30:25.603 +08:00] [WARN] [storage/minio_chunk_manager.go:425] ["failed to list with prefix"] [bucket=bucket-name] [prefix=backup/mybackup/binlogs/delta_log/451699909127432302/451699909127432303/451699909127432419/] [error="Get \"https://minio.com/bucket-name/?delimiter=%2F&encoding-type=url&fetch-owner=true&list-type=2&prefix=backup%2Fmybackup%2Fbinlogs%2Fdelta_log%2F451699909127432302%2F451699909127432303%2F451699909127432419%2F\": context canceled"]
[2024/08/26 16:30:25.604 +08:00] [WARN] [storage/minio_chunk_manager.go:235] ["MinioChunkManager Exist errResponse"] [bucket=bucket-name] [filePath=backup/mybackup/binlogs/delta_log/451699909127432302/451699909127432303/451699909127432419/] [errResponse="Error response code ."]
[2024/08/26 16:30:25.604 +08:00] [WARN] [storage/minio_chunk_manager.go:243] ["failed to stat object"] [path=backup/mybackup/binlogs/delta_log/451699909127432302/451699909127432303/451699909127432419/] [error="Get \"https://minio.com/bucket-name/?delimiter=%2F&encoding-type=url&fetch-owner=true&list-type=2&prefix=backup%2Fmybackup%2Fbinlogs%2Fdelta_log%2F451699909127432302%2F451699909127432303%2F451699909127432419%2F\": context canceled"]
[2024/08/26 16:30:25.605 +08:00] [WARN] [core/backup_impl_restore_backup.go:885] ["check binlog exist fail"] [error="Get \"https://minio.com/bucket-name/?delimiter=%2F&encoding-type=url&fetch-owner=true&list-type=2&prefix=backup%2Fmybackup%2Fbinlogs%2Fdelta_log%2F451699909127432302%2F451699909127432303%2F451699909127432419%2F\": context canceled"]
[2024/08/26 16:30:25.605 +08:00] [ERROR] [core/backup_impl_restore_backup.go:369] ["fail to get partition backup binlog files"] [backup_db_name=test] [backup_collection_name=test] [target_db_name=test] [target_collection_name=testCollection3_recover] [skipDiskQuotaCheck=false] [error="Get \"https://minio.com/bucket-name/?delimiter=%2F&encoding-type=url&fetch-owner=true&list-type=2&prefix=backup%2Fmybackup%2Fbinlogs%2Fdelta_log%2F451699909127432302%2F451699909127432303%2F451699909127432419%2F\": context canceled"] [partition=_default] [stack="github.com/zilliztech/milvus-backup/core.(*BackupContext).executeRestoreBackupTask.func1\n\t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:369\ngithub.com/zilliztech/milvus-backup/internal/common.(*WorkerPool).work.func1\n\t/home/runner/work/milvus-backup/milvus-backup/internal/common/workerpool.go:70\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/sync@v0.3.0/errgroup/errgroup.go:75"]
[2024/08/26 16:30:25.605 +08:00] [ERROR] [core/backup_impl_restore_backup.go:371] ["executeRestoreCollectionTask failed"] [TargetDBName=test] [TargetCollectionName=testCollection3_recover] [error="Get \"https://minio.com/bucket-name/?delimiter=%2F&encoding-type=url&fetch-owner=true&list-type=2&prefix=backup%2Fmybackup%2Fbinlogs%2Fdelta_log%2F451699909127432302%2F451699909127432303%2F451699909127432419%2F\": context canceled"] [stack="github.com/zilliztech/milvus-backup/core.(*BackupContext).executeRestoreBackupTask.func1\n\t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:371\ngithub.com/zilliztech/milvus-backup/internal/common.(*WorkerPool).work.func1\n\t/home/runner/work/milvus-backup/milvus-backup/internal/common/workerpool.go:70\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/sync@v0.3.0/errgroup/errgroup.go:75"]
[2024/08/26 16:30:25.605 +08:00] [ERROR] [core/backup_impl_restore_backup.go:335] ["execute restore collection fail"] [backupId=958fc2d6-637e-11ef-9b59-8ea5ac62d0b8] [error="workerpool: execute job no binlog to import, input=[paths:\"backup/mybackup/binlogs/insert_log/451699909126010749/451699909126010750/451699909126211091/\" paths:\"\" ]: invalid parameter"] [stack="github.com/zilliztech/milvus-backup/core.(*BackupContext).RestoreBackup\n\t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:335\ngithub.com/zilliztech/milvus-backup/cmd.glob..func7\n\t/home/runner/work/milvus-backup/milvus-backup/cmd/restore.go:84\ngithub.com/spf13/cobra.(*Command).execute\n\t/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.5.0/command.go:876\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.5.0/command.go:990\ngithub.com/spf13/cobra.(*Command).Execute\n\t/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.5.0/command.go:918\ngithub.com/zilliztech/milvus-backup/cmd.Execute\n\t/home/runner/work/milvus-backup/milvus-backup/cmd/root.go:35\nmain.main\n\t/home/runner/work/milvus-backup/milvus-backup/main.go:24\nruntime.main\n\t/opt/hostedtoolcache/go/1.18.10/x64/src/runtime/proc.go:250"]
workerpool: execute job no binlog to import, input=[paths:"backup/mybackup/binlogs/insert_log/451699909126010749/451699909126010750/451699909126211091/" paths:"" ]: invalid parameter
duration:7 s
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 22 (8 by maintainers)
@elpsycongroo Actullay it is supported.
minio.backupUseSSL, I didn’t add it in backup.yaml. You can refer to the code:@wayblink
./milvus-backup create -n mybackup -d testcreate.log./milvus-backup restore --restore_index -n mybackup -s _recoverrestore.logsome information is masked.