milvus-backup: [Bug]: [Very Urgent] restore_collection: bulk insert failed: misaligned binlog count
Current Behavior
Happened during restore,
[error="restore_collection: restore data: restore_collection: restore partition data v1: restore_collection: restore not L0 groups: restore_collection: wait worker pool: workerpool: execute job restore_collection: restore data v1 bulk insert via grpc: restore_collection: bulk insert failed: misaligned binlog count, field102:102, field0:103: importing data failed"] [stack="github.com/zilliztech/milvus-backup/internal/common.(*WorkerPool).work.func1\n\t/home/runner/work/milvus-backup/milvus-backup/internal/common/workerpool.go:71\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/sync@v0.10.0/errgroup/errgroup.go:78"]
Expected Behavior
No response
Steps To Reproduce
Environment
Anything else?
No response
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 17
Hi @thainguyen-khc
Thank you for your feedback—we completely understand how critical reliable backup and restore processes are for any database system.
As mentioned earlier, although the actual data is not lost, older versions of the backup tool might backup some unnecessary binlogs due to the lack of a precise binlog list API in earlier versions of Milvus. This could cause confusion and inconsistencies during restore.
To address this:
We strongly recommend upgrading to milvus-backup v0.5.5 to take advantage of these improvements.
Please let us know if you have additional questions.
Hi all,
I’m experiencing the same issue as @Andy6132024 . I’m running Milvus 2.5.10 with milvus-backup tool v0.5.3. A reliable backup and restore process is essential and crucial for any database system. I feel very uncertain about the current strategy used by the milvus-backup tool, especially when backups fail to restore properly. This raises serious concerns about potential data loss. This is also very critical and urgent for me.
This manual fix can be quite complex.
If the data still exists in the source Milvus cluster, re-running the backup is usually a much simpler and safer approach.
Yes, you’re right — this issue doesn’t mean the original binlogs are bad, just that some extra invalid binlogs may have been included due to recent network issues during DataNode uploads.
If the backup was created during a stable period, the problem likely won’t occur. But since it’s hard to know whether a DataNode had recent retries, manual binlog count checks may be needed.
The upcoming v0.5.4 backup tool will include automatic checks to catch this during backup.
That’s a great question.
The reason this issue can feel hit-or-miss is because invalid (dirty) binlogs don’t always exist—their presence depends on specific runtime conditions. These binlogs are usually created when a network error occurs during a DataNode’s upload of binlogs to object storage. In such cases, the DataNode retries the upload, but the previously uploaded (incomplete or outdated) binlogs are not immediately cleaned up. Instead, they remain in storage until the next garbage collection (GC) cycle.
If a backup is taken during this window—after a failed upload but before GC removes the old files—the backup might capture these invalid binlogs. That’s why the issue doesn’t always occur, even when using the same collection and Milvus version.
As described in Issue #476, our long-term solution is for the backup tool (as of Milvus 2.5.9 and above) to always fetch the binlog file list from the Proxy node, rather than listing files directly from object storage. This ensures only valid binlogs are included in the backup.
For older versions of Milvus, we’ve also added a safety check #636 : if the backup detects a mismatch in binlog counts, it will now fail explicitly with an error, rather than silently producing an invalid backup. This helps avoid confusion and reduces the risk of restoring incomplete data.