milvus-backup: [Bug]: [Very Urgent] restore_collection: bulk insert failed: misaligned binlog count

Current Behavior

Happened during restore,

[error="restore_collection: restore data: restore_collection: restore partition data v1: restore_collection: restore not L0 groups: restore_collection: wait worker pool: workerpool: execute job restore_collection: restore data v1 bulk insert via grpc: restore_collection: bulk insert failed: misaligned binlog count, field102:102, field0:103: importing data failed"] [stack="github.com/zilliztech/milvus-backup/internal/common.(*WorkerPool).work.func1\n\t/home/runner/work/milvus-backup/milvus-backup/internal/common/workerpool.go:71\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/sync@v0.10.0/errgroup/errgroup.go:78"]

Expected Behavior

No response

Steps To Reproduce


Environment


Anything else?

No response

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 17

Most upvoted comments

Hi @thainguyen-khc

Thank you for your feedback—we completely understand how critical reliable backup and restore processes are for any database system.

As mentioned earlier, although the actual data is not lost, older versions of the backup tool might backup some unnecessary binlogs due to the lack of a precise binlog list API in earlier versions of Milvus. This could cause confusion and inconsistencies during restore.

To address this:

  • In Milvus v2.5.8, we introduced a new API to accurately retrieve the list of binlogs.
  • Starting from milvus-backup v0.5.4, the tool uses this API to ensure only the necessary binlogs are backed up.
  • Furthermore, when using v0.5.4 or above of the backup tool with Milvus versions earlier than 2.5.8, we’ve implemented an extra consistency check during backup. If inconsistent binlogs are detected, the tool will now fail fast and return an error, rather than silently completing with potential issues. This prevents incorrect backups from being created in the first place.

We strongly recommend upgrading to milvus-backup v0.5.5 to take advantage of these improvements.

Please let us know if you have additional questions.

Hi all,

I’m experiencing the same issue as @Andy6132024 . I’m running Milvus 2.5.10 with milvus-backup tool v0.5.3. A reliable backup and restore process is essential and crucial for any database system. I feel very uncertain about the current strategy used by the milvus-backup tool, especially when backups fail to restore properly. This raises serious concerns about potential data loss. This is also very critical and urgent for me.

This manual fix can be quite complex.

If the data still exists in the source Milvus cluster, re-running the backup is usually a much simpler and safer approach.

Yes, you’re right — this issue doesn’t mean the original binlogs are bad, just that some extra invalid binlogs may have been included due to recent network issues during DataNode uploads.

If the backup was created during a stable period, the problem likely won’t occur. But since it’s hard to know whether a DataNode had recent retries, manual binlog count checks may be needed.

The upcoming v0.5.4 backup tool will include automatic checks to catch this during backup.

May I ask why this felt like a hit-or-miss issue?

That’s a great question.

The reason this issue can feel hit-or-miss is because invalid (dirty) binlogs don’t always exist—their presence depends on specific runtime conditions. These binlogs are usually created when a network error occurs during a DataNode’s upload of binlogs to object storage. In such cases, the DataNode retries the upload, but the previously uploaded (incomplete or outdated) binlogs are not immediately cleaned up. Instead, they remain in storage until the next garbage collection (GC) cycle.

If a backup is taken during this window—after a failed upload but before GC removes the old files—the backup might capture these invalid binlogs. That’s why the issue doesn’t always occur, even when using the same collection and Milvus version.

As described in Issue #476, our long-term solution is for the backup tool (as of Milvus 2.5.9 and above) to always fetch the binlog file list from the Proxy node, rather than listing files directly from object storage. This ensures only valid binlogs are included in the backup.

For older versions of Milvus, we’ve also added a safety check #636 : if the backup detects a mismatch in binlog counts, it will now fail explicitly with an error, rather than silently producing an invalid backup. This helps avoid confusion and reduces the risk of restoring incomplete data.