wandb: [CLI]: sync offline-run artifacts get duplicated on web-server

Describe the bug

When syncing an offline run to the web via the sync command, the artifacts get duplicated.

In my code I have a loop logging each time a table artifact:

wandb.log({'table': table = wandb.Table(...})

When I use an online run I get the wanted output of a table with versions (each representing a loop iteration) as follows: In this example, the loop performed the python code above 8 times creating versions 0-7.

The problem starts when I train it with WANDB_MODE=offline and then run the following command:

wandb sync offline-run-<run-id>

The first time it will work correctly and will upload all versions up to the current loop status (lets say versions 0-2), when the loop will continue, if I’ll run the command again, instead of uploading versions 3-7, it will upload 0-2 again and start the counting from version 3, ending up with versions 0-10, where versions 0-2 are the same as versions 3-5.

Additional Files

No response

Environment

WandB version: 0.12.21

OS: CentOS Stream 8

Python version: 3.9.12

Versions of relevant libraries:

Additional Context

No response