wandb: [CLI]: wandb sync with --append option fails to sync offline runs with the same id

Describe the bug

I run my experiments in the offline mode. That results in multiple wandb folders such as: ./wandb/offline-run-20230701_162008-giyrbe5p ./wandb/offline-run-20230705_154815-giyrbe5p which is expected. However, the second part of the run (newer logs) won’t get synced when running: wandb sync --sync-all --include-offline --include-synced --append

Since the second folder is just the continuation of the first one, how can I sync it into the same run?

Additional Files

No response

Environment

WandB version: 0.15.5

OS:

Python version: 3.8.12

Versions of relevant libraries:

Additional Context

No response

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Reactions: 2
  • Comments: 17 (3 by maintainers)

Most upvoted comments

Hey, @streethagore , our eng is actively working on this request - resuming a run from a checkpoint via the graphQL API. We’ll keep you updated on this thread once this is merged and deployed.

Hello @anmolmann , that still doesn’t solve the issue for offline runs (or am I missing something?)

Basically in my wandb dir I have for example:

wandb/offline-run-20250218_134501-f2yl6k90  # (first run)
wandb/offline-run-20250218_184624-f2yl6k90  # (second run)

the second most recent run being a rewind of the first from the most recent checkpoint (say 20k).

Syncing the first run works fine: Image

then calling wandb sync wandb/offline-run-20250218_184624-f2yl6m90 simply erases logs of the first one (so typically loss curves start at log step zero, and show an initial value for the training step of 20k (the rewound-from step): Image

Calling wandb sync wandb/offline-run-20250218_184624-f2yl6m90 --append or wandb sync --id f2yl6m90 --append wandb/offline-run-20250218_184624-f2yl6m90 does not change anything.

Running 0.19.6

Related/Duplicates?: #3841 #4624 #2423