wandb: [CLI]: wandb sync with --append option fails to sync offline runs with the same id
Describe the bug
I run my experiments in the offline mode. That results in multiple wandb folders such as:
./wandb/offline-run-20230701_162008-giyrbe5p
./wandb/offline-run-20230705_154815-giyrbe5p
which is expected. However, the second part of the run (newer logs) won’t get synced when running:
wandb sync --sync-all --include-offline --include-synced --append
Since the second folder is just the continuation of the first one, how can I sync it into the same run?
Additional Files
No response
Environment
WandB version: 0.15.5
OS:
Python version: 3.8.12
Versions of relevant libraries:
Additional Context
No response
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Reactions: 2
- Comments: 17 (3 by maintainers)
Hey, @streethagore , our eng is actively working on this request - resuming a run from a checkpoint via the graphQL API. We’ll keep you updated on this thread once this is merged and deployed.
Hello @anmolmann , that still doesn’t solve the issue for offline runs (or am I missing something?)
Basically in my
wandbdir I have for example:the second most recent run being a rewind of the first from the most recent checkpoint (say 20k).
Syncing the first run works fine:
then calling
wandb sync wandb/offline-run-20250218_184624-f2yl6m90simply erases logs of the first one (so typically loss curves start at log step zero, and show an initial value for the training step of 20k (the rewound-from step):Calling
wandb sync wandb/offline-run-20250218_184624-f2yl6m90 --appendorwandb sync --id f2yl6m90 --append wandb/offline-run-20250218_184624-f2yl6m90does not change anything.Running
0.19.6Related/Duplicates?: #3841 #4624 #2423