wandb: [Bug]: Core service dropping data, sync won't recover

Describe the bug

Hi. I was training a run, and half way through I saw the print out about the core service. So I added the line wandb.require("core") and restarted training. It resumed training for a few epochs, it was logging and everything seemed fine.

This morning I came in and saw the run was still going locally, but had stopped syncing to Wandb. Weird. So I stopped the run, updated Wandb from 0.17.X to 0.18, and manually sync’ed all the run folders. The sync said it worked, but the data is not showing up in the UI.

Can you help me figure out if there’s a way to sync the missing data, or figure out why it’s not working? I’ve attached the debug logs, although I’m not sure if they append everything everytime. The run is here. Thank you for your help.

debug-cli.cmccarth.log debug.log debug-internal.log

$ wandb --version
wandb, version 0.18.0
$ python --version
Python 3.10.14
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 22.04.4 LTS
Release:        22.04
Codename:       jammy

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Reactions: 1
  • Comments: 15 (7 by maintainers)

Most upvoted comments

Let’s keep the issue open a little longer, in case anyone else comes across this behavior.

Got it, thank you for the detailed explanation and for pushing a partial fix! At this point I’ll probably freeze my Wandb version to <0.18 until after my paper deadline, but I’ll update to 0.18.2 after the deadline and let you know if I run into similar issues.

Thanks again!