wandb: [App] wandb sync for offline runs doesn't upload all data until run is finished.

I faced the same issue.

I run the code on the Slurm platform with time limitations. After the dead time comes, the whole run is crushed immediately. After that, I run wandb sync ./wandb/offline-run-xxx and the figures of metrics are shown in GUI normally, however, the table contains nothing about either config parameters and metrics.

The strange thing is that I also run wandb sync --view --verbose wandb/offline-run-xxx.wandb and I found the config parameters are actually already contained in the xxx.wandb file. But the files/config.yaml file only contains the system parameters like software version information without the needed experiments’ parameters.

So I think this issue might be fixed by allowing upload the config parameters saved in xxx.wandb file to the GUI table? It is so inconvenient to check the xxx.wandb file for obtaining the config parameters after offline runs crushed.

HI @yihong-chen, currently the ticket is in our queue to do, but I don’t have a timeline for it yet

Hi @vanpelt. I have created a minimal example to explain the issue. I used signal.SIGTERM to mimic slurm behavior when the time limit is reached. I ran the following code two times, the second time with os.environ["WANDB_MODE"] = "offline" commented.

import os
import signal
import argparse
import wandb
import time
os.environ["WANDB_MODE"] = "offline"

parser = argparse.ArgumentParser()
parser.add_argument('--seed', type=int, default=0)
parser.add_argument('--epochs', default=100, type=int)
args = parser.parse_args()

wandbid = wandb.util.generate_id()
print(wandbid)
wandb.init(project="test_project", id=wandbid, resume="allow", dir='./')
wandb.config.update(args, allow_val_change=True)

wandb.define_metric('Loss', summary="min")
for ep in range(args.epochs):
    wandb.log({'Loss': 0.0}, step=ep)
    time.sleep(1)
    print(ep)
    if ep==30: os.kill(os.getpgid(0), signal.SIGTERM)

After syncing an offline run the Table page is as follows.

The output of the tree command is as follows.

wandb
├── debug-cli.log
├── debug-internal.log -> offline-run-20220328_093814-1q6a1n1p/logs/debug-internal.log
├── debug.log -> offline-run-20220328_093814-1q6a1n1p/logs/debug.log
├── latest-run -> offline-run-20220328_093814-1q6a1n1p
├── offline-run-20220328_093814-1q6a1n1p
│   ├── files
│   │   ├── config.yaml
│   │   ├── output.log
│   │   ├── requirements.txt
│   │   └── wandb-metadata.json
│   ├── logs
│   │   ├── debug-internal.log
│   │   └── debug.log
│   ├── run-1q6a1n1p.wandb
│   ├── tmp
│   │   └── code
│   └── wandb
└── run-20220328_093725-20tgsy55
    ├── files
    │   ├── code
    │   │   └── debug_wandb
    │   │       └── debug_wandb.py
    │   ├── config.yaml
    │   ├── diff.patch
    │   ├── output.log
    │   ├── requirements.txt
    │   ├── wandb-metadata.json
    │   └── wandb-summary.json
    ├── logs
    │   ├── debug-internal.log
    │   └── debug.log
    ├── run-20tgsy55.wandb
    └── tmp
        └── code

Note that the arguments seed and epochs appear correctly in both wandb/run-20220328_093725-20tgsy55/run-20tgsy55.wandb and wandb/offline-run-20220328_093814-1q6a1n1p/run-1q6a1n1p.wandb.

Finally, I am using wandb==0.12.9.

Thank you.

I am also a computecanada user and run codes on slurm platform. Slurm terminates a job once we exhaust the time limit. Oftentimes, this happens before the training is complete or wandb.finish() is called. Since the compute node doesn’t have an internet connection, we use an offline mode for wandb and use wandb sync to sync an incomplete offline run. We cannot see config in wandb overview and Table tabs in this particular case. However, the config is stored in the run-xxxxxxxx.wandb file.