vscode-dvc: Live plots do not automatically update on Codespaces

Looks like something weird is happening with our watchers on Codedspaces. Local dev version runs fine for me.

How to reproduce

I’ve tried both - GPU and non-GPU setup. Two repositories: Get Started and Hackathon.

I’m getting only the first dot for the first epoch and then only at the very end it updates everything:

https://user-images.githubusercontent.com/3659196/218223477-c201c726-d819-4e23-bef5-13e7486cbdb5.mov

https://user-images.githubusercontent.com/3659196/218223521-e099b00e-6e8f-4b2a-a043-61f5f35c2e20.mov

Environment

Extension version: 0.6.4 DVC Version:

DVC version: 2.44.0 (deb)
-------------------------
Platform: Python 3.10.8 on Linux-5.4.0-1101-azure-x86_64-with-glibc2.31
Subprojects:

Supports:
        azure (adlfs = 2023.1.0, knack = 0.10.1, azure-identity = 1.12.0),
        gdrive (pydrive2 = 1.15.0),
        gs (gcsfs = 2023.1.0),
        hdfs (fsspec = 2023.1.0, pyarrow = 11.0.0),
        http (aiohttp = 3.8.3, aiohttp-retry = 2.8.3),
        https (aiohttp = 3.8.3, aiohttp-retry = 2.8.3),
        oss (ossfs = 2021.8.0),
        s3 (s3fs = 2023.1.0, boto3 = 1.24.59),
        ssh (sshfs = 2023.1.0),
        webdav (webdav4 = 0.9.8),
        webdavs (webdav4 = 0.9.8),
        webhdfs (fsspec = 2023.1.0)
Cache types: hardlink, symlink
Cache directory: ext4 on /dev/loop0
Caches: local
Remotes: https
Workspace directory: ext4 on /dev/loop0
Repo: dvc, git

dvclive==2.0.2

About this issue

Original URL
State: closed
Created a year ago
Comments: 18 (17 by maintainers)

Most upvoted comments

For the record here. Got a response in community/community:

mattseddon on Mar 1, 2023

@alex000kim @daavoo do you also see this behavior?

Yes, I saw this today in https://github.com/iterative/example-get-started-experiments My workaround when demoing is to keep pressing the “Refresh” button.

alex000kim on Feb 16, 2023

There seems to be an issue with watching heavily-nested folder structures on Codespaces. Below is a demo of the depth where events stop getting generated:

FWIW I have checked to see if this was a Linux issue by creating the devcontainer from example-get-started-experiments locally with Docker. Training performance was TERRIBLE but I could not recreate the issue. All events were created as expected.

mattseddon on Feb 22, 2023

metrics.json is named something else. Users not working with DVCLive.

These 2 are the same. DVCLive has metrics.json hardcoded as name, users can’t change it

daavoo on Feb 21, 2023

Thanks @daavoo as a patch we can rely on metrics.json being updated 🙏🏻.

I can think of a few cases when this won’t work:

metrics.json is still too deep/events are still dropped. E.g DVCLive dir is set to model/results/stage/train.
metrics.json is named something else.
Users not working with DVCLive.

mattseddon on Feb 20, 2023

Not a viable option but partly confirms the above theory. In this demo I have patched the DVCLiveCallback to touch the main dvc.yaml in the after_epoch function:

dvclive/metrics.json always gets updated on next_step

daavoo on Feb 20, 2023