dvc: `dvc plots diff` gives 'ERROR: unexpected error'

Bug Report

Please provide information about your setup

I’m getting failed to read 'my_project/data/metrics/metrics.json' on 'master' when I run dvc metrics diff master --show-md -vvv in a clean repo/on Github Actions, and it ends up failing with ERROR: unexpected error - 'data'. However, when I run it locally it works fine. I’ve tried dvc push -a -d --run-cache, as well as using an older version of DVC to see if that changes anything. I’m not sure what the next step is in debugging this anymore, nor what the problem is 😕 @steffansluis

https://discordapp.com/channels/485586884165107732/485596304961962003/753655147149787137

@pared#9484 I suddenly don’t seem to be able to reproduce the error with dvc metrics either locally or on GH actions, it now breaks both locally and remote on dvc plots diff ... with the same error: ERROR: unexpected error - 'data' @steffansluis

https://discordapp.com/channels/485586884165107732/485596304961962003/753722495147704370

Output of dvc version:

$ dvc version
DVC version: 1.6.6 (pip)
---------------------------------
Platform: Python 3.8.2 on Linux-5.4.0-7634-generic-x86_64-with-glibc2.29
Supports: azure, http, https, ssh
Cache types: hardlink, symlink
Cache directory: ext4 on /dev/mapper/data-root
Workspace directory: ext4 on /dev/mapper/data-root
Repo: dvc, git

Additional Information (if any):

As reproduced by @pared: https://discordapp.com/channels/485586884165107732/485596304961962003/753722565134123118

#!/bin/bash

rm -rf repo storage origin copy
main=$(pwd)

mkdir repo remote origin

pushd origin
git init --quiet --bare
popd

pushd repo
git init --quiet
git remote add origin $main/origin

dvc init --quiet
dvc remote add -d str $main/storage
echo -e "import json\nwith open('data', 'r') as fd:\n  val=int(fd.read())\nwith open('metric.json', 'w') as fd:\n  json.dump({'m':val}, fd)" > code.py

git add -A 
git commit -m "initial"

echo 1 >> data

dvc run -d data --plots metric.json -n run python code.py

git add -A
git commit -m "first run"
git push origin master
dvc push
dvc push --run-cache

popd

git clone origin copy

pushd copy

git checkout -b branch

echo -e "import json\nwith open('data', 'r') as fd:\n  val=int(fd.read())\nwith open('metric.json', 'w') as fd:\n  json.dump({'m':val+1}, fd)" > code.py

dvc repro run
dvc plots diff master

About this issue

Original URL
State: closed
Created 4 years ago
Reactions: 4
Comments: 24 (21 by maintainers)

Most upvoted comments

@alexdmiller @steffansluis seems like we found root cause for the unexpected error issue, it should be fixed in #4762. Can I ask you to check whether it still exists after 1.9.1?

pared on Oct 28, 2020

@pared I spoke to soon 😞

git fetch --prune
dvc pull --run-cache

fixes the error, but it generates a report that doesn’t take the master branch into account, so I end up with all my metrics being labeled as new, no diffs, and a confusion matrix that only includes the branch that my PR is about. I have reverted back to my old workaround for now.

steffansluis on Oct 14, 2020

@pared It works with dvc --version: 1.8.1+15b1e7 and

git fetch --prune
dvc pull --run-cache

steffansluis on Oct 7, 2020

@pared sorry for the delay in my response. Here is the verbose output of the command. Note that since I last posted, I changed the name of the plot to classification-plot.csv.

$ dvc plots show metrics/classification-plot.csv -v
2020-10-05 11:12:03,909 DEBUG: Check for update is enabled.
2020-10-05 11:12:04,059 ERROR: unexpected error - 'metrics/classification-plot.csv'
------------------------------------------------------------
Traceback (most recent call last):
  File "dvc/main.py", line 76, in main
  File "dvc/command/plots.py", line 54, in run
  File "dvc/command/plots.py", line 84, in _func
  File "dvc/repo/plots/__init__.py", line 88, in show
  File "dvc/repo/plots/__init__.py", line 88, in <genexpr>
KeyError: 'metrics/classification-plot.csv'
------------------------------------------------------------

Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!
DVC version: 1.7.9 (osxpkg)
---------------------------------
Platform: Python 3.7.9 on Darwin-19.5.0-x86_64-i386-64bit
Supports: All remotes
Cache types: reflink, hardlink, symlink
Cache directory: apfs on /dev/disk1s1
Workspace directory: apfs on /dev/disk1s1
Repo: dvc, git

And just to remind, if I run without specifying a target, it works:

$ dvc plots show
file:///Users/miller/Projects/dvc-project/plots.html

It seems like I’m getting the target name wrong. I’ve tried many different target names, but I get the same error. Here is the relevant portion of my dvc.yaml file:

  train-classifier:
    cmd: python3 train/train_classifier.py
    deps:
    - ../common
    - train/train_classifier.py
    outs:
    - models/classification/classification-jit.pth
    metrics:
    - metrics/classification-summary.json
    plots:
    - metrics/classification-plot.csv:
        x: epoch
        x_label: Epoch

alexdmiller on Oct 5, 2020

@efiop That seems to only pull the cache for the current branch. When running this on CI, the example provided in the README failed and I had to modify it to the following to get it to work:

BRANCH=$(git branch --show-current)

git fetch --prune
git checkout master
dvc pull --run-cache

git checkout $BRANCH
dvc pull --run-cache

Note after edit: I did not test the example 1:1, I’m not actually running dvc repro on CI, however I would expect that does not affect the situation presented here.

steffansluis on Sep 22, 2020