dvc: import: does not work with repositories using params
Importing some data from parametrized repository does not work.
reproduction:
#!/bin/bash
set -ex
pushd $TMPDIR
wsp=test_wspace
imp_repo=import_repo
rep=test_repo
rm -rf $wsp && mkdir $wsp && pushd $wsp
base=$(pwd)
mkdir $base/$imp_repo
mkdir $base/$rep
pushd $base/$imp_repo
git init
dvc init
echo data >> data
echo "m: 1" >> params.yaml
dvc add data
dvc run -d data -p m -o out -n train cp data out
git add -A
git commit -am "init"
pushd $base/$rep
git init
dvc init
dvc import $base/$imp_repo out -o my_out -v
Result:
dvc.parsing.ResolveError: failed to parse 'vars' in 'dvc.yaml': 'params.yaml' does not exist
On the first sight the culprit seems to be dvc/repo/stage.py::_maybe_collect_from_dvc_yaml
which passes dvc.yaml
to StageLoad.load_all
, while it seems that it should be a path to the file.
https://github.com/iterative/dvc/blob/d882e21b9b013f69abff960a8bad62d822024b59/dvc/repo/stage.py#L66
Kudos to @oslobowl for discovering the problem.
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Reactions: 7
- Comments: 24 (12 by maintainers)
@mnrozhkov downgrading to
2.8.1
seems to help, we are trying to pinpoint the exact change introducing the bug.Looks like the issue is how we use
wdir
instead of git repoβs root, which happens here: https://github.com/iterative/dvc/blob/d882e21b9b013f69abff960a8bad62d822024b59/dvc/parsing/context.py#L402This is definitely a serious bug, and may need serious consideration on how to fix on filesystem side. Assigning it myself.
@efiop Hi again π So, a colleague of mine figured out that we forgot to use the --upgrade flag in the pip install you posted above. Now everything works π Thank you very much for helping! Have a great weekend!
@efiop Thanks for the explanation Ruslan π we are cheering for you and we are very grateful for the great support we get with issues π
I tried what you said, and I still get the same error unfortunately:
@efiop Yes, sorry. Using 2.11.0 I just get another error: ERROR: failed update data - Failed to clone repo βhttps://github.com/me/my_pipeline.gitβ to β/tmp/tmpb9j9yyrudvc-cloneβ
bisect
points me to e8e1c76504895f846ef5724c02f5414f4a251f7f as being the culprit.