dvc: dvc exp push: pushing/pulling cache fails if params.yaml not in workspace

Bug Report

Description

If there is no params.yaml in the workspace and interpolation is used in dvc.yaml, dvc exp push will fail to push the cache for the experiments.

Reproduce

Fork and clone https://github.com/dberenbaum/params-test and then run:

dvc exp run
rm params.yaml
dvc exp push origin

You will see an error like ERROR: failed to push cache: failed to parse 'stages.params_test.cmd' in 'dvc.yaml': Could not find 'params'.

Expected

Since params.yaml is present in the experiment, it should be possible to push it. Can we interpolate using the state of the experiment rather than the workspace?

About this issue

Original URL
State: closed
Created a year ago
Reactions: 2
Comments: 15 (9 by maintainers)

Most upvoted comments

Sorry, it happens that the problem reproduces just with DVC 2 and vanishes in DVC 3. It outputs the error message like this ERROR: failed to collect 'workspace' - failed to parse 'stages.main.cmd' in 'dvc.yaml': Could not find 'parameters.parameter_name', but it does not prevent normal work of dvc pull/push. Probably this issue could be closed, thank you very much!

Danila89 on Dec 1, 2023

@Danila89 I don’t have any updates unfortunately. Have you tried adding params.yaml to your workspace? I think it should work once you have a params.yaml file with the expected parameters.

It will, but it is inconvenient. Different experiments have different parameters at params.yaml (I mean not just different values, but different parameter names as well). And for dvc exp pull or dvc gc I have to get the proper params.yaml first. Using garbage collection when many experiments are deleted becomes literally impossible(

Danila89 on Nov 27, 2023

This can also happen during pull, and not only on the first run, since parameters may change frequently (esp with hydra), and any missing parameter in the workspace causes a failure. See https://discord.com/channels/485586884165107732/485596304961962003/1137288948914339860.

dberenbaum on Aug 5, 2023