snakemake: `ancient()` statements are arbitrarily ignored when using a workflow as a `module:`
Snakemake version Occurs in v9.1.1 (and older)
Describe the bug
snakemake seems to arbitrarily ignore the ancient() setting in rule inputs when sub-workflows are called using the module: functionality.
Minimal example
In the case below, rule inputs with ancient statement within module_hydropower may be re-triggered, even if the file exists.
configfile: workflow.source_path("./test_config.yaml")
module module_hydropower:
snakefile:
"../../workflow/Snakefile"
config:
config["module_hydropower"]
use rule * from module_hydropower as module_hydropower_*
rule all:
message:
"Generate CF timeseries for a small country."
default_target: True
input:
"results/by_shape_id/hydro_dam_cf.parquet",
Additional context This issue was identified in this PR https://github.com/calliope-project/module_hydropower/pull/1
Here is a failing example: module_hydropower.zip. You should be able to re-run our case by running the test workflow in tests/integration
This project uses data that requires a specific API key. To allow for CI tests (and to deter constant downloads) we use ancient in rules that would re-download data using the API, and include minimal test datasets in the integration test. Unfortunately, snakemake is ignoring these pre-existing files during github’s CI testing, leading to the failure.
About this issue
- Original URL
- State: open
- Created 2 years ago
- Reactions: 3
- Comments: 15 (5 by maintainers)
@Hocnonsense to prevent changes from impacting the debugging of this issue, I created a tag
https://github.com/calliope-project/module_hydropower/tree/snakemake-ancient-issue
I’ll try to see if specifying
ancientvia the CLI results in the same problem.update: providing
ancientthrough the CLI via--consider-ancientdoes not get around this issue 🥲🥲🥲That’s right. But the problem is that the rule
powerplants_adjust_locationwasn’t included into the DAG. Ifoutput.adjusted_powerplants="results/adjusted_powerplants.parquet"is needed, the name will be print to shell.related to this rule:
, while
related to this rule:
The former one don’t need
resources/automatic/hydrobasin_global_01.parquetbut a series files: resources/automatic/hydrobasin_af_01.parquet resources/automatic/hydrobasin_ar_01.parquet resources/automatic/hydrobasin_as_01.parquet resources/automatic/hydrobasin_au_01.parquet resources/automatic/hydrobasin_eu_01.parquet resources/automatic/hydrobasin_gr_01.parquet resources/automatic/hydrobasin_na_01.parquet resources/automatic/hydrobasin_sa_01.parquet resources/automatic/hydrobasin_si_01.parquet, while the latter one needresources/automatic/hydrobasin_global_01.parquet.