snakemake: `ancient()` statements are arbitrarily ignored when using a workflow as a `module:`

Snakemake version Occurs in v9.1.1 (and older)

Describe the bug snakemake seems to arbitrarily ignore the ancient() setting in rule inputs when sub-workflows are called using the module: functionality.

Minimal example

In the case below, rule inputs with ancient statement within module_hydropower may be re-triggered, even if the file exists.

configfile: workflow.source_path("./test_config.yaml")

module module_hydropower:
    snakefile:
        "../../workflow/Snakefile"
    config:
        config["module_hydropower"]

use rule * from module_hydropower as module_hydropower_*

rule all:
    message:
        "Generate CF timeseries for a small country."
    default_target: True
    input:
        "results/by_shape_id/hydro_dam_cf.parquet",

Additional context This issue was identified in this PR https://github.com/calliope-project/module_hydropower/pull/1

Here is a failing example: module_hydropower.zip. You should be able to re-run our case by running the test workflow in tests/integration

This project uses data that requires a specific API key. To allow for CI tests (and to deter constant downloads) we use ancient in rules that would re-download data using the API, and include minimal test datasets in the integration test. Unfortunately, snakemake is ignoring these pre-existing files during github’s CI testing, leading to the failure.

About this issue

  • Original URL
  • State: open
  • Created 2 years ago
  • Reactions: 3
  • Comments: 15 (5 by maintainers)

Most upvoted comments

@Hocnonsense to prevent changes from impacting the debugging of this issue, I created a tag

https://github.com/calliope-project/module_hydropower/tree/snakemake-ancient-issue

I’ll try to see if specifying ancient via the CLI results in the same problem.

update: providing ancient through the CLI via --consider-ancient does not get around this issue 🥲🥲🥲

That’s right. But the problem is that the rule powerplants_adjust_location wasn’t included into the DAG. If output.adjusted_powerplants="results/adjusted_powerplants.parquet" is needed, the name will be print to shell.

I repeated the steps above, but using snakemake -np.

The command shows that ancient is not respected:

host: fedora
Building DAG of jobs...
Job stats:
job                                                   count
--------------------------------------------------  -------
module_hydropower_basins_combine_continents               1
module_hydropower_basins_extract_pfafstetter_level        9
module_hydropower_download_basin                          9
module_hydropower_download_cutout                         1
total                                                    20

related to this rule:

rule all:
    message:
        "Generate CF timeseries for a small country."
    default_target: True
    input:
        "results/by_shape_id/hydro_dam_cf.parquet",
        "results/by_shape_id/hydro_run_of_river_cf.parquet",

, while

and these is the output:

Building DAG of jobs...
Job stats:
job                                               count
----------------------------------------------  -------
all                                                   1
module_hydropower_powerplants_adjust_location         1
module_hydropower_powerplants_get_cf_per_shape        1
module_hydropower_powerplants_get_inflow_m3           1
module_hydropower_powerplants_get_inflow_mwh          1
total                                                 5

related to this rule:

rule all:
    message:
        "Generate CF timeseries for a small country."
    default_target: True
    input:
        "results/by_shape_id/hydro_dam_cf.parquet",

The former one don’t need resources/automatic/hydrobasin_global_01.parquet but a series files: resources/automatic/hydrobasin_af_01.parquet resources/automatic/hydrobasin_ar_01.parquet resources/automatic/hydrobasin_as_01.parquet resources/automatic/hydrobasin_au_01.parquet resources/automatic/hydrobasin_eu_01.parquet resources/automatic/hydrobasin_gr_01.parquet resources/automatic/hydrobasin_na_01.parquet resources/automatic/hydrobasin_sa_01.parquet resources/automatic/hydrobasin_si_01.parquet, while the latter one need resources/automatic/hydrobasin_global_01.parquet.