snakemake: Jobs updated by checkpoint are not getting run

Snakemake version works:

• 5.20.0
• 5.25.0
• 5.26.0
• 5.26.1

fails:

• HEAD (5.27.4+1.g4a63a269)
• 5.27.4
• 5.27.0

Describe the bug The test run for our DTR-phage-pipeline has stopped working recently. It has a few checkpoints and 160+ total jobs, so it’s not the simplest test case, but it did work in earlier version.

Instead of completing in one go, it stops after each checkpoint. DEbugging output indicates that the downstream jobs are getting updated, but somehow it stops anyway. Restarting snakemake continues where it left off.

Logs Here is the end of the output from the first run:

checkpoint generate_bins_and_stats:
    input: test/test-output/25m/1D/v2/kmer_binning/bin_membership.tsv
    output: test/test-output/25m/1D/v2/kmer_binning/bin_stats.csv, test/test-output/25m/1D/v2/kmer_binning/bins
    jobid: 11
Downstream jobs will be updated after completion.

[Thu Nov  5 14:15:24 2020]
rule plot_umap_freq_map_bins:
    input: test/test-output/25m/1D/v2/kmer_binning/bin_membership.tsv
    output: test/test-output/25m/1D/v2/kmer_binning/kmer_comp.umap.bins.png
    jobid: 9
                                                                                                                                                                                                                 
python /mnt/data0/jmeppley/projects/DTR/TBD/DTR-phage-pipeline/scripts/plot_umap_with_hdbscan_labels.py -o test/test-output/25m/1D/v2/kmer_binning/kmer_comp.umap.bins.png -s 7 -a 0.2 test/test-output/25m/1D/v2/kmer_binning/bin_membership.tsv
No kaiju DB found in /media/sfodata/databases/kaiju/nr_euk.
Skipping Kaiju
No kaiju DB found in /media/sfodata/databases/kaiju/nr_euk.
Skipping Kaiju
Job counts:
        count   jobs
        1       generate_bins_and_stats
        1
Activating conda environment: /mnt/data0/jmeppley/projects/DTR/TBD/DTR-phage-pipeline/.snakemake/conda/53e51836
Updating job 12 (combine_bin_cluster_read_info).
Replace combine_bin_cluster_read_info with dynamic branch combine_bin_cluster_read_info
updating depending job all
Updating job 13 (create_bin_cluster_read_lists).
Replace create_bin_cluster_read_lists with dynamic branch create_bin_cluster_read_lists
updating depending job combine_all_polished_ref_reads
updating depending job aggregate_prodigal_statistics
updating depending job combine_bin_cluster_strand_counts_into_table
updating depending job combine_dtr_aligns
updating depending job all
Updating job 21 (build_bin_cluster_summary_table).
Replace build_bin_cluster_summary_table with dynamic branch build_bin_cluster_summary_table
updating depending job combine_all_draft_stats
Updating job 0 (all).
Replace all with dynamic branch all
[Thu Nov  5 14:15:25 2020]
Finished job 11.
11 of 86 steps (13%) done
[Thu Nov  5 14:15:26 2020]
Finished job 8.
12 of 86 steps (14%) done
Mapping 869 data points...
Saving test/test-output/25m/1D/v2/kmer_binning/kmer_comp.umap.bins.notext.png
Mapping 869 data points...
[Thu Nov  5 14:15:26 2020]
Finished job 7.
13 of 86 steps (15%) done
bin -1 (66 reads)
bin 0 (315 reads)
bin 1 (79 reads)
bin 2 (45 reads)
bin 3 (29 reads)
bin 4 (12 reads)
bin 5 (27 reads)
bin 6 (15 reads)
bin 7 (42 reads)
bin 8 (42 reads)
bin 9 (20 reads)
bin 10 (13 reads)
bin 11 (36 reads)
bin 12 (28 reads)
bin 13 (25 reads)
bin 14 (16 reads)
bin 15 (18 reads)
bin 16 (29 reads)
bin 17 (12 reads)
Saving test/test-output/25m/1D/v2/kmer_binning/kmer_comp.umap.bins.png
Saving test/test-output/25m/1D/v2/kmer_binning/kmer_comp.umap.bins.nolabels.png
[Thu Nov  5 14:15:31 2020]
Finished job 9.
14 of 86 steps (16%) done
Complete log: /mnt/data0/jmeppley/projects/DTR/TBD/DTR-phage-pipeline/.snakemake/log/2020-11-05T141437.844467.snakemake.log
unlocking
removing lock
removing lock
removed all locks

Despite reporting 16% done, it just stops.

Minimal example I’m working on a simpler example. For now it should be easy to reproduce by cloning the linked repo:

$ git clone https://github.com/nanoporetech/DTR-phage-pipeline
$ cd DTR-phage-pipeline
$ mamba create -p ./env -f ./environment.yaml
$ conda activate ./env
$ snakemake --use-conda --conda-frontend mamba -j 20 -p

Additional context Oddly, if the first few rules (before the checkpoint) have already been completed, then the jobs after the checkpoint get run as they should. The erroneous stopping only occurs when starting with no output files.