snakemake: `mem_mb` and `mem_mib` do not match `mem` when `mem` is specified with unit.

Snakemake version

Version 8.9.0

Describe the bug

When mem is specified with a unit, the mem_mb and mem_mib resources do not make sense. For example, when the request is mem="2G", mem_mb is 1907 and mem_mib is 954.

If 2G is supposed to be gigabyes, then mem_mb should be 2000 and mem_mib should be 1907. If 2G is supposed to be gibibytes, then mem_mb should be 2147 and mem_mib should be 2048.

Logs

[Fri Mar 22 14:02:58 2024]
rule test:
    output: test.txt
    jobid: 1
    reason: Missing output files: test.txt
    threads: 6
    resources: mem_mb=1907, mem_mib=954, disk_mb=1000, disk_mib=954, tmpdir=<TBD>, lsf_project=acc_LOAD, lsf_queue=premium, runtime=2880, mem=2G

Minimal example

Here is a test rule:

rule test:
  output: "test.txt"
  threads: 6
  resources:
    runtime="2d",
    mem="2G"
  shell: ".echo test > {output}"

Additional context

This may be partially related to #2695 I am using the lsf executor plugin, and I am trying to troubleshoot if this is partially due to something I did in the plugin (mem_mib is not shown when not using the cluster and mem_mb is what mem_mib should be if 2G is gigabytes), but I don’t remember doing anything that could change the printed resources.

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Reactions: 1
  • Comments: 24 (22 by maintainers)

Commits related to this issue

Most upvoted comments

Thanks @hermidalc, we’ll see if the PR is merged into snakemake/snakemake:main

Hey did this effect disk_mib in addition to mem_mib?

Everything that effects mem_mb or mem_mib effects disk_mb or disk_mib. The same functions are at play.

Thanks. Please make sure you work on the mem_disk_fix branch in your fork.

I think I have fixed everything including the update of the mib values, but I have not run any tests.

I switched round to math.ceil, so your example should be 4769. But once again, I have not tested. The changes are in 3 (!!!) different files, unfortunately.

I don’t know why this is so difficult to fix. I’ll take a look at the relevant code.

It’s probably here:

https://github.com/snakemake/snakemake/blob/main/snakemake/resources.py#L632-L644

Or it could be an upstream bug in humanfriendly.

I am bumping this issue since it is still present and is rather severe.

I just checked my code in snakemake-executor-plugin-lsf and I am not updating the resources, just accessing them. I am basically doing the same thing as the Slurm executor.