snakemake: `mem_mb` and `mem_mib` do not match `mem` when `mem` is specified with unit.
Snakemake version
Version 8.9.0
Describe the bug
When mem is specified with a unit, the mem_mb and mem_mib resources do not make sense. For example, when the request is mem="2G", mem_mb is 1907 and mem_mib is 954.
If 2G is supposed to be gigabyes, then mem_mb should be 2000 and mem_mib should be 1907. If 2G is supposed to be gibibytes, then mem_mb should be 2147 and mem_mib should be 2048.
Logs
[Fri Mar 22 14:02:58 2024]
rule test:
output: test.txt
jobid: 1
reason: Missing output files: test.txt
threads: 6
resources: mem_mb=1907, mem_mib=954, disk_mb=1000, disk_mib=954, tmpdir=<TBD>, lsf_project=acc_LOAD, lsf_queue=premium, runtime=2880, mem=2G
Minimal example
Here is a test rule:
rule test:
output: "test.txt"
threads: 6
resources:
runtime="2d",
mem="2G"
shell: ".echo test > {output}"
Additional context
This may be partially related to #2695
I am using the lsf executor plugin, and I am trying to troubleshoot if this is partially due to something I did in the plugin (mem_mib is not shown when not using the cluster and mem_mb is what mem_mib should be if 2G is gigabytes), but I don’t remember doing anything that could change the printed resources.
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Reactions: 1
- Comments: 24 (22 by maintainers)
Commits related to this issue
- fix: `mem` and `disk` inference fixes (#3040) <!--Add a description of your PR here--> This fixes #2774. 1. When inferring memory or disk from a string, *_mb now divides the bytes by 1e6 (SI) ... — committed to snakemake/snakemake by BEFH 2 years ago
- fix: `mem` and `disk` inference fixes (#3040) <!--Add a description of your PR here--> This fixes #2774. 1. When inferring memory or disk from a string, *_mb now divides the bytes by 1e6 (SI) ... — committed to vandalt/snakemake by BEFH 2 years ago
Thanks @hermidalc, we’ll see if the PR is merged into snakemake/snakemake:main
Everything that effects mem_mb or mem_mib effects
disk_mbordisk_mib. The same functions are at play.Thanks. Please make sure you work on the mem_disk_fix branch in your fork.
I think I have fixed everything including the update of the
mibvalues, but I have not run any tests.I switched
roundtomath.ceil, so your example should be 4769. But once again, I have not tested. The changes are in 3 (!!!) different files, unfortunately.I don’t know why this is so difficult to fix. I’ll take a look at the relevant code.
It’s probably here:
https://github.com/snakemake/snakemake/blob/main/snakemake/resources.py#L632-L644
Or it could be an upstream bug in
humanfriendly.I am bumping this issue since it is still present and is rather severe.
I just checked my code in snakemake-executor-plugin-lsf and I am not updating the resources, just accessing them. I am basically doing the same thing as the Slurm executor.