singularity-hpc: Some biocontainer images fail to load

Describe the bug

$ module load quay.io/biocontainers/fastp
Lmod Warning:  MODULEPATH directory: "~/singularity-hpc/modules" has too many non-modulefiles (413). Please make sure that modulefiles are in their own directory and not mixed
in with non-modulefiles (e.g. source code)

Lmod has detected the following error:  Unable to load module because of error when evaluating modulefile:
     ~/singularity-hpc/modules/quay.io/biocontainers/fastp/0.23.2--h5f740d0_3/module.lua: [string "-- Lmod Module..."]:98: unfinished string near '"deb-list : gcc-8-base_8.3.0-6_amd64.deb'
     Please check the modulefile and especially if there is a line number specified in the above message 
While processing the following module(s):

The too many modulefiles warning also happens for tools that are working (trinity for example). But this one has a deb-list error and therefore does not work

To Reproduce Steps to reproduce the behavior: Install via:

shpc install quay.io/biocontainers/fastp

Expected behavior The tool loads without warning or errors.

Version of Singularity and Singularity Registry HPC Client

$ shpc --version
0.1.22
$ module --version

Modules based on Lua: Version 8.7.24  2023-05-04 15:12 -05:00
    by Robert McLay mclay@tacc.utexas.edu 
$ singularity --version
apptainer version 1.1.8

Here is a link to the generated fastp module.lua file: https://bin.disroot.org/?440b21e6dfca18ca#9JNBgP8Uuo65TYs27eckSy4AFqn9ZUFVPRsjLKR3gUV2

About this issue

Original URL
State: closed
Created a year ago
Comments: 43 (31 by maintainers)

Most upvoted comments

Is this something I can easily try out myself? Or do the module files need to be regenerated?

Probably yes, I can outline a strategy I have in mind, and you can evaluate how much support you would need. Suppose you have a directory path modules/ which contains all the modules + spurious files from the SHPC installation.

You could create a sibling directory wrappers/, where you would move all of the wrapper scripts from all bin/ subdirectories of modules/, preserving the same subdirectory tree; this can be automated using a bash script.

Then, look at line 70 of the singularity lua template: you would use a substitution tool such as grep to look into each modulefile for the line containing prepend_path, and then substitute the string pathJoin(moduleDir, "bin") with another general string that points to the wrappers/ tree.

Apologies, I have a few meetings starting, so I am unable to further detail how to implement point 2. (it might need a bit of Lua magic).

Please shout up for further support 😃

Mattias, also note that Vanessa has a PR open on point 1. above: https://github.com/singularityhub/singularity-hpc/pull/654

I am aiming to review it ASAP this week.

marcodelapierre on Aug 28, 2023

My pleasure Mattias.

And shout out to Audrey, our containers&modules&more bio-devops wizard!

marcodelapierre on Aug 28, 2023

Hi @mdehollander . Here we don’t serve the entire biocontainers repository through shpc. Only the ones that have been requested by users - a few dozens so far. So no performance issue.

muffato on Aug 23, 2023

I tried the strip-parens-templates branch and can confirm the module file work now:

$ module load quay.io/biocontainers/fastp
Lmod Warning:  MODULEPATH directory: "~/singularity-hpc/modules" has too many non-modulefiles (464). Please make sure that modulefiles are in their own directory and not mixed
in with non-modulefiles (e.g. source code)

$ which fastp
~/singularity-hpc/modules/quay.io/biocontainers/fastp/0.23.4--h5f740d0_0/bin/fastp

So thanks 🎉

The part that gave an error now looks like this:

whatis("busybox    : BusyBox v1.32.1 (2021-04-13 11:15:36 UTC) multi-call binary.")
whatis("deb-list    : gcc-8-base_8.3.0-6_amd64.deb, libc6_2.28-10_amd64.deb, libgcc1_1%3a8.3.0-6_amd64.deb, bash_5.0-4_amd64.deb, libc-bin_2.28-10_amd64.deb, libtinfo6_6.1+20181013-2+deb10u2_amd64.deb, ncurses-base_6.1+20181013-2+deb10u2_all.deb, base-files_10.3+deb10u9_amd64.deb")
whatis("glibc    : GNU C Library (Debian GLIBC 2.28-10) stable release version 2.28.")

There are indeed many files in the modules folder, and not starting with a . I haven’t installed many modules, but there are more than 600 files. Most files are in the bin folders. I don’t notice any slowness when loading a module, so indeed not a big issue. Happy that loading the biocontainer modules works now!

Edit: I did import modules from a /cvmfs mount, but but interrupted it as well, but still quite some bioconductor modules where imported as a module. When I remove them, the list is much shorter and there is no lmod warning anymore

mdehollander on Jun 7, 2023

Yours looks right! There’s no closing ") on both the first line and the last line. Not sure why that showed up in the original.

georgiastuart on Jun 5, 2023