pyxis: pyxis doesn't use environment variables defined in enroot .env files

Hi!

Let’s say you have an enroot .env file that references an environment variable:

$ cat /etc/enroot/environ.d/test.env
TEST_VAR=${TEST_VAR}

When using enroot directly, the environment variable in the container is set to what it’s been set on the host:

$ enroot start debian sh -c 'echo TEST_VAR=${TEST_VAR}'
TEST_VAR=
$ TEST_VAR=foobar enroot start debian sh -c 'echo TEST_VAR=${TEST_VAR}'
TEST_VAR=foobar

With Slurm, when using srun, the environment variable is also correctly passed to the compute node:

$ TEST_VAR=foobar srun sh -c 'echo TEST_VAR=${TEST_VAR}'
TEST_VAR=foobar

And when running enroot through srun, things work too:

$ TEST_VAR=foobar srun enroot start debian sh -c 'echo TEST_VAR=${TEST_VAR}'
TEST_VAR=foobar

But when using pyxis, it looks like the enroot .env file is ignored:

$ TEST_VAR=foobar srun --container-image=debian sh -c 'echo TEST_VAR=${TEST_VAR}'
pyxis: importing docker image ...
TEST_VAR=

I assume this is because when pyxis calls enroot, it is in a context where TEST_VAR is not defined, so it does parse the /etc/enroot/environ.d/test.env file, but substitutes an empty value to the variable.

I wanted to verify that by using a default value fallback in the variable definition, but it looks like enroot may be parsing this literally 😄. So it doesn’t prove the “undefined environment variable” hypothesis, but it does indicate that the /etc/enroot/environ.d/test.env file is used:

$ cat /etc/enroot/environ.d/test.env
TEST_VAR=${TEST_VAR:-fallback}
$ TEST_VAR=foobar srun --container-image=debian sh -c 'echo TEST_VAR=${TEST_VAR}'
pyxis: importing docker image ...
TEST_VAR=${TEST_VAR:-fallback}

Use case

We typically define a number of environment variables to point to various storage locations for our users. Those paths depend on a number of factors and are different for each user (exactly like $HOME). We would like to be able to automatically pass those environment variable to containers executed via pyxis/enroot and have the file systems mounted as well.

The goal is to have something like this:

$ cat /etc/enroot/environ.d/scratch.env
SCRATCH=${SCRATCH}

$ cat /etc/enroot/mounts.d/scratch.fstab
${SCRATCH}              ${SCRATCH}        none    x-create=dir,bind,rw,nosuid

and to have the user-specific $SCRATCH folder automatically mounted in containers.

Although all of that works well when containers are started by enroot directly, it doesn’t work when they’re started through pyxis, as the $SCRATCH environment variable is not defined when the container is initialized.