super-linter: Parallel cannot identify the default shell

Is there an existing issue for this?

I have searched the existing issues

Current Behavior

Running super-linter images locally fails with multiple error message /bin/sh: BuildFileArrays: not found.

x-ref: https://github.com/super-linter/super-linter/issues/5335#issuecomment-1970164621

Expected Behavior

The parallel command should not fail in this way and the /bin/sh: BuildFileArrays: not found. should not occur.

Super-Linter version

Tested with slim-6.2.0 and slim-latest

Relevant log output

Full log.txt

{
    "Seq": 42,
    "Host": ":",
    "Starttime": 1709166454.902,
    "JobRuntime": 0.036,
    "Send": 0,
    "Receive": 0,
    "Exitval": 127,
    "Signal": 0,
    "Command": "BuildFileArrays /tmp/lint/test/linters/xml/xsd_bad_1.xsd /tmp/lint/test/linters/xml/xsd_good_1.xsd /tmp/lint/test/linters/yaml/yml_bad_1.yml /tmp/lint/test/linters/yaml/yml_good_1.yml /tmp/lint/test/run-super-linter-tests.sh /tmp/lint/test/validate-docker-labels.sh /tmp/lint/version.txt /tmp/lint",
    "V": [
      "/tmp/lint/test/linters/xml/xsd_bad_1.xsd",
      "/tmp/lint/test/linters/xml/xsd_good_1.xsd",
      "/tmp/lint/test/linters/yaml/yml_bad_1.yml",
      "/tmp/lint/test/linters/yaml/yml_good_1.yml",
      "/tmp/lint/test/run-super-linter-tests.sh",
      "/tmp/lint/test/validate-docker-labels.sh",
      "/tmp/lint/version.txt",
      "/tmp/lint"
    ],
    "Stdout": "",
    "Stderr": "/bin/sh: BuildFileArrays: not found
"
  }

...

/bin/sh: BuildFileArrays: not found
/bin/sh: BuildFileArrays: not found
/bin/sh: BuildFileArrays: not found
/bin/sh: BuildFileArrays: not found
/bin/sh: BuildFileArrays: not found
------
2024-02-29 00:27:35 [FATAL]   Error when building the list of files and directories to lint.
2024-02-29 00:27:35 [DEBUG]   Captured exit code: 1
2024-02-29 00:27:35 [DEBUG]   Removing temporary files and directories
2024-02-29 00:27:35 [DEBUG]   LOG_FILE_PATH: /tmp/lint/super-linter.log
2024-02-29 00:27:35 [DEBUG]   Skipping the moving of the log file from /tmp/tmp.k9y3EiHek2 to /tmp/lint/super-linter.log
make: *** [lint-codebase] Error 1

Steps To Reproduce

Clone this repo (super-linter/super-linter)
git rev-parse HEAD https://github.com/super-linter/super-linter/commit/6c1f40c9031146346142102c3ecca4954671735e
make SUPER_LINTER_TEST_CONTAINER_URL=ghcr.io/super-linter/super-linter:slim-latest@sha256:5c73796a5ba0a5b6811712bfb2a58cc39346a92ccafd28f570ab46ea16e68e61 lint-codebase | tee log.txt

Anything else?

I believe this is occurring because the SHELL environment variable is not set and gnu parallel is choosing to use /bin/sh as a default even though bash is the desired shell. I suspect this can be resolved by exporting env var SHELL=/bin/bash (or setting this in the dockerfile).

Example from testing in the slim-latest image:

e7bc280fd46d:/# doit() { echo "$@"; }
e7bc280fd46d:/# export -f doit
e7bc280fd46d:/# parallel --will-cite doit ::: 1 2 3
/bin/sh: doit: not found
/bin/sh: doit: not found
/bin/sh: doit: not found
e7bc280fd46d:/# SHELL=$(type -p bash) parallel --will-cite doit ::: 1 2 3
1
2
3

About this issue

Original URL
State: open
Created 4 months ago
Reactions: 1
Comments: 19

Most upvoted comments

Maybe we can try with PARALLEL_SHELL first?

ferrarimarco on Mar 2, 2024

Ah! We crossed the streams 😃

Maybe we have better luck with the slim image?
make IMAGE=slim docker
make lint-codebase
Yes, this worked.

Another observation about export -f doit: I don’t think that the Dash shell support exporting functions, does it?

AFAIK, this is bash-specific. I don’t know what (if any) other shells support it. But it is how super-linter is making the function available to parallel: https://github.com/super-linter/super-linter/blob/main/lib/functions/buildFileList.sh#L511

Yeah, I think so.

Also, I’d like to reproduce this in our CI environment somehow, so we can have feedback about the proposed fix. I wasn’t able to reproduce this in a Linux-based environment.

You could configure a macos arm runner to run a test of a built amd64 image under emulation? https://github.blog/2023-10-02-introducing-the-new-apple-silicon-powered-m1-macos-larger-runner-for-github-actions/

Yeah, saw that. We dynamically generate part of the workflow matrix, so I’m thinking about the best way to do this 😉

ferrarimarco on Feb 29, 2024

The GNU parallel manual states, in line with what you observed in its source code:

$PARALLEL_SHELL

Use this shell for the commands run by GNU parallel:

- $PARALLEL_SHELL. If undefined use:
- The shell that started GNU parallel. If that cannot be determined:
- $SHELL. If undefined use:
- /bin/sh

The shell that started GNU Parallel is /usr/bin/env bash, as the hashbang specifies, so it’s probably falling the

However, it seems under emulation parent_shell may not work?

case you listed above.

ferrarimarco on Feb 29, 2024