runtime: Introduced in v3.1.15 - PublishSingleFile binary fails to start (cannot extract bundle) when run by systemd

Description

As of released runtime version 3.1.15, SDK v3.1.409 and SDK v3.1.115, a ‘PublishSingleFile’ binary fails to start when launched by systemd.

The AppHost can no longer extract the single file bundle:

May 27 12:24:48 fv-az72-422 systemd[1]: Started BundlingIssue.
May 27 12:24:48 fv-az72-422 BundlingIssue[1894]: Failure processing application bundle.
May 27 12:24:48 fv-az72-422 BundlingIssue[1894]: Failed to determine location for extracting embedded files
May 27 12:24:48 fv-az72-422 BundlingIssue[1894]: DOTNET_BUNDLE_EXTRACT_BASE_DIR is not set, and a read-write cache directory couldn't be created.
May 27 12:24:48 fv-az72-422 BundlingIssue[1894]: A fatal error was encountered. Could not extract contents of the bundle

Interestingly, the binary runs just fine when launched as root using sudo, or run as a lower-privileged user, it’s only when launched via systemd that the failure occurs.

This issue does not happen in 408 or prior 40* versions.

I’ve created a repo at https://github.com/enclave-alistair/netcore31-409-singlefile-systemd that has a minimal recreate for the problem, along with precise steps in a Github Action that verifies the published binary runs fine as root, but not under systemd.

I’ve defined a build matrix in Github Actions that executes 401 through 409, 114 and 115 and the run only fails on 409 + 115. See this latest result set for evidence: https://github.com/enclave-alistair/netcore31-409-singlefile-systemd/actions/runs/882234200.

Introduced by fix for CVE-2021-31204?

I find it curious that a CVE related to single-file applications (https://github.com/dotnet/announcements/issues/185) was addressed in 3.1.15, and this is where the break occurs. I do wonder whether systemd has tmp folder management applied in such a way that causes bundle extraction to fail when the CVE fix is applied.

This issue is kind of a big deal for us to be honest, because we are now between a rock and a hard place of not wanting to ship code with a known vulnerability, but also needing to run under systemd on linux.

Configuration

Recreated on:

  • SDK 3.1.409 / 3.1.115 - Runtime version 3.1.15
  • Ubuntu 20.04
  • x64
  • I do not believe it is specific to the OS platform, but am not certain.

Regression?

This issue was introduced in the most recent release 3.1.15.

Other information

Systemd Unit

Unit File that fails is described below:

The path I use for the binary file does not make a difference.

[Unit]
Description=BundlingIssue
After=network.target

[Service]
ExecStart=/usr/bin/mybinary

[Install]
WantedBy=multi-user.target

DOTNET_BUNDLE_EXTRACT_BASE_DIR Workaround

If I specify a DOTNET_BUNDLE_EXTRACT_BASE_DIR environment variable in the unit file, then this issue does not occur, but it shouldn’t be required.

[Unit]
Description=BundlingIssue
After=network.target

[Service]
Environment="DOTNET_BUNDLE_EXTRACT_BASE_DIR=/var/tmp/bundle-extract"
ExecStart=/usr/bin/mybinary

[Install]
WantedBy=multi-user.target

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 16 (9 by maintainers)

Most upvoted comments

Personally I agree it’s not a good idea to embed knowledge of systemd’s “specialness” into the runtime.

Since in order to run a service under systemd, you must define a .service file anyway, in which there are clear ways to define an Environment variable that does use the correct user’s home directory (via %h) in the same way as $HOME, that workaround may be sufficient, e.g.:

[Service]
Environment="DOTNET_BUNDLE_EXTRACT_BASE_DIR=%h/.net"

I would propose two things:

  • Can we improve the message displayed to the user by the runtime if $HOME is not defined? At the moment it just says “Failed to determine location”, but if it said “No $HOME variable is available, so extract location could not be determined.”, that might make it clearer what is going on.
  • There should probably be explicit documentation somewhere that setting the environment variable with %h is the recommended workaround to use when running under systemd, to avoid users accidentally blundering into using /var/tmp or /tmp, which reintroduces the previous security flaw.