runtime: Exit code 139 on 'dotnet new' command for Alpine 3.9

As part of the work to add support for .NET Docker images on Alpine 3.9 (see dotnet/dotnet-docker#933), we’ve encountered an intermittent issue in our tests. The issue is caused by running the dotnet new command in an Alpine 3.9 Docker container which results in a 139 exit code with nothing written out to standard error.

The issue is limited to .NET Core SDK 2.x running on Alpine 3.9. It does not repro on .NET Core 3.0. It also does not repro with Alpine 3.8.

We’ve encountered intermittent failures in our automated builds with these tests that suggests it’s this issue but we haven’t yet been able to confirm definitively since they have not failed after adding exit code information to the diagnostics. But we have been able to repro this manually albeit in a way that is not the same way that the builds execute the tests.

The repro involves running our test script which executes our test in a Linux container. The test then creates its own Docker container for Alpine 3.9 and executes the dotnet new command in that. So this is a nested Docker scenario. However, just running the test by itself doesn’t repro the issue. You need to execute the test script twice in parallel. That’s where it differs from what our automated builds do. It also doesn’t make much sense since the dotnet new commands are being executed in distinct Docker containers and should not be interfering with each other. Nevertheless, that’s how I’m able to repro the issue.

Repro Steps (Requires a machine with Docker for Windows installed.)

1. Clone the dotnet/dotnet-docker repo.
2. Change your working directory to the root of the repo.
3. In a PowerShell window, run ./build-and-test.ps1 -VersionFilter 2.2* -OSFilter alpine3.9* -SkipTesting
4. Open the tests\Microsoft.DotNet.Docker.Tests\ImageTests.cs file and comment out the [Theory] attribute for all the methods except for VerifyRuntimeImage_AppScenario.
5. Run docker build -t testrunner -f ./tests/Dockerfile.linux.testrunner .
6. Create a new Console App project with the following code:

using System;
using System.Diagnostics;
using System.Text;
using System.Threading.Tasks;

namespace ConsoleApp
{
    class Program
    {
        static void Main(string[] args)
        {
            RunAsync().Wait();
        }

        static async Task RunAsync()
        {
            int i = 0;
            while (true)
            {
                Console.WriteLine("Iteration " + i++);

                var task1 = ExecuteAsync();
                var task2 = ExecuteAsync();
                await Task.WhenAll(task1, task2);
            }
        }

        static Task ExecuteAsync()
        {
            TaskCompletionSource<bool> tcs = new TaskCompletionSource<bool>();

            Task.Run(() =>
            {
                var process = new Process();
                process.StartInfo = new ProcessStartInfo
                {
                    FileName = "docker",
                    Arguments = "run --rm -v /var/run/docker.sock:/var/run/docker.sock -w /src testrunner pwsh -File ./tests/run-tests.ps1 -VersionFilter 2.2* -OSFilter alpine3.9* -ArchitectureFilter amd64",
                    RedirectStandardError = true,
                    RedirectStandardOutput = true,
                };
                StringBuilder stdOutput = new StringBuilder();
                process.OutputDataReceived += new DataReceivedEventHandler((sender, e) => stdOutput.AppendLine(e.Data));

                StringBuilder stdError = new StringBuilder();
                process.ErrorDataReceived += new DataReceivedEventHandler((sender, e) => stdError.AppendLine(e.Data));

                process.EnableRaisingEvents = true;
                process.Start();
                process.BeginOutputReadLine();
                process.BeginErrorReadLine();
                process.WaitForExit();

                int exitCode = process.ExitCode;

                if (exitCode != 0)
                {
                    var stdErr = stdError.ToString();
                    Debug.WriteLine(stdOutput.ToString());
                    Debug.WriteLine("");
                    Debug.WriteLine("");
                    Debug.WriteLine(stdErr);
                    tcs.SetException(new Exception());
                }
                else
                {
                    tcs.SetResult(true);
                }
            });

            return tcs.Task;

        }
    }
}

1. Run this project in Debug mode in VS.

Result: An exception will be thrown. The Debug window output will contain the following output. It usually will repro on the first or second iteration but may take more. Each iteration might take 1 and a half minutes to run.

Starting test execution, please wait...
Failed   Microsoft.DotNet.Docker.Tests.ImageTests.VerifyRuntimeImage_AppScenario(imageData: Arch='Amd64', Version='2.2', VersionString='2.2', HasSdk='True', IsArm='False', OS='alpine3.9', Rid='linux-musl-x64', RuntimeDepsVersion='2.2', SdkOS='alpine3.9', SdkVersion='2.2')
Exit code: 139
Error Message:
 System.InvalidOperationException : Failed to execute docker run --name 2.2-create-console-131967895354341618 -w /app -p 80 mcr.microsoft.com/dotnet/core-nightly/sdk:2.2-alpine3.9 dotnet new console --framework netcoreapp2.2

Stack Trace:
   at Microsoft.DotNet.Docker.Tests.DockerHelper.Execute(String args, Boolean ignoreErrors, Boolean autoRetry, ITestOutputHelper outputHelper) in /src/tests/Microsoft.DotNet.Docker.Tests/DockerHelper.cs:line 92
   at Microsoft.DotNet.Docker.Tests.DockerHelper.ExecuteWithLogging(String args, Boolean ignoreErrors, Boolean autoRetry) in /src/tests/Microsoft.DotNet.Docker.Tests/DockerHelper.cs:line 142
   at Microsoft.DotNet.Docker.Tests.DockerHelper.Run(String image, String name, String command, String workdir, String publishArgs, Boolean detach, Boolean runAsContainerAdministrator, Boolean skipAutoCleanup) in /src/tests/Microsoft.DotNet.Docker.Tests/DockerHelper.cs:line 222
   at Microsoft.DotNet.Docker.Tests.ImageScenarioVerifier.CreateTestAppWithSdkImage(String appType) in /src/tests/Microsoft.DotNet.Docker.Tests/ImageScenarioVerifier.cs:line 123
   at Microsoft.DotNet.Docker.Tests.ImageScenarioVerifier.Execute() in /src/tests/Microsoft.DotNet.Docker.Tests/ImageScenarioVerifier.cs:line 37
   at Microsoft.DotNet.Docker.Tests.ImageTests.VerifyRuntimeImage_AppScenario(ImageData imageData) in /src/tests/Microsoft.DotNet.Docker.Tests/ImageTests.cs:line 145
--- End of stack trace from previous location where exception was thrown ---
Standard Output Messages:
 Executing: docker pull mcr.microsoft.com/dotnet/core-nightly/sdk:2.2-alpine3.9
 2.2-alpine3.9: Pulling from dotnet/core-nightly/sdk
 Digest: sha256:18630987e8752d76988513f0b0d27535b4b51968055d41a03b53599ccd2cff4e
 Status: Image is up to date for mcr.microsoft.com/dotnet/core-nightly/sdk:2.2-alpine3.9
 Execution Elapsed Time: 00:00:00.6170324
 Executing: docker run --name 2.2-create-console-131967895354341618 -w /app -p 80 mcr.microsoft.com/dotnet/core-nightly/sdk:2.2-alpine3.9 dotnet new console --framework netcoreapp2.2
 Getting ready...
 The template "Console Application" was created successfully.
 
 Processing post-creation actions...
 Running 'dotnet restore' on /app/app.csproj...
   Restoring packages for /app/app.csproj...
   Generating MSBuild file /app/obj/app.csproj.nuget.g.props.
   Generating MSBuild file /app/obj/app.csproj.nuget.g.targets.
   Restore completed in 305.04 ms for /app/app.csproj.
 
 Restore succeeded.
 Executing: docker container rm -f 2.2-create-console-131967895354341618
 2.2-create-console-131967895354341618
 Execution Elapsed Time: 00:00:00.1486305

Results File: /src/tests/Microsoft.DotNet.Docker.Tests/TestResults/_ec417107f69b_2019-03-11_14_52_24.trx

Total tests: 1. Passed: 0. Failed: 1. Skipped: 0.
Test execution time: 11.1232 Seconds

[xUnit.net 00:00:09.3049272]     Microsoft.DotNet.Docker.Tests.ImageTests.VerifyRuntimeImage_AppScenario(imageData: Arch='Amd64', Version='2.2', VersionString='2.2', HasSdk='True', IsArm='False', OS='alpine3.9', Rid='linux-musl-x64', RuntimeDepsVersion='2.2', SdkOS='alpine3.9', SdkVersion='2.2') [FAIL]
Test Run Failed.
Tests Failed
At /src/tests/run-tests.ps1:77 char:32
+     if ($LASTEXITCODE -ne 0) { throw "Tests Failed" }
+                                ~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo          : OperationStopped: (Tests Failed:String) [], RuntimeException
+ FullyQualifiedErrorId : Tests Failed

Notice the output contains Exit code: 139. This is output by the tests\Microsoft.DotNet.Docker.Tests\DockerHelper.cs class when it runs dotnet new via the docker run command.

As part of troubleshooting the issue, if you make any code changes to the repo, just re-run the command in step 5, and the change will be incorporated by the tests when they’re executed.