extensions: Unhandled exception in IHostedService does not stop the host

Describe the bug

If an unhandled exception occurs inside an implementation of a Microsoft.Extensions.Hosting.BackgroundService I would expect it to exit the application.

Using the generic host builder in a .net core 2.1 console application, I register hosted services that override BackgroundService. Each service returns a task that performs a long running operation. If an unhandled exception is thrown inside the task the host is not exited but continues to run until a Ctrl^C is triggered.

To Reproduce

Steps to reproduce the behavior:

1. Using version ‘2.2.0’ of package ‘Microsoft.Extensions.Hosting’
2. Run this code

internal class Program
    {
        public static async Task Main(string[] args)
        {
            var host = new HostBuilder()
                .ConfigureServices(services => services.AddHostedService<TestService>())
                .Build();

            await host.RunAsync();
        }
    }

    public class TestService : BackgroundService
    {
        protected override async Task ExecuteAsync(CancellationToken stoppingToken)
        {
            try
            {
                await Task.Run(() =>
                    {
                        try
                        {
                            Thread.Sleep(5000);                        
                            throw new SystemException("Something went wrong!");
                        }
                        catch (Exception e)
                        {
                            Console.WriteLine(e);
                            throw;
                        }
                    }, stoppingToken);
            }
            catch (Exception e)
            {
                Console.WriteLine(e);
                throw;
            }
        }
    }

1. See error The exception is correctly handled and rethrown but is consumed by the host.

Application started. Press Ctrl+C to shut down.
System.SystemException: Something went wrong!
   at TestHostBuilder.TestService.<>c.<ExecuteAsync>b__0_0() in Program.cs:line 31
Hosting environment: Production
Content root path: TestHostBuilder\bin\Debug\netcoreapp2.1\
System.SystemException: Something went wrong!
   at TestHostBuilder.TestService.<>c.<ExecuteAsync>b__0_0() in Program.cs:line 31
   at System.Threading.Tasks.Task`1.InnerInvoke()
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location where exception was thrown ---
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot)
--- End of stack trace from previous location where exception was thrown ---
   at TestHostBuilder.TestService.ExecuteAsync(CancellationToken stoppingToken) in Program.cs:line 27

Expected behavior

I expected the host to exit on receiving the unhandled exception. See example below

Additional context

What I don’t understand is why when I run the following hosted service using “System.Reactive” Version=“4.1.3” the exception does bubble up to the host and the application does exit.

public class TestService2 : BackgroundService
        {
            protected override async Task ExecuteAsync(CancellationToken stoppingToken)
            {
                try
                {
                    await Task.Run(() =>
                    {
                        try
                        {
                            Observable.Interval(TimeSpan.FromSeconds(1))
                                .Subscribe(x =>
                                {
                                    if (x > 4)
                                    {
                                        throw new SystemException("Something went wrong!");
                                    }
                                });
                        }
                        catch (Exception e)
                        {
                            Console.WriteLine(e);
                            throw;
                        }
                    }, stoppingToken);
                }
                catch (Exception e)
                {
                    Console.WriteLine(e);
                    throw;
                }
            }
        }

Output

Application started. Press Ctrl+C to shut down.
Hosting environment: Production
Content root path: TestHostBuilder\bin\Debug\netcoreapp2.1\

Unhandled Exception: System.SystemException: Something went wrong!
   at TestHostBuilder.TestService.TestService2.<>c.<ExecuteAsync>b__0_1(Int64 x) in Program.cs:line 66
   at System.Reactive.AnonymousSafeObserver`1.OnNext(T value) in D:\a\1\s\Rx.NET\Source\src\System.Reactive\AnonymousSafeObserver.cs:line 44
   at System.Reactive.Concurrency.Scheduler.<>c__67`1.<SchedulePeriodic>b__67_0(ValueTuple`2 t) in D:\a\1\s\Rx.NET\Source\src\System.Reactive\Concurrency\Scheduler.Services.Emulation.cs:line 79
   at System.Reactive.Concurrency.DefaultScheduler.PeriodicallyScheduledWorkItem`1.<>c.<Tick>b__5_0(PeriodicallyScheduledWorkItem`1 closureWorkItem) in D:\a\1\s\Rx.NET\Source\src\System.Reactive\Concurrency\DefaultScheduler.cs:line 127
   at System.Reactive.Concurrency.AsyncLock.Wait(Object state, Delegate delegate, Action`2 action) in D:\a\1\s\Rx.NET\Source\src\System.Reactive\Concurrency\AsyncLock.cs:line 93
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location where exception was thrown ---
   at System.Threading.TimerQueueTimer.CallCallback()
   at System.Threading.TimerQueueTimer.Fire()
   at System.Threading.TimerQueue.FireNextTimers()

C:\Program Files\dotnet\dotnet.exe (process 38600) exited with code 0.
To automatically close the console when debugging stops, enable Tools->Options->Debugging->Automatically close the console when debugging stops.
Press any key to close this window . . .

I believe the issue may be related to the implementation of BackgroundService

        /// <summary>
        /// Triggered when the application host is ready to start the service.
        /// </summary>
        /// <param name="cancellationToken">Indicates that the start process has been aborted.</param>
        public virtual Task StartAsync(CancellationToken cancellationToken)
        {
            // Store the task we're executing
            _executingTask = ExecuteAsync(_stoppingCts.Token);

            // If the task is completed then return it, this will bubble cancellation and failure to the caller
            if (_executingTask.IsCompleted)
            {
                return _executingTask;
            }

            // Otherwise it's running
            return Task.CompletedTask;
        }

If I update the code to always return the executing task and not Task.CompletedTaskthen it works as expected using both TestService and TestService2.

// <summary>
        /// Triggered when the application host is ready to start the service.
        /// </summary>
        /// <param name="cancellationToken">Indicates that the start process has been aborted.</param>
        public virtual Task StartAsync(CancellationToken cancellationToken)
        {
            // Store the task we're executing
            _executingTask = ExecuteAsync(_stoppingCts.Token);

            return _executingTask;
        }