MassTransit: Kafka rider health check goes from ready -> degarded -> ready when broker dies

Contact Details

No response

Version

8.x (develop, pre-release)

On which operating system(s) are you experiencing the issue?

Windows, Linux

Using which broker(s) did you encounter the issue?

Kafka

What are the steps required to reproduce the issue?

1. Run Normaly, Lose connection to the broker at random time
2. Get Errors, Health check still ready
3. Sometime client handler is dispose and throws the process
...

What is the expected behavior?

Health check should be Degraded. Client handler should fail safely and reconnect when possibe

What actually happened?

Running in production as usuall untill something happens at random times.

Related log output, including any exceptions

System.AggregateException : One or more errors occurred. (One or more errors occurred. (Local: Broker handle destroyed))
  ----> System.AggregateException : One or more errors occurred. (Local: Broker handle destroyed)
  ----> Confluent.Kafka.KafkaException : Local: Broker handle destroyed
   at MassTransit.Middleware.Agent.Stop(StopContext context) in C:\Users\Nico\source\repos\MassTransit\src\MassTransit.Abstractions\Middleware\Middleware\Agent.cs:line 87
   at MassTransit.Middleware.Supervisor.StopSupervisor(StopSupervisorContext context) in C:\Users\Nico\source\repos\MassTransit\src\MassTransit.Abstractions\Middleware\Middleware\Supervisor.cs:line 102
   at MassTransit.Middleware.Agent.Stop(StopContext context) in C:\Users\Nico\source\repos\MassTransit\src\MassTransit.Abstractions\Middleware\Middleware\Agent.cs:line 87
   at MassTransit.Middleware.Supervisor.StopSupervisor(StopSupervisorContext context) in C:\Users\Nico\source\repos\MassTransit\src\MassTransit.Abstractions\Middleware\Middleware\Supervisor.cs:line 102
   at MassTransit.Middleware.Agent.Stop(StopContext context) in C:\Users\Nico\source\repos\MassTransit\src\MassTransit.Abstractions\Middleware\Middleware\Agent.cs:line 87
   at MassTransit.Transports.TransportPipeContextSupervisor`1.StopSupervisor(StopSupervisorContext context) in C:\Users\Nico\source\repos\MassTransit\src\MassTransit\Transports\TransportPipeContextSupervisor.cs:line 41
   at MassTransit.Middleware.Agent.Stop(StopContext context) in C:\Users\Nico\source\repos\MassTransit\src\MassTransit.Abstractions\Middleware\Middleware\Agent.cs:line 87
   at MassTransit.Transports.ReceiveTransport`1.ReceiveTransportAgent.StopAgent(StopContext context) in C:\Users\Nico\source\repos\MassTransit\src\MassTransit\Transports\ReceiveTransport.cs:line 105
   at MassTransit.Middleware.Agent.Stop(StopContext context) in C:\Users\Nico\source\repos\MassTransit\src\MassTransit.Abstractions\Middleware\Middleware\Agent.cs:line 87
   at MassTransit.Transports.ReceiveEndpoint.Stop(Boolean removed, CancellationToken cancellationToken) in C:\Users\Nico\source\repos\MassTransit\src\MassTransit\Transports\ReceiveEndpoint.cs:line 175
   at MassTransit.Transports.ReceiveEndpointCollection.StopAgent(StopContext context) in C:\Users\Nico\source\repos\MassTransit\src\MassTransit\Transports\ReceiveEndpointCollection.cs:line 106
   at MassTransit.Middleware.Agent.Stop(StopContext context) in C:\Users\Nico\source\repos\MassTransit\src\MassTransit.Abstractions\Middleware\Middleware\Agent.cs:line 87
   at MassTransit.KafkaIntegration.KafkaRider.RiderAgent.StopAgent(StopContext context) in C:\Users\Nico\source\repos\MassTransit\src\Transports\MassTransit.KafkaIntegration\KafkaIntegration\KafkaRider.cs:line 113
   at MassTransit.Middleware.Agent.Stop(StopContext context) in C:\Users\Nico\source\repos\MassTransit\src\MassTransit.Abstractions\Middleware\Middleware\Agent.cs:line 87
   at MassTransit.Transports.RiderCollection.Handle.StopAsync(Boolean remove, CancellationToken cancellationToken) in C:\Users\Nico\source\repos\MassTransit\src\MassTransit\Transports\RiderCollection.cs:line 179
   at MassTransit.Transports.RiderCollection.StopAgent(StopContext context) in C:\Users\Nico\source\repos\MassTransit\src\MassTransit\Transports\RiderCollection.cs:line 138
   at MassTransit.Middleware.Agent.Stop(StopContext context) in C:\Users\Nico\source\repos\MassTransit\src\MassTransit.Abstractions\Middleware\Middleware\Agent.cs:line 87
   at MassTransit.Transports.BaseHost.Stop(CancellationToken cancellationToken) in C:\Users\Nico\source\repos\MassTransit\src\MassTransit\Transports\BaseHost.cs:line 152
   at MassTransit.MassTransitBus.Handle.StopAsync(CancellationToken cancellationToken) in C:\Users\Nico\source\repos\MassTransit\src\MassTransit\MassTransitBus.cs:line 384
   at MassTransit.MassTransitBus.Handle.StopAsync(CancellationToken cancellationToken) in C:\Users\Nico\source\repos\MassTransit\src\MassTransit\MassTransitBus.cs:line 400
   at MassTransit.MassTransitBus.StopAsync(CancellationToken cancellationToken) in C:\Users\Nico\source\repos\MassTransit\src\MassTransit\MassTransitBus.cs:line 263
   at MassTransit.MassTransitHostedService.DisposeAsync() in C:\Users\Nico\source\repos\MassTransit\src\MassTransit\MassTransitHostedService.cs:line 37
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngineScope.<DisposeAsync>g__Await|22_0(Int32 i, ValueTask vt, List`1 toDispose)
   at MassTransit.KafkaIntegration.Tests.HealthCheck_Specs.should_be_degraded_when_lost_connection() in C:\Users\Nico\source\repos\MassTransit\tests\MassTransit.KafkaIntegration.Tests\HealthCheck_Specs.cs:line 93
   at NUnit.Framework.Internal.TaskAwaitAdapter.GenericAdapter`1.BlockUntilCompleted()
   at NUnit.Framework.Internal.MessagePumpStrategy.NoMessagePumpStrategy.WaitForCompletion(AwaitAdapter awaiter)
   at NUnit.Framework.Internal.AsyncToSyncAdapter.Await(Func`1 invoke)
   at NUnit.Framework.Internal.Commands.TestMethodCommand.RunTestMethod(TestExecutionContext context)
   at NUnit.Framework.Internal.Commands.TestMethodCommand.Execute(TestExecutionContext context)
   at NUnit.Framework.Internal.Commands.BeforeAndAfterTestCommand.<>c__DisplayClass1_0.<Execute>b__0()
   at NUnit.Framework.Internal.Commands.DelegatingTestCommand.RunTestMethodInThreadAbortSafeZone(TestExecutionContext context, Action action)
--AggregateException

--KafkaException
   at Confluent.Kafka.Impl.SafeKafkaHandle.ConsumerClose()
   at Confluent.Kafka.Consumer`2.Close()
   at MassTransit.KafkaIntegration.KafkaMessageConsumer`2.ActiveAndActualAgentsCompleted(StopContext context) in C:\Users\Nico\source\repos\MassTransit\src\Transports\MassTransit.KafkaIntegration\KafkaIntegration\KafkaMessageConsumer.cs:line 117

19:44:44.300-D Running health checks
19:44:44.302-D Running health check masstransit-bus
19:44:44.312-E Health check masstransit-bus with status Unhealthy completed after 9.462ms with message 'Not ready: not started'
19:44:44.313-D Health check data for masstransit-bus:
    Endpoints: loopback://localhost/DESKTOPFVPA0MK_ReSharperTestRunner_bus_qtbyyyguhzcfh4ihbdphw6k9n7: Unhealthy - not ready, loopback://localhost/kafka/health-check/HealthCheck_Specs: Unhealthy - not ready
    HealthCheckName: masstransit-bus

19:44:44.316-D Health check processing with combined status Unhealthy completed after 16.2988ms
{
  "status": "Unhealthy",
  "results": {
    "masstransit-bus": {
      "status": "Unhealthy",
      "description": "Not ready: not started",
      "data": {
        "Endpoints": {
          "loopback://localhost/DESKTOPFVPA0MK_ReSharperTestRunner_bus_qtbyyyguhzcfh4ihbdphw6k9n7": {
            "status": "Unhealthy",
            "description": "not ready"
          },
          "loopback://localhost/kafka/health-check/HealthCheck_Specs": {
            "status": "Unhealthy",
            "description": "not ready"
          }
        }
      }
    }
  }
}
19:44:45.367-I Creating topics: health-check with 1 partitions and 1 replicas
19:44:45.508-D Starting bus instances: IBus
19:44:45.509-D Starting bus: loopback://localhost/
19:44:45.509-D Endpoint Ready: loopback://localhost/DESKTOPFVPA0MK_ReSharperTestRunner_bus_qtbyyyguhzcfh4ihbdphw6k9n7
19:44:45.525-D Endpoint Ready: loopback://localhost/kafka/health-check/HealthCheck_Specs
19:44:45.525-I Bus started: loopback://localhost/
19:44:45.648-D Running health checks
19:44:45.648-D Running health check masstransit-bus
19:44:45.649-D Health check masstransit-bus with status Healthy completed after 0.3539ms with message 'Ready'
19:44:45.649-D Health check data for masstransit-bus:
    Endpoints: loopback://localhost/DESKTOPFVPA0MK_ReSharperTestRunner_bus_qtbyyyguhzcfh4ihbdphw6k9n7: Healthy - ready, loopback://localhost/kafka/health-check/HealthCheck_Specs: Healthy - ready
    HealthCheckName: masstransit-bus

19:44:45.649-D Health check processing with combined status Healthy completed after 0.5868ms
{
  "status": "Healthy",
  "results": {
    "masstransit-bus": {
      "status": "Healthy",
      "description": "Ready",
      "data": {
        "Endpoints": {
          "loopback://localhost/DESKTOPFVPA0MK_ReSharperTestRunner_bus_qtbyyyguhzcfh4ihbdphw6k9n7": {
            "status": "Healthy",
            "description": "ready"
          },
          "loopback://localhost/kafka/health-check/HealthCheck_Specs": {
            "status": "Healthy",
            "description": "ready"
          }
        }
      }
    }
  }
}
19:44:47.259-W Consumer [rdkafka-c81fa82d-4dd6-4ee6-a4c0-683374cc5aee] error (Local_Transport): GroupCoordinator: localhost:9092: Disconnected (after 1712ms in state UP) on health-check
19:44:47.260-W Consumer [rdkafka-c81fa82d-4dd6-4ee6-a4c0-683374cc5aee] error (Local_Transport): localhost:9092/1: Disconnected: verify that security.protocol is correctly configured, broker might require SASL authentication (after 1715ms in state UP) on health-check
19:44:47.260-W Consumer [rdkafka-c81fa82d-4dd6-4ee6-a4c0-683374cc5aee] error (Local_AllBrokersDown): 2/2 brokers are down on health-check
19:44:47.261-D Consumer Stopping: loopback://localhost/kafka/health-check/HealthCheck_Specs (Consume Loop Exited)
19:44:57.474-D Running health checks
19:44:57.475-D Running health check masstransit-bus
19:44:57.475-D Health check masstransit-bus with status Healthy completed after 0.0548ms with message 'Ready'
19:44:57.475-D Health check data for masstransit-bus:
    Endpoints: loopback://localhost/DESKTOPFVPA0MK_ReSharperTestRunner_bus_qtbyyyguhzcfh4ihbdphw6k9n7: Healthy - ready, loopback://localhost/kafka/health-check/HealthCheck_Specs: Healthy - ready
    HealthCheckName: masstransit-bus

19:44:57.475-D Health check processing with combined status Healthy completed after 0.1544ms
19:44:58.477-D Running health checks
19:44:58.477-D Running health check masstransit-bus
19:44:58.478-D Health check masstransit-bus with status Healthy completed after 0.0886ms with message 'Ready'
19:44:58.478-D Health check data for masstransit-bus:
    Endpoints: loopback://localhost/DESKTOPFVPA0MK_ReSharperTestRunner_bus_qtbyyyguhzcfh4ihbdphw6k9n7: Healthy - ready, loopback://localhost/kafka/health-check/HealthCheck_Specs: Healthy - ready
    HealthCheckName: masstransit-bus

19:44:58.478-D Health check processing with combined status Healthy completed after 0.2749ms
19:44:59.486-D Running health checks
19:44:59.486-D Running health check masstransit-bus
19:44:59.486-D Health check masstransit-bus with status Healthy completed after 0.0858ms with message 'Ready'
19:44:59.486-D Health check data for masstransit-bus:
    Endpoints: loopback://localhost/DESKTOPFVPA0MK_ReSharperTestRunner_bus_qtbyyyguhzcfh4ihbdphw6k9n7: Healthy - ready, loopback://localhost/kafka/health-check/HealthCheck_Specs: Healthy - ready
    HealthCheckName: masstransit-bus

19:44:59.486-D Health check processing with combined status Healthy completed after 0.2006ms
19:45:00.489-D Running health checks
19:45:00.489-D Running health check masstransit-bus
19:45:00.489-D Health check masstransit-bus with status Healthy completed after 0.0782ms with message 'Ready'
19:45:00.489-D Health check data for masstransit-bus:
    Endpoints: loopback://localhost/DESKTOPFVPA0MK_ReSharperTestRunner_bus_qtbyyyguhzcfh4ihbdphw6k9n7: Healthy - ready, loopback://localhost/kafka/health-check/HealthCheck_Specs: Healthy - ready
    HealthCheckName: masstransit-bus

19:45:00.489-D Health check processing with combined status Healthy completed after 0.2263ms
19:45:01.491-D Running health checks
19:45:01.491-D Running health check masstransit-bus
19:45:01.491-D Health check masstransit-bus with status Healthy completed after 0.09ms with message 'Ready'
19:45:01.491-D Health check data for masstransit-bus:
    Endpoints: loopback://localhost/DESKTOPFVPA0MK_ReSharperTestRunner_bus_qtbyyyguhzcfh4ihbdphw6k9n7: Healthy - ready, loopback://localhost/kafka/health-check/HealthCheck_Specs: Healthy - ready
    HealthCheckName: masstransit-bus

19:45:01.491-D Health check processing with combined status Healthy completed after 0.2233ms
19:45:02.492-D Running health checks
19:45:02.492-D Running health check masstransit-bus
19:45:02.493-D Health check masstransit-bus with status Healthy completed after 0.0871ms with message 'Ready'
19:45:02.493-D Health check data for masstransit-bus:
    Endpoints: loopback://localhost/DESKTOPFVPA0MK_ReSharperTestRunner_bus_qtbyyyguhzcfh4ihbdphw6k9n7: Healthy - ready, loopback://localhost/kafka/health-check/HealthCheck_Specs: Healthy - ready
    HealthCheckName: masstransit-bus

19:45:02.493-D Health check processing with combined status Healthy completed after 0.2173ms
19:45:03.504-D Running health checks
19:45:03.504-D Running health check masstransit-bus
19:45:03.504-D Health check masstransit-bus with status Healthy completed after 0.0791ms with message 'Ready'
19:45:03.504-D Health check data for masstransit-bus:
    Endpoints: loopback://localhost/DESKTOPFVPA0MK_ReSharperTestRunner_bus_qtbyyyguhzcfh4ihbdphw6k9n7: Healthy - ready, loopback://localhost/kafka/health-check/HealthCheck_Specs: Healthy - ready
    HealthCheckName: masstransit-bus

19:45:03.504-D Health check processing with combined status Healthy completed after 0.2159ms
19:45:04.515-D Running health checks
19:45:04.515-D Running health check masstransit-bus
19:45:04.515-D Health check masstransit-bus with status Healthy completed after 0.1863ms with message 'Ready'
19:45:04.515-D Health check data for masstransit-bus:
    Endpoints: loopback://localhost/DESKTOPFVPA0MK_ReSharperTestRunner_bus_qtbyyyguhzcfh4ihbdphw6k9n7: Healthy - ready, loopback://localhost/kafka/health-check/HealthCheck_Specs: Healthy - ready
    HealthCheckName: masstransit-bus

19:45:04.515-D Health check processing with combined status Healthy completed after 0.2836ms
19:45:05.523-D Running health checks
19:45:05.523-D Running health check masstransit-bus
19:45:05.523-D Health check masstransit-bus with status Healthy completed after 0.0762ms with message 'Ready'
19:45:05.523-D Health check data for masstransit-bus:
    Endpoints: loopback://localhost/DESKTOPFVPA0MK_ReSharperTestRunner_bus_qtbyyyguhzcfh4ihbdphw6k9n7: Healthy - ready, loopback://localhost/kafka/health-check/HealthCheck_Specs: Healthy - ready
    HealthCheckName: masstransit-bus

19:45:05.523-D Health check processing with combined status Healthy completed after 0.1707ms
19:45:06.528-D Running health checks
19:45:06.528-D Running health check masstransit-bus
19:45:06.528-D Health check masstransit-bus with status Healthy completed after 0.0912ms with message 'Ready'
19:45:06.528-D Health check data for masstransit-bus:
    Endpoints: loopback://localhost/DESKTOPFVPA0MK_ReSharperTestRunner_bus_qtbyyyguhzcfh4ihbdphw6k9n7: Healthy - ready, loopback://localhost/kafka/health-check/HealthCheck_Specs: Healthy - ready
    HealthCheckName: masstransit-bus

19:45:06.528-D Health check processing with combined status Healthy completed after 0.239ms
19:45:07.539-D Running health checks
19:45:07.539-D Running health check masstransit-bus
19:45:07.539-D Health check masstransit-bus with status Healthy completed after 0.081ms with message 'Ready'
19:45:07.539-D Health check data for masstransit-bus:
    Endpoints: loopback://localhost/DESKTOPFVPA0MK_ReSharperTestRunner_bus_qtbyyyguhzcfh4ihbdphw6k9n7: Healthy - ready, loopback://localhost/kafka/health-check/HealthCheck_Specs: Healthy - ready
    HealthCheckName: masstransit-bus

19:45:07.539-D Health check processing with combined status Healthy completed after 0.1809ms
19:45:08.551-D Running health checks
19:45:08.551-D Running health check masstransit-bus
19:45:08.551-D Health check masstransit-bus with status Healthy completed after 0.0868ms with message 'Ready'
19:45:08.551-D Health check data for masstransit-bus:
    Endpoints: loopback://localhost/DESKTOPFVPA0MK_ReSharperTestRunner_bus_qtbyyyguhzcfh4ihbdphw6k9n7: Healthy - ready, loopback://localhost/kafka/health-check/HealthCheck_Specs: Healthy - ready
    HealthCheckName: masstransit-bus

19:45:08.551-D Health check processing with combined status Healthy completed after 0.2657ms
19:45:09.555-D Running health checks
19:45:09.555-D Running health check masstransit-bus
19:45:09.555-D Health check masstransit-bus with status Healthy completed after 0.0608ms with message 'Ready'
19:45:09.555-D Health check data for masstransit-bus:
    Endpoints: loopback://localhost/DESKTOPFVPA0MK_ReSharperTestRunner_bus_qtbyyyguhzcfh4ihbdphw6k9n7: Healthy - ready, loopback://localhost/kafka/health-check/HealthCheck_Specs: Healthy - ready
    HealthCheckName: masstransit-bus

19:45:09.555-D Health check processing with combined status Healthy completed after 0.1661ms
19:45:10.565-D Running health checks
19:45:10.565-D Running health check masstransit-bus
19:45:10.566-D Health check masstransit-bus with status Healthy completed after 0.0904ms with message 'Ready'
19:45:10.566-D Health check data for masstransit-bus:
    Endpoints: loopback://localhost/DESKTOPFVPA0MK_ReSharperTestRunner_bus_qtbyyyguhzcfh4ihbdphw6k9n7: Healthy - ready, loopback://localhost/kafka/health-check/HealthCheck_Specs: Healthy - ready
    HealthCheckName: masstransit-bus

19:45:10.566-D Health check processing with combined status Healthy completed after 0.1981ms
19:45:11.578-D Running health checks
19:45:11.578-D Running health check masstransit-bus
19:45:11.578-D Health check masstransit-bus with status Healthy completed after 0.0746ms with message 'Ready'
19:45:11.578-D Health check data for masstransit-bus:
    Endpoints: loopback://localhost/DESKTOPFVPA0MK_ReSharperTestRunner_bus_qtbyyyguhzcfh4ihbdphw6k9n7: Healthy - ready, loopback://localhost/kafka/health-check/HealthCheck_Specs: Healthy - ready
    HealthCheckName: masstransit-bus

19:45:11.578-D Health check processing with combined status Healthy completed after 0.1712ms
19:45:12.486-D Stopping bus instances: IBus
19:45:12.487-D Stopping bus: loopback://localhost/
19:45:12.490-D Endpoint Stopping: loopback://localhost/kafka/health-check/HealthCheck_Specs
19:45:12.491-D Consumer Stopping: loopback://localhost/kafka/health-check/HealthCheck_Specs (Stop Receive Transport)
19:45:12.495-D Endpoint Completed: loopback://localhost/kafka/health-check/HealthCheck_Specs
19:45:12.499-D Consumer Completed: loopback://localhost/kafka/health-check/HealthCheck_Specs: 0 received, 0 concurrent
19:45:12.502-D Endpoint Faulted: loopback://localhost/kafka/health-check/HealthCheck_Specs

MassTransit.KafkaConnectionException: ReceiveTransport faulted: health-check
 ---> System.AggregateException: One or more errors occurred. (One or more errors occurred. (One or more errors occurred. (One or more errors occurred. (Local: Broker handle destroyed))))
 ---> System.AggregateException: One or more errors occurred. (One or more errors occurred. (One or more errors occurred. (Local: Broker handle destroyed)))
 ---> System.AggregateException: One or more errors occurred. (One or more errors occurred. (Local: Broker handle destroyed))
 ---> System.AggregateException: One or more errors occurred. (Local: Broker handle destroyed)
 ---> Confluent.Kafka.KafkaException: Local: Broker handle destroyed
   at Confluent.Kafka.Impl.SafeKafkaHandle.ConsumerClose()
   at Confluent.Kafka.Consumer`2.Close()
   at MassTransit.KafkaIntegration.KafkaMessageConsumer`2.ActiveAndActualAgentsCompleted(StopContext context) in C:\Users\Nico\source\repos\MassTransit\src\Transports\MassTransit.KafkaIntegration\KafkaIntegration\KafkaMessageConsumer.cs:line 117
   --- End of inner exception stack trace ---
   --- End of inner exception stack trace ---
   at MassTransit.Middleware.Agent.Stop(StopContext context) in C:\Users\Nico\source\repos\MassTransit\src\MassTransit.Abstractions\Middleware\Middleware\Agent.cs:line 87
   at MassTransit.Middleware.Supervisor.StopSupervisor(StopSupervisorContext context) in C:\Users\Nico\source\repos\MassTransit\src\MassTransit.Abstractions\Middleware\Middleware\Supervisor.cs:line 102
   at MassTransit.Middleware.Agent.Stop(StopContext context) in C:\Users\Nico\source\repos\MassTransit\src\MassTransit.Abstractions\Middleware\Middleware\Agent.cs:line 87
   at MassTransit.Middleware.Supervisor.StopSupervisor(StopSupervisorContext context) in C:\Users\Nico\source\repos\MassTransit\src\MassTransit.Abstractions\Middleware\Middleware\Supervisor.cs:line 102
   at MassTransit.Middleware.Agent.Stop(StopContext context) in C:\Users\Nico\source\repos\MassTransit\src\MassTransit.Abstractions\Middleware\Middleware\Agent.cs:line 87
   at MassTransit.Transports.TransportPipeContextSupervisor`1.StopSupervisor(StopSupervisorContext context) in C:\Users\Nico\source\repos\MassTransit\src\MassTransit\Transports\TransportPipeContextSupervisor.cs:line 41
   at MassTransit.Middleware.Agent.Stop(StopContext context) in C:\Users\Nico\source\repos\MassTransit\src\MassTransit.Abstractions\Middleware\Middleware\Agent.cs:line 87
   at MassTransit.Transports.ReceiveTransport`1.ReceiveTransportAgent.StopAgent(StopContext context) in C:\Users\Nico\source\repos\MassTransit\src\MassTransit\Transports\ReceiveTransport.cs:line 105
   at MassTransit.Middleware.Agent.Stop(StopContext context) in C:\Users\Nico\source\repos\MassTransit\src\MassTransit.Abstractions\Middleware\Middleware\Agent.cs:line 87
   at MassTransit.Transports.ReceiveEndpoint.Stop(Boolean removed, CancellationToken cancellationToken) in C:\Users\Nico\source\repos\MassTransit\src\MassTransit\Transports\ReceiveEndpoint.cs:line 175
   at MassTransit.Transports.ReceiveEndpointCollection.StopAgent(StopContext context) in C:\Users\Nico\source\repos\MassTransit\src\MassTransit\Transports\ReceiveEndpointCollection.cs:line 106
   at MassTransit.Middleware.Agent.Stop(StopContext context) in C:\Users\Nico\source\repos\MassTransit\src\MassTransit.Abstractions\Middleware\Middleware\Agent.cs:line 87
   at MassTransit.KafkaIntegration.KafkaRider.RiderAgent.StopAgent(StopContext context) in C:\Users\Nico\source\repos\MassTransit\src\Transports\MassTransit.KafkaIntegration\KafkaIntegration\KafkaRider.cs:line 113
   at MassTransit.Middleware.Agent.Stop(StopContext context) in C:\Users\Nico\source\repos\MassTransit\src\MassTransit.Abstractions\Middleware\Middleware\Agent.cs:line 87
   at MassTransit.Transports.RiderCollection.Handle.StopAsync(Boolean remove, CancellationToken cancellationToken) in C:\Users\Nico\source\repos\MassTransit\src\MassTransit\Transports\RiderCollection.cs:line 179
   at MassTransit.Transports.RiderCollection.StopAgent(StopContext context) in C:\Users\Nico\source\repos\MassTransit\src\MassTransit\Transports\RiderCollection.cs:line 138
   at MassTransit.Middleware.Agent.Stop(StopContext context) in C:\Users\Nico\source\repos\MassTransit\src\MassTransit.Abstractions\Middleware\Middleware\Agent.cs:line 87
   at MassTransit.Transports.BaseHost.Stop(CancellationToken cancellationToken) in C:\Users\Nico\source\repos\MassTransit\src\MassTransit\Transports\BaseHost.cs:line 152
   at MassTransit.MassTransitBus.Handle.StopAsync(CancellationToken cancellationToken) in C:\Users\Nico\source\repos\MassTransit\src\MassTransit\MassTransitBus.cs:line 384
   --- End of inner exception stack trace ---
   --- End of inner exception stack trace ---
   at MassTransit.KafkaIntegration.Middleware.KafkaConsumerFilter`2.Send(ConsumerContext context, IPipe`1 next) in C:\Users\Nico\source\repos\MassTransit\src\Transports\MassTransit.KafkaIntegration\KafkaIntegration\Middleware\KafkaConsumerFilter.cs:line 34
   at MassTransit.KafkaIntegration.Middleware.KafkaConsumerFilter`2.Send(ConsumerContext context, IPipe`1 next) in C:\Users\Nico\source\repos\MassTransit\src\Transports\MassTransit.KafkaIntegration\KafkaIntegration\Middleware\KafkaConsumerFilter.cs:line 43
   at MassTransit.Middleware.ReceiveEndpointDependencyFilter`1.Send(TContext context, IPipe`1 next) in C:\Users\Nico\source\repos\MassTransit\src\MassTransit\Middleware\ReceiveEndpointDependencyFilter.cs:line 23
   at MassTransit.Agents.PipeContextSupervisor`1.Send(IPipe`1 pipe, CancellationToken cancellationToken) in C:\Users\Nico\source\repos\MassTransit\src\MassTransit\Agents\PipeContextSupervisor.cs:line 55
   at MassTransit.Agents.PipeContextSupervisor`1.Send(IPipe`1 pipe, CancellationToken cancellationToken) in C:\Users\Nico\source\repos\MassTransit\src\MassTransit\Agents\PipeContextSupervisor.cs:line 61
   at MassTransit.Agents.PipeContextSupervisor`1.Send(IPipe`1 pipe, CancellationToken cancellationToken) in C:\Users\Nico\source\repos\MassTransit\src\MassTransit\Agents\PipeContextSupervisor.cs:line 68
   at MassTransit.Transports.ReceiveTransport`1.ReceiveTransportAgent.RunTransport() in C:\Users\Nico\source\repos\MassTransit\src\MassTransit\Transports\ReceiveTransport.cs:line 195
   --- End of inner exception stack trace ---
19:45:12.520-D ReceiveTransport Cannot Retry: loopback://localhost/kafka/health-check/HealthCheck_Specs

MassTransit.KafkaConnectionException: ReceiveTransport faulted: health-check
 ---> System.AggregateException: One or more errors occurred. (One or more errors occurred. (One or more errors occurred. (One or more errors occurred. (Local: Broker handle destroyed))))
 ---> System.AggregateException: One or more errors occurred. (One or more errors occurred. (One or more errors occurred. (Local: Broker handle destroyed)))
 ---> System.AggregateException: One or more errors occurred. (One or more errors occurred. (Local: Broker handle destroyed))
 ---> System.AggregateException: One or more errors occurred. (Local: Broker handle destroyed)
 ---> Confluent.Kafka.KafkaException: Local: Broker handle destroyed
   at Confluent.Kafka.Impl.SafeKafkaHandle.ConsumerClose()
   at Confluent.Kafka.Consumer`2.Close()
   at MassTransit.KafkaIntegration.KafkaMessageConsumer`2.ActiveAndActualAgentsCompleted(StopContext context) in C:\Users\Nico\source\repos\MassTransit\src\Transports\MassTransit.KafkaIntegration\KafkaIntegration\KafkaMessageConsumer.cs:line 117
   --- End of inner exception stack trace ---
   --- End of inner exception stack trace ---
   at MassTransit.Middleware.Agent.Stop(StopContext context) in C:\Users\Nico\source\repos\MassTransit\src\MassTransit.Abstractions\Middleware\Middleware\Agent.cs:line 87
   at MassTransit.Middleware.Supervisor.StopSupervisor(StopSupervisorContext context) in C:\Users\Nico\source\repos\MassTransit\src\MassTransit.Abstractions\Middleware\Middleware\Supervisor.cs:line 102
   at MassTransit.Middleware.Agent.Stop(StopContext context) in C:\Users\Nico\source\repos\MassTransit\src\MassTransit.Abstractions\Middleware\Middleware\Agent.cs:line 87
   at MassTransit.Middleware.Supervisor.StopSupervisor(StopSupervisorContext context) in C:\Users\Nico\source\repos\MassTransit\src\MassTransit.Abstractions\Middleware\Middleware\Supervisor.cs:line 102
   at MassTransit.Middleware.Agent.Stop(StopContext context) in C:\Users\Nico\source\repos\MassTransit\src\MassTransit.Abstractions\Middleware\Middleware\Agent.cs:line 87
   at MassTransit.Transports.TransportPipeContextSupervisor`1.StopSupervisor(StopSupervisorContext context) in C:\Users\Nico\source\repos\MassTransit\src\MassTransit\Transports\TransportPipeContextSupervisor.cs:line 41
   at MassTransit.Middleware.Agent.Stop(StopContext context) in C:\Users\Nico\source\repos\MassTransit\src\MassTransit.Abstractions\Middleware\Middleware\Agent.cs:line 87
   at MassTransit.Transports.ReceiveTransport`1.ReceiveTransportAgent.StopAgent(StopContext context) in C:\Users\Nico\source\repos\MassTransit\src\MassTransit\Transports\ReceiveTransport.cs:line 105
   at MassTransit.Middleware.Agent.Stop(StopContext context) in C:\Users\Nico\source\repos\MassTransit\src\MassTransit.Abstractions\Middleware\Middleware\Agent.cs:line 87
   at MassTransit.Transports.ReceiveEndpoint.Stop(Boolean removed, CancellationToken cancellationToken) in C:\Users\Nico\source\repos\MassTransit\src\MassTransit\Transports\ReceiveEndpoint.cs:line 175
   at MassTransit.Transports.ReceiveEndpointCollection.StopAgent(StopContext context) in C:\Users\Nico\source\repos\MassTransit\src\MassTransit\Transports\ReceiveEndpointCollection.cs:line 106
   at MassTransit.Middleware.Agent.Stop(StopContext context) in C:\Users\Nico\source\repos\MassTransit\src\MassTransit.Abstractions\Middleware\Middleware\Agent.cs:line 87
   at MassTransit.KafkaIntegration.KafkaRider.RiderAgent.StopAgent(StopContext context) in C:\Users\Nico\source\repos\MassTransit\src\Transports\MassTransit.KafkaIntegration\KafkaIntegration\KafkaRider.cs:line 113
   at MassTransit.Middleware.Agent.Stop(StopContext context) in C:\Users\Nico\source\repos\MassTransit\src\MassTransit.Abstractions\Middleware\Middleware\Agent.cs:line 87
   at MassTransit.Transports.RiderCollection.Handle.StopAsync(Boolean remove, CancellationToken cancellationToken) in C:\Users\Nico\source\repos\MassTransit\src\MassTransit\Transports\RiderCollection.cs:line 179
   at MassTransit.Transports.RiderCollection.StopAgent(StopContext context) in C:\Users\Nico\source\repos\MassTransit\src\MassTransit\Transports\RiderCollection.cs:line 138
   at MassTransit.Middleware.Agent.Stop(StopContext context) in C:\Users\Nico\source\repos\MassTransit\src\MassTransit.Abstractions\Middleware\Middleware\Agent.cs:line 87
   at MassTransit.Transports.BaseHost.Stop(CancellationToken cancellationToken) in C:\Users\Nico\source\repos\MassTransit\src\MassTransit\Transports\BaseHost.cs:line 152
   at MassTransit.MassTransitBus.Handle.StopAsync(CancellationToken cancellationToken) in C:\Users\Nico\source\repos\MassTransit\src\MassTransit\MassTransitBus.cs:line 384
   --- End of inner exception stack trace ---
   --- End of inner exception stack trace ---
   at MassTransit.KafkaIntegration.Middleware.KafkaConsumerFilter`2.Send(ConsumerContext context, IPipe`1 next) in C:\Users\Nico\source\repos\MassTransit\src\Transports\MassTransit.KafkaIntegration\KafkaIntegration\Middleware\KafkaConsumerFilter.cs:line 34
   at MassTransit.KafkaIntegration.Middleware.KafkaConsumerFilter`2.Send(ConsumerContext context, IPipe`1 next) in C:\Users\Nico\source\repos\MassTransit\src\Transports\MassTransit.KafkaIntegration\KafkaIntegration\Middleware\KafkaConsumerFilter.cs:line 43
   at MassTransit.Middleware.ReceiveEndpointDependencyFilter`1.Send(TContext context, IPipe`1 next) in C:\Users\Nico\source\repos\MassTransit\src\MassTransit\Middleware\ReceiveEndpointDependencyFilter.cs:line 23
   at MassTransit.Agents.PipeContextSupervisor`1.Send(IPipe`1 pipe, CancellationToken cancellationToken) in C:\Users\Nico\source\repos\MassTransit\src\MassTransit\Agents\PipeContextSupervisor.cs:line 55
   at MassTransit.Agents.PipeContextSupervisor`1.Send(IPipe`1 pipe, CancellationToken cancellationToken) in C:\Users\Nico\source\repos\MassTransit\src\MassTransit\Agents\PipeContextSupervisor.cs:line 61
   at MassTransit.Agents.PipeContextSupervisor`1.Send(IPipe`1 pipe, CancellationToken cancellationToken) in C:\Users\Nico\source\repos\MassTransit\src\MassTransit\Agents\PipeContextSupervisor.cs:line 68
   at MassTransit.Transports.ReceiveTransport`1.ReceiveTransportAgent.RunTransport() in C:\Users\Nico\source\repos\MassTransit\src\MassTransit\Transports\ReceiveTransport.cs:line 195
   --- End of inner exception stack trace ---
   at MassTransit.Transports.ReceiveTransport`1.ReceiveTransportAgent.RunTransport() in C:\Users\Nico\source\repos\MassTransit\src\MassTransit\Transports\ReceiveTransport.cs:line 208
   at MassTransit.Transports.ReceiveTransport`1.ReceiveTransportAgent.Run() in C:\Users\Nico\source\repos\MassTransit\src\MassTransit\Transports\ReceiveTransport.cs:line 128
19:45:12.493-W Bus stop faulted: loopback://localhost/

System.AggregateException: One or more errors occurred. (One or more errors occurred. (Local: Broker handle destroyed))
 ---> System.AggregateException: One or more errors occurred. (Local: Broker handle destroyed)
 ---> Confluent.Kafka.KafkaException: Local: Broker handle destroyed
   at Confluent.Kafka.Impl.SafeKafkaHandle.ConsumerClose()
   at Confluent.Kafka.Consumer`2.Close()
   at MassTransit.KafkaIntegration.KafkaMessageConsumer`2.ActiveAndActualAgentsCompleted(StopContext context) in C:\Users\Nico\source\repos\MassTransit\src\Transports\MassTransit.KafkaIntegration\KafkaIntegration\KafkaMessageConsumer.cs:line 117
   --- End of inner exception stack trace ---
   --- End of inner exception stack trace ---
   at MassTransit.Middleware.Agent.Stop(StopContext context) in C:\Users\Nico\source\repos\MassTransit\src\MassTransit.Abstractions\Middleware\Middleware\Agent.cs:line 87
   at MassTransit.Middleware.Supervisor.StopSupervisor(StopSupervisorContext context) in C:\Users\Nico\source\repos\MassTransit\src\MassTransit.Abstractions\Middleware\Middleware\Supervisor.cs:line 102
   at MassTransit.Middleware.Agent.Stop(StopContext context) in C:\Users\Nico\source\repos\MassTransit\src\MassTransit.Abstractions\Middleware\Middleware\Agent.cs:line 87
   at MassTransit.Middleware.Supervisor.StopSupervisor(StopSupervisorContext context) in C:\Users\Nico\source\repos\MassTransit\src\MassTransit.Abstractions\Middleware\Middleware\Supervisor.cs:line 102
   at MassTransit.Middleware.Agent.Stop(StopContext context) in C:\Users\Nico\source\repos\MassTransit\src\MassTransit.Abstractions\Middleware\Middleware\Agent.cs:line 87
   at MassTransit.Transports.TransportPipeContextSupervisor`1.StopSupervisor(StopSupervisorContext context) in C:\Users\Nico\source\repos\MassTransit\src\MassTransit\Transports\TransportPipeContextSupervisor.cs:line 41
   at MassTransit.Middleware.Agent.Stop(StopContext context) in C:\Users\Nico\source\repos\MassTransit\src\MassTransit.Abstractions\Middleware\Middleware\Agent.cs:line 87
   at MassTransit.Transports.ReceiveTransport`1.ReceiveTransportAgent.StopAgent(StopContext context) in C:\Users\Nico\source\repos\MassTransit\src\MassTransit\Transports\ReceiveTransport.cs:line 105
   at MassTransit.Middleware.Agent.Stop(StopContext context) in C:\Users\Nico\source\repos\MassTransit\src\MassTransit.Abstractions\Middleware\Middleware\Agent.cs:line 87
   at MassTransit.Transports.ReceiveEndpoint.Stop(Boolean removed, CancellationToken cancellationToken) in C:\Users\Nico\source\repos\MassTransit\src\MassTransit\Transports\ReceiveEndpoint.cs:line 175
   at MassTransit.Transports.ReceiveEndpointCollection.StopAgent(StopContext context) in C:\Users\Nico\source\repos\MassTransit\src\MassTransit\Transports\ReceiveEndpointCollection.cs:line 106
   at MassTransit.Middleware.Agent.Stop(StopContext context) in C:\Users\Nico\source\repos\MassTransit\src\MassTransit.Abstractions\Middleware\Middleware\Agent.cs:line 87
   at MassTransit.KafkaIntegration.KafkaRider.RiderAgent.StopAgent(StopContext context) in C:\Users\Nico\source\repos\MassTransit\src\Transports\MassTransit.KafkaIntegration\KafkaIntegration\KafkaRider.cs:line 113
   at MassTransit.Middleware.Agent.Stop(StopContext context) in C:\Users\Nico\source\repos\MassTransit\src\MassTransit.Abstractions\Middleware\Middleware\Agent.cs:line 87
   at MassTransit.Transports.RiderCollection.Handle.StopAsync(Boolean remove, CancellationToken cancellationToken) in C:\Users\Nico\source\repos\MassTransit\src\MassTransit\Transports\RiderCollection.cs:line 179
   at MassTransit.Transports.RiderCollection.StopAgent(StopContext context) in C:\Users\Nico\source\repos\MassTransit\src\MassTransit\Transports\RiderCollection.cs:line 138
   at MassTransit.Middleware.Agent.Stop(StopContext context) in C:\Users\Nico\source\repos\MassTransit\src\MassTransit.Abstractions\Middleware\Middleware\Agent.cs:line 87
   at MassTransit.Transports.BaseHost.Stop(CancellationToken cancellationToken) in C:\Users\Nico\source\repos\MassTransit\src\MassTransit\Transports\BaseHost.cs:line 152
   at MassTransit.MassTransitBus.Handle.StopAsync(CancellationToken cancellationToken) in C:\Users\Nico\source\repos\MassTransit\src\MassTransit\MassTransitBus.cs:line 384


----- sometimes this:


  failed after: 0 retries
  Expected: True
  But was:  False
   at MassTransit.KafkaIntegration.Tests.HealthCheck_Specs.should_be_degraded_when_lost_connection() in C:\Users\Nico\source\repos\MassTransit\tests\MassTransit.KafkaIntegration.Tests\HealthCheck_Specs.cs:line 96

   at MassTransit.KafkaIntegration.Tests.HealthCheck_Specs.should_be_degraded_when_lost_connection() in C:\Users\Nico\source\repos\MassTransit\tests\MassTransit.KafkaIntegration.Tests\HealthCheck_Specs.cs:line 96
   at MassTransit.KafkaIntegration.Tests.HealthCheck_Specs.should_be_degraded_when_lost_connection() in C:\Users\Nico\source\repos\MassTransit\tests\MassTransit.KafkaIntegration.Tests\HealthCheck_Specs.cs:line 93
   at NUnit.Framework.Internal.TaskAwaitAdapter.GenericAdapter`1.BlockUntilCompleted()
   at NUnit.Framework.Internal.MessagePumpStrategy.NoMessagePumpStrategy.WaitForCompletion(AwaitAdapter awaiter)
   at NUnit.Framework.Internal.AsyncToSyncAdapter.Await(Func`1 invoke)
   at NUnit.Framework.Internal.Commands.TestMethodCommand.RunTestMethod(TestExecutionContext context)
   at NUnit.Framework.Internal.Commands.TestMethodCommand.Execute(TestExecutionContext context)
   at NUnit.Framework.Internal.Commands.BeforeAndAfterTestCommand.<>c__DisplayClass1_0.<Execute>b__0()
   at NUnit.Framework.Internal.Commands.DelegatingTestCommand.RunTestMethodInThreadAbortSafeZone(TestExecutionContext context, Action action)


19:48:01.835-D Running health checks
19:48:01.838-D Running health check masstransit-bus
19:48:01.849-E Health check masstransit-bus with status Unhealthy completed after 10.5955ms with message 'Not ready: not started'
19:48:01.850-D Health check data for masstransit-bus:
    Endpoints: loopback://localhost/DESKTOPFVPA0MK_ReSharperTestRunner_bus_a1ioyyguhzcf79g8bdphw6qiy4: Unhealthy - not ready, loopback://localhost/kafka/health-check/HealthCheck_Specs: Unhealthy - not ready
    HealthCheckName: masstransit-bus

19:48:01.853-D Health check processing with combined status Unhealthy completed after 17.3761ms
{
  "status": "Unhealthy",
  "results": {
    "masstransit-bus": {
      "status": "Unhealthy",
      "description": "Not ready: not started",
      "data": {
        "Endpoints": {
          "loopback://localhost/DESKTOPFVPA0MK_ReSharperTestRunner_bus_a1ioyyguhzcf79g8bdphw6qiy4": {
            "status": "Unhealthy",
            "description": "not ready"
          },
          "loopback://localhost/kafka/health-check/HealthCheck_Specs": {
            "status": "Unhealthy",
            "description": "not ready"
          }
        }
      }
    }
  }
}
19:48:02.912-I Creating topics: health-check with 1 partitions and 1 replicas
19:48:03.051-D Starting bus instances: IBus
19:48:03.051-D Starting bus: loopback://localhost/
19:48:03.052-D Endpoint Ready: loopback://localhost/DESKTOPFVPA0MK_ReSharperTestRunner_bus_a1ioyyguhzcf79g8bdphw6qiy4
19:48:03.067-D Endpoint Ready: loopback://localhost/kafka/health-check/HealthCheck_Specs
19:48:03.067-I Bus started: loopback://localhost/
19:48:03.193-D Running health checks
19:48:03.194-D Running health check masstransit-bus
19:48:03.194-D Health check masstransit-bus with status Healthy completed after 0.3872ms with message 'Ready'
19:48:03.194-D Health check data for masstransit-bus:
    Endpoints: loopback://localhost/DESKTOPFVPA0MK_ReSharperTestRunner_bus_a1ioyyguhzcf79g8bdphw6qiy4: Healthy - ready, loopback://localhost/kafka/health-check/HealthCheck_Specs: Healthy - ready
    HealthCheckName: masstransit-bus

19:48:03.194-D Health check processing with combined status Healthy completed after 0.6254ms
{
  "status": "Healthy",
  "results": {
    "masstransit-bus": {
      "status": "Healthy",
      "description": "Ready",
      "data": {
        "Endpoints": {
          "loopback://localhost/DESKTOPFVPA0MK_ReSharperTestRunner_bus_a1ioyyguhzcf79g8bdphw6qiy4": {
            "status": "Healthy",
            "description": "ready"
          },
          "loopback://localhost/kafka/health-check/HealthCheck_Specs": {
            "status": "Healthy",
            "description": "ready"
          }
        }
      }
    }
  }
}
19:48:04.505-W Consumer [rdkafka-f32937a4-4f1b-486d-8f91-8ca69b597931] error (Local_Transport): GroupCoordinator: localhost:9092: Disconnected (after 1404ms in state UP) on health-check
19:48:04.505-W Consumer [rdkafka-f32937a4-4f1b-486d-8f91-8ca69b597931] error (Local_Transport): localhost:9092/1: Disconnected: verify that security.protocol is correctly configured, broker might require SASL authentication (after 1418ms in state UP) on health-check
19:48:04.505-W Consumer [rdkafka-f32937a4-4f1b-486d-8f91-8ca69b597931] error (Local_AllBrokersDown): 2/2 brokers are down on health-check
19:48:04.506-W Consumer [rdkafka-f32937a4-4f1b-486d-8f91-8ca69b597931] error (Local_Transport): GroupCoordinator: localhost:9092: Disconnected (after 1ms in state APIVERSION_QUERY, 1 identical error(s) suppressed) on health-check
19:48:04.522-W Consumer [rdkafka-f32937a4-4f1b-486d-8f91-8ca69b597931] error (Local_Transport): localhost:9092/1: Disconnected while requesting ApiVersion: might be caused by incorrect security.protocol configuration (connecting to a SSL listener?) or broker version is < 0.10 (see api.version.request) (after 0ms in state APIVERSION_QUERY) on health-check
19:48:04.584-D Consumer Stopping: loopback://localhost/kafka/health-check/HealthCheck_Specs (Consume Loop Exited)
19:48:04.590-D Endpoint Completed: loopback://localhost/kafka/health-check/HealthCheck_Specs
19:48:04.594-D Consumer Completed: loopback://localhost/kafka/health-check/HealthCheck_Specs: 0 received, 0 concurrent
19:48:05.596-D Endpoint Ready: loopback://localhost/kafka/health-check/HealthCheck_Specs
19:48:05.598-W Consumer [] error (Local_Transport): localhost:9092/bootstrap: Disconnected while requesting ApiVersion: might be caused by incorrect security.protocol configuration (connecting to a SSL listener?) or broker version is < 0.10 (see api.version.request) (after 1ms in state APIVERSION_QUERY) on health-check
19:48:05.598-W Consumer [] error (Local_AllBrokersDown): 1/1 brokers are down on health-check
19:48:05.702-D Consumer Stopping: loopback://localhost/kafka/health-check/HealthCheck_Specs (Consume Loop Exited)
19:48:05.703-D Endpoint Completed: loopback://localhost/kafka/health-check/HealthCheck_Specs
19:48:05.703-D Consumer Completed: loopback://localhost/kafka/health-check/HealthCheck_Specs: 0 received, 0 concurrent
19:48:06.716-D Endpoint Ready: loopback://localhost/kafka/health-check/HealthCheck_Specs
19:48:06.718-W Consumer [] error (Local_Transport): localhost:9092/bootstrap: Disconnected while requesting ApiVersion: might be caused by incorrect security.protocol configuration (connecting to a SSL listener?) or broker version is < 0.10 (see api.version.request) (after 0ms in state APIVERSION_QUERY) on health-check
19:48:06.718-W Consumer [] error (Local_AllBrokersDown): 1/1 brokers are down on health-check
19:48:06.823-D Consumer Stopping: loopback://localhost/kafka/health-check/HealthCheck_Specs (Consume Loop Exited)
19:48:06.824-D Endpoint Completed: loopback://localhost/kafka/health-check/HealthCheck_Specs
19:48:06.824-D Consumer Completed: loopback://localhost/kafka/health-check/HealthCheck_Specs: 0 received, 0 concurrent
19:48:07.838-D Endpoint Ready: loopback://localhost/kafka/health-check/HealthCheck_Specs
19:48:07.840-W Consumer [] error (Local_Transport): localhost:9092/bootstrap: Disconnected while requesting ApiVersion: might be caused by incorrect security.protocol configuration (connecting to a SSL listener?) or broker version is < 0.10 (see api.version.request) (after 1ms in state APIVERSION_QUERY) on health-check
19:48:07.840-W Consumer [] error (Local_AllBrokersDown): 1/1 brokers are down on health-check
19:48:07.945-D Consumer Stopping: loopback://localhost/kafka/health-check/HealthCheck_Specs (Consume Loop Exited)
19:48:07.946-D Endpoint Completed: loopback://localhost/kafka/health-check/HealthCheck_Specs
19:48:07.946-D Consumer Completed: loopback://localhost/kafka/health-check/HealthCheck_Specs: 0 received, 0 concurrent
19:48:08.951-D Endpoint Ready: loopback://localhost/kafka/health-check/HealthCheck_Specs
19:48:08.953-W Consumer [] error (Local_Transport): localhost:9092/bootstrap: Disconnected while requesting ApiVersion: might be caused by incorrect security.protocol configuration (connecting to a SSL listener?) or broker version is < 0.10 (see api.version.request) (after 1ms in state APIVERSION_QUERY) on health-check
19:48:08.953-W Consumer [] error (Local_AllBrokersDown): 1/1 brokers are down on health-check
19:48:09.058-D Consumer Stopping: loopback://localhost/kafka/health-check/HealthCheck_Specs (Consume Loop Exited)
19:48:09.059-D Endpoint Completed: loopback://localhost/kafka/health-check/HealthCheck_Specs
19:48:09.059-D Consumer Completed: loopback://localhost/kafka/health-check/HealthCheck_Specs: 0 received, 0 concurrent
19:48:10.073-D Endpoint Ready: loopback://localhost/kafka/health-check/HealthCheck_Specs
19:48:10.075-W Consumer [] error (Local_Transport): localhost:9092/bootstrap: Disconnected while requesting ApiVersion: might be caused by incorrect security.protocol configuration (connecting to a SSL listener?) or broker version is < 0.10 (see api.version.request) (after 1ms in state APIVERSION_QUERY) on health-check
19:48:10.075-W Consumer [] error (Local_AllBrokersDown): 1/1 brokers are down on health-check
19:48:10.181-D Consumer Stopping: loopback://localhost/kafka/health-check/HealthCheck_Specs (Consume Loop Exited)
19:48:10.181-D Endpoint Completed: loopback://localhost/kafka/health-check/HealthCheck_Specs
19:48:10.181-D Consumer Completed: loopback://localhost/kafka/health-check/HealthCheck_Specs: 0 received, 0 concurrent
19:48:11.196-D Endpoint Ready: loopback://localhost/kafka/health-check/HealthCheck_Specs
19:48:11.198-W Consumer [] error (Local_Transport): localhost:9092/bootstrap: Disconnected while requesting ApiVersion: might be caused by incorrect security.protocol configuration (connecting to a SSL listener?) or broker version is < 0.10 (see api.version.request) (after 0ms in state APIVERSION_QUERY) on health-check
19:48:11.198-W Consumer [] error (Local_AllBrokersDown): 1/1 brokers are down on health-check
19:48:11.305-D Consumer Stopping: loopback://localhost/kafka/health-check/HealthCheck_Specs (Consume Loop Exited)
19:48:11.306-D Endpoint Completed: loopback://localhost/kafka/health-check/HealthCheck_Specs
19:48:11.306-D Consumer Completed: loopback://localhost/kafka/health-check/HealthCheck_Specs: 0 received, 0 concurrent
19:48:12.316-D Endpoint Ready: loopback://localhost/kafka/health-check/HealthCheck_Specs
19:48:12.318-W Consumer [] error (Local_Transport): localhost:9092/bootstrap: Disconnected while requesting ApiVersion: might be caused by incorrect security.protocol configuration (connecting to a SSL listener?) or broker version is < 0.10 (see api.version.request) (after 0ms in state APIVERSION_QUERY) on health-check
19:48:12.318-W Consumer [] error (Local_AllBrokersDown): 1/1 brokers are down on health-check
19:48:12.423-D Consumer Stopping: loopback://localhost/kafka/health-check/HealthCheck_Specs (Consume Loop Exited)
19:48:12.424-D Endpoint Completed: loopback://localhost/kafka/health-check/HealthCheck_Specs
19:48:12.424-D Consumer Completed: loopback://localhost/kafka/health-check/HealthCheck_Specs: 0 received, 0 concurrent
19:48:13.432-D Endpoint Ready: loopback://localhost/kafka/health-check/HealthCheck_Specs
19:48:13.434-W Consumer [] error (Local_Transport): localhost:9092/bootstrap: Disconnected while requesting ApiVersion: might be caused by incorrect security.protocol configuration (connecting to a SSL listener?) or broker version is < 0.10 (see api.version.request) (after 0ms in state APIVERSION_QUERY) on health-check
19:48:13.434-W Consumer [] error (Local_AllBrokersDown): 1/1 brokers are down on health-check
19:48:13.538-D Consumer Stopping: loopback://localhost/kafka/health-check/HealthCheck_Specs (Consume Loop Exited)
19:48:13.539-D Endpoint Completed: loopback://localhost/kafka/health-check/HealthCheck_Specs
19:48:13.539-D Consumer Completed: loopback://localhost/kafka/health-check/HealthCheck_Specs: 0 received, 0 concurrent
19:48:14.550-D Endpoint Ready: loopback://localhost/kafka/health-check/HealthCheck_Specs
19:48:14.976-D Running health checks
19:48:14.976-D Running health check masstransit-bus
19:48:14.976-D Health check masstransit-bus with status Healthy completed after 0.0565ms with message 'Ready'
19:48:14.976-D Health check data for masstransit-bus:
    Endpoints: loopback://localhost/DESKTOPFVPA0MK_ReSharperTestRunner_bus_a1ioyyguhzcf79g8bdphw6qiy4: Healthy - ready, loopback://localhost/kafka/health-check/HealthCheck_Specs: Healthy - ready
    HealthCheckName: masstransit-bus

19:48:14.976-D Health check processing with combined status Healthy completed after 0.1565ms
19:48:15.985-D Running health checks
19:48:15.985-D Running health check masstransit-bus
19:48:15.985-D Health check masstransit-bus with status Healthy completed after 0.0775ms with message 'Ready'
19:48:15.985-D Health check data for masstransit-bus:
    Endpoints: loopback://localhost/DESKTOPFVPA0MK_ReSharperTestRunner_bus_a1ioyyguhzcf79g8bdphw6qiy4: Healthy - ready, loopback://localhost/kafka/health-check/HealthCheck_Specs: Healthy - ready
    HealthCheckName: masstransit-bus

19:48:15.985-D Health check processing with combined status Healthy completed after 0.2062ms
19:48:16.606-W Consumer [] error (Local_Transport): localhost:9092/bootstrap: Connect to ipv4#127.0.0.1:9092 failed: Unknown error (after 2056ms in state CONNECT) on health-check
19:48:16.606-W Consumer [] error (Local_AllBrokersDown): 1/1 brokers are down on health-check
19:48:16.621-D Consumer Stopping: loopback://localhost/kafka/health-check/HealthCheck_Specs (Consume Loop Exited)
19:48:16.622-D Endpoint Completed: loopback://localhost/kafka/health-check/HealthCheck_Specs
19:48:16.622-D Consumer Completed: loopback://localhost/kafka/health-check/HealthCheck_Specs: 0 received, 0 concurrent
19:48:16.992-D Running health checks
19:48:16.992-D Running health check masstransit-bus
19:48:16.993-W Health check masstransit-bus with status Degraded completed after 0.3667ms with message 'Degraded Endpoints: HealthCheck_Specs'
19:48:16.993-D Health check data for masstransit-bus:
    Endpoints: loopback://localhost/DESKTOPFVPA0MK_ReSharperTestRunner_bus_a1ioyyguhzcf79g8bdphw6qiy4: Healthy - ready, loopback://localhost/kafka/health-check/HealthCheck_Specs: Degraded - stopped (delivered 0 messages)
    HealthCheckName: masstransit-bus

19:48:16.993-D Health check processing with combined status Degraded completed after 0.5306ms
19:48:17.629-D Endpoint Ready: loopback://localhost/kafka/health-check/HealthCheck_Specs
{
  "status": "Degraded",
  "results": {
    "masstransit-bus": {
      "status": "Degraded",
      "description": "Degraded Endpoints: HealthCheck_Specs",
      "data": {
        "Endpoints": {
          "loopback://localhost/DESKTOPFVPA0MK_ReSharperTestRunner_bus_a1ioyyguhzcf79g8bdphw6qiy4": {
            "status": "Healthy",
            "description": "ready"
          },
          "loopback://localhost/kafka/health-check/HealthCheck_Specs": {
            "status": "Degraded",
            "description": "stopped (delivered 0 messages)"
          }
        }
      }
    }
  }
}
19:48:18.002-D Running health checks
19:48:18.002-D Running health check masstransit-bus
19:48:18.002-D Health check masstransit-bus with status Healthy completed after 0.0661ms with message 'Ready'
19:48:18.002-D Health check data for masstransit-bus:
    Endpoints: loopback://localhost/DESKTOPFVPA0MK_ReSharperTestRunner_bus_a1ioyyguhzcf79g8bdphw6qiy4: Healthy - ready, loopback://localhost/kafka/health-check/HealthCheck_Specs: Healthy - ready
    HealthCheckName: masstransit-bus

19:48:18.002-D Health check processing with combined status Healthy completed after 0.1551ms
19:48:18.017-D Stopping bus instances: IBus
19:48:18.018-D Stopping bus: loopback://localhost/
19:48:18.021-D Endpoint Stopping: loopback://localhost/kafka/health-check/HealthCheck_Specs
19:48:18.022-D Consumer Stopping: loopback://localhost/kafka/health-check/HealthCheck_Specs (Stop Receive Transport)
19:48:18.065-D Endpoint Completed: loopback://localhost/kafka/health-check/HealthCheck_Specs
19:48:18.065-D Consumer Completed: loopback://localhost/kafka/health-check/HealthCheck_Specs: 0 received, 0 concurrent
19:48:18.068-D Endpoint Stopping: loopback://localhost/DESKTOPFVPA0MK_ReSharperTestRunner_bus_a1ioyyguhzcf79g8bdphw6qiy4
19:48:18.068-D Consumer Stopping: loopback://localhost/DESKTOPFVPA0MK_ReSharperTestRunner_bus_a1ioyyguhzcf79g8bdphw6qiy4 (Stop Receive Transport)
19:48:18.071-D Endpoint Completed: loopback://localhost/DESKTOPFVPA0MK_ReSharperTestRunner_bus_a1ioyyguhzcf79g8bdphw6qiy4
19:48:18.071-D Consumer Completed: loopback://localhost/DESKTOPFVPA0MK_ReSharperTestRunner_bus_a1ioyyguhzcf79g8bdphw6qiy4: 0 received, 0 concurrent
19:48:18.073-I Bus stopped: loopback://localhost/

Link to repository that demonstrates/reproduces the issue

https://github.com/nicomoya123/MassTransit/blob/8f5cbc94cd78db43181532b0f95c4a968b20ebee/tests/MassTransit.KafkaIntegration.Tests/HealthCheck_Specs.cs#L55C11-L55C13

About this issue

Original URL
State: closed
Created 9 months ago
Comments: 33 (17 by maintainers)

Most upvoted comments

Are you stopping the contatiners post partitions assignment ?

Starting all
healthy
Stopping containers
should not become healthy anymore

I pretty much sure your test was doing the same thing

NooNameR on Oct 15, 2023