stan.net: Cannot re-connect to server after nats-streaming-server is restarted

I have encountered the same situation in version 0.3.0. when my Stan connection entered to Connection Closed Mode it never comes to connect state and all publish requests throw the Connection Close Exception.

this is my code to register stan Connection:

public static class Extensions
    {
        private static IStanConnection _stanConnection; 
        public static IServiceCollection AddStanClient(this IServiceCollection services, StanSettings settings)
        {

            var stanOptions = StanOptions.GetDefaultOptions();
            var natsOptions = ConnectionFactory.GetDefaultOptions();
            var stanClientId = $"{settings.ClientId}-{Guid.NewGuid()}";
            var stanClusterId = settings.ClusterId;
            natsOptions.Servers = settings.NatsServers.ToArray();
            natsOptions.AllowReconnect = true;
            natsOptions.MaxReconnect = Options.ReconnectForever;
            natsOptions.PingInterval = 2000;
            natsOptions.MaxPingsOut = 2;
            natsOptions.ReconnectWait = 1000;
            natsOptions.ReconnectBufferSize = Options.ReconnectBufferSizeUnbounded;
            natsOptions.Timeout = 4000;
             
            natsOptions.ClosedEventHandler += (object sender, ConnEventArgs arg) =>
            {
                Console.WriteLine($"connection lost on connection {arg.Conn.ConnectedId}" +
                                  $"{Environment.NewLine} {arg.Error.StackTrace}");
            };

            natsOptions.ReconnectedEventHandler += (object sender, ConnEventArgs arg) =>
            {
                stanOptions.NatsConn = arg.Conn;
                Console.WriteLine($"connection Recovered by connection id {arg.Conn.ConnectedId}");
            }; 
            
            
            stanOptions.NatsConn = new ConnectionFactory().CreateConnection(natsOptions);
            stanOptions.ConnectionLostEventHandler += (object sender, StanConnLostHandlerArgs arg) => {
                Console.WriteLine($"connection lost on connection {arg.Connection.NATSConnection.ConnectedId}" +
                                  $"{Environment.NewLine} {arg.ConnectionException.StackTrace}"); 
            };

            _stanConnection = new StanConnectionFactory().CreateConnection(stanClusterId, stanClientId, stanOptions); 
            services.AddSingleton<IStanConnection>( _stanConnection);

            return services;
        }
    }

and I use this connection like this :

static Task Main(string[] args)
        {
            var host = CreateHostBuilder(args).Build();

            var scope = host.Services.CreateScope();
            

            var watch = new Stopwatch();
            watch.Start();
            while (true)
            {
                var stanConnection = scope.ServiceProvider.GetRequiredService<IStanConnection>();
                try
                {
                    stanConnection.Publish("test", new byte[0x1]);
                    Console.WriteLine("{0}. Published message", watch.Elapsed);
                }
                catch (Exception e)
                {
                    Console.WriteLine("{0} - Type:{1}. On Publish: exception message: {2}", watch.Elapsed, e.GetType(), e.Message);
                }
                finally
                {
                    Thread.Sleep(1000);
                }
            }
            Console.WriteLine("Hello World!");

            return host.StartAsync();
        }


        static IHostBuilder CreateHostBuilder(string[] args) =>
            Host.CreateDefaultBuilder(args)
                .ConfigureServices((_, services) =>
                    services.AddStanClient(new StanSettings
                    {
                        ClientId = "uniq123",
                        ClusterId = "test-cluster",
                        NatsServers = new List<string>{"nats://localhost:4222"}
                    })); 

    }
    ```
    and this is the output of my code when I stop the Stan service and after a while, I start it again :
    
00:00:13.7476360. Published message
00:00:14.7542784. Published message
00:00:15.7602945. Published message
00:00:16.7658160. Published message
00:00:17.7862428. Published message
00:00:18.8041595. Published message
00:00:20.6005542 - Type:STAN.Client.StanConnectionException. On Publish: exception message: The NATS connection is reconnecting
00:00:21.6487055 - Type:STAN.Client.StanConnectionException. On Publish: exception message: The NATS connection is reconnecting
00:00:22.7012163 - Type:STAN.Client.StanConnectionException. On Publish: exception message: The NATS connection is reconnecting
00:00:23.7444252 - Type:STAN.Client.StanConnectionException. On Publish: exception message: The NATS connection is reconnecting
00:00:24.7905985 - Type:STAN.Client.StanConnectionException. On Publish: exception message: The NATS connection is reconnecting
00:00:25.8348325 - Type:STAN.Client.StanConnectionException. On Publish: exception message: The NATS connection is reconnecting
00:00:26.8816522 - Type:STAN.Client.StanConnectionException. On Publish: exception message: The NATS connection is reconnecting
00:00:27.9295129 - Type:STAN.Client.StanConnectionException. On Publish: exception message: The NATS connection is reconnecting
00:00:28.9737369 - Type:STAN.Client.StanConnectionException. On Publish: exception message: The NATS connection is reconnecting
00:00:30.0154864 - Type:STAN.Client.StanConnectionException. On Publish: exception message: The NATS connection is reconnecting
00:00:31.0607351 - Type:STAN.Client.StanConnectionException. On Publish: exception message: The NATS connection is reconnecting
00:00:32.1057689 - Type:STAN.Client.StanConnectionException. On Publish: exception message: The NATS connection is reconnecting
00:00:33.1495434 - Type:STAN.Client.StanConnectionException. On Publish: exception message: The NATS connection is reconnecting
00:00:34.1919119 - Type:STAN.Client.StanConnectionException. On Publish: exception message: The NATS connection is reconnecting
00:00:35.2404412 - Type:STAN.Client.StanConnectionClosedException. On Publish: exception message: Connection closed.
00:00:36.3016650 - Type:STAN.Client.StanConnectionClosedException. On Publish: exception message: Connection closed.
00:00:37.3464719 - Type:STAN.Client.StanConnectionClosedException. On Publish: exception message: Connection closed.
00:00:38.3887135 - Type:STAN.Client.StanConnectionClosedException. On Publish: exception message: Connection closed.
00:00:39.4637382 - Type:STAN.Client.StanConnectionClosedException. On Publish: exception message: Connection closed.
00:00:40.5152720 - Type:STAN.Client.StanConnectionClosedException. On Publish: exception message: Connection closed.
00:00:41.5576260 - Type:STAN.Client.StanConnectionClosedException. On Publish: exception message: Connection closed.
connection Recovered by connection id NDIVMUUHKJN644WRMAFRWW7GB5TD3ZPYVE2CIJWFMCLJDXCWIBDURO6L
00:00:42.6050295 - Type:STAN.Client.StanConnectionClosedException. On Publish: exception message: Connection closed.
00:00:43.6454686 - Type:STAN.Client.StanConnectionClosedException. On Publish: exception message: Connection closed.
00:00:44.6864267 - Type:STAN.Client.StanConnectionClosedException. On Publish: exception message: Connection closed.
00:00:45.7310466 - Type:STAN.Client.StanConnectionClosedException. On Publish: exception message: Connection closed.

Many thanks for your time spent on this issue! Is kind of odd and I can never imagine a practical situation in which I would not like that a STAN client to recover when ping failed but the enclosed NATS connection is alive. I was led by the impression that STAN connection should be long lived (and act as a singleton) similar to NATS connection object. In my prod app, I was registering the STAN connection as a singleton with the DI container. So, to conclude: you are telling me that:

Either:

1. disable (set very high timeout/no of pings) ping on STAN connection, but leave it on for NATS connection
2. or, recreate STAN connection every time when the underlying connection to NATS is closed.

Could you reconsider the architectural decision of not recovering STAN connection in case that NATS streaming server is running in embedded mode and STAN client using a not self-managed NATS connection and make somehow transparent for the STAN client how the NATS server is ran?

I will try to reproduce this test using the java streaming server client and see if the behaviour is similar.