npgsql: UseSslStream causes every other connection to an RDS database to fail

I am using a managed PostgreSQL database hosted on Amazon’s Relational Database Service (RDS). When I create an SSL connection to the database with the UseSslStream option turned on (which is the default), every other connection to the database fails to open. That is, the second, fourth, sixth, etc. connections fail to open. But, if I turn off the UseSslStream option, all the connections successfully open.

This is the exception that I run into.

Unhandled Exception: Npgsql.NpgsqlException: Failed to establish a connection to 'rds.server.internal'. ---> System.IO.IOException: Authentication failed because the remote party has closed the transport stream.
   at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncReq
uest)
   at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.ForceAuthentication(Boolean receiveFirst, Byte[] buffer, AsyncProtocolRequest asyncRe
quest)
   at System.Net.Security.SslState.ProcessAuthentication(LazyAsyncResult lazyResult)
   at System.Net.Security.SslStream.AuthenticateAsClient(String targetHost, X509CertificateCollection clientCertificates
, SslProtocols enabledSslProtocols, Boolean checkCertificateRevocation)
   at Npgsql.NpgsqlClosedState.Open(NpgsqlConnector context, Int32 timeout)
   --- End of inner exception stack trace ---
   at Npgsql.NpgsqlClosedState.Open(NpgsqlConnector context, Int32 timeout)
   at Npgsql.NpgsqlConnector.Open()
   at Npgsql.NpgsqlConnectorPool.GetPooledConnector(NpgsqlConnection Connection)
   at Npgsql.NpgsqlConnectorPool.RequestPooledConnectorInternal(NpgsqlConnection Connection)
   at Npgsql.NpgsqlConnectorPool.RequestConnector(NpgsqlConnection Connection)
   at Npgsql.NpgsqlConnection.Open()
   at SslTest.Program.Main(String[] args) in z:\SslTest\SslTest\Program.cs:line 42

This is what I see in the RDS log.

2015-04-09 01:24:32 UTC:10.0.16.60(52531):[unknown]@[unknown]:[15208]:LOG: could not accept SSL connection: session id context uninitialized
2015-04-09 01:24:32 UTC:10.0.16.60(52533):[unknown]@[unknown]:[15210]:LOG: could not accept SSL connection: session id context uninitialized
2015-04-09 01:24:32 UTC:10.0.16.60(52535):[unknown]@[unknown]:[15212]:LOG: could not accept SSL connection: session id context uninitialized
2015-04-09 01:24:32 UTC:10.0.16.60(52537):[unknown]@[unknown]:[15214]:LOG: could not accept SSL connection: session id context uninitialized

Here is some code to reproduce the issue.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using Npgsql;

namespace SslTest
{
    class Program
    {
        static void Main(string[] args)
        {
            bool useSslStream = false;
            if (args.Contains("-s"))
            {
                useSslStream = true;
            }
            Console.WriteLine("Use SSL Stream: {0}", useSslStream);

            string connectionString = "Server=rds.server.internal;Port=5432;Database=test;User Id=ssl_test;Password='redacted';SSL=true;SslMode=Require;";

            NpgsqlConnection[] connections = new NpgsqlConnection[10];

            for (int i = 0; i < 10; ++i)
            {
                connections[i] = new NpgsqlConnection(connectionString)
                {
                    UseSslStream = useSslStream
                };

                try
                {
                    connections[i].Open();
                    Console.WriteLine("Opened connection {0}", i);
                }
                catch
                {
                    Console.WriteLine("Failed to open connection {0}", i);
                }
            }

            foreach (NpgsqlConnection connection in connections)
            {
                connection.Close();
            }
        }
    }
}

Strangely enough, when I run my code on a PostgreSQL server that I manage myself (i.e. not on RDS), there are no issues. That makes me wonder whether the bug is in Npgsql’s SSL implementation or in Amazon’s PostgreSQL setup.

About this issue

  • Original URL
  • State: closed
  • Created 9 years ago
  • Reactions: 1
  • Comments: 46 (21 by maintainers)

Most upvoted comments

I tried to reproduce the problem myself and it is definitely a bug in RDS. .NET’s SslStream uses Session tickets (https://tools.ietf.org/html/rfc5077) to resume sessions. For some reasons, when SslStream sends a Client Hello TLS message that includes a session ticket it obtained before, RDS immediately closes the TCP stream without any alert message and appends to the log:

LOG: could not accept SSL connection: session id context uninitialized

I have no idea what that means, but according to RFC 5077, if the server doesn’t accept the ticket, it should do a normal full handshake instead and not end the connection. Doing a quick Google search https://www.google.se/search?q=“session+id+context+uninitialized”, it seems the issue is caused when one forgets to initialize OpenSSL in a correct manner.

In the WARNINGS section at https://www.openssl.org/docs/ssl/SSL_CTX_set_session_id_context.html:

If the session id context is not set on an SSL/TLS server and client certificates are used, stored sessions will not be reused but a fatal error will be flagged and the handshake will fail.

SslStream seems to delete session tickets that failed to create a new connection, so that’s why it works each other time (client does not send a resumption ticket) and doesn’t work each other time (client sends a resumption ticket).

The solution seems to simply set UseSslStream to false until Amazon fixes the bug. @matthewrwilton, did you submit an Amazon bug? Otherwise this information above should be helpful.

@roji, @Emill, @jk-shah Thank you for your determined detective work in tracking down such a difficult bug’s root cause. You have all gone above and beyond. I never would have even suspected that the problem would lie in PostgreSQL.