openssl: Fatal - Unexpected Message returned with OpenSSL 3.0

Checked, /etc/ssl/openssl.cnf on the servers is vanilla, checked against another install. postconf output shows no settings referring to an openssl.cnf, and shows tls_ssl_options = (empty).

No need. I’ve identified the issue. It is an OpenSSL 3.0 session cache management regression.

You don’t need to downgrade Postfix or OpenSSL. It suffices to just set:

smtpd_tls_session_cache_database =

which is its default and recommended value. Yes, clients that don’t support tickets won’t be able to make effective use of TLS resumption, but that’s unlikely to be a major issue under realistic message rates. See the follow-up to the “postfix-users” list.

Shared the -s 0 pcap (3.7MB) via email.

Checked a failing transmission again, and the ec_point_formats don’t show up in Wireshark’s UI for the ServerHello, e.g. frame 206 in the pcap. The only extension I see is renegotiation_info. The “random” in that ServerHello seems odd, rendered as text it shows DDOWNGRD?

0000   6d 44 ed c5 31 c8 f4 82 be ba 09 61 0b 23 5d 7d   mD..1......a.#]}
0010   96 7b bf 44 44 4f 57 4e 47 52 44 01               .{.DDOWNGRD.

Let me know if there’s anything I can do to help here.

A successful Server Hello response does contain the ec_point_formats. Note that here the session id is echoed.

TLSv1.2 Record Layer: Handshake Protocol: Server Hello
    Content Type: Handshake (22)
    Version: TLS 1.2 (0x0303)
    Length: 89
    Handshake Protocol: Server Hello
        Handshake Type: Server Hello (2)
        Length: 85
        Version: TLS 1.2 (0x0303)
        Random: 1288f92d608a9b53273ff9dc4bdf8ba1b218a7f5da9769f6444f574e47524401
            GMT Unix Time: Nov  9, 1979 04:36:13.000000000 W. Europe Standard Time
            Random Bytes: 608a9b53273ff9dc4bdf8ba1b218a7f5da9769f6444f574e47524401
        Session ID Length: 32
        Session ID: e9bcd6a0eb609186631f6544acd285173ce763660df481634d6349ac4f9af263
        Cipher Suite: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (0xc02f)
        Compression Method: null (0)
        Extensions Length: 13
        Extension: renegotiation_info (len=1)
            Type: renegotiation_info (65281)
            Length: 1
            Renegotiation Info extension
                Renegotiation info extension length: 0
        Extension: ec_point_formats (len=4)
            Type: ec_point_formats (11)
            Length: 4
            EC point formats Length: 3
            Elliptic curves point formats (3)
                EC point format: uncompressed (0)
                EC point format: ansiX962_compressed_prime (1)
                EC point format: ansiX962_compressed_char2 (2)
        [JA3S Fullstring: 771,49199,65281-11]
        [JA3S: 303951d4c50efb2e991652225a6f02b1]

I know barely enough to be dangerous, clueless would not surprise me! It was a bit of a crash job when I created these servers some years ago as a Postfix noob. The relevant Postfix TLS config (from our code repo) below.

smtpd_tls_session_cache_database = btree:${data_directory}/smtpd_scache
# 2019-08-28, https://ssl-config.mozilla.org/#server=postfix&server-version=3.3.0&config=old
smtpd_use_tls = yes
smtpd_tls_security_level = may 
smtpd_tls_auth_only = yes
smtpd_tls_cert_file = /etc/ssl/certs/server.cer
smtpd_tls_key_file = /etc/ssl/priv/server.key
smtpd_tls_mandatory_protocols = !SSLv2, !SSLv3
smtpd_tls_protocols = !SSLv2, !SSLv3
smtpd_tls_mandatory_ciphers = medium
# not actually 1024 bits, this applies to all DHE >= 1024 bits
smtpd_tls_dh1024_param_file = /etc/ssl/dhparam.pem
tls_medium_cipherlist = ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-CHACHA20-POLY1305:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES256-SHA256:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA
tls_preempt_cipherlist = yes
# End Mozilla config
smtpd_tls_loglevel = 1
smtpd_tls_received_header = yes
## END TLS Server config

I’ll need to shorten de pcap a bit to send via email, will get to that tomorrow! I can then check the postconf output as well. The code repo has neither tls_ssl_options nor smtpd_tls_always_issue_session_ids.

Again, thank you very much for spending your valuable time on this!

Indeed the Postfix tlsproxy supports multiple concurrent non-blocking SSL connections in a single thread via an event loop. The current not_resumable bit is a fragile mechanism that needs rework, but the internal cache is now exhibiting a few too many warts and needs some TLC. 😦

If neither @tmshort nor @mattcaswell volunteers to craft the PR the way they prefer, I might have to do it myself, but full disclosure, I might be inclined to rip out the time ordering code. Is it demonstrably a win? What is its justification?

I don’t see a compelling case made in #8687 as to why it was a good idea.

Technical nit: Postfix does not respond to ClientHello messages, OpenSSL does.

That aside, you’ve not been very specific in your error report:

• Is the client offering TLS 1.3 in its supported_versions extension, or only TLS 1.2?
• Is the Java client sending a non-empty session id?
• Do you have tshark decodes of the client and server TLS HELLO messages? (Please collect and post, or else provide a full PCAP capture of a failed handshake, without packet truncation, i.e. -s 0 option of tcpdump).

With TLS 1.3, the server needs to echo the client’s “legacy” session id. With TLS 1.2, the server signals non-support for session resumption by returning an empty session-id.

If the Java clients are refusing to interoperate with empty session ids in the server response, barring additional evidence, I rather think the bug is on the Java side. A PCAP file of the failed handshake would be useful here, if you’re willing to “leak” the IP addresses and server SNI involved.