openssl: OpenSSL doesn't properly encode NewSessionTicket when using SSL_CTX_set_tlsext_ticket_key_cb callback that returns 0

I’m working with OpenSSL 1.1.1q.

We have our own build of OpenSSL.

We have custom session ticket handling logic that’s configured through SSL_CTX_set_tlsext_ticket_key_cb. On the server, this callback logic can sometimes return 0 which

indicates that it was not possible to set/retrieve a session ticket and the SSL/TLS session will continue by negotiating a set of cryptographic parameters or using the alternate SSL/TLS resumption mechanism, session ids.

We recently enabled TLS 1.3 on this service. Clients, also on TLS 1.3, started seeing new session ticket processing errors like

error:1416E09F:SSL routines:tls_process_new_session_ticket:length mismatch

This only occurs when the callback from SSL_CTX_set_tlsext_ticket_key_cb returns 0.

Here’s a server repro (that I’ve adapted from https://wiki.openssl.org/index.php/Simple_TLS_Server). You’ll need to plug in your own cert/key paths.

#include <arpa/inet.h>
#include <openssl/err.h>
#include <openssl/ssl.h>
#include <stdio.h>
#include <string.h>
#include <sys/socket.h>
#include <unistd.h>

int create_socket(int port) {
  int s;
  struct sockaddr_in6 addr;
  addr.sin6_family = AF_INET6;
  addr.sin6_port = htons(port);
  addr.sin6_addr = in6addr_any;
  s = socket(AF_INET6, SOCK_STREAM, 0);
  if (s < 0) {
    perror("Unable to create socket");
    exit(EXIT_FAILURE);
  }
  if (bind(s, (struct sockaddr*)&addr, sizeof(addr)) < 0) {
    perror("Unable to bind");
    exit(EXIT_FAILURE);
  }
  if (listen(s, 1) < 0) {
    perror("Unable to listen");
    exit(EXIT_FAILURE);
  }
  return s;
}

SSL_CTX* create_context() {
  const SSL_METHOD* method;
  SSL_CTX* ctx;
  method = TLS_server_method();
  ctx = SSL_CTX_new(method);
  if (!ctx) {
    perror("Unable to create SSL context");
    ERR_print_errors_fp(stderr);
    exit(EXIT_FAILURE);
  }
  return ctx;
}

int handle_session_ticket(
    SSL* s,
    unsigned char key_name[16],
    unsigned char iv[EVP_MAX_IV_LENGTH],
    EVP_CIPHER_CTX* ctx,
    HMAC_CTX* hctx,
    int enc /* when set to 1 */) {
  return 0;
}

void configure_context(SSL_CTX* ctx) {
  if (SSL_CTX_use_certificate_file(ctx, "cert.pem", SSL_FILETYPE_PEM) <= 0) {
    ERR_print_errors_fp(stderr);
    exit(EXIT_FAILURE);
  }
  if (SSL_CTX_use_PrivateKey_file(ctx, "key.pem", SSL_FILETYPE_PEM) <= 0) {
    ERR_print_errors_fp(stderr);
    exit(EXIT_FAILURE);
  }
  SSL_CTX_set_session_cache_mode(
      ctx,
      SSL_SESS_CACHE_SERVER | SSL_SESS_CACHE_NO_INTERNAL |
          SSL_SESS_CACHE_NO_AUTO_CLEAR);
  SSL_CTX_set_tlsext_ticket_key_cb(ctx, handle_session_ticket);
}

int main(int argc, char** argv) {
  int sock;
  SSL_CTX* ctx;
  ctx = create_context();
  configure_context(ctx);
  sock = create_socket(4433);
  while (1) {
    struct sockaddr_in addr;
    unsigned int len = sizeof(addr);
    SSL* ssl;
    const char reply[] = "test\n";
    int client = accept(sock, (struct sockaddr*)&addr, &len);
    if (client < 0) {
      perror("Unable to accept");
      exit(EXIT_FAILURE);
    }
    ssl = SSL_new(ctx);
    SSL_set_fd(ssl, client);

    if (SSL_accept(ssl) <= 0) {
      ERR_print_errors_fp(stderr);
    } else {
      SSL_write(ssl, reply, strlen(reply));
    }

    SSL_shutdown(ssl);
    SSL_free(ssl);
    close(client);
  }

  close(sock);
  SSL_CTX_free(ctx);
}

Once this is running, we can use openssl s_client to repro the client side:

> openssl version
OpenSSL 1.1.1k  FIPS 25 Mar 2021
> openssl s_client -connect [::1]:4433 -msg
[...]
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
[...]
Verify return code: 0 (ok)
---
<<< ??? [length 0005]
    17 03 03 00 1d
<<< TLS 1.3 [length 0001]
    16
<<< TLS 1.3, Handshake [length 000c], NewSessionTicket
    04 00 00 08 00 00 00 00 00 00 00 00
>>> ??? [length 0005]
    17 03 03 00 13
>>> TLS 1.3 [length 0001]
    15
>>> TLS 1.3, Alert [length 0002], fatal decode_error
    02 32
139758528358208:error:1416E09F:SSL routines:tls_process_new_session_ticket:length mismatch:ssl/statem/statem_clnt.c:2592:

If my understanding is correct, in the messages above, the 16 indicates the handshake ContentType, 4 indicates the new_session_ticket HandshakeType, the 00 00 08 are the Handshake message’s length.

The following bytes are the NewSessionTicket message. The first 6 bytes might be the encoding from the statem_srvr.c (which also doesn’t seem to match the uint32 ticket_lifetime; and uint32 ticket_age_add; definitions). I don’t know what the final two are, maybe extensions? In any case, I think the client is expecting bytes for the ticket_nonce and/or ticket fields and that’s why it fails while decoding.