cluster-api-k3s: Control plane load balancer SSL health check fails

is it possible to teach ELB to support TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 ?

confirmed: no

I methodically compared the difference in behavior between a CAPA kubeadm based deployment and a cluster-api-k3s based one.

I ran a sample Go program with config generated by https://ssl-config.mozilla.org/#server=go&version=1.19&config=intermediate&hsts=false&guideline=5.6

which is the same config generator used by k8s itself, see https://github.com/k3s-io/k3s/blob/f8b661d590ecd1ed2ed04b3c51ff5e6d67cb092b/pkg/cli/server/server.go#L380

// generated 2022-12-13, Mozilla Guideline v5.6, Go 1.14.4, intermediate configuration, no HSTS
// https://ssl-config.mozilla.org/#server=go&version=1.14.4&config=intermediate&hsts=false&guideline=5.6
package main

import (
	"crypto/tls"
	"fmt"
	"log"
	"net/http"
)

func main() {
	mux := http.NewServeMux()
	mux.HandleFunc("/", func(w http.ResponseWriter, req *http.Request) {
		w.Write([]byte("This server is running the Mozilla intermediate configuration.\n"))
	})

	cfg := &tls.Config{
		MinVersion:               tls.VersionTLS10,
		PreferServerCipherSuites: true,
		CipherSuites: []uint16{
			tls.TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,
			tls.TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,
			tls.TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,
			tls.TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,
			tls.TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,
			tls.TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,
			tls.TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256,
			tls.TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,
			tls.TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,
			tls.TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,
			tls.TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,
			tls.TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,
			tls.TLS_RSA_WITH_AES_128_GCM_SHA256,
			tls.TLS_RSA_WITH_AES_256_GCM_SHA384,
			tls.TLS_RSA_WITH_AES_128_CBC_SHA256,
			tls.TLS_RSA_WITH_AES_128_CBC_SHA,
			tls.TLS_RSA_WITH_AES_256_CBC_SHA,
			tls.TLS_RSA_WITH_3DES_EDE_CBC_SHA,
		},
	}

	srv := &http.Server{
		Addr:      ":6443",
		Handler:   mux,
		TLSConfig: cfg,
		// Consider setting ReadTimeout, WriteTimeout, and IdleTimeout
		// to prevent connections from taking resources indefinitely.
	}

	log.Fatal(srv.ListenAndServeTLS(
		"/root/apiserver.crt",
		"/root/apiserver.key",
	))
}

I shut down the api-server and ran it on port 6443, looking at the TLS handshakes performed by the AWS LB using tcpdump.

Let’s call HostA the host running kubeadm and Host3 running k3s.

The same program worked on HostA and didn’t work on Host3. The only difference were the TLS keypairs on the two machienes. I copied the keypair from HostA to Host3 and it worked on Host3 too.

tcpdump revealed that the negotiated cipher suite was TLS_RSA_WITH_AES_128_GCM_SHA256 (0x009c)

I modified the test program to include the ciphers included by default by k3s:

        cfg := &tls.Config{
                MinVersion: tls.VersionTLS12,
                CipherSuites: []uint16{
                        tls.TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,
                        tls.TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,
                        tls.TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,
                        tls.TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,
                        tls.TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,
                        tls.TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,
                        tls.TLS_RSA_WITH_AES_128_GCM_SHA256,
                },
        }

It didn’t work until I added tls.TLS_RSA_WITH_AES_128_GCM_SHA256, which is similar to the included tls.TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 but it doesn’t have the ECDHE prefix.

This cipher is not chosen when I use the certificate created during k3s initialization. I conclude the certificate is not compatible with ECDHE.

TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 means:

• the certificate contains a RSA public key.
• the key exchange will be done using ECDHE (Elliptic-curve Diffie–Hellman)
• the symmetric cipher used after the key exchange will be AES-GCM with a 128 bit key.
• PRF (pseudo-random function) for key exchange is SHA256.

I have layperson knowledge about how TLS works and I assumed that the DH exchange didn’t depend on the asymmetric crypto used to verify the certificate signature (here RSA I assume).

Yesterday I did quickly check the CA certificate and I found nothing unusual:

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 0 (0x0)
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN = kubernetes
        Validity
            Not Before: Dec 12 14:21:18 2022 GMT
            Not After : Dec  9 14:26:18 2032 GMT
        Subject: CN = kubernetes
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                RSA Public-Key: (2048 bit)
                Modulus: <.......>
                Exponent: 65537 (0x10001)

However, when looking at the actual certificate that used by the server (which is signed by the CA but is not the CA certificate), I can see it’s using elliptic curve crypto:

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 8063256806938441330 (0x6fe671584e50f672)
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN = kubernetes
        Validity
            Not Before: Dec 12 14:21:18 2022 GMT
            Not After : Dec 12 14:26:55 2023 GMT
        Subject: CN = kube-apiserver
        Subject Public Key Info:
            Public Key Algorithm: id-ecPublicKey
                Public-Key: (256 bit)
                pub:
                    04:55:59:f2:0e:57:4d:4b:ed:a6:47:38:ee:6d:43:
                    20:6e:45:b8:a8:44:a9:19:2d:01:0f:da:72:7c:8a:
                    03:d5:17:ba:55:8a:79:26:67:97:80:65:4a:3d:54:
                    37:ef:3d:af:98:83:cc:09:80:19:a9:3b:7b:11:8c:
                    42:25:d0:d5:69
                ASN1 OID: prime256v1
                NIST CURVE: P-256

The file name is /var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt and presumably created either by the k3s installer or by the k3s binary itself on first run.

---

This to investigate:

• what’s the easiest way to let k3s create the server certificate using Public Key Algorithm: rsaEncryption
• is it possible to teach ELB to support TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 ?
• how hard would it be to configure the ELB to use TCP healthcheck instead of “SSL” healthcheck? who creates that LB? CAPA?

another example; TLS_DHE_RSA_WITH_AES_128_GCM_SHA256 is supported by k3s, and we can use it locally with:

$ curl -k https://localhost:6443 --ciphers DHE-RSA-AES128-GCM-SHA256

but it fails with:

$ curl -k https://localhost:6443 --ciphers DHE-RSA-AES128-GCM-SHA256  --tlsv1.2 --tls-max 1.2
curl: (35) error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure

I find the TLS cipher/version matrix very hard to keep up with; I’ll try to figure out which ciphers need to be enabled so that it k3s works with TLS1.2