k3s: Invalid certificate for https://127.0.0.1:6443

Hi,

first of all, thanks for k3s 👍 Currently, I am testing it as an alternative to minikube 😃

I see OpenSSL complaining about invalid certificate when accessing https://127.0.0.1:6443 with ca.crt extracted from kubectl get secrets ....

Reproduction:

Setup

$ docker-compose up -d --scale node=3
$ cp kubeconfig.yaml ~/.kube/config
$ kubectl get nodes # works great :+1: 

Extract ca.crt

$ kubectl get secrets default-token-XXXXX -o go-template='{{index .data "ca.crt" | base64decode}}'  | tee ca.crt

Show issuer

$ openssl x509 -in ca.crt -noout -subject -issuer
subject=CN = k3s-token-ca@1549801496
issuer=CN = k3s-token-ca@1549801496

Show server certs

$ openssl s_client -showcerts -connect 127.0.0.1:6443 < /dev/null &> apiserver.crt

depth=0 O = k3s-org, CN = cattle
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 O = k3s-org, CN = cattle
verify error:num=21:unable to verify the first certificate
verify return:1
CONNECTED(00000003)
---
Certificate chain
 0 s:/O=k3s-org/CN=cattle
   i:/O=k3s-org/CN=k3s-ca
-----BEGIN CERTIFICATE-----
MIIDCDCCAfCgAwIBAgIIR4ZQtTBbrXAwDQYJKoZIhvcNAQELBQAwIzEQMA4GA1UE
....
sy5YAOPhEVA6WESx1xWmdDpsAmvBFsdEPdP88pg8jHSB0tkZ9L7BIc/4X6+q0QPZ
1Ls9bSy5vEJUWoL/XY3UNDmP5ki9VkfWOBVHlWz1RIIeugZzFrD9fDiR11AxhtP3
l179G9cbE4GJ3nT7
-----END CERTIFICATE-----
---
Server certificate
subject=/O=k3s-org/CN=cattle
issuer=/O=k3s-org/CN=k3s-ca
---
No client certificate CA names sent
Peer signing digest: SHA512
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 1353 bytes and written 269 bytes
Verification error: unable to verify the first certificate
---
New, TLSv1.2, Cipher is ECDHE-RSA-AES128-GCM-SHA256
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES128-GCM-SHA256
    Session-ID: EF5219395CABD1D9BDE462EAEFF21A7A6C350EF955978231E7A5CC4F504E6D3B
    Session-ID-ctx:
    Master-Key: KEY
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket:
    ........

    Start Time: 1549802771
    Timeout   : 7200 (sec)
    Verify return code: 21 (unable to verify the first certificate)
    Extended master secret: no
---
DONE

Verifying cert fails

$ openssl verify -verbose -CAfile ca.crt apiserver.crt
O = k3s-org, CN = cattle
error 20 at 0 depth lookup: unable to get local issuer certificate
error apiserver.crt: verification failed

curl’ing fails with provided cert

$ curl -vv --cacert ca.crt  https://localhost:6443/
*   Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 6443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: ca.crt
  CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS alert, Server hello (2):
* SSL certificate problem: unable to get local issuer certificate
* Curl_http_done: called premature == 1
* stopped the pause stream!
* Closing connection 0
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.haxx.se/docs/sslcerts.html

curl performs SSL certificate verification by default, using a "bundle"
 of Certificate Authority (CA) public keys (CA certs). If the default
 bundle file isn't adequate, you can specify an alternate file
 using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
 the bundle, the certificate verification probably failed due to a
 problem with the certificate (it might be expired, or the name might
 not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
 the -k (or --insecure) option.

curl’ing works with -k

$ curl -vv -k  https://localhost:6443/
*   Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 6443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: O=k3s-org; CN=localhost
*  start date: Feb 10 12:24:58 2019 GMT
*  expire date: Feb 10 12:24:58 2020 GMT
*  issuer: O=k3s-org; CN=k3s-ca
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
> GET / HTTP/1.1
> Host: localhost:6443
> User-Agent: curl/7.52.1
> Accept: */*
> 
< HTTP/1.1 401 Unauthorized
< Content-Type: application/json
< Www-Authenticate: Basic realm="kubernetes-master"
< Date: Sun, 10 Feb 2019 12:50:13 GMT
< Content-Length: 165
< 
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {
    
  },
  "status": "Failure",
  "message": "Unauthorized",
  "reason": "Unauthorized",
  "code": 401
* Curl_http_done: called premature == 0
* Connection #0  to host localhost left intact
}

Is anyone seeing similar behaviour?

It is very likely I am missing something. Hints/feedback is much appreciated 💜

Kind regards, Peter