stubby: TLS failure for Cloudflare/TLS3

(all on the same machine)

$ kdig -d @1.1.1.1 +tls-ca +tls-host=cloudflare-dns.com example.com
;; DEBUG: Querying for owner(example.com.), class(1), type(1), server(1.1.1.1), port(853), protocol(TCP)
;; DEBUG: TLS, imported 149 system certificates
;; DEBUG: TLS, received certificate hierarchy:
;; DEBUG:  #1, C=US,ST=CA,L=San Francisco,O=Cloudflare\, Inc.,CN=*.cloudflare-dns.com
;; DEBUG:      SHA-256 PIN: yioEpqeR4WtDwE9YxNVnCEkTxIjx6EEIwFSQW+lJsbc=
;; DEBUG:  #2, C=US,O=DigiCert Inc,CN=DigiCert ECC Secure Server CA
;; DEBUG:      SHA-256 PIN: PZXN3lRAy+8tBKk2Ox6F7jIlnzr2Yzmwqc3JnyfXoCw=
;; DEBUG: TLS, skipping certificate PIN check
;; DEBUG: TLS, The certificate is trusted.
;; TLS session (TLS1.2)-(ECDHE-ECDSA-SECP256R1)-(AES-256-GCM)
;; ->>HEADER<<- opcode: QUERY; status: NOERROR; id: 13298
;; Flags: qr rd ra; QUERY: 1; ANSWER: 1; AUTHORITY: 0; ADDITIONAL: 1

;; EDNS PSEUDOSECTION:
;; Version: 0; flags: ; UDP size: 1452 B; ext-rcode: NOERROR
;; PADDING: 408 B

;; QUESTION SECTION:
;; example.com.                 IN      A

;; ANSWER SECTION:
example.com.            1906    IN      A       93.184.216.34

;; Received 468 B
;; Time 2018-09-16 13:48:02 AEST
;; From 1.1.1.1@853(TCP) in 19.9 ms

Using stubby, via dig @127.0.0.1 -p 5353 example.com

$ stubby -v 7
[03:49:12.130173] STUBBY: Read config from file /etc/stubby/stubby.yml
[03:49:12.130371] STUBBY: DNSSEC Validation is OFF
[03:49:12.130377] STUBBY: Transport list is:
[03:49:12.130379] STUBBY:   - TLS
[03:49:12.130381] STUBBY: Privacy Usage Profile is Strict (Authentication required)
[03:49:12.130383] STUBBY: (NOTE a Strict Profile only applies when TLS is the ONLY transport!!)
[03:49:12.130385] STUBBY: Starting DAEMON....
[03:49:22.769877] STUBBY: 1.1.1.1                                  : Conn opened: TLS - Strict Profile
[03:49:22.812921] STUBBY: 1.1.1.1                                  : Conn closed: TLS - *Failure*
[03:49:22.813053] STUBBY: 1.0.0.1                                  : Conn opened: TLS - Strict Profile
[03:49:22.813081] STUBBY: 1.1.1.1                                  : Conn closed: TLS - Resps=     0, Timeouts  =     0, Curr_auth =   None, Keepalive(ms)=     0
[03:49:22.813112] STUBBY: 1.1.1.1                                  : Upstream   : TLS - Resps=     0, Timeouts  =     0, Best_auth =   None
[03:49:22.813130] STUBBY: 1.1.1.1                                  : Upstream   : TLS - Conns=     0, Conn_fails=     1, Conn_shuts=      0, Backoffs     =     0
[03:49:22.857642] STUBBY: 1.0.0.1                                  : Conn closed: TLS - *Failure*
[03:49:22.857757] STUBBY: 1.1.1.1                                  : Conn opened: TLS - Strict Profile
[03:49:22.857784] STUBBY:    *FAILURE* no valid transports or upstreams available!
[03:49:22.857946] STUBBY: 1.0.0.1                                  : Conn closed: TLS - Resps=     0, Timeouts  =     0, Curr_auth =   None, Keepalive(ms)=     0
[03:49:22.857978] STUBBY: 1.0.0.1                                  : Upstream   : TLS - Resps=     0, Timeouts  =     0, Best_auth =   None
[03:49:22.858006] STUBBY: 1.0.0.1                                  : Upstream   : TLS - Conns=     0, Conn_fails=     1, Conn_shuts=      0, Backoffs     =     0

stubby.yml

resolution_type: GETDNS_RESOLUTION_STUB

dns_transport_list:
  - GETDNS_TRANSPORT_TLS

tls_authentication: GETDNS_AUTHENTICATION_REQUIRED

tls_query_padding_blocksize: 128

edns_client_subnet_private: 1

round_robin_upstreams: 1

idle_timeout: 10000

listen_addresses:
  - 127.0.0.1@5353
  - 0::1@5353

appdata_dir: "/var/cache/stubby"

upstream_recursive_servers:
  - address_data: 1.1.1.1
    tls_auth_name: "cloudflare-dns.com"
  - address_data: 1.0.0.1
    tls_auth_name: "cloudflare-dns.com"

Any ideas what I am doing wrong? This was working until I updated my system recently, with OpenSSL 1.1.1 11 Sep 2018… so potentially this is solely my problem, but I’d appreciate any pointers of where to look.

About this issue

Original URL
State: closed
Created 6 years ago
Comments: 30 (10 by maintainers)

Most upvoted comments

@obendev I talked to Cloudflare and it seems they are running an experimental version of TLS 1.3 (just like we were on dnsovertls3.sinodun.com) which is why it is incompatible with development versions of OpenSSL 1.1.1. They plan to roll this back to support just TLS 1.2 soon and will only add TLS 1.3 support when GnuTLS has the RFC compatible implementation.

Until then stubby users who want to use Cloudflare will have to downgrade to an earlier version of OpenSSL (1.0 or it seems 1.1.1-pre5 works).

Stubby doesn’t have the equivalent of +tls1_2, that’s the fix we need to add 😃

saradickinson on Sep 19, 2018

V. v. Sorry - I was unexpectedly out the office recently. There is a new Windows installer built against OpenSSL 1.1.1. available now: https://dnsprivacy.org/wiki/display/DP/Windows+installer+for+Stubby

Please test!

saradickinson on Nov 5, 2018

I’m going to try to get this done early next week.

It is possible to build stubby on Windows - however it requires Mingw. There are some slightly outdated instructions for how to [build getdns on Windows here](url https://getdnsapi.net/documentation/readme/#microsoft-windows-8.1). You need to add ---with-stubby --without-libidn --without-libidn2 to the configure line. I’ll try to add some instructions to the Stubby README too.

saradickinson on Oct 4, 2018

@dcousens Few things going on here I think:

Note that Cloudflare does not yet support TLS 1.3, so as suggested I would normally say use 145.100.185.18 but I also see a handshake failure (‘invalid parameter’) to that when I use the very latest OpenSSL too. Strangely I can connection to 9.9.9.9 and many other servers using the same client which all successfully fall back to TLS 1.2 (I’m not limiting the cipher suites). We’ll upgrade our server to the latest OpenSSLL and re-test…
You can always use Wireshark to look at just the handshake in detail
If you have the getdns_query or getdns_server_mon tools installed you can also use them in a similar way to kdig to do single queries.
To downgrade you will need to rebuild getdns to link to a different OpenSSL. What OS are you using and how are you building Stubby and Openssl?

saradickinson on Sep 17, 2018

Apologies - I didn’t get this done last week and I’m out the office this week. Top of the list for next week though!

saradickinson on Oct 15, 2018

Can confirm testing Cloudflare with Stubby built against OpenSSL 1.1.1 works

saradickinson on Oct 1, 2018

GnuTLS released version 3.6.4 with TLS 1.3 support 😊 https://lists.gnupg.org/pipermail/gnutls-help/2018-September/004457.html Now its Cloudflares turn. 😊

Update: Cloudflare just released TLS 1.3 support: https://developers.cloudflare.com/1.1.1.1/dns-over-tls/

Cloudflare’s DNS over TLS supports TLS 1.3 and TLS 1.2.

obendev on Sep 29, 2018

Okay… As far as I understand correctly:

TLS 1.3 does work with stubby and openssl >= 1.1.1 since it supports globally installed versions. Test: dnsovertls3.sinodun.com
However it does not work with Cloudflare, because Cloudflare supports only TLS 1.2 (https://developers.cloudflare.com/1.1.1.1/dns-over-tls/)
Therefore we have to wait until Cloudflare supports TLS 1.3 to use with stubby and openssl 1.1.1 (Thanks to @saradickinson for offering to ping Cloudflare, it would be really cool if you keep us updated)
Workaround for stubby with openssl 1.1.1 and Servers who not support TLS 1.3 yet: Add option +tls1_2 (https://github.com/getdnsapi/stubby/issues/132#issuecomment-422387086)
stubby needs to be updated with openssl 1.1.1 support, we talked about that in getdnsapi/stubby#130

I guess @dcousens should have enough solutions for now? 🤣

obendev on Sep 18, 2018

@obendev We’ve upgraded the server to OpenSSL-1.1.1 and dnsovertls3.sinodun.com now works for me with a OpenSSL 1.1.1 client - it was a compatibility issue (e.g. OpenSSL-1.1.1-pre5 worked, but OpenSSL-1.1.1-pre9 didn’t). NOTE: Cloudflare does not work with a OpenSSL-1.1.1 client unless the client is restricted to use TLS 1.2: openssl s_client -connect 145.100.185.18:853 - works for me openssl s_client -connect 1.1.1.1:853 - fails openssl s_client -connect 1.1.1.1:853 +tls1_2 - works for me

…BUT as has been pointed out the OpenSSL API call to control the 1.3 cipher suites has changed and the current getdns code doesn’t have a way of disabling TLS 1.3 in OpenSSL 1.1.1 😦 So until we release a bug fix update to address this or Cloudflare fully support TLS 1.3 you won’t be able to use Cloudflare with Stubby built against OpenSSL 1.1.1. I’ll ping Cloudflare and ask about TLS 1.3 support…

saradickinson on Sep 18, 2018

I think you might be able to downgrade by installing unbound 1.7.3-6 and getdns 1.4.2-1, but not really sure. There was no system-wide OpenSSL rebuild this time since the new version remained ABI and API compatible, but apparently softwares being rebuilt against the new version can no longer work with the previous one.

So if you are lucky that means only getdns here, but if not it won’t be easy to downgrade.

Also, have you tried openssl s_client to check whether this work or not?

ArchangeGabriel on Sep 18, 2018

If you install anything by the build in package manager: Could you try purging the packages and reinstalling them? openssl and stubby. And first try the openssl version you were using before you upgraded to 1.1.1

But as I and Sara said before, you can try dnsovertls3.sinodun.com to test TLS 1.3 with stubby.

On most Linux distros, it should just use the system OpenSSL, (…)

If this is not the case on your system your problem should not occur at all.

And then again there is the thing with linking getdns to a different openssl version yourself.

obendev on Sep 18, 2018

Hm. Unfortunately, I can not help you there. You could try the DNS Server dnsovertls3.sinodun.com with openssl 1.1.1 to narrow down the problem.

Maybe @ArchangeGabriel can give you further advice on that topic.

obendev on Sep 17, 2018