core: DoT dns resolution stopped working - Related to Let's Encrypt Root CA Expiry
Important notices
Before you add a new report, we ask you kindly to acknowledge the following:
- I have read the contributing guide lines at https://github.com/opnsense/core/blob/master/CONTRIBUTING.md
- I am convinced that my issue is new after having checked both open and closed issues at https://github.com/opnsense/core/issues?q=is%3Aissue
Describe the bug
The root CA DST Root CA X3 expired on September 30, 2021.
See: https://letsencrypt.org/docs/dst-root-ca-x3-expiration-september-2021/
Ever since that date, the DNS over TLS feature of Unbound in OPNsense has stopped working for me.
My DoT servers all use a Let’s Encrypt certificate and have valid certificate chains.
I believe something in OPNsense is not correctly detecting a valid certificate chain for these servers.
Certificate chain was verified using multiple methods:
To Reproduce
Steps to reproduce the behaviour:
- Go to OPNsense --> Services --> Unbound DNS --> DNS over TLS
- Click on [+] buton to ADD a DoT server with the following details:
Enabled: checked
IP Address: 145.100.185.15
Port: 853
Hostname: dnsovertls.sinodun.com
- Click the [Apply] button and then restart Unbound service
- On a client machine which uses the OPNsense device as a DNS server, perform a dns query
ping www.google.com
- Go to OPNsense --> Services --> Unbound DNS --> Log file
Expected behaviour
I expect the DNS query in step 4 to work and return an IP address for the fqdn www.google.com.
Instead, I get:
~> ping www.google.com
ping: www.google.com: Temporary failure in name resolution
Describe alternatives you considered
DNS resolution works fine by using a DoT server which doesn’t use a Let’s Encrypt certificate, e.g. Quad9’s 9.9.9.9:853 or Cloudflare’s 1.1.1.1:853.
Relevant log files
Proof that certificate chain is valid for 145.100.185.15:853
~> kdig -d @145.100.185.15 +tls-ca +tls-host=dnsovertls.sinodun.com www.google.com
;; DEBUG: Querying for owner(www.google.com.), class(1), type(1), server(145.100.185.15), port(853), protocol(TCP)
;; DEBUG: TLS, imported 512 system certificates
;; DEBUG: TLS, received certificate hierarchy:
;; DEBUG: #1, CN=dnsovertls.sinodun.com
;; DEBUG: SHA-256 PIN: 62lKu9HsDVbyiPenApnc4sfmSYTHOVfFgL3pyB+cBL4=
;; DEBUG: #2, C=US,O=Let's Encrypt,CN=R3
;; DEBUG: SHA-256 PIN: jQJTbIh0grw0/1TkHSumWb+Fs0Ggogr621gT3PvPKG0=
;; DEBUG: #3, C=US,O=Internet Security Research Group,CN=ISRG Root X1
;; DEBUG: SHA-256 PIN: C5+lpZ7tcVwmwQIMcRtPbsQtWLABXhQzejna0wHFr8M=
;; DEBUG: TLS, skipping certificate PIN check
;; DEBUG: TLS, The certificate is trusted.
;; TLS session (TLS1.2)-(ECDHE-SECP256R1)-(RSA-SHA512)-(AES-256-GCM)
;; ->>HEADER<<- opcode: QUERY; status: NOERROR; id: 63636
;; Flags: qr rd ra; QUERY: 1; ANSWER: 1; AUTHORITY: 0; ADDITIONAL: 1
;; EDNS PSEUDOSECTION:
;; Version: 0; flags: ; UDP size: 1232 B; ext-rcode: NOERROR
;; PADDING: 405 B
;; QUESTION SECTION:
;; www.google.com. IN A
;; ANSWER SECTION:
www.google.com. 243 IN A 172.217.168.196
;; Received 468 B
;; Time 2021-10-04 14:50:57 BST
;; From 145.100.185.15@853(TCP) in 20.4 ms
Unbound log msgs:
2021-10-04T15:04:16 unbound[51738] [51738:2] notice: ssl handshake failed 145.100.185.15 port 853
2021-10-04T15:04:16 unbound[51738] [51738:2] error: ssl handshake failed crypto error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed
2021-10-04T15:04:16 unbound[51738] [51738:3] notice: ssl handshake failed 145.100.185.15 port 853
2021-10-04T15:04:16 unbound[51738] [51738:3] error: ssl handshake failed crypto error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed
2021-10-04T15:04:16 unbound[51738] [51738:2] notice: ssl handshake failed 145.100.185.15 port 853
2021-10-04T15:04:16 unbound[51738] [51738:2] error: ssl handshake failed crypto error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed
2021-10-04T15:04:16 unbound[51738] [51738:3] notice: ssl handshake failed 145.100.185.15 port 853
2021-10-04T15:04:16 unbound[51738] [51738:3] error: ssl handshake failed crypto error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed
2021-10-04T15:04:16 unbound[51738] [51738:2] notice: ssl handshake failed 145.100.185.15 port 853
2021-10-04T15:04:16 unbound[51738] [51738:2] error: ssl handshake failed crypto error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed
2021-10-04T15:04:16 unbound[51738] [51738:2] info: 192.168.0.10 www.google.com. AAAA IN
2021-10-04T15:04:16 unbound[51738] [51738:2] info: 192.168.0.10 www.google.com. A IN
2021-10-04T15:04:16 unbound[51738] [51738:0] info: start of service (unbound 1.13.2).
Additional context
I’ve tried several DoT servers and all of them which use a Let’s Encrypt certificate show the same behaviour.
- 91.239.100.100:853 - anycast.censurfridns.dk
- 145.100.185.15:853 - dnsovertls.sinodun.com
- 145.100.185.16:853 - dnsovertls1.sinodun.com
- 185.49.141.37:853 - getdnsapi.net
Environment
Software version used and hardware type if relevant, e.g.:
OPNsense 21.7.3_3-amd64 FreeBSD 12.1-RELEASE-p20-HBSD OpenSSL 1.1.1l 24 Aug 2021
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 24 (21 by maintainers)
Commits related to this issue
- Trust / Authorities - prevent expired certificates from being flushed to disk to avoid non valid paths being trusted. (ref https://github.com/opnsense/core/issues/5257) ca-root-nss should be valid at... — committed to opnsense/core by AdSchellevis 3 years ago
- System / Trust / Authorities - do not flush intermediate certificates by default into the local trust store. as discussed in https://github.com/opnsense/core/issues/5257 When someone adds an intermed... — committed to opnsense/core by AdSchellevis 3 years ago
- Trust / Authorities - prevent expired or intermediate certificates from being flushed to disk by default to avoid non valid paths being trusted. PR: https://github.com/opnsense/core/issues/5257 — committed to opnsense/core by AdSchellevis 3 years ago
- Trust / Authorities - prevent expired or intermediate certificates from being flushed to disk by default to avoid non valid paths being trusted. PR: https://github.com/opnsense/core/issues/5257 — committed to opnsense/core by AdSchellevis 3 years ago
- Trust / Authorities - prevent expired or intermediate certificates from being flushed to disk by default to avoid non valid paths being trusted. PR: https://github.com/opnsense/core/issues/5257 — committed to DynFi/opnsense-core by AdSchellevis 3 years ago
@kulikov-a this https://github.com/opnsense/core/commit/5b9d7baccba979b5b41858b572145ee826dc389e should be it then. Exclude intermediates by default, optionally enable via System->Settings->General
Thank you so much.
That worked.
The bottom cert which I removed was:
Am happy to close this issue unless it needs to be left open until an official solution is put in place.