nextdns: Logs of "context deadline exceeded", broken pipes, and eventual process death

Context

• nextdns version 1.32.2
• EdgeRouter X v2.0.9-hotfix.2

$ /config/nextdns/nextdns config
log-queries false
cache-max-age 0s
hardened-privacy false
report-client-info true
bogus-priv true
setup-router true
listen localhost:53
max-ttl 0s
timeout 5s
cache-size 10MB
discovery-dns 
detect-captive-portals false
use-hosts true
auto-activate true
control /var/run/nextdns.sock
config 10.11.12.0/24=abc123
config def456

For the past week or so I’ve been having problems on and off with the client on my router (and Windows host on a separate network, but I’m going to focus on the router). Starting yesterday, things have gotten much worse and the client is now repeatedly dying. The home network is becoming unusable. 😢

These are the sorts of messages filling my logs:

Jun  2 16:07:50 ubnt nextdns[16175]: Starting NextDNS 1.32.2/linux on :53
Jun  2 16:07:50 ubnt nextdns[16175]: Starting mDNS discovery
Jun  2 16:07:50 ubnt nextdns[16175]: Listening on TCP/:53
Jun  2 16:07:50 ubnt nextdns[16175]: Listening on UDP/:53
Jun  2 16:07:54 ubnt nextdns[16175]: Connected 45.90.28.0:443 (con=11ms tls=3941ms, TCP, TLS13)
Jun  2 16:07:55 ubnt nextdns[16175]: Setting up router
Jun  2 16:07:55 ubnt nextdns[16175]: Activating
Jun  2 16:07:56 ubnt nextdns[16175]: Connected [2001:19f0:ac01:170:5400:2ff:fec8:72c8]:443 (con=9ms tls=1474ms, TCP, TLS13)
Jun  2 16:07:56 ubnt nextdns[16175]: Switching endpoint: https://dns.nextdns.io.#104.238.181.28,2001:19f0:ac01:170:5400:2ff:fec8:72c8
Jun  2 16:07:57 ubnt nextdns[16175]: Query 10.19.47.121 UDP A doh.dns.apple.com. (qry=35/res=12) 7264ms : doh resolve: context deadline exceeded
Jun  2 16:07:57 ubnt nextdns[16175]: Query 10.19.47.121 UDP 65 apple.com. (qry=27/res=12) 7270ms : doh resolve: context deadline exceeded
…
Jun  2 16:16:01 ubnt nextdns[16175]: Connected 104.238.181.28:443 (con=0ms tls=0ms, TCP, )
Jun  2 16:16:41 ubnt nextdns[16175]: Connected 104.238.181.28:443 (con=19ms tls=0ms, TCP, )
Jun  2 16:18:19 ubnt nextdns[16175]: Query 10.19.45.95 UDP AAAA www.apple.com. (qry=31/res=12) cached HTTP/2.0: doh resolve: context deadline exceeded
Jun  2 16:18:19 ubnt nextdns[16175]: Query 10.19.45.95 UDP A www.apple.com. (qry=31/res=12) cached HTTP/2.0: doh resolve: context deadline exceeded
…
Jun  2 16:25:04 ubnt nextdns[16175]: Received signal: broken pipe (ignored)
Jun  2 16:25:59 ubnt nextdns[16175]: Received signal: broken pipe (ignored)
Jun  2 16:26:02 ubnt nextdns[16175]: message repeated 5 times: [ Received signal: broken pipe (ignored)]
Jun  2 16:26:03 ubnt nextdns[16175]: Received signal: broken pipe (ignored)
Jun  2 16:26:04 ubnt nextdns[16175]: Received signal: broken pipe (ignored)
Jun  2 16:26:08 ubnt nextdns[16175]: message repeated 21 times: [ Received signal: broken pipe (ignored)]
Jun  2 16:26:09 ubnt nextdns[16175]: Endpoint provider failed: &{dns.nextdns.io. https://dns.nextdns.io#45.90.28.0,2a07:a8c0::,45.90.30.0,2a07:a8c1::}: exchange: roundtrip: write tcp [2001:558:6045:42:15ee:b4d7:5357:f202]:46832->[2a07:a8c0::]:443: write: broken pipe
Jun  2 16:26:09 ubnt nextdns[16175]: Received signal: broken pipe (ignored)
Jun  2 16:26:09 ubnt nextdns[16175]: message repeated 5 times: [ Received signal: broken pipe (ignored)]
Jun  2 16:26:12 ubnt nextdns[16175]: Connected [2001:19f0:ac01:170:5400:2ff:fec8:72c8]:443 (con=11ms tls=1471ms, TCP, TLS13)
Jun  2 16:26:12 ubnt nextdns[16175]: Switching endpoint: https://dns.nextdns.io.#104.238.181.28,2001:19f0:ac01:170:5400:2ff:fec8:72c8
Jun  2 16:26:12 ubnt nextdns[16175]: Query 10.19.45.81 UDP A gateway.icloud.com. (qry=36/res=12) 74721ms : doh resolve: context deadline exceeded

After a while, the client is killed by the oom-killer, which #505 is about.

I’ve upgraded all the things as a first step. I was running v1.9.4 on EdgeOS v2.0.9 (no hotfix) for many months without issue.