dns: DNS resolution failed until restart of container

Hi! First of all, thank you for your amazing work, I’m a big fan of your project.

I have a question for you: does the LRU cache saves empty responses? I’m asking this because something strange happened yesterday.

FYI I upgraded to v2.0.0-beta about ~2 weeks ago. Everything was working fine until yesterday.

For context, some of my services resolve very often (like once every second) a specific hostname—let’s call it acme.com. So they are basically hitting the LRU most of the time, if not all the time.

Yesterday that hostname couldn’t get resolved anymore by my services for some reason. After I noticed the issue I tried:

> ping acme.com
ping: acme.com: Name or service not known

But at the same time I could resolve google.com

> ping google.com
PING google.com(fra24s04-in-x0e.1e100.net (2a00:1450:4001:827::200e)) 56 data bytes
64 bytes from fra24s04-in-x0e.1e100.net (2a00:1450:4001:827::200e): icmp_seq=1 ttl=118 time=5.11 ms
64 bytes from fra24s04-in-x0e.1e100.net (2a00:1450:4001:827::200e): icmp_seq=2 ttl=118 time=5.25 ms
64 bytes from fra24s04-in-x0e.1e100.net (2a00:1450:4001:827::200e): icmp_seq=3 ttl=118 time=5.33 ms

Once I restarted the DNS container, the hostname could get resolved once again right away.

My config is very simple:

version: "3.7"

services:
  dns-server:
    image: qmcgaw/dns:v2.0.0-beta
    container_name: dns-server
    restart: always
    environment:
      - BLOCK_MALICIOUS=on
      - BLOCK_SURVEILLANCE=on
      - BLOCK_ADS=on
      - METRICS_TYPE=prometheus
    ports:
      - 127.0.0.1:53:53/udp
      - 172.17.0.1:53:53/udp
      - 172.17.0.1:9892:9090/tcp
    network_mode: bridge

Those are the last logs that got written before I restarted my DNS container. I got some warnings but I don’t think they are important. FYI the issue started the 28th around 17:30.

2021/10/27 21:46:39 WARN cannot exchange over DoT connection: read tcp 172.17.0.2:35246->1.1.1.1:853: i/o timeout
2021/10/27 23:00:01 WARN cannot exchange over DoT connection: read tcp 172.17.0.2:53620->1.1.1.1:853: i/o timeout
2021/10/28 01:54:45 WARN cannot exchange over DoT connection: read tcp 172.17.0.2:60710->1.1.1.1:853: i/o timeout
2021/10/28 03:42:38 WARN cannot exchange over DoT connection: read tcp 172.17.0.2:50438->1.1.1.1:853: i/o timeout
2021/10/28 03:54:29 WARN cannot exchange over DoT connection: read tcp 172.17.0.2:54982->1.0.0.1:853: i/o timeout
2021/10/28 09:49:23 WARN cannot exchange over DoT connection: read tcp 172.17.0.2:36820->1.0.0.1:853: i/o timeout
2021/10/28 11:09:58 WARN cannot exchange over DoT connection: read tcp 172.17.0.2:36598->1.1.1.1:853: i/o timeout
2021/10/28 12:46:54 WARN cannot exchange over DoT connection: read tcp 172.17.0.2:33552->1.0.0.1:853: i/o timeout
2021/10/28 13:26:33 WARN cannot exchange over DoT connection: read tcp 172.17.0.2:43944->1.0.0.1:853: i/o timeout
2021/10/28 21:17:11 WARN cannot exchange over DoT connection: read tcp 172.17.0.2:52100->1.1.1.1:853: i/o timeout
2021/10/28 21:29:20 INFO planned periodic restart of DNS server
2021/10/28 21:29:20 INFO downloading and building DNS block lists
2021/10/28 21:29:21 INFO 1220920 hostnames blocked overall
2021/10/28 21:29:21 INFO 29956 IP addresses blocked overall
2021/10/28 21:29:21 INFO 2482 IP networks blocked overall
2021/10/28 21:29:21 INFO starting DNS server
2021/10/28 21:29:21 INFO DNS server listening on :53

Could it be possible that the DNS server saved an empty resolution in the LRU during one of those timeouts and then kept serving that empty resolution forever to my services? What do you think?

Thank you very much for your help.