routedns: panic: runtime error: invalid memory address or nil pointer dereference

I’ve been receiving the following error running routedns on a public server with DOH exposed.

Ubuntu 20.04.3 LTS (GNU/Linux 5.4.0-90-generic x86_64) go version go1.17.3 linux/amd64

Dec 20 16:40:25 server routedns[203538]: time="2021-12-20T16:40:25Z" level=info msg="starting listener" addr=":443" id=local-doh protocol=doh
Dec 20 16:40:25 server routedns[203538]: time="2021-12-20T16:40:25Z" level=info msg="starting listener" addr=":53" id=local-udp protocol=udp
Dec 20 16:40:25 server routedns[203538]: time="2021-12-20T16:40:25Z" level=info msg="starting listener" addr=":53" id=local-tcp protocol=tcp
Dec 20 17:06:44 server routedns[203538]: panic: runtime error: invalid memory address or nil pointer dereference
Dec 20 17:06:44 server routedns[203538]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x817afe]
Dec 20 17:06:44 server routedns[203538]: goroutine 1993 [running]:
Dec 20 17:06:44 server routedns[203538]: github.com/miekg/dns.(*OPT).copy(0xc003b58980)
Dec 20 17:06:44 server routedns[203538]:         /opt/routedns/pkg/mod/github.com/miekg/dns@v1.1.43/ztypes.go:866 +0xfe
Dec 20 17:06:44 server routedns[203538]: github.com/miekg/dns.(*Msg).CopyTo(0xc00577cbd0, 0xc00577cc60)
Dec 20 17:06:44 server routedns[203538]:         /opt/routedns/pkg/mod/github.com/miekg/dns@v1.1.43/msg.go:1076 +0x277
Dec 20 17:06:44 server routedns[203538]: github.com/miekg/dns.(*Msg).Copy(...)
Dec 20 17:06:44 server routedns[203538]:         /opt/routedns/pkg/mod/github.com/miekg/dns@v1.1.43/msg.go:1050
Dec 20 17:06:44 server routedns[203538]: github.com/folbricht/routedns.(*Cache).Resolve(0xc0000b3810, 0xc00017b5f0, {{0xc0037f15cc, 0x8, 0x0}})
Dec 20 17:06:44 server routedns[203538]:         /opt/routedns/pkg/mod/github.com/folbricht/routedns@v0.1.0/cache.go:133 +0x434
Dec 20 17:06:44 server routedns[203538]: github.com/folbricht/routedns.listenHandler.func1({0xb59158, 0xc000157180}, 0xc00017b5f0)
Dec 20 17:06:44 server routedns[203538]:         /opt/routedns/pkg/mod/github.com/folbricht/routedns@v0.1.0/dnslistener.go:71 +0x60b
Dec 20 17:06:44 server routedns[203538]: github.com/miekg/dns.HandlerFunc.ServeDNS(0xc001757500, {0xb59158, 0xc000157180}, 0xc00017b5f0)
Dec 20 17:06:44 server routedns[203538]:         /opt/routedns/pkg/mod/github.com/miekg/dns@v1.1.43/server.go:37 +0x2f
Dec 20 17:06:44 server routedns[203538]: github.com/miekg/dns.(*Server).serveDNS(0xc0000d05a0, {0xc001757500, 0x80, 0xc000090298}, 0xc000157180)
Dec 20 17:06:44 server routedns[203538]:         /opt/routedns/pkg/mod/github.com/miekg/dns@v1.1.43/server.go:651 +0x4da
Dec 20 17:06:44 server routedns[203538]: github.com/miekg/dns.(*Server).serveTCPConn(0xc0000d05a0, 0x0, {0xb592b8, 0xc0000b6728})
Dec 20 17:06:44 server routedns[203538]:         /opt/routedns/pkg/mod/github.com/miekg/dns@v1.1.43/server.go:559 +0x26c
Dec 20 17:06:44 server routedns[203538]: created by github.com/miekg/dns.(*Server).serveTCP
Dec 20 17:06:44 server routedns[203538]:         /opt/routedns/pkg/mod/github.com/miekg/dns@v1.1.43/server.go:460 +0x2d9

About this issue

Original URL
State: open
Created 3 years ago
Comments: 46 (24 by maintainers)

Most upvoted comments

All of the known race conditions should be addressed on master. Not sure if there’s one more lurking somewhere. Would you be able to run it off master with -race to see if anything shows with your config?

folbricht on Apr 12, 2023

https://github.com/folbricht/routedns/pull/290 (which I just merged) should help somewhat, there’s one more issue in QUIC that I need to find.

folbricht on Apr 1, 2023

Perfect, this issue was just introduced. Pushed a fix to the same branch just now.

folbricht on Mar 5, 2023

Perfect, that helped so much! The fix for it should be on the issue-198 branch. Pretty sure this is going to take care of it, but it’d be nice if you could still run it with -race in case there are any others left.

folbricht on Mar 5, 2023

I fixed the one data race I was able to reproduce in the request-dedup component (https://github.com/folbricht/routedns/pull/270) It’s not entirely clear if this will fix your issue as well but it’s worth a try. It’s on master and also backported to the patch-issue-198 branch.

folbricht on Dec 27, 2022

Thank you. This actually failed in a different spot, a little bit earlier. I updated the branch to catch this one as well, and it should be the last possible failure place. If you run this for a bit, it’d be nice if you could grep the logs for NILOPTREC. That’ll tell me how often it happens and where.

This more and more looks like a data-race in on of the elements before. The one’s in your config that use concurrency are cache and fastest. I’ll try to replicate this in my own config

folbricht on Nov 21, 2022

That’s actually a different issue. It was introduced recently with the blocklist logging changes and should be fixed in https://github.com/folbricht/routedns/pull/203 now.

The hunt continues…

folbricht on Jan 12, 2022

Thank you. I put a quick patch on the patch-issue-198 branch that should help you until I have a proper fix. It’s only a workaround for now. I’ll look deeper, there could well be a bug in the DoT code, it deals with padding in that area so I’ll review that part.

folbricht on Dec 23, 2021