redisson: Regression: DNS resolver failing to find valid DNS record

Expected behavior

The DNS resolver should find valid DNS records. The hostname I’m having trouble with worked in 3.5.0, it is failing in newer versions.

I opened a ticket with Netty first thinking it was a problem with the resolver itself, but after trying different Netty versions, it appears to be a configuration issue with how Reddison is calling the resolver.

See https://github.com/netty/netty/issues/8261

Actual behavior

Exception thrown:

Caused by: io.netty.resolver.dns.DnsNameResolverContext$SearchDomainUnknownHostException: Search domain query failed. Original hostname: 'host.toplevel' failed to resolve 'host.toplevel.search.domain' after 7 queries 
	at io.netty.resolver.dns.DnsNameResolverContext.finishResolve(DnsNameResolverContext.java:721)
	at io.netty.resolver.dns.DnsNameResolverContext.tryToFinishResolve(DnsNameResolverContext.java:663)
	at io.netty.resolver.dns.DnsNameResolverContext.query(DnsNameResolverContext.java:306)
	at io.netty.resolver.dns.DnsNameResolverContext.query(DnsNameResolverContext.java:295)
	at io.netty.resolver.dns.DnsNameResolverContext.tryToFinishResolve(DnsNameResolverContext.java:636)
	at io.netty.resolver.dns.DnsNameResolverContext$3.operationComplete(DnsNameResolverContext.java:342)
	at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507)
	at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:481)
	at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:420)
	at io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:104)
	at io.netty.resolver.dns.DnsQueryContext.setSuccess(DnsQueryContext.java:197)
	at io.netty.resolver.dns.DnsQueryContext.finish(DnsQueryContext.java:180)
	at io.netty.resolver.dns.DnsNameResolver$DnsResponseHandler.channelRead(DnsNameResolver.java:969)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1412)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:943)
	at io.netty.channel.nio.AbstractNioMessageChannel$NioMessageUnsafe.read(AbstractNioMessageChannel.java:93)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:886)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.lang.Thread.run(Thread.java:748)

Steps to reproduce

  1. Configure a top level domain someDomain on a DNS server you own
  2. Configure a host under the new top level domain someHost.someDomain
  3. Configure multiple resolvers on the DNS client machine that will run the Netty code. i.e. 8.8.8.8, 192.168.1.1, and 10.0.0.1 (I have 3 resolvers configured, each pointing to different DNS masters - global DNS, local personal private network, company private network over a VPN)
  4. Configure the search domain to not match the top level domain, i.e. search.otherDomain on the DNS client machine that will run the Netty code
  5. Configure redisson to connect to someHost.someDomain
  6. failure during configuration.

Redis version

Redisson version

Trying to migrate from 3.5.0 to the latest version

Redisson configuration

About this issue

  • Original URL
  • State: closed
  • Created 6 years ago
  • Comments: 50 (20 by maintainers)

Commits related to this issue

Most upvoted comments

I’m experiencing the same issue with Redisson 3.13.4 which brings in Netty 4.1.51 Final. I’m not using Kubernetes. I’m using docker-compose + docker on a Google Compute Engine VM. It’s failing to resolve an unqualified DNS name on my VPC which is resolved correctly by dig/nslookup/ping/etc.

The container is built from jetty:9-jdk14-slim, which I think (?) means it’s based on debian.

I’m using dirt-simple Redisson configuration:

private val config = Config().apply { useSingleServer().address = "redis://my-host:6379" }

My container’s /etc/resolv.conf looks like this:

`search c.some-domain.internal. google.internal.

nameserver 127.0.0.11

options ndots:0 `

The host VM image is from Debian 10. The host VM’s /etc/resolv.conf looks like this:

`domain c.some-domain.internal

search c.some-domain.internal. google.internal.

nameserver 169.254.169.254`

I worked around it for now by passing a fully-qualified name for my-host. Would love to know why it doesn’t work though.

Setting -Djava.net.preferIPv4Stack=true -Djava.net.preferIPv4Addresses=true did not help. My solution was to use tcp mode for DNS request as reported here.

similar issue happened again @mrniko

JDK 1.8 Redisson 3.9 with default netty (4.1.29.Final)

2018-12-06 17:25:10.746 ERROR 1 --- [redisson-netty-4-1] org.redisson.connection.DNSMonitor       : Unable to resolve mb.y.redis.sohucs.com
io.netty.resolver.dns.DnsResolveContext$SearchDomainUnknownHostException: Search domain query failed. Original hostname: 'someHost.someDomain' failed to resolve 'someHost.someDomain.search.otherDomain' after 2 queries 
  at io.netty.resolver.dns.DnsResolveContext.finishResolve(DnsResolveContext.java:816) [netty-resolver-dns-4.1.29.Final.jar!/:4.1.29.Final]
  at io.netty.resolver.dns.DnsResolveContext.tryToFinishResolve(DnsResolveContext.java:777) [netty-resolver-dns-4.1.29.Final.jar!/:4.1.29.Final]
  at io.netty.resolver.dns.DnsResolveContext.query(DnsResolveContext.java:310) [netty-resolver-dns-4.1.29.Final.jar!/:4.1.29.Final]
  at io.netty.resolver.dns.DnsResolveContext.query(DnsResolveContext.java:299) [netty-resolver-dns-4.1.29.Final.jar!/:4.1.29.Final]
  at io.netty.resolver.dns.DnsResolveContext.access$500(DnsResolveContext.java:62) [netty-resolver-dns-4.1.29.Final.jar!/:4.1.29.Final]
  at io.netty.resolver.dns.DnsResolveContext$3.operationComplete(DnsResolveContext.java:356) [netty-resolver-dns-4.1.29.Final.jar!/:4.1.29.Final]
  at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511) [netty-common-4.1.29.Final.jar!/:4.1.29.Final]
  at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:504) [netty-common-4.1.29.Final.jar!/:4.1.29.Final]
  at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:483) [netty-common-4.1.29.Final.jar!/:4.1.29.Final]
  at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424) [netty-common-4.1.29.Final.jar!/:4.1.29.Final]
  at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:121) [netty-common-4.1.29.Final.jar!/:4.1.29.Final]
  at io.netty.resolver.dns.DnsQueryContext.setFailure(DnsQueryContext.java:216) [netty-resolver-dns-4.1.29.Final.jar!/:4.1.29.Final]
  at io.netty.resolver.dns.DnsQueryContext.access$300(DnsQueryContext.java:43) [netty-resolver-dns-4.1.29.Final.jar!/:4.1.29.Final]
  at io.netty.resolver.dns.DnsQueryContext$4.run(DnsQueryContext.java:166) [netty-resolver-dns-4.1.29.Final.jar!/:4.1.29.Final]
  at io.netty.util.concurrent.PromiseTask$RunnableAdapter.call(PromiseTask.java:38) [netty-common-4.1.29.Final.jar!/:4.1.29.Final]
  at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:127) [netty-common-4.1.29.Final.jar!/:4.1.29.Final]
  at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) [netty-common-4.1.29.Final.jar!/:4.1.29.Final]
  at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404) [netty-common-4.1.29.Final.jar!/:4.1.29.Final]
  at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:446) [netty-transport-4.1.29.Final.jar!/:4.1.29.Final]
  at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884) [netty-common-4.1.29.Final.jar!/:4.1.29.Final]
  at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-common-4.1.29.Final.jar!/:4.1.29.Final]
  at java.lang.Thread.run(Thread.java:748) [na:1.8.0_181]
Caused by: io.netty.resolver.dns.DnsNameResolverTimeoutException: [/172.16.40.1:53] query timed out after 5000 milliseconds (no stack trace available)

We have also encountered this problem, it seems to be related to this: Racy conntrack and DNS lookup timeouts We are currently attempting to upgrade Alpine to version 3.18.3 and are in the testing phase, not sure if it will work.

I have ready test on alpine 3.18.3 , it can not fix the problem , because redisson dns not implement by musl , it depends on netty dns resovler.

This issue’s fix seems already be included in the 3.18.0 release. redisson-3.18.0