nameko: Eventlet 0.20.0 breaks dns resolution in nameko

I seem to have hit a very strange bug running nameko inside of docker-compose At this point, i’m able to get around the issue, but I would like to know if eventlet or docker is doing something wrong.

Docker: Version 1.12.3 (13776) Channel: Stable 583d1b8ffe

nameko version 2.3.1

Traceback from inside the container.

[root@ba50a39f5dbb app]# nameko run --config ./cfg/nameko.yaml blah.blah.service:blahService
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/eventlet-0.20.0-py2.7.egg/eventlet/hubs/poll.py", line 115, in wait
    listener.cb(fileno)
  File "/usr/lib/python2.7/site-packages/eventlet-0.20.0-py2.7.egg/eventlet/greenthread.py", line 214, in main
    result = function(*args, **kwargs)
  File "build/bdist.linux-x86_64/egg/nameko/utils.py", line 175, in call
    return getattr(item, name)(*args, **kwargs)
  File "build/bdist.linux-x86_64/egg/nameko/messaging.py", line 191, in setup
    verify_amqp_uri(self.amqp_uri)
  File "build/bdist.linux-x86_64/egg/nameko/amqp.py", line 43, in verify_amqp_uri
    with transport.establish_connection():
  File "/usr/lib/python2.7/site-packages/kombu-4.0.2-py2.7.egg/kombu/transport/pyamqp.py", line 130, in establish_connection
    conn.connect()
  File "/usr/lib/python2.7/site-packages/amqp-2.1.4-py2.7.egg/amqp/connection.py", line 294, in connect
    self.transport.connect()
  File "/usr/lib/python2.7/site-packages/amqp-2.1.4-py2.7.egg/amqp/transport.py", line 120, in connect
    self._connect(self.host, self.port, self.connect_timeout)
  File "/usr/lib/python2.7/site-packages/amqp-2.1.4-py2.7.egg/amqp/transport.py", line 150, in _connect
    host, port, 0, socket.SOCK_STREAM, SOL_TCP,
  File "/usr/lib/python2.7/site-packages/eventlet-0.20.0-py2.7.egg/eventlet/support/greendns.py", line 464, in getaddrinfo
    raise socket.gaierror(socket.EAI_NONAME, 'No address found')
gaierror: [Errno -2] No address found

The host dns resolves with a non authoritive response, but all dns functions work correctly from the container itself.

Pinning the version of eventlet to 0.16.1 seems to stop the issue from occurring.

Is the eventlet dns client implementation known to have issues?

thanks in advance! 😃 Rob

About this issue

Original URL
State: closed
Created 8 years ago
Comments: 23 (10 by maintainers)

Commits related to this issue

Pin dnspython below 2.0 See https://github.com/nameko/nameko/issues/392 — committed to Emplocity/nameko-prometheus by zsiciarz 4 years ago

Most upvoted comments

There are a couple of threads of conversation happening here now. Let me try to summarise:

Kombu 4.0 requires eventlet 0.2.0, but nameko pins kombu to 3.0.37, so this is irrelevant. #378 is not related.
Eventlet 0.2.0 bundles dnspython and seems to use it to monkeypatch the socket library’s DNS functions. Unfortunately dnspython doesn’t have exactly the same behaviour – in particular, it won’t resolve a fully qualified domain name without a . at the end if there’s a search domain specified. From what I understand about DNS that’s the correct behaviour, but the fact that socket.getaddrinfo behaves differently suggests there’s bug is in eventlet’s assumption that they’re equivalent.

mattbennett on Dec 20, 2016

Hi @manfred-chebli & @rgardam have you docker “linked” your broker container to your service container? Or do you just rely on Docker embedded DNS server by simply using the name (broker in your case) of the container?

Once we’ve upgraded to master version of Eventlet (before they cut 0.20 release, we needed it to use a fix to HTTPS body parsing https://github.com/eventlet/eventlet/blob/master/NEWS#L8) we noticed that Docker embedded DNS name resolution does not work any more.

Since eventlet vendored dnspython I’ve opened issue directly with dnspython here https://github.com/rthalley/dnspython/issues/219 but they said this is expected behaviour. There is a gist to reproduce the issue.

The workaround we’ve employed for now is to trick dnspython into thinking that our container names are fully qualified domains by adding period . to the url. In your case it would be amqp://user:pass@broker.:5672

I think by using dnspython for dns resolution eventlet broke some functionality.

kooba on Dec 20, 2016

@mattbennett This was a pebcak. Problem was solved by adding a sleep script to the container entrypoint to properly wait for the broker to startup. Sorry for the confusion.

alexander-mayr on Jan 24, 2019

Thanks for the quick resolution on this! 😃

rgardam on Jan 3, 2017

The fix is included in Eventlet v0.20.1 version that is already available on PyPI. If you pinned requirements to 0.19 or installed zip from long url above, please update to eventlet==0.20.1 now.

Fly safe.

temoto on Jan 3, 2017

I have raised this as a bug with eventlet https://github.com/eventlet/eventlet/issues/363

In the mean time, the workarounds are:

pin eventlet down to <0.2.0 in your application
resolve using a strictly fully-qualified domain name (i.e. put a . at the end)

For the moment I think it’s too restrictive to pin eventlet down in nameko itself.

mattbennett on Dec 20, 2016

@kooba It seems like inside of newer versions of docker-compose the link: option doesn’t do what it used to do.

Previously docker would create /etc/hosts entries, but this is not the case and it now only uses the internal dns.

I will try your trick.

rgardam on Dec 20, 2016