pika: Tornado adapter: passive exchange.declare scenario where close hangs

Updated bug description:

When a channel is closed due to a passive declaration of an exchange that does not exist, processing of the resulting channel closure hangs.

---

Original description (kept so the context of earlier comments is not lost):

Not sure exactly when it started happening, but lately, we’ve been getting this message, a lot, when an AMQP client tries to passively declare a number of exchanges that are not present.

Correct behaviour is that the AMQP broker closes the channel with a “not found” exception, and our code retries opening the channel. It seems pika trips up on the latter bit, confusing the AMQP broker which then responds by closing the entire connection.

We’re using the Tornado client adapter on Python 2.7.

2018-02-05 01:02:34,929 pika.connection[connection.py:1382] WARNING Disconnected from RabbitMQ at rabbitmq:5672 (504): CHANNEL_ERROR - second 'channel.open' seen
2018-02-05 01:02:35,042 pika.adapters.base_connection[base_connection.py: 212] INFO Connecting to 172.18.0.4:5672
2018-02-05 01:02:35,047 pika.credentials[credentials.py:  71] INFO Erasing stored credential values
2018-02-05 01:02:35,763 pika.adapters.base_connection[base_connection.py: 180] WARNING Socket closed when connection was open
2018-02-05 01:02:35,764 pika.adapters.base_connection[base_connection.py: 308] WARNING Connection is closed but not stopping IOLoop
2018-02-05 01:02:35,764 pika.connection[connection.py:1382] WARNING Disconnected from RabbitMQ at rabbitmq:5672 (504): CHANNEL_ERROR - second 'channel.open' seen
2018-02-05 01:02:38,288 pika.adapters.base_connection[base_connection.py: 212] INFO Connecting to 172.18.0.4:5672
2018-02-05 01:02:38,295 pika.credentials[credentials.py:  71] INFO Erasing stored credential values
2018-02-05 01:02:38,995 pika.adapters.base_connection[base_connection.py: 180] WARNING Socket closed when connection was open
2018-02-05 01:02:38,996 pika.adapters.base_connection[base_connection.py: 308] WARNING Connection is closed but not stopping IOLoop
2018-02-05 01:02:38,997 pika.connection[connection.py:1382] WARNING Disconnected from RabbitMQ at rabbitmq:5672 (504): CHANNEL_ERROR - second 'channel.open' seen

I’ll also point out that the “Connection is closed but not stopping IOLoop” is annoying; the IOLoop does not belong to pika, so it isn’t pika’s place to say whether the IOLoop should be running or stopped.