websockets: Default compression settings of 10.0 trigger bug in AWS API Gateway server

Hello,

I’ve noticed an issue that appears to have started with websockets version 10.0. I have a websocket server run by the AWS API gateway. I use the websockets library to connect with my python program. Under 9.1 there was no issue receiving messages, but I noticed that as of 10.0 I stopped receiving the messages the server continuously sends (verified through wscat).

When I run with the logger I’d observe the connection being made as expected:

async with websockets.connect(my_url) as websocket:

DEBUG, > Upgrade: websocket
DEBUG, > Connection: Upgrade
DEBUG, > Sec-WebSocket-Key: cSPm5HYtYSZU3BoudrY91w==
DEBUG, > Sec-WebSocket-Version: 13
DEBUG, > Sec-WebSocket-Extensions: permessage-deflate; server_max_window_bits=12; client_max_window_bits=12
DEBUG, > User-Agent: Python/3.7 websockets/10.0
DEBUG, < HTTP/1.1 101 Switching Protocols
DEBUG, < Date: Wed, 06 Oct 2021 17:57:43 GMT
DEBUG, < Connection: upgrade
DEBUG, < upgrade: websocket
DEBUG, < sec-websocket-accept: e9Ztme4JDr4o30+FPAUZwoJMZcE=
DEBUG, < sec-websocket-extensions: permessage-deflate;server_max_window_bits=12
DEBUG, = connection is OPEN

Note that the server does not reply with any compression options. I think this is valid if the server doesn’t support compression.

I then wait on messages like so:

while True:
    msg = await websocket.recv()

Everything appears normal from a connection perspective, except messages should be flowing regularly instead of just default ping / pong style responses:

DEBUG, %% sending keepalive ping
DEBUG, > PING 3c 93 7f 61 [binary, 4 bytes]
DEBUG, < PONG 3c 93 7f 61 [binary, 4 bytes]

I discovered if I set compression=None in connect ie:

async with websockets.connect(my_url, compression=None) as websocket:

Then messages from my server flow as usual as they did in 9.1.

It seems like compression should be something AWS supports on their side and will also engage with their support. However, since things appear to work with wscat, I think there may also be an issue with the websockets library itself in supporting servers that do not support compression.

To help debugging, I’m hosting a websocket server through the API Gateway which outputs regular messages to any connected client and does not require any authentication. I won’t guarantee it will stay up indefinitely (sorry to any future readers!)

To test, run: wscat -c wss://n36vwxc045.execute-api.us-east-2.amazonaws.com/test This should work and you’ll see the output < {"thanks": "for investigating this"} at about 1/2 Hz.

If you use the following code with the websockets 10.0 library to read the messages:

import asyncio
import websockets
import socket

import logging

logging.basicConfig(
    format="%(message)s",
    level=logging.DEBUG,
)

def main() -> None:
    asyncio.get_event_loop().run_until_complete(
        listen_forever(
            "wss://n36vwxc045.execute-api.us-east-2.amazonaws.com/test"
        )
    )


async def listen_forever(
    url: str
) -> None:
    """
    Listen for a websocket message, reconnecting if something bad happens.

    Args:
      url (str): The URL to which to connect.

    """
    # outer loop restarted every time the connection fails
    while True:
        try:
            async with websockets.connect(
                url,
                # compression=None  # UNCOMMENT THIS LINE TO GET MESSAGES
            ) as websocket:
                print('Websocket connected, waiting for messages...')
                while True:
                    msg = await websocket.recv()
                    print(msg)
        except socket.gaierror as exp:
            print(exp)
            continue
        except ConnectionRefusedError as exp:
            print(exp)
            continue
        except websockets.exceptions.ConnectionClosedOK:
            print(
                'Server closed connection (2 hour timeout?), reconnecting.'
            )
            continue
        except websockets.exceptions.InvalidStatusCode as exp:
            print(
                'Server returned an InvalidStatusCode - {}. Reconnecting.'.format(
                    exp
                )
            )
            continue
        except asyncio.streams.IncompleteReadError as exp:
            print(
                'Server an IncompleteReadError - {}. Reconnecting.'.format(
                    exp
                )
            )
            continue
        except Exception as exp:
            print(
                'Unhandled exception - {}. Reconnecting.'.format(
                    exp
                )
            )
            continue


if __name__ == '__main__':
    main()

You’ll observe no message output unless you uncomment line 35.

Thanks for all your great work on this project!