httpx: h11._util.RemoteProtocolError: can't handle event type ConnectionClosed when role=SERVER and state=SEND_RESPONSE

There are two web servers. testapp.py

from fastapi import FastAPI
from starlette.requests import Request
from starlette.responses import JSONResponse
import http3

app = FastAPI()
http_client = http3.AsyncClient()

@app.middleware('http')
async def sso_middleware(request: Request, call_next):
    r = await http_client.post('http://127.0.0.1:8001')
    if r.status_code != 200:
        return JSONResponse({'ok': 0, 'data': {'status_code': r.status_code}})
    ret = r.json()
    await r.close()
    print(ret)
    response = await call_next(request)
    return response

@app.get('/')
def index(request: Request):
    return {"ok": 1, "data": "welcome to test app!"}

if __name__ == "__main__":
    import uvicorn
    uvicorn.run(app, host="127.0.0.1", port=8000)
    pass

testapp1.py

from fastapi import FastAPI
from starlette.requests import Request

app = FastAPI()

@app.get('/')
@app.post('/')
def index(request: Request):
    return {"ok": 1, "data": "welcome to test app 11111111111!"}

if __name__ == "__main__":
    import uvicorn
    uvicorn.run(app, host="127.0.0.1", port=8001)
    pass

The reproduce steps:

1. run the two servers
2. open the http://127.0.0.1:8000/ in browser
3. refresh the page. the response is 200 and ok.
4. then stop refreshing, do some other works for seconds
5. switch back and refresh again, the first refresh result is 500 most of the time

This happens, not every time. The logs again:

INFO: ('127.0.0.1', 14501) - "GET / HTTP/1.1" 200
{'ok': 1, 'data': 'welcome to test app 11111111111!'}
INFO: ('127.0.0.1', 14501) - "GET / HTTP/1.1" 200
INFO: ('127.0.0.1', 14553) - "GET / HTTP/1.1" 500
ERROR: Exception in ASGI application
Traceback (most recent call last):
  File "C:\Users\lch\.virtualenvs\authcenter-5r0y-06W\lib\site-packages\uvicorn\protocols\http\h11_impl.py", line 370, in run_asgi
    result = await app(self.scope, self.receive, self.send)
  File "C:\Users\lch\.virtualenvs\authcenter-5r0y-06W\lib\site-packages\starlette\applications.py", line 133, in __call__
    await self.error_middleware(scope, receive, send)
  File "C:\Users\lch\.virtualenvs\authcenter-5r0y-06W\lib\site-packages\starlette\middleware\errors.py", line 122, in __call__
    raise exc from None
  File "C:\Users\lch\.virtualenvs\authcenter-5r0y-06W\lib\site-packages\starlette\middleware\errors.py", line 100, in __call__
    await self.app(scope, receive, _send)
  File "C:\Users\lch\.virtualenvs\authcenter-5r0y-06W\lib\site-packages\starlette\middleware\base.py", line 25, in __call__
    response = await self.dispatch_func(request, self.call_next)
  File "testapp.py", line 13, in sso_middleware
    r = await http_client.post('http://127.0.0.1:8001')
  File "C:\Users\lch\.virtualenvs\authcenter-5r0y-06W\lib\site-packages\http3\client.py", line 406, in post
    timeout=timeout,
  File "C:\Users\lch\.virtualenvs\authcenter-5r0y-06W\lib\site-packages\http3\client.py", line 548, in request
    timeout=timeout,
  File "C:\Users\lch\.virtualenvs\authcenter-5r0y-06W\lib\site-packages\http3\client.py", line 145, in send
    allow_redirects=allow_redirects,
  File "C:\Users\lch\.virtualenvs\authcenter-5r0y-06W\lib\site-packages\http3\client.py", line 178, in send_handling_redirects
    request, verify=verify, cert=cert, timeout=timeout
  File "C:\Users\lch\.virtualenvs\authcenter-5r0y-06W\lib\site-packages\http3\dispatch\connection_pool.py", line 130, in send
    raise exc
  File "C:\Users\lch\.virtualenvs\authcenter-5r0y-06W\lib\site-packages\http3\dispatch\connection_pool.py", line 121, in send
    request, verify=verify, cert=cert, timeout=timeout
  File "C:\Users\lch\.virtualenvs\authcenter-5r0y-06W\lib\site-packages\http3\dispatch\connection.py", line 59, in send
    response = await self.h11_connection.send(request, timeout=timeout)
  File "C:\Users\lch\.virtualenvs\authcenter-5r0y-06W\lib\site-packages\http3\dispatch\http11.py", line 58, in send
    http_version, status_code, headers = await self._receive_response(timeout)
  File "C:\Users\lch\.virtualenvs\authcenter-5r0y-06W\lib\site-packages\http3\dispatch\http11.py", line 130, in _receive_response
    event = await self._receive_event(timeout)
  File "C:\Users\lch\.virtualenvs\authcenter-5r0y-06W\lib\site-packages\http3\dispatch\http11.py", line 161, in _receive_event
    event = self.h11_state.next_event()
  File "C:\Users\lch\.virtualenvs\authcenter-5r0y-06W\lib\site-packages\h11\_connection.py", line 439, in next_event
    exc._reraise_as_remote_protocol_error()
  File "C:\Users\lch\.virtualenvs\authcenter-5r0y-06W\lib\site-packages\h11\_util.py", line 72, in _reraise_as_remote_protocol_error
    raise self
  File "C:\Users\lch\.virtualenvs\authcenter-5r0y-06W\lib\site-packages\h11\_connection.py", line 422, in next_event
    self._process_event(self.their_role, event)
  File "C:\Users\lch\.virtualenvs\authcenter-5r0y-06W\lib\site-packages\h11\_connection.py", line 238, in _process_event
    self._cstate.process_event(role, type(event), server_switch_event)
  File "C:\Users\lch\.virtualenvs\authcenter-5r0y-06W\lib\site-packages\h11\_state.py", line 238, in process_event
    self._fire_event_triggered_transitions(role, event_type)
  File "C:\Users\lch\.virtualenvs\authcenter-5r0y-06W\lib\site-packages\h11\_state.py", line 253, in _fire_event_triggered_transitions
    .format(event_type.__name__, role, self.states[role]))
h11._util.RemoteProtocolError: can't handle event type ConnectionClosed when role=SERVER and state=SEND_RESPONSE
{'ok': 1, 'data': 'welcome to test app 11111111111!'}
INFO: ('127.0.0.1', 14570) - "GET / HTTP/1.1" 200

Sorry for my poor English.

After upgrading from httpx 0.18.2 to 0.21.1 we also suffer from this problem again. As a workaround we pinned httpx to the old version where everything seems to work fine.

fyi @marns93

I don’t think there are any actual behavioral bugs being discussed in this thread. It’s just about how to give a better error message. So I think either there’s something wrong with your server setup that’s causing it to close connections abruptly without sending responses, or else you’ve found a different bug in httpx and should probably open a new issue so your problem doesn’t get lost in the noise.

I can reproduce it. I’ve narrowed it down to this “caller” script:

import asyncio
import sys

import httpx

http_client = httpx.AsyncClient()


async def request(port):
    print("Performing request")
    resp = await http_client.get(f"http://localhost:{port}")
    if resp.status_code != 200:
        raise Exception("Unexpected non-200 response")
    print("Got response", resp.content, "\n")
    await resp.close()


async def main(port, timeout=5):
    await request(port)
    await asyncio.sleep(timeout)
    await request(port)


if __name__ == "__main__":
    if len(sys.argv) < 2:
        print("Please specify a port number to connect to")
    else:
        port = sys.argv[1]
        asyncio.run(main(port))

It is not reproducible against a simple Flask WSGI server:

from flask import Flask

app = Flask(__name__)


@app.route("/", methods=["GET", "POST"])
def index():
    return {"ok": 1, "data": "welcome to test app 11111111111!"}


if __name__ == "__main__":
    app.run(port=5000)

The script outputs:

Performing request
Got response b'{"data":"welcome to test app 11111111111!","ok":1}\n'

Performing request
Got response b'{"data":"welcome to test app 11111111111!","ok":1}\n

But it fails with ASGI servers, interestingly it fails for both FastAPI+Uvicorn and Quart+Hypercorn:

from fastapi import FastAPI

app = FastAPI()


@app.get("/")
@app.post("/")
def index(request):
    return {"ok": 1, "data": "welcome to test app 11111111111!"}


if __name__ == "__main__":
    import uvicorn
    uvicorn.run(app, host="127.0.0.1", port=8001)

The Quart code:

from quart import Quart

app = Quart(__name__)


@app.route("/", methods=["GET", "POST"])
async def hello():
    return "Hello, World!"


if __name__ == "__main__":
    app.run(port=5001)

For both implementations the script’s output is:

Performing request
Got response b'Hello, World!'

Performing request
Traceback (most recent call last):
  File "/Users/yeray/.pyenv/versions/httpx/lib/python3.7/site-packages/h11/_state.py", line 249, in _fire_event_triggered_transitions
    new_state = EVENT_TRIGGERED_TRANSITIONS[role][state][event_type]
KeyError: <class 'h11._events.ConnectionClosed'>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "client_script.py", line 29, in <module>
    asyncio.run(main(port))
  File "/Users/yeray/.pyenv/versions/3.7.3/lib/python3.7/asyncio/runners.py", line 43, in run
    return loop.run_until_complete(main)
  File "/Users/yeray/.pyenv/versions/3.7.3/lib/python3.7/asyncio/base_events.py", line 584, in run_until_complete
    return future.result()
  File "client_script.py", line 21, in main
    await request(port)
  File "client_script.py", line 11, in request
    resp = await http_client.get(f"http://localhost:{port}")
  File "/Users/yeray/code/personal/_forks/httpx/httpx/client.py", line 316, in get
    timeout=timeout,
  File "/Users/yeray/code/personal/_forks/httpx/httpx/client.py", line 548, in request
    timeout=timeout,
  File "/Users/yeray/code/personal/_forks/httpx/httpx/client.py", line 144, in send
    allow_redirects=allow_redirects,
  File "/Users/yeray/code/personal/_forks/httpx/httpx/client.py", line 177, in send_handling_redirects
    request, verify=verify, cert=cert, timeout=timeout
  File "/Users/yeray/code/personal/_forks/httpx/httpx/dispatch/connection_pool.py", line 128, in send
    raise exc
  File "/Users/yeray/code/personal/_forks/httpx/httpx/dispatch/connection_pool.py", line 119, in send
    request, verify=verify, cert=cert, timeout=timeout
  File "/Users/yeray/code/personal/_forks/httpx/httpx/dispatch/connection.py", line 54, in send
    response = await self.h11_connection.send(request, timeout=timeout)
  File "/Users/yeray/code/personal/_forks/httpx/httpx/dispatch/http11.py", line 58, in send
    http_version, status_code, headers = await self._receive_response(timeout)
  File "/Users/yeray/code/personal/_forks/httpx/httpx/dispatch/http11.py", line 130, in _receive_response
    event = await self._receive_event(timeout)
  File "/Users/yeray/code/personal/_forks/httpx/httpx/dispatch/http11.py", line 161, in _receive_event
    event = self.h11_state.next_event()
  File "/Users/yeray/.pyenv/versions/httpx/lib/python3.7/site-packages/h11/_connection.py", line 439, in next_event
    exc._reraise_as_remote_protocol_error()
  File "/Users/yeray/.pyenv/versions/httpx/lib/python3.7/site-packages/h11/_util.py", line 72, in _reraise_as_remote_protocol_error
    raise self
  File "/Users/yeray/.pyenv/versions/httpx/lib/python3.7/site-packages/h11/_connection.py", line 422, in next_event
    self._process_event(self.their_role, event)
  File "/Users/yeray/.pyenv/versions/httpx/lib/python3.7/site-packages/h11/_connection.py", line 238, in _process_event
    self._cstate.process_event(role, type(event), server_switch_event)
  File "/Users/yeray/.pyenv/versions/httpx/lib/python3.7/site-packages/h11/_state.py", line 238, in process_event
    self._fire_event_triggered_transitions(role, event_type)
  File "/Users/yeray/.pyenv/versions/httpx/lib/python3.7/site-packages/h11/_state.py", line 253, in _fire_event_triggered_transitions
    .format(event_type.__name__, role, self.states[role]))
h11._util.RemoteProtocolError: can't handle event type ConnectionClosed when role=SERVER and state=SEND_RESPONSE

@vyahello Yes, this fix along with some others will be released soon, probably as 0.7.9. 😃 If you need them now you can install from master.

Here’s a simple reproducible example. It uses two separate files and requires fastapi and httpx.

FastAPI Server:

from fastapi import FastAPI

app = FastAPI()

@app.get("/hello")
async def hello():
    return "World"

# To Run:
# gunicorn -k uvicorn.workers.UvicornWorker -b 0.0.0.0:80  test:app

Client:

#!/usr/bin/env python3

import httpx


def main():
    try:
        while True:
            r = httpx.get("http://localhost/hello")
            print(r.text)
    except KeyboardInterrupt:
        print("done")


if __name__ == "__main__":
    main()

In one terminal run the server (I’m using gunicorn with a uvicorn worker since that’s how I normally run my FastAPI applications).

In a second terminal, launch the test (I just did ./test.py). Let it run for a second and then kill the server with a CTRL-C in the first terminal. The test client, in the second terminal, should die with the expected h11._util.RemoteProtocolError exception. In a quick test (~5 tries) I got the expected error 4 times and a ConnectionRefused error once, which kind of makes sense since it appears to be a race condition of whether or not the server finishes sending the response before it dies.

For what it’s worth I’m running on macOS (Mojave) with Homebrew Python3.7 in a virtual environment with the latest httpx and fastapi.

Relevant snippet from the h11 docs:

Connection.next_event Raises: RemoteProtocolError – The peer has misbehaved. You should close the connection (possibly after sending some kind of 4xx response)

I could reproduce with this snippet:

import asyncio
async def main():
    async with AsyncClient() as session:
        await asyncio.wait([
           asyncio.create_task(session.get('http://localhost:80'))
           for _ in range(1000)
        ])
asyncio.run(main())

the ConnectionClosed seems to be generated only in this place in h11:

    def _extract_next_receive_event(self):
        state = self.their_state
        # We don't pause immediately when they enter DONE, because even in
        # DONE state we can still process a ConnectionClosed() event. But
        # if we have data in our buffer, then we definitely aren't getting
        # a ConnectionClosed() immediately and we need to pause.
        if state is DONE and self._receive_buffer:
            return PAUSED
        if state is MIGHT_SWITCH_PROTOCOL or state is SWITCHED_PROTOCOL:
            return PAUSED
        assert self._reader is not None
        event = self._reader(self._receive_buffer)
        if event is None:
            if not self._receive_buffer and self._receive_buffer_closed:
                # In some unusual cases (basically just HTTP/1.0 bodies), EOF
                # triggers an actual protocol event; in that case, we want to
                # return that event, and then the state will change and we'll
                # get called again to generate the actual ConnectionClosed().
                if hasattr(self._reader, "read_eof"):
                    event = self._reader.read_eof()
                else:
                    event = ConnectionClosed()
        if event is None:
            event = NEED_DATA
        return event

and _receive_buffer_closed seems to be closed only in

    def receive_data(self, data):
        if data:
            if self._receive_buffer_closed:
                raise RuntimeError("received close, then received more data?")
            self._receive_buffer += data
        else:
            self._receive_buffer_closed = True

meaning it was called with an empty data

in turn, this comes from httpcore:

    async def _receive_event(self, timeout: float = None) -> H11Event:
        while True:
            with map_exceptions({h11.RemoteProtocolError: RemoteProtocolError}):
                event = self._h11_state.next_event()

            if event is h11.NEED_DATA:
                data = await self._network_stream.read(
                    self.READ_NUM_BYTES, timeout=timeout
                )
                self._h11_state.receive_data(data)
            else:
                return event

so _network_stream.read returned empty data

as I’m using asyncio, it’s an AsyncIOStream with this code:

    async def read(self, max_bytes: int, timeout: float = None) -> bytes:
        exc_map = {
            TimeoutError: ReadTimeout,
            anyio.BrokenResourceError: ReadError,
        }
        with map_exceptions(exc_map):
            with anyio.fail_after(timeout):
                try:
                    return await self._stream.receive(max_bytes=max_bytes)
                except anyio.EndOfStream:  # pragma: nocover
                    return b""

so I guess we reached EndOfStream on the client side then emitted the transition (their_role=SERVER, state=SEND_RESPONSE, event_type=ConnectionClosed) which is not managed.

When programming on the server side, it means client reached ConnectionClosed while the server is the sending the response/client is receiving, I think the bug does not happen since the server would not be calling receive_data while sending the response.

On the client side, it means the server reached ConnectionClosed while it is actually sending/while the client is receiving, and it trigger this bug.

Ok, nailed it: https://github.com/encode/starlette/issues/919

Hm. Got it. Thanks for the heads up.

I have my load test requests coming in to my server, I then make requests to a vendor server in order to answer the original request. Upon “requesting” these vendor requests, I see this issue.

Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/h11/_state.py", line 249, in _fire_event_triggered_transitions
new_state = EVENT_TRIGGERED_TRANSITIONS[role][state][event_type]
KeyError: <class 'h11._events.ConnectionClosed'>

Do you have any idea where to start looking, or are there known scenarios where this error comes up?

This exception could be seen if your web server closes a tcp connection prematurely (being overloaded could certainly cause this), or if a server dies, although it could just be a bug in the server code.

When using httpx you must capture certain exceptions when legitimate failuremodes occur (like readtimeouts, connect failures, etc).

This code would look something like this.

import httpx
try: 
   httpx.get(website)
except httpx.exceptions.HTTPError as e:
    print ("got unwanted exception [%s]" % e)

Unfortunately, seeing the stacktrace is because httpx is leaking an exception type from a dependent package (h11). For the moment, you must also do something like this:

import httpx
import h11
try: 
   httpx.get(website)
except httpx.exceptions.HTTPError as e:
    print ("got unwanted exception [%s]" % e)
except h11.RemoteProtocolError as e:
    print ("got unwanted exception [%s]" % e)

You can then decide how to handle the failure mode (close connection, retry, warn user, etc.).

Thanks @iwoloschin, that’s very helpful. I was able to reproduce using the even simpler setup below (plain ASGI app + uvicorn):

# app.py
async def app(scope, receive, send):
    assert scope["type"] == "http"
    await send(
        {
            "type": "http.response.start",
            "status": 200,
            "headers": [[b"content-type", b"text/plain"]],
        }
    )
    await send({"type": "http.response.body", "body": b"Hello, world!"})

# uvicorn app:app

I received the h11 error 3 times out of 5, the 2 other failures were an OSError. I’m running the same setup as you (except for Python 3.7 via pyenv).

Another piece of info: I get the same result whether I use the top-level API level, a Client, or an AsyncClient.

I’ll still second a vote for encapsulating exceptions to only throw HTTPX exceptions.

Agreed, yup.

Closed via #145 Released to PyPI as 0.6.8

Okay, so I think the right approach would be for us to have an equivelent is_connection_dropped on the Reader class in https://github.com/encode/httpx/blob/master/httpx/concurrency.py replacing the existing logic of “suck it and see”.

Does that make sense to you too, @yeraydiazdiaz? #143 is a great start - nice test case for the issue that can still be used here.

(Actually I think we’ll also want to combine the Reader and Writer interfaces into a single SocketConnection, but that’s a different story.)

I think this is caused by the client code in http3.