prefect: Client sometimes fails with `httpx.LocalProtocolError`

First check

• I added a descriptive title to this issue.
• I used the GitHub search to find a similar issue and didn’t find it.
• I searched the Prefect documentation for this issue.
• I checked that this issue is related to Prefect and not one of its dependencies.

Bug summary

Occasionally it seems agent is not able to retrieve the jobs to run from Prefect Cloud. It may be hours until the issue appears.

The problem had been discovered, when we received a “Late” scheduled flow notification and consequently by checking the agent logs.

The error appeared the same day we upgraded from 2.4.0 to 2.6.5 via pip.

Prefect slack thread for reference

Reproduction

1. Prefect agent 2.6.5 (or 2.6.6) running on AWS EC2 with 22.04.1 LTS (Jammy Jellyfish) x86_64
2. `pip install prefect==2.6.6`
3. `prefect agent start -q 'my_queue'`
4. tail the logs until one sees the failed scheduled flows

Error

22:24:31.719 | ERROR   | prefect.agent - Invalid input ConnectionInputs.SEND_HEADERS in state ConnectionState.CLOSED
Traceback (most recent call last):
  File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/h2/connection.py", line 224, in process_input
    func, target_state = self._transitions[(self.state, input_)]
KeyError: (<ConnectionState.CLOSED: 3>, <ConnectionInputs.SEND_HEADERS: 0>)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/httpcore/_async/http2.py", line 108, in handle_async_request
    await self._send_request_headers(request=request, stream_id=stream_id)
  File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/httpcore/_async/http2.py", line 205, in _send_request_headers
    self._h2_state.send_headers(stream_id, headers, end_stream=end_stream)
  File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/h2/connection.py", line 766, in send_headers
    self.state_machine.process_input(ConnectionInputs.SEND_HEADERS)
  File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/h2/connection.py", line 228, in process_input
    raise ProtocolError(
h2.exceptions.ProtocolError: Invalid input ConnectionInputs.SEND_HEADERS in state ConnectionState.CLOSED
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/httpx/_transports/default.py", line 60, in map_httpcore_exceptions
    yield
  File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/httpx/_transports/default.py", line 353, in handle_async_request
    resp = await self._pool.handle_async_request(req)
  File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/httpcore/_async/connection_pool.py", line 253, in handle_async_request
    raise exc
  File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/httpcore/_async/connection_pool.py", line 237, in handle_async_request
    response = await connection.handle_async_request(request)
  File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/httpcore/_async/connection.py", line 90, in handle_async_request
    return await self._connection.handle_async_request(request)
  File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/httpcore/_async/http2.py", line 144, in handle_async_request
    raise LocalProtocolError(exc)  # pragma: nocover
httpcore.LocalProtocolError: Invalid input ConnectionInputs.SEND_HEADERS in state ConnectionState.CLOSED
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/prefect/agent.py", line 154, in get_and_submit_flow_runs
    queue_runs = await self.client.get_runs_in_work_queue(
  File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/prefect/client/orion.py", line 759, in get_runs_in_work_queue
    response = await self._client.post(
  File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/httpx/_client.py", line 1842, in post
    return await self.request(
  File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/httpx/_client.py", line 1527, in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
  File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/prefect/client/base.py", line 159, in send
    await super().send(*args, **kwargs)
  File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/httpx/_client.py", line 1614, in send
    response = await self._send_handling_auth(
  File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/httpx/_client.py", line 1642, in _send_handling_auth
    response = await self._send_handling_redirects(
  File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/httpx/_client.py", line 1679, in _send_handling_redirects
    response = await self._send_single_request(request)
  File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/httpx/_client.py", line 1716, in _send_single_request
    response = await transport.handle_async_request(request)
  File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/httpx/_transports/default.py", line 352, in handle_async_request
    with map_httpcore_exceptions():
  File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/httpx/_transports/default.py", line 77, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.LocalProtocolError: Invalid input ConnectionInputs.SEND_HEADERS in state ConnectionState.CLOSED
22:24:31.800 | ERROR   | prefect.agent - Invalid input ConnectionInputs.SEND_HEADERS in state ConnectionState.CLOSED
Traceback (most recent call last):
  File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/h2/connection.py", line 224, in process_input
    func, target_state = self._transitions[(self.state, input_)]
KeyError: (<ConnectionState.CLOSED: 3>, <ConnectionInputs.SEND_HEADERS: 0>)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/httpcore/_async/http2.py", line 108, in handle_async_request
    await self._send_request_headers(request=request, stream_id=stream_id)
  File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/httpcore/_async/http2.py", line 205, in _send_request_headers
    self._h2_state.send_headers(stream_id, headers, end_stream=end_stream)
  File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/h2/connection.py", line 766, in send_headers
    self.state_machine.process_input(ConnectionInputs.SEND_HEADERS)
  File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/h2/connection.py", line 228, in process_input
    raise ProtocolError(
h2.exceptions.ProtocolError: Invalid input ConnectionInputs.SEND_HEADERS in state ConnectionState.CLOSED
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/httpx/_transports/default.py", line 60, in map_httpcore_exceptions
    yield
  File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/httpx/_transports/default.py", line 353, in handle_async_request
    resp = await self._pool.handle_async_request(req)
  File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/httpcore/_async/connection_pool.py", line 253, in handle_async_request
    raise exc
  File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/httpcore/_async/connection_pool.py", line 237, in handle_async_request
    response = await connection.handle_async_request(request)
  File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/httpcore/_async/connection.py", line 90, in handle_async_request
    return await self._connection.handle_async_request(request)
  File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/httpcore/_async/http2.py", line 144, in handle_async_request
    raise LocalProtocolError(exc)  # pragma: nocover
httpcore.LocalProtocolError: Invalid input ConnectionInputs.SEND_HEADERS in state ConnectionState.CLOSED
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/prefect/agent.py", line 154, in get_and_submit_flow_runs
    queue_runs = await self.client.get_runs_in_work_queue(
  File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/prefect/client/orion.py", line 759, in get_runs_in_work_queue
    response = await self._client.post(
  File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/httpx/_client.py", line 1842, in post
    return await self.request(
  File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/httpx/_client.py", line 1527, in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
  File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/prefect/client/base.py", line 159, in send
    await super().send(*args, **kwargs)
  File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/httpx/_client.py", line 1614, in send
    response = await self._send_handling_auth(
  File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/httpx/_client.py", line 1642, in _send_handling_auth
    response = await self._send_handling_redirects(
  File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/httpx/_client.py", line 1679, in _send_handling_redirects
    response = await self._send_single_request(request)
  File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/httpx/_client.py", line 1716, in _send_single_request
    response = await transport.handle_async_request(request)
  File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/httpx/_transports/default.py", line 352, in handle_async_request
    with map_httpcore_exceptions():
  File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/httpx/_transports/default.py", line 77, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.LocalProtocolError: Invalid input ConnectionInputs.SEND_HEADERS in state ConnectionState.CLOSED

Versions

Version:             2.6.6
API version:         0.8.3
Python version:      3.10.6
Git commit:          87767cda
Built:               Thu, Nov 3, 2022 1:15 PM
OS/Arch:             linux/x86_64
Profile:             production
Server type:         cloud

Additional context

output of pip install prefect==2.6.6

$ pip install prefect==2.6.6
Requirement already satisfied: prefect==2.6.6 in ./venv/lib/python3.10/site-packages (2.6.6)
Requirement already satisfied: click<8.2,>=8.0 in ./venv/lib/python3.10/site-packages (from prefect==2.6.6) (8.1.3)
Requirement already satisfied: pendulum>=2.1.2 in ./venv/lib/python3.10/site-packages (from prefect==2.6.6) (2.1.2)
Requirement already satisfied: aiofiles>=0.7.0 in ./venv/lib/python3.10/site-packages (from prefect==2.6.6) (22.1.0)
Requirement already satisfied: pytz>=2021.1 in ./venv/lib/python3.10/site-packages (from prefect==2.6.6) (2022.2.1)
Requirement already satisfied: apprise>=1.0.0 in ./venv/lib/python3.10/site-packages (from prefect==2.6.6) (1.0.0)
Requirement already satisfied: kubernetes>=24.2.0 in ./venv/lib/python3.10/site-packages (from prefect==2.6.6) (24.2.0)
Requirement already satisfied: pydantic>=1.10.0 in ./venv/lib/python3.10/site-packages (from prefect==2.6.6) (1.10.2)
Requirement already satisfied: rich>=11.0 in ./venv/lib/python3.10/site-packages (from prefect==2.6.6) (12.5.1)
Requirement already satisfied: anyio>=3.4.0 in ./venv/lib/python3.10/site-packages (from prefect==2.6.6) (3.6.1)
Requirement already satisfied: asgi-lifespan>=1.0 in ./venv/lib/python3.10/site-packages (from prefect==2.6.6) (1.0.1)
Requirement already satisfied: cloudpickle>=2.0 in ./venv/lib/python3.10/site-packages (from prefect==2.6.6) (2.2.0)
Requirement already satisfied: asyncpg>=0.23 in ./venv/lib/python3.10/site-packages (from prefect==2.6.6) (0.26.0)
Requirement already satisfied: aiohttp>=3.8.1 in ./venv/lib/python3.10/site-packages (from prefect==2.6.6) (3.8.3)
Requirement already satisfied: sqlalchemy[asyncio]!=1.4.33,>=1.4.20 in ./venv/lib/python3.10/site-packages (from prefect==2.6.6) (1.4.41)
Requirement already satisfied: cryptography>=36.0.1 in ./venv/lib/python3.10/site-packages (from prefect==2.6.6) (38.0.1)
Requirement already satisfied: pathspec>=0.8.0 in ./venv/lib/python3.10/site-packages (from prefect==2.6.6) (0.10.1)
Requirement already satisfied: toml>=0.10.0 in ./venv/lib/python3.10/site-packages (from prefect==2.6.6) (0.10.2)
Requirement already satisfied: coolname>=1.0.4 in ./venv/lib/python3.10/site-packages (from prefect==2.6.6) (1.1.0)
Requirement already satisfied: uvicorn>=0.14.0 in ./venv/lib/python3.10/site-packages (from prefect==2.6.6) (0.18.3)
Requirement already satisfied: python-slugify>=5.0 in ./venv/lib/python3.10/site-packages (from prefect==2.6.6) (6.1.2)
Requirement already satisfied: readchar>=3.0.6 in ./venv/lib/python3.10/site-packages (from prefect==2.6.6) (4.0.3)
Requirement already satisfied: orjson>=3.7 in ./venv/lib/python3.10/site-packages (from prefect==2.6.6) (3.8.0)
Requirement already satisfied: httpx[http2]>=0.23 in ./venv/lib/python3.10/site-packages (from prefect==2.6.6) (0.23.0)
Requirement already satisfied: docker>=4.0 in ./venv/lib/python3.10/site-packages (from prefect==2.6.6) (6.0.0)
Requirement already satisfied: fsspec>=2022.5.0 in ./venv/lib/python3.10/site-packages (from prefect==2.6.6) (2022.8.2)
Requirement already satisfied: typer>=0.4.1 in ./venv/lib/python3.10/site-packages (from prefect==2.6.6) (0.6.1)
Requirement already satisfied: alembic>=1.7.5 in ./venv/lib/python3.10/site-packages (from prefect==2.6.6) (1.8.1)
Requirement already satisfied: aiosqlite>=0.17.0 in ./venv/lib/python3.10/site-packages (from prefect==2.6.6) (0.17.0)
Requirement already satisfied: pyyaml>=5.4.1 in ./venv/lib/python3.10/site-packages (from prefect==2.6.6) (6.0)
Requirement already satisfied: croniter>=1.0.12 in ./venv/lib/python3.10/site-packages (from prefect==2.6.6) (1.3.7)
Requirement already satisfied: packaging>=21.3 in ./venv/lib/python3.10/site-packages (from prefect==2.6.6) (21.3)
Requirement already satisfied: jsonpatch>=1.32 in ./venv/lib/python3.10/site-packages (from prefect==2.6.6) (1.32)
Requirement already satisfied: griffe>=0.20.0 in ./venv/lib/python3.10/site-packages (from prefect==2.6.6) (0.22.2)
Requirement already satisfied: fastapi>=0.70 in ./venv/lib/python3.10/site-packages (from prefect==2.6.6) (0.85.0)
Requirement already satisfied: typing-extensions>=4.1.0 in ./venv/lib/python3.10/site-packages (from prefect==2.6.6) (4.3.0)
Requirement already satisfied: aiosignal>=1.1.2 in ./venv/lib/python3.10/site-packages (from aiohttp>=3.8.1->prefect==2.6.6) (1.2.0)
Requirement already satisfied: frozenlist>=1.1.1 in ./venv/lib/python3.10/site-packages (from aiohttp>=3.8.1->prefect==2.6.6) (1.3.1)
Requirement already satisfied: charset-normalizer<3.0,>=2.0 in ./venv/lib/python3.10/site-packages (from aiohttp>=3.8.1->prefect==2.6.6) (2.1.1)
Requirement already satisfied: multidict<7.0,>=4.5 in ./venv/lib/python3.10/site-packages (from aiohttp>=3.8.1->prefect==2.6.6) (6.0.2)
Requirement already satisfied: attrs>=17.3.0 in ./venv/lib/python3.10/site-packages (from aiohttp>=3.8.1->prefect==2.6.6) (22.1.0)
Requirement already satisfied: async-timeout<5.0,>=4.0.0a3 in ./venv/lib/python3.10/site-packages (from aiohttp>=3.8.1->prefect==2.6.6) (4.0.2)
Requirement already satisfied: yarl<2.0,>=1.0 in ./venv/lib/python3.10/site-packages (from aiohttp>=3.8.1->prefect==2.6.6) (1.8.1)
Requirement already satisfied: Mako in ./venv/lib/python3.10/site-packages (from alembic>=1.7.5->prefect==2.6.6) (1.2.3)
Requirement already satisfied: idna>=2.8 in ./venv/lib/python3.10/site-packages (from anyio>=3.4.0->prefect==2.6.6) (3.4)
Requirement already satisfied: sniffio>=1.1 in ./venv/lib/python3.10/site-packages (from anyio>=3.4.0->prefect==2.6.6) (1.3.0)
Requirement already satisfied: six in ./venv/lib/python3.10/site-packages (from apprise>=1.0.0->prefect==2.6.6) (1.16.0)
Requirement already satisfied: requests-oauthlib in ./venv/lib/python3.10/site-packages (from apprise>=1.0.0->prefect==2.6.6) (1.3.1)
Requirement already satisfied: requests in ./venv/lib/python3.10/site-packages (from apprise>=1.0.0->prefect==2.6.6) (2.28.1)
Requirement already satisfied: markdown in ./venv/lib/python3.10/site-packages (from apprise>=1.0.0->prefect==2.6.6) (3.4.1)
Requirement already satisfied: python-dateutil in ./venv/lib/python3.10/site-packages (from croniter>=1.0.12->prefect==2.6.6) (2.8.2)
Requirement already satisfied: cffi>=1.12 in ./venv/lib/python3.10/site-packages (from cryptography>=36.0.1->prefect==2.6.6) (1.15.1)
Requirement already satisfied: websocket-client>=0.32.0 in ./venv/lib/python3.10/site-packages (from docker>=4.0->prefect==2.6.6) (1.4.1)
Requirement already satisfied: urllib3>=1.26.0 in ./venv/lib/python3.10/site-packages (from docker>=4.0->prefect==2.6.6) (1.26.12)
Requirement already satisfied: starlette==0.20.4 in ./venv/lib/python3.10/site-packages (from fastapi>=0.70->prefect==2.6.6) (0.20.4)
Requirement already satisfied: certifi in ./venv/lib/python3.10/site-packages (from httpx[http2]>=0.23->prefect==2.6.6) (2022.9.24)
Requirement already satisfied: httpcore<0.16.0,>=0.15.0 in ./venv/lib/python3.10/site-packages (from httpx[http2]>=0.23->prefect==2.6.6) (0.15.0)
Requirement already satisfied: rfc3986[idna2008]<2,>=1.3 in ./venv/lib/python3.10/site-packages (from httpx[http2]>=0.23->prefect==2.6.6) (1.5.0)
Requirement already satisfied: h2<5,>=3 in ./venv/lib/python3.10/site-packages (from httpx[http2]>=0.23->prefect==2.6.6) (4.1.0)
Requirement already satisfied: jsonpointer>=1.9 in ./venv/lib/python3.10/site-packages (from jsonpatch>=1.32->prefect==2.6.6) (2.3)
Requirement already satisfied: google-auth>=1.0.1 in ./venv/lib/python3.10/site-packages (from kubernetes>=24.2.0->prefect==2.6.6) (2.11.1)
Requirement already satisfied: setuptools>=21.0.0 in ./venv/lib/python3.10/site-packages (from kubernetes>=24.2.0->prefect==2.6.6) (59.6.0)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in ./venv/lib/python3.10/site-packages (from packaging>=21.3->prefect==2.6.6) (3.0.9)
Requirement already satisfied: pytzdata>=2020.1 in ./venv/lib/python3.10/site-packages (from pendulum>=2.1.2->prefect==2.6.6) (2020.1)
Requirement already satisfied: text-unidecode>=1.3 in ./venv/lib/python3.10/site-packages (from python-slugify>=5.0->prefect==2.6.6) (1.3)
Requirement already satisfied: pygments<3.0.0,>=2.6.0 in ./venv/lib/python3.10/site-packages (from rich>=11.0->prefect==2.6.6) (2.13.0)
Requirement already satisfied: commonmark<0.10.0,>=0.9.0 in ./venv/lib/python3.10/site-packages (from rich>=11.0->prefect==2.6.6) (0.9.1)
Requirement already satisfied: greenlet!=0.4.17 in ./venv/lib/python3.10/site-packages (from sqlalchemy[asyncio]!=1.4.33,>=1.4.20->prefect==2.6.6) (1.1.3)
Requirement already satisfied: h11>=0.8 in ./venv/lib/python3.10/site-packages (from uvicorn>=0.14.0->prefect==2.6.6) (0.12.0)
Requirement already satisfied: pycparser in ./venv/lib/python3.10/site-packages (from cffi>=1.12->cryptography>=36.0.1->prefect==2.6.6) (2.21)
Requirement already satisfied: pyasn1-modules>=0.2.1 in ./venv/lib/python3.10/site-packages (from google-auth>=1.0.1->kubernetes>=24.2.0->prefect==2.6.6) (0.2.8)
Requirement already satisfied: rsa<5,>=3.1.4 in ./venv/lib/python3.10/site-packages (from google-auth>=1.0.1->kubernetes>=24.2.0->prefect==2.6.6) (4.9)
Requirement already satisfied: cachetools<6.0,>=2.0.0 in ./venv/lib/python3.10/site-packages (from google-auth>=1.0.1->kubernetes>=24.2.0->prefect==2.6.6) (5.2.0)
Requirement already satisfied: hpack<5,>=4.0 in ./venv/lib/python3.10/site-packages (from h2<5,>=3->httpx[http2]>=0.23->prefect==2.6.6) (4.0.0)
Requirement already satisfied: hyperframe<7,>=6.0 in ./venv/lib/python3.10/site-packages (from h2<5,>=3->httpx[http2]>=0.23->prefect==2.6.6) (6.0.1)
Requirement already satisfied: MarkupSafe>=0.9.2 in ./venv/lib/python3.10/site-packages (from Mako->alembic>=1.7.5->prefect==2.6.6) (2.1.1)
Requirement already satisfied: oauthlib>=3.0.0 in ./venv/lib/python3.10/site-packages (from requests-oauthlib->apprise>=1.0.0->prefect==2.6.6) (3.2.1)
Requirement already satisfied: pyasn1<0.5.0,>=0.4.6 in ./venv/lib/python3.10/site-packages (from pyasn1-modules>=0.2.1->google-auth>=1.0.1->kubernetes>=24.2.0->prefect==2.6.6) (0.4.8)