core: `aioesphomeapi` floods of errors+nodes slow

The problem

Since 2023.2, log is flooded with errors like below. ESPHome nodes go randomly offline, and became slow-responsive.

Things which are certain:

• Before HA 2023.2 everything was nice and snappy, no errors in the logs
• ESPHome nodes have up-to date firmware (at the time of writing this 2023.2.4)
• HA 2023.3 didn’t fix this.

What version of Home Assistant Core has the issue?

2023.3.0

What was the last working version of Home Assistant Core?

2023.1.x

What type of installation are you running?

Home Assistant OS

Integration causing the issue

aioesphomeapi

Link to integration documentation on our website

https://www.home-assistant.io/integrations/esphome/

Diagnostics information

No response

Example YAML snippet

No response

Anything in the logs that might be useful for us?

2023-03-02 15:40:11.629 WARNING (MainThread) [aioesphomeapi.connection] lampa-wc-fent @ 192.168.20.11: Connection error occurred: 
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/aioesphomeapi/connection.py", line 537, in send_message_await_response_complex
    await fut
asyncio.exceptions.CancelledError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/aioesphomeapi/connection.py", line 536, in send_message_await_response_complex
    async with async_timeout.timeout(timeout):
  File "/usr/local/lib/python3.10/site-packages/async_timeout/__init__.py", line 129, in __aexit__
    self._do_exit(exc_type)
  File "/usr/local/lib/python3.10/site-packages/async_timeout/__init__.py", line 212, in _do_exit
    raise asyncio.TimeoutError
asyncio.exceptions.TimeoutError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/aioesphomeapi/connection.py", line 320, in _keep_alive_loop
    await self._ping()
  File "/usr/local/lib/python3.10/site-packages/aioesphomeapi/connection.py", line 645, in _ping
    await self.send_message_await_response(PingRequest(), PingResponse)
  File "/usr/local/lib/python3.10/site-packages/aioesphomeapi/connection.py", line 554, in send_message_await_response
    res = await self.send_message_await_response_complex(
  File "/usr/local/lib/python3.10/site-packages/aioesphomeapi/connection.py", line 539, in send_message_await_response_complex
    raise TimeoutAPIError(
aioesphomeapi.core.TimeoutAPIError: Timeout waiting for response for <class 'api_pb2.PingRequest'> after 10.0s
2023-03-02 15:40:12.462 WARNING (MainThread) [aioesphomeapi.connection] lampa-venti-zuhany @ 172.22.215.32: Connection error occurred: 
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/aioesphomeapi/connection.py", line 537, in send_message_await_response_complex
    await fut
asyncio.exceptions.CancelledError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/aioesphomeapi/connection.py", line 536, in send_message_await_response_complex
    async with async_timeout.timeout(timeout):
  File "/usr/local/lib/python3.10/site-packages/async_timeout/__init__.py", line 129, in __aexit__
    self._do_exit(exc_type)
  File "/usr/local/lib/python3.10/site-packages/async_timeout/__init__.py", line 212, in _do_exit
    raise asyncio.TimeoutError
asyncio.exceptions.TimeoutError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/aioesphomeapi/connection.py", line 320, in _keep_alive_loop
    await self._ping()
  File "/usr/local/lib/python3.10/site-packages/aioesphomeapi/connection.py", line 645, in _ping
    await self.send_message_await_response(PingRequest(), PingResponse)
  File "/usr/local/lib/python3.10/site-packages/aioesphomeapi/connection.py", line 554, in send_message_await_response
    res = await self.send_message_await_response_complex(
  File "/usr/local/lib/python3.10/site-packages/aioesphomeapi/connection.py", line 539, in send_message_await_response_complex
    raise TimeoutAPIError(
aioesphomeapi.core.TimeoutAPIError: Timeout waiting for response for <class 'api_pb2.PingRequest'> after 10.0s

Additional information

Accesing HA user interface through a reverse proxy became also painfully slow - not sure if it has to do with this, but the symptoms appeared in the same time.

https://github.com/home-assistant/core/issues/88535 https://github.com/home-assistant/core/issues/85432#issuecomment-1422633517 https://github.com/home-assistant/core/issues/87158#issuecomment-1436493301

https://discord.com/channels/330944238910963714/1012073425638015037/1061867739863597127

The aioesphomeapi lib has also been optimized reduce latency and improve performance as well on dev (will help all esphome devices not just ones with a Bluetooth proxy). At this point any future improvements are probably going to come from ESP – IDF 5 and the protobuf cpp switch (bdraco — 2023.01.09. 5:42)