core: Client unable to keep up with pending messages

Read before posting

https://github.com/home-assistant/core/issues/95386

The problem

this error at log. Systemwasnt load too high cpu all the time about 15-25% and memory 30-40%

What version of Home Assistant Core has the issue?

2023.2.5 and newer

What was the last working version of Home Assistant Core?

No response

What type of installation are you running?

Home Assistant OS

Integration causing the issue

Home Assistant WebSocket API

Link to integration documentation on our website

https://www.home-assistant.io/integrations/websocket_api

Diagnostics information

No response

Example YAML snippet

No response

Anything in the logs that might be useful for us?

Logger: homeassistant.components.websocket_api.http.connection
Source: components/websocket_api/http.py:159
Integration: Home Assistant WebSocket API (documentation, issues)
First occurred: 16:01:05 (2 occurrences)
Last logged: 22:04:18

[546291820880] Client unable to keep up with pending messages. Stayed over 512 for 5 seconds. The system's load is too high or an integration is misbehaving
[547007151456] Client unable to keep up with pending messages. Stayed over 512 for 5 seconds. The system's load is too high or an integration is misbehaving

Additional information

No response

About this issue

  • Original URL
  • State: closed
  • Created a year ago
  • Comments: 44 (22 by maintainers)

Commits related to this issue

Most upvoted comments

Same problem here. Problem arrived after upgrading to 2023.6

The device referred to is an iPad Mini 5.

Home Assistant 2023.6.2 Supervisor 2023.06.2 Operating System 10.2 Grensesnitt 20230608.0 – latest

Logger: homeassistant.components.websocket_api.http.connection
Source: components/websocket_api/http.py:172 
Integration: Home Assistant WebSocket API (documentation, issues) 
First occurred: 10:13:29 (1 occurrences) 
Last logged: 10:13:29

[139728829651024] Daniel from 192.168.40.155 (Mozilla/5.0 (iPad; CPU OS 16_4_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Home Assistant/2023.4 (io.robbie.HomeAssistant; build:2023.460; iPadOS 16.4.1) Mobile/HomeAssistant, like Safari): Client unable to keep up with pending messages. Reached 4096 pending messages. The system's load is too high or an integration is misbehaving; Last message was: {"id":4666,"type":"event","event":{"type":"removed","notifications":{"retracted":{"message":"To set up Inngangsdør:21066 in the Home App, scan the QR code or enter the following code:\n### 871-43-438\n![image](/api/homekit/pairingqr?retracted)","notification_id":"retracted","title":"HomeKit Pairing","created_at":"2023-06-15T21:29:05.059035+00:00"}}}}

And

Logger: homeassistant.components.websocket_api.http.connection
Source: components/websocket_api/http.py:53 
Integration: Home Assistant WebSocket API (documentation, issues) 
First occurred: 10:13:31 (6 occurrences) 
Last logged: 10:13:45

[139727070132176] Daniel from 192.168.40.155 (Mozilla/5.0 (iPad; CPU OS 16_4_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Home Assistant/2023.4 (io.robbie.HomeAssistant; build:2023.460; iPadOS 16.4.1) Mobile/HomeAssistant, like Safari): Client unable to keep up with pending messages. Reached 4096 pending messages. The system's load is too high or an integration is misbehaving; Last message was: {"id":3446,"type":"result","success":true,"result":null}
[139727071724816] Daniel from 192.168.40.155 (Mozilla/5.0 (iPad; CPU OS 16_4_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Home Assistant/2023.4 (io.robbie.HomeAssistant; build:2023.460; iPadOS 16.4.1) Mobile/HomeAssistant, like Safari): Client unable to keep up with pending messages. Reached 4096 pending messages. The system's load is too high or an integration is misbehaving; Last message was: {"id":3446,"type":"result","success":true,"result":null}
[139727069618192] Daniel from 192.168.40.155 (Mozilla/5.0 (iPad; CPU OS 16_4_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Home Assistant/2023.4 (io.robbie.HomeAssistant; build:2023.460; iPadOS 16.4.1) Mobile/HomeAssistant, like Safari): Client unable to keep up with pending messages. Reached 4096 pending messages. The system's load is too high or an integration is misbehaving; Last message was: {"id":3186,"type":"result","success":true,"result":null}
[139727077667280] Daniel from 192.168.40.155 (Mozilla/5.0 (iPad; CPU OS 16_4_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Home Assistant/2023.4 (io.robbie.HomeAssistant; build:2023.460; iPadOS 16.4.1) Mobile/HomeAssistant, like Safari): Client unable to keep up with pending messages. Reached 4096 pending messages. The system's load is too high or an integration is misbehaving; Last message was: {"id":3446,"type":"result","success":true,"result":null}
[139727285572048] Daniel from 192.168.40.155 (Mozilla/5.0 (iPad; CPU OS 16_4_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Home Assistant/2023.4 (io.robbie.HomeAssistant; build:2023.460; iPadOS 16.4.1) Mobile/HomeAssistant, like Safari): Client unable to keep up with pending messages. Reached 4096 pending messages. The system's load is too high or an integration is misbehaving; Last message was: {"id":4827,"type":"result","success":true,"result":null}

Perhaps I should create a different PR, but starting 2023.6.0/2023.6.1, I’m seeing this error as well.

023-06-12 14:37:47.168 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [139897668178256] Tommy from 110.38.44.177 (Mozilla/5.0 (Linux; Android 10; Lenovo TB-8705F Build/QP1A.190711.020; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/112.0.5615.135 Safari/537.36): Client unable to keep up with pending messages. Reached 4096 pending messages. The system's load is too high or an integration is misbehaving; Last message was: {"id":4120,"type":"event","event":{"type":"added","notifications":{"mail_1234":{"message":"Hit DISMISS after you have retrieved the mail","notification_id":"mail_1234","title":"You Have Mail!","created_at":"2023-06-12T18:37:46.200542+00:00"}}}}

It seems I only get this error when a persistent notification is sent (showing up in the “Notifications” sidebar). I have a Windows Desktop Chrome, a few iOS Companion Apps, and an Android Lenovo FKB connected to HA.

I also have a problem with the Android/Lenovo Tablet, whereby at the UI, when I use the sidebar Notifications to clear the notifications, they are cleared/dismissed, but the sidebar continues to indicate I still have a notification (circle with number of notifications pending). This maybe because of this error. I don’t have this issue with the other browsers/companion app.

While its a generic error and we can never fix every instance of it on every device. These two PRs will improve the situation for mobile / narrow devices:

https://github.com/home-assistant/frontend/pull/16953 https://github.com/home-assistant/core/pull/94780

Understood. Yet already a few hours since I’ve disabled all persistent notifications and the issue has not returned neither on mobile nor on PC chrome browser. I’ll need some more time to collect the callgrind and py-spys.

callgrind.out.1682337757952494.txt

i added .txt because this file format isnt suported here