core: Nest integration high CPU usage on armv7 / raspberry pi in pubsub subscriber native code (fixed in 2023.10.x)
The problem
When I enable the Google Nest integration, I see a continous CPU load of python3 of about 65% Without this integration CPU load of python3 is just about 2%
Any recommendation to optimize the load of the Google Nest integration?
What version of Home Assistant Core has the issue?
2022.2.9
What was the last working version of Home Assistant Core?
No response
What type of installation are you running?
Home Assistant Container
Integration causing the issue
nest
Link to integration documentation on our website
https://www.home-assistant.io/integrations/nest
Diagnostics information
{
"home_assistant": {
"installation_type": "Home Assistant Container",
"version": "2022.2.9",
"dev": false,
"hassio": false,
"virtualenv": false,
"python_version": "3.9.7",
"docker": true,
"arch": "armv7l",
"timezone": "Europe/Brussels",
"os_name": "Linux",
"os_version": "5.10.63-v7l+",
"run_as_root": true
},
"custom_components": {
"config_editor": {
"version": "3.0",
"requirements": []
},
"hacs": {
"version": "1.22.0",
"requirements": [
"aiogithubapi>=21.11.0"
]
},
"bosch": {
"version": "0.17.3",
"requirements": [
"bosch-thermostat-client==0.17.3"
]
},
"afvalbeheer": {
"version": "4.9.2",
"requirements": [
"rsa",
"pycryptodome"
]
},
"authenticated": {
"version": "21.9.0",
"requirements": []
}
},
"integration_manifest": {
"domain": "nest",
"name": "Nest",
"config_flow": true,
"dependencies": [
"ffmpeg",
"http",
"media_source"
],
"documentation": "https://www.home-assistant.io/integrations/nest",
"requirements": [
"python-nest==4.2.0",
"google-nest-sdm==1.7.1"
],
"codeowners": [
"@allenporter"
],
"quality_scale": "platinum",
"dhcp": [
{
"macaddress": "18B430*"
},
{
"macaddress": "641666*"
},
{
"macaddress": "D8EB46*"
},
{
"macaddress": "1C53F9*"
}
],
"iot_class": "cloud_push",
"is_built_in": true
},
"data": {
"subscriber": {
"start": 1,
"message_received": 7,
"message_acked": 7
},
"devices": [
{
"data": {
"name": "**REDACTED**",
"type": "sdm.devices.types.DOORBELL",
"assignee": "**REDACTED**",
"traits": {
"sdm.devices.traits.Info": {
"customName": "**REDACTED**"
},
"sdm.devices.traits.CameraLiveStream": {
"maxVideoResolution": {
"width": 640,
"height": 480
},
"videoCodecs": [
"H264"
],
"audioCodecs": [
"AAC"
],
"supportedProtocols": [
"RTSP"
]
},
"sdm.devices.traits.CameraImage": {
"maxImageResolution": {
"width": 1920,
"height": 1200
}
},
"sdm.devices.traits.CameraPerson": {},
"sdm.devices.traits.CameraSound": {},
"sdm.devices.traits.CameraMotion": {},
"sdm.devices.traits.CameraEventImage": {},
"sdm.devices.traits.DoorbellChime": {}
},
"parentRelations": [
{
"parent": "**REDACTED**",
"displayName": "**REDACTED**"
}
]
},
"command": {
"sdm.devices.commands.CameraLiveStream.GenerateRtspStream": 1
},
"event_media": {
"event": 2,
"event.new": 2,
"event.fetch": 2,
"fetch_image": 2,
"fetch_image.skip": 2,
"event.notify": 2
}
}
]
}
}
Example YAML snippet
No response
Anything in the logs that might be useful for us?
No response
Additional information
No response
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Reactions: 4
- Comments: 93 (44 by maintainers)
Fantastic.That meets the description of the issue I was seeing when debugging.
Last time I bumped grpc there were some other dependency issues for protobuf 4. If these aren’t resolved yet it may not just be a trivial bump, but either way I’ll manage it.
Still an issue - I am running a new install on a raspberry pi 4, hosted via docker compose.
spotted high CPU last night and narrowed it down to the nest integration. CPU is around 30% with best enabled, 1-2% with it disabled.
i assume we are needing to wait for a fix further upstream based on the comment history, but I just wanted to confirm that it’s still an issue on the latest HA version.
I have been seeing a jump from 5% CPU without the Nest integration to 40% with the integration.
I have been running HomeAssistantOS 32 bit on a raspberry pi 4. The other day I migrated my HAOS to the 64 bit version and my CPU usage stays at 5% while running the Nest integration. Yay!
I don’t know if this is already known but I thought it may be helpful to someone.
That’s great news. The lib bump with come out with 2023.10.x
Did the same 64-Bit swap as rgerbranda, my cpu finally lowered to like 5% usage and the low 50’s in temp since summer last year.
I switched to Raspberry Pi OS (64-bit) (thanks @Photoexploration ) and now the Nest integratrion is working fine! The CPU load is back to normal.
Done, sharing also another log from strace showing some issue in
futexinstruction usage (some sort of thread synchronization locking but with already expired absolute timeouts), which should help them to better trace the source of the issue.However, I think I will move to a 64bit OS as soon as possible (thanks @Photoexploration ! ), in order to workaround the problem and reduce my RPi power usage.
As an addition, I tried to profile both the main HA process and threads using the CPU with “strace -Tcfp <pid>”, and they always reports an high “% temp” on futex syscall, with also a lot of errors… :
(from main process)
(from the thread constantly using the CPU)
(from the thread using CPU intermittently)
so, if I’m interpreting it correctly, it should confirm that there are still some issues on threading/polling management on grpc library on ARM.
Hi, I just updated HA (docker installation) to 2023.1.0, but my RPI 3 CPU usage has been increased from ~50% to ~100%… from what I can see from “htop”, there are three HA threads which cause all the CPU load, with one of them constantly using 50% of the CPU, and the other two, at alternated times, using the remaining 50% .
(PIDs 48, 53 and 64)
Python profiler still shows nothing, so the issue is still in some native code:
I tried to retrieve some information for such threads adding “gdb” to the docker image and performing a “thread apply all bt” command on the HA’s python process, and all three threads report the library “/usr/local/lib/python3.10/site-packages/grpc/_cython/cygrpc.cpython-310-arm-linux-gnueabihf.so” in their stacks, so there are still some other issues in the grpc library:
I also tried to enable grpc logs (GRPC_TRACE=api and GRPC_VERBOSITY=info environmental variable), but it seems that the polling problem reported in the https://github.com/googleapis/python-pubsub/issues/728 has been fixed, as the container grpc library is 1.51.1 and logs reports a poll message every 200ms, as it should:
so there should still be something wrong on grpcimplementation…
grpc 1.51.1 was integrated in https://github.com/home-assistant/core/pull/83420
Perhaps we can confirm folks see this helps on a dev build and we can consider this closed.
Hey! Remember this is a volunteer project. I bunch a pretty significant amount of effort into diagnosing and profiling this, finding a potential cause, and filed the issue in the upstream project.
The right answer is I disabled the addon for now, running 100% cpu for months on a row is unacceptable, hoped a little more effort from the devs