core: Unresponsive integrations can kill HA core during startup stalling system in CoreState.Startup

Update:

When either the hardware or the cloud service for an integration present within core.config_entries fails to respond to HA during startup, HA can die and HA OS gets stuck in CoreState.STARTUP effectively bricking the HA installation.

  • Dead TPLink (device)
  • Dead Plum Lightpads (dead cloud)

configuration.yaml Example:

plum_lightpad:
  username: anyfakeaccount@fake-test.com
  password: 123456-doesnt-matter-plum-servers-are-dead

Original:

The problem

Same as #53605… Core does not shutdown because this thread won’t quit.

Doesn’t appear to be a way can force core to restart from ssh using “ha” command.

Restarting supervisor is possible, however supervisor doesn’t kill core and fix the issue.

What is version of Home Assistant Core has the issue?

core-2021.9.7

What was the last working version of Home Assistant Core?

No response

What type of installation are you running?

Home Assistant OS

Integration causing the issue

plum

Link to integration documentation on our website

No response

Example YAML snippet

No response

Anything in the logs that might be useful for us?

result = self.fn(*self.args, **self.kwargs)
  File "/usr/src/homeassistant/homeassistant/components/itunes/media_player.py", line 193, in setup_platform
    ItunesDevice(
  File "/usr/src/homeassistant/homeassistant/components/itunes/media_player.py", line 230, in __init__
    self.update()
  File "/usr/src/homeassistant/homeassistant/components/itunes/media_player.py", line 271, in update
    now_playing = self.client.now_playing()
  File "/usr/src/homeassistant/homeassistant/components/itunes/media_player.py", line 113, in now_playing
    return self._request("GET", "/now_playing")
  File "/usr/src/homeassistant/homeassistant/components/itunes/media_player.py", line 93, in _request
    response = requests.get(url, timeout=DEFAULT_TIMEOUT)
  File "/usr/local/lib/python3.9/site-packages/requests/api.py", line 76, in get
    return request('get', url, params=params, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.9/site-packages/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 394, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/local/lib/python3.9/site-packages/urllib3/connection.py", line 234, in request
    super(HTTPConnection, self).request(method, url, body=body, headers=headers)
  File "/usr/local/lib/python3.9/http/client.py", line 1279, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/lib/python3.9/http/client.py", line 1325, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.9/http/client.py", line 1274, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.9/http/client.py", line 1034, in _send_output
    self.send(msg)
  File "/usr/local/lib/python3.9/http/client.py", line 974, in send
    self.connect()
  File "/usr/local/lib/python3.9/site-packages/urllib3/connection.py", line 200, in connect
    conn = self._new_conn()
  File "/usr/local/lib/python3.9/site-packages/urllib3/connection.py", line 169, in _new_conn
    conn = connection.create_connection(
  File "/usr/local/lib/python3.9/site-packages/urllib3/util/connection.py", line 73, in create_connection
    for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
  File "/usr/local/lib/python3.9/socket.py", line 954, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
2021-09-28 11:24:46 WARNING (Thread-4) [homeassistant.util.executor] Thread[SyncWorker_2] is still running at shutdown: File "/usr/local/lib/python3.9/threading.py", line 930, in _bootstrap
    self._bootstrap_inner()
  File "/usr/local/lib/python3.9/threading.py", line 973, in _bootstrap_inner
    self.run()
  File "/usr/local/lib/python3.9/threading.py", line 910, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.9/concurrent/futures/thread.py", line 77, in _worker
    work_item.run()
  File "/usr/local/lib/python3.9/concurrent/futures/thread.py", line 52, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/src/homeassistant/homeassistant/components/itunes/media_player.py", line 193, in setup_platform
    ItunesDevice(
  File "/usr/src/homeassistant/homeassistant/components/itunes/media_player.py", line 230, in __init__
    self.update()
  File "/usr/src/homeassistant/homeassistant/components/itunes/media_player.py", line 271, in update
    now_playing = self.client.now_playing()
  File "/usr/src/homeassistant/homeassistant/components/itunes/media_player.py", line 113, in now_playing
    return self._request("GET", "/now_playing")
  File "/usr/src/homeassistant/homeassistant/components/itunes/media_player.py", line 93, in _request
    response = requests.get(url, timeout=DEFAULT_TIMEOUT)
  File "/usr/local/lib/python3.9/site-packages/requests/api.py", line 76, in get
    return request('get', url, params=params, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.9/site-packages/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 394, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/local/lib/python3.9/site-packages/urllib3/connection.py", line 234, in request
    super(HTTPConnection, self).request(method, url, body=body, headers=headers)
  File "/usr/local/lib/python3.9/http/client.py", line 1279, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/lib/python3.9/http/client.py", line 1325, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.9/http/client.py", line 1274, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.9/http/client.py", line 1034, in _send_output
    self.send(msg)
  File "/usr/local/lib/python3.9/http/client.py", line 974, in send
    self.connect()
  File "/usr/local/lib/python3.9/site-packages/urllib3/connection.py", line 200, in connect
    conn = self._new_conn()
  File "/usr/local/lib/python3.9/site-packages/urllib3/connection.py", line 169, in _new_conn
    conn = connection.create_connection(
  File "/usr/local/lib/python3.9/site-packages/urllib3/util/connection.py", line 73, in create_connection
    for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
  File "/usr/local/lib/python3.9/socket.py", line 954, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
2021-09-28 11:24:47 WARNING (MainThread) [homeassistant.components.webhook] Received message for unregistered webhook 7be1f507fbb4c5debcfc75f67bec8a453fa67d65056cda2d4973b7ace8965ed4 from 192.168.1.1

Additional information

Supervisor logs

21-09-28 11:19:54 INFO (MainThread) [supervisor.resolution.checks.base] Run check for IssueType.TRUST/ContextType.CORE
21-09-28 11:19:55 INFO (MainThread) [supervisor.host.sound] Updating PulseAudio information
21-09-28 11:19:55 INFO (MainThread) [supervisor.resolution.checks.base] Run check for IssueType.TRUST/ContextType.SUPERVISOR
21-09-28 11:19:55 INFO (MainThread) [supervisor.host.manager] Host information reload completed
21-09-28 11:19:56 INFO (MainThread) [supervisor.api.middleware.security] /supervisor/logs access from core_ssh
21-09-28 11:19:57 INFO (MainThread) [supervisor.resolution.checks.base] Run check for IssueType.PWNED/ContextType.ADDON
21-09-28 11:19:57 WARNING (MainThread) [supervisor.addons.options] Option 'anonymous' does not exist in the schema for Mosquitto broker (core_mosquitto)
21-09-28 11:19:57 INFO (MainThread) [supervisor.resolution.checks.base] Run check for IssueType.FREE_SPACE/ContextType.SYSTEM
21-09-28 11:19:57 INFO (MainThread) [supervisor.resolution.check] System checks complete
21-09-28 11:19:57 INFO (MainThread) [supervisor.resolution.evaluate] Starting system evaluation with state CoreState.RUNNING
21-09-28 11:20:00 INFO (MainThread) [supervisor.resolution.evaluate] System evaluation complete
21-09-28 11:20:00 INFO (MainThread) [supervisor.resolution.fixup] Starting system autofix at state CoreState.RUNNING
21-09-28 11:20:00 INFO (MainThread) [supervisor.resolution.fixup] System autofix complete
21-09-28 11:20:55 INFO (MainThread) [supervisor.api.middleware.security] /supervisor/logs access from core_ssh
21-09-28 11:21:24 INFO (MainThread) [supervisor.api.middleware.security] /core/logs access from core_ssh
21-09-28 11:21:47 WARNING (MainThread) [supervisor.misc.tasks] Watchdog miss API response from Home Assistant
21-09-28 11:23:48 ERROR (MainThread) [supervisor.misc.tasks] Watchdog found a problem with Home Assistant API!
21-09-28 11:23:48 INFO (SyncWorker_1) [supervisor.docker.interface] Restarting ghcr.io/home-assistant/raspberrypi4-64-homeassistant
21-09-28 11:23:53 INFO (MainThread) [supervisor.homeassistant.core] Wait until Home Assistant is ready
21-09-28 11:24:19 INFO (MainThread) [supervisor.resolution.evaluate] Starting system evaluation with state CoreState.RUNNING
21-09-28 11:24:20 INFO (MainThread) [supervisor.resolution.evaluate] System evaluation complete
21-09-28 11:24:44 INFO (MainThread) [supervisor.homeassistant.api] Updated Home Assistant API token
21-09-28 11:24:45 ERROR (MainThread) [supervisor.homeassistant.api] Error on call https://172.30.32.1:8123/api/config: Server disconnected
21-09-28 11:24:45 ERROR (MainThread) [supervisor.homeassistant.api] Error on call https://172.30.32.1:8123/api/config: Cannot connect to host 172.30.32.1:8123 ssl:False [Connection reset by peer]
21-09-28 11:25:18 ERROR (MainThread) [supervisor.homeassistant.api] Error on call https://172.30.32.1:8123/api/config:
21-09-28 11:25:21 ERROR (MainThread) [supervisor.homeassistant.api] Error on call https://172.30.32.1:8123/api/config:
21-09-28 11:25:50 ERROR (MainThread) [supervisor.homeassistant.api] Error on call https://172.30.32.1:8123/api/config:
21-09-28 11:25:57 ERROR (MainThread) [supervisor.homeassistant.api] Error on call https://172.30.32.1:8123/api/config:
21-09-28 11:26:26 ERROR (MainThread) [supervisor.homeassistant.api] Error on call https://172.30.32.1:8123/api/config:
21-09-28 11:26:33 ERROR (MainThread) [supervisor.homeassistant.api] Error on call https://172.30.32.1:8123/api/config:
21-09-28 11:26:58 ERROR (MainThread) [supervisor.homeassistant.api] Error on call https://172.30.32.1:8123/api/config:
21-09-28 11:27:09 ERROR (MainThread) [supervisor.homeassistant.api] Error on call https://172.30.32.1:8123/api/config:
21-09-28 11:27:22 INFO (MainThread) [supervisor.api.middleware.security] /core/info access from core_ssh
21-09-28 11:27:30 ERROR (MainThread) [supervisor.homeassistant.api] Error on call https://172.30.32.1:8123/api/config:
21-09-28 11:27:45 ERROR (MainThread) [supervisor.homeassistant.api] Error on call https://172.30.32.1:8123/api/config:
21-09-28 11:28:02 ERROR (MainThread) [supervisor.homeassistant.api] Error on call https://172.30.32.1:8123/api/config:
21-09-28 11:28:21 ERROR (MainThread) [supervisor.homeassistant.api] Error on call https://172.30.32.1:8123/api/config:
21-09-28 11:28:38 ERROR (MainThread) [supervisor.homeassistant.api] Error on call https://172.30.32.1:8123/api/config:
21-09-28 11:28:57 ERROR (MainThread) [supervisor.homeassistant.api] Error on call https://172.30.32.1:8123/api/config:
21-09-28 11:29:10 ERROR (MainThread) [supervisor.homeassistant.api] Error on call https://172.30.32.1:8123/api/config:
21-09-28 11:29:33 ERROR (MainThread) [supervisor.homeassistant.api] Error on call https://172.30.32.1:8123/api/config:
21-09-28 11:29:42 ERROR (MainThread) [supervisor.homeassistant.api] Error on call https://172.30.32.1:8123/api/config:
21-09-28 11:30:09 ERROR (MainThread) [supervisor.homeassistant.api] Error on call https://172.30.32.1:8123/api/config:
21-09-28 11:30:14 ERROR (MainThread) [supervisor.homeassistant.api] Error on call https://172.30.32.1:8123/api/config:
21-09-28 11:30:45 ERROR (MainThread) [supervisor.homeassistant.api] Error on call https://172.30.32.1:8123/api/config:
21-09-28 11:30:50 ERROR (MainThread) [supervisor.homeassistant.api] Error on call https://172.30.32.1:8123/api/config:
21-09-28 11:31:21 ERROR (MainThread) [supervisor.homeassistant.api] Error on call https://172.30.32.1:8123/api/config:
21-09-28 11:31:22 ERROR (MainThread) [supervisor.homeassistant.api] Error on call https://172.30.32.1:8123/api/config:
21-09-28 11:31:54 ERROR (MainThread) [supervisor.homeassistant.api] Error on call https://172.30.32.1:8123/api/config:
21-09-28 11:31:57 ERROR (MainThread) [supervisor.homeassistant.api] Error on call https://172.30.32.1:8123/api/config:
21-09-28 11:32:26 ERROR (MainThread) [supervisor.homeassistant.api] Error on call https://172.30.32.1:8123/api/config:
21-09-28 11:32:33 ERROR (MainThread) [supervisor.homeassistant.api] Error on call https://172.30.32.1:8123/api/config:
21-09-28 11:33:02 ERROR (MainThread) [supervisor.homeassistant.api] Error on call https://172.30.32.1:8123/api/config:
21-09-28 11:33:09 ERROR (MainThread) [supervisor.homeassistant.api] Error on call https://172.30.32.1:8123/api/config:
21-09-28 11:33:34 ERROR (MainThread) [supervisor.homeassistant.api] Error on call https://172.30.32.1:8123/api/config:
21-09-28 11:33:45 ERROR (MainThread) [supervisor.homeassistant.api] Error on call https://172.30.32.1:8123/api/config:
21-09-28 11:34:06 ERROR (MainThread) [supervisor.homeassistant.api] Error on call https://172.30.32.1:8123/api/config:
21-09-28 11:34:21 ERROR (MainThread) [supervisor.homeassistant.api] Error on call https://172.30.32.1:8123/api/config:
21-09-28 11:34:38 ERROR (MainThread) [supervisor.homeassistant.api] Error on call https://172.30.32.1:8123/api/config:
21-09-28 11:34:57 ERROR (MainThread) [supervisor.homeassistant.api] Error on call https://172.30.32.1:8123/api/config:
21-09-28 11:35:14 ERROR (MainThread) [supervisor.homeassistant.api] Error on call https://172.30.32.1:8123/api/config:
21-09-28 11:35:33 ERROR (MainThread) [supervisor.homeassistant.api] Error on call https://172.30.32.1:8123/api/config:
21-09-28 11:35:46 ERROR (MainThread) [supervisor.homeassistant.api] Error on call https://172.30.32.1:8123/api/config:
21-09-28 11:36:09 ERROR (MainThread) [supervisor.homeassistant.api] Error on call https://172.30.32.1:8123/api/config:
21-09-28 11:36:18 ERROR (MainThread) [supervisor.homeassistant.api] Error on call https://172.30.32.1:8123/api/config:
21-09-28 11:36:45 ERROR (MainThread) [supervisor.homeassistant.api] Error on call https://172.30.32.1:8123/api/config:
21-09-28 11:36:50 ERROR (MainThread) [supervisor.homeassistant.api] Error on call https://172.30.32.1:8123/api/config:
21-09-28 11:37:21 ERROR (MainThread) [supervisor.homeassistant.api] Error on call https://172.30.32.1:8123/api/config:
21-09-28 11:37:26 ERROR (MainThread) [supervisor.homeassistant.api] Error on call https://172.30.32.1:8123/api/config:
21-09-28 11:37:57 ERROR (MainThread) [supervisor.homeassistant.api] Error on call https://172.30.32.1:8123/api/config:
21-09-28 12:04:24 INFO (MainThread) [supervisor.api.middleware.security] /supervisor/logs access from core_ssh
21-09-28 12:04:46 INFO (MainThread) [supervisor.api.middleware.security] /core/logs access from core_ssh
21-09-28 12:05:23 INFO (MainThread) [supervisor.api.middleware.security] /backups access from core_ssh
21-09-28 12:19:46 INFO (MainThread) [supervisor.resolution.check] Starting system checks with state CoreState.RUNNING
21-09-28 12:19:46 INFO (MainThread) [supervisor.resolution.checks.base] Run check for IssueType.SECURITY/ContextType.CORE
21-09-28 12:19:46 INFO (MainThread) [supervisor.resolution.checks.base] Run check for IssueType.TRUST/ContextType.PLUGIN
21-09-28 12:19:46 INFO (MainThread) [supervisor.resolution.checks.base] Run check for IssueType.TRUST/ContextType.CORE
21-09-28 12:19:46 INFO (MainThread) [supervisor.resolution.checks.base] Run check for IssueType.TRUST/ContextType.SUPERVISOR
21-09-28 12:19:46 INFO (MainThread) [supervisor.resolution.checks.base] Run check for IssueType.PWNED/ContextType.ADDON
21-09-28 12:19:46 INFO (MainThread) [supervisor.resolution.checks.base] Run check for IssueType.FREE_SPACE/ContextType.SYSTEM
21-09-28 12:19:46 INFO (MainThread) [supervisor.resolution.check] System checks complete
21-09-28 12:19:46 INFO (MainThread) [supervisor.resolution.evaluate] Starting system evaluation with state CoreState.RUNNING
21-09-28 12:19:47 INFO (MainThread) [supervisor.resolution.evaluate] System evaluation complete
21-09-28 12:19:47 INFO (MainThread) [supervisor.resolution.fixup] Starting system autofix at state CoreState.RUNNING
21-09-28 12:19:47 INFO (MainThread) [supervisor.resolution.fixup] System autofix complete
21-09-28 12:23:57 INFO (MainThread) [supervisor.api.middleware.security] /core/logs access from core_ssh
21-09-28 12:24:10 INFO (MainThread) [supervisor.api.middleware.security] /supervisor/logs access from core_ssh

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 21 (11 by maintainers)

Most upvoted comments

Also, simply changing “disabled_by”: null to “disabled_by”: “user” allows the system to come up fully, and leaves the config entry available in case we’d like to try the Plum service again occasionally.

Apparently this is the way to disable integrations which are not set up in configuration.yaml.

+1
rjroch on Oct 2, 2021

So the problem is sort of understood, but definitely needs to be debugged and fixed.

+1
Tam-Lin on Oct 1, 2021
Read more comments on GitHub
← core: Unreliable sensor data (data usage) since firmware 7.50 - filter/block that data
core: Unsupported system - Connectivity check disabled after update to core 2022.11.0 →