core: Tibber integration losses his connection and does not get it back
The problem
I’m using the latest release of HA (2023.03.1) and since then the Tibber integration has lost it’s connection twice.
See log for details.
What version of Home Assistant Core has the issue?
2023.3.1
What was the last working version of Home Assistant Core?
2023.2.5
What type of installation are you running?
Home Assistant OS
Integration causing the issue
Tibber
Link to integration documentation on our website
No response
Diagnostics information
config_entry-tibber-3f24869fce4711c38ea58585b2147299.json.txt
Tibber log crash 2023-03-06.txt
Example YAML snippet
No response
Anything in the logs that might be useful for us?
2023-03-06 11:00:50.091 DEBUG (MainThread) [homeassistant.components.tibber.sensor] Manually updated Opsaheden 44 data
2023-03-06 11:00:50.108 DEBUG (MainThread) [tibber.tibber_home] Data received for 9ca92063-a177-40f3-........
2023-03-06 11:00:50.390 DEBUG (MainThread) [tibber] Watchdog: Connection is alive
2023-03-06 11:00:55.391 DEBUG (MainThread) [tibber] Watchdog: Connection is alive
2023-03-06 11:01:00.366 DEBUG (MainThread) [homeassistant.components.tibber.sensor] Manually updated Opsaheden 44 data
2023-03-06 11:01:00.380 DEBUG (MainThread) [tibber.tibber_home] Data received for 9ca92063-a177-40f3-........
2023-03-06 11:01:00.393 DEBUG (MainThread) [tibber] Watchdog: Connection is alive
2023-03-06 11:01:05.395 DEBUG (MainThread) [tibber] Watchdog: Connection is alive
2023-03-06 11:01:10.163 DEBUG (MainThread) [homeassistant.components.tibber.sensor] Manually updated Opsaheden 44 data
2023-03-06 11:01:10.178 DEBUG (MainThread) [tibber.tibber_home] Data received for 9ca92063-a177-40f3-............
2023-03-06 11:01:10.397 DEBUG (MainThread) [tibber] Watchdog: Connection is alive
2023-03-06 11:01:15.399 DEBUG (MainThread) [tibber] Watchdog: Connection is alive
2023-03-06 11:01:19.268 ERROR (MainThread) [tibber.tibber_home] Error in rt_subscribe
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/tibber/tibber_home.py", line 415, in _start
async for data in self._tibber_control.sub_manager.session.subscribe(
File "/usr/local/lib/python3.10/site-packages/gql/client.py", line 1064, in subscribe
async for result in inner_generator:
File "/usr/local/lib/python3.10/site-packages/gql/client.py", line 960, in _subscribe
async for result in inner_generator:
File "/usr/local/lib/python3.10/site-packages/gql/transport/websockets_base.py", line 388, in subscribe
answer_type, execution_result = await listener.get()
File "/usr/local/lib/python3.10/site-packages/gql/transport/websockets_base.py", line 56, in get
raise item
gql.transport.exceptions.TransportServerError: No pong received after 5.0 seconds
2023-03-06 11:01:20.401 ERROR (MainThread) [tibber] Watchdog: Connection is down
2023-03-06 11:01:44.313 ERROR (MainThread) [tibber] Error connecting to Tibber: Cannot connect to host api.tibber.com:443 ssl:default [Try again]
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/aiohttp/connector.py", line 1152, in _create_direct_connection
hosts = await asyncio.shield(host_resolved)
File "/usr/local/lib/python3.10/site-packages/aiohttp/connector.py", line 874, in _resolve_host
addrs = await self._resolver.resolve(host, port, family=self._family)
File "/usr/local/lib/python3.10/site-packages/aiohttp/resolver.py", line 33, in resolve
infos = await self._loop.getaddrinfo(
File "/usr/local/lib/python3.10/asyncio/base_events.py", line 863, in getaddrinfo
return await self.run_in_executor(
File "/usr/local/lib/python3.10/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
File "/usr/local/lib/python3.10/socket.py", line 955, in getaddrinfo
for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -3] Try again
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/tibber/__init__.py", line 214, in _execute
resp = await self.websession.post(self.api_endpoint, **post_args)
File "/usr/local/lib/python3.10/site-packages/aiohttp/client.py", line 536, in _request
conn = await self._connector.connect(
File "/usr/local/lib/python3.10/site-packages/aiohttp/connector.py", line 540, in connect
proto = await self._create_connection(req, traces, timeout)
File "/usr/local/lib/python3.10/site-packages/aiohttp/connector.py", line 901, in _create_connection
_, proto = await self._create_direct_connection(req, traces, timeout)
File "/usr/local/lib/python3.10/site-packages/aiohttp/connector.py", line 1166, in _create_direct_connection
raise ClientConnectorError(req.connection_key, exc) from exc
aiohttp.client_exceptions.ClientConnectorError: Cannot connect to host api.tibber.com:443 ssl:default [Try again]
2023-03-06 11:01:44.318 ERROR (MainThread) [tibber] Error connecting to Tibber: Cannot connect to host api.tibber.com:443 ssl:default [Try again]
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/aiohttp/connector.py", line 1152, in _create_direct_connection
hosts = await asyncio.shield(host_resolved)
File "/usr/local/lib/python3.10/site-packages/aiohttp/connector.py", line 1152, in _create_direct_connection
hosts = await asyncio.shield(host_resolved)
File "/usr/local/lib/python3.10/site-packages/aiohttp/connector.py", line 874, in _resolve_host
addrs = await self._resolver.resolve(host, port, family=self._family)
File "/usr/local/lib/python3.10/site-packages/aiohttp/resolver.py", line 33, in resolve
infos = await self._loop.getaddrinfo(
File "/usr/local/lib/python3.10/asyncio/base_events.py", line 863, in getaddrinfo
return await self.run_in_executor(
File "/usr/local/lib/python3.10/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
File "/usr/local/lib/python3.10/socket.py", line 955, in getaddrinfo
for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -3] Try again
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/tibber/__init__.py", line 214, in _execute
resp = await self.websession.post(self.api_endpoint, **post_args)
File "/usr/local/lib/python3.10/site-packages/aiohttp/client.py", line 536, in _request
conn = await self._connector.connect(
File "/usr/local/lib/python3.10/site-packages/aiohttp/connector.py", line 540, in connect
proto = await self._create_connection(req, traces, timeout)
File "/usr/local/lib/python3.10/site-packages/aiohttp/connector.py", line 901, in _create_connection
_, proto = await self._create_direct_connection(req, traces, timeout)
File "/usr/local/lib/python3.10/site-packages/aiohttp/connector.py", line 1166, in _create_direct_connection
raise ClientConnectorError(req.connection_key, exc) from exc
aiohttp.client_exceptions.ClientConnectorError: Cannot connect to host api.tibber.com:443 ssl:default [Try again]
2023-03-06 11:01:44.323 ERROR (MainThread) [homeassistant] Error doing job: Task exception was never retrieved
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/aiohttp/connector.py", line 1152, in _create_direct_connection
hosts = await asyncio.shield(host_resolved)
File "/usr/local/lib/python3.10/site-packages/aiohttp/connector.py", line 1152, in _create_direct_connection
hosts = await asyncio.shield(host_resolved)
File "/usr/local/lib/python3.10/site-packages/aiohttp/connector.py", line 874, in _resolve_host
addrs = await self._resolver.resolve(host, port, family=self._family)
File "/usr/local/lib/python3.10/site-packages/aiohttp/resolver.py", line 33, in resolve
infos = await self._loop.getaddrinfo(
File "/usr/local/lib/python3.10/asyncio/base_events.py", line 863, in getaddrinfo
return await self.run_in_executor(
File "/usr/local/lib/python3.10/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
File "/usr/local/lib/python3.10/socket.py", line 955, in getaddrinfo
for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -3] Try again
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/tibber/tibber_home.py", line 433, in _start
await asyncio.gather(
File "/usr/local/lib/python3.10/site-packages/tibber/tibber_home.py", line 198, in update_info
if data := await self._tibber_control.execute(UPDATE_INFO % self._home_id):
File "/usr/local/lib/python3.10/site-packages/tibber/__init__.py", line 184, in execute
res := await self._execute(document, variable_values, timeout=timeout)
File "/usr/local/lib/python3.10/site-packages/tibber/__init__.py", line 221, in _execute
return await self._execute(
File "/usr/local/lib/python3.10/site-packages/tibber/__init__.py", line 221, in _execute
return await self._execute(
File "/usr/local/lib/python3.10/site-packages/tibber/__init__.py", line 214, in _execute
resp = await self.websession.post(self.api_endpoint, **post_args)
File "/usr/local/lib/python3.10/site-packages/aiohttp/client.py", line 536, in _request
conn = await self._connector.connect(
File "/usr/local/lib/python3.10/site-packages/aiohttp/connector.py", line 540, in connect
proto = await self._create_connection(req, traces, timeout)
File "/usr/local/lib/python3.10/site-packages/aiohttp/connector.py", line 901, in _create_connection
_, proto = await self._create_direct_connection(req, traces, timeout)
File "/usr/local/lib/python3.10/site-packages/aiohttp/connector.py", line 1166, in _create_direct_connection
raise ClientConnectorError(req.connection_key, exc) from exc
aiohttp.client_exceptions.ClientConnectorError: Cannot connect to host api.tibber.com:443 ssl:default [Try again]
2023-03-06 11:02:09.302 ERROR (MainThread) [tibber] Error in watchdog connect, will retry. Retry count: 2
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/tibber/__init__.py", line 160, in _rt_watchdog
await self.rt_connect()
File "/usr/local/lib/python3.10/site-packages/tibber/__init__.py", line 115, in rt_connect
await self.sub_manager.connect_async()
File "/usr/local/lib/python3.10/site-packages/gql/client.py", line 632, in connect_async
await self.transport.connect()
File "/usr/local/lib/python3.10/site-packages/gql/transport/websockets_base.py", line 490, in connect
self.websocket = await asyncio.wait_for(
File "/usr/local/lib/python3.10/asyncio/tasks.py", line 445, in wait_for
return fut.result()
File "/usr/local/lib/python3.10/asyncio/tasks.py", line 650, in _wrap_awaitable
return (yield from awaitable.__await__())
File "/usr/local/lib/python3.10/site-packages/websockets/legacy/client.py", line 659, in __await_impl_timeout__
return await asyncio.wait_for(self.__await_impl__(), self.open_timeout)
File "/usr/local/lib/python3.10/asyncio/tasks.py", line 445, in wait_for
return fut.result()
File "/usr/local/lib/python3.10/site-packages/websockets/legacy/client.py", line 663, in __await_impl__
_transport, _protocol = await self._create_connection()
File "/usr/local/lib/python3.10/asyncio/base_events.py", line 1036, in create_connection
infos = await self._ensure_resolved(
File "/usr/local/lib/python3.10/asyncio/base_events.py", line 1418, in _ensure_resolved
return await loop.getaddrinfo(host, port, family=family, type=type,
File "/usr/local/lib/python3.10/asyncio/base_events.py", line 863, in getaddrinfo
return await self.run_in_executor(
File "/usr/local/lib/python3.10/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
File "/usr/local/lib/python3.10/socket.py", line 955, in getaddrinfo
for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -3] Try again
2023-03-06 11:02:09.311 ERROR (MainThread) [tibber] Watchdog: Connection is down
2023-03-06 11:02:54.329 ERROR (MainThread) [tibber] Error in watchdog connect, will retry. Retry count: 3
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/tibber/__init__.py", line 160, in _rt_watchdog
await self.rt_connect()
File "/usr/local/lib/python3.10/site-packages/tibber/__init__.py", line 115, in rt_connect
await self.sub_manager.connect_async()
File "/usr/local/lib/python3.10/site-packages/gql/client.py", line 632, in connect_async
await self.transport.connect()
File "/usr/local/lib/python3.10/site-packages/gql/transport/websockets_base.py", line 490, in connect
self.websocket = await asyncio.wait_for(
File "/usr/local/lib/python3.10/asyncio/tasks.py", line 445, in wait_for
return fut.result()
File "/usr/local/lib/python3.10/asyncio/tasks.py", line 650, in _wrap_awaitable
return (yield from awaitable.__await__())
File "/usr/local/lib/python3.10/site-packages/websockets/legacy/client.py", line 659, in __await_impl_timeout__
return await asyncio.wait_for(self.__await_impl__(), self.open_timeout)
File "/usr/local/lib/python3.10/asyncio/tasks.py", line 445, in wait_for
return fut.result()
File "/usr/local/lib/python3.10/site-packages/websockets/legacy/client.py", line 663, in __await_impl__
_transport, _protocol = await self._create_connection()
File "/usr/local/lib/python3.10/asyncio/base_events.py", line 1036, in create_connection
infos = await self._ensure_resolved(
File "/usr/local/lib/python3.10/asyncio/base_events.py", line 1418, in _ensure_resolved
return await loop.getaddrinfo(host, port, family=family, type=type,
File "/usr/local/lib/python3.10/asyncio/base_events.py", line 863, in getaddrinfo
return await self.run_in_executor(
File "/usr/local/lib/python3.10/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
File "/usr/local/lib/python3.10/socket.py", line 955, in getaddrinfo
for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -3] Try again
2023-03-06 11:02:54.334 ERROR (MainThread) [tibber] Watchdog: Connection is down
2023-03-06 11:03:15.348 ERROR (MainThread) [tibber] Error in watchdog connect, will retry. Retry count: 4
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/tibber/__init__.py", line 160, in _rt_watchdog
await self.rt_connect()
File "/usr/local/lib/python3.10/site-packages/tibber/__init__.py", line 115, in rt_connect
await self.sub_manager.connect_async()
File "/usr/local/lib/python3.10/site-packages/gql/client.py", line 632, in connect_async
await self.transport.connect()
File "/usr/local/lib/python3.10/site-packages/gql/transport/websockets_base.py", line 490, in connect
self.websocket = await asyncio.wait_for(
File "/usr/local/lib/python3.10/asyncio/tasks.py", line 445, in wait_for
return fut.result()
File "/usr/local/lib/python3.10/asyncio/tasks.py", line 650, in _wrap_awaitable
return (yield from awaitable.__await__())
File "/usr/local/lib/python3.10/site-packages/websockets/legacy/client.py", line 659, in __await_impl_timeout__
return await asyncio.wait_for(self.__await_impl__(), self.open_timeout)
File "/usr/local/lib/python3.10/asyncio/tasks.py", line 445, in wait_for
return fut.result()
File "/usr/local/lib/python3.10/site-packages/websockets/legacy/client.py", line 663, in __await_impl__
_transport, _protocol = await self._create_connection()
File "/usr/local/lib/python3.10/asyncio/base_events.py", line 1036, in create_connection
infos = await self._ensure_resolved(
File "/usr/local/lib/python3.10/asyncio/base_events.py", line 1418, in _ensure_resolved
return await loop.getaddrinfo(host, port, family=family, type=type,
File "/usr/local/lib/python3.10/asyncio/base_events.py", line 863, in getaddrinfo
return await self.run_in_executor(
File "/usr/local/lib/python3.10/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
File "/usr/local/lib/python3.10/socket.py", line 955, in getaddrinfo
for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -3] Try again
2023-03-06 11:03:15.355 ERROR (MainThread) [tibber] Watchdog: Connection is down
2023-03-06 11:03:21.842 ERROR (MainThread) [snitun.multiplexer.core] Ping fails, no response from peer
2023-03-06 11:03:27.865 ERROR (MainThread) [snitun.client.client_peer] Can't connect to SniTun server eu-west-2-1.ui.nabu.casa:443 with: [Errno -3] Try again
2023-03-06 11:03:27.869 ERROR (MainThread) [hass_nabucasa.remote] Connection problem to snitun server
2023-03-06 11:03:33.881 ERROR (MainThread)
Additional information
No response
About this issue
- Original URL
- State: closed
- Created a year ago
- Reactions: 2
- Comments: 156 (24 by maintainers)
@Danielhiversen I am not a coder even I work with software systems, I would provide if I had the skills in this. We are dependent on you for this feature 😦 Be care full starting a “sole” project if you cannot maintain it somehow. People believe in this and buy i.e, Tibber Pulse hardware, but also spend a lot of time on error tracking - on each person level of knowledge - in order to support your highly appreciated time on this plugin.
Sure. Here you go. I created the automation directly from the UI. Hope ot works for you.
alias: Tibber failsafe description: “” trigger:
Fixed by https://github.com/home-assistant/core/pull/96276
I’ve noticed that core is still using pytibber 0.27.0 and not 0.27.1. I have no dropouts or errors when manually using 0.27.1 by adding the integration to custom components and modifying the manifest.json to use pytibber 0.27.1.
Can someone please try the following and check if it solves the problem for them?:
I’ve now added more extensive logs for this specific user to show whether the server thinks it sent data to the client or not. That should allow us to see what happened just before the client closes connection next time.
I can check for the server logs if you experience any outages to see what happened there. I will need to have your home id, user id, IP or email address used to register at tibber. You can contact me via toni@tibber.com.
2023.8
@toni Sounds great that you improve the reconnect issues on your (tibber) side.
However as @LAPDragnet says, and been concluded by me and others, it is Data Fetching that stops within Tibber integration after Interruption of Connection to Tibber Servers… Why Tibber integration wont recover seems to be that the watchdog (if present) in Tibber Integration do not detect that no data is received even Tibber Integration log says connection is alive. Connection is alive but no payload/measure data is seen in the log. As long developer @Danielhiversen or someone else with skills wont look into this recovery wont work every time there is x-time network connection or service interruption to Tibber servers.
@papnak after detecting the problem with clients reconnecting in bursts leading to connection issues we stopped all Websocket API deployments and are working with an improved/delayed shutdown process (to avoid bursts to the new servers). The implementation is now done but we need to invest time in reviewing and testing this thoroughly to avoid troubles.
I suspect the issue you might be running into with fast refreshing the page is that some of the websocket connections stay open and do not get cleaned out (which is of course wrong but might happen if the client keeps it open). The maximum number of connections we allow per authentication token is 2. See Subscribing to data with websockets. If you exceed that you can no longer connect until the old connections are closed.
Chiming in here to mention that we did a deploy of the websocket server yesterday at around 16:25 CEST, and that would result in disconnects. Clients should reconnect and recover straight away, but if people are seeing actual timeouts during those reconnects, that is suspect. There should be no (general) downtime during a deploy. However, the quick burst of clients reconnecting is hard to handle, so we require clients to use random reconnection delays to not hammer our API, and some do.
For the people who saw timeouts yesterday evening, did the integration eventually recover, and if so, after how long time, or did you have to restart it?
Same for me, froze at exactly the same time. I am in Norway with a steady wireless network (Ubiquiti with several APs throughout the house, Tibber Pulse is locked to a dedicated AP so bad wifi is definitely not the issue for me). As many others, the issues started after update to 2023.3.X.
@ShogunMan Good point 👍 I have done a smaller fix in 0.27.1. I do not think that is the main issue, but will anyway update HA to use it.
I am from Sweden with rather rock stable 500 Mbit fiber and the Tibber Pulse connected via Unifi 6 LR APs. I always update when there is a new HA update, so running 2023.4.6 now.
To have a positive note, the summary is that my Tibber Integration has worked much better since March. But then I have added two stop gap solution automations after the discussions here that solves the resulting end user problem of “stalled Tibber sensor data” that makes energy readings die.
Look at the monthly energy summaries for this year where the Tibber integration “stalling” can be seen from the perspective of my wife 😉
Look at January and February with big gaps where the integration stalled, and I did not notice it before doing a manual reload.
Then in March I realized I needed to look at this and started posting here - getting good advice resulting in creating the “Tibber integration restart every night” automation that reload the config every night at 03:30. YAML attached below.
So, looking at the march energy stats we can see “stalls” on the 6:th and14:th of march that are “corrected” in the 03:30 reload of the config the day after. But then the energy for the day it stalled ends up on the next day (at 03:30) making the day after looking bad, and the day it died look “good”…
Then after more advice here I added the automation for “Tibber – Auto Reload” (YAML attached) in late March that reloads the configuration for Tiber if the “average power” metric from Tibber has not changed for an hour. So looking at April the end user (wife and kids) experience is good as can be seen. But then it reloads the config every night, AND the auto reload goes in if it has been stale for over an hour and reloads it. But these stop gap automations makes the resulting problem of the stalls go away at least 😃
For the state now - the last time the “Auto Reload” was triggered was on the 19:th of April (see screenshot), but it fixed the stall without any noticable metric loss as can be seen. But the last month (since I added the Auto Reload automation) the end result at least is stable as I do not have any “bad days” with lost energy consumption metrics since mid March…
And a note for using Tibber in the HA energy dashboards… Until the friendly advice from @IvovanWilligen in early March, I was using “accumulated consumption” from Tibber for the Energy dashboard but changed to “last meter consumption” that I recommend if anyone has set it up the way I did with accumulated! When using accumulated instead of last meter you get negative readings the day when it recovers.
A thought, if everyone was running something similar to the “Auto Reload” automation, it would be interesting to see if there is a correlation when it happens that could indicate an upstream problem in the Tibber APIs. Or if’s a local problem that has made the integration stall…
And - as images and graphs are fun, the Tibber problem from the Grafana perspective for 2023 where you can se the odd flat plateaus when Tibber Pulse integration has stalled in the “Energi” graph… And yes, winters are dark in Sweden 😃
Some Swedish translations for anyone interested in the other stuff 😉 Energi = Energy (kind of obvious) Värmepump = Heat Pump Innomhus = Indoor Utomhus = Outdoor
@RicardP Would be great if you could help out. Any pull request is more than welcome.
This is something I do in my spare time. Unfortunately, I do not have much time to spend on this at the moment. I do some work on it, and have had some progress. So if no other fixes it, I will fix it at some point
Im sharing logs for one user that experienced outage on 14th April.
Client seems to repeatedly close connection with code 1000 which means “The client closes the socket and the connection by dispatching a 1000: Normal Closure close event to the server indicating a normal closure” (https://github.com/enisdenjo/graphql-ws/blob/master/PROTOCOL.md).
We should try to figure out why the client thinks the connection needs to be closed.
Having the full tibber integration under custom_components seems to solve the problem, at least for me. After removing it from custom components and relying on the core integration, I started experiencing hangups again.
I have now tried to circumvent this by creating an automation that reloads Tibber if the state of the Pulse “Power” hasn’t changed in 5 Minutes. This should mean that the maximum lost time would be 5 Minutes which is OK I guess