core: Sensor Community Sensors changes from available to unavailable every few minutes

This issue is still very topical. The problem occurs regularly for me as well. Please do not close this issue yet.

If the API is unreachable or overloaded, the last value should be preserved for a period of time.

• Using Restful integration instead with boosted timeout (120s) and reasonable scan interval (≥5 min, I used 10min) gives me much more stable results.
• Consider using “bbox” api endpoint to grab data from multiple nearby sensors (and sensor types) at once in a single request https://data.sensor.community/airrohr/v1/filter/box=min_latitude,min_longitude,max_latitude,max_longitude

Results, not perfect but huge improvement is visible: (while pulling data from more sensors at the same time)

I found this discussion about real-time local use of the data: https://forum.sensor.community/t/using-a-custom-server-api-for-real-time-data/1532. Looks interesting as a starting point.

May be we should consider another approach. I understand there is an option to import the sensor data directly into a local program. That should also be more in line with the HA philosophy: processing locally as much as possible.

I understand your reasoning and the generic preference of local API. But in this integration‘s case, the cloud polling (also) makes sense. Think about the people who don’t run their own sensors, but want to ingest data from someone in their neighborhood. Like me 😉 After all, that‘s why sensor owners share the data with the Luftdaten network, right?

2022.3.5 also: ssl.SSLWantReadError: The operation did not complete (read) (_ssl.c:1129), asyncio.exceptions.CancelledError, httpcore.ConnectTimeout and httpx.ConnectTimeout

https://community.home-assistant.io/t/luftdaten-info-local-api-sensors/46647

Could also be interesting

Looks like the problem is timeouts on the Luftdaten API, i see 3 possible solutions:

• ignore and return last value
• increase timeout
• add a retry

May be we should consider another approach. I understand there is an option to import the sensor data directly into a local program. That should also be more in line with the HA philosophy: processing locally as much as possible.

If this is a bridge too far for the moment, I think increasing timeout or adding one or more retries might help. Somehow I think we should not opt for ignore and return last value

In addition a code review might be a good option. I do not understand the API good enough, but from the logging I get the impression that there is more happening than just a time-out.

• a lot of noise in the logbook is generated from this integration

At least this part of the issue can be prevented by configuring the logger component appropriately. Add this to your configuration.yaml:

logger:
  default: warning
  logs:
    homeassistant.components.luftdaten: fatal

Same issue with core-2022.3.4

Logger: homeassistant.components.luftdaten Source: components/luftdaten/init.py:43 Integration: Sensor.Community (documentation, issues) First occurred: 18:31:57 (5 occurrences) Last logged: 19:12:17

Unexpected error fetching luftdaten_53519 data: Traceback (most recent call last): File “/usr/local/lib/python3.9/site-packages/anyio/streams/tls.py”, line 108, in _call_sslobject_method result = func(*args) File “/usr/local/lib/python3.9/ssl.py”, line 888, in read v = self._sslobj.read(len) ssl.SSLWantReadError: The operation did not complete (read) (_ssl.c:2633)

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File “/usr/local/lib/python3.9/site-packages/httpcore/backends/asyncio.py”, line 31, in read return await self._stream.receive(max_bytes=max_bytes) File “/usr/local/lib/python3.9/site-packages/anyio/streams/tls.py”, line 171, in receive data = await self._call_sslobject_method(self._ssl_object.read, max_bytes) File “/usr/local/lib/python3.9/site-packages/anyio/streams/tls.py”, line 115, in _call_sslobject_method data = await self.transport_stream.receive() File “/usr/local/lib/python3.9/site-packages/anyio/_backends/_asyncio.py”, line 1105, in receive await self._protocol.read_event.wait() File “/usr/local/lib/python3.9/asyncio/locks.py”, line 226, in wait await fut asyncio.exceptions.CancelledError

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File “/usr/local/lib/python3.9/site-packages/httpcore/_exceptions.py”, line 8, in map_exceptions yield File “/usr/local/lib/python3.9/site-packages/httpcore/backends/asyncio.py”, line 33, in read return b"" File “/usr/local/lib/python3.9/site-packages/anyio/_core/_tasks.py”, line 103, in exit raise TimeoutError TimeoutError

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File “/usr/local/lib/python3.9/site-packages/httpx/_transports/default.py”, line 60, in map_httpcore_exceptions yield File “/usr/local/lib/python3.9/site-packages/httpx/_transports/default.py”, line 308, in handle_async_request resp = await self._pool.handle_async_request(req) File “/usr/local/lib/python3.9/site-packages/httpcore/_async/connection_pool.py”, line 244, in handle_async_request raise exc File “/usr/local/lib/python3.9/site-packages/httpcore/_async/connection_pool.py”, line 228, in handle_async_request response = await connection.handle_async_request(request) File “/usr/local/lib/python3.9/site-packages/httpcore/_async/connection.py”, line 90, in handle_async_request return await self._connection.handle_async_request(request) File “/usr/local/lib/python3.9/site-packages/httpcore/_async/http11.py”, line 102, in handle_async_request raise exc File “/usr/local/lib/python3.9/site-packages/httpcore/_async/http11.py”, line 81, in handle_async_request ) = await self._receive_response_headers(**kwargs) File “/usr/local/lib/python3.9/site-packages/httpcore/_async/http11.py”, line 143, in _receive_response_headers event = await self._receive_event(timeout=timeout) File “/usr/local/lib/python3.9/site-packages/httpcore/_async/http11.py”, line 172, in _receive_event data = await self._network_stream.read( File “/usr/local/lib/python3.9/site-packages/httpcore/backends/asyncio.py”, line 33, in read return b"" File “/usr/local/lib/python3.9/contextlib.py”, line 137, in exit self.gen.throw(typ, value, traceback) File “/usr/local/lib/python3.9/site-packages/httpcore/_exceptions.py”, line 12, in map_exceptions raise to_exc(exc) httpcore.ReadTimeout

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File “/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py”, line 187, in _async_refresh self.data = await self._async_update_data() File “/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py”, line 147, in _async_update_data return await self.update_method() File “/usr/src/homeassistant/homeassistant/components/luftdaten/init.py”, line 43, in async_update await sensor_community.get_data() File “/usr/local/lib/python3.9/site-packages/luftdaten/init.py”, line 29, in get_data response = await client.get(str(url)) File “/usr/local/lib/python3.9/site-packages/httpx/_client.py”, line 1736, in get return await self.request( File “/usr/local/lib/python3.9/site-packages/httpx/_client.py”, line 1513, in request return await self.send(request, auth=auth, follow_redirects=follow_redirects) File “/usr/local/lib/python3.9/site-packages/httpx/_client.py”, line 1600, in send response = await self._send_handling_auth( File “/usr/local/lib/python3.9/site-packages/httpx/_client.py”, line 1628, in _send_handling_auth response = await self._send_handling_redirects( File “/usr/local/lib/python3.9/site-packages/httpx/_client.py”, line 1665, in _send_handling_redirects response = await self._send_single_request(request) File “/usr/local/lib/python3.9/site-packages/httpx/_client.py”, line 1702, in _send_single_request response = await transport.handle_async_request(request) File “/usr/local/lib/python3.9/site-packages/httpx/_transports/default.py”, line 308, in handle_async_request resp = await self._pool.handle_async_request(req) File “/usr/local/lib/python3.9/contextlib.py”, line 137, in exit self.gen.throw(typ, value, traceback) File “/usr/local/lib/python3.9/site-packages/httpx/_transports/default.py”, line 77, in map_httpcore_exceptions raise mapped_exc(message) from exc httpx.ReadTimeout