core: Netatmo Error handling request AND Update for camera.netatmo_presence_fails AND Too many retries AND taking over 10 seconds

The problem

Allready discussed this error on Discord with cgtobi. Putting it in github to track the progress of the issue out of curiosisty. No hurry everyhing works. But the system seems to be unhappy. The log is shouting daily:

Error handling request Update for camera.netatmo_presence_achtertuin fails Update of camera.netatmo_presence_achtertuin is taking over 10 seconds Too many retries

Environment

Multiple devices in house displaying the lovelace frontend using the offical homeassistant app.

arch	armv7l
dev	false
docker	true
hassio	true
os_name	Linux
os_version	4.19.114-v7l
python_version	3.7.7
timezone	Europe/Amsterdam
version	0.109.6
virtualenv	false

• Home Assistant Core release with the issue: Unknown
• Last working Home Assistant Core release (if known): It works
• Operating environment (Home Assistant/Supervised/Docker/venv): Home Assistant 0.109.6 Supervisor: 222 and HassOS 3.13
• Integration causing this issue: Netatmo
• Link to integration documentation on our website: https://www.home-assistant.io/integrations/netatmo/

Problem-relevant configuration.yaml

netatmo: client_id: thesecretid client_secret: thesecretcode

Traceback/Error logs

2020-05-09 17:20:31 ERROR (MainThread) [homeassistant.components.netatmo] Error during webhook registration - 400 - Bad request - Allowed ports for webhooks are 443 and 80. (21) when accessing 'https://api.netatmo.com/api/addwebhook'
2020-05-09 22:35:16 WARNING (MainThread) [homeassistant.helpers.entity] Update of camera.netatmo_presence_achtertuin is taking over 10 seconds
2020-05-09 22:51:16 WARNING (MainThread) [homeassistant.helpers.entity] Update of camera.netatmo_presence_achtertuin is taking over 10 seconds
2020-05-09 22:51:24 ERROR (SyncWorker_11) [pyatmo.auth] Too many retries
2020-05-09 22:51:24 ERROR (MainThread) [homeassistant.helpers.entity] Update for camera.netatmo_presence_achtertuin fails
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 279, in async_update_ha_state
    await self.async_device_update()
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 472, in async_device_update
    await self.hass.async_add_executor_job(self.update)
  File "/usr/local/lib/python3.7/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/src/homeassistant/homeassistant/components/netatmo/camera.py", line 212, in update
    self._data.update()
  File "/usr/src/homeassistant/homeassistant/util/__init__.py", line 240, in wrapper
    result = method(*args, **kwargs)
  File "/usr/src/homeassistant/homeassistant/components/netatmo/camera.py", line 261, in update
    self.camera_data = pyatmo.CameraData(self.auth, size=100)
  File "/usr/local/lib/python3.7/site-packages/pyatmo/camera.py", line 30, in __init__
    resp = self.authData.post_request(url=_GETHOMEDATA_REQ, params=postParams)
  File "/usr/local/lib/python3.7/site-packages/pyatmo/auth.py", line 168, in post_request
    if "application/json" in resp.headers.get("content-type")
AttributeError: 'NoneType' object has no attribute 'headers'
2020-05-09 23:38:16 WARNING (MainThread) [homeassistant.helpers.entity] Update of camera.netatmo_presence_achtertuin is taking over 10 seconds
2020-05-09 23:38:24 ERROR (SyncWorker_6) [pyatmo.auth] Too many retries
2020-05-09 23:38:24 ERROR (MainThread) [homeassistant.helpers.entity] Update for camera.netatmo_presence_achtertuin fails
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 279, in async_update_ha_state
    await self.async_device_update()
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 472, in async_device_update
    await self.hass.async_add_executor_job(self.update)
  File "/usr/local/lib/python3.7/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/src/homeassistant/homeassistant/components/netatmo/camera.py", line 212, in update
    self._data.update()
  File "/usr/src/homeassistant/homeassistant/util/__init__.py", line 240, in wrapper
    result = method(*args, **kwargs)
  File "/usr/src/homeassistant/homeassistant/components/netatmo/camera.py", line 261, in update
    self.camera_data = pyatmo.CameraData(self.auth, size=100)
  File "/usr/local/lib/python3.7/site-packages/pyatmo/camera.py", line 30, in __init__
    resp = self.authData.post_request(url=_GETHOMEDATA_REQ, params=postParams)
  File "/usr/local/lib/python3.7/site-packages/pyatmo/auth.py", line 168, in post_request
    if "application/json" in resp.headers.get("content-type")
AttributeError: 'NoneType' object has no attribute 'headers'
2020-05-10 00:10:17 WARNING (MainThread) [homeassistant.helpers.entity] Update of camera.netatmo_presence_achtertuin is taking over 10 seconds
2020-05-10 00:10:48 WARNING (MainThread) [homeassistant.helpers.entity] Update of camera.netatmo_presence_achtertuin is taking over 10 seconds
2020-05-10 00:10:56 ERROR (SyncWorker_17) [pyatmo.auth] Too many retries
2020-05-10 00:10:56 ERROR (MainThread) [homeassistant.helpers.entity] Update for camera.netatmo_presence_achtertuin fails
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 279, in async_update_ha_state
    await self.async_device_update()
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 472, in async_device_update
    await self.hass.async_add_executor_job(self.update)
  File "/usr/local/lib/python3.7/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/src/homeassistant/homeassistant/components/netatmo/camera.py", line 217, in update
    cid=self._camera_id
  File "/usr/local/lib/python3.7/site-packages/pyatmo/camera.py", line 284, in camera_urls
    temp_local_url = check_url(vpn_url)
  File "/usr/local/lib/python3.7/site-packages/pyatmo/camera.py", line 277, in check_url
    resp = self.authData.post_request(url=f"{url}/command/ping")
  File "/usr/local/lib/python3.7/site-packages/pyatmo/auth.py", line 168, in post_request
    if "application/json" in resp.headers.get("content-type")
AttributeError: 'NoneType' object has no attribute 'headers'
2020-05-10 00:11:03 ERROR (MainThread) [aiohttp.server] Error handling request
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/urllib3/response.py", line 437, in _error_catcher
    yield
  File "/usr/local/lib/python3.7/site-packages/urllib3/response.py", line 519, in read
    data = self._fp.read(amt) if not fp_closed else b""
  File "/usr/local/lib/python3.7/http/client.py", line 457, in read
    n = self.readinto(b)
  File "/usr/local/lib/python3.7/http/client.py", line 501, in readinto
    n = self.fp.readinto(b)
  File "/usr/local/lib/python3.7/socket.py", line 589, in readinto
    return self._sock.recv_into(b)
ConnectionResetError: [Errno 104] Connection reset by peer

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/requests/models.py", line 751, in generate
    for chunk in self.raw.stream(chunk_size, decode_content=True):
  File "/usr/local/lib/python3.7/site-packages/urllib3/response.py", line 576, in stream
    data = self.read(amt=amt, decode_content=decode_content)
  File "/usr/local/lib/python3.7/site-packages/urllib3/response.py", line 541, in read
    raise IncompleteRead(self._fp_bytes_read, self.length_remaining)
  File "/usr/local/lib/python3.7/contextlib.py", line 130, in __exit__
    self.gen.throw(type, value, traceback)
  File "/usr/local/lib/python3.7/site-packages/urllib3/response.py", line 455, in _error_catcher
    raise ProtocolError("Connection broken: %r" % e, e)
urllib3.exceptions.ProtocolError: ("Connection broken: ConnectionResetError(104, 'Connection reset by peer')", ConnectionResetError(104, 'Connection reset by peer'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/netatmo/camera.py", line 99, in camera_image
    f"{self._localurl}/live/snapshot_720.jpg", timeout=10
  File "/usr/local/lib/python3.7/site-packages/requests/api.py", line 76, in get
    return request('get', url, params=params, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 530, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 683, in send
    r.content
  File "/usr/local/lib/python3.7/site-packages/requests/models.py", line 829, in content
    self._content = b''.join(self.iter_content(CONTENT_CHUNK_SIZE)) or b''
  File "/usr/local/lib/python3.7/site-packages/requests/models.py", line 754, in generate
    raise ChunkedEncodingError(e)
requests.exceptions.ChunkedEncodingError: ("Connection broken: ConnectionResetError(104, 'Connection reset by peer')", ConnectionResetError(104, 'Connection reset by peer'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/aiohttp/web_protocol.py", line 418, in start
    resp = await task
  File "/usr/local/lib/python3.7/site-packages/aiohttp/web_app.py", line 458, in _handle
    resp = await handler(request)
  File "/usr/local/lib/python3.7/site-packages/aiohttp/web_middlewares.py", line 119, in impl
    return await handler(request)
  File "/usr/src/homeassistant/homeassistant/components/http/real_ip.py", line 39, in real_ip_middleware
    return await handler(request)
  File "/usr/src/homeassistant/homeassistant/components/http/ban.py", line 73, in ban_middleware
    return await handler(request)
  File "/usr/src/homeassistant/homeassistant/components/http/auth.py", line 127, in auth_middleware
    return await handler(request)
  File "/usr/src/homeassistant/homeassistant/components/http/view.py", line 125, in handle
    result = await result
  File "/usr/src/homeassistant/homeassistant/components/camera/__init__.py", line 490, in get
    return await self.handle(request, camera)
  File "/usr/src/homeassistant/homeassistant/components/camera/__init__.py", line 507, in handle
    image = await camera.async_camera_image()
  File "/usr/src/homeassistant/homeassistant/components/camera/__init__.py", line 376, in async_camera_image
    return await self.hass.async_add_executor_job(self.camera_image)
  File "/usr/local/lib/python3.7/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/src/homeassistant/homeassistant/components/netatmo/camera.py", line 118, in camera_image
    cid=self._camera_id
  File "/usr/local/lib/python3.7/site-packages/pyatmo/camera.py", line 284, in camera_urls
    temp_local_url = check_url(vpn_url)
  File "/usr/local/lib/python3.7/site-packages/pyatmo/camera.py", line 277, in check_url
    resp = self.authData.post_request(url=f"{url}/command/ping")
  File "/usr/local/lib/python3.7/site-packages/pyatmo/auth.py", line 158, in post_request
    f"{resp.status_code} - "
KeyError: 502

Additional information

Using 2 netatmo presence’s. Using the new config flow (:8123/config/integrations> choose yaml config) Multiple display’s in the house (official app on android and ios).

Sometimes netatmo’s don’t show camera picture in lovelace when the ios/android app wasn’t used for a while and lost connection. You need to force refresh then the get the camera’s again, while other integrations just (seem to) display, but offcourse their an other type of integration.