core: Essent API "Client Error: Forbidden for url"

The problem

unable to start integration

What is version of Home Assistant Core has the issue?

core-2021.8.2

What was the last working version of Home Assistant Core?

No response

What type of installation are you running?

Home Assistant Container

Integration causing the issue

essent

Link to integration documentation on our website

https://www.home-assistant.io/integrations/essent/

Example YAML snippet

sensor:
- platform: essent
  username: !secret ESSENT_USER
  password: !secret ESSENT_PASS

Anything in the logs that might be useful for us?

Logger: homeassistant.components.sensor
Source: components/essent/sensor.py:76
Integration: Sensor (documentation, issues)
First occurred: 18:51:33 (1 occurrences)
Last logged: 18:51:33
Error while setting up essent platform for sensor

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/entity_platform.py", line 249, in _async_setup_platform
    await asyncio.shield(task)
  File "/usr/local/lib/python3.9/concurrent/futures/thread.py", line 52, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/src/homeassistant/homeassistant/components/essent/sensor.py", line 26, in setup_platform
    essent = EssentBase(username, password)
  File "/usr/src/homeassistant/homeassistant/components/essent/sensor.py", line 63, in __init__
    self.update()
  File "/usr/src/homeassistant/homeassistant/util/__init__.py", line 277, in wrapper
    result = method(*args, **kwargs)
  File "/usr/src/homeassistant/homeassistant/components/essent/sensor.py", line 76, in update
    essent = PyEssent(self._username, self._password)
  File "/usr/local/lib/python3.9/site-packages/pyessent/pyessent.py", line 114, in __init__
    PyEssent.User.authenticate_user(username, password)
  File "/usr/local/lib/python3.9/site-packages/pyessent/pyessent.py", line 108, in authenticate_user
    r.raise_for_status()
  File "/usr/local/lib/python3.9/site-packages/requests/models.py", line 943, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://api.essent.nl/selfservice/user/authenticateUser

Additional information

I tried the link to essent you stated here: https://github.com/home-assistant/core/issues/24016#issuecomment-494692046 and that works.

About this issue

Original URL
State: closed
Created 3 years ago
Comments: 15 (6 by maintainers)

Most upvoted comments

Well, that was a helpful start. It seems they have completely revamped their entire API though, like, it doesn’t even use XML anymore but JSON instead. I don’t really have time to rewrite this integration from scratch (because frankly, that’s what seemingly needs to be done), I’m afraid. But at least whoever has time should have a less painful time than I had at first given it’s all JSON now.

I’ve been doing a bit more digging tonight, and the SOAP / XML api is still there and still in use, despite this new JSON API (for how long I’m not sure)

Something in the new UI is calling https://api.essent.nl/support/isCustomerAuthenticated?_=XXXXXX (where XXXXXX is some id)

And using the same session in the proxy I’m using I was able to call all the XML endpoints used in PyEssent to get my current meter readings.

So once I work out where this ID is coming from (and indeed if it’s even needed) I don’t think a re-write would be required just yet but hopefully just refactoring the login methods would be sufficient that the script has the correct authentication in place for the legacy XML API.

Will report back here with more findings, I’ve forked PyEssent as if I have the energy / time I may take a stab at this 😃

Update: From what I can tell the _=XXXXX query param is not needed 😃 So this just leaves refactoring the authenticate_user method.

craigtsmith on Aug 18, 2021