core: Ecobee integration unable to generate refresh token
The problem
Ecobee integration stopped working. Sensors and climate entities became undefined.
Environment
arch | x86_64
-- | --
dev | false
docker | true
hassio | true
os_name | Linux
os_version | 4.19.107
python_version | 3.7.7
timezone | America/Chicago
version | 0.108.3
virtualenv | false
Home Assistant 0.108.3 Supervisor 217 HassOS 3.12
- Home Assistant Core release with the issue: 0.108.3
- Last working Home Assistant Core release (if known): 0.108.3
- Operating environment (Home Assistant/Supervised/Docker/venv): Supervised
- Integration causing this issue: ecobee
- Link to integration documentation on our website: https://www.home-assistant.io/integrations/ecobee/
Problem-relevant configuration.yaml
configured via configuration flow in ui.
Traceback/Error logs
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/pyecobee/__init__.py", line 518, in _request
response.raise_for_status()
File "/usr/local/lib/python3.7/site-packages/requests/models.py", line 941, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: https://api.ecobee.com/token?grant_type=refresh_token&refresh_token=xxx&client_id=xxx
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/config_entries.py", line 216, in async_setup
hass, self
File "/usr/src/homeassistant/homeassistant/components/ecobee/__init__.py", line 58, in async_setup_entry
if not await data.refresh():
File "/usr/src/homeassistant/homeassistant/components/ecobee/__init__.py", line 105, in refresh
if await self._hass.async_add_executor_job(self.ecobee.refresh_tokens):
File "/usr/local/lib/python3.7/concurrent/futures/thread.py", line 57, in run
result = self.fn(*self.args, **self.kwargs)
File "/usr/local/lib/python3.7/site-packages/pyecobee/__init__.py", line 153, in refresh_tokens
auth_request=True,
File "/usr/local/lib/python3.7/site-packages/pyecobee/__init__.py", line 533, in _request
"ecobee tokens invalid; re-authentication required"
pyecobee.errors.InvalidTokenError: ecobee tokens invalid; re-authentication required```
Additional information
I’ve had this happen once before. I can remove the integration and reinstall it to solve the problem The link given above appears to be resolving properly when I ran a curl command to it so I am eliminating local network issues. Is this a transient error occurring randomly? Or is it a regular timeout associated with the ecobee api? Is there anything I can do to prevent it in the future?
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Reactions: 5
- Comments: 113 (47 by maintainers)
@quielb This is actually a different issue (you are receiving the ExpiredTokenError, not the InvalidTokenError) and I will explain why it is happening for the benefit of the thread:
When you request tokens from ecobee you get two important tokens: an access token and a refresh token. The access token lasts for one hour. The refresh token lasts for one year OR until it is used to request a new access token.
In the normal course of things, the integration updates its values either every three minutes or when you take an action (i.e. set the temperature, change the mode, etc.). If the request indicates that the access token has expired, then the ExpiredTokenError is raised.
If the Error is raised from a regular every-three-minute update, the Error is caught and the tokens are refreshed and you get a new access token, which all happens transparently to the user.
Currently, other ExpiredTokenErrors are not caught (e.g. any action on the thermostat - setting e.g. temp or mode) and this Error is raised. The reason that ecobee goes on working for you through the day is that the next update refreshes the token.
This error should only show up rarely as you need to be right at the end of the access token validity period, and THEN take an action for the access token to have been expired but not-yet-refreshed.
Once I solve the InvalidTokenError that keeps popping up, I plan to introduce a mechanism to the integration that will catch all ExpiredTokenErrors, refresh, and then automatically retry the failed action so that the action completes transparently for the user.
As a workaround, is there a way to detect this and just start a re-auth workflow without having to delete and re-add the whole integration? Sorry if I’m messing up terminology. This is what used to happen. The front end (non-Lovelace, non-integration) would show that Ecobee isn’t authorized and you would get prompted to do it again.
Alrighty, that took longer to reproduce then I had hoped but here we are. I was able to reproduce by following these steps:
Now, I have debug logs of all those steps. Going to try and pull in the parts around pyecobee and homeassistant.components.ecobee. I will still have the full logs so we can search through them more but I don’t really want to try and sanitize the entire log.
Note that I masked the tokens just incase any of them are still valid. I left the last 4 characters and tried to keep them consistant. If the last 4 are the same then you should be able to assume they are fully the same.
@vinniefalco that is exactly what I am saying. Home Assistant contacts ecobee.com, which then contacts your thermostat, which then relays data to ecobee.com, which Home Assistant then polls for data.
Your ecobee is logged in to your ecobee account when you first set it up (it won’t work otherwise). The fact that your ecobee is behind your firewall is irrelevant. You were under the wrong impression - your ecobee is connected to the cloud (that’s how the ecobee app on your phone works too).
I just pushed a PR that will make the debug logging for the dependency more explicit so that we can trace more precisely where the problem lies. @rsnodgrass may be correct that there is a race condition going on here but we can’t know for sure until we see where the failing lies, in HA or in the dependency. I’ve asked for this to get put into a point release but we will see. Savvy testers could update the dependency themselves to 0.2.7 to get the improved debug logging.
You don’t need to remove the app. Just delete and re-add the integration, put in the dev api key, add the app using the 4 character code and it will replace the existing app. I have to do this a lot and ecobee’s website runs like garbage so I do whatever I can to make the process quicker.
While it’s not ideal, the only way I have been able to get around this issue is by removing the integration, removing the app from the ecobee portal, and then re-adding the ecobee integration (via the UI), including the whole pin authorization process.
Any updates on this? Should I start digging a little?
@MartinHjelmare Can I run something by you here?
As background: ecobee’s API uses an auth token and a refresh token. The auth token is good for 1 hour, the refresh token is good for 1 year, or until it is used to refresh tokens, when it is then expired (a new refresh token is issued along with the new auth token on a successful refresh). When the auth token expires, pyecobee raises an exception and the integration refreshes the tokens. The new refresh token gets stored in the config entry by the refresh method:
At startup of HA, the integration loads the stored refresh token from the entry, then refreshes tokens to get a new access token. As long as it hasn’t been 1 year since HA was started, the refresh procedure should yield a new access token.
The problem (that has shown up in this issue and several other issues since the conversion to config entries) is that occasionally when a user restarts HA, the refresh token raises an invalid exception (meaning, either it has expired after 1 year or it has already been used to refresh tokens - this exception gets raised in pyecobee when the API response indicates that the tokens have become invalid).
I have been digging trying to figure out why this would be happening. One of the only things I can think of is that the following (from the refresh method above) is not always actually updating the entry with the new refresh token:
So, when HA restarts, it loads an old refresh token from the entry and the ecobee API flags it as invalid.
Any thoughts/suggestions?
Hi, has there been any progress or workarounds for this specific issue? Every now and then one of my automations will fail due to expired token. I think some kind of try/catch with a retry would be the easiest solution,
Any others in this thread still experiencing the InvalidTokenError? I’m wondering if this has been solved through PRs elsewhere in HA.
Thanks for the clarification. To be honest I didn’t really compare my stack trace to the one already posted. I think I just let my frustration with the expired token get to me. I think I must have the worst luck of anybody on the planet. You mention that the error should show up rarely, but for me it is almost a daily occurrence. I have 3 or 4 automations that adjust the thermostat throughout the day. It seems that one of them fails, almost daily. If I can get 2 days without the expired token error, I’ve gotten luck. Looking forward to your fix.
@lmatter Sorry, thanks for bearing with me, getting the kinks worked out. Try the version attached to this message please! ecobee_cc_20200120.tar.gz
@rsnodgrass @j4ys0n
If anyone is feeling adventurous and wants to help me by testing, I’ve attached a tar.gz of the ecobee integration that you should be able to drop into custom_components that includes a possible fix. Before you do the following, delete the existing config entry for ecobee (you will need to add the integration and do the config flow again after loading this custom code) and stop Home Assistant.
ecobee.tar.gz
Then, start HA, and add the ecobee integration.
Technical details: instead of saving the API_KEY and REFRESH_TOKEN to the config entry, I’m using the inbuilt load_json and save_json utils to retrieve and save them from ecobee.conf in the config directory. Please post results of testing here. The code is super dirty at the moment and needs to be cleaned up before PR’ing, but I want to see if this generally solves the problem before going further.
another upgrade, another re-activate
was only a few revs back, not sure what exactly. current: Host Operating System | Home Assistant OS 5.8 Supervisor Version | 2020.12.6 Home Assistant 2020.12.0
deleted, recreated app on ecobee dev … all back, still annoyed. 😃
As you can see above, I think I’ve tracked this to an underlying issue in the way that the refreshed keys are saved to the config entry. It does not appear to have anything to do with the integration or pyecobee code as such. If someone with more knowledge of the workings of HA wants to take a run at tracking down the issue, I would be happy to assist.
I ended up just removing and re-adding the integration… not ideal since ecobee’s website is garbage… but it worked.
@marthoc is there anything else in the logs you may want to see around this? To your note that this indicates it is not the ecobee integration I would point out that although I did not consider my smartthings integration in this test it does exhibit the same problem especially when I upgrade to 0.112.x
Ok. It’s not clear why that happens.
@rmevans9 Thank you for this, I think it is the clearest indication that my theory is correct: the code to update the config entry isn’t (always) doing so.
At 20:59:03 we see that the tokens are successfully refreshed (extra debug code I added to pyecobee confirms that step succeeds and returns True) - refresh token retrieved is nNo0.
After the reboot, at 21:04:15 when a refresh is attempted, refresh token 8aGD is used, which is the token from the previous refresh.
@MartinHjelmare I hate to keep tagging you on this issue, but I think this is the clearest indication I have yet that it’s not the ecobee integration code that is failing here…
@rmevans9 you need to have manually updated python-ecobee-api to get the debug logging, since it has not been included into a released version (even as of 0.112.3 released today, as far as I can tell).
It’s definitely the same issue as in the previous issue linked. That last issue was closed after things became less than friendly.
Here’s what I need from anyone experiencing the issue: enable debug logging for both the integration and pyecobee. Wait until the error happens. Then send me your home-assistant.log. I need to see the sequence of events leading up to the error occurring to trace where the problem lies.
Your help is needed!