core: ForecastSolar can't recover when API rate limit is hit

Have a same issue after update home assistant to version 2024.1.5. Updated on Saturday 20.1. evening. Since this time the Forecast.Solar integration has an error.

Knut here, possibly it would be a way to check responses with HTTP code 429 for retry at

or the headers

This should be stored somewhere and checked before next call. (This does not even have to be deleted, as every call in the far future will always be after this timestamp)

The zone holds with IP ... or API key ... the reason/scope, e.g. for logging.

Not sure if it’s same error but this integration is not working anymore for some time now! This is in the logs:

Update: disabling the integration for at least an hour and then reenabling makes the integration work again. So indeed it seems it just can’t recover when API limit is hit.

I’m running into the problem as well, with an anonymous account (so no API Key); from the Code it’s quite clear that it should only updaten hourly when no API-Key is set. I believe the problem for me (and maybe for others as well) is, that if the API call fails, then Home Assistant tries to re-initialize the Integration after 1 Minute (or 90 seconds, which I believe is more correct), then receives the same error & after 1 Minute tries again, and again, and again… …pushing the next possible working call further into the future.

It would be probably be best, going full circle with the issue, to not let the Integration go into an erronious state with this error, so that the Integration itself can handle the interval, without being re-initialized all over.

I did some research to get more clarity about where things go wrong and especially how the problem can persist for days.

By default, if you make more than 12 requests to the API in an hour, you will get a rate limit error and this can be caused, for example, by many restarts of Home Assistant. But what I noticed is that with a rate limit exception, an error appears in the logs every minute, so it may try to execute a request to the API every minute.

As a result, the reset time is always pushed forward (rolling reset time), which puts you in a limbo and the problem does not solve itself after waiting 1 hour and can last for days.

I have ran into the same senario as above, in my case the error was quite quick to resolve.

1. Disable the solar integration,
2. The command: curl -v -H ‘Accept: text/csv’ ‘https://api.forecast.solar/estimate/watthours/day/52/12/37/0/5.67’
3. I noticed that it is restricting the data to the IP address of the WAN side.
4. so I just reset the internet connection to get a new public address, enabled the integration and all was well.
5. for static ip addresses it may be much more difficult to get a new IP, or you can make use of a VPN to change the device`s out IP or just block the device for a hour or so, but this step is a bit annoying to do. I do agree to have a waiting period active with the integration or check past requests and not request more than required. that will help to minimise the requests as mentioned above.

I am encountering the same issue. I tried disabling the integration for 12 hours (overnight) to ensure it did not exceed the API call limit. I re-enabled it, and the same issue occurred immediately on the first API call as the integration was starting.

I’ve also had the same, I wonder if either failed accesses are counted against you as a tally (e.g. I had this happening for 24+ hours before I noticed, so I would have accumulated 3 * 24 * 40 = 2880 rejected requests, which at 12 calls per IP per hour is going to take 10 days to clear 😭 Equally it could just be a bug in the rate limiting at Forecast.Solar.

I hit the limiting too. Because of the internal integration reloads, it never recovered. I cannot confirm your calculation because my test calls showed that a rate-limiting would end in about 1–2 hours in the future.

I got it back to work:

1. I disabled the integration to disable the API calls.
2. Test calls: With an example API call, you can check when you are allowed to call again:

curl -v -H 'Accept: text/csv' 'https://api.forecast.solar/estimate/watthours/day/52/12/37/0/5.67'

As already mentioned above you will get details when you are allowed to try next, what are the limits and the current calls which are registered:

- x-ratelimit-period
- x-ratelimit-limit
- x-retry-at

1. when you are able to get information with the curl call

2024-02-15;6707
2024-02-16;12691

1. enable integration: I enabled it one by one because of several planes/strings. Just to make sure, re-enabling everything at once could possibly trigger the limit again.
2. Integration is working again.

Anyway: The RatelimitException (HTTP Codes 429) should not be treated as an integration fault. Maybe just log an unsuccessful call and a hint for old data, but without reloading or restarting a new attempt.

For now, this behavior leads to never ending reloads and API limiting.

Depending on how sophisticated you want the checks to be:

• https://doc.forecast.solar/for_developers#api_key
• https://doc.forecast.solar/api:misc#check_location

As I said here, I think (independent of concrete implementation because not familiar with HA) about an abstract logic during integration installation, boot up, normal run mode etc.

• Integration installation comes e.g. with a default .retry-at flag file with 1970-01-01 00:00:00 in it
• On API fetch, the actual system timestamp is checked against “retry at” time in the flag file
  • if “now” is after “retry at”, fetch
  • if not, just skip
• If then at some point in time a 429 response comes up, the header “retry at” or response body “retry at” will be written to the flag file and is thus simply observed at the next “run”.

Then the integration can work as now, if it runs for days/weeks fine, the (last) “retry at” is (far) before “now” and it runs smoothly 😃

This won’t fix it, especially for users who have previously set up the integration but simply made too many requests. The problem lies in the updateCoordinator and the function that retrieves the data.

./Klaas

Not sure if it’s same error but this integration is not working anymore for some time now! This is in the logs:

Update: disabling the integration for at least an hour and then reenabling makes the integration work again. So indeed it seems it just can’t recover when API limit is hit.

Same here, blocked HA for one hour from internet, reconnected again…voila!

An idea to solve this is to adjust the update_interval to a time delta when receiving this exception, which after the reset restores the update_interval to the old situation upon a successful API call.

We have already done some testing with this, but ran into some problems with the coordinator and have not yet been able to figure it out / solve it.

https://github.com/home-assistant-libs/forecast_solar/blob/master/forecast_solar/exceptions.py shows that the ForecastSolarRatelimit exception being thrown includes reset_at

Looking at https://github.com/home-assistant/core/blob/dev/homeassistant/components/forecast_solar/coordinator.py around line 67, it needs to catch ForecastSolarRatelimit and adjust the time at which the next retry can be done.

Further looking at https://github.com/home-assistant/core/blob/dev/homeassistant/helpers/update_coordinator.py I can see that there is update_interval which controls the frequency of the polls, but next_refresh isn’t available so I can’t see how to make the update coordinator delay until the desired time - setting the update interval only affects the refresh after the next.