ha-solcast-solar: problem with migration from Ozzie's version

The problem

Hi, I’ve just installed the latest version 4.2.3 by following the recommended HACS recommended on https://github.com/BJReplay/ha-solcast-solar over the top of 2nd to last version that ozzie published to HACS (which was working perfectly but I was liking the new features coming out)

Looking at the logs; solcast returns 429/try again later; but what I don’t understand is when I enter

curl https://api.solcast.com.au/rooftop_sites\?format\=json\&api_key\=\*\*\*\*\*\*Hbie_n (with correct api_key I get the sites JSON data) from the home assistant terminal it works; and I see the site json file; similarly from another webbrowser on another IP address I also gets the sites file.

Is it possible that solcast for home assistant is getting 429 from the solcast api; but the home assistant terminal isn’t?

What version of Solcast Integration has the issue?

v4.2.3

What version of Home Assistant are you using?

2024.10.3

Diagnostics information

No response

Logs debug/info/error output

2024-10-21 20:54:24.661 DEBUG (MainThread) [custom_components.solcast_solar] Auto-update options: {'auto_update': 0}
2024-10-21 20:54:24.661 DEBUG (MainThread) [custom_components.solcast_solar] Estimate to use options: {'key_estimate': 'estimate'}
2024-10-21 20:54:24.661 DEBUG (MainThread) [custom_components.solcast_solar] Attribute options: {'attr_brk_estimate': True, 'attr_brk_estimate10': True, 'attr_brk_estimate90': True, 'attr_brk_site': True, 'attr_brk_halfhourly': True, 'attr_brk_hourly': True, 'attr_brk_detailed': False}
2024-10-21 20:54:24.661 DEBUG (MainThread) [custom_components.solcast_solar] Custom sensor options: {'customhoursensor': 1}
2024-10-21 20:54:24.661 DEBUG (MainThread) [custom_components.solcast_solar] Hard limit: {'hard_limit': 100000}
2024-10-21 20:54:24.661 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] Configuration directory is /config
2024-10-21 20:54:24.662 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] Sites cache does not yet exist
2024-10-21 20:54:24.662 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] Connecting to https://api.solcast.com.au/rooftop_sites?format=json&api_key=******Hbie_n
2024-10-21 20:54:24.674 WARNING (MainThread) [custom_components.solcast_solar.solcastapi] HTTP session returned status 429/Try again later in __sites_data(), trying cache
2024-10-21 20:54:24.674 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] Will retry get sites, retry 1
2024-10-21 20:54:29.698 WARNING (MainThread) [custom_components.solcast_solar.solcastapi] HTTP session returned status 429/Try again later in __sites_data(), trying cache
2024-10-21 20:54:29.699 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] Will retry get sites, retry 2
2024-10-21 20:54:34.712 WARNING (MainThread) [custom_components.solcast_solar.solcastapi] HTTP session returned status 429/Try again later in __sites_data(), trying cache
2024-10-21 20:54:34.712 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] Will retry get sites, retry 3
2024-10-21 20:54:39.729 WARNING (MainThread) [custom_components.solcast_solar.solcastapi] HTTP session returned status 429/Try again later in __sites_data(), trying cache
2024-10-21 20:54:39.729 WARNING (MainThread) [custom_components.solcast_solar.solcastapi] Retries exhausted gathering sites, last call result: 429/Try again later, using cached data if it exists
2024-10-21 20:54:39.729 ERROR (MainThread) [custom_components.solcast_solar.solcastapi] Cached sites are not yet available for ******Hbie_n to cope with API call failure
2024-10-21 20:54:39.729 ERROR (MainThread) [custom_components.solcast_solar.solcastapi] At least one successful API 'get sites' call is needed, so the integration will not function correctly
2024-10-21 20:54:39.729 ERROR (MainThread) [custom_components.solcast_solar.solcastapi] https://api.solcast.com.au HTTP status error 404/Not found in __sites_data() while gathering sites
2024-10-21 20:54:39.730 ERROR (MainThread) [custom_components.solcast_solar.solcastapi] Exception in __sites_data(): HTTP __sites_data() error: gathering sites: Traceback (most recent call last):
  File "/config/custom_components/solcast_solar/solcastapi.py", line 542, in __sites_data
    raise Exception("HTTP __sites_data() error: gathering sites")
Exception: HTTP __sites_data() error: gathering sites

Additional information

No response

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 30 (6 by maintainers)

Most upvoted comments

heh 😃 funny story; chatgpt literally just told me to put the same code in

I simplified the user agent to Mozilla/5.0. Works. Reverted. No works.

Deffo a v4.2.4 ‘emergency’ bump. I will stash the prevailing v4.2.4 multi-hard limit proposal and work this @BJReplay.

Sooo… @BJReplay. Add this header to all Solcast calls? Push as v4.2.4? Solcast must have pushed a change.

I’m thinking headers as well (but I’m not across aiohttp) - just having a Quick Look at the code but will need to head off soon for the night (hello from Hobart BTW)

From what I’ve had a look at our logs, most of the API calls on the 21st UTC, were between 6am and 12am (UTC), which is the time where most of the 429’s are coming from.

It varies day to day, but it looks like the 6am API calls (Europe) and 12am (Americas), are the majority of API calls, which makes sense given it’s about dawn in the respective timezones, but it also means that those are the periods where the majority of 429’s are likely to occur.

The header stuff is interesting, because for the logs, I saw it work in both instances throughout the day. So I can’t say for sure what would have allowed some request to go through and others not.

I’ve replied to @billy-solcast with my site details via support@solcast

Hey all,

Adding some context here. We don’t do any specific limiting/restricting on headers, as all of the work is done by the WAF. So it could be entirely out of coincidence.

@niggles1973 if you want to send me (support@solcast.com) your toolkit account details, I can have a look at our logs and trace back from there.

Pleasing per-site/API key hard limit calculation times on forecast fetch. Ahh… Thought it would be quicker given the optimisation, but it is what it is. Fast.

2024-10-21 23:06:41.395 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] HTTP data call processing took 0.065 seconds
2024-10-21 23:06:41.396 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] Hard limit for individual API keys: True
2024-10-21 23:06:41.397 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] Hard limit for API key ******sxdNS5: 5.0
2024-10-21 23:06:41.399 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] Earliest period: 2024-10-21 00:00:00, latest period: 2024-10-28 23:30:00
2024-10-21 23:06:41.405 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] Hard limit for API key ******bvvHxW: 5.0
2024-10-21 23:06:41.408 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] Earliest period: 2024-10-21 00:00:00, latest period: 2024-10-28 23:30:00
2024-10-21 23:06:41.413 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] Build hard limit processing took 0.017 seconds for dampened
2024-10-21 23:06:41.475 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] Build per-site and total processing took 0.025 seconds for dampened
2024-10-21 23:06:41.482 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] Hard limit for API key ******sxdNS5: 5.0
2024-10-21 23:06:41.484 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] Earliest period: 2024-10-21 00:00:00, latest period: 2024-10-28 23:30:00
2024-10-21 23:06:41.490 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] Hard limit for API key ******bvvHxW: 5.0
2024-10-21 23:06:41.492 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] Earliest period: 2024-10-21 00:00:00, latest period: 2024-10-28 23:30:00
2024-10-21 23:06:41.519 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] Build hard limit processing took 0.037 seconds for un-dampened
2024-10-21 23:06:41.695 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] Build per-site and total processing took 0.039 seconds for un-dampened
2024-10-21 23:06:41.710 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] Forecast data from 2024-10-21 to 2024-10-28 contains all 48 intervals
2024-10-21 23:06:41.710 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] Calculating splines

PR for multi-API-key hard limit concurs. 429s resolved. All 200s, baby. Specific request headers for the win with aiohttp.

I am off and developing again…

I won’t know until I rebase on the proposed 4.2.4 code, but I am tipping four hours of 429s will melt away. (Maybe we should have called it v4.2.9… 🤣)

Welcome aboard, @niggles1973, BTW!