ha-solcast-solar: Integration failing to start after migrate from the old Solcast
The problem
Hello, after moved from the old Solcast integration i saw that it cannot actually start:
Also in the logs provided bellow there is KeyError: 'pv_estimate10'
I tried to reinstall from scratch but didnt help.
What version of Solcast Integration has the issue?
v4.3.2
What version of Home Assistant are you using?
core-2025.4.1
Diagnostics information
2025-04-10 10:26:57.876 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] Task build_data took 0.002 seconds
2025-04-10 10:26:57.878 WARNING (MainThread) [custom_components.solcast_solar.solcastapi] Forecast data for 2025-04-10 contains 0 of 48 intervals, so is missing forecast data
2025-04-10 10:26:57.878 WARNING (MainThread) [custom_components.solcast_solar.solcastapi] Forecast data for 2025-04-11 contains 0 of 48 intervals, so is missing forecast data
2025-04-10 10:26:57.878 WARNING (MainThread) [custom_components.solcast_solar.solcastapi] Forecast data for 2025-04-12 contains 0 of 48 intervals, so is missing forecast data
2025-04-10 10:26:57.878 WARNING (MainThread) [custom_components.solcast_solar.solcastapi] Forecast data for 2025-04-13 contains 0 of 48 intervals, so is missing forecast data
2025-04-10 10:26:57.878 WARNING (MainThread) [custom_components.solcast_solar.solcastapi] Forecast data for 2025-04-14 contains 0 of 48 intervals, so is missing forecast data
2025-04-10 10:26:57.879 WARNING (MainThread) [custom_components.solcast_solar.solcastapi] Forecast data for 2025-04-15 contains 0 of 48 intervals, so is missing forecast data
2025-04-10 10:26:57.879 WARNING (MainThread) [custom_components.solcast_solar.solcastapi] Forecast data for 2025-04-16 contains 0 of 48 intervals, so is missing forecast data
2025-04-10 10:26:57.879 WARNING (MainThread) [custom_components.solcast_solar.solcastapi] Forecast data for 2025-04-17 contains 0 of 48 intervals, so is missing forecast data
2025-04-10 10:26:57.879 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] Task recalculate_splines took 0.000 seconds
2025-04-10 10:27:08.005 DEBUG (MainThread) [custom_components.solcast_solar] Auto-update options: {'auto_update': 1}
2025-04-10 10:27:08.005 DEBUG (MainThread) [custom_components.solcast_solar] Estimate to use options: {'key_estimate': 'estimate'}
2025-04-10 10:27:08.005 DEBUG (MainThread) [custom_components.solcast_solar] Attribute options: {'attr_brk_estimate': True, 'attr_brk_estimate10': True, 'attr_brk_estimate90': True, 'attr_brk_site': True, 'attr_brk_halfhourly': True, 'attr_brk_hourly': True, 'attr_brk_detailed': False}
2025-04-10 10:27:08.005 DEBUG (MainThread) [custom_components.solcast_solar] Custom sensor options: {'customhoursensor': 1}
2025-04-10 10:27:08.005 DEBUG (MainThread) [custom_components.solcast_solar] Hard limit: {'hard_limit_api': '100.0'}
2025-04-10 10:27:08.014 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] Configuration directory is /config
2025-04-10 10:27:08.014 DEBUG (MainThread) [custom_components.solcast_solar] Session headers: {'Accept': 'application/json', 'User-Agent': 'ha-solcast-solar-integration/4.3'}
2025-04-10 10:27:08.022 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] Sites cache exists for ******7IIvXE
2025-04-10 10:27:08.022 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] Connecting to https://api.solcast.com.au/rooftop_sites?format=json&api_key=******7IIvXE
2025-04-10 10:27:08.559 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] HTTP session returned status 200/Success
2025-04-10 10:27:08.560 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] Sites data: {'sites': [{'name': 'Home East', 'resource_id': '88f2-02da-9eb0-f5d2', 'capacity': 8, 'capacity_dc': 1.85, 'longitude': **.******, 'latitude': **.******, 'azimuth': 70, 'tilt': 21, 'install_date': '2022-10-21T21:00:00.0000000Z', 'loss_factor': 0.9, 'tags': ['sofia bulgaria', ' '], 'api_key': '******7IIvXE'}, {'name': 'Home South', 'resource_id': 'fa54-68a8-2e71-d1fa', 'capacity': 8, 'capacity_dc': 3.7, 'longitude': **.******, 'latitude': **.******, 'azimuth': 160, 'tilt': 15, 'install_date': '2022-10-22T21:00:00.0000000Z', 'loss_factor': 0.9, 'tags': [], 'api_key': '******7IIvXE'}], 'page_count': 1, 'current_page': 1, 'total_records': 2}
2025-04-10 10:27:08.560 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] Sites loaded
2025-04-10 10:27:08.560 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] Checking rekey for ******7IIvXE
2025-04-10 10:27:08.560 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] Writing sites cache for ******7IIvXE
2025-04-10 10:27:08.568 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] Usage cache exists for ******7IIvXE
2025-04-10 10:27:08.571 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] Usage cache for ******7IIvXE last reset 2025-04-10 03:00:00
2025-04-10 10:27:08.571 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] Usage loaded
2025-04-10 10:27:08.572 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] API counter for ******7IIvXE is 0/10
2025-04-10 10:27:08.598 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] Data cache /config/solcast.json exists, file type is <class 'dict'>
2025-04-10 10:27:08.598 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] Dampened data loaded
2025-04-10 10:27:08.610 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] Data cache /config/solcast-undampened.json exists, file type is <class 'dict'>
2025-04-10 10:27:08.610 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] Un-dampened data loaded
2025-04-10 10:27:08.615 ERROR (MainThread) [custom_components.solcast_solar.solcastapi] Exception in build_data(): 'pv_estimate10': Traceback (most recent call last):
File "/config/custom_components/solcast_solar/solcastapi.py", line 2869, in build_data
forecast["pv_estimate10"],
~~~~~~~~^^^^^^^^^^^^^^^^^
KeyError: 'pv_estimate10'
Logs debug/info/error output
2025-04-10 09:21:53.072 WARNING (MainThread) [custom_components.solcast_solar.solcastapi] Forecast data for 2025-04-10 contains 0 of 48 intervals, so is missing forecast data
2025-04-10 09:21:53.072 WARNING (MainThread) [custom_components.solcast_solar.solcastapi] Forecast data for 2025-04-11 contains 0 of 48 intervals, so is missing forecast data
2025-04-10 09:21:53.072 WARNING (MainThread) [custom_components.solcast_solar.solcastapi] Forecast data for 2025-04-12 contains 0 of 48 intervals, so is missing forecast data
2025-04-10 09:21:53.073 WARNING (MainThread) [custom_components.solcast_solar.solcastapi] Forecast data for 2025-04-13 contains 0 of 48 intervals, so is missing forecast data
2025-04-10 09:21:53.073 WARNING (MainThread) [custom_components.solcast_solar.solcastapi] Forecast data for 2025-04-14 contains 0 of 48 intervals, so is missing forecast data
2025-04-10 09:21:53.073 WARNING (MainThread) [custom_components.solcast_solar.solcastapi] Forecast data for 2025-04-15 contains 0 of 48 intervals, so is missing forecast data
2025-04-10 09:21:53.073 WARNING (MainThread) [custom_components.solcast_solar.solcastapi] Forecast data for 2025-04-16 contains 0 of 48 intervals, so is missing forecast data
2025-04-10 09:21:53.073 WARNING (MainThread) [custom_components.solcast_solar.solcastapi] Forecast data for 2025-04-17 contains 0 of 48 intervals, so is missing forecast data
2025-04-10 09:21:58.978 ERROR (MainThread) [custom_components.solcast_solar.solcastapi] Exception in build_data(): 'pv_estimate10': Traceback (most recent call last):
File "/config/custom_components/solcast_solar/solcastapi.py", line 2869, in build_data
forecast["pv_estimate10"],
~~~~~~~~^^^^^^^^^^^^^^^^^
KeyError: 'pv_estimate10'
Additional information
No response
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 39 (9 by maintainers)
You sharing that code is important for other ancient Oziee users should they attempt migrating to BJ, @Stormhand. Thanks again.
The json structure has six keys in a
dict.forecasts,detailedForecasts,siteinfo,last_updated,api_usedandversion. Theforecastslist has the 50th percentile values (or presumably the selected estimate), and thedetailedForecastslist has 10th, 50th and 90th percentile, presumably the sum of all rooftop sites, and then thesiteinfokey has what we consider to be “modern day”solcast.jsondata, except in this case forecast history is not present. v3 didn’t do history.I’ve not seen this structure before. It is clearly a ‘missing link’ release between v3 and v4 in terms of data format.
The migration fucked up when presented with this file, and I am yet to feed it into my dev instance to determine exactly how it fell drunk into a gutter. That it had a
siteinfokey should have made the migration happy as a clam.But that you’re out of API calls doesn’t tell me whether the integration would have actually recovered from this “failed” migration.
My dev instance doesn’t run out of API calls, but maybe I need to set it up to do so, and then attempt a migration from the
solcast.jsonthat you’ve shared. The weekend looms, and I need a good science project.Really important stuff sharing this 🎁😘. Thanks again. Most of Ozzie’s legacy turns into a fun forensic dig.
Edit: (More for my own remembering…)
dictkeys that are useless. Waste of memory and storage, and I/O to read/write the file.versionkey in thedict, with v2 probably the first evolution after the original, which did not feature aversionkey. The migration can look for this and understand exactly the keys it needs to grab, and also exactly the actions that need to successfully follow (get history), and by extension can log EXACTLY the situation if this all falls down drunk into a gutter. This way the person calling the cops can make a super-accurate up-front statement.solcast.jsonv3 then I really want it…deleted both files and restarted. This is in the log now:
and here the solcast_solar archive solcast_solar.zip
@autoSteve will tell you, please can you turn debug on for the solcast integration - instructions in the issue log template
Otherwise very difficult to identify the issue
To put it another way, the only time you need to set the hard limit is when you have far more panel capacity than inverter capacity and Solcast is producing forecasts that are too high.
If your Solcast config is set up correctly you should not need to mess with a hard limit. Review the readme on that front, but in short a hard limit is only required for circumstances where it is not possible to set up the Solcast rooftop sites effectively.
This can happen when PV strings are “over-sized”. Example, there’s 14kW of generation power possible from the panel strings on the roof (two strings, different azimuth for each, 7kW + 7kW) yet the one inverter will stop at 8kW. With the sun in the right place in the sky it’s possible that more than 8kW could be produced by the panels and the inverter will “clip” the output. If that’s your situation then you’d see the sensor for inverter output stop at maximum, yet the forecast from Solcast would exceed that 8kW.
Make sense?
Glad things worked out for you.
No worries, im happy its working now 😄 Talking for thread-jacking… i didnt fully understand the hard limit. My inverter is Solaredge 8kw. Should i set 8 instead of the default 100?
Maybe it was v3.1.2 that was too powerful?
That commit sees the config migration from v3 to v4.
But recreating ancient history that has been nuked from orbit is tricky…
The history at that point is most murky, @BJReplay, and the commits are even harder to come by.
v3.0.47 has
solcast.jsonv2. There are no changelog mentions of that transition prior. Then Ozzie mentions an unreleased 3.0.55./v3.1.x, then (somewhat grumpily) v4 is born with v4 ofsolcast.json. I suspect that a v3 ofsolcast.jsonnever made it into the wild.I am hopeful that a v3 never made it into the wild… but who knows? 🤷
Ozzie’s v3 was chaotic, but that settled somewhat in v4, where there is a bloody good history of the changes. As you recall, v4 was actually called a v3 branch, which you renamed to
main, but I don’t recall that branch ever carrying v3 Git history.Don’t forget to change the
50setting back to10…Sadly you can’t see your limit at Solcast.com.au any more. The Solcast folks got rid of an API call, so we’re blind. Looking more carefully at the logged messages (the
jsonreturningNoneis the give-away) that is definitely an, “API says no” set of messages, so you do have just ten calls.Early adopters got 50. BJ has 50. Newer users got just 10. I have 10, just missing out…
For development you’d think that would suck, right? It used to. Then I built an API simulator that can give me unlimited calls. I giggle every time this reaches the 4000 call mark in a day.
1,118 calls so far for today. Soft. I had the laptop off because day job. Dev is updating forecast every 43 seconds at the moment… 😂
Thanks for the help, i disabled it for now. How can i see my limit? My account is 3 years old but even changing the config to 50 requests didnt help.
P.S. Wonderful job on this integration! Just read the documentation and it looks fantastic with so many new features 😃
solcast.json okey, i extracted solcast.json from yesterday(before migration). The key was like @autoSteve said - to download the backup, not copy it from the NAS 😃
Interesting. Also missing SiteInfo at the start of solcast.json.
Do you happen to have automatic backups? If so, it would be interesting to see the solcast files from the backup.
Do you happen to know what version you upgraded from?
@autoSteve failed upgrade? api_used at the tail of the file? No SiteInfo at the head?