ha-solcast-solar: Integration failing to start after migrate from the old Solcast

The problem

Hello, after moved from the old Solcast integration i saw that it cannot actually start:

Image

Also in the logs provided bellow there is KeyError: 'pv_estimate10'

I tried to reinstall from scratch but didnt help.

What version of Solcast Integration has the issue?

v4.3.2

What version of Home Assistant are you using?

core-2025.4.1

Diagnostics information

2025-04-10 10:26:57.876 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] Task build_data took 0.002 seconds
2025-04-10 10:26:57.878 WARNING (MainThread) [custom_components.solcast_solar.solcastapi] Forecast data for 2025-04-10 contains 0 of 48 intervals, so is missing forecast data
2025-04-10 10:26:57.878 WARNING (MainThread) [custom_components.solcast_solar.solcastapi] Forecast data for 2025-04-11 contains 0 of 48 intervals, so is missing forecast data
2025-04-10 10:26:57.878 WARNING (MainThread) [custom_components.solcast_solar.solcastapi] Forecast data for 2025-04-12 contains 0 of 48 intervals, so is missing forecast data
2025-04-10 10:26:57.878 WARNING (MainThread) [custom_components.solcast_solar.solcastapi] Forecast data for 2025-04-13 contains 0 of 48 intervals, so is missing forecast data
2025-04-10 10:26:57.878 WARNING (MainThread) [custom_components.solcast_solar.solcastapi] Forecast data for 2025-04-14 contains 0 of 48 intervals, so is missing forecast data
2025-04-10 10:26:57.879 WARNING (MainThread) [custom_components.solcast_solar.solcastapi] Forecast data for 2025-04-15 contains 0 of 48 intervals, so is missing forecast data
2025-04-10 10:26:57.879 WARNING (MainThread) [custom_components.solcast_solar.solcastapi] Forecast data for 2025-04-16 contains 0 of 48 intervals, so is missing forecast data
2025-04-10 10:26:57.879 WARNING (MainThread) [custom_components.solcast_solar.solcastapi] Forecast data for 2025-04-17 contains 0 of 48 intervals, so is missing forecast data
2025-04-10 10:26:57.879 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] Task recalculate_splines took 0.000 seconds
2025-04-10 10:27:08.005 DEBUG (MainThread) [custom_components.solcast_solar] Auto-update options: {'auto_update': 1}
2025-04-10 10:27:08.005 DEBUG (MainThread) [custom_components.solcast_solar] Estimate to use options: {'key_estimate': 'estimate'}
2025-04-10 10:27:08.005 DEBUG (MainThread) [custom_components.solcast_solar] Attribute options: {'attr_brk_estimate': True, 'attr_brk_estimate10': True, 'attr_brk_estimate90': True, 'attr_brk_site': True, 'attr_brk_halfhourly': True, 'attr_brk_hourly': True, 'attr_brk_detailed': False}
2025-04-10 10:27:08.005 DEBUG (MainThread) [custom_components.solcast_solar] Custom sensor options: {'customhoursensor': 1}
2025-04-10 10:27:08.005 DEBUG (MainThread) [custom_components.solcast_solar] Hard limit: {'hard_limit_api': '100.0'}
2025-04-10 10:27:08.014 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] Configuration directory is /config
2025-04-10 10:27:08.014 DEBUG (MainThread) [custom_components.solcast_solar] Session headers: {'Accept': 'application/json', 'User-Agent': 'ha-solcast-solar-integration/4.3'}
2025-04-10 10:27:08.022 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] Sites cache exists for ******7IIvXE
2025-04-10 10:27:08.022 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] Connecting to https://api.solcast.com.au/rooftop_sites?format=json&api_key=******7IIvXE
2025-04-10 10:27:08.559 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] HTTP session returned status 200/Success
2025-04-10 10:27:08.560 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] Sites data: {'sites': [{'name': 'Home East', 'resource_id': '88f2-02da-9eb0-f5d2', 'capacity': 8, 'capacity_dc': 1.85, 'longitude': **.******, 'latitude': **.******, 'azimuth': 70, 'tilt': 21, 'install_date': '2022-10-21T21:00:00.0000000Z', 'loss_factor': 0.9, 'tags': ['sofia bulgaria', ' '], 'api_key': '******7IIvXE'}, {'name': 'Home South', 'resource_id': 'fa54-68a8-2e71-d1fa', 'capacity': 8, 'capacity_dc': 3.7, 'longitude': **.******, 'latitude': **.******, 'azimuth': 160, 'tilt': 15, 'install_date': '2022-10-22T21:00:00.0000000Z', 'loss_factor': 0.9, 'tags': [], 'api_key': '******7IIvXE'}], 'page_count': 1, 'current_page': 1, 'total_records': 2}
2025-04-10 10:27:08.560 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] Sites loaded
2025-04-10 10:27:08.560 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] Checking rekey for ******7IIvXE
2025-04-10 10:27:08.560 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] Writing sites cache for ******7IIvXE
2025-04-10 10:27:08.568 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] Usage cache exists for ******7IIvXE
2025-04-10 10:27:08.571 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] Usage cache for ******7IIvXE last reset 2025-04-10 03:00:00
2025-04-10 10:27:08.571 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] Usage loaded
2025-04-10 10:27:08.572 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] API counter for ******7IIvXE is 0/10
2025-04-10 10:27:08.598 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] Data cache /config/solcast.json exists, file type is <class 'dict'>
2025-04-10 10:27:08.598 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] Dampened data loaded
2025-04-10 10:27:08.610 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] Data cache /config/solcast-undampened.json exists, file type is <class 'dict'>
2025-04-10 10:27:08.610 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] Un-dampened data loaded
2025-04-10 10:27:08.615 ERROR (MainThread) [custom_components.solcast_solar.solcastapi] Exception in build_data(): 'pv_estimate10': Traceback (most recent call last):
  File "/config/custom_components/solcast_solar/solcastapi.py", line 2869, in build_data
    forecast["pv_estimate10"],
    ~~~~~~~~^^^^^^^^^^^^^^^^^
KeyError: 'pv_estimate10'

Logs debug/info/error output

2025-04-10 09:21:53.072 WARNING (MainThread) [custom_components.solcast_solar.solcastapi] Forecast data for 2025-04-10 contains 0 of 48 intervals, so is missing forecast data
2025-04-10 09:21:53.072 WARNING (MainThread) [custom_components.solcast_solar.solcastapi] Forecast data for 2025-04-11 contains 0 of 48 intervals, so is missing forecast data
2025-04-10 09:21:53.072 WARNING (MainThread) [custom_components.solcast_solar.solcastapi] Forecast data for 2025-04-12 contains 0 of 48 intervals, so is missing forecast data
2025-04-10 09:21:53.073 WARNING (MainThread) [custom_components.solcast_solar.solcastapi] Forecast data for 2025-04-13 contains 0 of 48 intervals, so is missing forecast data
2025-04-10 09:21:53.073 WARNING (MainThread) [custom_components.solcast_solar.solcastapi] Forecast data for 2025-04-14 contains 0 of 48 intervals, so is missing forecast data
2025-04-10 09:21:53.073 WARNING (MainThread) [custom_components.solcast_solar.solcastapi] Forecast data for 2025-04-15 contains 0 of 48 intervals, so is missing forecast data
2025-04-10 09:21:53.073 WARNING (MainThread) [custom_components.solcast_solar.solcastapi] Forecast data for 2025-04-16 contains 0 of 48 intervals, so is missing forecast data
2025-04-10 09:21:53.073 WARNING (MainThread) [custom_components.solcast_solar.solcastapi] Forecast data for 2025-04-17 contains 0 of 48 intervals, so is missing forecast data
2025-04-10 09:21:58.978 ERROR (MainThread) [custom_components.solcast_solar.solcastapi] Exception in build_data(): 'pv_estimate10': Traceback (most recent call last):
  File "/config/custom_components/solcast_solar/solcastapi.py", line 2869, in build_data
    forecast["pv_estimate10"],
    ~~~~~~~~^^^^^^^^^^^^^^^^^
KeyError: 'pv_estimate10'

Additional information

No response

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 39 (9 by maintainers)

Most upvoted comments

You sharing that code is important for other ancient Oziee users should they attempt migrating to BJ, @Stormhand. Thanks again.

The json structure has six keys in a dict. forecasts, detailedForecasts, siteinfo, last_updated, api_used and version. The forecasts list has the 50th percentile values (or presumably the selected estimate), and the detailedForecasts list has 10th, 50th and 90th percentile, presumably the sum of all rooftop sites, and then the siteinfo key has what we consider to be “modern day” solcast.json data, except in this case forecast history is not present. v3 didn’t do history.

I’ve not seen this structure before. It is clearly a ‘missing link’ release between v3 and v4 in terms of data format.

The migration fucked up when presented with this file, and I am yet to feed it into my dev instance to determine exactly how it fell drunk into a gutter. That it had a siteinfo key should have made the migration happy as a clam.

But that you’re out of API calls doesn’t tell me whether the integration would have actually recovered from this “failed” migration.

My dev instance doesn’t run out of API calls, but maybe I need to set it up to do so, and then attempt a migration from the solcast.json that you’ve shared. The weekend looms, and I need a good science project.

Really important stuff sharing this 🎁😘. Thanks again. Most of Ozzie’s legacy turns into a fun forensic dig.

Edit: (More for my own remembering…)

  • Theory: The migration did what it needed to do, but then history could not be downloaded because out of API calls (because new integration wants history), so fell drunk into gutter and crashed. Someone called cops.
  • Theory: Migration occurred, but got it horribly wrong and crashed. Unlikely.
  • Likely: Migration doesn’t clean up dict keys that are useless. Waste of memory and storage, and I/O to read/write the file.
  • Assertion: Solcast v3.0.x actually did eventually carry a version key in the dict, with v2 probably the first evolution after the original, which did not feature a version key. The migration can look for this and understand exactly the keys it needs to grab, and also exactly the actions that need to successfully follow (get history), and by extension can log EXACTLY the situation if this all falls down drunk into a gutter. This way the person calling the cops can make a super-accurate up-front statement.
  • Assertion: File version 3 is still a complete mystery. Never been seen. Did it exist, or was there a jump to v4 for Solcast v4.0.x? If someone has a solcast.json v3 then I really want it…

deleted both files and restarted. This is in the log now:

2025-04-10 12:35:23.133 WARNING (MainThread) [custom_components.solcast_solar.solcastapi] There is no solcast.json to load, so fetching solar forecast, including past forecasts
2025-04-10 12:35:23.134 WARNING (MainThread) [custom_components.solcast_solar.solcastapi] API polling limit exhausted, not getting forecast for site 88f2-02da-9eb0-f5d2, API used is 10/10
2025-04-10 12:35:23.134 ERROR (MainThread) [custom_components.solcast_solar.solcastapi] No valid data was returned for estimated_actuals so this will cause issues (API limit may be exhausted, or Solcast might have a problem)
2025-04-10 12:35:23.134 ERROR (MainThread) [custom_components.solcast_solar.solcastapi] API did not return a json object, returned `None`

2025-04-10 12:35:23.135 WARNING (MainThread) [custom_components.solcast_solar.solcastapi] Forecast update for site 88f2-02da-9eb0-f5d2 failed so not getting remaining sites
2025-04-10 12:35:23.135 WARNING (MainThread) [custom_components.solcast_solar.solcastapi] Forecast has not been updated
2025-04-10 12:35:23.136 WARNING (MainThread) [custom_components.solcast_solar.solcastapi] Forecast data for 2025-04-10 contains 0 of 48 intervals, so is missing forecast data
2025-04-10 12:35:23.137 WARNING (MainThread) [custom_components.solcast_solar.solcastapi] Forecast data for 2025-04-11 contains 0 of 48 intervals, so is missing forecast data
2025-04-10 12:35:23.137 WARNING (MainThread) [custom_components.solcast_solar.solcastapi] Forecast data for 2025-04-12 contains 0 of 48 intervals, so is missing forecast data
2025-04-10 12:35:23.137 WARNING (MainThread) [custom_components.solcast_solar.solcastapi] Forecast data for 2025-04-13 contains 0 of 48 intervals, so is missing forecast data
2025-04-10 12:35:23.137 WARNING (MainThread) [custom_components.solcast_solar.solcastapi] Forecast data for 2025-04-14 contains 0 of 48 intervals, so is missing forecast data
2025-04-10 12:35:23.137 WARNING (MainThread) [custom_components.solcast_solar.solcastapi] Forecast data for 2025-04-15 contains 0 of 48 intervals, so is missing forecast data
2025-04-10 12:35:23.138 WARNING (MainThread) [custom_components.solcast_solar.solcastapi] Forecast data for 2025-04-16 contains 0 of 48 intervals, so is missing forecast data
2025-04-10 12:35:23.138 WARNING (MainThread) [custom_components.solcast_solar.solcastapi] Forecast data for 2025-04-17 contains 0 of 48 intervals, so is missing forecast data

and here the solcast_solar archive solcast_solar.zip

@autoSteve will tell you, please can you turn debug on for the solcast integration - instructions in the issue log template

Otherwise very difficult to identify the issue

To put it another way, the only time you need to set the hard limit is when you have far more panel capacity than inverter capacity and Solcast is producing forecasts that are too high.

If your Solcast config is set up correctly you should not need to mess with a hard limit. Review the readme on that front, but in short a hard limit is only required for circumstances where it is not possible to set up the Solcast rooftop sites effectively.

This can happen when PV strings are “over-sized”. Example, there’s 14kW of generation power possible from the panel strings on the roof (two strings, different azimuth for each, 7kW + 7kW) yet the one inverter will stop at 8kW. With the sun in the right place in the sky it’s possible that more than 8kW could be produced by the panels and the inverter will “clip” the output. If that’s your situation then you’d see the sensor for inverter output stop at maximum, yet the forecast from Solcast would exceed that 8kW.

Make sense?

Glad things worked out for you.

No worries, im happy its working now 😄 Talking for thread-jacking… i didnt fully understand the hard limit. My inverter is Solaredge 8kw. Should i set 8 instead of the default 100?

Maybe it was v3.1.2 that was too powerful?

That commit sees the config migration from v3 to v4.

But recreating ancient history that has been nuked from orbit is tricky…

The history at that point is most murky, @BJReplay, and the commits are even harder to come by.

v3.0.47 has solcast.json v2. There are no changelog mentions of that transition prior. Then Ozzie mentions an unreleased 3.0.55./v3.1.x, then (somewhat grumpily) v4 is born with v4 of solcast.json. I suspect that a v3 of solcast.json never made it into the wild.

I am hopeful that a v3 never made it into the wild… but who knows? 🤷

Ozzie’s v3 was chaotic, but that settled somewhat in v4, where there is a bloody good history of the changes. As you recall, v4 was actually called a v3 branch, which you renamed to main, but I don’t recall that branch ever carrying v3 Git history.

Don’t forget to change the 50 setting back to 10…

Sadly you can’t see your limit at Solcast.com.au any more. The Solcast folks got rid of an API call, so we’re blind. Looking more carefully at the logged messages (the json returning None is the give-away) that is definitely an, “API says no” set of messages, so you do have just ten calls.

Early adopters got 50. BJ has 50. Newer users got just 10. I have 10, just missing out…

For development you’d think that would suck, right? It used to. Then I built an API simulator that can give me unlimited calls. I giggle every time this reaches the 4000 call mark in a day.

1,118 calls so far for today. Soft. I had the laptop off because day job. Dev is updating forecast every 43 seconds at the moment… 😂

2025-04-10 10:01:40.062 INFO Getting forecast update for site 2222-2222-2222-2222
2025-04-10 10:01:40.062 DEBUG Polling API for site 2222-2222-2222-2222, last day 2025-04-17, 172 hours
2025-04-10 10:01:40.062 DEBUG Fetching forecast
2025-04-10 10:01:40.082 DEBUG Fetch data url https://api.solcast.com.au/rooftop_sites/2222-2222-2222-2222/forecasts?format=json&api_key=******1&hours=172
2025-04-10 10:01:40.082 DEBUG API returned data, API counter incremented from 1117 to 1118
2025-04-10 10:01:40.082 DEBUG Writing API usage cache file: /workspaces/homeAssistant-core/config/solcast-usage-******1.json
2025-04-10 10:01:40.084 DEBUG Task fetch_data took 0.022 seconds
2025-04-10 10:01:40.084 DEBUG 345 records returned
2025-04-10 10:01:40.086 DEBUG Task load_new_data took 0.001 seconds
2025-04-10 10:01:40.087 DEBUG Task apply_dampening took 0.001 seconds
2025-04-10 10:01:40.108 DEBUG Task sort_and_prune took 0.021 seconds
2025-04-10 10:01:40.108 DEBUG Forecasts dictionary length 4513 (1058 un-dampened)

Thanks for the help, i disabled it for now. How can i see my limit? My account is 3 years old but even changing the config to 50 requests didnt help.

P.S. Wonderful job on this integration! Just read the documentation and it looks fantastic with so many new features 😃

solcast.json okey, i extracted solcast.json from yesterday(before migration). The key was like @autoSteve said - to download the backup, not copy it from the NAS 😃

Interesting. Also missing SiteInfo at the start of solcast.json.

Do you happen to have automatic backups? If so, it would be interesting to see the solcast files from the backup.

Do you happen to know what version you upgraded from?

@autoSteve failed upgrade? api_used at the tail of the file? No SiteInfo at the head?