core: Enphase Envoy - Timeouts reading large number of inverters

Home Assistant release with the issue: 0.103.5

Last working Home Assistant release (if known): N/A - Hasn’t worked since at least .98 when I first started getting things running

Operating environment (Hass.io/Docker/Windows/etc.): hass.io running inside an Ubuntu VM on a large multi-Xeon vmWare server

Integration: https://www.home-assistant.io/integrations/enphase_envoy/

Description of problem: I have an Enphase Envoy solar system, controlled by an “S” model controller. While summary production numbers are being produced, individual inverter statistics are stuck as “Unknown”. My Home Assistant logs are filling up with two relevant errors:

2020-01-02 18:57:33 ERROR (MainThread) [homeassistant.helpers.entity] Update for sensor.envoy_solar_controllerenvoy_inverter_121731044406 fails
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 281, in async_update_ha_state
    await self.async_device_update()
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 459, in async_device_update
    await self.async_update()
  File "/usr/src/homeassistant/homeassistant/components/enphase_envoy/sensor.py", line 133, in async_update
    self._state = inverters[serial_number]
KeyError: 'ControllerEnvoy'

and

2020-01-02 18:53:01 WARNING (MainThread) [homeassistant.helpers.entity] Update of sensor.envoy_solar_controllerenvoy_inverter_121731037388 is taking over 10 seconds

I just picked an example of each type of error. I have 24 panels and microinverters so they all are generating the same 2 error types.

When I use another linux machine to manually run envoy_reader (the underpinning of this integration) downloaded from GitHub I receive the following output:

Enter the Envoy IP address or host name, or press enter to use 'envoy' as default: 
Enter the Username for Inverter data authentication, or press enter to use 'envoy' as default: 
Enter the Password for Inverter data authentication, or press enter to use the default password: 
Reading...
production:              3863
consumption:             3873
daily_production:        0
daily_consumption:       0
seven_days_production:   0
seven_days_consumption:  0
lifetime_production:     0
lifetime_consumption:    -3
inverters_production:   {'121731037388': [158, '2020-01-02 19:59:28'], '121731038085': [151, '2020-01-02 19:59:33'], '121731023211': [156, '2020-01-02 19:59:25'], '121731023758': [154, '2020-01-02 19:59:29'], '121731044407': [159, '2020-01-02 19:59:36'], '121731044406': [157, '2020-01-02 19:59:10'], '121731037503': [154, '2020-01-02 19:59:07'], '121731044863': [162, '2020-01-02 19:59:14'], '121731045676': [156, '2020-01-02 19:59:20'], '121731040860': [156, '2020-01-02 19:59:10'], '121731032959': [157, '2020-01-02 19:59:03'], '121731044153': [155, '2020-01-02 19:59:23'], '121731044143': [157, '2020-01-02 19:59:21'], '121731032724': [153, '2020-01-02 19:59:18'], '121731040984': [160, '2020-01-02 19:59:16'], '121731037408': [157, '2020-01-02 19:59:15'], '121731023280': [152, '2020-01-02 19:59:11'], '121731029397': [157, '2020-01-02 19:59:05'], '121731023151': [159, '2020-01-02 19:59:25'], '121731044656': [153, '2020-01-02 19:59:33'], '121731023488': [158, '2020-01-02 19:59:37'], '121731045122': [159, '2020-01-02 19:59:28'], '121731045846': [163, '2020-01-02 19:59:34'], '121731044648': [158, '2020-01-02 19:59:05']}

Clearly the inverters are returning data thru the local API. But for some reason the HA integration isn’t properly ingesting the data.

Problem-relevant configuration.yaml entries:

# Solar System Interface
sensor:
  - platform: enphase_envoy
    ip_address: 192.168.71.65
    name: Envoy Solar Controller
    monitored_conditions:
      - production
      - inverters

Additional information: Willing to test out and help debug as needed. My Python skills are rudimentary so I am not sure where things are borked.