pyatv: After homeassistant restart pyatv/atv addon remote service no longer works.

Describe the bug After HA restart the remote service no longer works. I can not send commands and the remote service is set to off and I am unable to turn it on again until I reboot my apple tv.

To Reproduce

• Setup pyatv within homeassistant, confirm the remote service is working.
• Restart HA.
• See the remote entity change to off and when trying to turn it on again, it will go back to off
• restart ATV
• try turning on the remote entity (it could take a few seconds after reboot)
• everything should be working again.

Expected behavior Remote entity should turn back to on after restart of HA.

System Setup (please complete the following information):

• OS: HassOS 3.11
• Python: 3.7.6
• pyatv: 0.4.0a16

Additional info Logs:

2020-02-18 20:53:58 DEBUG (MainThread) [custom_components.apple_tv] Starting connect loop
2020-02-18 20:53:58 DEBUG (MainThread) [custom_components.apple_tv] Updating state: connected=False, disconnected=False
2020-02-18 20:54:00 DEBUG (MainThread) [custom_components.apple_tv] Updating state: connected=False, disconnected=False
2020-02-18 20:54:01 DEBUG (MainThread) [custom_components.apple_tv] Updating state: connected=True, disconnected=False
2020-02-18 20:54:01 DEBUG (MainThread) [custom_components.apple_tv] Changing address to 192.168.1.50
2020-02-18 20:54:01 DEBUG (MainThread) [custom_components.apple_tv] Connect loop ended
2020-02-18 20:54:51 DEBUG (MainThread) [custom_components.apple_tv] Not starting connect loop (False, True)
2020-02-18 20:59:11 DEBUG (MainThread) [custom_components.apple_tv] Not starting connect loop (False, True)
2020-02-18 21:01:07 DEBUG (MainThread) [custom_components.apple_tv] Not starting connect loop (False, True)