core: iAquaLink integration goes offline with SSL error

The problem

The iAquaLink integration keeps going offline. The logs show it’s complaining about an SSL error.

Environment

  • Home Assistant Core release with the issue: 0.114.3
  • Last working Home Assistant Core release (if known): It MAY have been working on 0.114.2
  • Operating environment (OS/Container/Supervised/Core): HassOS 4.12 / HA 0.114.3
  • Integration causing this issue: iAquaLink
  • Link to integration documentation on our website: https://www.home-assistant.io/integrations/iaqualink/

Problem-relevant configuration.yaml

Integrated through the configuration UI – no additional yaml provided.

Traceback/Error logs

It started with this error:

2020-08-24 19:23:02 WARNING (MainThread) [iaqualink] <- 504 Gateway Time-out - https://iaqualink-api.realtime.io/v1/mobile/session.json?actionID=command&command=get_home&serial=[REDACTED]&sessionID=[REDACTED]
2020-08-24 19:23:03 ERROR (MainThread) [iaqualink] Unhandled exception: 0, message='Attempt to decode JSON with unexpected mimetype: text/html', url=URL('https://iaqualink-api.realtime.io/v1/mobile/session.json?actionID=command&command=get_home&serial=[REDACTED]&sessionID=[REDACTED]')
2020-08-24 19:23:03 ERROR (MainThread) [iaqualink] Traceback (most recent call last):
2020-08-24 19:23:03 ERROR (MainThread) [iaqualink]   File "/usr/local/lib/python3.8/site-packages/iaqualink/system.py", line 77, in update
2020-08-24 19:23:03 ERROR (MainThread) [iaqualink]     await self._parse_home_response(r1)
2020-08-24 19:23:03 ERROR (MainThread) [iaqualink]   File "/usr/local/lib/python3.8/site-packages/iaqualink/system.py", line 96, in _parse_home_response
2020-08-24 19:23:03 ERROR (MainThread) [iaqualink]     data = await response.json()
2020-08-24 19:23:03 ERROR (MainThread) [iaqualink]   File "/usr/local/lib/python3.8/site-packages/aiohttp/client_reqrep.py", line 1026, in json
2020-08-24 19:23:03 ERROR (MainThread) [iaqualink]     raise ContentTypeError(
2020-08-24 19:23:03 ERROR (MainThread) [iaqualink] aiohttp.client_exceptions.ContentTypeError: 0, message='Attempt to decode JSON with unexpected mimetype: text/html', url=URL('https://iaqualink-api.realtime.io/v1/mobile/session.json?actionID=command&command=get_home&serial=[REDACTED]&sessionID=[REDACTED]')
2020-08-24 19:23:03 ERROR (MainThread) [iaqualink] 
2020-08-24 19:23:03 WARNING (MainThread) [homeassistant.components.iaqualink] Failed to refresh iAqualink state

But then I repeatedly got THIS error (this is the most recent one of the dozens in the log over the past 24 hours):

2020-08-26 08:08:38 ERROR (MainThread) [iaqualink] Unhandled exception: Cannot connect to host iaqualink-api.realtime.io:443 ssl:default [[SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:1108)]
2020-08-26 08:08:38 ERROR (MainThread) [iaqualink] Traceback (most recent call last):
2020-08-26 08:08:38 ERROR (MainThread) [iaqualink]   File "/usr/local/lib/python3.8/site-packages/aiohttp/connector.py", line 936, in _wrap_create_connection
2020-08-26 08:08:38 ERROR (MainThread) [iaqualink]     return await self._loop.create_connection(*args, **kwargs)  # type: ignore  # noqa
2020-08-26 08:08:38 ERROR (MainThread) [iaqualink]   File "/usr/local/lib/python3.8/asyncio/base_events.py", line 1050, in create_connection
2020-08-26 08:08:38 ERROR (MainThread) [iaqualink]     transport, protocol = await self._create_connection_transport(
2020-08-26 08:08:38 ERROR (MainThread) [iaqualink]   File "/usr/local/lib/python3.8/asyncio/base_events.py", line 1080, in _create_connection_transport
2020-08-26 08:08:38 ERROR (MainThread) [iaqualink]     await waiter
2020-08-26 08:08:38 ERROR (MainThread) [iaqualink]   File "/usr/local/lib/python3.8/asyncio/sslproto.py", line 529, in data_received
2020-08-26 08:08:38 ERROR (MainThread) [iaqualink]     ssldata, appdata = self._sslpipe.feed_ssldata(data)
2020-08-26 08:08:38 ERROR (MainThread) [iaqualink]   File "/usr/local/lib/python3.8/asyncio/sslproto.py", line 189, in feed_ssldata
2020-08-26 08:08:38 ERROR (MainThread) [iaqualink]     self._sslobj.do_handshake()
2020-08-26 08:08:38 ERROR (MainThread) [iaqualink]   File "/usr/local/lib/python3.8/ssl.py", line 944, in do_handshake
2020-08-26 08:08:38 ERROR (MainThread) [iaqualink]     self._sslobj.do_handshake()
2020-08-26 08:08:38 ERROR (MainThread) [iaqualink] ssl.SSLError: [SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:1108)
2020-08-26 08:08:38 ERROR (MainThread) [iaqualink] 
2020-08-26 08:08:38 ERROR (MainThread) [iaqualink] The above exception was the direct cause of the following exception:
2020-08-26 08:08:38 ERROR (MainThread) [iaqualink] 
2020-08-26 08:08:38 ERROR (MainThread) [iaqualink] Traceback (most recent call last):
2020-08-26 08:08:38 ERROR (MainThread) [iaqualink]   File "/usr/local/lib/python3.8/site-packages/iaqualink/system.py", line 75, in update
2020-08-26 08:08:38 ERROR (MainThread) [iaqualink]     r1 = await self.aqualink.send_home_screen_request(self.serial)
2020-08-26 08:08:38 ERROR (MainThread) [iaqualink]   File "/usr/local/lib/python3.8/site-packages/iaqualink/client.py", line 157, in send_home_screen_request
2020-08-26 08:08:38 ERROR (MainThread) [iaqualink]     r = await self._send_session_request(
2020-08-26 08:08:38 ERROR (MainThread) [iaqualink]   File "/usr/local/lib/python3.8/site-packages/iaqualink/client.py", line 152, in _send_session_request
2020-08-26 08:08:38 ERROR (MainThread) [iaqualink]     return await self._send_request(url)
2020-08-26 08:08:38 ERROR (MainThread) [iaqualink]   File "/usr/local/lib/python3.8/site-packages/iaqualink/client.py", line 79, in _send_request
2020-08-26 08:08:38 ERROR (MainThread) [iaqualink]     r = await self.session.request(
2020-08-26 08:08:38 ERROR (MainThread) [iaqualink]   File "/usr/local/lib/python3.8/site-packages/aiohttp/client.py", line 480, in _request
2020-08-26 08:08:38 ERROR (MainThread) [iaqualink]     conn = await self._connector.connect(
2020-08-26 08:08:38 ERROR (MainThread) [iaqualink]   File "/usr/local/lib/python3.8/site-packages/aiohttp/connector.py", line 523, in connect
2020-08-26 08:08:38 ERROR (MainThread) [iaqualink]     proto = await self._create_connection(req, traces, timeout)
2020-08-26 08:08:38 ERROR (MainThread) [iaqualink]   File "/usr/local/lib/python3.8/site-packages/aiohttp/connector.py", line 858, in _create_connection
2020-08-26 08:08:38 ERROR (MainThread) [iaqualink]     _, proto = await self._create_direct_connection(
2020-08-26 08:08:38 ERROR (MainThread) [iaqualink]   File "/usr/local/lib/python3.8/site-packages/aiohttp/connector.py", line 1004, in _create_direct_connection
2020-08-26 08:08:38 ERROR (MainThread) [iaqualink]     raise last_exc
2020-08-26 08:08:38 ERROR (MainThread) [iaqualink]   File "/usr/local/lib/python3.8/site-packages/aiohttp/connector.py", line 980, in _create_direct_connection
2020-08-26 08:08:38 ERROR (MainThread) [iaqualink]     transp, proto = await self._wrap_create_connection(
2020-08-26 08:08:38 ERROR (MainThread) [iaqualink]   File "/usr/local/lib/python3.8/site-packages/aiohttp/connector.py", line 941, in _wrap_create_connection
2020-08-26 08:08:38 ERROR (MainThread) [iaqualink]     raise ClientConnectorSSLError(req.connection_key, exc) from exc
2020-08-26 08:08:38 ERROR (MainThread) [iaqualink] aiohttp.client_exceptions.ClientConnectorSSLError: Cannot connect to host iaqualink-api.realtime.io:443 ssl:default [[SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:1108)]
2020-08-26 08:08:38 ERROR (MainThread) [iaqualink] 
2020-08-26 08:08:38 WARNING (MainThread) [homeassistant.components.iaqualink] Failed to refresh iAqualink state
2020-08-26 08:09:08 ERROR (MainThread) [iaqualink] Unhandled exception: Cannot connect to host iaqualink-api.realtime.io:443 ssl:default [[SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:1108)]
2020-08-26 08:09:08 ERROR (MainThread) [iaqualink] Traceback (most recent call last):
2020-08-26 08:09:08 ERROR (MainThread) [iaqualink]   File "/usr/local/lib/python3.8/site-packages/aiohttp/connector.py", line 936, in _wrap_create_connection
2020-08-26 08:09:08 ERROR (MainThread) [iaqualink]     return await self._loop.create_connection(*args, **kwargs)  # type: ignore  # noqa
2020-08-26 08:09:08 ERROR (MainThread) [iaqualink]   File "/usr/local/lib/python3.8/asyncio/base_events.py", line 1050, in create_connection
2020-08-26 08:09:08 ERROR (MainThread) [iaqualink]     transport, protocol = await self._create_connection_transport(
2020-08-26 08:09:08 ERROR (MainThread) [iaqualink]   File "/usr/local/lib/python3.8/asyncio/base_events.py", line 1080, in _create_connection_transport
2020-08-26 08:09:08 ERROR (MainThread) [iaqualink]     await waiter
2020-08-26 08:09:08 ERROR (MainThread) [iaqualink]   File "/usr/local/lib/python3.8/asyncio/sslproto.py", line 529, in data_received
2020-08-26 08:09:08 ERROR (MainThread) [iaqualink]     ssldata, appdata = self._sslpipe.feed_ssldata(data)
2020-08-26 08:09:08 ERROR (MainThread) [iaqualink]   File "/usr/local/lib/python3.8/asyncio/sslproto.py", line 189, in feed_ssldata
2020-08-26 08:09:08 ERROR (MainThread) [iaqualink]     self._sslobj.do_handshake()
2020-08-26 08:09:08 ERROR (MainThread) [iaqualink]   File "/usr/local/lib/python3.8/ssl.py", line 944, in do_handshake
2020-08-26 08:09:08 ERROR (MainThread) [iaqualink]     self._sslobj.do_handshake()
2020-08-26 08:09:08 ERROR (MainThread) [iaqualink] ssl.SSLError: [SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:1108)
2020-08-26 08:09:08 ERROR (MainThread) [iaqualink] 
2020-08-26 08:09:08 ERROR (MainThread) [iaqualink] The above exception was the direct cause of the following exception:
2020-08-26 08:09:08 ERROR (MainThread) [iaqualink] 
2020-08-26 08:09:08 ERROR (MainThread) [iaqualink] Traceback (most recent call last):
2020-08-26 08:09:08 ERROR (MainThread) [iaqualink]   File "/usr/local/lib/python3.8/site-packages/iaqualink/system.py", line 75, in update
2020-08-26 08:09:08 ERROR (MainThread) [iaqualink]     r1 = await self.aqualink.send_home_screen_request(self.serial)
2020-08-26 08:09:08 ERROR (MainThread) [iaqualink]   File "/usr/local/lib/python3.8/site-packages/iaqualink/client.py", line 157, in send_home_screen_request
2020-08-26 08:09:08 ERROR (MainThread) [iaqualink]     r = await self._send_session_request(
2020-08-26 08:09:08 ERROR (MainThread) [iaqualink]   File "/usr/local/lib/python3.8/site-packages/iaqualink/client.py", line 152, in _send_session_request
2020-08-26 08:09:08 ERROR (MainThread) [iaqualink]     return await self._send_request(url)
2020-08-26 08:09:08 ERROR (MainThread) [iaqualink]   File "/usr/local/lib/python3.8/site-packages/iaqualink/client.py", line 79, in _send_request
2020-08-26 08:09:08 ERROR (MainThread) [iaqualink]     r = await self.session.request(
2020-08-26 08:09:08 ERROR (MainThread) [iaqualink]   File "/usr/local/lib/python3.8/site-packages/aiohttp/client.py", line 480, in _request
2020-08-26 08:09:08 ERROR (MainThread) [iaqualink]     conn = await self._connector.connect(
2020-08-26 08:09:08 ERROR (MainThread) [iaqualink]   File "/usr/local/lib/python3.8/site-packages/aiohttp/connector.py", line 523, in connect
2020-08-26 08:09:08 ERROR (MainThread) [iaqualink]     proto = await self._create_connection(req, traces, timeout)
2020-08-26 08:09:08 ERROR (MainThread) [iaqualink]   File "/usr/local/lib/python3.8/site-packages/aiohttp/connector.py", line 858, in _create_connection
2020-08-26 08:09:08 ERROR (MainThread) [iaqualink]     _, proto = await self._create_direct_connection(
2020-08-26 08:09:08 ERROR (MainThread) [iaqualink]   File "/usr/local/lib/python3.8/site-packages/aiohttp/connector.py", line 1004, in _create_direct_connection
2020-08-26 08:09:08 ERROR (MainThread) [iaqualink]     raise last_exc
2020-08-26 08:09:08 ERROR (MainThread) [iaqualink]   File "/usr/local/lib/python3.8/site-packages/aiohttp/connector.py", line 980, in _create_direct_connection
2020-08-26 08:09:08 ERROR (MainThread) [iaqualink]     transp, proto = await self._wrap_create_connection(
2020-08-26 08:09:08 ERROR (MainThread) [iaqualink]   File "/usr/local/lib/python3.8/site-packages/aiohttp/connector.py", line 941, in _wrap_create_connection
2020-08-26 08:09:08 ERROR (MainThread) [iaqualink]     raise ClientConnectorSSLError(req.connection_key, exc) from exc
2020-08-26 08:09:08 ERROR (MainThread) [iaqualink] aiohttp.client_exceptions.ClientConnectorSSLError: Cannot connect to host iaqualink-api.realtime.io:443 ssl:default [[SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:1108)]
2020-08-26 08:09:08 ERROR (MainThread) [iaqualink]

Additional information

This SSL error only started yesterday, well after I upgraded to 0.114.3. It is POSSIBLE that it is a server issue, but hard to tell from here.

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Comments: 61 (3 by maintainers)

Most upvoted comments

@markiper I’m having a zoom call with them in 5 minutes to talk about it. I’ll give you more details after we’re done.

+1
derekatkins on Sep 3, 2020

Yes, that is my observation as well. So long as it has the 34.224 and 54.174 addresses it all works. When it changes to something else (13.225 for you, 99.86 for me) it throws errors.

+1
derekatkins on Sep 3, 2020

as long as the p-api.iaqualink.net point to the 34.224 and 54.174 IP addresses, everything works properly; when it changes to the 13.225 set of IP addresses is when the problem happens. Why the change on their end is what I don’t understand (load balancing?); luckily for the past 30 minutes there hasn’t been any changes, so HA has also been stable.

+1
markiper on Sep 3, 2020

OOOH! I was just able to reproduce this manually:

curl -1 -0 -v 'https://iaqualink-api.realtime.io/'
*   Trying 99.86.100.129...
* TCP_NODELAY set
* Connected to iaqualink-api.realtime.io (99.86.100.129) port 443 (#0)
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* error:1408F10B:SSL routines:ssl3_get_record:wrong version number
* Closing connection 0
curl: (35) error:1408F10B:SSL routines:ssl3_get_record:wrong version number

Compared to a successful output:

curl -1 -0 -v 'https://iaqualink-api.realtime.io/'
*   Trying 54.174.124.202...
* TCP_NODELAY set
* Connected to iaqualink-api.realtime.io (54.174.124.202) port 443 (#0)
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: CN=*.realtime.io
*  start date: Feb 21 00:00:00 2020 GMT
*  expire date: Mar 21 12:00:00 2021 GMT
*  subjectAltName: host "iaqualink-api.realtime.io" matched cert's "*.realtime.io"
*  issuer: C=US; O=Amazon; OU=Server CA 1B; CN=Amazon
*  SSL certificate verify ok.
> GET / HTTP/1.0
> Host: iaqualink-api.realtime.io
> User-Agent: curl/7.61.1
> Accept: */*
> 
< HTTP/1.1 200 OK
< Date: Thu, 03 Sep 2020 16:13:29 GMT
< Content-Type: text/html; charset=utf-8
< Connection: close
< Server: Apache/2.4.34 (Unix) mod_perl/2.0.10 Perl/v5.20.2
< 
* TLSv1.2 (IN), TLS alert, close notify (256):
* Closing connection 0
* TLSv1.2 (OUT), TLS alert, close notify (256):
<html><body><h1>It works!</h1><div id='upstreamID'>HA-0A</div></body></html>

I think we’ve found their bad server!! Their DNS info does change periodically. When I got the bad curl output I saw:

host iaqualink-api.realtime.io
iaqualink-api.realtime.io is an alias for iaqualink-api.iaqualink.net.
iaqualink-api.iaqualink.net is an alias for p-api.iaqualink.net.
p-api.iaqualink.net has address 99.86.100.128
p-api.iaqualink.net has address 99.86.100.101
p-api.iaqualink.net has address 99.86.100.129
p-api.iaqualink.net has address 99.86.100.25

Whereas during good times I see:

host iaqualink-api.realtime.io
iaqualink-api.realtime.io is an alias for iaqualink-api.iaqualink.net.
iaqualink-api.iaqualink.net is an alias for p-api.iaqualink.net.
p-api.iaqualink.net has address 34.224.76.141
p-api.iaqualink.net has address 54.174.124.202

Does anyone have a contact at realtime.io? I suspect this is an error on one of their servers, at least 99.86.100.129 is being problematic.

+1
derekatkins on Sep 3, 2020
Read more comments on GitHub
← core: I can't control my SamsungTV
core: iBeacon never changes from "home" regardless if kilometers away or battery off. →