core: Mazda Integration Rate Limit Lock Out - Does not recover

The problem

A couple days ago Mazda Services went down for “emergency services” my log captured that error 22 times. Following that it appears that Login now fails in the Mazda integration due to exceeding the rate limit. It has been 4 days now and the integration has not recovered.

I would say this appears to be that because login was failing because the service was down and the response was not expected it kept retrying, resulting in the Mazda service blocking the client due to rate limit.

Restarting Home Assistant did fix the problem.

Either Mazda is not sending a retry-after, the integration is ignoring it, or once the integration gets the rate limit error it just never tries again.

What version of Home Assistant Core has the issue?

core-2023.7.2

What was the last working version of Home Assistant Core?

No response

What type of installation are you running?

Home Assistant Container

Integration causing the issue

Mazda Connected Services

Link to integration documentation on our website

https://www.home-assistant.io/integrations/mazda

Diagnostics information

removing diagnostic log because it logs personal info

Example YAML snippet

No response

Anything in the logs that might be useful for us?

Logger: pymazda.connection
Source: components/mazda/__init__.py:72
First occurred: July 13, 2023 at 7:55:09 PM (1 occurrences)
Last logged: July 13, 2023 at 7:55:09 PM

Login failed: OVER_RATE_LIMIT

----------------------------------
emergency maintenance log





Last logged: July 13, 2023 at 7:53:48 PM

Unknown error occurred during Mazda update request: Request failed: MyMazda is temporarily unavailable due to emergency maintenance.
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/mazda/__init__.py", line 154, in async_update_data
    vehicles = await with_timeout(mazda_client.get_vehicles())
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/mazda/__init__.py", line 57, in with_timeout
    return await task
           ^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/pymazda/client.py", line 27, in get_vehicles
    vec_base_infos_response = await self.controller.get_vec_base_infos()
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/pymazda/controller.py", line 21, in get_vec_base_infos
    return await self.connection.api_request("POST", "remoteServices/getVecBaseInfos/v4", body_dict={"internaluserid": "__INTERNAL_ID__"}, needs_keys=True, needs_auth=True)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/pymazda/connection.py", line 163, in api_request
    return await self.__api_request_retry(method, uri, query_dict, body_dict, needs_keys, needs_auth, num_retries=0)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/pymazda/connection.py", line 170, in __api_request_retry
    await self.__ensure_keys_present()
  File "/usr/local/lib/python3.11/site-packages/pymazda/connection.py", line 258, in __ensure_keys_present
    await self.__retrieve_keys()
  File "/usr/local/lib/python3.11/site-packages/pymazda/connection.py", line 273, in __retrieve_keys
    response = await self.api_request("POST", "service/checkVersion", needs_keys=False, needs_auth=False)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/pymazda/connection.py", line 163, in api_request
    return await self.__api_request_retry(method, uri, query_dict, body_dict, needs_keys, needs_auth, num_retries=0)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/pymazda/connection.py", line 178, in __api_request_retry
    return await self.__send_api_request(method, uri, query_dict, body_dict, needs_keys, needs_auth)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/pymazda/connection.py", line 252, in __send_api_request
    raise MazdaException("Request failed: " + response_json["error"])
pymazda.exceptions.MazdaException: Request failed: MyMazda is temporarily unavailable due to emergency maintenance.

Additional information

No response