core: 404 Error - Unable to set up integration

Glad that it was helpful! I’ve seen so many bad bug reports in my life, that I try to do better myself 😃 You’re right about the versions. I had to update the Cloud Key firmware, because I wasn’t on Unify OS before. And as they removed user management from the Network app (which I had updated) and moved it to the OS, I had to update the firmware to be able to create an additional user for HA use.

My current versions are

UniFi-CloudKey-Gen2-Plus Firmware 2.1.11

Network Version 7.2.95

@Kane610 I’ve been looking into this problem now for two days, I think I found the reason for the /api/auth/login received 404 Not Found error situation. As I don’t have experience with Python or HA development, I’m not unfortunately sure how to dev & test a proper PR for this, but hopefully the information here will help the bug to be fixed.

I own a CloudKey Gen 2 Plus, and hard restarting it does not solve the problem with it, like it for some reason seems to do with the UDM-Pro. I didn’t find a single issue comment anywhere which would have said that CKG2+ started to work after a hard reset, only UDM-Pros did.

TL;DR

HTTP GET https://UNIFI_OS_IP/ sets a cookie that does not work with with the subsequent HTTP POST https://UNIFI_OS_IP/api/auth/login, resulting as a 404 Not Found reply from the controller

Explanation

The authentication endpoint differs between the old Unifi Network application and the new Unify OS versions. Because of this the Unifi integration first checks the software version and then tries to log in.

However both of those calls use the same session, and as the first call already sets a cookie, it gets sent along the second request.

Here are some examples obtained by putting a mitmproxy between HA and the Unifi OS controller, and exported for curl:

First request:

curl -H 'User-Agent: HomeAssistant/2022.12.9 aiohttp/3.8.1 Python/3.10' -H 'Accept: */*' --compressed https://UNIFY_OS_IP/ -k

Unifi OS responds HTTP 200 with a header

Set-Cookie: TOKEN=first_token_string

This causes the second authentication request to look like this:

curl -H 'User-Agent: HomeAssistant/2022.12.9 aiohttp/3.8.1 Python/3.10' -H 'Accept: */*' --compressed -H 'Cookie: TOKEN=first_token_string' -H 'Content-Type: application/json' -X POST https://UNIFI_OS_IP/api/auth/login -d '{"username":"my_secret_username","password":"my_secret_password","remember":true}' -k

And this will cause the infamous HTTP 404 Not Found response.

However, if we modify the request so that we take the cookie header away:

curl -H 'User-Agent: HomeAssistant/2022.12.9 aiohttp/3.8.1 Python/3.10' -H 'Accept: */*' --compressed -H 'Content-Type: application/json' -X POST https://UNIFI_OS_IP/api/auth/login -d '{"username":"my_secret_username","password":"my_secret_password","remember":true}' -k

We get HTTP 200 with a JSON response body, and a Set-Cookie header which will work for later API requests.

Fix

The first request for the checking whether we are dealing with Unifi OS or not (HTTP GET https://UNIFI_OS_IP/) shouldn’t be done with the same session/CookieJar as the other requests. Session properly starts only after a successful login attempt, not before it. However as I’m not so familiar with the techs used here, I’m not sure if this is something that should be fixed on the HA Unifi integration side or on the aiounifi side

Verification

I was able to verify that this fixes the Unifi integration setup by setting a mitmproxy reverse proxy with this command:

mitmproxy -p 443 --mode reverse:https://UNIFI_OS_IP --ssl-insecure -s remove-logic-check-cookie.py

and this script remove-logic-check-cookie.py:

from mitmproxy import http
from mitmproxy import ctx

class RemoveLoginCheckCookie:
    filter = "https://UNIFY_OS_IP/"

    def response(self, flow: http.HTTPFlow) -> None:
        if (self.filter == flow.request.pretty_url and ('Set-Cookie') in flow.response.headers):
            del flow.response.headers['Set-Cookie']

addons = [RemoveLoginCheckCookie()]

Now when I try to setup the HA Unifi integration by pointing host to my laptop running mitmproxy, everything works.

Notes

~I have tested this only using curl, so it might still be that this does not fix things in the core integration. As I have absolutely no idea on how to setup a proper dev env for HA, and unfortunately also not enough time, I couldn’t test this properly there. But this at least sounds like it could be the source of these problems.~

And no, I really don’t understand how hard resetting a UDM-Pro helps with this 😄

Thanks for the deep dive! I will look into it shortly 😃