ubios-cert: FW 3.2.7 breaks RADIUS certificates

Allright, I “think” I got to the end of it by modifying the right key/values in the mongoDB. This seems to survive the System config reload, the service unifi restart and the full UDM hardware reboot.

⚠️ Again, I’m not responsible for you breaking your MongoDB, have some backups before!

The following one-liners are used to load the right certs in the right collection by updating existing values (this mongosh command looks for “key: radius” combo to upsert values in place -which creates them if not present, you might want to check that you only have one “key: radius” combo before ending up with more than one document):

mongo --quiet --port 27117 ace --eval "var certContent='$(base64 -w 0 /data/udapi-config/raddb/certs/server.pem)'; db.setting.findOneAndUpdate({key : \"radius\"}, {\$set: {'server_certificate': certContent}}, {upsert: true, returnNewDocument: true})"
mongo --quiet --port 27117 ace --eval "var certContent='$(base64 -w 0 /data/udapi-config/raddb/certs/server-key.pem)'; db.setting.findOneAndUpdate({key : \"radius\"}, {\$set: {'server_certificate_key': certContent}}, {upsert: true, returnNewDocument: true})"
mongo --quiet --port 27117 ace --eval "var certContent='$(base64 -w 0 /data/udapi-config/raddb/certs/ca.pem)'; db.setting.findOneAndUpdate({key : \"radius\"}, {\$set: {'ca_certificate': certContent}}, {upsert: true, returnNewDocument: true})"

One thing I haven’t tested is if the other changes to the JSON files are necessary if this is done first. It might be the case that only this change to the MongoDB is needed ultimately.

Also, I’m not sure how this will fare with updates, Ubiquiti has already changed all certs w/o telling its users, so this might have to be adjusted with future updates of Unifi OS.

Allright, I found the solution (which I had in front of my eyes for a while, except I missed a layer of b64 encoding).

Turns out, with 3.2.7, the certificates (CA, Server, and a bunch of others) have been moved to the udapi-server configuration state/file in /data/udapi-config/ubios-udapi-server/ubios-udapi-server.state. This is confirmed by the service being launched as /usr/bin/ubios-udapi-server -c /data/udapi-config/ubios-udapi-server/ubios-udapi-server.state --silent.

The JSON file elements where this can be found are .services.radiusServer.caCertificate.crt .services.radiusServer.serverCertificate.crt and .services.radiusServer.serverCertificate.key but they are not in PEM format, they are Base64 encoded over their PEM format.

Upon service start, the certificates are taken from the ubios-udapi-server.state JSON file, decoded from Base64 and written to /etc/freeradius/3.0/certs/ as ca.pem server.pem and server-key.pem accordingly.

As such, it’s possible to replace these entries in the JSON file and (probably?) make them persistent in this configuration.

Here are one-liners using jq examples to load your existing certificates into the JSON configuration file, provided that you already have a ca.pem (CA Certificate content in PEM format), server.pem (Server Certificate content in PEM format) and server-key.pem (Server Key Content in PEM format).

⚠️DO BACKUP YOUR JSON CONFIG FILE, I AM NOT RESPONSIBLE FOR YOU BREAKING YOUR UNIFI HARDWARE.⚠️

jq --arg filecontent "$(cat /data/udapi-config/raddb/certs/ca.pem)" '.services.radiusServer.caCertificate.crt = $filecontent | (.services.radiusServer.caCertificate.crt |= @base64)' /data/udapi-config/ubios-udapi-server/ubios-udapi-server.state > /data/udapi-config/ubios-udapi-server/ubios-udapi-server.new.state ; mv /data/udapi-config/ubios-udapi-server/ubios-udapi-server.new.state /data/udapi-config/ubios-udapi-server/ubios-udapi-server.state

jq --arg filecontent "$(cat /data/udapi-config/raddb/certs/server.pem)" '.services.radiusServer.serverCertificate.crt = $filecontent | (.services.radiusServer.serverCertificate.crt |= @base64)' /data/udapi-config/ubios-udapi-server/ubios-udapi-server.state > /data/udapi-config/ubios-udapi-server/ubios-udapi-server.new.state ; mv /data/udapi-config/ubios-udapi-server/ubios-udapi-server.new.state /data/udapi-config/ubios-udapi-server/ubios-udapi-server.state

jq --arg filecontent "$(cat /data/udapi-config/raddb/certs/server-key.pem)" '.services.radiusServer.serverCertificate.key = $filecontent | (.services.radiusServer.serverCertificate.key |= @base64)' /data/udapi-config/ubios-udapi-server/ubios-udapi-server.state > /data/udapi-config/ubios-udapi-server/ubios-udapi-server.new.state ; mv /data/udapi-config/ubios-udapi-server/ubios-udapi-server.new.state /data/udapi-config/ubios-udapi-server/ubios-udapi-server.state

Note that this is probably also true for the unifi-core certificates, but I haven’t looked into it specifically.

So why did Ubiquiti make that change? I can’t tel, but hopefully it is because eventually that JSON config file (that seem to hold the entirety of the UDM config) will be modified by the Web UI and allow the user to load their own certificates ? One can dream 😃

That being said, forcing a new CA chain onto all their users without any early warning is certainly not professional, with many users complaining about connection refusals from Apple (and Android) devices that suddently get a certificate change (for another untrusted one, mind you) https://community.ui.com/questions/UniFi-OS-3-2-7-CA-Certificate-changes-break-Radius-backed-802-1x-authentication/9a5d4fb9-f791-4bec-81b5-3b9cc0c7853a#answer/23a7c99a-e1f4-410f-95fb-670b4fb426d3

All thanks @ouaibe . This did solve the problem for me.

@ouaibe Great analysis!

Will have a look into the web server certificate in the other issue e#62 - the JSON contains another cert, suitable for web server, which is (again) the standard self-signed cert by UI (at least on my machine).

So far, restarting unifi-core did not break the web certificate. And I can only support what you write in your final paragraph…

but, maybe this is just a small step we have to overcome on the path to having it managed automatically.

Allright, I found the solution (which I had in front of my eyes for a while, except I missed a layer of b64 encoding).

Turns out, with 3.2.7, the certificates (CA, Server, and a bunch of others) have been moved to the udapi-server configuration state/file in /data/udapi-config/ubios-udapi-server/ubios-udapi-server.state. This is confirmed by the service being launched as /usr/bin/ubios-udapi-server -c /data/udapi-config/ubios-udapi-server/ubios-udapi-server.state --silent.

The JSON file elements where this can be found are .services.radiusServer.caCertificate.crt .services.radiusServer.serverCertificate.crt and .services.radiusServer.serverCertificate.key but they are not in PEM format, they are Base64 encoded over their PEM format.

Upon service start, the certificates are taken from the ubios-udapi-server.state JSON file, decoded from Base64 and written to /etc/freeradius/3.0/certs/ as ca.pem server.pem and server-key.pem accordingly.

As such, it’s possible to replace these entries in the JSON file and (probably?) make them persistent in this configuration.

Here are one-liners using jq examples to load your existing certificates into the JSON configuration file, provided that you already have a ca.pem (CA Certificate content in PEM format), server.pem (Server Certificate content in PEM format) and server-key.pem (Server Key Content in PEM format).

⚠️DO BACKUP YOUR JSON CONFIG FILE, I AM NOT RESPONSIBLE FOR YOU BREAKING YOUR UNIFI HARDWARE.⚠️

jq --arg filecontent "$(cat /data/udapi-config/raddb/certs/ca.pem)" '.services.radiusServer.caCertificate.crt = $filecontent | (.services.radiusServer.caCertificate.crt |= @base64)' /data/udapi-config/ubios-udapi-server/ubios-udapi-server.state > /data/udapi-config/ubios-udapi-server/ubios-udapi-server.new.state ; mv /data/udapi-config/ubios-udapi-server/ubios-udapi-server.new.state /data/udapi-config/ubios-udapi-server/ubios-udapi-server.state

jq --arg filecontent "$(cat /data/udapi-config/raddb/certs/server.pem)" '.services.radiusServer.serverCertificate.crt = $filecontent | (.services.radiusServer.serverCertificate.crt |= @base64)' /data/udapi-config/ubios-udapi-server/ubios-udapi-server.state > /data/udapi-config/ubios-udapi-server/ubios-udapi-server.new.state ; mv /data/udapi-config/ubios-udapi-server/ubios-udapi-server.new.state /data/udapi-config/ubios-udapi-server/ubios-udapi-server.state

jq --arg filecontent "$(cat /data/udapi-config/raddb/certs/server-key.pem)" '.services.radiusServer.serverCertificate.key = $filecontent | (.services.radiusServer.serverCertificate.key |= @base64)' /data/udapi-config/ubios-udapi-server/ubios-udapi-server.state > /data/udapi-config/ubios-udapi-server/ubios-udapi-server.new.state ; mv /data/udapi-config/ubios-udapi-server/ubios-udapi-server.new.state /data/udapi-config/ubios-udapi-server/ubios-udapi-server.state

Note that this is probably also true for the unifi-core certificates, but I haven’t looked into it specifically.

So why did Ubiquiti make that change? I can’t tel, but hopefully it is because eventually that JSON config file (that seem to hold the entirety of the UDM config) will be modified by the Web UI and allow the user to load their own certificates ? One can dream 😃

That being said, forcing a new CA chain onto all their users without any early warning is certainly not professional, with many users complaining about connection refusals from Apple (and Android) devices that suddently get a certificate change (for another untrusted one, mind you) https://community.ui.com/questions/UniFi-OS-3-2-7-CA-Certificate-changes-break-Radius-backed-802-1x-authentication/9a5d4fb9-f791-4bec-81b5-3b9cc0c7853a#answer/23a7c99a-e1f4-410f-95fb-670b4fb426d3

I know this is “late” in the discussion, but I can also confirm that the certs are being replaced by udapi-server starting process:

From journalctl -u udapi-server:

...
ubios-udapi-server[1467192]: svc-radius-server-service:      +(services):   Start deleted->running service radiusServer
ubios-udapi-server[1467192]: svc-radius-server-service: applying full configuration
ubios-udapi-server[1467192]: svc-radius-server-service: using provisioned server certificate
ubios-udapi-server[1467192]: svc-radius-server-service: using provisioned server key
ubios-udapi-server[1467192]: svc-radius-server-service: using provisioned CA certificate
ubios-udapi-server[1467192]: svc-radius-server-service: starting the service
...

And using inotifywait to monitor file changes in the certs directory:

/etc/freeradius/3.0/certs# inotifywait -mr .
Setting up watches.  Beware: since -r was given, this may take a while!
Watches established.


./ MODIFY server.pem
./ OPEN server.pem
./ MODIFY server.pem
./ CLOSE_WRITE,CLOSE server.pem
./ MODIFY server-key.pem
./ OPEN server-key.pem
./ MODIFY server-key.pem
./ CLOSE_WRITE,CLOSE server-key.pem
./ MODIFY ca.pem
./ OPEN ca.pem
./ MODIFY ca.pem
./ CLOSE_WRITE,CLOSE ca.pem
./ OPEN server.pem
./ ACCESS server.pem
./ CLOSE_NOWRITE,CLOSE server.pem
./ OPEN ca.pem
./ ACCESS ca.pem
./ CLOSE_NOWRITE,CLOSE ca.pem
./ OPEN ca.pem
./ ACCESS ca.pem
./ CLOSE_NOWRITE,CLOSE ca.pem
./ OPEN server-key.pem
./ ACCESS server-key.pem
./ CLOSE_NOWRITE,CLOSE server-key.pem
./ OPEN dh
./ ACCESS dh
./ CLOSE_NOWRITE,CLOSE dh
./ OPEN,ISDIR
./ ACCESS,ISDIR
./ CLOSE_NOWRITE,CLOSE,ISDIR

That being said, I’m unable to find the source of these certificates, nor which exact process does the replacement.