supervisor: Supervised installation not working on Docker 20.x
Describe the issue
Upgrading to the latest Debian 10 stable on ‘healthy supported supervised’ installation makes HA’s components never come back again: supervisor, observer and all dockers including HA
Steps to reproduce
- Have working Debian 10 installation
- apt-get update && apt-get upgrade (docker-ce and containerd especially to version 20.x)
- reboot (even it’s not needed as dockers doesn’t come back up)
After that, the only working ‘docker ps’ docker will be observer with no possibility to come back online.
restarting of the supervisor and doing ha su repair helps to get HA (and only HA, without addons) back online
But it’s not persistent and lost after reboot.
Enviroment details
- Operating System:: Debian 10.7
- Supervisor version:: latest (2020.12.06)
- Home Assistant version: latest (0.118.5)
Supervisor logs
Supervisor logs
Supervisor doensn't come back up
System Information
System Information
arch: amd64
channel: stable
docker: 20.10.0
features:
- reboot
- shutdown
- services
- network
- hostname
hassos: null
homeassistant: 0.118.5
hostname: brix
logging: info
machine: qemux86-64
operating_system: Debian GNU/Linux 10 (buster)
supervisor: 2020.12.6
supported: true
supported_arch:
- amd64
- i386
timezone: Europe/Warsaw
What’s quick-fix for those that have their whole house down, is to downgrade docker:
apt install docker-ce=5:19.03.14~3-0~debian-buster
and restore full-snapshot…
EDIT: Full rollback as in alert until fixed:
apt install docker-ce=5:19.03.14~3-0~debian-buster
apt install docker-ce-cli=5:19.03.14~3-0~debian-buster
apt install containerd.io=1.3.9-1
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Reactions: 47
- Comments: 33 (12 by maintainers)
Please do not respond with “same problem here”, instead click on the 👍 reaction on the first post.
We are aware of the issue but will not have time to fix it until after the Home Assistant Conference.
I think you need to rethink what an “alert” means. There’s nothing on the homeassistant.io website, nothing in the blog, nothing in the forums.
ok, fine, I get that you don’t support ubuntu (and completely get that you can’t support every operating system that some user arbitrarily decides they want to use), but it wouldn’t take a lot to update the initial post and the alert (what alert?) to point users at relevant user experiences on unsupported operating systems. That’s just helping the community help itself, and that’s always a good thing.
Until support is implemented, here’s the command to prevent packages from being upgraded:
And the command to revert that:
Today I decided to switch from HASS OS to Supervised Mode. Luckily I was made aware of the alert on https://alerts.home-assistant.io (https://alerts.home-assistant.io/#docker_2010.markdown). Guess how? By a user comment in a Home Assistant facebook group…
My two cents even I was lucky for today and could just downgrade the freshly installed docker as mentioned in the alert:
So I have to agree with @jon-hedgerows by saying the communication part has a bit room for improvement. Maybe in general, even users running the Supervised mode need take care of a lot more things as documentation is saying, that´s the deal.
For history. Docker will release soon 20.10.1, which solves the issue with systemd, and we can work around the other bug with the docker-library. 20.10.0 will never get working
@balloob maybe it’s worth to post a blog regarding this or some social-media (FB/Twitter)? I’m afraid most of users that has ‘clean Supervised’ will just apt upgrade like I did and face few hours of downtime (it took 4 for me before I decided to skip debugging and restored full snapshot with 21 addons).
Instructions for that has been added to the bottom of this: https://alerts.home-assistant.io/#docker_2010.markdown
I can confirm it works - forced Supervisor update, apt upgrade and reboot worked (as usually) - flawlessly.
I also confirm that it works with the last docker-ce package
Docker 20.10.1 was released with an updated systemd unit file, and should probably fix this issue
Following on what others say, it might be a good thing to integrate those alerts into HA, like a notification.
This are the commands to rollback in ubuntu 18.04 (bionic)
and this, for prevent packages from being upgraded:
apt-mark hold containerd.io=1.3.9-1 docker-ce=5:19.03.14~3-0~ubuntu-bionic docker-ce-cli=5:19.03.14~3-0~ubuntu-bionicFor other distributions, you can check with
cat /etc/os-releaseand you can replace the commands that contain ubuntu-bionic with your version, in this format:
%ID%-%VERSION_CODENAME%(mods: if you want to merge this comment in the first want, you re welcome!)
To add some more context to this issue witch is also described in the alert
The issue has 2 parts.
Filtering has been removed from the API that lists container images, with that change the Supervisor is not able to handle the response correctly. It looks like that will be resolved in https://github.com/docker/docker-py/commit/1757c974fa3a05b0e9b783af85242b18df09d05d which is the library the Supervisor uses to communicate with Docker.
The second part, is changes to the systemd service for Docker, that change results in the service for
hassio-supervisorbeing blocked at startup.The quick-fix given in the issue description will for the most part work, but it’s not enough, see the alert for a full rollback of Docker.