core: 2022.7.x docker container broken on 1st gen Raspberry Pi

The problem

I’m trying to run home-assistant:stable with the docker compose method on an Raspberry Pi Model B Rev 2 (512MB RAM / armv6l). This is obviously not a good platform, but it’s what I had available for a test. The OS is Raspbian/bullseye, with docker ce 20.10.17

Running docker compose up, the container starts, but the homeassistant inside seems to crash:

Attaching to homeassistant
homeassistant  | s6-rc: info: service s6rc-oneshot-runner: starting
homeassistant  | s6-rc: info: service s6rc-oneshot-runner successfully started
homeassistant  | s6-rc: info: service fix-attrs: starting
homeassistant  | s6-rc: info: service fix-attrs successfully started
homeassistant  | s6-rc: info: service legacy-cont-init: starting
homeassistant  | s6-rc: info: service legacy-cont-init successfully started
homeassistant  | s6-rc: info: service legacy-services: starting
homeassistant  | services-up: info: copying legacy longrun home-assistant (no readiness notification)
homeassistant  | s6-rc: info: service legacy-services successfully started
homeassistant  | [22:08:03] INFO: Home Assistant Core finish process exit code 256
homeassistant  | [22:08:03] INFO: Home Assistant Core finish process received signal 4
homeassistant  | [22:08:09] INFO: Home Assistant Core finish process exit code 256
homeassistant  | [22:08:09] INFO: Home Assistant Core finish process received signal 4

A plain Home Assistant Core venv starts successfully on the same hardware, as does the previous release of the docker container using 2022.6.6@sha256:c2e7c0bb84e6f84d1c62ff34d9f286fb17486ccf2760a5612f65128848c59316.

What version of Home Assistant Core has the issue?

2022.7.5

What was the last working version of Home Assistant Core?

2022.6.6

What type of installation are you running?

Home Assistant Container

Integration causing the issue

No response

Link to integration documentation on our website

No response

Diagnostics information

When looking at the resulting core file with gdb, it reports an illegal instruction:

Core was generated by `python3 -m homeassistant --config /config'.
Program terminated with signal SIGILL, Illegal instruction.
#0  0xb58fe7fa in ?? ()

I assume the build uses some compiler flag that creates instructions not supported by the BCM2835?

Example YAML snippet

No response

Anything in the logs that might be useful for us?

No response

Additional information

No response

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Reactions: 16
  • Comments: 49 (16 by maintainers)

Most upvoted comments

I’ve finally created a pull request I’ve noted that the builder used, if the architecture is armhf, in the actions will provide with the QEMU_CPU=arm1176 as an ARG. The only thing needed was to add an ARG QEMU_CPU in the dockerfile and that should trigger the right process.

I tested it by the same process

Added

ARG QEMU_CPU at the beginning of the dockerfile after the ENV S6…

then build it with docker buildx create --platform linux/arm/v6 --use //needed only one time docker buildx build --platform=linux/arm/v6 --build-arg QEMU_CPU=arm1176 --build-arg BUILD_FROM=ghcr.io/home-assistant/armhf-homeassistant-base:2022.11.0 -t homeassistant-home .

hope it will be merged soon so that we will be able to use the offical image from the repository.

thank you @Jojo-1000 without your analysis I would never been able to identify what was going on

+6
Gerigot on Jan 19, 2023

TL;DR

I think the Github Actions are configured wrong to use armv7 instead of armv6 for the wheels packages. From the logs of armhf base core images (2022.12.9): Collecting numpy==1.23.2 #13 85.81 Downloading https://wheels.home-assistant.io/musllinux/numpy-1.23.2-cp310-cp310-musllinux_1_2_armv7l.whl

Long story (for troubleshooting in case that is not it)

I ran into this same issue on raspberry pi 1B rev2. It is very easy to reproduce with the built docker containers:

pi@raspberrypi:~ $ docker exec -ti homeassistant bash
bash-5.1# python
Python 3.10.7 (main, Nov 24 2022, 12:49:35) [GCC 11.2.1 20220219] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
Illegal instruction (core dumped)
bash-5.1#

Because it was suggested that the wheels might be broken, I tried to only pull the wheels directly (my poor pi 1b does not like to compile or download the full requirements). So I created a Dockerfile which only installs the numpy package from the same repository as the Github Actions use and built the container.

Dockerfile:

ARG BUILD_FROM
FROM ghcr.io/home-assistant/armhf-homeassistant-base:2022.11.0@sha256:37b3385d3a10a30344ba73f4d5551b7ecb63aca1606b1586e67f34624f592d6c

# Synchronize with homeassistant/core.py:async_stop
ENV \
    S6_SERVICES_GRACETIME=220000

WORKDIR /usr/src

## Setup Home Assistant Core dependencies
#COPY requirements.txt homeassistant/
#COPY homeassistant/package_constraints.txt homeassistant/homeassistant/

# test install numpy package
RUN pip3 install --no-cache-dir --no-index --only-binary=:all: --find-links "https://wheels.home-assistant.io/musllinux/" --use-deprecated=legacy-resolver numpy
Build output 

pi@raspberrypi:~ $ docker build .
Sending build context to Docker daemon   2.56kB
Step 1/5 : ARG BUILD_FROM
Step 2/5 : FROM ghcr.io/home-assistant/armhf-homeassistant-base:2022.11.0@sha256:37b3385d3a10a30344ba73f4d5551b7ecb63aca1606b1586e67f34624f592d6c
 ---> cc6d4d2f7195
Step 3/5 : ENV     S6_SERVICES_GRACETIME=220000
 ---> Using cache
 ---> 70e2347fa3b1
Step 4/5 : WORKDIR /usr/src
 ---> Using cache
 ---> 7ab57f3baea6
Step 5/5 : RUN pip3 install --no-cache-dir --no-index --only-binary=:all: --find-links "https://wheels.home-assistant.io/musllinux/" --use-deprecated=legacy-resolver numpy
 ---> Running in 50736869aad6
Looking in links: https://wheels.home-assistant.io/musllinux/
Collecting numpy
  Downloading https://wheels.home-assistant.io/musllinux/numpy-1.23.2-cp310-cp310-musllinux_1_2_armv6l.whl (10.2 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 10.2/10.2 MB 556.4 kB/s eta 0:00:00
Installing collected packages: numpy
Successfully installed numpy-1.23.2
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
Removing intermediate container 50736869aad6
 ---> 0d13df698c3d
Successfully built 0d13df698c3d

The goal was just to get a simple system where I may have a chance of manually building the single package or swap it to a different source. So I tried to run the import-crash code again and to my surprise it was working:

docker run -ti 0d13df698c3d bash
s6-rc: info: service s6rc-oneshot-runner: starting
s6-rc: info: service s6rc-oneshot-runner successfully started
s6-rc: info: service fix-attrs: starting
s6-rc: info: service fix-attrs successfully started
s6-rc: info: service legacy-cont-init: starting
s6-rc: info: service legacy-cont-init successfully started
s6-rc: info: service legacy-services: starting
s6-rc: info: service legacy-services successfully started
bash-5.1# python
Python 3.10.7 (main, Nov 24 2022, 12:49:35) [GCC 11.2.1 20220219] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
>>> numpy.max([1, 2, 3])
3
>>>

I tried to pull the exact same image that was built in the Actions for that version and compared the numpy versions. That is where I noticed that the Actions are pulling from armv7, while my working version uses armv6:

Not working:

Collecting numpy==1.23.2
#13 85.81   Downloading https://wheels.home-assistant.io/musllinux/numpy-1.23.2-cp310-cp310-musllinux_1_2_armv7l.whl

Working:

Collecting numpy
  Downloading https://wheels.home-assistant.io/musllinux/numpy-1.23.2-cp310-cp310-musllinux_1_2_armv6l.whl (10.2 MB)

This is the same for all of the packages (although I did not test that every single one is working).

So I think this easily explains the illegal instruction errors, unless I am mistaken and the Actions are only used for CI and the uploaded docker images come from somewhere else. I am not very familiar with the Actions workflows, but this should be pretty easy to fix without any code changes and without rebuilding the wheels. I noticed that the architecture everywhere is set to arm in the logs, so maybe that is why it defaults to armv7?

+5
Jojo-1000 on Jan 4, 2023

I confirmed this problem still exists on 2022.11.5. I tried to build 2022.11.5 on my QNAP NAS but so far I haven’t been able to. I’m a noob when it comes to building from source. I tried downloading the HA core repo and adding the build command to my docker compose but it gives me an error saying I need to check my yaml. I will build on my machine if I can figure it out.

+4
Wetzel402 on Dec 5, 2022

So I made it work on my raspberry pi zero w with arch armv6l, I had to compile it from the source and from another pc because the raspberry was too loaded and it struggled to create the image.

steps that I did: Checkout from github the core repository

//Modified the Dockerfile FROM ghcr.io/home-assistant/armhf-homeassistant-base:2022.11.0

//added ENV QEMU_CPU=arm1176 // this is used to made it download the armv6 wheels instead of armv7

//run on a faster machine docker buildx create --platform linux/arm/v6 --use docker buildx build --platform=linux/arm/v6 -o type=tar,dest=out.tar .

//send “out.tar” file over to the raspberry pi zero w and run cat out.tar | docker import - custom/homeassistant

//custom/homeassistant is the image tag you can use whatever you want…

with this I could install the newer version

Unfortunately I couldn’t find how and if it’s possible to modify it in the main github actions to make it run on raspberry pi zero w directly

EDIT: it’s not necessary to replace ${WHEELS_LINK} because is present in the armhf-homeassitant-base image where the build begins 😉

I’m digging more into the details of the base Image and what I noticed is that the layers of the armhf-homeassistant-base are full of RUN |7 BUILD_ARCH=armhf QEMU_CPU=arm1176 I think that maybe there is a solution for my and many others problem with raspberry’s v6 version

UPDATE: I think I found a way to get the parameter I will make a pull request if it works.

+3
Gerigot on Jan 18, 2023

I’ve tested it on my raspberry pi zero W and it works like a charm I’m not sure about ti other but I think this issue can be closed as resolved.

+2
Gerigot on Jan 22, 2023

@Metaln00b

Already available:

No flags needed

https://github.com/home-assistant/core/pkgs/container/home-assistant/64544700?tag=2023.2.0.dev20230120

Thanks all guys for support !!

+2
dudusandi on Jan 20, 2023

I don’t think there is an issue with the Alpine base image. I think the issue is specific to wheels when building HA core. The linuxserver.io image works for those of us with armv7 CPUs and uses different wheels than the OEM version. Separately @Jojo-1000, seems to have found another issue where the wrong numpy wheel is being collected for armv6.

+2
Wetzel402 on Jan 13, 2023

Okay, I built an image on my QNAP NAS with the following dockerfile…

ARG BUILD_FROM
FROM ghcr.io/home-assistant/armhf-homeassistant-base:2022.11.0

# Synchronize with homeassistant/core.py:async_stop
ENV \
    S6_SERVICES_GRACETIME=220000

WORKDIR /usr/src

## Setup Home Assistant Core dependencies
COPY requirements.txt homeassistant/
COPY homeassistant/package_constraints.txt homeassistant/homeassistant/
RUN \
    apk --update --upgrade add \
  build-base \
  cairo \
  cairo-dev \
  cargo \
  curl \
  freetype-dev \
  gcc \
  gdk-pixbuf-dev \
  gettext \
  jpeg-dev \
  lcms2-dev \
  libffi-dev \
  libxml2-dev \
  libxslt-dev \
  musl-dev \
  openjpeg-dev \
  openssl-dev \
  pango-dev \
  poppler-utils \
  postgresql-client \
  postgresql-dev \
  py-cffi \
  python3-dev \
  rust \
  tcl-dev \
  tiff-dev \
  tk-dev \
  zlib-dev
RUN \
    pip3 install "setuptools<58.0" && \
    pip3 install -v --no-binary=:all: --find-links "${WHEELS_LINKS}" \
    -r homeassistant/requirements.txt --use-deprecated=legacy-resolver
COPY requirements_all.txt home_assistant_frontend-* homeassistant/
RUN \
    if ls homeassistant/home_assistant_frontend*.whl 1> /dev/null 2>&1; then \
        pip3 install -v --no-cache-dir --no-index home-assistant-frontend-*.whl; \
    fi \
    && pip3 install -v --no-cache-dir --no-index --only-binary=:all: --find-links "${WHEELS_LINKS}" \
    -r homeassistant/requirements_all.txt --use-deprecated=legacy-resolver

## Setup Home Assistant Core
COPY . homeassistant/
RUN \
    pip3 install -v --no-binary=:all: --find-links "${WHEELS_LINKS}" \
    -e ./homeassistant --use-deprecated=legacy-resolver \
    && python3 -m compileall homeassistant/homeassistant

# Home Assistant S6-Overlay
COPY rootfs /

WORKDIR /config

This image starts setting up integrations before giving the same errors when it tries to set up domain file_upload.

@samitheberber, I had no idea linuxserver.io published a HA image. I grabbed the latest and can confirm it works. I just updated 6 months worth of HA 👍

+2
Wetzel402 on Jan 5, 2023

I noticed that linuxserver.io is providing own wheels, so got latest version working by switching to their container. I was thinking trying to figure out how to get wheels built by myself, but decided to go with this easy route for now.

+2
samitheberber on Jan 5, 2023

I’ve done some investigation about this issue, the issue is deeper than cryptography and numpy. It is expanded on all packages with compiled (С/C++/Rust) code. When I recompiled packages on my Pi 0W, it seems to work. But what is strange - I tried to build through Github Actions and packages also work from it. I don’t know why packages from Home Assistant wheels index doesn’t work, maybe it was temporary compiler issue (like https://github.com/piwheels/packages/issues/276). Can someone else try to compile numpy or any other failing packages on Github Actions and confirm/refute my theory? If the theory is true, we should ask developers to rebuild all armhf packages and issue will be solved.

+2
AloneLiberty on Oct 22, 2022

Can confirm Raspberry Pi 1b works with 2022.6.7 as well, but it does not work with 2022.7.*

+2
richardzone on Jul 23, 2022

Edited subject to clarify this is a new issue in 2022.7, and the problem persists in 2022.7.5

It’s possible to start python3 itself inside the container, so I’d assume the problem is in some module that gets pulled in by homeassistant:

$ docker ps
CONTAINER ID   IMAGE                                          COMMAND   CREATED              STATUS          PORTS     NAMES
e92eae05a31e   ghcr.io/home-assistant/home-assistant:latest   "/init"   About a minute ago   Up 56 seconds             homeassistant
$ docker exec e92eae05a31e python3 -V
Python 3.10.5
+2
abochmann on Jul 17, 2022

Yes it has been merged almost 20 hours ago but we had to wait until a build was triggered.

That happened 2 hours ago with a development version 2023.2.0.dev20230120 as said by @dudusandi

I will try it and I hope it will be soon present also in a stable version

@Metaln00b if you are using the :dev Image Tag you should stop your container delete the image and then pull the new one. Otherwise you can use the 2023.2.0.dev20230120 image Tag so that you’ll have the specific build that solves the problem.

+1
Gerigot on Jan 20, 2023

It’s gemerged, but what to do when you to compose on :dev? Is a flag necessary?

+1
Metaln00b on Jan 19, 2023

TL;DR

I think the Github Actions are configured wrong to use armv7 instead of armv6 for the wheels packages. From the logs of armhf base core images (2022.12.9): Collecting numpy==1.23.2 #13 85.81 Downloading https://wheels.home-assistant.io/musllinux/numpy-1.23.2-cp310-cp310-musllinux_1_2_armv7l.whl

Long story (for troubleshooting in case that is not it)

I ran into this same issue on raspberry pi 1B rev2. It is very easy to reproduce with the built docker containers:

pi@raspberrypi:~ $ docker exec -ti homeassistant bash
bash-5.1# python
Python 3.10.7 (main, Nov 24 2022, 12:49:35) [GCC 11.2.1 20220219] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
Illegal instruction (core dumped)
bash-5.1#

Because it was suggested that the wheels might be broken, I tried to only pull the wheels directly (my poor pi 1b does not like to compile or download the full requirements). So I created a Dockerfile which only installs the numpy package from the same repository as the Github Actions use and built the container.

Dockerfile:

ARG BUILD_FROM
FROM ghcr.io/home-assistant/armhf-homeassistant-base:2022.11.0@sha256:37b3385d3a10a30344ba73f4d5551b7ecb63aca1606b1586e67f34624f592d6c

# Synchronize with homeassistant/core.py:async_stop
ENV \
    S6_SERVICES_GRACETIME=220000

WORKDIR /usr/src

## Setup Home Assistant Core dependencies
#COPY requirements.txt homeassistant/
#COPY homeassistant/package_constraints.txt homeassistant/homeassistant/

# test install numpy package
RUN pip3 install --no-cache-dir --no-index --only-binary=:all: --find-links "https://wheels.home-assistant.io/musllinux/" --use-deprecated=legacy-resolver numpy

Build output The goal was just to get a simple system where I may have a chance of manually building the single package or swap it to a different source. So I tried to run the import-crash code again and to my surprise it was working:

docker run -ti 0d13df698c3d bash
s6-rc: info: service s6rc-oneshot-runner: starting
s6-rc: info: service s6rc-oneshot-runner successfully started
s6-rc: info: service fix-attrs: starting
s6-rc: info: service fix-attrs successfully started
s6-rc: info: service legacy-cont-init: starting
s6-rc: info: service legacy-cont-init successfully started
s6-rc: info: service legacy-services: starting
s6-rc: info: service legacy-services successfully started
bash-5.1# python
Python 3.10.7 (main, Nov 24 2022, 12:49:35) [GCC 11.2.1 20220219] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
>>> numpy.max([1, 2, 3])
3
>>>

I tried to pull the exact same image that was built in the Actions for that version and compared the numpy versions. That is where I noticed that the Actions are pulling from armv7, while my working version uses armv6:

Not working:

Collecting numpy==1.23.2
#13 85.81   Downloading https://wheels.home-assistant.io/musllinux/numpy-1.23.2-cp310-cp310-musllinux_1_2_armv7l.whl

Working:

Collecting numpy
  Downloading https://wheels.home-assistant.io/musllinux/numpy-1.23.2-cp310-cp310-musllinux_1_2_armv6l.whl (10.2 MB)

This is the same for all of the packages (although I did not test that every single one is working).

So I think this easily explains the illegal instruction errors, unless I am mistaken and the Actions are only used for CI and the uploaded docker images come from somewhere else. I am not very familiar with the Actions workflows, but this should be pretty easy to fix without any code changes and without rebuilding the wheels. I noticed that the architecture everywhere is set to arm in the logs, so maybe that is why it defaults to armv7?

I think you’re right and probably the only thing that is missing in the configuration of the armhf config is an environment variable, QEMU_CPU=arm1176, that will force the build container to think is on armv6 so that the build will just work for armv6 and then in the build for v7 remove that variable.

I’m trying to build an image on my linux computer using docker buildx and send it to my raspberry pi zero w if it works I will update this message.

+1
Gerigot on Jan 12, 2023

@Jojo-1000, my QNAP NAS is arm v7 so that makes sense then why its working for me. I wonder if the linuxserver.io image has “good” wheels but is using the wrong numpy version like you observed for the HA in house image.

+1
Wetzel402 on Jan 5, 2023

Bricked my native python install trying to upgrade and interestingly it was getting stuck on building cryptography (for some reason no wheel available) - likely related to the rust dependency: https://github.com/piwheels/packages/issues/276#issue-1105998174

Cryptography which broke a lot of builds and wheels last year: https://github.com/pyca/cryptography/issues/5771

So now fresh install, trying docker and came across this issue

interestingly jwt imports cryptography https://github.com/GehirnInc/python-jwt/blob/fd684745b2de884a32e6d7a423e659e4d262fb27/setup.py#L27

Although I cant find that numpy does

+1
HarvsG on Sep 21, 2022

Interestingly this issue is also referenced in the wheels issue 276. Issue 276 and the numpy issue were both closed around January 17 but I’m not seeing a clear commit in the commit history.

As @kuznetsss pointed out, HA has it’s own wheels it would be worth while to verify the fix implemented Jan 17 is also in the HA variant.

+1
Wetzel402 on Sep 12, 2022

Also can confirm that import numpy inside container still causes a crash for version 2022.9.1 on Odroid C2 (aarch64 CPU)

bash-5.1# python
Python 3.10.5 (main, Jun 22 2022, 16:10:59) [GCC 11.2.1 20220219] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
Illegal instruction (core dumped)

I found the issue https://github.com/numpy/numpy/issues/18131 in numpy repo but it has been fixed.

+1
kuznetsss on Sep 9, 2022

So the way I understand it is this: piwheels is a repository offering all sorts of pre-compiled Python packages, aka wheels, specifically compiled for Raspis. piwheels.org’s cloud provider, Mythic Beasts Pi, updated their OS images. This update contained a gcc version which, when compiling Python packages for piwheels.org repositories, would produce binaries that are incompatible with armv6 Raspis, e.g. Raspi 1 and Raspi 2. This issue has bin fixed more than half a year ago, and all affected wheels have since been rebuilt.

I found that Home Assistant has it’s own wheels repo https://github.com/home-assistant/wheels So maybe if the problem is related to wheels it would be useful to create an issue in that repo. What do you think?

+1
kuznetsss on Sep 9, 2022

I am still seeing a boot loop as of version 2022.9.

+1
Wetzel402 on Sep 8, 2022

Importing crypto libs seems so be the problem:

Python 3.10.5 (main, Jun 22 2022, 16:42:00) [GCC 11.2.1 20220219] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from cryptography.hazmat.primitives.asymmetric import ec, padding
Segmentation fault

But this is wrapped in a try/except block… so it might be a red herring: https://github.com/jpadilla/pyjwt/blob/0bef0fbff5c245668578a43774d8620bdba4a6f7/jwt/algorithms.py#L18-L57

+1
larsxschneider on Aug 6, 2022

According to the debugger it breaks here: https://github.com/home-assistant/core/blob/2022.8.1/homeassistant/__main__.py#L69

bash-5.1# python3 -m pdb -m homeassistant --config /config
> /usr/src/homeassistant/homeassistant/__main__.py(1)<module>()
-> """Start Home Assistant."""
(Pdb) break /usr/src/homeassistant/homeassistant/__main__.py:69
Breakpoint 1 at /usr/src/homeassistant/homeassistant/__main__.py:69
(Pdb) c
> /usr/src/homeassistant/homeassistant/__main__.py(69)get_arguments()
-> from . import config as config_util
(Pdb) n
Segmentation fault

Stepping into this call just gives me “frozen” gibberish:

-> from . import config as config_util
(Pdb) s
--Call--
> <frozen importlib._bootstrap>(404)parent()
(Pdb) s
> <frozen importlib._bootstrap>(407)parent()
(Pdb) s
> <frozen importlib._bootstrap>(408)parent()
(Pdb) s

But poking around the config utils I found that import jwt creates a segfault: https://github.com/home-assistant/core/blob/2022.8.1/homeassistant/auth/__init__.py#L10

Does this make any sense to someone? 😄

+1
larsxschneider on Aug 6, 2022
Read more comments on GitHub
← core: 2022.7.x causes issue with IMAP sensor
core: [2022.8.1] ONVIF integration failing with "Couldn't connect to camera x, please verify that the credentials are correct. Error: Invalid data" →