pi-gen: Buster builds broken on non-arm hosts

So as per @Chaser’s comment, one way to produce a working image from a 64-bit host is to use ./build-docker.sh with this modification to the Dockerfile:

diff --git a/Dockerfile b/Dockerfile
index 706a5fb..cf9aac4 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -1,4 +1,4 @@
-FROM debian:buster
+FROM i386/debian:buster
 
 ENV DEBIAN_FRONTEND noninteractive

@Chaser @rkubes thanks for your input, really appreciated! I now tested it myself too and can also confirm that indeed is just Qemu that needs to be 32 bits, not the kernel of the host system.

I used the included Dockerfile from pi-gen and used the i386/debian:buster base image instead to bring 32 bits binaries (including Qemu) as @Chaser suggested. Worked fine on actual RPI.

Not sure if there’s any other tests that we can do to prove there are no other issues.

It is not clear at the moment how to test 100% reliably, however the SSL certificates test is a very good indicator as far as I can tell because it provides a tangible control case.

I will send a PR to update the included Dockerfile. @Chaser thanks a lot again for your input, I didn’t know there were i386 images for Docker out there, I would have tested that for sure otherwise.

EDIT: @ryanteck might be interested. You don’t need to setup a VM nor a 32-bits kernel for your host build system, just make sure you are installing the i386 version of Qemu in multiarch.

@hhromic - Clean execution of todays pi-gen mainline https://github.com/RPi-Distro/pi-gen/commit/114353035142d43798a564a552d1c21fa3e91935

As is pulls down qemu-user-static amd64

Get:103 http://cdn-fastly.deb.debian.org/debian buster/main amd64 qemu-user-static amd64 1:3.1+dfsg-8~deb10u1 [21.1 MB]

Changing to FROM i386/debian:buster

Get:103 http://cdn-fastly.deb.debian.org/debian buster/main i386 qemu-user-static i386 1:3.1+dfsg-8~deb10u1 [22.5 MB]

The build completed successfully. Hopefully this helps.

Hi all, we were able to overcome the issue of SSL certs only by rehashing the ssl certs with c_rehash

Specifically

# Patch Issue with ssl certs as per https://github.com/RPi-Distro/pi-gen/issues/271
on_chroot << EOF
echo Patching certs...
c_rehash /etc/ssl/certs
EOF

Credit to Keith Tate as well 😃

As indicated below by @hhromic its not a complete solution.

Maybe my solution is useful for some people: I have decided to take the approach of using Vagrant with Virtualbox because of the fact I can contain everything in a quite portable vm with all dependencies inside (including proxy-cache for the packages). Just create a file Vagrantfile in the root of pi-gen repo like this:

$run = <<"SCRIPT"
echo ">>> Generating rpi image ... $@"
export DEBIAN_FRONTEND=noninteractive
export RPIGEN_DIR="${1:-/home/vagrant/rpi-gen}"
export APT_PROXY='http://127.0.0.1:3142' 
# Prepare. Copy the repo to another location to run as root
rsync -a --delete --exclude 'work' --exclude 'deploy' /vagrant/  ${RPIGEN_DIR}/
cd ${RPIGEN_DIR}
# Clean previous builds. Start always from scratch (the proxy helps here!)
sudo umount --recursive work/*/stage*/rootfs/{dev,proc,sys} || true
# Delete old builds
sudo rm -rf work/*
# Build it again
sudo --preserve-env=APT_PROXY ./build.sh
# Copy images back to host
[ -d deploy ] && cp -vR deploy /vagrant/
SCRIPT

Vagrant.configure("2") do |config|
  # All Vagrant configuration is done here. The most common configuration
  # options are documented and commented below. For a complete reference,
  # please see the online documentation at vagrantup.com.  

  config.vm.define :rpigen do |rpigen|
      # Every Vagrant virtual environment requires a box to build off of.
      rpigen.vm.box = "jriguera/rpibuilder-buster-10.0-i386"
      rpigen.vm.provision "shell" do |s|
        s.inline = $run
        s.args = "#{ENV['WORK_DIR']}"
      end
  end
end

and run vagrant up . It will start downloading the Virtualbox base image (based on Debian Buster i386) and after done, it will run the build.sh script of the repo. Once done, it will put the images in the deploy folder. If the process fails, you can run again with vagrant provision. vagrant destroy will delete the vm and its contents. The source is here: https://github.com/jriguera/packer-rpibuilder-vagrant so you can customize it and create your own Raspbian builder vm.

@hhromic I usually run my builds in a VM running Ubuntu 18.04. I don’t use docker. I can confirm using the 64-bit 18.04 I will get the SSL error message. I then changed nothing else other than installing the 32-bit qemu-user-static package and rebuilt the image. Once deployed, I was able to get HTML content without any SSL error messages. All that to say, I also think having the 32-bit qemu is all that’s needed. Not sure if there’s any other tests that we can do to prove there are no other issues.

@Chaser that is a workaround for the ca-certificates package only, not a solution to the actual problem described in this issue. I wouldn’t advice advertising it as a solution.

The problem affects any program using the readdir() syscall, not just ca-certificates and the effects are of varying nature. The SSL certificates issue is just one manifestation. Another (as the original post indicated) are icons not being correctly rendered in the Desktop.

It is unknown/unverified what other packages might be affected. Therefore the safest solution for now is to build using a 32-bits host (be ARM or non-ARM) as indicated before.

Hi @XECDesign , so I can confirm now that this issue is specifically for hosts with 64 bits kernels no matter if arm or not. I made a testing VM with vanilla Debian i386 (32 bits kernel) and the generated image works fine on real hardware (tested with RPI 3B).

To verify this I built a control “broken” Buster image with a 64 bits Debian host using Docker and I did get SSL problems with curl (with GitHub and other websites too), for example:

$ curl -sSL https://github.com
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

Then I built another Buster image using a 32 bits Debian host and the curl command above worked fine on the same hardware and same network almost at the same time.

Aside, I also noticed a minor issue with the Qemu version shipped with Debian Stretch, where the man-db package being installed for Buster in the image triggers many of these errors:

qemu: Unsupported syscall: 383

This is a manifestation of the following bug: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=891109

Fortunately this is resolved in the current Qemu version shipped with Debian Buster, therefore building with a Debian Buster host will not show any errors. I will send a PR to update the Dockerfile for this.

In summary, I wanted to let you know that this bug is not affecting 32 bits build hosts, no matter if they are arm or not (at least for me). For now you can use a 32 bits build host and pi-gen will generate a working image.

Hope this is useful for future readers!

Depending on the version of qemu and whether you’re using docker, server images might not exhibit issues, but I still wouldn’t recommend it until qemu is fixed.

There has been some good progress upstream. One of the issues has been fixed and another has a fix in the pipeline (https://gitlab.com/qemu-project/qemu/-/issues/633). Not sure how long it will be before there’s an official release with both fixes.

EDIT: I should mention that the fixes only make it work with i686 qemu. amd64 still won’t work, but that seems to be a glibc and/or kernel issue that might not be fixable. I’m not sure what the current state of that is.

Just an update for someone getting here:

I confirm that I just built an arm64 bullseye lite (stage2) image using build.sh on the arm64 branch of this repo. My machine is an amd64 debian bullseye. After dumping the image on a real RPi4 and running curl -sSL https://github.com I got proper html code (so I guess it is indeed fixed)

Just hit this issue while doing something else and the previous workaround didn’t work. It looks like at least on some distributions it’s no longer necessary to copy the qemu binary into the chroot. In my case, it was using the system’s qemu binary rather than the 32bit one I was copying into the chroot.

The workaround I’m using right now is to override the qemu path binfmt uses with a local 32bit copy (edit as appropriate in your case):

if [ "$(dpkg --print-architecture)-$ARCH" = "amd64-armhf" ] && [ ! -e /proc/sys/fs/binfmt_misc/sbuild-arm ]; then
	echo ":sbuild-arm:M::\x7f\x45\x4c\x46\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x28\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:$(realpath "qemu/qemu-arm-static"):OCF" > /proc/sys/fs/binfmt_misc/register
fi

Then to remove that override:

if [ -e /proc/sys/fs/binfmt_misc/sbuild-arm ]; then
	echo "-1" > /proc/sys/fs/binfmt_misc/sbuild-arm
fi

Edit: It looks like this is the relevant change that causes the different behaviour: https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1815100

So, maybe that override with the F flag is a good approach for pi-gen to use in general, to avoid even having to copy the binary in the first place. Pi-gen would just need to check that multiarch support is enabled and that the dependencies are installed, then fetch the binary from somewhere else - debian or ubuntu repos.

@hhromic - was using Codebuild docker image - https://github.com/aws/aws-codebuild-docker-images/blob/master/ubuntu/standard/2.0/Dockerfile

uname -a output:

Linux 046536f42b8e 4.14.123-86.109.amzn1.x86_64 #1 SMP Mon Jun 10 19:44:53 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

@hhromic - understood, curl works as expected. HTML content received.

Thanks @hhromic updated my comment to be clear its not a solution.

Are there tests that should be done to confirm issues? I have just built an image on a EC2 ARM (64bit) instance (a1.2xlarge) running ubuntu 18.04 LTS. I would like to do some sanity checks on it.

Thanks for looking into it. Much appreciated.