minikube: Starting Minikube messes up binfmt config for Docker, making it unable to run cross-architecture images

What Happened?

I was planning using Minikube with docker driver, on an Apple M1 chip.

My Docker Engine, when freshly started supports the following emulators:

% docker run --privileged --rm tonistiigi/binfmt
{
  "supported": [
    "linux/arm64",
    "linux/amd64",
    "linux/386",
    "linux/arm/v7",
    "linux/arm/v6"
  ],
  "emulators": [
    "mac-macho-arm64",
    "mac-macho-x86_64",
    "mac-universal-arm64",
    "mac-universal-x86_64",
    "qemu-arm",
    "qemu-i386",
    "qemu-x86_64",
    "rosetta",
    "rosetta​",
    "rosetta​​"
  ]
}
% minikube start
😄  minikube v1.32.0 on Darwin 14.1.1 (arm64)
✨  Automatically selected the docker driver. Other choices: qemu2, virtualbox, ssh
📌  Using Docker Desktop driver with root privileges
👍  Starting control plane node minikube in cluster minikube
🚜  Pulling base image ...
🔥  Creating docker container (CPUs=2, Memory=8100MB) ...
🐳  Preparing Kubernetes v1.28.3 on Docker 24.0.7 ...
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔗  Configuring bridge CNI (Container Networking Interface) ...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🔎  Verifying Kubernetes components...
🌟  Enabled addons: storage-provisioner, default-storageclass
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default
% docker run --privileged --rm tonistiigi/binfmt
{
  "supported": [
    "linux/arm64"
  ],
  "emulators": [
    "python3.10"
  ]
}

Minikube messes up the binfmt config, so from that point on, it’s only possible to run images of the same architecture as minikube itself (in my case linux/arm64).

This invasive behaviour is a showstopper for me to use Minikube, despite it being lovely and I’d really like to switch my K3S based stack to Minikube.

My attempts to find a workaround, or try to restore the Binfmt settings manually failed, so reaching out with an issue.

Attach the log file

log.txt

Operating System

macOS (Default)

Driver

Docker

About this issue

  • Original URL
  • State: open
  • Created 7 months ago
  • Reactions: 1
  • Comments: 18 (3 by maintainers)

Most upvoted comments

@fbuetler @sugymt I moved removing the file until after installing systemd, if you would mind retrying with --base-image gcr.io/k8s-minikube/kicbase-builds:v0.0.42-1703092832-17830

@fbuetler @sugymt Could you try with --base-image gcr.io/k8s-minikube/kicbase-builds:v0.0.42-1702948046-17830 and see if my fix resolves it

There is no need to install python binfmt for minikube, so we should just remove this “feature” from ubuntu

Then we don’t to restore any pre-existing config from the docker host that was wiped, including WSL (or Rosetta)

Running into a similar issue here. My setup is on a Mac M2 using lima + vz + rosetta for the Docker VM.

With minikube 1.32.0 (with docker driver), once I start minikube it somehow breaks rosetta in the lima VM. Same configuration works fine on minikube 1.31.2.