linuxkit: cannot boot from usb iso image on bare metal

Description The computer that is booting off the usb drive is unable to mount the root filesystem during the boot sequence

Steps to reproduce the issue:

git clone github.com/linuxkit
cd linuxkit/examples
mkdir wireguard && mv wireguard.yml wireguard/
linuxkit build --format iso-bios wireguard.yml
Insert 16 GB usb flash drive into computer
sudo dd if=wireguard.iso of=/dev/rdisk2 bs=1m
Take out flash drive when complete
Put flash drive into motherboard or computer capable of booting from usb drive
Power on and watch boot sequence

Describe the results you received: Kernel panic - not syncing - VFS: Unable to mount root fs, unknown block(0,0)

Describe the results you expected: I expected to see the text below during the boot.

    1.740835] Freeing unused kernel memory: 1512K
[    1.763587] Write protecting the kernel read-only data: 18432k
[    1.763587] Write protecting the kernel read-only data: 18432k
[    1.765206] Freeing unused kernel memory: 2024K
[    1.765206] Freeing unused kernel memory: 2024K
[    1.768204] Freeing unused kernel memory: 1100K
[    1.768204] Freeing unused kernel memory: 1100K

Welcome to LinuxKit

                        ##         .
                  ## ## ##        ==
               ## ## ## ## ##    ===
           /"""""""""""""""""\___/ ===
          {                       /  ===-
           \______ O           __/
             \    \         __/
              \____\_______/

[    2.346960] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[    2.346960] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[    2.552594] clocksource: Switched to clocksource tsc
[    2.552594] clocksource: Switched to clocksource tsc
[    2.723036] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[    2.723036] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready

Additional information you deem important (e.g. issue happens only occasionally):

Works fine when booting from hyperkit and virtual machine (vmware fusion). Just not bare metal.

Bare metal machine boots from other live images without issue.

The usb drive is the only storage device in the system.

Wireguard Config

kernel:
  image: linuxkit/kernel:4.14.58
  cmdline: "console=tty0 console=ttyS0 console=ttyAMA0"
init:
  - linuxkit/init:v0.6
  - linuxkit/runc:v0.6
  - linuxkit/containerd:v0.6
  - linuxkit/ca-certificates:v0.6
onboot:
  - name: sysctl
    image: linuxkit/sysctl:v0.6
  - name: dhcpcd
    image: linuxkit/dhcpcd:v0.6
    command: ["/sbin/dhcpcd", "--nobackground", "-f", "/dhcpcd.conf", "-1"]
  - name: wg0
    image: linuxkit/ip:v0.6
    net: new
    binds:
      - /etc/wireguard:/etc/wireguard
    command: ["/usr/bin/lk-wg-config", "configure", "/etc/wireguard/wg0.conf"]
    runtime:
      interfaces:
        - name: wg0
          add: wireguard
          createInRoot: true
      bindNS:
          net: /run/netns/wg0
  - name: wg1
    image: linuxkit/ip:v0.6
    net: new
    binds:
      - /etc/wireguard:/etc/wireguard
    command: ["/usr/bin/lk-wg-config", "configure", "/etc/wireguard/wg1.conf"]
    runtime:
      interfaces:
        - name: wg1
          add: wireguard
          createInRoot: true
      bindNS:
          net: /run/netns/wg1
services:
  - name: getty
    image: linuxkit/getty:v0.6
    env:
     - INSECURE=true
    net: /run/netns/wg1
  - name: rngd
    image: linuxkit/rngd:v0.6
  - name: nginx
    image: nginx:1.13.8-alpine
    net: /run/netns/wg0
    capabilities:
     - CAP_NET_BIND_SERVICE
     - CAP_CHOWN
     - CAP_SETUID
     - CAP_SETGID
     - CAP_DAC_OVERRIDE
files:
  - path: etc/wireguard/wg0.conf
    contents: |
      [Interface]
      Address = 192.168.2.1/24
      DNS = 192.168.2.2
      PrivateKey = KG7EKkHDkp7THfW5mOXcZzo3RbGAjq+ARMfJGFm5G1s=
      ListenPort = 51820
      [Peer]
      PublicKey = ZP7h1OjA1oIAmsAvMKujp2RAAC/f0kY814b3Xq6j/0Y=
      AllowedIPs = 0.0.0.0/0
      Endpoint = 127.0.0.1:51821
  - path: etc/wireguard/wg1.conf
    contents: |
      [Interface]
      Address = 192.168.2.2/24
      PrivateKey = SCGCKDuTm4PMOw+LXdK/2s8mxnv145QHOohKRq3vc2A=
      ListenPort = 51821
      [Peer]
      PublicKey = AcS5t3PC5nL/oj0sYhc3yFpDlRaXoJ0mfEq6iq0rFF4=
      AllowedIPs = 0.0.0.0/0
      Endpoint = 127.0.0.1:51820
trust:
  org:
    - linuxkit
    - library

About this issue

Original URL
State: open
Created 6 years ago
Comments: 23 (10 by maintainers)

Most upvoted comments

FWIW, I’ve got this working by recompiling the linuxkit kernel and enabling the following options (among others, but I think these are the most likely candidates):

CONFIG_USB_XHCI_HCD=y
CONFIG_USB_XHCI_PCI=y
CONFIG_USB_XHCI_PLATFORM=y
CONFIG_USB_EHCI_HCD=y
CONFIG_USB_EHCI_ROOT_HUB_TT=y
CONFIG_USB_OHCI_HCD=y
CONFIG_USB_OHCI_HCD_PCI=y
CONFIG_USB_OHCI_HCD_PLATFORM=y
CONFIG_USB_UHCI_HCD=y
CONFIG_USB_STORAGE=y

All the above options are compiled as modules by default, which doesn’t help when the kernel itself is still booting (and in linuxkit AFAICT there’s no initrd either).

In my case the boot was from a virtual IPMI media, which apparently is exposed as XHCI_HCD device. Building the driver in-kernel made the kernel recognize /dev/sr0 and the boot could continue.

If I can suggest, I’d enable those options as kernel builtins by default for the future, as I think the scenario I’ve described is not unlikely when deploying linuxkit.

waldner on May 10, 2022

@jcudit that looks to be the same issue you captured. I captured the output here

I haven’t tried anything different since the creation of this issue. Many new kernel updates have been released since I last posted so I’ll give it a retry and post an update.

Might It it also be some bios setting that affects how usb storage is detected?

logicminds on Jan 28, 2019

I’m having a similar issue and have captured the console output. See attached video.

It seems as if /dev/sr0 is not available at the time that the boot process expects to access the root filesystem located on the kernel+iso output file. The video lists only /dev/ram* devices as available.

I’ve tried adding root=/dev/ram0 and other variants but it eventually errors out as there is no filesystem to be read. I think the fix here will be to figure out how to make /dev/sr0 accessible at such an early stage of the boot sequence.

ISO oops.mp4.gz

@logicminds thanks for opening this. I don’t want to hijack your thread here, so can you confirm if what is shown in the video is the same issue as what you’ve encountered?

@rn hopefully this video is all that is needed to diagnose the issue. Let me know if there is anything further you’d like to see and I’ll arrange for it to be posted to this ticket within the week.

jcudit on Oct 1, 2018