meta-raspberrypi: Kernel modules not loaded during boot

Description Observed that some kernel modules are not being loaded in the latest kernel 5.15.34-v7.

So I have built a core-image-base with the tip of the master (0135a02) and while trying access the camera using Picamera got some errors. The errors mainly complain about mmal drivers not present.

root@raspberrypi3:~# python3
Python 3.10.4 (main, Mar 23 2022, 20:25:24) [GCC 11.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from picamera import PiCamera
>>> camera = PiCamera()
mmal: mmal_vc_shm_init: could not initialize vc shared memory service
mmal: mmal_vc_component_create: failed to initialise shm for 'vc.camera_info' (7:EIO)
mmal: mmal_component_create_core: could not create component 'vc.camera_info' (7)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.10/site-packages/picamera/camera.py", line 408, in __init__
    self._init_revision(options)
  File "/usr/lib/python3.10/site-packages/picamera/camera.py", line 480, in _init_revision
    with mo.MMALCameraInfo() as camera_info:
  File "/usr/lib/python3.10/site-packages/picamera/mmalobj.py", line 2425, in __init__
    super(MMALCameraInfo, self).__init__()
  File "/usr/lib/python3.10/site-packages/picamera/mmalobj.py", line 696, in __init__
    mmal_check(
  File "/usr/lib/python3.10/site-packages/picamera/exc.py", line 184, in mmal_check
    raise PiCameraMMALError(status, prefix)
picamera.exc.PiCameraMMALError: Failed to create MMAL component b'vc.camera_info': I/O error
>>> 
>>> 
root@raspberrypi3:~#

After digging through my system found an older build (don’t know why didn’t delete it but thankfully it gave some insight into the issue), I tried booting that image and everything seems to working fine.

So I checked out to the commit which the older build was using (63a3d8cb17c5d1affe8f2848f45fcc6a706f9412), and the camera worked fine(though I had to make few changes, which are not significant for this issue). While analyzing the bootlogs found that the latest build (0135a02) doesn’t load all the drivers.

Also I have observed that the kernel module are compressed in the 5.15.34 kernel, eg: root@raspberrypi3:~# ls /lib/modules/5.15.34-v7/kernel/drivers/usb/gadget/libcomposite.ko.xz and while trying load the modules using modprobe getting the following error:

root@raspberrypi3:~# ls /lib/modules/5.15.34-v7/kernel/drivers/usb/gadget/legacy/
g_acm_ms.ko.xz        g_cdc.ko.xz           g_hid.ko.xz           g_midi.ko.xz          g_printer.ko.xz       g_webcam.ko.xz        gadgetfs.ko.xz
g_audio.ko.xz         g_ether.ko.xz         g_mass_storage.ko.xz  g_multi.ko.xz         g_serial.ko.xz        g_zero.ko.xz
root@raspberrypi3:~# modprobe gadgetfs
modprobe: FATAL: Module gadgetfs not found in directory /lib/modules/5.15.34-v7

My question is what and where the changes have happened to the kernel between 63a3d8cb17c5d1affe8f2848f45fcc6a706f9412 (5.10) and 0135a02 (5.15) , so that I can look into and adapt the changes required ?

Note: All the commit which are mentioned above are of this repo.

Logs I have attached the bootlogs of both the kernels. lsmod logs

5.15.34

root@raspberrypi3:~# lsmod
Module                  Size  Used by
root@raspberrypi3:~#

5.10.81

root@raspberrypi3:~# lsmod
Module                  Size  Used by
rfcomm                 49152  2
cmac                   16384  3
algif_hash             16384  1
nfc                    86016  0
aes_arm_bs             24576  2
crypto_simd            16384  1 aes_arm_bs
cryptd                 24576  2 crypto_simd
algif_skcipher         16384  1
af_alg                 28672  6 algif_hash,algif_skcipher
bnep                   20480  2
hci_uart               40960  1
btbcm                  16384  1 hci_uart
bluetooth             421888  31 hci_uart,bnep,btbcm,rfcomm
ecdh_generic           16384  2 bluetooth
ecc                    36864  1 ecdh_generic
ipv6                  503808  26
brcmfmac              331776  0
brcmutil               24576  1 brcmfmac
sha256_generic         16384  0
bcm2835_v4l2           49152  0
cfg80211              782336  1 brcmfmac
bcm2835_codec          40960  0
bcm2835_isp            32768  0
v4l2_mem2mem           36864  1 bcm2835_codec
rfkill                 32768  4 bluetooth,nfc,cfg80211
bcm2835_mmal_vchiq     36864  3 bcm2835_isp,bcm2835_codec,bcm2835_v4l2
videobuf2_dma_contig    20480  2 bcm2835_isp,bcm2835_codec
videobuf2_vmalloc      16384  1 bcm2835_v4l2
videobuf2_memops       16384  2 videobuf2_dma_contig,videobuf2_vmalloc
videobuf2_v4l2         32768  4 bcm2835_isp,bcm2835_codec,bcm2835_v4l2,v4l2_mem2mem
videobuf2_common       61440  5 bcm2835_isp,bcm2835_codec,bcm2835_v4l2,v4l2_mem2mem,videobuf2_v4l2
raspberrypi_hwmon      16384  0
videodev              253952  6 bcm2835_isp,bcm2835_codec,videobuf2_common,bcm2835_v4l2,v4l2_mem2mem,videobuf2_v4l2
mc                     45056  6 bcm2835_isp,bcm2835_codec,videobuf2_common,videodev,v4l2_mem2mem,videobuf2_v4l2
vc_sm_cma              32768  2 bcm2835_isp,bcm2835_mmal_vchiq
uio_pdrv_genirq        16384  0
uio                    20480  1 uio_pdrv_genirq
fixed                  16384  0
root@raspberrypi3:~#

Additional details (revisions used, host distro, etc.):

Some more details of the builds

meta-openembedded commit hash : 64156a6f8f3b4527c4732c91936aee0a167acb48
poky commit hash : 5ec6839709f1f9411a8334306c563675246fff35
local.conf

MACHINE = "raspberrypi3"
VIDEO_CAMERA = "1"
ENABLE_UART = "1"
RPI_USE_U_BOOT = "1"
GPU_FREQ = "250"
GPU_MEM = "128"

IMAGE_INSTALL:append = "python3-picamera openssh libcamera libffi libgphoto2"

bblayers.conf

BBLAYERS ?= " \
  /home/bulbasaur/HDD-2-2/RPI-camera-project-23042022/poky/meta \
  /home/bulbasaur/HDD-2-2/RPI-camera-project-23042022/poky/meta-poky \
  /home/bulbasaur/HDD-2-2/RPI-camera-project-23042022/poky/meta-yocto-bsp \
  /home/bulbasaur/HDD-2-2/RPI-camera-project-23042022/meta-raspberrypi \
  /home/bulbasaur/HDD-2-2/RPI-camera-project-23042022/meta-openembedded/meta-python \
  /home/bulbasaur/HDD-2-2/RPI-camera-project-23042022/meta-openembedded/meta-oe \
  /home/bulbasaur/HDD-2-2/RPI-camera-project-23042022/meta-openembedded/meta-multimedia \
  "

bootlog-v5_15_34-kernel.txt bootlog-v5_10_18-kernel.txt

About this issue

Original URL
State: closed
Created 2 years ago
Comments: 21 (14 by maintainers)

Commits related to this issue

kmod: Enable xz compression The RaspberryPi default Linux kernel configuration uses xz for compressing kernel modules[1]. This was introduced in 5.15. We need the respective support in kmod too. [1]... — committed to agherzan/meta-raspberrypi by agherzan 2 years ago
kmod: Enable xz compression The RaspberryPi default Linux kernel configuration uses xz for compressing kernel modules[1]. This was introduced in 5.15. We need the respective support in kmod too. [1]... — committed to agherzan/meta-raspberrypi by agherzan 2 years ago
Revert "kmod: Enable xz compression" This reverts commit c97a9e34abf23d871903481ac1867efec9abf090. — committed to agherzan/meta-raspberrypi by deleted user 2 years ago
kmod: Enable xz compression The RaspberryPi default Linux kernel configuration uses xz for compressing kernel modules[1]. This was introduced in 5.15. We need the respective support in kmod too. [1]... — committed to onsolutionjames/meta-raspberrypi by agherzan 2 years ago

Most upvoted comments

tl;dr: PACKAGECONFIG:append:pn-kmod = " xz"

While I can’t answer your question regarding the specific commit which introduced the changes to the kernel I might be able to help you loading the kernel modules.

I was in a similar situation and no kernel modules had been loaded after building a new image with latest openembedded-core, meta-openembedded, and meta-raspberrypi versions.

In my case there was no output on my screen but the root cause was the missing kernel modules.

Analysis

root@raspberrypi4-64:~# lsmod
Module                  Size  Used by

root@raspberrypi4-64:~# find /lib/modules/5.15.34-v8/kernel -type f | head
/lib/modules/5.15.34-v8/kernel/drivers/connector/cn.ko.xz
/lib/modules/5.15.34-v8/kernel/drivers/w1/masters/ds2490.ko.xz
/lib/modules/5.15.34-v8/kernel/drivers/w1/masters/w1-gpio.ko.xz
/lib/modules/5.15.34-v8/kernel/drivers/w1/masters/ds1wm.ko.xz
/lib/modules/5.15.34-v8/kernel/drivers/w1/masters/ds2482.ko.xz
/lib/modules/5.15.34-v8/kernel/drivers/w1/slaves/w1_ds2438.ko.xz
/lib/modules/5.15.34-v8/kernel/drivers/w1/slaves/w1_ds28e04.ko.xz
/lib/modules/5.15.34-v8/kernel/drivers/w1/slaves/w1_ds2431.ko.xz
/lib/modules/5.15.34-v8/kernel/drivers/w1/slaves/w1_smem.ko.xz
/lib/modules/5.15.34-v8/kernel/drivers/w1/slaves/w1_ds2781.ko.xz

root@raspberrypi4-64:~# kmod --version
kmod version 29
-ZSTD -XZ +ZLIB -LIBCRYPTO -EXPERIMENTAL

root@raspberrypi4-64:~# which kmod
/bin/kmod

Notice that XZ is disabled for kmod.

Find the recipe providing /bin/kmod:

$ oe-pkgdata-util find-path /bin/kmod
kmod: /bin/kmod

Ok, could have guessed that it is kmod.

Relevant line in the recipe: https://github.com/openembedded/openembedded-core/blob/5e7d09142da82c37aeab22c34d5314187c90bd84/meta/recipes-kernel/kmod/kmod_29.bb#L35

Fix

Enable XZ for kmod:

$ echo 'PACKAGECONFIG:append:pn-kmod = " xz"' >> meta-foo/conf/distro/foo-distro.conf

After building a new image XZ was enabled for kmod but the kernel modules still haven’t been loaded:

root@raspberrypi4-64:~# kmod --version
kmod version 29
-ZSTD +XZ +ZLIB -LIBCRYPTO -EXPERIMENTAL

root@raspberrypi4-64:~# lsmod
Module                  Size  Used by

However, after generating modules.dep and map files followed by a reboot the kernel modules where finally loaded.

root@raspberrypi4-64:~# depmod --all
root@raspberrypi4-64:~# reboot
root@raspberrypi4-64:~# lsmod
Module                  Size  Used by
rfcomm                 53248  2
cmac                   16384  3
algif_hash             20480  1
aes_arm64              16384  3
algif_skcipher         20480  1
af_alg                 32768  6 algif_hash,algif_skcipher
bnep                   28672  2
hci_uart               49152  1
btbcm                  28672  1 hci_uart
bluetooth             483328  31 hci_uart,btbcm,bnep,rfcomm
ecdh_generic           16384  2 bluetooth
ecc                    36864  1 ecdh_generic
rpivid_mem             16384  0
brcmfmac              339968  0
bcm2835_codec          53248  0
bcm2835_v4l2           45056  0
bcm2835_isp            32768  0
brcmutil               24576  1 brcmfmac
v3d                    86016  5
v4l2_mem2mem           45056  1 bcm2835_codec
bcm2835_mmal_vchiq     40960  3 bcm2835_codec,bcm2835_v4l2,bcm2835_isp
videobuf2_dma_contig    24576  2 bcm2835_codec,bcm2835_isp
gpu_sched              49152  1 v3d
videobuf2_vmalloc      20480  1 bcm2835_v4l2
videobuf2_memops       20480  2 videobuf2_vmalloc,videobuf2_dma_contig
videobuf2_v4l2         32768  4 bcm2835_codec,bcm2835_v4l2,v4l2_mem2mem,bcm2835_isp
raspberrypi_hwmon      16384  0
i2c_brcmstb            20480  0
videobuf2_common       69632  8 bcm2835_codec,videobuf2_vmalloc,videobuf2_dma_contig,videobuf2_v4l2,bcm2835_v4l2,v4l2_mem2mem,videobuf2_memops,bcm2835_isp
videodev              266240  6 bcm2835_codec,videobuf2_v4l2,bcm2835_v4l2,videobuf2_common,v4l2_mem2mem,bcm2835_isp
vc_sm_cma              36864  2 bcm2835_mmal_vchiq,bcm2835_isp
mc                     61440  6 videodev,bcm2835_codec,videobuf2_v4l2,videobuf2_common,v4l2_mem2mem,bcm2835_isp
hid_lg_g15             20480  0
uio_pdrv_genirq        16384  0
cfg80211              876544  1 brcmfmac
uio                    24576  1 uio_pdrv_genirq
nvmem_rmem             16384  0
rfkill                 36864  4 bluetooth,cfg80211
sch_fq_codel           20480  6
fuse                  139264  1
ipv6                  561152  38

MatthiasGugel on Apr 26, 2022