edgetpu: Building gasket-dkms broken on linux kernel 6.5.0 (Ubuntu 23.10)

Description

Just upgraded my machines with PCIe TPUs from Ubuntu 23.04 to 23.10, upgrading from kernel 6.2 to 6.5.0.

apt/dpkg output:

...
Setting up python3-distupgrade (1:23.10.10) ...
Setting up gasket-dkms (1.0-18) ...
debconf: unable to initialize frontend: Dialog
debconf: (No usable dialog-like program is installed, so the dialog based frontend cannot be used. at /usr/share/perl5/Debconf/FrontEnd/Dialog.pm line 78.)
debconf: falling back to frontend: Readline
Removing old gasket-1.0 DKMS files...
Deleting module gasket-1.0 completely from the DKMS tree.
Loading new gasket-1.0 DKMS files...
Deprecated feature: REMAKE_INITRD (/usr/src/gasket-1.0/dkms.conf)
Building for 6.5.0-10-generic
Building initial module for 6.5.0-10-generic
Deprecated feature: REMAKE_INITRD (/var/lib/dkms/gasket/1.0/source/dkms.conf)
ERROR: Cannot create report: [Errno 17] File exists: '/var/crash/gasket-dkms.0.crash'
Error! Bad return status for module build on kernel: 6.5.0-10-generic (x86_64)
Consult /var/lib/dkms/gasket/1.0/build/make.log for more information.
dpkg: error processing package gasket-dkms (--configure):
 installed gasket-dkms package post-installation script subprocess returned error exit status 10
Setting up ubuntu-release-upgrader-core (1:23.10.10) ...
Processing triggers for dbus (1.14.10-1ubuntu1) ...
Errors were encountered while processing:
 gasket-dkms
needrestart is being skipped since dpkg has failed
E: Sub-process /usr/bin/dpkg returned an error code (1)

The mentioned crash log /var/crash/gasket-dkms.0.crash:

ProblemType: Package
DKMSBuildLog:
 DKMS make.log for gasket-1.0 for kernel 6.5.0-10-generic (x86_64)
 Thu Nov  9 21:35:37 UTC 2023
 make: Entering directory '/usr/src/linux-headers-6.5.0-10-generic'
 warning: the compiler differs from the one used to build the kernel
   The kernel was built by: x86_64-linux-gnu-gcc-13 (Ubuntu 13.2.0-4ubuntu3) 13.2.0
   You are using:           gcc-13 (Ubuntu 13.2.0-4ubuntu3) 13.2.0
   CC [M]  /var/lib/dkms/gasket/1.0/build/gasket_core.o
   CC [M]  /var/lib/dkms/gasket/1.0/build/gasket_ioctl.o
   CC [M]  /var/lib/dkms/gasket/1.0/build/gasket_interrupt.o
   CC [M]  /var/lib/dkms/gasket/1.0/build/gasket_page_table.o
 /var/lib/dkms/gasket/1.0/build/gasket_core.c: In function ‘gasket_register_device’:
 /var/lib/dkms/gasket/1.0/build/gasket_core.c:1841:41: error: passing argument 1 of ‘class_create’ from incompatible pointer type [-Werror=incompatible-pointer-types]
  1841 |                 class_create(driver_desc->module, driver_desc->name);
       |                              ~~~~~~~~~~~^~~~~~~~
       |                                         |
       |                                         struct module *
 In file included from ./include/linux/device.h:31,
                  from ./include/linux/cdev.h:8,
                  from /var/lib/dkms/gasket/1.0/build/gasket_core.h:11,
                  from /var/lib/dkms/gasket/1.0/build/gasket_core.c:12:
 ./include/linux/device/class.h:230:54: note: expected ‘const char *’ but argument is of type ‘struct module *’
   230 | struct class * __must_check class_create(const char *name);
       |                                          ~~~~~~~~~~~~^~~~
 /var/lib/dkms/gasket/1.0/build/gasket_core.c:1841:17: error: too many arguments to function ‘class_create’
  1841 |                 class_create(driver_desc->module, driver_desc->name);
       |                 ^~~~~~~~~~~~
 ./include/linux/device/class.h:230:29: note: declared here
   230 | struct class * __must_check class_create(const char *name);
       |                             ^~~~~~~~~~~~
 cc1: some warnings being treated as errors
 make[2]: *** [scripts/Makefile.build:251: /var/lib/dkms/gasket/1.0/build/gasket_core.o] Error 1
 make[2]: *** Waiting for unfinished jobs....
 make[1]: *** [/usr/src/linux-headers-6.5.0-10-generic/Makefile:2037: /var/lib/dkms/gasket/1.0/build] Error 2
 make: *** [Makefile:234: __sub-make] Error 2
 make: Leaving directory '/usr/src/linux-headers-6.5.0-10-generic'
DKMSKernelVersion: 6.5.0-10-generic
Date: Thu Nov  9 21:35:39 2023
DuplicateSignature: dkms:gasket-dkms:1.0-18:/var/lib/dkms/gasket/1.0/build/gasket_core.c:1841:41: error: passing argument 1 of ‘class_create’ from incompatible pointer type [-Werror=incompatible-pointer-types]
Package: gasket-dkms 1.0-18
PackageVersion: 1.0-18
SourcePackage: gasket-dkms
Title: gasket-dkms 1.0-18: gasket kernel module failed to build
Click to expand!

Issue Type

Build/Install

Operating System

Ubuntu

Coral Device

No response

Other Devices

No response

Programming Language

No response

Relevant Log Output

No response

About this issue

  • Original URL
  • State: closed
  • Created 8 months ago
  • Reactions: 2
  • Comments: 37

Most upvoted comments

@themana

sudo apt update
sudo apt upgrade
sudo apt install devscripts debhelper -y

Then clone and the rest

❯ git clone https://github.com/google/gasket-driver.git
Cloning into 'gasket-driver'...
...
❯ cd gasket-driver; debuild -us -uc -tc -b; cd ..
...
dpkg-deb: building package 'gasket-dkms' in '../gasket-dkms_1.0-18_all.deb'.
...
❯ ls -l gasket-dkms*
-rw-r--r-- 1 dewet dewet 49000 Jan 17 13:17 gasket-dkms_1.0-18_all.deb
-rw-r--r-- 1 dewet dewet  1788 Jan 17 13:18 gasket-dkms_1.0-18_amd64.build
-rw-r--r-- 1 dewet dewet  5642 Jan 17 13:17 gasket-dkms_1.0-18_amd64.buildinfo
-rw-r--r-- 1 dewet dewet  1017 Jan 17 13:17 gasket-dkms_1.0-18_amd64.changes

You can install that .deb on any systems that need to build the kernel module, and it will be rebuilt automatically with newer kernel packages being installed:

❯ sudo dpkg -i gasket-dkms_1.0-18_all.deb
...
Setting up gasket-dkms (1.0-18) ...
...
Building for 6.5.0-14-generic
Building initial module for 6.5.0-14-generic
...
depmod...
Time: 0h:00m:10s
❯ sudo modprobe apex
❯ lsmod | grep apex
apex                   28672  0
gasket                135168  1 apex
+15
alienatedsec on Jan 24, 2024

Stopped working here too. How to build the correct package for kernel 6.5?

@Mikescotland As documented in the earlier notes:

❯ sudo apt install devscripts debhelper dh-dkms -y
...
❯ git clone https://github.com/google/gasket-driver.git
Cloning into 'gasket-driver'...
...
❯ cd gasket-driver; debuild -us -uc -tc -b; cd ..
...
dpkg-deb: building package 'gasket-dkms' in '../gasket-dkms_1.0-18_all.deb'.
...
❯ ls -l gasket-dkms*
-rw-r--r-- 1 dewet dewet 49000 Jan 17 13:17 gasket-dkms_1.0-18_all.deb
-rw-r--r-- 1 dewet dewet  1788 Jan 17 13:18 gasket-dkms_1.0-18_amd64.build
-rw-r--r-- 1 dewet dewet  5642 Jan 17 13:17 gasket-dkms_1.0-18_amd64.buildinfo
-rw-r--r-- 1 dewet dewet  1017 Jan 17 13:17 gasket-dkms_1.0-18_amd64.changes

You can install that .deb on any systems that need to build the kernel module, and it will be rebuilt automatically with newer kernel packages being installed:

❯ sudo dpkg -i gasket-dkms_1.0-18_all.deb
...
Setting up gasket-dkms (1.0-18) ...
...
Building for 6.5.0-14-generic
Building initial module for 6.5.0-14-generic
...
depmod...
Time: 0h:00m:10s
❯ sudo modprobe apex
❯ lsmod | grep apex
apex                   28672  0
gasket                135168  1 apex
+10
dewet22 on Jan 17, 2024

When will this be available via https://packages.cloud.google.com/apt ?

It’s been fixed but still not available in stable for over 3 months.

I really not want to install all the dev dependencies on a production node.

+5
kub3let on Dec 15, 2023

FYI Workaround for Proxmox (debian based) systems here: https://forum.proxmox.com/threads/update-error-with-coral-tpu-drivers.136888/#post-608975

+4
bbccdd on Dec 11, 2023

Make sure you have the dkms debhelper package installed: sudo apt install dh-dkms

Some Proxmox instances require more effort 😉

sudo apt install dh-dkms devscripts git

+3
alienatedsec on Nov 25, 2023

Thanks @alienatedsec and @dewet22 this worked great. Not sure why they havent updated to stable though. Sure would make things much easier

+2
kymocode on Jan 31, 2024

This bug is now appearing in the HWE kernel in Ubuntu 22.04 as well. The “stable” repo isn’t seeming very table anymore.

What do we need to do to get it packaged and updated in the Debian stable repo?

If you give me write access to the Google Cloud repos, I’ll gladly update it myself.

+2
drikster80 on Jan 11, 2024

This was fixed a few months ago in https://github.com/google/gasket-driver/ already; for my use case I just built the .deb from that repo directly and deployed across my Coral-bearing machines.

I am trying to build it for Promox, can you guide me on how to build it ?

+2
arevindh on Nov 24, 2023

This was fixed a few months ago in https://github.com/google/gasket-driver/ already; for my use case I just built the .deb from that repo directly and deployed across my Coral-bearing machines.

+2
dewet22 on Nov 10, 2023

Thank you @alienatedsec ! My issue was that I had added it to the blocklist. I eventually got it working!

+1
nglessner on Apr 22, 2024

Is there any apex? It doesn’t have to be apex_0 for dual edge and it’s likely you will see only one. @nglessner

+1
alienatedsec on Apr 19, 2024

Can I just ask @alienatedsec @bbccdd @arevindh if your Proxmox installs are still working with your Coral?

Sorry for the late response @bobmarley2021 - I don’t know why I missed this message - regardless, my proxmox cluster and all three nodes have the latest versions and work just fine. I documented my effort in this comment for frigate installation on proxmox with Coral support

Installing Coral drivers - go to the shell of the host and type the following commands

apt update
apt upgrade
apt install pve-headers-$(uname -r)
apt install proxmox-default-headers
apt install dh-dkms devscripts git
git clone  https://github.com/google/gasket-driver
cd gasket-driver
debuild -us -uc -tc -b -d
cd ..
dpkg -i gasket-dkms_1.0-18_all.deb
reboot

Once rebooted, ensure the apex_0 device is in /dev/.

ls -alh /dev/apex_0 
crw-rw---- 1 root root 120, 0 Mar 20 14:51 /dev/apex_0

Any future kernel updates should automatically include gasket-dkms after above steps.

Follow the rest here https://github.com/blakeblackshear/frigate/discussions/5448#discussioncomment-8855247

+1
alienatedsec on Apr 19, 2024

Can I just ask @alienatedsec @bbccdd @arevindh if your Proxmox installs are still working with your Coral?

I have:

  1. Built gasket-dkms from source as per above
  2. Tried reverting to a previous kernel (eg. 6.2.16-20-pve)

Modules seem loaded:

apex                   28672  0
gasket                135168  1 apex

No dice…

~ # ls /dev/apex*
zsh: no matches found: /dev/apex*

I’ve gone so far as to order another Coral today as I cannot even get this thing recognised and flashed by webcoral now. Starting to suspect a faulty unit.

+1
bobmarley2021 on Feb 21, 2024

Thank you a ton @dewet22 and @alienatedsec, that did the trick! Very lucky I found this thread after a good deal of hunting. Cheers!

+1
keptin on Jan 30, 2024

@alienatedsec That worked perfectly, thank you so much

+1
themana on Jan 25, 2024

Make sure you have the dkms debhelper package installed: sudo apt install dh-dkms

Some Proxmox instances require more effort 😉

sudo apt install dh-dkms devscripts git

Yep , I did post in Double take discord server

+1
arevindh on Nov 25, 2023
Read more comments on GitHub
← edgetpu: AttributeError: 'Delegate' object has no attribute '_library'
edgetpu: bulk out for header failed. Abort. Unknown: USB transfer error 1 [LibUsbDataOutCallback] →