jetson-containers: Docker fails to create container after upgrading docker on Jetpack 4.9

We have released a fix for this, here are the steps to run it:

distribution=$(. /etc/os-release;echo $ID$VERSION_ID) \
   && curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add - \
   && curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | sudo tee /etc/apt/sources.list.d/nvidia-docker.list

sudo apt-get update
sudo apt-get install nvidia-docker2=2.8.0-1

Let me know if that works for you guys.

I ran into this today as well (what a bad time to update), and spent a couple of hours fiddling with things to attempt to fix it.

Downgrading is probably the easiest approach at the moment. The newest version of the nvidia-docker-toolkit that fixes this problem is currently in experimental, and packages haven’t even been built for arm64 yet (I let them know on that linked issue). You can download a .deb file of the last version of Docker 19 at https://launchpad.net/ubuntu/bionic/arm64/docker.io/19.03.6-0ubuntu1~18.04.3 .

You’ll probably also have to downgrade containerd to 1.5.2 by doing

apt install containerd=1.5.2-0ubuntu1~18.04.3

You may also want to pin docker.io to version 19 and containerd to 1.5.2 for now so it doesn’t get updated again until they sort things out (or until Jetpack 5.0 releases next year sometime and we get a slightly less crusty version of Ubuntu). You can do that by editing /etc/apt/preferences and adding:

Package: docker.io
Pin: version 19.03*
Pin-Priority: 1001

Package: containerd
Pin: version 1.5.2*
Pin-Priority: 1001

JetPack 4.6 on Jetson Nano - I’m getting the same error during building of containers (setup for runtime = nvidia) which were fine previously. Have issues with all nvidia containers which includes: nvcr.io/nvidia/l4t-pytorch:r32.6.1-pth1.9-py3, nvcr.io/nvidia/l4t-base:r32.6.1 & dustynv/ros:noetic-ros-base-l4t-r32.6.1

Is this related perhaps? https://github.com/NVIDIA/nvidia-container-runtime/issues/157

I had an error when I tried to downgrade to Docker 19 before I downgraded containerd, so you should downgrade containerd first,

$ sudo apt install containerd=1.5.2-0ubuntu1~18.04.3

and then Docker 19.

$ sudo apt install ./docker.io_19.03.6-0ubuntu1_18.04.3_arm64.deb

Also, you can fix the version of docker.io and containerd in the following way, without editting /etc/apt/preferences

$ sudo apt-mark hold docker.io containerd

If the problem is solved and you want to upgrade them,

$ sudo apt-mark unhold docker.io containerd

Yes, that seems to be the same issue. I will look through that thread and update any solitons/updates here. Thanks.

We have released a fix for this, here are the steps to run it:

distribution=$(. /etc/os-release;echo $ID$VERSION_ID) \
   && curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add - \
   && curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | sudo tee /etc/apt/sources.list.d/nvidia-docker.list

sudo apt-get update
sudo apt-get install nvidia-docker2=2.8.0-1

Let me know if that works for you guys.

Facing issue and the above instructions does not solve the problem. Environment: Xavier NX

$printenv | grep JETPACK
JETSON_JETPACK=4.6
$python3
Python 3.6.9 (default, Mar 15 2022, 13:55:28) 
[GCC 8.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
\>\>\> import torch
\>\>\> torch.cuda.is_available()
True
\>\>\> 
$ sudo docker run --rm --runtime nvidia -it nvcr.io/nvidia/l4t-pytorch:r32.6.1-pth1.9-py3

root@18d14c453369:/# python3
Python 3.6.9 (default, Jan 26 2021, 15:33:00) 
[GCC 8.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
\> \> \> import torch
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.6/dist-packages/torch/__init__.py", line 196, in <module>
    _load_global_deps()
  File "/usr/local/lib/python3.6/dist-packages/torch/__init__.py", line 149, in _load_global_deps
    ctypes.CDLL(lib_path, mode=ctypes.RTLD_GLOBAL)
  File "/usr/lib/python3.6/ctypes/__init__.py", line 348, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: libcurand.so.10: cannot open shared object file: No such file or directory

@JeremieBourque1 I am unaware of any updates that have fixed this issue. I still have used this thread’s solution as:

1. Downgrading containerd to containerd=1.5.2-0ubuntu1~18.04.3
2. Downloading the docker.io package, then running this command: ./docker.io_19.03.6-0ubuntu1_18.04.3_arm64.deb
3. Marking the docker.io package and containerd package as hold by doing: sudo apt-mark hold docker.io containerd

If you do find that an update has been released that addresses this issue, please post it here!