podman: ERRO[0328] 'overlay' is not supported over extfs at "/var/lib/containers/storage/overlay"

[ No idea what caused it, no reproducer. Filing as placeholder because an issue search finds no hits, and this seems worth putting on the radar.]

Long-running job. System load average was ~6 but I’m not sure why. podman commands were taking O(1 minute) to complete. For grins I ran a ps:

podman ps -a
ERRO[0328] 'overlay' is not supported over extfs at "/var/lib/containers/storage/overlay"                                          
error creating libpod runtime: kernel does not support overlay fs: 'overlay' is not supported over extfs at "/var/lib/containers/storage/overlay": backing file system is unsupported for this graph driver

Right around that time, I saw the same error in the log of the long-running job; it was running podman logs CONTAINERNAME. I ^C’ed the job, waited for a prompt, can no longer reproduce. But on this system, podman ps -a is still taking O(minute), and something called exe seems to be hogging RAM.

I do not expect any action to be taken on this. I am filing in hopes that someone sees this in the future, finds this issue, and can add actually useful information.

podman-0.9.2-1536796590.gitfacab2ae.fc28.x86_64

About this issue

Original URL
State: closed
Created 6 years ago
Comments: 20 (13 by maintainers)

Most upvoted comments

FWIW happened to me on Arch after updating kernel. Things go wonky if you don’t reboot after kernel replacement.

simpleauthority on Jul 26, 2020

FWIW, just seen this when the overlay module was unavailable in a VM. Installing it and modprobing it didn’t help until I removed /var/run/containers/storage/overlay/overlay-false … presumably why @simpleauthority needed a reboot…

srd424 on Aug 11, 2020

@debarshiray after seeing your comment here I ran the following command

cp ~/.config/containers ~/.config/containers-BAK

After this I can run podman commands again without issue. It rebuilt the ~/.config/containers directory.

Here are the differences:

$ diff -aur containers containers-BAK/
diff -aur containers/libpod.conf containers-BAK/libpod.conf
--- containers/libpod.conf	2019-01-24 10:42:25.986409018 -0700
+++ containers-BAK/libpod.conf	2019-01-01 10:17:51.290997908 -0700
@@ -4,7 +4,6 @@
 conmon_path = ["/usr/libexec/podman/conmon", "/usr/libexec/crio/conmon", "/usr/local/lib/podman/conmon", "/usr/local/libexec/crio/conmon", "/usr/bin/conmon", "/usr/sbin/conmon", "/usr/lib/crio/bin/conmon"]
 conmon_env_vars = ["PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"]
 cgroup_manager = "cgroupfs"
-init_path = "/usr/libexec/podman/catatonit"
 static_dir = "/var/home/jonessean/.local/share/containers/storage/libpod"
 tmp_dir = "/run/user/1000/libpod/tmp"
 max_log_size = -1
diff -aur containers/storage.conf containers-BAK/storage.conf
--- containers/storage.conf	2019-01-24 10:42:25.949408857 -0700
+++ containers-BAK/storage.conf	2019-01-01 10:16:38.205998187 -0700
@@ -1,6 +1,4 @@
-[storage]
-  driver = "overlay"
-  runroot = "/run/user/1000"
-  graphroot = "/var/home/jonessean/.local/share/containers/storage"
-  [storage.options]
-    mount_program = "/usr/bin/fuse-overlayfs"
+RunRoot = "/run/user/1000"
+GraphRoot = "/var/home/jonessean/.local/share/containers/storage"
+GraphDriverName = "overlay"
+GraphDriverOptions = ["overlay.mount_program=/usr/bin/fuse-overlayfs"]

seandjones92 on Jan 24, 2019

@agners You need to set in your libpod.conf:

[storage.options]
     mount_program = "/usr/bin/fuse-overlayfs"

@rhatdan I think this is a mount options thing - I don’t know if your multi-driver patch fixes it?

mheon on Aug 29, 2019

I had the same problem - I had an old config generated by some previous version of libpod in ~/.config/containers/ - removing that directory and letting libpod recreate it fixed the problem for me.

mizdebsk on Feb 2, 2019

is the job running in a container and/var/lib/containers/storage is not a volume?

It looks like the issue is podman trying to use overlayfs on top of another overlayfs file system and that won’t work

giuseppe on Nov 19, 2018