podman: Silverblue: Podman complains kernel not supporting overlay fs (backing file system is unsupported for this graph driver)

/kind bug

Description Podman is completly broken on Fedora Silverblue. I did not do anything, but just tried to use podman. It throws me the error shown below or is just stuck and does not continue.

Steps to reproduce the issue: Run podman login or basically any other command.

Describe the results you received:

$ podman login …                   
ERRO[0000] User-selected graph driver "vfs" overwritten by graph driver "overlay" from database - delete libpod local files to resolve

Describe the results you expected: No error.

Additional information you deem important (e.g. issue happens only occasionally): Happens always.

Output of podman version:

does not work

podman -v gives:

podman version 2.1.1

Output of podman info --debug:

impossible to run (no output)

Package info (e.g. output of rpm -q podman or apt list podman): This is Fedora Silverblue 33. ostree://fedora:fedora/33/x86_64/silverblue Version: 33.20201207.0 (2020-12-07T01:04:03Z)

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide?

Yes

Additional environment details (AWS, VirtualBox, physical, etc.):

I did not change anything. I’ve set REGISTRY_AUTH_FILE, but that should not affect the whole system. Podman could be used a few months ago. I did not test it in the meantime, but now everything fails.

So I’ve obviously looked that up and found e.g. this: https://github.com/containers/podman/issues/7501 https://github.com/containers/podman/issues/7396

Nothing helped. E.g. I could delete (move) ~/.local/share/containers, but of course I don’t want to loose all my container and b) it actually sometimes shows a different behavior: When I simply run a podman ps it shows me an empty terminal. It get’s stuck. No output, nothing. It also does not exit though (unless I Ctrl+C it, of course.) I.e. the thing it advises me: Deleting libpod actually did not help to fix this.

Also tried configuring ~/.config/containers/storage.conf, but I don’t know what to set there. Just work as usual in Silverblue. I don’t know or care what storage driver you use. 😄

Even toolbox fails fundamentally:

$ toolbox list                  
Error: failed to get the Podman version

Unfortunately podman info --debug is also broken. (Does not output anything/does not exist) Only podman -v works.

Reported downstream: https://bugzilla.redhat.com/show_bug.cgi?id=1907228

Edit: BTW, I can see it re-creates the $HOME/.local/share/containers dir when I start it though. Actually, only one file is created though: $HOME/.local/share/containers/storage/libpod/bolt_state.db

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 23 (17 by maintainers)

Most upvoted comments

I’m seeing the same issue since podman on debian testing upgraded to 3.4.6+ds1-1:

$ podman version
ERRO[0000] User-selected graph driver "vfs" overwritten by graph driver "overlay" from database - delete libpod local files to resolve 
Version:      3.4.6
API Version:  3.4.6
Go Version:   go1.18.1
Built:        Wed Dec 31 16:00:00 1969
OS/Arch:      linux/amd64

$ dpkg -s podman
Package: podman
Status: install ok installed
Priority: optional
Section: admin
Installed-Size: 34862
Maintainer: Debian Go Packaging Team <pkg-go-maintainers@lists.alioth.debian.org>
Architecture: amd64
Source: libpod
Version: 3.4.6+ds1-1
Depends: libc6 (>= 2.33), libdevmapper1.02.1 (>= 2:1.02.97), libgpgme11 (>= 1.4.1), libseccomp2 (>= 2.4.1), conmon (>= 2.0.18~), containernetworking-plugins (>= 0.8.7), golang-github-containers-common, crun | runc (>= 1.0.0~rc92~)
Recommends: buildah (>= 1.21.0), fuse-overlayfs (>= 1.0.0~), slirp4netns (>= 0.4.1~), catatonit | tini | dumb-init, uidmap, golang-github-containernetworking-plugin-dnsname
Suggests: containers-storage, docker-compose, iptables
Breaks: buildah (<< 1.10.1-6), fuse-overlayfs (<< 0.7.1), slirp4netns (<< 0.4.1)
Conffiles:
 /etc/cni/net.d/87-podman-bridge.conflist a87c090f17c5274af878e7106e969b60
 /etc/containers/libpod.conf ceec5a77b5f6a56d212eeed7b707d322
Description: engine to run OCI-based containers in Pods
 Podman is an engine for running OCI-based containers in Pods.
 Podman provides a CLI interface for managing Pods, Containers, and
 Container Images.
 .
 At a high level, the scope of libpod and podman is the following:
  * Support multiple image formats including the OCI and Docker image
    formats.
  * Support for multiple means to download images including trust & image
    verification.
  * Container image management (managing image layers, overlay filesystems,
    etc).
  * Full management of container lifecycle.
  * Support for pods to manage groups of containers together.
  * Resource isolation of containers and pods.
  * Support for a Docker-compatible CLI interface through Podman.
 .
 Podman is a daemon-less alternative to Docker.
Built-Using: containerd (= 1.6.2~ds1-1), docker-registry (= 2.8.1+ds1-1), docker.io (= 20.10.14+dfsg1-1), golang-1.18 (= 1.18.1-1), golang-dbus (= 5.0.6-1), golang-fsnotify (= 1.5.1-1), golang-ginkgo (= 1.14.2-1), golang-github-acarl005-stripansi (= 0.0~git20180116.5a71ef0-2), golang-github-appc-cni (= 1.0.1-3), golang-github-beorn7-perks (= 1.0.1-1), golang-github-blang-semver (= 4.0.0-1), golang-github-buger-goterm (= 0.0+git20181115.c206103-2), golang-github-cespare-xxhash (= 2.1.1-2), golang-github-checkpoint-restore-go-criu (= 5.3.0-2), golang-github-chzyer-readline (= 1.4.39.g2972be2-3), golang-github-containerd-stargz-snapshotter (= 0.8.0-9), golang-github-containernetworking-plugins (= 1.1.0+ds1-1), golang-github-containers-buildah (= 1.23.1+ds1-2), golang-github-containers-common (= 0.44.5+ds1-1), golang-github-containers-image (= 5.16.0-3), golang-github-containers-ocicrypt (= 1.0.3-1), golang-github-containers-psgo (= 1.5.2-1), golang-github-containers-storage (= 1.37.1+ds1-1), golang-github-coreos-bbolt (= 1.3.5-1), golang-github-coreos-go-iptables (= 0.6.0-1), golang-github-coreos-go-systemd (= 22.3.2-1), golang-github-cyphar-filepath-securejoin (= 0.2.3-1), golang-github-davecgh-go-spew (= 1.1.1-2), golang-github-disiqueira-gotree (= 3.0.2-2), golang-github-docker-docker-credential-helpers (= 0.6.4+ds1-1), golang-github-docker-go-connections (= 0.4.0-3), golang-github-docker-go-metrics (= 0.0.1-1), golang-github-docker-go-units (= 0.4.0-4), golang-github-docker-libtrust (= 0.0~git20150526.0.9cbd2a1-3.1), golang-github-dtylman-scp (= 0.0~git20181017.f3000a3-3), golang-github-fsouza-go-dockerclient (= 1.6.6-1), golang-github-fullsailor-pkcs7 (= 0.0~git20210826.33d0574-2), golang-github-ghodss-yaml (= 1.0.0-1.1), golang-github-go-logr-logr (= 0.4.0-1), golang-github-google-go-cmp (= 0.5.7-1), golang-github-google-go-intervals (= 0.0.2-2), golang-github-google-gofuzz (= 1.2.0-1), golang-github-google-shlex (= 0.0~git20191202.e7afc7f-1), golang-github-google-uuid (= 1.3.0-1), golang-github-gorilla-handlers (= 1.4.2-1), golang-github-gorilla-mux (= 1.8.0-1), golang-github-hashicorp-errwrap (= 1.1.0-1), golang-github-hashicorp-go-multierror (= 1.1.1-1), golang-github-influxdata-tail (= 1.0.0+git20180327.c434825-4), golang-github-ishidawataru-sctp (= 0.0+git20210707.9a39160-1), golang-github-jinzhu-copier (= 0.3.2-2), golang-github-json-iterator-go (= 1.1.12-1), golang-github-juju-ansiterm (= 0.0~git20210929.8b71cc9-1), golang-github-klauspost-compress (= 1.13.6+ds1-2), golang-github-klauspost-pgzip (= 1.2.5-1), golang-github-lunixbochs-vtclean (= 1.0.0-1), golang-github-manifoldco-promptui (= 0.8.0-2), golang-github-mattn-go-colorable (= 0.1.12-1), golang-github-mattn-go-isatty (= 0.0.14-1), golang-github-mattn-go-runewidth (= 0.0.13-2), golang-github-mitchellh-mapstructure (= 1.4.3-1), golang-github-moby-sys (= 0.0~git20220215.b8d8fab-2), golang-github-moby-term (= 0.0~git20210619.3f7ff69-1), golang-github-modern-go-concurrent (= 1.0.3-1.1), golang-github-modern-go-reflect2 (= 1.0.2-2), golang-github-morikuni-aec (= 1.0.0-2), golang-github-opencontainers-go-digest (= 1.0.0-1), golang-github-opencontainers-image-spec (= 1.0.2-2), golang-github-opencontainers-runtime-tools (= 0.9.0+dfsg-2), golang-github-opencontainers-selinux (= 1.10.0+ds1-1), golang-github-opencontainers-specs (= 1.0.2.66.g20a2d97-1), golang-github-openshift-imagebuilder (= 1.2.1+ds1-3), golang-github-pkg-errors (= 0.9.1-2), golang-github-pmezard-go-difflib (= 1.0.0-3), golang-github-proglottis-gpgme (= 0.1.1-1), golang-github-prometheus-client-golang (= 1.11.1-1), golang-github-prometheus-client-model (= 0.2.0-3), golang-github-prometheus-common (= 0.32.1-8), golang-github-prometheus-procfs (= 0.7.3-2), golang-github-rivo-uniseg (= 0.2.0-1), golang-github-safchain-ethtool (= 0.2.0-1), golang-github-spf13-cobra (= 1.4.0-2), golang-github-spf13-pflag (= 1.0.5-2), golang-github-ulikunitz-xz (= 0.5.6-2), golang-github-vbatts-tar-split (= 0.11.1-2), golang-github-vbauerster-mpb (= 7.3.2-1), golang-github-vishvananda-netlink (= 1.1.0.125.gf243826-4), golang-github-vishvananda-netns (= 0.0~git20211101.5004558-1), golang-github-vividcortex-ewma (= 1.1.1-2), golang-github-xeipuuv-gojsonpointer (= 0.0~git20190905.02993c4-2), golang-github-xeipuuv-gojsonreference (= 0.0~git20180127.bd5ef7b-2), golang-github-xeipuuv-gojsonschema (= 1.2.0-3), golang-go-patricia (= 2.3.1-1), golang-go-zfs (= 2.1.1.49.gf784269-1), golang-go.crypto (= 1:0.0~git20220315.3147a52-1), golang-gocapability-dev (= 0.0+git20200815.42c35b4-1), golang-gogoprotobuf (= 1.3.2-1), golang-golang-x-net (= 1:0.0+git20220225.27dd868+dfsg-1), golang-golang-x-sync (= 0.0~git20210220.036812b-1), golang-golang-x-sys (= 0.0~git20211216.1d35b9e-1), golang-golang-x-term (= 0.0~git20210615.6886f2d-1), golang-golang-x-text (= 0.3.7-1), golang-golang-x-xerrors (= 0.0~git20200804.5ec99f8-1), golang-gomega (= 1.10.3-1), golang-google-genproto (= 0.0~git20200413.b5235f6-1), golang-google-grpc (= 1.27.1-1), golang-google-protobuf (= 1.27.1-1), golang-gopkg-inf.v0 (= 0.9.1-1), golang-gopkg-square-go-jose.v2 (= 2.6.0-1), golang-gopkg-tomb.v1 (= 0.0~git20141024.0.dd63297-7), golang-gopkg-yaml.v3 (= 3.0.0~git20210107.496545a-1), golang-goprotobuf (= 1.3.4-2), golang-k8s-klog (= 2.5.0-2), golang-k8s-sigs-structured-merge-diff (= 4.1.2+ds1-1), golang-logrus (= 1.7.0-2), golang-protobuf-extensions (= 1.0.1-1), golang-toml (= 1.0.0-2), golang-yaml.v2 (= 2.4.0-2), rootlesskit (= 1.0.0-1), runc (= 1.1.1+ds1-1)
Homepage: https://github.com/containers/podman

I have no /etc/containers/storage.conf file:

$ sudo ls /etc/containers/
libpod.conf  policy.json  registries.conf  registries.conf.d

and no $HOME/.config/containers/storage.conf either:

$ ls $HOME/.config/containers
ls: cannot access '/home/fishy/.config/containers': No such file or directory

creating ~/.config/containers/storage.conf fixed it for me:

$ cat ~/.config/containers/storage.conf 
[storage]
driver = "overlay"

[storage.options.overlay]
mount_program = "/usr/bin/fuse-overlayfs"

$ podman version
Version:      3.4.6
API Version:  3.4.6
Go Version:   go1.18.1
Built:        Wed Dec 31 16:00:00 1969
OS/Arch:      linux/amd64

so sounds like it’s an issue with the default config?

fishy on Apr 19, 2022

Well… then I get the “old” error back as it seems… 😜 (twice, even 😃 )

Though for some reason the old error is also still displayed for some awkward reason. 😆

$ podman version                        
ERRO[0000] User-selected graph driver "vfs" overwritten by graph driver "overlay" from database - delete libpod local files to resolve 
ERRO[0000] User-selected graph driver "vfs" overwritten by graph driver "overlay" from database - delete libpod local files to resolve 
Error: kernel does not support overlay fs: 'overlay' is not supported over extfs at "/var/home/rugk/.local/share/containers/storage/overlay": backing file system is unsupported for this graph driver

storage.conf:

[storage]
driver = "vfs"
#[storage.options]
#mount_program = "/usr/bin/fuse-overlayfs"

rugk on Dec 13, 2020

It upgraded to 3.4.6 earlier today, which looks pretty new to me?

This issue also only started happening after today’s upgrade (I can’t remember what’s the version prior to the upgrade), so this looks like a regression in 3.4.6 to me (or a regression on debian’s packaging of podman 3.4.6)

fishy on Apr 19, 2022

Alright. It is complaining that it wants overlay - sorry, must have misread. Looks like we need to change back to overlay and add:

[storage.options.overlay]
mount_program = "/usr/bin/fuse-overlayfs"

That will force the use of fuse-overlayfs, not kernel overlay - rootless users can’t mount Overlay (usually - there are a few distros that maintain non-upstreamed patches that allow that) but can mount Fuse FSes, which is the probably cause of your error.

On Sun, Dec 13, 2020 at 5:10 PM rugk notifications@github.com wrote:

Well… then I get the “old” error back as it seems…

$ podman version

ERRO[0000] User-selected graph driver “vfs” overwritten by graph driver “overlay” from database - delete libpod local files to resolve

ERRO[0000] User-selected graph driver “vfs” overwritten by graph driver “overlay” from database - delete libpod local files to resolve

Error: kernel does not support overlay fs: ‘overlay’ is not supported over extfs at “/var/home/rugk/.local/share/containers/storage/overlay”: backing file system is unsupported for this graph driver

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

mheon on Dec 14, 2020