podman: [podman] kernel does not support overlay fs

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description podman complains that my kernel doesn’t support overlayfs. But it does (the kernel does and all the tools are installed and in search-path).

Steps to reproduce the issue:

~/podman/usr/local/bin/podman --log-level=debug --conmon /home/podman/podman/usr/local/libexec/crio/conmon --network-cmd-path /home/podman/podman/usr/local/bin/slirp4netns --runtime /home/podman/podman/usr/local/sbin/runc --storage-driver overlay run busybox sh

Describe the results you received:


INFO[0000] running as rootless
DEBU[0000] Initializing boltdb state at /home/podman/.local/share/containers/storage/libpod/bolt_state.db
DEBU[0000] Using graph driver overlay
DEBU[0000] Using graph root /home/podman/.local/share/containers/storage
DEBU[0000] Using run root /home/podman/rundir
DEBU[0000] Using static dir /home/podman/.local/share/containers/storage/libpod
DEBU[0000] Using tmp dir /home/podman/rundir/libpod/tmp
DEBU[0000] Using volume path /home/podman/.local/share/containers/storage/volumes
DEBU[0000] Set libpod namespace to ""
DEBU[0000] Not configuring container store
DEBU[0000] Using slirp4netns netmode
INFO[0000] running as rootless
DEBU[0000] Initializing boltdb state at /home/podman/.local/share/containers/storage/libpod/bolt_state.db
DEBU[0000] Using graph driver overlay
DEBU[0000] Using graph root /home/podman/.local/share/containers/storage
DEBU[0000] Using run root /home/podman/rundir
DEBU[0000] Using static dir /home/podman/.local/share/containers/storage/libpod
DEBU[0000] Using tmp dir /home/podman/rundir/libpod/tmp
DEBU[0000] Using volume path /home/podman/.local/share/containers/storage/volumes
DEBU[0000] Set libpod namespace to ""
DEBU[0000] [graphdriver] trying provided driver "overlay"
DEBU[0000] overlay test mount with multiple lowers failed mountfrom re-exec error: exit status 1: output: operation not permitted
DEBU[0000] overlay test mount with a single lower failed mountfrom re-exec error: exit status 1: output: operation not permitted
ERRO[0000] 'overlay' is not supported over extfs at "/home/podman/.local/share/containers/storage/overlay"
ERRO[0000] error creating libpod runtime: kernel does not support overlay fs: 'overlay' is not supported over extfs at "/home/podman/.local/share/containers/storage/overlay": backing file system is unsupported for this graph driver

Additional information you deem important (e.g. issue happens only occasionally):

fuse3 and fuse-overlayfs are installed and working. I’m using ext4 only. Might be connected with https://github.com/containers/libpod/issues/545

Output of podman version:

INFO[0000] running as rootless
Version:            1.2.0-dev
RemoteAPI Version:  1
Go Version:         go1.11.5
Git Commit:         c7aa191c8bcb83e8f5064d458d6604fcdf0e41d9
Built:              Thu Mar 14 19:52:43 2019
OS/Arch:            linux/amd64

Output of podman info --debug:

~/podman/usr/local/bin/podman --log-level=debug --conmon /home/podman/podman/usr/local/libexec/crio/conmon --network-cmd-path /home/podman/podman/usr/local/bin/slirp4netns --runtime /home/podman/podman/usr/local/sbin/runc info --debug
INFO[0000] running as rootless
DEBU[0000] Initializing boltdb state at /home/podman/.local/share/containers/storage/libpod/bolt_state.db
DEBU[0000] Using graph driver vfs
DEBU[0000] Using graph root /home/podman/.local/share/containers/storage
DEBU[0000] Using run root /home/podman/rundir
DEBU[0000] Using static dir /home/podman/.local/share/containers/storage/libpod
DEBU[0000] Using tmp dir /home/podman/rundir/libpod/tmp
DEBU[0000] Using volume path /home/podman/.local/share/containers/storage/volumes
DEBU[0000] Set libpod namespace to ""
DEBU[0000] [graphdriver] trying provided driver "vfs"
debug:
  compiler: gc
  git commit: c7aa191c8bcb83e8f5064d458d6604fcdf0e41d9
  go version: go1.11.5
  podman version: 1.2.0-dev
host:
  BuildahVersion: 1.8-dev
  Conmon:
    package: Unknown
    path: /home/podman/podman/usr/local/libexec/crio/conmon
    version: 'conmon version , commit: cc2b49590a485da9bd358440f92f219dfd6b230f'
  Distribution:
    distribution: debian
    version: "9"
  MemFree: 34703171584
  MemTotal: 38205562880
  OCIRuntime:
    package: Unknown
    path: /home/podman/podman/usr/local/sbin/runc
    version: |-
      runc version 1.0.0-rc6+dev
      commit: 2b18fe1d885ee5083ef9f0838fee39b62d653e30
      spec: 1.0.1-dev
  SwapFree: 8586784768
  SwapTotal: 8586784768
  arch: amd64
  cpus: 8
  hostname: pgsql
  kernel: 4.19.0-0.bpo.2-amd64
  os: linux
  rootless: true
  uptime: 103h 35m 21.42s (Approximately 4.29 days)
insecure registries:
  registries: []
registries:
  registries:
  - docker.io
  - quay.io
store:
  ConfigFile: /home/podman/.config/containers/storage.conf
  ContainerStore:
    number: 0
  GraphDriverName: vfs
  GraphOptions: null
  GraphRoot: /home/podman/.local/share/containers/storage
  GraphStatus: {}
  ImageStore:
    number: 0
  RunRoot: /home/podman/rundir
  VolumePath: /home/podman/.local/share/containers/storage/volumes

Additional environment details (AWS, VirtualBox, physical, etc.):

Debian 9 on VirtualBox 6 on Windows 10 on AMD 64

About this issue

Original URL
State: closed
Created 5 years ago
Comments: 25 (24 by maintainers)

Commits related to this issue

utils: split generation and writing of storage.conf split the generation for the default storage.conf and when we write it if not existing for a rootless user. This is necessary because during the s... — committed to giuseppe/libpod by giuseppe 5 years ago
utils: split generation and writing of storage.conf split the generation for the default storage.conf and when we write it if not existing for a rootless user. This is necessary because during the s... — committed to muayyad-alsadi/libpod by giuseppe 5 years ago
rootless: write storage overrides to the conf file make sure the user overrides are stored in the configuration file when first created. Closes: https://github.com/containers/libpod/issues/2659 Sig... — committed to giuseppe/libpod by giuseppe 5 years ago
rootless: write storage overrides to the conf file make sure the user overrides are stored in the configuration file when first created. Closes: https://github.com/containers/libpod/issues/2659 Sig... — committed to lsm5/podman by giuseppe 5 years ago

Most upvoted comments

Does this work if you run it as root?

If yes, then cleanup your homedir, and try again.

rm -rf ~/.config/container ~/.local/share/containers

+14

rhatdan on Mar 15, 2019

Tested, works.

[storage]
  driver = "overlay"
  runroot = "/tmp/run-1000"
  graphroot = "/home/tobwen/.local/share/containers/storage"
  [storage.options]
    mount_program = "/home/tobwen/podman/usr/local/bin/fuse-overlayfs"

tobwen on Oct 21, 2019

@giuseppe This looks like the storage.conf is not setting the override paths into the storage.conf file that it creates.

rhatdan on Oct 19, 2019

Oh god, I’m so sorry, I didn’t see the first run. I am so sorry. It works as expected now.

Maybe we should warn the user about two commands? I apologize very much for my boundless stupidity.

tobwen on Mar 17, 2019