podman: Podman + runc: Cannot mount read-only filesystem on host as a volume

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Trying to use rootless containers on ubuntu 21.10 with podman 3.2.1 using the default “runc” fails.

Installing “crun” makes it work again, with runc the volume mounts are failing (OCI permission denied)

Steps to reproduce the issue:

limactl start examples/podman.yaml
export CONTAINER_HOST=unix://$HOME/podman.sock
podman --remote run -it -v $PWD/hello:/hello docker.io/busybox cat /hello/world

Describe the results you received:

Error: error preparing container e552b677afd051af9dccb902ea038e527445a5daa1d08d59623364907700d497 for attach: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: rootfs_linux.go:76: mounting "/home/anders/lima/hello" to rootfs at "/hello" caused: operation not permitted: OCI permission denied

Describe the results you expected:

Hello, World!

Additional information you deem important (e.g. issue happens only occasionally):

It works after installing crun, it fails with only runc (which is the apt default)

limactl shell podman sudo apt install -y crun

Output of podman version:

WARN[0000] The cgroupv2 manager is set to systemd but there is no systemd user session available 
WARN[0000] For using systemd, you may need to login using an user session 
WARN[0000] Alternatively, you can enable lingering with: `loginctl enable-linger 1000` (possibly as root) 
WARN[0000] Falling back to --cgroup-manager=cgroupfs    
Version:      3.2.1
API Version:  3.2.1
Go Version:   go1.16.7
Built:        Thu Jan  1 00:00:00 1970
OS/Arch:      linux/amd64

Output of podman info --debug:

(paste your output here)

Package info (e.g. output of rpm -q podman or apt list podman):

podman/impish,now 3.2.1+ds1-2ubuntu3 amd64 [installed]

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/master/troubleshooting.md)

Yes

Additional environment details (AWS, VirtualBox, physical, etc.):

Ubuntu 21.10

runc version 1.0.1-0ubuntu2
spec: 1.0.2-dev
go: go1.16.5
libseccomp: 2.5.1

https://packages.ubuntu.com/impish/podman

https://packages.ubuntu.com/impish/runc

About this issue

Original URL
State: closed
Created 3 years ago
Comments: 30 (29 by maintainers)

Links to this issue

Commits related to this issue

Fix failure with rw bind mount of a ro fuse As reported in [1], in a case where read-only fuse (sshfs) mount is used as a volume without specifying ro flag, the kernel fails to remount it (when addin... — committed to kolyshkin/runc by kolyshkin 3 years ago
Fix failure with rw bind mount of a ro fuse As reported in [1], in a case where read-only fuse (sshfs) mount is used as a volume without specifying ro flag, the kernel fails to remount it (when addin... — committed to kolyshkin/runc by kolyshkin 3 years ago
Fix failure with rw bind mount of a ro fuse As reported in [1], in a case where read-only fuse (sshfs) mount is used as a volume without specifying ro flag, the kernel fails to remount it (when addin... — committed to kolyshkin/runc by kolyshkin 3 years ago
Fix failure with rw bind mount of a ro fuse As reported in [1], in a case where read-only fuse (sshfs) mount is used as a volume without specifying ro flag, the kernel fails to remount it (when addin... — committed to kolyshkin/runc by kolyshkin 3 years ago
Fix failure with rw bind mount of a ro fuse As reported in [1], in a case where read-only fuse (sshfs) mount is used as a volume without specifying ro flag, the kernel fails to remount it (when addin... — committed to kolyshkin/runc by kolyshkin 3 years ago
Fix failure with rw bind mount of a ro fuse As reported in [1], in a case where read-only fuse (sshfs) mount is used as a volume without specifying ro flag, the kernel fails to remount it (when addin... — committed to kolyshkin/runc by kolyshkin 3 years ago
[1.0] Fix failure with rw bind mount of a ro fuse As reported in [1], in a case where read-only fuse (sshfs) mount is used as a volume without specifying ro flag, the kernel fails to remount it (when... — committed to kolyshkin/runc by kolyshkin 3 years ago

Most upvoted comments

@kolyshkin : I can confirm that this new runc binary fixes the issue, where the old runc still shows it.

https://github.com/opencontainers/runc/commit/d7846bc19a51b21b46803b4fd6a68b7ca31c33af (actual commit, as per PR)

https://github.com/opencontainers/runc/commit/ecf09297 (merge commit, per binary)

working:

runc version 1.0.0+dev
commit: v1.0.0-379-gecf09297
spec: 1.0.2-dev
go: go1.17.3
libseccomp: 2.5.2

ubuntu: (not working)

runc version 1.0.1-0ubuntu2
spec: 1.0.2-dev
go: go1.16.5
libseccomp: 2.5.1

Validated by creating a lima VM with podman.yaml, and then installing and selecting “runc” as the engine in the config

anders@lima-podman:/home/anders/lima$ more /etc/containers/containers.conf 
[engine]
runtime = "runc"

afbjorklund on Nov 18, 2021

@afbjorklund can you test the fix? https://github.com/opencontainers/runc/pull/3283

(You can get the static binary by navigating to Checks -> validate, when at the very bottom of the page under “Artifacts” you’ll find “release-NNNNN” file which contains pre-built binaries).

kolyshkin on Nov 18, 2021

On the command line we use fusermount rather than mount, maybe something similar is needed for this remount ?

What runtime does is a bind mount, so it does not really matter if this is a fuse mount or not.

Anyway, I am fixing this in runc (and the most complicated part is writing a test case).

kolyshkin on Nov 17, 2021

The way crun implemented it is to first try the mount, and if it fails then check the mount options for the source and attempt again with ro if it is present there.

How to check it beforehand? Does docker adds ro anytime the source is mounted as read only?

giuseppe on Nov 10, 2021