moby: lsetxattr /dev/mqueue operation not permitted when using docker userns with selinux-enabled

Output of docker version:

Client:
 Version:      1.10.2
 API version:  1.22
 Go version:   go1.5.3
 Git commit:   e0e49ff
 Built:        Tue Feb 23 06:11:53 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.10.2
 API version:  1.22
 Go version:   go1.5.3
 Git commit:   e0e49ff
 Built:        Tue Feb 23 06:11:53 2016
 OS/Arch:      linux/amd64

Output of docker info:

Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 0
Server Version: 1.10.2
Storage Driver: btrfs
 Build Version: Btrfs v3.19.1
 Library Version: 101
Execution Driver: native-0.2
Logging Driver: json-file
Plugins:
 Volume: local
 Network: host bridge null
Kernel Version: 4.1.12-32.el7uek.x86_64
Operating System: Oracle Linux Server 7.2
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 47.16 GiB

Output of /etc/subuid and /etc/subgid:

/etc/subuid
dockremap:100000:65536
/etc/subgid
dockremap:100000:65536

List the steps to reproduce the issue:

  1. Run the docker daemon with --selinux-enabled --userns-remap=default
  2. docker run -it oraclelinux:7.2 bash

Describe the results you received:

Timestamp: 2016-02-29 21:24:14.706357216 -0500 EST
Code: System error

Message: operation not permitted

Frames:

---
0: setupRootfs
Package: github.com/opencontainers/runc/libcontainer
File: rootfs_linux.go@40

---
1: Init
Package: github.com/opencontainers/runc/libcontainer.(*linuxStandardInit)
File: standard_init_linux.go@57

---
2: StartInitialization
Package: github.com/opencontainers/runc/libcontainer.(*LinuxFactory)
File: factory_linux.go@240

---
3: initializer
Package: github.com/docker/docker/daemon/execdriver/native
File: init.go@35

---
4: Init
Package: github.com/docker/docker/pkg/reexec
File: reexec.go@26

---
5: main
Package: main
File: docker.go@18

---
6: main
Package: runtime
File: proc.go@111

---
7: goexit
Package: runtime
File: asm_amd64.s@1721
docker: Error response from daemon: Cannot start container 2b45ea01fc0f0b1ff5b2caffc5183d1379506a401577db0494f2ed3837cc9947: [9] System error: operation not permitted

Describe the results you expected: Container should start successfully.

Provide additional info you think is important: This is happening when we are trying to setFileLabel on /dev/mqueue in mountToRootfs on file vendor/src/github.com/opencontainers/runc/libcontainer/rootfs_linux.go

 97 func mountToRootfs(m *configs.Mount, rootfs, mountLabel string) error {
...
112         case "mqueue":
113                 if err := os.MkdirAll(dest, 0755); err != nil {
114                         return err
115                 }
116                 if err := mountPropagate(m, rootfs, mountLabel); err != nil {
117                         // older kernels do not support labeling of /dev/mqueue
118                         if err := mountPropagate(m, rootfs, ""); err != nil {
119                                 return err
120                         }
121                 }
122                 return label.SetFileLabel(dest, mountLabel)
...

Stracing the docker daemon narrowed it down to the following:

...
9311  lsetxattr("/var/lib/docker/100000.100000/btrfs/subvolumes/bf3a1a7300fbb3bab5ac89f982cdf733e20a1525c4b5005beef93fe203ec6942-init/usr/lib/systemd/system/sysinit.target.wants/dev-mqueue.mount", "security.selinux", "system_u:object_r:svirt_sandbox_file_t:s0:c232,c472", 51, 0) = 0
9621  lsetxattr("/var/lib/docker/100000.100000/btrfs/subvolumes/bf3a1a7300fbb3bab5ac89f982cdf733e20a1525c4b5005beef93fe203ec6942/dev/mqueue", "security.selinux", "system_u:object_r:svirt_sandbox_file_t:s0:c232,c472", 51, 0) = -1 EPERM (Operation not permitted)
...

Narrow it down further by tracing the linux kernel:

we seem to fail at the following function at fs/inode.c:

bool inode_owner_or_capable(const struct inode *inode)
{
        struct user_namespace *ns;

        if (uid_eq(current_fsuid(), inode->i_uid))
                return true;

        ns = current_user_ns();
        if (ns_capable(ns, CAP_FOWNER) && kuid_has_mapping(ns, inode->i_uid))
                return true;
        return false;
}

and kuid_has_mapping returns FALSE that is because inode->i_uid = 0

userns is configured and compiled in the kernel:

# grep CONFIG_USER_NS /boot/config-4.1.12-32.el7uek.x86_64
CONFIG_USER_NS=y

Hoping that maybe someone could provide some pointers here, thanks in advance.

About this issue

  • Original URL
  • State: open
  • Created 8 years ago
  • Comments: 21 (12 by maintainers)

Commits related to this issue

Most upvoted comments

I am seeing this on CentOS 7, 1.12.6. I cannot just tell my users to not use selinux.

Apologies if this is the wrong place to ask, but I am trying to use kubernetes on AWS (via CloudFormation), which uses CoreOS. Currently even the latest alpha release (https://coreos.com/releases/) is running Docker 1.10.3. This issue happens to me when I try to run a service via a ReplicationController and a private docker repository (quay.io).

May 03 22:21:50 <scrubbed> kernel: SELinux: mount invalid.  Same superblock, different security settings for (dev mqueue, type mqueue)
May 03 22:21:50 <scrubbed> kubelet-wrapper[1234]: I0503 22:21:50.629272    1234 provider.go:91] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 03 22:21:50 <scrubbed> kubelet-wrapper[1234]: E0503 22:21:50.945204    1234 pod_workers.go:138] Error syncing pod 6e664d92-117d-11e6-86cd-12595a8629bd, skipping: failed to "StartContainer" for "event-service" with ErrImagePull: "image pull failed for <scrubbed>, this may be because there are no credentials on this request.  details: (Error: Status 403 trying to pull repository <scrubbed>: \"{\\\"error\\\": \\\"Permission Denied\\\"}\")"
May 03 22:21:51 <scrubbed> kernel: IPv6: eth0: IPv6 duplicate address fe80::42:aff:fe02:1208 detected!
May 03 22:21:51 <scrubbed> kubelet-wrapper[1234]: E0503 22:21:51.609254    1234 pod_workers.go:138] Error syncing pod 6e664d92-117d-11e6-86cd-12595a8629bd, skipping: failed to "StartContainer" for "event-service" with ImagePullBackOff: "Back-off pulling image \"<scrubbed>\""

What are my options here? Should I disable --selinux-enabled flag? Is there any way to upgrade to Docker 1.11.0 on CoreOS?