rexray: Can't use s3fs plugin with other user than root

Summary

I need to start a container with other user than root (like graylog for example) and use rexray/s3fs as a persistent volume. But it not works. I can only access the persistent volume as root user from inside the container.

Bug Reports

Version

docker plugin ls
ID                  NAME                DESCRIPTION               ENABLED
19a00812f486        rexray/s3fs:0.8.2   REX-Ray for Amazon S3FS   true

Expected Behavior

I expect to user rexray/s3fs plugin in containers that runs with other users than root.

Actual Behavior

This is what occurs if I try to access a volume from rexray/s3fs plugin in a container running with a non root user:

graylog@33098acdd9b2:/usr/share/graylog$ ls -lh data/
ls: cannot access data/config: Permission denied
total 12K
d????????? ? ?       ?          ?            ? config
drwxr-xr-x 2 graylog graylog 4.0K Apr  4 12:58 contentpacks
drwxr-xr-x 2 root    root    4.0K Apr  4 12:58 journal
drwxr-xr-x 2 root    root    4.0K Apr  4 12:58 log

Steps To Reproduce

Create a s3 bucket
Install rexray/s3fs v.0.8.2
Run graylog container with graylog user:

docker run -ti --user=graylog --rm -v my.s3.volume:/usr/share/graylog/data/config graylog2/server:2.2.3-1 /bin/bash

Try to access /usr/share/graylog/data/config from within container

Configuration Files

docker plugin inspect rexray/s3fs:0.8.2 
[
    {
        "Config": {
            "Args": {
                "Description": "",
                "Name": "",
                "Settable": null,
                "Value": null
            },
            "Description": "REX-Ray for Amazon S3FS",
            "Documentation": "https://github.com/codedellemc/rexray/.docker/plugin/s3fs",
            "Entrypoint": [
                "/rexray.sh",
                "rexray",
                "start",
                "-f",
                "--nopid"
            ],
            "Env": [
                {
                    "Description": "",
                    "Name": "REXRAY_FSTYPE",
                    "Settable": [
                        "value"
                    ],
                    "Value": "ext4"
                },
                {
                    "Description": "",
                    "Name": "REXRAY_LOGLEVEL",
                    "Settable": [
                        "value"
                    ],
                    "Value": "warn"
                },
                {
                    "Description": "",
                    "Name": "REXRAY_PREEMPT",
                    "Settable": [
                        "value"
                    ],
                    "Value": "false"
                },
                {
                    "Description": "",
                    "Name": "S3FS_ACCESSKEY",
                    "Settable": [
                        "value"
                    ],
                    "Value": ""
                },
                {
                    "Description": "",
                    "Name": "S3FS_REGION",
                    "Settable": [
                        "value"
                    ],
                    "Value": ""
                },
                {
                    "Description": "",
                    "Name": "S3FS_SECRETKEY",
                    "Settable": [
                        "value"
                    ],
                    "Value": ""
                }
            ],
            "Interface": {
                "Socket": "rexray.sock",
                "Types": [
                    "docker.volumedriver/1.0"
                ]
            },
            "IpcHost": false,
            "Linux": {
                "AllowAllDevices": true,
                "Capabilities": [
                    "CAP_SYS_ADMIN"
                ],
                "Devices": null
            },
            "Mounts": [
                {
                    "Description": "",
                    "Destination": "/dev",
                    "Name": "",
                    "Options": [
                        "rbind"
                    ],
                    "Settable": null,
                    "Source": "/dev",
                    "Type": "bind"
                }
            ],
            "Network": {
                "Type": "host"
            },
            "PidHost": false,
            "PropagatedMount": "/var/lib/libstorage/volumes",
            "User": {},
            "WorkDir": "",
            "rootfs": {
                "diff_ids": [
                    "sha256:a7f0d37906c7f57b73a838cee49fe6068628bd7613eea766a12ca914c0921aaf"
                ],
                "type": "layers"
            }
        },
        "Enabled": true,
        "Id": "19a00812f486d4965c1b7347ed7746b473bd48028b6142c05fbc487e2733bbaf",
        "Name": "rexray/s3fs:0.8.2",
        "PluginReference": "docker.io/rexray/s3fs:0.8.2",
        "Settings": {
            "Args": [],
            "Devices": [],
            "Env": [
                "REXRAY_FSTYPE=ext4",
                "REXRAY_LOGLEVEL=warn",
                "REXRAY_PREEMPT=false",
                "S3FS_ACCESSKEY=myaccesskey",
                "S3FS_REGION=sa-east-1",
                "S3FS_SECRETKEY=mysecretkey"
            ],
            "Mounts": [
                {
                    "Description": "",
                    "Destination": "/dev",
                    "Name": "",
                    "Options": [
                        "rbind"
                    ],
                    "Settable": null,
                    "Source": "/dev",
                    "Type": "bind"
                }
            ]
        }
    }
]

Logs

How could I get docker plugin logs??

About this issue

Original URL
State: open
Created 7 years ago
Comments: 28 (11 by maintainers)

Most upvoted comments

I am trying rexray and minio and have a similar ask.

I have different containers in a swarm cluster and some of them wants to run as non root. Others check the file mode for security.

i tried different options to no avail, there is allways some container complaining.

1/ umask

Umask allows to set the file mode in the volume but containers like traefik when using acme (letsencrypt) will check that the file containing the configuration is only accessible to root (0600).

2/ gid/uid & mp_umask

this would allow to mount with rights to another user and group and to set the access rights at the mountpoint level. Unfortunately most containers are running as root.

3/ chown behavior of s3fs

when playing with different containers i had often the issue that the container couldn’t chown the mount point (input/output error). This happens:

when the container is started: it tries to copy the content of the folder if it exists to the volume. When doing that it will set the rights accordingly. So it tries to do a chown of the mount point, which lamentably fails. The workaround i found for that is using the “:nocopy” volume option
Some containers will start as root an use su-exec or gosu to set the process as another user (i.e. couchdb). They will most often use the required entrypoint to do some chown and chmod. These entrypoints will fail when they try to set rights on the mount point

My ask

So to my ask…

rexray allows to change user but on a global level, this will affect any volume provided by this rexray instance (volume plugin). I really don’t feel like setting different plugin for the different needs of the containers (in my small swarm it could be 3 or 4 but i can imagine situation where there are many more).

What would be realy flexible is to be able to define a set of options at the volume level. In this sense the following would be a minimum:

group id
user id

it would be good to also be able to control umask and/or mp_umask (maybe specific to s3fs there)

cblomart on May 11, 2020

This is a relatively old issue but thought I should update with the info I found as I do have a solution for my issue (could not write to an S3FS backed rexray/plugin volume as a user other than root from within a container).

There is now a LINUX_VOLUME_FILEMODE=0777 flag which can be set for the s3fs plugin. After setting this I was able to mount a volume in a container and read/write to the container as a user other than root (apache in my case).

My plugin install command:

docker plugin install rexray/s3fs \
  LINUX_VOLUME_FILEMODE=0777
  S3FS_ACCESSKEY=<aws-access-key> \
  S3FS_SECRETKEY=<aws-secret-key> \
  S3FS_REGION=<aws-region> \
  S3FS_OPTIONS=use_cache=/tmp,allow_other,use_rrs

I am using this with docker-compose, my relevant compose info:

version: '2'
services:
    app:
      build:
        context: ./app/
      env_file:
        - .env   # pass full .env file into container.
      volumes:
        - mybucket:/opt/share
      ports:
        - "${APP_PORT}:80"
      depends_on:
        - db
        - redis-sentinel
      # NOTICE: the sys_admin capability is needed to bind mount.
      cap_add:
        - sys_admin
        
volumes:
    mybucket:
      external: true

dradux on Mar 17, 2018

@ConstantinElse 0.11.1 is worked.

zironycho on Sep 3, 2018