dokku: Dokku cannot work with user namespaces

Pushing to Dokku fails with following message:

 $ git push dokku
Counting objects: 99, done.
Compressing objects: 100% (54/54), done.
Writing objects: 100% (99/99), 22.41 KiB | 11.21 MiB/s, done.
Total 99 (delta 39), reused 99 (delta 39)
-----> Cleaning up...
-----> Building my-app from herokuish...
-----> Adding BUILD_ENV to build environment...
remote: chown: changing ownership of '/cache': Operation not permitted
To localhost:my-app
 ! [remote rejected] master -> master (pre-receive hook declined)
error: failed to push some refs to 'dokku@localhost:my-app'

When docker on the host uses user namespaces to increase isolation of containers and improve security.

Since /cache is a mounted volume and Dokku attempt to run chown on it fails:

remote: + local PHASE=BUILD
remote: + local FILE_PREFIX=DOCKER_OPTIONS_
remote: + local PHASE_FILE_PATH=/home/dokku/my-app/DOCKER_OPTIONS_BUILD
remote: + local output=
remote: + [[ -f /home/dokku/my-app/DOCKER_OPTIONS_BUILD ]]
remote: + echo -n ''
remote: + local 'DOCKER_ARGS= --env=USER=herokuishuser'
remote: + [[ -n 1 ]]
remote: + DOCKER_ARGS+=' -e TRACE=true '
remote: + declare -a ARG_ARRAY
remote: + eval 'ARG_ARRAY=( --env=USER=herokuishuser -e TRACE=true )'
remote: ++ ARG_ARRAY=(--env=USER=herokuishuser -e TRACE=true)
remote: ++ docker run --label=dokku -d -v /home/dokku/my-app/cache:/cache -e CACHE_PATH=/cache --env=USER=herokuishuser -e TRACE=true dokku/my-app:latest /build
remote: + cid=5af13e1a9ee6b1be0991f3f70fa40d6f9fbb1f398e0cd2e092498f8d6873506a
remote: + docker attach 5af13e1a9ee6b1be0991f3f70fa40d6f9fbb1f398e0cd2e092498f8d6873506a
remote: : Operation not permitted
remote: chown: changing ownership of '/cache/node/cache/node_modules/circular-json/LICENSE.txt': Operation not permitted
...(a lot more files ommitted)...

I’m not sure where it comes from but it clearly won’t work.

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Reactions: 1
  • Comments: 24 (8 by maintainers)

Commits related to this issue

Most upvoted comments

Hello! We have the same problem with docker + userns-remap + dokku. And, looks like, we have a workaround to fix it.

Information about enviroment

OS version:

# cat /etc/os-release 
NAME="Ubuntu"
VERSION="16.04.5 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04.5 LTS"
VERSION_ID="16.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
VERSION_CODENAME=xenial
UBUNTU_CODENAME=xenial

Docker version:

# docker version
Client:
 Version:           18.09.3
 API version:       1.39
 Go version:        go1.10.8
 Git commit:        774a1f4
 Built:             Thu Feb 28 06:40:58 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          18.09.3
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.10.8
  Git commit:       774a1f4
  Built:            Thu Feb 28 05:59:55 2019
  OS/Arch:          linux/amd64
  Experimental:     true

Docker. daemon.json

# cat /etc/docker/daemon.json 
{
  "userns-remap" : "dockremap"
}

Docker. User for userns-remap

# grep dockremap /etc/passwd /etc/group
/etc/passwd:dockremap:x:1000:1000::/home/dockremap:
/etc/group:dockremap:x:1000:

# grep dockremap /etc/subuid /etc/subgid 
/etc/subuid:dockremap:165536:65536
/etc/subgid:dockremap:165536:65536

Dokku version and user

# dokku version
0.14.6

# grep dokku /etc/passwd /etc/group
/etc/passwd:dokku:x:1001:1001:,,,:/home/dokku:/bin/bash
/etc/group:adm:x:4:syslog,dokku
/etc/group:docker:x:999:dokku
/etc/group:dokku:x:1001:

Problem details

Dokku. Example postgres service creation

# env POSTGRES_IMAGE='mdillon/postgis' POSTGRES_IMAGE_VERSION='latest' dokku postgres:create example
latest: Pulling from mdillon/postgis
f7e2b70d04ae: Pull complete 
027ad848ac9c: Pull complete 
7c040ef66643: Pull complete 
b891079ad2eb: Pull complete 
cb64a97e42d9: Pull complete 
1b88625f7d89: Pull complete 
a6ac0b663e77: Pull complete 
594497f0a694: Pull complete 
ca7201b6a21f: Pull complete 
48cdfad3f2fd: Pull complete 
912fb62e7390: Pull complete 
1e6365c64609: Pull complete 
eda829b73ec7: Pull complete 
1dafb86732d6: Pull complete 
ad1854653222: Pull complete 
410147bb9559: Pull complete 
ee8d11af0d8f: Pull complete 
Digest: sha256:ac5dd18c43b0ee89320764684aa5c2da5c3c2ce54cf01a6caab21f905302d1b5
Status: Downloaded newer image for mdillon/postgis:latest
       Waiting for container to be ready
       Creating container database
OCI runtime exec failed: exec failed: container_linux.go:344: starting container process caused "process_linux.go:91: executing setns process caused \"exit status 21\"": unknown
       Already exists
       Securing connection to database
Error response from daemon: page not found
Error: failed to start containers:

Docker container. Logs

# docker logs dokku.postgres.example
chown: changing ownership of '/var/lib/postgresql/data': Operation not permitted
chown: changing ownership of '/var/lib/postgresql/data': Operation not permitted
chown: changing ownership of '/var/lib/postgresql/data': Operation not permitted

Docker container. Volumes (inspect)

# docker inspect dokku.postgres.example | grep -A1 -B1 /var/lib/dokku/
            "Binds": [
                "/var/lib/dokku/services/postgres/example/data:/var/lib/postgresql/data"
            ],
--
                "Type": "bind",
                "Source": "/var/lib/dokku/services/postgres/example/data",
                "Destination": "/var/lib/postgresql/data",

# stat /var/lib/dokku/services/postgres/example/data
  File: '/var/lib/dokku/services/postgres/example/data'
  Size: 4096      	Blocks: 8          IO Block: 4096   directory
Device: 900h/2304d	Inode: 11016339    Links: 2
Access: (0755/drwxr-xr-x)  Uid: ( 1001/   dokku)   Gid: ( 1001/   dokku)
Access: 2019-03-26 20:45:21.960752485 +0300
Modify: 2019-03-26 20:45:18.304720264 +0300
Change: 2019-03-26 20:45:18.304720264 +0300
 Birth: -

Dokku. Example postgres service destruction

# dokku postgres:destroy example
 !     WARNING: Potentially Destructive Action
 !     This command will destroy example Postgres service.
 !     To proceed, type "example"

> example
=====> Deleting example
=====> Stopping container
       Container stopped
       Removing container
       Removing data
chmod: /data: Operation not permitted
chmod: /data: Operation not permitted

# dokku postgres:list
NAME     VERSION  STATUS  EXPOSED PORTS  LINKS
example  missing  -       -

Dokku. Example postgres service destruction with manual permissions fix

# chown -R 165536:165536 /var/lib/dokku/services/postgres/example/data 

# dokku postgres:destroy example -f 
=====> Deleting example
 !     Service is already stopped
       Removing data
=====> Postgres container deleted: example

# dokku postgres:list
 !     There are no Postgres services

Dokku. Example postgres service creation with trace on

Service deployment:

# env POSTGRES_IMAGE='mdillon/postgis' POSTGRES_IMAGE_VERSION='latest' dokku postgres:create example
...
+ mkdir -p /var/lib/dokku/services/postgres/example
+ mkdir -p /var/lib/dokku/services/postgres/example/data
...
++ docker run --name dokku.postgres.example -v /var/lib/dokku/services/postgres/example/data:/var/lib/postgresql/data -e POSTGRES_PASSWORD=b217f0f7e52e0f7ab679004803d81445 --env-file=/var/lib/dokku/services/postg
res/example/ENV -d --restart always --label dokku=service --label dokku.service=postgres mdillon/postgis:latest
...
++ docker ps -aq --no-trunc --filter status=exited --filter 'name=^/dokku.postgres.example$' --format '{{ .ID }}'
+ PREVIOUS_ID=
+ docker start ''
Error response from daemon: page not found
Error: failed to start containers:

Docker container:

# docker ps
CONTAINER ID        IMAGE                    COMMAND                  CREATED             STATUS                         PORTS               NAMES
d2c3255e9976        mdillon/postgis:latest   "docker-entrypoint.s…"   6 minutes ago       Restarting (1) 6 minutes ago                       dokku.postgres.example

# docker logs dokku.postgres.example
chown: changing ownership of '/var/lib/postgresql/data': Operation not permitted

Volume permissions fix. Docker container check. Example service destruction.

# chown -R 165536:165536 /var/lib/dokku/services/postgres/example/data
#

# docker restart dokku.postgres.example
dokku.postgres.example

# docker logs dokku.postgres.example
chown: changing ownership of '/var/lib/postgresql/data': Operation not permitted
chown: changing ownership of '/var/lib/postgresql/data': Operation not permitted
chown: changing ownership of '/var/lib/postgresql/data': Operation not permitted
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process
...
fixing permissions on existing directory /var/lib/postgresql/data ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting dynamic shared memory implementation ... posix
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok

Success. You can now start the database server using:

    pg_ctl -D /var/lib/postgresql/data -l logfile start
...
server started
...

# dokku postgres:destroy example -f
+ export DOKKU_HOST_ROOT=/home/dokku
+ DOKKU_HOST_ROOT=/home/dokku
+ export DOKKU_DISTRO
++ . /etc/os-release
...
=====> Postgres container deleted: example
+ implemented=1
+ [[ 1 -eq 0 ]]
+ [[ 1 -eq 0 ]]
+ exit 0

Workaround fix

Dokku user. uid/gid fix

# usermod -u 165536 dokku
# groupmod -g 165536 dokku

# grep dokku /etc/passwd /etc/group
/etc/passwd:dokku:x:165536:165536:,,,:/home/dokku:/bin/bash
/etc/group:adm:x:4:syslog,dokku
/etc/group:docker:x:999:dokku
/etc/group:dokku:x:165536:

Dokku folders. Permission fix

# find /home/dokku -gid 1001 -exec chown :dokku {} +
# find /home/dokku -uid 1001 -exec chown dokku: {} +
# find /var/lib/dokku -gid 1001 -exec chown :dokku {} +
# find /var/lib/dokku -uid 1001 -exec chown dokku: {} +

Dokku foldrers. Permissions check

# ls -la /home/dokku/
total 56
drwxr-xr-x 6 dokku dokku 4096 Mar 26 20:51 .
drwxr-xr-x 3 root  root  4096 Mar 26 20:43 ..
drwxr-xr-x 2 dokku dokku 4096 Mar 25 17:38 .basher
-rw-r--r-- 1 dokku dokku  220 Mar 25 17:38 .bash_logout
-rw-r--r-- 1 dokku dokku 3771 Mar 25 17:38 .bashrc
drwxr-xr-x 2 dokku dokku 4096 Mar 26 21:01 .dokkurc
-rw-r--r-- 1 dokku dokku   10 Mar 25 19:03 HOSTNAME
drwxr-xr-x 3 dokku dokku 4096 Mar 25 21:24 .parallel
-rw-r--r-- 1 dokku dokku  655 Mar 25 17:38 .profile
-rw------- 1 dokku dokku 1024 Mar 26 21:01 .rnd
drwxr-xr-x 2 dokku root  4096 Mar 25 19:13 .ssh
-rw-r--r-- 1 dokku root    15 Mar 25 17:38 .sshcommand
-rw-r--r-- 1 root  root     7 Mar 25 17:38 VERSION
-rw-r--r-- 1 dokku dokku   10 Mar 26 20:44 VHOST

# ls -la /var/lib/dokku/
total 40
drwxr-xr-x  7 root  root  4096 Mar 26 20:44 .
drwxr-xr-x 47 root  root  4096 Mar 25 20:45 ..
drwxr-xr-x  6 root  root  4096 Mar 26 20:44 config
drwxr-xr-x  4 dokku dokku 4096 Mar 25 17:38 core-plugins
drwxr-xr-x  6 dokku dokku 4096 Mar 25 17:38 data
-rw-r--r--  1 root  root    41 Feb 19 19:55 GIT_REV
drwxr-xr-x  4 dokku dokku 4096 Mar 25 17:38 plugins
drwxr-xr-x  3 root  root  4096 Mar 26 20:44 services
-rw-r--r--  1 root  root     7 Feb 19 19:55 STABLE_VERSION
-rw-r--r--  1 root  root     7 Feb 19 19:55 VERSION

Workaround fix check

Example dokku service creation

# env POSTGRES_IMAGE='mdillon/postgis' POSTGRES_IMAGE_VERSION='latest' dokku postgres:create example
       Waiting for container to be ready
       Creating container database
       Securing connection to database
=====> Postgres container created: example
=====> Container Information
       Config dir:          /var/lib/dokku/services/postgres/example/config
       Data dir:            /var/lib/dokku/services/postgres/example/data
       Dsn:                 postgres://postgres:f8f4766df184fb2352f5710e7dc907a8@dokku-postgres-example:5432/example
       Exposed ports:       -                        
       Id:                  d3b699114b852682967b54e81e2fad15d60835cde81ceb5ccbe5980904281bac
       Internal ip:         172.17.0.2               
       Links:               -                        
       Service root:        /var/lib/dokku/services/postgres/example
       Status:              running

Volume permission check

# stat /var/lib/dokku/services/postgres/example/data
  File: '/var/lib/dokku/services/postgres/example/data'
  Size: 4096      	Blocks: 8          IO Block: 4096   directory
Device: 900h/2304d	Inode: 7908253     Links: 19
Access: (0700/drwx------)  Uid: (166535/ UNKNOWN)   Gid: (165536/   dokku)
Access: 2019-03-26 21:17:49.509977803 +0300
Modify: 2019-03-26 21:17:49.525977945 +0300
Change: 2019-03-26 21:17:49.525977945 +0300
 Birth: -

Docker container and dokku service status check

# docker ps
CONTAINER ID        IMAGE                    COMMAND                  CREATED             STATUS              PORTS               NAMES
b712d13b4fd8        mdillon/postgis:latest   "docker-entrypoint.s…"   28 seconds ago      Up 16 seconds       5432/tcp            dokku.postgres.example

# dokku postgres:list
NAME     VERSION                 STATUS   EXPOSED PORTS  LINKS
example  mdillon/postgis:latest  running  -              -

Example dokku service destruction

# dokku postgres:destroy example
 !     WARNING: Potentially Destructive Action
 !     This command will destroy example Postgres service.
 !     To proceed, type "example"

> example
=====> Deleting example
=====> Stopping container
       Container stopped
       Removing container
       Removing data
=====> Postgres container deleted: example

# dokku postgres:list
 !     There are no Postgres services

# docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES

Major points

disable-chown option can’t solve the issue (by our estimate), because dokku could create volume directories in some cases. Any application in a docker container doesn’t have permissions for use mounted volumes with or without disable-chown (if a volume has been created by dokku).

@jakubgs, maybe our workaround will be useful for you.

@josegonzalez, please could you check our workaround? Maybe you already have a dokku-way solution for docker + userns-remap + dokku. Or maybe our workaround isn’t goot and will cause problems in the future.

We would appreciate any information on the topic. Thanks.

+3
mindfl on Mar 27, 2019
Read more comments on GitHub
← dokku: Dokku 0.5.6 returns "Please run: sudo dokku plugin:install" on each plugin request
dokku: Dokku deploys errors with gzip: stdin: invalid compressed data--format violated →