dokku: Pushs and rebuilds fail after server was running out of disk space.
Our server was running out of disk space due to some docker diff files piling up (it seems they need to be cleaned out regularly, but that is a different topic). After making some space, ps:rebuild
is not working anymore, it just returns immediately with no output. And we can also not push anymore to the server, we get:
! [remote rejected] dokku -> master (pre-receive hook declined)
error: failed to push some refs to 'dokku@.....:.....'
Help is greatly appreciated.
dokku report -----> uname: Linux ubuntu-512mb-fra1-01 3.13.0-77-generic #121-Ubuntu SMP Wed Jan 20 10:50:42 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux -----> memory: total used free shared buffers cached Mem: 994 647 346 10 19 111 -/+ buffers/cache: 515 478 Swap: 999 112 887 -----> docker version: Client: Version: 17.05.0-ce API version: 1.29 Go version: go1.7.5 Git commit: 89658be Built: Thu May 4 22:06:06 2017 OS/Arch: linux/amd64
Server: Version: 17.05.0-ce API version: 1.29 (minimum version 1.12) Go version: go1.7.5 Git commit: 89658be Built: Thu May 4 22:06:06 2017 OS/Arch: linux/amd64 Experimental: false -----> docker daemon info: Containers: 12 Running: 5 Paused: 0 Stopped: 7 Images: 32 Server Version: 17.05.0-ce Storage Driver: aufs Root Dir: /var/lib/docker/aufs Backing Filesystem: extfs Dirs: 1133 Dirperm1 Supported: false Logging Driver: json-file Cgroup Driver: cgroupfs Plugins: Volume: local Network: bridge host macvlan null overlay Swarm: inactive Runtimes: runc Default Runtime: runc Init Binary: docker-init containerd version: 9048e5e50717ea4497b757314bad98ea3763c145 runc version: 9c2d8d184e5da67c95d601382adf14862e4f2228 init version: 949e6fa Security Options: apparmor Kernel Version: 3.13.0-77-generic Operating System: Ubuntu 14.04.5 LTS OSType: linux Architecture: x86_64 CPUs: 1 Total Memory: 994MiB Name: ubuntu-512mb-fra1-01 ID: 5XQF:UC63:3ZUV:GI2Y:JEWP:NJHA:NQ6X:5IKC:RRFA:XLTA:IILX:HBUZ Docker Root Dir: /var/lib/docker Debug Mode (client): true Debug Mode (server): false Registry: https://index.docker.io/v1/ Experimental: false Insecure Registries: 127.0.0.0/8 Live Restore Enabled: false
WARNING: No swap limit support -----> sigil version: 0.4.0 -----> herokuish version: herokuish: 0.3.31 buildpacks: heroku-buildpack-multi v1.0.0 heroku-buildpack-ruby v163 heroku-buildpack-nodejs v99 heroku-buildpack-clojure v76 heroku-buildpack-python v109 heroku-buildpack-java v52 heroku-buildpack-gradle v22 heroku-buildpack-grails v21 heroku-buildpack-scala v76 heroku-buildpack-play v26 heroku-buildpack-php v121 heroku-buildpack-go v69 heroku-buildpack-erlang fa17af9 buildpack-nginx v8 -----> dokku version: 0.5.7 -----> dokku plugins: plugn: 0.3.0 00_dokku-standard 0.5.7 enabled dokku core standard plugin 20_events 0.5.7 enabled dokku core events logging plugin apps 0.5.7 enabled dokku core apps plugin build-env 0.5.7 enabled dokku core build-env plugin certs 0.5.7 enabled dokku core certificate management plugin checks 0.5.7 enabled dokku core checks plugin common 0.5.7 enabled dokku core common plugin config 0.5.7 enabled dokku core config plugin docker-options 0.5.7 enabled dokku core docker-options plugin domains 0.5.7 enabled dokku core domains plugin enter 0.5.7 enabled dokku core enter plugin git 0.5.7 enabled dokku core git plugin letsencrypt 0.8.6 enabled Automated installation of let’s encrypt TLS certificates logs 0.5.7 enabled dokku core logs plugin memcached 1.0.0 enabled dokku memcached service plugin named-containers 0.5.7 enabled dokku core named containers plugin nginx-vhosts 0.5.7 enabled dokku core nginx-vhosts plugin plugin 0.5.7 enabled dokku core plugin plugin postgres 1.0.0 enabled dokku postgres service plugin proxy 0.5.7 enabled dokku core proxy plugin ps 0.5.7 enabled dokku core ps plugin shell 0.5.7 enabled dokku core shell plugin storage 0.5.7 enabled dokku core storage plugin tags 0.5.7 enabled dokku core tags plugin tar 0.5.7 enabled dokku core tar plugin
Environment details (AWS, VirtualBox, physical, etc.): Digital Ocean
About this issue
- Original URL
- State: closed
- Created 7 years ago
- Comments: 25 (10 by maintainers)
Using postgres as an example, if you ever have to recover from suddenly deleting/losing all your containers/images/volumes, fear not! Dokku stores the data on disk, so as long as you aren’t mucking about in your
/var/lib/dokku
directory, the following will get your services running again:Once your services are running, a
ps:rebuild
will properly rebuild your applications.