dokku: Stuck on procfile

Description of problem

I have no idea how to describe what’s happening. I have multiple apps in dock, which are deployed on multiple servers, and I haven’t made any changes to any of the apps, but I’m assuming that Ubuntu or Dokku or Docker or the buildpack has made updates.

This is the part where the deploy gets stuck:

       Using release configuration from last framework (Ruby).
-----> Discovering process types
       Procfile declares types -> sidekiq, web, webpacker

It gets stuck here, I can’t stop it with dokku ps:stop or with docker stop and it just hangs.

Any idea of how to debug this better or if anyone else has experienced this issue lately?

dokku report $APP_NAME

➜  ~ dokku report ulsproduction
-----> uname: Linux shibuya 5.13.0-39-generic #44~20.04.1-Ubuntu SMP Thu Mar 24 16:43:35 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
-----> memory: 
                     total        used        free      shared  buff/cache   available
       Mem:         644940        3090      634282          78        7566      637847
       Swap:          2047           0        2047
-----> docker version: 
       Client: Docker Engine - Community
        Version:           20.10.14
        API version:       1.41
        Go version:        go1.16.15
        Git commit:        a224086
        Built:             Thu Mar 24 01:48:02 2022
        OS/Arch:           linux/amd64
        Context:           default
        Experimental:      true
       
       Server: Docker Engine - Community
        Engine:
         Version:          20.10.14
         API version:      1.41 (minimum version 1.12)
         Go version:       go1.16.15
         Git commit:       87a90dc
         Built:            Thu Mar 24 01:45:53 2022
         OS/Arch:          linux/amd64
         Experimental:     false
        containerd:
         Version:          1.5.11
         GitCommit:        3df54a852345ae127d1fa3092b95168e4a88e2f8
        runc:
         Version:          1.0.3
         GitCommit:        v1.0.3-0-gf46b6ba
        docker-init:
         Version:          0.19.0
         GitCommit:        de40ad0
-----> docker daemon info: 
       Client:
        Context:    default
        Debug Mode: true
        Plugins:
         app: Docker App (Docker Inc., v0.9.1-beta3)
         buildx: Docker Buildx (Docker Inc., v0.8.1-docker)
         scan: Docker Scan (Docker Inc., v0.17.0)
       
       Server:
        Containers: 38
         Running: 22
         Paused: 0
         Stopped: 16
        Images: 97
        Server Version: 20.10.14
        Storage Driver: overlay2
         Backing Filesystem: extfs
         Supports d_type: true
         Native Overlay Diff: true
         userxattr: false
        Logging Driver: loki
        Cgroup Driver: cgroupfs
        Cgroup Version: 1
        Plugins:
         Volume: local
         Network: bridge host ipvlan macvlan null overlay
         Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
        Swarm: inactive
        Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
        Default Runtime: runc
        Init Binary: docker-init
        containerd version: 3df54a852345ae127d1fa3092b95168e4a88e2f8
        runc version: v1.0.3-0-gf46b6ba
        init version: de40ad0
        Security Options:
         apparmor
         seccomp
          Profile: default
        Kernel Version: 5.13.0-39-generic
        Operating System: Ubuntu 20.04.4 LTS
        OSType: linux
        Architecture: x86_64
        CPUs: 40
        Total Memory: 629.8GiB
        Name: shibuya
        ID: CNKP:JS5H:D4ZV:ZSOE:CGXZ:XXIF:3K56:OXFO:GXOB:46QI:Y57R:DBOP
        Docker Root Dir: /var/lib/docker
        Debug Mode: false
        Registry: https://index.docker.io/v1/
        Labels:
        Experimental: false
        Insecure Registries:
         127.0.0.0/8
        Live Restore Enabled: false
       
-----> git version: git version 2.25.1
-----> sigil version: 0.8.1build+dce1c9e
-----> herokuish version: 
       herokuish: 0.5.34
       buildpacks:
         heroku-buildpack-multi     v1.0.0
         heroku-buildpack-ruby      v236
         heroku-buildpack-nodejs    v189
         heroku-buildpack-clojure   v87
         heroku-buildpack-python    v205
         heroku-buildpack-java      v69
         heroku-buildpack-gradle    v36
         heroku-buildpack-scala     v91
         heroku-buildpack-play      v26
         heroku-buildpack-php       v205
         heroku-buildpack-go        v160
         heroku-buildpack-nginx     v16
         buildpack-null             v3
-----> dokku version: dokku version 0.27.0
-----> plugn version: plugn: 0.8.2build+1b2e2b4
-----> dokku plugins: 
         00_dokku-standard    0.27.0 enabled    dokku core standard plugin
         20_events            0.27.0 enabled    dokku core events logging plugin
         app-json             0.27.0 enabled    dokku core app-json plugin
         apps                 0.27.0 enabled    dokku core apps plugin
         builder              0.27.0 enabled    dokku core builder plugin
         builder-dockerfile   0.27.0 enabled    dokku core builder-dockerfile plugin
         builder-herokuish    0.27.0 enabled    dokku core builder-herokuish plugin
         builder-null         0.27.0 enabled    dokku core builder-null plugin
         builder-pack         0.27.0 enabled    dokku core builder-pack plugin
         buildpacks           0.27.0 enabled    dokku core buildpacks plugin
         certs                0.27.0 enabled    dokku core certificate management plugin
         checks               0.27.0 enabled    dokku core checks plugin
         common               0.27.0 enabled    dokku core common plugin
         config               0.27.0 enabled    dokku core config plugin
         cron                 0.27.0 enabled    dokku core cron plugin
         docker-options       0.27.0 enabled    dokku core docker-options plugin
         domains              0.27.0 enabled    dokku core domains plugin
         enter                0.27.0 enabled    dokku core enter plugin
         git                  0.27.0 enabled    dokku core git plugin
         logs                 0.27.0 enabled    dokku core logs plugin
         network              0.27.0 enabled    dokku core network plugin
         nginx-vhosts         0.27.0 enabled    dokku core nginx-vhosts plugin
         plugin               0.27.0 enabled    dokku core plugin plugin
         proxy                0.27.0 enabled    dokku core proxy plugin
         ps                   0.27.0 enabled    dokku core ps plugin
         redis                1.13.0 enabled    dokku redis service plugin
         registry             0.27.0 enabled    dokku core registry plugin
         repo                 0.27.0 enabled    dokku core repo plugin
         resource             0.27.0 enabled    dokku core resource plugin
         run                  0.27.0 enabled    dokku core run plugin
         scheduler            0.25.6 enabled    dokku core scheduler plugin
         scheduler-docker-local 0.27.0 enabled    dokku core scheduler-docker-local plugin
         scheduler-null       0.27.0 enabled    dokku core scheduler-null plugin
         shell                0.27.0 enabled    dokku core shell plugin
         ssh-keys             0.27.0 enabled    dokku core ssh-keys plugin
         storage              0.27.0 enabled    dokku core storage plugin
         trace                0.27.0 enabled    dokku core trace plugin
=====> ulsproduction app-json information
       App json computed selected:    app.json
       App json global selected:      app.json
       App json selected:             
=====> ulsproduction app information
       App created at:                1649458705
       App deploy source:             
       App deploy source metadata:    
       App dir:                       /home/dokku/ulsproduction
       App locked:                    true
=====> ulsproduction builder information
       Builder build dir:             
       Builder computed build dir:    
       Builder computed selected:     
       Builder global build dir:      
       Builder global selected:       
       Builder selected:              
=====> ulsproduction builder-dockerfile information
       Builder dockerfile computed dockerfile path: Dockerfile               
       Builder dockerfile global dockerfile path: Dockerfile               
       Builder dockerfile dockerfile path:                          
=====> ulsproduction builder-pack information
       Builder pack computed projecttoml path: project.toml             
       Builder pack global projecttoml path: project.toml             
       Builder pack projecttoml path:                          
=====> ulsproduction buildpacks information
       Buildpacks computed stack:     gliderlabs/herokuish:latest-20
       Buildpacks global stack:       
       Buildpacks list:               
       Buildpacks stack:              
=====> ulsproduction ssl information
       Ssl dir:                       /home/dokku/ulsproduction/tls
       Ssl enabled:                   false                    
       Ssl hostnames:                                          
       Ssl expires at:                                         
       Ssl issuer:                                             
       Ssl starts at:                                          
       Ssl subject:                                            
       Ssl verified:                                           
=====> ulsproduction checks information
       Checks disabled list:          none                     
       Checks skipped list:           none                     
=====> ulsproduction cron information
       Cron task count:               0
=====> ulsproduction docker options information
       Docker options build:          --link dokku.redis.ulsproduction:dokku-redis-ulsproduction 
       Docker options deploy:         --link dokku.redis.ulsproduction:dokku-redis-ulsproduction --restart=on-failure:10 
       Docker options run:            --link dokku.redis.ulsproduction:dokku-redis-ulsproduction 
=====> ulsproduction domains information
       Domains app enabled:           true                     
       Domains app vhosts:            ulsproduction.shibuya    
       Domains global enabled:        true                     
       Domains global vhosts:         shibuya                  
=====> ulsproduction git information
       Git deploy branch:             master                   
       Git global deploy branch:      master                   
       Git keep git dir:              false                    
       Git rev env var:               GIT_REV                  
       Git sha:                       518c055                  
       Git last updated at:                                    
=====> ulsproduction logs information
       Logs computed max size:        10m
       Logs global max size:          10m
       Logs global vector sink:       
       Logs max size:                 
       Logs vector sink:              
=====> ulsproduction network information
       Network attach post create:           
       Network attach post deploy:           
       Network bind all interfaces:          false
       Network computed attach post create:  
       Network computed attach post deploy:  
       Network computed bind all interfaces: false
       Network computed initial network:     
       Network computed tld:                 
       Network global attach post create:    
       Network global attach post deploy:    
       Network global bind all interfaces:   false
       Network global initial network:       
       Network global tld:                   
       Network initial network:              
       Network static web listener:          
       Network tld:                          
       Network web listeners:                172.17.0.7:5000 172.17.0.3:5000 172.17.0.4:5000
=====> ulsproduction nginx information
       Nginx access log format:                                
       Nginx access log path:         /var/log/nginx/ulsproduction-access.log
       Nginx bind address ipv4:                                
       Nginx bind address ipv6:       ::                       
       Nginx client max body size:                             
       Nginx disable custom config:   false                    
       Nginx error log path:          /var/log/nginx/ulsproduction-error.log
       Nginx global hsts:             true                     
       Nginx computed hsts:           true                     
       Nginx hsts:                                             
       Nginx hsts include subdomains: true                     
       Nginx hsts max age:            15724800                 
       Nginx hsts preload:            false                    
       Nginx proxy buffer size:       4096                     
       Nginx proxy buffering:         on                       
       Nginx proxy buffers:           8 4096                   
       Nginx proxy busy buffers size: 8192                     
       Nginx proxy read timeout:      60s                      
       Nginx last visited at:         1649450513               
       Nginx x forwarded for value:   $remote_addr             
       Nginx x forwarded port value:  $server_port             
       Nginx x forwarded proto value: $scheme                  
       Nginx x forwarded ssl:                                  
=====> ulsproduction proxy information
       Proxy enabled:                 true
       Proxy port map:                http:80:5000
       Proxy type:                    nginx
Error: No such object: 148506f0e8cb1e52abb7d056b585f09da43dec345568341a888df66fbc4f44e3
Error: No such object: f5676be626903869d99507e92287a811c9ead596d723d9e882f31b4b473c7b59
Error: No such object: 4b133d26321828dfe4054afe855336a1c191e2b0a7d979fee443cc1086bf9f74
Error: No such object: d46f395af359e67e1b0c812f6b542debac556652556903f4ebdee6829c53c6c6
Error: No such object: 45bec4983329f4ad97a07d5b24c607789b424f41a3922d3e8bd17ab81ec1cffd
=====> ulsproduction ps information
       Deployed:                      true
       Processes:                     5
       Ps can scale:                  true
       Ps computed procfile path:     Procfile
       Ps global procfile path:       Procfile
       Ps procfile path:              
       Ps restart policy:             on-failure:10
       Restore:                       false
       Running:                       false
       Status sidekiq 1:              missing (CID: 4b133d26321)
       Status sidekiq 2:              missing (CID: d46f395af35)
       Status web 1:                  missing (CID: 45bec498332)
       Status web 2:                  missing (CID: f5676be6269)
       Status web 3:                  missing (CID: 148506f0e8c)
=====> ulsproduction registry information
       Registry computed image repo:      dokku/ulsproduction
       Registry computed push on release: false
       Registry computed server:          
       Registry global push on release:   
       Registry global server:            
       Registry image repo:               
       Registry push on release:          
       Registry server:                   
       Registry tag version:              
=====> ulsproduction resource information
=====> ulsproduction scheduler information
       Scheduler computed selected:   docker-local
       Scheduler global selected:     docker-local
       Scheduler selected:            
=====> ulsproduction scheduler-docker-local information
       Scheduler docker local disable chown:                          
       Scheduler docker local parallel schedule count:                          
=====> ulsproduction storage information
       Storage build mounts:                                   
       Storage deploy mounts:                                  
       Storage run mounts:                   

Additional information

➜  ~ dokku ps:inspect ulsproduction
Error: No such container: 4b133d26321828dfe4054afe855336a1c191e2b0a7d979fee443cc1086bf9f74
Error: No such container: d46f395af359e67e1b0c812f6b542debac556652556903f4ebdee6829c53c6c6
Error: No such container: 45bec4983329f4ad97a07d5b24c607789b424f41a3922d3e8bd17ab81ec1cffd
Error: No such container: f5676be626903869d99507e92287a811c9ead596d723d9e882f31b4b473c7b59
Error: No such container: 148506f0e8cb1e52abb7d056b585f09da43dec345568341a888df66fbc4f44e3
[]
2022/04/08 16:37:20 exit status 1
 !     exit status 1

Output of failing deploy after running: dokku trace:off

-----> Detecting rails configuration
       
       ###### WARNING:
       
       Your app was upgraded to bundler 2.2.33.
       Previously you had a successful deploy with bundler 2.3.10.
       
       If you see problems related to the bundler version please refer to:
       https://devcenter.heroku.com/articles/bundler-version#known-upgrade-issues
       
       
       ###### WARNING:
       
       There is a more recent Ruby version available for you to use:
       
       2.7.5
       
       The latest version will include security and bug fixes. We always recommend
       running the latest version of your minor release.
       
       Please upgrade your Ruby version.
       
       For all available Ruby versions see:
       https://devcenter.heroku.com/articles/ruby-support#supported-runtimes
       
       
       Using release configuration from last framework (Ruby).
-----> Discovering process types
       Procfile declares types -> sidekiq, web, webpacker```

Output of failing deploy after running: dokku trace:on

➜  ~ dokku trace:on 
-----> Enabling trace mode