kamal-proxy: Randomly fails to become healthy

We’ve been using kamal with kamal-proxy for a few weeks now and overall it works great. We do have a random case (1 in 15?) where the deployment fails because the new version doesn’t become healthy according to Kamal, although the logs do show correct status code 200 for the healthcheck. Attached the logs of such a failure:

2024-11-22T08:19:10.7613515Z 2024-11-22T08:18:40.620061000Z {"time":"2024-11-22T09:18:40.619937363+01:00","level":"INFO","msg":"Request","path":"/up","status":502,"dur":0,"method":"GET","req_content_length":0,"req_content_type":"","resp_content_length":626,"resp_content_type":"text/html","remote_addr":"172.18.0.3:52158","user_agent":"Go-http-client/1.1","cache":"miss","query":""}
2024-11-22T08:19:10.7616212Z 2024-11-22T08:18:41.619040000Z {"time":"2024-11-22T09:18:41.618760816+01:00","level":"INFO","msg":"Unable to proxy request","path":"/up","error":"dial tcp 127.0.0.1:3000: connect: connection refused"}
2024-11-22T08:19:10.7618864Z 2024-11-22T08:18:41.632961000Z {"time":"2024-11-22T09:18:41.629230642+01:00","level":"INFO","msg":"Request","path":"/up","status":502,"dur":10,"method":"GET","req_content_length":0,"req_content_type":"","resp_content_length":626,"resp_content_type":"text/html","remote_addr":"172.18.0.3:52166","user_agent":"Go-http-client/1.1","cache":"miss","query":""}
2024-11-22T08:19:10.7620828Z 2024-11-22T08:18:42.275092000Z [21] Puma starting in cluster mode...
2024-11-22T08:19:10.7621369Z 2024-11-22T08:18:42.276457000Z [21] * Puma version: 6.4.3 (ruby 3.1.6-p260) ("The Eagle of Durango")
2024-11-22T08:19:10.7621866Z 2024-11-22T08:18:42.277440000Z [21] *  Min threads: 3
2024-11-22T08:19:10.7622224Z 2024-11-22T08:18:42.277496000Z [21] *  Max threads: 3
2024-11-22T08:19:10.7622593Z 2024-11-22T08:18:42.277526000Z [21] *  Environment: staging
2024-11-22T08:19:10.7622974Z 2024-11-22T08:18:42.278793000Z [21] *   Master PID: 21
2024-11-22T08:19:10.7623347Z 2024-11-22T08:18:42.279629000Z [21] *      Workers: 2
2024-11-22T08:19:10.7623991Z 2024-11-22T08:18:42.280388000Z [21] *     Restarts: (✔) hot (✖) phased
2024-11-22T08:19:10.7624421Z 2024-11-22T08:18:42.281572000Z [21] * Preloading application
2024-11-22T08:19:10.7624867Z 2024-11-22T08:18:42.281619000Z [21] * Listening on http://0.0.0.0:3000
2024-11-22T08:19:10.7625292Z 2024-11-22T08:18:42.281651000Z [21] Use Ctrl-C to stop
2024-11-22T08:19:10.7625738Z 2024-11-22T08:18:42.312462000Z [21] - Worker 0 (PID: 36) booted in 0.02s, phase: 0
2024-11-22T08:19:10.7626247Z 2024-11-22T08:18:42.328429000Z [21] - Worker 1 (PID: 37) booted in 0.03s, phase: 0
2024-11-22T08:19:10.7626948Z 2024-11-22T08:18:47.622613000Z {"time":"2024-11-22T09:18:47.620309603+01:00","level":"INFO","msg":"Unable to proxy request","path":"/up","error":"context canceled"}
2024-11-22T08:19:10.7628306Z 2024-11-22T08:18:47.623511000Z {"time":"2024-11-22T09:18:47.620907601+01:00","level":"INFO","msg":"Request","path":"/up","status":502,"dur":5001,"method":"GET","req_content_length":0,"req_content_type":"","resp_content_length":626,"resp_content_type":"text/html","remote_addr":"172.18.0.3:52172","user_agent":"Go-http-client/1.1","cache":"miss","query":""}
2024-11-22T08:19:10.7629858Z 2024-11-22T08:18:53.380502000Z {"time":"2024-11-22T09:18:53.379131964+01:00","level":"INFO","msg":"Unable to proxy request","path":"/up","error":"context canceled"}
2024-11-22T08:19:10.7631211Z 2024-11-22T08:18:53.381686000Z {"time":"2024-11-22T09:18:53.379478313+01:00","level":"INFO","msg":"Request","path":"/up","status":502,"dur":5755,"method":"GET","req_content_length":0,"req_content_type":"","resp_content_length":626,"resp_content_type":"text/html","remote_addr":"172.18.0.3:52184","user_agent":"Go-http-client/1.1","cache":"miss","query":""}
2024-11-22T08:19:10.7632528Z 2024-11-22T08:18:58.388605000Z {"time":"2024-11-22T09:18:58.383575978+01:00","level":"INFO","msg":"Unable to proxy request","path":"/up","error":"context canceled"}
2024-11-22T08:19:10.7633872Z 2024-11-22T08:18:58.389712000Z {"time":"2024-11-22T09:18:58.383848059+01:00","level":"INFO","msg":"Request","path":"/up","status":502,"dur":4999,"method":"GET","req_content_length":0,"req_content_type":"","resp_content_length":626,"resp_content_type":"text/html","remote_addr":"172.18.0.3:46734","user_agent":"Go-http-client/1.1","cache":"miss","query":""}
2024-11-22T08:19:10.7635860Z 2024-11-22T08:18:59.716690000Z {"time":"2024-11-22T09:18:59.716474635+01:00","level":"INFO","msg":"Request","path":"/up","status":200,"dur":1331,"method":"GET","req_content_length":0,"req_content_type":"","resp_content_length":73,"resp_content_type":"text/html; charset=utf-8","remote_addr":"172.18.0.3:46748","user_agent":"Go-http-client/1.1","cache":"miss","query":""}
2024-11-22T08:19:10.7637841Z 2024-11-22T08:18:59.729143000Z {"time":"2024-11-22T09:18:59.728924097+01:00","level":"INFO","msg":"Request","path":"/up","status":200,"dur":9,"method":"GET","req_content_length":0,"req_content_type":"","resp_content_length":73,"resp_content_type":"text/html; charset=utf-8","remote_addr":"172.18.0.3:46750","user_agent":"Go-http-client/1.1","cache":"miss","query":""}
2024-11-22T08:19:19.7084400Z   INFO [fdc801ec] Running docker container ls --all --filter name=^platform-web-staging-18258ef278ae34862e4efeb855abf1960a3a0e37$ --quiet | xargs docker inspect --format '{{json .State.Health}}' on staging.toolsfactory.nl
2024-11-22T08:19:19.7086005Z   INFO [fdc801ec] Finished in 0.453 seconds with exit status 0 (successful).
2024-11-22T08:19:19.7086594Z  ERROR null
2024-11-22T08:19:19.7087661Z   INFO [bb6b6f8e] Running docker container ls --all --filter name=^platform-web-staging-18258ef278ae34862e4efeb855abf1960a3a0e37$ --quiet | xargs docker stop on staging.toolsfactory.nl
2024-11-22T08:19:19.7088948Z   INFO [bb6b6f8e] Finished in 8.501 seconds with exit status 0 (successful).
2024-11-22T08:19:19.7089717Z Releasing the deploy lock...
2024-11-22T08:19:20.0281473Z   Finished all in 68.9 seconds
2024-11-22T08:19:20.0305461Z   ERROR (SSHKit::Command::Failed): Exception while executing on host staging.toolsfactory.nl: docker exit status: 1
2024-11-22T08:19:20.0306650Z docker stdout: Nothing written
2024-11-22T08:19:20.0307308Z docker stderr: Error: target failed to become healthy

What could be the cause of this and how can we prevent it?

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 16

Most upvoted comments

Thanks. Normal deploys are healthy within 20 seconds, so I guess I’ll have to do some more digging to fix this problem for me.

@frenkel thanks for sending those along. From these logs, it does look like the application is taking more than 30s to reach a healthy state. The default deployment timeout is 30s, and you can see where it stops trying 30s into the deployment, at this line:

{"time":"2024-12-02T10:41:30.401027238Z","level":"INFO","msg":"Target failed to become healthy","hosts":["development.example.com","development0.example.com","development1.example.com","development2.example.com","development3.example.com","development4.example.com","development5.example.com","development6.example.com","development7.example.com","development8.example.com","development9.example.com","signup.example.com","demo.example.com"],"target":"a7535f905149:80"}

Kamal Proxy is behaving as expected here.

If your application sometimes takes longer than 30s to start, then increasing the timeout is the right thing to do. It may simply need more time to reliably be ready than it’s currently getting.

However if you feel the application should be starting more quickly, then I think you’ll need to investigate that on the application side. Given that the deployments are working sometimes, it could be worth comparing the logs when it succeeds – if it’s often fairly close to that 30s mark, maybe you’re just seeing small fluctuations in startup time that cause it to sometimes run over?

Thanks for the reply, but it works fine in the other 14 out of 15 deploys. I use thruster and port 80, so no need to change the app_port.