kamal: [Kamal 2] Error: target failed to become healthy
I’m using kamal 2 and a fresh config and a fresh dockerfile.
Logs end with
"User-Agent" on xxx.xxx.x.xx
ERROR Failed to boot web on xxx.xxx.x.xx
INFO First web container is unhealthy on xxx.xxx.x.xx, not booting any other roles
INFO [c43fba2b] Running docker container ls --all --filter name=^app-web-616499a7f846ac1b9e793e4b65fc0d620975af34_uncommitted_5974b526eb0bface$ --quiet | xargs docker logs --timestamps 2>&1 on xxx.xxx.x.xx
INFO [c43fba2b] Finished in 0.086 seconds with exit status 0 (successful).
ERROR 2024-10-03T10:12:09.396377559Z {"time":"2024-10-03T10:12:09.396233974Z","level":"INFO","msg":"Server started","http":":80"}
2024-10-03T10:12:10.030154028Z => Booting Puma
2024-10-03T10:12:10.030170436Z => Rails 7.1.3.2 application starting in devint
2024-10-03T10:12:10.030172811Z => Run `bin/rails server --help` for more startup options
2024-10-03T10:12:10.047581359Z {"time":"2024-10-03T10:12:10.047492864Z","level":"INFO","msg":"Unable to proxy request","path":"/up","error":"dial tcp 127.0.0.1:3000: connect: connection refused"}
2024-10-03T10:12:10.047600740Z {"time":"2024-10-03T10:12:10.047563654Z","level":"INFO","msg":"Request","path":"/up","status":502,"dur":0,"method":"GET","req_content_length":0,"req_content_type":"","resp_content_length":0,"resp_content_type":"text/plain; charset=utf-8","remote_addr":"172.19.0.2:44120","user_agent":"Go-http-client/1.1","cache":"miss","query":""}
2024-10-03T10:12:10.586403629Z Puma starting in single mode...
2024-10-03T10:12:10.586417892Z * Puma version: 6.4.2 (ruby 3.3.0-p0) ("The Eagle of Durango")
2024-10-03T10:12:10.586420350Z * Min threads: 40
2024-10-03T10:12:10.586422283Z * Max threads: 40
2024-10-03T10:12:10.586424126Z * Environment: devint
2024-10-03T10:12:10.586426017Z * PID: 17
2024-10-03T10:12:10.586552488Z * Listening on http://0.0.0.0:3000
2024-10-03T10:12:10.603315796Z Use Ctrl-C to stop
2024-10-03T10:12:11.048120983Z {"time":"2024-10-03T10:12:11.048046409Z","level":"INFO","msg":"Request","path":"/up","status":301,"dur":1,"method":"GET","req_content_length":0,"req_content_type":"","resp_content_length":0,"resp_content_type":"text/html; charset=utf-8","remote_addr":"172.19.0.2:44120","user_agent":"Go-http-client/1.1","cache":"miss","query":""}
2024-10-03T10:12:12.047512910Z {"time":"2024-10-03T10:12:12.04732821Z","level":"INFO","msg":"Request","path":"/up","status":301,"dur":0,"method":"GET","req_content_length":0,"req_content_type":"","resp_content_length":0,"resp_content_type":"text/html; charset=utf-8","remote_addr":"172.19.0.2:44120","user_agent":"Go-http-client/1.1","cache":"miss","query":""}
2024-10-03T10:12:13.047846408Z {"time":"2024-10-03T10:12:13.047729937Z","level":"INFO","msg":"Request","path":"/up","status":301,"dur":0,"method":"GET","req_content_length":0,"req_content_type":"","resp_content_length":0,"resp_content_type":"text/html; charset=utf-8","remote_addr":"172.19.0.2:44120","user_agent":"Go-http-client/1.1","cache":"miss","query":""}
2024-10-03T10:12:14.047807686Z {"time":"2024-10-03T10:12:14.047593626Z","level":"INFO","msg":"Request","path":"/up","status":301,"dur":0,"method":"GET","req_content_length":0,"req_content_type":"","resp_content_length":0,"resp_content_type":"text/html; charset=utf-8","remote_addr":"172.19.0.2:44120","user_agent":"Go-http-client/1.1","cache":"miss","query":""}
2024-10-03T10:12:15.047478119Z {"time":"2024-10-03T10:12:15.047384139Z","level":"INFO","msg":"Request","path":"/up","status":301,"dur":0,"method":"GET","req_content_length":0,"req_content_type":"","resp_content_length":0,"resp_content_type":"text/html; charset=utf-8","remote_addr":"172.19.0.2:44120","user_agent":"Go-http-client/1.1","cache":"miss","query":""}
2024-10-03T10:12:16.047969551Z {"time":"2024-10-03T10:12:16.047850513Z","level":"INFO","msg":"Request","path":"/up","status":301,"dur":0,"method":"GET","req_content_length":0,"req_content_type":"","resp_content_length":0,"resp_content_type":"text/html; charset=utf-8","remote_addr":"172.19.0.2:44120","user_agent":"Go-http-client/1.1","cache":"miss","query":""}
2024-10-03T10:12:17.047529146Z {"time":"2024-10-03T10:12:17.047391056Z","level":"INFO","msg":"Request","path":"/up","status":301,"dur":0,"method":"GET","req_content_length":0,"req_content_type":"","resp_content_length":0,"resp_content_type":"text/html; charset=utf-8","remote_addr":"172.19.0.2:44120","user_agent":"Go-http-client/1.1","cache":"miss","query":""}
2024-10-03T10:12:18.047754441Z {"time":"2024-10-03T10:12:18.047662475Z","level":"INFO","msg":"Request","path":"/up","status":301,"dur":0,"method":"GET","req_content_length":0,"req_content_type":"","resp_content_length":0,"resp_content_type":"text/html; charset=utf-8","remote_addr":"172.19.0.2:44120","user_agent":"Go-http-client/1.1","cache":"miss","query":""}
2024-10-03T10:12:19.047525119Z {"time":"2024-10-03T10:12:19.047406314Z","level":"INFO","msg":"Request","path":"/up","status":301,"dur":0,"method":"GET","req_content_length":0,"req_content_type":"","resp_content_length":0,"resp_content_type":"text/html; charset=utf-8","remote_addr":"172.19.0.2:44120","user_agent":"Go-http-client/1.1","cache":"miss","query":""}
2024-10-03T10:12:20.047555951Z {"time":"2024-10-03T10:12:20.047457548Z","level":"INFO","msg":"Request","path":"/up","status":301,"dur":0,"method":"GET","req_content_length":0,"req_content_type":"","resp_content_length":0,"resp_content_type":"text/html; charset=utf-8","remote_addr":"172.19.0.2:44120","user_agent":"Go-http-client/1.1","cache":"miss","query":""}
2024-10-03T10:12:21.048005701Z {"time":"2024-10-03T10:12:21.047904734Z","level":"INFO","msg":"Request","path":"/up","status":301,"dur":0,"method":"GET","req_content_length":0,"req_content_type":"","resp_content_length":0,"resp_content_type":"text/html; charset=utf-8","remote_addr":"172.19.0.2:44120","user_agent":"Go-http-client/1.1","cache":"miss","query":""}
2024-10-03T10:12:22.047418577Z {"time":"2024-10-03T10:12:22.047325291Z","level":"INFO","msg":"Request","path":"/up","status":301,"dur":0,"method":"GET","req_content_length":0,"req_content_type":"","resp_content_length":0,"resp_content_type":"text/html; charset=utf-8","remote_addr":"172.19.0.2:44120","user_agent":"Go-http-client/1.1","cache":"miss","query":""}
2024-10-03T10:12:23.047766252Z {"time":"2024-10-03T10:12:23.047654357Z","level":"INFO","msg":"Request","path":"/up","status":301,"dur":0,"method":"GET","req_content_length":0,"req_content_type":"","resp_content_length":0,"resp_content_type":"text/html; charset=utf-8","remote_addr":"172.19.0.2:44120","user_agent":"Go-http-client/1.1","cache":"miss","query":""}
2024-10-03T10:12:24.047382298Z {"time":"2024-10-03T10:12:24.047286567Z","level":"INFO","msg":"Request","path":"/up","status":301,"dur":0,"method":"GET","req_content_length":0,"req_content_type":"","resp_content_length":0,"resp_content_type":"text/html; charset=utf-8","remote_addr":"172.19.0.2:44120","user_agent":"Go-http-client/1.1","cache":"miss","query":""}
2024-10-03T10:12:25.048162887Z {"time":"2024-10-03T10:12:25.048084503Z","level":"INFO","msg":"Request","path":"/up","status":301,"dur":0,"method":"GET","req_content_length":0,"req_content_type":"","resp_content_length":0,"resp_content_type":"text/html; charset=utf-8","remote_addr":"172.19.0.2:44120","user_agent":"Go-http-client/1.1","cache":"miss","query":""}
2024-10-03T10:12:26.047750958Z {"time":"2024-10-03T10:12:26.047666361Z","level":"INFO","msg":"Request","path":"/up","status":301,"dur":0,"method":"GET","req_content_length":0,"req_content_type":"","resp_content_length":0,"resp_content_type":"text/html; charset=utf-8","remote_addr":"172.19.0.2:44120","user_agent":"Go-http-client/1.1","cache":"miss","query":""}
2024-10-03T10:12:27.047461131Z {"time":"2024-10-03T10:12:27.04729404Z","level":"INFO","msg":"Request","path":"/up","status":301,"dur":0,"method":"GET","req_content_length":0,"req_content_type":"","resp_content_length":0,"resp_content_type":"text/html; charset=utf-8","remote_addr":"172.19.0.2:44120","user_agent":"Go-http-client/1.1","cache":"miss","query":""}
2024-10-03T10:12:28.047998914Z {"time":"2024-10-03T10:12:28.047815193Z","level":"INFO","msg":"Request","path":"/up","status":301,"dur":0,"method":"GET","req_content_length":0,"req_content_type":"","resp_content_length":0,"resp_content_type":"text/html; charset=utf-8","remote_addr":"172.19.0.2:44120","user_agent":"Go-http-client/1.1","cache":"miss","query":""}
2024-10-03T10:12:29.047534102Z {"time":"2024-10-03T10:12:29.04739252Z","level":"INFO","msg":"Request","path":"/up","status":301,"dur":0,"method":"GET","req_content_length":0,"req_content_type":"","resp_content_length":0,"resp_content_type":"text/html; charset=utf-8","remote_addr":"172.19.0.2:44120","user_agent":"Go-http-client/1.1","cache":"miss","query":""}
2024-10-03T10:12:30.047297465Z {"time":"2024-10-03T10:12:30.047186348Z","level":"INFO","msg":"Request","path":"/up","status":301,"dur":0,"method":"GET","req_content_length":0,"req_content_type":"","resp_content_length":0,"resp_content_type":"text/html; charset=utf-8","remote_addr":"172.19.0.2:44120","user_agent":"Go-http-client/1.1","cache":"miss","query":""}
2024-10-03T10:12:31.047697501Z {"time":"2024-10-03T10:12:31.047600294Z","level":"INFO","msg":"Request","path":"/up","status":301,"dur":0,"method":"GET","req_content_length":0,"req_content_type":"","resp_content_length":0,"resp_content_type":"text/html; charset=utf-8","remote_addr":"172.19.0.2:44120","user_agent":"Go-http-client/1.1","cache":"miss","query":""}
2024-10-03T10:12:32.047398803Z {"time":"2024-10-03T10:12:32.047280902Z","level":"INFO","msg":"Request","path":"/up","status":301,"dur":0,"method":"GET","req_content_length":0,"req_content_type":"","resp_content_length":0,"resp_content_type":"text/html; charset=utf-8","remote_addr":"172.19.0.2:44120","user_agent":"Go-http-client/1.1","cache":"miss","query":""}
2024-10-03T10:12:33.047837629Z {"time":"2024-10-03T10:12:33.047736866Z","level":"INFO","msg":"Request","path":"/up","status":301,"dur":0,"method":"GET","req_content_length":0,"req_content_type":"","resp_content_length":0,"resp_content_type":"text/html; charset=utf-8","remote_addr":"172.19.0.2:44120","user_agent":"Go-http-client/1.1","cache":"miss","query":""}
2024-10-03T10:12:34.047391468Z {"time":"2024-10-03T10:12:34.047251666Z","level":"INFO","msg":"Request","path":"/up","status":301,"dur":0,"method":"GET","req_content_length":0,"req_content_type":"","resp_content_length":0,"resp_content_type":"text/html; charset=utf-8","remote_addr":"172.19.0.2:44120","user_agent":"Go-http-client/1.1","cache":"miss","query":""}
2024-10-03T10:12:35.047979415Z {"time":"2024-10-03T10:12:35.047800354Z","level":"INFO","msg":"Request","path":"/up","status":301,"dur":0,"method":"GET","req_content_length":0,"req_content_type":"","resp_content_length":0,"resp_content_type":"text/html; charset=utf-8","remote_addr":"172.19.0.2:44120","user_agent":"Go-http-client/1.1","cache":"miss","query":""}
2024-10-03T10:12:36.047563982Z {"time":"2024-10-03T10:12:36.047367517Z","level":"INFO","msg":"Request","path":"/up","status":301,"dur":0,"method":"GET","req_content_length":0,"req_content_type":"","resp_content_length":0,"resp_content_type":"text/html; charset=utf-8","remote_addr":"172.19.0.2:44120","user_agent":"Go-http-client/1.1","cache":"miss","query":""}
2024-10-03T10:12:37.047533440Z {"time":"2024-10-03T10:12:37.04743844Z","level":"INFO","msg":"Request","path":"/up","status":301,"dur":0,"method":"GET","req_content_length":0,"req_content_type":"","resp_content_length":0,"resp_content_type":"text/html; charset=utf-8","remote_addr":"172.19.0.2:44120","user_agent":"Go-http-client/1.1","cache":"miss","query":""}
2024-10-03T10:12:38.047773791Z {"time":"2024-10-03T10:12:38.047710488Z","level":"INFO","msg":"Request","path":"/up","status":301,"dur":0,"method":"GET","req_content_length":0,"req_content_type":"","resp_content_length":0,"resp_content_type":"text/html; charset=utf-8","remote_addr":"172.19.0.2:44120","user_agent":"Go-http-client/1.1","cache":"miss","query":""}
INFO [0d99e151] Running docker container ls --all --filter name=^app-web-616499a7f846ac1b9e793e4b65fc0d620975af34_uncommitted_5974b526eb0bface$ --quiet | xargs docker inspect --format '{{json .State.Health}}' on xxx.xxx.x.xx
INFO [0d99e151] Finished in 0.079 seconds with exit status 0 (successful).
ERROR null
INFO [92917103] Running docker container ls --all --filter name=^app-web-616499a7f846ac1b9e793e4b65fc0d620975af34_uncommitted_5974b526eb0bface$ --quiet | xargs docker stop on xxx.xxx.x.xx
INFO [92917103] Finished in 12.182 seconds with exit status 0 (successful).
Releasing the deploy lock...
Finished all in 158.3 seconds
ERROR (SSHKit::Command::Failed): Exception while executing on host xxx.xxx.x.xx: docker exit status: 1
docker stdout: Nothing written
docker stderr: Error: target failed to become healthy
Dockerfile:
# syntax = docker/dockerfile:1
# Make sure RUBY_VERSION matches the Ruby version in .ruby-version and Gemfile
ARG RUBY_VERSION=3.3.0
FROM ruby:$RUBY_VERSION-slim AS base
# Rails app lives here
WORKDIR /rails
# Set production environment
ENV BUNDLE_DEPLOYMENT="1" \
BUNDLE_PATH="/usr/local/bundle" \
#BUNDLE_WITHOUT="development:test" \
RAILS_ENV="devint"
# Update gems and bundler
RUN gem update --system --no-document && \
gem install -N bundler
# Throw-away build stage to reduce size of final image
FROM base AS build
# Install packages needed to build gems and node modules
RUN apt-get update -qq && \
apt-get install --no-install-recommends -y build-essential curl default-libmysqlclient-dev libvips node-gyp pkg-config python-is-python3
# Install Node.js
ARG NODE_VERSION=18.15.0
ENV PATH=/usr/local/node/bin:$PATH
RUN curl -sL https://github.com/nodenv/node-build/archive/master.tar.gz | tar xz -C /tmp/ && \
/tmp/node-build-master/bin/node-build "${NODE_VERSION}" /usr/local/node && \
rm -rf /tmp/node-build-master
# Install application gems
COPY Gemfile Gemfile.lock ./
RUN bundle install && \
bundle exec bootsnap precompile --gemfile && \
rm -rf ~/.bundle/ "${BUNDLE_PATH}"/ruby/*/cache "${BUNDLE_PATH}"/ruby/*/bundler/gems/*/.git
# Install node modules
COPY package.json package-lock.json ./
RUN npm install
# Copy application code
COPY . .
# Precompile bootsnap code for faster boot times
RUN bundle exec bootsnap precompile app/ lib/
# Adjust binfiles to be executable on Linux
RUN chmod +x bin/* && \
sed -i "s/\r$//g" bin/* && \
sed -i 's/ruby\.exe$/ruby/' bin/*
# Precompiling assets for production without requiring secret RAILS_MASTER_KEY
RUN SECRET_KEY_BASE_DUMMY=1 ./bin/rails assets:precompile
RUN SECRET_KEY_BASE_DUMMY=1 ./bin/rails tailwindcss:build
# Final stage for app image
FROM base
# Install packages needed for deployment
RUN apt-get update -qq && \
apt-get install --no-install-recommends -y curl default-mysql-client imagemagick libsqlite3-0 libvips && \
rm -rf /var/lib/apt/lists /var/cache/apt/archives
# Copy built artifacts: gems, application
COPY --from=build "${BUNDLE_PATH}" "${BUNDLE_PATH}"
COPY --from=build /rails /rails
# Run and own only the runtime files as a non-root user for security
RUN groupadd --system --gid 1000 rails && \
useradd rails --uid 1000 --gid 1000 --create-home --shell /bin/bash && \
chown -R 1000:1000 db log storage tmp
USER 1000:1000
# Entrypoint prepares the database.
ENTRYPOINT ["/rails/bin/docker-entrypoint"]
# Start the server by default, this can be overwritten at runtime
EXPOSE 80
CMD ["bundle", "exec", "thrust", "./bin/rails", "server"]
deploy.yml
# Name of your application. Used to uniquely configure containers.
service: app
# Name of the container image.
image: mycomp/myapp
# Deploy to these servers.
servers:
web:
- xxx.xxx.x.xx
# job:
# hosts:
# - 192.168.0.1
# cmd: bin/jobs
# Enable SSL auto certification via Let's Encrypt (and allow for multiple apps on one server).
# Set ssl: false if using something like Cloudflare to terminate SSL (but keep host!).
proxy:
ssl: false
host: devint.mydomain.ee
registry:
username: myusername
password:
- KAMAL_REGISTRY_PASSWORD
# Configure builder setup.
builder:
arch: amd64
context: .
ssh:
user: rando
# Inject ENV variables into containers (secrets come from .kamal/secrets).
#
# env:
# clear:
# DB_HOST: 192.168.0.2
# secret:
# - RAILS_MASTER_KEY
env:
clear:
DB_HOST: xxx.xxx.x.xx
RAILS_LOG_TO_STDOUT: 1
RUBY_YJIT_ENABLE: 1
RAILS_SERVE_STATIC_FILES: true
RAILS_MAX_THREADS: 40
secret:
- RAILS_MASTER_KEY
- MYSQL_ROOT_PASSWORD
asset_path: /rails/public/assets
accessories:
db:
image: mariadb:11.4
host: xxx.xxx.x.xx
port: 3306
env:
clear:
MYSQL_ROOT_HOST: '%'
secret:
- MYSQL_ROOT_PASSWORD
files:
# - config/mysql/production.cnf:/etc/mysql/my.cnf
- db/production.sql:/docker-entrypoint-initdb.d/setup.sql
directories:
- data:/var/lib/mysql
What am I missing? devint domain root path serves blue 404 page I assume comes from kamal-proxy.
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 72 (8 by maintainers)
I also faced this problem while migrating my Rails 7.2.1 app from Kama 1.9 to 2.1. On comparing, rails 7 and 8 boilerplates generated by
rails new, found that rails 8 has bothassume_sslandforce_ssltrue, while rails 7 has onlyforce_ssltrue.So just making
assume_ssltrue in my rails 7.2 app made the deployment go through smoothly with Kamal 2.1.In summary just set the following in your production.rb:
This way you can avoid all the other configuration on
/upendpointI’ve also encountered this problem. The problem comes from the “/up” route which must remain reachable without SSL. You just need to add a configuration line in
production.rb. I actually found this configuration in a fresh rails 8 app.# production.rbUse
config.force_ssltogether withconfig.assume_ssl. Then Rails will still act as though you’re on SSL, which is important for setting secure cookies etc. This is the default for Rails 8+.It’s because Kamal doesn’t care if your app is a Rails app. You can deploy any dockerized app, in any language.
Kamal asks docker to build the whatever app with the Dockerfile, push the resulting image to any registry you config, and asks his remote partner Kamal-server to pull it and deploy it on your server. Kamal doesn’t know your app is a rails’ app.
Kamal needs your app to be healthy to finish. If your app isn’t responding properly to the health check (like misconfiguration about SSL), Kamal can’t finish. So, your app’s responsibility is to respond 200 to the health check request with SSL.
Got hung up on this with a Django install. This post got me up and running. https://www.coryzue.com/writing/kamal-django/
Summary is create a middleware and put it first in the list to bypass the
ALLOWED_HOSTScheck and HTTPS redirects.@jjatinggoyal it worked thanks, also thank you all of you that take some minutes to help me to solve this issue.
@guilhermecaixeta Have you tried using
thrustergem?Gemfile
Dockerfile
Running on Rails 7, I also had to make sure
force_sslwas off, and/uproute was properly configured.production.rb
routes.rb
If you look at the bottom of the Dockerfile, you can see
The container is supposed to be using basecamp/thruster which by default should expose port 80. Nowhere has port 3000 been defined. Adding your suggestion of app_port makes things worse.
2024-10-03T14:23:01.810967693Z 2024-10-03 14:23:01 +0000 HTTP parse error, malformed request: #<Puma::HttpParserError: Invalid HTTP format, parsing fails. Are you trying to open an SSL connection to a non-SSL Puma?>. Thruster is included in the gemfile.@dhh, thruster doesn’t need any fancy config, does it?
I ran into this for a different reason, in case anyone else encounters this.
My problem was that my target instance was under-provisioned (Digital Ocean Basic / 512 MB / 1 vCPU).
The behavior is that an initial
kamal deployorkamal setupwould be successful, but subsequent deploys would not. Thedocker exec kamal-proxy kamal-proxy deploy my-app-here-webstep would hang (for much longer than 30 seconds, which should have been a hint to me) and then ultimately complain with this error.I noticed it was also quite slow to connect to the target box and/or run commands on an active shell. Running
topshowed that swap was very active. So, probably what happens is that running two containers during handoff eats up more resources and grinds the health check to a halt and blue-green cannot be verified.It’s an easy gotcha if you’re experimenting with Kamala on the cheapest cloud instance you can find, which is what I was doing 😄
This is what worked for me on Rails 7.2.2, I mean setting
config.assume_ssltotrue.@jjatinggoyal you’re the MVP! 😄 Thanks!
@dhh Thanks for explaining how healtchecks are used for gap-less deployments. That actually makes a lot of sense and inspired me to find a solution for this.
Thanks to the post by @scuml , I have this solution for Nextjs users struggling with getting healthchecks working on Kamal 2.0.
Place this in your middleware. Point your healthcheck at
/up.I don’t see any path to deployment without a working healthcheck (whether that’s /up or HEALTHCHECK inside the container) since that’s the mechanism we need to ensure the gap-less deploy. Otherwise you’re going to be throwing 500 errors from the proxy for the duration of the app boot, which isn’t what anyone wants.
On a fresh Hertz instance with a Rails 7.2.1 application and Kamal 2, I had to set
config.force_ssl = falseto get the healthcheck working.I’ve deployed a new Rails 8 app onto it and I didn’t need to set that. I’m not sure why.
Expanding on previous comment - see https://nts.strzibny.name/upgrading-to-kamal-2/ - try
config.host_authorization = { exclude: ->(request) { request.path == "/up" } }(I didn’t end up requiring this, YMMV).Also do you have
config.force_ssl = false?By default, Kamal Proxy checks port 80. Since you are using port 3000, specify the app_port setting in the proxy configuration:
This is the setting that solved the health check problem for me.
I’v found what was the cause of my issues, it’s kinda edge case but may help someone because it was very hard to trace. I have the engine which is used in the app, and there was a line inside Engine class within initializer, require for non existing file. I have no idea why I haven’t seen any error on build stage but container for sure was not healthy.
@easydatawarehousing instructions worked for me, after hours of poking around. The issue was I needed to go into DigitalOcean managed Postgres instance and adjust my “Trusted Sources” so the “Droplet” could access the database.
From my DigitalOcean “Droplet” (not even from within the Docker container)
Another peculiar thing that helped me understand what was going on is that
kamal app bootworked fine butkamal app stopthenkamal app startwould report the start as failing.It’s been a rough two days with Kamal 2 + Rails 8. I see the vision, though. I think there needs to be a lot better handoff between Rails 8, Kamal 2 proxy, and Kamal CLI output. In this case I’d hope to see a “Couldn’t connect to database” message.
I’m back! The problem was not related to Kamal. I was using a firewall in my EC2 that was not allowing the proxy to check if the server was healthy. After some tweaks to the firewall rules, it finally worked.
@pftg I think my settings are correct. The SSL Setting in Cloudflare is set to Full, the production.rb contains assume_ssl and force_ssl set as true, and the proxy config contains ssl: true. Am I missing something?
Yep, I found that as well https://github.com/basecamp/kamal/issues/1041#issuecomment-2408919805
Was definitely what I needed, but I also have other setup steps running in my
bin/docker-entrypointso I had to adjust mydeploy_timeoutin the kamal config:This stopped the following from happening again:
I recall us investigating the healthcheck feature in the past and finding it wasn’t suitable for our needs, especially around the gap-less deploys. Don’t recall the specifics, though. Maybe @djmb remembers why we discarded the healthcheck as an option.
Either way, it’s not something that’s on the menu to change in the short term. Tons of successful deployments in the wild now that are built around using /up as the healthcheck.
Yeah this is still a problem for other frameworks which don’t let you do HTTP/S handling on a per-route basis. For instance, I am trying to host a Next 14 project and running into this problem.
I was able to fix this error on my end.
production.rb
config.force_ssl = falsedeploy.yml
Disabled force_ssl from Rails to make way for Kamal2 SSL.
Problem: After changing the
Dockerfile, I had the container health problem again.Solution: This time, I increased
deploy_timeout, healthcheck’sintervalandtimeoutdurations (to wait before starting the health check and increased timeout from 3 seconds to 30 seconds), which helped@kishaningithub I don’t know why the deployment process still sent request to “/up” it’s really strange for me.
But I think this log line is the correct health check through docker :
INFO [68e47201] Running docker container ls --all --filter name=^wordtrial-web-eb801b29d0ab51fb77c80f988f0ea128838ade00$ --quiet | xargs docker inspect --format '{{json .State.Health}}' on 139.59.18.46But we can see some warning in the “/up” logs. Maybe your app doesn’t boot properly? Maybe you should check if your app can start alone by starting it manually.
kamal app images,kamal app startIf you can access to your app, your app is booting, but the health check doesn’t pass. I already have this issue and the solution was to permit non-ssl (http) request to “/up” in my case. As your health check path is “/” I suggest you to add a dedicated health check’s route for permitting non-ssl request only on this path.
@kishaningithub Yes you are right. Few lines after in the documentation describes this :
I think you can pass the health check barrier with a custom health check. As you should prove aliveness of the container, you should be good.
@dhh I am facing this same issue while deploying a wordpress site with
proxy: falseLogs of
kamal deployIf you notice below the proxy looks alive and it is continously pinging
/upendpoint. This is incorrect behaviour whenproxy: falseKamal config file
Docker file
My bad, so I was using this gem https://github.com/rameerez/allgood to cover more ground with my health checks, and some of my checks defined there were failing, but without a proper error to understand the cause.
Everything works after removing it!
@guilhermecaixeta or you can also add thruster. You will need some change in the entrypoint file.
Yeah, you can’t run both versions on the same box at the same time.