kamal: [Kamal 2] Error: target failed to become healthy

I’m using kamal 2 and a fresh config and a fresh dockerfile.

Logs end with

 "User-Agent" on xxx.xxx.x.xx
 ERROR Failed to boot web on xxx.xxx.x.xx
  INFO First web container is unhealthy on xxx.xxx.x.xx, not booting any other roles
  INFO [c43fba2b] Running docker container ls --all --filter name=^app-web-616499a7f846ac1b9e793e4b65fc0d620975af34_uncommitted_5974b526eb0bface$ --quiet | xargs docker logs --timestamps 2>&1 on xxx.xxx.x.xx
  INFO [c43fba2b] Finished in 0.086 seconds with exit status 0 (successful).
 ERROR 2024-10-03T10:12:09.396377559Z {"time":"2024-10-03T10:12:09.396233974Z","level":"INFO","msg":"Server started","http":":80"}
2024-10-03T10:12:10.030154028Z => Booting Puma
2024-10-03T10:12:10.030170436Z => Rails 7.1.3.2 application starting in devint
2024-10-03T10:12:10.030172811Z => Run `bin/rails server --help` for more startup options
2024-10-03T10:12:10.047581359Z {"time":"2024-10-03T10:12:10.047492864Z","level":"INFO","msg":"Unable to proxy request","path":"/up","error":"dial tcp 127.0.0.1:3000: connect: connection refused"}
2024-10-03T10:12:10.047600740Z {"time":"2024-10-03T10:12:10.047563654Z","level":"INFO","msg":"Request","path":"/up","status":502,"dur":0,"method":"GET","req_content_length":0,"req_content_type":"","resp_content_length":0,"resp_content_type":"text/plain; charset=utf-8","remote_addr":"172.19.0.2:44120","user_agent":"Go-http-client/1.1","cache":"miss","query":""}
2024-10-03T10:12:10.586403629Z Puma starting in single mode...
2024-10-03T10:12:10.586417892Z * Puma version: 6.4.2 (ruby 3.3.0-p0) ("The Eagle of Durango")
2024-10-03T10:12:10.586420350Z *  Min threads: 40
2024-10-03T10:12:10.586422283Z *  Max threads: 40
2024-10-03T10:12:10.586424126Z *  Environment: devint
2024-10-03T10:12:10.586426017Z *          PID: 17
2024-10-03T10:12:10.586552488Z * Listening on http://0.0.0.0:3000
2024-10-03T10:12:10.603315796Z Use Ctrl-C to stop
2024-10-03T10:12:11.048120983Z {"time":"2024-10-03T10:12:11.048046409Z","level":"INFO","msg":"Request","path":"/up","status":301,"dur":1,"method":"GET","req_content_length":0,"req_content_type":"","resp_content_length":0,"resp_content_type":"text/html; charset=utf-8","remote_addr":"172.19.0.2:44120","user_agent":"Go-http-client/1.1","cache":"miss","query":""}
2024-10-03T10:12:12.047512910Z {"time":"2024-10-03T10:12:12.04732821Z","level":"INFO","msg":"Request","path":"/up","status":301,"dur":0,"method":"GET","req_content_length":0,"req_content_type":"","resp_content_length":0,"resp_content_type":"text/html; charset=utf-8","remote_addr":"172.19.0.2:44120","user_agent":"Go-http-client/1.1","cache":"miss","query":""}
2024-10-03T10:12:13.047846408Z {"time":"2024-10-03T10:12:13.047729937Z","level":"INFO","msg":"Request","path":"/up","status":301,"dur":0,"method":"GET","req_content_length":0,"req_content_type":"","resp_content_length":0,"resp_content_type":"text/html; charset=utf-8","remote_addr":"172.19.0.2:44120","user_agent":"Go-http-client/1.1","cache":"miss","query":""}
2024-10-03T10:12:14.047807686Z {"time":"2024-10-03T10:12:14.047593626Z","level":"INFO","msg":"Request","path":"/up","status":301,"dur":0,"method":"GET","req_content_length":0,"req_content_type":"","resp_content_length":0,"resp_content_type":"text/html; charset=utf-8","remote_addr":"172.19.0.2:44120","user_agent":"Go-http-client/1.1","cache":"miss","query":""}
2024-10-03T10:12:15.047478119Z {"time":"2024-10-03T10:12:15.047384139Z","level":"INFO","msg":"Request","path":"/up","status":301,"dur":0,"method":"GET","req_content_length":0,"req_content_type":"","resp_content_length":0,"resp_content_type":"text/html; charset=utf-8","remote_addr":"172.19.0.2:44120","user_agent":"Go-http-client/1.1","cache":"miss","query":""}
2024-10-03T10:12:16.047969551Z {"time":"2024-10-03T10:12:16.047850513Z","level":"INFO","msg":"Request","path":"/up","status":301,"dur":0,"method":"GET","req_content_length":0,"req_content_type":"","resp_content_length":0,"resp_content_type":"text/html; charset=utf-8","remote_addr":"172.19.0.2:44120","user_agent":"Go-http-client/1.1","cache":"miss","query":""}
2024-10-03T10:12:17.047529146Z {"time":"2024-10-03T10:12:17.047391056Z","level":"INFO","msg":"Request","path":"/up","status":301,"dur":0,"method":"GET","req_content_length":0,"req_content_type":"","resp_content_length":0,"resp_content_type":"text/html; charset=utf-8","remote_addr":"172.19.0.2:44120","user_agent":"Go-http-client/1.1","cache":"miss","query":""}
2024-10-03T10:12:18.047754441Z {"time":"2024-10-03T10:12:18.047662475Z","level":"INFO","msg":"Request","path":"/up","status":301,"dur":0,"method":"GET","req_content_length":0,"req_content_type":"","resp_content_length":0,"resp_content_type":"text/html; charset=utf-8","remote_addr":"172.19.0.2:44120","user_agent":"Go-http-client/1.1","cache":"miss","query":""}
2024-10-03T10:12:19.047525119Z {"time":"2024-10-03T10:12:19.047406314Z","level":"INFO","msg":"Request","path":"/up","status":301,"dur":0,"method":"GET","req_content_length":0,"req_content_type":"","resp_content_length":0,"resp_content_type":"text/html; charset=utf-8","remote_addr":"172.19.0.2:44120","user_agent":"Go-http-client/1.1","cache":"miss","query":""}
2024-10-03T10:12:20.047555951Z {"time":"2024-10-03T10:12:20.047457548Z","level":"INFO","msg":"Request","path":"/up","status":301,"dur":0,"method":"GET","req_content_length":0,"req_content_type":"","resp_content_length":0,"resp_content_type":"text/html; charset=utf-8","remote_addr":"172.19.0.2:44120","user_agent":"Go-http-client/1.1","cache":"miss","query":""}
2024-10-03T10:12:21.048005701Z {"time":"2024-10-03T10:12:21.047904734Z","level":"INFO","msg":"Request","path":"/up","status":301,"dur":0,"method":"GET","req_content_length":0,"req_content_type":"","resp_content_length":0,"resp_content_type":"text/html; charset=utf-8","remote_addr":"172.19.0.2:44120","user_agent":"Go-http-client/1.1","cache":"miss","query":""}
2024-10-03T10:12:22.047418577Z {"time":"2024-10-03T10:12:22.047325291Z","level":"INFO","msg":"Request","path":"/up","status":301,"dur":0,"method":"GET","req_content_length":0,"req_content_type":"","resp_content_length":0,"resp_content_type":"text/html; charset=utf-8","remote_addr":"172.19.0.2:44120","user_agent":"Go-http-client/1.1","cache":"miss","query":""}
2024-10-03T10:12:23.047766252Z {"time":"2024-10-03T10:12:23.047654357Z","level":"INFO","msg":"Request","path":"/up","status":301,"dur":0,"method":"GET","req_content_length":0,"req_content_type":"","resp_content_length":0,"resp_content_type":"text/html; charset=utf-8","remote_addr":"172.19.0.2:44120","user_agent":"Go-http-client/1.1","cache":"miss","query":""}
2024-10-03T10:12:24.047382298Z {"time":"2024-10-03T10:12:24.047286567Z","level":"INFO","msg":"Request","path":"/up","status":301,"dur":0,"method":"GET","req_content_length":0,"req_content_type":"","resp_content_length":0,"resp_content_type":"text/html; charset=utf-8","remote_addr":"172.19.0.2:44120","user_agent":"Go-http-client/1.1","cache":"miss","query":""}
2024-10-03T10:12:25.048162887Z {"time":"2024-10-03T10:12:25.048084503Z","level":"INFO","msg":"Request","path":"/up","status":301,"dur":0,"method":"GET","req_content_length":0,"req_content_type":"","resp_content_length":0,"resp_content_type":"text/html; charset=utf-8","remote_addr":"172.19.0.2:44120","user_agent":"Go-http-client/1.1","cache":"miss","query":""}
2024-10-03T10:12:26.047750958Z {"time":"2024-10-03T10:12:26.047666361Z","level":"INFO","msg":"Request","path":"/up","status":301,"dur":0,"method":"GET","req_content_length":0,"req_content_type":"","resp_content_length":0,"resp_content_type":"text/html; charset=utf-8","remote_addr":"172.19.0.2:44120","user_agent":"Go-http-client/1.1","cache":"miss","query":""}
2024-10-03T10:12:27.047461131Z {"time":"2024-10-03T10:12:27.04729404Z","level":"INFO","msg":"Request","path":"/up","status":301,"dur":0,"method":"GET","req_content_length":0,"req_content_type":"","resp_content_length":0,"resp_content_type":"text/html; charset=utf-8","remote_addr":"172.19.0.2:44120","user_agent":"Go-http-client/1.1","cache":"miss","query":""}
2024-10-03T10:12:28.047998914Z {"time":"2024-10-03T10:12:28.047815193Z","level":"INFO","msg":"Request","path":"/up","status":301,"dur":0,"method":"GET","req_content_length":0,"req_content_type":"","resp_content_length":0,"resp_content_type":"text/html; charset=utf-8","remote_addr":"172.19.0.2:44120","user_agent":"Go-http-client/1.1","cache":"miss","query":""}
2024-10-03T10:12:29.047534102Z {"time":"2024-10-03T10:12:29.04739252Z","level":"INFO","msg":"Request","path":"/up","status":301,"dur":0,"method":"GET","req_content_length":0,"req_content_type":"","resp_content_length":0,"resp_content_type":"text/html; charset=utf-8","remote_addr":"172.19.0.2:44120","user_agent":"Go-http-client/1.1","cache":"miss","query":""}
2024-10-03T10:12:30.047297465Z {"time":"2024-10-03T10:12:30.047186348Z","level":"INFO","msg":"Request","path":"/up","status":301,"dur":0,"method":"GET","req_content_length":0,"req_content_type":"","resp_content_length":0,"resp_content_type":"text/html; charset=utf-8","remote_addr":"172.19.0.2:44120","user_agent":"Go-http-client/1.1","cache":"miss","query":""}
2024-10-03T10:12:31.047697501Z {"time":"2024-10-03T10:12:31.047600294Z","level":"INFO","msg":"Request","path":"/up","status":301,"dur":0,"method":"GET","req_content_length":0,"req_content_type":"","resp_content_length":0,"resp_content_type":"text/html; charset=utf-8","remote_addr":"172.19.0.2:44120","user_agent":"Go-http-client/1.1","cache":"miss","query":""}
2024-10-03T10:12:32.047398803Z {"time":"2024-10-03T10:12:32.047280902Z","level":"INFO","msg":"Request","path":"/up","status":301,"dur":0,"method":"GET","req_content_length":0,"req_content_type":"","resp_content_length":0,"resp_content_type":"text/html; charset=utf-8","remote_addr":"172.19.0.2:44120","user_agent":"Go-http-client/1.1","cache":"miss","query":""}
2024-10-03T10:12:33.047837629Z {"time":"2024-10-03T10:12:33.047736866Z","level":"INFO","msg":"Request","path":"/up","status":301,"dur":0,"method":"GET","req_content_length":0,"req_content_type":"","resp_content_length":0,"resp_content_type":"text/html; charset=utf-8","remote_addr":"172.19.0.2:44120","user_agent":"Go-http-client/1.1","cache":"miss","query":""}
2024-10-03T10:12:34.047391468Z {"time":"2024-10-03T10:12:34.047251666Z","level":"INFO","msg":"Request","path":"/up","status":301,"dur":0,"method":"GET","req_content_length":0,"req_content_type":"","resp_content_length":0,"resp_content_type":"text/html; charset=utf-8","remote_addr":"172.19.0.2:44120","user_agent":"Go-http-client/1.1","cache":"miss","query":""}
2024-10-03T10:12:35.047979415Z {"time":"2024-10-03T10:12:35.047800354Z","level":"INFO","msg":"Request","path":"/up","status":301,"dur":0,"method":"GET","req_content_length":0,"req_content_type":"","resp_content_length":0,"resp_content_type":"text/html; charset=utf-8","remote_addr":"172.19.0.2:44120","user_agent":"Go-http-client/1.1","cache":"miss","query":""}
2024-10-03T10:12:36.047563982Z {"time":"2024-10-03T10:12:36.047367517Z","level":"INFO","msg":"Request","path":"/up","status":301,"dur":0,"method":"GET","req_content_length":0,"req_content_type":"","resp_content_length":0,"resp_content_type":"text/html; charset=utf-8","remote_addr":"172.19.0.2:44120","user_agent":"Go-http-client/1.1","cache":"miss","query":""}
2024-10-03T10:12:37.047533440Z {"time":"2024-10-03T10:12:37.04743844Z","level":"INFO","msg":"Request","path":"/up","status":301,"dur":0,"method":"GET","req_content_length":0,"req_content_type":"","resp_content_length":0,"resp_content_type":"text/html; charset=utf-8","remote_addr":"172.19.0.2:44120","user_agent":"Go-http-client/1.1","cache":"miss","query":""}
2024-10-03T10:12:38.047773791Z {"time":"2024-10-03T10:12:38.047710488Z","level":"INFO","msg":"Request","path":"/up","status":301,"dur":0,"method":"GET","req_content_length":0,"req_content_type":"","resp_content_length":0,"resp_content_type":"text/html; charset=utf-8","remote_addr":"172.19.0.2:44120","user_agent":"Go-http-client/1.1","cache":"miss","query":""}
  INFO [0d99e151] Running docker container ls --all --filter name=^app-web-616499a7f846ac1b9e793e4b65fc0d620975af34_uncommitted_5974b526eb0bface$ --quiet | xargs docker inspect --format '{{json .State.Health}}' on xxx.xxx.x.xx
  INFO [0d99e151] Finished in 0.079 seconds with exit status 0 (successful).
 ERROR null
  INFO [92917103] Running docker container ls --all --filter name=^app-web-616499a7f846ac1b9e793e4b65fc0d620975af34_uncommitted_5974b526eb0bface$ --quiet | xargs docker stop on xxx.xxx.x.xx
  INFO [92917103] Finished in 12.182 seconds with exit status 0 (successful).
Releasing the deploy lock...
  Finished all in 158.3 seconds
  ERROR (SSHKit::Command::Failed): Exception while executing on host xxx.xxx.x.xx: docker exit status: 1
docker stdout: Nothing written
docker stderr: Error: target failed to become healthy

Dockerfile:

# syntax = docker/dockerfile:1

# Make sure RUBY_VERSION matches the Ruby version in .ruby-version and Gemfile
ARG RUBY_VERSION=3.3.0
FROM ruby:$RUBY_VERSION-slim AS base

# Rails app lives here
WORKDIR /rails

# Set production environment
ENV BUNDLE_DEPLOYMENT="1" \
    BUNDLE_PATH="/usr/local/bundle" \
    #BUNDLE_WITHOUT="development:test" \
    RAILS_ENV="devint"

# Update gems and bundler
RUN gem update --system --no-document && \
    gem install -N bundler


# Throw-away build stage to reduce size of final image
FROM base AS build

# Install packages needed to build gems and node modules
RUN apt-get update -qq && \
    apt-get install --no-install-recommends -y build-essential curl default-libmysqlclient-dev libvips node-gyp pkg-config python-is-python3

# Install Node.js
ARG NODE_VERSION=18.15.0
ENV PATH=/usr/local/node/bin:$PATH
RUN curl -sL https://github.com/nodenv/node-build/archive/master.tar.gz | tar xz -C /tmp/ && \
    /tmp/node-build-master/bin/node-build "${NODE_VERSION}" /usr/local/node && \
    rm -rf /tmp/node-build-master

# Install application gems
COPY Gemfile Gemfile.lock ./
RUN bundle install && \
    bundle exec bootsnap precompile --gemfile && \
    rm -rf ~/.bundle/ "${BUNDLE_PATH}"/ruby/*/cache "${BUNDLE_PATH}"/ruby/*/bundler/gems/*/.git

# Install node modules
COPY package.json package-lock.json ./
RUN npm install

# Copy application code
COPY . .

# Precompile bootsnap code for faster boot times
RUN bundle exec bootsnap precompile app/ lib/

# Adjust binfiles to be executable on Linux
RUN chmod +x bin/* && \
    sed -i "s/\r$//g" bin/* && \
    sed -i 's/ruby\.exe$/ruby/' bin/*

# Precompiling assets for production without requiring secret RAILS_MASTER_KEY
RUN SECRET_KEY_BASE_DUMMY=1 ./bin/rails assets:precompile
RUN SECRET_KEY_BASE_DUMMY=1 ./bin/rails tailwindcss:build


# Final stage for app image
FROM base

# Install packages needed for deployment
RUN apt-get update -qq && \
    apt-get install --no-install-recommends -y curl default-mysql-client imagemagick libsqlite3-0 libvips && \
    rm -rf /var/lib/apt/lists /var/cache/apt/archives

# Copy built artifacts: gems, application
COPY --from=build "${BUNDLE_PATH}" "${BUNDLE_PATH}"
COPY --from=build /rails /rails

# Run and own only the runtime files as a non-root user for security
RUN groupadd --system --gid 1000 rails && \
    useradd rails --uid 1000 --gid 1000 --create-home --shell /bin/bash && \
    chown -R 1000:1000 db log storage tmp
USER 1000:1000

# Entrypoint prepares the database.
ENTRYPOINT ["/rails/bin/docker-entrypoint"]

# Start the server by default, this can be overwritten at runtime
EXPOSE 80
CMD ["bundle", "exec", "thrust", "./bin/rails", "server"]

deploy.yml

# Name of your application. Used to uniquely configure containers.
service: app
# Name of the container image.
image: mycomp/myapp

# Deploy to these servers.
servers:
  web:
    -  xxx.xxx.x.xx
  # job:
  #   hosts:
  #     - 192.168.0.1
  #   cmd: bin/jobs

# Enable SSL auto certification via Let's Encrypt (and allow for multiple apps on one server).
# Set ssl: false if using something like Cloudflare to terminate SSL (but keep host!).
proxy:
  ssl: false
  host: devint.mydomain.ee

registry:
  username: myusername
  password:
    - KAMAL_REGISTRY_PASSWORD

# Configure builder setup.
builder:
  arch: amd64
  context: .

ssh:
  user: rando


# Inject ENV variables into containers (secrets come from .kamal/secrets).
#
# env:
#   clear:
#     DB_HOST: 192.168.0.2
#   secret:
#     - RAILS_MASTER_KEY

env:
  clear:
    DB_HOST:  xxx.xxx.x.xx
    RAILS_LOG_TO_STDOUT: 1
    RUBY_YJIT_ENABLE: 1
    RAILS_SERVE_STATIC_FILES: true
    RAILS_MAX_THREADS: 40
  secret:
    - RAILS_MASTER_KEY
    - MYSQL_ROOT_PASSWORD


asset_path: /rails/public/assets

accessories:
  db:
    image: mariadb:11.4
    host: xxx.xxx.x.xx
    port: 3306
    env:
      clear:
        MYSQL_ROOT_HOST: '%'
      secret:
        - MYSQL_ROOT_PASSWORD
    files:
      #       - config/mysql/production.cnf:/etc/mysql/my.cnf
      - db/production.sql:/docker-entrypoint-initdb.d/setup.sql
    directories:
      - data:/var/lib/mysql

What am I missing? devint domain root path serves blue 404 page I assume comes from kamal-proxy.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 72 (8 by maintainers)

Most upvoted comments

I also faced this problem while migrating my Rails 7.2.1 app from Kama 1.9 to 2.1. On comparing, rails 7 and 8 boilerplates generated by rails new, found that rails 8 has both assume_ssl and force_ssl true, while rails 7 has only force_ssl true.

So just making assume_ssl true in my rails 7.2 app made the deployment go through smoothly with Kamal 2.1.

In summary just set the following in your production.rb:

config.assume_ssl = true
config.force_ssl = true

This way you can avoid all the other configuration on /up endpoint

I’ve also encountered this problem. The problem comes from the “/up” route which must remain reachable without SSL. You just need to add a configuration line in production.rb. I actually found this configuration in a fresh rails 8 app.

# production.rb

# Skip http-to-https redirect for the default health check endpoint.
  config.ssl_options = { redirect: { exclude: ->(request) { request.path == “/up” } }

Use config.force_ssl together with config.assume_ssl. Then Rails will still act as though you’re on SSL, which is important for setting secure cookies etc. This is the default for Rails 8+.

Is there any reason why Kamal 2 doesn’t add the assume_ssl line?

It’s because Kamal doesn’t care if your app is a Rails app. You can deploy any dockerized app, in any language.

Kamal asks docker to build the whatever app with the Dockerfile, push the resulting image to any registry you config, and asks his remote partner Kamal-server to pull it and deploy it on your server. Kamal doesn’t know your app is a rails’ app.

Kamal needs your app to be healthy to finish. If your app isn’t responding properly to the health check (like misconfiguration about SSL), Kamal can’t finish. So, your app’s responsibility is to respond 200 to the health check request with SSL.

Got hung up on this with a Django install. This post got me up and running. https://www.coryzue.com/writing/kamal-django/

Summary is create a middleware and put it first in the list to bypass the ALLOWED_HOSTS check and HTTPS redirects.

# <project>/middleware.py

from django.http import HttpResponse

class HealthCheckMiddleware:
    def __init__(self, get_response):
        self.get_response = get_response

    def __call__(self, request):
        if request.path == "/up":
            # Or perform any "real" health checking, if needed
            response = HttpResponse("OK")
        else:
            response = self.get_response(request)

        return response

@jjatinggoyal it worked thanks, also thank you all of you that take some minutes to help me to solve this issue.

@guilhermecaixeta Have you tried using thruster gem?

Gemfile

# Add HTTP asset caching/compression and X-Sendfile acceleration to Puma [https://github.com/basecamp/thruster/]
gem "thruster", require: false

Dockerfile

# Start the server by default, this can be overwritten at runtime
EXPOSE 3000
CMD ["./bin/thrust", "./bin/rails", "server"]

Running on Rails 7, I also had to make sure force_ssl was off, and /up route was properly configured.

production.rb

  config.force_ssl = false
  config.host_authorization = { exclude: ->(request) { request.path == "/up" } }

routes.rb

  # Reveal health status on /up that returns 200 if the app boots with no exceptions, otherwise 500.
  # Can be used by load balancers and uptime monitors to verify that the app is live.
  get "up" => "rails/health#show", as: :rails_health_check

By default, Kamal Proxy checks port 80. Since you are using port 3000, specify the app_port setting in the proxy configuration:

proxy:
  ssl: true
  host: api.abc.com
  app_port: 3000

If you look at the bottom of the Dockerfile, you can see

# Start the server by default, this can be overwritten at runtime
EXPOSE 80
CMD ["bundle", "exec", "thrust", "./bin/rails", "server"]

The container is supposed to be using basecamp/thruster which by default should expose port 80. Nowhere has port 3000 been defined. Adding your suggestion of app_port makes things worse. 2024-10-03T14:23:01.810967693Z 2024-10-03 14:23:01 +0000 HTTP parse error, malformed request: #<Puma::HttpParserError: Invalid HTTP format, parsing fails. Are you trying to open an SSL connection to a non-SSL Puma?> . Thruster is included in the gemfile.

@dhh, thruster doesn’t need any fancy config, does it?

I ran into this for a different reason, in case anyone else encounters this.

My problem was that my target instance was under-provisioned (Digital Ocean Basic / 512 MB / 1 vCPU).

The behavior is that an initial kamal deploy or kamal setup would be successful, but subsequent deploys would not. The docker exec kamal-proxy kamal-proxy deploy my-app-here-web step would hang (for much longer than 30 seconds, which should have been a hint to me) and then ultimately complain with this error.

I noticed it was also quite slow to connect to the target box and/or run commands on an active shell. Running top showed that swap was very active. So, probably what happens is that running two containers during handoff eats up more resources and grinds the health check to a halt and blue-green cannot be verified.

It’s an easy gotcha if you’re experimenting with Kamala on the cheapest cloud instance you can find, which is what I was doing 😄

config.assume_ssl = true
config.force_ssl = true

This is what worked for me on Rails 7.2.2, I mean setting config.assume_ssl to true.

@jjatinggoyal you’re the MVP! 😄 Thanks!

@dhh Thanks for explaining how healtchecks are used for gap-less deployments. That actually makes a lot of sense and inspired me to find a solution for this.

Thanks to the post by @scuml , I have this solution for Nextjs users struggling with getting healthchecks working on Kamal 2.0.

Place this in your middleware. Point your healthcheck at /up.

import { NextRequest, NextResponse } from "next/server";

export function middleware(request: NextRequest) {

  if (request.nextUrl.pathname === "/up") {
    return new Response("OK", { status: 200 });
  }

  return NextResponse.next();
}

I don’t see any path to deployment without a working healthcheck (whether that’s /up or HEALTHCHECK inside the container) since that’s the mechanism we need to ensure the gap-less deploy. Otherwise you’re going to be throwing 500 errors from the proxy for the duration of the app boot, which isn’t what anyone wants.

On a fresh Hertz instance with a Rails 7.2.1 application and Kamal 2, I had to set config.force_ssl = false to get the healthcheck working.

I’ve deployed a new Rails 8 app onto it and I didn’t need to set that. I’m not sure why.

Expanding on previous comment - see https://nts.strzibny.name/upgrading-to-kamal-2/ - try config.host_authorization = { exclude: ->(request) { request.path == "/up" } } (I didn’t end up requiring this, YMMV).

Also do you have config.force_ssl = false?

By default, Kamal Proxy checks port 80. Since you are using port 3000, specify the app_port setting in the proxy configuration:

proxy:
  ssl: true
  host: api.abc.com
  app_port: 3000

ssl

This is the setting that solved the health check problem for me.

# production.rb
# Assume all access to the app is happening through a SSL-terminating reverse proxy.
  config.assume_ssl = true

I’v found what was the cause of my issues, it’s kinda edge case but may help someone because it was very hard to trace. I have the engine which is used in the app, and there was a line inside Engine class within initializer, require for non existing file. I have no idea why I haven’t seen any error on build stage but container for sure was not healthy.

    initializer 'my_foo_engine.extensions', before: :load_config_initializers do |_app|
      require 'extensions'
      require 'non existing file here'
    end

@easydatawarehousing instructions worked for me, after hours of poking around. The issue was I needed to go into DigitalOcean managed Postgres instance and adjust my “Trusted Sources” so the “Droplet” could access the database.

From my DigitalOcean “Droplet” (not even from within the Docker container)

$ cat .kamal/apps/<your-app-name>/env/roles/web.env
$ source .kamal/apps/<your-app-name>/env/roles/web.env
$ pg_isready -h $DB_HOST -p $DB_PORT -U $DB_USER -d $DB_USER # `man pg_isready` for help

# Before the fix I was getting:
> myapp-db-1-do-user-18470523-0.j.db.ondigitalocean.com:25060 - no response

# After the fix I get:
pg_isready -h $DB_HOST -p $DB_PORT -U $DB_USER -d $DB_USER
myapp-db-1-do-user-18470523-0.j.db.ondigitalocean.com:25060 - accepting connections

Another peculiar thing that helped me understand what was going on is that kamal app boot worked fine but kamal app stop then kamal app start would report the start as failing.

It’s been a rough two days with Kamal 2 + Rails 8. I see the vision, though. I think there needs to be a lot better handoff between Rails 8, Kamal 2 proxy, and Kamal CLI output. In this case I’d hope to see a “Couldn’t connect to database” message.

I’m back! The problem was not related to Kamal. I was using a firewall in my EC2 that was not allowing the proxy to check if the server was healthy. After some tweaks to the firewall rules, it finally worked.

@pftg I think my settings are correct. The SSL Setting in Cloudflare is set to Full, the production.rb contains assume_ssl and force_ssl set as true, and the proxy config contains ssl: true. Am I missing something?

config.assume_ssl = true config.force_ssl = true

Was definitely what I needed, but I also have other setup steps running in my bin/docker-entrypoint so I had to adjust my deploy_timeout in the kamal config:

# Deploy timeout
#
# How long to wait for a container to become ready, default 30:
deploy_timeout: 60

This stopped the following from happening again:

2024-11-16T15:43:30.463083222Z {"time":"2024-11-16T15:43:30.462675726Z","level":"INFO","msg":"Unable to proxy request","path":"/up","error":"dial tcp [::1]:3000: connect: connection refused"}
2024-11-16T15:43:30.464958159Z {"time":"2024-11-16T15:43:30.463473194Z","level":"INFO","msg":"Request","path":"/up","status":502,"dur":2,"method":"GET","req_content_length":0,"req_content_type":"","resp_content_length":0,"resp_content_type":"text/plain; charset=utf-8","remote_addr":"172.18.0.2:39756","user_agent":"Go-http-client/1.1","cache":"miss","query":""}
2024-11-16T15:43:31.214349949Z => Booting Puma
2024-11-16T15:43:31.214412520Z => Rails 8.0.0.rc2 application starting in staging 
2024-11-16T15:43:31.214419656Z => Run `bin/rails server --help` for more startup options
2024-11-16T15:43:31.471120468Z {"time":"2024-11-16T15:43:31.47029138Z","level":"INFO","msg":"Unable to proxy request","path":"/up","error":"dial tcp [::1]:3000: connect: connection refused"}
2024-11-16T15:43:31.471175057Z {"time":"2024-11-16T15:43:31.470470487Z","level":"INFO","msg":"Request","path":"/up","status":502,"dur":6,"method":"GET","req_content_length":0,"req_content_type":"","resp_content_length":0,"resp_content_type":"text/plain; charset=utf-8","remote_addr":"172.18.0.2:39756","user_agent":"Go-http-client/1.1","cache":"miss","query":""}
[...]
Releasing the deploy lock...
  Finished all in 404.0 seconds
  ERROR (SSHKit::Command::Failed): Exception while executing on host ***: docker exit status: 1
docker stdout: Nothing written
docker stderr: Error: target failed to become healthy

I recall us investigating the healthcheck feature in the past and finding it wasn’t suitable for our needs, especially around the gap-less deploys. Don’t recall the specifics, though. Maybe @djmb remembers why we discarded the healthcheck as an option.

Either way, it’s not something that’s on the menu to change in the short term. Tons of successful deployments in the wild now that are built around using /up as the healthcheck.

Yeah this is still a problem for other frameworks which don’t let you do HTTP/S handling on a per-route basis. For instance, I am trying to host a Next 14 project and running into this problem.

I was able to fix this error on my end.

production.rb config.force_ssl = false

deploy.yml

proxy:
  ssl: true
  host: example.com
  # kamal-proxy connects to your container over port 80, use `app_port` to specify a different port.
  app_port: 3000

Disabled force_ssl from Rails to make way for Kamal2 SSL.

Problem: After changing the Dockerfile, I had the container health problem again.

Solution: This time, I increased deploy_timeout, healthcheck’s interval and timeout durations (to wait before starting the health check and increased timeout from 3 seconds to 30 seconds), which helped

@kishaningithub I don’t know why the deployment process still sent request to “/up” it’s really strange for me.

But I think this log line is the correct health check through docker : INFO [68e47201] Running docker container ls --all --filter name=^wordtrial-web-eb801b29d0ab51fb77c80f988f0ea128838ade00$ --quiet | xargs docker inspect --format '{{json .State.Health}}' on 139.59.18.46

But we can see some warning in the “/up” logs. Maybe your app doesn’t boot properly? Maybe you should check if your app can start alone by starting it manually.

  • You can see your images apps with : kamal app images,
  • You can start your app with : kamal app start
  • After that, try to access your app by launching a console.

If you can access to your app, your app is booting, but the health check doesn’t pass. I already have this issue and the solution was to permit non-ssl (http) request to “/up” in my case. As your health check path is “/” I suggest you to add a dedicated health check’s route for permitting non-ssl request only on this path.

@BarnabeD But the problem is they have removed the health check from the global config in kamal 2 and made it to be part of “proxy”. When we set the value of “proxy” as “false” there is no way to specify a health check

Ref Link kamal-deploy.org/docs/upgrading/configuration-changes#healthchecks

Screenshot image

@kishaningithub Yes you are right. Few lines after in the documentation describes this :

Non-proxy roles

For roles that do not run the proxy, you can set a custom Docker healthcheck via the options.

servers: web: … jobs: options: health-cmd: bin/jobs-healthy

For those containers, Kamal will wait for the healthy status if they have a healthcheck or running if they don’t.

You can set a readiness_delay, which is used when we see the running status. We’ll wait that long and confirm the container is still running before continuing.

I think you can pass the health check barrier with a custom health check. As you should prove aliveness of the container, you should be good.

@dhh I am facing this same issue while deploying a wordpress site with proxy: false

Logs of kamal deploy

If you notice below the proxy looks alive and it is continously pinging /up endpoint. This is incorrect behaviour when proxy: false

2024-10-08T06:27:20.226839066Z 172.18.0.3 - - [08/Oct/2024:06:27:20 +0000] "GET /up HTTP/1.1" 404 35861 "http://9f3ecb8ffd88:80/up" "Go-http-client/1.1"
2024-10-08T06:27:21.039613333Z [Tue Oct 08 06:27:21.038484 2024] [php:warn] [pid 22:tid 22] [client 172.18.0.3:55608] PHP Warning:  Attempt to read property "is_valid" on null in /var/www/html/wp-content/plugins/luique-plugin/admin/dashboard-theme-activation.php on line 27
2024-10-08T06:27:21.039664440Z [Tue Oct 08 06:27:21.038538 2024] [php:warn] [pid 22:tid 22] [client 172.18.0.3:55608] PHP Warning:  Attempt to read property "support_renew_link" on null in /var/www/html/wp-content/plugins/luique-plugin/admin/dashboard-theme-activation.php on line 31
2024-10-08T06:27:21.078548953Z 172.18.0.3 - - [08/Oct/2024:06:27:21 +0000] "GET /up HTTP/1.1" 301 385 "-" "Go-http-client/1.1"
2024-10-08T06:27:21.087763451Z [Tue Oct 08 06:27:21.087639 2024] [php:warn] [pid 22:tid 22] [client 172.18.0.3:55608] PHP Warning:  Attempt to read property "is_valid" on null in /var/www/html/wp-content/plugins/luique-plugin/admin/dashboard-theme-activation.php on line 27, referer: http://9f3ecb8ffd88:80/up
2024-10-08T06:27:21.087906525Z [Tue Oct 08 06:27:21.087851 2024] [php:warn] [pid 22:tid 22] [client 172.18.0.3:55608] PHP Warning:  Attempt to read property "support_renew_link" on null in /var/www/html/wp-content/plugins/luique-plugin/admin/dashboard-theme-activation.php on line 31, referer: http://9f3ecb8ffd88:80/up
2024-10-08T06:27:21.209349936Z 172.18.0.3 - - [08/Oct/2024:06:27:21 +0000] "GET /up HTTP/1.1" 404 35861 "http://9f3ecb8ffd88:80/up" "Go-http-client/1.1"
2024-10-08T06:27:22.037620774Z [Tue Oct 08 06:27:22.036877 2024] [php:warn] [pid 23:tid 23] [client 172.18.0.3:55610] PHP Warning:  Attempt to read property "is_valid" on null in /var/www/html/wp-content/plugins/luique-plugin/admin/dashboard-theme-activation.php on line 27
2024-10-08T06:27:22.037676742Z [Tue Oct 08 06:27:22.036916 2024] [php:warn] [pid 23:tid 23] [client 172.18.0.3:55610] PHP Warning:  Attempt to read property "support_renew_link" on null in /var/www/html/wp-content/plugins/luique-plugin/admin/dashboard-theme-activation.php on line 31
2024-10-08T06:27:22.072997578Z 172.18.0.3 - - [08/Oct/2024:06:27:22 +0000] "GET /up HTTP/1.1" 301 385 "-" "Go-http-client/1.1"
2024-10-08T06:27:22.082678689Z [Tue Oct 08 06:27:22.082553 2024] [php:warn] [pid 23:tid 23] [client 172.18.0.3:55610] PHP Warning:  Attempt to read property "is_valid" on null in /var/www/html/wp-content/plugins/luique-plugin/admin/dashboard-theme-activation.php on line 27, referer: http://9f3ecb8ffd88:80/up
2024-10-08T06:27:22.082820318Z [Tue Oct 08 06:27:22.082769 2024] [php:warn] [pid 23:tid 23] [client 172.18.0.3:55610] PHP Warning:  Attempt to read property "support_renew_link" on null in /var/www/html/wp-content/plugins/luique-plugin/admin/dashboard-theme-activation.php on line 31, referer: http://9f3ecb8ffd88:80/up
2024-10-08T06:27:22.198417534Z 172.18.0.3 - - [08/Oct/2024:06:27:22 +0000] "GET /up HTTP/1.1" 404 35861 "http://9f3ecb8ffd88:80/up" "Go-http-client/1.1"
2024-10-08T06:27:23.038643181Z [Tue Oct 08 06:27:23.038086 2024] [php:warn] [pid 24:tid 24] [client 172.18.0.3:55612] PHP Warning:  Attempt to read property "is_valid" on null in /var/www/html/wp-content/plugins/luique-plugin/admin/dashboard-theme-activation.php on line 27
2024-10-08T06:27:23.038710088Z [Tue Oct 08 06:27:23.038136 2024] [php:warn] [pid 24:tid 24] [client 172.18.0.3:55612] PHP Warning:  Attempt to read property "support_renew_link" on null in /var/www/html/wp-content/plugins/luique-plugin/admin/dashboard-theme-activation.php on line 31
2024-10-08T06:27:23.079937251Z 172.18.0.3 - - [08/Oct/2024:06:27:23 +0000] "GET /up HTTP/1.1" 301 385 "-" "Go-http-client/1.1"
2024-10-08T06:27:23.088756298Z [Tue Oct 08 06:27:23.088657 2024] [php:warn] [pid 24:tid 24] [client 172.18.0.3:55612] PHP Warning:  Attempt to read property "is_valid" on null in /var/www/html/wp-content/plugins/luique-plugin/admin/dashboard-theme-activation.php on line 27, referer: http://9f3ecb8ffd88:80/up
2024-10-08T06:27:23.089577211Z [Tue Oct 08 06:27:23.088819 2024] [php:warn] [pid 24:tid 24] [client 172.18.0.3:55612] PHP Warning:  Attempt to read property "support_renew_link" on null in /var/www/html/wp-content/plugins/luique-plugin/admin/dashboard-theme-activation.php on line 31, referer: http://9f3ecb8ffd88:80/up
2024-10-08T06:27:23.201792327Z 172.18.0.3 - - [08/Oct/2024:06:27:23 +0000] "GET /up HTTP/1.1" 404 35861 "http://9f3ecb8ffd88:80/up" "Go-http-client/1.1"
2024-10-08T06:27:24.057137478Z [Tue Oct 08 06:27:24.056870 2024] [php:warn] [pid 19:tid 19] [client 172.18.0.3:55626] PHP Warning:  Attempt to read property "is_valid" on null in /var/www/html/wp-content/plugins/luique-plugin/admin/dashboard-theme-activation.php on line 27
2024-10-08T06:27:24.058564102Z [Tue Oct 08 06:27:24.057091 2024] [php:warn] [pid 19:tid 19] [client 172.18.0.3:55626] PHP Warning:  Attempt to read property "support_renew_link" on null in /var/www/html/wp-content/plugins/luique-plugin/admin/dashboard-theme-activation.php on line 31
2024-10-08T06:27:24.113243837Z 172.18.0.3 - - [08/Oct/2024:06:27:24 +0000] "GET /up HTTP/1.1" 301 385 "-" "Go-http-client/1.1"
2024-10-08T06:27:24.132089890Z [Tue Oct 08 06:27:24.131690 2024] [php:warn] [pid 19:tid 19] [client 172.18.0.3:55626] PHP Warning:  Attempt to read property "is_valid" on null in /var/www/html/wp-content/plugins/luique-plugin/admin/dashboard-theme-activation.php on line 27, referer: http://9f3ecb8ffd88:80/up
2024-10-08T06:27:24.132457011Z [Tue Oct 08 06:27:24.132208 2024] [php:warn] [pid 19:tid 19] [client 172.18.0.3:55626] PHP Warning:  Attempt to read property "support_renew_link" on null in /var/www/html/wp-content/plugins/luique-plugin/admin/dashboard-theme-activation.php on line 31, referer: http://9f3ecb8ffd88:80/up
  INFO [8c74c232] Running docker container ls --all --filter name=^wordtrial-web-5d4897ade85123848c0171c695f868f073dd7dd0$ --quiet | xargs docker inspect --format '{{json .State.Health}}' on 139.59.18.46
  INFO [8c74c232] Finished in 0.103 seconds with exit status 0 (successful).
 ERROR null
  INFO [a3309027] Running docker container ls --all --filter name=^wordtrial-web-5d4897ade85123848c0171c695f868f073dd7dd0$ --quiet | xargs docker stop on 139.59.18.46
  INFO [a3309027] Finished in 1.374 seconds with exit status 0 (successful).
Releasing the deploy lock...
  Finished all in 85.1 seconds
  ERROR (SSHKit::Command::Failed): Exception while executing on host 139.59.18.46: docker exit status: 1
docker stdout: Nothing written
docker stderr: Error: target failed to become healthy

Kamal config file

service: wordtrial

image: kishaningithub/wordtrial

servers:
  web:
    - 139.59.18.46

proxy: false

registry:
  server: ghcr.io
  username: kishaningithub
  password:
    - KAMAL_REGISTRY_PASSWORD

builder:
  arch: amd64

env:
  clear:
    WORDPRESS_DB_HOST: wordtrial-db:3306
    WORDPRESS_DB_USER: wordpress
    WORDPRESS_DB_NAME: wordpress
  secret:
    - WORDPRESS_DB_PASSWORD

volumes:
  - wordpress_data:/var/www/html

accessories:
  db:
    image: mariadb:11.4
    host: 139.59.18.46
    port: "127.0.0.1:3306:3306"
    options:
      restart: unless-stopped
    env:
      clear:
        MARIADB_DATABASE: wordpress
        MARIADB_USER: wordpress
      secret:
        - MARIADB_ROOT_PASSWORD
        - MARIADB_PASSWORD
    directories:
      - data:/var/lib/mysql

Docker file

FROM wordpress:6.6.2-apache

# https://www.aptgetlife.co.uk/docker-wordpress-increase-php-max-file-size/
COPY ./config/wordpress/uploads.ini /usr/local/etc/php/conf.d/

My bad, so I was using this gem https://github.com/rameerez/allgood to cover more ground with my health checks, and some of my checks defined there were failing, but without a proper error to understand the cause.

Everything works after removing it!

@guilhermecaixeta or you can also add thruster. You will need some change in the entrypoint file.

Yeah, you can’t run both versions on the same box at the same time.