caddy: Caddy Server Running for 3-4 weeks, and then all requests fail

I’ve been running a Caddy server now for ~2 months. In that time, the server has gotten stuck twice, pegs the CPU, and requires a restart in order to fix it. During that time, the server is non-responsive. The server seems to run well for about 3 or so weeks, and then this happens. A simple restart fixes it.

I’m running it on Google Compute Engine on a pretty low end config, with ~1 GB of memory.

I’m redirecting ~1000 domains, but the traffic is pretty low.

Here is my Caddyfile:

{
        email {$EMAIL}

        on_demand_tls {
                ask {$TLS_ASK_URL}
        }

        # Global log configuration
        log {
                output stdout
        }
}

:80 {
        route {
                respond /ping 200
                redir https://{host}{uri}
        }
}

:443 {
        tls {
                on_demand
        }
        route {
                rewrite * /redirect/handle?host={http.request.host}&path={path}
                reverse_proxy /* https://{$REVERSE_PROXY_HOST} {
                        header_up Host {$REVERSE_PROXY_HOST}
                }
        }
}

Here is my Dockerfile:

FROM caddy:2.6.4-alpine as cpr-redirect-server

COPY Caddyfile /etc/caddy/Caddyfile

image

I’m not sure how to go about debugging this. Please advise.

About this issue

  • Original URL
  • State: closed
  • Created a year ago
  • Reactions: 2
  • Comments: 36 (17 by maintainers)

Most upvoted comments

Okay so from that, it looks like the S3Proxy gets stuck in a crazy write loop which is taking up 75% of the time in syscalls. The rest of the time looks to be in the Go runtime/scheduler and GC. This seems like a bug with the S3Proxy plugin, not with Caddy.