caddy: reverse_proxy: memory leak (recycling memory buffers problem)

1. Environment

1a. Operating system and version

Ubuntu 20.04.2 LTS (5.4.0-65-generic x86_64)

1b. Caddy version (run `caddy version` or paste commit SHA)

2.4.5

1c. Go version (if building Caddy from source; run `go version`)

1.16.8

2. Description

There is a problem with recycling memory buffers in Go (see post at Cloudflare’s blog: https://blog.cloudflare.com/recycling-memory-buffers-in-go/). We have rising memory usage up to 32+ Gb for 72 hours uptime with loading 40-60 RPS.

Here is the diagram of memory usage from profiler: 001

The solution is to change direct usage bytes.Buffer to sync.Pool technique (see https://riptutorial.com/go/example/16314/sync-pool)

We had this problem in our code and this solution significantly decreased memory usage.

About this issue

Original URL
State: closed
Created 3 years ago
Comments: 16 (16 by maintainers)

Most upvoted comments

Hi @dtelyukh - Do you happen to have a reproducible testcase and configuration that makes triggering the memory leak easy? I’d be happy to contribute a patch if @mholt and @francislavoie consider this is a real concern for Caddy. Thanks! 😄

copyResponse seems to be using a pool already: https://github.com/caddyserver/caddy/blob/master/modules/caddyhttp/reverseproxy/streaming.go#L161-L164

https://github.com/caddyserver/caddy/blob/master/modules/caddyhttp/reverseproxy/streaming.go#L276-L285

It will be interesting to compare sync.Pool vs specialized solutions like https://github.com/gallir/bytebufferpool in case the affected code path is a common one.

jimen0 on Oct 4, 2021

Oh, really sync.Pool is already used. But there is some leak in metricsHandler.

I have reproduced this problem on vanilla caddy.

Caddyfile:

{
	log {
		level DEBUG
	}
}

https://metrics.guix.localhost {
	metrics
}

https://guix.localhost {
	tls internal

	reverse_proxy {
		header_up host guix.gnu.org
		to https://guix.gnu.org
	}
	
	push / {
		/static/base/css/elements.css
		/static/base/css/common.css
	}
	log {
		format json
		level DEBUG
	}
}

Run

docker run -d -p 80:80 -p 443:443 \                                                                                                                            
    -v $PWD/caddydocker/Caddyfile:/etc/caddy/Caddyfile \
    caddy

Then it should constantly fetch URL metrics.guix.localhost (and some times guix.localhost). Memory usage will grow.

dtelyukh on Oct 5, 2021