traefik: wasm plugin memory leak in Traefik v3
Welcome!
- Yes, I’ve searched similar issues on GitHub and didn’t find any.
- Yes, I’ve searched similar issues on the Traefik community forum and didn’t find any.
What did you do?
add a simple wasm traefik middleware plugin to a router. run ab benchmark to make some request you will see memory increase stop the benchmark and wait you will see the memory keep leaks, no matter how long you waited
the more concurrent requests make, the more memory leak.
@juliens @zetaab could you plz take a look at this?
here’s the traefik memory usage grafana graph from our real server, after enabling wasm based plugin:
steps to reproduce
go version 1.23.1 under Linux amd64 machine
git clone https://github.com/ttys3/traefik-wasm-plugin-memory-leak-demo.git
cd traefik-wasm-plugin-memory-leak-demo
pushd plugins-local/src/github.com/traefik/plugindemowasm/
# build wasm plugin
./build.sh
# you will see generated plugin.wasm now
# now we go back to traefik-wasm-plugin-memory-leak-demo dir
popd
# download newest traefik
curl -LO https://github.com/traefik/traefik/releases/download/v3.1.4/traefik_v3.1.4_linux_amd64.tar.gz
tar xvf traefik_v3.1.4_linux_amd64.tar.gz
# run the demo backend service, it is a simple http static server
./run-downstream-service.sh
# run traefik
./run.sh
now install ps_mem if you do not have it
sudo pip install ps_mem
now we open a new terminal to watch the memory
./watch-mem.sh
we 'll see that the init memory usage is:
go we run ./bench.sh in the repo to make some requests (you should install ab - the Apache HTTP server benchmarking tool, if you os does not installed it, or other bench tools also OK)
adjust params of
abif you system does not have too much free memory, in case OOM
./bench.sh
Benchmarking localhost (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Completed 1000 requests
Finished 1000 requests
Server Software: ECAcc
Server Hostname: localhost
Server Port: 6688
Document Path: /
Document Length: 1256 bytes
Concurrency Level: 100
Time taken for tests: 6.276 seconds
Complete requests: 1000
Failed requests: 0
Total transferred: 1632250 bytes
HTML transferred: 1256000 bytes
Requests per second: 159.33 [#/sec] (mean)
Time per request: 627.634 [ms] (mean)
Time per request: 6.276 [ms] (mean, across all concurrent requests)
Transfer rate: 253.97 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.4 0 2
Processing: 184 537 905.7 200 4695
Waiting: 184 536 905.4 200 4695
Total: 184 537 906.0 200 4696
Percentage of the requests served within a certain time (ms)
50% 200
66% 213
75% 248
80% 321
90% 2040
95% 3256
98% 3400
99% 3542
100% 4696 (longest request)
What did you see instead?
traefik memory keeps growing and never free
now we run
go tool pprof http://127.0.0.1:8080/debug/pprof/heap
type web and enter, will see the heap graph:
and here is the pprof file:
pprof.traefik.alloc_objects.alloc_space.inuse_objects.inuse_space.009.pb.gz
and also the allocs
allocs pprof file
pprof.traefik.alloc_objects.alloc_space.inuse_objects.inuse_space.010.pb.gz
note:
the plugindemowasm plugin here is intended to demostrate the memory leak issue.
not for real usage.
What version of Traefik are you using?
./traefik version
Version: 3.1.4
Codename: comte
Go version: go1.23.1
Built: 2024-09-19T13:47:17Z
OS/Arch: linux/amd64
What is your environment & configuration?
# Static configuration
log:
level: TRACE
global:
checkNewVersion: false
sendAnonymousUsage: false
api:
dashboard: true
insecure: true
debug: true
metrics:
prometheus: {}
entryPoints:
web:
address: ":6688"
providers:
# https://doc.traefik.io/traefik/providers/file/
file:
watch: true
filename: "dyn.yaml"
experimental:
localPlugins:
demowasm:
moduleName: github.com/traefik/plugindemowasm
Add more configuration information here.
dyn.yaml
# Dynamic configuration
http:
routers:
my-router:
rule: Host(`localhost`)
service: service-foo
entryPoints:
- web
middlewares:
- demowasm
services:
service-foo:
loadBalancer:
passHostHeader: false
servers:
- url: http://www.example.com/
middlewares:
demowasm:
plugin:
demowasm:
headers:
X-Demo: testxxxxx
X-URL: 'http://example.localhost/'
test 2
if we disable the wasm based plugin in dyn.yaml :
diff --git a/dyn.yaml b/dyn.yaml
index a1ce928..f71cdec 100755
--- a/dyn.yaml
+++ b/dyn.yaml
@@ -8,7 +8,7 @@ http:
entryPoints:
- web
middlewares:
- - demowasm
+ # - demowasm
services:
service-foo:
and run ./bench.sh again, only cost about100MB memory:
maybe related
https://github.com/jcchavezs/coraza-http-wasm-traefik/issues/9
https://community.traefik.io/t/traefik-v3-coraza-waf-high-resource-usage/22814/8
https://github.com/tetratelabs/proxy-wasm-go-sdk/issues/450
also tried https://github.com/wasilibs/nottinygc, but no difference
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 19 (18 by maintainers)
Commits related to this issue
- fix: memory leak due to not closing the middleware when spawning a new one. Fixes #11119 — committed to jcchavezs/traefik by deleted user 2 years ago
prev I was wrong, the plugin init memory leak still exists:
no http request + make changes to
dyn.yamlwith 51 service has the wasm plugin middleware enabled for 100 timesthe memory leak keeps, the problem exists, because, new wasm host and guest created each time, but the memory not GC ed. we have to call
nethttp.middleware.Close()(which will callhandler.middleware.Close()-> wazero.Runtime.Close() ) to free the memoryso here comes the second commit: https://github.com/traefik/http-wasm-host-go/pull/2
the memory leak version (orig traefik):
4 GC round after we stop to change dyn.yaml:
with my PR’s second commit:
0 GC round after we stop to change dyn.yaml :
2 GC round after we stop to change dyn.yaml :
init startup with 51 service has the wasm plugin middleware enabled memory usage:
after ./update-dyn-config.sh run:
after 2 more GC rounds:
compare test result: the PR code works fine for dynamic config loading casued wasm plugin leak
@juliens here’s the PR https://github.com/traefik/http-wasm-host-go/pull/2
be aware that: traefik make uses traefik/http-wasm-host-go
verify_wasip1branch’s code, and has go mod replace directive in go.mod file.https://github.com/traefik/traefik/blob/d317cd90fcb579722ab29c35410c10b6a135f949/go.mod#L379
so I fired the PR targeting
verify_wasip1branch too.