traefik: wasm plugin memory leak in Traefik v3

Welcome!

  • Yes, I’ve searched similar issues on GitHub and didn’t find any.
  • Yes, I’ve searched similar issues on the Traefik community forum and didn’t find any.

What did you do?

add a simple wasm traefik middleware plugin to a router. run ab benchmark to make some request you will see memory increase stop the benchmark and wait you will see the memory keep leaks, no matter how long you waited

the more concurrent requests make, the more memory leak.

@juliens @zetaab could you plz take a look at this?

here’s the traefik memory usage grafana graph from our real server, after enabling wasm based plugin:

image


steps to reproduce

go version 1.23.1 under Linux amd64 machine

git clone https://github.com/ttys3/traefik-wasm-plugin-memory-leak-demo.git

cd traefik-wasm-plugin-memory-leak-demo

pushd plugins-local/src/github.com/traefik/plugindemowasm/
# build wasm plugin
./build.sh

# you will see generated plugin.wasm now

# now we go back to traefik-wasm-plugin-memory-leak-demo dir
popd

# download newest traefik
curl -LO https://github.com/traefik/traefik/releases/download/v3.1.4/traefik_v3.1.4_linux_amd64.tar.gz
tar xvf traefik_v3.1.4_linux_amd64.tar.gz

# run the demo backend service, it is a simple http static server
./run-downstream-service.sh

# run traefik
./run.sh

now install ps_mem if you do not have it

sudo pip install ps_mem

now we open a new terminal to watch the memory

./watch-mem.sh

we 'll see that the init memory usage is:

image

go we run ./bench.sh in the repo to make some requests (you should install ab - the Apache HTTP server benchmarking tool, if you os does not installed it, or other bench tools also OK)

adjust params of ab if you system does not have too much free memory, in case OOM

./bench.sh

Benchmarking localhost (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Completed 1000 requests
Finished 1000 requests


Server Software:        ECAcc
Server Hostname:        localhost
Server Port:            6688

Document Path:          /
Document Length:        1256 bytes

Concurrency Level:      100
Time taken for tests:   6.276 seconds
Complete requests:      1000
Failed requests:        0
Total transferred:      1632250 bytes
HTML transferred:       1256000 bytes
Requests per second:    159.33 [#/sec] (mean)
Time per request:       627.634 [ms] (mean)
Time per request:       6.276 [ms] (mean, across all concurrent requests)
Transfer rate:          253.97 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.4      0       2
Processing:   184  537 905.7    200    4695
Waiting:      184  536 905.4    200    4695
Total:        184  537 906.0    200    4696

Percentage of the requests served within a certain time (ms)
  50%    200
  66%    213
  75%    248
  80%    321
  90%   2040
  95%   3256
  98%   3400
  99%   3542
 100%   4696 (longest request)

What did you see instead?

traefik memory keeps growing and never free

image

image

image

now we run

go tool pprof http://127.0.0.1:8080/debug/pprof/heap

type web and enter, will see the heap graph:

image

and here is the pprof file:

pprof.traefik.alloc_objects.alloc_space.inuse_objects.inuse_space.009.pb.gz

and also the allocs

image

allocs pprof file

pprof.traefik.alloc_objects.alloc_space.inuse_objects.inuse_space.010.pb.gz

note:

the plugindemowasm plugin here is intended to demostrate the memory leak issue.

not for real usage.

What version of Traefik are you using?

 ./traefik version                    
Version:      3.1.4
Codename:     comte
Go version:   go1.23.1
Built:        2024-09-19T13:47:17Z
OS/Arch:      linux/amd64

What is your environment & configuration?

# Static configuration

log:
  level: TRACE

global:
  checkNewVersion: false
  sendAnonymousUsage: false

api:
  dashboard: true
  insecure: true
  debug: true

metrics:
  prometheus: {}

entryPoints:
  web:
   address: ":6688"

providers:
  # https://doc.traefik.io/traefik/providers/file/
  file:
    watch: true
    filename: "dyn.yaml"

experimental:
  localPlugins:
    demowasm:
      moduleName: github.com/traefik/plugindemowasm

Add more configuration information here.

dyn.yaml

# Dynamic configuration

http:
  routers:
    my-router:
      rule: Host(`localhost`)
      service: service-foo
      entryPoints:
        - web
      middlewares:
        - demowasm

  services:
   service-foo:
      loadBalancer:
        passHostHeader: false
        servers:
          - url: http://www.example.com/

  middlewares:
    demowasm:
      plugin:
        demowasm:
          headers:
            X-Demo: testxxxxx
            X-URL: 'http://example.localhost/'

test 2

if we disable the wasm based plugin in dyn.yaml :

diff --git a/dyn.yaml b/dyn.yaml
index a1ce928..f71cdec 100755
--- a/dyn.yaml
+++ b/dyn.yaml
@@ -8,7 +8,7 @@ http:
       entryPoints:
         - web
       middlewares:
-        - demowasm
+        # - demowasm
 
   services:
    service-foo:

and run ./bench.sh again, only cost about100MB memory:

image


maybe related

https://github.com/jcchavezs/coraza-http-wasm-traefik/issues/9

https://community.traefik.io/t/traefik-v3-coraza-waf-high-resource-usage/22814/8

https://github.com/tetratelabs/proxy-wasm-go-sdk/issues/450

also tried https://github.com/wasilibs/nottinygc, but no difference

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 19 (18 by maintainers)

Commits related to this issue

Most upvoted comments

prev I was wrong, the plugin init memory leak still exists:

no http request + make changes to dyn.yaml with 51 service has the wasm plugin middleware enabled for 100 times

the memory leak keeps, the problem exists, because, new wasm host and guest created each time, but the memory not GC ed. we have to call nethttp.middleware.Close() (which will call handler.middleware.Close() -> wazero.Runtime.Close() ) to free the memory

so here comes the second commit: https://github.com/traefik/http-wasm-host-go/pull/2

the memory leak version (orig traefik):

4 GC round after we stop to change dyn.yaml:

image


with my PR’s second commit:

0 GC round after we stop to change dyn.yaml : image

2 GC round after we stop to change dyn.yaml : image


init startup with 51 service has the wasm plugin middleware enabled memory usage:

image

after ./update-dyn-config.sh run: image

after 2 more GC rounds:

image

compare test result: the PR code works fine for dynamic config loading casued wasm plugin leak

@juliens here’s the PR https://github.com/traefik/http-wasm-host-go/pull/2

be aware that: traefik make uses traefik/http-wasm-host-go verify_wasip1 branch’s code, and has go mod replace directive in go.mod file.

https://github.com/traefik/traefik/blob/d317cd90fcb579722ab29c35410c10b6a135f949/go.mod#L379

so I fired the PR targeting verify_wasip1 branch too.