envoy: Lua memory leak
Hi, I am having a problem with lua scripts. When using envoy.lua http filter envoy allocates memory after each request, bot does not free it. In my case, after around 920K request, 1GB is allocated and envoy crashes.
This happens even when I just use the respond function:
function envoy_on_request(request_handle)
request_handle:respond({[":status"] = "200"})
end
But it does not happen when I use print:
function envoy_on_request(request_handle)
print("hello")
end
Since I am using a docker image, I dont have a stacktrace of this error, but here is full console output. envoy_output.txt
I have also tried using “collectgarbage()” function but it doesn’t help, just slows down whole envoy.
About this issue
- Original URL
- State: closed
- Created 6 years ago
- Comments: 23 (12 by maintainers)
Commits related to this issue
- lua filter: fix memory leak due to circular runtime reference See code comments in the diff for more details. Fixes https://github.com/envoyproxy/envoy/issues/3570 Signed-off-by: Matt Klein <mklein... — committed to envoyproxy/envoy by deleted user 6 years ago
- lua filter: fix memory leak due to circular runtime reference (#3855) See code comments in the diff for more details. Fixes https://github.com/envoyproxy/envoy/issues/3570 Signed-off-by: Matt K... — committed to envoyproxy/envoy by deleted user 6 years ago
@yolken-stripe thanks a ton for digging into this. Let me page this code in and I will get back to you. cc @dio since he might also be interested in looking at this.
@mattklein123 I have partially solved this issue. When I was changing or adding headers the error was caused by not having enough file descriptors.
But the other part is caused by lua not freeing memory either when: 1) request_handle:respond() function is called e.g. script1.lua.txt This does not cause lua to log an error, just allocates 1KB of memory and does not free it. Stats and visualized differential dumps: stats1.txt, gv1.pdf.
2) when lua gets a bad argument e.g. script2.lua.txt This causes lua to log an error, also allocates 1KB of memory per request and does not free it. Log:
[2018-06-18 14:43:53.826][17303][error][lua] source/extensions/filters/http/lua/lua_filter.cc:512] script log: script2.lua:3: bad argument #2 to 'replace' (string expected, got nil)Stats and visualized differential dumps: stats2.txt, gv2.pdf.At the end, both of these issues cause Segmentation fault. In those scripts I don’t use global variables, and NOP scripts work fine.
@lwhile Seems to be lack of symbol information, you can use
bazel -c dbgto build envoy