falcon: Mysterious SIGKILL errors about Async::Container::Forked
we have many error logs that look like:
{
"time": "2025-04-24T23:16:40+00:00",
"severity": "error",
"oid": 53504,
"pid": 1,
"subject": "Async::Container::Forked",
"status": "pid 11231 SIGKILL (signal 9)"
}
- do you know what they mean? they might be coming from the
async-containerdependency, but it seems core tofalconwhat with pid 1 reporting on the status of a much higher pid, and there is nothing else running in the affected k8s pods. - should we be worried about them? or might they be somewhat innocuous?
- if we should maybe be worried about them, do you know how we can get more information about the issue?
we are running 2 falcon process forks per pod on 16 pods and getting ~190 such errors per day. sample ps output:
root 1 0.0 0.2 788804 153064 ? Ssl Apr23 0:23 ruby /app/bin/falcon-host /app/falcon.rb
root 2658293 21.6 1.3 1557960 841836 ? Sl 00:11 1:28 shimmy (C=23/3.42K R=2/3.42K L=0.273)
root 2664265 21.3 0.4 999304 316392 ? Sl 00:17 0:18 shimmy (C=9/611 R=3/610 L=0.195)
also curious what the statistics in the forked process names mean 🙂
thanks!
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Reactions: 1
- Comments: 45 (21 by maintainers)
You can configure falcon to dump the memory when a leak is detected, and then analyze it:
https://github.com/socketry/falcon/blob/main/examples/supervisor/readme.md
https://github.com/Shopify/heap-profiler
Thanks for clarifying that.
I just released Async v2.25.0 which includes basic support for offloading
IO.select. Can you try that and let me know if it helps?also: turns out caching httpx sessions in a thread-local variable was unnecessary. we’re caching them in a class variable now and it works great.
solved, going back to your earlier tip to thread-local cache connection instances 😅
I am happy to hear that you are moving forward. Let me know how you get on.
Apologies, I don’t think it’s documented, I just expected every system to have
psavailable, but I can add a warning if it’s not available.These are probably context switching / non-blocking IO - are you using
Async::HTTPorNet::HTTP? They are probably not the cause.For sure, a core dump would help identify the issue.
Thanks for your report and questions.
Firstly, the easy one - the meaning of the statistics string is now documented: https://github.com/socketry/falcon/commit/59e4db8640b3b8234579b890d62052815d0cff95 - let me know if that isn’t clear enough.