falcon: Mysterious SIGKILL errors about Async::Container::Forked

we have many error logs that look like:

{
  "time": "2025-04-24T23:16:40+00:00",
  "severity": "error",
  "oid": 53504,
  "pid": 1,
  "subject": "Async::Container::Forked",
  "status": "pid 11231 SIGKILL (signal 9)"
}
  1. do you know what they mean? they might be coming from the async-container dependency, but it seems core to falcon what with pid 1 reporting on the status of a much higher pid, and there is nothing else running in the affected k8s pods.
  2. should we be worried about them? or might they be somewhat innocuous?
  3. if we should maybe be worried about them, do you know how we can get more information about the issue?

we are running 2 falcon process forks per pod on 16 pods and getting ~190 such errors per day. sample ps output:

root           1  0.0  0.2 788804 153064 ?       Ssl  Apr23   0:23 ruby /app/bin/falcon-host /app/falcon.rb
root     2658293 21.6  1.3 1557960 841836 ?      Sl   00:11   1:28 shimmy (C=23/3.42K R=2/3.42K L=0.273)
root     2664265 21.3  0.4 999304 316392 ?       Sl   00:17   0:18 shimmy (C=9/611 R=3/610 L=0.195)

also curious what the statistics in the forked process names mean 🙂

thanks!

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Reactions: 1
  • Comments: 45 (21 by maintainers)

Most upvoted comments

You can configure falcon to dump the memory when a leak is detected, and then analyze it:

https://github.com/socketry/falcon/blob/main/examples/supervisor/readme.md

https://github.com/Shopify/heap-profiler

Thanks for clarifying that.

I just released Async v2.25.0 which includes basic support for offloading IO.select. Can you try that and let me know if it helps?

also: turns out caching httpx sessions in a thread-local variable was unnecessary. we’re caching them in a class variable now and it works great.

httpx already does the heavy lifting of managing the whole thread-safety / fork-safety / fiber-safety so you don’t have to 😃

solved, going back to your earlier tip to thread-local cache connection instances 😅

I am happy to hear that you are moving forward. Let me know how you get on.

Apologies, I don’t think it’s documented, I just expected every system to have ps available, but I can add a warning if it’s not available.

also wondering if it would be related to very long outbound http calls, a not insignificant number of which time out after 15, 40, or 60 seconds

These are probably context switching / non-blocking IO - are you using Async::HTTP or Net::HTTP? They are probably not the cause.

For sure, a core dump would help identify the issue.

Thanks for your report and questions.

Firstly, the easy one - the meaning of the statistics string is now documented: https://github.com/socketry/falcon/commit/59e4db8640b3b8234579b890d62052815d0cff95 - let me know if that isn’t clear enough.