webhook: Too many open files

After a few hours of running I get “Too many open files” error, and webhook dies.

OS: debian buster, Webhook installed from official apt sources

ls -l /proc/11294/fd | wc -l result: 721 open socket: lrwx------ 1 root root 64 Apr 24 21:44 718 -> ‘socket:[3390743]’ lrwx------ 1 root root 64 Apr 24 21:44 719 -> ‘socket:[3390857]’

my conf:

[
  {
    "id": "minio-webhook",
    "execute-command": "/root/webhook/rclone-sync.sh",
    "command-working-directory": "/root/webhook/",
    "pass-arguments-to-command":
    [
      {
            "source": "payload",
            "name": "Key"
      }
    ]
  }
]

Any ideas how to fix it?

thanks

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 27 (7 by maintainers)

Most upvoted comments

Looks like they’re rolling out the fix upstream https://github.com/minio/minio/issues/9460#issuecomment-619766606

😃

adnanh on Apr 27, 2020

BTW, the Go docs for http.Server.SetKeepAlivesEnabled discourages disabling keep-alives:

Only very resource-constrained environments or servers in the process of shutting down should disable them.

I still believe the underlying issue is a bug upstream in minio/Go.

moorereason on Apr 26, 2020

Awesome, thank you! Regarding the existence check, we’ll be rolling out a new feature that can help with that, keep an eye on #356

adnanh on Apr 29, 2020

latest minio solved this ‘issue’ 😃 thanks for your help guys

dyipon on Apr 29, 2020

I was curious how one might work around this ulimit issue if you happened to be hosting a super-busy webhook instance. 🚀

I found this recommendation from MongoDB on updating the webhook service unit file in systemd:

[Service]
# (open files)
LimitNOFILE=64000

moorereason on Apr 27, 2020

well, i think keep-alive is not essential for a ‘simple’ webhook service, because these requests are not speed-relevant (or it is, +5msec for an TCP connection opening is not a big deal).

It would superb if we can switch keep-alive settings in config. Thanks a lot.

dyipon on Apr 26, 2020

no worries, systemctl automatically restart webhook on failure.

I thought I installed the latest version, but debian apt package version is 2.6.9-1. Now I leeched the latest the binary, but connections still kept alive if minio gives incorrect header: “error parsing body payload due to unsupported content type header:” “error extracting command arguments: couldn’t retrieve argument for {Source:payload Name:Key EnvName: Base64Decode:false}”

dyipon on Apr 25, 2020