hugo: watch with lots of files dies

$ find . -type f | wc -l
    8149

29 pages created
0 tags index created
1 topics index created
in 37 ms
Watching for changes in /Users/philips/trees/ifup-hugo/content
Web Server is available at http://localhost:1313
Press ctrl+c to stop
panic: listen tcp :1313: too many open files

goroutine 27 [running]:
runtime.panic(0x3d2a20, 0x210b68240)
    /usr/local/go/src/pkg/runtime/panic.c:266 +0xb6
github.com/spf13/hugo/commands.serve(0x521)
    /home/parallels/.gopathBuild/src/github.com/spf13/hugo/commands/server.go:69 +0x3fa
created by github.com/spf13/hugo/commands.NewWatcher
    /home/parallels/.gopathBuild/src/github.com/spf13/hugo/commands/hugo.go:194 +0x28f

goroutine 1 [semacquire]:
sync.runtime_Semacquire(0x210ad7cd8)
    /usr/local/go/src/pkg/runtime/sema.goc:199 +0x30
sync.(*WaitGroup).Wait(0x210b92180)
    /usr/local/go/src/pkg/sync/waitgroup.go:127 +0x14b
github.com/spf13/hugo/commands.NewWatcher(0x521, 0x0, 0x0)
    /home/parallels/.gopathBuild/src/github.com/spf13/hugo/commands/hugo.go:197 +0x29f
github.com/spf13/hugo/commands.server(0x8bc520, 0x210aa4e40, 0x0, 0x3)
    /home/parallels/.gopathBuild/src/github.com/spf13/hugo/commands/server.go:53 +0x24a
github.com/spf13/cobra.(*Command).execute(0x8bc520, 0x210a35020, 0x3, 0x3, 0x0, ...)
    /home/parallels/.gopathBuild/src/github.com/spf13/cobra/command.go:273 +0x116
github.com/spf13/cobra.(*Command).findAndExecute(0x8bc440, 0x210a35010, 0x4, 0x4, 0x0, ...)
    /home/parallels/.gopathBuild/src/github.com/spf13/cobra/command.go:259 +0xa8
github.com/spf13/cobra.(*Command).Execute(0x8bc440, 0x0, 0x0)
    /home/parallels/.gopathBuild/src/github.com/spf13/cobra/command.go:308 +0x66e
github.com/spf13/hugo/commands.Execute()
    /home/parallels/.gopathBuild/src/github.com/spf13/hugo/commands/hugo.go:52 +0x2f
main.main()
    /home/parallels/.gopathBuild/src/github.com/spf13/hugo/main.go:21 +0x1a

goroutine 24 [chan receive]:
github.com/howeyc/fsnotify.(*Watcher).purgeEvents(0x210b508f0)
    /home/parallels/.gopathBuild/src/github.com/howeyc/fsnotify/fsnotify.go:21 +0x41
created by github.com/howeyc/fsnotify.NewWatcher
    /home/parallels/.gopathBuild/src/github.com/howeyc/fsnotify/fsnotify_bsd.go:103 +0x2fd

goroutine 23 [syscall]:
syscall.Syscall6(0x16b, 0x2f, 0x0, 0x0, 0x210b01000, ...)
    /usr/local/go/src/pkg/syscall/asm_darwin_amd64.s:41 +0x5
syscall.kevent(0x2f, 0x0, 0x0, 0x210b01000, 0xa, ...)
    /usr/local/go/src/pkg/syscall/zsyscall_darwin_amd64.go:199 +0x85
syscall.Kevent(0x2f, 0x0, 0x0, 0x0, 0x210b01000, ...)
    /usr/local/go/src/pkg/syscall/syscall_bsd.go:431 +0x9d
github.com/howeyc/fsnotify.(*Watcher).readEvents(0x210b508f0)
    /home/parallels/.gopathBuild/src/github.com/howeyc/fsnotify/fsnotify_bsd.go:328 +0x31b
created by github.com/howeyc/fsnotify.NewWatcher
    /home/parallels/.gopathBuild/src/github.com/howeyc/fsnotify/fsnotify_bsd.go:102 +0x2e6

goroutine 25 [select]:
github.com/spf13/hugo/commands.func·006()
    /home/parallels/.gopathBuild/src/github.com/spf13/hugo/commands/hugo.go:172 +0x23c
created by github.com/spf13/hugo/commands.NewWatcher
    /home/parallels/.gopathBuild/src/github.com/spf13/hugo/commands/hugo.go:185 +0x1de

goroutine 26 [finalizer wait]:
runtime.park(0xc600, 0x8c4268, 0x8c0d68)
    /usr/local/go/src/pkg/runtime/proc.c:1342 +0x66
runfinq()
    /usr/local/go/src/pkg/runtime/mgc0.c:2276 +0x84
runtime.goexit()
    /usr/local/go/src/pkg/runtime/proc.c:1394

About this issue

Original URL
State: closed
Created 10 years ago
Comments: 26 (17 by maintainers)

Commits related to this issue

Skip directories like node_modules from the watchlist A local `node_modules` directory can easily contain tens of thousands of files, easily exhausting the tiny default max open files limit especiall... — committed to gohugoio/hugo by anthonyfok 9 years ago
Use osext on GitHub Fixes #922 add finalize command and tests to change draft status of specified content to false and set date to Now() undraft command Allow hyphens in shortcode name Fixes #929... — committed to mohae/hugo by bep 9 years ago
Skip directories like node_modules from the watchlist A local `node_modules` directory can easily contain tens of thousands of files, easily exhausting the tiny default max open files limit especiall... — committed to tychoish/hugo by anthonyfok 9 years ago

Most upvoted comments

Hi @chibicode,

The following appear to be one of the most comprehensive info on max open files limit on Mac OS X. Please take a look:

http://docs.basho.com/riak/latest/ops/tuning/open-files-limit/#Mac-OS-X

In particular, OS X Yosemite (10.10), which came out in October 2014, eight months after the aforementioned fixes, has made changes to how this limit is set.

You may also try this for a quick fix:

$ sudo sysctl -w kern.maxfiles=65536
$ sudo sysctl -w kern.maxfilesperproc=65536
$ ulimit -n 65536 65536

Please let us know how it works! Thanks!

anthonyfok on Mar 10, 2015

Regarding my May 14 comment, I found this command (sudo launchctl limit maxfiles 1000000 1000000) that lets me increase the maximum number of open file descriptors for the current session, and then hugo server --watch works!

hypertexthero on Jun 29, 2016

@nathany:

Though perhaps fsnotify will use fsevents

That would be awesome! 😄

anthonyfok on May 16, 2016

@anthonyfok I don’t see why not. Though perhaps fsnotify will use fsevents, or a new replacement for fsnotify.

nathany on May 16, 2016

Could Hugo use this?

FSEvents for Go (OS X) - Clean room implementation https://github.com/fsnotify/fsevents

anthonyfok on May 15, 2016

Hi @anthonyfok, wow, thank you so much for such a thorough response. I did try ulimit -n but yes, I couldn’t get it to go past 10240. I tried your commands but the last one returned limit: setrlimit failed: invalid argument.

$ hugo check ulimit
Current rLimit: {4864 9223372036854775807}
Attempting to increase limit
rLimit after change: {10240 10240}

$ sudo sysctl -w kern.maxfiles=65536
kern.maxfiles: 12288 -> 65536

$ sudo sysctl -w kern.maxfilesperproc=65536
kern.maxfiles: 65536 -> 65536

$ ulimit -n 65536 65536
limit: setrlimit failed: invalid argument

Also thanks for https://github.com/spf13/hugo/commit/b9b70fb6b085e5141078bee80f342d2a9fdc2f5e - although this looks like a huge hack and I now feel bad for requesting this.

Sorry I’m on a hurry, but really appreciated your response.

chibicode on Mar 11, 2015