go: cmd/gofmt: parallel processing causing "too many open files" errors

What version of Go are you using (`go version`)?

$ go version
go version go1.18beta2 darwin/arm64

What did you do?

Execute gofmt -w . on decent sized project (1660 go files in my case).

What did you expect to see?

No errors.

What did you see instead?

Long list of errors:

…
open aaa/bbb/ccc.go: too many open files
open iii/jjj/kkk.go: too many open files
open uuu/vvv/www.go: too many open files
fcntl xxx/yyy/zzz/test: too many open files

Maybe related to

https://tip.golang.org/doc/go1.18#gofmt

gofmt now reads and formats input files concurrently […]

$ ulimit -a
-t: cpu time (seconds)              unlimited
-f: file size (blocks)              unlimited
-d: data seg size (kbytes)          unlimited
-s: stack size (kbytes)             8176
-c: core file size (blocks)         0
-v: address space (kbytes)          unlimited
-l: locked-in-memory size (kbytes)  unlimited
-u: processes                       2666
-n: file descriptors                256

About this issue

Original URL
State: closed
Created 2 years ago
Reactions: 2
Comments: 22 (17 by maintainers)

Most upvoted comments

The limit is per-process - I’d be surprised if initializing a main Go program managed to open 56 file descriptors and keep them open. I can imagine a few, like stdin/stdout/stderr, but certainly nothing close to get from 200 to 256.

mvdan on Feb 15, 2022

If a hard limit of 200 open files doesn’t cause a slow-down on this machine, I imagine it will be plenty for practically all other machines.

I could see it causing a slowdown when formatting a large repo hosted on a very slow disk (such as an NFS mount).

That said, if the disk is slow then everything will be slow, so it’s not clear to me why we would need to optimize gofmt in particular for that case.

bcmills on Feb 15, 2022