go-internal: testscript: "signal: killed" exec errors on MacOS 12

     --- FAIL: TestScript/flags (0.11s)
        testscript.go:429: > exec shfmt -h
            [signal: killed]
            FAIL: testdata/script/flags.txtar:1: unexpected command failure

I’ve seen this in a number of projects of mine, like:

• https://github.com/burrowers/garble/pull/614
• https://github.com/mvdan/sh/pull/934#issuecomment-1287113938
• https://github.com/mvdan/sh/commit/461be7eb20ff0eca28755287f192b258f0ac3a34

@rvagg mentions the same crash in https://github.com/ipld/go-car/pull/364, and in the past, others like @mr-joshcrane have mentioned the same error on Slack.

This must be something going wrong with either testscript or Go, because for example, that TestScript/flags test from above was just running exec shfmt -h, showing the help output from a Go program. You can see that the testscript file is rather boring, so it’s not doing anything particularly worrying.

Personally, I’ve worked around this by downgrading from macos-latest on GitHub Actions (which switched to macos-12 late last year) to macos-11, which seems to make the failures go away entirely. But of course that’s not a complete fix.

I first hoped that this would be fixed in Go 1.20 with https://go-review.googlesource.com/c/go/+/460476, and that may still be true, given that there are four distinct os/exec bugs for Mac there. But it’s just a good guess, I haven’t verified this yet - nor do I have a Mac machine to test with. Help would be appreciated.

The only other recent mentions of “signal: killed” upstream for Mac are https://github.com/golang/go/issues/57418 and https://github.com/golang/go/issues/57239, and they both seem to point to processes being OOM-killed by the system. This could be the case for us as well, perhaps either due to the OS version upgrade changing the OOM behavior, or perhaps because the macos-12 GitHub machines have less available memory. But I’d also find it hard to believe, given that testscript doesn’t use a particularly high amount of memory.

Filing this issue to track investigation and progress.