go: runtime/cgo: pthread_create failed: Resource temporarily unavailable

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (`go version`)?

go version go1.10 linux/amd64

Does this issue reproduce with the latest release?

go1.10 is latest

What operating system and processor architecture are you using (`go env`)?

GOARCH=“amd64” GOOS=“linux”

What did you do?

If possible, provide a recipe for reproducing the error. A complete runnable program is good. A link on play.golang.org is best.

my process starts around 500 child processes. The number of os level threads is creeping up slowly until it reaches around 10k, at which point child processes start to die with the below message.

Process limits seem set sufficiently high Limit Soft Limit Hard Limit Units
Max processes 257093 257093 processes

$ cat /proc/sys/kernel/threads-max 514187

What did you expect to see?

no crash 😉

What did you see instead?

runtime/cgo: pthread_create failed: Resource temporarily unavailable SIGABRT: abort PC=0x7f24685ab428 m=44 sigcode=18446744073709551610 goroutine 0 [idle]: runtime: unknown pc 0x7f24685ab428 stack: frame={sp:0x7f2407ffea08, fp:0x0} stack=[0x7f24077ff2f0,0x7f2407ffeef0) 00007f2407ffe908: 00007f2468d84168 00007f2407ffea68 00007f2407ffe918: 00007f2468b67b1f 0000000000000002 00007f2407ffe928: 00007f2468d79a80 0000000000000005 00007f2407ffe938: 0000000000f021e0 00007f23d80008c0 00007f2407ffe948: 00000000000000f1 0000000000000011 00007f2407ffe958: 0000000000000000 0000000000c2597a 00007f2407ffe968: 00007f2468b6cac6 0000000000000005 00007f2407ffe978: 0000000000000000 0000000100000000 00007f2407ffe988: 00007f246857cde0 00007f2407ffeb20 00007f2407ffe998: 00007f2468b74923 000000ffffffffff 00007f2407ffe9a8: 0000000000000000 0000000000000000 00007f2407ffe9b8: 0000000000000000 2525252525252525 00007f2407ffe9c8: 2525252525252525 0000000000000000 00007f2407ffe9d8: 00007f246893b700 0000000000c2597a 00007f2407ffe9e8: 00007f23d80008c0 00000000000000f1 00007f2407ffe9f8: 0000000000000011 0000000000000000 00007f2407ffea08: <00007f24685ad02a 0000000000000020 00007f2407ffea18: 0000000000000000 0000000000000000 00007f2407ffea28: 0000000000000000 0000000000000000 00007f2407ffea38: 0000000000000000 0000000000000000 00007f2407ffea48: 0000000000000000 0000000000000000 00007f2407ffea58: 0000000000000000 0000000000000000 00007f2407ffea68: 0000000000000000 0000000000000000 00007f2407ffea78: 0000000000000000 0000000000000000 00007f2407ffea88: 0000000000000000 0000000000000000 00007f2407ffea98: 0000000000000000 0000000000000000 00007f2407ffeaa8: 00007f24685eebff 00007f246893b540 00007f2407ffeab8: 0000000000000001 00007f246893b5c3 00007f2407ffeac8: 00000000000000f1 0000000000000011 00007f2407ffead8: 00007f24685f0409 000000000000000a 00007f2407ffeae8: 00007f246866d2dd 000000000000000a 00007f2407ffeaf8: 00007f246893c770 0000000000000000 runtime: unknown pc 0x7f24685ab428 stack: frame={sp:0x7f2407ffea08, fp:0x0} stack=[0x7f24077ff2f0,0x7f2407ffeef0) 00007f2407ffe908: 00007f2468d84168 00007f2407ffea68 00007f2407ffe918: 00007f2468b67b1f 0000000000000002 00007f2407ffe928: 00007f2468d79a80 0000000000000005 00007f2407ffe938: 0000000000f021e0 00007f23d80008c0 00007f2407ffe948: 00000000000000f1 0000000000000011 00007f2407ffe958: 0000000000000000 0000000000c2597a 00007f2407ffe968: 00007f2468b6cac6 0000000000000005 00007f2407ffe978: 0000000000000000 0000000100000000 00007f2407ffe988: 00007f246857cde0 00007f2407ffeb20 00007f2407ffe998: 00007f2468b74923 000000ffffffffff 00007f2407ffe9a8: 0000000000000000 0000000000000000 00007f2407ffe9b8: 0000000000000000 2525252525252525 00007f2407ffe9c8: 2525252525252525 0000000000000000 00007f2407ffe9d8: 00007f246893b700 0000000000c2597a 00007f2407ffe9e8: 00007f23d80008c0 00000000000000f1 00007f2407ffe9f8: 0000000000000011 0000000000000000 00007f2407ffea08: <00007f24685ad02a 0000000000000020 00007f2407ffea18: 0000000000000000 0000000000000000 00007f2407ffea28: 0000000000000000 0000000000000000 00007f2407ffea38: 0000000000000000 0000000000000000 00007f2407ffea48: 0000000000000000 0000000000000000 00007f2407ffea58: 0000000000000000 0000000000000000 00007f2407ffea68: 0000000000000000 0000000000000000 00007f2407ffea78: 0000000000000000 0000000000000000 00007f2407ffea88: 0000000000000000 0000000000000000 00007f2407ffea98: 0000000000000000 0000000000000000 00007f2407ffeaa8: 00007f24685eebff 00007f246893b540 00007f2407ffeab8: 0000000000000001 00007f246893b5c3 00007f2407ffeac8: 00000000000000f1 0000000000000011 00007f2407ffead8: 00007f24685f0409 000000000000000a 00007f2407ffeae8: 00007f246866d2dd 000000000000000a 00007f2407ffeaf8: 00007f246893c770 0000000000000000 goroutine 632 [running]: runtime.systemstack_switch() /opt/go/1.10.0/go/src/runtime/asm_amd64.s:363 fp=0xc4204f6d50 sp=0xc4204f6d48 pc=0x457270 runtime.gcMarkTermination(0x3ff75e93c8506a48) /opt/go/1.10.0/go/src/runtime/mgc.go:1647 +0x407 fp=0xc4204f6f20 sp=0xc4204f6d50 pc=0x41a907 runtime.gcMarkDone() /opt/go/1.10.0/go/src/runtime/mgc.go:1513 +0x22c fp=0xc4204f6f48 sp=0xc4204f6f20 pc=0x41a49c runtime.gcBgMarkWorker(0xc420048500) /opt/go/1.10.0/go/src/runtime/mgc.go:1912 +0x2e7 fp=0xc4204f6fd8 sp=0xc4204f6f48 pc=0x41b417 runtime.goexit() /opt/go/1.10.0/go/src/runtime/asm_amd64.s:2361 +0x1 fp=0xc4204f6fe0 sp=0xc4204f6fd8 pc=0x459de1 created by runtime.gcBgMarkStartWorkers /opt/go/1.10.0/go/src/runtime/mgc.go:1723 +0x79

About this issue

Original URL
State: closed
Created 6 years ago
Reactions: 2
Comments: 29 (13 by maintainers)

Links to this issue

docker swarm not restarting unhealthy selenium hub containers - Stack Overflow

Commits related to this issue

Disable cgo xref: https://github.com/golang/go/issues/24484 — committed to kubeguard/guard by tamalsaha 5 years ago

Most upvoted comments

I was trying to solve a hackrRank problem question. What I Expected: Test failed (or something similar )

What I Got: runtime/cgo: runtime/cgo: pthread_create failed: Resource temporarily unavailable pthread_create failed: Resource temporarily unavailable SIGABRT: abort PC=0x7febdf25d7bb m=12 sigcode=18446744073709551610

goroutine 0 [idle]: runtime: unknown pc 0x7febdf25d7bb stack: frame={sp:0x7feb9bffc850, fp:0x0} stack=[0x7feb99ffd288,0x7feb9bffce88) 00007feb9bffc750: 0000000000000005 0000ffff00001fa0 00007feb9bffc760: 00007feb9bffcc60 00007febd8d2f010 00007feb9bffc770: 0000000000000000 0000000000b16de0 00007feb9bffc780: 0000000000000000 00007febdf41cb00 00007feb9bffc790: 0000000000000005 0000000000000000 00007feb9bffc7a0: 00007feb9bffcb40 00007febdf236f98 00007feb9bffc7b0: 00007feb9bffcb70 00007febdf4234ba 00007feb9bffc7c0: 0000000000000000 0000000000000000 00007feb9bffc7d0: 0000000000000000 00007feb9bffcc60 00007feb9bffc7e0: 2525252525252525 2525252525252525 00007feb9bffc7f0: 000000ffffffffff 0000000000000000 00007feb9bffc800: 000000ffffffffff 0000000000000000 00007feb9bffc810: 4d6c6f72746e6f43 6f6e3d7265747361 00007feb9bffc820: 7273752f3a6e6962 732f6c61636f6c2f 00007feb9bffc830: 7273752f3a6e6962 622f6c61636f6c2f 00007feb9bffc840: 2f7273752f3a6e69 73752f3a6e696273 00007feb9bffc850: <0000000000000000 6e69622f3a6e6962 00007feb9bffc860: 6e75720000000000 6f67632f656d6974 00007feb9bffc870: 0000000000000000 0000000000000000 00007feb9bffc880: 000000c0000289c0 000000000000000d 00007feb9bffc890: 000000c000026500 0000000000000015 00007feb9bffc8a0: 000000c0000a01e0 0000000000000027 00007feb9bffc8b0: 0405060700010203 0c0d0e0f08090a0b 00007feb9bffc8c0: 000000c000112940 000000c000112e00 00007feb9bffc8d0: fffffffe7fffffff ffffffffffffffff 00007feb9bffc8e0: ffffffffffffffff ffffffffffffffff 00007feb9bffc8f0: ffffffffffffffff ffffffffffffffff 00007feb9bffc900: ffffffffffffffff ffffffffffffffff 00007feb9bffc910: ffffffffffffffff ffffffffffffffff 00007feb9bffc920: ffffffffffffffff ffffffffffffffff 00007feb9bffc930: ffffffffffffffff ffffffffffffffff 00007feb9bffc940: ffffffffffffffff ffffffffffffffff runtime: unknown pc 0x7febdf25d7bb stack: frame={sp:0x7feb9bffc850, fp:0x0} stack=[0x7feb99ffd288,0x7feb9bffce88) 00007feb9bffc750: 0000000000000005 0000ffff00001fa0 00007feb9bffc760: 00007feb9bffcc60 00007febd8d2f010 00007feb9bffc770: 0000000000000000 0000000000b16de0 00007feb9bffc780: 0000000000000000 00007febdf41cb00 00007feb9bffc790: 0000000000000005 0000000000000000 00007feb9bffc7a0: 00007feb9bffcb40 00007febdf236f98 00007feb9bffc7b0: 00007feb9bffcb70 00007febdf4234ba 00007feb9bffc7c0: 0000000000000000 0000000000000000 00007feb9bffc7d0: 0000000000000000 00007feb9bffcc60 00007feb9bffc7e0: 2525252525252525 2525252525252525 00007feb9bffc7f0: 000000ffffffffff 0000000000000000 00007feb9bffc800: 000000ffffffffff 0000000000000000 00007feb9bffc810: 4d6c6f72746e6f43 6f6e3d7265747361 00007feb9bffc820: 7273752f3a6e6962 732f6c61636f6c2f 00007feb9bffc830: 7273752f3a6e6962 622f6c61636f6c2f 00007feb9bffc840: 2f7273752f3a6e69 73752f3a6e696273 00007feb9bffc850: <0000000000000000 6e69622f3a6e6962 00007feb9bffc860: 6e75720000000000 6f67632f656d6974 00007feb9bffc870: 0000000000000000 0000000000000000 00007feb9bffc880: 000000c0000289c0 000000000000000d 00007feb9bffc890: 000000c000026500 0000000000000015 00007feb9bffc8a0: 000000c0000a01e0 0000000000000027 00007feb9bffc8b0: 0405060700010203 0c0d0e0f08090a0b 00007feb9bffc8c0: 000000c000112940 000000c000112e00 00007feb9bffc8d0: fffffffe7fffffff ffffffffffffffff 00007feb9bffc8e0: ffffffffffffffff ffffffffffffffff 00007feb9bffc8f0: ffffffffffffffff ffffffffffffffff 00007feb9bffc900: ffffffffffffffff ffffffffffffffff 00007feb9bffc910: ffffffffffffffff ffffffffffffffff 00007feb9bffc920: ffffffffffffffff ffffffffffffffff 00007feb9bffc930: ffffffffffffffff ffffffffffffffff 00007feb9bffc940: ffffffffffffffff ffffffffffffffff

goroutine 1 [semacquire]: sync.runtime_Semacquire(0xc0002f5438) /usr/local/go/src/runtime/sema.go:56 +0x42 sync.(*WaitGroup).Wait(0xc0002f5430) /usr/local/go/src/sync/waitgroup.go:130 +0x64 cmd/go/internal/work.(*Builder).Do(0xc0000bd860, 0xc0000c7040) /usr/local/go/src/cmd/go/internal/work/exec.go:186 +0x3c5 cmd/go/internal/work.runBuild(0xea3280, 0xc0000201a0, 0x1, 0x1) /usr/local/go/src/cmd/go/internal/work/build.go:387 +0x6e2 main.main() /usr/local/go/src/cmd/go/main.go:189 +0x57f

goroutine 6 [syscall]: os/signal.signal_recv(0x0) /usr/local/go/src/runtime/sigqueue.go:147 +0x9c os/signal.loop() /usr/local/go/src/os/signal/signal_unix.go:23 +0x22 created by os/signal.init.0 /usr/local/go/src/os/signal/signal_unix.go:29 +0x41

goroutine 21 [semacquire]: sync.runtime_SemacquireMutex(0xecabb4, 0xc0004de000, 0x1) /usr/local/go/src/runtime/sema.go:71 +0x47 sync.(*Mutex).lockSlow(0xecabb0) /usr/local/go/src/sync/mutex.go:138 +0xfc sync.(*Mutex).Lock(…) /usr/local/go/src/sync/mutex.go:81 sync.(*RWMutex).Lock(0xecabb0) /usr/local/go/src/sync/rwmutex.go:98 +0x97 syscall.forkExec(0xc0004ac000, 0x2a, 0xc0004ae020, 0x2, 0x2, 0xc0003fad68, 0x17, 0x19221bee00010200, 0xc000408380) /usr/local/go/src/syscall/exec_unix.go:193 +0x200 syscall.StartProcess(…) /usr/local/go/src/syscall/exec_unix.go:248 os.startProcess(0xc0004ac000, 0x2a, 0xc0004ae020, 0x2, 0x2, 0xc0003faf00, 0x0, 0x0, 0x0) /usr/local/go/src/os/exec_posix.go:51 +0x2b0 os.StartProcess(0xc0004ac000, 0x2a, 0xc0004ae020, 0x2, 0x2, 0xc0003faf00, 0x17, 0x0, 0x0) /usr/local/go/src/os/exec.go:102 +0x7c os/exec.(*Cmd).Start(0xc0004b2000, 0xc0004b6001, 0xc0004a0060) /usr/local/go/src/os/exec/exec.go:416 +0x50c os/exec.(*Cmd).Run(0xc0004b2000, 0xc0004a0060, 0x16) /usr/local/go/src/os/exec/exec.go:338 +0x2b cmd/go/internal/work.(*Builder).toolID(0xc0000bd860, 0xa3c6df, 0x7, 0xb, 0xc0003fb3c8) /usr/local/go/src/cmd/go/internal/work/buildid.go:193 +0x44d cmd/go/internal/work.(*Builder).buildActionID(0xc0000bd860, 0xc0002f3400, 0x0, 0x0, 0x0, 0x0) /usr/local/go/src/cmd/go/internal/work/exec.go:242 +0x1215 cmd/go/internal/work.(*Builder).build(0xc0000bd860, 0xc0002f3400, 0x0, 0x0) /usr/local/go/src/cmd/go/internal/work/exec.go:397 +0x52a6 cmd/go/internal/work.(*Builder).Do.func2(0xc0002f3400) /usr/local/go/src/cmd/go/internal/work/exec.go:117 +0x36d cmd/go/internal/work.(*Builder).Do.func3(0xc0002f5430, 0xc0000bd860, 0xc00036c940) /usr/local/go/src/cmd/go/internal/work/exec.go:177 +0x79 created by cmd/go/internal/work.(*Builder).Do /usr/local/go/src/cmd/go/internal/work/exec.go:164 +0x3a1

goroutine 22 [select]: cmd/go/internal/work.(*Builder).Do.func3(0xc0002f5430, 0xc0000bd860, 0xc00036c940) /usr/local/go/src/cmd/go/internal/work/exec.go:167 +0xf6 created by cmd/go/internal/work.(*Builder).Do /usr/local/go/src/cmd/go/internal/work/exec.go:164 +0x3a1

goroutine 23 [semacquire]: syscall.forkAndExecInChild1(0xc000434060, 0xc0004380e0, 0x3, 0x3, 0xc00046c000, 0x18, 0x18, 0x0, 0x0, 0xc0003fed68, …) /usr/local/go/src/syscall/exec_linux.go:180 +0x1f4 syscall.forkAndExecInChild(0xc000434060, 0xc0004380e0, 0x3, 0x3, 0xc00046c000, 0x18, 0x18, 0x0, 0x0, 0xc0003fed68, …) /usr/local/go/src/syscall/exec_linux.go:72 +0xcf syscall.forkExec(0xc000434000, 0x2a, 0xc000438020, 0x2, 0x2, 0xc0003fed68, 0x17, 0x3a7c51d000010200, 0xc000456000) /usr/local/go/src/syscall/exec_unix.go:201 +0x35b syscall.StartProcess(…) /usr/local/go/src/syscall/exec_unix.go:248 os.startProcess(0xc000434000, 0x2a, 0xc000438020, 0x2, 0x2, 0xc0003fef00, 0x0, 0x0, 0x0) /usr/local/go/src/os/exec_posix.go:51 +0x2b0 os.StartProcess(0xc000434000, 0x2a, 0xc000438020, 0x2, 0x2, 0xc0003fef00, 0x17, 0x0, 0x0) /usr/local/go/src/os/exec.go:102 +0x7c os/exec.(*Cmd).Start(0xc000444000, 0xc000446001, 0xc000406060) /usr/local/go/src/os/exec/exec.go:416 +0x50c os/exec.(*Cmd).Run(0xc000444000, 0xc000406060, 0x16) /usr/local/go/src/os/exec/exec.go:338 +0x2b cmd/go/internal/work.(*Builder).toolID(0xc0000bd860, 0xa3c6df, 0x7, 0xb, 0xc0003ff3c8) /usr/local/go/src/cmd/go/internal/work/buildid.go:193 +0x44d cmd/go/internal/work.(*Builder).buildActionID(0xc0000bd860, 0xc0002f2b40, 0x0, 0x0, 0x0, 0x0) /usr/local/go/src/cmd/go/internal/work/exec.go:242 +0x1215 cmd/go/internal/work.(*Builder).build(0xc0000bd860, 0xc0002f2b40, 0x0, 0x0) /usr/local/go/src/cmd/go/internal/work/exec.go:397 +0x52a6 cmd/go/internal/work.(*Builder).Do.func2(0xc0002f2b40) /usr/local/go/src/cmd/go/internal/work/exec.go:117 +0x36d cmd/go/internal/work.(*Builder).Do.func3(0xc0002f5430, 0xc0000bd860, 0xc00036c940) /usr/local/go/src/cmd/go/internal/work/exec.go:177 +0x79 created by cmd/go/internal/work.(*Builder).Do /usr/local/go/src/cmd/go/internal/work/exec.go:164 +0x3a1

goroutine 24 [semacquire]: os.(*Process).wait(0xc000434090, 0xa84348, 0xa84350, 0xa84340) /usr/local/go/src/os/exec_unix.go:37 +0x75 os.(*Process).Wait(…) /usr/local/go/src/os/exec.go:125 os/exec.(*Cmd).Wait(0xc0000b2160, 0x0, 0x0) /usr/local/go/src/os/exec/exec.go:501 +0x60 os/exec.(*Cmd).Run(0xc0000b2160, 0xc000394060, 0x16) /usr/local/go/src/os/exec/exec.go:341 +0x5c cmd/go/internal/work.(*Builder).toolID(0xc0000bd860, 0xa3c6df, 0x7, 0xb, 0xc0003a33c8) /usr/local/go/src/cmd/go/internal/work/buildid.go:193 +0x44d cmd/go/internal/work.(*Builder).buildActionID(0xc0000bd860, 0xc0002f2140, 0x0, 0x0, 0x0, 0x0) /usr/local/go/src/cmd/go/internal/work/exec.go:242 +0x1215 cmd/go/internal/work.(*Builder).build(0xc0000bd860, 0xc0002f2140, 0x0, 0x0) /usr/local/go/src/cmd/go/internal/work/exec.go:397 +0x52a6 cmd/go/internal/work.(*Builder).Do.func2(0xc0002f2140) /usr/local/go/src/cmd/go/internal/work/exec.go:117 +0x36d cmd/go/internal/work.(*Builder).Do.func3(0xc0002f5430, 0xc0000bd860, 0xc00036c940) /usr/local/go/src/cmd/go/internal/work/exec.go:177 +0x79 created by cmd/go/internal/work.(*Builder).Do /usr/local/go/src/cmd/go/internal/work/exec.go:164 +0x3a1

goroutine 25 [semacquire]: os.(*Process).wait(0xc0003aa120, 0xa84348, 0xa84350, 0xa84340) /usr/local/go/src/os/exec_unix.go:37 +0x75 os.(*Process).Wait(…) /usr/local/go/src/os/exec.go:125 os/exec.(*Cmd).Wait(0xc0001b5e40, 0x0, 0x0) /usr/local/go/src/os/exec/exec.go:501 +0x60 os/exec.(*Cmd).Run(0xc0001b5e40, 0xc00023cd50, 0x16) /usr/loc

Here’s the code

package main

import ( “bufio” “fmt” “io” “os” “strconv” “strings” )

// Complete the migratoryBirds function below. func migratoryBirds(arr []int32) int32 { var min int32 = 0 return min }

func main() { reader := bufio.NewReaderSize(os.Stdin, 16 * 1024 * 1024)

stdout, err := os.Create(os.Getenv("OUTPUT_PATH"))
checkError(err)

defer stdout.Close()

writer := bufio.NewWriterSize(stdout, 16 * 1024 * 1024)

arrCount, err := strconv.ParseInt(strings.TrimSpace(readLine(reader)), 10, 64)
checkError(err)

arrTemp := strings.Split(strings.TrimSpace(readLine(reader)), " ")

var arr []int32

for i := 0; i < int(arrCount); i++ {
    arrItemTemp, err := strconv.ParseInt(arrTemp[i], 10, 64)
    checkError(err)
    arrItem := int32(arrItemTemp)
    arr = append(arr, arrItem)
}

result := migratoryBirds(arr)

fmt.Fprintf(writer, "%d\n", result)

writer.Flush()

}

func readLine(reader *bufio.Reader) string { str, _, err := reader.ReadLine() if err == io.EOF { return “” }

return strings.TrimRight(string(str), "\r\n")

}

func checkError(err error) { if err != nil { panic(err) } }

zainabb12345 on Nov 6, 2020

As of today it gets reproduced on 4.19.0-6-cloud-amd64 #1 SMP Debian 4.19.67-2+deb10u2 (2019-11-11) x86_64 GNU/Linux ( I know, its pretty old kernel for end of 2023 era)

One alternative to chpst -m is ulimit -s since underlying it is doing the same “setrlimit(RLIMIT_STACK, …)” for -m flag.

e.g.ulimit -s 100000000000 && <program path>

pPanda-beta on Nov 7, 2023

In a Go program that uses cgo, new threads are created using pthread_create. If pthread_create fails, the Go runtime will retry up to 20 times. The relevant code is at https://golang.org/src/runtime/cgo/gcc_libinit.c#L91 .

ianlancetaylor on Oct 18, 2018

I’m able to reproduce this if I use chpst with the following program:

package main 
import "C"
func main() {
}

Running with chpst -m 1000000000 <program path> will crash with the following stacktrace:

runtime/cgo: pthread_create failed: Resource temporarily unavailable

goroutine 1 [running]:
runtime.systemstack_switch()
        /home/linuxbrew/.linuxbrew/Cellar/go/1.16.3_1/libexec/src/runtime/asm_amd64.s:339 fp=0xc000050788 sp=0xc000050780 pc=0x45b9c0
runtime.main()
        /home/linuxbrew/.linuxbrew/Cellar/go/1.16.3_1/libexec/src/runtime/proc.go:144 +0x89 fp=0xc0000507e0 sp=0xc000050788 pc=0x431ce9
runtime.goexit()
        /home/linuxbrew/.linuxbrew/Cellar/go/1.16.3_1/libexec/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc0000507e8 sp=0xc0000507e0 pc=0x45d7a1

strace seems to indicate that one of the calls to mmap is miscalculating available memory. When using chpst, there’s a failed mmap call of mmap(NULL, 1000005632, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0). If you don’t use chpst, that call is instead mmap(NULL, 8392704, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0).

It’s not clear to me if this is the same root cause as what was originally reported, but gut feeling says no. I can open a separate issue if desired.

rfratto on Jul 30, 2021

Things like making parallel DNS requests from a large number of goroutines can also contribute. I managed to drive the number of threads down by queuing requests, but

$ env CGO_ENABLED=0 go build

really fixed the issue for me.

Am Fr., 19. Okt. 2018 um 14:38 Uhr schrieb SjonHortensius < notifications@github.com>:

@xianglinghui https://github.com/xianglinghui - I have the same issue but as @ianlancetaylor https://github.com/ianlancetaylor explained

In a Go program that uses cgo, new threads are created using pthread_create

Eg. if you disable cgo, new threads will not use pthread_create but another mechanism.

For my application (which contains a very small number of concurrent go-routines and no explicit cgo usage but a lot of os.Exec calls) disabling cgo fixed a lot of Resource temporarily unavailable crashes as well. I’m not sure what causes this and how expected it is - but I’m just disabling cgo from now on

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/golang/go/issues/24484#issuecomment-431348065, or mute the thread https://github.com/notifications/unsubscribe-auth/AALulpms_fZX_Vkp1u0BPR0rhYuNaCBbks5umcfegaJpZM4S2QK1 .

fiber on Oct 19, 2018

compiling with CGO_ENABLED=0 and setting a low GOMAXPROCS did reduce the overall number of threads to about 1/3rd. Instead of peaking out at 12k threads, I’m now down to 4.5k, but still (very) slowly creeping up. Still investigating.

fiber on Apr 11, 2018

I believe there is no cgo being used outside the standard library. I have compiled with CGO_ENABLED=0 now and will monitor the situation.

fiber on Mar 22, 2018

The most common reason for a thread to be created is because all the existing threads are blocked in system calls or in calls to C code via cgo (the error message shows that your application uses cgo). cgo calls would be the first place to look. See if any of those calls do not return.

ianlancetaylor on Mar 22, 2018