avalanchego: 4GB memory not enough / crasher

Describe the bug I set up a dedicated AWS EC2 linux instance with 4GB of RAM (listed in requirements) w/o swap to run an avalanchego (avalanche/1.1.5) validator. It bootstrapped pretty quickly and then crashed a number of hours later. I lost the scrollback in my terminal, and there wasn’t anything interesting in the log.

I ran it again and watched it over the last 24 hours, and piped stderr/stdout to a file. I noticed:

1. memory usage slowly climbed until 4GB:
2. it then crashed.

To Reproduce

./avalanchego --log-dir ./log/ --api-keystore-enabled=false --api-health-enabled=true --api-info-enabled=true --api-keystore-enabled=false --public-ip [IP I USED] &>> ./log/output.log

Expected behavior server running fine on a machine with 4GB of RAM per avalanche validator docs

Screenshots

An hour or so before it crashed:

My friend says his validator is also using more than 4GB of RAM, but his machine can handle it:

Operating System

NAME="Amazon Linux"
VERSION="2"
ID="amzn"
ID_LIKE="centos rhel fedora"
VERSION_ID="2"
PRETTY_NAME="Amazon Linux 2"
ANSI_COLOR="0;33"
CPE_NAME="cpe:2.3:o:amazon:amazon_linux:2"
HOME_URL="https://amazonlinux.com/"

Additional context

I saw a previously closed issue (https://github.com/ava-labs/avalanchego/issues/717) from a user who ran an older version of avalanchego and then upgraded, maybe causing some DB issues. To be clear, this is the only version I’ve run.

from outputting to a log:

INFO [02-10|15:15:27] <C Chain> snow/engine/snowman/transitive.go#116: bootstrapping finished with VEtPRUjubZYc5gu4cxc68r8BYSQs2Z2Pnin9iaxdcqFEcNZXX as the last accepted block
INFO [02-11|20:43:44] api/info/service.go#182: Info: IsBootstrapped called with chain: X
INFO [02-11|20:45:14] api/info/service.go#182: Info: IsBootstrapped called with chain: X
runtime/cgo: pthread_create failed: Resource temporarily unavailable
SIGABRT: abort
PC=0x7f9f979bbb20 m=8 sigcode=18446744073709551610

goroutine 0 [idle]:
runtime: unknown pc 0x7f9f979bbb20
stack: frame={sp:0x7f9f68dfa7b0, fp:0x0} stack=[0x7f9f685fb288,0x7f9f68dfae88)
00007f9f68dfa6b0:  0000000000000000  0000000000000000 
00007f9f68dfa6c0:  0000000000000000  0000000000000000 
00007f9f68dfa6d0:  0000000000000000  0000000001b8ce80 
00007f9f68dfa6e0:  0000000000000000  0000000000000000 

... and then about 40,000 more lines of go stack

By submitting this issue I agree to the Terms and Conditions of the Developer Accelerator Program.