gvisor: Memory leak in procfs in VFS2

Below is a simple way to reproduce this issue:

sh -c 'while true; do ls -l /proc; done'

After launching above command, we can see that the amount of memory taken by runsc-sandbox process keeps increasing (e.g. 1.5G after about 4min 29sec):

   PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 80839 root      20   0   64.0t   1.5g  23296 S  80.0  0.8   4:29.73 exe 

runsc version: release-20200907.0-38-gb6ca96b9b9e7

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Comments: 17 (6 by maintainers)

Commits related to this issue

Most upvoted comments

@ayushr2 Thanks for all the works! I really appreciate it. I tried this patch with below commands:

$ sh -c 'while true; do find /proc | xargs readlink; done' > /dev/null 2>&1 &
$ sh -c 'while true; do ls -l /proc; done' > /dev/null 2>&1 &

It looks runsc may crash sometimes, below is the backtrace I got:

I0930 11:29:21.533552   22826 main.go:207] ***************************
I0930 11:29:21.533584   22826 main.go:208] Args: [./runsc --platform=ptrace --debug-log=runsc.log --vfs2=true create test-container]
I0930 11:29:21.533602   22826 main.go:209] Version release-20200921.0-76-g5e246d0933e4
I0930 11:29:21.533614   22826 main.go:210] PID: 22826
I0930 11:29:21.533623   22826 main.go:211] UID: 0, GID: 0
I0930 11:29:21.533632   22826 main.go:212] Configuration:
I0930 11:29:21.533641   22826 main.go:213] 		RootDir: /var/run/runsc
I0930 11:29:21.533649   22826 main.go:214] 		Platform: ptrace
I0930 11:29:21.533658   22826 main.go:215] 		FileAccess: 0, overlay: false
I0930 11:29:21.533668   22826 main.go:216] 		Network: 0, logging: false
I0930 11:29:21.533678   22826 main.go:217] 		Strace: false, max size: 1024, syscalls: 
I0930 11:29:21.533687   22826 main.go:218] 		VFS2 enabled: true
I0930 11:29:21.533696   22826 main.go:219] ***************************
......
panic: Incrementing non-positive ref count 0xc00055f110 owned by *kernfs.Dentry

goroutine 3557 [running]:
panic(0xf197e0, 0xc0008da190)
	GOROOT/src/runtime/panic.go:1064 +0x46d fp=0xc00042b3f0 sp=0xc00042b338 pc=0x43438d
gvisor.dev/gvisor/pkg/sentry/fsimpl/kernfs.(*DentryRefs).IncRef(0xc00055f110)
	bazel-out/k8-fastbuild/bin/pkg/sentry/fsimpl/kernfs/dentry_refs.go:71 +0xcc fp=0xc00042b458 sp=0xc00042b3f0 pc=0x7a921c
gvisor.dev/gvisor/pkg/sentry/vfs.(*Dentry).IncRef(...)
	pkg/sentry/vfs/dentry.go:146
gvisor.dev/gvisor/pkg/sentry/vfs.(*FileDescription).Init(0xc000412a30, 0x12d8cc0, 0xc000412a20, 0xc000030800, 0xc00054e3c0, 0xc00055f118, 0xc00042b4dc, 0xc000226000, 0x10055f110)
	pkg/sentry/vfs/file_description.go:146 +0xd8 fp=0xc00042b490 sp=0xc00042b458 pc=0x6cef58
gvisor.dev/gvisor/pkg/sentry/fsimpl/kernfs.NewGenericDirectoryFD(0xc00054e3c0, 0xc00055f110, 0xc0003ddb20, 0xc0003ddb60, 0xc00042b578, 0x1, 0x3b00, 0x1, 0x1b5ec40)
	pkg/sentry/fsimpl/kernfs/fd_impl_util.go:90 +0xdc fp=0xc00042b4f8 sp=0xc00042b490 pc=0x7a9f3c
gvisor.dev/gvisor/pkg/sentry/fsimpl/kernfs.(*StaticDirectory).Open(0xc0003ddb00, 0x12cc8e0, 0xc0002ae000, 0xc000242900, 0xc00055f110, 0x30800, 0xc00054a1d8, 0x0, 0x0)
	pkg/sentry/fsimpl/kernfs/inode_impl_util.go:646 +0x6d fp=0xc00042b550 sp=0xc00042b4f8 pc=0x7b4a8d
gvisor.dev/gvisor/pkg/sentry/fsimpl/proc.(*taskOwnedInode).Open(0xc0002c1ba0, 0x12cc8e0, 0xc0002ae000, 0xc000242900, 0xc00055f110, 0x30800, 0x0, 0x0, 0x40)
	<autogenerated>:1 +0x93 fp=0xc00042b5a8 sp=0xc00042b550 pc=0xc09623
gvisor.dev/gvisor/pkg/sentry/fsimpl/kernfs.(*Filesystem).OpenAt(0xc00054a180, 0x12cc8e0, 0xc0002ae000, 0xc000242900, 0x30800, 0x0, 0x0, 0x0)
	pkg/sentry/fsimpl/kernfs/filesystem.go:429 +0x2d1 fp=0xc00042b7d0 sp=0xc00042b5a8 pc=0x7addc1
gvisor.dev/gvisor/pkg/sentry/vfs.(*VirtualFilesystem).OpenAt(0xc00019e188, 0x12cc8e0, 0xc0002ae000, 0xc00040ecc0, 0xc00042ba18, 0xc00042ba00, 0x0, 0xc00054e240, 0xc000590000)
	pkg/sentry/vfs/vfs.go:416 +0x159 fp=0xc00042b960 sp=0xc00042b7d0 pc=0x6e1be9
gvisor.dev/gvisor/pkg/sentry/syscalls/linux/vfs2.openat(0xc0002ae000, 0x300000009, 0x55ade7d8b368, 0xc000030900, 0x0, 0x0, 0x0, 0x0, 0x0)
	pkg/sentry/syscalls/linux/vfs2/filesystem.go:179 +0x265 fp=0xc00042bab0 sp=0xc00042b960 pc=0xc6dcb5
gvisor.dev/gvisor/pkg/sentry/syscalls/linux/vfs2.Openat(0xc0002ae000, 0x9, 0x55ade7d8b368, 0x30900, 0x0, 0x55ade7d87a50, 0x11, 0x0, 0x5b72, 0x40000002, ...)
	pkg/sentry/syscalls/linux/vfs2/filesystem.go:158 +0x55 fp=0xc00042bb08 sp=0xc00042bab0 pc=0xc6d965
gvisor.dev/gvisor/pkg/sentry/kernel.(*Task).executeSyscall(0xc0002ae000, 0x101, 0x9, 0x55ade7d8b368, 0x30900, 0x0, 0x55ade7d87a50, 0x11, 0x0, 0x10e5560, ...)
	pkg/sentry/kernel/task_syscall.go:104 +0x122 fp=0xc00042bc68 sp=0xc00042bb08 pc=0x8f45f2
gvisor.dev/gvisor/pkg/sentry/kernel.(*Task).doSyscallInvoke(0xc0002ae000, 0x101, 0x9, 0x55ade7d8b368, 0x30900, 0x0, 0x55ade7d87a50, 0x11, 0x55ade7d87a50, 0x11)
	pkg/sentry/kernel/task_syscall.go:239 +0x66 fp=0xc00042bcf0 sp=0xc00042bc68 pc=0x8f57d6
gvisor.dev/gvisor/pkg/sentry/kernel.(*Task).doSyscallEnter(0xc0002ae000, 0x101, 0x9, 0x55ade7d8b368, 0x30900, 0x0, 0x55ade7d87a50, 0x11, 0x129a301, 0x100000000000000)
	pkg/sentry/kernel/task_syscall.go:199 +0x96 fp=0xc00042bd50 sp=0xc00042bcf0 pc=0x8f5366
gvisor.dev/gvisor/pkg/sentry/kernel.(*Task).doSyscall(0xc0002ae000, 0x2, 0xc0002ae000)
	pkg/sentry/kernel/task_syscall.go:174 +0x15c fp=0xc00042be10 sp=0xc00042bd50 pc=0x8f4cfc
gvisor.dev/gvisor/pkg/sentry/kernel.(*runApp).execute(0x0, 0xc0002ae000, 0x129a340, 0x0)
	pkg/sentry/kernel/task_run.go:275 +0xc18 fp=0xc00042bf60 sp=0xc00042be10 pc=0x8e96b8
gvisor.dev/gvisor/pkg/sentry/kernel.(*Task).run(0xc0002ae000, 0xef)
	pkg/sentry/kernel/task_run.go:93 +0x18b fp=0xc00042bfd0 sp=0xc00042bf60 pc=0x8e83cb
runtime.goexit()
	src/runtime/asm_amd64.s:1373 +0x1 fp=0xc00042bfd8 sp=0xc00042bfd0 pc=0x467da1
created by gvisor.dev/gvisor/pkg/sentry/kernel.(*Task).Start
	pkg/sentry/kernel/task_start.go:318 +0xfe