go: sync: sync.Map keys will never be garbage collected

If you add a sm.Range(…) between sm.Store and sm.Delete, the memory leaks on go 1.14, too.

One bug at a time mate

Here is a simpler reproduction

(~/src) % env GODEBUG=gctrace=1 ./leak
gc 1 @0.015s 4%: 0.024+3.3+0.018 ms clock, 0.096+0.37/3.1/0.077+0.074 ms cpu, 5->5->3 MB, 6 MB goal, 4 P
gc 2 @0.032s 5%: 0.009+4.8+0.031 ms clock, 0.038+0.45/4.3/5.4+0.12 ms cpu, 8->8->6 MB, 9 MB goal, 4 P
gc 3 @0.069s 6%: 0.014+11+0.017 ms clock, 0.058+0.31/10/7.3+0.070 ms cpu, 16->17->12 MB, 17 MB goal, 4 P
gc 4 @0.152s 5%: 0.012+34+0.018 ms clock, 0.050+4.6/16/16+0.074 ms cpu, 33->34->25 MB, 34 MB goal, 4 P
gc 5 @0.317s 5%: 0.013+48+0.018 ms clock, 0.055+3.8/37/63+0.075 ms cpu, 67->67->50 MB, 68 MB goal, 4 P
gc 6 @0.652s 5%: 0.014+92+0.052 ms clock, 0.058+0.66/91/157+0.20 ms cpu, 135->135->100 MB, 136 MB goal, 4 P
gc 7 @1.322s 6%: 0.014+203+0.020 ms clock, 0.057+42/192/357+0.080 ms cpu, 269->270->200 MB, 270 MB goal, 4 P
gc 8 @2.721s 7%: 0.014+479+0.019 ms clock, 0.056+120/478/955+0.079 ms cpu, 539->541->400 MB, 540 MB goal, 4 P
gc 9 @5.619s 10%: 0.016+1205+0.026 ms clock, 0.064+759/1202/2360+0.10 ms cpu, 1079->1081->799 MB, 1080 MB goal, 4 P
^C
(~/src) % cat leak.go 
package main

import "sync"

func main() {
	var sm sync.Map

	var value [16]byte

	for i := 0; i < 1<<26; i++ {
		sm.Store(i, value)
		sm.Delete(i - 1)
	}
}

Keys in the read map cannot be purged immediately, but they are only retained until the cost to rebuild the map has been amortized away. That is an intentional tradeoff in the design.

But perhaps we are missing some sort of cache-miss tracking for deleted keys. If you have a real-world program for which that causes a problem, please open a new issue and we can at least see if there is a straightforward fix for it.

(But also bear in mind that sync.Map is intended to solve a specific pattern in the Go standard library. It is surely not optimal for all “concurrent map” use-cases, and if it’s not a good fit for your use-case it is totally fine to use something else instead. sync.Map should almost never appear in exported APIs, so in most cases it is easy to swap out for something else.)

@davecheney I’ve added a Debug method in sync.Map:

func (m *Map) Debug() {
	m.mu.Lock()
	println("sync.Map read:")
	for k := range m.read.Load().(readOnly).m {
		println(k)
	}
	println("sync.Map dirty:")
	for k := range m.dirty {
		println(k)
	}
	println("sync.Map debug end.")
	m.mu.Unlock()
}

And the example code to call Debug: https://play.golang.org/p/H3fkSdAw-JJ Output:

sync.Map read:
(0x1064920,0xc00000c060)
(0x1064920,0xc00000c0e0)
(0x1064920,0xc00000c140)
(0x1064920,0xc00000c100)
(0x1064920,0xc00000c120)
(0x1064920,0xc00000c160)
(0x1064920,0xc00000c040)
(0x1064920,0xc00000c080)
(0x1064920,0xc00000c0a0)
(0x1064920,0xc00000c0c0)
sync.Map dirty:
sync.Map debug end.

As you can see, the keys are not deleted.

@gopherbot, please backport to 1.15. This is a regression from Go 1.14.7, and can cause difficult-to-predict memory leaks in existing code.

Here is a simpler reproduction

(~/src) % env GODEBUG=gctrace=1 ./leak
gc 1 @0.015s 4%: 0.024+3.3+0.018 ms clock, 0.096+0.37/3.1/0.077+0.074 ms cpu, 5->5->3 MB, 6 MB goal, 4 P
gc 2 @0.032s 5%: 0.009+4.8+0.031 ms clock, 0.038+0.45/4.3/5.4+0.12 ms cpu, 8->8->6 MB, 9 MB goal, 4 P
gc 3 @0.069s 6%: 0.014+11+0.017 ms clock, 0.058+0.31/10/7.3+0.070 ms cpu, 16->17->12 MB, 17 MB goal, 4 P
gc 4 @0.152s 5%: 0.012+34+0.018 ms clock, 0.050+4.6/16/16+0.074 ms cpu, 33->34->25 MB, 34 MB goal, 4 P
gc 5 @0.317s 5%: 0.013+48+0.018 ms clock, 0.055+3.8/37/63+0.075 ms cpu, 67->67->50 MB, 68 MB goal, 4 P
gc 6 @0.652s 5%: 0.014+92+0.052 ms clock, 0.058+0.66/91/157+0.20 ms cpu, 135->135->100 MB, 136 MB goal, 4 P
gc 7 @1.322s 6%: 0.014+203+0.020 ms clock, 0.057+42/192/357+0.080 ms cpu, 269->270->200 MB, 270 MB goal, 4 P
gc 8 @2.721s 7%: 0.014+479+0.019 ms clock, 0.056+120/478/955+0.079 ms cpu, 539->541->400 MB, 540 MB goal, 4 P
gc 9 @5.619s 10%: 0.016+1205+0.026 ms clock, 0.064+759/1202/2360+0.10 ms cpu, 1079->1081->799 MB, 1080 MB goal, 4 P
^C
(~/src) % cat leak.go 
package main

import "sync"

func main() {
	var sm sync.Map

	var value [16]byte

	for i := 0; i < 1<<26; i++ {
		sm.Store(i, value)
		sm.Delete(i - 1)
	}
}

If you add a sm.Range(…) between sm.Store and sm.Delete, the memory leaks on go 1.14, too.

@iand Sure, example code: https://play.golang.org/p/lHKYvukGbpZ Also I added the Debug method above in both go 1.15 and 1.14. On go 1.15, output:

sync.Map read:
sync.Map dirty:
(0x1064560,0xc00000c080)
(0x1064560,0xc00000c0a0)
(0x1064560,0xc00000c0c0)
(0x1064560,0xc00000c100)
(0x1064560,0xc00000c120)
(0x1064560,0xc00000c140)
(0x1064560,0xc00000c160)
(0x1064560,0xc00000c040)
(0x1064560,0xc00000c0e0)
(0x1064560,0xc00000c060)
sync.Map debug end.

On go 1.14, output:

sync.Map read:
sync.Map dirty:
(0x10603e0,0xc00000c100)
(0x10603e0,0xc00000c0c0)
(0x10603e0,0xc00000c140)
(0x10603e0,0xc00000c040)
(0x10603e0,0xc00000c080)
sync.Map debug end.

You can see that, if there’s no read operation on map, the keys in dirty part can be deleted.