Hello everyone !

Since we switch our prediction platform to Tensorflow we also experienced a memory leak.

After finding that out, we used go tool pprof in order to found where leaks where happening but it was useless. Impossible to find the leak using pprof. In fact, pprof’s inuse_space sum output was completely different from top’s. Everything seemed to be fine on pprof but top was showing a constant rise in memory usage.

We tried to figured out if our code could be the issue and it is not. No leak at all without any call to TF and still nothing of a memory leak shown by pprof.

We tried to abstract as much as possible and finished with this piece of code.

package main

import (
	"fmt"
	tf "github.com/tensorflow/tensorflow/tensorflow/go"
	"runtime"
	"time"
)

func printStats(mem runtime.MemStats) {

	runtime.ReadMemStats(&mem)

	fmt.Println("mem.Alloc:", mem.Alloc)

	fmt.Println("mem.TotalAlloc:", mem.TotalAlloc)

	fmt.Println("mem.HeapAlloc:", mem.HeapAlloc)

	fmt.Println("mem.NumGC:", mem.NumGC)

	fmt.Println("-----")

}

func loadModel(modelPath string, tags []string) {
	model, err := tf.LoadSavedModel(modelPath, tags, nil)
	if err != nil {
		fmt.Println("error OPENING model")
		return
	}
	err = model.Session.Close()
	if err != nil {
		fmt.Println("error CLOSING model")
		return
	}
}

func main() {
	var mem runtime.MemStats

	printStats(mem)
	tags := []string{"serve"}
	for i := 0; i < 300; i++ {
		loadModel("./path_to_model/", tags)
		fmt.Println("loading model: => ", i)
		printStats(mem)
	}
	fmt.Println("done")
	printStats(mem)
	fmt.Println("call to garbage collector")
	runtime.GC()
	printStats(mem)
	fmt.Println("sleeping 10 minutes")
	time.Sleep(time.Minute * 10)
	fmt.Println("complete done")
}

We are basically loading a model 300 times, closing them right after and then calling the Garbage Collector at the end of the process to force the memory pointed by useless pointers to be freed. However, whether it be with TF 1.15.2 or 2.0.1 we are experiencing something weird. The models are, as weird as it is, being freed some time to times. I let you run this program several times and see what top says on the process’s memory.

We are expecting it to decrease as soon as the GC is called. On both 1.15.2 & 2.0.1, sometimes it decreases, sometimes it does not and we think it is linked with some useless pointers still used by TF somewhere, preventing the GC to free them. Note that with our recents tests we can point that this issue seems to happens more often with TF 2.0.1 than with 1.15.2.

In our case and with our model, the program start at around 100-200Mo, reach 1.5Go at the end of the 300 models loading and then when the GC is called, the inuse memory shown by top drops back to 150Mo or sometimes doesn’t and stays at 1.5Go.

All of this lead us to believe that the issue is from the C++ code OR from the CGo bindings of Tensorflow. We couldn’t figured out anything else. An investigation would be appreciated 🚀

Execute the following code, a memory leak will occur

for i := 0; i < 100; i++ {
	tf.LoadSavedModel("/model/fm_model_v1", []string{"serve"}, nil)
	time.sleep(10 * time.Second)
}

I have the same issue, but I use tf.LoadSavedModel, I tried assign model to nil, defer session.close(), golang garbage collection, none of them works, go pprof or cmd top both show the same result, loadsavedmodel did not release memory.

in my case, i had to choose one of hundreds model file and load it to inference, I know there’s a mechanism stop my session.close or model release due to I’m sending request frequently

is there anything i can do? I choose golang and try prevent from python memory leak nightmare, now I put myself into another one, sight

Hi There,

We are checking to see if you still need help on this, as you are using an older version of tensorflow which is officially considered end of life . We recommend that you upgrade to the latest 2.x version and let us know if the issue still persists in newer versions. Please open a new issue for any help you need against 2.x, and we will get you the right help.

This issue will be closed automatically 7 days from now. If you still need help with this issue, please provide us with more information.

It seems that tensorflow has no limit on cpu utilization by default，memory usage will increase until OOM if you don’t add limit. Try add tf_config = tf.ConfigProto(inter_op_parallelism_threads=1,intra_op_parallelism_threads=1)) sess=tf.Session(config=tf_config) and it works for me.

I have narrow it down to NewTensor also creating memory leak. It seems the golang garbage collector does not clear the whole unused Tensor.