tensorflow: tf.config.LogicalDeviceConfiguration memory_limit does not really work ?

System information

• Custom code
• Linux Ubuntu 20.04 (tested also 18.04)
• Computer (not mobile device)
• TensorFlow installed from pip install:
• TensorFlow version 2.7 (tested also 2.4,2.5,2.6,2.8)
• Python version: 3.7 (tested also 3.8)
• Bazel version (if compiling from source): unbuntu 20.04 no self compil
• GCC/Compiler version (if compiling from source):
• CUDA/cuDNN version: 11.2.2/8.2.0.1
• GPU model and memory: NVIDIA RTX3090 24GB (tested also RTX2060 anX GTX1660) Driver 460.x

Hello I successfully ran the profiler tool on ma classification model to profile the maximum memory usage. Because I want to use different CNN on a same GPU. But I’m really baffled by the results of the profiler. Let me explain

I have a NVIDIA RTX3090 with 24GB memory so for my small CNN I set 512 memory limit in my code before all use with this code :
tf.config.set_logical_device_configuration(gpus[0],[tf.config.LogicalDeviceConfiguration(memory_limit=512)])

It seems to work because of the tensorflow logs 2022-01-19 16:24:13.615890: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1525] Created device /job:localhost/replica:0/task:0/device:GPU:0 with **512 MB memory:** -> device: 0, name: GeForce RTX 3090, pci bus id: 0000:2d:00.0, compute capability: 8.6

Nvidia-smi shows that GPU use 419MiB are used

Then I start a batch to make the inference on the classification model with batch size = 1 and tensorboard shows that the model use about 100MiB

so theoretically I could have set a small memory limit (under 512) but … here is the real use of the memory given by Nvidia-smi is 1869MiB !

I tested the code in Tensorflow 2.4,2.5,2.6,2.7 and 2.8. With different CUDA CUDNN but it is the same. Memory limit seems to be applied at Tensorflow level but not at the real GPU memory ( Nvidia level ). Did I miss something ? It would be very usefull to be able to manage the memory of a model !