openvino: [Good First Issue][GPU]: Cannot load model when cache directory is running out of disk space

Context

GPU plugin utilizes cache files when loading models. When the cache is full, loading another model fails with:

~/pgladkow/openvino_repro/build$ ./test_ov
terminate called after throwing an instance of 'ov::Exception'
  what():  cldnn program build failed! clCreateProgramWithBinary
Aborted 

This task regards solving that issue.

What needs to be done?

The cache should be cleared when not needed. Details should be discussed with people from Contact Points section.

Steps to reproduce:

1. Get ov.cpp (attachment)
2. Get CMakeLists.txt (attachment)
3. Mount cache directory to have a limited disk space (10m)

sudo mount -t tmpfs -o size=10m tmpfs tmp_cache

1. Run OpenVINO app

~/pgladkow/openvino_repro/build$ cmake ../ovms
-- Configuring done
-- Generating done
-- Build files have been written to: /opt/home/k8sworker/pgladkow/openvino_repro/build
~/pgladkow/openvino_repro/build$ cmake --build .
[100%] Built target test_ov
~/pgladkow/openvino_repro/build$ ./test_ov
Hello 

1. Verify that cache directory is full

~/pgladkow/openvino_repro$ ls tmp_cache/

1. Run OV app once again - it should fail this time

~/pgladkow/openvino_repro/build$ ./test_ov
terminate called after throwing an instance of 'ov::Exception'
  what():  cldnn program build failed! clCreateProgramWithBinary
Aborted 

Directory structure from the original reproducer:

~/pgladkow/openvino_repro$ tree
.
├── build
│   ├── CMakeCache.txt
│   ├── CMakeFiles
│   │   ├── 3.16.3
│   │   │   ├── CMakeCCompiler.cmake
│   │   │   ├── CMakeCXXCompiler.cmake
│   │   │   ├── CMakeDetermineCompilerABI_C.bin
│   │   │   ├── CMakeDetermineCompilerABI_CXX.bin
│   │   │   ├── CMakeSystem.cmake
│   │   │   ├── CompilerIdC
│   │   │   │   ├── a.out
│   │   │   │   ├── CMakeCCompilerId.c
│   │   │   │   └── tmp
│   │   │   └── CompilerIdCXX
│   │   │       ├── a.out
│   │   │       ├── CMakeCXXCompilerId.cpp
│   │   │       └── tmp
│   │   ├── cmake.check_cache
│   │   ├── CMakeDirectoryInformation.cmake
│   │   ├── CMakeError.log
│   │   ├── CMakeOutput.log
│   │   ├── CMakeTmp
│   │   ├── Makefile2
│   │   ├── Makefile.cmake
│   │   ├── progress.marks
│   │   ├── TargetDirectories.txt
│   │   └── test_ov.dir
│   │       ├── build.make
│   │       ├── cmake_clean.cmake
│   │       ├── CXX.includecache
│   │       ├── DependInfo.cmake
│   │       ├── depend.internal
│   │       ├── depend.make
│   │       ├── flags.make
│   │       ├── link.txt
│   │       ├── progress.make
│   │       └── src
│   │           └── ov.cpp.o
│   ├── cmake_install.cmake
│   ├── Makefile
│   └── test_ov
├── ovms
│   ├── CMakeLists.txt
│   └── src
│       ├── brain-tumor-segmentation-0002-2
│       │   └── 1
│       │       ├── brain-tumor-segmentation-0002.onnx
│       │       └── description.txt
│       ├── dummy
│       │   └── 1
│       │       ├── dummy.bin
│       │       └── dummy.xml
│       ├── inception-resnet-v2-tf
│       │   └── 1
│       │       ├── inception-resnet-v2-tf.bin
│       │       ├── inception-resnet-v2-tf.mapping
│       │       └── inception-resnet-v2-tf.xml
│       ├── ov.cpp
│       └── ssdlite_mobilenet_v2_ov
│           └── 1
│               ├── ssdlite_mobilenet_v2_ov.bin
│               ├── ssdlite_mobilenet_v2_ov.mapping
│               └── ssdlite_mobilenet_v2_ov.xml
└── tmp_cache 

reproducer.zip

Resources

• What is OpenVINO?
• Contribution guide
• Blog post on contributing to OpenVINO
• User documentation

Contact points

@pgladkows @vladimir-paramuzov

Ticket: 104958