cml: cml workflow with gpu fails with LD_LIBRARY_PATH error

Thank you for your measured response @0x2b3bfa0, that was quite a confusing and unhelpful message!

Ah - my mistake, actually what I did was the following, actually the reverse of what is mentioned in that comment (I missed this). https://stackoverflow.com/a/67642774

https://github.com/tensorflow/tensorflow/issues/45848#issuecomment-829557887

This appears to work, but still uncertain why the error exists at all

Perhaps relevant, but why is tensorflow 2.5.0 asking for libcusolver.so.11 if it doesn’t exist?

https://stackoverflow.com/questions/63199164/how-to-install-libcusolver-so-11

As noted in comments there is no version 11.0 of cuSolver in the CUDA 11.0 release

🤔 Can you please try running sudo find / -name 'libcusolver.*' 2>/dev/null on your train step?

Apologies for delay, tf 2.2.0 works with base0 image(!), but tf 2.5.0 fails with the following:

poetry run python -c "import tensorflow as tf; print('Num GPUs Available: ', len(tf.config.list_physical_devices('GPU')))" || true
  nvidia-smi || true  # run this to see full gpu diagnostics if one exists
  shell: sh -e {0}
2021-07-12 16:39:41.664649: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0
2021-07-12 16:39:42.759446: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcuda.so.1
2021-07-12 16:39:44.101362: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-07-12 16:39:44.102174: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1733] Found device 0 with properties: 
pciBusID: 0000:00:1e.0 name: Tesla T4 computeCapability: 7.5
coreClock: 1.59GHz coreCount: 40 deviceMemorySize: 14.75GiB deviceMemoryBandwidth: 298.08GiB/s
2021-07-12 16:39:44.102265: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0
2021-07-12 16:39:44.105130: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcublas.so.11
2021-07-12 16:39:44.105210: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcublasLt.so.11
2021-07-12 16:39:44.106387: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcufft.so.10
2021-07-12 16:39:44.106740: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcurand.so.10
2021-07-12 16:39:44.107206: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcusolver.so.11'; dlerror: libcusolver.so.11: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/nvidia/lib:/usr/local/nvidia/lib64
2021-07-12 16:39:44.108043: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcusparse.so.11
2021-07-12 16:39:44.108241: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudnn.so.8
2021-07-12 16:39:44.108267: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1766] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
Num GPUs Available:  0

There is actually, the above is for 2.2.0 and the 2.5.0 had LD_LIBRARY_PATH errors of a slightly different kind, let me find it

@btjones-me, can you please try using docker://dvcorg/cml:0-dvc2-base0-gpu instead of docker://dvcorg/cml:0-dvc2-base1-gpu for your train job? As per the documentation, we provide base0 images with CUDA 10 and CuDNN 7 for compatibility with Tensorflow versions lower than 2.4.