tensorflow: tf.io.gfile / GCS fails to work on OpenSUSE
System information
- Have I written custom code (as opposed to using a stock example script provided in TensorFlow): Yes
- OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux OpenSUSE Tumbleweed
- TensorFlow installed from (source or binary): Binary (conda)
- TensorFlow version (use command below): unknown 2.1.0
- Python version: 3.7.5
- CUDA/cuDNN version: N/A
- GPU model and memory: N/A
Describe the current behavior
import tensorflow as tf
tf.io.gfile.listdir("gs://some-bucket") # replace w/ bucket of your choice
This code gives an error:
2020-06-01 15:43:56.684531: W tensorflow/core/platform/cloud/google_auth_provider.cc:178] All attempts to get a Google authentication bearer token failed, returning an empty token. Retrieving token from files failed with "Unavailable: Error executing an HTTP request: libcurl code 77 meaning 'Problem with the SSL CA cert (path? access rights?)', error details: error setting certificate verify locations:
CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: none". Retrieving token from GCE failed with "Aborted: All 10 retry attempts failed. The last failure: Unavailable: Error executing an HTTP request: libcurl code 6 meaning 'Couldn't resolve host name', error details: Couldn't resolve host 'metadata'".
After that it hangs for a while, and then raises a NotFoundError
.
I believe this is because the libcurl packaged with tensorflow doesn’t know where to find the ca-certificates bundle file on OpenSUSE, which is at /etc/ssl/ca-bundle.pem
rather than /etc/ssl/certs/ca-certificates.crt
. Also, I installed through miniconda so there’s another equivalent file at $CONDA_PREFIX/ssl/cacert.pem
. Neither of these seems to be found by tensorflow.
This code suggests that the bundle file’s location can be customized with the CURL_CA_BUNDLE
env variable. However, this doesn’t change the behavior as far as i can tell; the error is still raised.
Describe the expected behavior It should list the contents of the bucket.
About this issue
- Original URL
- State: open
- Created 4 years ago
- Reactions: 1
- Comments: 18 (6 by maintainers)
ping @jhseu … this is still an issue. Anyone?
fixing CURL certificates path as suggested in the code mentioned above seems to work for me for Tensorflow 2.4.1 and Fedora 28. added the environment variable in .bashrc :
I ran into this again on RHEL 7 with TF 2.2 and 2.3 and dug a bit:
Hence for TF to work the CA cert MUST be at
/etc/ssl/certs/ca-certificates.crt
and it is NOT possible to change that.The only solutions are to use the system curl (see TF_SYSTEM_LIBS, but that is partly broken), make TF use the CMake install of curl (but I doubt that is gonna happen) or implement the location detection in Bazel
Colab is not using OpenSUSE. There has been no PR here, so this is still valid.