io: Unable to load libtensorflow_io_plugins.so for aarch64 tensorflow-io-gcs-filesystem wheels

Description

The aarch64 architecture wheels for tensorflow-io-gcs-filesystem are missing the libtensorflow_io_plugins.so file. Here is a reproducible demonstration of this problem for aarch64 but not for amd64 using Docker for tensorflow-io-gcs-filesystem v0.29.0 installed through tensorflow-aarch64. However it is noted that this library is not used in the passing example for the amd64 wheel.

Minimal failing example for `arm64`

$ docker run --rm -ti --platform linux/arm64 python:3.9 /bin/bash
root@38df62ecc9c2:/# python -m venv venv && . venv/bin/activate
(venv) root@38df62ecc9c2:/# python -m pip --quiet install --upgrade pip setuptools wheel
(venv) root@38df62ecc9c2:/# python -m pip --quiet install 'tensorflow-aarch64==2.11.0'
(venv) root@38df62ecc9c2:/# python -m pip list | grep tensor
tensorboard                  2.11.0
tensorboard-data-server      0.6.1
tensorboard-plugin-wit       1.8.1
tensorflow-aarch64           2.11.0
tensorflow-estimator         2.11.0
tensorflow-io-gcs-filesystem 0.29.0
(venv) root@38df62ecc9c2:/# python -c 'import tensorflow_io'
/venv/lib/python3.9/site-packages/tensorflow_io/python/ops/__init__.py:98: UserWarning: unable to load libtensorflow_io_plugins.so: unable to open file: libtensorflow_io_plugins.so, from paths: ['/venv/lib/python3.9/site-packages/tensorflow_io/python/ops/libtensorflow_io_plugins.so']
caused by: ["[Errno 2] The file to load file system plugin from does not exist.: '/venv/lib/python3.9/site-packages/tensorflow_io/python/ops/libtensorflow_io_plugins.so'"]
  warnings.warn(f"unable to load libtensorflow_io_plugins.so: {e}")
/venv/lib/python3.9/site-packages/tensorflow_io/python/ops/__init__.py:104: UserWarning: file system plugins are not loaded: unable to open file: libtensorflow_io.so, from paths: ['/venv/lib/python3.9/site-packages/tensorflow_io/python/ops/libtensorflow_io.so']
caused by: ['/venv/lib/python3.9/site-packages/tensorflow_io/python/ops/libtensorflow_io.so: cannot open shared object file: No such file or directory']
  warnings.warn(f"file system plugins are not loaded: {e}")
(venv) root@38df62ecc9c2:/# python -c 'import tensorflow'  # for consistency with passing example
/venv/lib/python3.9/site-packages/tensorflow_io/python/ops/__init__.py:98: UserWarning: unable to load libtensorflow_io_plugins.so: unable to open file: libtensorflow_io_plugins.so, from paths: ['/venv/lib/python3.9/site-packages/tensorflow_io/python/ops/libtensorflow_io_plugins.so']
caused by: ["[Errno 2] The file to load file system plugin from does not exist.: '/venv/lib/python3.9/site-packages/tensorflow_io/python/ops/libtensorflow_io_plugins.so'"]
  warnings.warn(f"unable to load libtensorflow_io_plugins.so: {e}")
/venv/lib/python3.9/site-packages/tensorflow_io/python/ops/__init__.py:104: UserWarning: file system plugins are not loaded: unable to open file: libtensorflow_io.so, from paths: ['/venv/lib/python3.9/site-packages/tensorflow_io/python/ops/libtensorflow_io.so']
caused by: ['/venv/lib/python3.9/site-packages/tensorflow_io/python/ops/libtensorflow_io.so: cannot open shared object file: No such file or directory']
  warnings.warn(f"file system plugins are not loaded: {e}")
(venv) root@38df62ecc9c2:/# find /venv/ -type f -iname "libtensorflow_io_plugins.so"
(venv) root@38df62ecc9c2:/#

Minimal working example for `amd64`

$ docker run --rm -ti --platform linux/amd64 python:3.9 /bin/bash
root@41f993536792:/# python -m venv venv && . venv/bin/activate
(venv) root@41f993536792:/# python -m pip --quiet install --upgrade pip setuptools wheel
(venv) root@41f993536792:/# python -m pip --quiet install 'tensorflow-cpu==2.11.0'
(venv) root@41f993536792:/# python -m pip list | grep tensor
tensorboard                  2.11.0
tensorboard-data-server      0.6.1
tensorboard-plugin-wit       1.8.1
tensorflow-cpu               2.11.0
tensorflow-estimator         2.11.0
tensorflow-io-gcs-filesystem 0.29.0
(venv) root@41f993536792:/# python -c 'import tensorflow'
2023-01-03 19:55:09.854903: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F AVX512_VNNI FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-01-03 19:55:09.922735: I tensorflow/core/util/port.cc:104] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
(venv) root@41f993536792:/# find /venv/ -type f -iname "libtensorflow_io_plugins.so"
(venv) root@41f993536792:/# find /venv/ -type f -iname "libtensorflow_io*.so"
/venv/lib/python3.9/site-packages/tensorflow_io_gcs_filesystem/core/python/ops/libtensorflow_io_gcs_filesystem.so
(venv) root@41f993536792:/#

Expectation

For the aarch64 wheels to be able to be used without the warning

/venv/lib/python3.9/site-packages/tensorflow_io/python/ops/__init__.py:98: UserWarning: unable to load libtensorflow_io_plugins.so: unable to open file: libtensorflow_io_plugins.so, from paths: ['/venv/lib/python3.9/site-packages/tensorflow_io/python/ops/libtensorflow_io_plugins.so']
caused by: ["[Errno 2] The file to load file system plugin from does not exist.: '/venv/lib/python3.9/site-packages/tensorflow_io/python/ops/libtensorflow_io_plugins.so'"]
  warnings.warn(f"unable to load libtensorflow_io_plugins.so: {e}")
/venv/lib/python3.9/site-packages/tensorflow_io/python/ops/__init__.py:104: UserWarning: file system plugins are not loaded: unable to open file: libtensorflow_io.so, from paths: ['/venv/lib/python3.9/site-packages/tensorflow_io/python/ops/libtensorflow_io.so']
caused by: ['/venv/lib/python3.9/site-packages/tensorflow_io/python/ops/libtensorflow_io.so: cannot open shared object file: No such file or directory']
  warnings.warn(f"file system plugins are not loaded: {e}")

Related Issues

(I’m creating this one as the others don’t have easily reproducible examples)

Issue #1532
Issue #1643
Issue #1689
Issue #1735
Issue #1754

About this issue

Original URL
State: closed
Created a year ago
Reactions: 6
Comments: 26 (5 by maintainers)

Most upvoted comments

@matthewfeickert There was a bug in the packaging of tensorflow-io-gcs-filesystem for AARCH64 that led to parts of tensorflow-io being incorrectly included. This has been corrected in the build scripts and will resolve the issue in the next release. In the meantime as a workaround you can

python -m pip install tensorflow-io 
python -m pip uninstall -y tensorflow-io

Installing and then removing tensorflow-io will clean up the install of tensorflow-io-gcs-filesystem by removing the incorrectly included files and leave you with a working system. I am sorry for the inconvenience.

elfringham on Feb 9, 2023

Im having the same issue as it cant find libtensorflow_io_plugins.so. Im currently using M1 mac.

guoz23 on Jan 29, 2023

Thanks very much @elfringham for the summary! That’s quite useful to understand and I appreciate you taking the time to explain it. Thanks for all your work!

matthewfeickert on Feb 9, 2023

@trueToastedCode I believe that the M1 support is still a WIP. But that is a completely separate issue to this one.

elfringham on Jul 28, 2023

@trueToastedCode It was closed because this works.

$ docker run --rm -ti --platform linux/arm64 python:3.9 /bin/bash
root@9dd720c550b9:/# python -m venv venv && . venv/bin/activate
(venv) root@9dd720c550b9:/# python -m pip --quiet install --upgrade pip setuptools wheel
(venv) root@9dd720c550b9:/# python -m pip --quiet install 'tensorflow-aarch64==2.12.0'
(venv) root@9dd720c550b9:/# python -m pip list | grep tensor
tensorboard                  2.12.3
tensorboard-data-server      0.7.1
tensorflow-aarch64           2.12.0
tensorflow-estimator         2.12.0
tensorflow-io-gcs-filesystem 0.32.0
(venv) root@9dd720c550b9:/# python -c 'import tensorflow'
(venv) root@9dd720c550b9:/# python -m pip --quiet install 'tensorflow-io'
(venv) root@9dd720c550b9:/# python -c 'import tensorflow_io'
(venv) root@9dd720c550b9:/#

elfringham on Jul 25, 2023

Yeah, using the extra workflow works well

$ docker run --privileged --rm tonistiigi/binfmt --install arm64  # I'm on an amd64 machine
$ docker run --rm -ti --platform linux/arm64 python:3.10 /bin/bash
root@61821132ad7d:/# python -m venv venv && . venv/bin/activate
(venv) root@61821132ad7d:/# python -m pip --quiet install --upgrade pip setuptools wheel
(venv) root@61821132ad7d:/# python -m pip --quiet install --upgrade 'tensorflow-io[tensorflow]==0.31.0'
(venv) root@61821132ad7d:/# python -c 'import tensorflow_io; print(tensorflow_io)'
<module 'tensorflow_io' from '/venv/lib/python3.10/site-packages/tensorflow_io/__init__.py'>
(venv) root@61821132ad7d:/# find /venv/ -type f -iname "libtensorflow_io_plugins.so"
/venv/lib/python3.10/site-packages/tensorflow_io/python/ops/libtensorflow_io_plugins.so
(venv) root@61821132ad7d:/#

so I’m going to close this as I think this is sufficiently resolved. 👍

matthewfeickert on Mar 14, 2023

I am working on x86 , windows 11. Python 3.11 tensorflow-io=0.31.0

The error:

NotImplementedError: unable to open file: libtensorflow_io.so, from paths: [‘C:\Users\bchai\PycharmProjects\AudioML\venv\Lib\site-packages\tensorflow_io\python\ops\libtensorflow_io.so’] caused by: [‘C:\Users\bchai\PycharmProjects\AudioML\venv\Lib\site-packages\tensorflow_io\python\ops\libtensorflow_io.so not found’]

The file does exist at this location. I have tried downgrading tensorflow-io to 0.30.0 but that did not help either

BARKHAC on May 20, 2023

.so’ (no such file) still exists， I am is arm64 linux, I can build the .so file from source code?Thanks

wendaovswendao on Mar 5, 2023

@elfringham Thanks for the pointer. Mind elaborating more about the issue and the fix? I’m having a similar issue with centos7 where the libtensorflow_io_plugins.so contains no symbol except a few which makes tf throwing a warning not being able to find a TF_InitPlugins symbol.

MrAta on Feb 20, 2023

Reinstall tensorflow-io can help:

pip3 install --force-reinstall tensorflow-io

thanhle7 on Feb 9, 2023