io: S3 filesystem pure virtual method called; terminate called without an active exception
I am getting a core dump during interpreter teardown, when using the s3 filesystem. Can I please be given guidance how to handle this issue. Please see script to reproduce inside docker:
FROM tensorflow/tensorflow:2.14.0-gpu
The following environment variables are set
"AWS_ACCESS_KEY_ID": xxx,
"AWS_SECRET_ACCESS_KEY": xxx,
"AWS_ENDPOINT_URL_S3": xxx,
"AWS_REGION": "us-east-1",
"S3_USE_HTTPS": "1",
"S3_VERIFY_SSL": "1",
"S3_DISABLE_MULTI_PART_DOWNLOAD": "0",
"S3_ENDPOINT": xxx,
import os
import tensorflow as tf
import tensorflow_io as tfio
def illustrate_core_dump():
print(f"tf version: {tf.__version__}")
print(f"tfio version: {tfio.__version__}")
filename = f"{os.environ['CLOUD_MOUNT']}/tmp/test_tfrecord.tfrecord"
assert filename.startswith("s3://"), "problem appears to be be for s3 filesystem only"
ds = tf.data.TFRecordDataset(filename, "GZIP")
for i in ds:
print(f"i.shape: {i.shape}")
if __name__ == "__main__":
illustrate_core_dump()
print("reaches here successfully")
print("something broken during destruction and tf")
# during interpreter teardown if s3 filesystem used we will get
# pure virtual method called
# terminate called without an active exception
# Aborted (core dumped)
# gs:// and file:// do not exhibit this issue which don't rely on tfio
TF_CPP_MIN_LOG_LEVEL=0 python notebooks/illustrate_core_dump.py
2024-01-01 18:07:11.253238: E tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:9342] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2024-01-01 18:07:11.253287: E tensorflow/compiler/xla/stream_executor/cuda/cuda_fft.cc:609] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2024-01-01 18:07:11.253323: E tensorflow/compiler/xla/stream_executor/cuda/cuda_blas.cc:1518] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2024-01-01 18:07:11.262384: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
tf version: 2.14.0
tfio version: 0.35.0
2024-01-01 18:07:14.402239: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:894] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2024-01-01 18:07:14.413303: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:894] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2024-01-01 18:07:14.416545: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:894] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2024-01-01 18:07:14.421598: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:894] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2024-01-01 18:07:14.423868: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:894] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2024-01-01 18:07:14.426098: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:894] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2024-01-01 18:07:15.494277: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:894] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2024-01-01 18:07:15.496519: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:894] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2024-01-01 18:07:15.498484: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:894] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2024-01-01 18:07:15.500342: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1886] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 13589 MB memory: -> device: 0, name: Tesla T4, pci bus id: 0000:00:04.0, compute capability: 7.5
i.shape: ()
reaches here successfully
something broken during destruction and tf
pure virtual method called
terminate called without an active exception
Aborted (core dumped)
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Reactions: 1
- Comments: 24 (4 by maintainers)
Commits related to this issue
- Update build.yml Bump Ubuntu version for Linux Wheel to address issue #1912 https://github.com/tensorflow/io/issues/1912 — committed to yarri-oss/io by yarri-oss 2 years ago
- Update build.yml (#2005) Bump Ubuntu version for Linux Wheel to address issue #1912 https://github.com/tensorflow/io/issues/1912 — committed to tensorflow/io by yarri-oss 2 years ago
- Update build.yml (#2005) Bump Ubuntu version for Linux Wheel to address issue #1912 https://github.com/tensorflow/io/issues/1912 — committed to teskobif7/io by teskobif7 2 years ago
We are still having this issue with tensorflow-io 0.37.1, please help to reopen this issue. @yarri-oss
I am still seeing
on
0.37.1. anyone else?This problem persists in
tensorflow-io==0.37.0Please fix, this is rendering s3 based io unusable without resorting to old versionstensorflow-io==0.34.0 # workstensorflow-io==0.35.0 # crashingCan we please verify why the latest so exhibiting this issue. Thank you
@rivershah I pulled the
ubuntu:22.04image from dockerhuband installed all the packages and bazel as instructed in https://github.com/tensorflow/io/blob/master/docs/development.md#ubuntu-2204 (without the
sudo)I then followed the instructions at https://github.com/tensorflow/io/blob/master/docs/development.md#python-wheels:
Then, within the same container, I was able to validate tf-io’s S3 filesystem functionality by trying to checkpoint a model to S3.
I’ll need to do some additional work to reproduce the failure I got when copying the generated tf-io wheel out into a different container, since I’ve terminated all of that setup now.
With tensorflow-cpu fixed at 2.14.0, I can reproduce the issue by changing tensorflow-io from 0.34.0 to 0.35.0.
This is my script:
If I run
pip install tensorflow-io==0.34.0 tensorflow-cpu==2.14.0 && python main.pyAnd the process exited immediately with status 0.
If I run
pip install tensorflow-io==0.35.0 tensorflow-cpu==2.14.0 && python main.pyand the process exited with status 134.
So, I went through every single commit between 0.34.0 and 0.35.0. This was the only commit that seems to touch s3: https://github.com/tensorflow/io/pull/1857/files.
Does anybody know how to build TensorFlow myself so I can see if reverting the commit fixes the issue?
cc @yongtang
update: I found the doc! https://github.com/tensorflow/io/blob/master/docs/development.md
Also still seeing the same issue after upgrading to v0.37.1
Which users? I posted the issue and it repros as per above
End users have confirmed this issue fixed.
@spolloni If you can post a repro (with S3 bucket blob) we can investigate further. I would prefer a new issue be opened against your specific repro tho.
+1, also running into this issue
This is blocking us from upgrading the tensorstore version. A quick fix will be much appreciated!
As an update, I followed the build instructions for tensorflow-io (Ubuntu 22.04 and then Python Wheels), and discovered that this particular
pure virtual method callederror does not occur when I use a locally built wheel for tensorflow-io.Note: The link in the docker build instructions is broken - https://github.com/tensorflow/io/blob/master/docs/development.md#docker - and the latest image in tfsigio/tfio is about 2 years old.