tensorflow: Illegal instruction (core dumped)

System information

  • OS Platform and Distribution: Ubuntu 20.04 LTS
  • TensorFlow installed from: source
  • TensorFlow version: 2.4.1
  • Python version: 3.8.5
  • Installed using: pip
  • Bazel version: 4.0.0
  • GCC/Compiler version: 8.4.0
  • CUDA/cuDNN version: 11.1/8
  • GPU model and memory: GeForce GTX 1080 Ti (11.7G)

Describe the problem At first, I tried installing tensorflow from pip repositories, but it crashed with Illegal instruction (core dumped) upon importing. I’ve tracked this down to my CPU (Intel G4400) not supporting AVX instructions (https://github.com/tensorflow/tensorflow/issues/17411#issuecomment-608027554), so I built tensorflow from source, using the tools described above. I can successfully import tensorflow now, but when I try running the deepdream tutorial: https://www.tensorflow.org/tutorials/generative/deepdream.

Provide the exact sequence of commands / steps that you executed before running into the problem

  1. Downloaded notebook from https://www.tensorflow.org/tutorials/generative/deepdream
  2. jupyter notebook
  3. Click “Run All”
  4. Illegal instruction (core dumped)

Alternatively:

  1. Downloaded notebook from https://www.tensorflow.org/tutorials/generative/deepdream
  2. ipython nbconvert --to script deepdream.ipynb
  3. python3 deepdream.py
  4. Illegal instruction (core dumped)

Any other info / logs This is the full log of the “alternative” sequence:

$ python3 deepdream.py                                                                                                                                                                                                                          
2021-02-18 15:28:33.944636: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2021-02-18 15:28:35.342188: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2021-02-18 15:28:35.342976: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2021-02-18 15:28:35.372123: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-02-18 15:28:35.372799: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:01:00.0 name: GeForce GTX 1080 Ti computeCapability: 6.1
coreClock: 1.645GHz coreCount: 28 deviceMemorySize: 10.91GiB deviceMemoryBandwidth: 451.17GiB/s
2021-02-18 15:28:35.372953: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-02-18 15:28:35.373537: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 1 with properties: 
pciBusID: 0000:04:00.0 name: GeForce GTX 1080 Ti computeCapability: 6.1
coreClock: 1.6575GHz coreCount: 28 deviceMemorySize: 10.92GiB deviceMemoryBandwidth: 451.17GiB/s
2021-02-18 15:28:35.373661: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-02-18 15:28:35.374241: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 2 with properties: 
pciBusID: 0000:06:00.0 name: GeForce GTX 1080 Ti computeCapability: 6.1
coreClock: 1.6575GHz coreCount: 28 deviceMemorySize: 10.92GiB deviceMemoryBandwidth: 451.17GiB/s
2021-02-18 15:28:35.374363: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-02-18 15:28:35.374945: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 3 with properties: 
pciBusID: 0000:08:00.0 name: GeForce GTX 1080 Ti computeCapability: 6.1
coreClock: 1.6575GHz coreCount: 28 deviceMemorySize: 10.92GiB deviceMemoryBandwidth: 451.17GiB/s
2021-02-18 15:28:35.375035: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2021-02-18 15:28:35.419511: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2021-02-18 15:28:35.419904: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2021-02-18 15:28:35.430928: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2021-02-18 15:28:35.431822: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2021-02-18 15:28:35.440650: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.11
2021-02-18 15:28:35.453660: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2021-02-18 15:28:35.454131: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2021-02-18 15:28:35.454448: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-02-18 15:28:35.456315: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-02-18 15:28:35.458227: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-02-18 15:28:35.460016: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-02-18 15:28:35.461794: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-02-18 15:28:35.463552: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-02-18 15:28:35.465343: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-02-18 15:28:35.467095: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-02-18 15:28:35.468689: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0, 1, 2, 3
2021-02-18 15:28:35.469399: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  SSE3 SSE4.1 SSE4.2
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2021-02-18 15:28:35.469726: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2021-02-18 15:28:36.139881: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-02-18 15:28:36.140462: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:01:00.0 name: GeForce GTX 1080 Ti computeCapability: 6.1
coreClock: 1.645GHz coreCount: 28 deviceMemorySize: 10.91GiB deviceMemoryBandwidth: 451.17GiB/s
2021-02-18 15:28:36.140558: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-02-18 15:28:36.141120: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 1 with properties: 
pciBusID: 0000:04:00.0 name: GeForce GTX 1080 Ti computeCapability: 6.1
coreClock: 1.6575GHz coreCount: 28 deviceMemorySize: 10.92GiB deviceMemoryBandwidth: 451.17GiB/s
2021-02-18 15:28:36.141188: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-02-18 15:28:36.141733: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 2 with properties: 
pciBusID: 0000:06:00.0 name: GeForce GTX 1080 Ti computeCapability: 6.1
coreClock: 1.6575GHz coreCount: 28 deviceMemorySize: 10.92GiB deviceMemoryBandwidth: 451.17GiB/s
2021-02-18 15:28:36.141793: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-02-18 15:28:36.142330: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 3 with properties: 
pciBusID: 0000:08:00.0 name: GeForce GTX 1080 Ti computeCapability: 6.1
coreClock: 1.6575GHz coreCount: 28 deviceMemorySize: 10.92GiB deviceMemoryBandwidth: 451.17GiB/s
2021-02-18 15:28:36.142348: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2021-02-18 15:28:36.142369: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2021-02-18 15:28:36.142380: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2021-02-18 15:28:36.142389: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2021-02-18 15:28:36.142399: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2021-02-18 15:28:36.142408: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.11
2021-02-18 15:28:36.142418: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2021-02-18 15:28:36.142427: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2021-02-18 15:28:36.142482: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-02-18 15:28:36.143058: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-02-18 15:28:36.143631: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-02-18 15:28:36.144209: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-02-18 15:28:36.144794: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-02-18 15:28:36.145431: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-02-18 15:28:36.146029: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-02-18 15:28:36.146610: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-02-18 15:28:36.147133: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0, 1, 2, 3
2021-02-18 15:28:36.147166: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2021-02-18 15:28:40.439754: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-02-18 15:28:40.439843: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 1 2 3 
2021-02-18 15:28:40.439851: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N Y Y Y 
2021-02-18 15:28:40.439856: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 1:   Y N Y Y 
2021-02-18 15:28:40.439859: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 2:   Y Y N Y 
2021-02-18 15:28:40.439863: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 3:   Y Y Y N 
2021-02-18 15:28:40.440129: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-02-18 15:28:40.440819: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-02-18 15:28:40.441455: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-02-18 15:28:40.442052: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-02-18 15:28:40.442640: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-02-18 15:28:40.443222: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-02-18 15:28:40.443766: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 6064 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:01:00.0, compute capability: 6.1)
2021-02-18 15:28:40.444097: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-02-18 15:28:40.444766: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-02-18 15:28:40.445324: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 6122 MB memory) -> physical GPU (device: 1, name: GeForce GTX 1080 Ti, pci bus id: 0000:04:00.0, compute capability: 6.1)
2021-02-18 15:28:40.445584: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-02-18 15:28:40.446200: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-02-18 15:28:40.446752: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 6122 MB memory) -> physical GPU (device: 2, name: GeForce GTX 1080 Ti, pci bus id: 0000:06:00.0, compute capability: 6.1)
2021-02-18 15:28:40.446992: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-02-18 15:28:40.447637: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-02-18 15:28:40.448199: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:3 with 6122 MB memory) -> physical GPU (device: 3, name: GeForce GTX 1080 Ti, pci bus id: 0000:08:00.0, compute capability: 6.1)
2021-02-18 15:28:44.046400: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
2021-02-18 15:28:44.124782: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 3299990000 Hz
2021-02-18 15:28:44.980720: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
Illegal instruction (core dumped)

(I believe this is a build/install problem rather than a bug because I’ve already encountered this error in the context of an install issue)

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 24 (4 by maintainers)

Most upvoted comments

Well, I think I may have answered my own (and possibly the OP’s question). I ran my script using gdb. Python receives a SIGILL,(illegal instruction) in libcudnn_cnn_train.so.8

Disassembly:

│-->0x7ffd5ebbf2a7      vmovdqa 0x3c1831(%rip),%ymm0        # 0x7ffd5ef80ae0                                                                                                          
│   0x7ffd5ebbf2af      lea    0x53ce33a(%rip),%rax        # 0x7ffd63f8d5f0                                                                                                           
│   0x7ffd5ebbf2b6      lea    0x53ce4cb(%rip),%rdx        # 0x7ffd63f8d788                                                                                                           
│   0x7ffd5ebbf2bd      lea    0x3bdda4(%rip),%rcx        # 0x7ffd5ef7d068                                                                                                            
│   0x7ffd5ebbf2c4      lea    0x542f375(%rip),%rdi        # 0x7ffd63fee640

vmovdqa is an AVX instruction.

So it looks like the problem is that CUDNN is compiled using AVX instructions even though Tensorflow is not. When Tensorflow calls into that DLL, it crashes with SIGILL. Now to see if I can find a version of CUDNN without AVX that works with the build…

Just for the record I compiled with cudnn-11.2-linux-x64-v8.1.1.33. I don’t see anything in the release notes that explicitly states AVX is required.

Thank you @nicholastoddsmith , I did encounter the same issue as you. I built tensorflow from source (such a pain sometimes 😃 ) from a workstation with a 1080Ti and a processor without AVX support. I was happy to finish the build after some hours but finally got the “Illegal Instruction” when trying to run a tensorflow training (importing tensorflow was not an issue); For the record, I list here slightly more details for the deassemble;

As you suggest, running gdb on the program with a backtrace I got :

Thread 37 "python3" received signal SIGILL, Illegal instruction.
[Switching to Thread 0x7ffeabfff700 (LWP 11822)]
0x00007ffb3abbf2a7 in ?? () from /usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8

Then, within gdb , I did

(gdb) disassemble 0x00007ffb3abbf2a7, +20
=> 0x00007ffb3abbf2a7:  vmovdqa 0x3c1831(%rip),%ymm0        # 0x7ffb3af80ae0
   0x00007ffb3abbf2af:  lea    0x53ce33a(%rip),%rax        # 0x7ffb3ff8d5f0
   0x00007ffb3abbf2b6:  lea    0x53ce4cb(%rip),%rdx        # 0x7ffb3ff8d788

Anyway thank you for pointing to that . It helped me understanding the issue. One last note, I was using an installation with libcudnn 8.1.1.33-1+cuda11.2 and tensorflow 2.6.2 built successfully from the source.

And I noticed afterwhile the release notes of cudnn where they mention

Starting in version 8.1, cuDNN uses AVX intrinsics on the x86_64 architecture; users of this architecture without support for AVX intrinsics may see illegal instruction errors.

Yes, that’s true .

+3
jeremyfix on Jan 10, 2022

I found an older CUDNN version that doesn’t use AVX instructions. You can check by downloading the CUDNN package, disassembling the DLL in question, and grepping for AVX instructions. I used:

objdump -d cuda/lib64/libcudnn.so.7 | grep vmovdqa

That didn’t turn up any hits on cudnn-10.1-linux-x64-v7.6.5.32, while equivalent DLLs in newer packages did.

I swapped out the toolkits and libraries and tried compiling with r2.5, but it failed… the structure of the CUDNN library changed in more recent versions and tensorflow r2.5 depends on that new layout. Switching to the r2.4 branch worked. However, I also needed to switch to GCC 7 instead of 9.

Final configuration that worked (no more SIGILL crash now!):

cudnn-10.1-linux-x64-v7.6.5.32 cuda_10.1.243_418.87.00_linux GCC 7 tensorflow r2.4

Going forward, if all future CUDNN releases require AVX support, then non-AVX computers will be stuck at Tensorflow 2.4.

Hope this is helpful to other.

+3
nogilnick on Apr 23, 2021

Can I say “me too”, in case it helps you with debugging this? Here’s the result of running the two-line script above:

$ python3
Python 3.9.2 (default, Feb 28 2021, 17:03:44) 
[GCC 10.2.1 20210110] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import tensorflow as tf
2021-03-24 14:44:39.111867: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
>>> print("Num GPUs Available: ", len(tf.config.list_physical_devices('GPU')))
2021-03-24 14:44:49.652549: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2021-03-24 14:44:49.653460: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2021-03-24 14:44:49.691660: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-03-24 14:44:49.692173: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:41:00.0 name: GeForce RTX 2080 Ti computeCapability: 7.5
coreClock: 1.755GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s
2021-03-24 14:44:49.692196: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2021-03-24 14:44:49.695547: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2021-03-24 14:44:49.695655: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2021-03-24 14:44:49.696506: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2021-03-24 14:44:49.696911: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2021-03-24 14:44:49.697576: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.11
2021-03-24 14:44:49.698125: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2021-03-24 14:44:49.698215: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2021-03-24 14:44:49.698351: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-03-24 14:44:49.698969: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-03-24 14:44:49.699433: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
Num GPUs Available:  1
>>>

I have also just built TensorFlow from source (see my blog post if you want to see the details of how I did that).

Interestingly, the result of running grep . /sys/bus/pci/devices/0000\:*/numa_node is lots of -1s and one 0; the only 0 corresponds to the device 09:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981/PM983, whereas the GPU returns a -1.

+2
juliangilbey on Mar 24, 2021

It is working, but because I’ve switched machines and can no longer reproduce the issue - I’m personally fine with closing the issue, but can’t speak for others in this thread.

Thanks for your time and effort!

+1
tmladek on Feb 27, 2023

Ok.In that case shall we close it as resolved as I can see it working fine now? Thanks!

+1
SuryanarayanaY on Feb 13, 2023
Read more comments on GitHub
← tensorflow: If TMP is not set, then the build system defaults to the wrong path
tensorflow: Illegal instruction (core dumped) after running import tensorflow →