keras: Non-OK-status: Internal: invalid configuration argument Aborted (core dumped)

Please make sure that this is a Bug or a Feature Request and provide all applicable information asked by the template. If your issue is an implementation question, please ask your question on StackOverflow or on the Keras Slack channel instead of opening a GitHub issue.

System information

  • Have I written custom code (as opposed to using example directory):
  • OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux Ubuntu 16.04
  • TensorFlow backend (yes / no): yes
  • TensorFlow version: v1.14.0-rc1-22-gaf24dc91b5 1.14.0
  • Keras version: 2.2.4
  • Python version: 3.6
  • CUDA/cuDNN version: Cuda compilation tools, release 10.0, V10.0.130
  • GPU model and memory: 2 gpus, each 11 GB

You can obtain the TensorFlow version with:
python -c “import tensorflow as tf; print(tf.GIT_VERSION, tf.VERSION)”
You can obtain the Keras version with:
python -c ‘import keras as k; print(k.version)’

Describe the current behavior
I was using below code to build a LSTM model.

    left = Input(shape=(128, 3072), dtype='float32', name='Input-Left')
    right = Input(shape=(128, 3072), dtype='float32', name='Input-Right')
    lstm = Bidirectional(LSTM(units=768,
                              activation='tanh'),
                         name='Bidirectional-LSTM')
    l_lstm = lstm(left)
    r_lstm = lstm(right)
    subtracted = Subtract(name='Subtract')([l_lstm, r_lstm])
    abs_subtracted = Lambda(function=backend.abs)(subtracted)
    mul = Multiply(name='multiplication')([l_lstm, r_lstm])
    concat = concatenate([abs_subtracted, mul])
    output = Dense(units=1)(concat)
    model = Model(inputs=[left, right],
                  outputs=output)
    model = multi_gpu_model(model, gpus=2)
    model.compile(loss='mean_squared_error',
                  optimizer='Adam',
                  metrics=['acc'])

Describe the expected behavior
expect the code run without error.

Code to reproduce the issue
Provide a reproducible test case that is the bare minimum necessary to generate the problem.

Other info / logs
Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached.

get the following error

2019-07-11 00:34:47.259516: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-07-11 00:34:47.261497: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-07-11 00:34:47.263346: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-07-11 00:34:47.263979: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-07-11 00:34:47.264617: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0, 1
2019-07-11 00:34:49.404341: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-07-11 00:34:49.404385: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187]      0 1 
2019-07-11 00:34:49.404390: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0:   N Y 
2019-07-11 00:34:49.404394: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 1:   Y N 
2019-07-11 00:34:49.404729: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-07-11 00:34:49.405481: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-07-11 00:34:49.406172: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-07-11 00:34:49.406916: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9428 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:01:00.0, compute capability: 6.1)
2019-07-11 00:34:49.407465: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-07-11 00:34:49.408128: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 10039 MB memory) -> physical GPU (device: 1, name: GeForce GTX 1080 Ti, pci bus id: 0000:02:00.0, compute capability: 6.1)
2019-07-11 00:34:49.942669: F ./tensorflow/core/kernels/random_op_gpu.h:227] Non-OK-status: CudaLaunchKernel(FillPhiloxRandomKernelLaunch<Distribution>, num_blocks, block_size, 0, d.stream(), gen, data, size, dist) status: Internal: invalid configuration argument
Aborted (core dumped)

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Reactions: 1
  • Comments: 24 (2 by maintainers)

Most upvoted comments

for me, its cause i import torch and tf at the same time. you have to import tf before torch so tf can use GPU correctly. see https://github.com/tensorflow/tensorflow/issues/27487

Closing this issue. I was able to resolve this issue. I think it is compatibility issue between tensorflow-gpu and cuda.

  • CUDA 10.0 and Tensorflow-gpu 1.14: No Error
  • CUDA 9.2 and Tensorflow-gpu 1.14: CudaLaunchKernel error.
  • CUDA 9.2 and Tensorflow-gpu 1.12: No Error.

Experiencing the same issue with CUDA 10.0 and Tensorflow-gpu 1.14. I don’t think it is version related issue. At least not between CUDA and tensorflow gpu

CUDA10.0 and tensorflow-2.0 with the same error.

I am using CUDA 10.0 and Tensorflow-gpu 1.14 but still got this issue, 2019-10-17 10:28:25.101386: F ./tensorflow/core/kernels/random_op_gpu.h:227] Non-OK-status: CudaLaunchKernel(FillPhiloxRandomKernelLaunch<Distribution>, num_blocks, block_size, 0, d.stream(), gen, data, size, dist) status: Internal: invalid configuration argument

Closing this issue. I was able to resolve this issue. I think it is compatibility issue between tensorflow-gpu and cuda.

  • CUDA 10.0 and Tensorflow-gpu 1.14: No Error
  • CUDA 9.2 and Tensorflow-gpu 1.14: CudaLaunchKernel error.
  • CUDA 9.2 and Tensorflow-gpu 1.12: No Error.

Hi, I use CUDA 10.0 and Tensorflow-gpu 1.14 but still got the same issue.