tensorflow: failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED

System information

  • OS Platform and Distribution (e.g., Linux Ubuntu 16.04):
Linux 4.4.0-75-generic #96-Ubuntu SMP Thu Apr 20 09:56:33 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
  • TensorFlow version (use command below):
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcublas.so.8.0 locally
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcudnn.so.5 locally
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcufft.so.8.0 locally
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcuda.so.1 locally
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcurand.so.8.0 locally
v1.0.0-65-g4763edf-dirty 1.0.1

  • CUDA/cuDNN version: 8.0

  • GPU model and memory:

name: GeForce GTX 980
major: 5 minor: 2 memoryClockRate (GHz) 1.2785
pciBusID 0000:01:00.0
Total memory: 3.94GiB
Free memory: 145.50MiB

Describe the problem

If I change the order of device usage, it would report error

Source code / logs

  • If I use GPU first then CPU, it would be fine
with tf.device('/gpu:0'):
    a = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[2, 3], name='a')
    b = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[3, 2], name='b')
    c = tf.matmul(a, b)
with tf.device('/cpu:0'):
    e = tf.constant([1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18], shape=[2, 9],dtype=tf.float32, name='e')
    f = tf.matmul(c,e)
sess = tf.Session(config=tf.ConfigProto(log_device_placement=True))
print(sess.run(f))
  • But if I use CPU first then GPU, it return error
with tf.device('/cpu:0'):
    a = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[2, 3], name='a')
    b = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[3, 2], name='b')
    c = tf.matmul(a, b)
with tf.device('/gpu:0'):
    e = tf.constant([1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18], shape=[2, 9],dtype=tf.float32, name='e')
    f = tf.matmul(c,e)
sess = tf.Session(config=tf.ConfigProto(log_device_placement=True))
print(sess.run(f))
  • the error dump below
    print(sess.run(f))
  File "/home/pika/.local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 767, in run
    run_metadata_ptr)
  File "/home/pika/.local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 965, in _run
    feed_dict_string, options, run_metadata)
  File "/home/pika/.local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1015, in _do_run
    target_list, options, run_metadata)
  File "/home/pika/.local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1035, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InternalError: Blas SGEMM launch failed : a.shape=(2, 2), b.shape=(2, 9), m=2, n=9, k=2
         [[Node: MatMul_1 = MatMul[T=DT_FLOAT, transpose_a=false, transpose_b=false, _device="/job:localhost/replica:0/task:0/gpu:0"](MatMul/_1, e)]]

Caused by op 'MatMul_1', defined at:
  File "m1_n0teb00k/tensorflow_palyground.py", line 13, in <module>
    f = tf.matmul(c,e)
  File "/home/pika/.local/lib/python3.5/site-packages/tensorflow/python/ops/math_ops.py", line 1765, in matmul
    a, b, transpose_a=transpose_a, transpose_b=transpose_b, name=name)
  File "/home/pika/.local/lib/python3.5/site-packages/tensorflow/python/ops/gen_math_ops.py", line 1454, in _mat_mul
    transpose_b=transpose_b, name=name)
  File "/home/pika/.local/lib/python3.5/site-packages/tensorflow/python/framework/op_def_library.py", line 763, in apply_op
    op_def=op_def)
  File "/home/pika/.local/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 2327, in create_op
    original_op=self._default_original_op, op_def=op_def)
  File "/home/pika/.local/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 1226, in __init__
    self._traceback = _extract_stack()

InternalError (see above for traceback): Blas SGEMM launch failed : a.shape=(2, 2), b.shape=(2, 9), m=2, n=9, k=2
         [[Node: MatMul_1 = MatMul[T=DT_FLOAT, transpose_a=false, transpose_b=false, _device="/job:localhost/replica:0/task:0/gpu:0"](MatMul/_1, e)]]

About this issue

  • Original URL
  • State: closed
  • Created 7 years ago
  • Comments: 67 (8 by maintainers)

Commits related to this issue

Most upvoted comments

This seems to possibly be an out-of-memory being masked by the CUBLAS_STATUS_NOT_INITIALIZED error. When I have low memory and ask for a new session for detection I hit this error, when I clear the gpu of other processes and free memory then I do not get the cublas error. Since by default it seems tensorflow sucks up nearly all available memory this could happen a lot I imagine.

root@746bb093df44:/data/jeremy/tensorflow/models# python /usr/lib/python2.7/dist-packages/variant/ml/tf_detect.py
cwd /data/jeremy/tensorflow/models/object_detection
using gpu0
2017-09-23 14:37:50.992104: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.
2017-09-23 14:37:50.992170: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.
2017-09-23 14:37:51.072360: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:893] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2017-09-23 14:37:51.072588: I tensorflow/core/common_runtime/gpu/gpu_device.cc:955] Found device 0 with properties: 
name: Quadro P6000
major: 6 minor: 1 memoryClockRate (GHz) 1.645
pciBusID 0000:00:05.0
Total memory: 23.87GiB
Free memory: 805.44MiB
2017-09-23 14:37:51.072625: I tensorflow/core/common_runtime/gpu/gpu_device.cc:976] DMA: 0 
2017-09-23 14:37:51.072643: I tensorflow/core/common_runtime/gpu/gpu_device.cc:986] 0:   Y 
2017-09-23 14:37:51.072664: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1045] Creating TensorFlow device (/gpu:0) -> (device: 0, name: Quadro P6000, pci bus id: 0000:00:05.0)
2017-09-23 14:37:56.768781: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED
2017-09-23 14:37:56.768888: W tensorflow/stream_executor/stream.cc:1756] attempting to perform BLAS operation using StreamExecutor without BLAS support
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/variant/ml/tf_detect.py", line 264, in <module>
    test_detect()
  File "/usr/lib/python2.7/dist-packages/variant/ml/tf_detect.py", line 123, in test_detect
    feed_dict={image_tensor: image_np_expanded})
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 895, in run
    run_metadata_ptr)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1124, in _run
    feed_dict_tensor, options, run_metadata)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1321, in _do_run
    options, run_metadata)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1340, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InternalError: Blas SGEMM launch failed : m=36300, n=80, k=64
	 [[Node: FirstStageFeatureExtractor/InceptionResnetV2/InceptionResnetV2/Conv2d_3b_1x1/convolution = Conv2D[T=DT_FLOAT, data_format="NHWC", padding="SAME", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/gpu:0"](FirstStageFeatureExtractor/InceptionResnetV2/InceptionResnetV2/MaxPool_3a_3x3/MaxPool, FirstStageFeatureExtractor/InceptionResnetV2/Conv2d_3b_1x1/weights/read)]]
	 [[Node: SecondStagePostprocessor/BatchMultiClassNonMaxSuppression/MultiClassNonMaxSuppression/ChangeCoordinateFrame_52/Scale/concat/_391 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/cpu:0", send_device="/job:localhost/replica:0/task:0/gpu:0", send_device_incarnation=1, tensor_name="edge_21463_SecondStagePostprocessor/BatchMultiClassNonMaxSuppression/MultiClassNonMaxSuppression/ChangeCoordinateFrame_52/Scale/concat", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"]()]]

Caused by op u'FirstStageFeatureExtractor/InceptionResnetV2/InceptionResnetV2/Conv2d_3b_1x1/convolution', defined at:
  File "/usr/lib/python2.7/dist-packages/variant/ml/tf_detect.py", line 70, in <module>
    tf.import_graph_def(od_graph_def, name='')
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/importer.py", line 313, in import_graph_def
    op_def=op_def)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 2630, in create_op
    original_op=self._default_original_op, op_def=op_def)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 1204, in __init__
    self._traceback = self._graph._extract_stack()  # pylint: disable=protected-access

InternalError (see above for traceback): Blas SGEMM launch failed : m=36300, n=80, k=64
	 [[Node: FirstStageFeatureExtractor/InceptionResnetV2/InceptionResnetV2/Conv2d_3b_1x1/convolution = Conv2D[T=DT_FLOAT, data_format="NHWC", padding="SAME", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/gpu:0"](FirstStageFeatureExtractor/InceptionResnetV2/InceptionResnetV2/MaxPool_3a_3x3/MaxPool, FirstStageFeatureExtractor/InceptionResnetV2/Conv2d_3b_1x1/weights/read)]]
	 [[Node: SecondStagePostprocessor/BatchMultiClassNonMaxSuppression/MultiClassNonMaxSuppression/ChangeCoordinateFrame_52/Scale/concat/_391 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/cpu:0", send_device="/job:localhost/replica:0/task:0/gpu:0", send_device_incarnation=1, tensor_name="edge_21463_SecondStagePostprocessor/BatchMultiClassNonMaxSuppression/MultiClassNonMaxSuppression/ChangeCoordinateFrame_52/Scale/concat", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"]()]]
+40
jeremy-rutman on Sep 23, 2017

2019-02-11 16:42:08.094047: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.101327: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.101418: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.101615: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.101720: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.102197: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.103385: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.104762: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.104859: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.105022: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.105433: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.105519: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.105676: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.106412: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.106499: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.106657: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.107383: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.107484: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.107643: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.108376: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.108460: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.108614: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.109330: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.109414: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.109581: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.110328: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.110414: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.110580: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.111328: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.111415: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.111605: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.112232: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.112308: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.112436: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.113061: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.113125: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.113253: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.113871: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.113948: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.114089: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.114680: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.114756: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.114885: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.115542: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.115621: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.115766: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.116345: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.116420: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.116550: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.117171: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.117259: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.117402: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.118083: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.118165: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.118299: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.118901: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.118977: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.119106: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.119737: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.119833: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.119962: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.120649: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.120728: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.120858: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.121487: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.121563: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.121691: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.122289: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.122336: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.122395: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.122427: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.122497: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.122542: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.122602: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.122634: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.122694: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.122727: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.122799: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.122830: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.122900: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.122931: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.123011: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.123042: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.123101: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.123133: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.123191: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.123223: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.123281: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.123312: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.123370: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.123400: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.123508: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.123543: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.123605: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.123637: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.123695: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.123727: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.123818: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.123851: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.123908: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.123939: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.123997: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.124029: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.124086: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.124117: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.124449: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.124484: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.124529: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.124560: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.124743: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.124810: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.124914: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.124950: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.124993: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.125016: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.125133: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.125200: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.125310: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.125334: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.125391: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.125488: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.125521: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.125582: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.125680: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.125712: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.125755: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.125778: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.125890: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.125933: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.125957: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.126070: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.126113: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.126147: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.126276: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.126300: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.126341: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.126365: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.126489: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.126550: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.126658: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.126682: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.126723: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.126746: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.126871: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.126933: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.127350: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.127376: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.127440: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.127552: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.127577: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.127637: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.127734: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.127758: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.127809: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.127831: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.127942: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.127985: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.128007: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.128137: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.128161: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.128202: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.128224: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.128352: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.128396: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.128419: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.128530: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.128574: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.128652: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.128828: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.128852: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.128894: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.128936: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.129078: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.129103: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.129123: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.129143: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.129184: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.129335: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.129577: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.129602: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.129623: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.129652: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.129672: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.129692: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.129714: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.129734: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.129754: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.129782: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.129802: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.129822: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.129970: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.129993: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.130048: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.130382: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.130688: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.130802: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.131137: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.131180: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.131378: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.131546: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.131786: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.131895: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.132214: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.132258: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.132455: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.132634: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.132884: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.132993: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.133313: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.133356: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.133551: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.133711: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.133943: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.134050: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.134379: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.134431: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.134626: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.134804: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.135074: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.135191: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.135561: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.135615: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.135829: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.136008: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.136240: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.136347: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.136664: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.136707: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.136903: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.137078: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.137309: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.137414: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.137790: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.137845: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.138066: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.138253: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.138479: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.138585: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.138909: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.138963: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.139151: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.139324: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.139574: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.139691: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.140016: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.140059: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.140252: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.140448: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.140677: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.140783: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.141096: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.141139: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.141347: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.141534: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.141763: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.141870: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.142180: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.142224: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.142416: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.142589: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.142863: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.142998: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.143331: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.143374: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.143604: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.143795: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.144019: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.144125: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.144437: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.144480: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.144671: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.144842: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.145070: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.145176: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.145485: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.145527: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.145749: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.145940: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.146176: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.146282: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.146593: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.146636: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.146826: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.146997: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.147220: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.147326: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.147694: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.147739: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.147956: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.148159: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.148412: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.148519: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.148843: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.148887: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.149081: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.149268: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.149508: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.149627: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.149936: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.149979: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.150171: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.150344: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.150566: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.150672: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.151034: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.151091: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.151298: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.151482: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.151715: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.151822: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.152163: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.152188: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.152417: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.152609: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.152871: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.152914: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.153053: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.153094: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.153277: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.153309: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.153541: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.153574: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.153708: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.155756: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.155960: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.156480: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2019-02-11 16:42:08.156626: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED Floating point exception (core dumped)

+35
wxtyixiao on Feb 11, 2019

In my case it was not a memory error, I fixed it by removing the cache folder ~/.nv/

+25
stbnps on Jul 16, 2018

I fixed this by running as root.

+20
DhashS on Jan 25, 2018

Check out #5354, #3600, #4196, http://stackoverflow.com/questions/38303974/tensorflow-running-error-with-cublas Let me know if those don’t provide solution to your problem.

+19
aselle on Apr 27, 2017

It has been 14 days with no activity and the awaiting response label was assigned. Is this still an issue?

This in fact is still an issue, I followed TF GPU install for 18.04/CUDA 10.1, and for whatever reason the default linked library for libcublas10 is still libcublas10=10.2.2.89-1. So whatever pin that was done above did not work.

For others who encounter this cublas init error, do this:

  1. first check libcublas10 version: dpkg -l | grep libcublas
  2. if version is 10.2.2.89, remove it: sudo apt remove libcublas10
  3. install downgraded version: sudo apt install libcublas10=10.2.1.243-1
  4. it might say that it has to remove a bunch of cuda packages when you downgrade, I thought it might be an issue, but it did not impact my CUDA 10.1 installation.
+17
SkookumAsFrig on Apr 21, 2020

Observed the similar problem with libcublas10=10.2.2.89-1 / libcublas-dev=10.2.2.89-1. Downgrading to either libcublas10=10.2.1.243-1 / libcublas-dev=10.2.1.243-1 or libcublas10=10.1.0.105-1 / libcublas-dev=10.1.0.105-1 + export TF_FORCE_GPU_ALLOW_GROWTH=true (Limiting GPU memory growth) has fixed the issue. ~/.nv was empty.

+10
artemry-nv on Dec 6, 2019

I have tried ‘rm ~/.nv’ and ‘config = tf.ConfigProto() config.gpu_options.allow_growth = True’, no one work.

+6
wxtyixiao on Feb 11, 2019

Heads up, if you’re facing this problem in a conda env, make sure to pin the conda package cudatoolkit=10.1.243. We faced an issue where pytorch (also in our env) was installing a more recent cudatoolkit, and that was causing this CUBLAS_STATUS_NOT_INITIALIZED error when running tensorflow 2.1 and 2.2.

+5
ankitvgupta on Jun 3, 2020

I’m on cuda-11-0 and tensorflow 2.4 and got the same errror. First of all, removing the chache folder ~/.nv did noting for me. Secondly, while

physical_devices = tf.config.list_physical_devices('GPU')
tf.config.experimental.set_memory_growth(physical_devices[0], True)

got my code running, the actual error was that I had libcudnn8=8.1.1.33 installed (which seems to rely on cuda-11-2). So downgrading libcudnn to 8.0.5.39 was the actual fix.

+4
AxelBohm on Mar 5, 2021

It is still broken for TF2.1rc2, and I confirmed that adding TF_FORCE_GPU_ALLOW_GROWTH=true to docker file doesn’t solve the issue. Also confirmed that changing base docker image to nvidia/cuda:10.1-cudnn7-runtime-ubuntu18.04 and installing tensorflow-gpu==2.1.0rc2 works as expected. Probably need to downgrade cuda to 10.1

+4
vlasenkoalexey on Jan 8, 2020

Had the same problem with libcublas10=10.2.2.89-1, solved it by downgrading to 10.1.0.105-1

+4
cataclysmus on Nov 22, 2019

As @jeremy-rutman said, this is likely an out-of-memory issue. When I get this error, I can fix it every time by unplugging my external monitor. This frees up GPU memory immediately, and I’m then able to run predictions with TF. Another fix that seems to work is if I close my million Chrome tabs.

Note that running as root did not fix it for me.

+4
harvitronix on Mar 2, 2018

This didnt work for me.

I have checked all my installation. Made sure my LD_LIBRARY_PATH is what it should be. Kept my memory down to a minimum. Nothing seems to be working.

+4
ashleylid on Nov 20, 2017

I am experiencing this too, on tensorflow 2.1. A short bit of code to check if the problem persists is:

import tensorflow as tf
print(tf.matmul([[1., 2.],[3., 4.]], [[1., 2.],[3., 4.]]))

Attempting downgrade now.

+3
thomasaarholt on Feb 10, 2020

Looks like latest change wasn’t pushed to docker hub: https://hub.docker.com/layers/tensorflow/tensorflow/2.1.0rc2-gpu/images/sha256-e328b120aba6af551e1b9124cd7eeee3f05e4d41d115c139614f1ca4350c7953 https://github.com/tensorflow/tensorflow/blob/master/tensorflow/tools/dockerfiles/dockerfiles/devel-gpu.Dockerfile

Meaning that this will work if you build from docker file, but won’t work if you pull container from docker hub. Probably not a huge issue sine TF2.1 is going to be released this week.

+3
vlasenkoalexey on Jan 8, 2020

In my case with rtx3070 & tensorflow 2.4.1, below code worked for me

physical_devices = tf.config.experimental.list_physical_devices('GPU')
assert len(physical_devices) > 0, "Not enough GPU hardware devices available"
config = tf.config.experimental.set_memory_growth(physical_devices[0], True)
+2
joonb14 on Apr 20, 2021

Nothing changes when compiling TF from source on CUDA 10.1.243. So in the meantime I’m using this workaround:

def is_rtx_card():
    for device in device_lib.list_local_devices():
        if device.device_type == "GPU" and "RTX" in device.physical_device_desc:
            return True
    return False


# TF 2.2 is not working with RTX cards without this workaround...
if is_rtx_card():
    print("WARNING: RTX card detected! Using experimental device growth!")
    tf.config.experimental.set_memory_growth(tf.config.experimental.list_physical_devices('GPU')[0], True)

Which work if you compiled TF 2.2 from source with CUDA 10.2 (doesn’t work with CUDA 10.1.x)

+2
EKami on Jun 11, 2020

Also maybe not directly relevant to cublas package version, but I did an apt upgrade today and it obliterated the tensorflow GPU packages in terms of compatibility. The ones impacted were libcublas10 and libcudnn (went from 6 to 7 cuda 10.2), even though I am on 10.1. So I thought I would share some useful commands for fixing things like this:

  1. check package version, say we are fixing libcudnn: dpkg -l | grep libcudnn
  2. remove incorrectly installed new ones: sudo apt remove libcudnn7 libcudnn7-dev libcudnn7-doc
  3. search for available libcudnn packages: sudo apt-cache policy libcudnn7
  4. install the correct one, for your CUDA version (don’t forget the -dev version too): sudo apt install libcudnn7=7.6.5.32-1+cuda10.1
  5. put these annoying packages on hold: sudo apt-mark hold libcublas10 reference links: https://chadrick-kwag.net/downgrading-cudnn-version-to-match-cuda-version/ https://askubuntu.com/questions/18654/how-to-prevent-updating-of-a-specific-package
+2
SkookumAsFrig on Apr 23, 2020

Had to downgrade to tensorflow version 2.0 pip install tensorflow-gpu==2.0

+2
samon11 on Feb 4, 2020

In my case it was caused by running it under Windows with session_config.gpu_options.allow_growth = True without this I got another error, that was masked by failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED

+2
mpekalski on Apr 6, 2018

Just an FYI. Solved this issue on TF 2.2.0, Cuda 10.1 by removing existing libcublas and installing a libcublas 10.2.1.243-1 instead. sample for Red Hat/CentOS:

yum remove libcublas
yum install libcublas10-10.2.1.243-1.x86_64
+1
Denisolt on Aug 27, 2020

i found another way to solve this problem , i think the problem is that GPU havent been completely released . so first check the nvidia used by all the processing lsof /dev/nvidia*

lsof: WARNING: can’t stat() tracefs file system /sys/kernel/debug/tracing Output information may be incomplete. lsof: WARNING: can’t stat() fuse.gvfsd-fuse file system /run/user/108/gvfs Output information may be incomplete. COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME t2t-train 24674 dell mem CHR 195,255 28064 /dev/nvidiactl t2t-train 24674 dell mem CHR 195,0 17699 /dev/nvidia0 t2t-train 24674 dell mem CHR 241,0 32956 /dev/nvidia-uvm t2t-train 24674 dell mem CHR 195,1 44248 /dev/nvidia1 t2t-train 24674 dell mem CHR 195,2 2640 /dev/nvidia2

and then kill all the process kill -9 24674 and restart your work again

good luck

+1
paulrich1234 on May 14, 2020

@SkookumAsFrig Thanks for your solution! Downgrading libcublas worked for me.

+1
rremigius on Apr 22, 2020

I wonder why it is not possible to emit a more useful, detailed error message?

+1
jelmd on Feb 25, 2020

CUBLAS_STATUS_NOT_INITIALIZED can mask multiple issues, quite commonly is it a bad setup, especially when it crashes on initialization. I’d be nice to have a better error message.

TF2.1 is using CUDA10.1 while TF2.0 is on CUDA10.0. So in some cases just updating TF2.0 -> T2.1 won’t work, you’d also need to update CUDA and other libraries, and even possibly GPU drivers.

The easiest way to make sure that setup is correct: Make sure that you have https://github.com/NVIDIA/nvidia-docker installed. Run docker run --gpus all tensorflow/tensorflow:2.1.0-gpu-py3 python -c ‘import tensorflow as tf; from tensorflow.python.client import device_lib; device_lib.list_local_devices()’

Hope this helps.

+1
vlasenkoalexey on Feb 10, 2020

In my case it was not a memory error, I fixed it by removing the cache folder ~/.nv/

+1 - worked on linux

+1
aescart1 on Feb 26, 2019
Read more comments on GitHub
← tensorflow: Failed to convert weights to 8 bit precision: "Quantize weights tool only supports tflite models with one subgraph"
tensorflow: failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED on a AWS p2.xlarge instance →