tensorflow: tf.keras.Sequential() fails

System information

Running the most basic instruction fails, for example from the documentation page https://www.tensorflow.org/api_docs/python/tf/keras/Sequential
OS Platform and Distribution: Arch Linux kernel 5.7.10-arch1-1 (linux@archlinux) (gcc version 10.1.0 (GCC), GNU ld (GNU Binutils) 2.34.0)
TensorFlow installed from (source or binary): binary package (official Arch package)
TensorFlow version (use command below): tensorflow-cuda 2.3.0
Python version: Python 3.8.4
CUDA/cuDNN version: cuda 11.0
GPU model and memory: GeForce GTX 950M, Driver Version: 450.57 - 2004MiB

Describe the current behavior Start python then run:


import tensorflow as tf

m = tf.keras.Sequential()

The last line fails with the following error messages:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.8/site-packages/tensorflow/python/training/tracking/base.py", line 457, in _method_wrapper
    result = method(self, *args, **kwargs)
  File "/usr/lib/python3.8/site-packages/tensorflow/python/keras/engine/sequential.py", line 116, in __init__
    super(functional.Functional, self).__init__(  # pylint: disable=bad-super-call
  File "/usr/lib/python3.8/site-packages/tensorflow/python/training/tracking/base.py", line 457, in _method_wrapper
    result = method(self, *args, **kwargs)
  File "/usr/lib/python3.8/site-packages/tensorflow/python/keras/engine/training.py", line 308, in __init__
    self._init_batch_counters()
  File "/usr/lib/python3.8/site-packages/tensorflow/python/training/tracking/base.py", line 457, in _method_wrapper
    result = method(self, *args, **kwargs)
  File "/usr/lib/python3.8/site-packages/tensorflow/python/keras/engine/training.py", line 317, in _init_batch_counters
    self._train_counter = variables.Variable(0, dtype='int64', aggregation=agg)
  File "/usr/lib/python3.8/site-packages/tensorflow/python/ops/variables.py", line 262, in __call__
    return cls._variable_v2_call(*args, **kwargs)
  File "/usr/lib/python3.8/site-packages/tensorflow/python/ops/variables.py", line 244, in _variable_v2_call
    return previous_getter(
  File "/usr/lib/python3.8/site-packages/tensorflow/python/ops/variables.py", line 237, in <lambda>
    previous_getter = lambda **kws: default_variable_creator_v2(None, **kws)
  File "/usr/lib/python3.8/site-packages/tensorflow/python/ops/variable_scope.py", line 2633, in default_variable_creator_v2
    return resource_variable_ops.ResourceVariable(
  File "/usr/lib/python3.8/site-packages/tensorflow/python/ops/variables.py", line 264, in __call__
    return super(VariableMetaclass, cls).__call__(*args, **kwargs)
  File "/usr/lib/python3.8/site-packages/tensorflow/python/ops/resource_variable_ops.py", line 1507, in __init__
    self._init_from_args(
  File "/usr/lib/python3.8/site-packages/tensorflow/python/ops/resource_variable_ops.py", line 1661, in _init_from_args
    handle = eager_safe_variable_handle(
  File "/usr/lib/python3.8/site-packages/tensorflow/python/ops/resource_variable_ops.py", line 242, in eager_safe_variable_handle
    return _variable_handle_from_shape_and_dtype(
  File "/usr/lib/python3.8/site-packages/tensorflow/python/ops/resource_variable_ops.py", line 174, in _variable_handle_from_shape_and_dtype
    gen_logging_ops._assert(  # pylint: disable=protected-access
  File "/usr/lib/python3.8/site-packages/tensorflow/python/ops/gen_logging_ops.py", line 49, in _assert
    _ops.raise_from_not_ok_status(e, name)
  File "/usr/lib/python3.8/site-packages/tensorflow/python/framework/ops.py", line 6843, in raise_from_not_ok_status
    six.raise_from(core._status_to_exception(e.code, message), None)
  File "<string>", line 3, in raise_from
tensorflow.python.framework.errors_impl.InvalidArgumentError: assertion failed: [0] [Op:Assert] name: EagerVariableNameReuse

Describe the expected behavior

An empty sequential model is created, no error.

Standalone code to reproduce the issue

import tensorflow as tf

m = tf.keras.Sequential()

About this issue

Original URL
State: closed
Created 4 years ago
Reactions: 6
Comments: 46 (8 by maintainers)

Most upvoted comments

Same problem with python 3.7, tf 2.3 , CUDA 10.1 and cuDnn 7.6, windows 10 python 3.7, tf-nigthly (1-SEP-2020) , CUDA 11.0 and cuDnn 8.03, windows 10

also replicated by just tf.Variable((2,3))

the problem does not happen in tensorflow 2.2

amitport on Sep 2, 2020

This is the failing assertion which seems to be GPU independent. Do you know if you’re doing something weird in your model?

from tensorflow.keras.models import Sequential
model = Sequential()

seems to work fine for me though, on tf-nightly.

sanjoy on Aug 12, 2020

I have same issue too. I have 2 laptop. One has rtx2060 mobile and other gtx860M. I installed linux a while of time ago. before that it is worked smoothly but after i install windows 10 back its doesn’t working anymore I m facing with this error on gtx860m. rtx one just working fine with these versions. but gtx one not working whatever i do visual studio 2019 community with c++ Cuda: 10.1 update2 Cudnn: 7.6.5 Windows10 ( two laptop has same installation versions) I tried all possiblities. I tried python 3.7, python 3.8, anaconda versions, i tried driver version wich comes with cuda, i tried updating driver version to 451. I tried tensorflow 2.2.0 and 2.3.0. and always i installed windows 10 from zero when trying different combination.

import tensorflow as tf
train_loss = tf.keras.metrics.Mean(name=“train_loss”)

import tensorflow as tf
m = tf.keras.Sequential()

or trying to train a model it s not working and giving this error

	self._init_from_args(
  File "C:\Users\mehmet\Anaconda3\lib\site-packages\tensorflow\python\ops\resource_variable_ops.py", line 1661, in _init_from_args
	handle = eager_safe_variable_handle(
  File "C:\Users\mehmet\Anaconda3\lib\site-packages\tensorflow\python\ops\resource_variable_ops.py", line 242, in eager_safe_variable_handle
	return _variable_handle_from_shape_and_dtype(
  File "C:\Users\mehmet\Anaconda3\lib\site-packages\tensorflow\python\ops\resource_variable_ops.py", line 174, in _variable_handle_from_shape_and_dtype
	gen_logging_ops._assert(  # pylint: disable=protected-access
  File "C:\Users\mehmet\Anaconda3\lib\site-packages\tensorflow\python\ops\gen_logging_ops.py", line 49, in _assert
	_ops.raise_from_not_ok_status(e, name)
  File "C:\Users\mehmet\Anaconda3\lib\site-packages\tensorflow\python\framework\ops.py", line 6843, in raise_from_not_ok_status
	six.raise_from(core._status_to_exception(e.code, message), None)
  File "<string>", line 3, in raise_from
InvalidArgumentError: assertion failed: [0] [Op:Assert] name: EagerVariableNameReuse

or stoping like this.

2020-08-05 17:36:46.242249: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudart64_101.dll
2020-08-05 17:36:48.741475: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library nvcuda.dll
2020-08-05 17:36:46.242249: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudart64_101.dll
2020-08-05 17:36:48.741475: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library nvcuda.dll
2020-08-05 17:36:49.608631: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: GeForce GTX 860M computeCapability: 5.0
coreClock: 1.0195GHz coreCount: 5 deviceMemorySize: 4.00GiB deviceMemoryBandwidth: 74.65GiB/s
2020-08-05 17:36:49.608692: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudart64_101.dll
2020-08-05 17:36:49.613753: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cublas64_10.dll
2020-08-05 17:36:49.618772: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cufft64_10.dll
2020-08-05 17:36:49.620362: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library curand64_10.dll
2020-08-05 17:36:49.626329: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cusolver64_10.dll
2020-08-05 17:36:49.629463: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cusparse64_10.dll
2020-08-05 17:36:49.641028: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudnn64_7.dll
2020-08-05 17:36:49.641171: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0
2020-08-05 17:36:49.641781: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN)to use the following CPU instructions in performance-critical operations:  AVX2
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2020-08-05 17:36:49.659086: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x18a99c5aff0 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-08-05 17:36:49.659142: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
2020-08-05 17:36:49.659427: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: GeForce GTX 860M computeCapability: 5.0
coreClock: 1.0195GHz coreCount: 5 deviceMemorySize: 4.00GiB deviceMemoryBandwidth: 74.65GiB/s
2020-08-05 17:36:49.659465: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudart64_101.dll
2020-08-05 17:36:49.659487: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cublas64_10.dll
2020-08-05 17:36:49.659506: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cufft64_10.dll
2020-08-05 17:36:49.659523: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library curand64_10.dll
2020-08-05 17:36:49.659539: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cusolver64_10.dll
2020-08-05 17:36:49.659555: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cusparse64_10.dll
2020-08-05 17:36:49.659576: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudnn64_7.dll
2020-08-05 17:36:49.659646: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0
2020-08-05 17:36:49.751107: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1257] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-08-05 17:36:49.751144: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1263]  	0
2020-08-05 17:36:49.751154: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1276] 0:   N
2020-08-05 17:36:49.751380: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1402] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3121 MB memory) -> physical GPU (device: 0, name: GeForce GTX 860M, pci bus id: 0000:01:00.0, compute capability: 5.0)
2020-08-05 17:36:49.755399: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x18a9c483ba0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2020-08-05 17:36:49.755435: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): GeForce GTX 860M, Compute Capability 5.0

mcagricaliskan on Aug 5, 2020

@gowthamkpr Right now I’m running with the following (downgraded) versions

cuda 10.2.89-5
cudnn 7.6.5.32-4
tensorflow-cuda 2.2.0-1

Waiting for the next update that would guarantee that my code runs.

alexn11 on Aug 17, 2020

Seems to be working when deactivating the GPU:

import os
os.environ["CUDA_VISIBLE_DEVICES"] = "-1"
import tensorflow as tf
m = tf.keras.Sequential()

At that point I get the following output:

2020-07-29 12:57:51.389675: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcuda.so.1
2020-07-29 12:57:51.410142: E tensorflow/stream_executor/cuda/cuda_driver.cc:314] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected
2020-07-29 12:57:51.410208: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:169] retrieving CUDA diagnostic information for host: myhostname
2020-07-29 12:57:51.410220: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:176] hostname: myhostname
2020-07-29 12:57:51.410407: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:200] libcuda reported version is: 450.57.0
2020-07-29 12:57:51.410449: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:204] kernel reported version is: 450.57.0
2020-07-29 12:57:51.410459: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:310] kernel version seems to match DSO: 450.57.0
2020-07-29 12:57:51.410912: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN)to use the following CPU instructions in performance-critical operations:  SSE3 SSE4.1 SSE4.2 AVX AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2020-07-29 12:57:51.419007: I tensorflow/core/platform/profile_utils/cpu_utils.cc:104] CPU Frequency: 2300800000 Hz
2020-07-29 12:57:51.419293: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55d524787410 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-07-29 12:57:51.419314: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
2020-07-29 12:57:51.420774: I tensorflow/core/common_runtime/process_util.cc:146] Creating new thread pool with default inter op setting: 2. Tune using inter_op_parallelism_threads for best performance.

The model has been created:

>>> m
<tensorflow.python.keras.engine.sequential.Sequential object at 0x7ff7a49cfc40>

alexn11 on Jul 29, 2020