tensorflow: Unexpected warning during GeneratorDataset iterator finalization

System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow):
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 14.04
Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device: N/A
TensorFlow installed from (source or binary): binary
TensorFlow version (use command below): ('v1.12.0-0-ga6d8ffae09', '1.12.0')
Python version: 2.7
Bazel version (if compiling from source): N/A
GCC/Compiler version (if compiling from source): N/A
CUDA/cuDNN version: N/A
GPU model and memory:

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 410.73       Driver Version: 410.73       CUDA Version: 10.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Quadro M2000M       Off  | 00000000:01:00.0  On |                  N/A |
| N/A   57C    P0    N/A /  N/A |    832MiB /  4043MiB |      4%      Default |
+-------------------------------+----------------------+----------------------+

Describe the current behavior When using from_generator to create a tf.data.Dataset instance, reading from a one_shot_iterator and not explicitly closing the session object, I observe a warning:

 W tensorflow/core/kernels/data/generator_dataset_op.cc:78] Error occurred when finalizing GeneratorDataset iterator: Failed precondition: Python interpreter state is not initialized. The process may be terminated.

Describe the expected behavior Would expect the program to finish without a warning.

Code to reproduce the issue

import itertools

import tensorflow as tf

def gen():
    for i in itertools.count(1):
        yield (i, [1] * i)

ds = tf.data.Dataset.from_generator(
    gen, (tf.int64, tf.int64), (tf.TensorShape([]), tf.TensorShape([None])))

iterator = ds.make_one_shot_iterator()

sess = tf.Session()
iterator = iterator.get_next()

sess.run(iterator)
# sess.close()   <-- no warning is shown if closing the session explicitly

Other info / logs

2018-12-25 23:43:21.645869: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2018-12-25 23:43:21.684610: W tensorflow/core/kernels/data/generator_dataset_op.cc:78] Error occurred when finalizing GeneratorDataset iterator: Failed precondition: Python interpreter state is not initialized. The process may be terminated.
	 [[{{node PyFunc}} = PyFunc[Tin=[DT_INT64], Tout=[DT_INT64], token="pyfunc_2"](arg0)]]

About this issue

Original URL
State: closed
Created 6 years ago
Reactions: 1
Comments: 24 (15 by maintainers)

Most upvoted comments

@jvishnuvardhan we have this issue in 2.0b1

bionicles on Aug 15, 2019

I’m having the same issue using TF 2.2.0 and Python 3.8.4. I’m trying to run a simplified convnet with the following code:

import numpy as np
import tensorflow as tf
from tensorflow import keras
import os
from keras import backend as K
from keras.models import Sequential
from keras.layers import Activation
from keras.layers.core import Dense, Flatten
from keras.optimizers import Adam
from keras.metrics import categorical_crossentropy
from keras.preprocessing.image import ImageDataGenerator
from keras.layers.normalization import BatchNormalization
from keras.layers.convolutional import Conv2D, MaxPooling2D
from sklearn.metrics import confusion_matrix
import itertools
import matplotlib.pyplot as plt

train_path = os.path.join(os.path.dirname(__file__), "./CNN Data/Training Set")
valid_path = os.path.join(os.path.dirname(__file__), "./CNN Data/Validation Set")
test_path = os.path.join(os.path.dirname(__file__), "./CNN Data/Testing Set")

train_batches = ImageDataGenerator().flow_from_directory(train_path, target_size=(251,388), classes=['Cortex','Medulla', 'Pelvis'], batch_size=6)
valid_batches = ImageDataGenerator().flow_from_directory(valid_path, target_size=(251,388), classes=['Cortex','Medulla', 'Pelvis'], batch_size=7)
test_batches = ImageDataGenerator().flow_from_directory(test_path, target_size=(251,388), classes=['Cortex','Medulla', 'Pelvis'], batch_size=7)

model = keras.Sequential([
    Conv2D(64, (3,3), strides=(1, 1), activation='relu', padding='same', input_shape=(251,388, 1)),
    Conv2D(64, (3,3), activation='relu', padding='same'),
    MaxPooling2D(pool_size=(2,2), strides=(2,2), padding='valid'),
    Flatten(),
    Dense(3, activation='softmax')
])

EPOCHS = 20
INIT_LR = 0.001

model.compile(Adam(learning_rate=INIT_LR), loss='categorical_crossentropy', metrics=['accuracy'])
model.fit(
    train_batches, steps_per_epoch=36, epochs=EPOCHS, verbose=1,
    validation_data=valid_batches, validation_steps=18
)

Running the above code yields the following error message:

tensorflow.python.framework.errors_impl.NotFoundError: No algorithm worked! [[node sequential/conv2d/Conv2D (defined at c:/…/TinyNet/Reproduce the Error.py:43) ]] [Op:__inference_train_function_684]

Function call stack: train_function

2020-07-30 14:39:37.905975: W tensorflow/core/kernels/data/generator_dataset_op.cc:103] Error occurred when finalizing GeneratorDataset iterator: Failed precondition: Python interpreter state is not initialized. The process may be terminated. [[{{node PyFunc}}]]

I’ve played around with this code some more and noticed that removing the input_shape argument from the first conv layer eliminates the “Failed precondition” error, allowing the program to run normally; likewise, putting another layer (e.g. MaxPooling2D) first and including input_shape causes the error to occur. Furthermore, if I run a single image through the network (which involves removing the ImageDataGenerator().flow_from_directory()), the program runs normally even with the input_shape argument in the first layer.

TonyD-BME on Aug 18, 2020

@bionicles Could you open a new issue by filling issue template and also provide a standalone code to reproduce the issue? As the origianal issue with TF1.x was resolved, it is better to open a new issue so that it will help the community that are using TF2.0. Thanks!

jvishnuvardhan on Aug 15, 2019

I meet the same problem in tf 2.2 and python 3.6

I have the same issue as well

Manoj-M-97 on Jun 23, 2020