hub: USE latest version can't be used with MirroredStrategy ("Trying to access a placeholder that is not supposed to be executed.")

The latest versions of USE throw this error when used in a MirroredStrategy. (tf: 2.1.0, keras: 2.2.4-tf, hub: 0.7.0)

import tensorflow as tf
import tensorflow.keras as keras
import tensorflow_hub as hub
import tensorflow_text as text

#This USE models fail with " InvalidArgumentError:  assertion failed: [Trying to access a placeholder that is not supposed to be executed"
LM = "https://tfhub.dev/google/universal-sentence-encoder-multilingual-large/3"
#LM = "https://tfhub.dev/google/universal-sentence-encoder-multilingual/3"
#LM = "https://tfhub.dev/google/universal-sentence-encoder-large/5"
#LM = "https://tfhub.dev/google/universal-sentence-encoder/4"
DIM = 512

strategy = tf.distribute.MirroredStrategy()
with strategy.scope():
    model = keras.models.Sequential()
    model.add(
        hub.KerasLayer(LM,
                       output_shape=DIM,
                       input_shape=[],
                       dtype=tf.string)
    )
    model.add(keras.layers.Dense(1, activation='sigmoid'))
    model.compile(optimizer="adam", loss="binary_crossentropy")

model.summary()

Throws:

INFO:tensorflow:Using MirroredStrategy with devices ('/job:localhost/replica:0/task:0/device:GPU:0',)
---------------------------------------------------------------------------
InvalidArgumentError                      Traceback (most recent call last)
<ipython-input-2-6b41a257fb9c> in <module>()
     32                        output_shape=DIM,
     33                        input_shape=[],
---> 34                        dtype=tf.string)
     35     )
     36     model.add(keras.layers.Dense(1, activation='sigmoid'))

12 frames
/usr/local/lib/python3.6/dist-packages/six.py in raise_from(value, from_value)

InvalidArgumentError:  assertion failed: [Trying to access a placeholder that is not supposed to be executed. This means you are executing a graph generated from cross-replica context in an in-replica context.]
	 [[node Assert/Assert (defined at /usr/local/lib/python3.6/dist-packages/tensorflow_hub/module_v2.py:95) ]] [Op:__inference_restored_function_body_45855]

Function call stack:
restored_function_body

Interestingly, NNLM works fine and also USE-large v3 (albeit some warnings, not sure if they affect it’s later performance). Also, if OneDeviceStrategy or no strategy is used everything works.

Check this colab with code and test cases: https://colab.research.google.com/drive/1_YaGYje4tXPyQDx_hYaj9VNLAA5hi3Dg

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 23 (5 by maintainers)

Most upvoted comments

Here is a minimum reproducible example for tf.distribute.MirroredStrategy() with USE latest version which throws up an error.

import tensorflow as  tf
import tensorflow_hub as hub
import numpy as np
import datetime, os
import pandas as pd
import datetime, os
import tensorflow.keras as keras 

## These are my tensorflow , keras , tensorflow_hub and python versions: 

print(tf.__version__, hub.__version__, tf.keras.__version__, sys.version)
## 2.2.0 0.8.0  2.3.0-tf  3.8.0 (default, Nov  6 2019, 21:49:08) 

## Text and labels for classification
text = [

    "SOCCER",
    "CRICKET",
    "BASKETBALL",

    "400 METRES SPRINT",
    "JAVELIN THROW",
    "TRIPLE JUMP",


    "FREESTYLE RELAY",
    "BACKSTROKE",
    "MEDLEY RELAY",

    "SAILING",
    "ROWING",
    "SURFING",
]

label = [0,0,0,1,1,1,2,2,2,3,3,3]

# Definition of hub.KerasLayer with USE large
module_obj = 'https://tfhub.dev/google/universal-sentence-encoder-large/5'
embed =        hub.KerasLayer(module_obj, trainable= True)

def create_model():
    return tf.keras.models.Sequential(
            [
                tf.keras.layers.Input(shape = [], dtype=tf.string),
                embed,
                tf.keras.layers.Dense(4,activation='softmax')]
            )


mirrored_strategy = tf.distribute.MirroredStrategy()
with mirrored_strategy.scope():
    model = create_model()
    model.compile(optimizer = 'adam', loss = 'categorical_crossentropy', metrics = ['accuracy'])

Output after this :

INFO:tensorflow:Using MirroredStrategy with devices ('/job:localhost/replica:0/task:0/device:GPU:0', '/job:localhost/replica:0/task:0/device:GPU:1')
INFO:tensorflow:Using MirroredStrategy with devices ('/job:localhost/replica:0/task:0/device:GPU:0', '/job:localhost/replica:0/task:0/device:GPU:1')
ValueError: Variable (<tf.Variable 'Embeddings/sharded_0:0' shape=(13334, 320) dtype=float32, numpy=
array([[-1.1641915e+00, -4.0162230e+00, -1.0308318e+00, ...,
1.1064901e-01, 2.9283254e+00, 1.4386557e+00],
[-3.7163803e-01, -1.8228565e-01, -4.0679860e-01, ...,
1.6463491e-01, -1.2669672e-01, -1.8727861e-01],
[ 6.4132787e-02, 2.4964781e-01, -5.7500858e-02, ...,
2.7740359e-01, -7.3342794e-01, -1.5283586e-01],
...,
[-4.7713269e-02, 1.7435255e-02, 1.7971721e-01, ...,
-2.6605502e-02, -6.8220109e-02, 5.4901250e-02],
[-8.2942925e-02, 8.3685674e-02, 4.9772050e-02, ...,
9.7135836e-03, -3.4118034e-02, -7.6729544e-03],
[-1.7106453e-02, 9.3901977e-02, -1.6374167e-02, ...,
4.9962573e-02, 9.2947654e-02, -1.7278243e-03]], dtype=float32)>) was not created in the distribution strategy scope of (<tensorflow.python.distribute.mirrored_strategy.MirroredStrategy object at 0x7f79a45c9790>). It is most likely due to not all layers or the model or optimizer being created outside the distribution strategy scope. Try to make sure your code looks similar to the following.
with strategy.scope():
model=_create_model()
model.compile(...)

@arnoegw , @MorganR : Any insights on how to resolve this error.

freshforlife on Jun 11, 2021

Hi all, the issue was because this model was converted from TF1 SavedModel to TF2 SavedModel. There’s some tricky issues with loading such models under tf.distribute.Strategy. The team is working on a TF2 native version of the model.

crccw on Jul 10, 2020

Confirmed: there is an issue. Thank you, @eduardofv, for the very clear and reproducible report!

@mercarikaicheung, there is no useful update yet, sorry.

arnoegw on Mar 11, 2020