hub: Segmentation fault running universal-sentence-encoder-multilingual-qa example code

Ubuntu 18.04
python3.6

seaborn               0.9.
tensorboard           1.14.0       
tensorflow-estimator  1.14.0       
tensorflow-gpu        1.14.0       
tensorflow-hub        0.5.0 
tf-sentencepiece      0.1.82.1

I am getting a seg fault when I run the example code from https://tfhub.dev/google/universal-sentence-encoder-multilingual-qa/1

import tensorflow as tf
import tensorflow_hub as hub
import numpy as np
import tf_sentencepiece

questions = ["What is your age?"]
responses = ["I am 20 years old.", "good morning"]
response_contexts = ["I will be 21 next year.", "great day."]

# Set up graph.
g = tf.Graph()
with g.as_default():
  print('About to segfault')
  module = hub.Module("https://tfhub.dev/google/universal-sentence-encoder-multilingual-large-qa/1")
  print('If you see me the segfault didn't happen')
  question_embeddings = module(
    dict(input=questions),
    signature="question_encoder", as_dict=True)

  response_embeddings = module(
    dict(input=responses,
         context=response_contexts),
    signature="response_encoder", as_dict=True)

  init_op = tf.group([tf.global_variables_initializer(), tf.tables_initializer()])
g.finalize()

# Initialize session.
session = tf.Session(graph=g)
session.run(init_op)

# Compute embeddings.
question_results = session.run(question_embeddings)
response_results = session.run(response_embeddings)

np.inner(question_results["outputs"], response_result["outputs"])

The fault seems to be thrown during or after download of the model on this line: module = hub.Module("https://tfhub.dev/google/universal-sentence-encoder-multilingual-qa/1")

Resulting output:

/home/myname/tensorflow3/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:516: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
/home/myname/tensorflow3/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:517: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/home/myname/tensorflow3/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:518: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
/home/myname/tensorflow3/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:519: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/home/myname/tensorflow3/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:520: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint32 = np.dtype([("qint32", np.int32, 1)])
/home/myname/tensorflow3/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  np_resource = np.dtype([("resource", np.ubyte, 1)])
/home/myname/tensorflow3/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:541: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
/home/myname/tensorflow3/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:542: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/home/myname/tensorflow3/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:543: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
/home/myname/tensorflow3/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:544: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/home/myname/tensorflow3/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:545: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint32 = np.dtype([("qint32", np.int32, 1)])
/home/myname/tensorflow3/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:550: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  np_resource = np.dtype([("resource", np.ubyte, 1)])
WARNING: Logging before flag parsing goes to stderr.
W0807 07:50:11.738195 139921118234432 deprecation_wrapper.py:119] From /home/myname/tensorflow3/lib/python3.6/site-packages/tf_sentencepiece/sentencepiece_processor_ops.py:259: The name tf.NotDifferentiable is deprecated. Please use tf.no_gradient instead.

About to segfault
Segmentation fault (core dumped)

Here is what comes up in /var/log/kern.log when the code is executed:

Aug  7 07:10:17 mycomputer kernel: [70181.395777] audit: type=1400 audit(1565172617.110:117): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="/usr/bin/evince-previewer" pid=8330 comm="apparmor_parser"
Aug  7 07:10:17 mycomputer kernel: [70181.410178] audit: type=1400 audit(1565172617.126:118): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="/usr/bin/evince-previewer//sanitized_helper" pid=8330 comm="apparmor_parser"
Aug  7 07:10:17 mycomputer kernel: [70181.411100] audit: type=1400 audit(1565172617.126:119): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="/usr/bin/evince-thumbnailer" pid=8330 comm="apparmor_parser"
Aug  7 07:11:28 mycomputer kernel: [70252.500375] printk: systemd: 35 output lines suppressed due to ratelimiting
Aug  7 07:11:53 mycomputer kernel: [70277.764820] audit: type=1400 audit(1565172713.474:120): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="libreoffice-oopslash" pid=16867 comm="apparmor_parser"
Aug  7 07:11:53 mycomputer kernel: [70277.789818] audit: type=1400 audit(1565172713.502:121): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="libreoffice-senddoc" pid=16871 comm="apparmor_parser"
Aug  7 07:11:53 mycomputer kernel: [70278.250343] audit: type=1400 audit(1565172713.962:122): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="libreoffice-soffice" pid=16875 comm="apparmor_parser"
Aug  7 07:11:53 mycomputer kernel: [70278.274262] audit: type=1400 audit(1565172713.986:123): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="libreoffice-soffice//gpg" pid=16875 comm="apparmor_parser"
Aug  7 07:11:54 mycomputer kernel: [70278.305612] audit: type=1400 audit(1565172714.018:124): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="libreoffice-xpdfimport" pid=16879 comm="apparmor_parser"
Aug  7 07:41:33 mycomputer kernel: [72057.574792] perf: interrupt took too long (3180 > 3163), lowering kernel.perf_event_max_sample_rate to 62750
Aug  7 07:50:13 mycomputer kernel: [72577.373066] python[21789]: segfault at 30 ip 00007f418d2b087d sp 00007ffd0197bfe0 error 4 in _sentencepiece_processor_ops.so.1.14.0[7f418d1fd000+df000]
Aug  7 07:50:13 mycomputer kernel: [72577.373073] Code: 55 b9 01 00 00 00 41 54 55 49 89 fd 53 48 89 f3 48 81 ec c0 01 00 00 64 48 8b 04 25 28 00 00 00 48 89 84 24 b8 01 00 00 31 c0 <48> 8b 46 30 48 c7 44 24 08 00 00 00 00 48 8d 6c 24 30 48 c7 44 24

About this issue

Original URL
State: closed
Created 5 years ago
Comments: 22 (8 by maintainers)

Most upvoted comments

sentencepiece==0.1.83
tf-sentencepiece==0.1.3

As described above, you need

tf-sentencepiece==0.1.83

vbardiovskyg on Aug 27, 2019

Thanks, @vbardiovskyg,

The code that I am running is the one shared by Google here: https://tfhub.dev/google/universal-sentence-encoder-multilingual-large/1

I haven’t made any changes to the code.

I am running a fresh VM from google cloud platform with Ubuntu 18.04.

Sometimes the code gets stuck at this line: embed = hub.Module("https://tfhub.dev/google/universal-sentence-encoder-multilingual-large/1") and it never comes out of it, and at times I get errors like: tensorflow.python.framework.errors_impl.NotFoundError: Graph ops missing from the python registry ({'SentencepieceEncodeSparse'}) are also absent from the c++ registry.

Package Versions:

sentencepiece==0.1.83
tf-sentencepiece==0.1.3
tf-estimator-nightly==1.14.0.dev2019060501
tensorboard==1.14.0
tensorflow==1.14.0
tensorflow-estimator==1.14.0
tensorflow-hub==0.5.0

I have also tried with tensorflow==1.13.1 It doesn’t work 😦

JawadAr on Aug 27, 2019

Does it solve your issue if you also carefully uninstall tensorflow and install again?

!pip uninstall tf-sentencepiece
!pip uninstall tensorflow

!pip install tf-sentencepiece 
!pip install tensorflow

This works for me on Google colab.

vbardiovskyg on Aug 23, 2019

Did you

import tf_sentencepiece

vbardiovskyg on Aug 23, 2019

@vbardiovskyg It is running on Google Colab but it is giving this error (NotFoundError: Graph ops missing from the python registry ({‘SentencepieceEncodeSparse’}) are also absent from the c++ registry.) on a google VM. Can you please direct me to the solution? I have searched everywhere for this but cannot find any solution, thanks.

JawadAr on Aug 23, 2019

Correction: It appears downgrading to tensorflow[-gpu] 1.13.1 or 1.12.0 is sufficient, with no change in version of tf-sentencepiece.

arnoegw on Aug 7, 2019

Confirmed for the given PIP packages (with or without -gpu), with the following minimal repro:

import tensorflow as tf
import tensorflow_hub as hub
import tf_sentencepiece
module = hub.Module("https://tfhub.dev/google/universal-sentence-encoder-multilingual-qa/1")

Turns out this is an issue between tensorflow 1.14.0 and tf-sentencepiece 0.1.82.1 that is already tracked internally at Google. The issue can be reproduced without any use of the tensorflow_hub library or TF Hub modules.

Until this is fixed, the only advice I can forward is to go back to PIP packages tf-sentencepiece 0.1.82 (no trailing .1) and a compatible TF version, such as tensorflow[-gpu] 1.13.1 (not .2). I am going to update the module documentation accordingly.

arnoegw on Aug 16, 2019