tensorflow: TensorFlow Lite causes segementation faults in C++ when creating threads if fully statically linked

System information

• Linux Ubuntu 18.04
• TensorFlow Binary (via pip)
• Tested on 2.1.0, 2.2.0, 2.3.0 & all recent master branches of tensorflow/tensorflow:devel up to my last comment
  • I have a local CI script tracking this issue
• Tested on x86_64 as well as ARM builds

Problem

• Segmentaion fault on invoke() when running any model that spawns threads
  • Only occurs if TensorFlow Lite library has been statically linked
  • Not an issue in older versions (pre-2.0.0)

Problem Reproduction

• Standard Tensorflow Lite library build
  • Spin up tensorflow/tensorflow:devel docker container
  • cd tensorflow_src
  • ./tensorflow/lite/tools/make/download_dependencies.sh
  • ./tensorflow/lite/tools/make/build_lib.sh
  • Grab TensorFlow and Flatbuffer headers, as well as Tensorflow .a file
• Compile tflite c++ minimal example
  • GCC 7.5
  • Static
    • g++ minimal.cc -I../tensorflow_headers -I../flatbuffers_include -Wl,--whole-archive -lpthread -Wl,--no-whole-archive -pthread -g -static -L../tensorflow_lib -ltensorflow-lite -ldl -o tflite_min_static
  • Dynamic
    • g++ minimal.cc -I../tensorflow_headers -I../flatbuffers_include -Wl,--whole-archive -lpthread -Wl,--no-whole-archive -pthread -g -L../tensorflow_lib -ltensorflow-lite -ldl -o tflite_min_dynamic
• Make some test models
  • Convolutional model creates extra threads, non-convolutional model does not

import tensorflow as tf
from tensorflow.keras.layers import Conv2D, Flatten, Input, Dense
from tensorflow.keras.models import Model

### With a convolution (creates threads on invoke())
inputs = Input(shape=[10, 5, 1])
x = inputs
x = Conv2D(32, (3, 3))(x)
x = Flatten()(x)
x = Dense(1)(x)

model = Model(inputs, x)
model.compile()

# Convert the model.
converter = tf.lite.TFLiteConverter.from_keras_model(model)
converter.optimizations = [tf.lite.Optimize.DEFAULT]
tflite_model = converter.convert()

# Save the TF Lite model.
with tf.io.gfile.GFile("min_model_with_conv.tflite", "wb") as f:
    f.write(tflite_model)

### Without convolution (no extra threading)
inputs = Input(shape=[10, 5, 1])
x = inputs
# x = Conv2D(32, (3, 3))(x)
x = Flatten()(x)
x = Dense(1)(x)

model = Model(inputs, x)
model.compile()

# Convert the model.
converter = tf.lite.TFLiteConverter.from_keras_model(model)
converter.optimizations = [tf.lite.Optimize.DEFAULT]
tflite_model = converter.convert()

# Save the TF Lite model.
with tf.io.gfile.GFile("min_model_no_conv.tflite", "wb") as f:
    f.write(tflite_model)

• Run minimal examples
  • tflite_min_dynamic min_model_no_conv.tflite: pass
  • tflite_min_dynamic min_model_with_conv.tflite: pass
  • tflite_min_static min_model_no_conv.tflite: pass
  • tflite_min_static min_model_with_conv.tflite: segmentation fault
• From gdb --args tflite_min_static min_model_with_conv.tflite

Node   2 Operator Builtin Code   9 FULLY_CONNECTED
  Inputs: 6 2 -1
  Outputs: 7
[New Thread 0x7fffff7b0700 (LWP 2331)]
[New Thread 0x7ffffefa0700 (LWP 2332)]

Thread 2 "tflite_min_stat" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffff7b0700 (LWP 2331)]
0x0000000000000000 in ?? ()

Bazel Reproduction Using the minimal example above, the problem can be reproduced under Bazel by adding the following build rule:

cc_binary(
    name = "tflite_minimal_bazel",
    srcs = ["minimal.cc"],
    deps = [
        ":framework",
        "//tensorflow/lite/kernels:builtin_ops_all_linked", # For kernels/register.h
    ],
    features = ["fully_static_link"], # For full static link of standard library
    linkstatic=True,
)

Files for Reproduction See here.