tensorflow: XNNPACK delegate performs much slower than default TFLite backend if multi-threading is configured according to documentation

System information

• Have I written custom code (as opposed to using a stock example script provided in TensorFlow): Yes
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux Ubuntu 16.04, Android 10
• Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device: Any Android smarthphone
• TensorFlow installed from (source or binary): Source
• TensorFlow version (use command below): 2.3.0
• Python version: -
• Bazel version (if compiling from source): 3.1.0
• GCC/Compiler version (if compiling from source): GCC 5.4.0 / Clang shipped with Android NDK 21
• CUDA/cuDNN version: -
• GPU model and memory: -

When TFLite is built with XNNPACK, performance improvement is expected. However, seems like the code provided in /tensorflow/lite/examples/minimal with a minor change of interpreter settings leads to performance degradation comparing to the default build.

So here is the code taken from minimal example with my changes:

  ...

  // Load model
  std::unique_ptr<tflite::FlatBufferModel> model =
      tflite::FlatBufferModel::BuildFromFile(filename);
  TFLITE_MINIMAL_CHECK(model != nullptr);

  // Build the interpreter
  tflite::ops::builtin::BuiltinOpResolver resolver;
  InterpreterBuilder builder(*model, resolver);
  std::unique_ptr<Interpreter> interpreter;
  builder(&interpreter);
  TFLITE_MINIMAL_CHECK(interpreter != nullptr);

  // Allocate tensor buffers.
  TFLITE_MINIMAL_CHECK(interpreter->AllocateTensors() == kTfLiteOk);
  printf("=== Pre-invoke Interpreter State ===\n");
  tflite::PrintInterpreterState(interpreter.get());

  // Set number of threads (added by me)
  interpreter->SetNumThreads(8);

  // Run inference
  TFLITE_MINIMAL_CHECK(interpreter->Invoke() == kTfLiteOk);
  printf("\n\n=== Post-invoke Interpreter State ===\n");
  tflite::PrintInterpreterState(interpreter.get());

  ...

This code performs slower if executed with TFLite + XNNPACK build. I’ve tested it both on x64 desktop and arm64 Android using ResNet-34 FP32 TFLite model and observed the exact same performance degradation.

I was able to fix the behavior and achieve 30% performance improvement only after I spent few hours with TFLite code and found out that tflite::Interpreter::SetNumThreads is not applied to XNNPACK delegate (maybe to other delegates as well), because XNNPACK delegate is only initialized in builder(&interpreter) with number of threads passed to this invocation and then is not being updated on interpreter->SetNumThreads(8) call! In the case is illustrated by the code above), XNNPACK work in single-threaded mode or so. So the fix is to initialize interpreter as the following:

  builder(&interpreter, 8);

Then XNNPACK really introduces significant performance improvement.

I’m OK with the solution that I found, but I was really confused with this issue and spent almost a day figuring out why can’t I achieve claimed performance, because neither official documentation nor TFLite code commentary does not mention InterpreterBuilder’s num_threads argument as something necessary or mention it at least. Thus, following “Tweak the number of threads” documentation section in combination with using XNNPACK will lead anyone to this pitfall and will result with a very poor performance.

If it’s need I can provide more standalone example, point to the parts of TFLite code, which are responsible for this behavior, and detailed measurements obtained on different devices.