tensorflow: TF 2.4.0 crashes on startup with IndexError assuming that sys.argv[0] exists when it may be hosted by C++ (regression from 2.3.1)

System information

  • Have I written custom code (as opposed to using a stock example script provided in TensorFlow): Yes
  • OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Windows 10, Linux Debian 10
  • TensorFlow installed from (source or binary): pip install tensorflow-gpu==2.4.0
  • TensorFlow version (use command below): v2.4.0-rc4-71-g582c8d236cb 2.4.0
  • Python version: 3.7.9
  • CUDA/cuDNN version: 11.0, 8.0.4 for 11.0
  • GPU model and memory: NVIDIA GTX 1080

Describe the current behavior

TF 2.4.0 crashes on startup with:

File "...\site-packages\tensorflow\python\distribute\combinations.py", line 159, in GPUCombination
GPU_TEST = re.search(r"(test_gpu|test_xla_gpu)$", sys.argv[0])
IndexError: list index out of range

Describe the expected behavior sys.argv is checked to be non-empty before indexing (no bug on TF 2.3.1)

Standalone code to reproduce the issue

  1. Ensure that PYTHONHOME is set
  2. Py_Initialize()
  3. Append our module’s directory to the system path: (PyImport_ImportModule(“sys”), PyObject_GetAttrString(sys, “path”), PyUnicode_FromString(“…”), PyList_Append(sysPath, …))
  4. Import our module: PyImport_ImportModule(“…”)
  5. Our module calls “import tensorflow as tf”, which then crashes

Python-only mock repro

import sys
sys.argv.clear()
import tensorflow as tf

Workaround

# Work around https://github.com/tensorflow/tensorflow/issues/45994
import sys
if not sys.argv:
  sys.argv.append("(C++)")

import tensorflow as tf

Notes

It may help to write a unit test/build verification test that mocks this scenario by clearing sys.argv before importing tensorflow.

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Comments: 15 (4 by maintainers)

Commits related to this issue

Most upvoted comments

This repro, importing tensorflow in embedded python, seems fixed (via https://github.com/tensorflow/tensorflow/commit/c8cb67f4c27ec312d228a344ed124bb34cbee74b, @scdub). Thanks!