tensorflow: TF 2.4.0 crashes on startup with IndexError assuming that sys.argv[0] exists when it may be hosted by C++ (regression from 2.3.1)
System information
- Have I written custom code (as opposed to using a stock example script provided in TensorFlow): Yes
- OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Windows 10, Linux Debian 10
- TensorFlow installed from (source or binary): pip install tensorflow-gpu==2.4.0
- TensorFlow version (use command below): v2.4.0-rc4-71-g582c8d236cb 2.4.0
- Python version: 3.7.9
- CUDA/cuDNN version: 11.0, 8.0.4 for 11.0
- GPU model and memory: NVIDIA GTX 1080
Describe the current behavior
TF 2.4.0 crashes on startup with:
File "...\site-packages\tensorflow\python\distribute\combinations.py", line 159, in GPUCombination
GPU_TEST = re.search(r"(test_gpu|test_xla_gpu)$", sys.argv[0])
IndexError: list index out of range
Describe the expected behavior sys.argv is checked to be non-empty before indexing (no bug on TF 2.3.1)
Standalone code to reproduce the issue
- Ensure that PYTHONHOME is set
- Py_Initialize()
- Append our module’s directory to the system path: (PyImport_ImportModule(“sys”), PyObject_GetAttrString(sys, “path”), PyUnicode_FromString(“…”), PyList_Append(sysPath, …))
- Import our module: PyImport_ImportModule(“…”)
- Our module calls “import tensorflow as tf”, which then crashes
Python-only mock repro
import sys
sys.argv.clear()
import tensorflow as tf
Workaround
# Work around https://github.com/tensorflow/tensorflow/issues/45994
import sys
if not sys.argv:
sys.argv.append("(C++)")
import tensorflow as tf
Notes
It may help to write a unit test/build verification test that mocks this scenario by clearing sys.argv before importing tensorflow.
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Comments: 15 (4 by maintainers)
Commits related to this issue
- * avoiding tensorflow bug (see https://github.com/tensorflow/tensorflow/issues/45994#issue-775109542) — committed to mooch443/trex by mooch443 3 years ago
This repro, importing tensorflow in embedded python, seems fixed (via https://github.com/tensorflow/tensorflow/commit/c8cb67f4c27ec312d228a344ed124bb34cbee74b, @scdub). Thanks!