mmdetection: FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmpautvz0c6/tmp_8n_2d_c.py'

Checklist

1. I have searched related issues but cannot get the expected help.
2. The bug has not been fixed in the latest version.

Describe the bug I have trained and tested successfully before, but after a break of connection with SSH server, I couldn’t train or test anymore. The break happened while training a model.

Environment

sys.platform: linux Python: 3.7.7 (default, May 7 2020, 21:25:33) [GCC 7.3.0] CUDA available: True CUDA_HOME: /lustre/opt/cascadelake/linux-centos7-x86_64/gcc-4.8.5/cuda-10.1.243-42lx5ihyhh6h27eu2z2ft5w4lyotgvy6 NVCC: Cuda compilation tools, release 10.1, V10.1.243 GPU 0: Tesla V100-SXM3-32GB GCC: gcc (Spack GCC) 7.4.0 PyTorch: 1.4.0 PyTorch compiling details: PyTorch built with:

• GCC 7.3
• Intel® Math Kernel Library Version 2020.0.0 Product Build 20191122 for Intel® 64 architecture applications
• Intel® MKL-DNN v0.21.1 (Git Hash 7d2fd500bc78936d1d648ca713b901012f470dbc)
• OpenMP 201511 (a.k.a. OpenMP 4.5)
• NNPACK is enabled
• CUDA Runtime 10.1
• NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_37,code=compute_37
• CuDNN 7.6.3
• Magma 2.5.1
• Build settings: BLAS=MKL, BUILD_NAMEDTENSOR=OFF, BUILD_TYPE=Release, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -fopenmp -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -O2 -fPIC -Wno-narrowing -Wall -Wextra -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Wno-stringop-overflow, DISABLE_NUMA=1, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, USE_CUDA=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_STATIC_DISPATCH=OFF,

TorchVision: 0.5.0 OpenCV: 4.2.0 MMCV: 0.5.1 MMDetection: 2.0.0+514a644 MMDetection Compiler: GCC 7.4 MMDetection CUDA Compiler: 10.1

Error traceback If applicable, paste the error trackback here.

Traceback (most recent call last):
  File "./tools/train.py", line 159, in <module>
    main()
  File "./tools/train.py", line 68, in main
    cfg = Config.fromfile(args.config)
  File "/lustre/home/acct-eejxh/eejxh/.local/lib/python3.7/site-packages/mmcv/utils/config.py", line 165, in fromfile
    cfg_dict, cfg_text = Config._file2dict(filename)
  File "/lustre/home/acct-eejxh/eejxh/.local/lib/python3.7/site-packages/mmcv/utils/config.py", line 125, in _file2dict
    _cfg_dict, _cfg_text = Config._file2dict(osp.join(cfg_dir, f))
  File "/lustre/home/acct-eejxh/eejxh/.local/lib/python3.7/site-packages/mmcv/utils/config.py", line 95, in _file2dict
    mod = import_module(temp_module_name)
  File "/lustre/home/acct-eejxh/eejxh/.conda/envs/mmdetection/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 724, in exec_module
  File "<frozen importlib._bootstrap_external>", line 860, in get_code
  File "<frozen importlib._bootstrap_external>", line 791, in source_to_code
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
ValueError: source code string cannot contain null bytes
Exception ignored in: <function _TemporaryFileCloser.__del__ at 0x2ad6a75a20e0>
Traceback (most recent call last):
  File "/lustre/home/acct-eejxh/eejxh/.conda/envs/mmdetection/lib/python3.7/tempfile.py", line 448, in __del__
    self.close()
  File "/lustre/home/acct-eejxh/eejxh/.conda/envs/mmdetection/lib/python3.7/tempfile.py", line 444, in close
    unlink(self.name)
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmpautvz0c6/tmp_8n_2d_c.py'

Bug fix If you have already identified the reason, you can provide the information here. If you are willing to create a PR to fix it, please also leave a comment here and that would be much appreciated!