tensorflow-directml: dml_command_recorder.cc:366 Check failed: (((HRESULT)((dml_device_->GetDeviceRemovedReason()))) >= 0) == true (0 vs. 1
System Information
- Windows 10 Version 2004 / Build 19041.508)
- WSL2 cat /etc/debian_version > 10.0
- Python 3.7.7
- TensorFlow-DirectML Version tensorflow_directml-1.15.3.dev200911-cp37-cp37m-win_amd64.whl -GPU AMD Radeon VII - rn-rad-win-20-20-01-05
Host System
--------------------------------------------------------------------------------
Windows 10 Version : Windows 10 Home 64-bit (10.0, Build 19041) (19041.vb_release.191206-1406)
Processor : AMD Ryzen 7 3700X 8-Core Processor (16 CPUs), ~3.6GHz
Memory : 65536MB RAM
DirectX Version : DirectX 12
Python Environment
--------------------------------------------------------------------------------
Python Version : 3.7.7
TensorFlow-DirectML : 1.15.3.dev200911
DirectX Device
--------------------------------------------------------------------------------
Description : AMD Radeon VII
Manufacturer : Advanced Micro Devices, Inc.
Chip Type : AMD Radeon Graphics Processor (0x66AF)
Dedicated Memory : 33270 MB
Driver Version : 27.20.2001.5003
Driver Model : WDDM 2.7
Driver Date : 29/06/2020 03:00:00
Feature Levels : 12_1,12_0,11_1,11_0,10_1,10_0,9_3,9_2,9_1
DirectX Device
--------------------------------------------------------------------------------
Description : AMD Radeon VII
Manufacturer : Advanced Micro Devices, Inc.
Chip Type : AMD Radeon Graphics Processor (0x66AF)
Dedicated Memory : 33270 MB
Driver Version : 27.20.2001.5003
Driver Model : WDDM 2.7
Driver Date : 29/06/2020 03:00:00
Feature Levels : 12_1,12_0,11_1,11_0,10_1,10_0,9_3,9_2,9_1
Repro Details
In [1]: from ai_benchmark import AIBenchmark
In [2]: benchmark = AIBenchmark(use_CPU=None, verbose_level=1)
>> AI-Benchmark-v.0.1.2
>> Let the AI Games begin..
In [3]: results = benchmark.run()
Describe the current behavior
14/19. ICNet
14.1 - inference | batch=5, size=1024x1536: 330 ± 6 ms
2020-09-15 10:35:21.207385: F tensorflow/core/common_runtime/dml/dml_command_recorder.cc:366] Check failed: (((HRESULT)((dml_device_->GetDeviceRemovedReason()))) >= 0) == true (0 vs. 1)
Describe the expected behavior bench passed successfully.
Code to reproduce the issue –>Repro Details
Other info / logs
conda list
(directml) PS F:\DSML\GPT2\gpt-2> conda list
# packages in environment at F:\DSML\Soft\Anaconda\envs\directml:
#
# Name Version Build Channel
_tflow_select 2.2.0 eigen
absl-py 0.9.0 py37_0
ai-benchmark 0.1.2 pypi_0 pypi
astor 0.8.0 py37_0
attrs 19.3.0 py_0
backcall 0.1.0 py37_0
blas 1.0 mkl
bleach 3.1.4 py_0
blinker 1.4 py37_0
brotlipy 0.7.0 py37he774522_1000
ca-certificates 2020.6.20 hecda079_0 conda-forge
cachetools 4.1.0 py_1
certifi 2020.6.20 py37hc8dfbb8_0 conda-forge
cffi 1.14.0 py37h7a1dbc1_0
chardet 3.0.4 py37_1003
click 7.1.2 py_0
colorama 0.4.3 py_0
cryptography 2.9.2 py37h7a1dbc1_0
cycler 0.10.0 pypi_0 pypi
decorator 4.4.2 py_0
defusedxml 0.6.0 py_0
entrypoints 0.3 py37_0
fire 0.3.1 pyh9f0ad1d_0 conda-forge
gast 0.2.2 py37_0
google-auth 1.17.2 py_0
google-auth-oauthlib 0.4.1 py_2
google-pasta 0.2.0 py_0
grpcio 1.27.2 py37h351948d_0
h5py 2.10.0 py37h5e291fa_0
hdf5 1.10.4 h7ebc959_0
icc_rt 2019.0.0 h0cc432a_1
idna 2.10 py_0
importlib_metadata 1.5.0 py37_0
intel-openmp 2020.1 216
ipykernel 5.1.4 py37h39e3cac_0
ipython 7.13.0 py37h5ca1d4c_0
ipython_genutils 0.2.0 py37_0
jedi 0.17.0 py37_0
jinja2 2.11.2 py_0
jsonschema 3.2.0 py37_0
jupyter_client 6.1.3 py_0
jupyter_contrib_core 0.3.3 py_2 conda-forge
jupyter_contrib_nbextensions 0.5.1 py37_0 conda-forge
jupyter_core 4.6.3 py37_0
jupyter_highlight_selected_word 0.2.0 py37_1000 conda-forge
jupyter_latex_envs 1.4.4 py37_1000 conda-forge
jupyter_nbextensions_configurator 0.4.1 py37hc8dfbb8_1 conda-forge
jupyterthemes 0.20.0 pypi_0 pypi
keras-applications 1.0.8 py_1
keras-preprocessing 1.1.0 py_1
kiwisolver 1.2.0 pypi_0 pypi
lesscpy 0.14.0 pypi_0 pypi
libiconv 1.15 hfa6e2cd_1006 conda-forge
libprotobuf 3.12.3 h7bd577a_0
libsodium 1.0.16 h9d3ae62_0
libxml2 2.9.10 h5d81f1c_2 conda-forge
libxslt 1.1.33 h579f668_1 conda-forge
lxml 4.5.2 py37h8ba8a40_0 conda-forge
m2w64-gcc-libgfortran 5.3.0 6
m2w64-gcc-libs 5.3.0 7
m2w64-gcc-libs-core 5.3.0 7
m2w64-gmp 6.1.0 2
m2w64-libwinpthread-git 5.0.0.4634.697f757 2
markdown 3.1.1 py37_0
markupsafe 1.1.1 py37he774522_0
matplotlib 3.3.0 pypi_0 pypi
mistune 0.8.4 py37he774522_0
mkl 2020.1 216
mkl-service 2.3.0 py37hb782905_0
mkl_fft 1.1.0 py37h45dec08_0
mkl_random 1.1.1 py37h47e9c7a_0
msys2-conda-epoch 20160418 1
nbconvert 5.6.1 py37_0
nbformat 5.0.6 py_0
notebook 6.0.3 py37_0
numpy 1.18.5 py37h6530119_0
numpy-base 1.18.5 py37hc3f5095_0
oauthlib 3.1.0 py_0
openssl 1.1.1g he774522_1 conda-forge
opt_einsum 3.1.0 py_0
pandoc 2.2.3.2 0
pandocfilters 1.4.2 py37_1
parso 0.7.0 py_0
pickleshare 0.7.5 py37_0
pillow 7.2.0 pypi_0 pypi
pip 20.0.2 py37_3
ply 3.11 pypi_0 pypi
powershell_shortcut 0.0.1 3
prometheus_client 0.7.1 py_0
prompt-toolkit 3.0.4 py_0
prompt_toolkit 3.0.4 0
protobuf 3.12.3 py37h33f27b4_0
psutil 5.7.2 pypi_0 pypi
py-cpuinfo 7.0.0 pypi_0 pypi
pyasn1 0.4.8 py_0
pyasn1-modules 0.2.7 py_0
pycparser 2.20 py_2
pygments 2.6.1 py_0
pyjwt 1.7.1 py37_0
pyopenssl 19.1.0 py_1
pyparsing 2.4.7 pypi_0 pypi
pyreadline 2.1 py37_1
pyrsistent 0.16.0 py37he774522_0
pysocks 1.7.1 py37_1
python 3.7.7 h81c818b_4
python-dateutil 2.8.1 py_0
python_abi 3.7 1_cp37m conda-forge
pywin32 227 py37he774522_1
pywinpty 0.5.7 py37_0
pyyaml 5.3.1 py37h8055547_0 conda-forge
pyzmq 18.1.1 py37ha925a31_0
regex 2020.7.14 py37h4ab8f01_0 conda-forge
requests 2.24.0 py_0
requests-oauthlib 1.3.0 py_0
rsa 4.0 py_0
scipy 1.5.0 py37h9439919_0
send2trash 1.5.0 py37_0
setuptools 46.4.0 py37_0
six 1.14.0 py37_0
sqlite 3.31.1 h2a8f88b_1
tensorboard 1.15.0 pypi_0 pypi
tensorboard-plugin-wit 1.6.0 py_0
tensorflow 1.15.0 eigen_py37h9f89a44_0
tensorflow-base 1.15.0 eigen_py37h07d2309_0
tensorflow-directml 1.15.3.dev200911 pypi_0 pypi
tensorflow-estimator 1.15.1 pyh2649769_0
termcolor 1.1.0 py37_1
terminado 0.8.3 py37_0
testpath 0.4.4 py_0
tornado 6.0.4 py37he774522_1
tqdm 4.48.2 pyh9f0ad1d_0 conda-forge
traitlets 4.3.3 py37_0
urllib3 1.25.9 py_0
vc 14.1 h0510ff6_4
vs2015_runtime 14.16.27012 hf0eaf9b_2
wcwidth 0.1.9 py_0
webencodings 0.5.1 py37_1
werkzeug 0.16.1 py_0
wheel 0.34.2 py37_0
win_inet_pton 1.1.0 py37_0
wincertstore 0.2 py37_0
winpty 0.4.3 4
wrapt 1.12.1 py37he774522_1
yaml 0.2.5 he774522_0 conda-forge
zeromq 4.3.1 h33f27b4_3
zipp 3.1.0 py_0
zlib 1.2.11 h62dcd97_4
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Comments: 15 (5 by maintainers)
I also ran into this problem with ai_benchmark. My GPU is Rx 580.
Thanks for this @Herobring . We’ll investigate and come back to you.