keras: Doesn't work with cudnn v7.1.1.5

@taneta Thanks! that fixed my problem using the downgrade (cudnn 7.1 to 7.0) you gave, here’s just a paste of it with the typos $=s and 9.1=9.0 fix

now my code in docker works with tf 1.6 cuda 9.0 and cudnn 7.0

============================================

USER root

RUN apt-get update && apt-get install -y --allow-downgrades --no-install-recommends
libcudnn7=7.0.5.15-1+cuda9.0 libcudnn7-dev=7.0.5.15-1+cuda9.0 && rm -rf /var/lib/apt/lists/*

RUN apt-get update

My env:windows-10 64 bit +python 3.5+ visual studio 2017 community + cuda toolkit 9.0+cuDNN v7.1.1 (Feb 28, 2018), for CUDA 9.0+tensorflow 1.6-gpu
My failure is

2018-03-13 20:44:03.100572: E C:\tf_jenkins\workspace\rel-win\M\windows-gpu\PY\35\tensorflow\stream_executor\cuda\cuda_dnn.cc:378] Loaded runtime CuDNN library: 7101 (compatibility version 7100) but source was compiled with 7003 (compatibility version 7000).  If using a binary install, upgrade your CuDNN library to match.  If building from sources, make sure the library loaded at runtime matches a compatible version specified during compile configuration.

replace Download cuDNN v7.1.1 (Feb 28, 2018), for CUDA 9.0 by cuDNN v7.0.5 (Dec 5, 2017), for CUDA 9.0 is ok.

So new is windows-10 64 bit +python 3.5+ visual studio 2017 community + cuda toolkit 9.0+cuDNN v7.0.5 (Dec 5, 2017), for CUDA 9.0+tensorflow 1.6-gpu

@zhaoyang10 Yes, I understand that this problem because of cudnn version. What I don’t know is how to refer to nvidia/cuda Docker image with cuDNN v7.0.x.

Keras Dockerfile is built on of top Nvidia container: ARG cuda_version=9.0 ARG cudnn_version=7 FROM nvidia/cuda:${cuda_version}-cudnn${cudnn_version}-devel

But nvidia/cuda:$9.0-cudnn7-devel container now includes cuDNN v7.1.1.5 version and I don’t now how to request the earlier one.

tensorflow-gpu==1.5.0 to tensorflow-gpu==1.9.0 solved in my case

@taneta I ran into the same problem running nvidia-docker after upgrading to CUDA 9.1 and CUDNN 7.1. Here is how I was able to fix the issue:

As @zhaoyang10 mentioned, you need to downgrade CUDNN.

First, you can search for available versions using apt-cache madison libcudnn7. Pick an appropriate version (7.0.x) and then run the following command to downgrade, replacing the CUDNN version with your chosen one (I used an official NVIDIA docker file as a reference):

apt-get update && apt-get install -y --allow-downgrades --no-install-recommends \ 
    libcudnn7=7.0.5.15-1+cuda9.1 \
    libcudnn7-dev=7.0.5.15-1+cuda9.1 && \
    rm -rf /var/lib/apt/lists/*

(Then run a quick apt-get update to refresh the lists)

Hope that helps!

I actually have the same problem as you @taneta. Have you found a solution, yet?

It’s a problem about versions. You can solve this by installing cuDNN v7.0.x.

tensorflow-gpu==1.5.0 to tensorflow-gpu==1.9.0 solved in my case

Thanks very much . when I change cudnn , it still happen same issue. when I change version of tensorflow 1.5 to 1.9 ,It solved. thanks

still I’m comfused that I need a lower version cudnn . why update TF 1.5 t 1.9 can solve problem . I thought 1.5 should to 1.3

so i have the same error and i ftry to use this :apt-get install -y --allow-downgrades --no-install-recommends libcudnn7=7.0.5.15-1+cuda9.0 libcudnn7-dev=7.0.5.15-1+cuda9.0 && rm -rf /var/lib/apt/lists/* But it doesn’t work . Error said: can’t find lib 7.0.5.15-1. So i realize that error happened just because of my libcudnn version . So i remove the early version and install libcudnn7.0.5.15-1 and it’s work. i remove the old version by : rm /usr/local/cuda/include/cudnn.h rm /usr/local/cuda/lib64/libcudnn* ( cuda maybe rename like cuda9.0 or something else but it’s in /usr/local/…) so then i install that again with different version : http://docs.nvidia.com/deeplearning/sdk/cudnn-install/index.html So here is my solution. Hope it can help 😄