serving: configure in Docker container ends with java.lang.RuntimeException: Unrecoverable error while evaluating node

Hi, I’m on Mac OS and trying to get into the serving tutorials running a docker container. I am following these steps.

When running configure (using all default options) it stops with a Runtime Exception (see below).

This line seems to be a bit strange to me but i have no idea what to do:

REPOSITORY_DIRECTORY:@jpeg' (requested by nodes 'REPOSITORY:@jpeg

Any help is appreciated!

root@a9c8b25976dd:/serving/tensorflow# ./configure 
Please specify the location of python. [Default is /usr/bin/python]: 
Please specify optimization flags to use during compilation [Default is -march=native]: 
Do you wish to use jemalloc as the malloc implementation? [Y/n] 
jemalloc enabled
Do you wish to build TensorFlow with Google Cloud Platform support? [y/N] 
No Google Cloud Platform support will be enabled for TensorFlow
Do you wish to build TensorFlow with Hadoop File System support? [y/N] 
No Hadoop File System support will be enabled for TensorFlow
Do you wish to build TensorFlow with the XLA just-in-time compiler (experimental)? [y/N] 
No XLA support will be enabled for TensorFlow
Found possible Python library paths:
  /usr/local/lib/python2.7/dist-packages
  /usr/lib/python2.7/dist-packages
Please input the desired Python library path to use.  Default is [/usr/local/lib/python2.7/dist-packages]

Using python library path: /usr/local/lib/python2.7/dist-packages
Do you wish to build TensorFlow with OpenCL support? [y/N] 
No OpenCL support will be enabled for TensorFlow
Do you wish to build TensorFlow with CUDA support? [y/N] 
No CUDA support will be enabled for TensorFlow
Configuration finished
INFO: Reading 'startup' options from /root/.bazelrc: --batch
INFO: Starting clean (this may take a while). Consider using --expunge_async if the clean takes more than several minutes.

INFO: Reading 'startup' options from /root/.bazelrc: --batch
java.lang.RuntimeException: Unrecoverable error while evaluating node 'REPOSITORY_DIRECTORY:@jpeg' (requested by nodes 'REPOSITORY:@jpeg')
	at com.google.devtools.build.skyframe.ParallelEvaluator$Evaluate.run(ParallelEvaluator.java:429)
	at com.google.devtools.build.lib.concurrent.AbstractQueueVisitor$WrappedRunnable.run(AbstractQueueVisitor.java:501)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: Invalid EvalException:
java.lang.InterruptedException
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
	at java.util.concurrent.Semaphore.acquire(Semaphore.java:312)
	at com.google.devtools.build.lib.bazel.repository.downloader.HttpDownloader.download(HttpDownloader.java:196)
	at com.google.devtools.build.lib.bazel.repository.skylark.SkylarkRepositoryContext.downloadAndExtract(SkylarkRepositoryContext.java:594)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.google.devtools.build.lib.syntax.FuncallExpression.callMethod(FuncallExpression.java:316)
	at com.google.devtools.build.lib.syntax.FuncallExpression.invokeObjectMethod(FuncallExpression.java:732)
	at com.google.devtools.build.lib.syntax.FuncallExpression.invokeObjectMethod(FuncallExpression.java:784)
	at com.google.devtools.build.lib.syntax.FuncallExpression.doEval(FuncallExpression.java:770)
	at com.google.devtools.build.lib.syntax.Expression.eval(Expression.java:48)
	at com.google.devtools.build.lib.syntax.ExpressionStatement.doExec(ExpressionStatement.java:46)
	at com.google.devtools.build.lib.syntax.Statement.exec(Statement.java:37)
	at com.google.devtools.build.lib.syntax.UserDefinedFunction.call(UserDefinedFunction.java:136)
	at com.google.devtools.build.lib.syntax.BaseFunction.call(BaseFunction.java:439)
	at com.google.devtools.build.lib.bazel.repository.skylark.SkylarkRepositoryFunction.fetch(SkylarkRepositoryFunction.java:106)
	at com.google.devtools.build.lib.rules.repository.RepositoryDelegatorFunction.compute(RepositoryDelegatorFunction.java:155)
	at com.google.devtools.build.skyframe.ParallelEvaluator$Evaluate.run(ParallelEvaluator.java:370)
	at com.google.devtools.build.lib.concurrent.AbstractQueueVisitor$WrappedRunnable.run(AbstractQueueVisitor.java:501)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

	at com.google.devtools.build.lib.syntax.EvalException.<init>(EvalException.java:112)
	at com.google.devtools.build.lib.syntax.EvalException$EvalExceptionWithJavaCause.<init>(EvalException.java:209)
	at com.google.devtools.build.lib.syntax.EvalException$EvalExceptionWithJavaCause.<init>(EvalException.java:217)
	at com.google.devtools.build.lib.syntax.FuncallExpression.callMethod(FuncallExpression.java:344)
	at com.google.devtools.build.lib.syntax.FuncallExpression.invokeObjectMethod(FuncallExpression.java:732)
	at com.google.devtools.build.lib.syntax.FuncallExpression.invokeObjectMethod(FuncallExpression.java:784)
	at com.google.devtools.build.lib.syntax.FuncallExpression.doEval(FuncallExpression.java:770)
	at com.google.devtools.build.lib.syntax.Expression.eval(Expression.java:48)
	at com.google.devtools.build.lib.syntax.ExpressionStatement.doExec(ExpressionStatement.java:46)
	at com.google.devtools.build.lib.syntax.Statement.exec(Statement.java:37)
	at com.google.devtools.build.lib.syntax.UserDefinedFunction.call(UserDefinedFunction.java:136)
	at com.google.devtools.build.lib.syntax.BaseFunction.call(BaseFunction.java:439)
	at com.google.devtools.build.lib.bazel.repository.skylark.SkylarkRepositoryFunction.fetch(SkylarkRepositoryFunction.java:106)
	at com.google.devtools.build.lib.rules.repository.RepositoryDelegatorFunction.compute(RepositoryDelegatorFunction.java:155)
	at com.google.devtools.build.skyframe.ParallelEvaluator$Evaluate.run(ParallelEvaluator.java:370)
	... 4 more
java.lang.RuntimeException: Unrecoverable error while evaluating node 'REPOSITORY_DIRECTORY:@jpeg' (requested by nodes 'REPOSITORY:@jpeg')
	at com.google.devtools.build.skyframe.ParallelEvaluator$Evaluate.run(ParallelEvaluator.java:429)
	at com.google.devtools.build.lib.concurrent.AbstractQueueVisitor$WrappedRunnable.run(AbstractQueueVisitor.java:501)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: Invalid EvalException:
java.lang.InterruptedException
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
	at java.util.concurrent.Semaphore.acquire(Semaphore.java:312)
	at com.google.devtools.build.lib.bazel.repository.downloader.HttpDownloader.download(HttpDownloader.java:196)
	at com.google.devtools.build.lib.bazel.repository.skylark.SkylarkRepositoryContext.downloadAndExtract(SkylarkRepositoryContext.java:594)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.google.devtools.build.lib.syntax.FuncallExpression.callMethod(FuncallExpression.java:316)
	at com.google.devtools.build.lib.syntax.FuncallExpression.invokeObjectMethod(FuncallExpression.java:732)
	at com.google.devtools.build.lib.syntax.FuncallExpression.invokeObjectMethod(FuncallExpression.java:784)
	at com.google.devtools.build.lib.syntax.FuncallExpression.doEval(FuncallExpression.java:770)
	at com.google.devtools.build.lib.syntax.Expression.eval(Expression.java:48)
	at com.google.devtools.build.lib.syntax.ExpressionStatement.doExec(ExpressionStatement.java:46)
	at com.google.devtools.build.lib.syntax.Statement.exec(Statement.java:37)
	at com.google.devtools.build.lib.syntax.UserDefinedFunction.call(UserDefinedFunction.java:136)
	at com.google.devtools.build.lib.syntax.BaseFunction.call(BaseFunction.java:439)
	at com.google.devtools.build.lib.bazel.repository.skylark.SkylarkRepositoryFunction.fetch(SkylarkRepositoryFunction.java:106)
	at com.google.devtools.build.lib.rules.repository.RepositoryDelegatorFunction.compute(RepositoryDelegatorFunction.java:155)
	at com.google.devtools.build.skyframe.ParallelEvaluator$Evaluate.run(ParallelEvaluator.java:370)
	at com.google.devtools.build.lib.concurrent.AbstractQueueVisitor$WrappedRunnable.run(AbstractQueueVisitor.java:501)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

	at com.google.devtools.build.lib.syntax.EvalException.<init>(EvalException.java:112)
	at com.google.devtools.build.lib.syntax.EvalException$EvalExceptionWithJavaCause.<init>(EvalException.java:209)
	at com.google.devtools.build.lib.syntax.EvalException$EvalExceptionWithJavaCause.<init>(EvalException.java:217)
	at com.google.devtools.build.lib.syntax.FuncallExpression.callMethod(FuncallExpression.java:344)
	at com.google.devtools.build.lib.syntax.FuncallExpression.invokeObjectMethod(FuncallExpression.java:732)
	at com.google.devtools.build.lib.syntax.FuncallExpression.invokeObjectMethod(FuncallExpression.java:784)
	at com.google.devtools.build.lib.syntax.FuncallExpression.doEval(FuncallExpression.java:770)
	at com.google.devtools.build.lib.syntax.Expression.eval(Expression.java:48)
	at com.google.devtools.build.lib.syntax.ExpressionStatement.doExec(ExpressionStatement.java:46)
	at com.google.devtools.build.lib.syntax.Statement.exec(Statement.java:37)
	at com.google.devtools.build.lib.syntax.UserDefinedFunction.call(UserDefinedFunction.java:136)
	at com.google.devtools.build.lib.syntax.BaseFunction.call(BaseFunction.java:439)
	at com.google.devtools.build.lib.bazel.repository.skylark.SkylarkRepositoryFunction.fetch(SkylarkRepositoryFunction.java:106)
	at com.google.devtools.build.lib.rules.repository.RepositoryDelegatorFunction.compute(RepositoryDelegatorFunction.java:155)
	at com.google.devtools.build.skyframe.ParallelEvaluator$Evaluate.run(ParallelEvaluator.java:370)
	... 4 more

About this issue

  • Original URL
  • State: closed
  • Created 7 years ago
  • Reactions: 2
  • Comments: 25 (8 by maintainers)

Most upvoted comments

I can confirm, this issue seems resolved: ./configure and build runs smoothly, starting the server also works. ✅

Calling the server gives me another error FAILED_PRECONDITION, details="Default serving signature key not found." - if there’s still anybody here able to give a hint in the right direction, I would appreciate it:

bazel-bin/tensorflow_serving/example/inception_client --server localhost:9000 --image Picture.jpg 
D0207 11:28:41.984146287    6541 ev_posix.c:101]             Using polling engine: poll
Traceback (most recent call last):
  File "/serving/bazel-bin/tensorflow_serving/example/inception_client.runfiles/tf_serving/tensorflow_serving/example/inception_client.py", line 56, in <module>
    tf.app.run()
  File "/serving/bazel-bin/tensorflow_serving/example/inception_client.runfiles/org_tensorflow/tensorflow/python/platform/app.py", line 44, in run
    _sys.exit(main(_sys.argv[:1] + flags_passthrough))
  File "/serving/bazel-bin/tensorflow_serving/example/inception_client.runfiles/tf_serving/tensorflow_serving/example/inception_client.py", line 51, in main
    result = stub.Predict(request, 10.0)  # 10 secs timeout
  File "/usr/local/lib/python2.7/dist-packages/grpc/beta/_client_adaptations.py", line 300, in __call__
    self._request_serializer, self._response_deserializer)
  File "/usr/local/lib/python2.7/dist-packages/grpc/beta/_client_adaptations.py", line 198, in _blocking_unary_unary
    raise _abortion_error(rpc_error_call)
grpc.framework.interfaces.face.face.AbortionError: AbortionError(code=StatusCode.FAILED_PRECONDITION, details="Default serving signature key not found.")
E0207 11:28:46.020743693    6541 chttp2_transport.c:1810]    close_transport: {"created":"@1486466926.020710033","description":"FD shutdown","file":"src/core/lib/iomgr/ev_poll_posix.c","file_line":427}
+5
k-schreiber on Feb 7, 2017

@kirilg you’re right, syncing up with the latest head solved the issue. Thanks!

+1
brookwc on Feb 14, 2017

Changes merged and the submodule is now updated. Please give it a try.

Closing this issue since the original problem was resolved. If there are still any problems with the models submodule, please reopen in a new bug.

+1
kirilg on Feb 1, 2017

FYI, we have a fix for the specific file here: https://github.com/tensorflow/models/commit/e5079c839058ff40dcbd15515a9cfb462fabbc2a#diff-1f23067444bd897aed5e55f64b63fe76, but the submodule tf_model hasn’t been synced yet. When this pull request ( https://github.com/tensorflow/serving/pull/304) is merged, the file should be updated.

On Tue, Jan 31, 2017 at 9:30 AM, k-schreiber notifications@github.com wrote:

Alright, getting there. Can you advise on any compatible versions of TF and Inception?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/tensorflow/serving/issues/301#issuecomment-276432649, or mute the thread https://github.com/notifications/unsubscribe-auth/AVjoBei2ytocGnmCwbFDPLqEYESGSoRWks5rX2-igaJpZM4LrE2J .

+1
lilao on Feb 1, 2017

Thanks. Had a feeling it was a certificates issue.

apt-get install ca-certificates-java update-ca-certificates -f

solved the problem for me.

+1
aerrity on Jan 30, 2017

I was able to get ./configure to work by running the following two commands (in the running Docker container). apt-get install ca-certificates-java update-ca-certificates -f

The issue I had was none of the github dependencies would load, as they used https transport.

I’m building from the Docker file at: tensorflow_serving/tools/docker/Dockerfile.devel This image is built using Ubuntu 14.04.5 LTS (per /etc/issue) and openjdk version “1.8.0_111” (in spite of the bazel install docs saying openjdk is not available for this version of Ubuntu) Host is: Docker for Mac. Also was successful in getting ./configure to work by installing the Oracle JDK and reinstalling bazel.

+1
mountaintom on Jan 30, 2017

The same story. Running Docker on MacOS Sierra via Docker for MacOS. Things that I tried with no success:

  • complete reinstallation, including Docker itself
  • bazel version 0.4.2 -> 0.4.3

It seems that the bug came up recently, because the last week I was able to run ./configure

+1
vaskinyy on Jan 24, 2017
Read more comments on GitHub
← serving: Cannot load saved model from S3 bucket after tensorflow-serving 2.7
serving: Docker with GPU failed call to cuInit: CUresult(-1)  →