kubeflow: Jupyter Notebook - istio side car readiness probe fail (status code 503)

/kind bug

What steps did you take and what happened: Launching a jupyter notebook with a simple PVC bounded on the home dir, creates a running pod that fails the readiness probe.

  Normal   Created                 35s   kubelet, REDACTED  Created container istio-init
  Normal   Started                 35s   kubelet, REDACTED  Started container istio-init
  Normal   Pulled                  32s   kubelet, REDACTED  Container image "gcr.io/kubeflow-images-public/tensorflow-1.14.0-notebook-cpu:v0.7.0" already present on machine
  Normal   Created                 32s   kubelet, REDACTED  Created container test-nb
  Normal   Started                 32s   kubelet, REDACTED  Started container test-nb
  Normal   Pulled                  32s   kubelet, REDACTED  Container image "docker.io/istio/proxyv2:1.1.6" already present on machine
  Normal   Created                 32s   kubelet, REDACTED  Created container istio-proxy
  Normal   Started                 32s   kubelet, REDACTED  Started container istio-proxy
  Warning  Unhealthy               30s   kubelet, REDACTED  Readiness probe failed: HTTP probe failed with statuscode: 503

What did you expect to happen: Readiness probe should be executed correctly

Environment:

Kubeflow version: (version number can be found at the bottom left corner of the Kubeflow dashboard): 0.7
kfctl version: (use kfctl version): 0.7
Kubernetes platform: (e.g. minikube) EKS
Kubernetes version: (use kubectl version): 1.14.7
OS (e.g. from /etc/os-release): Amazon Linux (Centos)

About this issue

Original URL
State: closed
Created 5 years ago
Reactions: 2
Comments: 28 (8 by maintainers)

Most upvoted comments

I have the same issue.

gugumituo on Jun 4, 2021

I am having the same problem as @supertetelman - mounted without any volume the home directory for the jovyan user is all owned by root - which prevents launching the notebook since jovyan lacks permissions to alter root user owned files and directories.

Working on migrating my existing notebook container build to meet some of the launch criteria in the interim.

humantargetjoe on May 18, 2020