binderhub: Containers stuck at ContainerCreating

I have set up a BinderHub using a modified version of the instructions at https://zero-to-jupyterhub.readthedocs.io. I can load up the main page, but the process does not advance beyond the “Waiting” status, and the log there just shows “Waiting for build to start…”, even after waiting more than an hour.

Looking at the pods, I see a container whose name starts with build-binder, which has been stuck at the ContainerCreating status. Strangely, it appears in the default namespace, even though everything else (BinderHub, JupyterHub) is under a different namespace (sheff-hub).

I can’t access its log, but describe shows that mounting a volume has failed, because of a non-existent secret:

MountVolume.SetUp failed for volume "docker-push-secret" : secrets "binder-push-secret" not found

(full output attached at end)

Indeed, that secret does not exist in the default namespace, but it does in sheff-hub:

NAMESPACE     NAME                                  TYPE                                  DATA   AGE
default       default-token-2fq72                   kubernetes.io/service-account-token   3      4h
sheff-hub     binder-push-secret                    Opaque                                1      2h
sheff-hub     binder-secret                         Opaque                                2      2h
sheff-hub     binderhub-token-wkfzx                 kubernetes.io/service-account-token   3      2h
sheff-hub     default-token-zcdbw                   kubernetes.io/service-account-token   3      4h
sheff-hub     hub-secret                            Opaque                                2      2h
sheff-hub     hub-token-6dd7s                       kubernetes.io/service-account-token   3      2h
sheff-hub     sheff-hub-image-cleaner-token-kfmh5   kubernetes.io/service-account-token   3      2h

(table snipped to remove secrets from kube-system, kube-public namespaces)

This is on OS X, with

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.4", GitCommit:"c27b913fddd1a6c480c229191a087698aa92f0b1", GitTreeState:"clean", BuildDate:"2019-03-01T23:35:19Z", GoVersion:"go1.12", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.8", GitCommit:"4e209c9383fa00631d124c8adcc011d617339b3c", GitTreeState:"clean", BuildDate:"2019-02-28T18:40:05Z", GoVersion:"go1.10.8", Compiler:"gc", Platform:"linux/amd64"}

I also had to use the workaround in #809, because I was encountering the same error when trying to install binderhub, but am not getting the authentication error now.

Thanks in advice for any insight or advice!

This is the full output of describing the container:

$ kubectl describe pod build-binder-2dexamples-2drequirements-55ab5c-a73ba1
Name:               build-binder-2dexamples-2drequirements-55ab5c-a73ba1
Namespace:          default
Priority:           0
PriorityClassName:  <none>
Node:               aks-nodepool1-33022287-1/10.240.0.6
Start Time:         Mon, 18 Mar 2019 15:03:56 +0000
Labels:             component=binderhub-build
                    name=build-binder-2dexamples-2drequirements-55ab5c-a73ba1
Annotations:        binder-repo: https://github.com/binder-examples/requirements
Status:             Pending
IP:                 
Containers:
  builder:
    Container ID:  
    Image:         jupyter/repo2docker:0.8.0
    Image ID:      
    Port:          <none>
    Host Port:     <none>
    Args:
      jupyter-repo2docker
      --ref
      a73ba121c9847fa38b7c4153230b9bfa9eecfaa7
      --image
      binder-2dexamples-2drequirements-55ab5c:a73ba121c9847fa38b7c4153230b9bfa9eecfaa7
      --no-clean
      --no-run
      --json-logs
      --user-name
      jovyan
      --user-id
      1000
      --push
      https://github.com/binder-examples/requirements
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Limits:
      memory:  0
    Requests:
      memory:  0
    Environment:
      KUBERNETES_PORT_443_TCP_ADDR:  sheffhubcl-shefftesthub-eb9e00-2228b749.hcp.westeurope.azmk8s.io
      KUBERNETES_PORT:               tcp://sheffhubcl-shefftesthub-eb9e00-2228b749.hcp.westeurope.azmk8s.io:443
      KUBERNETES_PORT_443_TCP:       tcp://sheffhubcl-shefftesthub-eb9e00-2228b749.hcp.westeurope.azmk8s.io:443
      KUBERNETES_SERVICE_HOST:       sheffhubcl-shefftesthub-eb9e00-2228b749.hcp.westeurope.azmk8s.io
    Mounts:
      /root/.docker from docker-push-secret (rw)
      /var/run/docker.sock from docker-socket (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-2fq72 (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  docker-socket:
    Type:          HostPath (bare host directory volume)
    Path:          /var/run/docker.sock
    HostPathType:  Socket
  docker-push-secret:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  binder-push-secret
    Optional:    false
  default-token-2fq72:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-2fq72
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason       Age                   From                               Message
  ----     ------       ----                  ----                               -------
  Normal   Scheduled    83m                   default-scheduler                  Successfully assigned default/build-binder-2dexamples-2drequirements-55ab5c-a73ba1 to aks-nodepool1-33022287-1
  Warning  FailedMount  7m59s (x45 over 83m)  kubelet, aks-nodepool1-33022287-1  MountVolume.SetUp failed for volume "docker-push-secret" : secrets "binder-push-secret" not found
  Warning  FailedMount  118s (x36 over 81m)   kubelet, aks-nodepool1-33022287-1  Unable to mount volumes for pod "build-binder-2dexamples-2drequirements-55ab5c-a73ba1_default(0d3243e8-498f-11e9-8d40-fe8b4ef2ccd8)": timeout expired waiting for volumes to attach or mount for pod "default"/"build-binder-2dexamples-2drequirements-55ab5c-a73ba1". list of unmounted volumes=[docker-push-secret]. list of unattached volumes=[docker-socket docker-push-secret default-token-2fq72]

About this issue

Original URL
State: closed
Created 5 years ago
Comments: 22 (14 by maintainers)

Most upvoted comments

Trying this again from scratch, it seems to work (the containers are created in the expected namespace). I was following the same instructions, and the only difference I can see is in the description of the binder deployment. I now have

Annotations:            deployment.kubernetes.io/revision: 2

instead of

Annotations:            deployment.kubernetes.io/revision: 4

(it’s possible that I updated the deployment more times when I first tried this)

I guess I must have made a mistake somewhere along the way the first time, so this can be closed as far as I’m concerned!

ageorgou on Apr 1, 2019

Just tried a new setup on OVH (new cluster and new BinderHub), but this time choosing 1.15 (the lowest they offer at the moment) as the Kubernetes version.

And it’s working out of the box without having to follow the steps mentioned in https://github.com/jupyterhub/binderhub/issues/810#issuecomment-651089134.

We should probably open a new issue to keep track of this. It looks like it is related to some rbac changes in the latest versions?

jtpio on Jun 29, 2020

Thanks for chasing this done. I’ll close this here now.

We can move this discussion to http://discourse.jupyter.org/ (where most of the people who hang out here also hang out, as well as more people) if you want to keep exchanging debugging tips. We are trying to streamline the different discussion places and want to use the issues on GitHub repos for technical discussions on how to change the contents of the repo. More general discussions should go to discourse so that they are easier to find, better indexed by google and generally more accessible than being hidden in the bowls of a GitHub repository 😉

Thanks!

betatim on Apr 1, 2019

The variable seems to be set correctly “in the pod”:

# echo $BUILD_NAMESPACE
sheff-hub

ageorgou on Mar 19, 2019

Hmmm, I dont know why the pod has ended up there really, but it is because its replicaset is there i assume, and the replicaset is there because its deployment is there, but why is the deployment located in the default namespace?

Did you install the deployment without using helm? helm will add the namespace field to the deployment’s metadata.namespace field, but without it added it will install in default

I know a dirty fix that only cures this symptom but probably will just could be the first of many symptoms of something that needs fixing. You could edit the deployments namespace with kubectl edit.

Oh oh wait this pod is perhaps created by binderhub and isnt managed by a deployment/replicaset…

Hmmm then why did binderhub not start the pod in the correct namespace? Hmmm… How does binderhub know what namespace it should spawn things in? Probably by being deployed with helm and allowing its template to render {{ .Release.Namespace }} or something to an environment variable it reads. Then, again id wonder if you perhaps installed this without rendering the templates using helm.

A stream of thoughts written down without edit from a mobile, from a person sitting on a train, on a baggage shelf, in sweden, the world, in the milkyway, in our cluster of galaxies.

❤️

consideRatio on Mar 18, 2019