kubespawner: kubespawner stopped working after kubernetes upgrade from 1.15 to 1.16

The patch can be done also by the command below.

kubectl patch deploy -n $NAMESPACE hub --type json --patch '[{"op": "replace", "path": "/spec/template/spec/containers/0/command", "value": ["bash", "-c", "\nmkdir -p ~/hotfix\ncp -r /usr/local/lib/python3.6/dist-packages/kubespawner ~/hotfix\nls -R ~/hotfix\npatch ~/hotfix/kubespawner/spawner.py << EOT\n72c72\n<             key=lambda x: x.last_timestamp,\n---\n>             key=lambda x: x.last_timestamp and x.last_timestamp.timestamp() or 0.,\nEOT\n\nPYTHONPATH=$HOME/hotfix jupyterhub --config /srv/jupyterhub_config.py --upgrade-db\n"]}]'

@dmpe Change deployment hub’s .spec.template.spec.containers[0].command to something like this:

kubectl edit deploy -n $NAMESPACE hub

...
      containers:
      - command:
        - bash
        - -c
        - |
          mkdir -p ~/hotfix
          cp -r /usr/local/lib/python3.6/dist-packages/kubespawner ~/hotfix
          ls -R ~/hotfix
          patch ~/hotfix/kubespawner/spawner.py << EOT
          72c72
          <             key=lambda x: x.last_timestamp,
          ---
          >             key=lambda x: x.last_timestamp and x.last_timestamp.timestamp() or 0.,
          EOT

          PYTHONPATH=$HOME/hotfix jupyterhub --config /srv/jupyterhub_config.py --upgrade-db
        env:
        - name: PYTHONUNBUFFERED
          value: "1"
        - name: HELM_RELEASE_NAME
...

As @apoliakevitch says event objects sometimes have null lastTimestamp actually.

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.1", GitCommit:"d647ddbd755faf07169599a625faf302ffc34458", GitTreeState:"clean", BuildDate:"2019-10-02T17:01:15Z", GoVersion:"go1.12.10", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.1", GitCommit:"d647ddbd755faf07169599a625faf302ffc34458", GitTreeState:"clean", BuildDate:"2019-10-02T16:51:36Z", GoVersion:"go1.12.10", Compiler:"gc", Platform:"linux/amd64"}

$ kubectl get events -ojsonpath="{range .items[*]}{.lastTimestamp}{'\t'}{.message}{'\n'}{end}" | expand -t 24
2019-10-04T00:03:52Z    waiting for a volume to be created, either by external provisioner "rook-ceph.rbd.csi.ceph.com" or manually created by system administrator
2019-10-04T00:03:52Z    External provisioner is provisioning volume for claim "jhub/claim-test"
<nil>                   pod has unbound immediate PersistentVolumeClaims
<nil>                   pod has unbound immediate PersistentVolumeClaims
<nil>                   Successfully assigned jhub/jupyter-test to niu.mtk.nao.ac.jp
2019-10-04T00:03:57Z    AttachVolume.Attach succeeded for volume "pvc-06076af8-529e-4e0b-a9af-d366c9e99f61"
2019-10-04T00:04:10Z    Container image "jupyterhub/k8s-network-tools:0.8.2" already present on machine
2019-10-04T00:04:10Z    Created container block-cloud-metadata
2019-10-04T00:04:10Z    Started container block-cloud-metadata
2019-10-04T00:04:11Z    Container image "jupyterhub/k8s-singleuser-sample:0.8.2" already present on machine
2019-10-04T00:04:12Z    Created container notebook
2019-10-04T00:04:12Z    Started container notebook
2019-10-04T00:04:15Z    Stopping container notebook
2019-10-04T00:02:34Z    No matching pods found
2019-10-04T00:02:34Z    No matching pods found

So null checking can avoid this problem for the moment.

root@hub-d59bc9d4f-48c5l:/srv/jupyterhub# diff -c a b
*** a   2019-10-04 00:14:01.892597740 +0000
--- b   2019-10-04 00:13:46.252338190 +0000
***************
*** 69,75 ****
      def events(self):
          return sorted(
              self.resources.values(),
!             key=lambda x: x.last_timestamp,
          )


--- 69,75 ----
      def events(self):
          return sorted(
              self.resources.values(),
!             key=lambda x: x.last_timestamp and x.last_timestamp.timestamp() or 0.,
          )

It is resolved in the 0.9.0-beta.3 though, right?

If the problem is due to a breaking change in the K8s API (as opposed to a bug in kubespawner) the fix will either need to wait for the upstream Python client package (there’s already an open issue: https://github.com/kubernetes-client/python/issues/973) or the fix will need to use methods which haven’t been broken.

If anyone currently running k8s 1.16 has time to investigate that would be very helpful!

I can confirm that this works in version 0.9.0-beta.3

For me the proposed change just changes the error:

  File "/opt/app-root/lib/python3.6/site-packages/kubespawner/spawner.py", line 73, in events
    key=lambda x: x.last_timestamp if x.last_timestamp is not None else 0.,
TypeError: '<' not supported between instances of 'float' and 'datetime.datetime'

Now complains about comparing float rather than NoneType.

Great! Would one of you like to open a pull request?

In my case (bare metal installation), I fixed the null problem using the @michitaro patch and the following helm instruction:

...
singleuser:
  # Mandatory for Bare Metal installation
  cloudMetadata:
    enabled: true
...

The patch can be done also by the command below.

kubectl patch deploy -n $NAMESPACE hub --type json --patch '[{"op": "replace", "path": "/spec/template/spec/containers/0/command", "value": ["bash", "-c", "\nmkdir -p ~/hotfix\ncp -r /usr/local/lib/python3.6/dist-packages/kubespawner ~/hotfix\nls -R ~/hotfix\npatch ~/hotfix/kubespawner/spawner.py << EOT\n72c72\n<             key=lambda x: x.last_timestamp,\n---\n>             key=lambda x: x.last_timestamp and x.last_timestamp.timestamp() or 0.,\nEOT\n\nPYTHONPATH=$HOME/hotfix jupyterhub --config /srv/jupyterhub_config.py --upgrade-db\n"]}]'

If is just to avoid error on sort, maybe should use datetime.datetime.fromtimestamp(0) rather than just 0..

You could also use this docker image. It comes with the patch included. Since there’s no pull request yet, I’ll try to open one.