microk8s: kubectl logs and exec fail

On a clean VM running ubuntu 18.04, after installing microk8s 1.17 (incidentally, 1.18 showed the same behavior), I’m unable to kubectl logs <pod name> or kubectl exec <pod name>. Other commands, like kubectl get all work just fine. The issue arises with an application I am developing, but can be reproduced just by trying to get the logs from the registry container, which is what’s shown here.

inspection-report-20200630_054324.tar.gz

Initial setup:

sudo snap install docker
sudo snap install microk8s --classic --channel=1.17/stable
sudo groupadd docker
sudo usermod -a -G microk8s,docker $USER
su - $USER
sudo iptables -P FORWARD ACCEPT
microk8s.enable storage dns registry

get all works:

microk8s.kubectl get all -A
NAMESPACE            NAME                                        READY   STATUS    RESTARTS   AGE
container-registry   pod/registry-d7d7c8bc9-8k6j8                1/1     Running   0          22s
kube-system          pod/coredns-9b8997588-sh2gg                 1/1     Running   0          40s
kube-system          pod/hostpath-provisioner-7b9cb5cdb4-lj2z7   1/1     Running   0          40s

NAMESPACE            NAME                 TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                  AGE
container-registry   service/registry     NodePort    10.152.183.141   <none>        5000:32000/TCP           22s
default              service/kubernetes   ClusterIP   10.152.183.1     <none>        443/TCP                  67s
kube-system          service/kube-dns     ClusterIP   10.152.183.10    <none>        53/UDP,53/TCP,9153/TCP   40s

NAMESPACE            NAME                                   READY   UP-TO-DATE   AVAILABLE   AGE
container-registry   deployment.apps/registry               1/1     1            1           23s
kube-system          deployment.apps/coredns                1/1     1            1           40s
kube-system          deployment.apps/hostpath-provisioner   1/1     1            1           41s

NAMESPACE            NAME                                              DESIRED   CURRENT   READY   AGE
container-registry   replicaset.apps/registry-d7d7c8bc9                1         1         1       23s
kube-system          replicaset.apps/coredns-9b8997588                 1         1         1       40s
kube-system          replicaset.apps/hostpath-provisioner-7b9cb5cdb4   1         1         1       41s

But logs fails:

$ microk8s kubectl logs -n container-registry registry-d7d7c8bc9-8k6j8
error: You must be logged in to the server (the server has asked for the client to provide credentials ( pods/log registry-d7d7c8bc9-8k6j8))

Verbose version of the logs command:

$ microk8s kubectl logs -n container-registry registry-d7d7c8bc9-8k6j8 -v=9
I0630 05:43:04.864564   10322 loader.go:375] Config loaded from file:  /var/snap/microk8s/1489/credentials/client.config
I0630 05:43:04.870006   10322 round_trippers.go:423] curl -k -v -XGET  -H "Authorization: Basic YWRtaW46YUhkUWFtWk9aMDlQZFUxek0yOUZRbkFyTTFoTVQxcG5kWFJNVDNBd1ZqTjVUSEZ3YlVSMlVtWjBORDBL" -H "Accept: application/json, */*" -H "User-Agent: kubectl/v1.17.7 (linux/amd64) kubernetes/b445510" 'https://127.0.0.1:16443/api/v1/namespaces/container-registry/pods/registry-d7d7c8bc9-8k6j8'
I0630 05:43:04.876268   10322 round_trippers.go:443] GET https://127.0.0.1:16443/api/v1/namespaces/container-registry/pods/registry-d7d7c8bc9-8k6j8 200 OK in 6 milliseconds
I0630 05:43:04.876312   10322 round_trippers.go:449] Response Headers:
I0630 05:43:04.876322   10322 round_trippers.go:452]     Cache-Control: no-cache, private
I0630 05:43:04.876327   10322 round_trippers.go:452]     Content-Type: application/json
I0630 05:43:04.876351   10322 round_trippers.go:452]     Content-Length: 2925
I0630 05:43:04.876869   10322 round_trippers.go:452]     Date: Tue, 30 Jun 2020 05:43:04 GMT
I0630 05:43:04.876920   10322 request.go:1017] Response Body: {"kind":"Pod","apiVersion":"v1","metadata":{"name":"registry-d7d7c8bc9-8k6j8","generateName":"registry-d7d7c8bc9-","namespace":"container-registry","selfLink":"/api/v1/namespaces/container-registry/pods/registry-d7d7c8bc9-8k6j8","uid":"384c62b9-1971-4813-8413-aac9d50d08fd","resourceVersion":"396","creationTimestamp":"2020-06-30T05:40:00Z","labels":{"app":"registry","pod-template-hash":"d7d7c8bc9"},"ownerReferences":[{"apiVersion":"apps/v1","kind":"ReplicaSet","name":"registry-d7d7c8bc9","uid":"0cd2786e-e89b-49ea-957f-43bbfc74c700","controller":true,"blockOwnerDeletion":true}]},"spec":{"volumes":[{"name":"registry-data","persistentVolumeClaim":{"claimName":"registry-claim"}},{"name":"default-token-rc5ld","secret":{"secretName":"default-token-rc5ld","defaultMode":420}}],"containers":[{"name":"registry","image":"cdkbot/registry-amd64:2.6","ports":[{"name":"registry","containerPort":5000,"protocol":"TCP"}],"env":[{"name":"REGISTRY_HTTP_ADDR","value":":5000"},{"name":"REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY","value":"/var/lib/registry"},{"name":"REGISTRY_STORAGE_DELETE_ENABLED","value":"yes"}],"resources":{},"volumeMounts":[{"name":"registry-data","mountPath":"/var/lib/registry"},{"name":"default-token-rc5ld","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent"}],"restartPolicy":"Always","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","serviceAccountName":"default","serviceAccount":"default","nodeName":"ecr-baseline","securityContext":{},"schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}],"priority":0,"enableServiceLinks":true},"status":{"phase":"Running","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2020-06-30T05:40:01Z"},{"type":"Ready","status":"True","lastProbeTime":null,"lastTransitionTime":"2020-06-30T05:40:13Z"},{"type":"ContainersReady","status":"True","lastProbeTime":null,"lastTransitionTime":"2020-06-30T05:40:13Z"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2020-06-30T05:40:01Z"}],"hostIP":"192.168.37.129","podIP":"10.1.31.4","podIPs":[{"ip":"10.1.31.4"}],"startTime":"2020-06-30T05:40:01Z","containerStatuses":[{"name":"registry","state":{"running":{"startedAt":"2020-06-30T05:40:12Z"}},"lastState":{},"ready":true,"restartCount":0,"image":"docker.io/cdkbot/registry-amd64:2.6","imageID":"docker.io/cdkbot/registry-amd64@sha256:9dc5ad9e0b3ed59afa4febe8d663fd545628c60886a2c9fcd0e186e5cf6343ef","containerID":"containerd://7e14ff48ff18f6470975ee0b31f599f3a5b86c922bd544db6a0015e558220d0d","started":true}],"qosClass":"BestEffort"}}
I0630 05:43:04.883059   10322 round_trippers.go:423] curl -k -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: kubectl/v1.17.7 (linux/amd64) kubernetes/b445510" -H "Authorization: Basic YWRtaW46YUhkUWFtWk9aMDlQZFUxek0yOUZRbkFyTTFoTVQxcG5kWFJNVDNBd1ZqTjVUSEZ3YlVSMlVtWjBORDBL" 'https://127.0.0.1:16443/api/v1/namespaces/container-registry/pods/registry-d7d7c8bc9-8k6j8/log'
I0630 05:43:04.887857   10322 round_trippers.go:443] GET https://127.0.0.1:16443/api/v1/namespaces/container-registry/pods/registry-d7d7c8bc9-8k6j8/log 401 Unauthorized in 4 milliseconds
I0630 05:43:04.887895   10322 round_trippers.go:449] Response Headers:
I0630 05:43:04.887901   10322 round_trippers.go:452]     Date: Tue, 30 Jun 2020 05:43:04 GMT
I0630 05:43:04.887905   10322 round_trippers.go:452]     Cache-Control: no-cache, private
I0630 05:43:04.887910   10322 round_trippers.go:452]     Content-Type: application/json
I0630 05:43:04.887913   10322 round_trippers.go:452]     Content-Length: 276
I0630 05:43:04.887952   10322 request.go:1017] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"the server has asked for the client to provide credentials ( pods/log registry-d7d7c8bc9-8k6j8)","reason":"Unauthorized","details":{"name":"registry-d7d7c8bc9-8k6j8","kind":"pods/log"},"code":401}
I0630 05:43:04.888316   10322 helpers.go:203] server response object: [{
  "metadata": {},
  "status": "Failure",
  "message": "the server has asked for the client to provide credentials ( pods/log registry-d7d7c8bc9-8k6j8)",
  "reason": "Unauthorized",
  "details": {
    "name": "registry-d7d7c8bc9-8k6j8",
    "kind": "pods/log"
  },
  "code": 401
}]
F0630 05:43:04.888383   10322 helpers.go:114] error: You must be logged in to the server (the server has asked for the client to provide credentials ( pods/log registry-d7d7c8bc9-8k6j8))