kubernetes: Unable to register node with API server with RBAC authorization-mode

Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/.):

Yes, request for help. Refered - https://kubernetes.io/docs/admin/authorization/rbac/ to enable RBAC on API Server

What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): RBAC authorization Forbidden - unable to register node VM.

---

Is this a BUG REPORT or FEATURE REQUEST? (choose one):

Kubernetes version (use kubectl version):

root@master [ ~ ]# kubectl version  
Client Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.6", GitCommit:"7fa1c1756d8bc963f1a389f4a6937dc71f08ada2", GitTreeState:"clean", BuildDate:"2017-06-16T18:34:20Z", GoVersion:"go1.7.6", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"8+", GitVersion:"v1.8.0-alpha.0.772+7831a5426fca83", GitCommit:"7831a5426fca83d5f9a66341b7a07a64178f8807", GitTreeState:"clean", BuildDate:"2017-06-16T05:37:06Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}

Environment:

• Cloud provider or hardware configuration: vsphere
• OS (e.g. from /etc/os-release): photon
• Kernel (e.g. uname -a): Linux master 4.4.8-esx #1-photon SMP Tue Jun 7 08:04:49 UTC 2016 x86_64 GNU/Linux
• Install tools:
• Others:

What happened: Started API server with RBAC enabled on the master node. used "--authorization-mode=RBAC" flag.

spec from kube-apiserver.json

    "spec": {
        "containers": [
            {
                "command": [
                    "/hyperkube",
                    "apiserver",
                    "--address=127.0.0.1",
                    "--etcd-servers=http://127.0.0.1:2379",
                    "--cloud-provider=vsphere",
                    "--admission-control=NamespaceLifecycle,LimitRanger,ServiceAccount,PersistentVolumeLabel,ResourceQuota",
                    "--service-cluster-ip-range=10.0.0.0/16",
                    "--client-ca-file=/srv/kubernetes/ca.pem",
                    "--tls-cert-file=/srv/kubernetes/apiserver.pem",
                    "--tls-private-key-file=/srv/kubernetes/apiserver-key.pem",
                    "--secure-port=443",
                    "--storage-backend=etcd2",
                    "--allow-privileged",
                    "--authorization-mode=RBAC",
                    "--v=4",
                    "--cloud-config=/etc/kubernetes/vsphere.conf"
                ],

Node VMs are not able to register to the Master node.

Logs from Node VM.

{"log":"I0617 06:33:11.702223     697 kubelet_node_status.go:80] Attempting to register node node1\n","stream":"stderr","time":"2017-06-17T06:33:11.702884548Z"}
{"log":"E0617 06:33:11.704772     697 kubelet_node_status.go:104] Unable to register node \"node1\" with API server: User \"kubernetes-node_certificate\" cannot create nodes at the cluster scope. (post nodes)\n","stream":"stderr","time":"2017-06-17T06:33:11.704902558Z"}
{"log":"E0617 06:33:11.706422     697 event.go:200] Server rejected event '\u0026v1.Event{TypeMeta:v1.TypeMeta{Kind:\"\", APIVersion:\"\"}, ObjectMeta:v1.ObjectMeta{Name:\"node1.14c8d50d732a19e2\", GenerateName:\"\", Namespace:\"default\", SelfLink:\"\", UID:\"\", ResourceVersion:\"\", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{sec:0, nsec:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:\"\"}, InvolvedObject:v1.ObjectReference{Kind:\"Node\", Namespace:\"\", Name:\"node1\", UID:\"node1\", APIVersion:\"\", ResourceVersion:\"\", FieldPath:\"\"}, Reason:\"NodeHasSufficientDisk\", Message:\"Node node1 status is now: NodeHasSufficientDisk\", Source:v1.EventSource{Component:\"kubelet\", Host:\"node1\"}, FirstTimestamp:v1.Time{Time:time.Time{sec:63633277929, nsec:844120034, loc:(*time.Location)(0x91f42a0)}}, LastTimestamp:v1.Time{Time:time.Time{sec:63633277991, nsec:702191698, loc:(*time.Location)(0x91f42a0)}}, Count:15, Type:\"Normal\"}': 'User \"kubernetes-node_certificate\" cannot patch events in the namespace \"default\". (patch events node1.14c8d50d732a19e2)' (will not retry!)\n","stream":"stderr","time":"2017-06-17T06:33:11.706869462Z"}
{"log":"E0617 06:33:11.707987     697 event.go:200] Server rejected event '\u0026v1.Event{TypeMeta:v1.TypeMeta{Kind:\"\", APIVersion:\"\"}, ObjectMeta:v1.ObjectMeta{Name:\"node1.14c8d50d732a3bef\", GenerateName:\"\", Namespace:\"default\", SelfLink:\"\", UID:\"\", ResourceVersion:\"\", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{sec:0, nsec:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:\"\"}, InvolvedObject:v1.ObjectReference{Kind:\"Node\", Namespace:\"\", Name:\"node1\", UID:\"node1\", APIVersion:\"\", ResourceVersion:\"\", FieldPath:\"\"}, Reason:\"NodeHasSufficientMemory\", Message:\"Node node1 status is now: NodeHasSufficientMemory\", Source:v1.EventSource{Component:\"kubelet\", Host:\"node1\"}, FirstTimestamp:v1.Time{Time:time.Time{sec:63633277929, nsec:844128751, loc:(*time.Location)(0x91f42a0)}}, LastTimestamp:v1.Time{Time:time.Time{sec:63633277991, nsec:702199899, loc:(*time.Location)(0x91f42a0)}}, Count:15, Type:\"Normal\"}': 'User \"kubernetes-node_certificate\" cannot patch events in the namespace \"default\". (patch events node1.14c8d50d732a3bef)' (will not retry!)\n","stream":"stderr","time":"2017-06-17T06:33:11.708452386Z"}
{"log":"E0617 06:33:11.710181     697 event.go:200] Server rejected event '\u0026v1.Event{TypeMeta:v1.TypeMeta{Kind:\"\", APIVersion:\"\"}, ObjectMeta:v1.ObjectMeta{Name:\"node1.14c8d50d732ab476\", GenerateName:\"\", Namespace:\"default\", SelfLink:\"\", UID:\"\", ResourceVersion:\"\", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{sec:0, nsec:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:\"\"}, InvolvedObject:v1.ObjectReference{Kind:\"Node\", Namespace:\"\", Name:\"node1\", UID:\"node1\", APIVersion:\"\", ResourceVersion:\"\", FieldPath:\"\"}, Reason:\"NodeHasNoDiskPressure\", Message:\"Node node1 status is now: NodeHasNoDiskPressure\", Source:v1.EventSource{Component:\"kubelet\", Host:\"node1\"}, FirstTimestamp:v1.Time{Time:time.Time{sec:63633277929, nsec:844159606, loc:(*time.Location)(0x91f42a0)}}, LastTimestamp:v1.Time{Time:time.Time{sec:63633277991, nsec:702208029, loc:(*time.Location)(0x91f42a0)}}, Count:15, Type:\"Normal\"}': 'User \"kubernetes-node_certificate\" cannot patch events in the namespace \"default\". (patch events node1.14c8d50d732ab476)' (will not retry!)\n","stream":"stderr","time":"2017-06-17T06:33:11.710360503Z"}
{"log":"E0617 06:33:12.143901     697 reflector.go:190] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: Failed to list *v1.Pod: User \"kubernetes-node_certificate\" cannot list pods at the cluster scope. (get pods)\n","stream":"stderr","time":"2017-06-17T06:33:12.144158629Z"}
{"log":"E0617 06:33:12.145993     697 reflector.go:190] k8s.io/kubernetes/pkg/kubelet/kubelet.go:408: Failed to list *v1.Node: User \"kubernetes-node_certificate\" cannot list nodes at the cluster scope. (get nodes)\n","stream":"stderr","time":"2017-06-17T06:33:12.146180664Z"}
{"log":"E0617 06:33:12.146045     697 reflector.go:190] k8s.io/kubernetes/pkg/kubelet/kubelet.go:400: Failed to list *v1.Service: User \"kubernetes-node_certificate\" cannot list services at the cluster scope. (get services)\n","stream":"stderr","time":"2017-06-17T06:33:12.146203213Z"}

Logs from Master VM

{"log":"I0617 06:32:41.983168       1 rbac.go:114] RBAC DENY: user \"kubernetes-node_certificate\" groups [\"kubernetes-anywhere\" \"system:authenticated\"] cannot \"list\" resource \"nodes\" cluster-wide\n","stream":"stderr","time":"2017-06-17T06:32:41.983227779Z"}
{"log":"I0617 06:32:41.983208       1 authorization.go:58] Forbidden: \"/api/v1/nodes?fieldSelector=metadata.name%3Dnode1\u0026resourceVersion=0\", Reason: \"\"\n","stream":"stderr","time":"2017-06-17T06:32:41.983246584Z"}
{"log":"I0617 06:32:41.983287       1 wrap.go:42] GET /api/v1/nodes?fieldSelector=metadata.name%3Dnode1\u0026resourceVersion=0: (518.851µs) 403 [[hyperkube/v1.8.0 (linux/amd64) kubernetes/7831a54] 10.162.57.253:53774]\n","stream":"stderr","time":"2017-06-17T06:32:41.983304058Z"}
{"log":"I0617 06:32:41.983505       1 wrap.go:42] GET /apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings?resourceVersion=0: (269.326µs) 200 [[hyperkube/v1.8.0 (linux/amd64) kubernetes/7831a54] 127.0.0.1:41580]\n","stream":"stderr","time":"2017-06-17T06:32:41.983561467Z"}
{"log":"I0617 06:32:41.983754       1 rbac.go:114] RBAC DENY: user \"kubernetes-node_certificate\" groups [\"kubernetes-anywhere\" \"system:authenticated\"] cannot \"list\" resource \"pods\" cluster-wide\n","stream":"stderr","time":"2017-06-17T06:32:41.983813506Z"}
{"log":"I0617 06:32:41.983791       1 authorization.go:58] Forbidden: \"/api/v1/pods?fieldSelector=spec.nodeName%3Dnode1\u0026resourceVersion=0\", Reason: \"\"\n","stream":"stderr","time":"2017-06-17T06:32:41.983829892Z"}

What you expected to happen: Node should be registered to the master node. Are we missing any configuration in the node VMs?

How to reproduce it (as minimally and precisely as possible): Just enabled “–authorization-mode=RBAC” on the API server on master node. No configuration change on the node VMs.

Anything else we need to know:

cc: @SandeepPissay