aws-ebs-csi-driver: RegisterPlugin error -- failed to get plugin info using RPC GetInfo at socket /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock

/kind bug

What happened?

All cluster nodes have the following lines in the kubelet log:

Nov 06 20:55:06 nodename kubelet[776]: I1106 20:55:06.366541     776 operation_generator.go:193] parsed scheme: ""
Nov 06 20:55:06 nodename kubelet[776]: I1106 20:55:06.366580     776 operation_generator.go:193] scheme "" not registered, fallback to default scheme
Nov 06 20:55:06 nodename kubelet[776]: I1106 20:55:06.366603     776 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{/var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock 0  <nil>}] <nil>}
Nov 06 20:55:06 nodename kubelet[776]: I1106 20:55:06.366614     776 clientconn.go:577] ClientConn switching balancer to "pick_first"
Nov 06 20:55:06 nodename kubelet[776]: E1106 20:55:06.368047     776 goroutinemap.go:150] Operation for "/var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock" failed. No retries permitted until 2019-11-06 20:57:08.368022193 +0000 UTC m=+871.375258670 (durationBeforeRetry 2m2s). Error: "RegisterPlugin error -- failed to get plugin info using RPC GetInfo at socket /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock, err: rpc error: code = Unimplemented desc = unknown service pluginregistration.Registration"

It works though: ebs volumes are successfully created, mounted, and destroyed, when necessary.

And the driver looks registered:

$ kubectl get csidrivers.storage.k8s.io
NAME              CREATED AT
ebs.csi.aws.com   2019-10-07T21:03:19Z

And the file socket looks like exists there

$ ls -la /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
srwxr-xr-x 1 root root 0 Nov  6 20:42 /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock

What you expected to happen?

I think there should be no such error message

How to reproduce it (as minimally and precisely as possible)?

aws-ebs-csi-driver was installed using the following values (manifests generated from heml chart then applied):

image:
  tag: "v0.4.0"

enableVolumeScheduling: true

enableVolumeResizing: true

resources:
  limits:
    cpu: 100m
    memory: 128Mi
  requests:
    cpu: 100m
    memory: 128Mi

Anything else we need to know?:

Environment

  • Kubernetes version (use kubectl version):
Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.0", GitCommit:"2bd9643cee5b3b3a5ecbd3af49d09018f0773c77", GitTreeState:"clean", BuildDate:"2019-09-18T14:36:53Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.0", GitCommit:"2bd9643cee5b3b3a5ecbd3af49d09018f0773c77", GitTreeState:"clean", BuildDate:"2019-09-18T14:27:17Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}
  • Driver version: 0.4.0

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Reactions: 10
  • Comments: 33 (18 by maintainers)

Most upvoted comments

https://github.com/kubernetes-csi/node-driver-registrar#usage

Note that before Kubernetes v1.17, if the csi socket is in the /var/lib/kubelet/plugins/ path, kubelet may log a lot of harmless errors regarding grpc GetInfo call not implemented (fix in kubernetes/kubernetes#84533). The /var/lib/kubelet/csi-plugins/ path is preferred in Kubernetes versions prior to v1.17.

I have the same problem on kubernetes 1.15.3 and driver version 0.4.0 kubelet error logs as follows: Dec 9 22:59:12 ip-10-0-8-162 kubelet: E1209 22:59:12.830347 24463 goroutinemap.go:150] Operation for "/var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock" failed. No retries permitted until 2019-12-09 23:01:14.830320874 +0800 CST m=+2205.924636419 (durationBeforeRetry 2m2s). Error: "RegisterPlugin error -- failed to get plugin info using RPC GetInfo at socket /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock, err: rpc error: code = Unimplemented desc = unknown service pluginregistration.Registration"

Is there any possible reason?

Hi @zerkms, please upgrade to v2.7.0+ which likely fixes this issue.

/reopen

@zerkms thx for reporting the issue. My comment was just a small findings I got while debugging other issues. More time still need to be spent on this issue to root cause the problem