openyurt: [BUG] openyurt v1.2.0 - pool-coordinator not working because of missing images and invalid certificates

What happened:

When I deploy the openyurt helm chart as described, the pool coordinator does not start because google_containers/etcd and google_containers/kube-apiserver do not exist on docker hub.

After replacing the images with the following:

poolCoordinator:
        apiserverImage:
          registry: registry.k8s.io
          repository: kube-apiserver
          tag: v1.22.17
        etcdImage:
          registry: quay.io
          repository: coreos/etcd
          tag: v3.5.0

the pool-coorinatior runs in CrashLoopBackup because the certificates are not correct:

kube-apiserver

I0131 17:10:46.948544       1 server.go:553] external host was not specified, using 192.168.88.248
I0131 17:10:46.949264       1 server.go:161] Version: v1.22.17
I0131 17:10:48.087533       1 shared_informer.go:240] Waiting for caches to sync for node_authorizer
I0131 17:10:48.089422       1 plugins.go:158] Loaded 11 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.
I0131 17:10:48.089444       1 plugins.go:161] Loaded 10 validating admission controller(s) successfully in the following order: LimitRanger,PodSecurity,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.
I0131 17:10:48.091339       1 plugins.go:158] Loaded 11 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.
I0131 17:10:48.091359       1 plugins.go:161] Loaded 10 validating admission controller(s) successfully in the following order: LimitRanger,PodSecurity,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.
W0131 17:10:48.255284       1 clientconn.go:1326] [core] grpc: addrConn.createTransport failed to connect to {127.0.0.1:12379 127.0.0.1 <nil> 0 <nil>}. Err: connection error: desc = "transport: authentication handshake failed: x509: certificate is valid for 10.110.120.6, 10.110.120.6, not 127.0.0.1". Reconnecting...
W0131 17:10:49.102574       1 clientconn.go:1326] [core] grpc: addrConn.createTransport failed to connect to {127.0.0.1:12379 127.0.0.1 <nil> 0 <nil>}. Err: connection error: desc = "transport: authentication handshake failed: x509: certificate is valid for 10.110.120.6, 10.110.120.6, not 127.0.0.1". Reconnecting...
W0131 17:10:49.272927       1 clientconn.go:1326] [core] grpc: addrConn.createTransport failed to connect to {127.0.0.1:12379 127.0.0.1 <nil> 0 <nil>}. Err: connection error: desc = "transport: authentication handshake failed: x509: certificate is valid for 10.110.120.6, 10.110.120.6, not 127.0.0.1". Reconnecting...

etcd

{"level":"info","ts":"2023-01-31T17:10:47.354Z","caller":"etcdmain/etcd.go:72","msg":"Running: ","args":["etcd","--advertise-client-urls=https://0.0.0.0:12379","--listen-client-urls=https://0.0.0.0:12379","--cert-file=/etc/kubernetes/pki/etcd-server.crt","--client-cert-auth=true","--max-txn-ops=102400","--data-dir=/var/lib/etcd","--max-request-bytes=100000000","--key-file=/etc/kubernetes/pki/etcd-server.key","--listen-metrics-urls=http://0.0.0.0:12381","--snapshot-count=10000","--trusted-ca-file=/etc/kubernetes/pki/ca.crt"]}
{"level":"info","ts":"2023-01-31T17:10:47.354Z","caller":"embed/etcd.go:131","msg":"configuring peer listeners","listen-peer-urls":["http://localhost:2380"]}
{"level":"info","ts":"2023-01-31T17:10:47.355Z","caller":"embed/etcd.go:139","msg":"configuring client listeners","listen-client-urls":["https://0.0.0.0:12379"]}
{"level":"info","ts":"2023-01-31T17:10:47.356Z","caller":"embed/etcd.go:307","msg":"starting an etcd server","etcd-version":"3.5.0","git-sha":"946a5a6f2","go-version":"go1.16.3","go-os":"linux","go-arch":"amd64","max-cpu-set":4,"max-cpu-available":4,"member-initialized":false,"name":"default","data-dir":"/var/lib/etcd","wal-dir":"","wal-dir-dedicated":"","member-dir":"/var/lib/etcd/member","force-new-cluster":false,"heartbeat-interval":"100ms","election-timeout":"1s","initial-election-tick-advance":true,"snapshot-count":10000,"snapshot-catchup-entries":5000,"initial-advertise-peer-urls":["http://localhost:2380"],"listen-peer-urls":["http://localhost:2380"],"advertise-client-urls":["https://0.0.0.0:12379"],"listen-client-urls":["https://0.0.0.0:12379"],"listen-metrics-urls":["http://0.0.0.0:12381"],"cors":["*"],"host-whitelist":["*"],"initial-cluster":"default=http://localhost:2380","initial-cluster-state":"new","initial-cluster-token":"etcd-cluster","quota-size-bytes":2147483648,"pre-vote":true,"initial-corrupt-check":false,"corrupt-check-time-interval":"0s","auto-compaction-mode":"periodic","auto-compaction-retention":"0s","auto-compaction-interval":"0s","discovery-url":"","discovery-proxy":"","downgrade-check-interval":"5s"}
{"level":"warn","ts":"2023-01-31T17:10:47.356Z","caller":"etcdserver/server.go:342","msg":"exceeded recommended request limit","max-request-bytes":100000000,"max-request-size":"100 MB","recommended-request-bytes":10485760,"recommended-request-size":"10 MB"}
{"level":"warn","ts":1675185047.3563454,"caller":"fileutil/fileutil.go:57","msg":"check file permission","error":"directory \"/var/lib/etcd\" exist, but the permission is \"dtrwxrwxrwx\". The recommended permission is \"-rwx------\" to prevent possible unprivileged access to the data"}
{"level":"info","ts":"2023-01-31T17:10:47.356Z","caller":"etcdserver/backend.go:81","msg":"opened backend db","path":"/var/lib/etcd/member/snap/db","took":"282.354µs"}
{"level":"info","ts":"2023-01-31T17:10:47.460Z","caller":"etcdserver/raft.go:448","msg":"starting local member","local-member-id":"8e9e05c52164694d","cluster-id":"cdf818194e3a8c32"}
{"level":"info","ts":"2023-01-31T17:10:47.460Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8e9e05c52164694d switched to configuration voters=()"}
{"level":"info","ts":"2023-01-31T17:10:47.460Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8e9e05c52164694d became follower at term 0"}
{"level":"info","ts":"2023-01-31T17:10:47.460Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"newRaft 8e9e05c52164694d [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]"}
{"level":"info","ts":"2023-01-31T17:10:47.460Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8e9e05c52164694d became follower at term 1"}
{"level":"info","ts":"2023-01-31T17:10:47.460Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8e9e05c52164694d switched to configuration voters=(10276657743932975437)"}
{"level":"warn","ts":"2023-01-31T17:10:47.460Z","caller":"auth/store.go:1220","msg":"simple token is not cryptographically signed"}
{"level":"info","ts":"2023-01-31T17:10:47.549Z","caller":"mvcc/kvstore.go:415","msg":"kvstore restored","current-rev":1}
{"level":"info","ts":"2023-01-31T17:10:47.550Z","caller":"etcdserver/quota.go:94","msg":"enabled backend quota with default value","quota-name":"v3-applier","quota-size-bytes":2147483648,"quota-size":"2.1 GB"}
{"level":"info","ts":"2023-01-31T17:10:47.550Z","caller":"etcdserver/server.go:843","msg":"starting etcd server","local-member-id":"8e9e05c52164694d","local-server-version":"3.5.0","cluster-version":"to_be_decided"}
{"level":"info","ts":"2023-01-31T17:10:47.550Z","caller":"etcdserver/server.go:728","msg":"started as single-node; fast-forwarding election ticks","local-member-id":"8e9e05c52164694d","forward-ticks":9,"forward-duration":"900ms","election-ticks":10,"election-timeout":"1s"}
{"level":"info","ts":"2023-01-31T17:10:47.551Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8e9e05c52164694d switched to configuration voters=(10276657743932975437)"}
{"level":"info","ts":"2023-01-31T17:10:47.552Z","caller":"membership/cluster.go:393","msg":"added member","cluster-id":"cdf818194e3a8c32","local-member-id":"8e9e05c52164694d","added-peer-id":"8e9e05c52164694d","added-peer-peer-urls":["http://localhost:2380"]}
{"level":"info","ts":"2023-01-31T17:10:47.555Z","caller":"embed/etcd.go:687","msg":"starting with client TLS","tls-info":"cert = /etc/kubernetes/pki/etcd-server.crt, key = /etc/kubernetes/pki/etcd-server.key, client-cert=, client-key=, trusted-ca = /etc/kubernetes/pki/ca.crt, client-cert-auth = true, crl-file = ","cipher-suites":[]}
{"level":"info","ts":"2023-01-31T17:10:47.649Z","caller":"embed/etcd.go:580","msg":"serving peer traffic","address":"127.0.0.1:2380"}
{"level":"info","ts":"2023-01-31T17:10:47.650Z","caller":"embed/etcd.go:552","msg":"cmux::serve","address":"127.0.0.1:2380"}
{"level":"info","ts":"2023-01-31T17:10:47.649Z","caller":"embed/etcd.go:276","msg":"now serving peer/client/metrics","local-member-id":"8e9e05c52164694d","initial-advertise-peer-urls":["http://localhost:2380"],"listen-peer-urls":["http://localhost:2380"],"advertise-client-urls":["https://0.0.0.0:12379"],"listen-client-urls":["https://0.0.0.0:12379"],"listen-metrics-urls":["http://0.0.0.0:12381"]}
{"level":"info","ts":"2023-01-31T17:10:47.649Z","caller":"embed/etcd.go:762","msg":"serving metrics","address":"http://0.0.0.0:12381"}
{"level":"info","ts":"2023-01-31T17:10:48.161Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8e9e05c52164694d is starting a new election at term 1"}
{"level":"info","ts":"2023-01-31T17:10:48.161Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8e9e05c52164694d became pre-candidate at term 1"}
{"level":"info","ts":"2023-01-31T17:10:48.161Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8e9e05c52164694d received MsgPreVoteResp from 8e9e05c52164694d at term 1"}
{"level":"info","ts":"2023-01-31T17:10:48.162Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8e9e05c52164694d became candidate at term 2"}
{"level":"info","ts":"2023-01-31T17:10:48.162Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8e9e05c52164694d received MsgVoteResp from 8e9e05c52164694d at term 2"}
{"level":"info","ts":"2023-01-31T17:10:48.162Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8e9e05c52164694d became leader at term 2"}
{"level":"info","ts":"2023-01-31T17:10:48.162Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"raft.node: 8e9e05c52164694d elected leader 8e9e05c52164694d at term 2"}
{"level":"info","ts":"2023-01-31T17:10:48.162Z","caller":"etcdserver/server.go:2476","msg":"setting up initial cluster version using v2 API","cluster-version":"3.5"}
{"level":"info","ts":"2023-01-31T17:10:48.162Z","caller":"embed/serve.go:98","msg":"ready to serve client requests"}
{"level":"info","ts":"2023-01-31T17:10:48.162Z","caller":"etcdserver/server.go:2027","msg":"published local member to cluster through raft","local-member-id":"8e9e05c52164694d","local-member-attributes":"{Name:default ClientURLs:[https://0.0.0.0:12379]}","request-path":"/0/members/8e9e05c52164694d/attributes","cluster-id":"cdf818194e3a8c32","publish-timeout":"7s"}
{"level":"info","ts":"2023-01-31T17:10:48.163Z","caller":"membership/cluster.go:531","msg":"set initial cluster version","cluster-id":"cdf818194e3a8c32","local-member-id":"8e9e05c52164694d","cluster-version":"3.5"}
{"level":"info","ts":"2023-01-31T17:10:48.163Z","caller":"api/capability.go:75","msg":"enabled capabilities for version","cluster-version":"3.5"}
{"level":"info","ts":"2023-01-31T17:10:48.163Z","caller":"etcdserver/server.go:2500","msg":"cluster version is updated","cluster-version":"3.5"}
{"level":"info","ts":"2023-01-31T17:10:48.163Z","caller":"etcdmain/main.go:47","msg":"notifying init daemon"}
{"level":"info","ts":"2023-01-31T17:10:48.163Z","caller":"etcdmain/main.go:53","msg":"successfully notified init daemon"}
{"level":"info","ts":"2023-01-31T17:10:48.166Z","caller":"embed/serve.go:188","msg":"serving client traffic securely","address":"[::]:12379"}
{"level":"warn","ts":"2023-01-31T17:10:48.349Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"127.0.0.1:39106","server-name":"","error":"remote error: tls: bad certificate"}
{"level":"warn","ts":"2023-01-31T17:10:48.355Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"127.0.0.1:39122","server-name":"","error":"tls: failed to verify client certificate: x509: certificate specifies an incompatible key usage"}
WARNING: 2023/01/31 17:10:48 [core] grpc: addrConn.createTransport failed to connect to {0.0.0.0:12379 0.0.0.0:12379 <nil> 0 <nil>}. Err: connection error: desc = "transport: authentication handshake failed: remote error: tls: bad certificate". Reconnecting...
{"level":"warn","ts":"2023-01-31T17:10:49.102Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"127.0.0.1:39138","server-name":"","error":"remote error: tls: bad certificate"}
{"level":"warn","ts":"2023-01-31T17:10:49.272Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"127.0.0.1:39152","server-name":"","error":"remote error: tls: bad certificate"}
{"level":"warn","ts":"2023-01-31T17:10:49.457Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"127.0.0.1:39156","server-name":"","error":"tls: failed to verify client certificate: x509: certificate specifies an incompatible key usage"}
WARNING: 2023/01/31 17:10:49 [core] grpc: addrConn.createTransport failed to connect to {0.0.0.0:12379 0.0.0.0:12379 <nil> 0 <nil>}. Err: connection error: desc = "transport: authentication handshake failed: remote error: tls: bad certificate". Reconnecting...
{"level":"warn","ts":"2023-01-31T17:10:50.117Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"127.0.0.1:39162","server-name":"","error":"remote error: tls: bad certificate"}
{"level":"warn","ts":"2023-01-31T17:10:50.999Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"127.0.0.1:39176","server-name":"","error":"tls: failed to verify client certificate: x509: certificate specifies an incompatible key usage"}
WARNING: 2023/01/31 17:10:50 [core] grpc: addrConn.createTransport failed to connect to {0.0.0.0:12379 0.0.0.0:12379 <nil> 0 <nil>}. Err: connection error: desc = "transport: authentication handshake failed: remote error: tls: bad certificate". Reconnecting...
{"level":"warn","ts":"2023-01-31T17:10:51.083Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"127.0.0.1:39182","server-name":"","error":"remote error: tls: bad certificate"}
{"level":"warn","ts":"2023-01-31T17:10:51.756Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"127.0.0.1:39190","server-name":"","error":"remote error: tls: bad certificate"}
{"level":"warn","ts":"2023-01-31T17:10:53.344Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"127.0.0.1:39206","server-name":"","error":"tls: failed to verify client certificate: x509: certificate specifies an incompatible key usage"}
WARNING: 2023/01/31 17:10:53 [core] grpc: addrConn.createTransport failed to connect to {0.0.0.0:12379 0.0.0.0:12379 <nil> 0 <nil>}. Err: connection error: desc = "transport: authentication handshake failed: remote error: tls: bad certificate". Reconnecting...
{"level":"warn","ts":"2023-01-31T17:10:53.909Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"127.0.0.1:48360","server-name":"","error":"remote error: tls: bad certificate"}
{"level":"warn","ts":"2023-01-31T17:10:53.975Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"127.0.0.1:48370","server-name":"","error":"remote error: tls: bad certificate"}
{"level":"warn","ts":"2023-01-31T17:10:57.254Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"127.0.0.1:48384","server-name":"","error":"tls: failed to verify client certificate: x509: certificate specifies an incompatible key usage"}
WARNING: 2023/01/31 17:10:57 [core] grpc: addrConn.createTransport failed to connect to {0.0.0.0:12379 0.0.0.0:12379 <nil> 0 <nil>}. Err: connection error: desc = "transport: authentication handshake failed: remote error: tls: bad certificate". Reconnecting...
{"level":"warn","ts":"2023-01-31T17:10:57.386Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"127.0.0.1:48398","server-name":"","error":"remote error: tls: bad certificate"}
{"level":"warn","ts":"2023-01-31T17:10:57.876Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"127.0.0.1:48408","server-name":"","error":"remote error: tls: bad certificate"}
{"level":"warn","ts":"2023-01-31T17:11:03.072Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"127.0.0.1:48418","server-name":"","error":"tls: failed to verify client certificate: x509: certificate specifies an incompatible key usage"}
WARNING: 2023/01/31 17:11:03 [core] grpc: addrConn.createTransport failed to connect to {0.0.0.0:12379 0.0.0.0:12379 <nil> 0 <nil>}. Err: connection error: desc = "transport: authentication handshake failed: remote error: tls: bad certificate". Reconnecting...
{"level":"warn","ts":"2023-01-31T17:11:03.752Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"127.0.0.1:48430","server-name":"","error":"remote error: tls: bad certificate"}
{"level":"warn","ts":"2023-01-31T17:11:03.773Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"127.0.0.1:41384","server-name":"","error":"remote error: tls: bad certificate"}
{"level":"warn","ts":"2023-01-31T17:11:10.232Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"127.0.0.1:41392","server-name":"","error":"remote error: tls: bad certificate"}
{"level":"warn","ts":"2023-01-31T17:11:11.227Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"127.0.0.1:41404","server-name":"","error":"remote error: tls: bad certificate"}
{"level":"warn","ts":"2023-01-31T17:11:11.246Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"127.0.0.1:41406","server-name":"","error":"remote error: tls: bad certificate"}
{"level":"warn","ts":"2023-01-31T17:11:12.241Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"127.0.0.1:41416","server-name":"","error":"remote error: tls: bad certificate"}
{"level":"warn","ts":"2023-01-31T17:11:13.022Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"127.0.0.1:41428","server-name":"","error":"remote error: tls: bad certificate"}
{"level":"warn","ts":"2023-01-31T17:11:13.776Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"127.0.0.1:46976","server-name":"","error":"remote error: tls: bad certificate"}
{"level":"warn","ts":"2023-01-31T17:11:14.594Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"127.0.0.1:46990","server-name":"","error":"tls: failed to verify client certificate: x509: certificate specifies an incompatible key usage"}
WARNING: 2023/01/31 17:11:14 [core] grpc: addrConn.createTransport failed to connect to {0.0.0.0:12379 0.0.0.0:12379 <nil> 0 <nil>}. Err: connection error: desc = "transport: authentication handshake failed: remote error: tls: bad certificate". Reconnecting...

How to reproduce it (as minimally and precisely as possible):

deploy openyurt by using the Manually Setup instructions and add a edge-worker node.

Environment:

• OpenYurt version: 1.2.0
• Kubernetes version (use kubectl version): 1.22.17
• OS (e.g: cat /etc/os-release): ubuntu 20.04
• Kernel (e.g. uname -a): 5.4.0-137-generic
• Install tools: manual/ansible

others

/kind bug