kubernetes: HA Kubernetes (v1.17.3) kubeadm init failure

What happened:

Hey everyone! Not sure why this is happening, I have not tried to set up HA k8s before this. Running normal kubeadm init works… but I am trying to use HA Kubernetes and getting these following errors. Any pointers would be GREAT.

While installing HA kubernetes via command line with kubeadm, I am repeadetly blocked by kubelet message:

kubeadm init --config config.yaml --upload-certs

Contents of config.yaml:

apiVersion: kubeadm.k8s.io/v1beta2
bootstrapTokens:
- groups:
  - system:bootstrappers:kubeadm:default-node-token
  token: abcdef.0123456789abcdef
  ttl: 24h0m0s
  usages:
  - signing
  - authentication
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress:
  bindPort: 6444 #Changed this to 6444 per some documents I read through on GitHub issues... also tried with 6443
nodeRegistration:
  criSocket: /var/run/dockershim.sock
  name: 
  taints:
  - effect: NoSchedule
    key: node-role.kubernetes.io/master
---
apiServer:
  timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta2
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controllerManager: {}
dns:
  type: CoreDNS
etcd:
  local:
    dataDir: /var/lib/etcd
imageRepository: k8s.gcr.io
kind: ClusterConfiguration
controlPlaneEndpoint: "LoadBalancerDNS:6443"
kubernetesVersion: v1.17.3
networking:
  dnsDomain: cluster.local
  serviceSubnet: 10.96.0.0/12
scheduler: {}
---
address: 0.0.0.0
apiVersion: kubelet.config.k8s.io/v1beta1
authentication:
  anonymous:
    enabled: false
  webhook:
    cacheTTL: 2m0s
    enabled: true
  x509:
    clientCAFile: /etc/kubernetes/pki/ca.crt
authorization:
  mode: Webhook
  webhook:
    cacheAuthorizedTTL: 5m0s
    cacheUnauthorizedTTL: 30s
cgroupDriver: systemd
cgroupsPerQOS: true
clusterDNS:
- 10.96.0.10
clusterDomain: cluster.local
configMapAndSecretChangeDetectionStrategy: Watch
containerLogMaxFiles: 5
containerLogMaxSize: 10Mi
contentType: application/vnd.kubernetes.protobuf
cpuCFSQuota: true
cpuCFSQuotaPeriod: 100ms
cpuManagerPolicy: none
cpuManagerReconcilePeriod: 10s
enableControllerAttachDetach: true
enableDebuggingHandlers: true
enforceNodeAllocatable:
- pods
eventBurst: 10
eventRecordQPS: 5
evictionHard:
  imagefs.available: 15%
  memory.available: 100Mi
  nodefs.available: 10%
  nodefs.inodesFree: 5%
evictionPressureTransitionPeriod: 5m0s
failSwapOn: false
fileCheckFrequency: 20s
hairpinMode: promiscuous-bridge
healthzBindAddress: 127.0.0.1
healthzPort: 10248
httpCheckFrequency: 20s
imageGCHighThresholdPercent: 85
imageGCLowThresholdPercent: 80
imageMinimumGCAge: 2m0s
iptablesDropBit: 15
iptablesMasqueradeBit: 14
kind: KubeletConfiguration
kubeAPIBurst: 10
kubeAPIQPS: 5
makeIPTablesUtilChains: true
maxOpenFiles: 1000000
maxPods: 110
nodeLeaseDurationSeconds: 80 #changed from 40
nodeStatusReportFrequency: 1m0s
nodeStatusUpdateFrequency: 10s
oomScoreAdj: -999
podPidsLimit: -1
port: 10250
registryBurst: 10
registryPullQPS: 5
resolvConf: /etc/resolv.conf
rotateCertificates: true
runtimeRequestTimeout: 4m0s
serializeImagePulls: true
staticPodPath: /etc/kubernetes/manifests
streamingConnectionIdleTimeout: 4h0m0s
syncFrequency: 1m0s
topologyManagerPolicy: none
volumeStatsAggPeriod: 1m0s

Failed to list *v1.Node: Get https://load-balancer-dns:6443/api/v1/nodes?fieldSelector=metadata.name%3DCurrentMasterNodeName&limit=500&resourceVersion=0: x509: certificate signed by unknown authority

This is the same for all: Failed to list *v1beta1.CSIDriver Failed to list *v1.Service Failed to list *v1.Node Failed to get status for pod "etcd-NodeNameMaster

For all of them, this is ending with x509: certificate signed by unknown authority

What you expected to happen:

I expected Kubernetes to set up the first control plane node for me and use the F5 load balancer that I provided

How to reproduce it (as minimally and precisely as possible):

Follow all docs here:

• https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/high-availability/ My set up is for a stacked control plane

Anything else we need to know?:

Tried the following and set these params: net.bridge.bridge-nf-call-iptables=1 net.bridge.bridge-nf-call-ip6tables=1 net.ipv4.ip_forward = 1

Driver for Docker and kubelet: systemd

Environment:

• Kubernetes version (use kubectl version): kubeadm kubelet kubectl versions: v1.17.3 Docker version 19.03.11

• Cloud provider or hardware configuration: Bare-metal with an F5 load balancer instead of HAproxy LB

• OS (e.g: cat /etc/os-release): NAME=“Red Hat Enterprise Linux Server” VERSION=“7.7 (Maipo)” ID=“rhel” ID_LIKE=“fedora” VARIANT=“Server” VARIANT_ID=“server” VERSION_ID=“7.7”

• Kernel (e.g. uname -a): Linux MasterNode 3.10.0-1062.18.1.el7.x86_64 #1 SMP Wed Feb 12 14:08:31 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

• Install tools: yum

• Network plugin and version (if this is a network-related bug):

• Others:

Jun 10 11:19:30 MasterNode kubelet[3195]: Flag --cgroup-driver has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-
Jun 10 11:19:30 MasterNode kubelet[3195]: Flag --cgroup-driver has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-
Jun 10 11:19:30 MasterNode kubelet[3195]: I0610 11:19:30.964228    3195 server.go:416] Version: v1.17.3
Jun 10 11:19:30 MasterNode kubelet[3195]: I0610 11:19:30.964703    3195 plugins.go:100] No cloud provider specified.
Jun 10 11:19:30 MasterNode kubelet[3195]: I0610 11:19:30.964939    3195 server.go:821] Client rotation is on, will bootstrap in background
Jun 10 11:19:30 MasterNode kubelet[3195]: I0610 11:19:30.967369    3195 certificate_store.go:129] Loading cert/key pair from "/var/lib/kubelet/pki/kubelet-client-current.pem".
Jun 10 11:19:31 MasterNode kubelet[3195]: I0610 11:19:31.054304    3195 server.go:641] --cgroups-per-qos enabled, but --cgroup-root was not specified.  defaulting to /
Jun 10 11:19:31 MasterNode kubelet[3195]: I0610 11:19:31.054653    3195 container_manager_linux.go:265] container manager verified user specified cgroup-root exists: []
Jun 10 11:19:31 MasterNode kubelet[3195]: I0610 11:19:31.054687    3195 container_manager_linux.go:270] Creating Container Manager object based on Node Config: {RuntimeCgroupsName: SystemCgroupsName: KubeletCgroupsName: Con
Jun 10 11:19:31 MasterNode kubelet[3195]: I0610 11:19:31.054789    3195 fake_topology_manager.go:29] [fake topologymanager] NewFakeManager
Jun 10 11:19:31 MasterNode kubelet[3195]: I0610 11:19:31.054794    3195 container_manager_linux.go:305] Creating device plugin manager: true
Jun 10 11:19:31 MasterNode kubelet[3195]: I0610 11:19:31.054826    3195 fake_topology_manager.go:39] [fake topologymanager] AddHintProvider HintProvider:  &{kubelet.sock /var/lib/kubelet/device-plugins/ map[] {0 0} <nil> {{
Jun 10 11:19:31 MasterNode kubelet[3195]: I0610 11:19:31.054872    3195 state_mem.go:36] [cpumanager] initializing new in-memory state store
Jun 10 11:19:31 MasterNode kubelet[3195]: I0610 11:19:31.054979    3195 state_mem.go:84] [cpumanager] updated default cpuset: ""
Jun 10 11:19:31 MasterNode kubelet[3195]: I0610 11:19:31.054985    3195 state_mem.go:92] [cpumanager] updated cpuset assignments: "map[]"
Jun 10 11:19:31 MasterNode kubelet[3195]: I0610 11:19:31.054994    3195 fake_topology_manager.go:39] [fake topologymanager] AddHintProvider HintProvider:  &{{0 0} 0x6e9bc50 10000000000 0xc000ace6c0 <nil> <nil> <nil> <nil> m
Jun 10 11:19:31 MasterNode kubelet[3195]: I0610 11:19:31.055060    3195 kubelet.go:286] Adding pod path: /etc/kubernetes/manifests
Jun 10 11:19:31 MasterNode kubelet[3195]: I0610 11:19:31.055115    3195 kubelet.go:311] Watching apiserver
Jun 10 11:19:31 MasterNode kubelet[3195]: I0610 11:19:31.069097    3195 client.go:75] Connecting to docker on unix:///var/run/docker.sock
Jun 10 11:19:31 MasterNode kubelet[3195]: I0610 11:19:31.069444    3195 client.go:104] Start docker client with request timeout=4m0s
Jun 10 11:19:31 MasterNode kubelet[3195]: E0610 11:19:31.076542    3195 reflector.go:153] k8s.io/kubernetes/pkg/kubelet/kubelet.go:449: Failed to list *v1.Service: Get https://LoadBalancerIP:6443/api/v1/services?limit
Jun 10 11:19:31 MasterNode kubelet[3195]: E0610 11:19:31.077278    3195 reflector.go:153] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: Failed to list *v1.Pod: Get https://LoadBalancerIP:6443/api/v1/pods?field
Jun 10 11:19:31 MasterNode kubelet[3195]: E0610 11:19:31.077500    3195 reflector.go:153] k8s.io/kubernetes/pkg/kubelet/kubelet.go:458: Failed to list *v1.Node: Get https://LoadBalancerIP:6443/api/v1/nodes?fieldSelect
Jun 10 11:19:31 MasterNode kubelet[3195]: W0610 11:19:31.079822    3195 docker_service.go:563] Hairpin mode set to "promiscuous-bridge" but kubenet is not enabled, falling back to "hairpin-veth"
Jun 10 11:19:31 MasterNode kubelet[3195]: I0610 11:19:31.079844    3195 docker_service.go:240] Hairpin mode set to "hairpin-veth"
Jun 10 11:19:31 MasterNode kubelet[3195]: W0610 11:19:31.079937    3195 cni.go:237] Unable to update cni config: no networks found in /etc/cni/net.d
Jun 10 11:19:31 MasterNode kubelet[3195]: W0610 11:19:31.083724    3195 cni.go:237] Unable to update cni config: no networks found in /etc/cni/net.d
Jun 10 11:19:31 MasterNode kubelet[3195]: I0610 11:19:31.083778    3195 docker_service.go:255] Docker cri networking managed by cni
Jun 10 11:19:31 MasterNode kubelet[3195]: W0610 11:19:31.083836    3195 cni.go:237] Unable to update cni config: no networks found in /etc/cni/net.d
Jun 10 11:19:31 MasterNode kubelet[3195]: I0610 11:19:31.093083    3195 docker_service.go:260] Docker Info: &{ID:CKKA:YUJL:2557:3IQE:7MRG:35J3:B2MN:GC3H:3WMM:FCV4:R2BU:TQW4 Containers:8 ContainersRunning:8 ContainersPaused:
Jun 10 11:19:31 MasterNode kubelet[3195]: I0610 11:19:31.093148    3195 docker_service.go:273] Setting cgroupDriver to systemd
Jun 10 11:19:31 MasterNode kubelet[3195]: I0610 11:19:31.103239    3195 remote_runtime.go:59] parsed scheme: ""
Jun 10 11:19:31 MasterNode kubelet[3195]: I0610 11:19:31.103260    3195 remote_runtime.go:59] scheme "" not registered, fallback to default scheme
Jun 10 11:19:31 MasterNode kubelet[3195]: I0610 11:19:31.103297    3195 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{/var/run/dockershim.sock 0  <nil>}] <nil>}
Jun 10 11:19:31 MasterNode kubelet[3195]: I0610 11:19:31.103321    3195 clientconn.go:577] ClientConn switching balancer to "pick_first"
Jun 10 11:19:31 MasterNode kubelet[3195]: I0610 11:19:31.103352    3195 remote_image.go:50] parsed scheme: ""
Jun 10 11:19:31 MasterNode kubelet[3195]: I0610 11:19:31.103358    3195 remote_image.go:50] scheme "" not registered, fallback to default scheme
Jun 10 11:19:31 MasterNode kubelet[3195]: I0610 11:19:31.103366    3195 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{/var/run/dockershim.sock 0  <nil>}] <nil>}
Jun 10 11:19:31 MasterNode kubelet[3195]: I0610 11:19:31.103370    3195 clientconn.go:577] ClientConn switching balancer to "pick_first"
Jun 10 11:19:32 MasterNode kubelet[3195]: E0610 11:19:32.083448    3195 reflector.go:153] k8s.io/kubernetes/pkg/kubelet/kubelet.go:449: Failed to list *v1.Service: Get https://LoadBalancerIP:6443/api/v1/services?limit
Jun 10 11:19:32 MasterNode kubelet[3195]: E0610 11:19:32.086800    3195 reflector.go:153] k8s.io/kubernetes/pkg/kubelet/kubelet.go:458: Failed to list *v1.Node: Get https://LoadBalancerIP:6443/api/v1/nodes?fieldSelect
Jun 10 11:19:32 MasterNode kubelet[3195]: E0610 11:19:32.086872    3195 reflector.go:153] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: Failed to list *v1.Pod: Get https://LoadBalancerIP:6443/api/v1/pods?field
Jun 10 11:19:33 MasterNode kubelet[3195]: E0610 11:19:33.090523    3195 reflector.go:153] k8s.io/kubernetes/pkg/kubelet/kubelet.go:449: Failed to list *v1.Service: Get https://LoadBalancerIP:6443/api/v1/services?limit
Jun 10 11:19:33 MasterNode kubelet[3195]: E0610 11:19:33.094073    3195 reflector.go:153] k8s.io/kubernetes/pkg/kubelet/kubelet.go:458: Failed to list *v1.Node: Get https://LoadBalancerIP:6443/api/v1/nodes?fieldSelect
Jun 10 11:19:33 MasterNode kubelet[3195]: E0610 11:19:33.094838    3195 reflector.go:153] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: Failed to list *v1.Pod: Get https://LoadBalancerIP:6443/api/v1/pods?field
Jun 10 11:19:34 MasterNode kubelet[3195]: E0610 11:19:34.097327    3195 reflector.go:153] k8s.io/kubernetes/pkg/kubelet/kubelet.go:449: Failed to list *v1.Service: Get https://LoadBalancerIP:6443/api/v1/services?limit
Jun 10 11:19:34 MasterNode kubelet[3195]: E0610 11:19:34.102600    3195 reflector.go:153] k8s.io/kubernetes/pkg/kubelet/kubelet.go:458: Failed to list *v1.Node: Get https://LoadBalancerIP:6443/api/v1/nodes?fieldSelect
Jun 10 11:19:34 MasterNode kubelet[3195]: E0610 11:19:34.103424    3195 reflector.go:153] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: Failed to list *v1.Pod: Get https://LoadBalancerIP:6443/api/v1/pods?field
Jun 10 11:19:34 MasterNode kubelet[3195]: E0610 11:19:34.108470    3195 aws_credentials.go:77] while getting AWS credentials NoCredentialProviders: no valid providers in chain. Deprecated.
Jun 10 11:19:34 MasterNode kubelet[3195]: For verbose messaging see aws.Config.CredentialsChainVerboseErrors
Jun 10 11:19:34 MasterNode kubelet[3195]: I0610 11:19:34.121933    3195 kuberuntime_manager.go:211] Container runtime docker initialized, version: 19.03.11, apiVersion: 1.40.0
Jun 10 11:19:34 MasterNode kubelet[3195]: I0610 11:19:34.135482    3195 server.go:1113] Started kubelet
Jun 10 11:19:34 MasterNode kubelet[3195]: E0610 11:19:34.135999    3195 kubelet.go:1302] Image garbage collection failed once. Stats initialization may not have completed yet: failed to get imageFs info: unable to find data
Jun 10 11:19:34 MasterNode kubelet[3195]: I0610 11:19:34.136895    3195 fs_resource_analyzer.go:64] Starting FS ResourceAnalyzer
Jun 10 11:19:34 MasterNode kubelet[3195]: I0610 11:19:34.138046    3195 server.go:144] Starting to listen on 0.0.0.0:10250
Jun 10 11:19:34 MasterNode kubelet[3195]: I0610 11:19:34.140208    3195 server.go:384] Adding debug handlers to kubelet server.
Jun 10 11:19:34 MasterNode kubelet[3195]: I0610 11:19:34.151892    3195 volume_manager.go:265] Starting Kubelet Volume Manager
Jun 10 11:19:34 MasterNode kubelet[3195]: I0610 11:19:34.152428    3195 desired_state_of_world_populator.go:138] Desired state populator starts to run