metrics-server: Error from server (ServiceUnavailable): the server is currently unable to handle the request (get nodes.metrics.k8s.io)

Hello all,

I would like to ask about the networking issue (aggregration layer) when installing metric-server on kubernetes. The architecture that I use here consists of 1 master node and 2 worker nodes as follows

Each node is connected without a firewall. I tried to deploy using Metric Server version 0.4.1 with the following server metric deployment.apps configuration (kubectl edit deployments.apps -n kube-system metrics-server)

apiVersion: apps/v1                                                                                                                                          
kind: Deployment                                                                                                                                             
metadata:                                                                                                                                                    
  annotations:                                                                                                                                               
    deployment.kubernetes.io/revision: "13"                                                                                                                  
    kubectl.kubernetes.io/last-applied-configuration: |                                                                                                      
      {"apiVersion":"apps/v1","kind":"Deployment","metadata":{"annotations":{},"labels":{"k8s-app":"metrics-server"},"name":"metrics-server","namespace":"kub
system"},"spec":{"selector":{"matchLabels":{"k8s-app":"metrics-server"}},"strategy":{"rollingUpdate":{"maxUnavailable":0}},"template":{"metadata":{"labels":{
8s-app":"metrics-server"}},"spec":{"containers":[{"args":["--cert-dir=/tmp","--secure-port=4443","--kubelet-preferred-address-types=InternalIP,ExternalIP,Hos
ame","--kubelet-use-node-status-port"],"image":"k8s.gcr.io/metrics-server/metrics-server:v0.4.1","imagePullPolicy":"IfNotPresent","livenessProbe":{"failureTh
shold":3,"httpGet":{"path":"/livez","port":"https","scheme":"HTTPS"},"periodSeconds":10},"name":"metrics-server","ports":[{"containerPort":4443,"name":"https
"protocol":"TCP"}],"readinessProbe":{"failureThreshold":3,"httpGet":{"path":"/readyz","port":"https","scheme":"HTTPS"},"periodSeconds":10},"securityContext":
readOnlyRootFilesystem":true,"runAsNonRoot":true,"runAsUser":1000},"volumeMounts":[{"mountPath":"/tmp","name":"tmp-dir"}]}],"nodeSelector":{"kubernetes.io/os
"linux"},"priorityClassName":"system-cluster-critical","serviceAccountName":"metrics-server","volumes":[{"emptyDir":{},"name":"tmp-dir"}]}}}}                
  creationTimestamp: "2021-05-05T17:13:02Z"                                                                                                                  
  generation: 13                                                                                                                                             
  labels:                                                                                                                                                    
    k8s-app: metrics-server                                                                                                                                  
  name: metrics-server                                                                                                                                       
  namespace: kube-system                                                                                                                                     
  resourceVersion: "7519983"                                                                                                                                 
  uid: 3e80a7e9-4992-472d-b3f4-9181417aa07f                                                                                                                  
spec:                                                                                                                                                        
  progressDeadlineSeconds: 600                                                                                                                               
  replicas: 1                                                                                                                                                
  revisionHistoryLimit: 10                                                                                                                                   
  selector:                                                                                                                                                  
    matchLabels:                                                                                                                                             
      k8s-app: metrics-server                                                                                                                                
  strategy:                                                                                                                                                  
    rollingUpdate:                                                                                                                                           
      maxSurge: 25%                                                                                                                                          
      maxUnavailable: 0 
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        k8s-app: metrics-server
    spec:
      containers:
      - args:
        - --cert-dir=/tmp
        - --secure-port=4443
        - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
        - --kubelet-insecure-tls=true
        - --kubelet-use-node-status-port
        - --metric-resolution=30s
        image: k8s.gcr.io/metrics-server/metrics-server:v0.4.1
        imagePullPolicy: Always
        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /livez
            port: https
            scheme: HTTPS
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        name: metrics-server
        ports:
        - containerPort: 4443
          hostPort: 4443
          name: https
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /readyz
            port: https
            scheme: HTTPS
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        resources: {}
        securityContext:
          readOnlyRootFilesystem: true
          runAsNonRoot: true
          runAsUser: 1000
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /tmp
          name: tmp-dir
      dnsPolicy: ClusterFirst
      hostNetwork: true
      nodeSelector:
        kubernetes.io/os: linux
      priorityClassName: system-cluster-critical
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      serviceAccount: metrics-server
      serviceAccountName: metrics-server
      terminationGracePeriodSeconds: 30
      volumes:
      - emptyDir: {}
        name: tmp-dir	  
[]

The configuration used above keeps all services running

When we look at the log of the pods, it displays the following message

And when looking at the apiservice (kubectl describe apiservice v1beta1.metrics.k8s.io) displays the following error message

  Version Priority:  100
Status:
  Conditions:
    Last Transition Time:  2021-05-05T17:13:02Z
    **Message:               failing or missing response from https://10.220.37.42:4443/apis/metrics.k8s.io/v1beta1: Get "https://10.220.37.42:4443/apis/metrics.k8s.io/v1beta1": dial tcp 10.220.37.42:4443: connect: no route to host
    Reason:                FailedDiscoveryCheck**
    Status:                False
    Type:                  Available
Events:                    <none>`

I’ve tried adding an args (–enable-aggregator-routing = true) as shown in the guide here https://github.com/kubernetes-sigs/metrics-server/issues/281 but it still returns the same error message

What you expected to happen: Networking running well and kubectl top node gave output running well

Environment:

• Kubernetes on-premise version v1.20.4
• Container Network Setup using CNI

/kind bug