proxmox-cloud-controller-manager: CCM does not labelize nodes

Bug Report

Description

Since I’ve removed kube-proxy and let routing to be done by Cilium, my CCM does not labelize nodes anymore. I made a fresh install.

Logs

Here are some logs from my proxmox-cloud-controller-manager pod :

I1021 20:09:44.245894       1 serving.go:348] Generated self-signed cert in-memory
I1021 20:09:44.836589       1 serving.go:348] Generated self-signed cert in-memory
W1021 20:09:44.836642       1 client_config.go:618] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I1021 20:09:45.281997       1 requestheader_controller.go:244] Loaded a new request header values for RequestHeaderAuthRequestController
I1021 20:09:45.283103       1 controllermanager.go:168] Version: v0.2.0
I1021 20:09:45.634437       1 requestheader_controller.go:169] Starting RequestHeaderAuthRequestController
I1021 20:09:45.634500       1 shared_informer.go:311] Waiting for caches to sync for RequestHeaderAuthRequestController
I1021 20:09:45.634529       1 configmap_cafile_content.go:202] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file"
I1021 20:09:45.634561       1 shared_informer.go:311] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I1021 20:09:45.634578       1 configmap_cafile_content.go:202] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"
I1021 20:09:45.634584       1 shared_informer.go:311] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I1021 20:09:45.635263       1 tlsconfig.go:200] "Loaded serving cert" certName="Generated self signed cert" certDetail="\"localhost@1697918984\" [serving] validServingFor=[127.0.0.1,localhost,localhost] issuer=\"localhost-ca@1697918983\" (2023-10-21 19:09:43 +0000 UTC to 2024-10-20 19:09:43 +0000 UTC (now=2023-10-21 20:09:45.635180211 +0000 UTC))"
I1021 20:09:45.636389       1 named_certificates.go:53] "Loaded SNI cert" index=0 certName="self-signed loopback" certDetail="\"apiserver-loopback-client@1697918985\" [serving] validServingFor=[apiserver-loopback-client] issuer=\"apiserver-loopback-client-ca@1697918984\" (2023-10-21 19:09:44 +0000 UTC to 2024-10-20 19:09:44 +0000 UTC (now=2023-10-21 20:09:45.636344007 +0000 UTC))"
I1021 20:09:45.636684       1 secure_serving.go:210] Serving securely on :10258
I1021 20:09:45.637374       1 leaderelection.go:250] attempting to acquire leader lease kube-system/cloud-controller-manager-proxmox...
I1021 20:09:45.636900       1 tlsconfig.go:240] "Starting DynamicServingCertificateController"
I1021 20:09:45.734941       1 shared_informer.go:318] Caches are synced for RequestHeaderAuthRequestController
I1021 20:09:45.734939       1 shared_informer.go:318] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I1021 20:09:45.735291       1 shared_informer.go:318] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I1021 20:09:45.735365       1 tlsconfig.go:178] "Loaded client CA" index=0 certName="client-ca::kube-system::extension-apiserver-authentication::client-ca-file,client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" certDetail="\"front-proxy-ca\" [] validServingFor=[front-proxy-ca] issuer=\"<self>\" (2023-10-21 12:28:31 +0000 UTC to 2033-10-18 12:28:31 +0000 UTC (now=2023-10-21 20:09:45.73533554 +0000 UTC))"
I1021 20:09:45.735995       1 tlsconfig.go:200] "Loaded serving cert" certName="Generated self signed cert" certDetail="\"localhost@1697918984\" [serving] validServingFor=[127.0.0.1,localhost,localhost] issuer=\"localhost-ca@1697918983\" (2023-10-21 19:09:43 +0000 UTC to 2024-10-20 19:09:43 +0000 UTC (now=2023-10-21 20:09:45.73597482 +0000 UTC))"
I1021 20:09:45.736573       1 named_certificates.go:53] "Loaded SNI cert" index=0 certName="self-signed loopback" certDetail="\"apiserver-loopback-client@1697918985\" [serving] validServingFor=[apiserver-loopback-client] issuer=\"apiserver-loopback-client-ca@1697918984\" (2023-10-21 19:09:44 +0000 UTC to 2024-10-20 19:09:44 +0000 UTC (now=2023-10-21 20:09:45.736536011 +0000 UTC))"
I1021 20:09:45.736688       1 tlsconfig.go:178] "Loaded client CA" index=0 certName="client-ca::kube-system::extension-apiserver-authentication::client-ca-file,client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" certDetail="\"kubernetes\" [] validServingFor=[kubernetes] issuer=\"<self>\" (2023-10-21 12:28:30 +0000 UTC to 2033-10-18 12:28:30 +0000 UTC (now=2023-10-21 20:09:45.736668497 +0000 UTC))"
I1021 20:09:45.736887       1 tlsconfig.go:178] "Loaded client CA" index=1 certName="client-ca::kube-system::extension-apiserver-authentication::client-ca-file,client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" certDetail="\"front-proxy-ca\" [] validServingFor=[front-proxy-ca] issuer=\"<self>\" (2023-10-21 12:28:31 +0000 UTC to 2033-10-18 12:28:31 +0000 UTC (now=2023-10-21 20:09:45.736866132 +0000 UTC))"
I1021 20:09:45.738574       1 tlsconfig.go:200] "Loaded serving cert" certName="Generated self signed cert" certDetail="\"localhost@1697918984\" [serving] validServingFor=[127.0.0.1,localhost,localhost] issuer=\"localhost-ca@1697918983\" (2023-10-21 19:09:43 +0000 UTC to 2024-10-20 19:09:43 +0000 UTC (now=2023-10-21 20:09:45.73855163 +0000 UTC))"
I1021 20:09:45.739395       1 named_certificates.go:53] "Loaded SNI cert" index=0 certName="self-signed loopback" certDetail="\"apiserver-loopback-client@1697918985\" [serving] validServingFor=[apiserver-loopback-client] issuer=\"apiserver-loopback-client-ca@1697918984\" (2023-10-21 19:09:44 +0000 UTC to 2024-10-20 19:09:44 +0000 UTC (now=2023-10-21 20:09:45.739372846 +0000 UTC))"
I1021 20:10:02.179596       1 leaderelection.go:260] successfully acquired lease kube-system/cloud-controller-manager-proxmox
I1021 20:10:02.179991       1 event.go:307] "Event occurred" object="kube-system/cloud-controller-manager-proxmox" fieldPath="" kind="Lease" apiVersion="coordination.k8s.io/v1" type="Normal" reason="LeaderElection" message="proxmox-cloud-controller-manager-85d56dfbd6-449h4_94ace4d2-576d-44dc-8428-f77aa93d5c80 became leader"
I1021 20:10:03.124143       1 cloud.go:64] clientset initialized
I1021 20:10:03.156349       1 cloud.go:83] proxmox initialized
W1021 20:10:03.156373       1 controllermanager.go:314] "node-route-controller" is disabled
W1021 20:10:03.156381       1 controllermanager.go:314] "cloud-node-controller" is disabled
I1021 20:10:03.156387       1 controllermanager.go:318] Starting "cloud-node-lifecycle-controller"
I1021 20:10:03.160296       1 controllermanager.go:337] Started "cloud-node-lifecycle-controller"
W1021 20:10:03.160317       1 controllermanager.go:314] "service-lb-controller" is disabled
I1021 20:10:03.160511       1 node_lifecycle_controller.go:113] Sending events to api server
E1021 20:34:05.883135       1 leaderelection.go:332] error retrieving resource lock kube-system/cloud-controller-manager-proxmox: Get "https://10.96.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/cloud-controller-manager-proxmox?timeout=5s": net/http: request canceled (Client.Timeout exceeded while awaiting headers)

Environment

  • Plugin version: v0.2.0
  • Kubernetes version:
Client Version: v1.27.3
Kustomize Version: v5.0.1
Server Version: v1.27.7

Additionnal information

I’ve created my K8S cluster with these Ansible steps :

    - name: Create init conf file (for adding serverTLSBootstrap option)
      copy:
        dest: /etc/kubernetes/kubeadm-init.yaml
        content: |
          apiVersion: kubeadm.k8s.io/v1beta3
          kind: ClusterConfiguration
          controlPlaneEndpoint: "{{ kube_endpoint }}:6443"
          ---
          apiVersion: kubelet.config.k8s.io/v1beta1
          kind: KubeletConfiguration
          serverTLSBootstrap: true
        mode: 0644
    - name: Initialize Kubernetes cluster
      shell: kubeadm init --upload-certs --skip-phases=addon/kube-proxy --config /etc/kubernetes/kubeadm-init.yaml | tee kubeadm-init-`date '+%Y-%m-%d_%H-%M-%S'`.out

I’ve installed Cilium with these Ansible steps :

  - name: Add cilium helm repository
    kubernetes.core.helm_repository:
      name: cilium
      repo_url: "https://helm.cilium.io/"

  - name: Retreive cilium CNI version
    shell: curl -s https://raw.githubusercontent.com/cilium/cilium/main/stable.txt
    register: cilium_version
    changed_when: false

  - name: Install cilium chart
    kubernetes.core.helm:
      name: cilium
      namespace: kube-system
      chart_ref: cilium/cilium
      chart_version: "{{ cilium_version.stdout }}"
      values:
        kubeProxyReplacement: true
        bpf.masquerade: true
        k8sServiceHost: "{{ kube_endpoint }}"
        k8sServicePort: 6443

Here are the Ansible steps I used for deploying CCM and CSI plugin :

  - name: Install Proxmox CCM chart
    kubernetes.core.helm:
      name: proxmox-cloud-controller-manager
      namespace: kube-system
      chart_ref: oci://ghcr.io/sergelogvinov/charts/proxmox-cloud-controller-manager
      values:
        config:
          clusters:
            - url: "{{ proxmox_url }}"
              insecure: false
              token_id: "kubernetes@pve!ccm"
              token_secret: "xxxxxxxxxxxxxxxxx"
              region: main
        enabledControllers:
          - cloud-node-lifecycle
        nodeSelector:
          node-role.kubernetes.io/control-plane: ""
        tolerations:
          - key: node-role.kubernetes.io/control-plane
            effect: NoSchedule

  - name: Create Proxmox CSI namespace
    kubernetes.core.k8s:
      state: present
      definition:
        api_version: v1
        kind: Namespace
        metadata:
          name: csi-proxmox
          labels:
            app.kubernetes.io/managed-by: Helm
            pod-security.kubernetes.io/enforce: privileged
          annotations:
            meta.helm.sh/release-name: proxmox-csi-plugin
            meta.helm.sh/release-namespace: csi-proxmox

  - name: Install Proxmox CSI chart
    kubernetes.core.helm:
      name: proxmox-csi-plugin
      namespace: csi-proxmox
      chart_ref: oci://ghcr.io/sergelogvinov/charts/proxmox-csi-plugin
      values:
        config:
          clusters:
            - url: "{{ proxmox_url }}"
              insecure: false
              token_id: "kubernetes-csi@pve!csi"
              token_secret: "yyyyyyyyyyyyyyyyyyy"
              region: main
        node:
          nodeSelector:
          tolerations:
            - operator: Exists
        nodeSelector:
          node-role.kubernetes.io/control-plane: ""
        tolerations:
          - key: node-role.kubernetes.io/control-plane
            effect: NoSchedule
        storageClass:
          - name: proxmox-data
            storage: local
            reclaimPolicy: Delete
            fstype: ext4
            cache: none

Here is one of my worker node :

apiVersion: v1
kind: Node
metadata:
  annotations:
    kubeadm.alpha.kubernetes.io/cri-socket: unix:///var/run/containerd/containerd.sock
    node.alpha.kubernetes.io/ttl: "0"
    volumes.kubernetes.io/controller-managed-attach-detach: "true"
  creationTimestamp: "2023-10-21T12:35:47Z"
  labels:
    beta.kubernetes.io/arch: amd64
    beta.kubernetes.io/os: linux
    kubernetes.io/arch: amd64
    kubernetes.io/hostname: kube-wk-1
    kubernetes.io/os: linux
    node-role.kubernetes.io/worker: ""
  name: kube-wk-1
  resourceVersion: "103848"
  uid: 7b112df8-f186-48c9-921b-eec6aa1c0833
spec: {}
status:
  addresses:
  - address: 192.168.1.104
    type: InternalIP
  - address: kube-wk-1
    type: Hostname
  allocatable:
    cpu: "2"
    ephemeral-storage: "47265173836"
    hugepages-1Gi: "0"
    hugepages-2Mi: "0"
    memory: 1705900Ki
    pods: "110"
  capacity:
    cpu: "2"
    ephemeral-storage: 51285996Ki
    hugepages-1Gi: "0"
    hugepages-2Mi: "0"
    memory: 1808300Ki
    pods: "110"
  conditions:
  - lastHeartbeatTime: "2023-10-21T12:44:17Z"
    lastTransitionTime: "2023-10-21T12:44:17Z"
    message: Cilium is running on this node
    reason: CiliumIsUp
    status: "False"
    type: NetworkUnavailable
  - lastHeartbeatTime: "2023-10-21T20:48:06Z"
    lastTransitionTime: "2023-10-21T12:35:47Z"
    message: kubelet has sufficient memory available
    reason: KubeletHasSufficientMemory
    status: "False"
    type: MemoryPressure
  - lastHeartbeatTime: "2023-10-21T20:48:06Z"
    lastTransitionTime: "2023-10-21T12:35:47Z"
    message: kubelet has no disk pressure
    reason: KubeletHasNoDiskPressure
    status: "False"
    type: DiskPressure
  - lastHeartbeatTime: "2023-10-21T20:53:12Z"
    lastTransitionTime: "2023-10-21T12:35:47Z"
    message: kubelet has sufficient PID available
    reason: KubeletHasSufficientPID
    status: "False"
    type: PIDPressure
  - lastHeartbeatTime: "2023-10-21T20:53:12Z"
    lastTransitionTime: "2023-10-21T12:43:21Z"
    message: kubelet is posting ready status
    reason: KubeletReady
    status: "True"
    type: Ready
  daemonEndpoints:
    kubeletEndpoint:
      Port: 10250
  images:
  - names:
    - quay.io/cilium/cilium@sha256:e5ca22526e01469f8d10c14e2339a82a13ad70d9a359b879024715540eef4ace
    sizeBytes: 184477234
  - names:
    - ghcr.io/sergelogvinov/proxmox-csi-node@sha256:3ebc0d60f1d664dd1550ec1779c8929936a3eb700e6da643e50a43b8118be4d9
    - ghcr.io/sergelogvinov/proxmox-csi-node:v0.3.0
    sizeBytes: 28682577
  - names:
    - ghcr.io/postfinance/kubelet-csr-approver@sha256:bafbb479878906bfb69d635144027b90d42e58fb52d12b0663f77a58dd4fe416
    - ghcr.io/postfinance/kubelet-csr-approver:v1.0.5
    sizeBytes: 27379324
  - names:
    - quay.io/cilium/operator-generic@sha256:c9613277b72103ed36e9c0d16b9a17cafd507461d59340e432e3e9c23468b5e2
    sizeBytes: 25151466
  - names:
    - ghcr.io/sergelogvinov/proxmox-cloud-controller-manager@sha256:954ec00288cbed35afc9a6abf8b0bb28f812942b5df7b636e31ad8a38933d15d
    - ghcr.io/sergelogvinov/proxmox-cloud-controller-manager:v0.2.0
    sizeBytes: 17003810
  - names:
    - registry.k8s.io/coredns/coredns@sha256:a0ead06651cf580044aeb0a0feba63591858fb2e43ade8c9dea45a6a89ae7e5e
    - registry.k8s.io/coredns/coredns:v1.10.1
    sizeBytes: 16190758
  - names:
    - registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:cd21e19cd8bbd5bc56f1b4f1398a436e7897da2995d6d036c9729be3f4e456e6
    - registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.9.0
    sizeBytes: 10755934
  - names:
    - registry.k8s.io/sig-storage/livenessprobe@sha256:82adbebdf5d5a1f40f246aef8ddbee7f89dea190652aefe83336008e69f9a89f
    - registry.k8s.io/sig-storage/livenessprobe:v2.11.0
    sizeBytes: 9696710
  - names:
    - registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db
    - registry.k8s.io/pause:3.6
    sizeBytes: 301773
  nodeInfo:
    architecture: amd64
    bootID: 3af35a66-2cf4-4b27-b041-6a4d44b2b52f
    containerRuntimeVersion: containerd://1.6.24
    kernelVersion: 5.14.0-284.30.1.el9_2.x86_64
    kubeProxyVersion: v1.27.3
    kubeletVersion: v1.27.3
    machineID: fb1a9895e56540c4b29f780679468485
    operatingSystem: linux
    osImage: Rocky Linux 9.2 (Blue Onyx)
    systemUUID: fb1a9895-e565-40c4-b29f-780679468485

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 24 (11 by maintainers)

Most upvoted comments

@morsik I’m truly sorry to hear that.

I’ve updated the documentation https://github.com/sergelogvinov/proxmox-cloud-controller-manager/blob/main/docs/install.md#requirements

Thank you for contributing to the project!

@sergelogvinov thank you! It works great after I discovered this simple change, but it real pain to understand why I’m getting node IP errors and then I hit source code and found this issue.

BTW, this is not true at all:

If your node has multiple IP addresses

You explicitly look for that annotation in your source code! I had single IP address and it still didn’t worked for that very reason!