proxmox-cloud-controller-manager: CCM does not labelize nodes
Bug Report
Description
Since I’ve removed kube-proxy and let routing to be done by Cilium, my CCM does not labelize nodes anymore. I made a fresh install.
Logs
Here are some logs from my proxmox-cloud-controller-manager pod :
I1021 20:09:44.245894 1 serving.go:348] Generated self-signed cert in-memory
I1021 20:09:44.836589 1 serving.go:348] Generated self-signed cert in-memory
W1021 20:09:44.836642 1 client_config.go:618] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
I1021 20:09:45.281997 1 requestheader_controller.go:244] Loaded a new request header values for RequestHeaderAuthRequestController
I1021 20:09:45.283103 1 controllermanager.go:168] Version: v0.2.0
I1021 20:09:45.634437 1 requestheader_controller.go:169] Starting RequestHeaderAuthRequestController
I1021 20:09:45.634500 1 shared_informer.go:311] Waiting for caches to sync for RequestHeaderAuthRequestController
I1021 20:09:45.634529 1 configmap_cafile_content.go:202] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file"
I1021 20:09:45.634561 1 shared_informer.go:311] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I1021 20:09:45.634578 1 configmap_cafile_content.go:202] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"
I1021 20:09:45.634584 1 shared_informer.go:311] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I1021 20:09:45.635263 1 tlsconfig.go:200] "Loaded serving cert" certName="Generated self signed cert" certDetail="\"localhost@1697918984\" [serving] validServingFor=[127.0.0.1,localhost,localhost] issuer=\"localhost-ca@1697918983\" (2023-10-21 19:09:43 +0000 UTC to 2024-10-20 19:09:43 +0000 UTC (now=2023-10-21 20:09:45.635180211 +0000 UTC))"
I1021 20:09:45.636389 1 named_certificates.go:53] "Loaded SNI cert" index=0 certName="self-signed loopback" certDetail="\"apiserver-loopback-client@1697918985\" [serving] validServingFor=[apiserver-loopback-client] issuer=\"apiserver-loopback-client-ca@1697918984\" (2023-10-21 19:09:44 +0000 UTC to 2024-10-20 19:09:44 +0000 UTC (now=2023-10-21 20:09:45.636344007 +0000 UTC))"
I1021 20:09:45.636684 1 secure_serving.go:210] Serving securely on :10258
I1021 20:09:45.637374 1 leaderelection.go:250] attempting to acquire leader lease kube-system/cloud-controller-manager-proxmox...
I1021 20:09:45.636900 1 tlsconfig.go:240] "Starting DynamicServingCertificateController"
I1021 20:09:45.734941 1 shared_informer.go:318] Caches are synced for RequestHeaderAuthRequestController
I1021 20:09:45.734939 1 shared_informer.go:318] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I1021 20:09:45.735291 1 shared_informer.go:318] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I1021 20:09:45.735365 1 tlsconfig.go:178] "Loaded client CA" index=0 certName="client-ca::kube-system::extension-apiserver-authentication::client-ca-file,client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" certDetail="\"front-proxy-ca\" [] validServingFor=[front-proxy-ca] issuer=\"<self>\" (2023-10-21 12:28:31 +0000 UTC to 2033-10-18 12:28:31 +0000 UTC (now=2023-10-21 20:09:45.73533554 +0000 UTC))"
I1021 20:09:45.735995 1 tlsconfig.go:200] "Loaded serving cert" certName="Generated self signed cert" certDetail="\"localhost@1697918984\" [serving] validServingFor=[127.0.0.1,localhost,localhost] issuer=\"localhost-ca@1697918983\" (2023-10-21 19:09:43 +0000 UTC to 2024-10-20 19:09:43 +0000 UTC (now=2023-10-21 20:09:45.73597482 +0000 UTC))"
I1021 20:09:45.736573 1 named_certificates.go:53] "Loaded SNI cert" index=0 certName="self-signed loopback" certDetail="\"apiserver-loopback-client@1697918985\" [serving] validServingFor=[apiserver-loopback-client] issuer=\"apiserver-loopback-client-ca@1697918984\" (2023-10-21 19:09:44 +0000 UTC to 2024-10-20 19:09:44 +0000 UTC (now=2023-10-21 20:09:45.736536011 +0000 UTC))"
I1021 20:09:45.736688 1 tlsconfig.go:178] "Loaded client CA" index=0 certName="client-ca::kube-system::extension-apiserver-authentication::client-ca-file,client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" certDetail="\"kubernetes\" [] validServingFor=[kubernetes] issuer=\"<self>\" (2023-10-21 12:28:30 +0000 UTC to 2033-10-18 12:28:30 +0000 UTC (now=2023-10-21 20:09:45.736668497 +0000 UTC))"
I1021 20:09:45.736887 1 tlsconfig.go:178] "Loaded client CA" index=1 certName="client-ca::kube-system::extension-apiserver-authentication::client-ca-file,client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" certDetail="\"front-proxy-ca\" [] validServingFor=[front-proxy-ca] issuer=\"<self>\" (2023-10-21 12:28:31 +0000 UTC to 2033-10-18 12:28:31 +0000 UTC (now=2023-10-21 20:09:45.736866132 +0000 UTC))"
I1021 20:09:45.738574 1 tlsconfig.go:200] "Loaded serving cert" certName="Generated self signed cert" certDetail="\"localhost@1697918984\" [serving] validServingFor=[127.0.0.1,localhost,localhost] issuer=\"localhost-ca@1697918983\" (2023-10-21 19:09:43 +0000 UTC to 2024-10-20 19:09:43 +0000 UTC (now=2023-10-21 20:09:45.73855163 +0000 UTC))"
I1021 20:09:45.739395 1 named_certificates.go:53] "Loaded SNI cert" index=0 certName="self-signed loopback" certDetail="\"apiserver-loopback-client@1697918985\" [serving] validServingFor=[apiserver-loopback-client] issuer=\"apiserver-loopback-client-ca@1697918984\" (2023-10-21 19:09:44 +0000 UTC to 2024-10-20 19:09:44 +0000 UTC (now=2023-10-21 20:09:45.739372846 +0000 UTC))"
I1021 20:10:02.179596 1 leaderelection.go:260] successfully acquired lease kube-system/cloud-controller-manager-proxmox
I1021 20:10:02.179991 1 event.go:307] "Event occurred" object="kube-system/cloud-controller-manager-proxmox" fieldPath="" kind="Lease" apiVersion="coordination.k8s.io/v1" type="Normal" reason="LeaderElection" message="proxmox-cloud-controller-manager-85d56dfbd6-449h4_94ace4d2-576d-44dc-8428-f77aa93d5c80 became leader"
I1021 20:10:03.124143 1 cloud.go:64] clientset initialized
I1021 20:10:03.156349 1 cloud.go:83] proxmox initialized
W1021 20:10:03.156373 1 controllermanager.go:314] "node-route-controller" is disabled
W1021 20:10:03.156381 1 controllermanager.go:314] "cloud-node-controller" is disabled
I1021 20:10:03.156387 1 controllermanager.go:318] Starting "cloud-node-lifecycle-controller"
I1021 20:10:03.160296 1 controllermanager.go:337] Started "cloud-node-lifecycle-controller"
W1021 20:10:03.160317 1 controllermanager.go:314] "service-lb-controller" is disabled
I1021 20:10:03.160511 1 node_lifecycle_controller.go:113] Sending events to api server
E1021 20:34:05.883135 1 leaderelection.go:332] error retrieving resource lock kube-system/cloud-controller-manager-proxmox: Get "https://10.96.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/cloud-controller-manager-proxmox?timeout=5s": net/http: request canceled (Client.Timeout exceeded while awaiting headers)
Environment
- Plugin version: v0.2.0
- Kubernetes version:
Client Version: v1.27.3
Kustomize Version: v5.0.1
Server Version: v1.27.7
Additionnal information
I’ve created my K8S cluster with these Ansible steps :
- name: Create init conf file (for adding serverTLSBootstrap option)
copy:
dest: /etc/kubernetes/kubeadm-init.yaml
content: |
apiVersion: kubeadm.k8s.io/v1beta3
kind: ClusterConfiguration
controlPlaneEndpoint: "{{ kube_endpoint }}:6443"
---
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
serverTLSBootstrap: true
mode: 0644
- name: Initialize Kubernetes cluster
shell: kubeadm init --upload-certs --skip-phases=addon/kube-proxy --config /etc/kubernetes/kubeadm-init.yaml | tee kubeadm-init-`date '+%Y-%m-%d_%H-%M-%S'`.out
I’ve installed Cilium with these Ansible steps :
- name: Add cilium helm repository
kubernetes.core.helm_repository:
name: cilium
repo_url: "https://helm.cilium.io/"
- name: Retreive cilium CNI version
shell: curl -s https://raw.githubusercontent.com/cilium/cilium/main/stable.txt
register: cilium_version
changed_when: false
- name: Install cilium chart
kubernetes.core.helm:
name: cilium
namespace: kube-system
chart_ref: cilium/cilium
chart_version: "{{ cilium_version.stdout }}"
values:
kubeProxyReplacement: true
bpf.masquerade: true
k8sServiceHost: "{{ kube_endpoint }}"
k8sServicePort: 6443
Here are the Ansible steps I used for deploying CCM and CSI plugin :
- name: Install Proxmox CCM chart
kubernetes.core.helm:
name: proxmox-cloud-controller-manager
namespace: kube-system
chart_ref: oci://ghcr.io/sergelogvinov/charts/proxmox-cloud-controller-manager
values:
config:
clusters:
- url: "{{ proxmox_url }}"
insecure: false
token_id: "kubernetes@pve!ccm"
token_secret: "xxxxxxxxxxxxxxxxx"
region: main
enabledControllers:
- cloud-node-lifecycle
nodeSelector:
node-role.kubernetes.io/control-plane: ""
tolerations:
- key: node-role.kubernetes.io/control-plane
effect: NoSchedule
- name: Create Proxmox CSI namespace
kubernetes.core.k8s:
state: present
definition:
api_version: v1
kind: Namespace
metadata:
name: csi-proxmox
labels:
app.kubernetes.io/managed-by: Helm
pod-security.kubernetes.io/enforce: privileged
annotations:
meta.helm.sh/release-name: proxmox-csi-plugin
meta.helm.sh/release-namespace: csi-proxmox
- name: Install Proxmox CSI chart
kubernetes.core.helm:
name: proxmox-csi-plugin
namespace: csi-proxmox
chart_ref: oci://ghcr.io/sergelogvinov/charts/proxmox-csi-plugin
values:
config:
clusters:
- url: "{{ proxmox_url }}"
insecure: false
token_id: "kubernetes-csi@pve!csi"
token_secret: "yyyyyyyyyyyyyyyyyyy"
region: main
node:
nodeSelector:
tolerations:
- operator: Exists
nodeSelector:
node-role.kubernetes.io/control-plane: ""
tolerations:
- key: node-role.kubernetes.io/control-plane
effect: NoSchedule
storageClass:
- name: proxmox-data
storage: local
reclaimPolicy: Delete
fstype: ext4
cache: none
Here is one of my worker node :
apiVersion: v1
kind: Node
metadata:
annotations:
kubeadm.alpha.kubernetes.io/cri-socket: unix:///var/run/containerd/containerd.sock
node.alpha.kubernetes.io/ttl: "0"
volumes.kubernetes.io/controller-managed-attach-detach: "true"
creationTimestamp: "2023-10-21T12:35:47Z"
labels:
beta.kubernetes.io/arch: amd64
beta.kubernetes.io/os: linux
kubernetes.io/arch: amd64
kubernetes.io/hostname: kube-wk-1
kubernetes.io/os: linux
node-role.kubernetes.io/worker: ""
name: kube-wk-1
resourceVersion: "103848"
uid: 7b112df8-f186-48c9-921b-eec6aa1c0833
spec: {}
status:
addresses:
- address: 192.168.1.104
type: InternalIP
- address: kube-wk-1
type: Hostname
allocatable:
cpu: "2"
ephemeral-storage: "47265173836"
hugepages-1Gi: "0"
hugepages-2Mi: "0"
memory: 1705900Ki
pods: "110"
capacity:
cpu: "2"
ephemeral-storage: 51285996Ki
hugepages-1Gi: "0"
hugepages-2Mi: "0"
memory: 1808300Ki
pods: "110"
conditions:
- lastHeartbeatTime: "2023-10-21T12:44:17Z"
lastTransitionTime: "2023-10-21T12:44:17Z"
message: Cilium is running on this node
reason: CiliumIsUp
status: "False"
type: NetworkUnavailable
- lastHeartbeatTime: "2023-10-21T20:48:06Z"
lastTransitionTime: "2023-10-21T12:35:47Z"
message: kubelet has sufficient memory available
reason: KubeletHasSufficientMemory
status: "False"
type: MemoryPressure
- lastHeartbeatTime: "2023-10-21T20:48:06Z"
lastTransitionTime: "2023-10-21T12:35:47Z"
message: kubelet has no disk pressure
reason: KubeletHasNoDiskPressure
status: "False"
type: DiskPressure
- lastHeartbeatTime: "2023-10-21T20:53:12Z"
lastTransitionTime: "2023-10-21T12:35:47Z"
message: kubelet has sufficient PID available
reason: KubeletHasSufficientPID
status: "False"
type: PIDPressure
- lastHeartbeatTime: "2023-10-21T20:53:12Z"
lastTransitionTime: "2023-10-21T12:43:21Z"
message: kubelet is posting ready status
reason: KubeletReady
status: "True"
type: Ready
daemonEndpoints:
kubeletEndpoint:
Port: 10250
images:
- names:
- quay.io/cilium/cilium@sha256:e5ca22526e01469f8d10c14e2339a82a13ad70d9a359b879024715540eef4ace
sizeBytes: 184477234
- names:
- ghcr.io/sergelogvinov/proxmox-csi-node@sha256:3ebc0d60f1d664dd1550ec1779c8929936a3eb700e6da643e50a43b8118be4d9
- ghcr.io/sergelogvinov/proxmox-csi-node:v0.3.0
sizeBytes: 28682577
- names:
- ghcr.io/postfinance/kubelet-csr-approver@sha256:bafbb479878906bfb69d635144027b90d42e58fb52d12b0663f77a58dd4fe416
- ghcr.io/postfinance/kubelet-csr-approver:v1.0.5
sizeBytes: 27379324
- names:
- quay.io/cilium/operator-generic@sha256:c9613277b72103ed36e9c0d16b9a17cafd507461d59340e432e3e9c23468b5e2
sizeBytes: 25151466
- names:
- ghcr.io/sergelogvinov/proxmox-cloud-controller-manager@sha256:954ec00288cbed35afc9a6abf8b0bb28f812942b5df7b636e31ad8a38933d15d
- ghcr.io/sergelogvinov/proxmox-cloud-controller-manager:v0.2.0
sizeBytes: 17003810
- names:
- registry.k8s.io/coredns/coredns@sha256:a0ead06651cf580044aeb0a0feba63591858fb2e43ade8c9dea45a6a89ae7e5e
- registry.k8s.io/coredns/coredns:v1.10.1
sizeBytes: 16190758
- names:
- registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:cd21e19cd8bbd5bc56f1b4f1398a436e7897da2995d6d036c9729be3f4e456e6
- registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.9.0
sizeBytes: 10755934
- names:
- registry.k8s.io/sig-storage/livenessprobe@sha256:82adbebdf5d5a1f40f246aef8ddbee7f89dea190652aefe83336008e69f9a89f
- registry.k8s.io/sig-storage/livenessprobe:v2.11.0
sizeBytes: 9696710
- names:
- registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db
- registry.k8s.io/pause:3.6
sizeBytes: 301773
nodeInfo:
architecture: amd64
bootID: 3af35a66-2cf4-4b27-b041-6a4d44b2b52f
containerRuntimeVersion: containerd://1.6.24
kernelVersion: 5.14.0-284.30.1.el9_2.x86_64
kubeProxyVersion: v1.27.3
kubeletVersion: v1.27.3
machineID: fb1a9895e56540c4b29f780679468485
operatingSystem: linux
osImage: Rocky Linux 9.2 (Blue Onyx)
systemUUID: fb1a9895-e565-40c4-b29f-780679468485
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 24 (11 by maintainers)
@morsik I’m truly sorry to hear that.
I’ve updated the documentation https://github.com/sergelogvinov/proxmox-cloud-controller-manager/blob/main/docs/install.md#requirements
Thank you for contributing to the project!
@sergelogvinov thank you! It works great after I discovered this simple change, but it real pain to understand why I’m getting node IP errors and then I hit source code and found this issue.
BTW, this is not true at all:
You explicitly look for that annotation in your source code! I had single IP address and it still didn’t worked for that very reason!