kubernetes: kubelet without deprecated `--cloud-provider=external` do not get proper taint and are not CCM initialized

What happened?

The kubelet CLI docs state that --cloud-provider=external is deprecated and will go away in1.24:

–cloud-provider string

| The provider for cloud services. Set to empty string for running with no cloud provider. If set, the cloud provider > determines the name of the node (consult cloud provider documentation to determine if and how the hostname is used). (DEPRECATED: will be removed in 1.24 or later, in favor of removing cloud provider code from Kubelet.)

The docs for running a CCM, on the other hand, say that --cloud-provider=external is required, in order to get the proper node.cloudprovider.kubernetes.io/uninitialized taint.

I launched a single-node 1.23 cluster without --cloud-provider=external, and while I did not get the deprecation warnings, the node was not properly tainted, and the CCM I am running never seems to get the correct calls leading to populating the providerID and removing the taint.

I do not know if this is a bug in the controller-manager and/or kubelet, or that we need to do something else to get this to work, and thus missing documentation.

What did you expect to happen?

node to get the correct taint, k8s to initialize the node.

How can we reproduce it (as minimally and precisely as possible)?

1. Launch a single node cluster without --cloud-provider=external
2. See that the node does not have the taint
3. Launch a CCM
4. See that it does not set the provider ID

Anything else we need to know?

No response

Kubernetes version

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.1", GitCommit:"86ec240af8cbd1b60bcc4c03c20da9b98005b92e", GitTreeState:"clean", BuildDate:"2021-12-16T11:33:37Z", GoVersion:"go1.17.5", Compiler:"gc", Platform:"darwin/arm64"}
Server Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.4", GitCommit:"e6c093d87ea4cbb530a7b2ae91e54c0842d8308a", GitTreeState:"clean", BuildDate:"2022-02-16T12:32:02Z", GoVersion:"go1.17.7", Compiler:"gc", Platform:"linux/amd64"}

Cloud provider

OS version

# On Linux:
$ cat /etc/os-release
NAME="Ubuntu"
VERSION="18.04.6 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.6 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic

$ uname -a
Linux k8s-master-01 5.4.0-107-generic #121~18.04.1-Ubuntu SMP Thu Mar 24 17:21:33 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Install tools

Container runtime (CRI) and version (if applicable)

Related plugins (CNI, CSI, …) and versions (if applicable)