kubernetes: Cloud Controller Manger doesn't query cloud provider for node name, causing the node to be removed

What happened:

Launch a node with container linux CoreOS-stable-1911.3.0 on aws, with customized ignition configs.
The hostname turns to be ip-10-3-18-1, instead of the full private dns ip-10-3-18-1.us-west-1.compute.internal because /etc/hostname is not set.
kubelet starts with --cloud-provider=external and skips this code path. So it’s not able to set the private dns as the node name.
When CCM starts, it tries to read the node name from the node spec.
CCM calls GetInstanceProviderID() and fails.
CCM calls getNodeAddressesByProviderIDOrName() and fails too.
CCM then removes the node.

What you expected to happen:

Since now the kubelet runs with --cloud-provider=external, no one is executing that code path to query the cloud provider to get the node name anymore. However this code path still needs to be executed by someone to get the correct node name from the cloud provider for the node. I think the CCM might need to query the cloud provider for the full node hostname in case the hostname given by the kubelet is not the full hostname (in AWS case).

How to reproduce it (as minimally and precisely as possible): Launch a container linux with non-empty ignition config in the user data, then the hostname won’t be the full private-dns. Then launch kubelet with --cloud-provider=external and CCM will reproduce the issue described above.

Anything else we need to know?:

Environment:

Kubernetes version:

Server Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.2", GitCommit:"bb9ffb1654d4a729bb4cec18ff088eacc153c239", GitTreeState:"clean", BuildDate:"2018-08-07T23:08:19Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}```
- Cloud provider or hardware configuration: 
AWS
- OS 
```NAME="Container Linux by CoreOS"
ID=coreos
VERSION=1911.3.0
VERSION_ID=1911.3.0
BUILD_ID=2018-11-05-1815
PRETTY_NAME="Container Linux by CoreOS 1911.3.0 (Rhyolite)"
ANSI_COLOR="38;5;75"
HOME_URL="https://coreos.com/"
BUG_REPORT_URL="https://issues.coreos.com"
COREOS_BOARD="amd64-usr"

Kernel: Linux ip-10-3-20-13 4.14.78-coreos #1 SMP Mon Nov 5 17:42:07 UTC 2018 x86_64 Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz GenuineIntel GNU/Linux
Install tools: Internal k8s installer tool based on terraform
Others:

This issue can be mitigated by telling the igntion config to set the /etc/hostname to the private-dns (by curl http://169.254.169.254/latest/meta-data/hostname). Or just use the coreos-metadata service.

/cc @Quentin-M

@kubernetes/sig-aws-misc @andrewsykim

/kind bug

About this issue

Original URL
State: open
Created 6 years ago
Comments: 40 (32 by maintainers)

Most upvoted comments

This remains an issue on latest k8s. Updated code path in kubelet: https://github.com/kubernetes/kubernetes/blob/53a7922e4572f48d41b144ae21dad7516500cb8d/cmd/kubelet/app/server.go#L992-L994

I think this also poses a problem for the out-of-tree migration. It seems to me that e.g. the OpenStack legacy cloud provider doesn’t guarantee that the node name and hostname match: https://github.com/kubernetes/kubernetes/blob/53a7922e4572f48d41b144ae21dad7516500cb8d/staging/src/k8s.io/legacy-cloud-providers/openstack/openstack_instances.go#L71-L79

So, if one tries to upgrade a node from --cloud-provider=openstack to --cloud-provider=external, the old node name (not necesarily hostname) may not match the new node name (hostname), which will cause the kubelet to be unable to find its node.

From https://github.com/openshift/machine-config-operator/pull/2401#issuecomment-776878881 it sounds like this also affects VMware and AWS depending on configuration. (perhaps @nckturner can confirm impact on AWS)

Wanted to check if this is on the external cloud provider migration radar and if there is a blessed migration path?

ehashman on Jul 15, 2021

This issue affects both the AWS and OpenStack in-tree cloud providers, both of which return instance.Name in CurrentNodeName(), which is not necessarily the same as kubelet’s default of the FQDN hostname.

We can work round this by setting the hostname to be whatever the in-tree cloud provider previously returned. However, this feels like a kludge because:

We’re changing the hostname of an existing host which may have other consequences, some of which may be outside of our ability to predict if they involve, e.g. third-party drivers.
It makes AWS and OpenStack configuration snowflakes
It perpetuates the problem beyond the use of the in-tree provider which originally caused it
We’re not solving the problem, just deferring it indefinitely

An idea I’ve seen is to request it from the CCM. However that may have a bootstrapping problem as kubelet communicates with the CCM via the API, so it would have to identify itself somehow in order to retrieve the correct information.

I wonder if a simpler idea might be to persist kubelet’s node name as local state, e.g. in /etc/kubernetes/node. We could initialise that value appropriately if it doesn’t exist, then pass its contents to kubelet via --hostname-override subsequently. We could ensure for upgrade that it is initialised with the current node name for existing nodes, but for all future nodes it is initialised with hostname. This would allow us to eventually remove the upgrade logic, and would also insulate us from cloud configuration changes which affect hostname (although nobody should be doing that anyway).

Or even simpler, launching kubelet with --hostname-override iff /etc/kubernetes/node exists. Then we can create this file only on AWS/OpenStack nodes which previously ran the in-tree cloud provider.

mdbooth on Jul 16, 2021