eks-anywhere: Why don't my control plane get the IP number defined in the cluster manifest?

What happened:

I ran eksctl anywhere create with my cluster config, and it hangs on creating workload cluster, after creating the etcd node it never gets a ready condition from the control plane, and I wonder if it is because it doesn’t get the IP number specified in the manifest.

What you expected to happen:

The workload cluster to be built.

How to reproduce it (as minimally and precisely as possible):

eksctl anywhere create cluster -f cluster-manifest.yaml

My cluster manifest:

apiVersion: anywhere.eks.amazonaws.com/v1alpha1
kind: Cluster
metadata:
  name: mgmt
spec:
  clusterNetwork:
    cniConfig:
      cilium: {}
    pods:
      cidrBlocks:
      - 192.168.0.0/16
    services:
      cidrBlocks:
      - 10.96.0.0/12
  proxyConfiguration:
    httpProxy: "xxxxxx:3128"
    httpsProxy: "xxxxxx:3128"
  controlPlaneConfiguration:
    count: 1
    endpoint:
      host: "10.56.235.70"
    machineGroupRef:
      kind: VSphereMachineConfig
      name: mgmt-cp
  datacenterRef:
    kind: VSphereDatacenterConfig
    name: mgmt
  externalEtcdConfiguration:
    count: 1
    machineGroupRef:
      kind: VSphereMachineConfig
      name: mgmt-etcd
  kubernetesVersion: "1.23"
  managementCluster:
    name: mgmt
  workerNodeGroupConfigurations:
  - count: 1
    machineGroupRef:
      kind: VSphereMachineConfig
      name: mgmt
    name: md-0

---
apiVersion: anywhere.eks.amazonaws.com/v1alpha1
kind: VSphereDatacenterConfig
metadata:
  name: mgmt
spec:
  datacenter: "My_lab"
  insecure: true
  network: "lab_vmm_domain/nwname|ap_PSI|epg_EKS_ANYWHERE"
  server: "nwlabvc01-102935-domainname"
  thumbprint: "52:DA:7C:3E:78:8C:C8:F9:19:FD:2D:4F:14:F1:F8:4D:93:F1:B3:8F"

---
apiVersion: anywhere.eks.amazonaws.com/v1alpha1
kind: VSphereMachineConfig
metadata:
  name: mgmt-cp
spec:
  datastore: "nwlabESX02-Loc-DS-2"
  diskGiB: 25
  folder: "EKS-A/Templates"
  memoryMiB: 8192
  numCPUs: 2
  osFamily: bottlerocket
  resourcePool: "Resources"
  template: "bottlerocket-vmware-k8s-1.23-x86_64-1.9.2-b8074d44"
  users:
  - name: ec2-user
    sshAuthorizedKeys:
    - ssh-rsa 

---
apiVersion: anywhere.eks.amazonaws.com/v1alpha1
kind: VSphereMachineConfig
metadata:
  name: mgmt
spec:
  datastore: "nwlabESX02-Loc-DS-2"
  diskGiB: 25
  folder: "EKS-A/Templates"
  memoryMiB: 8192
  numCPUs: 2
  osFamily: bottlerocket
  resourcePool: "Resources"
  template: "bottlerocket-vmware-k8s-1.23-x86_64-1.9.2-b8074d44"
  users:
  - name: ec2-user
    sshAuthorizedKeys:
    - ssh-rsa 

---
apiVersion: anywhere.eks.amazonaws.com/v1alpha1
kind: VSphereMachineConfig
metadata:
  name: mgmt-etcd
spec:
  datastore: "nwlabESX02-Loc-DS-2"
  diskGiB: 25
  folder: "EKS-A/Templates"
  memoryMiB: 8192
  numCPUs: 2
  osFamily: bottlerocket
  resourcePool: "Resources"
  template: "bottlerocket-vmware-k8s-1.23-x86_64-1.9.2-b8074d44"
  users:
  - name: ec2-user
    sshAuthorizedKeys:
    - ssh-rsa 

Anything else we need to know?:

My dhcp config:

option domain-name "mydomain";
option domain-name-servers ns1, ns2;

default-lease-time 600;
max-lease-time 7200;

ddns-update-style none;

authoritative;

subnet 10.56.235.64 netmask 255.255.255.224 {
  range 10.56.235.80 10.56.235.90;
  option routers 10.56.235.65;
}

As dhcp delivers addresses in the 80-90 range, my etcd node gets one of those, and the control node also gets one of those, instead of .70 that I have specified in the manifest above. Why??

What is this statement expecting?? Executing command {"cmd": "/usr/bin/docker exec -i eksa_1668166906815177748 kubectl wait --timeout 3600.00s --for=condition=ControlPlaneReady clusters.cluster.x-k8s.io/mgmt --kubeconfig mgmt/generated/mgmt.kind.kubeconfig -n eksa-system"}

Is it listening on that endpoint address, then something is wrong, as it’s not instatiated by the provisioner with the IP address in the manifest.

Environment:

• EKS Anywhere Release: eksctl version 0.115.0 eksctl anywhere version v0.11.4

• EKS Distro Release: How to I find out?