operator: Error running cluster on M1 / ARM Mac OS for local development

I am currently following the steps outlined in this guide to create a local cluster for development purposes. However, I’m encountering errors when executing kind and kubectl, which are essential for creating and managing the cluster.

The errors I’m facing include:

  • build/_output/bin/kind: cannot execute binary file
  • exec format error: ./build/_output/bin/kubectl

Expected Behavior

The cluster should be created when I run make cluster-create, followed by interacting with it through kubectl

Current Behavior

The first step make cluster-create fails with the aforementioned errors.

Possible Solution

I suspect the issue might be related to compatibility with Apple Silicon (M1). In an attempt to resolve this, I made changes to the Makefile. It sort of got the cluster up and running though followed by an error in a later step:

I am explaining the rationale behind my changes.

  • The make cluster-create command depends on kubectl and kind so those os specific binaries are fetched beforehand.

kubectl

  • For kubectl, I modified the command to fetch the binary to use the darwin distribution instead of linux, making it compatible with MacOS.
  • The current command in makefile curl -L https://storage.googleapis.com/kubernetes-release/release/v1.25.6/bin/linux/$(ARCH)/kubectl where linux is hardcoded.
  • There is already a BUILDOS variable already present in our makefile but there is no use for it. So I replaced linux/$(ARCH)/kubectl with $(BUILDOS)/$(ARCH)/kubectl to make it OS independent.
  • Is there any side effect of this?

kind

  • For kind, I directly installed the binary using sh -c "GOBIN=$(CURDIR)/$(BINDIR) go install sigs.k8s.io/kind", ensuring that I had a MacOS-compatible kind binary available.
  • The current command in makefile is $(CONTAINERIZED) $(CALICO_BUILD) sh -c "GOBIN=/go/src/$(PACKAGE_NAME)/$(BINDIR) go install sigs.k8s.io/kind"
    • It appears that the kind binary is downloaded inside the Linux-based Docker container and then stored in the build/_output/bin directory, which is mounted inside the container with read/write permissions. This setup allows the binary to be accessible both inside and outside the container, essentially persisting the Linux distribution for later use.
    • But again, in subsequent commands when kind create cluster is invoked, I have a kind binary for linux which is not compatible.
  • But, does this impact the kind control plane somehow as I am getting this error when I run the operator against the local cluster described below?

This KUBECONFIG=./kubeconfig.yaml go run ./ --enable-leader-election=false command is not working as expected and gives the following error.

level":"error","ts":"2023-10-15T00:46:28+06:00","logger":"setup","msg":"problem running manager","error":"failed to wait for policy-recommendation-controller caches to sync: timed out waiting for cache to be synced"

Am I heading in the right direction or what should I do to run the cluster on Apple Silicon (M1)?

Steps to Reproduce (for bugs)

  1. Clone the repo
  2. Make sure docker is running
  3. Follow this guide, and try to run in Apple Silicon (M1)

Context

I am trying to make a local cluster for development purposes.

Your Environment

  • Operating System and version: MacOS Sonoma Version 14.0
  • Link to your project (optional): cluster-create fix

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 15 (13 by maintainers)

Most upvoted comments

Sorry there hasn’t been any response here for a while.

For 1: If you want to make the suggested change that would be good.

For 2: I think you are suggesting switch to

sh -c "GOBIN=$(CURDIR)/$(BINDIR) go install sigs.k8s.io/kind"

I don’t think that is something we would want in general, since it would no longer be containerized which is something we want to maintain. I’d be ok with a conditional based on BUILDOS, perhaps if BUILDOS != linux then instruct user to copy a functional kind binary to $(BINDIR)/kind

For 3: You could request the projectcalico/calico to push the node image for arm on master builds. Another option would be to have a make target that switches the versions to ease creating a build with latest (or some other tag). Maybe something like the following

set-calico-version:
  sed -i -e "s/version: .*$$/version: $(VERSION)/" config/calico_versions.yml
  make gen-versions-calico

If you’re getting what looks like a functional cluster from make cluster-create then I think you’re on the right path. After I run that command I get

> export KUBECONFIG=kubeconfig.yaml 
> kubectl get pods -A
NAMESPACE            NAME                                         READY   STATUS    RESTARTS   AGE
kube-system          coredns-558bd4d5db-4s6jh                     0/1     Pending   0          54s
kube-system          coredns-558bd4d5db-j26ws                     0/1     Pending   0          54s
kube-system          etcd-kind-control-plane                      1/1     Running   0          68s
kube-system          kube-apiserver-kind-control-plane            1/1     Running   0          68s
kube-system          kube-controller-manager-kind-control-plane   1/1     Running   0          68s
kube-system          kube-proxy-67d9l                             1/1     Running   0          35s
kube-system          kube-proxy-8h8v4                             1/1     Running   0          35s
kube-system          kube-proxy-rvw7f                             1/1     Running   0          54s
kube-system          kube-proxy-z8b9p                             1/1     Running   0          35s
kube-system          kube-scheduler-kind-control-plane            1/1     Running   0          68s
local-path-storage   local-path-provisioner-5545dd49d7-wvj9w      0/1     Pending   0          54s

Also if I look at the crds in the cluster I see the following. I’m including this because I’m wondering if they were not created as they should have been based on the error you’ve received.

> kubectl get crds | grep operator
amazoncloudintegrations.operator.tigera.io              2023-10-17T13:28:27Z
apiservers.operator.tigera.io                           2023-10-17T13:28:27Z
applicationlayers.operator.tigera.io                    2023-10-17T13:28:27Z
authentications.operator.tigera.io                      2023-10-17T13:28:27Z
compliances.operator.tigera.io                          2023-10-17T13:28:27Z
egressgateways.operator.tigera.io                       2023-10-17T13:28:27Z
imagesets.operator.tigera.io                            2023-10-17T13:28:27Z
installations.operator.tigera.io                        2023-10-17T13:28:27Z
intrusiondetections.operator.tigera.io                  2023-10-17T13:28:27Z
logcollectors.operator.tigera.io                        2023-10-17T13:28:27Z
logstorages.operator.tigera.io                          2023-10-17T13:28:27Z
managementclusterconnections.operator.tigera.io         2023-10-17T13:28:27Z
managementclusters.operator.tigera.io                   2023-10-17T13:28:27Z
managers.operator.tigera.io                             2023-10-17T13:28:27Z
monitors.operator.tigera.io                             2023-10-17T13:28:27Z
policyrecommendations.operator.tigera.io                2023-10-17T13:28:27Z
tenants.operator.tigera.io                              2023-10-17T13:28:27Z
tigerastatuses.operator.tigera.io                       2023-10-17T13:28:27Z