operator: Error running cluster on M1 / ARM Mac OS for local development
I am currently following the steps outlined in this guide to create a local cluster for development purposes. However, I’m encountering errors when executing kind and kubectl, which are essential for creating and managing the cluster.
The errors I’m facing include:
build/_output/bin/kind: cannot execute binary fileexec format error: ./build/_output/bin/kubectl
Expected Behavior
The cluster should be created when I run make cluster-create, followed by interacting with it through kubectl
Current Behavior
The first step make cluster-create fails with the aforementioned errors.
Possible Solution
I suspect the issue might be related to compatibility with Apple Silicon (M1). In an attempt to resolve this, I made changes to the Makefile. It sort of got the cluster up and running though followed by an error in a later step:
I am explaining the rationale behind my changes.
- The
make cluster-createcommand depends onkubectlandkindso those os specific binaries are fetched beforehand.
kubectl
- For kubectl, I modified the command to fetch the binary to use the darwin distribution instead of linux, making it compatible with MacOS.
- The current command in makefile
curl -L https://storage.googleapis.com/kubernetes-release/release/v1.25.6/bin/linux/$(ARCH)/kubectlwhere linux is hardcoded. - There is already a
BUILDOSvariable already present in our makefile but there is no use for it. So I replacedlinux/$(ARCH)/kubectlwith$(BUILDOS)/$(ARCH)/kubectlto make it OS independent. - Is there any side effect of this?
kind
- For kind, I directly installed the binary using
sh -c "GOBIN=$(CURDIR)/$(BINDIR) go install sigs.k8s.io/kind", ensuring that I had a MacOS-compatible kind binary available. - The current command in makefile is
$(CONTAINERIZED) $(CALICO_BUILD) sh -c "GOBIN=/go/src/$(PACKAGE_NAME)/$(BINDIR) go install sigs.k8s.io/kind"- It appears that the
kindbinary is downloaded inside the Linux-based Docker container and then stored in thebuild/_output/bindirectory, which is mounted inside the container with read/write permissions. This setup allows the binary to be accessible both inside and outside the container, essentially persisting the Linux distribution for later use. - But again, in subsequent commands when
kind create clusteris invoked, I have akindbinary for linux which is not compatible.
- It appears that the
- But, does this impact the
kind control planesomehow as I am getting this error when I run the operator against the local cluster described below?
This KUBECONFIG=./kubeconfig.yaml go run ./ --enable-leader-election=false command is not working as expected and gives the following error.
level":"error","ts":"2023-10-15T00:46:28+06:00","logger":"setup","msg":"problem running manager","error":"failed to wait for policy-recommendation-controller caches to sync: timed out waiting for cache to be synced"
Am I heading in the right direction or what should I do to run the cluster on Apple Silicon (M1)?
Steps to Reproduce (for bugs)
- Clone the repo
- Make sure docker is running
- Follow this guide, and try to run in Apple Silicon (M1)
Context
I am trying to make a local cluster for development purposes.
Your Environment
- Operating System and version: MacOS Sonoma Version 14.0
- Link to your project (optional): cluster-create fix
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 15 (13 by maintainers)
Sorry there hasn’t been any response here for a while.
For 1: If you want to make the suggested change that would be good.
For 2: I think you are suggesting switch to
I don’t think that is something we would want in general, since it would no longer be containerized which is something we want to maintain. I’d be ok with a conditional based on BUILDOS, perhaps if BUILDOS != linux then instruct user to copy a functional kind binary to
$(BINDIR)/kindFor 3: You could request the projectcalico/calico to push the node image for arm on master builds. Another option would be to have a make target that switches the versions to ease creating a build with latest (or some other tag). Maybe something like the following
If you’re getting what looks like a functional cluster from
make cluster-createthen I think you’re on the right path. After I run that command I getAlso if I look at the crds in the cluster I see the following. I’m including this because I’m wondering if they were not created as they should have been based on the error you’ve received.