linkerd2: linkerd cli doesn't work on our macs

What is the issue?

you always get a timeout

How can it be reproduced?

linkerd check --pre

Logs, error output, etc

Error rendering install manifest: Get “https://k8s-test-ci-rs.otenv.com:6443/api/v1/namespaces/linkerd/configmaps/linkerd-config?timeout=30s”: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers

output of linkerd check -o short

would be the same

Environment

mac, k8s 1.20

Possible solution

No response

Additional context

[Mike Bell](https://app.slack.com/team/U02UXTN060H) [Today at 1:18 PM](https://linkerd.slack.com/archives/C89RTCWJF/p1644959935619059)
mbell ~/git/astp/linkerd/install [main]$ $ linkerd check --pre    
Error rendering install manifest: Get "https://k8s-test-ci-rs.otenv.com:6443/api/v1/namespaces/linkerd/configmaps/linkerd-config?timeout=30s": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)%
white_check_mark
eyes
raised_hands





61 replies

[Steve Gray](https://app.slack.com/team/U02KZDEC9F1)  [1 hour ago](https://linkerd.slack.com/archives/C89RTCWJF/p1644960136367659?thread_ts=1644959935.619059&cid=C89RTCWJF)
Not a trust issue or anything like that. The error appears to imply your client side had issues/timeouts when reading the content map from the Kubernetes APIs. Does your account/credential set have access to that API, or do you need access to the cluster to be via a VPN?
You can also retry with --verbose on the check command to see.
Effectively it's doing the same as the below, using your local Kubeconfig/currently selected context. (It actually does this with the same code that's in the Kubectl command line tools too/the Go SDK).
kubectl get configmap linkerd-config -n linkerd (edited) 

[Mike Bell](https://app.slack.com/team/U02UXTN060H)  [1 hour ago](https://linkerd.slack.com/archives/C89RTCWJF/p1644960268466199?thread_ts=1644959935.619059&cid=C89RTCWJF)
I have full admin rights

[Mike Bell](https://app.slack.com/team/U02UXTN060H)  [1 hour ago](https://linkerd.slack.com/archives/C89RTCWJF/p1644960274652339?thread_ts=1644959935.619059&cid=C89RTCWJF)
and 0 issues with kubectl

[Mike Bell](https://app.slack.com/team/U02UXTN060H)  [1 hour ago](https://linkerd.slack.com/archives/C89RTCWJF/p1644960289516459?thread_ts=1644959935.619059&cid=C89RTCWJF)
mbell ~/.kube $ $ kubectl get configmap linkerd-config -n linkerd
Error from server (NotFound): namespaces "linkerd" not found

[Mike Bell](https://app.slack.com/team/U02UXTN060H)  [1 hour ago](https://linkerd.slack.com/archives/C89RTCWJF/p1644960296933559?thread_ts=1644959935.619059&cid=C89RTCWJF)
(which is correct since this is the pre check)

[Steve Gray](https://app.slack.com/team/U02KZDEC9F1)  [1 hour ago](https://linkerd.slack.com/archives/C89RTCWJF/p1644960331868009?thread_ts=1644959935.619059&cid=C89RTCWJF)
Yeah, so its odd that it's timing out when being run via the linkerd tools. Can you dump snippets for:
kubectl version
and
linkerd version

[Mike Bell](https://app.slack.com/team/U02UXTN060H)  [1 hour ago](https://linkerd.slack.com/archives/C89RTCWJF/p1644960349123489?thread_ts=1644959935.619059&cid=C89RTCWJF)
mbell ~/.kube $ $ kubectl version
Client Version: [version.Info](http://version.info/){Major:"1", Minor:"22", GitVersion:"v1.22.1", GitCommit:"632ed300f2c34f6d6d15ca4cef3d3c7073412212", GitTreeState:"clean", BuildDate:"2021-08-19T15:38:26Z", GoVersion:"go1.16.6", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: [version.Info](http://version.info/){Major:"1", Minor:"20", GitVersion:"v1.20.11", GitCommit:"27522a29febbcc4badac257763044d0d90c11abd", GitTreeState:"clean", BuildDate:"2021-09-15T19:16:25Z", GoVersion:"go1.15.15", Compiler:"gc", Platform:"linux/amd64"}
WARNING: version difference between client (1.22) and server (1.20) exceeds the supported minor version skew of +/-1

[Mike Bell](https://app.slack.com/team/U02UXTN060H)  [1 hour ago](https://linkerd.slack.com/archives/C89RTCWJF/p1644960365695509?thread_ts=1644959935.619059&cid=C89RTCWJF)
mbell ~/.kube $ $ linkerd version
Client version: stable-2.11.1
Server version: unavailable

[Mike Bell](https://app.slack.com/team/U02UXTN060H)  [1 hour ago](https://linkerd.slack.com/archives/C89RTCWJF/p1644960396584739?thread_ts=1644959935.619059&cid=C89RTCWJF)
(and yes I know the server is a bit old, however the linkerd stable currently supports this version)

[Steve Gray](https://app.slack.com/team/U02KZDEC9F1)  [1 hour ago](https://linkerd.slack.com/archives/C89RTCWJF/p1644960443742309?thread_ts=1644959935.619059&cid=C89RTCWJF)
Hrm, that shouldn't cause a problem. Try with the
--verbose --pre
flags on the linkerd check and see what comes up. It might be choking on something else, and then timing out.

[Steve Gray](https://app.slack.com/team/U02KZDEC9F1)  [1 hour ago](https://linkerd.slack.com/archives/C89RTCWJF/p1644960475799319?thread_ts=1644959935.619059&cid=C89RTCWJF)
i.e. Has the Kubernetes API certificate expired and something, somewhere is hanging - but Kubectl is just ignoring it or something (wild guess, probably not it)

[Mike Bell](https://app.slack.com/team/U02UXTN060H)  [1 hour ago](https://linkerd.slack.com/archives/C89RTCWJF/p1644960479021569?thread_ts=1644959935.619059&cid=C89RTCWJF)
yep, while waiting, I tried (just the --pre)

[Mike Bell](https://app.slack.com/team/U02UXTN060H)  [1 hour ago](https://linkerd.slack.com/archives/C89RTCWJF/p1644960483506099?thread_ts=1644959935.619059&cid=C89RTCWJF)
on multiple other clusters

[Mike Bell](https://app.slack.com/team/U02UXTN060H)  [1 hour ago](https://linkerd.slack.com/archives/C89RTCWJF/p1644960488889129?thread_ts=1644959935.619059&cid=C89RTCWJF)
so no, no cert issue per se

[Mike Bell](https://app.slack.com/team/U02UXTN060H)  [1 hour ago](https://linkerd.slack.com/archives/C89RTCWJF/p1644960518491659?thread_ts=1644959935.619059&cid=C89RTCWJF)
(or at least whatever it is is universal to our clusters)

[Mike Bell](https://app.slack.com/team/U02UXTN060H)  [1 hour ago](https://linkerd.slack.com/archives/C89RTCWJF/p1644960543671409?thread_ts=1644959935.619059&cid=C89RTCWJF)
--verbose doesn't print a thing more than non verbose

[Mike Bell](https://app.slack.com/team/U02UXTN060H)  [1 hour ago](https://linkerd.slack.com/archives/C89RTCWJF/p1644960545734459?thread_ts=1644959935.619059&cid=C89RTCWJF)
:wink:

[Mike Bell](https://app.slack.com/team/U02UXTN060H)  [1 hour ago](https://linkerd.slack.com/archives/C89RTCWJF/p1644960555401379?thread_ts=1644959935.619059&cid=C89RTCWJF)
Error rendering install manifest: Get "https://k8s-sp-ci-rs.otenv.com:6443/api/v1/namespaces/linkerd/configmaps/linkerd-config?timeout=30s": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)%

[Mike Bell](https://app.slack.com/team/U02UXTN060H)  [1 hour ago](https://linkerd.slack.com/archives/C89RTCWJF/p1644960565761019?thread_ts=1644959935.619059&cid=C89RTCWJF)
(as you can see this is another cluster)

[Mike Bell](https://app.slack.com/team/U02UXTN060H)  [1 hour ago](https://linkerd.slack.com/archives/C89RTCWJF/p1644960733682869?thread_ts=1644959935.619059&cid=C89RTCWJF)
curl also fails (no timeout) without -k, but before you go "aha" - the difference is we provide a certificate authority in our .kube/config

[Mike Bell](https://app.slack.com/team/U02UXTN060H)  [1 hour ago](https://linkerd.slack.com/archives/C89RTCWJF/p1644960760158869?thread_ts=1644959935.619059&cid=C89RTCWJF)
(curl with -k works fine, though of course it gives me an unauthorized error)

[Steve Gray](https://app.slack.com/team/U02KZDEC9F1)  [1 hour ago](https://linkerd.slack.com/archives/C89RTCWJF/p1644960991954599?thread_ts=1644959935.619059&cid=C89RTCWJF)
Try explicitly passing the Kubeconfig file to use with --kubeconfig <blah>
Its odd that its not picking up that, as I imagine that'd be fairly standard practice :confused:

[Mike Bell](https://app.slack.com/team/U02UXTN060H)  [1 hour ago](https://linkerd.slack.com/archives/C89RTCWJF/p1644961058246449?thread_ts=1644959935.619059&cid=C89RTCWJF)
mbell ~/.kube $ $ linkerd check --pre --kubeconfig ~/.kube/config

[Mike Bell](https://app.slack.com/team/U02UXTN060H)  [1 hour ago](https://linkerd.slack.com/archives/C89RTCWJF/p1644961060060539?thread_ts=1644959935.619059&cid=C89RTCWJF)
same result

[Mike Bell](https://app.slack.com/team/U02UXTN060H)  [1 hour ago](https://linkerd.slack.com/archives/C89RTCWJF/p1644961096733729?thread_ts=1644959935.619059&cid=C89RTCWJF)
- cluster:
  certificate-authority: certs/test-ci-rs/k8s-ca.crt
  server: https://k8s-test-ci-rs.otenv.com:6443/
 name: test-ci-rs
contexts:

[Mike Bell](https://app.slack.com/team/U02UXTN060H)  [1 hour ago](https://linkerd.slack.com/archives/C89RTCWJF/p1644961102905149?thread_ts=1644959935.619059&cid=C89RTCWJF)
is the relevant part of the config file

[Steve Gray](https://app.slack.com/team/U02KZDEC9F1)  [1 hour ago](https://linkerd.slack.com/archives/C89RTCWJF/p1644961170855499?thread_ts=1644959935.619059&cid=C89RTCWJF)
Try making that path absolute :wink:

[Steve Gray](https://app.slack.com/team/U02KZDEC9F1)  [1 hour ago](https://linkerd.slack.com/archives/C89RTCWJF/p1644961256994049?thread_ts=1644959935.619059&cid=C89RTCWJF)
I suspect (random psychic debug hat moment :male_mage: )  - that Linkerd is not finding that file relative to its own location, whereas Kubectl is, and that's the problem. 99.99% of people would have that be an absolute path or directly embedded as certificate-authority-data (encoded base64 of the content) directly in the YAML, and thats potentially why the mixed outcome.

[Steve Gray](https://app.slack.com/team/U02KZDEC9F1)  [1 hour ago](https://linkerd.slack.com/archives/C89RTCWJF/p1644961564180089?thread_ts=1644959935.619059&cid=C89RTCWJF)
:awaits-in-silence:

[Mike Bell](https://app.slack.com/team/U02UXTN060H)  [1 hour ago](https://linkerd.slack.com/archives/C89RTCWJF/p1644961578961739?thread_ts=1644959935.619059&cid=C89RTCWJF)
or the mac version is compiled with the wrong cgo library

[Mike Bell](https://app.slack.com/team/U02UXTN060H)  [1 hour ago](https://linkerd.slack.com/archives/C89RTCWJF/p1644961583494329?thread_ts=1644959935.619059&cid=C89RTCWJF)
Ive run into that before

[Mike Bell](https://app.slack.com/team/U02UXTN060H)  [1 hour ago](https://linkerd.slack.com/archives/C89RTCWJF/p1644961592960639?thread_ts=1644959935.619059&cid=C89RTCWJF)
(weird dns issues)

[Mike Bell](https://app.slack.com/team/U02UXTN060H)  [1 hour ago](https://linkerd.slack.com/archives/C89RTCWJF/p1644961659196799?thread_ts=1644959935.619059&cid=C89RTCWJF)
I'll try the aboslute path...

[Mike Bell](https://app.slack.com/team/U02UXTN060H)  [1 hour ago](https://linkerd.slack.com/archives/C89RTCWJF/p1644961737805519?thread_ts=1644959935.619059&cid=C89RTCWJF)
absolute doesnt work either

[Steve Gray](https://app.slack.com/team/U02KZDEC9F1)  [44 minutes ago](https://linkerd.slack.com/archives/C89RTCWJF/p1644961840770109?thread_ts=1644959935.619059&cid=C89RTCWJF)
Move it out to the certificate-authority-data key as a string (when you base64 encode it, it should look like a string with LS0t at the start, which is the first few bytes when correctly encoded). That should hopefully put it to bed. You'll probably need to take a sec too if this does/doesn't work to raise a github issue for either of "doesn't work with pathed certs" or "my cluster wont connect" :slightly_smiling_face:

[Mike Bell](https://app.slack.com/team/U02UXTN060H)  [44 minutes ago](https://linkerd.slack.com/archives/C89RTCWJF/p1644961850721059?thread_ts=1644959935.619059&cid=C89RTCWJF)
Which is what I would assume - eg that the client library for k8s usees the same code as kubectl :wink:

[Steve Gray](https://app.slack.com/team/U02KZDEC9F1)  [43 minutes ago](https://linkerd.slack.com/archives/C89RTCWJF/p1644961900777039?thread_ts=1644959935.619059&cid=C89RTCWJF)
Yeah, its the same library - but Kubectl does a whole wonderful word of junk when it comes to converting those pseudo-relative paths into actual paths, using a concept called location of origin etc. There's a lot of history behind that code it seems .

[Mike Bell](https://app.slack.com/team/U02UXTN060H)  [39 minutes ago](https://linkerd.slack.com/archives/C89RTCWJF/p1644962135013069?thread_ts=1644959935.619059&cid=C89RTCWJF)
nope no cigar

[Mike Bell](https://app.slack.com/team/U02UXTN060H)  [39 minutes ago](https://linkerd.slack.com/archives/C89RTCWJF/p1644962148309319?thread_ts=1644959935.619059&cid=C89RTCWJF)
changed it to cert-authiority-data: base64, and kubectl get nodes continues to function

[Mike Bell](https://app.slack.com/team/U02UXTN060H)  [39 minutes ago](https://linkerd.slack.com/archives/C89RTCWJF/p1644962150446289?thread_ts=1644959935.619059&cid=C89RTCWJF)
but no joy

[Mike Bell](https://app.slack.com/team/U02UXTN060H)  [39 minutes ago](https://linkerd.slack.com/archives/C89RTCWJF/p1644962180067649?thread_ts=1644959935.619059&cid=C89RTCWJF)
A friend of mine is trying to recompile this for the mac, because there are infinite issues with go's dns library and the mac

[Steve Gray](https://app.slack.com/team/U02KZDEC9F1)  [36 minutes ago](https://linkerd.slack.com/archives/C89RTCWJF/p1644962350431769?thread_ts=1644959935.619059&cid=C89RTCWJF)
Whilst anecdotal, I'm on the latest MacOS and stable-2.11.1 and there's no issues hitting clusters I've built with OCI, AWS EKS or Azure AKS (I'm just rotating between all three at the moment). Anything overly special about the cluster or the way it's built? The stable versions been out for a long time, so it'd be odd that we'd not seen more reports of it if it was a go runtime issue.

[Steve Gray](https://app.slack.com/team/U02KZDEC9F1)  [35 minutes ago](https://linkerd.slack.com/archives/C89RTCWJF/p1644962389217939?thread_ts=1644959935.619059&cid=C89RTCWJF)
you could drop your kubeconfig into a docker ubuntu instance and try the CLI from there on Linux, that'd narrow down if its a MacOS thing.

[Mike Bell](https://app.slack.com/team/U02UXTN060H)  [12 minutes ago](https://linkerd.slack.com/archives/C89RTCWJF/p1644963741448639?thread_ts=1644959935.619059&cid=C89RTCWJF)
it's the mac cli build

[Mike Bell](https://app.slack.com/team/U02UXTN060H)  [12 minutes ago](https://linkerd.slack.com/archives/C89RTCWJF/p1644963753573969?thread_ts=1644959935.619059&cid=C89RTCWJF)
I did it fine on linux a few minutes ago

[Mike Bell](https://app.slack.com/team/U02UXTN060H)  [12 minutes ago](https://linkerd.slack.com/archives/C89RTCWJF/p1644963764926859?thread_ts=1644959935.619059&cid=C89RTCWJF)
There are many many known issues of the DNS library in GO

[Mike Bell](https://app.slack.com/team/U02UXTN060H)  [12 minutes ago](https://linkerd.slack.com/archives/C89RTCWJF/p1644963769521599?thread_ts=1644959935.619059&cid=C89RTCWJF)
breaking on mac apps

[Steve Gray](https://app.slack.com/team/U02KZDEC9F1)  [10 minutes ago](https://linkerd.slack.com/archives/C89RTCWJF/p1644963863588599?thread_ts=1644959935.619059&cid=C89RTCWJF)
That was very much the case in the dim and distant past, but it's been a fairly long time since I'd heard anything around that? The 'ole resolve.conf / CGO issues  were swapped out for libSystem on Go on the mac a long time back.

[Mike Bell](https://app.slack.com/team/U02UXTN060H)  [10 minutes ago](https://linkerd.slack.com/archives/C89RTCWJF/p1644963882361419?thread_ts=1644959935.619059&cid=C89RTCWJF)
yes, that's what I'm talking about

[Steve Gray](https://app.slack.com/team/U02KZDEC9F1)  [9 minutes ago](https://linkerd.slack.com/archives/C89RTCWJF/p1644963908580599?thread_ts=1644959935.619059&cid=C89RTCWJF)
Yeah, but that's an issue that was fixed, IIRC, around go 1.14/5-ish.

[Steve Gray](https://app.slack.com/team/U02KZDEC9F1)  [9 minutes ago](https://linkerd.slack.com/archives/C89RTCWJF/p1644963946861839?thread_ts=1644959935.619059&cid=C89RTCWJF)
Not saying using a different resolver isn't fixing your particular problem, but that its something endemic seems unlikely - given how many users who work with Linkerd use a Mac as their mainstay platform.

[Mike Bell](https://app.slack.com/team/U02UXTN060H)  [8 minutes ago](https://linkerd.slack.com/archives/C89RTCWJF/p1644963971244549?thread_ts=1644959935.619059&cid=C89RTCWJF)
it's endemic "here" - and the only thing that I can guess is bespoke is our cisco vpn

[Mike Bell](https://app.slack.com/team/U02UXTN060H)  [8 minutes ago](https://linkerd.slack.com/archives/C89RTCWJF/p1644963983681219?thread_ts=1644959935.619059&cid=C89RTCWJF)
(only bespoke in the sense not everyone uses it)

[Steve Gray](https://app.slack.com/team/U02KZDEC9F1)  [6 minutes ago](https://linkerd.slack.com/archives/C89RTCWJF/p1644964107835639?thread_ts=1644959935.619059&cid=C89RTCWJF)
Cisco's VPN product does a wide range of exciting things to the local network stack, and I'd wager that's almost certainly going to be where the problem lies. Used to use that nightmare at a previous employer, and we were generally just grateful if Email still worked - but it was definitely secure, even most staff struggled to access things :slightly_smiling_face:

[Steve Gray](https://app.slack.com/team/U02KZDEC9F1)  [5 minutes ago](https://linkerd.slack.com/archives/C89RTCWJF/p1644964151521279?thread_ts=1644959935.619059&cid=C89RTCWJF)
Your docker instance would have the advantage of being on the far end of that, and thus not directly exposed to the magic. I imagine if you used a parallels VM on your mac inside your mac, and tried the same, you'd find it worked too.

[Steve Gray](https://app.slack.com/team/U02KZDEC9F1)  [4 minutes ago](https://linkerd.slack.com/archives/C89RTCWJF/p1644964256496969?thread_ts=1644959935.619059&cid=C89RTCWJF)
AnyConnect has always sucked on MacOS (I assume it's AC, [@Mike Bell](https://linkerd.slack.com/team/U02UXTN060H)?)

[Mike Bell](https://app.slack.com/team/U02UXTN060H)  [3 minutes ago](https://linkerd.slack.com/archives/C89RTCWJF/p1644964278297069?thread_ts=1644959935.619059&cid=C89RTCWJF)
yes it is

[Steve Gray](https://app.slack.com/team/U02KZDEC9F1)  [2 minutes ago](https://linkerd.slack.com/archives/C89RTCWJF/p1644964330095449?thread_ts=1644959935.619059&cid=C89RTCWJF)
Unfortunately changing the way linkerd is built for that issue is unlikely, as I imagine switching back to effectively using the internal DNS go framework code and resolver.conf would hurt more people than it helps :disappointed: (edited) 

[Steve Gray](https://app.slack.com/team/U02KZDEC9F1)  [2 minutes ago](https://linkerd.slack.com/archives/C89RTCWJF/p1644964359564269?thread_ts=1644959935.619059&cid=C89RTCWJF)
and as you're aware, its not a runtime toggle.

[Mike Bell](https://app.slack.com/team/U02UXTN060H)  [1 minute ago](https://linkerd.slack.com/archives/C89RTCWJF/p1644964394478159?thread_ts=1644959935.619059&cid=C89RTCWJF)
yup
New

[Steve Gray](https://app.slack.com/team/U02KZDEC9F1)  [< 1 minute ago](https://linkerd.slack.com/archives/C89RTCWJF/p1644964442778219?thread_ts=1644959935.619059&cid=C89RTCWJF)
Maybe put it on the Github issues list anyway, maybe folks can consider doing a second MacOS build without.

Would you like to work on fixing this bug?

maybe