crossplane: Function: certificate signed by unknown authority "Crossplane"
What happened?
We are experiencing an intermittent problem with crossplane and the registration of the patch-and-transform function. We are constantly spinning down and spinning up clusters and we’re now seeing this relatively regularly.
When the problem occurs we are seeing the following message appear in our logs and if we try to inspect the resource using crossplane beta trace:
crossplane-6f458f9b86-plp42 universal-crossplane 2024-03-05T15:29:40Z DEBUG crossplane Event(v1.ObjectReference{Kind:“XCompositeExample”, Namespace:“”, Name:“test05-qh6kd”, UID:“e152b643-b2f7-4161-a893-fab54b1265d0”, APIVersion:“platform.mydomain.com/v1alpha1”, ResourceVersion:“110305”, FieldPath:“”}): type: ‘Warning’ reason: ‘ComposeResources’ cannot compose resources: cannot run Composition pipeline step “patch-and-transform”: cannot run Function “crossplane-contrib-function-patch-and-transform”: rpc error: code = Unavailable desc = last connection error: connection error: desc = “transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "Crossplane")”
How can we reproduce it?
As I mention, it’s difficult to reproduce since I can spin up a cluster using exactly the same IaC branch and get a functioning deployment.
However at present it’s very difficult to diagnose what the problem actually is. Other than the above message there is very little in the logs and my generate approach is to try to compare a broken cluster with a working cluster. I’ve looked at the TLS secrets associated with the pods and the certificates look fine. I’ve deleted the upbound-system namespace and let everything get recreated and the problem still persists.
If someone can shed some light on how to diagnose this problem further that would be great.
What environment did it happen in?
Crossplane version: 0.14.4 and 0.15.0 Kubernetes version: v1.27.8 GKE Function crossplane-contrib-function-patch-and-transform: 0.3.0
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Reactions: 11
- Comments: 17 (4 by maintainers)
I just upgraded helm chart to 1.17.2 and am still fighting with this issue 😦
No. I think you misunderstand the question. We are deploying crossplane through gitops, so how do we ensure this doesn’t occur. I don’t want to manually have to intervene manually and ideally I don’t want to have to put ordering into our end state definitions.
We have encountered this same issue intermittently when using function-go-templating in Crossplane v1.15.0. After trying several approaches, we found a solution that involved both re-creating certain secrets and restarting the Crossplane pods.
Solution
Delete Existing Secrets
Remove the existing TLS secrets:
Recreate Secrets
Create the secrets again with empty data:
Restart Crossplane Pods
Delete the Crossplane pods to allow them to restart and regenerate the necessary certificates:
How It Works
Implementing these steps completely eliminated the issues we were facing with function-go-templating in Crossplane v1.15.0.
We are facing the same issue, both when setting up the cluster with ArgoCD and with a script. Even when the functions and providers and installed after Crossplane is fully ready it can still occur. We do not uninstall Crossplane, we install it once, then install the providers and functions. At this point, the
crossplane-root-cais populated by Crossplane’s init container, so the certificate should be properly signed.Cleaning the secrets and rolling out the pods does not necessarily solve this issue, sometime it does, sometime it doesn’t; the proper order seems yet unclear. Could this be an issue with having multiple Crossplane replicas that don’t sync correctly during the init?
@jbw976 Just spitballing after hitting the same issue but is it possible that the fix implemented only works if the functions are passed to the helm chart using the parameters in the values file but not if they are deployed otherwise?
@jbw976 In our case we managed to fix this issue by deleting all the Crossplane secrets with the
tls-serverortls-clientsuffix, we’ve seen in N times already, the biggest issue is that it happens randomly. We have almost the same situation as previously mentioned, we’re dynamically creating k8s clusters.The issue also touches function and provider pods. I’ve observed that the IPs from the error message all point to the
konnectivity-agentpods in kube-system namespace, it’s an IBM cluster (IKS, k8s v1.31.2).We’re investigating this issue right now, we’ll get back with more info if we find anything, maybe it’s just issue on IBM side
Same issue encountered with the v1.18.2, suddenly our compositions stopped being reconciled (no status set on XRC) and there’s a lot of events:
EDIT: it may be related to the fact that we have some TLS errors in the crossplane pod:
Looking at this and I am wondering if we are getting an incorrect
NotFoundhere from the controller runtime cache where we try to get thecrossplane-root-casecret and regenerate the ca secret leaving previously generated function secrets out of sync.I couldn’t reproduce it yet, tried with multiple replicas, low cpu etc, but I wouldn’t be surprised if 👆 is the root cause. If/once we could validate this (by somehow slowing down the controller runtime cache), fix should be straightforward I believe.