crossplane: Function: certificate signed by unknown authority "Crossplane"

What happened?

We are experiencing an intermittent problem with crossplane and the registration of the patch-and-transform function. We are constantly spinning down and spinning up clusters and we’re now seeing this relatively regularly.

When the problem occurs we are seeing the following message appear in our logs and if we try to inspect the resource using crossplane beta trace:

crossplane-6f458f9b86-plp42 universal-crossplane 2024-03-05T15:29:40Z DEBUG crossplane Event(v1.ObjectReference{Kind:“XCompositeExample”, Namespace:“”, Name:“test05-qh6kd”, UID:“e152b643-b2f7-4161-a893-fab54b1265d0”, APIVersion:“platform.mydomain.com/v1alpha1”, ResourceVersion:“110305”, FieldPath:“”}): type: ‘Warning’ reason: ‘ComposeResources’ cannot compose resources: cannot run Composition pipeline step “patch-and-transform”: cannot run Function “crossplane-contrib-function-patch-and-transform”: rpc error: code = Unavailable desc = last connection error: connection error: desc = “transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "Crossplane")”

How can we reproduce it?

As I mention, it’s difficult to reproduce since I can spin up a cluster using exactly the same IaC branch and get a functioning deployment.

However at present it’s very difficult to diagnose what the problem actually is. Other than the above message there is very little in the logs and my generate approach is to try to compare a broken cluster with a working cluster. I’ve looked at the TLS secrets associated with the pods and the certificates look fine. I’ve deleted the upbound-system namespace and let everything get recreated and the problem still persists.

If someone can shed some light on how to diagnose this problem further that would be great.

What environment did it happen in?

Crossplane version: 0.14.4 and 0.15.0 Kubernetes version: v1.27.8 GKE Function crossplane-contrib-function-patch-and-transform: 0.3.0

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Reactions: 11
  • Comments: 17 (4 by maintainers)

Most upvoted comments

I just upgraded helm chart to 1.17.2 and am still fighting with this issue 😦

No. I think you misunderstand the question. We are deploying crossplane through gitops, so how do we ensure this doesn’t occur. I don’t want to manually have to intervene manually and ideally I don’t want to have to put ordering into our end state definitions.

We have encountered this same issue intermittently when using function-go-templating in Crossplane v1.15.0. After trying several approaches, we found a solution that involved both re-creating certain secrets and restarting the Crossplane pods.


Solution

  1. Delete Existing Secrets
    Remove the existing TLS secrets:

    kubectl delete secret -n crossplane-system crossplane-tls-client crossplane-tls-server
    
  2. Recreate Secrets
    Create the secrets again with empty data:

    kubectl create secret generic crossplane-tls-client --namespace=crossplane-system --type=Opaque
    kubectl create secret generic crossplane-tls-server --namespace=crossplane-system --type=Opaque
    
  3. Restart Crossplane Pods
    Delete the Crossplane pods to allow them to restart and regenerate the necessary certificates:

    kubectl delete pods -n crossplane-system -l app=crossplane
    

How It Works

  • The secrets are initially re-created with empty data.
  • Once the Crossplane pods start up again, the init container in crossplane deployment automatically populates these secrets with new certificates.
  • This process effectively refreshes the certificate data and resolves the intermittent errors.

Implementing these steps completely eliminated the issues we were facing with function-go-templating in Crossplane v1.15.0.

Image

We are facing the same issue, both when setting up the cluster with ArgoCD and with a script. Even when the functions and providers and installed after Crossplane is fully ready it can still occur. We do not uninstall Crossplane, we install it once, then install the providers and functions. At this point, the crossplane-root-ca is populated by Crossplane’s init container, so the certificate should be properly signed.

Cleaning the secrets and rolling out the pods does not necessarily solve this issue, sometime it does, sometime it doesn’t; the proper order seems yet unclear. Could this be an issue with having multiple Crossplane replicas that don’t sync correctly during the init?

@jbw976 Just spitballing after hitting the same issue but is it possible that the fix implemented only works if the functions are passed to the helm chart using the parameters in the values file but not if they are deployed otherwise?

@jbw976 In our case we managed to fix this issue by deleting all the Crossplane secrets with the tls-server or tls-client suffix, we’ve seen in N times already, the biggest issue is that it happens randomly. We have almost the same situation as previously mentioned, we’re dynamically creating k8s clusters.

The issue also touches function and provider pods. I’ve observed that the IPs from the error message all point to the konnectivity-agent pods in kube-system namespace, it’s an IBM cluster (IKS, k8s v1.31.2).

We’re investigating this issue right now, we’ll get back with more info if we find anything, maybe it’s just issue on IBM side

Same issue encountered with the v1.18.2, suddenly our compositions stopped being reconciled (no status set on XRC) and there’s a lot of events:

50s                        Warning   ComposeResources                  REDACTED/redacted                                               cannot compose resources: cannot run Composition pipeline step "k8s_provider_config": cannot run Function "crossplane-contrib-function-go-templating": rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of \"crypto/rsa: verification error\" while trying to verify candidate authority certificate \"Crossplane\")"

EDIT: it may be related to the fact that we have some TLS errors in the crossplane pod:

{"level":"info","ts":"2025-01-14T11:51:17Z","logger":"crossplane","msg":"Beta feature enabled","flag":"EnableBetaCompositionWebhookSchemaValidation"}
{"level":"info","ts":"2025-01-14T11:51:17Z","logger":"crossplane","msg":"Alpha feature enabled","flag":"EnableAlphaUsages"}
{"level":"info","ts":"2025-01-14T11:51:17Z","logger":"crossplane","msg":"Beta feature enabled","flag":"EnableBetaDeploymentRuntimeConfigs"}
2025/01/14 11:51:19 http: TLS handshake error from 172.30.255.129:41964: EOF
Warning: ControllerConfig.pkg.crossplane.io/v1alpha1 is deprecated. Use DeploymentRuntimeConfig from pkg.crossplane.io/v1beta1 instead.
2025/01/14 11:51:28 http: TLS handshake error from 172.30.200.174:36792: EOF

Looking at this and I am wondering if we are getting an incorrect NotFound here from the controller runtime cache where we try to get the crossplane-root-ca secret and regenerate the ca secret leaving previously generated function secrets out of sync.

I couldn’t reproduce it yet, tried with multiple replicas, low cpu etc, but I wouldn’t be surprised if 👆 is the root cause. If/once we could validate this (by somehow slowing down the controller runtime cache), fix should be straightforward I believe.