crossplane: Crossplane fails to synchronize claims with XRs

What happened?

From time to time I see that claims and XRs loose sync such as here (see second resource):

❯ kubectl get xirsas
NAME                                           API                           SYNCED   READY   COMPOSITION                                         AGE
acm-pca-issuer-nw-eu-west-3-main               infra.nexthink.com/v1alpha3   True     True    custom.policy.xirsas.infra.nexthink.com             9m30s
acm-pca-issuer-nw-us-east-2-main               infra.nexthink.com/v1alpha3                    custom.policy.xirsas.infra.nexthink.com             9m31s
collector-traffic-nw-eu-west-3-main            infra.nexthink.com/v1alpha3   True     True    custom.auth.xirsas.infra.nexthink.com               22h
[...]

And this does not change until I do a rollout restart of crossplane deployment. Logs in crossplane deployment look to be on a loop continuously prompting logs such these:

crossplane-7c898b5fdf-rf7mg universal-crossplane {"level":"info","ts":"2024-02-16T10:15:46Z","logger":"crossplane","msg":"Enqueueing composite resource because managed resource changed","controller":"defined/compositeresourcedefinition.apiextensions.crossplane.io","request":{"name":"xcertmanagers.infra.nexthink.com"},"uid":"c3eb8136-bcc4-4c98-8832-d782e62cb2b2","version":"168408426","name":"xcertmanagers.infra.nexthink.com","name":"cert-manager-nw-eu-west-3-main","mrGVK":"infra.nexthink.com/v1alpha1, Kind=XAcmPcaIssuer","mrName":"pca-plugin-nw-eu-west-3-main"}
crossplane-7c898b5fdf-rf7mg universal-crossplane {"level":"info","ts":"2024-02-16T10:15:46Z","logger":"crossplane","msg":"Enqueueing composite resource because managed resource changed","controller":"defined/compositeresourcedefinition.apiextensions.crossplane.io","request":{"name":"xcertmanagers.infra.nexthink.com"},"uid":"c3eb8136-bcc4-4c98-8832-d782e62cb2b2","version":"168408426","name":"xcertmanagers.infra.nexthink.com","name":"cert-manager-nw-eu-west-3-main","mrGVK":"infra.nexthink.com/v1alpha1, Kind=XAcmPcaIssuer","mrName":"pca-plugin-nw-eu-west-3-main"}
crossplane-7c898b5fdf-rf7mg universal-crossplane {"level":"info","ts":"2024-02-16T10:15:46Z","logger":"crossplane","msg":"Enqueueing composite resource because managed resource changed","controller":"defined/compositeresourcedefinition.apiextensions.crossplane.io","request":{"name":"xcertmanagers.infra.nexthink.com"},"uid":"c3eb8136-bcc4-4c98-8832-d782e62cb2b2","version":"168408426","name":"xcertmanagers.infra.nexthink.com","name":"cert-manager-nw-eu-west-3-main","mrGVK":"infra.nexthink.com/v1alpha1, Kind=XAcmPcaIssuer","mrName":"pca-plugin-nw-eu-west-3-main"}
crossplane-7c898b5fdf-rf7mg universal-crossplane {"level":"info","ts":"2024-02-16T10:15:46Z","logger":"crossplane","msg":"Enqueueing composite resource because managed resource changed","controller":"defined/compositeresourcedefinition.apiextensions.crossplane.io","request":{"name":"xcertmanagers.infra.nexthink.com"},"uid":"c3eb8136-bcc4-4c98-8832-d782e62cb2b2","version":"168408426","name":"xcertmanagers.infra.nexthink.com","name":"cert-manager-nw-eu-west-3-main","mrGVK":"infra.nexthink.com/v1alpha1, Kind=XAcmPcaIssuer","mrName":"pca-plugin-nw-eu-west-3-main"}

How can we reproduce it?

What environment did it happen in?

Crossplane version: universal-crossplane-1.14.5-up.1

EKS v1.27.9-eks-5e0fdde

Relevant PRs

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 19 (18 by maintainers)

Most upvoted comments

@negz yes, I can reproduce it:

add - --enable-realtime-compositions to crossplane deployment args

Install configuration-aws-network from the marketplace: https://marketplace.upbound.io/configurations/upbound/configuration-aws-network/v0.24.0

apiVersion: pkg.crossplane.io/v1
kind: Configuration
metadata:
  name: configuration-aws-network
spec:
  package: xpkg.upbound.io/upbound/configuration-aws-network:v0.24.0

Apply a sample XR: https://github.com/upbound/configuration-aws-network/blob/main/examples/xnetworks/configuration-aws-network.yaml:

apiVersion: aws.platform.upbound.io/v1alpha1
kind: XNetwork
metadata:
  name: configuration-aws-network
spec:
  parameters:
    id: configuration-aws-network
    region: us-west-2
    vpcCidrBlock: 192.168.0.0/16
    subnets:
      - availabilityZone: us-west-2a
        type: public
        cidrBlock: 192.168.0.0/18
      - availabilityZone: us-west-2b
        type: public
        cidrBlock: 192.168.64.0/18
      - availabilityZone: us-west-2a
        type: private
        cidrBlock: 192.168.128.0/18
      - availabilityZone: us-west-2b
        type: private
        cidrBlock: 192.168.192.0/18

The composite will not have any events on it:

$ kubectl get composite 
NAME                                                         SYNCED   READY   COMPOSITION   AGE
xnetwork.aws.platform.upbound.io/configuration-aws-network                                  16s
$ kubectl describe xnetwork.aws.platform.upbound.io/configuration-aws-network 
Name:         configuration-aws-network
Namespace:    
Labels:       <none>
Annotations:  <none>
API Version:  aws.platform.upbound.io/v1alpha1
Kind:         XNetwork
Metadata:
  Creation Timestamp:  2025-04-23T13:40:10Z
  Generation:          1
  Resource Version:    2135594
  UID:                 0a1886dc-39ac-4206-a3f1-875eaf6e1935
Spec:
  Composition Update Policy:  Automatic
  Parameters:
    Deletion Policy:       Delete
    Id:                    configuration-aws-network
    Provider Config Name:  default
    Region:                us-west-2
    Subnets:
      Availability Zone:  us-west-2a
      Cidr Block:         192.168.0.0/18
      Type:               public
      Availability Zone:  us-west-2b
      Cidr Block:         192.168.64.0/18
      Type:               public
      Availability Zone:  us-west-2a
      Cidr Block:         192.168.128.0/18
      Type:               private
      Availability Zone:  us-west-2b
      Cidr Block:         192.168.192.0/18
      Type:               private
    Vpc Cidr Block:       192.168.0.0/16
Events:                   <none>

/fresh