keycloak: Keycloak operator tries to manipulate Secret which is not managed by Keycloak

Before reporting an issue

I have searched existing issues
I have reproduced the issue with the latest nightly release

Area

operator

Describe the bug

In the Keycloak operator v22.0.5 logs errors are reported about the Secret which is not managed neither referenced in the Keycloak CR.

Operator reports attempts to manipulate Secret keycloak-bindinfo-cs-keycloak-tls-secret, even though such Secret is not referenced in the Keycloak CR. However, subject Secret indeed exists in the same namespace, but it’s owned by other operator running in the same namespace.

There might be some name collision or wildcard name matching* since Secret cs-keycloak-tls-secret is actually used in the Keycloak CR and it contains the TLS certificate.

Yet, the Secret keycloak-bindinfo-cs-keycloak-tls-secret (which has the same suffix in the name) is not listed in Keycloak CR and is being managed by other operator (actually it is the exact copy of the cs-keycloak-tls-secret, but used for different purposes).

apiVersion: k8s.keycloak.org/v2alpha1
kind: Keycloak
metadata:
  name: cs-keycloak
  namespace: <redacted>
  labels:
    operator.ibm.com/opreq-control: 'true'
spec:
  db:
    host: keycloak-edb-cluster-rw
    passwordSecret:
      key: password
      name: keycloak-edb-cluster-app
    usernameSecret:
      key: username
      name: keycloak-edb-cluster-app
    vendor: postgres
  hostname:
    strict: false
  http:
    tlsSecret: cs-keycloak-tls-secret      <--- HERE
  ingress:
    className: openshift-default
    enabled: true
  instances: 1
  unsupported:
    podTemplate:
      spec:
        containers:
          - resources:
              limits:
                cpu: 1000m
                memory: 1Gi
              requests:
                cpu: 1000m
                memory: 1Gi
status:
  conditions:
    - lastTransitionTime: '2023-10-27T12:39:27.282365273Z'
      message: ''
      observedGeneration: 3
      status: 'True'
      type: Ready
    - lastTransitionTime: '2023-10-19T09:48:42.726396161Z'
      message: ''
      observedGeneration: 3
      status: 'False'
      type: HasErrors
    - lastTransitionTime: '2023-10-27T12:38:40.397258184Z'
      message: ''
      observedGeneration: 3
      status: 'False'
      type: RollingUpdate
  instances: 1
  observedGeneration: 3
  selector: >-
    app=keycloak,app.kubernetes.io/managed-by=keycloak-operator,app.kubernetes.io/instance=cs-keycloak

Secret cs-keycloak-tls-secret:

kind: Secret
apiVersion: v1
metadata:
  name: cs-keycloak-tls-secret
  namespace: <redacted>
  uid: f4ef6450-76f9-42bb-b701-13d0fee2903f
  resourceVersion: '70485'
  creationTimestamp: '2023-10-19T09:46:13Z'
  labels:
    controller.cert-manager.io/fao: 'true'
    operator.ibm.com/managedBy-opbi: original
    operator.ibm.com/watched-by-cert-manager: ''
    operator.keycloak.org/component: watched-secret
    rhbk-operator-2205-rc.keycloak-bindinfo/bindinfo: 'true'
  annotations:
    cert-manager.io/alt-names: >-
      cs-keycloak-service,cs-keycloak-service.rhbk-operator-2205-rc,cs-keycloak-service.rhbk-operator-2205-rc.svc,cs-keycloak-service.rhbk-operator-2205-rc.svc.cluster.local
    cert-manager.io/certificate-name: cs-keycloak-tls-cert
    cert-manager.io/common-name: cs-keycloak-service
    cert-manager.io/ip-sans: ''
    cert-manager.io/issuer-group: ''
    cert-manager.io/issuer-kind: Issuer
    cert-manager.io/issuer-name: cs-ca-issuer
    cert-manager.io/uri-sans: ''
  managedFields:
    - manager: cert-manager-certificates-issuing
      operation: Apply
      apiVersion: v1
      time: '2023-10-19T09:46:13Z'
      fieldsType: FieldsV1
      fieldsV1:
        'f:data':
          'f:ca.crt': {}
          'f:tls.crt': {}
          'f:tls.key': {}
        'f:metadata':
          'f:annotations':
            'f:cert-manager.io/alt-names': {}
            'f:cert-manager.io/certificate-name': {}
            'f:cert-manager.io/common-name': {}
            'f:cert-manager.io/ip-sans': {}
            'f:cert-manager.io/issuer-group': {}
            'f:cert-manager.io/issuer-kind': {}
            'f:cert-manager.io/issuer-name': {}
            'f:cert-manager.io/uri-sans': {}
          'f:labels':
            'f:controller.cert-manager.io/fao': {}
        'f:type': {}
    - manager: fabric8-kubernetes-client
      operation: Update
      apiVersion: v1
      time: '2023-10-19T09:46:29Z'
      fieldsType: FieldsV1
      fieldsV1:
        'f:metadata':
          'f:labels':
            'f:operator.keycloak.org/component': {}
    - manager: manager
      operation: Update
      apiVersion: v1
      time: '2023-10-19T09:46:30Z'
      fieldsType: FieldsV1
      fieldsV1:
        'f:metadata':
          'f:labels':
            'f:operator.ibm.com/managedBy-opbi': {}
            'f:operator.ibm.com/watched-by-cert-manager': {}
            'f:rhbk-operator-2205-rc.keycloak-bindinfo/bindinfo': {}
data:
  ca.crt: >-
           <redacted>
  tls.crt: >-
           <redacted>
  tls.key: >-
           <redacted>
type: kubernetes.io/tls

Secret `keycloak-bindinfo-cs-keycloak-tls-secret`

kind: Secret
apiVersion: v1
metadata:
  name: keycloak-bindinfo-cs-keycloak-tls-secret
  namespace: <redacted>
  uid: 7c25d3a0-7df5-453d-889e-a5ecaf9c466d
  resourceVersion: '5248821'
  creationTimestamp: '2023-10-19T09:46:30Z'
  labels:
    controller.cert-manager.io/fao: 'true'
    operator.ibm.com/managedBy-opbi: copy
    operator.ibm.com/watched-by-cert-manager: ''
    rhbk-operator-2205-rc.keycloak-bindinfo/bindinfo: 'true'
  ownerReferences:
    - apiVersion: operator.ibm.com/v1alpha1
      kind: OperandRequest
      name: fixed-single-ar-cp4inav-rhbk-operator-2205-rc-keycloak
      uid: bc68b51e-f35f-40b3-8cea-acb75ec1fb37
      controller: true
      blockOwnerDeletion: true
  managedFields:
    - manager: manager
      operation: Update
      apiVersion: v1
      time: '2023-10-27T11:25:00Z'
      fieldsType: FieldsV1
      fieldsV1:
        'f:data':
          .: {}
          'f:ca.crt': {}
          'f:tls.crt': {}
          'f:tls.key': {}
        'f:metadata':
          'f:labels':
            .: {}
            'f:controller.cert-manager.io/fao': {}
            'f:operator.ibm.com/managedBy-opbi': {}
            'f:operator.ibm.com/watched-by-cert-manager': {}
            'f:rhbk-operator-2205-rc.keycloak-bindinfo/bindinfo': {}
          'f:ownerReferences':
            .: {}
            'k:{"uid":"bc68b51e-f35f-40b3-8cea-acb75ec1fb37"}': {}
        'f:type': {}
data:
  ca.crt: >-
       <redacted>
  tls.crt: >-
    <redacted>
  tls.key: >-
    <redacted>
type: kubernetes.io/tls

Complete stack trace:

2023-10-26 16:37:49,261 INFO  [org.key.ope.con.KeycloakController] (ReconcilerExecutor-keycloakcontroller-75) --- Reconciliation finished successfully
2023-10-26 17:21:08,697 INFO  [org.key.ope.con.WatchedSecretsStore] (-1618489605-pool-9-thread-4119) No CRs watching "keycloak-bindinfo-cs-keycloak-tls-secret" Secret, cleaning up labels
2023-10-26 17:21:08,721 INFO  [org.key.ope.con.WatchedSecretsStore] (-1618489605-pool-9-thread-4119) No CRs watching "keycloak-bindinfo-cs-keycloak-tls-secret" Secret, cleaning up labels
2023-10-26 17:21:08,746 INFO  [org.key.ope.con.WatchedSecretsStore] (-1618489605-pool-9-thread-4118) No CRs watching "keycloak-bindinfo-cs-keycloak-tls-secret" Secret, cleaning up labels
2023-10-26 17:21:08,767 INFO  [org.key.ope.con.WatchedSecretsStore] (-1618489605-pool-9-thread-4118) No CRs watching "keycloak-bindinfo-cs-keycloak-tls-secret" Secret, cleaning up labels
2023-10-26 17:21:08,784 ERROR [io.fab.kub.cli.inf.imp.cac.SharedProcessor] (-1618489605-pool-9-thread-4118) v1/namespaces/rhbk-operator-2205-rc/secrets failed invoking InformerEventSource{resourceClass: Secret} event handler: Failure executing: PATCH at: https://172.30.0.1:443/api/v1/namespaces/rhbk-operator-2205-rc/secrets/keycloak-bindinfo-cs-keycloak-tls-secret. Message: Operation cannot be fulfilled on secrets "keycloak-bindinfo-cs-keycloak-tls-secret": the object has been modified; please apply your changes to the latest version and try again. Received status: Status(apiVersion=v1, code=409, details=StatusDetails(causes=[], group=null, kind=secrets, name=keycloak-bindinfo-cs-keycloak-tls-secret, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=Operation cannot be fulfilled on secrets "keycloak-bindinfo-cs-keycloak-tls-secret": the object has been modified; please apply your changes to the latest version and try again, metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=Conflict, status=Failure, additionalProperties={}).: io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: PATCH at: https://172.30.0.1:443/api/v1/namespaces/rhbk-operator-2205-rc/secrets/keycloak-bindinfo-cs-keycloak-tls-secret. Message: Operation cannot be fulfilled on secrets "keycloak-bindinfo-cs-keycloak-tls-secret": the object has been modified; please apply your changes to the latest version and try again. Received status: Status(apiVersion=v1, code=409, details=StatusDetails(causes=[], group=null, kind=secrets, name=keycloak-bindinfo-cs-keycloak-tls-secret, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=Operation cannot be fulfilled on secrets "keycloak-bindinfo-cs-keycloak-tls-secret": the object has been modified; please apply your changes to the latest version and try again, metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=Conflict, status=Failure, additionalProperties={}).
	at io.fabric8.kubernetes.client.KubernetesClientException.copyAsCause(KubernetesClientException.java:238)
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.waitForResult(OperationSupport.java:518)
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handleResponse(OperationSupport.java:535)
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handlePatch(OperationSupport.java:430)
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handlePatch(OperationSupport.java:408)
	at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.handlePatch(BaseOperation.java:713)
	at io.fabric8.kubernetes.client.dsl.internal.HasMetadataOperation.lambda$patch$2(HasMetadataOperation.java:232)
	at io.fabric8.kubernetes.client.dsl.internal.HasMetadataOperation.patch(HasMetadataOperation.java:237)
	at io.fabric8.kubernetes.client.dsl.internal.HasMetadataOperation.edit(HasMetadataOperation.java:66)
	at io.fabric8.kubernetes.client.dsl.internal.HasMetadataOperation.edit(HasMetadataOperation.java:45)
	at org.keycloak.operator.controllers.WatchedSecretsStore.cleanObsoleteLabelFromSecret(WatchedSecretsStore.java:181)
	at org.keycloak.operator.controllers.WatchedSecretsStore.lambda$getWatchedSecretsEventSource$8(WatchedSecretsStore.java:210)
	at io.javaoperatorsdk.operator.processing.event.source.informer.InformerEventSource.propagateEvent(InformerEventSource.java:196)
	at io.javaoperatorsdk.operator.processing.event.source.informer.InformerEventSource.onDelete(InformerEventSource.java:150)
	at io.javaoperatorsdk.operator.processing.event.source.informer.InformerEventSource.onDelete(InformerEventSource.java:73)
	at io.fabric8.kubernetes.client.informers.impl.cache.ProcessorListener$DeleteNotification.handle(ProcessorListener.java:122)
	at io.fabric8.kubernetes.client.informers.impl.cache.ProcessorListener.add(ProcessorListener.java:50)
	at io.fabric8.kubernetes.client.informers.impl.cache.SharedProcessor.lambda$distribute$0(SharedProcessor.java:91)
	at io.fabric8.kubernetes.client.informers.impl.cache.SharedProcessor.lambda$distribute$1(SharedProcessor.java:114)
	at io.fabric8.kubernetes.client.utils.internal.SerialExecutor.lambda$execute$0(SerialExecutor.java:58)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: PATCH at: https://172.30.0.1:443/api/v1/namespaces/rhbk-operator-2205-rc/secrets/keycloak-bindinfo-cs-keycloak-tls-secret. Message: Operation cannot be fulfilled on secrets "keycloak-bindinfo-cs-keycloak-tls-secret": the object has been modified; please apply your changes to the latest version and try again. Received status: Status(apiVersion=v1, code=409, details=StatusDetails(causes=[], group=null, kind=secrets, name=keycloak-bindinfo-cs-keycloak-tls-secret, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=Operation cannot be fulfilled on secrets "keycloak-bindinfo-cs-keycloak-tls-secret": the object has been modified; please apply your changes to the latest version and try again, metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=Conflict, status=Failure, additionalProperties={}).
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.requestFailure(OperationSupport.java:671)
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.requestFailure(OperationSupport.java:651)
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.assertResponseCode(OperationSupport.java:600)
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.lambda$handleResponse$0(OperationSupport.java:560)
	at java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:646)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
	at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
	at io.fabric8.kubernetes.client.http.StandardHttpClient.lambda$completeOrCancel$10(StandardHttpClient.java:140)
	at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
	at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
	at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
	at io.fabric8.kubernetes.client.http.ByteArrayBodyHandler.onBodyDone(ByteArrayBodyHandler.java:52)
	at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
	at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
	at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
	at io.fabric8.kubernetes.client.vertx.VertxHttpRequest.lambda$consumeBytes$1(VertxHttpRequest.java:122)
	at io.vertx.core.impl.ContextInternal.dispatch(ContextInternal.java:264)
	at io.vertx.core.impl.ContextInternal.dispatch(ContextInternal.java:246)
	at io.vertx.core.http.impl.HttpEventHandler.handleEnd(HttpEventHandler.java:76)
	at io.vertx.core.http.impl.HttpClientResponseImpl.handleEnd(HttpClientResponseImpl.java:250)
	at io.vertx.core.http.impl.Http1xClientConnection$StreamImpl.lambda$new$0(Http1xClientConnection.java:444)
	at io.vertx.core.streams.impl.InboundBuffer.handleEvent(InboundBuffer.java:255)
	at io.vertx.core.streams.impl.InboundBuffer.write(InboundBuffer.java:134)
	at io.vertx.core.http.impl.Http1xClientConnection$StreamImpl.handleEnd(Http1xClientConnection.java:708)
	at io.vertx.core.impl.EventLoopContext.lambda$execute$2(EventLoopContext.java:78)
	at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174)
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167)
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:569)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	... 1 more

2023-10-26 17:21:08,794 INFO  [org.key.ope.con.WatchedSecretsStore] (-1618489605-pool-9-thread-4119) No CRs watching "keycloak-bindinfo-cs-keycloak-tls-secret" Secret, cleaning up labels
2023-10-26 17:21:08,814 INFO  [org.key.ope.con.WatchedSecretsStore] (-1618489605-pool-9-thread-4119) No CRs watching "keycloak-bindinfo-cs-keycloak-tls-secret" Secret, cleaning up labels
2023-10-26 17:21:08,827 INFO  [org.key.ope.con.WatchedSecretsStore] (-1618489605-pool-9-thread-4118) No CRs watching "keycloak-bindinfo-cs-keycloak-tls-secret" Secret, cleaning up labels
2023-10-26 17:21:08,841 INFO  [org.key.ope.con.WatchedSecretsStore] (-1618489605-pool-9-thread-4118) No CRs watching "keycloak-bindinfo-cs-keycloak-tls-secret" Secret, cleaning up labels
2023-10-26 17:21:08,855 INFO  [org.key.ope.con.WatchedSecretsStore] (-1618489605-pool-9-thread-4119) No CRs watching "keycloak-bindinfo-cs-keycloak-tls-secret" Secret, cleaning up labels
2023-10-26 17:21:08,870 INFO  [org.key.ope.con.WatchedSecretsStore] (-1618489605-pool-9-thread-4119) No CRs watching "keycloak-bindinfo-cs-keycloak-tls-secret" Secret, cleaning up labels
2023-10-26 17:21:08,884 INFO  [org.key.ope.con.WatchedSecretsStore] (-1618489605-pool-9-thread-4118) No CRs watching "keycloak-bindinfo-cs-keycloak-tls-secret" Secret, cleaning up labels
2023-10-26 17:21:08,900 INFO  [org.key.ope.con.WatchedSecretsStore] (-1618489605-pool-9-thread-4118) No CRs watching "keycloak-bindinfo-cs-keycloak-tls-secret" Secret, cleaning up labels
2023-10-27 02:23:04,586 INFO  [org.key.ope.con.WatchedSecretsStore] (-1618489605-pool-9-thread-4326) No CRs watching "keycloak-bindinfo-cs-keycloak-tls-secret" Secret, cleaning up labels
2023-10-27 02:23:04,618 INFO  [org.key.ope.con.WatchedSecretsStore] (-1618489605-pool-9-thread-4326) No CRs watching "keycloak-bindinfo-cs-keycloak-tls-secret" Secret, cleaning up labels
2023-10-27 02:23:04,640 ERROR [io.fab.kub.cli.inf.imp.cac.SharedProcessor] (-1618489605-pool-9-thread-4326) v1/namespaces/rhbk-operator-2205-rc/secrets failed invoking InformerEventSource{resourceClass: Secret} event handler: Failure executing: PATCH at: https://172.30.0.1:443/api/v1/namespaces/rhbk-operator-2205-rc/secrets/keycloak-bindinfo-cs-keycloak-tls-secret. Message: Operation cannot be fulfilled on secrets "keycloak-bindinfo-cs-keycloak-tls-secret": the object has been modified; please apply your changes to the latest version and try again. Received status: Status(apiVersion=v1, code=409, details=StatusDetails(causes=[], group=null, kind=secrets, name=keycloak-bindinfo-cs-keycloak-tls-secret, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=Operation cannot be fulfilled on secrets "keycloak-bindinfo-cs-keycloak-tls-secret": the object has been modified; please apply your changes to the latest version and try again, metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=Conflict, status=Failure, additionalProperties={}).: io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: PATCH at: https://172.30.0.1:443/api/v1/namespaces/rhbk-operator-2205-rc/secrets/keycloak-bindinfo-cs-keycloak-tls-secret. Message: Operation cannot be fulfilled on secrets "keycloak-bindinfo-cs-keycloak-tls-secret": the object has been modified; please apply your changes to the latest version and try again. Received status: Status(apiVersion=v1, code=409, details=StatusDetails(causes=[], group=null, kind=secrets, name=keycloak-bindinfo-cs-keycloak-tls-secret, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=Operation cannot be fulfilled on secrets "keycloak-bindinfo-cs-keycloak-tls-secret": the object has been modified; please apply your changes to the latest version and try again, metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=Conflict, status=Failure, additionalProperties={}).
	at io.fabric8.kubernetes.client.KubernetesClientException.copyAsCause(KubernetesClientException.java:238)
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.waitForResult(OperationSupport.java:518)
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handleResponse(OperationSupport.java:535)
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handlePatch(OperationSupport.java:430)
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handlePatch(OperationSupport.java:408)
	at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.handlePatch(BaseOperation.java:713)
	at io.fabric8.kubernetes.client.dsl.internal.HasMetadataOperation.lambda$patch$2(HasMetadataOperation.java:232)
	at io.fabric8.kubernetes.client.dsl.internal.HasMetadataOperation.patch(HasMetadataOperation.java:237)
	at io.fabric8.kubernetes.client.dsl.internal.HasMetadataOperation.edit(HasMetadataOperation.java:66)
	at io.fabric8.kubernetes.client.dsl.internal.HasMetadataOperation.edit(HasMetadataOperation.java:45)
	at org.keycloak.operator.controllers.WatchedSecretsStore.cleanObsoleteLabelFromSecret(WatchedSecretsStore.java:181)
	at org.keycloak.operator.controllers.WatchedSecretsStore.lambda$getWatchedSecretsEventSource$8(WatchedSecretsStore.java:210)
	at io.javaoperatorsdk.operator.processing.event.source.informer.InformerEventSource.propagateEvent(InformerEventSource.java:196)
	at io.javaoperatorsdk.operator.processing.event.source.informer.InformerEventSource.onAddOrUpdate(InformerEventSource.java:175)
	at io.javaoperatorsdk.operator.processing.event.source.informer.InformerEventSource.onAdd(InformerEventSource.java:121)
	at io.javaoperatorsdk.operator.processing.event.source.informer.InformerEventSource.onAdd(InformerEventSource.java:73)
	at io.fabric8.kubernetes.client.informers.impl.cache.ProcessorListener$AddNotification.handle(ProcessorListener.java:103)
	at io.fabric8.kubernetes.client.informers.impl.cache.ProcessorListener.add(ProcessorListener.java:50)
	at io.fabric8.kubernetes.client.informers.impl.cache.SharedProcessor.lambda$distribute$0(SharedProcessor.java:91)
	at io.fabric8.kubernetes.client.informers.impl.cache.SharedProcessor.lambda$distribute$1(SharedProcessor.java:114)
	at io.fabric8.kubernetes.client.utils.internal.SerialExecutor.lambda$execute$0(SerialExecutor.java:58)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: PATCH at: https://172.30.0.1:443/api/v1/namespaces/rhbk-operator-2205-rc/secrets/keycloak-bindinfo-cs-keycloak-tls-secret. Message: Operation cannot be fulfilled on secrets "keycloak-bindinfo-cs-keycloak-tls-secret": the object has been modified; please apply your changes to the latest version and try again. Received status: Status(apiVersion=v1, code=409, details=StatusDetails(causes=[], group=null, kind=secrets, name=keycloak-bindinfo-cs-keycloak-tls-secret, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=Operation cannot be fulfilled on secrets "keycloak-bindinfo-cs-keycloak-tls-secret": the object has been modified; please apply your changes to the latest version and try again, metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=Conflict, status=Failure, additionalProperties={}).
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.requestFailure(OperationSupport.java:671)
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.requestFailure(OperationSupport.java:651)
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.assertResponseCode(OperationSupport.java:600)
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.lambda$handleResponse$0(OperationSupport.java:560)
	at java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:646)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
	at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
	at io.fabric8.kubernetes.client.http.StandardHttpClient.lambda$completeOrCancel$10(StandardHttpClient.java:140)
	at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
	at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
	at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
	at io.fabric8.kubernetes.client.http.ByteArrayBodyHandler.onBodyDone(ByteArrayBodyHandler.java:52)
	at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
	at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
	at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
	at io.fabric8.kubernetes.client.vertx.VertxHttpRequest.lambda$consumeBytes$1(VertxHttpRequest.java:122)
	at io.vertx.core.impl.ContextInternal.dispatch(ContextInternal.java:264)
	at io.vertx.core.impl.ContextInternal.dispatch(ContextInternal.java:246)
	at io.vertx.core.http.impl.HttpEventHandler.handleEnd(HttpEventHandler.java:76)
	at io.vertx.core.http.impl.HttpClientResponseImpl.handleEnd(HttpClientResponseImpl.java:250)
	at io.vertx.core.http.impl.Http1xClientConnection$StreamImpl.lambda$new$0(Http1xClientConnection.java:444)
	at io.vertx.core.streams.impl.InboundBuffer.handleEvent(InboundBuffer.java:255)
	at io.vertx.core.streams.impl.InboundBuffer.write(InboundBuffer.java:134)
	at io.vertx.core.http.impl.Http1xClientConnection$StreamImpl.handleEnd(Http1xClientConnection.java:708)
	at io.vertx.core.impl.EventLoopContext.execute(EventLoopContext.java:76)
	at io.vertx.core.impl.ContextBase.execute(ContextBase.java:232)
	at io.vertx.core.http.impl.Http1xClientConnection.handleResponseEnd(Http1xClientConnection.java:945)
	at io.vertx.core.http.impl.Http1xClientConnection.handleHttpMessage(Http1xClientConnection.java:814)
	at io.vertx.core.http.impl.Http1xClientConnection.handleMessage(Http1xClientConnection.java:778)
	at io.vertx.core.net.impl.ConnectionBase.read(ConnectionBase.java:158)
	at io.vertx.core.net.impl.VertxHandler.channelRead(VertxHandler.java:153)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
	at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1383)
	at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1246)
	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1295)
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529)
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	... 1 more

2023-10-27 02:23:04,652 INFO  [org.key.ope.con.WatchedSecretsStore] (-1618489605-pool-9-thread-4325) No CRs watching "keycloak-bindinfo-cs-keycloak-tls-secret" Secret, cleaning up labels
2023-10-27 02:23:04,681 INFO  [org.key.ope.con.WatchedSecretsStore] (-1618489605-pool-9-thread-4325) No CRs watching "keycloak-bindinfo-cs-keycloak-tls-secret" Secret, cleaning up labels
2023-10-27 02:23:04,705 INFO  [org.key.ope.con.WatchedSecretsStore] (-1618489605-pool-9-thread-4326) No CRs watching "keycloak-bindinfo-cs-keycloak-tls-secret" Secret, cleaning up labels
2023-10-27 02:23:04,732 INFO  [org.key.ope.con.WatchedSecretsStore] (-1618489605-pool-9-thread-4326) No CRs watching "keycloak-bindinfo-cs-keycloak-tls-secret" Secret, cleaning up labels
2023-10-27 02:23:04,754 INFO  [org.key.ope.con.WatchedSecretsStore] (-1618489605-pool-9-thread-4325) No CRs watching "keycloak-bindinfo-cs-keycloak-tls-secret" Secret, cleaning up labels
2023-10-27 02:23:04,779 INFO  [org.key.ope.con.WatchedSecretsStore] (-1618489605-pool-9-thread-4325) No CRs watching "keycloak-bindinfo-cs-keycloak-tls-secret" Secret, cleaning up labels

Version

22.0.5

Expected behavior

No errors in the Keycloak operator Pod

Actual behavior

Observed errors in the Keycloak operator Pod, related to the failed attempts of update on the K8s Secret owned by some other operator.

How to Reproduce?

Use the provided Keycloak CR

Anything else?

No response

About this issue

Original URL
State: open
Created 8 months ago
Reactions: 1
Comments: 31 (18 by maintainers)

Commits related to this issue

test: ensuring we won't modify a secret for which we don't own the label closes #24353 Signed-off-by: Steve Hawkins <shawkins@redhat.com> — committed to shawkins/keycloak by shawkins 7 months ago
test: ensuring we won't modify a secret for which we don't own the label closes #24353 Signed-off-by: Steve Hawkins <shawkins@redhat.com> — committed to shawkins/keycloak by shawkins 7 months ago

Most upvoted comments

@shawkins what would be the purpose of reverse mapping? Should not the keycloak reference the secret?

It does, but basically requires reprocessing the entire CR to determine them, so instead we’ve added a simpler lookup on our statefulsets. I took https://github.com/keycloak/keycloak/pull/25338 a step further to use the cache from statefulset dependentresource for statefulset lookups. I suppose we could take that another step further and use an index. For now though the itemStore behavior needs some thought - I’ve added a couple of workarounds for the operator sdk and quarkus extension issue.

shawkins on Dec 7, 2023

I need to make an addendum - due to the Kubernetes SSA issue with handling secret data vs stringdata future versions of the JOSDK will not by default use SSA for Secrets. Since this logic path doesn’t touch the data, we should be able to specifically enable SSA - cc @csviri

@pgodowski can you try your scenario against main and confirm that the event loop issue is solved? If so we can leave this or a new issue open as a reminder to enable Secret SSA later.

It also sounds like we’ll simply leave the labeling process as is, and won’t offer an opt-out of watching at this time.

shawkins on Nov 17, 2023

The fields on the copied Secrets are still marked as owned by the Operator, aren’t they?

That really comes down to how the managed fields are handled by the other operator. In theory it could go out of its way to make the copy reflect the original, but if not then it will assume ownership of everything in the copied secret.

shawkins on Nov 7, 2023