spring-cloud-kubernetes: spring cloud kubernetes configuration watcher does not support ConfigMaps and Secrets from multiple namespaces

Describe the bug Using springcloud/spring-cloud-kubernetes-configuration-watcher:3.0.1-SNAPSHOT

The documentation is very unclear about using the configuration watcher. Especially about supporting multiple namespaces. The closest option I thought that should work is spring.cloud.kubernetes.reload.namespaces but this one is said as This functionality has been deprecated in the 2020.0 release. Please see the [Spring Cloud Kubernetes Configuration Watcher](https://docs.spring.io/spring-cloud-kubernetes/docs/3.0.1-SNAPSHOT/reference/html/#spring-cloud-kubernetes-configuration-watcher) controller for an alternative way to achieve the same functionality. So actually I don’t think I should be be using it, and use another parameter but no idea which one. anyways, let’s describe the bug with spring.cloud.kubernetes.reload.namespaces. When you provide a list of namespaces, only the first one in the list is taken into account.

Sample

SENARIO 1: I specify this list: namespace1,namespace2

I am using this deployment to deploy the watcher in the cw NS, and so it watches configmaps from 2 NS: namespace1 and namespace2

apiVersion: v1
kind: List
items:
  - apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: configuration-watcher-deployment
      namespace: cw
    spec:
      selector:
        matchLabels:
          app: configuration-watcher
      template:
        metadata:
          labels:
            app: configuration-watcher
        spec:
          serviceAccount: configuration-watcher-service
          containers:
            - name: configuration-watcher-container
              image: springcloud/spring-cloud-kubernetes-configuration-watcher:3.0.1-SNAPSHOT
              imagePullPolicy: IfNotPresent
              readinessProbe:
                httpGet:
                  port: 8888
                  path: /actuator/health/readiness
              livenessProbe:
                httpGet:
                  port: 8888
                  path: /actuator/health/liveness
              ports:
                - containerPort: 8888
              env:
                - name: logging.level.org.springframework.cloud.kubernetes
                  value: "TRACE"
                - name: spring.cloud.kubernetes.discovery.all-namespaces
                  value: "true"
                - name: spring.cloud.kubernetes.reload.enabled
                  value: "true"
                - name: spring.cloud.kubernetes.reload.namespaces
                  value: "namespace1,namespace2"

here is the 2 config maps:

apiVersion: v1
kind: ConfigMap
metadata:
  name: cm1
  namespace: namespace1
  labels:
    spring.cloud.kubernetes.config: "true"
data:
  application.properties: |-
    bean.message=Hello World from NS1!
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: cm2
  namespace: namespace2
  labels:
    spring.cloud.kubernetes.config: "true"
data:
  application.properties: |-
    bean.message=Hello World from NS2!

and how to create the SA and CRB

---
apiVersion: v1
kind: List
items:
  - apiVersion: v1
    kind: ServiceAccount
    metadata:
      labels:
        app: configuration-watcher
      name: configuration-watcher-service
      namespace: cw

  - apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRoleBinding
    metadata:
      creationTimestamp: null
      name: admin-a
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: ClusterRole
      name: cluster-admin
    subjects:
      - kind: ServiceAccount
        name: configuration-watcher-service
        namespace: cw

and the 3 NS

apiVersion: v1
kind: Namespace
metadata:
  name: cw
---
apiVersion: v1
kind: Namespace
metadata:
  name: namespace1
---
apiVersion: v1
kind: Namespace
metadata:
  name: namespace2

I put those files on github for your convenience: https://github.com/webrod/spring-cloud-watcher

out of the logs I can see it recognizes my 2 NS :

023-01-13T13:12:24.523Z DEBUG 1 --- [           main] o.s.c.k.c.c.KubernetesClientConfigUtils  : informer namespaces : [ra-demo-app, namespace1]
2023-01-13T13:12:24.528Z  INFO 1 --- [           main] sClientEventBasedConfigMapChangeDetector : Kubernetes event-based configMap change detector activated
2023-01-13T13:12:24.530Z DEBUG 1 --- [           main] sClientEventBasedConfigMapChangeDetector : added configmap informer for namespace : namespace1
2023-01-13T13:12:24.533Z DEBUG 1 --- [           main] sClientEventBasedConfigMapChangeDetector : added configmap informer for namespace : namespace2
2023-01-13T13:12:24.543Z  INFO 1 --- [           main] o.s.c.k.client.KubernetesClientUtils     : Created API client in the cluster.
2023-01-13T13:12:24.544Z DEBUG 1 --- [           main] o.s.c.k.c.c.KubernetesClientConfigUtils  : informer namespaces : [namespace1, namespace2]
2023-01-13T13:12:24.548Z  INFO 1 --- [           main] tesClientEventBasedSecretsChangeDetector : Kubernetes event-based secrets change detector activated
2023-01-13T13:12:24.552Z DEBUG 1 --- [           main] tesClientEventBasedSecretsChangeDetector : added secret informer for namespace : namespace1
2023-01-13T13:12:24.556Z DEBUG 1 --- [           main] tesClientEventBasedSecretsChangeDetector : added secret informer for namespace : namespace2

I can see out of the logs it detects the CM named cm1 from Namespace1: 2023-01-13T13:12:24.623Z DEBUG 1 --- [ool-14-thread-1] sClientEventBasedConfigMapChangeDetector : ConfigMap cm1 was added. but not cm2 from Namespace2! I can confirm a manual change of cm1 is detected but not in cm2

SENARIO 2: I specify this list: namespace2,namespace1 basically NS2 is now first. I do this by updating the above deployment with this part:

                - name: spring.cloud.kubernetes.reload.namespaces
                  value: "namespace2,namespace1"

The logs confirm that both NS have been detected, and with a different order:

2023-01-13T13:32:37.489Z DEBUG 1 --- [           main] o.s.c.k.c.c.KubernetesClientConfigUtils  : informer namespaces : [namespace2, namespace1]
2023-01-13T13:32:37.492Z  INFO 1 --- [           main] sClientEventBasedConfigMapChangeDetector : Kubernetes event-based configMap change detector activated
2023-01-13T13:32:37.493Z DEBUG 1 --- [           main] sClientEventBasedConfigMapChangeDetector : added configmap informer for namespace : namespace2
2023-01-13T13:32:37.496Z DEBUG 1 --- [           main] sClientEventBasedConfigMapChangeDetector : added configmap informer for namespace : namespace1

but this time it shows cm2 from the log and NOT cm1: 2023-01-13T13:32:37.564Z DEBUG 1 --- [ool-14-thread-1] sClientEventBasedConfigMapChangeDetector : ConfigMap cm2 was added. I can confirm a manual change of cm2 is detected but not in cm1

SUMMARY: It sounds like only the first from the list is really watched. The configuration-watcher-service SA has the ClusterRole. so I don’t think it is a permission issue, and we can see that anyways depending on the order, each CM from any of the 2 NS individually work. so it has access to those NS. and my CM1 and CM2 have the expected label: spring.cloud.kubernetes.config: true

Note that I also tried with enable-reload-filtering: true and adding this label to the CMs spring.cloud.kubernetes.config.informer.enabled: true but I have the same behavior

any idea?