spring-cloud-kubernetes: In rare scenarios ConfigMap update is not updated using polling strategy

Describe the bug Intermittently once pod is deployed, Any configMap resource update is not getting applied to the Application We see in problematic pods, we continously see below WARN messages 10:12:24.807 WARN — [TaskScheduler-1] o.s.c.k.c.c.reload.ConfigReloadUtil : The current number of ConfigMap PropertySources does not match the ones loaded from Kubernetes - No reload will take place

This does not happen normally, but happens sometimes after which its not recoverable, we need to go for full deployment to overcome.No new configMap updates will be applied.

We see that ConfigMap resource is updated in K8s. we could verify using kubectl describe cm <name> But K8s spring cloud is not able to accept or update the newly updated resource.

Version Details [INFO] ± org.springframework.cloud:spring-cloud-kubernetes-commons:jar:3.0.3:compile [INFO] | ± org.springframework.boot:spring-boot-autoconfigure:jar:3.1.4:compile [INFO] | ± org.springframework.cloud:spring-cloud-commons:jar:4.0.3:compile [INFO] | | - org.springframework.security:spring-security-crypto:jar:6.1.4:compile [INFO] | - org.springframework.cloud:spring-cloud-context:jar:4.0.3:compile [INFO] ± org.springframework.cloud:spring-cloud-kubernetes-fabric8-autoconfig:jar:3.0.3:compile [INFO] ± com.ciscospark:cisco-spark-starter-server:jar:4.1.8:compile [INFO] | ± com.ciscospark:cisco-spark-autoconfigure:jar:4.1.8:compile

[INFO] ± io.fabric8:kubernetes-client:jar:6.2.0:compile [INFO] | ± io.fabric8:kubernetes-client-api:jar:6.2.0:compile [INFO] | | ± io.fabric8:kubernetes-model-core:jar:6.2.0:compile [INFO] | | | - io.fabric8:kubernetes-model-common:jar:6.2.0:compile

Temp workaround Delete the pod, or go for full deployment(image, configmap)

logs snippet when issue happpened

10:12:24.806 WARN — [TaskScheduler-1] s.c.k.c.c.ConfigMapPropertySourceLocator : path support is deprecated and will be removed in a future release. Please use spring.config.import 10:12:24.806 DEBUG — [TaskScheduler-1] s.c.k.c.c.ConfigMapPropertySourceLocator : paths property sources : [] 10:12:24.806 DEBUG — [TaskScheduler-1] o.s.c.k.c.c.reload.ConfigReloadUtil : environment from locateMapPropertySources : ApplicationServletEnvironment {activeProfiles=[], defaultProfiles=[default], propertySources=[ {name=‘spring-app-name’}, BootstrapPropertySource {name=‘bootstrapProperties-configmap.dhruva-proxy-dynamic-config-map.wxc-dhruva’}, BootstrapPropertySource {name=‘bootstrapProperties-configmap.dhruva-proxy-static-config-map.wxc-dhruva’}, ConfigurationPropertySourcesPropertySource {name=‘configurationProperties’}, StubPropertySource {name=‘servletConfigInitParams’}, ServletContextPropertySource {name=‘servletContextInitParams’}, JndiPropertySource {name=‘jndiProperties’}, PropertiesPropertySource {name=‘systemProperties’}, OriginAwareSystemEnvironmentPropertySource {name=‘systemEnvironment’}, RandomValuePropertySource {name=‘random’}, CachedRandomPropertySource {name=‘cachedrandom’}, MapPropertySource {name=‘springCloudClientHostInfo’}, OriginTrackedMapPropertySource {name=‘applicationConfig: [classpath:/bootstrap.yaml]’}, OriginTrackedMapPropertySource {name=‘Config resource ‘class path resource [bootstrap.yaml]’ via location ‘optional:classpath:/’’}, MapPropertySource {name=‘springCloudDefaultProperties’}, OriginTrackedMapPropertySource {name=‘core-metrics-properties.yaml’}, OriginTrackedMapPropertySource {name=‘common-identity-client.yaml’}, OriginTrackedMapPropertySource {name=‘service-health-properties.yaml’}, ResourcePropertySource {name=‘class path resource [git.properties]’}, OriginTrackedMapPropertySource {name=‘filter-properties.yaml’}, OriginTrackedMapPropertySource {name=‘diagnostics-service-properties.yaml’}, OriginTrackedMapPropertySource {name=‘common-service-properties.yaml’}, {name=‘Management Server’}]} 10:12:24.806 DEBUG — [TaskScheduler-1] o.s.c.k.c.c.reload.ConfigReloadUtil : sources from locateMapPropertySources : [Fabric8ConfigMapPropertySource@232678616 {name=‘configmap…wxc-dhruva’, properties={}}] 10:12:24.806 DEBUG — [TaskScheduler-1] o.s.c.k.c.c.reload.ConfigReloadUtil : left size: 1 10:12:24.806 DEBUG — [TaskScheduler-1] o.s.c.k.c.c.reload.ConfigReloadUtil : Fabric8ConfigMapPropertySource@232678616 {name=‘configmap…wxc-dhruva’, properties={}} 10:12:24.806 DEBUG — [TaskScheduler-1] o.s.c.k.c.c.reload.ConfigReloadUtil : right size: 2 10:12:24.807 DEBUG — [TaskScheduler-1] o.s.c.k.c.c.reload.ConfigReloadUtil : Fabric8ConfigMapPropertySource@794279295 {name=‘configmap.dhruva-proxy-dynamic-config-map.wxc-dhruva’, properties={ ####MASKED### SECURITY REASONS

10:12:24.807 WARN — [TaskScheduler-1] o.s.c.k.c.c.reload.ConfigReloadUtil : The current number of ConfigMap PropertySources does not match the ones loaded from Kubernetes - No reload will take place

Sample app config

spring:
  application:
    name: dhruvaProxyApplication
  cloud:
    consul:
      enabled: false
    kubernetes:
      config:
        include-profile-specific-sources: false
      reload:
        enabled: true
        mode: polling
        period: 10000 # event based missed some config changes, testing polling
      discovery:
        enabled: false

JAVA_OPTS -Dspring.cloud.kubernetes.config.namespace=wxc-dhruva -Dspring.cloud.kubernetes.config.sources[0].name=dhruva-proxy-static-config-map -Dspring.cloud.kubernetes.config.sources[1].name=dhruva-proxy-dynamic-config-map

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 25 (13 by maintainers)

Most upvoted comments

finally I had some time to put up a test to reproduce this issue, sorry it takes so much and I am so slow at this, this unfortunately is not my main job and I have little time these days…

Anyway, I did reproduce it via a test. It is indeed a bug on our side, I am positive about this. The core issue is still in the fact that when there is an error, we generate an incorrect name. The thing you need to get an error twice and some very unlucky timings.

I am not going to bother you with details, I need to think on how to solve this one now. 😃

I have a PR in motion here, working on it…

as usual, time is the limiting factor for me… I will be looking at this one, but no timeline when exactly.