spring-cloud-kubernetes: In rare scenarios ConfigMap update is not updated using polling strategy
Describe the bug Intermittently once pod is deployed, Any configMap resource update is not getting applied to the Application We see in problematic pods, we continously see below WARN messages 10:12:24.807 WARN — [TaskScheduler-1] o.s.c.k.c.c.reload.ConfigReloadUtil : The current number of ConfigMap PropertySources does not match the ones loaded from Kubernetes - No reload will take place
This does not happen normally, but happens sometimes after which its not recoverable, we need to go for full deployment to overcome.No new configMap updates will be applied.
We see that ConfigMap resource is updated in K8s. we could verify using kubectl describe cm <name> But K8s spring cloud is not able to accept or update the newly updated resource.
Version Details [INFO] ± org.springframework.cloud:spring-cloud-kubernetes-commons:jar:3.0.3:compile [INFO] | ± org.springframework.boot:spring-boot-autoconfigure:jar:3.1.4:compile [INFO] | ± org.springframework.cloud:spring-cloud-commons:jar:4.0.3:compile [INFO] | | - org.springframework.security:spring-security-crypto:jar:6.1.4:compile [INFO] | - org.springframework.cloud:spring-cloud-context:jar:4.0.3:compile [INFO] ± org.springframework.cloud:spring-cloud-kubernetes-fabric8-autoconfig:jar:3.0.3:compile [INFO] ± com.ciscospark:cisco-spark-starter-server:jar:4.1.8:compile [INFO] | ± com.ciscospark:cisco-spark-autoconfigure:jar:4.1.8:compile
[INFO] ± io.fabric8:kubernetes-client:jar:6.2.0:compile [INFO] | ± io.fabric8:kubernetes-client-api:jar:6.2.0:compile [INFO] | | ± io.fabric8:kubernetes-model-core:jar:6.2.0:compile [INFO] | | | - io.fabric8:kubernetes-model-common:jar:6.2.0:compile
Temp workaround Delete the pod, or go for full deployment(image, configmap)
logs snippet when issue happpened
10:12:24.806 WARN — [TaskScheduler-1] s.c.k.c.c.ConfigMapPropertySourceLocator : path support is deprecated and will be removed in a future release. Please use spring.config.import 10:12:24.806 DEBUG — [TaskScheduler-1] s.c.k.c.c.ConfigMapPropertySourceLocator : paths property sources : [] 10:12:24.806 DEBUG — [TaskScheduler-1] o.s.c.k.c.c.reload.ConfigReloadUtil : environment from locateMapPropertySources : ApplicationServletEnvironment {activeProfiles=[], defaultProfiles=[default], propertySources=[ {name=‘spring-app-name’}, BootstrapPropertySource {name=‘bootstrapProperties-configmap.dhruva-proxy-dynamic-config-map.wxc-dhruva’}, BootstrapPropertySource {name=‘bootstrapProperties-configmap.dhruva-proxy-static-config-map.wxc-dhruva’}, ConfigurationPropertySourcesPropertySource {name=‘configurationProperties’}, StubPropertySource {name=‘servletConfigInitParams’}, ServletContextPropertySource {name=‘servletContextInitParams’}, JndiPropertySource {name=‘jndiProperties’}, PropertiesPropertySource {name=‘systemProperties’}, OriginAwareSystemEnvironmentPropertySource {name=‘systemEnvironment’}, RandomValuePropertySource {name=‘random’}, CachedRandomPropertySource {name=‘cachedrandom’}, MapPropertySource {name=‘springCloudClientHostInfo’}, OriginTrackedMapPropertySource {name=‘applicationConfig: [classpath:/bootstrap.yaml]’}, OriginTrackedMapPropertySource {name=‘Config resource ‘class path resource [bootstrap.yaml]’ via location ‘optional:classpath:/’’}, MapPropertySource {name=‘springCloudDefaultProperties’}, OriginTrackedMapPropertySource {name=‘core-metrics-properties.yaml’}, OriginTrackedMapPropertySource {name=‘common-identity-client.yaml’}, OriginTrackedMapPropertySource {name=‘service-health-properties.yaml’}, ResourcePropertySource {name=‘class path resource [git.properties]’}, OriginTrackedMapPropertySource {name=‘filter-properties.yaml’}, OriginTrackedMapPropertySource {name=‘diagnostics-service-properties.yaml’}, OriginTrackedMapPropertySource {name=‘common-service-properties.yaml’}, {name=‘Management Server’}]} 10:12:24.806 DEBUG — [TaskScheduler-1] o.s.c.k.c.c.reload.ConfigReloadUtil : sources from locateMapPropertySources : [Fabric8ConfigMapPropertySource@232678616 {name=‘configmap…wxc-dhruva’, properties={}}] 10:12:24.806 DEBUG — [TaskScheduler-1] o.s.c.k.c.c.reload.ConfigReloadUtil : left size: 1 10:12:24.806 DEBUG — [TaskScheduler-1] o.s.c.k.c.c.reload.ConfigReloadUtil : Fabric8ConfigMapPropertySource@232678616 {name=‘configmap…wxc-dhruva’, properties={}} 10:12:24.806 DEBUG — [TaskScheduler-1] o.s.c.k.c.c.reload.ConfigReloadUtil : right size: 2 10:12:24.807 DEBUG — [TaskScheduler-1] o.s.c.k.c.c.reload.ConfigReloadUtil : Fabric8ConfigMapPropertySource@794279295 {name=‘configmap.dhruva-proxy-dynamic-config-map.wxc-dhruva’, properties={ ####MASKED### SECURITY REASONS
10:12:24.807 WARN — [TaskScheduler-1] o.s.c.k.c.c.reload.ConfigReloadUtil : The current number of ConfigMap PropertySources does not match the ones loaded from Kubernetes - No reload will take place
Sample app config
spring:
application:
name: dhruvaProxyApplication
cloud:
consul:
enabled: false
kubernetes:
config:
include-profile-specific-sources: false
reload:
enabled: true
mode: polling
period: 10000 # event based missed some config changes, testing polling
discovery:
enabled: false
JAVA_OPTS -Dspring.cloud.kubernetes.config.namespace=wxc-dhruva -Dspring.cloud.kubernetes.config.sources[0].name=dhruva-proxy-static-config-map -Dspring.cloud.kubernetes.config.sources[1].name=dhruva-proxy-dynamic-config-map
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 25 (13 by maintainers)
finally I had some time to put up a test to reproduce this issue, sorry it takes so much and I am so slow at this, this unfortunately is not my main job and I have little time these days…
Anyway, I did reproduce it via a test. It is indeed a bug on our side, I am positive about this. The core issue is still in the fact that when there is an error, we generate an incorrect name. The thing you need to get an error twice and some very unlucky timings.
I am not going to bother you with details, I need to think on how to solve this one now. 😃
I have a PR in motion here, working on it…
as usual, time is the limiting factor for me… I will be looking at this one, but no timeline when exactly.