quarkus: Kubernetes Config fails on EKS/K8S 1.21 with Service Account Error

Describe the bug

We see these errors when deploying our native service using the latest Quarkus and the “kubernetes-config” extension: WARN: Error reading service account token from: [/var/run/secrets/kubernetes.io/serviceaccount/token]. Ignoring

ERROR: Failed to start application (with profile my-release) io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://10.100.0.1/api/v1/namespaces/platform-445-staging/configmaps/my-config. Message: configmaps “my-config” is forbidden: User “system:anonymous” cannot get resource “configmaps” in API group “” in the namespace “platform-445-staging”.

We are starting to think this is an issue between our K8S upgrade to 1.21 from 1.19. 1.19 - it was working here (no code/config changes) 1.20 - dont know if it works or not we only upgraded here to get to latest 1.21 - does not work

We know it worked before and as a quick test we deployed it into an older cluster (1.17) and it looks to work as-is.

Maybe it is related to this according to the upgrade log for K8S? Service account tokens bound to a pod is now a stable feature. The feature gates will be removed in 1.21 release. For more information, refer to notes below on the changelogs.

We double-checked and redeployed the same service as-is to 1.19 and 1.20 - both of those work as they did before.

Expected behavior

Deploys into EKS 1.21 as it did before and the configuration can read the config map.

Actual behavior

Fails to startup in EKS 1.21 with this error:

kubectl logs my-platform-client-c85ff7478-m7876 -n platform-445-staging

Aug 06, 2021 4:08:37 PM io.fabric8.kubernetes.client.Config WARN: Error reading service account token from: [/var/run/secrets/kubernetes.io/serviceaccount/token]. Ignoring. Aug 06, 2021 4:08:37 PM io.fabric8.kubernetes.client.Config WARN: Error reading service account token from: [/var/run/secrets/kubernetes.io/serviceaccount/token]. Ignoring. Aug 06, 2021 4:08:37 PM io.fabric8.kubernetes.client.Config WARN: Error reading service account token from: [/var/run/secrets/kubernetes.io/serviceaccount/token]. Ignoring. Aug 06, 2021 4:08:37 PM io.quarkus.runtime.ApplicationLifecycleManager run ERROR: Failed to start application (with profile my-release) io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://10.100.0.1/api/v1/namespaces/platform-445-staging/configmaps/my-config. Message: configmaps “my-config” is forbidden: User “system:anonymous” cannot get resource “configmaps” in API group “” in the namespace “platform-445-staging”. Received status: Status(apiVersion=v1, code=403, details=StatusDetails(causes=[], group=null, kind=configmaps, name=my-config, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=configmaps “my-config” is forbidden: User “system:anonymous” cannot get resource “configmaps” in API group “” in the namespace “platform-445-staging”, metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=Forbidden, status=Failure, additionalProperties={}).

How to Reproduce?

No response

Output of uname -a or ver

No response

Output of java -version

11

GraalVM version (if different from Java)

No response

Quarkus version or git rev

2.1.2 Final

Build tool (ie. output of mvnw --version or gradlew --version)

No response

Additional information

No response

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 15 (8 by maintainers)

Most upvoted comments

@geoand - tried to redeploy the working services under these conditions:

  • AWS EKS v1.21 (upgraded from v1.20)
  • Quarkus 2.3.0.Final
  • Native image being deployed
  • Kubernetes and Kubernetes-Config extensions are enabled.

Same issue as before…

$ kubectl logs fusion-platform-client-v1-585c988484-dnthm -n fusion-staging Oct 07, 2021 4:52:30 PM io.fabric8.kubernetes.client.Config WARN: Error reading service account token from: [/var/run/secrets/kubernetes.io/serviceaccount/token]. Ignoring. Oct 07, 2021 4:52:30 PM io.fabric8.kubernetes.client.Config WARN: Error reading service account token from: [/var/run/secrets/kubernetes.io/serviceaccount/token]. Ignoring. Oct 07, 2021 4:52:30 PM io.fabric8.kubernetes.client.Config WARN: Error reading service account token from: [/var/run/secrets/kubernetes.io/serviceaccount/token]. Ignoring. Oct 07, 2021 4:52:30 PM io.quarkus.runtime.ApplicationLifecycleManager run ERROR: Failed to start application (with profile fusion-release) io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://10.100.0.1/api/v1/namespaces/fusion-staging/configmaps/fusion-config. Message: configmaps "fusion-config" is forbidden: User "system:anonymous" cannot get resource "configmaps" in API group "" in the namespace "fusion-staging". Received status: Status(apiVersion=v1, code=403, details=StatusDetails(causes=[], group=null, kind=configmaps, name=fusion-config, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=configmaps "fusion-config" is forbidden: User "system:anonymous" cannot get resource "configmaps" in API group "" in the namespace "fusion-staging", metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=Forbidden, status=Failure, additionalProperties={}). at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:686) at io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:625) at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:565) at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:526) at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleGet(OperationSupport.java:493) at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleGet(OperationSupport.java:475) at io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleGet(BaseOperation.java:807) at io.fabric8.kubernetes.client.dsl.base.BaseOperation.getMandatory(BaseOperation.java:188) at io.fabric8.kubernetes.client.dsl.base.BaseOperation.get(BaseOperation.java:155) at io.fabric8.kubernetes.client.dsl.base.BaseOperation.get(BaseOperation.java:88) at io.quarkus.kubernetes.config.runtime.KubernetesConfigSourceProvider.getConfigMapConfigSources(KubernetesConfigSourceProvider.java:69) at io.quarkus.kubernetes.config.runtime.KubernetesConfigSourceProvider.getConfigSources(KubernetesConfigSourceProvider.java:45) at io.quarkus.runtime.configuration.ConfigUtils.addSourceProvider(ConfigUtils.java:178) at io.quarkus.runtime.configuration.ConfigUtils.addSourceProviders(ConfigUtils.java:192) at io.quarkus.runtime.generated.Config.readConfig(Config.zig:2639) at io.quarkus.deployment.steps.RuntimeConfigSetup.deploy(RuntimeConfigSetup.zig:60) at io.quarkus.runner.ApplicationImpl.doStart(ApplicationImpl.zig:513) at io.quarkus.runtime.Application.start(Application.java:101) at io.quarkus.runtime.ApplicationLifecycleManager.run(ApplicationLifecycleManager.java:101) at io.quarkus.runtime.Quarkus.run(Quarkus.java:66) at io.quarkus.runtime.Quarkus.run(Quarkus.java:42) at io.quarkus.runtime.Quarkus.run(Quarkus.java:119) at io.quarkus.runner.GeneratedMain.main(GeneratedMain.zig:29)

I will try a “fresh” EKS cluster as well once i get a chance just to make sure.

Also, just to confirm, v1.20 the same services will deploy fine.

+1
kjq on Oct 7, 2021
Read more comments on GitHub
← quarkus: KStreams extension does not support configure it in code
quarkus: Link.fromMethod does not take class' @Path into account →