bank-of-anthos: Non-GKE: Tomcat is not starting due to missing credentials despite metrics disabled

Describe the bug

I am trying to deploy bank-of-anthos on a non-GKE cluster (IBM Cloud Kubernetes Service). The java services keep crashing and do not start properly with the error The Application Default Credentials are not available. - despite having disabled metrics and tracing. Any tips on how to get the service running without Google Services are highly appreciated. Thank you!

To Reproduce

1. Set ENABLE_TRACING and ENABLE_METRICS in all deployment manifests to “false”
2. Run

   kubectl apply -f ./extras/jwt/jwt-secret.yaml
   kubectl apply -f ./kubernetes-manifests
   

3. Notice that the python services come up properly, but the java services all keep restarting and never get to a ready state.

   NAME                                  READY   STATUS             RESTARTS      AGE
   accounts-db-0                         1/1     Running            0             4m47s
   balancereader-5c99564794-w8jvn        0/1     CrashLoopBackOff   3 (35s ago)   4m56s
   contacts-6b44695c44-c9nnl             1/1     Running            0             4m56s
   frontend-6dcb6bc9fc-75kw5             1/1     Running            0             4m56s
   ledger-db-0                           1/1     Running            0             4m47s
   ledgerwriter-657f98d7c8-m67j4         0/1     CrashLoopBackOff   3 (28s ago)   4m56s
   loadgenerator-555cd877df-6dcp9        1/1     Running            0             4m56s
   transactionhistory-7c4d4794db-rd2mt   0/1     CrashLoopBackOff   3 (10s ago)   4m56s
   userservice-5c955dbb5f-llwmh          1/1     Running            0             4m56s
   

Logs

These are the logs from balancereader:

{"timestamp": "2023-03-16 09:35:38", "message": "logStarting | Starting BalanceReaderApplication using Java 17.0.4.1 with PID 1 (/app/classes started by ? in /)", "severity": "INFO"}
{"timestamp": "2023-03-16 09:35:38", "message": "logStartupProfileInfo | No active profile set, falling back to 1 default profile: "default"", "severity": "INFO"}
{"timestamp": "2023-03-16 09:35:52", "message": "registerRepositoriesIn | Bootstrapping Spring Data JPA repositories in DEFAULT mode.", "severity": "INFO"}
{"timestamp": "2023-03-16 09:35:54", "message": "registerRepositoriesIn | Finished Spring Data repository scanning in 1639 ms. Found 1 JPA repository interfaces.", "severity": "INFO"}
{"timestamp": "2023-03-16 09:36:01", "message": "initialize | Tomcat initialized with port(s): 8080 (http)", "severity": "INFO"}
{"timestamp": "2023-03-16 09:36:03", "message": "log | Initializing ProtocolHandler ["http-nio-8080"]", "severity": "INFO"}
{"timestamp": "2023-03-16 09:36:03", "message": "log | Starting service [Tomcat]", "severity": "INFO"}
{"timestamp": "2023-03-16 09:36:03", "message": "log | Starting Servlet engine: [Apache Tomcat/10.1.5]", "severity": "INFO"}
{"timestamp": "2023-03-16 09:36:05", "message": "log | Initializing Spring embedded WebApplicationContext", "severity": "INFO"}
{"timestamp": "2023-03-16 09:36:05", "message": "prepareWebApplicationContext | Root WebApplicationContext: initialization completed in 26851 ms", "severity": "INFO"}
{"timestamp": "2023-03-16 09:36:07", "message": "gcpProjectIdProvider | The default project ID is null", "severity": "INFO"}
{"timestamp": "2023-03-16 09:36:07", "message": "<init> | No core credentials are set. Service-specific credentials (e.g., spring.cloud.gcp.pubsub.credentials.*) should be used if your app uses services that require credentials.", "severity": "WARN"}
java.io.IOException: The Application Default Credentials are not available. They are available if running in Google Compute Engine. Otherwise, the environment variable GOOGLE_APPLICATION_CREDENTIALS must be defined pointing to a file defining the credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information.
        at com.google.auth.oauth2.DefaultCredentialsProvider.getDefaultCredentials(DefaultCredentialsProvider.java:127) ~[google-auth-library-oauth2-http-1.15.0.jar:?]
        at com.google.auth.oauth2.GoogleCredentials.getApplicationDefault(GoogleCredentials.java:129) ~[google-auth-library-oauth2-http-1.15.0.jar:?]
        at com.google.auth.oauth2.GoogleCredentials.getApplicationDefault(GoogleCredentials.java:101) ~[google-auth-library-oauth2-http-1.15.0.jar:?]
        at com.google.api.gax.core.GoogleCredentialsProvider.getCredentials(GoogleCredentialsProvider.java:70) ~[gax-2.23.0.jar:2.23.0]
        at com.google.cloud.spring.core.DefaultCredentialsProvider.<init>(DefaultCredentialsProvider.java:101) ~[spring-cloud-gcp-core-4.1.0.jar:4.1.0]
        at com.google.cloud.spring.autoconfigure.core.GcpContextAutoConfiguration.googleCredentials(GcpContextAutoConfiguration.java:56) ~[spring-cloud-gcp-autoconfigure-4.1.0.jar:4.1.0]
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) ~[?:?]
        at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[?:?]
        at java.lang.reflect.Method.invoke(Unknown Source) ~[?:?]
        at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:139) ~[spring-beans-6.0.4.jar:6.0.4]
        at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:653) ~[spring-beans-6.0.4.jar:6.0.4]
        at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:491) ~[spring-beans-6.0.4.jar:6.0.4]
        ...
{"timestamp": "2023-03-16 09:36:12", "message": "onStartup | Error starting Tomcat context. Exception: org.springframework.beans.factory.UnsatisfiedDependencyException. Message: Error creating bean with name 'webMvcObservationFilter' defined in class path resource [org/springframework/boot/actuate/autoconfigure/observation/web/servlet/WebMvcObservationAutoConfiguration.class]: Unsatisfied dependency expressed through method 'webMvcObservationFilter' parameter 0: Error creating bean with name 'observationRegistry' defined in class path resource [org/springframework/boot/actuate/autoconfigure/observation/ObservationAutoConfiguration.class]: Error creating bean with name 'defaultTracingObservationHandler' defined in class path resource [org/springframework/boot/actuate/autoconfigure/tracing/MicrometerTracingAutoConfiguration.class]: Unsatisfied dependency expressed through method 'defaultTracingObservationHandler' parameter 0: Error creating bean with name 'braveTracerBridge' defined in class path resource [org/springframework/boot/actuate/autoconfigure/tracing/BraveAutoConfiguration.class]: Unsatisfied dependency expressed through method 'braveTracerBridge' parameter 0: Error creating bean with name 'braveTracer' defined in class path resource [org/springframework/boot/actuate/autoconfigure/tracing/BraveAutoConfiguration.class]: Unsatisfied dependency expressed through method 'braveTracer' parameter 0: Error creating bean with name 'braveTracing' defined in class path resource [org/springframework/boot/actuate/autoconfigure/tracing/BraveAutoConfiguration.class]: Unsatisfied dependency expressed through method 'braveTracing' parameter 1: Error creating bean with name 'stackdriverSpanHandler' defined in class path resource [com/google/cloud/spring/autoconfigure/trace/StackdriverTraceAutoConfiguration.class]: Unsatisfied dependency expressed through method 'stackdriverSpanHandler' parameter 0: Error creating bean with name 'stackdriverReporter' defined in class path resource [com/google/cloud/spring/autoconfigure/trace/StackdriverTraceAutoConfiguration.class]: Unsatisfied dependency expressed through method 'stackdriverReporter' parameter 2: Error creating bean with name 'stackdriverSender' defined in class path resource [com/google/cloud/spring/autoconfigure/trace/StackdriverTraceAutoConfiguration.class]: Failed to instantiate [zipkin2.reporter.Sender]: Factory method 'stackdriverSender' threw exception with message: The Application Default Credentials are not available. They are available if running in Google Compute Engine. Otherwise, the environment variable GOOGLE_APPLICATION_CREDENTIALS must be defined pointing to a file defining the credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information.", "severity": "ERROR"}
{"timestamp": "2023-03-16 09:36:12", "message": "log | Stopping service [Tomcat]", "severity": "INFO"}
{"timestamp": "2023-03-16 09:36:12", "message": "refresh | Exception encountered during context initialization - cancelling refresh attempt: org.springframework.context.ApplicationContextException: Unable to start web server", "severity": "WARN"}
{"timestamp": "2023-03-16 09:36:12", "message": "logMessage | 

Error starting ApplicationContext. To display the condition evaluation report re-run your application with 'debug' enabled.", "severity": "INFO"}
{"timestamp": "2023-03-16 09:36:13", "message": "reportFailure | Application run failed", "severity": "ERROR"}
org.springframework.context.ApplicationContextException: Unable to start web server
        at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.onRefresh(ServletWebServerApplicationContext.java:164) ~[spring-boot-3.0.2.jar:3.0.2]
...
Caused by: org.springframework.boot.web.server.WebServerException: Unable to start embedded Tomcat

Environment

Kubernetes distribution, version: v1.25.6 IKS (IBM Cloud Kubernetes Service)

Additional context

Balance Reader Description:

Name:                   balancereader
Namespace:              bank-of-anthos
CreationTimestamp:      Wed, 15 Mar 2023 23:49:30 +0100
Labels:                 <none>
Annotations:            deployment.kubernetes.io/revision: 1
Selector:               app=balancereader
Replicas:               1 desired | 1 updated | 1 total | 0 available | 1 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  25% max unavailable, 25% max surge
Pod Template:
  Labels:           app=balancereader
  Service Account:  default
  Containers:
   balancereader:
    Image:      gcr.io/bank-of-anthos-ci/balancereader:v0.5.11
    Port:       <none>
    Host Port:  <none>
    Limits:
      cpu:     500m
      memory:  512Mi
    Requests:
      cpu:      100m
      memory:   256Mi
    Liveness:   http-get http://:8080/healthy delay=120s timeout=10s period=5s #success=1 #failure=3
    Readiness:  http-get http://:8080/ready delay=60s timeout=10s period=5s #success=1 #failure=3
    Startup:    http-get http://:8080/healthy delay=0s timeout=1s period=10s #success=1 #failure=30
    Environment Variables from:
      environment-config  ConfigMap  Optional: false
      ledger-db-config    ConfigMap  Optional: false
    Environment:
      VERSION:         v0.5.11
      PORT:            8080
      ENABLE_TRACING:  false
      ENABLE_METRICS:  false
      POLL_MS:         100
      CACHE_SIZE:      1000000
      JVM_OPTS:        -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -Xms256m -Xmx512m
      LOG_LEVEL:       info
      NAMESPACE:        (v1:metadata.namespace)
    Mounts:
      /tmp from tmp (rw)
      /tmp/.ssh from publickey (ro)
  Volumes:
   publickey:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  jwt-key
    Optional:    false
   tmp:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Progressing    True    NewReplicaSetAvailable
  Available      False   MinimumReplicasUnavailable
OldReplicaSets:  balancereader-5c99564794 (1/1 replicas created)
NewReplicaSet:   <none>
Events:
  Type    Reason             Age                   From                   Message
  ----    ------             ----                  ----                   -------
  Normal  ScalingReplicaSet  10m (x2 over 12h)     deployment-controller  Scaled down replica set balancereader-5c99564794 to 0 from 1
  Normal  ScalingReplicaSet  7m33s (x2 over 122m)  deployment-controller  Scaled up replica set balancereader-5c99564794 to 1 from 0

Exposure