external-secrets: Sometimes multiple errors occur and secret files are not synchronized properly in GKE
Version / env
- ESO
v0.4.4 - GKE
v1.21.6-gke.1500 - use
Workload Identity - use
SecretStore(don’t use ClusterSecretStore) - use
Secret Manager
Problem
When I kubectl describe external-secret, I get the following error
Events:
Type Reason AgeFromMessage
---- ------ ---- -----------
Normal Updated56m (x415 over 7h56m)external-secretsUpdated Secret
WarningUpdateFailed 49m (x2 over 53m)external-secrets could not get secret data from provider: key "development-exsec-my-credential" from ExternalSecret "my-external-secrets-operator": unable to access Secret from SecretManager Client: rpc error: code = PermissionDenied desc = Permission 'secretmanager.versions.access' denied for resource 'projects/MY_PROJECT/secrets/development-exsec-my-credential/versions/latest' (or it may not exist).
WarningUpdateFailed 43m (x12 over 53m) external-secrets could not get secret data from provider: key "development-exsec-my-session-key" from ExternalSecret "my-external-secrets-operator": unable to access Secret from SecretManager Client: rpc error: code = Canceled desc = grpc: the client connection is closing
WarningUpdateFailed 38m external-secrets could not get secret data from provider: key "development-exsec-my-api-key" from ExternalSecret "my-external-secrets-operator": unable to access Secret from SecretManager Client: rpc error: code = Canceled desc = grpc: the client connection is closing
WarningUpdateFailed 24m (x19 over 53m) external-secrets could not get secret data from provider: key "development-exsec-my-credential" from ExternalSecret "my-external-secrets-operator": unable to access Secret from SecretManager Client: rpc error: code = Canceled desc = grpc: the client connection is closing
WarningInvalidProviderClientConfig17m (x33 over 54m) external-secrets failed to create GCP secretmanager client: unable to generate gcp access token: rpc error: code = Canceled desc = grpc: the client connection is closing
Normal Updated12m (x14 over 45m) external-secretsUpdated Secret
WarningUpdateFailed 2m42s (x10 over 34m) external-secrets could not get secret data from provider: key "development-exsec-my-bugsnag-api-key" from ExternalSecret "my-external-secrets-operator": unable to access Secret from SecretManager Client: rpc error: code = Canceled desc = grpc: the client connection is closing
In summary, the following types of errors occur randomly.
msg: could not update Secret
err: could not get secret data from provider: key \\"development-exsec-admin-bugsnag-api-key-for-api\\" from ExternalSecret \\"admin-secrets\\": unable to access Secret from SecretManager Client: rpc error: code = Canceled desc = grpc: the client connection is closing
msg: could not update Secret
err: could not get secret data from provider: key "development-exsec-my-credential" from ExternalSecret "my-external-secrets-operator": unable to access Secret from SecretManager Client: rpc error: code = PermissionDenied desc = Permission 'secretmanager.versions.access' denied for resource 'projects/MY_PROJECT/secrets/development-exsec-my-credential/versions/latest' (or it may not exist).
msg: could not get provider client
err: failed to create GCP secretmanager client: unable to generate gcp access token: rpc error: code = Canceled desc = grpc: the client connection is closing
msg: could not close provider client
err: failed to create GCP secretmanager client: unable to generate gcp access token: rpc error: code = Canceled desc = grpc: the client connection is closing
Lack of authorization appears to be the problem, but the following are in place.
- Workload Identity settings (NodePool/GKE) have been done.
- GSA - KSA ties are in place.
- IAM role (Workload Identity/TokenCreator) is granted.
I believe that the privileges granted are not wrong because "most of the time the Secret is created successfully, but sometimes it fails.
By the way, here is the ESO Pod log. (Ignore the fact that it does not correspond to the above error.)
ESO Pod log (Click me)
{"level":"error","ts":1646971718.1400776,"logger":"controllers.ExternalSecret","msg":"could not close provider client","ExternalSecret":"app-dev-test1/my-secrets","SecretStore":"app-dev-test1/my-external-secrets-operator","error":"unable to close SecretManager client: rpc error: code = Canceled desc = grpc: the client connection is closing (and 4 other errors)","stacktrace":"github.com/external-secrets/external-secrets/pkg/controllers/externalsecret.(*Reconciler).Reconcile\\n\\t/home/runner/work/external-secrets/external-secrets/pkg/controllers/externalsecret/externalsecret_controller.go:246\\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\\n\\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.1/pkg/internal/controller/controller.go:114\\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\\n\\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.1/pkg/internal/controller/controller.go:311\\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\\n\\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.1/pkg/internal/controller/controller.go:266\\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\\n\\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.1/pkg/internal/controller/controller.go:227"}
{"level":"info","ts":1646971718.6454322,"logger":"controllers.ExternalSecret","msg":"reconciled secret","ExternalSecret":"app-dev-test1/my-secrets","SecretStore":"app-dev-test1/my-external-secrets-operator"}external-secrets-c4bc855d7-9wnlb {"level":"error","ts":1646971718.7890575,"logger":"controllers.ExternalSecret","msg":"could not get provider client","ExternalSecret":"app-dev-outgame-sandbox/admin-secrets","SecretStore":"app-dev-outgame-sandbox/admin-external-secrets-operator","error":"failed to create GCP secretmanager client: unable to generate gcp access token: rpc error: code = Canceled desc = grpc: the client connection is closing","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\\n\\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.1/pkg/internal/controller/controller.go:114\\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\\n\\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.1/pkg/internal/controller/controller.go:311\\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\\n\\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.1/pkg/internal/controller/controller.go:266\\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\\n\\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.1/pkg/internal/controller/controller.go:227"}
{"level":"error","ts":1646971721.8899071,"logger":"controllers.ExternalSecret","msg":"could not update Secret","ExternalSecret":"app-dev-test2/admin-secrets","SecretStore":"app-dev-test2/admin-external-secrets-operator","error":"could not get secret data from provider: key \\"development-exsec-admin-bugsnag-api-key-for-api\\" from ExternalSecret \\"admin-secrets\\": unable to access Secret from SecretManager Client: rpc error: code = Canceled desc = grpc: the client connection is closing","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\\n\\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.1/pkg/internal/controller/controller.go:114\\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\\n\\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.1/pkg/internal/controller/controller.go:311\\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\\n\\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.1/pkg/internal/controller/controller.go:266\\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\\n\\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.1/pkg/internal/controller/controller.go:227"}
{"level":"error","ts":1646971721.890896,"logger":"controllers.ExternalSecret","msg":"could not get provider client","ExternalSecret":"prometheus/prometheus-iap-client","SecretStore":"prometheus/external-secrets-operator","error":"failed to create GCP secretmanager client: unable to generate gcp access token: rpc error: code = Canceled desc = grpc: the client connection is closing","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\\n\\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.1/pkg/internal/controller/controller.go:114\\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\\n\\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.1/pkg/internal/controller/controller.go:311\\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\\n\\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.1/pkg/internal/controller/controller.go:266\\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\\n\\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.1/pkg/internal/controller/controller.go:227"}
{"level":"info","ts":1646971722.2506986,"logger":"controllers.ExternalSecret","msg":"reconciled secret","ExternalSecret":"prometheus/prometheus-iap-client","SecretStore":"prometheus/external-secrets-operator"}
{"level":"error","ts":1646971781.9834816,"logger":"controllers.ExternalSecret","msg":"could not update Secret","ExternalSecret":"app-dev-test3/admin-secrets","SecretStore":"app-dev-test3/admin-external-secrets-operator","error":"could not get secret data from provider: key \\"development-exsec-admin-bugsnag-api-key-for-frontend\\" from ExternalSecret \\"admin-secrets\\": unable to access Secret from SecretManager Client: rpc error: code = PermissionDenied desc = Permission 'secretmanager.versions.access' denied for resource 'projects/MY_PROJECT/secrets/development-exsec-admin-bugsnag-api-key-for-frontend/versions/latest' (or it may not exist).","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\\n\\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.1/pkg/internal/controller/controller.go:114\\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\\n\\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.1/pkg/internal/controller/controller.go:311\\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\\n\\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.1/pkg/internal/controller/controller.go:266\\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\\n\\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.1/pkg/internal/controller/controller.go:227"}
{"level":"error","ts":1646971810.505019,"logger":"controllers.ExternalSecret","msg":"could not update Secret","ExternalSecret":"app-dev-test4/admin-secrets","SecretStore":"app-dev-test4/admin-external-secrets-operator","error":"could not get secret data from provider: key \\"development-exsec-admin-bugsnag-api-key-for-frontend\\" from ExternalSecret \\"admin-secrets\\": unable to access Secret from SecretManager Client: rpc error: code = PermissionDenied desc = Permission 'secretmanager.versions.access' denied for resource 'projects/MY_PROJECT/secrets/development-exsec-admin-bugsnag-api-key-for-frontend/versions/latest' (or it may not exist).","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\\n\\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.1/pkg/internal/controller/controller.go:114\\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\\n\\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.1/pkg/internal/controller/controller.go:311\\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\\n\\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.1/pkg/internal/controller/controller.go:266\\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\\n\\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.1/pkg/internal/controller/controller.go:227"}
Concerns
We have two GKE clusters running on one GCP project, both running ESO.
Tentatively named dev cluster / staging cluster, this problem only occurs in the dev cluster.
The ESO startup configuration for the dev cluster and staging cluster is exactly the same.
The only difference is that the dev cluster has many external secrets and secret stores running because it is a development environment.
# dev cluster
❯ k get -A externalsecrets.external-secrets.io| wc -l
68
❯ k get -A secretstores.external-secrets.io | wc -l
47
# staging cluster
❯ k get -A externalsecrets.external-secrets.io| wc -l
9
❯ k get -A secretstores.external-secrets.io | wc -l
5
Our manifests
our manifests (Click me)
apiVersion: external-secrets.io/v1alpha1
kind: ExternalSecret
metadata:
labels:
app.kubernetes.io/component: admin-api
app.kubernetes.io/name: admin-api
name: admin-iap-client-secrets
namespace: app-dev-test0
spec:
data:
- remoteRef:
key: development-exsec-admin-iap-client
property: client_id
version: latest
secretKey: client_id
- remoteRef:
key: development-exsec-admin-iap-client
property: client_secret
version: latest
secretKey: client_secret
refreshInterval: 1m
secretStoreRef:
kind: SecretStore
name: admin-external-secrets-operator
---
apiVersion: external-secrets.io/v1alpha1
kind: ExternalSecret
metadata:
labels:
app.kubernetes.io/component: admin-api
app.kubernetes.io/name: admin-api
name: admin-secrets
namespace: app-dev-test0
spec:
data:
- remoteRef:
key: development-exsec-admin-bugsnag-api-key-for-api
version: latest
secretKey: bugsnag-api-key-for-api
- remoteRef:
key: development-exsec-admin-bugsnag-api-key-for-frontend
version: latest
secretKey: bugsnag-api-key-for-frontend
refreshInterval: 1m
secretStoreRef:
kind: SecretStore
name: admin-external-secrets-operator
target:
creationPolicy: Owner
name: admin-secrets
---
apiVersion: external-secrets.io/v1alpha1
kind: ExternalSecret
metadata:
labels:
app.kubernetes.io/component: api
app.kubernetes.io/name: my-api
name: my-secrets
namespace: app-dev-test0
spec:
data:
- remoteRef:
key: development-exsec-my-credential
version: latest
secretKey: credentials.json
- remoteRef:
key: development-exsec-my-session-key
version: latest
secretKey: session-key
- remoteRef:
key: development-exsec-my-api-key
version: latest
secretKey: api-key
- remoteRef:
key: development-exsec-my-bugsnag-api-key
version: latest
secretKey: bugsnag-api-key
refreshInterval: 1m
secretStoreRef:
kind: SecretStore
name: my-external-secrets-operator
target:
creationPolicy: Owner
name: my-secrets
apiVersion: external-secrets.io/v1alpha1
kind: SecretStore
metadata:
labels:
app.kubernetes.io/component: admin-api
app.kubernetes.io/name: admin-api
name: admin-external-secrets-operator
namespace: app-dev-test0
spec:
provider:
gcpsm:
auth:
workloadIdentity:
clusterLocation: asia-northeast1
clusterName: development
serviceAccountRef:
name: admin-external-secrets-operator
projectID: MY_PROJECT
---
apiVersion: external-secrets.io/v1alpha1
kind: SecretStore
metadata:
labels:
app.kubernetes.io/component: api
app.kubernetes.io/name: my-api
name: my-external-secrets-operator
namespace: app-dev-test0
spec:
provider:
gcpsm:
auth:
workloadIdentity:
clusterLocation: asia-northeast1
clusterName: development
serviceAccountRef:
name: my-external-secrets-operator
projectID: MY_PROJECT
What we tried
I tried and checked the following, but no improvement.
- Set
concurrent=10to ESO indev-cluster. - Set
.spec.refreshIntervalto5minexternalsecretsused indev-cluster(originally1m) - Check GCP Quota for Secret Manager API / IAM API (
staging-clusteris running in the same PJ so this should not be a problem)
I also took the debug log but it doesn’t seem to have any useful information.
debug log(Click me)
2022-03-11T06:41:03.937506827Z {"level":"debug","ts":1646980863.9366648,"logger":"controllers.ExternalSecret","msg":"reconciled secret","ExternalSecret":"app-dev-test0/admin-secrets","SecretStore":"app-dev-test0/admin-external-secrets-operator"}
2022-03-11T06:41:03.937595199Z {"level":"debug","ts":1646980863.9367306,"logger":"events","msg":"Normal","object":{"kind":"ExternalSecret","namespace":"app-dev-test0","name":"admin-secrets","uid":"5bdb1a3b-226b-41d3-b116-eaf5e55cc0c7","apiVersion":"external-secrets.io/v1alpha1","resourceVersion":"370114762"},"reason":"Updated","message":"Updated Secret"}
2022-03-11T06:41:04.059129393Z {"level":"debug","ts":1646980864.0589173,"logger":"controllers.ExternalSecret","msg":"skipping refresh","ExternalSecret":"app-dev-test0/admin-secrets","SecretStore":"app-dev-test0/admin-external-secrets-operator","rv":"1-472f358594e87ba4678b952d5f0d02af"}
2022-03-11T06:41:30.059628782Z {"level":"error","ts":1646980890.059401,"logger":"controllers.ExternalSecret","msg":"could not get provider client","ExternalSecret":"app-dev-test0/admin-iap-client-secrets","SecretStore":"app-dev-test0/admin-external-secrets-operator","error":"failed to create GCP secretmanager client: unable to generate gcp access token: rpc error: code = Canceled desc = grpc: the client connection is closing","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\\n\\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.1/pkg/internal/controller/controller.go:114\\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\\n\\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.1/pkg/internal/controller/controller.go:311\\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\\n\\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.1/pkg/internal/controller/controller.go:266\\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\\n\\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.1/pkg/internal/controller/controller.go:227"}
2022-03-11T06:41:30.060736443Z {"level":"debug","ts":1646980890.0605001,"logger":"events","msg":"Warning","object":{"kind":"ExternalSecret","namespace":"app-dev-test0","name":"admin-iap-client-secrets","uid":"500f9d9f-8d68-4e18-804b-9df866414a54","apiVersion":"external-secrets.io/v1alpha1","resourceVersion":"370112794"},"reason":"InvalidProviderClientConfig","message":"failed to create GCP secretmanager client: unable to generate gcp access token: rpc error: code = Canceled desc = grpc: the client connection is closing"}
2022-03-11T06:41:30.385601122Z {"level":"info","ts":1646980890.3854601,"logger":"controllers.ExternalSecret","msg":"reconciled secret","ExternalSecret":"app-dev-test0/admin-iap-client-secrets","SecretStore":"app-dev-test0/admin-external-secrets-operator"}
2022-03-11T06:41:30.386757849Z {"level":"debug","ts":1646980890.3862054,"logger":"events","msg":"Normal","object":{"kind":"ExternalSecret","namespace":"app-dev-test0","name":"admin-iap-client-secrets","uid":"500f9d9f-8d68-4e18-804b-9df866414a54","apiVersion":"external-secrets.io/v1alpha1","resourceVersion":"370115999"},"reason":"Updated","message":"Updated Secret"}
2022-03-11T06:41:30.494851945Z {"level":"debug","ts":1646980890.4946878,"logger":"controllers.ExternalSecret","msg":"skipping refresh","ExternalSecret":"app-dev-test0/admin-iap-client-secrets","SecretStore":"app-dev-test0/admin-external-secrets-operator","rv":"2-aa72504d829be8d0dc98bffd91af7175"}
2022-03-11T06:41:30.496088871Z {"level":"error","ts":1646980890.495925,"logger":"controllers.ExternalSecret","msg":"could not close provider client","ExternalSecret":"app-dev-test0/admin-iap-client-secrets","SecretStore":"app-dev-test0/admin-external-secrets-operator","error":"unable to close SecretManager client: rpc error: code = Canceled desc = grpc: the client connection is closing (and 4 other errors)","stacktrace":"github.com/external-secrets/external-secrets/pkg/controllers/externalsecret.(*Reconciler).Reconcile\\n\\t/home/runner/work/external-secrets/external-secrets/pkg/controllers/externalsecret/externalsecret_controller.go:189\\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\\n\\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.1/pkg/internal/controller/controller.go:114\\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\\n\\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.1/pkg/internal/controller/controller.go:311\\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\\n\\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.1/pkg/internal/controller/controller.go:266\\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\\n\\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.1/pkg/internal/controller/controller.go:227"}
2022-03-11T06:42:00.188171213Z {"level":"debug","ts":1646980920.1880045,"logger":"controllers.ExternalSecret","msg":"skipping refresh","ExternalSecret":"app-dev-test0/admin-iap-client-secrets","SecretStore":"app-dev-test0/admin-external-secrets-operator","rv":"2-aa72504d829be8d0dc98bffd91af7175"}
2022-03-11T06:42:03.827023557Z {"level":"debug","ts":1646980923.826863,"logger":"controllers.ExternalSecret","msg":"reconciled secret","ExternalSecret":"app-dev-test0/my-secrets","SecretStore":"app-dev-test0/my-external-secrets-operator"}
2022-03-11T06:42:03.829087766Z {"level":"debug","ts":1646980923.8288572,"logger":"events","msg":"Normal","object":{"kind":"ExternalSecret","namespace":"app-dev-test0","name":"my-secrets","uid":"78d5e757-064c-4214-981b-98889d1af799","apiVersion":"external-secrets.io/v1alpha1","resourceVersion":"370113132"},"reason":"Updated","message":"Updated Secret"}
2022-03-11T06:42:03.946847552Z {"level":"debug","ts":1646980923.9466808,"logger":"controllers.ExternalSecret","msg":"skipping refresh","ExternalSecret":"app-dev-test0/my-secrets","SecretStore":"app-dev-test0/my-external-secrets-operator","rv":"2-b8c4fa48ab589f778dd09a8f2b0ea8e7"}
2022-03-11T06:42:04.268476009Z {"level":"debug","ts":1646980924.2669353,"logger":"controllers.ExternalSecret","msg":"reconciled secret","ExternalSecret":"app-dev-test0/admin-secrets","SecretStore":"app-dev-test0/admin-external-secrets-operator"}
2022-03-11T06:42:04.268531595Z {"level":"debug","ts":1646980924.2683086,"logger":"events","msg":"Normal","object":{"kind":"ExternalSecret","namespace":"app-dev-test0","name":"admin-secrets","uid":"5bdb1a3b-226b-41d3-b116-eaf5e55cc0c7","apiVersion":"external-secrets.io/v1alpha1","resourceVersion":"370115711"},"reason":"Updated","message":"Updated Secret"}
2022-03-11T06:42:04.380377285Z {"level":"debug","ts":1646980924.3801599,"logger":"controllers.ExternalSecret","msg":"skipping refresh","ExternalSecret":"app-dev-test0/admin-secrets","SecretStore":"app-dev-test0/admin-external-secrets-operator","rv":"1-472f358594e87ba4678b952d5f0d02af"}
2022-03-11T06:43:03.803177998Z {"level":"debug","ts":1646980983.8029563,"logger":"controllers.SecretStore","msg":"validating","secretstore":"app-dev-test0/admin-external-secrets-operator"}
2022-03-11T06:43:03.901338841Z {"level":"debug","ts":1646980983.9011183,"logger":"events","msg":"Normal","object":{"kind":"SecretStore","namespace":"app-dev-test0","name":"admin-external-secrets-operator","uid":"c1e70348-046b-441d-9a25-1e49d9e113fb","apiVersion":"external-secrets.io/v1alpha1","resourceVersion":"370107419"},"reason":"Valid","message":"store validated"}
2022-03-11T06:43:04.602448342Z {"level":"debug","ts":1646980984.6022263,"logger":"controllers.ExternalSecret","msg":"reconciled secret","ExternalSecret":"app-dev-test0/admin-secrets","SecretStore":"app-dev-test0/admin-external-secrets-operator"}
2022-03-11T06:43:04.604262051Z {"level":"debug","ts":1646980984.6022446,"logger":"events","msg":"Normal","object":{"kind":"ExternalSecret","namespace":"app-dev-test0","name":"admin-secrets","uid":"5bdb1a3b-226b-41d3-b116-eaf5e55cc0c7","apiVersion":"external-secrets.io/v1alpha1","resourceVersion":"370116354"},"reason":"Updated","message":"Updated Secret"}
2022-03-11T06:43:04.716556753Z {"level":"debug","ts":1646980984.7163868,"logger":"controllers.ExternalSecret","msg":"skipping refresh","ExternalSecret":"app-dev-test0/admin-secrets","SecretStore":"app-dev-test0/admin-external-secrets-operator","rv":"1-472f358594e87ba4678b952d5f0d02af"}
2022-03-11T06:43:05.331476139Z {"level":"debug","ts":1646980985.3313165,"logger":"controllers.SecretStore","msg":"validating","secretstore":"app-dev-test0/my-external-secrets-operator"}
2022-03-11T06:43:05.439749825Z {"level":"debug","ts":1646980985.4394543,"logger":"events","msg":"Normal","object":{"kind":"SecretStore","namespace":"app-dev-test0","name":"my-external-secrets-operator","uid":"cff1fcff-67f6-4e43-b629-e24c86ae2c9b","apiVersion":"external-secrets.io/v1alpha1","resourceVersion":"370086544"},"reason":"Valid","message":"store validated"}
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Reactions: 1
- Comments: 18 (10 by maintainers)
Commits related to this issue
- Implements single Client as the default GCP SecretManager approach Fixes #818 Fixes #837 Signed-off-by: Gustavo Carvalho <gustavo.carvalho@container-solutions.com> — committed to external-secrets/external-secrets by gusfcarvalho 2 years ago
- Implementing Concurrency validation checks for providers that do not support it (like GCP SM) Fixes #818 Fixes #834 Signed-off-by: Gustavo Carvalho <gustavo.carvalho@container-solutions.com> — committed to gusfcarvalho/external-secrets by gusfcarvalho 2 years ago
- Implementing Concurrency validation checks for providers that do not support it (like GCP SM) Fixes #818 Fixes #834 Signed-off-by: Gustavo Carvalho <gustavo.carvalho@container-solutions.com> Fix sc... — committed to gusfcarvalho/external-secrets by gusfcarvalho 2 years ago
Hey @lirlia ! This can be a long shot, but can you try removing the
version: latestfield from your manifests? It seems to me that the issue is around this call on google grpc client. If you leave it empty, the latest version will be the one fetched.Edit: Nevermind, I just found out a better way to debug it. I think I’m moving closer to the issue.
I took a deeper look but couldn’t find any obvious issue from reading the code. My guess is that it’s specific to workload identity because the issue originates from the
iamcredentials.googleapis.comgrpc target. For some reason a connection pool seems to be re-used after it has been disposed. Based on your information it seems that the issue happens in ~3% of cases.Unfortunately i don’t have a GCP Account and can’t look deeper into that issue. @gusfcarvalho did you find some time to reproduce that issue?
I’ll debug this later today, and I’ll come back with any findings. It does seem to be a weird behavior, so my first guess would be that these messages show exactly while GCP is handling rotation (making latest version unavailable). I’ll take a further look into this, and if this is confirmed, I’ll come back to you as well. Probably there are ways to avoid this scenario within the provider code.
debug log in
dev clusterfull grpc-go debug log(Click me)