bank-vaults: kubebuilder.controller Reconciler error . failed to update vault status

Describe the bug kubebuilder.controller Reconciler error . failed to update vault status

To Reproduce Steps to reproduce the behavior:

operator version 0.4.13
vault version 1.1.0-beta2

Operator Logs

{"level":"info","ts":1553012934.9785693,"logger":"cmd","msg":"Go Version: go1.12"}
{"level":"info","ts":1553012934.9787564,"logger":"cmd","msg":"Go OS/Arch: linux/amd64"}
{"level":"info","ts":1553012934.978833,"logger":"cmd","msg":"operator-sdk Version: v0.4.1"}
{"level":"info","ts":1553012934.9788764,"logger":"cmd","msg":"Watched namespace: "}
{"level":"info","ts":1553012934.9836006,"logger":"leader","msg":"Trying to become the leader."}
{"level":"info","ts":1553012934.9845932,"logger":"cmd","msg":"Liveness probe listening on: 8080"}
{"level":"info","ts":1553012936.3835745,"logger":"leader","msg":"No pre-existing lock was found."}
{"level":"info","ts":1553012936.3945363,"logger":"leader","msg":"Became the leader."}
{"level":"info","ts":1553012936.6002703,"logger":"cmd","msg":"Registering Components."}
{"level":"info","ts":1553012936.6008043,"logger":"kubebuilder.controller","msg":"Starting EventSource","controller":"vault-controller","source":"kind source: /, Kind="}
{"level":"info","ts":1553012936.6010134,"logger":"cmd","msg":"Starting the Cmd."}
{"level":"info","ts":1553012936.7014093,"logger":"kubebuilder.controller","msg":"Starting Controller","controller":"vault-controller"}
{"level":"info","ts":1553012936.8016922,"logger":"kubebuilder.controller","msg":"Starting workers","controller":"vault-controller","worker count":1}
{"level":"info","ts":1553012936.8019593,"logger":"controller_vault","msg":"Reconciling Vault","Request.Namespace":"security","Request.Name":"vault"}
{"level":"info","ts":1553012977.1772525,"logger":"controller_vault","msg":"Reconciling Vault","Request.Namespace":"security","Request.Name":"vault"}
{"level":"error","ts":1553013005.4871998,"logger":"kubebuilder.controller","msg":"Reconciler error","controller":"vault-controller","request":"security/vault","error":"failed to update vault status: Operation cannot be fulfilled on vaults.vault.banzaicloud.com \"vault\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/pkg/mod/github.com/go-logr/zapr@v0.1.0/zapr.go:128\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.1.8/pkg/internal/controller/controller.go:215\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.1.8/pkg/internal/controller/controller.go:158\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20181128191346-49ce2735e507/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20181128191346-49ce2735e507/pkg/util/wait/wait.go:134\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20181128191346-49ce2735e507/pkg/util/wait/wait.go:88"}
{"level":"info","ts":1553013006.4886765,"logger":"controller_vault","msg":"Reconciling Vault","Request.Namespace":"security","Request.Name":"vault"}
{"level":"info","ts":1553013039.2430947,"logger":"controller_vault","msg":"Reconciling Vault","Request.Namespace":"security","Request.Name":"vault"}
{"level":"info","ts":1553013064.3822663,"logger":"controller_vault","msg":"Reconciling Vault","Request.Namespace":"security","Request.Name":"vault"}
{"level":"info","ts":1553013080.1126606,"logger":"controller_vault","msg":"Reconciling Vault","Request.Namespace":"security","Request.Name":"vault"}
{"level":"info","ts":1553013111.8073497,"logger":"controller_vault","msg":"Reconciling Vault","Request.Namespace":"security","Request.Name":"vault"}
{"level":"info","ts":1553013140.8785634,"logger":"controller_vault","msg":"Reconciling Vault","Request.Namespace":"security","Request.Name":"vault"}
{"level":"info","ts":1553013170.278676,"logger":"controller_vault","msg":"Reconciling Vault","Request.Namespace":"security","Request.Name":"vault"}
{"level":"info","ts":1553013193.3756018,"logger":"controller_vault","msg":"Reconciling Vault","Request.Namespace":"security","Request.Name":"vault"}
{"level":"info","ts":1553013215.6929455,"logger":"controller_vault","msg":"Reconciling Vault","Request.Namespace":"security","Request.Name":"vault"}
{"level":"info","ts":1553013229.283953,"logger":"controller_vault","msg":"Reconciling Vault","Request.Namespace":"security","Request.Name":"vault"}
{"level":"info","ts":1553013241.496228,"logger":"controller_vault","msg":"Reconciling Vault","Request.Namespace":"security","Request.Name":"vault"}
{"level":"info","ts":1553013260.8134034,"logger":"controller_vault","msg":"Reconciling Vault","Request.Namespace":"security","Request.Name":"vault"}
{"level":"info","ts":1553013280.4284825,"logger":"controller_vault","msg":"Reconciling Vault","Request.Namespace":"security","Request.Name":"vault"}
{"level":"info","ts":1553013298.9968777,"logger":"controller_vault","msg":"Reconciling Vault","Request.Namespace":"security","Request.Name":"vault"}
{"level":"info","ts":1553013316.3546042,"logger":"controller_vault","msg":"Reconciling Vault","Request.Namespace":"security","Request.Name":"vault"}
{"level":"info","ts":1553013332.4901135,"logger":"controller_vault","msg":"Reconciling Vault","Request.Namespace":"security","Request.Name":"vault"}
{"level":"info","ts":1553013354.4773872,"logger":"controller_vault","msg":"Reconciling Vault","Request.Namespace":"security","Request.Name":"vault"}
{"level":"info","ts":1553013373.1622124,"logger":"controller_vault","msg":"Reconciling Vault","Request.Namespace":"security","Request.Name":"vault"}
{"level":"info","ts":1553013394.0778363,"logger":"controller_vault","msg":"Reconciling Vault","Request.Namespace":"security","Request.Name":"vault"}
{"level":"info","ts":1553013410.878533,"logger":"controller_vault","msg":"Reconciling Vault","Request.Namespace":"security","Request.Name":"vault"}
{"level":"info","ts":1553013423.394313,"logger":"controller_vault","msg":"Reconciling Vault","Request.Namespace":"security","Request.Name":"vault"}
{"level":"info","ts":1553013437.8060305,"logger":"controller_vault","msg":"Reconciling Vault","Request.Namespace":"security","Request.Name":"vault"}
{"level":"info","ts":1553013453.9772651,"logger":"controller_vault","msg":"Reconciling Vault","Request.Namespace":"security","Request.Name":"vault"}
{"level":"info","ts":1553013473.8057735,"logger":"controller_vault","msg":"Reconciling Vault","Request.Namespace":"security","Request.Name":"vault"}
{"level":"info","ts":1553013486.3606265,"logger":"controller_vault","msg":"Reconciling Vault","Request.Namespace":"security","Request.Name":"vault"}

About this issue

Original URL
State: closed
Created 5 years ago
Comments: 40 (35 by maintainers)

Most upvoted comments

I’ve identified two issues during some local testing. First issue was that the order of pod names in the status was not consistent thus frequently triggering unnecessary updates. This was fixed in #546 .

The other issue was reading a stale cache in the reconciliation loop right after an update. My best guess is that in case we run out of the sync period, the controller immediately triggers a reconcile. Since the update operation does not update the local caches it is possible that the event from the API server does not arrive before the next loop starts, thus leaving us with a stale object. In order to avoid an error in this case #547 tries to load the object once again, which in my local tests eliminated the error.

pepov on Jul 12, 2019

Ok that is actually a change that happened sometime in the 0.4.1x series.

Now all vault pods are active and the ha is hanndled by vault itself.

So the 3/3 running containers in all pods is normal

Any request hitting a standby vault instance will be forwarded to the active one

You should check the status of the vault resource , which will tell you which one is active

Also if you exec in the pod and run vault status it will tell you which one is active.

You should probably update your vault image and check that you have the right settings in your config like api_addr since most of what I told you about vault ha is from my experience with newer versions and 0.11.6 might (or might not) miss some

I am on my phone now but I can give better directions tomorrow

primeroz on Jun 4, 2019

Thanks, will give a try and let you know.

gowthamsubbu on Jun 4, 2019

Anyhow we can change the default interval to 30s in the binary as well, that should be enough for all the operations to finish.

bonifaido on Jun 4, 2019