chaos-mesh: unable to recover networkchaos after deleting related CRD
Bug Report
What version of Kubernetes are you using?
Client Version: version.Info{Major:“1”, Minor:“15”, GitVersion:“v1.15.1”, GitCommit:“4485c6f18cee9a5d3c3b4e523bd27972b1b53892”, GitTreeState:“clean”, BuildDate:“2020-03-04T03:38:58Z”, GoVersion:“go1.12.7”, Compiler:“gc”, Platform:“darwin/amd64”} Server Version: version.Info{Major:“1”, Minor:“14+”, GitVersion:“v1.14.8-5”, GitCommit:“$Format:%H$”, GitTreeState:“”, BuildDate:“2020-03-04T12:29:45Z”, GoVersion:“go1.13.5”, Compiler:“gc”, Platform:“linux/amd64”}
What did you do?
- create a network partition chaos.
- testing and verify program behavior.
- delete related crd.
- run step 1-3 in many round.
- network partition didn’t recover, iptable rules still remained in the pod.
What did you expect to see? network partition recover after deleting related custom resource.
“controllers.NetworkChaos Removing self” should appear in log but didn’t.
What did you see instead?
error log below
2020-03-05T01:17:30.322Z ERROR controllers.NetworkChaos unable to update chaosctl status {"reconciler": "networkchaos", "action": "partition", "error": "Operation cannot be fulfilled on networkchaos.pingcap.com \"client-pns-partition\": StorageError: invalid object, Code: 4, Key: /registry/pingcap.com/networkchaos/arch-test/client-pns-partition, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: cde47ac9-5e7e-11ea-b448-5254002aa8c4, UID in object meta: "}
github.com/go-logr/zapr.(*zapLogger).Error
/go/pkg/mod/github.com/go-logr/zapr@v0.1.0/zapr.go:128
github.com/pingcap/chaos-mesh/controllers/twophase.(*Reconciler).Reconcile
/src/controllers/twophase/types.go:176
github.com/pingcap/chaos-mesh/controllers/networkchaos.(*Reconciler).scheduleNetworkChaos
/src/controllers/networkchaos/types.go:86
github.com/pingcap/chaos-mesh/controllers/networkchaos.(*Reconciler).Reconcile
/src/controllers/networkchaos/types.go:55
github.com/pingcap/chaos-mesh/controllers.(*NetworkChaosReconciler).Reconcile
/src/controllers/networkchaos_controller.go:43
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.4.0/pkg/internal/controller/controller.go:256
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.4.0/pkg/internal/controller/controller.go:232
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.4.0/pkg/internal/controller/controller.go:211
k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1
/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20191121015412-41065c7a8c2a/pkg/util/wait/wait.go:152
k8s.io/apimachinery/pkg/util/wait.JitterUntil
/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20191121015412-41065c7a8c2a/pkg/util/wait/wait.go:153
k8s.io/apimachinery/pkg/util/wait.Until
/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20191121015412-41065c7a8c2a/pkg/util/wait/wait.go:88
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Comments: 20 (15 by maintainers)
problem solved with latest image. Thanks everyone!
one suggestion: tag image with version or commit id. I can only find latest version in dockerhub.