kubernetes: Failed logs during Kubernetes pvc/pv binding
What happened: I used storageclass to create volume, during pvc and pv binding, failed logs were reported. It happens every time when creating pvc/pv, so there are too many failed logs in my cluster. failed logs:
E1218 07:57:01.256167 1 pv_protection_controller.go:116] PV pvc-781a999f-029a-11e9-bdc2-fa163ef4d7a2 failed with : Operation cannot be fulfilled on persistentvolumes "pvc-781a999f-029a-11e9-bdc2-fa163ef4d7a2": the object has been modified; please apply your changes to the latest version and try again
E1218 07:57:31.230257 1 pv_protection_controller.go:116] PV pvc-8c5a675f-029a-11e9-bdc2-fa163ef4d7a2 failed with : Operation cannot be fulfilled on persistentvolumes "pvc-8c5a675f-029a-11e9-bdc2-fa163ef4d7a2": the object has been modified; please apply your changes to the latest version and try again
E1218 09:11:40.652570 1 pv_protection_controller.go:116] PV pvc-ee11e6c7-02a4-11e9-9a7d-fa163ef2d3bd failed with : Operation cannot be fulfilled on persistentvolumes "pvc-ee11e6c7-02a4-11e9-9a7d-fa163ef2d3bd": the object has been modified; please apply your changes to the latest version and try again
I opened kube-controller log level to 5 and got follow logs:
I0108 02:20:54.327308 1 pv_protection_controller.go:203] Got event on PV pvc-0205077c-12ec-11e9-8d34-fa163ea8fbb3
I0108 02:20:54.327443 1 pv_controller_base.go:535] storeObjectUpdate updating volume "pvc-0205077c-12ec-11e9-8d34-fa163ea8fbb3" with version 1008493
I0108 02:20:54.327505 1 pv_controller.go:824] volume "pvc-0205077c-12ec-11e9-8d34-fa163ea8fbb3" entered phase "Bound"
I0108 02:20:54.327548 1 pv_controller.go:931] updating PersistentVolumeClaim[default/pvc-test]: binding to "pvc-0205077c-12ec-11e9-8d34-fa163ea8fbb3"
I0108 02:20:54.327648 1 pv_controller.go:963] volume "pvc-0205077c-12ec-11e9-8d34-fa163ea8fbb3" bound to claim "default/pvc-test"
I0108 02:20:54.332311 1 pv_protection_controller.go:166] Error adding protection finalizer to PV pvc-0205077c-12ec-11e9-8d34-fa163ea8fbb3: Operation cannot be fulfilled on persistentvolumes "pvc-0205077c-12ec-11e9-8d34-fa163ea8fbb3": the object has been modified; please apply your changes to the latest version and try again
I0108 02:20:54.332519 1 pv_protection_controller.go:126] Finished processing PV pvc-0205077c-12ec-11e9-8d34-fa163ea8fbb3 (13.40545ms)
E0108 02:20:54.332648 1 pv_protection_controller.go:116] PV pvc-0205077c-12ec-11e9-8d34-fa163ea8fbb3 failed with : Operation cannot be fulfilled on persistentvolumes "pvc-0205077c-12ec-11e9-8d34-fa163ea8fbb3": the object has been modified; please apply your changes to the latest version and try again
I0108 02:20:54.332971 1 pv_controller.go:974] updating PersistentVolumeClaim[default/pvc-test]: bound to "pvc-0205077c-12ec-11e9-8d34-fa163ea8fbb3"
I0108 02:20:54.333386 1 pv_protection_controller.go:123] Processing PV pvc-0205077c-12ec-11e9-8d34-fa163ea8fbb3
......
I0108 02:20:54.340808 1 pv_controller.go:645] synchronizing PersistentVolume[pvc-0205077c-12ec-11e9-8d34-fa163ea8fbb3]: all is bound
I0108 02:20:54.340830 1 pv_controller.go:803] updating PersistentVolume[pvc-0205077c-12ec-11e9-8d34-fa163ea8fbb3]: set phase Bound
I0108 02:20:54.340854 1 pv_controller.go:806] updating PersistentVolume[pvc-0205077c-12ec-11e9-8d34-fa163ea8fbb3]: phase Bound already set
I0108 02:20:54.340876 1 pv_protection_controller.go:203] Got event on PV pvc-0205077c-12ec-11e9-8d34-fa163ea8fbb3
I0108 02:20:54.341004 1 pv_protection_controller.go:169] Added protection finalizer to PV pvc-0205077c-12ec-11e9-8d34-fa163ea8fbb3
I0108 02:20:54.341053 1 pv_protection_controller.go:126] Finished processing PV pvc-0205077c-12ec-11e9-8d34-fa163ea8fbb3 (7.543352ms)
I0108 02:20:54.341076 1 pv_protection_controller.go:123] Processing PV pvc-0205077c-12ec-11e9-8d34-fa163ea8fbb3
What you expected to happen: why it reported “Error adding protection finalizer to PV” at first time? why it reported failed logs? Error/failed logs during successful operations should not occur. How to reproduce it (as minimally and precisely as possible): It happens every time when i use k8s pvc/pv Anything else we need to know?:
Environment:
- Kubernetes version (use
kubectl version): v1.11.2 and v1.12.3
/sig Storage
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Reactions: 1
- Comments: 17 (3 by maintainers)
I am using k8s v1.13.4 now, the issue still exist: E0411 02:17:53.404234 1 pv_protection_controller.go:116] PV pvc-fd256216-5bff-11e9-a069-fa163ec82c85 failed with : Operation cannot be fulfilled on persistentvolumes “pvc-fd256216-5bff-11e9-a069-fa163ec82c85”: the object has been modified; please apply your changes to the latest version and try again
I have the same issue using cinder storage class. the volume is created but never attached. Did you guys found a fix for this?
running openstack rocky, kubernets deployed using magnum.
storage class: https://kubernetes.io/docs/concepts/storage/storage-classes/#openstack-cinder mysql deployment: https://kubernetes.io/docs/tutorials/stateful-application/mysql-wordpress-persistent-volume/ kubectl get sc,pv,pvc,pod --all-namespaces: https://paste.xinu.at/UNp/
latest log and nothing after it:
Having the same issue but the volume isn’t attached to any node in my case.
pv_controller tries to bound pvc, succeed, updates the claim and since the volume is actually detached pv_controller does the same thing next loop. I don’t see any other related errors except the one posted above.