lvm-localpv: Volumes go missing.
What steps did you take and what happened: Created a pvc, which got formatted and mounted just fine. Then booted the machine, after bootup the volume is not to be found! This causes loss of data and is very severe.
What did you expect to happen: Not loosing volumes!
The output of the following commands will help us better understand what’s going on: (Pasting long output into a GitHub gist or other Pastebin is fine.)
kubectl logs -f openebs-lvm-controller-0 -n kube-system -c openebs-lvm-plugin
I0919 21:07:44.486525 1 main.go:136] LVM Driver Version :- 0.8.0 - commit :- 929ae4439f2da71a2d6ee5bda6a33dd2f7d424fc
I0919 21:07:44.486615 1 main.go:137] DriverName: local.csi.openebs.io Plugin: controller EndPoint: unix:///var/lib/csi/sockets/pluginproxy/csi.sock NodeID: SetIOLimits: false ContainerRuntime: containerd RIopsPerGB: [] WIopsPerGB: [] RBpsPerGB: [] WBpsPerGB: []
I0919 21:07:44.486646 1 driver.go:48] enabling volume access mode: SINGLE_NODE_WRITER
I0919 21:07:44.496097 1 controller.go:212] waiting for k8s & lvm node informer caches to be synced
I0919 21:07:44.596266 1 controller.go:216] synced k8s & lvm node informer caches
I0919 21:07:44.596300 1 controller.go:218] initializing csi provisioning leak protection controller
I0919 21:07:44.596895 1 leak_protection.go:129] "starting up csi pvc controller"
I0919 21:07:44.596916 1 shared_informer.go:240] Waiting for caches to sync for CSI Provisioner
I0919 21:07:44.597275 1 grpc.go:190] Listening for connections on address: &net.UnixAddr{Name:"//var/lib/csi/sockets/pluginproxy/csi.sock", Net:"unix"}
I0919 21:07:44.697073 1 shared_informer.go:247] Caches are synced for CSI Provisioner
I0919 21:07:44.991733 1 grpc.go:72] GRPC call: /csi.v1.Identity/Probe requests {}
I0919 21:07:44.999709 1 grpc.go:81] GRPC response: {}
I0919 21:07:45.001288 1 grpc.go:72] GRPC call: /csi.v1.Identity/GetPluginInfo requests {}
I0919 21:07:45.001331 1 grpc.go:81] GRPC response: {"name":"local.csi.openebs.io","vendor_version":"0.8.0"}
I0919 21:07:45.002064 1 grpc.go:72] GRPC call: /csi.v1.Identity/GetPluginCapabilities requests {}
I0919 21:07:45.002142 1 grpc.go:81] GRPC response: {"capabilities":[{"Type":{"Service":{"type":1}}},{"Type":{"Service":{"type":2}}}]}
I0919 21:07:45.002973 1 grpc.go:72] GRPC call: /csi.v1.Controller/ControllerGetCapabilities requests {}
I0919 21:07:45.003009 1 grpc.go:81] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":9}}},{"Type":{"Rpc":{"type":5}}},{"Type":{"Rpc":{"type":4}}}]}
I0919 21:07:45.325452 1 grpc.go:72] GRPC call: /csi.v1.Identity/Probe requests {}
I0919 21:07:45.325666 1 grpc.go:81] GRPC response: {}
I0919 21:07:45.326637 1 grpc.go:72] GRPC call: /csi.v1.Identity/GetPluginInfo requests {}
I0919 21:07:45.326779 1 grpc.go:81] GRPC response: {"name":"local.csi.openebs.io","vendor_version":"0.8.0"}
I0919 21:07:45.327324 1 grpc.go:72] GRPC call: /csi.v1.Identity/GetPluginCapabilities requests {}
I0919 21:07:45.327354 1 grpc.go:81] GRPC response: {"capabilities":[{"Type":{"Service":{"type":1}}},{"Type":{"Service":{"type":2}}}]}
I0919 21:07:45.328054 1 grpc.go:72] GRPC call: /csi.v1.Controller/ControllerGetCapabilities requests {}
I0919 21:07:45.328135 1 grpc.go:81] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":9}}},{"Type":{"Rpc":{"type":5}}},{"Type":{"Rpc":{"type":4}}}]}
I0919 21:07:45.673459 1 grpc.go:72] GRPC call: /csi.v1.Identity/GetPluginInfo requests {}
I0919 21:07:45.673695 1 grpc.go:81] GRPC response: {"name":"local.csi.openebs.io","vendor_version":"0.8.0"}
I0919 21:07:45.675001 1 grpc.go:72] GRPC call: /csi.v1.Identity/Probe requests {}
I0919 21:07:45.675122 1 grpc.go:81] GRPC response: {}
I0919 21:07:45.675962 1 grpc.go:72] GRPC call: /csi.v1.Controller/ControllerGetCapabilities requests {}
I0919 21:07:45.676000 1 grpc.go:81] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":9}}},{"Type":{"Rpc":{"type":5}}},{"Type":{"Rpc":{"type":4}}}]}
....
I0919 22:20:15.437512 1 grpc.go:72] GRPC call: /csi.v1.Controller/GetCapacity requests {"accessible_topology":{"segments":{"kubernetes.io/hostname":"alt-ksx-g-c01oco06","openebs.io/nodename":"alt-ksx-g-c01oco06"}},"parameters":{"fsType":"xfs","shared":"yes","thinProvision":"no","volgroup":"datavg"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{}}]}
I0919 22:20:15.437667 1 grpc.go:81] GRPC response: {"available_capacity":261124784128}
I0919 22:20:15.438064 1 grpc.go:72] GRPC call: /csi.v1.Controller/GetCapacity requests {"accessible_topology":{"segments":{"kubernetes.io/hostname":"alt-eos-g-c01oco03","openebs.io/nodename":"alt-eos-g-c01oco03"}},"parameters":{"fsType":"xfs","shared":"yes","thinProvision":"no","volgroup":"datavg"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{}}]}
I0919 22:20:15.438195 1 grpc.go:81] GRPC response: {"available_capacity":322118352896}
I0919 22:20:15.438548 1 grpc.go:72] GRPC call: /csi.v1.Controller/GetCapacity requests {"accessible_topology":{"segments":{"kubernetes.io/hostname":"alt-ebs-g-c01oco01","openebs.io/nodename":"alt-ebs-g-c01oco01"}},"parameters":{"fsType":"ext4","shared":"yes","thinProvision":"no","volgroup":"datavg"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{}}]}
I0919 22:20:15.438674 1 grpc.go:81] GRPC response: {"available_capacity":322118352896}
I0919 22:20:15.439082 1 grpc.go:72] GRPC call: /csi.v1.Controller/GetCapacity requests {"accessible_topology":{"segments":{"kubernetes.io/hostname":"alt-ksx-g-c01oco09","openebs.io/nodename":"alt-ksx-g-c01oco09"}},"parameters":{"fsType":"ext4","shared":"yes","thinProvision":"no","volgroup":"datavg"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{}}]}
I0919 22:20:15.439263 1 grpc.go:81] GRPC response: {"available_capacity":2171097579520}
I0919 22:20:15.439687 1 grpc.go:72] GRPC call: /csi.v1.Controller/GetCapacity requests {"accessible_topology":{"segments":{"kubernetes.io/hostname":"alt-eos-g-c01oco02","openebs.io/nodename":"alt-eos-g-c01oco02"}},"parameters":{"fsType":"xfs","shared":"yes","thinProvision":"no","volgroup":"datavg"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{}}]}
I0919 22:20:15.439817 1 grpc.go:81] GRPC response: {"available_capacity":322118352896}
I0919 22:20:15.440257 1 grpc.go:72] GRPC call: /csi.v1.Controller/GetCapacity requests {"accessible_topology":{"segments":{"kubernetes.io/hostname":"alt-ksx-g-c01oco06","openebs.io/nodename":"alt-ksx-g-c01oco06"}},"parameters":{"fsType":"ext4","shared":"yes","thinProvision":"no","volgroup":"datavg"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{}}]}
I0919 22:20:15.440385 1 grpc.go:81] GRPC response: {"available_capacity":261124784128}
I0919 22:20:15.440940 1 grpc.go:72] GRPC call: /csi.v1.Controller/GetCapacity requests {"accessible_topology":{"segments":{"kubernetes.io/hostname":"alt-ksx-g-c01oco04","openebs.io/nodename":"alt-ksx-g-c01oco04"}},"parameters":{"fsType":"xfs","shared":"yes","thinProvision":"no","volgroup":"datavg"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{}}]}
I0919 22:20:15.441111 1 grpc.go:81] GRPC response: {"available_capacity":261124784128}
I0919 22:20:15.441526 1 grpc.go:72] GRPC call: /csi.v1.Controller/GetCapacity requests {"accessible_topology":{"segments":{"kubernetes.io/hostname":"alt-ksx-g-c01oco07","openebs.io/nodename":"alt-ksx-g-c01oco07"}},"parameters":{"fsType":"ext4","shared":"yes","thinProvision":"no","volgroup":"datavg"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{}}]}
I0919 22:20:15.441646 1 grpc.go:81] GRPC response: {"available_capacity":2171097579520}
I0919 22:20:15.441994 1 grpc.go:72] GRPC call: /csi.v1.Controller/GetCapacity requests {"accessible_topology":{"segments":{"kubernetes.io/hostname":"alt-ebs-g-c01oco01","openebs.io/nodename":"alt-ebs-g-c01oco01"}},"parameters":{"fsType":"xfs","shared":"yes","thinProvision":"no","volgroup":"datavg"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{}}]}
I0919 22:20:15.442129 1 grpc.go:81] GRPC response: {"available_capacity":322118352896}
kubectl logs -f openebs-lvm-node-[xxxx] -n kube-system -c openebs-lvm-pluginkubectl get pods -n kube-system
k -n openebs get pods
NAME READY STATUS RESTARTS AGE
openebs-admission-server-894884484-gcc5f 1/1 Running 0 29h
openebs-apiserver-6494d57b96-xfltt 1/1 Running 0 77m
openebs-localpv-provisioner-5787758b59-tzs8l 1/1 Running 1 29h
openebs-lvm-localpv-controller-0 5/5 Running 0 76m
openebs-lvm-localpv-node-7x2kw 2/2 Running 6 5d4h
openebs-lvm-localpv-node-98kxr 2/2 Running 6 5d4h
openebs-lvm-localpv-node-9r2kd 2/2 Running 20 5d4h
openebs-lvm-localpv-node-h949v 2/2 Running 6 5d4h
openebs-lvm-localpv-node-jff5w 2/2 Running 6 5d4h
openebs-lvm-localpv-node-kccx5 2/2 Running 6 5d4h
openebs-lvm-localpv-node-ln7r4 2/2 Running 6 5d4h
openebs-lvm-localpv-node-pzxw4 2/2 Running 6 5d4h
openebs-lvm-localpv-node-qxwn9 2/2 Running 11 5d4h
openebs-lvm-localpv-node-qzxzb 2/2 Running 6 5d4h
openebs-lvm-localpv-node-trfvk 2/2 Running 6 5d4h
openebs-lvm-localpv-node-w25kv 2/2 Running 6 5d4h
openebs-lvm-localpv-node-wxv6w 2/2 Running 6 5d4h
openebs-lvm-localpv-node-xkmzj 2/2 Running 6 5d4h
openebs-ndm-5rq88 1/1 Running 5 10d
openebs-ndm-ftnwd 1/1 Running 3 10d
openebs-ndm-k9z5w 1/1 Running 3 10d
openebs-ndm-mxzss 1/1 Running 3 10d
openebs-ndm-operator-7bd7455b5b-khrmt 1/1 Running 0 29h
openebs-ndm-p79zz 1/1 Running 3 10d
openebs-ndm-p7f77 1/1 Running 3 10d
openebs-ndm-phfjw 1/1 Running 3 10d
openebs-ndm-zsqmp 1/1 Running 3 10d
openebs-provisioner-9f948db4-ljndj 1/1 Running 1 29h
openebs-snapshot-operator-68cf9946d6-vc5ls 2/2 Running 0 77m
I0919 21:47:33.032487 1 main.go:136] LVM Driver Version :- 0.8.0 - commit :- 929ae4439f2da71a2d6ee5bda6a33dd2f7d424fc
I0919 21:47:33.032679 1 main.go:137] DriverName: local.csi.openebs.io Plugin: agent EndPoint: unix:///plugin/csi.sock NodeID: alt-ksx-g-c01oco03 SetIOLimits: false ContainerRuntime: containerd RIopsPerGB: [] WIopsPerGB: [] RBpsPerGB: [] WBpsPerGB: []
I0919 21:47:33.032725 1 driver.go:48] enabling volume access mode: SINGLE_NODE_WRITER
I0919 21:47:33.033388 1 grpc.go:190] Listening for connections on address: &net.UnixAddr{Name:"//plugin/csi.sock", Net:"unix"}
I0919 21:47:33.036307 1 builder.go:107] Creating event broadcaster
I0919 21:47:33.036946 1 builder.go:107] Creating event broadcaster
I0919 21:47:33.037042 1 snapshot.go:165] Starting Snap controller
I0919 21:47:33.037049 1 snapshot.go:168] Waiting for informer caches to sync
I0919 21:47:33.036950 1 volume.go:265] Starting Vol controller
I0919 21:47:33.037584 1 volume.go:268] Waiting for informer caches to sync
I0919 21:47:33.055970 1 volume.go:209] Got add event for Vol pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca
I0919 21:47:33.056056 1 volume.go:209] Got add event for Vol pvc-63ebce02-7077-48ea-88b3-301b1cf7909d
I0919 21:47:33.067860 1 builder.go:116] Creating event broadcaster
I0919 21:47:33.068028 1 lvmnode.go:191] Starting Node controller
I0919 21:47:33.068466 1 lvmnode.go:194] Waiting for informer caches to sync
I0919 21:47:33.071142 1 lvmnode.go:127] Got add event for lvm node openebs/alt-ksx-g-c01oco03
I0919 21:47:33.137317 1 snapshot.go:172] Starting Snap workers
I0919 21:47:33.137350 1 snapshot.go:179] Started Snap workers
I0919 21:47:33.137728 1 volume.go:272] Starting Vol workers
I0919 21:47:33.137738 1 volume.go:279] Started Vol workers
I0919 21:47:33.137830 1 volume.go:98] lvm volume already provisioned
I0919 21:47:33.137840 1 volume.go:337] Successfully synced 'openebs/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca'
I0919 21:47:33.137832 1 volume.go:98] lvm volume already provisioned
I0919 21:47:33.137904 1 volume.go:337] Successfully synced 'openebs/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d'
I0919 21:47:33.168649 1 lvmnode.go:199] Starting Node workers
I0919 21:47:33.168667 1 lvmnode.go:206] Started Node workers
I0919 21:47:33.489540 1 lvmnode.go:100] lvm node controller: node volume groups updated current=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s:511996Mi Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s:199676Mi Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0}], required=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]
I0919 21:47:33.489636 1 lvmnode.go:110] lvm node controller: updating node object with &{TypeMeta:{Kind:LVMNode APIVersion:local.openebs.io/v1alpha1} ObjectMeta:{Name:alt-ksx-g-c01oco03 GenerateName: Namespace:openebs SelfLink: UID:e14c6a98-a813-4250-b3a4-b9362444f91d ResourceVersion:1305723725 Generation:4 CreationTimestamp:2021-08-23 12:07:57 +0000 UTC DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[] Annotations:map[] OwnerReferences:[{APIVersion:v1 Kind:Node Name:alt-ksx-g-c01oco03 UID:c61ff898-2ecb-401c-9c1d-63f9acbbe57f Controller:0xc00026a060 BlockOwnerDeletion:<nil>}] Finalizers:[] ClusterName: ManagedFields:[{Manager:lvm-driver Operation:Update APIVersion:local.openebs.io/v1alpha1 Time:2021-08-23 12:07:57 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"c61ff898-2ecb-401c-9c1d-63f9acbbe57f\"}":{".":{},"f:apiVersion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:volumeGroups":{}}}]} VolumeGroups:[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]}
I0919 21:47:33.497190 1 lvmnode.go:114] lvm node controller: updated node object openebs/alt-ksx-g-c01oco03
I0919 21:47:33.497209 1 lvmnode.go:274] Successfully synced 'openebs/alt-ksx-g-c01oco03'
I0919 21:47:33.754135 1 grpc.go:72] GRPC call: /csi.v1.Identity/GetPluginInfo requests {}
I0919 21:47:33.755194 1 grpc.go:81] GRPC response: {"name":"local.csi.openebs.io","vendor_version":"0.8.0"}
I0919 21:47:35.327404 1 grpc.go:72] GRPC call: /csi.v1.Node/NodeGetInfo requests {}
I0919 21:47:35.334632 1 grpc.go:81] GRPC response: {"accessible_topology":{"segments":{"kubernetes.io/hostname":"alt-ksx-g-c01oco03","openebs.io/nodename":"alt-ksx-g-c01oco03"}},"node_id":"alt-ksx-g-c01oco03"}
I0919 21:47:36.692782 1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"alertmanager-main-2","csi.storage.k8s.io/pod.namespace":"openshift-monitoring","csi.storage.k8s.io/pod.uid":"2a1d2646-68b5-4916-99b3-18d3918d4831","csi.storage.k8s.io/serviceAccount.name":"alertmanager-main","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"datavg","storage.kubernetes.io/csiProvisionerIdentity":"1631990632881-8081-local.csi.openebs.io"},"volume_id":"pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca"}
I0919 21:47:36.762805 1 mount_linux.go:366] Disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca]
E0919 21:47:36.927823 1 mount_linux.go:372] format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
)
E0919 21:47:36.927901 1 mount.go:72] lvm: failed to mount volume /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca [ext4] to /var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount, error format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
)
E0919 21:47:36.927997 1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
)
I0919 21:47:37.493233 1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"alertmanager-main-2","csi.storage.k8s.io/pod.namespace":"openshift-monitoring","csi.storage.k8s.io/pod.uid":"2a1d2646-68b5-4916-99b3-18d3918d4831","csi.storage.k8s.io/serviceAccount.name":"alertmanager-main","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"datavg","storage.kubernetes.io/csiProvisionerIdentity":"1631990632881-8081-local.csi.openebs.io"},"volume_id":"pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca"}
I0919 21:47:37.498794 1 mount_linux.go:366] Disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca]
E0919 21:47:37.499905 1 mount_linux.go:372] format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
)
E0919 21:47:37.499975 1 mount.go:72] lvm: failed to mount volume /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca [ext4] to /var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount, error format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
)
E0919 21:47:37.500016 1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
)
I0919 21:47:37.818527 1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"vault-1","csi.storage.k8s.io/pod.namespace":"vault","csi.storage.k8s.io/pod.uid":"160b5143-fb8d-4758-a829-6c92d8c719d7","csi.storage.k8s.io/serviceAccount.name":"vault","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"datavg","storage.kubernetes.io/csiProvisionerIdentity":"1631990632881-8081-local.csi.openebs.io"},"volume_id":"pvc-63ebce02-7077-48ea-88b3-301b1cf7909d"}
I0919 21:47:37.823301 1 mount_linux.go:366] Disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d]
E0919 21:47:37.824462 1 mount_linux.go:372] format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
)
E0919 21:47:37.824478 1 mount.go:72] lvm: failed to mount volume /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d [ext4] to /var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount, error format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
)
E0919 21:47:37.824495 1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
)
I0919 21:47:38.411354 1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"vault-1","csi.storage.k8s.io/pod.namespace":"vault","csi.storage.k8s.io/pod.uid":"160b5143-fb8d-4758-a829-6c92d8c719d7","csi.storage.k8s.io/serviceAccount.name":"vault","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"datavg","storage.kubernetes.io/csiProvisionerIdentity":"1631990632881-8081-local.csi.openebs.io"},"volume_id":"pvc-63ebce02-7077-48ea-88b3-301b1cf7909d"}
I0919 21:47:38.417422 1 mount_linux.go:366] Disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d]
E0919 21:47:38.418586 1 mount_linux.go:372] format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
)
E0919 21:47:38.418630 1 mount.go:72] lvm: failed to mount volume /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d [ext4] to /var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount, error format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
)
E0919 21:47:38.418687 1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
)
I0919 21:47:38.520481 1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"alertmanager-main-2","csi.storage.k8s.io/pod.namespace":"openshift-monitoring","csi.storage.k8s.io/pod.uid":"2a1d2646-68b5-4916-99b3-18d3918d4831","csi.storage.k8s.io/serviceAccount.name":"alertmanager-main","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"datavg","storage.kubernetes.io/csiProvisionerIdentity":"1631990632881-8081-local.csi.openebs.io"},"volume_id":"pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca"}
I0919 21:47:38.525817 1 mount_linux.go:366] Disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca]
E0919 21:47:38.526895 1 mount_linux.go:372] format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
) ...
I0919 21:48:33.198169 1 lvmnode.go:100] lvm node controller: node volume groups updated current=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s:511996Mi Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s:199676Mi Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0}], required=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]
I0919 21:48:33.198235 1 lvmnode.go:110] lvm node controller: updating node object with &{TypeMeta:{Kind:LVMNode APIVersion:local.openebs.io/v1alpha1} ObjectMeta:{Name:alt-ksx-g-c01oco03 GenerateName: Namespace:openebs SelfLink: UID:e14c6a98-a813-4250-b3a4-b9362444f91d ResourceVersion:1305723725 Generation:4 CreationTimestamp:2021-08-23 12:07:57 +0000 UTC DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[] Annotations:map[] OwnerReferences:[{APIVersion:v1 Kind:Node Name:alt-ksx-g-c01oco03 UID:c61ff898-2ecb-401c-9c1d-63f9acbbe57f Controller:0xc0006224c9 BlockOwnerDeletion:<nil>}] Finalizers:[] ClusterName: ManagedFields:[{Manager:lvm-driver Operation:Update APIVersion:local.openebs.io/v1alpha1 Time:2021-08-23 12:07:57 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"c61ff898-2ecb-401c-9c1d-63f9acbbe57f\"}":{".":{},"f:apiVersion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:volumeGroups":{}}}]} VolumeGroups:[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]}
I0919 21:48:33.205156 1 lvmnode.go:114] lvm node controller: updated node object openebs/alt-ksx-g-c01oco03
I0919 21:48:33.205174 1 lvmnode.go:274] Successfully synced 'openebs/alt-ksx-g-c01oco03'
I0919 21:48:40.776987 1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"alertmanager-main-2","csi.storage.k8s.io/pod.namespace":"openshift-monitoring","csi.storage.k8s.io/pod.uid":"2a1d2646-68b5-4916-99b3-18d3918d4831","csi.storage.k8s.io/serviceAccount.name":"alertmanager-main","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"datavg","storage.kubernetes.io/csiProvisionerIdentity":"1631990632881-8081-local.csi.openebs.io"},"volume_id":"pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca"}
I0919 21:48:40.793493 1 mount_linux.go:366] Disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca]
E0919 21:48:40.794577 1 mount_linux.go:372] format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
)
E0919 21:48:40.794632 1 mount.go:72] lvm: failed to mount volume /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca [ext4] to /var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount, error format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
)
E0919 21:48:40.794757 1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
)
I0919 21:48:41.690835 1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"vault-1","csi.storage.k8s.io/pod.namespace":"vault","csi.storage.k8s.io/pod.uid":"160b5143-fb8d-4758-a829-6c92d8c719d7","csi.storage.k8s.io/serviceAccount.name":"vault","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"datavg","storage.kubernetes.io/csiProvisionerIdentity":"1631990632881-8081-local.csi.openebs.io"},"volume_id":"pvc-63ebce02-7077-48ea-88b3-301b1cf7909d"}
I0919 21:48:41.697249 1 mount_linux.go:366] Disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d]
E0919 21:48:41.698425 1 mount_linux.go:372] format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
)
E0919 21:48:41.698442 1 mount.go:72] lvm: failed to mount volume /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d [ext4] to /var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount, error format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
)
E0919 21:48:41.698463 1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
)
I0919 21:58:33.193143 1 lvmnode.go:100] lvm node controller: node volume groups updated current=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s:511996Mi Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s:199676Mi Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0}], required=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]
I0919 21:58:33.193387 1 lvmnode.go:110] lvm node controller: updating node object with &{TypeMeta:{Kind:LVMNode APIVersion:local.openebs.io/v1alpha1} ObjectMeta:{Name:alt-ksx-g-c01oco03 GenerateName: Namespace:openebs SelfLink: UID:e14c6a98-a813-4250-b3a4-b9362444f91d ResourceVersion:1305723725 Generation:4 CreationTimestamp:2021-08-23 12:07:57 +0000 UTC DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[] Annotations:map[] OwnerReferences:[{APIVersion:v1 Kind:Node Name:alt-ksx-g-c01oco03 UID:c61ff898-2ecb-401c-9c1d-63f9acbbe57f Controller:0xc00026bce8 BlockOwnerDeletion:<nil>}] Finalizers:[] ClusterName: ManagedFields:[{Manager:lvm-driver Operation:Update APIVersion:local.openebs.io/v1alpha1 Time:2021-08-23 12:07:57 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"c61ff898-2ecb-401c-9c1d-63f9acbbe57f\"}":{".":{},"f:apiVersion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:volumeGroups":{}}}]} VolumeGroups:[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]}
I0919 21:58:33.200591 1 lvmnode.go:114] lvm node controller: updated node object openebs/alt-ksx-g-c01oco03
I0919 21:58:33.200709 1 lvmnode.go:274] Successfully synced 'openebs/alt-ksx-g-c01oco03'
I0919 21:59:33.197599 1 lvmnode.go:100] lvm node controller: node volume groups updated current=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s:511996Mi Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s:199676Mi Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0}], required=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]
I0919 21:59:33.197854 1 lvmnode.go:110] lvm node controller: updating node object with &{TypeMeta:{Kind:LVMNode APIVersion:local.openebs.io/v1alpha1} ObjectMeta:{Name:alt-ksx-g-c01oco03 GenerateName: Namespace:openebs SelfLink: UID:e14c6a98-a813-4250-b3a4-b9362444f91d ResourceVersion:1305723725 Generation:4 CreationTimestamp:2021-08-23 12:07:57 +0000 UTC DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[] Annotations:map[] OwnerReferences:[{APIVersion:v1 Kind:Node Name:alt-ksx-g-c01oco03 UID:c61ff898-2ecb-401c-9c1d-63f9acbbe57f Controller:0xc0006320f9 BlockOwnerDeletion:<nil>}] Finalizers:[] ClusterName: ManagedFields:[{Manager:lvm-driver Operation:Update APIVersion:local.openebs.io/v1alpha1 Time:2021-08-23 12:07:57 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"c61ff898-2ecb-401c-9c1d-63f9acbbe57f\"}":{".":{},"f:apiVersion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:volumeGroups":{}}}]} VolumeGroups:[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]}
I0919 21:59:33.203510 1 lvmnode.go:114] lvm node controller: updated node object openebs/alt-ksx-g-c01oco03
I0919 21:59:33.203600 1 lvmnode.go:274] Successfully synced 'openebs/alt-ksx-g-c01oco03'
I0919 21:59:55.071045 1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"alertmanager-main-2","csi.storage.k8s.io/pod.namespace":"openshift-monitoring","csi.storage.k8s.io/pod.uid":"2a1d2646-68b5-4916-99b3-18d3918d4831","csi.storage.k8s.io/serviceAccount.name":"alertmanager-main","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"datavg","storage.kubernetes.io/csiProvisionerIdentity":"1631990632881-8081-local.csi.openebs.io"},"volume_id":"pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca"}
I0919 21:59:55.077098 1 mount_linux.go:366] Disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca]
E0919 21:59:55.078420 1 mount_linux.go:372] format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
)
E0919 21:59:55.078595 1 mount.go:72] lvm: failed to mount volume /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca [ext4] to /var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount, error format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
)
E0919 21:59:55.078709 1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
)
I0919 21:59:56.077218 1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"vault-1","csi.storage.k8s.io/pod.namespace":"vault","csi.storage.k8s.io/pod.uid":"160b5143-fb8d-4758-a829-6c92d8c719d7","csi.storage.k8s.io/serviceAccount.name":"vault","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"datavg","storage.kubernetes.io/csiProvisionerIdentity":"1631990632881-8081-local.csi.openebs.io"},"volume_id":"pvc-63ebce02-7077-48ea-88b3-301b1cf7909d"}
I0919 21:59:56.081863 1 mount_linux.go:366] Disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d]
E0919 21:59:56.083050 1 mount_linux.go:372] format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
)
E0919 21:59:56.083154 1 mount.go:72] lvm: failed to mount volume /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d [ext4] to /var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount, error format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
)
E0919 21:59:56.083251 1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
)
I0919 22:00:33.194333 1 lvmnode.go:100] lvm node controller: node volume groups updated current=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s:511996Mi Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s:199676Mi Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0}], required=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]
I0919 22:00:33.194546 1 lvmnode.go:110] lvm node controller: updating node object with &{TypeMeta:{Kind:LVMNode APIVersion:local.openebs.io/v1alpha1} ObjectMeta:{Name:alt-ksx-g-c01oco03 GenerateName: Namespace:openebs SelfLink: UID:e14c6a98-a813-4250-b3a4-b9362444f91d ResourceVersion:1305723725 Generation:4 CreationTimestamp:2021-08-23 12:07:57 +0000 UTC DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[] Annotations:map[] OwnerReferences:[{APIVersion:v1 Kind:Node Name:alt-ksx-g-c01oco03 UID:c61ff898-2ecb-401c-9c1d-63f9acbbe57f Controller:0xc000131069 BlockOwnerDeletion:<nil>}] Finalizers:[] ClusterName: ManagedFields:[{Manager:lvm-driver Operation:Update APIVersion:local.openebs.io/v1alpha1 Time:2021-08-23 12:07:57 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"c61ff898-2ecb-401c-9c1d-63f9acbbe57f\"}":{".":{},"f:apiVersion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:volumeGroups":{}}}]} VolumeGroups:[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]}
I0919 22:00:33.200632 1 lvmnode.go:114] lvm node controller: updated node object openebs/alt-ksx-g-c01oco03
I0919 22:00:33.200795 1 lvmnode.go:274] Successfully synced 'openebs/alt-ksx-g-c01oco03'
I0919 22:01:33.192396 1 lvmnode.go:100] lvm node controller: node volume groups updated current=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s:511996Mi Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s:199676Mi Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0}], required=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]
I0919 22:01:33.192458 1 lvmnode.go:110] lvm node controller: updating node object with &{TypeMeta:{Kind:LVMNode APIVersion:local.openebs.io/v1alpha1} ObjectMeta:{Name:alt-ksx-g-c01oco03 GenerateName: Namespace:openebs SelfLink: UID:e14c6a98-a813-4250-b3a4-b9362444f91d ResourceVersion:1305723725 Generation:4 CreationTimestamp:2021-08-23 12:07:57 +0000 UTC DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[] Annotations:map[] OwnerReferences:[{APIVersion:v1 Kind:Node Name:alt-ksx-g-c01oco03 UID:c61ff898-2ecb-401c-9c1d-63f9acbbe57f Controller:0xc0006a808a BlockOwnerDeletion:<nil>}] Finalizers:[] ClusterName: ManagedFields:[{Manager:lvm-driver Operation:Update APIVersion:local.openebs.io/v1alpha1 Time:2021-08-23 12:07:57 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"c61ff898-2ecb-401c-9c1d-63f9acbbe57f\"}":{".":{},"f:apiVersion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:volumeGroups":{}}}]} VolumeGroups:[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]}
I0919 22:01:33.199002 1 lvmnode.go:114] lvm node controller: updated node object openebs/alt-ksx-g-c01oco03
I0919 22:01:33.199019 1 lvmnode.go:274] Successfully synced 'openebs/alt-ksx-g-c01oco03'
I0919 22:01:57.130951 1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"alertmanager-main-2","csi.storage.k8s.io/pod.namespace":"openshift-monitoring","csi.storage.k8s.io/pod.uid":"2a1d2646-68b5-4916-99b3-18d3918d4831","csi.storage.k8s.io/serviceAccount.name":"alertmanager-main","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"datavg","storage.kubernetes.io/csiProvisionerIdentity":"1631990632881-8081-local.csi.openebs.io"},"volume_id":"pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca"}
I0919 22:01:57.136043 1 mount_linux.go:366] Disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca]
E0919 22:01:57.137124 1 mount_linux.go:372] format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
)
E0919 22:01:57.137250 1 mount.go:72] lvm: failed to mount volume /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca [ext4] to /var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount, error format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
)
E0919 22:01:57.137358 1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
)
I0919 22:01:58.139894 1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"vault-1","csi.storage.k8s.io/pod.namespace":"vault","csi.storage.k8s.io/pod.uid":"160b5143-fb8d-4758-a829-6c92d8c719d7","csi.storage.k8s.io/serviceAccount.name":"vault","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"datavg","storage.kubernetes.io/csiProvisionerIdentity":"1631990632881-8081-local.csi.openebs.io"},"volume_id":"pvc-63ebce02-7077-48ea-88b3-301b1cf7909d"}
I0919 22:01:58.145250 1 mount_linux.go:366] Disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d]
E0919 22:01:58.146369 1 mount_linux.go:372] format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
)
E0919 22:01:58.146436 1 mount.go:72] lvm: failed to mount volume /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d [ext4] to /var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount, error format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
)
E0919 22:01:58.146499 1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
)
I0919 22:02:33.198541 1 lvmnode.go:100] lvm node controller: node volume groups updated current=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s:511996Mi Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s:199676Mi Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0}], required=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]
I0919 22:02:33.198810 1 lvmnode.go:110] lvm node controller: updating node object with &{TypeMeta:{Kind:LVMNode APIVersion:local.openebs.io/v1alpha1} ObjectMeta:{Name:alt-ksx-g-c01oco03 GenerateName: Namespace:openebs SelfLink: UID:e14c6a98-a813-4250-b3a4-b9362444f91d ResourceVersion:1305723725 Generation:4 CreationTimestamp:2021-08-23 12:07:57 +0000 UTC DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[] Annotations:map[] OwnerReferences:[{APIVersion:v1 Kind:Node Name:alt-ksx-g-c01oco03 UID:c61ff898-2ecb-401c-9c1d-63f9acbbe57f Controller:0xc000632248 BlockOwnerDeletion:<nil>}] Finalizers:[] ClusterName: ManagedFields:[{Manager:lvm-driver Operation:Update APIVersion:local.openebs.io/v1alpha1 Time:2021-08-23 12:07:57 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"c61ff898-2ecb-401c-9c1d-63f9acbbe57f\"}":{".":{},"f:apiVersion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:volumeGroups":{}}}]} VolumeGroups:[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]}
I0919 22:02:33.219623 1 lvmnode.go:114] lvm node controller: updated node object openebs/alt-ksx-g-c01oco03
I0919 22:02:33.219775 1 lvmnode.go:274] Successfully synced 'openebs/alt-ksx-g-c01oco03'
I0919 22:03:33.197462 1 lvmnode.go:100] lvm node controller: node volume groups updated current=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s:511996Mi Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s:199676Mi Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0}], required=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]
I0919 22:03:33.197666 1 lvmnode.go:110] lvm node controller: updating node object with &{TypeMeta:{Kind:LVMNode APIVersion:local.openebs.io/v1alpha1} ObjectMeta:{Name:alt-ksx-g-c01oco03 GenerateName: Namespace:openebs SelfLink: UID:e14c6a98-a813-4250-b3a4-b9362444f91d ResourceVersion:1305723725 Generation:4 CreationTimestamp:2021-08-23 12:07:57 +0000 UTC DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[] Annotations:map[] OwnerReferences:[{APIVersion:v1 Kind:Node Name:alt-ksx-g-c01oco03 UID:c61ff898-2ecb-401c-9c1d-63f9acbbe57f Controller:0xc0006221bb BlockOwnerDeletion:<nil>}] Finalizers:[] ClusterName: ManagedFields:[{Manager:lvm-driver Operation:Update APIVersion:local.openebs.io/v1alpha1 Time:2021-08-23 12:07:57 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"c61ff898-2ecb-401c-9c1d-63f9acbbe57f\"}":{".":{},"f:apiVersion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:volumeGroups":{}}}]} VolumeGroups:[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]}
I0919 22:03:33.204957 1 lvmnode.go:114] lvm node controller: updated node object openebs/alt-ksx-g-c01oco03
I0919 22:03:33.205147 1 lvmnode.go:274] Successfully synced 'openebs/alt-ksx-g-c01oco03'
I0919 22:03:59.214157 1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"alertmanager-main-2","csi.storage.k8s.io/pod.namespace":"openshift-monitoring","csi.storage.k8s.io/pod.uid":"2a1d2646-68b5-4916-99b3-18d3918d4831","csi.storage.k8s.io/serviceAccount.name":"alertmanager-main","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"datavg","storage.kubernetes.io/csiProvisionerIdentity":"1631990632881-8081-local.csi.openebs.io"},"volume_id":"pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca"}
I0919 22:03:59.221739 1 mount_linux.go:366] Disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca]
E0919 22:03:59.223076 1 mount_linux.go:372] format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
)
E0919 22:03:59.223095 1 mount.go:72] lvm: failed to mount volume /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca [ext4] to /var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount, error format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
)
E0919 22:03:59.223124 1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
)
I0919 22:04:00.225863 1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"vault-1","csi.storage.k8s.io/pod.namespace":"vault","csi.storage.k8s.io/pod.uid":"160b5143-fb8d-4758-a829-6c92d8c719d7","csi.storage.k8s.io/serviceAccount.name":"vault","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"datavg","storage.kubernetes.io/csiProvisionerIdentity":"1631990632881-8081-local.csi.openebs.io"},"volume_id":"pvc-63ebce02-7077-48ea-88b3-301b1cf7909d"}
I0919 22:04:00.230920 1 mount_linux.go:366] Disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d]
E0919 22:04:00.232305 1 mount_linux.go:372] format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
)
E0919 22:04:00.232440 1 mount.go:72] lvm: failed to mount volume /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d [ext4] to /var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount, error format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
)
E0919 22:04:00.232540 1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
)
I0919 22:04:33.193373 1 lvmnode.go:100] lvm node controller: node volume groups updated current=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s:511996Mi Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s:199676Mi Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0}], required=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]
I0919 22:04:33.193540 1 lvmnode.go:110] lvm node controller: updating node object with &{TypeMeta:{Kind:LVMNode APIVersion:local.openebs.io/v1alpha1} ObjectMeta:{Name:alt-ksx-g-c01oco03 GenerateName: Namespace:openebs SelfLink: UID:e14c6a98-a813-4250-b3a4-b9362444f91d ResourceVersion:1305723725 Generation:4 CreationTimestamp:2021-08-23 12:07:57 +0000 UTC DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[] Annotations:map[] OwnerReferences:[{APIVersion:v1 Kind:Node Name:alt-ksx-g-c01oco03 UID:c61ff898-2ecb-401c-9c1d-63f9acbbe57f Controller:0xc000043278 BlockOwnerDeletion:<nil>}] Finalizers:[] ClusterName: ManagedFields:[{Manager:lvm-driver Operation:Update APIVersion:local.openebs.io/v1alpha1 Time:2021-08-23 12:07:57 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"c61ff898-2ecb-401c-9c1d-63f9acbbe57f\"}":{".":{},"f:apiVersion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:volumeGroups":{}}}]} VolumeGroups:[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]}
I0919 22:04:33.199806 1 lvmnode.go:114] lvm node controller: updated node object openebs/alt-ksx-g-c01oco03
I0919 22:04:33.199958 1 lvmnode.go:274] Successfully synced 'openebs/alt-ksx-g-c01oco03'
I0919 22:05:33.201598 1 lvmnode.go:100] lvm node controller: node volume groups updated current=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s:511996Mi Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s:199676Mi Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0}], required=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]
I0919 22:05:33.201968 1 lvmnode.go:110] lvm node controller: updating node object with &{TypeMeta:{Kind:LVMNode APIVersion:local.openebs.io/v1alpha1} ObjectMeta:{Name:alt-ksx-g-c01oco03 GenerateName: Namespace:openebs SelfLink: UID:e14c6a98-a813-4250-b3a4-b9362444f91d ResourceVersion:1305723725 Generation:4 CreationTimestamp:2021-08-23 12:07:57 +0000 UTC DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[] Annotations:map[] OwnerReferences:[{APIVersion:v1 Kind:Node Name:alt-ksx-g-c01oco03 UID:c61ff898-2ecb-401c-9c1d-63f9acbbe57f Controller:0xc0003e00aa BlockOwnerDeletion:<nil>}] Finalizers:[] ClusterName: ManagedFields:[{Manager:lvm-driver Operation:Update APIVersion:local.openebs.io/v1alpha1 Time:2021-08-23 12:07:57 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"c61ff898-2ecb-401c-9c1d-63f9acbbe57f\"}":{".":{},"f:apiVersion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:volumeGroups":{}}}]} VolumeGroups:[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]}
I0919 22:05:33.209690 1 lvmnode.go:114] lvm node controller: updated node object openebs/alt-ksx-g-c01oco03
I0919 22:05:33.209786 1 lvmnode.go:274] Successfully synced 'openebs/alt-ksx-g-c01oco03'
I0919 22:06:01.309472 1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"alertmanager-main-2","csi.storage.k8s.io/pod.namespace":"openshift-monitoring","csi.storage.k8s.io/pod.uid":"2a1d2646-68b5-4916-99b3-18d3918d4831","csi.storage.k8s.io/serviceAccount.name":"alertmanager-main","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"datavg","storage.kubernetes.io/csiProvisionerIdentity":"1631990632881-8081-local.csi.openebs.io"},"volume_id":"pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca"}
I0919 22:06:01.315687 1 mount_linux.go:366] Disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca]
E0919 22:06:01.316691 1 mount_linux.go:372] format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
)
E0919 22:06:01.316707 1 mount.go:72] lvm: failed to mount volume /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca [ext4] to /var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount, error format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
)
E0919 22:06:01.316734 1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
)
I0919 22:06:02.317247 1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"vault-1","csi.storage.k8s.io/pod.namespace":"vault","csi.storage.k8s.io/pod.uid":"160b5143-fb8d-4758-a829-6c92d8c719d7","csi.storage.k8s.io/serviceAccount.name":"vault","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"datavg","storage.kubernetes.io/csiProvisionerIdentity":"1631990632881-8081-local.csi.openebs.io"},"volume_id":"pvc-63ebce02-7077-48ea-88b3-301b1cf7909d"}
I0919 22:06:02.321985 1 mount_linux.go:366] Disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d]
E0919 22:06:02.323009 1 mount_linux.go:372] format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
)
E0919 22:06:02.323099 1 mount.go:72] lvm: failed to mount volume /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d [ext4] to /var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount, error format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
)
E0919 22:06:02.323181 1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
)
I0919 22:06:33.192108 1 lvmnode.go:100] lvm node controller: node volume groups updated current=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s:511996Mi Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s:199676Mi Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0}], required=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]
I0919 22:06:33.192174 1 lvmnode.go:110] lvm node controller: updating node object with &{TypeMeta:{Kind:LVMNode APIVersion:local.openebs.io/v1alpha1} ObjectMeta:{Name:alt-ksx-g-c01oco03 GenerateName: Namespace:openebs SelfLink: UID:e14c6a98-a813-4250-b3a4-b9362444f91d ResourceVersion:1305723725 Generation:4 CreationTimestamp:2021-08-23 12:07:57 +0000 UTC DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[] Annotations:map[] OwnerReferences:[{APIVersion:v1 Kind:Node Name:alt-ksx-g-c01oco03 UID:c61ff898-2ecb-401c-9c1d-63f9acbbe57f Controller:0xc000212090 BlockOwnerDeletion:<nil>}] Finalizers:[] ClusterName: ManagedFields:[{Manager:lvm-driver Operation:Update APIVersion:local.openebs.io/v1alpha1 Time:2021-08-23 12:07:57 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"c61ff898-2ecb-401c-9c1d-63f9acbbe57f\"}":{".":{},"f:apiVersion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:volumeGroups":{}}}]} VolumeGroups:[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]}
I0919 22:06:33.203863 1 lvmnode.go:114] lvm node controller: updated node object openebs/alt-ksx-g-c01oco03
I0919 22:06:33.203885 1 lvmnode.go:274] Successfully synced 'openebs/alt-ksx-g-c01oco03'
I0919 22:07:33.197599 1 lvmnode.go:100] lvm node controller: node volume groups updated current=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s:511996Mi Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s:199676Mi Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0}], required=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]
I0919 22:07:33.197883 1 lvmnode.go:110] lvm node controller: updating node object with &{TypeMeta:{Kind:LVMNode APIVersion:local.openebs.io/v1alpha1} ObjectMeta:{Name:alt-ksx-g-c01oco03 GenerateName: Namespace:openebs SelfLink: UID:e14c6a98-a813-4250-b3a4-b9362444f91d ResourceVersion:1305723725 Generation:4 CreationTimestamp:2021-08-23 12:07:57 +0000 UTC DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[] Annotations:map[] OwnerReferences:[{APIVersion:v1 Kind:Node Name:alt-ksx-g-c01oco03 UID:c61ff898-2ecb-401c-9c1d-63f9acbbe57f Controller:0xc000212000 BlockOwnerDeletion:<nil>}] Finalizers:[] ClusterName: ManagedFields:[{Manager:lvm-driver Operation:Update APIVersion:local.openebs.io/v1alpha1 Time:2021-08-23 12:07:57 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"c61ff898-2ecb-401c-9c1d-63f9acbbe57f\"}":{".":{},"f:apiVersion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:volumeGroups":{}}}]} VolumeGroups:[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]}
I0919 22:07:33.204772 1 lvmnode.go:114] lvm node controller: updated node object openebs/alt-ksx-g-c01oco03
I0919 22:07:33.204789 1 lvmnode.go:274] Successfully synced 'openebs/alt-ksx-g-c01oco03'
I0919 22:08:03.336415 1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"alertmanager-main-2","csi.storage.k8s.io/pod.namespace":"openshift-monitoring","csi.storage.k8s.io/pod.uid":"2a1d2646-68b5-4916-99b3-18d3918d4831","csi.storage.k8s.io/serviceAccount.name":"alertmanager-main","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"datavg","storage.kubernetes.io/csiProvisionerIdentity":"1631990632881-8081-local.csi.openebs.io"},"volume_id":"pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca"}
I0919 22:08:03.343673 1 mount_linux.go:366] Disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca]
E0919 22:08:03.344630 1 mount_linux.go:372] format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
)
E0919 22:08:03.344645 1 mount.go:72] lvm: failed to mount volume /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca [ext4] to /var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount, error format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
)
E0919 22:08:03.344675 1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
)
I0919 22:08:04.353996 1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"vault-1","csi.storage.k8s.io/pod.namespace":"vault","csi.storage.k8s.io/pod.uid":"160b5143-fb8d-4758-a829-6c92d8c719d7","csi.storage.k8s.io/serviceAccount.name":"vault","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"datavg","storage.kubernetes.io/csiProvisionerIdentity":"1631990632881-8081-local.csi.openebs.io"},"volume_id":"pvc-63ebce02-7077-48ea-88b3-301b1cf7909d"}
I0919 22:08:04.364752 1 mount_linux.go:366] Disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d]
E0919 22:08:04.365823 1 mount_linux.go:372] format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
)
E0919 22:08:04.365942 1 mount.go:72] lvm: failed to mount volume /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d [ext4] to /var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount, error format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
)
E0919 22:08:04.366035 1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
)
I0919 22:08:33.195376 1 lvmnode.go:100] lvm node controller: node volume groups updated current=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s:511996Mi Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s:199676Mi Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0}], required=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]
I0919 22:08:33.195672 1 lvmnode.go:110] lvm node controller: updating node object with &{TypeMeta:{Kind:LVMNode APIVersion:local.openebs.io/v1alpha1} ObjectMeta:{Name:alt-ksx-g-c01oco03 GenerateName: Namespace:openebs SelfLink: UID:e14c6a98-a813-4250-b3a4-b9362444f91d ResourceVersion:1305723725 Generation:4 CreationTimestamp:2021-08-23 12:07:57 +0000 UTC DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[] Annotations:map[] OwnerReferences:[{APIVersion:v1 Kind:Node Name:alt-ksx-g-c01oco03 UID:c61ff898-2ecb-401c-9c1d-63f9acbbe57f Controller:0xc0006ac7e8 BlockOwnerDeletion:<nil>}] Finalizers:[] ClusterName: ManagedFields:[{Manager:lvm-driver Operation:Update APIVersion:local.openebs.io/v1alpha1 Time:2021-08-23 12:07:57 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"c61ff898-2ecb-401c-9c1d-63f9acbbe57f\"}":{".":{},"f:apiVersion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:volumeGroups":{}}}]} VolumeGroups:[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]}
I0919 22:08:33.205158 1 lvmnode.go:114] lvm node controller: updated node object openebs/alt-ksx-g-c01oco03
I0919 22:08:33.205327 1 lvmnode.go:274] Successfully synced 'openebs/alt-ksx-g-c01oco03'
I0919 22:09:33.195347 1 lvmnode.go:100] lvm node controller: node volume groups updated current=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s:511996Mi Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s:199676Mi Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0}], required=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]
I0919 22:09:33.195593 1 lvmnode.go:110] lvm node controller: updating node object with &{TypeMeta:{Kind:LVMNode APIVersion:local.openebs.io/v1alpha1} ObjectMeta:{Name:alt-ksx-g-c01oco03 GenerateName: Namespace:openebs SelfLink: UID:e14c6a98-a813-4250-b3a4-b9362444f91d ResourceVersion:1305723725 Generation:4 CreationTimestamp:2021-08-23 12:07:57 +0000 UTC DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[] Annotations:map[] OwnerReferences:[{APIVersion:v1 Kind:Node Name:alt-ksx-g-c01oco03 UID:c61ff898-2ecb-401c-9c1d-63f9acbbe57f Controller:0xc0005901da BlockOwnerDeletion:<nil>}] Finalizers:[] ClusterName: ManagedFields:[{Manager:lvm-driver Operation:Update APIVersion:local.openebs.io/v1alpha1 Time:2021-08-23 12:07:57 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"c61ff898-2ecb-401c-9c1d-63f9acbbe57f\"}":{".":{},"f:apiVersion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:volumeGroups":{}}}]} VolumeGroups:[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]}
I0919 22:09:33.204502 1 lvmnode.go:114] lvm node controller: updated node object openebs/alt-ksx-g-c01oco03
I0919 22:09:33.204669 1 lvmnode.go:274] Successfully synced 'openebs/alt-ksx-g-c01oco03'
I0919 22:10:05.398785 1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"alertmanager-main-2","csi.storage.k8s.io/pod.namespace":"openshift-monitoring","csi.storage.k8s.io/pod.uid":"2a1d2646-68b5-4916-99b3-18d3918d4831","csi.storage.k8s.io/serviceAccount.name":"alertmanager-main","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"datavg","storage.kubernetes.io/csiProvisionerIdentity":"1631990632881-8081-local.csi.openebs.io"},"volume_id":"pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca"}
I0919 22:10:05.405224 1 mount_linux.go:366] Disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca]
E0919 22:10:05.406422 1 mount_linux.go:372] format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
)
E0919 22:10:05.406541 1 mount.go:72] lvm: failed to mount volume /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca [ext4] to /var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount, error format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
)
E0919 22:10:05.406634 1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
)
I0919 22:10:06.407071 1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"vault-1","csi.storage.k8s.io/pod.namespace":"vault","csi.storage.k8s.io/pod.uid":"160b5143-fb8d-4758-a829-6c92d8c719d7","csi.storage.k8s.io/serviceAccount.name":"vault","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"datavg","storage.kubernetes.io/csiProvisionerIdentity":"1631990632881-8081-local.csi.openebs.io"},"volume_id":"pvc-63ebce02-7077-48ea-88b3-301b1cf7909d"}
I0919 22:10:06.412408 1 mount_linux.go:366] Disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d]
E0919 22:10:06.413575 1 mount_linux.go:372] format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
)
E0919 22:10:06.413705 1 mount.go:72] lvm: failed to mount volume /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d [ext4] to /var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount, error format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
)
E0919 22:10:06.413795 1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
)
I0919 22:10:33.195388 1 lvmnode.go:100] lvm node controller: node volume groups updated current=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s:511996Mi Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s:199676Mi Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0}], required=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]
I0919 22:10:33.195460 1 lvmnode.go:110] lvm node controller: updating node object with &{TypeMeta:{Kind:LVMNode APIVersion:local.openebs.io/v1alpha1} ObjectMeta:{Name:alt-ksx-g-c01oco03 GenerateName: Namespace:openebs SelfLink: UID:e14c6a98-a813-4250-b3a4-b9362444f91d ResourceVersion:1305723725 Generation:4 CreationTimestamp:2021-08-23 12:07:57 +0000 UTC DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[] Annotations:map[] OwnerReferences:[{APIVersion:v1 Kind:Node Name:alt-ksx-g-c01oco03 UID:c61ff898-2ecb-401c-9c1d-63f9acbbe57f Controller:0xc000622978 BlockOwnerDeletion:<nil>}] Finalizers:[] ClusterName: ManagedFields:[{Manager:lvm-driver Operation:Update APIVersion:local.openebs.io/v1alpha1 Time:2021-08-23 12:07:57 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"c61ff898-2ecb-401c-9c1d-63f9acbbe57f\"}":{".":{},"f:apiVersion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:volumeGroups":{}}}]} VolumeGroups:[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]}
I0919 22:10:33.209769 1 lvmnode.go:114] lvm node controller: updated node object openebs/alt-ksx-g-c01oco03
I0919 22:10:33.209910 1 lvmnode.go:274] Successfully synced 'openebs/alt-ksx-g-c01oco03'
I0919 22:11:33.195346 1 lvmnode.go:100] lvm node controller: node volume groups updated current=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s:511996Mi Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s:199676Mi Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0}], required=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]
I0919 22:11:33.195445 1 lvmnode.go:110] lvm node controller: updating node object with &{TypeMeta:{Kind:LVMNode APIVersion:local.openebs.io/v1alpha1} ObjectMeta:{Name:alt-ksx-g-c01oco03 GenerateName: Namespace:openebs SelfLink: UID:e14c6a98-a813-4250-b3a4-b9362444f91d ResourceVersion:1305723725 Generation:4 CreationTimestamp:2021-08-23 12:07:57 +0000 UTC DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[] Annotations:map[] OwnerReferences:[{APIVersion:v1 Kind:Node Name:alt-ksx-g-c01oco03 UID:c61ff898-2ecb-401c-9c1d-63f9acbbe57f Controller:0xc0000421f0 BlockOwnerDeletion:<nil>}] Finalizers:[] ClusterName: ManagedFields:[{Manager:lvm-driver Operation:Update APIVersion:local.openebs.io/v1alpha1 Time:2021-08-23 12:07:57 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"c61ff898-2ecb-401c-9c1d-63f9acbbe57f\"}":{".":{},"f:apiVersion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:volumeGroups":{}}}]} VolumeGroups:[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]}
I0919 22:11:33.206672 1 lvmnode.go:114] lvm node controller: updated node object openebs/alt-ksx-g-c01oco03
I0919 22:11:33.206754 1 lvmnode.go:274] Successfully synced 'openebs/alt-ksx-g-c01oco03'
I0919 22:12:07.452479 1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"alertmanager-main-2","csi.storage.k8s.io/pod.namespace":"openshift-monitoring","csi.storage.k8s.io/pod.uid":"2a1d2646-68b5-4916-99b3-18d3918d4831","csi.storage.k8s.io/serviceAccount.name":"alertmanager-main","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"datavg","storage.kubernetes.io/csiProvisionerIdentity":"1631990632881-8081-local.csi.openebs.io"},"volume_id":"pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca"}
I0919 22:12:07.457948 1 mount_linux.go:366] Disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca]
E0919 22:12:07.459041 1 mount_linux.go:372] format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
)
E0919 22:12:07.459053 1 mount.go:72] lvm: failed to mount volume /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca [ext4] to /var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount, error format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
)
E0919 22:12:07.459073 1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
)
I0919 22:12:08.468111 1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"vault-1","csi.storage.k8s.io/pod.namespace":"vault","csi.storage.k8s.io/pod.uid":"160b5143-fb8d-4758-a829-6c92d8c719d7","csi.storage.k8s.io/serviceAccount.name":"vault","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"datavg","storage.kubernetes.io/csiProvisionerIdentity":"1631990632881-8081-local.csi.openebs.io"},"volume_id":"pvc-63ebce02-7077-48ea-88b3-301b1cf7909d"}
I0919 22:12:08.473579 1 mount_linux.go:366] Disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d]
E0919 22:12:08.474930 1 mount_linux.go:372] format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
)
E0919 22:12:08.475057 1 mount.go:72] lvm: failed to mount volume /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d [ext4] to /var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount, error format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
)
E0919 22:12:08.475153 1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
)
I0919 22:12:33.199283 1 lvmnode.go:100] lvm node controller: node volume groups updated current=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s:511996Mi Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s:199676Mi Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0}], required=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]
I0919 22:12:33.199554 1 lvmnode.go:110] lvm node controller: updating node object with &{TypeMeta:{Kind:LVMNode APIVersion:local.openebs.io/v1alpha1} ObjectMeta:{Name:alt-ksx-g-c01oco03 GenerateName: Namespace:openebs SelfLink: UID:e14c6a98-a813-4250-b3a4-b9362444f91d ResourceVersion:1305723725 Generation:4 CreationTimestamp:2021-08-23 12:07:57 +0000 UTC DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[] Annotations:map[] OwnerReferences:[{APIVersion:v1 Kind:Node Name:alt-ksx-g-c01oco03 UID:c61ff898-2ecb-401c-9c1d-63f9acbbe57f Controller:0xc0001ab668 BlockOwnerDeletion:<nil>}] Finalizers:[] ClusterName: ManagedFields:[{Manager:lvm-driver Operation:Update APIVersion:local.openebs.io/v1alpha1 Time:2021-08-23 12:07:57 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"c61ff898-2ecb-401c-9c1d-63f9acbbe57f\"}":{".":{},"f:apiVersion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:volumeGroups":{}}}]} VolumeGroups:[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]}
I0919 22:12:33.208417 1 lvmnode.go:114] lvm node controller: updated node object openebs/alt-ksx-g-c01oco03
I0919 22:12:33.208537 1 lvmnode.go:274] Successfully synced 'openebs/alt-ksx-g-c01oco03'
I0919 22:13:33.194398 1 lvmnode.go:100] lvm node controller: node volume groups updated current=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s:511996Mi Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s:199676Mi Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0}], required=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]
I0919 22:13:33.194604 1 lvmnode.go:110] lvm node controller: updating node object with &{TypeMeta:{Kind:LVMNode APIVersion:local.openebs.io/v1alpha1} ObjectMeta:{Name:alt-ksx-g-c01oco03 GenerateName: Namespace:openebs SelfLink: UID:e14c6a98-a813-4250-b3a4-b9362444f91d ResourceVersion:1305723725 Generation:4 CreationTimestamp:2021-08-23 12:07:57 +0000 UTC DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[] Annotations:map[] OwnerReferences:[{APIVersion:v1 Kind:Node Name:alt-ksx-g-c01oco03 UID:c61ff898-2ecb-401c-9c1d-63f9acbbe57f Controller:0xc0001aa000 BlockOwnerDeletion:<nil>}] Finalizers:[] ClusterName: ManagedFields:[{Manager:lvm-driver Operation:Update APIVersion:local.openebs.io/v1alpha1 Time:2021-08-23 12:07:57 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"c61ff898-2ecb-401c-9c1d-63f9acbbe57f\"}":{".":{},"f:apiVersion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:volumeGroups":{}}}]} VolumeGroups:[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]}
I0919 22:13:33.202748 1 lvmnode.go:114] lvm node controller: updated node object openebs/alt-ksx-g-c01oco03
I0919 22:13:33.202825 1 lvmnode.go:274] Successfully synced 'openebs/alt-ksx-g-c01oco03'
I0919 22:14:09.561523 1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"alertmanager-main-2","csi.storage.k8s.io/pod.namespace":"openshift-monitoring","csi.storage.k8s.io/pod.uid":"2a1d2646-68b5-4916-99b3-18d3918d4831","csi.storage.k8s.io/serviceAccount.name":"alertmanager-main","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"datavg","storage.kubernetes.io/csiProvisionerIdentity":"1631990632881-8081-local.csi.openebs.io"},"volume_id":"pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca"}
I0919 22:14:09.566213 1 mount_linux.go:366] Disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca]
E0919 22:14:09.567230 1 mount_linux.go:372] format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
)
E0919 22:14:09.567328 1 mount.go:72] lvm: failed to mount volume /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca [ext4] to /var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount, error format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
)
E0919 22:14:09.567438 1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
)
I0919 22:14:10.571841 1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"vault-1","csi.storage.k8s.io/pod.namespace":"vault","csi.storage.k8s.io/pod.uid":"160b5143-fb8d-4758-a829-6c92d8c719d7","csi.storage.k8s.io/serviceAccount.name":"vault","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"datavg","storage.kubernetes.io/csiProvisionerIdentity":"1631990632881-8081-local.csi.openebs.io"},"volume_id":"pvc-63ebce02-7077-48ea-88b3-301b1cf7909d"}
I0919 22:14:10.577746 1 mount_linux.go:366] Disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d]
E0919 22:14:10.579041 1 mount_linux.go:372] format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
)
E0919 22:14:10.579115 1 mount.go:72] lvm: failed to mount volume /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d [ext4] to /var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount, error format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
)
E0919 22:14:10.579204 1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
)
I0919 22:14:33.194217 1 lvmnode.go:100] lvm node controller: node volume groups updated current=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s:511996Mi Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s:199676Mi Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0}], required=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]
I0919 22:14:33.194488 1 lvmnode.go:110] lvm node controller: updating node object with &{TypeMeta:{Kind:LVMNode APIVersion:local.openebs.io/v1alpha1} ObjectMeta:{Name:alt-ksx-g-c01oco03 GenerateName: Namespace:openebs SelfLink: UID:e14c6a98-a813-4250-b3a4-b9362444f91d ResourceVersion:1305723725 Generation:4 CreationTimestamp:2021-08-23 12:07:57 +0000 UTC DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[] Annotations:map[] OwnerReferences:[{APIVersion:v1 Kind:Node Name:alt-ksx-g-c01oco03 UID:c61ff898-2ecb-401c-9c1d-63f9acbbe57f Controller:0xc0001300e9 BlockOwnerDeletion:<nil>}] Finalizers:[] ClusterName: ManagedFields:[{Manager:lvm-driver Operation:Update APIVersion:local.openebs.io/v1alpha1 Time:2021-08-23 12:07:57 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"c61ff898-2ecb-401c-9c1d-63f9acbbe57f\"}":{".":{},"f:apiVersion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:volumeGroups":{}}}]} VolumeGroups:[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]}
I0919 22:14:33.203751 1 lvmnode.go:114] lvm node controller: updated node object openebs/alt-ksx-g-c01oco03
I0919 22:14:33.203929 1 lvmnode.go:274] Successfully synced 'openebs/alt-ksx-g-c01oco03'
I0919 22:15:33.197368 1 lvmnode.go:100] lvm node controller: node volume groups updated current=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s:511996Mi Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s:199676Mi Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0}], required=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:531498008576 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:1 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]
I0919 22:15:33.197592 1 lvmnode.go:110] lvm node controller: updating node object with &{TypeMeta:{Kind:LVMNode APIVersion:local.openebs.io/v1alpha1} ObjectMeta:{Name:alt-ksx-g-c01oco03 GenerateName: Namespace:openebs SelfLink: UID:e14c6a98-a813-4250-b3a4-b9362444f91d ResourceVersion:1305723725 Generation:4 CreationTimestamp:2021-08-23 12:07:57 +0000 UTC DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[] Annotations:map[] OwnerReferences:[{APIVersion:v1 Kind:Node Name:alt-ksx-g-c01oco03 UID:c61ff898-2ecb-401c-9c1d-63f9acbbe57f Controller:0xc00026a0c9 BlockOwnerDeletion:<nil>}] Finalizers:[] ClusterName: ManagedFields:[{Manager:lvm-driver Operation:Update APIVersion:local.openebs.io/v1alpha1 Time:2021-08-23 12:07:57 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"c61ff898-2ecb-401c-9c1d-63f9acbbe57f\"}":{".":{},"f:apiVersion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:volumeGroups":{}}}]} VolumeGroups:[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:531498008576 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:1 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]}
I0919 22:15:33.204602 1 lvmnode.go:114] lvm node controller: updated node object openebs/alt-ksx-g-c01oco03
I0919 22:15:33.204716 1 lvmnode.go:274] Successfully synced 'openebs/alt-ksx-g-c01oco03'
I0919 22:15:33.204919 1 lvmnode.go:139] Got update event for lvm node openebs/alt-ksx-g-c01oco03
I0919 22:15:33.227390 1 lvmnode.go:100] lvm node controller: node volume groups updated current=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s:511996Mi Format:BinarySI} Free:{i:{value:531498008576 scale:0} d:{Dec:<nil>} s:506876Mi Format:BinarySI} LVCount:1 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0}], required=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:531498008576 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:1 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]
I0919 22:15:33.227723 1 lvmnode.go:110] lvm node controller: updating node object with &{TypeMeta:{Kind: APIVersion:} ObjectMeta:{Name:alt-ksx-g-c01oco03 GenerateName: Namespace:openebs SelfLink: UID:e14c6a98-a813-4250-b3a4-b9362444f91d ResourceVersion:1305975699 Generation:5 CreationTimestamp:2021-08-23 12:07:57 +0000 UTC DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[] Annotations:map[] OwnerReferences:[{APIVersion:v1 Kind:Node Name:alt-ksx-g-c01oco03 UID:c61ff898-2ecb-401c-9c1d-63f9acbbe57f Controller:0xc00026a6f0 BlockOwnerDeletion:<nil>}] Finalizers:[] ClusterName: ManagedFields:[{Manager:lvm-driver Operation:Update APIVersion:local.openebs.io/v1alpha1 Time:2021-08-23 12:07:57 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"c61ff898-2ecb-401c-9c1d-63f9acbbe57f\"}":{".":{},"f:apiVersion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:volumeGroups":{}}}]} VolumeGroups:[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:531498008576 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:1 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]}
I0919 22:15:33.237650 1 lvmnode.go:114] lvm node controller: updated node object openebs/alt-ksx-g-c01oco03
I0919 22:15:33.237836 1 lvmnode.go:274] Successfully synced 'openebs/alt-ksx-g-c01oco03'
I0919 22:16:11.616127 1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"alertmanager-main-2","csi.storage.k8s.io/pod.namespace":"openshift-monitoring","csi.storage.k8s.io/pod.uid":"2a1d2646-68b5-4916-99b3-18d3918d4831","csi.storage.k8s.io/serviceAccount.name":"alertmanager-main","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"datavg","storage.kubernetes.io/csiProvisionerIdentity":"1631990632881-8081-local.csi.openebs.io"},"volume_id":"pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca"}
I0919 22:16:11.621182 1 mount_linux.go:366] Disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca]
E0919 22:16:11.622289 1 mount_linux.go:372] format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
)
E0919 22:16:11.622376 1 mount.go:72] lvm: failed to mount volume /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca [ext4] to /var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount, error format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
)
E0919 22:16:11.622450 1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
)
I0919 22:16:12.629181 1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"vault-1","csi.storage.k8s.io/pod.namespace":"vault","csi.storage.k8s.io/pod.uid":"160b5143-fb8d-4758-a829-6c92d8c719d7","csi.storage.k8s.io/serviceAccount.name":"vault","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"datavg","storage.kubernetes.io/csiProvisionerIdentity":"1631990632881-8081-local.csi.openebs.io"},"volume_id":"pvc-63ebce02-7077-48ea-88b3-301b1cf7909d"}
I0919 22:16:12.634759 1 mount_linux.go:366] Disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d]
E0919 22:16:12.638315 1 mount_linux.go:372] format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
)
E0919 22:16:12.638330 1 mount.go:72] lvm: failed to mount volume /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d [ext4] to /var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount, error format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
)
E0919 22:16:12.638354 1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
)
I0919 22:16:33.194321 1 lvmnode.go:100] lvm node controller: node volume groups updated current=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s:511996Mi Format:BinarySI} Free:{i:{value:531498008576 scale:0} d:{Dec:<nil>} s:506876Mi Format:BinarySI} LVCount:1 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0}], required=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:531498008576 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:1 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]
I0919 22:16:33.194582 1 lvmnode.go:110] lvm node controller: updating node object with &{TypeMeta:{Kind: APIVersion:} ObjectMeta:{Name:alt-ksx-g-c01oco03 GenerateName: Namespace:openebs SelfLink: UID:e14c6a98-a813-4250-b3a4-b9362444f91d ResourceVersion:1305975699 Generation:5 CreationTimestamp:2021-08-23 12:07:57 +0000 UTC DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[] Annotations:map[] OwnerReferences:[{APIVersion:v1 Kind:Node Name:alt-ksx-g-c01oco03 UID:c61ff898-2ecb-401c-9c1d-63f9acbbe57f Controller:0xc0001ab10d BlockOwnerDeletion:<nil>}] Finalizers:[] ClusterName: ManagedFields:[{Manager:lvm-driver Operation:Update APIVersion:local.openebs.io/v1alpha1 Time:2021-08-23 12:07:57 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"c61ff898-2ecb-401c-9c1d-63f9acbbe57f\"}":{".":{},"f:apiVersion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:volumeGroups":{}}}]} VolumeGroups:[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:531498008576 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:1 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]}
I0919 22:16:33.200940 1 lvmnode.go:114] lvm node controller: updated node object openebs/alt-ksx-g-c01oco03
I0919 22:16:33.201043 1 lvmnode.go:274] Successfully synced 'openebs/alt-ksx-g-c01oco03'
I0919 22:17:33.195250 1 lvmnode.go:100] lvm node controller: node volume groups updated current=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s:511996Mi Format:BinarySI} Free:{i:{value:531498008576 scale:0} d:{Dec:<nil>} s:506876Mi Format:BinarySI} LVCount:1 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0}], required=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:531498008576 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:1 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]
I0919 22:17:33.195449 1 lvmnode.go:110] lvm node controller: updating node object with &{TypeMeta:{Kind: APIVersion:} ObjectMeta:{Name:alt-ksx-g-c01oco03 GenerateName: Namespace:openebs SelfLink: UID:e14c6a98-a813-4250-b3a4-b9362444f91d ResourceVersion:1305975699 Generation:5 CreationTimestamp:2021-08-23 12:07:57 +0000 UTC DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[] Annotations:map[] OwnerReferences:[{APIVersion:v1 Kind:Node Name:alt-ksx-g-c01oco03 UID:c61ff898-2ecb-401c-9c1d-63f9acbbe57f Controller:0xc000800000 BlockOwnerDeletion:<nil>}] Finalizers:[] ClusterName: ManagedFields:[{Manager:lvm-driver Operation:Update APIVersion:local.openebs.io/v1alpha1 Time:2021-08-23 12:07:57 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"c61ff898-2ecb-401c-9c1d-63f9acbbe57f\"}":{".":{},"f:apiVersion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:volumeGroups":{}}}]} VolumeGroups:[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:531498008576 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:1 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]}
I0919 22:17:33.203085 1 lvmnode.go:114] lvm node controller: updated node object openebs/alt-ksx-g-c01oco03
I0919 22:17:33.203146 1 lvmnode.go:274] Successfully synced 'openebs/alt-ksx-g-c01oco03'
I0919 22:18:13.705268 1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"alertmanager-main-2","csi.storage.k8s.io/pod.namespace":"openshift-monitoring","csi.storage.k8s.io/pod.uid":"2a1d2646-68b5-4916-99b3-18d3918d4831","csi.storage.k8s.io/serviceAccount.name":"alertmanager-main","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"datavg","storage.kubernetes.io/csiProvisionerIdentity":"1631990632881-8081-local.csi.openebs.io"},"volume_id":"pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca"}
I0919 22:18:13.710264 1 mount_linux.go:366] Disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca]
E0919 22:18:13.711379 1 mount_linux.go:372] format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
)
E0919 22:18:13.711399 1 mount.go:72] lvm: failed to mount volume /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca [ext4] to /var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount, error format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
)
E0919 22:18:13.711423 1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
)
I0919 22:18:14.712131 1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"vault-1","csi.storage.k8s.io/pod.namespace":"vault","csi.storage.k8s.io/pod.uid":"160b5143-fb8d-4758-a829-6c92d8c719d7","csi.storage.k8s.io/serviceAccount.name":"vault","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"datavg","storage.kubernetes.io/csiProvisionerIdentity":"1631990632881-8081-local.csi.openebs.io"},"volume_id":"pvc-63ebce02-7077-48ea-88b3-301b1cf7909d"}
I0919 22:18:14.717477 1 mount_linux.go:366] Disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d]
E0919 22:18:14.718574 1 mount_linux.go:372] format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
)
E0919 22:18:14.718671 1 mount.go:72] lvm: failed to mount volume /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d [ext4] to /var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount, error format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
)
E0919 22:18:14.718739 1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
)
I0919 22:18:33.197552 1 lvmnode.go:100] lvm node controller: node volume groups updated current=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s:511996Mi Format:BinarySI} Free:{i:{value:531498008576 scale:0} d:{Dec:<nil>} s:506876Mi Format:BinarySI} LVCount:1 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0}], required=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:531498008576 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:1 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]
I0919 22:18:33.197616 1 lvmnode.go:110] lvm node controller: updating node object with &{TypeMeta:{Kind: APIVersion:} ObjectMeta:{Name:alt-ksx-g-c01oco03 GenerateName: Namespace:openebs SelfLink: UID:e14c6a98-a813-4250-b3a4-b9362444f91d ResourceVersion:1305975699 Generation:5 CreationTimestamp:2021-08-23 12:07:57 +0000 UTC DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[] Annotations:map[] OwnerReferences:[{APIVersion:v1 Kind:Node Name:alt-ksx-g-c01oco03 UID:c61ff898-2ecb-401c-9c1d-63f9acbbe57f Controller:0xc000212598 BlockOwnerDeletion:<nil>}] Finalizers:[] ClusterName: ManagedFields:[{Manager:lvm-driver Operation:Update APIVersion:local.openebs.io/v1alpha1 Time:2021-08-23 12:07:57 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"c61ff898-2ecb-401c-9c1d-63f9acbbe57f\"}":{".":{},"f:apiVersion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:volumeGroups":{}}}]} VolumeGroups:[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:531498008576 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:1 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]}
I0919 22:18:33.203316 1 lvmnode.go:114] lvm node controller: updated node object openebs/alt-ksx-g-c01oco03
I0919 22:18:33.203334 1 lvmnode.go:274] Successfully synced 'openebs/alt-ksx-g-c01oco03'
I0919 22:19:33.198358 1 lvmnode.go:100] lvm node controller: node volume groups updated current=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s:511996Mi Format:BinarySI} Free:{i:{value:531498008576 scale:0} d:{Dec:<nil>} s:506876Mi Format:BinarySI} LVCount:1 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0}], required=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:531498008576 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:1 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]
I0919 22:19:33.198425 1 lvmnode.go:110] lvm node controller: updating node object with &{TypeMeta:{Kind: APIVersion:} ObjectMeta:{Name:alt-ksx-g-c01oco03 GenerateName: Namespace:openebs SelfLink: UID:e14c6a98-a813-4250-b3a4-b9362444f91d ResourceVersion:1305975699 Generation:5 CreationTimestamp:2021-08-23 12:07:57 +0000 UTC DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[] Annotations:map[] OwnerReferences:[{APIVersion:v1 Kind:Node Name:alt-ksx-g-c01oco03 UID:c61ff898-2ecb-401c-9c1d-63f9acbbe57f Controller:0xc000130000 BlockOwnerDeletion:<nil>}] Finalizers:[] ClusterName: ManagedFields:[{Manager:lvm-driver Operation:Update APIVersion:local.openebs.io/v1alpha1 Time:2021-08-23 12:07:57 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"c61ff898-2ecb-401c-9c1d-63f9acbbe57f\"}":{".":{},"f:apiVersion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:volumeGroups":{}}}]} VolumeGroups:[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:531498008576 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:1 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]}
I0919 22:19:33.204133 1 lvmnode.go:114] lvm node controller: updated node object openebs/alt-ksx-g-c01oco03
I0919 22:19:33.204295 1 lvmnode.go:274] Successfully synced 'openebs/alt-ksx-g-c01oco03'
I0919 22:20:15.780801 1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"alertmanager-main-2","csi.storage.k8s.io/pod.namespace":"openshift-monitoring","csi.storage.k8s.io/pod.uid":"2a1d2646-68b5-4916-99b3-18d3918d4831","csi.storage.k8s.io/serviceAccount.name":"alertmanager-main","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"datavg","storage.kubernetes.io/csiProvisionerIdentity":"1631990632881-8081-local.csi.openebs.io"},"volume_id":"pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca"}
I0919 22:20:15.785438 1 mount_linux.go:366] Disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca]
E0919 22:20:15.786634 1 mount_linux.go:372] format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
)
E0919 22:20:15.786799 1 mount.go:72] lvm: failed to mount volume /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca [ext4] to /var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount, error format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
)
E0919 22:20:15.786892 1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
)
I0919 22:20:16.798260 1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"vault-1","csi.storage.k8s.io/pod.namespace":"vault","csi.storage.k8s.io/pod.uid":"160b5143-fb8d-4758-a829-6c92d8c719d7","csi.storage.k8s.io/serviceAccount.name":"vault","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"datavg","storage.kubernetes.io/csiProvisionerIdentity":"1631990632881-8081-local.csi.openebs.io"},"volume_id":"pvc-63ebce02-7077-48ea-88b3-301b1cf7909d"}
I0919 22:20:16.805115 1 mount_linux.go:366] Disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d]
E0919 22:20:16.806421 1 mount_linux.go:372] format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
)
E0919 22:20:16.806537 1 mount.go:72] lvm: failed to mount volume /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d [ext4] to /var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount, error format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
)
E0919 22:20:16.806623 1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
)
E0919 22:22:17.863627 1 mount.go:72] lvm: failed to mount volume /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca [ext4] to /var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount, error format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
)
E0919 22:22:17.863684 1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
)
I0919 22:22:18.863589 1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"vault-1","csi.storage.k8s.io/pod.namespace":"vault","csi.storage.k8s.io/pod.uid":"160b5143-fb8d-4758-a829-6c92d8c719d7","csi.storage.k8s.io/serviceAccount.name":"vault","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"datavg","storage.kubernetes.io/csiProvisionerIdentity":"1631990632881-8081-local.csi.openebs.io"},"volume_id":"pvc-63ebce02-7077-48ea-88b3-301b1cf7909d"}
I0919 22:22:18.869062 1 mount_linux.go:366] Disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d]
E0919 22:22:18.870243 1 mount_linux.go:372] format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
)
E0919 22:22:18.870264 1 mount.go:72] lvm: failed to mount volume /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d [ext4] to /var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount, error format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
)
E0919 22:22:18.870295 1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
)
I0919 22:22:33.195255 1 lvmnode.go:100] lvm node controller: node volume groups updated current=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s:511996Mi Format:BinarySI} Free:{i:{value:531498008576 scale:0} d:{Dec:<nil>} s:506876Mi Format:BinarySI} LVCount:1 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0}], required=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:531498008576 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:1 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]
I0919 22:22:33.195326 1 lvmnode.go:110] lvm node controller: updating node object with &{TypeMeta:{Kind: APIVersion:} ObjectMeta:{Name:alt-ksx-g-c01oco03 GenerateName: Namespace:openebs SelfLink: UID:e14c6a98-a813-4250-b3a4-b9362444f91d ResourceVersion:1305975699 Generation:5 CreationTimestamp:2021-08-23 12:07:57 +0000 UTC DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[] Annotations:map[] OwnerReferences:[{APIVersion:v1 Kind:Node Name:alt-ksx-g-c01oco03 UID:c61ff898-2ecb-401c-9c1d-63f9acbbe57f Controller:0xc0006ac5ea BlockOwnerDeletion:<nil>}] Finalizers:[] ClusterName: ManagedFields:[{Manager:lvm-driver Operation:Update APIVersion:local.openebs.io/v1alpha1 Time:2021-08-23 12:07:57 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"c61ff898-2ecb-401c-9c1d-63f9acbbe57f\"}":{".":{},"f:apiVersion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:volumeGroups":{}}}]} VolumeGroups:[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:531498008576 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:1 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]}
I0919 22:22:33.201762 1 lvmnode.go:114] lvm node controller: updated node object openebs/alt-ksx-g-c01oco03
I0919 22:22:33.201935 1 lvmnode.go:274] Successfully synced 'openebs/alt-ksx-g-c01oco03'
I0919 22:23:33.197380 1 lvmnode.go:100] lvm node controller: node volume groups updated current=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s:511996Mi Format:BinarySI} Free:{i:{value:531498008576 scale:0} d:{Dec:<nil>} s:506876Mi Format:BinarySI} LVCount:1 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0}], required=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:531498008576 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:1 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]
I0919 22:23:33.197443 1 lvmnode.go:110] lvm node controller: updating node object with &{TypeMeta:{Kind: APIVersion:} ObjectMeta:{Name:alt-ksx-g-c01oco03 GenerateName: Namespace:openebs SelfLink: UID:e14c6a98-a813-4250-b3a4-b9362444f91d ResourceVersion:1305975699 Generation:5 CreationTimestamp:2021-08-23 12:07:57 +0000 UTC DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[] Annotations:map[] OwnerReferences:[{APIVersion:v1 Kind:Node Name:alt-ksx-g-c01oco03 UID:c61ff898-2ecb-401c-9c1d-63f9acbbe57f Controller:0xc0000421f8 BlockOwnerDeletion:<nil>}] Finalizers:[] ClusterName: ManagedFields:[{Manager:lvm-driver Operation:Update APIVersion:local.openebs.io/v1alpha1 Time:2021-08-23 12:07:57 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"c61ff898-2ecb-401c-9c1d-63f9acbbe57f\"}":{".":{},"f:apiVersion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:volumeGroups":{}}}]} VolumeGroups:[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:531498008576 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:1 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]}
I0919 22:23:33.204502 1 lvmnode.go:114] lvm node controller: updated node object openebs/alt-ksx-g-c01oco03
I0919 22:23:33.204519 1 lvmnode.go:274] Successfully synced 'openebs/alt-ksx-g-c01oco03'
I0919 22:24:19.937883 1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"alertmanager-main-2","csi.storage.k8s.io/pod.namespace":"openshift-monitoring","csi.storage.k8s.io/pod.uid":"2a1d2646-68b5-4916-99b3-18d3918d4831","csi.storage.k8s.io/serviceAccount.name":"alertmanager-main","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"datavg","storage.kubernetes.io/csiProvisionerIdentity":"1631990632881-8081-local.csi.openebs.io"},"volume_id":"pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca"}
I0919 22:24:19.942809 1 mount_linux.go:366] Disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca]
E0919 22:24:19.943870 1 mount_linux.go:372] format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
)
...
I0919 22:25:33.199426 1 lvmnode.go:110] lvm node controller: updating node object with &{TypeMeta:{Kind: APIVersion:} ObjectMeta:{Name:alt-ksx-g-c01oco03 GenerateName: Namespace:openebs SelfLink: UID:e14c6a98-a813-4250-b3a4-b9362444f91d ResourceVersion:1305975699 Generation:5 CreationTimestamp:2021-08-23 12:07:57 +0000 UTC DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[] Annotations:map[] OwnerReferences:[{APIVersion:v1 Kind:Node Name:alt-ksx-g-c01oco03 UID:c61ff898-2ecb-401c-9c1d-63f9acbbe57f Controller:0xc000130079 BlockOwnerDeletion:<nil>}] Finalizers:[] ClusterName: ManagedFields:[{Manager:lvm-driver Operation:Update APIVersion:local.openebs.io/v1alpha1 Time:2021-08-23 12:07:57 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"c61ff898-2ecb-401c-9c1d-63f9acbbe57f\"}":{".":{},"f:apiVersion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:volumeGroups":{}}}]} VolumeGroups:[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:531498008576 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:1 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]}
I0919 22:25:33.220225 1 lvmnode.go:114] lvm node controller: updated node object openebs/alt-ksx-g-c01oco03
I0919 22:25:33.220325 1 lvmnode.go:274] Successfully synced 'openebs/alt-ksx-g-c01oco03'
kubectl get lvmvol -A -o yaml
apiVersion: v1
items:
- apiVersion: local.openebs.io/v1alpha1
kind: LVMVolume
metadata:
creationTimestamp: "2021-09-19T16:58:40Z"
finalizers:
- lvm.openebs.io/finalizer
generation: 3
labels:
kubernetes.io/nodename: alt-ksx-g-c01oco03
name: pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca
namespace: openebs
resourceVersion: "1304748995"
uid: 303e29b0-39f7-413a-9a1a-cecd32757061
spec:
capacity: "1073741824"
ownerNodeID: alt-ksx-g-c01oco03
shared: "yes"
thinProvision: "no"
vgPattern: ^datavg$
volGroup: datavg
status:
state: Ready
- apiVersion: local.openebs.io/v1alpha1
kind: LVMVolume
metadata:
creationTimestamp: "2021-09-19T17:48:47Z"
finalizers:
- lvm.openebs.io/finalizer
generation: 3
labels:
kubernetes.io/nodename: alt-ksx-g-c01oco03
name: pvc-63ebce02-7077-48ea-88b3-301b1cf7909d
namespace: openebs
resourceVersion: "1304943732"
uid: 79c320fc-4a6f-4c0e-b02e-0d433d9cf74f
spec:
capacity: "1073741824"
ownerNodeID: alt-ksx-g-c01oco03
shared: "yes"
thinProvision: "no"
vgPattern: ^datavg$
volGroup: datavg
status:
state: Ready
Anything else you would like to add: [Miscellaneous information that will assist in solving the issue.]
pvscan --cache
[root@alt-ksx-g-c01oco03 ~]# lvscan
ACTIVE '/dev/datavg/openebs' [5.00 GiB] inherit
ACTIVE '/dev/rootvg/swap_lv' [2.00 GiB] inherit
ACTIVE '/dev/rootvg/tmp_lv' [<8.79 GiB] inherit
ACTIVE '/dev/rootvg/var_lv' [150.00 GiB] inherit
ACTIVE '/dev/rootvg/home_lv' [1000.00 MiB] inherit
ACTIVE '/dev/rootvg/root_lv' [<6.84 GiB] inherit
k get pvc alertmanager-main-db-alertmanager-main-2 -o yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
annotations:
pv.kubernetes.io/bind-completed: "yes"
pv.kubernetes.io/bound-by-controller: "yes"
volume.beta.kubernetes.io/storage-provisioner: local.csi.openebs.io
volume.kubernetes.io/selected-node: alt-ksx-g-c01oco03
creationTimestamp: "2021-09-19T16:58:40Z"
finalizers:
- kubernetes.io/pvc-protection
labels:
alertmanager: main
app: alertmanager
app.kubernetes.io/instance: main
app.kubernetes.io/managed-by: prometheus-operator
app.kubernetes.io/name: alertmanager
app.kubernetes.io/version: 0.21.0
name: alertmanager-main-db-alertmanager-main-2
namespace: openshift-monitoring
resourceVersion: "1304749087"
uid: 630461fc-7dbb-4db1-a9fb-0b6f4035c6ca
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
storageClassName: openebs-lvmpv-ext4
volumeMode: Filesystem
volumeName: pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca
status:
accessModes:
- ReadWriteOnce
capacity:
storage: 1Gi
phase: Bound
k describe pod alertmanager-main-2 |tail -10
Warning FailedMount 34m kubelet Unable to attach or mount volumes: unmounted volumes=[alertmanager-main-db], unattached volumes=[alertmanager-trusted-ca-bundle kube-api-access-7drhb config-volume tls-assets alertmanager-main-db secret-alertmanager-main-tls secret-alertmanager-main-proxy secret-alertmanager-kube-rbac-proxy]: timed out waiting for the condition
Warning FailedMount 28m kubelet Unable to attach or mount volumes: unmounted volumes=[alertmanager-main-db], unattached volumes=[secret-alertmanager-main-proxy secret-alertmanager-kube-rbac-proxy alertmanager-trusted-ca-bundle kube-api-access-7drhb config-volume tls-assets alertmanager-main-db secret-alertmanager-main-tls]: timed out waiting for the condition
Warning FailedMount 23m (x4 over 39m) kubelet Unable to attach or mount volumes: unmounted volumes=[alertmanager-main-db], unattached volumes=[secret-alertmanager-kube-rbac-proxy alertmanager-trusted-ca-bundle kube-api-access-7drhb config-volume tls-assets alertmanager-main-db secret-alertmanager-main-tls secret-alertmanager-main-proxy]: timed out waiting for the condition
Warning FailedMount 21m kubelet Unable to attach or mount volumes: unmounted volumes=[alertmanager-main-db], unattached volumes=[alertmanager-main-db secret-alertmanager-main-tls secret-alertmanager-main-proxy secret-alertmanager-kube-rbac-proxy alertmanager-trusted-ca-bundle kube-api-access-7drhb config-volume tls-assets]: timed out waiting for the condition
Warning FailedMount 19m kubelet Unable to attach or mount volumes: unmounted volumes=[alertmanager-main-db], unattached volumes=[tls-assets alertmanager-main-db secret-alertmanager-main-tls secret-alertmanager-main-proxy secret-alertmanager-kube-rbac-proxy alertmanager-trusted-ca-bundle kube-api-access-7drhb config-volume]: timed out waiting for the condition
Warning FailedMount 10m (x23 over 41m) kubelet MountVolume.SetUp failed for volume "pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" : rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
)
Warning FailedMount 5m35s kubelet Unable to attach or mount volumes: unmounted volumes=[alertmanager-main-db], unattached volumes=[kube-api-access-7drhb config-volume tls-assets alertmanager-main-db secret-alertmanager-main-tls secret-alertmanager-main-proxy secret-alertmanager-kube-rbac-proxy alertmanager-trusted-ca-bundle]: timed out waiting for the condition
Warning FailedMount 62s (x2 over 30m) kubelet Unable to attach or mount volumes: unmounted volumes=[alertmanager-main-db], unattached volumes=[config-volume tls-assets alertmanager-main-db secret-alertmanager-main-tls secret-alertmanager-main-proxy secret-alertmanager-kube-rbac-proxy alertmanager-trusted-ca-bundle kube-api-access-7drhb]: timed out waiting for the condition
k get pv|grep pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca
pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca 1Gi RWO Delete Bound openshift-monitoring/alertmanager-main-db-alertmanager-main-2 openebs-lvmpv-ext4 5h30m
k get pv pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca -o yaml
apiVersion: v1
kind: PersistentVolume
metadata:
annotations:
pv.kubernetes.io/provisioned-by: local.csi.openebs.io
creationTimestamp: "2021-09-19T16:58:41Z"
finalizers:
- kubernetes.io/pv-protection
name: pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca
resourceVersion: "1304749080"
uid: c1f1c9dd-6440-459a-9e08-213e68657e45
spec:
accessModes:
- ReadWriteOnce
capacity:
storage: 1Gi
claimRef:
apiVersion: v1
kind: PersistentVolumeClaim
name: alertmanager-main-db-alertmanager-main-2
namespace: openshift-monitoring
resourceVersion: "1304748971"
uid: 630461fc-7dbb-4db1-a9fb-0b6f4035c6ca
csi:
driver: local.csi.openebs.io
fsType: ext4
volumeAttributes:
openebs.io/cas-type: localpv-lvm
openebs.io/volgroup: datavg
storage.kubernetes.io/csiProvisionerIdentity: 1631990632881-8081-local.csi.openebs.io
volumeHandle: pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: openebs.io/nodename
operator: In
values:
- alt-ksx-g-c01oco03
persistentVolumeReclaimPolicy: Delete
storageClassName: openebs-lvmpv-ext4
volumeMode: Filesystem
status:
phase: Bound
oot@alt-ksx-g-c01oco03 ~]# lvm version
LVM version: 2.02.187(2)-RHEL7 (2020-03-24)
Library version: 1.02.170-RHEL7 (2020-03-24)
Driver version: 4.37.1
Configuration: ./configure --build=x86_64-redhat-linux-gnu --host=x86_64-redhat-linux-gnu --program-prefix= --disable-dependency-tracking --prefix=/usr --exec-prefix=/usr --bindir=/usr/bin --sbindir=/usr/sbin --sysconfdir=/etc --datadir=/usr/share --includedir=/usr/include --libdir=/usr/lib64 --libexecdir=/usr/libexec --localstatedir=/var --sharedstatedir=/var/lib --mandir=/usr/share/man --infodir=/usr/share/info --with-default-dm-run-dir=/run --with-default-run-dir=/run/lvm --with-default-pid-dir=/run --with-default-locking-dir=/run/lock/lvm --with-usrlibdir=/usr/lib64 --enable-lvm1_fallback --enable-fsadm --with-pool=internal --enable-write_install --with-user= --with-group= --with-device-uid=0 --with-device-gid=6 --with-device-mode=0660 --enable-pkgconfig --enable-applib --enable-cmdlib --enable-dmeventd --enable-blkid_wiping --enable-python2-bindings --with-cluster=internal --with-clvmd=corosync --enable-cmirrord --with-udevdir=/usr/lib/udev/rules.d --enable-udev_sync --with-thin=internal --enable-lvmetad --with-cache=internal --enable-lvmpolld --enable-lvmlockd-dlm --enable-lvmlockd-sanlock --enable-dmfilemapd
[root@alt-ksx-g-c01oco03 ~]#
Environment:
- LVM Driver version:
- Kubernetes version (use
kubectl version): Server Version: version.Info{Major:“1”, Minor:“21”, GitVersion:“v1.21.1+9807387”, GitCommit:“98073871f173baaa04dc2bafab50effd62c308a6”, GitTreeState:“clean”, BuildDate:“2021-08-06T14:06:43Z”, GoVersion:“go1.16.6”, Compiler:“gc”, Platform:“linux/amd64”} - Kubernetes installer & version: openshift 4.8
- Cloud provider or hardware configuration: n/a
- OS (e.g. from
/etc/os-release): RHEL 7.x
About this issue
- Original URL
- State: open
- Created 3 years ago
- Reactions: 1
- Comments: 31 (4 by maintainers)
@davidkarlsen @gunzy83 In fact, we have reproduced this issue more than once. Procedure
After runing above steps circularly ,there are chances to reproduct the issue. The lvs will display <9 LVs , and the lost LV will disappear forever after reboot.
Ok, so I’ve tried creating a bunch of volumes and rebooting machines. Two changes:
I haven’t seen the problem since, so I think we close this and I’ll reopen if I can provoke it consistently.