longhorn: [BUG] Reboot node while volume expansion, will cause pod stuck at creating state

To avoid this issue was caused by environment, tested in a fresh environment with v1.4.x head images deployed, did 3 times test with steps, there were 2 times can reproduce this error.

Hi @shuo-wu , after using storage class with parameter -O ^metadata_csum added below, I can not reproduce volume corruption state

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: longhorn-test
provisioner: driver.longhorn.io
allowVolumeExpansion: true
reclaimPolicy: Delete
volumeBindingMode: Immediate
parameters:
  numberOfReplicas: "3"
  staleReplicaTimeout: "2880"
  fromBackup: ""
  fsType: "ext4"
  mkfsParams: "-O ^metadata_csum"

@innobead Right now the conclusion is, we can not directly fix this issue and the filesystem corruption may be hard to fix (I didn’t reproduce this case). But as I mentioned above, disabling metadata_csum can bypass this issue. Now I prefer to write a KB to inform this case and all workarounds (manual fsck or disabling metadata_csum at the beginning), and the upstream PR link, then close this ticket.

I tried do restore the superblock by steps and the volume still not get repaired, below was the reply when execute sudo e2fsck -b block_number /dev/xxx

e2fsck: Bad magic number in super-block while trying to open /dev/longhorn/pvc-1e4e47bb-a720-4438-88f4-ec3e719c9555

In addition, calculated md5sum of volume head image and volume snap images in all replicas, the result were all the same. Because they were all the same, I did not perform replica repair, thank you

It seems the mount point should be the target when running fsck as per the doc.

At least for EXT4, running fsck <mount point> would report error code 16 - usage or syntax error. I don’t think we can fix this issue by executing fsck before NodeExpandVolume

Some notes

1. Power outage right before superblock checksum update?
2. Data mismatch between replicas?
3. There are some patches recently fixing the superblock checksum error caused by ext4 volume online resize, e.g. PR. Not sure if it is related to this issue, because Longhorn re-expands the volume after the node comes back. Need more time to investigate it.

For a workaround, we can try if the error can be fixed by fsck.

cc @chriscchien @shuo-wu @innobead

In the PR, it mentioned that the reproducing step is online resizing is performed twice consecutively. But during the node reboot, we don’t know (and it’s hard to know) if the node really executes fs resizing before completely shutting down.

Based on my test, this checksum mismatching issue happens during the NodeStageVolume resizing. After this execution, running dumpe2fs <volume device path> would encounter the error, for example:

longhorn-csi-plugin-7lpkp:/ # dumpe2fs -h /dev/longhorn/pvc-b229a6a6-6338-4118-809a-b7c9110047c9
dumpe2fs 1.46.4 (18-Aug-2021)
Filesystem volume name:   <none>
Last mounted on:          <not available>
Filesystem UUID:          3d32fa57-f873-49b0-9061-3e451ef70a52
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal ext_attr resize_inode dir_index filetype needs_recovery extent 64bit flex_bg sparse_super large_file huge_file dir_nlink extra_isize metadata_csum
Filesystem flags:         signed_directory_hash
Default mount options:    user_xattr acl
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              196608
Block count:              786432
Reserved block count:     0
Overhead clusters:        30262
Free blocks:              756164
Free inodes:              196597
First block:              0
Block size:               4096
Fragment size:            4096
Group descriptor size:    64
Reserved GDT blocks:      255
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         8192
Inode blocks per group:   512
Flex block group size:    16
Filesystem created:       Wed Jan  4 12:40:17 2023
Last mount time:          Wed Jan  4 12:45:57 2023
Last write time:          Wed Jan  4 12:45:57 2023
Mount count:              2
Maximum mount count:      -1
Last checked:             Wed Jan  4 12:40:17 2023
Check interval:           0 (<none>)
Lifetime writes:          1061 kB
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:	          256
Required extra isize:     32
Desired extra isize:      32
Journal inode:            8
Default directory hash:   half_md4
Directory Hash Seed:      db0e4df7-de4b-4d21-b258-bb2a3d3b741d
Journal backup:           inode blocks
Checksum type:            crc32c
Checksum:                 0x45d31649
Journal features:         journal_64bit journal_checksum_v3
Total journal size:       64M
Total journal blocks:     16384
Max transaction length:   16384
Fast commit length:       0
Journal sequence:         0x00000008
Journal start:            17
Journal checksum type:    crc32c
Journal checksum:         0xdb860f47

dumpe2fs: Superblock checksum does not match superblock while trying to open /dev/longhorn/pvc-b229a6a6-6338-4118-809a-b7c9110047c9
*** Run e2fsck now!

Actually, NodeStageVolume will execute fsck before mounting. But the resizing requires a mount point, and itself is the cause of the error. Besides, running fsck requires volume being unmounted, we cannot execute fsck after the NodeStageVolume resizing or NodeExpandVolume resizing… In other word, I haven’t found a way to fix it.

BTW, the correct workaround is: Scaling down then re-scaling up the workload.

Some notes

1. Power outage right before superblock checksum update?
2. Data mismatch between replicas?
3. There are some patches recently fixing the superblock checksum error caused by ext4 volume online resize, e.g. PR. Not sure if it is related to this issue, because Longhorn re-expands the volume after the node comes back. Need more time to investigate it.

For a workaround, we can try if the error can be fixed by fsck.

cc @chriscchien @shuo-wu @innobead