longhorn: [BUG] Corruption using XFS after node restart or pod scale

We’ve identified the root cause of the filesystem corruption issue. In this report, we will explain the corruption occuring in an ext4 filesystem, and the same issue is also seen in a XFS filesystem.

Reproduce steps

The script can help reproduce the issue.

• Create a PVC containing 3 replicas
• Attach the volume to one node (worker-1)
• Format the longhorn volume by mkfs.ext4 -b 1024 /dev/longhorn/<pvc-name> on worker-1 node
• Run the script on worker-2 node (need to update the DEV path)

#!/bin/bash

DEV=/dev/longhorn/<pvc-name>

for x in {1..30}
do
  echo "============== delete $x =============="
  REPLICA=`kubectl -n longhorn-system get --no-headers=true replicas -o name -l longhornnode=rancher50-worker2`
  echo $REPLICA
  kubectl -n longhorn-system delete $REPLICA

  for i in {1..2000}
  do
    echo $i
    date
    umount /mnt/disk
    mount "$DEV" /mnt/disk
    if [ $? != 0 ]; then
      exit 1
    fi
    sleep 0.01
  done
done

Root Cause Analysis

The illustration shows how ext4 filesystem corruption and mount failure are caused. The root cause is a mismatch between the logical block size (1024 bytes) of the iSCSI block device and the copy-on-write (COW) block size (4096 bytes) in Longhorn replica.

The full story is

FIG 1 & 2 After deleting one replica, LH starts rebuilding the deleted replica, which is marked as WO mode, and then takes a snapshot. Meanwhile, the snapshot is empty. The new write io (offset=1024, length=1024) is incoming before finishing receiving the snapshot from the healthy replica (FIG2).

FIG 3 This write io also triggers the copy-on-write. The block size (replica.volume.sectorSize) used by copy-on-write operation is 4096 bytes. So, the data buffer contains two parts

• 0-1024 byte and 2048-4096 byte contains zero data reading from the empty snapshot
• 1024-2048 byte are the new data. The merged data buffer is written to the volume-head.

FIG 4 & FIG 5 After finishing receiving the snapshot from the healthy replica, the prune operation prunes the redundant blocks. So, the snapshots of the two replicas are identical, but the volume-heads are mismatched.

For a xfs filesystem, the default data block size is 4096 bytes, but the superblock size is 512. The block size information can be checked by xfs_info /dev/longhorn/<pvc-name>. Hence, the mismatch between the superblock size and Longhorn COW logic will cause the data corruption.

Impact

• Longhorn volume using xfs filesystem
  • All versions are impacted. By deafult, the superblock size is 512.
• Longhorn volume using ext4 filesystem
  • Ext4 filesystem on a Longhorn volume created by Longhorn < v1.3.0’s block size is mostly 4096 bytes, but it might be set to 1024 bytes from v1.3.0. The value is heuristically determined by the filesystem size and the expected usage of the filesystem (ref)

Solution

• Fix Longhorn replica COW logic during rebuilding (https://github.com/longhorn/longhorn-engine/pull/745)
• Set the xfs and ext4 filesystem block size to 4096 (https://github.com/longhorn/longhorn/issues/4594)

Related Issues

• https://github.com/longhorn/longhorn/issues/4354
• https://github.com/longhorn/longhorn/issues/3895

cc @shuo-wu @PhanLe1010 @joshimoo @innobead

I think this issue is addressed in https://github.com/longhorn/longhorn/issues/4354 and https://github.com/longhorn/longhorn/issues/4594

@derekbit other than minor version differences, the only difference I note is that you’re using EBS which would present a block device. I was using LVM, and had formatted the VG with xfs on my hypervisor, then passed it through to the worker node, and pointed Longhorn to it.

@stephanGarland I tried to reproduce the xfs filesystem corruption issue on AWS platform, but unfortunately, the corruption still cannot be reproduced. The StatefulSet workload writes data to a sqlite databse storing on the LH RWO volume. I scaled StatefulSet utilizing the PVC to 0 and back to 3 repeatedly as you mentioned. Every mount and xfs log recovery were always successful.

Here is my setting. Any suggestion is appreciated.

• Host OS: Ubuntu 20.04 (kernel 5.4.0-1041-aws)

• K8s: v1.24.4+rke2r1

• Longhorn: v1.2.4

• Storage Class

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: longhorn-xfs
provisioner: driver.longhorn.io
allowVolumeExpansion: true
reclaimPolicy: Delete
volumeBindingMode: Immediate
parameters:
  numberOfReplicas: "3"
  staleReplicaTimeout: "2880"
  fromBackup: ""
  fsType: "xfs"

• StatefulSet

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: volume-test
  namespace: default
spec:
  selector:
    matchLabels:
      app: volume-test
  serviceName: "volume-test"
  replicas: 3
  template:
    metadata:
      labels:
        app: volume-test
    spec:
      #restartPolicy: Always
      terminationGracePeriodSeconds: 10
      containers:
      - image: naturlich/ubuntu:fs-corruption
        name: volume-test
        command: ["/bin/bash", "/root/sqlite.sh"]
        imagePullPolicy: Always
        volumeMounts:
        - name: longhorn-volv
          mountPath: /data
  volumeClaimTemplates:
  - metadata:
      name: longhorn-volv
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: "longhorn-xfs"
      resources:
        requests:
          storage: 2Gi

• sqlite.sh

#!/bin/bash

rm -rf /data/test.db

sqlite3 /data/test.db <<EOF
create table helloworld (id INTEGER PRIMARY KEY,k TEXT,v TEXT);
EOF

for i in {1..100000};
do
KEY=`tr -dc A-Za-z0-9 </dev/urandom | head -c 4096`
VAL=`tr -dc A-Za-z0-9 </dev/urandom | head -c 4096`
sqlite3 /data/test.db <<EOF
insert into helloworld (k,v) values ('${KEY}','${VAL}');
EOF
done

cc @PhanLe1010 @shuo-wu @innobead

Not really sure if there is a bug in xfs. I think user can switch to use ext4 first and see if it still happens in ext4.

k3s_kubelet.log kw01_dmesg.log longhorn_shas.txt longhorn-support-bundle_a0d2f19d-cc3c-40ea-ac41-47baad381803_2022-02-10T00-00-40Z.zip

Here’s an organic event that just happened. I’m not sure what triggered it; I don’t believe I’ve touched this pod in a few days. Deleting the odd-out replica didn’t fix the problem. Weirdly, although the node dell03-k3s-worker-01 was the one reporting a different SHA (and thus the one I deleted), the dmesg error was from dell01-k3s-worker-01 - the other two reported no issues in their logs.

Unfortunately my most recent snapshot was also bad, so I had to delete the PVC and recreate it using a manual backup I performed a few weeks ago, copying the files in.

I checked the SSDs on the node for SMART errors but didn’t see any, FWIW.