longhorn: [BUG] Failed rebuilding a large replica of 3TB

Rebuilding and snapshot merge showed different behaviour. For snapshot merge, I saw progress at 20%, hours later at 40%, etc. For rebuilding, the UI show “Rebuilding …” as in pic in previous comment, but there’s no network transfer for this snapshot, so it’s effectively at 0% for 8 hours.

The issue is at GetFiemapExtents function in the client.go. I downloaded this fiemap.c, and run this against the source snapshot. It really took 7.5h, as well as load at 1cpu, and the process is hard to kill (the same behaviour when longhorn was running during initial 8h): File volume-snap-7432db6c-6aef-4bd7-901d-a4cd5ccea236.img has 1216060 extents: 0.00user 26854.01system 7:28:05elapsed 99%CPU (0avgtext+0avgdata 134160maxresident)k 50368inputs+0outputs (0major+33621minor)pagefaults 0swaps

Nvme disk rating with dbench is: Random Read/Write IOPS: 260k/233k. BW: 3210MiB/s / 2675MiB/s Average Latency (usec) Read/Write: 99.53/25.30 Sequential Read/Write: 3394MiB/s / 2660MiB/s Mixed Random Read/Write IOPS: 166k/55.4k

To show it’s not the structure of the extents that is an issue, I created a sparse file using your method above (but with 4k data size), alternating between data and holes, so 4k data, 4k hole, 4k data, 4k hole, etc. Then I ran the fiemap on it, it took 6h.

To show it’s not an OS/disk issue, I did same on CentOS 8.2 and different ssd disk (the above ran on Ubuntu 20.04.2 and nvme disk), it took 7h.

Then I did the same on XFS filesystem. It took 1s!!

So, the ext4 based sparse file probably has to traverse all the trees of the extents while xfs seems to keep this info somewhere. I don’t know if mkfs.ext4 has options so it can do like xfs. But now I retested the 1.9T production data using xfs. Initial write from outside source is 30% slower, but snapshot merging now takes less than 2h. Rebuilding doesn’t have that 8h lead time. Normal read/write performance is similar to ext4. Parallel checksum for source and target snapshot will also be helpful to reduce rebuilding when target snapshot exists.

Thanks for the help with this troubleshooting, the rebuilding steps you mentioned above is really helpful.

Verified with v1.1.2-rc1

Validation - Pass

Cluster Config : 1 Control plane, 3 worker nodes (4 vcpus, 8GB RAM) Host OS : Ubuntu 20.04

cat /proc/version
Linux version 5.4.0-73-generic (buildd@lcy01-amd64-019) (gcc version 9.3.0 (Ubuntu 9.3.0-17ubuntu1~20.04))

File System : ext4

---

Test with files created on workload fs : Created test files inside of the workload filesystem. Used the dd command to generate the data with the skip parameter. e.g. dd if=/dev/urandom of=file1.txt count=100 bs=1M conv=fsync skip=10k

Data / Version	v1.1.1	v1.1.2-rc1
2 Gi (2 files of 1 Gi) with 10K holes after each block	~1 min 55 sec	~1 min 42 sec
2 Gi (2000 files of 1M) with 10K holes after each block	~2 min 30 sec	~1 min 9 sec
10 Gi (10000 files of 1M) with 100K holes after each block	~12 min	~ 3 min 40 sec
50 Gi (50000 files of 1M) with 10k holes after each block	~39 min	~21 min
50 Gi ( 50 Files of 1G)	~1 hr 1 min 15 sec	~25 min

Test with above fio jobs : Used the above fio jobs to populate the data in the volume

• Best Case - 128k data 4k holes - 5G

  v1.1.1	v1.1.2-rc1
  5 min 4 sec	2 min 36 sec

• Worst Case - 4k data 4k holes - 5G

  v1.1.1	v1.1.2-rc1
  11 hr 58 min	2 hr 57 min

---

File System : XFS Data Size : 10 Gi with 10k holes introduced after each block

v1.1.1	v1.1.2-rc1
9 min 26 sec	3 min 11 sec

Note: The checksum of data is compared after each rebuild across all the replicas.

Creation of the Test Volume.

FIO jobs: You can use Job10 as a template if you only want to populate a single volume. Then update the engines in between the rebuild tests. i.e. test a single rebuild at a time. This way you only need to run fio once to populate the volume. To use this attach the <volume-name> for job 10 it’s rebuild-test to a random cluster node then run fio fio.job This populates the volume and creates the appropriate sparse file on the replicas.

[global]
bs=4k
iodepth=128
direct=1
sync=0
end_fsync=1
ioengine=libaio
randseed=1
randrepeat=0
# buffer_pattern="X"
group_reporting

# 4k bs
[job1]
bs=4k
rw=write:4k
filename=/dev/longhorn/rebuild-4k-old
name=data4k-holes4k-old
io_size=5G

[job2]
bs=4k
rw=write:4k
filename=/dev/longhorn/rebuild-4k-new
name=data4k-holes4k-new
io_size=5G


# 128k bs
[job4]
bs=128k
rw=write:4k
filename=/dev/longhorn/rebuild-128k-old
name=data128k-holes4k-old
io_size=5G

[job5]
bs=128k
rw=write:4k
filename=/dev/longhorn/rebuild-128k-new
name=data128k-holes4k-new
io_size=5G


# 512k bs
[job7]
bs=512k
rw=write:4k
filename=/dev/longhorn/rebuild-512k-old
name=data512k-holes4k-old
io_size=5G

[job8]
bs=512k
rw=write:4k
filename=/dev/longhorn/rebuild-512k-new
name=data512k-holes4k-new
io_size=5G

# 50GB test, need to switch engines between tests
[job10]
bs=128k
rw=write:4k
filename=/dev/longhorn/rebuild-test # specify the <volume-name>
name=test-total50gb-data128k-holes4k
io_size=50G # you can specify io_size to limit the volume to only be half written for example

@joshimoo Thanks for explaining the expected behavior and testing steps in details. The result looks good as per validation comment above https://github.com/longhorn/longhorn/issues/2507#issuecomment-866550145

Hi. They’re degraded because default replica count is 3, but I only have 2 nodes at the moment (slave-sys2/3). Most of the PVs are in ssd disks and have no issues because most PVs have size less than 50G. The one failing rebuilding is on NVMe. NVMe size is 6TB with default overprovisioning of 200%.

There were no issues creating 2 replicas the first time PV was created, I only realized it failed to resync and then to rebuild after 1 node was rebooted. Even recreating again a new PV with 2 replicas and then changing replica count to 1 and back to 2 failed to rebuild this PV.