cubefs: [Bug]: performance tuning help need

Contact Details

wayne.gao1@solidigm.com

Is there an existing issue for this?

  • I have searched all the existing issues

Priority

low (Default)

Environment

- CubeFS version:latest
- Deployment mode(docker or standalone or cluster):standalone
- Dependent components:
- OS kernel version(Ubuntu or CentOS):Ubuntu
- CPU/Memory:gen5 server and gen5 NVMe SSD
- Others:

Current Behavior

only 250MiB/s BW I am using gen5 NVMe SSD read BW is 14GB/s write is 9.5GB/s

root@salab-bncbeta03:~/wayne/cubefs/mdtest# fio -directory=/mnt/nvme2n1/cubefs/data/client/mnt -ioengine=psync -rw=read    -bs=128k  -direct=1     -group_reporting=1 -fallocate=none -time_based=1 -runtime=120 -name=test_file_c  -numjobs=4  -nrfiles=1 -size=1G
test_file_c: (g=0): rw=read, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=psync, iodepth=1
...
fio-3.28
Starting 4 processes
Jobs: 4 (f=4): [R(4)][100.0%][r=257MiB/s][r=2055 IOPS][eta 00m:00s]
test_file_c: (groupid=0, jobs=4): err= 0: pid=118386: Sun Jan 19 02:06:48 2025
  read: IOPS=2003, BW=250MiB/s (263MB/s)(29.4GiB/120002msec)
    clat (usec): min=1046, max=11665, avg=1988.32, stdev=357.32
     lat (usec): min=1047, max=11665, avg=1989.08, stdev=357.33
    clat percentiles (usec):
     |  1.00th=[ 1500],  5.00th=[ 1647], 10.00th=[ 1713], 20.00th=[ 1795],
     | 30.00th=[ 1860], 40.00th=[ 1909], 50.00th=[ 1958], 60.00th=[ 2008],
     | 70.00th=[ 2057], 80.00th=[ 2114], 90.00th=[ 2212], 95.00th=[ 2311],
     | 99.00th=[ 2769], 99.50th=[ 4686], 99.90th=[ 5997], 99.95th=[ 6915],
     | 99.99th=[ 8848]
   bw (  KiB/s): min=239104, max=287488, per=100.00%, avg=256680.95, stdev=2032.46, samples=956
   iops        : min= 1868, max= 2246, avg=2004.93, stdev=15.95, samples=956
  lat (msec)   : 2=58.31%, 4=41.04%, 10=0.65%, 20=0.01%
  cpu          : usr=0.85%, sys=2.60%, ctx=240498, majf=0, minf=477
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=240464,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=250MiB/s (263MB/s), 250MiB/s-250MiB/s (263MB/s-263MB/s), io=29.4GiB (31.5GB), run=120002-120002msec

Expected Behavior

how can I saturate my standbox NVMe performance as much as CubeFS SW can?

Steps To Reproduce

1. follow guide to deploy one standalone box
2. run fio directory test, BW is low

CubeFS Log

No response

Anything else? (Additional Context)

No response

About this issue

  • Original URL
  • State: open
  • Created 2 years ago
  • Comments: 16 (1 by maintainers)

Most upvoted comments

fio --directory=/mnt/cubefs --direct=0 -iodepth 1 -thread --rw=read --ioengine=libaio --bs=128k --size=1G --numjobs=128 -group_reporting -name=mytest61 fio --directory=/mnt/cubefs --direct=1 -iodepth 1 -thread --rw=read --ioengine=libaio --bs=128k --size=1G --numjobs=128 -group_reporting -name=mytest61

128个并发,我曾经测试出结果:3227MiB/s。单个客户端