cubefs: [Bug]: performance tuning help need
Contact Details
wayne.gao1@solidigm.com
Is there an existing issue for this?
- I have searched all the existing issues
Priority
low (Default)
Environment
- CubeFS version:latest
- Deployment mode(docker or standalone or cluster):standalone
- Dependent components:
- OS kernel version(Ubuntu or CentOS):Ubuntu
- CPU/Memory:gen5 server and gen5 NVMe SSD
- Others:
Current Behavior
only 250MiB/s BW I am using gen5 NVMe SSD read BW is 14GB/s write is 9.5GB/s
root@salab-bncbeta03:~/wayne/cubefs/mdtest# fio -directory=/mnt/nvme2n1/cubefs/data/client/mnt -ioengine=psync -rw=read -bs=128k -direct=1 -group_reporting=1 -fallocate=none -time_based=1 -runtime=120 -name=test_file_c -numjobs=4 -nrfiles=1 -size=1G
test_file_c: (g=0): rw=read, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=psync, iodepth=1
...
fio-3.28
Starting 4 processes
Jobs: 4 (f=4): [R(4)][100.0%][r=257MiB/s][r=2055 IOPS][eta 00m:00s]
test_file_c: (groupid=0, jobs=4): err= 0: pid=118386: Sun Jan 19 02:06:48 2025
read: IOPS=2003, BW=250MiB/s (263MB/s)(29.4GiB/120002msec)
clat (usec): min=1046, max=11665, avg=1988.32, stdev=357.32
lat (usec): min=1047, max=11665, avg=1989.08, stdev=357.33
clat percentiles (usec):
| 1.00th=[ 1500], 5.00th=[ 1647], 10.00th=[ 1713], 20.00th=[ 1795],
| 30.00th=[ 1860], 40.00th=[ 1909], 50.00th=[ 1958], 60.00th=[ 2008],
| 70.00th=[ 2057], 80.00th=[ 2114], 90.00th=[ 2212], 95.00th=[ 2311],
| 99.00th=[ 2769], 99.50th=[ 4686], 99.90th=[ 5997], 99.95th=[ 6915],
| 99.99th=[ 8848]
bw ( KiB/s): min=239104, max=287488, per=100.00%, avg=256680.95, stdev=2032.46, samples=956
iops : min= 1868, max= 2246, avg=2004.93, stdev=15.95, samples=956
lat (msec) : 2=58.31%, 4=41.04%, 10=0.65%, 20=0.01%
cpu : usr=0.85%, sys=2.60%, ctx=240498, majf=0, minf=477
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=240464,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=250MiB/s (263MB/s), 250MiB/s-250MiB/s (263MB/s-263MB/s), io=29.4GiB (31.5GB), run=120002-120002msec
Expected Behavior
how can I saturate my standbox NVMe performance as much as CubeFS SW can?
Steps To Reproduce
1. follow guide to deploy one standalone box
2. run fio directory test, BW is low
CubeFS Log
No response
Anything else? (Additional Context)
No response
About this issue
- Original URL
- State: open
- Created 2 years ago
- Comments: 16 (1 by maintainers)
fio --directory=/mnt/cubefs --direct=0 -iodepth 1 -thread --rw=read --ioengine=libaio --bs=128k --size=1G --numjobs=128 -group_reporting -name=mytest61 fio --directory=/mnt/cubefs --direct=1 -iodepth 1 -thread --rw=read --ioengine=libaio --bs=128k --size=1G --numjobs=128 -group_reporting -name=mytest61
128个并发,我曾经测试出结果:3227MiB/s。单个客户端