* Re: [SPDK] SPDK FIO poor 4K read performance
@ 2017-12-21 17:54 Kariuki, John K
0 siblings, 0 replies; 15+ messages in thread
From: Kariuki, John K @ 2017-12-21 17:54 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 15025 bytes --]
Mayank
That average latency (4K, random read, QD=1) your showing is very low. For a P3700 SSD that number should be in the low 80 usecs. I usually see numbers that low when my SSD is empty which usually indicates that the controller is returning the data without actually going to the NAND media to read data because it knows the media does not contain any data. Did you fill your SSD 2 times with sequential writes? What parameters did you use to fill your drive? In our presentation, we gave the time parameter based on an 800 GB P3700 SSD. Since your SSD is 2 TB you will need to multiply that by 2.5 so you should run with the following options ./perf -q 32 -s 131072 -w write -t 3000 to fill the SSD with sequential writes 2 times.
From: Mayank [mailto:mayank.patel(a)mayadata.io]
Sent: Tuesday, December 19, 2017 10:12 PM
To: Kariuki, John K <john.k.kariuki(a)intel.com>; Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] SPDK FIO poor 4K read performance
Hi John,
Thanks for sharing results.
In my latency data, QD=1.
Can you run perf with the following options(./perf -q 1 -s 4096 -t 600 -w randread) and share the output?
results.
[root(a)localhost spdk]# ./examples/nvme/perf/perf -q 1 -s 4096 -t 600 -w randread
Starting DPDK 17.11.0 initialization...
[ DPDK EAL parameters: perf -c 0x1 --file-prefix=spdk_pid24378 ]
EAL: Detected 112 lcore(s)
EAL: Probing VFIO support...
EAL: WARNING: Master core has no memory on local socket!
Initializing NVMe Controllers
EAL: PCI device 0000:18:00.0 on NUMA socket 0
EAL: probe driver: 8086:953 spdk_nvme
Attaching to NVMe Controller at 0000:18:00.0 [8086:0953]
Attached to NVMe Controller at 0000:18:00.0 [8086:0953]
Associating INTEL SSDPEDMD020T4 (CVFT4415009B2P0EGN ) with lcore 0
Initialization complete. Launching workers.
Starting thread on core 0
========================================================
Latency(us)
Device Information : IOPS MB/s Average min max
INTEL SSDPEDMD020T4 (CVFT4415009B2P0EGN ) from core 0: 158706.03 619.95 6.28 4.37 831.51
========================================================
Total : 158706.03 619.95 6.28 4.37 831.51
*** in this, latency is very less compared to fio!
Regarding PCI speed, snapshot from lspci for NVME SSD
LnkSta: Speed 8GT/s, Width x4, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
Can you perform randwrite with your setup, if possible?
On Wednesday 20 December 2017 03:17 AM, Kariuki, John K wrote:
Mayank
Thank you for sharing this information.
On the latency data (rand read 11.2k 45.9MB/s 88.83 usec), can you confirm that the QD=1?
I have a system with P3700 SSD (U.2 form factor) and I am getting a latency number of around 81 usecs with both perf and FIO.
SPDK PERF:
./perf -q 1 -s 4096 -t 600 -w randread -r 'trtype:PCIe traddr:0000:62:00.0'
Starting DPDK 17.08.0 initialization...
[ DPDK EAL parameters: perf -c 0x1 --file-prefix=spdk_pid43053 ]
EAL: Detected 112 lcore(s)
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
Initializing NVMe Controllers
EAL: PCI device 0000:62:00.0 on NUMA socket 0
EAL: probe driver: 8086:953 spdk_nvme
Attaching to NVMe Controller at 0000:62:00.0 [8086:0953]
Attached to NVMe Controller at 0000:62:00.0 [8086:0953]
Associating INTEL SSDPE2MD400G4 (CVFT6175001C400GGN ) with lcore 0
Initialization complete. Launching workers.
Starting thread on core 0
========================================================
Latency(us)
Device Information : IOPS MB/s Average min max
INTEL SSDPE2MD400G4 (CVFT6175001C400GGN ) from core 0: 12241.53 47.82 81.66 34.86 2891.90
========================================================
Total : 12241.53 47.82 81.66 34.86 2891.90
FIO:
RW=randrw MIX=100 BLK_SIZE=4096 IODEPTH=1 NUMJOBS=1 LD_PRELOAD=/home/john/spdk/examples/nvme/fio_plugin/fio_plugin fio fio.spdk_1ssd
filename1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk, iodepth=1
fio-3.2-84-g9b509
Starting 1 thread
Starting DPDK 17.08.0 initialization...
[ DPDK EAL parameters: fio -c 0x1 -m 512 --file-prefix=spdk_pid43241 ]
EAL: Detected 112 lcore(s)
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: PCI device 0000:62:00.0 on NUMA socket 0
EAL: probe driver: 8086:953 spdk_nvme
Jobs: 1 (f=0): [f(1)][100.0%][r=0KiB/s,w=0KiB/s][r=0,w=0 IOPS][eta 00m:00s]
filename1: (groupid=0, jobs=1): err= 0: pid=43356: Tue Dec 19 06:23:29 2017
read: IOPS=12.2k, BW=47.5MiB/s (49.8MB/s)(27.8GiB/600001msec)
slat (nsec): min=94, max=87632, avg=108.13, stdev=40.75
clat (nsec): min=196, max=3008.9k, avg=81777.09, stdev=38852.81
lat (usec): min=34, max=3009, avg=81.89, stdev=38.85
clat percentiles (usec):
| 50.000th=[ 78], 99.000th=[ 94], 99.900th=[ 241], 99.990th=[ 2278],
| 99.999th=[ 2671]
bw ( KiB/s): min=42738, max=50248, per=100.00%, avg=48802.24, stdev=1522.91, samples=1199
iops : min=10684, max=12562, avg=12200.43, stdev=380.76, samples=1199
lat (nsec) : 250=0.01%
lat (usec) : 50=0.01%, 100=99.78%, 250=0.12%, 500=0.04%, 750=0.01%
lat (usec) : 1000=0.01%
lat (msec) : 2=0.02%, 4=0.02%
cpu : usr=99.98%, sys=0.00%, ctx=51300, majf=0, minf=266
IO depths : 1=110.1%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwt: total=7298991,0,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=47.5MiB/s (49.8MB/s), 47.5MiB/s-47.5MiB/s (49.8MB/s-49.8MB/s), io=27.8GiB (29.9GB), run=600001-600001msec
Here is the FIO configuration, I usually use in my test environment
[global]
ioengine=spdk
direct=1
time_based=1
norandommap=1
ramp_time=60s
runtime=600s
thread=1
group_reporting=1
percentile_list=50:99:99.9:99.99:99.999
rw=${RW}
rwmixread=${MIX}
bs=${BLK_SIZE}
iodepth=${IODEPTH}
numjobs=${NUMJOBS}
[filename1]
filename=trtype=PCIe traddr=0000.62.00.0 ns=1
cpumask=0x1
I ran the test with your fio options and the latency is pretty consistent with perf
fio --time_based --name=benchmark --size=10G --runtime=600 '--filename=trtype=PCIe traddr=0000.62.00.0 ns=1' --ioengine=/home/john/spdk/examples/nvme/fio_plugin/fio_plugin --randrepeat=0 --iodepth=1 --thread --direct=1 --invalidate=1 --verify=0 --verify_fatal=0 --numjobs=1 --rw=randread --sync=1 --blocksize=4k --norandommap=1 --group_reporting
benchmark: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk, iodepth=1
fio-3.2-84-g9b509
Starting 1 thread
Starting DPDK 17.08.0 initialization...
[ DPDK EAL parameters: fio -c 0x1 -m 512 --file-prefix=spdk_pid43874 ]
EAL: Detected 112 lcore(s)
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: PCI device 0000:62:00.0 on NUMA socket 0
EAL: probe driver: 8086:953 spdk_nvme
Jobs: 1 (f=0): [f(1)][100.0%][r=4412KiB/s,w=0KiB/s][r=1103,w=0 IOPS][eta 00m:00s]
benchmark: (groupid=0, jobs=1): err= 0: pid=43989: Tue Dec 19 07:43:48 2017
read: IOPS=12.2k, BW=47.5MiB/s (49.8MB/s)(27.8GiB/600001msec)
slat (nsec): min=94, max=15837, avg=105.59, stdev=20.51
clat (usec): min=23, max=2947, avg=81.73, stdev=37.37
lat (usec): min=24, max=2947, avg=81.84, stdev=37.37
clat percentiles (usec):
| 1.00th=[ 63], 5.00th=[ 68], 10.00th=[ 72], 20.00th=[ 74],
| 30.00th=[ 75], 40.00th=[ 76], 50.00th=[ 78], 60.00th=[ 87],
| 70.00th=[ 89], 80.00th=[ 91], 90.00th=[ 92], 95.00th=[ 93],
| 99.00th=[ 94], 99.50th=[ 94], 99.90th=[ 221], 99.95th=[ 553],
| 99.99th=[ 2278]
bw ( KiB/s): min=42618, max=49384, per=99.98%, avg=48644.25, stdev=1407.57, samples=1199
iops : min=10654, max=12346, avg=12161.03, stdev=351.90, samples=1199
lat (usec) : 50=0.01%, 100=99.79%, 250=0.11%, 500=0.04%, 750=0.01%
lat (usec) : 1000=0.01%
lat (msec) : 2=0.02%, 4=0.02%
cpu : usr=100.00%, sys=0.00%, ctx=50106, majf=0, minf=280
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwt: total=7298286,0,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=47.5MiB/s (49.8MB/s), 47.5MiB/s-47.5MiB/s (49.8MB/s-49.8MB/s), io=27.8GiB (29.9GB), run=600001-600001msec
Can you run perf with the following options(./perf -q 1 -s 4096 -t 600 -w randread) and share the output?
Have you checked the PCI link capability and status to verify the Speed is 8GT/s and the width is x4?
On the sequential reads, you need to use a larger block size of 128K to reach the peak throughput.
From: Mayank [mailto:mayank.patel(a)mayadata.io]
Sent: Monday, December 18, 2017 11:33 PM
To: Kariuki, John K <john.k.kariuki(a)intel.com><mailto:john.k.kariuki(a)intel.com>; Storage Performance Development Kit <spdk(a)lists.01.org><mailto:spdk(a)lists.01.org>
Subject: Re: [SPDK] SPDK FIO poor 4K read performance
Hi John,
On Tuesday 19 December 2017 04:54 AM, Kariuki, John K wrote:
Mayank
Can you share your fio configuration file?
i used following fio command :
fio --time_based --time_based --name=benchmark --size=10G --runtime=600 '--filename=trtype=PCIe traddr=0000.18.00.0 ns=1' --ioengine=spdk/examples/nvme/fio_plugin/fio_plugin --randrepeat=0 --iodepth=32 --thread --direct=1 --invalidate=1 --verify=0 --verify_fatal=0 --numjobs=4 --rw=randwrite --sync=1 --blocksize=4k --norandommap=1 --group_reportin
Have you tried the latency performance test with the SPDK perf tool?
Yes, i tried it. Results for write with run-time 300sec is as below.
./examples/nvme/perf/perf -q 32 -s 4096 -t 300 -w write
========================================================
Latency(us)
Device Information : IOPS MB/s Average min max
INTEL SSDPEDMD020T4 (CVFT4415009B2P0EGN ) from core 0: 436776.09 1706.16 73.26 4.69 10563.94
========================================================
Total : 436776.09 1706.16 73.26 4.69 10563.94
################
+ ./examples/nvme/perf/perf -q 32 -s 4096 -t 300 -w read
========================================================
Latency(us)
Device Information : IOPS MB/s Average min max
INTEL SSDPEDMD020T4 (CVFT4415009B2P0EGN ) from core 0: 213554.89 834.20 149.84 4.05 9111.46
========================================================
Total : 213554.89 834.20 149.84 4.05 9111.46
Your random writes are pretty high. Did you precondition the SSD before the test? For the random write test you need to precondition the SSD by doing 90 minutes of random writes before testing.
Yes, I preconditioned SSD before test.
Here is a link to a video<https://www.youtube.com/watch?v=tkGE3pq5eIU&index=8&list=PLj-81kG3zG5ZIE-4CvqsvlFEHoOoWRIHZ> that highlights some performance best practices that you might want to check out.
Thanks for sharing video. I have already gone through this presentation and followed necessary steps mentioned in this presentation.
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Andrey Kuzmin
Sent: Monday, December 18, 2017 3:40 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org><mailto:spdk(a)lists.01.org>
Cc: mayank.patel(a)mayadata.io<mailto:mayank.patel(a)mayadata.io>
Subject: Re: [SPDK] SPDK FIO poor 4K read performance
On Tue, Dec 19, 2017, 01:27 Isaac Otsiabah <IOtsiabah(a)us.fujitsu.com<mailto:IOtsiabah(a)us.fujitsu.com>> wrote:
Hi Audrey,
Hi Isaac,
I guess you meant to address the original submission author, Mayank.
Regards,
Andrey
which
plugin are you running? examples/nvme/fio_plugin/fio_plugin or examples/bdev/fio_plugin/fio_plugin ? Is it on Fedora or Centos?
Isaac
From: SPDK [mailto:spdk-bounces(a)lists.01.org<mailto:spdk-bounces(a)lists.01.org>] On Behalf Of Andrey Kuzmin
Sent: Monday, December 18, 2017 5:59 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] SPDK FIO poor 4K read performance
On Mon, Dec 18, 2017, 16:08 Mayank <mayank.patel(a)mayadata.io<mailto:mayank.patel(a)mayadata.io>> wrote:
Hi All,
I have performed fio benchmark test with SPDK fio plugin for throughput test with iodepth = 32 and numjobs=4.
Latency results are as below:
rand read
11.2k
45.9MB/s
88.83 usec
rand write
126k
518MB/s
7.56 usec
Throughput results are as below:
Throughput (io:32 thr:4)
IOPS
Throuhgput
Latency (usec)
seq read
406k
1665MB/s
314.71
seq write
455k
1865MB/s
280.87
rand read
478k
1959MB/s
267.29
rand write
442k
1812MB/s
289.03
In results, 'm getting sequential read IOPS lesser than sequential write IOPS.
I am using NVME model number : INTEL SSDPEDMD020T4.
kernel : 4.13.13
fio version : 2.21
BTW, your fio version is couple of years behind.
can anybody tell me what could be the issue?
The latency data above clearly shows the reason: writes are being handled by the internal SSD cache of few usec latency, while reads are served off the real media, with typical (high-end) SSD latency of about 100 usec.
Regards,
Andrey
Thanks,
Mayank
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org<mailto:SPDK(a)lists.01.org>
https://lists.01.org/mailman/listinfo/spdk
--
Regards,
Andrey
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org<mailto:SPDK(a)lists.01.org>
https://lists.01.org/mailman/listinfo/spdk
--
Regards,
Andrey
[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 54783 bytes --]
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [SPDK] SPDK FIO poor 4K read performance
@ 2018-01-11 20:30 Mayank Patel
0 siblings, 0 replies; 15+ messages in thread
From: Mayank Patel @ 2018-01-11 20:30 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 22855 bytes --]
John
I performed pre-conditioning test with 13500 seconds for 4 times.
After pre-conditioning, i performed random write test with queue depth 32.
Command is as below:
./perf -q 32 -s 4096 -w randwrite -t 300
I got following result:
IOPS:292987 throughput:1144.48 MB/s latency:109.20 usec
On Wed, Jan 10, 2018 at 5:37 AM, Kariuki, John K <john.k.kariuki(a)intel.com>
wrote:
> Mayank
>
> That latency number is pretty low. I am still thinking that this has to do
> with pre-conditioning the drive. Can you change the pre-conditioning time
> for the random writes to 5400 X 2.5 = 13500 seconds? So basically apply the
> same scaling to the random write pre-conditioning as you applied when
> pre-conditioning for the other workloads.
>
> On my system, for 4K randwrites @ QD 32, I get IOPS = 85K, Bandwidth=334
> MB/s and latency=374 us for P3700 800 GB SSD.
>
>
>
>
>
> *From:* Mayank [mailto:mayank.patel(a)mayadata.io]
> *Sent:* Tuesday, January 09, 2018 12:11 AM
>
> *To:* Kariuki, John K <john.k.kariuki(a)intel.com>
> *Cc:* Storage Performance Development Kit <spdk(a)lists.01.org>
> *Subject:* Re: [SPDK] SPDK FIO poor 4K read performance
>
>
>
> Hi John,
>
> Thanks for reply.
>
> Yes, i did workload specific preconditioning for 90 minutes.
>
> I used following command to perform randwrite performance test.
>
> ./perf -q 32 -s 4096 -w randwrite -t 300
>
>
>
> After preconditioning of SSD, randread/read performance results are same
> as your results.
>
> In my first email, results with randread/randwrite were as below:
>
>
>
> rand read
>
> 11.2k
>
> 45.9MB/s
>
> 88.83 usec
>
> rand write
>
> 126k
>
> 518MB/s
>
> 7.56 usec
>
>
> Here, latency in random write is very low compared to random read. After
> preconditioning also, i am getting same numbers.
> Is it expected behavior?
>
> Can you run/share write (random/sequential) performance results for
> latency at your end?
>
> Thanks for help.
>
> Regards,
> Mayank
>
> On Monday 08 January 2018 09:44 PM, Kariuki, John K wrote:
>
> Mayank
>
> No problem. Thanks for the update.
>
> Did you do the workload specific pre-conditioning before the 100% random
> write test? For the 4K 100% random write workload, we do a workload
> specific preconditioning for 90 minutes, which can be done with perf
> command: *./perf -q 32 -s 4096 -w randwrite -t 5400*
>
> Can you also share the command inputs your using? I am looking at some of
> the older emails and the workload (-w) is set to write in runs instead of
> randwrite?
>
>
>
> *From:* Mayank Patel [mailto:mayank.patel(a)mayadata.io
> <mayank.patel(a)mayadata.io>]
> *Sent:* Monday, January 08, 2018 2:01 AM
> *To:* Kariuki, John K <john.k.kariuki(a)intel.com>
> <john.k.kariuki(a)intel.com>
> *Cc:* Storage Performance Development Kit <spdk(a)lists.01.org>
> <spdk(a)lists.01.org>
> *Subject:* Re: [SPDK] SPDK FIO poor 4K read performance
>
>
>
> Hi John,
>
> Thanks for suggesting preconditioning changes.
>
> I ran precondition test with your changes for 2 times. Now, i am getting
> read latency same as your results.
>
>
>
> Sorry i couldn't reply you on time.
>
>
>
> After filling up SSD, random read results with perf is as follow:
>
>
> Starting DPDK 17.11.0 initialization...
> [ DPDK EAL parameters: perf -c 0x1 --file-prefix=spdk_pid178910 ]
> EAL: Detected 112 lcore(s)
> EAL: Probing VFIO support...
> Initializing NVMe Controllers
> EAL: PCI device 0000:18:00.0 on NUMA socket 0
> EAL: probe driver: 8086:953 spdk_nvme
> Attaching to NVMe Controller at 0000:18:00.0 [8086:0953]
> Attached to NVMe Controller at 0000:18:00.0 [8086:0953]
> Associating INTEL SSDPEDMD020T4 (CVFT4415009B2P0EGN ) with lcore 0
> Initialization complete. Launching workers.
> Starting thread on core 0
> ========================================================
>
> Latency(us)
> Device Information : IOPS
> MB/s Average min max
> INTEL SSDPEDMD020T4 (CVFT4415009B2P0EGN ) from core 0: 11014.63
> 43.03 90.77 34.15 3730.32
> ========================================================
> Total : 11014.63
> 43.03 90.77 34.15 3730.32
>
>
>
> I've also ran fio test with QD=1
>
> benchmark: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T)
> 4096B-4096B, ioengine=spdk, iodepth=1
> fio-3.2
> Starting 1 thread
> Starting DPDK 17.11.0 initialization...
> [ DPDK EAL parameters: fio -c 0x1 -m 512 --file-prefix=spdk_pid225681 ]
> EAL: Detected 112 lcore(s)
> EAL: Probing VFIO support...
> EAL: PCI device 0000:18:00.0 on NUMA socket 0
> EAL: probe driver: 8086:953 spdk_nvme
>
> benchmark: (groupid=0, jobs=1): err= 0: pid=225812: Mon Jan 8 03:34:46
> 2018
> read: IOPS=11.2k, BW=43.8MiB/s (45.9MB/s)(25.6GiB/600001msec)
> slat (nsec): min=178, max=20412, avg=213.32, stdev=66.07
> clat (usec): min=4, max=3226, avg=88.66, stdev=46.20
> lat (usec): min=5, max=3226, avg=88.87, stdev=46.20
> clat percentiles (usec):
> | 1.00th=[ 69], 5.00th=[ 74], 10.00th=[ 78], 20.00th=[ 80],
> | 30.00th=[ 82], 40.00th=[ 83], 50.00th=[ 84], 60.00th=[ 94],
> | 70.00th=[ 96], 80.00th=[ 98], 90.00th=[ 99], 95.00th=[ 100],
> | 99.00th=[ 101], 99.50th=[ 109], 99.90th=[ 297], 99.95th=[ 889],
> | 99.99th=[ 2573]
> bw ( KiB/s): min=39799, max=55815, per=100.00%, avg=45083.50,
> stdev=1636.36, samples=1200
> iops : min= 9949, max=13953, avg=11270.50, stdev=409.05,
> samples=1200
> lat (usec) : 10=0.01%, 20=0.01%, 50=0.54%, 100=95.61%, 250=3.69%
> lat (usec) : 500=0.09%, 750=0.02%, 1000=0.01%
> lat (msec) : 2=0.02%, 4=0.02%
> cpu : usr=99.75%, sys=0.03%, ctx=157, majf=0, minf=2304
> IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%,
> >=64=0.0%
> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
> >=64=0.0%
> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
> >=64=0.0%
> issued rwt: total=6720907,0,0, short=0,0,0, dropped=0,0,0
> latency : target=0, window=0, percentile=100.00%, depth=1
>
> Run status group 0 (all jobs):
> READ: bw=43.8MiB/s (45.9MB/s), 43.8MiB/s-43.8MiB/s (45.9MB/s-45.9MB/s),
> io=25.6GiB (27.5GB), run=600001-600001msec
>
>
>
> Now, random read results are same as yours, but write results are still
> bothering me... Since latency in random write is very low compare to random
> read.
>
>
>
> Regards,
>
> Mayank
>
> On Thu, Dec 21, 2017 at 11:24 PM, Kariuki, John K <
> john.k.kariuki(a)intel.com> wrote:
>
> Mayank
>
> That average latency (4K, random read, QD=1) your showing is very low. For
> a P3700 SSD that number should be in the low 80 usecs. I usually see
> numbers that low when my SSD is empty which usually indicates that the
> controller is returning the data without actually going to the NAND media
> to read data because it knows the media does not contain any data. Did you
> fill your SSD 2 times with sequential writes? What parameters did you use
> to fill your drive? In our presentation, we gave the time parameter based
> on an 800 GB P3700 SSD. Since your SSD is 2 TB you will need to multiply
> that by 2.5 so you should run with the following options ./perf -q 32 -s
> 131072 -w write -t 3000 to fill the SSD with sequential writes 2 times.
>
>
>
> *From:* Mayank [mailto:mayank.patel(a)mayadata.io]
> *Sent:* Tuesday, December 19, 2017 10:12 PM
>
>
> *To:* Kariuki, John K <john.k.kariuki(a)intel.com>; Storage Performance
> Development Kit <spdk(a)lists.01.org>
> *Subject:* Re: [SPDK] SPDK FIO poor 4K read performance
>
>
>
> Hi John,
>
> Thanks for sharing results.
>
> In my latency data, QD=1.
>
> Can you run perf with the following options(./perf -q 1 -s 4096 -t 600 -w
> randread) and share the output?
>
> results.
>
> [root(a)localhost spdk]# ./examples/nvme/perf/perf -q 1 -s 4096 -t 600 -w
> randread
> Starting DPDK 17.11.0 initialization...
> [ DPDK EAL parameters: perf -c 0x1 --file-prefix=spdk_pid24378 ]
> EAL: Detected 112 lcore(s)
> EAL: Probing VFIO support...
> EAL: WARNING: Master core has no memory on local socket!
> Initializing NVMe Controllers
> EAL: PCI device 0000:18:00.0 on NUMA socket 0
> EAL: probe driver: 8086:953 spdk_nvme
> Attaching to NVMe Controller at 0000:18:00.0 [8086:0953]
> Attached to NVMe Controller at 0000:18:00.0 [8086:0953]
> Associating INTEL SSDPEDMD020T4 (CVFT4415009B2P0EGN ) with lcore 0
> Initialization complete. Launching workers.
> Starting thread on core 0
> ========================================================
>
> Latency(us)
> Device Information : IOPS
> MB/s Average min max
> INTEL SSDPEDMD020T4 (CVFT4415009B2P0EGN ) from core 0: 158706.03
> 619.95 6.28 4.37 831.51
> ========================================================
> Total : 158706.03
> 619.95 6.28 4.37 831.51
>
> *** in this, latency is very less compared to fio!
>
> Regarding PCI speed, snapshot from lspci for NVME SSD
>
> LnkSta: Speed 8GT/s, Width x4, TrErr- Train- SlotClk- DLActive-
> BWMgmt- ABWMgmt-
>
> Can you perform randwrite with your setup, if possible?
>
>
>
> On Wednesday 20 December 2017 03:17 AM, Kariuki, John K wrote:
>
> Mayank
>
> Thank you for sharing this information.
>
> On the latency data (rand read 11.2k 45.9MB/s 88.83 usec), can you
> confirm that the QD=1?
>
> I have a system with P3700 SSD (U.2 form factor) and I am getting a
> latency number of around 81 usecs with both perf and FIO.
>
> SPDK PERF:
>
> ./perf -q 1 -s 4096 -t 600 -w randread -r 'trtype:PCIe traddr:0000:62:00.0'
>
> Starting DPDK 17.08.0 initialization...
>
> [ DPDK EAL parameters: perf -c 0x1 --file-prefix=spdk_pid43053 ]
>
> EAL: Detected 112 lcore(s)
>
> EAL: No free hugepages reported in hugepages-1048576kB
>
> EAL: Probing VFIO support...
>
> Initializing NVMe Controllers
>
> EAL: PCI device 0000:62:00.0 on NUMA socket 0
>
> EAL: probe driver: 8086:953 spdk_nvme
>
> Attaching to NVMe Controller at 0000:62:00.0 [8086:0953]
>
> Attached to NVMe Controller at 0000:62:00.0 [8086:0953]
>
> Associating INTEL SSDPE2MD400G4 (CVFT6175001C400GGN ) with lcore 0
>
> Initialization complete. Launching workers.
>
> Starting thread on core 0
>
> ========================================================
>
>
> Latency(us)
>
> Device Information : IOPS
> MB/s Average min max
>
> INTEL SSDPE2MD400G4 (CVFT6175001C400GGN ) from core 0: 12241.53
> 47.82 81.66 34.86 2891.90
>
> ========================================================
>
> Total : 12241.53
> 47.82 81.66 34.86 2891.90
>
>
>
> FIO:
>
> RW=randrw MIX=100 BLK_SIZE=4096 IODEPTH=1 NUMJOBS=1
> LD_PRELOAD=/home/john/spdk/examples/nvme/fio_plugin/fio_plugin fio
> fio.spdk_1ssd
>
> filename1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T)
> 4096B-4096B, ioengine=spdk, iodepth=1
>
> fio-3.2-84-g9b509
>
> Starting 1 thread
>
> Starting DPDK 17.08.0 initialization...
>
> [ DPDK EAL parameters: fio -c 0x1 -m 512 --file-prefix=spdk_pid43241 ]
>
> EAL: Detected 112 lcore(s)
>
> EAL: No free hugepages reported in hugepages-1048576kB
>
> EAL: Probing VFIO support...
>
> EAL: PCI device 0000:62:00.0 on NUMA socket 0
>
> EAL: probe driver: 8086:953 spdk_nvme
>
> Jobs: 1 (f=0): [f(1)][100.0%][r=0KiB/s,w=0KiB/s][r=0,w=0 IOPS][eta
> 00m:00s]
>
> filename1: (groupid=0, jobs=1): err= 0: pid=43356: Tue Dec 19 06:23:29 2017
>
> read: IOPS=12.2k, BW=47.5MiB/s (49.8MB/s)(27.8GiB/600001msec)
>
> slat (nsec): min=94, max=87632, avg=108.13, stdev=40.75
>
> clat (nsec): min=196, max=3008.9k, avg=81777.09, stdev=38852.81
>
> lat (usec): min=34, max=3009, avg=81.89, stdev=38.85
>
> clat percentiles (usec):
>
> | 50.000th=[ 78], 99.000th=[ 94], 99.900th=[ 241], 99.990th=[
> 2278],
>
> | 99.999th=[ 2671]
>
> bw ( KiB/s): min=42738, max=50248, per=100.00%, avg=48802.24,
> stdev=1522.91, samples=1199
>
> iops : min=10684, max=12562, avg=12200.43, stdev=380.76,
> samples=1199
>
> lat (nsec) : 250=0.01%
>
> lat (usec) : 50=0.01%, 100=99.78%, 250=0.12%, 500=0.04%, 750=0.01%
>
> lat (usec) : 1000=0.01%
>
> lat (msec) : 2=0.02%, 4=0.02%
>
> cpu : usr=99.98%, sys=0.00%, ctx=51300, majf=0, minf=266
>
> IO depths : 1=110.1%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%,
> >=64=0.0%
>
> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
> >=64=0.0%
>
> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
> >=64=0.0%
>
> issued rwt: total=7298991,0,0, short=0,0,0, dropped=0,0,0
>
> latency : target=0, window=0, percentile=100.00%, depth=1
>
>
>
> Run status group 0 (all jobs):
>
> READ: bw=47.5MiB/s (49.8MB/s), 47.5MiB/s-47.5MiB/s (49.8MB/s-49.8MB/s),
> io=27.8GiB (29.9GB), run=600001-600001msec
>
>
>
> Here is the FIO configuration, I usually use in my test environment
>
> [global]
>
> ioengine=spdk
>
> direct=1
>
>
>
> time_based=1
>
> norandommap=1
>
> ramp_time=60s
>
> runtime=600s
>
> thread=1
>
> group_reporting=1
>
> percentile_list=50:99:99.9:99.99:99.999
>
>
>
> rw=${RW}
>
> rwmixread=${MIX}
>
> bs=${BLK_SIZE}
>
> iodepth=${IODEPTH}
>
> numjobs=${NUMJOBS}
>
>
>
> [filename1]
>
> filename=trtype=PCIe traddr=0000.62.00.0 ns=1
>
> cpumask=0x1
>
>
>
> I ran the test with your fio options and the latency is pretty consistent
> with perf
>
> fio --time_based --name=benchmark --size=10G --runtime=600
> '--filename=trtype=PCIe traddr=0000.62.00.0 ns=1'
> --ioengine=/home/john/spdk/examples/nvme/fio_plugin/fio_plugin
> --randrepeat=0 --iodepth=1 --thread --direct=1 --invalidate=1 --verify=0
> --verify_fatal=0 --numjobs=1 --rw=randread --sync=1 --blocksize=4k
> --norandommap=1 --group_reporting
>
> benchmark: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T)
> 4096B-4096B, ioengine=spdk, iodepth=1
>
> fio-3.2-84-g9b509
>
> Starting 1 thread
>
> Starting DPDK 17.08.0 initialization...
>
> [ DPDK EAL parameters: fio -c 0x1 -m 512 --file-prefix=spdk_pid43874 ]
>
> EAL: Detected 112 lcore(s)
>
> EAL: No free hugepages reported in hugepages-1048576kB
>
> EAL: Probing VFIO support...
>
> EAL: PCI device 0000:62:00.0 on NUMA socket 0
>
> EAL: probe driver: 8086:953 spdk_nvme
>
> Jobs: 1 (f=0): [f(1)][100.0%][r=4412KiB/s,w=0KiB/s][r=1103,w=0 IOPS][eta
> 00m:00s]
>
> benchmark: (groupid=0, jobs=1): err= 0: pid=43989: Tue Dec 19 07:43:48 2017
>
> read: IOPS=12.2k, BW=47.5MiB/s (49.8MB/s)(27.8GiB/600001msec)
>
> slat (nsec): min=94, max=15837, avg=105.59, stdev=20.51
>
> clat (usec): min=23, max=2947, avg=81.73, stdev=37.37
>
> lat (usec): min=24, max=2947, avg=81.84, stdev=37.37
>
> clat percentiles (usec):
>
> | 1.00th=[ 63], 5.00th=[ 68], 10.00th=[ 72], 20.00th=[ 74],
>
> | 30.00th=[ 75], 40.00th=[ 76], 50.00th=[ 78], 60.00th=[ 87],
>
> | 70.00th=[ 89], 80.00th=[ 91], 90.00th=[ 92], 95.00th=[ 93],
>
> | 99.00th=[ 94], 99.50th=[ 94], 99.90th=[ 221], 99.95th=[ 553],
>
> | 99.99th=[ 2278]
>
> bw ( KiB/s): min=42618, max=49384, per=99.98%, avg=48644.25,
> stdev=1407.57, samples=1199
>
> iops : min=10654, max=12346, avg=12161.03, stdev=351.90,
> samples=1199
>
> lat (usec) : 50=0.01%, 100=99.79%, 250=0.11%, 500=0.04%, 750=0.01%
>
> lat (usec) : 1000=0.01%
>
> lat (msec) : 2=0.02%, 4=0.02%
>
> cpu : usr=100.00%, sys=0.00%, ctx=50106, majf=0, minf=280
>
> IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%,
> >=64=0.0%
>
> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
> >=64=0.0%
>
> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
> >=64=0.0%
>
> issued rwt: total=7298286,0,0, short=0,0,0, dropped=0,0,0
>
> latency : target=0, window=0, percentile=100.00%, depth=1
>
>
>
> Run status group 0 (all jobs):
>
> READ: bw=47.5MiB/s (49.8MB/s), 47.5MiB/s-47.5MiB/s (49.8MB/s-49.8MB/s),
> io=27.8GiB (29.9GB), run=600001-600001msec
>
>
>
> Can you run perf with the following options(./perf -q 1 -s 4096 -t 600 -w
> randread) and share the output?
>
> Have you checked the PCI link capability and status to verify the Speed is
> 8GT/s and the width is x4?
>
> On the sequential reads, you need to use a larger block size of 128K to
> reach the peak throughput.
>
>
>
> *From:* Mayank [mailto:mayank.patel(a)mayadata.io <mayank.patel(a)mayadata.io>]
>
> *Sent:* Monday, December 18, 2017 11:33 PM
> *To:* Kariuki, John K <john.k.kariuki(a)intel.com>
> <john.k.kariuki(a)intel.com>; Storage Performance Development Kit
> <spdk(a)lists.01.org> <spdk(a)lists.01.org>
> *Subject:* Re: [SPDK] SPDK FIO poor 4K read performance
>
>
>
> Hi John,
>
>
>
> On Tuesday 19 December 2017 04:54 AM, Kariuki, John K wrote:
>
> Mayank
>
> Can you share your fio configuration file?
>
> i used following fio command :
>
> fio --time_based --time_based --name=benchmark --size=10G --runtime=600
> '--filename=trtype=PCIe traddr=0000.18.00.0 ns=1'
> --ioengine=spdk/examples/nvme/fio_plugin/fio_plugin --randrepeat=0
> --iodepth=32 --thread --direct=1 --invalidate=1 --verify=0 --verify_fatal=0
> --numjobs=4 --rw=randwrite --sync=1 --blocksize=4k --norandommap=1
> --group_reportin
>
> Have you tried the latency performance test with the SPDK perf tool?
>
> Yes, i tried it. Results for write with run-time 300sec is as below.
> ./examples/nvme/perf/perf -q 32 -s 4096 -t 300 -w write
> ========================================================
>
> Latency(us)
> Device Information : IOPS
> MB/s Average min max
> INTEL SSDPEDMD020T4 (CVFT4415009B2P0EGN ) from core 0: 436776.09
> 1706.16 73.26 4.69 10563.94
> ========================================================
> Total : 436776.09
> 1706.16 73.26 4.69 10563.94
> ################
> + ./examples/nvme/perf/perf -q 32 -s 4096 -t 300 -w read
> ========================================================
>
> Latency(us)
> Device Information : IOPS
> MB/s Average min max
> INTEL SSDPEDMD020T4 (CVFT4415009B2P0EGN ) from core 0: 213554.89
> 834.20 149.84 4.05 9111.46
> ========================================================
> Total : 213554.89
> 834.20 149.84 4.05 9111.46
>
>
>
>
> Your random writes are pretty high. Did you precondition the SSD before
> the test? For the random write test you need to precondition the SSD by
> doing 90 minutes of random writes before testing.
>
> Yes, I preconditioned SSD before test.
>
>
> Here is a link to a video
> <https://www.youtube.com/watch?v=tkGE3pq5eIU&index=8&list=PLj-81kG3zG5ZIE-4CvqsvlFEHoOoWRIHZ>
> that highlights some performance best practices that you might want to
> check out.
>
> Thanks for sharing video. I have already gone through this presentation
> and followed necessary steps mentioned in this presentation.
>
>
>
>
> *From:* SPDK [mailto:spdk-bounces(a)lists.01.org <spdk-bounces(a)lists.01.org>]
> *On Behalf Of *Andrey Kuzmin
> *Sent:* Monday, December 18, 2017 3:40 PM
> *To:* Storage Performance Development Kit <spdk(a)lists.01.org>
> <spdk(a)lists.01.org>
> *Cc:* mayank.patel(a)mayadata.io
> *Subject:* Re: [SPDK] SPDK FIO poor 4K read performance
>
>
>
>
>
> On Tue, Dec 19, 2017, 01:27 Isaac Otsiabah <IOtsiabah(a)us.fujitsu.com>
> wrote:
>
> Hi Audrey,
>
>
>
> Hi Isaac,
>
>
>
> I guess you meant to address the original submission author, Mayank.
>
>
>
> Regards,
>
> Andrey
>
>
>
> which
>
> plugin are you running? examples/nvme/fio_plugin/fio_plugin or
> examples/bdev/fio_plugin/fio_plugin ? Is it on Fedora or Centos?
>
>
>
> Isaac
>
> *From:* SPDK [mailto:spdk-bounces(a)lists.01.org] *On Behalf Of *Andrey
> Kuzmin
> *Sent:* Monday, December 18, 2017 5:59 AM
> *To:* Storage Performance Development Kit <spdk(a)lists.01.org>
> *Subject:* Re: [SPDK] SPDK FIO poor 4K read performance
>
>
>
>
>
> On Mon, Dec 18, 2017, 16:08 Mayank <mayank.patel(a)mayadata.io> wrote:
>
> Hi All,
>
> I have performed fio benchmark test with SPDK fio plugin for throughput
> test with iodepth = 32 and numjobs=4.
>
> Latency results are as below:
>
>
>
> rand read
>
> 11.2k
>
> 45.9MB/s
>
> 88.83 usec
>
> rand write
>
> 126k
>
> 518MB/s
>
> 7.56 usec
>
> Throughput results are as below:
>
>
>
> Throughput (io:32 thr:4)
>
> IOPS
>
> Throuhgput
>
> Latency (usec)
>
> seq read
>
> 406k
>
> 1665MB/s
>
> 314.71
>
> seq write
>
> 455k
>
> 1865MB/s
>
> 280.87
>
> rand read
>
> 478k
>
> 1959MB/s
>
> 267.29
>
> rand write
>
> 442k
>
> 1812MB/s
>
> 289.03
>
>
>
> In results, 'm getting sequential read IOPS lesser than sequential write
> IOPS.
>
> I am using NVME model number : INTEL SSDPEDMD020T4.
> kernel : 4.13.13
> fio version : 2.21
>
>
>
> BTW, your fio version is couple of years behind.
>
>
> can anybody tell me what could be the issue?
>
>
>
> The latency data above clearly shows the reason: writes are being handled
> by the internal SSD cache of few usec latency, while reads are served off
> the real media, with typical (high-end) SSD latency of about 100 usec.
>
>
>
> Regards,
>
> Andrey
>
>
>
> Thanks,
> Mayank
>
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
>
> --
>
> Regards,
> Andrey
>
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
>
> --
>
> Regards,
> Andrey
>
>
>
>
>
>
>
>
>
[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 59687 bytes --]
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [SPDK] SPDK FIO poor 4K read performance
@ 2018-01-10 0:07 Kariuki, John K
0 siblings, 0 replies; 15+ messages in thread
From: Kariuki, John K @ 2018-01-10 0:07 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 21191 bytes --]
Mayank
That latency number is pretty low. I am still thinking that this has to do with pre-conditioning the drive. Can you change the pre-conditioning time for the random writes to 5400 X 2.5 = 13500 seconds? So basically apply the same scaling to the random write pre-conditioning as you applied when pre-conditioning for the other workloads.
On my system, for 4K randwrites @ QD 32, I get IOPS = 85K, Bandwidth=334 MB/s and latency=374 us for P3700 800 GB SSD.
From: Mayank [mailto:mayank.patel(a)mayadata.io]
Sent: Tuesday, January 09, 2018 12:11 AM
To: Kariuki, John K <john.k.kariuki(a)intel.com>
Cc: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] SPDK FIO poor 4K read performance
Hi John,
Thanks for reply.
Yes, i did workload specific preconditioning for 90 minutes.
I used following command to perform randwrite performance test.
./perf -q 32 -s 4096 -w randwrite -t 300
After preconditioning of SSD, randread/read performance results are same as your results.
In my first email, results with randread/randwrite were as below:
rand read
11.2k
45.9MB/s
88.83 usec
rand write
126k
518MB/s
7.56 usec
Here, latency in random write is very low compared to random read. After preconditioning also, i am getting same numbers.
Is it expected behavior?
Can you run/share write (random/sequential) performance results for latency at your end?
Thanks for help.
Regards,
Mayank
On Monday 08 January 2018 09:44 PM, Kariuki, John K wrote:
Mayank
No problem. Thanks for the update.
Did you do the workload specific pre-conditioning before the 100% random write test? For the 4K 100% random write workload, we do a workload specific preconditioning for 90 minutes, which can be done with perf command: ./perf -q 32 -s 4096 -w randwrite -t 5400
Can you also share the command inputs your using? I am looking at some of the older emails and the workload (-w) is set to write in runs instead of randwrite?
From: Mayank Patel [mailto:mayank.patel(a)mayadata.io]
Sent: Monday, January 08, 2018 2:01 AM
To: Kariuki, John K <john.k.kariuki(a)intel.com><mailto:john.k.kariuki(a)intel.com>
Cc: Storage Performance Development Kit <spdk(a)lists.01.org><mailto:spdk(a)lists.01.org>
Subject: Re: [SPDK] SPDK FIO poor 4K read performance
Hi John,
Thanks for suggesting preconditioning changes.
I ran precondition test with your changes for 2 times. Now, i am getting read latency same as your results.
Sorry i couldn't reply you on time.
After filling up SSD, random read results with perf is as follow:
Starting DPDK 17.11.0 initialization...
[ DPDK EAL parameters: perf -c 0x1 --file-prefix=spdk_pid178910 ]
EAL: Detected 112 lcore(s)
EAL: Probing VFIO support...
Initializing NVMe Controllers
EAL: PCI device 0000:18:00.0 on NUMA socket 0
EAL: probe driver: 8086:953 spdk_nvme
Attaching to NVMe Controller at 0000:18:00.0 [8086:0953]
Attached to NVMe Controller at 0000:18:00.0 [8086:0953]
Associating INTEL SSDPEDMD020T4 (CVFT4415009B2P0EGN ) with lcore 0
Initialization complete. Launching workers.
Starting thread on core 0
========================================================
Latency(us)
Device Information : IOPS MB/s Average min max
INTEL SSDPEDMD020T4 (CVFT4415009B2P0EGN ) from core 0: 11014.63 43.03 90.77 34.15 3730.32
========================================================
Total : 11014.63 43.03 90.77 34.15 3730.32
I've also ran fio test with QD=1
benchmark: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk, iodepth=1
fio-3.2
Starting 1 thread
Starting DPDK 17.11.0 initialization...
[ DPDK EAL parameters: fio -c 0x1 -m 512 --file-prefix=spdk_pid225681 ]
EAL: Detected 112 lcore(s)
EAL: Probing VFIO support...
EAL: PCI device 0000:18:00.0 on NUMA socket 0
EAL: probe driver: 8086:953 spdk_nvme
benchmark: (groupid=0, jobs=1): err= 0: pid=225812: Mon Jan 8 03:34:46 2018
read: IOPS=11.2k, BW=43.8MiB/s (45.9MB/s)(25.6GiB/600001msec)
slat (nsec): min=178, max=20412, avg=213.32, stdev=66.07
clat (usec): min=4, max=3226, avg=88.66, stdev=46.20
lat (usec): min=5, max=3226, avg=88.87, stdev=46.20
clat percentiles (usec):
| 1.00th=[ 69], 5.00th=[ 74], 10.00th=[ 78], 20.00th=[ 80],
| 30.00th=[ 82], 40.00th=[ 83], 50.00th=[ 84], 60.00th=[ 94],
| 70.00th=[ 96], 80.00th=[ 98], 90.00th=[ 99], 95.00th=[ 100],
| 99.00th=[ 101], 99.50th=[ 109], 99.90th=[ 297], 99.95th=[ 889],
| 99.99th=[ 2573]
bw ( KiB/s): min=39799, max=55815, per=100.00%, avg=45083.50, stdev=1636.36, samples=1200
iops : min= 9949, max=13953, avg=11270.50, stdev=409.05, samples=1200
lat (usec) : 10=0.01%, 20=0.01%, 50=0.54%, 100=95.61%, 250=3.69%
lat (usec) : 500=0.09%, 750=0.02%, 1000=0.01%
lat (msec) : 2=0.02%, 4=0.02%
cpu : usr=99.75%, sys=0.03%, ctx=157, majf=0, minf=2304
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwt: total=6720907,0,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=43.8MiB/s (45.9MB/s), 43.8MiB/s-43.8MiB/s (45.9MB/s-45.9MB/s), io=25.6GiB (27.5GB), run=600001-600001msec
Now, random read results are same as yours, but write results are still bothering me... Since latency in random write is very low compare to random read.
Regards,
Mayank
On Thu, Dec 21, 2017 at 11:24 PM, Kariuki, John K <john.k.kariuki(a)intel.com<mailto:john.k.kariuki(a)intel.com>> wrote:
Mayank
That average latency (4K, random read, QD=1) your showing is very low. For a P3700 SSD that number should be in the low 80 usecs. I usually see numbers that low when my SSD is empty which usually indicates that the controller is returning the data without actually going to the NAND media to read data because it knows the media does not contain any data. Did you fill your SSD 2 times with sequential writes? What parameters did you use to fill your drive? In our presentation, we gave the time parameter based on an 800 GB P3700 SSD. Since your SSD is 2 TB you will need to multiply that by 2.5 so you should run with the following options ./perf -q 32 -s 131072 -w write -t 3000 to fill the SSD with sequential writes 2 times.
From: Mayank [mailto:mayank.patel(a)mayadata.io<mailto:mayank.patel(a)mayadata.io>]
Sent: Tuesday, December 19, 2017 10:12 PM
To: Kariuki, John K <john.k.kariuki(a)intel.com<mailto:john.k.kariuki(a)intel.com>>; Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] SPDK FIO poor 4K read performance
Hi John,
Thanks for sharing results.
In my latency data, QD=1.
Can you run perf with the following options(./perf -q 1 -s 4096 -t 600 -w randread) and share the output?
results.
[root(a)localhost spdk]# ./examples/nvme/perf/perf -q 1 -s 4096 -t 600 -w randread
Starting DPDK 17.11.0 initialization...
[ DPDK EAL parameters: perf -c 0x1 --file-prefix=spdk_pid24378 ]
EAL: Detected 112 lcore(s)
EAL: Probing VFIO support...
EAL: WARNING: Master core has no memory on local socket!
Initializing NVMe Controllers
EAL: PCI device 0000:18:00.0 on NUMA socket 0
EAL: probe driver: 8086:953 spdk_nvme
Attaching to NVMe Controller at 0000:18:00.0 [8086:0953]
Attached to NVMe Controller at 0000:18:00.0 [8086:0953]
Associating INTEL SSDPEDMD020T4 (CVFT4415009B2P0EGN ) with lcore 0
Initialization complete. Launching workers.
Starting thread on core 0
========================================================
Latency(us)
Device Information : IOPS MB/s Average min max
INTEL SSDPEDMD020T4 (CVFT4415009B2P0EGN ) from core 0: 158706.03 619.95 6.28 4.37 831.51
========================================================
Total : 158706.03 619.95 6.28 4.37 831.51
*** in this, latency is very less compared to fio!
Regarding PCI speed, snapshot from lspci for NVME SSD
LnkSta: Speed 8GT/s, Width x4, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
Can you perform randwrite with your setup, if possible?
On Wednesday 20 December 2017 03:17 AM, Kariuki, John K wrote:
Mayank
Thank you for sharing this information.
On the latency data (rand read 11.2k 45.9MB/s 88.83 usec), can you confirm that the QD=1?
I have a system with P3700 SSD (U.2 form factor) and I am getting a latency number of around 81 usecs with both perf and FIO.
SPDK PERF:
./perf -q 1 -s 4096 -t 600 -w randread -r 'trtype:PCIe traddr:0000:62:00.0'
Starting DPDK 17.08.0 initialization...
[ DPDK EAL parameters: perf -c 0x1 --file-prefix=spdk_pid43053 ]
EAL: Detected 112 lcore(s)
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
Initializing NVMe Controllers
EAL: PCI device 0000:62:00.0 on NUMA socket 0
EAL: probe driver: 8086:953 spdk_nvme
Attaching to NVMe Controller at 0000:62:00.0 [8086:0953]
Attached to NVMe Controller at 0000:62:00.0 [8086:0953]
Associating INTEL SSDPE2MD400G4 (CVFT6175001C400GGN ) with lcore 0
Initialization complete. Launching workers.
Starting thread on core 0
========================================================
Latency(us)
Device Information : IOPS MB/s Average min max
INTEL SSDPE2MD400G4 (CVFT6175001C400GGN ) from core 0: 12241.53 47.82 81.66 34.86 2891.90
========================================================
Total : 12241.53 47.82 81.66 34.86 2891.90
FIO:
RW=randrw MIX=100 BLK_SIZE=4096 IODEPTH=1 NUMJOBS=1 LD_PRELOAD=/home/john/spdk/examples/nvme/fio_plugin/fio_plugin fio fio.spdk_1ssd
filename1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk, iodepth=1
fio-3.2-84-g9b509
Starting 1 thread
Starting DPDK 17.08.0 initialization...
[ DPDK EAL parameters: fio -c 0x1 -m 512 --file-prefix=spdk_pid43241 ]
EAL: Detected 112 lcore(s)
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: PCI device 0000:62:00.0 on NUMA socket 0
EAL: probe driver: 8086:953 spdk_nvme
Jobs: 1 (f=0): [f(1)][100.0%][r=0KiB/s,w=0KiB/s][r=0,w=0 IOPS][eta 00m:00s]
filename1: (groupid=0, jobs=1): err= 0: pid=43356: Tue Dec 19 06:23:29 2017
read: IOPS=12.2k, BW=47.5MiB/s (49.8MB/s)(27.8GiB/600001msec)
slat (nsec): min=94, max=87632, avg=108.13, stdev=40.75
clat (nsec): min=196, max=3008.9k, avg=81777.09, stdev=38852.81
lat (usec): min=34, max=3009, avg=81.89, stdev=38.85
clat percentiles (usec):
| 50.000th=[ 78], 99.000th=[ 94], 99.900th=[ 241], 99.990th=[ 2278],
| 99.999th=[ 2671]
bw ( KiB/s): min=42738, max=50248, per=100.00%, avg=48802.24, stdev=1522.91, samples=1199
iops : min=10684, max=12562, avg=12200.43, stdev=380.76, samples=1199
lat (nsec) : 250=0.01%
lat (usec) : 50=0.01%, 100=99.78%, 250=0.12%, 500=0.04%, 750=0.01%
lat (usec) : 1000=0.01%
lat (msec) : 2=0.02%, 4=0.02%
cpu : usr=99.98%, sys=0.00%, ctx=51300, majf=0, minf=266
IO depths : 1=110.1%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwt: total=7298991,0,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=47.5MiB/s (49.8MB/s), 47.5MiB/s-47.5MiB/s (49.8MB/s-49.8MB/s), io=27.8GiB (29.9GB), run=600001-600001msec
Here is the FIO configuration, I usually use in my test environment
[global]
ioengine=spdk
direct=1
time_based=1
norandommap=1
ramp_time=60s
runtime=600s
thread=1
group_reporting=1
percentile_list=50:99:99.9:99.99:99.999
rw=${RW}
rwmixread=${MIX}
bs=${BLK_SIZE}
iodepth=${IODEPTH}
numjobs=${NUMJOBS}
[filename1]
filename=trtype=PCIe traddr=0000.62.00.0 ns=1
cpumask=0x1
I ran the test with your fio options and the latency is pretty consistent with perf
fio --time_based --name=benchmark --size=10G --runtime=600 '--filename=trtype=PCIe traddr=0000.62.00.0 ns=1' --ioengine=/home/john/spdk/examples/nvme/fio_plugin/fio_plugin --randrepeat=0 --iodepth=1 --thread --direct=1 --invalidate=1 --verify=0 --verify_fatal=0 --numjobs=1 --rw=randread --sync=1 --blocksize=4k --norandommap=1 --group_reporting
benchmark: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk, iodepth=1
fio-3.2-84-g9b509
Starting 1 thread
Starting DPDK 17.08.0 initialization...
[ DPDK EAL parameters: fio -c 0x1 -m 512 --file-prefix=spdk_pid43874 ]
EAL: Detected 112 lcore(s)
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: PCI device 0000:62:00.0 on NUMA socket 0
EAL: probe driver: 8086:953 spdk_nvme
Jobs: 1 (f=0): [f(1)][100.0%][r=4412KiB/s,w=0KiB/s][r=1103,w=0 IOPS][eta 00m:00s]
benchmark: (groupid=0, jobs=1): err= 0: pid=43989: Tue Dec 19 07:43:48 2017
read: IOPS=12.2k, BW=47.5MiB/s (49.8MB/s)(27.8GiB/600001msec)
slat (nsec): min=94, max=15837, avg=105.59, stdev=20.51
clat (usec): min=23, max=2947, avg=81.73, stdev=37.37
lat (usec): min=24, max=2947, avg=81.84, stdev=37.37
clat percentiles (usec):
| 1.00th=[ 63], 5.00th=[ 68], 10.00th=[ 72], 20.00th=[ 74],
| 30.00th=[ 75], 40.00th=[ 76], 50.00th=[ 78], 60.00th=[ 87],
| 70.00th=[ 89], 80.00th=[ 91], 90.00th=[ 92], 95.00th=[ 93],
| 99.00th=[ 94], 99.50th=[ 94], 99.90th=[ 221], 99.95th=[ 553],
| 99.99th=[ 2278]
bw ( KiB/s): min=42618, max=49384, per=99.98%, avg=48644.25, stdev=1407.57, samples=1199
iops : min=10654, max=12346, avg=12161.03, stdev=351.90, samples=1199
lat (usec) : 50=0.01%, 100=99.79%, 250=0.11%, 500=0.04%, 750=0.01%
lat (usec) : 1000=0.01%
lat (msec) : 2=0.02%, 4=0.02%
cpu : usr=100.00%, sys=0.00%, ctx=50106, majf=0, minf=280
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwt: total=7298286,0,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=47.5MiB/s (49.8MB/s), 47.5MiB/s-47.5MiB/s (49.8MB/s-49.8MB/s), io=27.8GiB (29.9GB), run=600001-600001msec
Can you run perf with the following options(./perf -q 1 -s 4096 -t 600 -w randread) and share the output?
Have you checked the PCI link capability and status to verify the Speed is 8GT/s and the width is x4?
On the sequential reads, you need to use a larger block size of 128K to reach the peak throughput.
From: Mayank [mailto:mayank.patel(a)mayadata.io]
Sent: Monday, December 18, 2017 11:33 PM
To: Kariuki, John K <john.k.kariuki(a)intel.com><mailto:john.k.kariuki(a)intel.com>; Storage Performance Development Kit <spdk(a)lists.01.org><mailto:spdk(a)lists.01.org>
Subject: Re: [SPDK] SPDK FIO poor 4K read performance
Hi John,
On Tuesday 19 December 2017 04:54 AM, Kariuki, John K wrote:
Mayank
Can you share your fio configuration file?
i used following fio command :
fio --time_based --time_based --name=benchmark --size=10G --runtime=600 '--filename=trtype=PCIe traddr=0000.18.00.0 ns=1' --ioengine=spdk/examples/nvme/fio_plugin/fio_plugin --randrepeat=0 --iodepth=32 --thread --direct=1 --invalidate=1 --verify=0 --verify_fatal=0 --numjobs=4 --rw=randwrite --sync=1 --blocksize=4k --norandommap=1 --group_reportin
Have you tried the latency performance test with the SPDK perf tool?
Yes, i tried it. Results for write with run-time 300sec is as below.
./examples/nvme/perf/perf -q 32 -s 4096 -t 300 -w write
========================================================
Latency(us)
Device Information : IOPS MB/s Average min max
INTEL SSDPEDMD020T4 (CVFT4415009B2P0EGN ) from core 0: 436776.09 1706.16 73.26 4.69 10563.94
========================================================
Total : 436776.09 1706.16 73.26 4.69 10563.94
################
+ ./examples/nvme/perf/perf -q 32 -s 4096 -t 300 -w read
========================================================
Latency(us)
Device Information : IOPS MB/s Average min max
INTEL SSDPEDMD020T4 (CVFT4415009B2P0EGN ) from core 0: 213554.89 834.20 149.84 4.05 9111.46
========================================================
Total : 213554.89 834.20 149.84 4.05 9111.46
Your random writes are pretty high. Did you precondition the SSD before the test? For the random write test you need to precondition the SSD by doing 90 minutes of random writes before testing.
Yes, I preconditioned SSD before test.
Here is a link to a video<https://www.youtube.com/watch?v=tkGE3pq5eIU&index=8&list=PLj-81kG3zG5ZIE-4CvqsvlFEHoOoWRIHZ> that highlights some performance best practices that you might want to check out.
Thanks for sharing video. I have already gone through this presentation and followed necessary steps mentioned in this presentation.
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Andrey Kuzmin
Sent: Monday, December 18, 2017 3:40 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org><mailto:spdk(a)lists.01.org>
Cc: mayank.patel(a)mayadata.io<mailto:mayank.patel(a)mayadata.io>
Subject: Re: [SPDK] SPDK FIO poor 4K read performance
On Tue, Dec 19, 2017, 01:27 Isaac Otsiabah <IOtsiabah(a)us.fujitsu.com<mailto:IOtsiabah(a)us.fujitsu.com>> wrote:
Hi Audrey,
Hi Isaac,
I guess you meant to address the original submission author, Mayank.
Regards,
Andrey
which
plugin are you running? examples/nvme/fio_plugin/fio_plugin or examples/bdev/fio_plugin/fio_plugin ? Is it on Fedora or Centos?
Isaac
From: SPDK [mailto:spdk-bounces(a)lists.01.org<mailto:spdk-bounces(a)lists.01.org>] On Behalf Of Andrey Kuzmin
Sent: Monday, December 18, 2017 5:59 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] SPDK FIO poor 4K read performance
On Mon, Dec 18, 2017, 16:08 Mayank <mayank.patel(a)mayadata.io<mailto:mayank.patel(a)mayadata.io>> wrote:
Hi All,
I have performed fio benchmark test with SPDK fio plugin for throughput test with iodepth = 32 and numjobs=4.
Latency results are as below:
rand read
11.2k
45.9MB/s
88.83 usec
rand write
126k
518MB/s
7.56 usec
Throughput results are as below:
Throughput (io:32 thr:4)
IOPS
Throuhgput
Latency (usec)
seq read
406k
1665MB/s
314.71
seq write
455k
1865MB/s
280.87
rand read
478k
1959MB/s
267.29
rand write
442k
1812MB/s
289.03
In results, 'm getting sequential read IOPS lesser than sequential write IOPS.
I am using NVME model number : INTEL SSDPEDMD020T4.
kernel : 4.13.13
fio version : 2.21
BTW, your fio version is couple of years behind.
can anybody tell me what could be the issue?
The latency data above clearly shows the reason: writes are being handled by the internal SSD cache of few usec latency, while reads are served off the real media, with typical (high-end) SSD latency of about 100 usec.
Regards,
Andrey
Thanks,
Mayank
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org<mailto:SPDK(a)lists.01.org>
https://lists.01.org/mailman/listinfo/spdk
--
Regards,
Andrey
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org<mailto:SPDK(a)lists.01.org>
https://lists.01.org/mailman/listinfo/spdk
--
Regards,
Andrey
[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 76686 bytes --]
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [SPDK] SPDK FIO poor 4K read performance
@ 2018-01-09 7:10 Mayank
0 siblings, 0 replies; 15+ messages in thread
From: Mayank @ 2018-01-09 7:10 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 26211 bytes --]
Hi John,
Thanks for reply.
Yes, i did workload specific preconditioning for 90 minutes.
I used following command to perform randwrite performance test.
./perf -q 32 -s 4096 -w randwrite -t 300
After preconditioning of SSD, randread/read performance results are same
as your results.
In my first email, results with randread/randwrite were as below:
rand read
11.2k
45.9MB/s
88.83 usec
rand write
126k
518MB/s
7.56 usec
Here, latency in random write is very low compared to random read. After
preconditioning also, i am getting same numbers.
Is it expected behavior?
Can you run/share write (random/sequential) performance results for
latency at your end?
Thanks for help.
Regards,
Mayank
On Monday 08 January 2018 09:44 PM, Kariuki, John K wrote:
>
> Mayank
>
> No problem. Thanks for the update.
>
> Did you do the workload specific pre-conditioning before the 100%
> random write test? For the 4K 100% random write workload, we do a
> workload specific preconditioning for 90 minutes, which can be done
> with perf command: *./perf -q 32 -s 4096 -w randwrite -t 5400*
>
> Can you also share the command inputs your using? I am looking at some
> of the older emails and the workload (-w) is set to write in runs
> instead of randwrite?
>
> *From:*Mayank Patel [mailto:mayank.patel(a)mayadata.io]
> *Sent:* Monday, January 08, 2018 2:01 AM
> *To:* Kariuki, John K <john.k.kariuki(a)intel.com>
> *Cc:* Storage Performance Development Kit <spdk(a)lists.01.org>
> *Subject:* Re: [SPDK] SPDK FIO poor 4K read performance
>
> Hi John,
>
> Thanks for suggesting preconditioning changes.
>
> I ran precondition test with your changes for 2 times. Now, i am
> getting read latency same as your results.
>
> Sorry i couldn't reply you on time.
>
> After filling up SSD, random read results with perf is as follow:
>
>
> Starting DPDK 17.11.0 initialization...
> [ DPDK EAL parameters: perf -c 0x1 --file-prefix=spdk_pid178910 ]
> EAL: Detected 112 lcore(s)
> EAL: Probing VFIO support...
> Initializing NVMe Controllers
> EAL: PCI device 0000:18:00.0 on NUMA socket 0
> EAL: probe driver: 8086:953 spdk_nvme
> Attaching to NVMe Controller at 0000:18:00.0 [8086:0953]
> Attached to NVMe Controller at 0000:18:00.0 [8086:0953]
> Associating INTEL SSDPEDMD020T4 (CVFT4415009B2P0EGN ) with lcore 0
> Initialization complete. Launching workers.
> Starting thread on core 0
> ========================================================
> Latency(us)
> Device Information : IOPS MB/s Average
> min max
> INTEL SSDPEDMD020T4 (CVFT4415009B2P0EGN ) from core 0: 11014.63
> 43.03 90.77 34.15 3730.32
> ========================================================
> Total : 11014.63
> 43.03 90.77 34.15 3730.32
>
> I've also ran fio test with QD=1
>
> benchmark: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B,
> (T) 4096B-4096B, ioengine=spdk, iodepth=1
> fio-3.2
> Starting 1 thread
> Starting DPDK 17.11.0 initialization...
> [ DPDK EAL parameters: fio -c 0x1 -m 512 --file-prefix=spdk_pid225681 ]
> EAL: Detected 112 lcore(s)
> EAL: Probing VFIO support...
> EAL: PCI device 0000:18:00.0 on NUMA socket 0
> EAL: probe driver: 8086:953 spdk_nvme
>
> benchmark: (groupid=0, jobs=1): err= 0: pid=225812: Mon Jan 8
> 03:34:46 2018
> read: IOPS=11.2k, BW=43.8MiB/s (45.9MB/s)(25.6GiB/600001msec)
> slat (nsec): min=178, max=20412, avg=213.32, stdev=66.07
> clat (usec): min=4, max=3226, avg=88.66, stdev=46.20
> lat (usec): min=5, max=3226, avg=88.87, stdev=46.20
> clat percentiles (usec):
> | 1.00th=[ 69], 5.00th=[ 74], 10.00th=[ 78], 20.00th=[ 80],
> | 30.00th=[ 82], 40.00th=[ 83], 50.00th=[ 84], 60.00th=[ 94],
> | 70.00th=[ 96], 80.00th=[ 98], 90.00th=[ 99], 95.00th=[ 100],
> | 99.00th=[ 101], 99.50th=[ 109], 99.90th=[ 297], 99.95th=[ 889],
> | 99.99th=[ 2573]
> bw ( KiB/s): min=39799, max=55815, per=100.00%, avg=45083.50,
> stdev=1636.36, samples=1200
> iops : min= 9949, max=13953, avg=11270.50, stdev=409.05,
> samples=1200
> lat (usec) : 10=0.01%, 20=0.01%, 50=0.54%, 100=95.61%, 250=3.69%
> lat (usec) : 500=0.09%, 750=0.02%, 1000=0.01%
> lat (msec) : 2=0.02%, 4=0.02%
> cpu : usr=99.75%, sys=0.03%, ctx=157, majf=0, minf=2304
> IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%,
> >=64=0.0%
> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
> >=64=0.0%
> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
> >=64=0.0%
> issued rwt: total=6720907,0,0, short=0,0,0, dropped=0,0,0
> latency : target=0, window=0, percentile=100.00%, depth=1
>
> Run status group 0 (all jobs):
> READ: bw=43.8MiB/s (45.9MB/s), 43.8MiB/s-43.8MiB/s
> (45.9MB/s-45.9MB/s), io=25.6GiB (27.5GB), run=600001-600001msec
>
> Now, random read results are same as yours, but write results are
> still bothering me... Since latency in random write is very low
> compare to random read.
>
> Regards,
>
> Mayank
>
> On Thu, Dec 21, 2017 at 11:24 PM, Kariuki, John K
> <john.k.kariuki(a)intel.com <mailto:john.k.kariuki(a)intel.com>> wrote:
>
> Mayank
>
> That average latency (4K, random read, QD=1) your showing is very
> low. For a P3700 SSD that number should be in the low 80 usecs. I
> usually see numbers that low when my SSD is empty which usually
> indicates that the controller is returning the data without
> actually going to the NAND media to read data because it knows the
> media does not contain any data. Did you fill your SSD 2 times
> with sequential writes? What parameters did you use to fill your
> drive? In our presentation, we gave the time parameter based on an
> 800 GB P3700 SSD. Since your SSD is 2 TB you will need to multiply
> that by 2.5 so you should run with the following options ./perf -q
> 32 -s 131072 -w write -t 3000 to fill the SSD with sequential
> writes 2 times.
>
> *From:*Mayank [mailto:mayank.patel(a)mayadata.io
> <mailto:mayank.patel(a)mayadata.io>]
> *Sent:* Tuesday, December 19, 2017 10:12 PM
>
>
> *To:* Kariuki, John K <john.k.kariuki(a)intel.com
> <mailto:john.k.kariuki(a)intel.com>>; Storage Performance
> Development Kit <spdk(a)lists.01.org <mailto:spdk(a)lists.01.org>>
> *Subject:* Re: [SPDK] SPDK FIO poor 4K read performance
>
> Hi John,
>
> Thanks for sharing results.
>
> In my latency data, QD=1.
>
> Can you run perf with the following options(./perf -q 1 -s 4096 -t
> 600 -w randread) and share the output?
>
> results.
>
> [root(a)localhost spdk]# ./examples/nvme/perf/perf -q 1 -s 4096 -t
> 600 -w randread
> Starting DPDK 17.11.0 initialization...
> [ DPDK EAL parameters: perf -c 0x1 --file-prefix=spdk_pid24378 ]
> EAL: Detected 112 lcore(s)
> EAL: Probing VFIO support...
> EAL: WARNING: Master core has no memory on local socket!
> Initializing NVMe Controllers
> EAL: PCI device 0000:18:00.0 on NUMA socket 0
> EAL: probe driver: 8086:953 spdk_nvme
> Attaching to NVMe Controller at 0000:18:00.0 [8086:0953]
> Attached to NVMe Controller at 0000:18:00.0 [8086:0953]
> Associating INTEL SSDPEDMD020T4 (CVFT4415009B2P0EGN ) with lcore 0
> Initialization complete. Launching workers.
> Starting thread on core 0
> ========================================================
> Latency(us)
> Device Information : IOPS MB/s Average
> min max
> INTEL SSDPEDMD020T4 (CVFT4415009B2P0EGN ) from core 0:
> 158706.03 619.95 6.28 4.37 831.51
> ========================================================
> Total : 158706.03 619.95 6.28 4.37 831.51
>
> *** in this, latency is very less compared to fio!
>
> Regarding PCI speed, snapshot from lspci for NVME SSD
>
> LnkSta: Speed 8GT/s, Width x4, TrErr- Train- SlotClk- DLActive-
> BWMgmt- ABWMgmt-
>
> Can you perform randwrite with your setup, if possible?
>
> On Wednesday 20 December 2017 03:17 AM, Kariuki, John K wrote:
>
> Mayank
>
> Thank you for sharing this information.
>
> On the latency data (rand read 11.2k 45.9MB/s 88.83 usec),
> can you confirm that the QD=1?
>
> I have a system with P3700 SSD (U.2 form factor) and I am
> getting a latency number of around 81 usecs with both perf and
> FIO.
>
> SPDK PERF:
>
> ./perf -q 1 -s 4096 -t 600 -w randread -r 'trtype:PCIe
> traddr:0000:62:00.0'
>
> Starting DPDK 17.08.0 initialization...
>
> [ DPDK EAL parameters: perf -c 0x1 --file-prefix=spdk_pid43053 ]
>
> EAL: Detected 112 lcore(s)
>
> EAL: No free hugepages reported in hugepages-1048576kB
>
> EAL: Probing VFIO support...
>
> Initializing NVMe Controllers
>
> EAL: PCI device 0000:62:00.0 on NUMA socket 0
>
> EAL: probe driver: 8086:953 spdk_nvme
>
> Attaching to NVMe Controller at 0000:62:00.0 [8086:0953]
>
> Attached to NVMe Controller at 0000:62:00.0 [8086:0953]
>
> Associating INTEL SSDPE2MD400G4 (CVFT6175001C400GGN ) with
> lcore 0
>
> Initialization complete. Launching workers.
>
> Starting thread on core 0
>
> ========================================================
>
> Latency(us)
>
> Device Information : IOPS MB/s Average
> min max
>
> INTEL SSDPE2MD400G4 (CVFT6175001C400GGN ) from core 0:
> 12241.53 47.82 81.66 34.86 2891.90
>
> ========================================================
>
> Total : 12241.53 47.82 81.66 34.86 2891.90
>
> FIO:
>
> RW=randrw MIX=100 BLK_SIZE=4096 IODEPTH=1 NUMJOBS=1
> LD_PRELOAD=/home/john/spdk/examples/nvme/fio_plugin/fio_plugin
> fio fio.spdk_1ssd
>
> filename1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W)
> 4096B-4096B, (T) 4096B-4096B, ioengine=spdk, iodepth=1
>
> fio-3.2-84-g9b509
>
> Starting 1 thread
>
> Starting DPDK 17.08.0 initialization...
>
> [ DPDK EAL parameters: fio -c 0x1 -m 512
> --file-prefix=spdk_pid43241 ]
>
> EAL: Detected 112 lcore(s)
>
> EAL: No free hugepages reported in hugepages-1048576kB
>
> EAL: Probing VFIO support...
>
> EAL: PCI device 0000:62:00.0 on NUMA socket 0
>
> EAL: probe driver: 8086:953 spdk_nvme
>
> Jobs: 1 (f=0): [f(1)][100.0%][r=0KiB/s,w=0KiB/s][r=0,w=0
> IOPS][eta 00m:00s]
>
> filename1: (groupid=0, jobs=1): err= 0: pid=43356: Tue Dec 19
> 06:23:29 2017
>
> read: IOPS=12.2k, BW=47.5MiB/s (49.8MB/s)(27.8GiB/600001msec)
>
> slat (nsec): min=94, max=87632, avg=108.13, stdev=40.75
>
> clat (nsec): min=196, max=3008.9k, avg=81777.09,
> stdev=38852.81
>
> lat (usec): min=34, max=3009, avg=81.89, stdev=38.85
>
> clat percentiles (usec):
>
> | 50.000th=[ 78], 99.000th=[ 94], 99.900th=[ 241],
> 99.990th=[ 2278],
>
> | 99.999th=[ 2671]
>
> bw ( KiB/s): min=42738, max=50248, per=100.00%,
> avg=48802.24, stdev=1522.91, samples=1199
>
> iops : min=10684, max=12562, avg=12200.43,
> stdev=380.76, samples=1199
>
> lat (nsec) : 250=0.01%
>
> lat (usec) : 50=0.01%, 100=99.78%, 250=0.12%, 500=0.04%,
> 750=0.01%
>
> lat (usec) : 1000=0.01%
>
> lat (msec) : 2=0.02%, 4=0.02%
>
> cpu : usr=99.98%, sys=0.00%, ctx=51300, majf=0, minf=266
>
> IO depths : 1=110.1%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%,
> 32=0.0%, >=64=0.0%
>
> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%,
> 64=0.0%, >=64=0.0%
>
> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%,
> 64=0.0%, >=64=0.0%
>
> issued rwt: total=7298991,0,0, short=0,0,0, dropped=0,0,0
>
> latency : target=0, window=0, percentile=100.00%, depth=1
>
> Run status group 0 (all jobs):
>
> READ: bw=47.5MiB/s (49.8MB/s), 47.5MiB/s-47.5MiB/s
> (49.8MB/s-49.8MB/s), io=27.8GiB (29.9GB), run=600001-600001msec
>
> Here is the FIO configuration, I usually use in my test
> environment
>
> [global]
>
> ioengine=spdk
>
> direct=1
>
> time_based=1
>
> norandommap=1
>
> ramp_time=60s
>
> runtime=600s
>
> thread=1
>
> group_reporting=1
>
> percentile_list=50:99:99.9:99.99:99.999
>
> rw=${RW}
>
> rwmixread=${MIX}
>
> bs=${BLK_SIZE}
>
> iodepth=${IODEPTH}
>
> numjobs=${NUMJOBS}
>
> [filename1]
>
> filename=trtype=PCIe traddr=0000.62.00.0 ns=1
>
> cpumask=0x1
>
> I ran the test with your fio options and the latency is pretty
> consistent with perf
>
> fio --time_based --name=benchmark --size=10G --runtime=600
> '--filename=trtype=PCIe traddr=0000.62.00.0 ns=1'
> --ioengine=/home/john/spdk/examples/nvme/fio_plugin/fio_plugin
> --randrepeat=0 --iodepth=1 --thread --direct=1 --invalidate=1
> --verify=0 --verify_fatal=0 --numjobs=1 --rw=randread --sync=1
> --blocksize=4k --norandommap=1 --group_reporting
>
> benchmark: (g=0): rw=randread, bs=(R) 4096B-4096B, (W)
> 4096B-4096B, (T) 4096B-4096B, ioengine=spdk, iodepth=1
>
> fio-3.2-84-g9b509
>
> Starting 1 thread
>
> Starting DPDK 17.08.0 initialization...
>
> [ DPDK EAL parameters: fio -c 0x1 -m 512
> --file-prefix=spdk_pid43874 ]
>
> EAL: Detected 112 lcore(s)
>
> EAL: No free hugepages reported in hugepages-1048576kB
>
> EAL: Probing VFIO support...
>
> EAL: PCI device 0000:62:00.0 on NUMA socket 0
>
> EAL: probe driver: 8086:953 spdk_nvme
>
> Jobs: 1 (f=0): [f(1)][100.0%][r=4412KiB/s,w=0KiB/s][r=1103,w=0
> IOPS][eta 00m:00s]
>
> benchmark: (groupid=0, jobs=1): err= 0: pid=43989: Tue Dec 19
> 07:43:48 2017
>
> read: IOPS=12.2k, BW=47.5MiB/s (49.8MB/s)(27.8GiB/600001msec)
>
> slat (nsec): min=94, max=15837, avg=105.59, stdev=20.51
>
> clat (usec): min=23, max=2947, avg=81.73, stdev=37.37
>
> lat (usec): min=24, max=2947, avg=81.84, stdev=37.37
>
> clat percentiles (usec):
>
> | 1.00th=[ 63], 5.00th=[ 68], 10.00th=[ 72],
> 20.00th=[ 74],
>
> | 30.00th=[ 75], 40.00th=[ 76], 50.00th=[ 78],
> 60.00th=[ 87],
>
> | 70.00th=[ 89], 80.00th=[ 91], 90.00th=[ 92],
> 95.00th=[ 93],
>
> | 99.00th=[ 94], 99.50th=[ 94], 99.90th=[ 221],
> 99.95th=[ 553],
>
> | 99.99th=[ 2278]
>
> bw ( KiB/s): min=42618, max=49384, per=99.98%,
> avg=48644.25, stdev=1407.57, samples=1199
>
> iops : min=10654, max=12346, avg=12161.03,
> stdev=351.90, samples=1199
>
> lat (usec) : 50=0.01%, 100=99.79%, 250=0.11%, 500=0.04%,
> 750=0.01%
>
> lat (usec) : 1000=0.01%
>
> lat (msec) : 2=0.02%, 4=0.02%
>
> cpu : usr=100.00%, sys=0.00%, ctx=50106, majf=0, minf=280
>
> IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%,
> 32=0.0%, >=64=0.0%
>
> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%,
> 64=0.0%, >=64=0.0%
>
> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%,
> 64=0.0%, >=64=0.0%
>
> issued rwt: total=7298286,0,0, short=0,0,0, dropped=0,0,0
>
> latency : target=0, window=0, percentile=100.00%, depth=1
>
> Run status group 0 (all jobs):
>
> READ: bw=47.5MiB/s (49.8MB/s), 47.5MiB/s-47.5MiB/s
> (49.8MB/s-49.8MB/s), io=27.8GiB (29.9GB), run=600001-600001msec
>
> Can you run perf with the following options(./perf -q 1 -s
> 4096 -t 600 -w randread) and share the output?
>
> Have you checked the PCI link capability and status to verify
> the Speed is 8GT/s and the width is x4?
>
> On the sequential reads, you need to use a larger block size
> of 128K to reach the peak throughput.
>
> *From:*Mayank [mailto:mayank.patel(a)mayadata.io]
> *Sent:* Monday, December 18, 2017 11:33 PM
> *To:* Kariuki, John K <john.k.kariuki(a)intel.com>
> <mailto:john.k.kariuki(a)intel.com>; Storage Performance
> Development Kit <spdk(a)lists.01.org> <mailto:spdk(a)lists.01.org>
> *Subject:* Re: [SPDK] SPDK FIO poor 4K read performance
>
> Hi John,
>
> On Tuesday 19 December 2017 04:54 AM, Kariuki, John K wrote:
>
> Mayank
>
> Can you share your fio configuration file?
>
> i used following fio command :
>
> fio --time_based --time_based --name=benchmark --size=10G
> --runtime=600 '--filename=trtype=PCIe traddr=0000.18.00.0
> ns=1' --ioengine=spdk/examples/nvme/fio_plugin/fio_plugin
> --randrepeat=0 --iodepth=32 --thread --direct=1 --invalidate=1
> --verify=0 --verify_fatal=0 --numjobs=4 --rw=randwrite
> --sync=1 --blocksize=4k --norandommap=1 --group_reportin
>
> Have you tried the latency performance test with the SPDK
> perf tool?
>
> Yes, i tried it. Results for write with run-time 300sec is as
> below.
> ./examples/nvme/perf/perf -q 32 -s 4096 -t 300 -w write
> ========================================================
> Latency(us)
> Device Information : IOPS MB/s Average
> min max
> INTEL SSDPEDMD020T4 (CVFT4415009B2P0EGN ) from core 0:
> 436776.09 1706.16 73.26 4.69 10563.94
> ========================================================
> Total :
> 436776.09 1706.16 73.26 4.69 10563.94
> ################
> + ./examples/nvme/perf/perf -q 32 -s 4096 -t 300 -w read
> ========================================================
> Latency(us)
> Device Information : IOPS MB/s Average
> min max
> INTEL SSDPEDMD020T4 (CVFT4415009B2P0EGN ) from core 0:
> 213554.89 834.20 149.84 4.05 9111.46
> ========================================================
> Total :
> 213554.89 834.20 149.84 4.05 9111.46
>
>
>
> Your random writes are pretty high. Did you precondition
> the SSD before the test? For the random write test you
> need to precondition the SSD by doing 90 minutes of random
> writes before testing.
>
> Yes, I preconditioned SSD before test.
>
> Here is a link to a video
> <https://www.youtube.com/watch?v=tkGE3pq5eIU&index=8&list=PLj-81kG3zG5ZIE-4CvqsvlFEHoOoWRIHZ>
> that highlights some performance best practices that you
> might want to check out.
>
> Thanks for sharing video. I have already gone through this
> presentation and followed necessary steps mentioned in this
> presentation.
>
> *From:*SPDK [mailto:spdk-bounces(a)lists.01.org] *On Behalf
> Of *Andrey Kuzmin
> *Sent:* Monday, December 18, 2017 3:40 PM
> *To:* Storage Performance Development Kit
> <spdk(a)lists.01.org> <mailto:spdk(a)lists.01.org>
> *Cc:* mayank.patel(a)mayadata.io
> <mailto:mayank.patel(a)mayadata.io>
> *Subject:* Re: [SPDK] SPDK FIO poor 4K read performance
>
> On Tue, Dec 19, 2017, 01:27 Isaac Otsiabah
> <IOtsiabah(a)us.fujitsu.com
> <mailto:IOtsiabah(a)us.fujitsu.com>> wrote:
>
> Hi Audrey,
>
> Hi Isaac,
>
> I guess you meant to address the original submission
> author, Mayank.
>
> Regards,
>
> Andrey
>
> which
>
> plugin are you running?
> examples/nvme/fio_plugin/fio_plugin or
> examples/bdev/fio_plugin/fio_plugin ? Is it on Fedora
> or Centos?
>
> Isaac
>
> *From:*SPDK [mailto:spdk-bounces(a)lists.01.org
> <mailto:spdk-bounces(a)lists.01.org>] *On Behalf Of
> *Andrey Kuzmin
> *Sent:* Monday, December 18, 2017 5:59 AM
> *To:* Storage Performance Development Kit
> <spdk(a)lists.01.org <mailto:spdk(a)lists.01.org>>
> *Subject:* Re: [SPDK] SPDK FIO poor 4K read performance
>
> On Mon, Dec 18, 2017, 16:08 Mayank
> <mayank.patel(a)mayadata.io
> <mailto:mayank.patel(a)mayadata.io>> wrote:
>
> Hi All,
>
> I have performed fio benchmark test with SPDK fio
> plugin for throughput test with iodepth = 32 and
> numjobs=4.
>
> Latency results are as below:
>
> rand read
>
>
>
> 11.2k
>
>
>
> 45.9MB/s
>
>
>
> 88.83 usec
>
> rand write
>
>
>
> 126k
>
>
>
> 518MB/s
>
>
>
> 7.56 usec
>
> Throughput results are as below:
>
> Throughput (io:32 thr:4)
>
>
>
> IOPS
>
>
>
> Throuhgput
>
>
>
> Latency (usec)
>
> seq read
>
>
>
> 406k
>
>
>
> 1665MB/s
>
>
>
> 314.71
>
> seq write
>
>
>
> 455k
>
>
>
> 1865MB/s
>
>
>
> 280.87
>
> rand read
>
>
>
> 478k
>
>
>
> 1959MB/s
>
>
>
> 267.29
>
> rand write
>
>
>
> 442k
>
>
>
> 1812MB/s
>
>
>
> 289.03
>
>
>
> In results, 'm getting sequential read IOPS lesser
> than sequential write IOPS.
>
> I am using NVME model number : INTEL SSDPEDMD020T4.
> kernel : 4.13.13
> fio version : 2.21
>
> BTW, your fio version is couple of years behind.
>
>
> can anybody tell me what could be the issue?
>
> The latency data above clearly shows the reason:
> writes are being handled by the internal SSD cache of
> few usec latency, while reads are served off the real
> media, with typical (high-end) SSD latency of about
> 100 usec.
>
> Regards,
>
> Andrey
>
>
>
> Thanks,
> Mayank
>
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org <mailto:SPDK(a)lists.01.org>
> https://lists.01.org/mailman/listinfo/spdk
>
> --
>
> Regards,
> Andrey
>
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org <mailto:SPDK(a)lists.01.org>
> https://lists.01.org/mailman/listinfo/spdk
>
> --
>
> Regards,
> Andrey
>
[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 123657 bytes --]
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [SPDK] SPDK FIO poor 4K read performance
@ 2018-01-08 16:14 Kariuki, John K
0 siblings, 0 replies; 15+ messages in thread
From: Kariuki, John K @ 2018-01-08 16:14 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 19575 bytes --]
Mayank
No problem. Thanks for the update.
Did you do the workload specific pre-conditioning before the 100% random write test? For the 4K 100% random write workload, we do a workload specific preconditioning for 90 minutes, which can be done with perf command: ./perf -q 32 -s 4096 -w randwrite -t 5400
Can you also share the command inputs your using? I am looking at some of the older emails and the workload (-w) is set to write in runs instead of randwrite?
From: Mayank Patel [mailto:mayank.patel(a)mayadata.io]
Sent: Monday, January 08, 2018 2:01 AM
To: Kariuki, John K <john.k.kariuki(a)intel.com>
Cc: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] SPDK FIO poor 4K read performance
Hi John,
Thanks for suggesting preconditioning changes.
I ran precondition test with your changes for 2 times. Now, i am getting read latency same as your results.
Sorry i couldn't reply you on time.
After filling up SSD, random read results with perf is as follow:
Starting DPDK 17.11.0 initialization...
[ DPDK EAL parameters: perf -c 0x1 --file-prefix=spdk_pid178910 ]
EAL: Detected 112 lcore(s)
EAL: Probing VFIO support...
Initializing NVMe Controllers
EAL: PCI device 0000:18:00.0 on NUMA socket 0
EAL: probe driver: 8086:953 spdk_nvme
Attaching to NVMe Controller at 0000:18:00.0 [8086:0953]
Attached to NVMe Controller at 0000:18:00.0 [8086:0953]
Associating INTEL SSDPEDMD020T4 (CVFT4415009B2P0EGN ) with lcore 0
Initialization complete. Launching workers.
Starting thread on core 0
========================================================
Latency(us)
Device Information : IOPS MB/s Average min max
INTEL SSDPEDMD020T4 (CVFT4415009B2P0EGN ) from core 0: 11014.63 43.03 90.77 34.15 3730.32
========================================================
Total : 11014.63 43.03 90.77 34.15 3730.32
I've also ran fio test with QD=1
benchmark: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk, iodepth=1
fio-3.2
Starting 1 thread
Starting DPDK 17.11.0 initialization...
[ DPDK EAL parameters: fio -c 0x1 -m 512 --file-prefix=spdk_pid225681 ]
EAL: Detected 112 lcore(s)
EAL: Probing VFIO support...
EAL: PCI device 0000:18:00.0 on NUMA socket 0
EAL: probe driver: 8086:953 spdk_nvme
benchmark: (groupid=0, jobs=1): err= 0: pid=225812: Mon Jan 8 03:34:46 2018
read: IOPS=11.2k, BW=43.8MiB/s (45.9MB/s)(25.6GiB/600001msec)
slat (nsec): min=178, max=20412, avg=213.32, stdev=66.07
clat (usec): min=4, max=3226, avg=88.66, stdev=46.20
lat (usec): min=5, max=3226, avg=88.87, stdev=46.20
clat percentiles (usec):
| 1.00th=[ 69], 5.00th=[ 74], 10.00th=[ 78], 20.00th=[ 80],
| 30.00th=[ 82], 40.00th=[ 83], 50.00th=[ 84], 60.00th=[ 94],
| 70.00th=[ 96], 80.00th=[ 98], 90.00th=[ 99], 95.00th=[ 100],
| 99.00th=[ 101], 99.50th=[ 109], 99.90th=[ 297], 99.95th=[ 889],
| 99.99th=[ 2573]
bw ( KiB/s): min=39799, max=55815, per=100.00%, avg=45083.50, stdev=1636.36, samples=1200
iops : min= 9949, max=13953, avg=11270.50, stdev=409.05, samples=1200
lat (usec) : 10=0.01%, 20=0.01%, 50=0.54%, 100=95.61%, 250=3.69%
lat (usec) : 500=0.09%, 750=0.02%, 1000=0.01%
lat (msec) : 2=0.02%, 4=0.02%
cpu : usr=99.75%, sys=0.03%, ctx=157, majf=0, minf=2304
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwt: total=6720907,0,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=43.8MiB/s (45.9MB/s), 43.8MiB/s-43.8MiB/s (45.9MB/s-45.9MB/s), io=25.6GiB (27.5GB), run=600001-600001msec
Now, random read results are same as yours, but write results are still bothering me... Since latency in random write is very low compare to random read.
Regards,
Mayank
On Thu, Dec 21, 2017 at 11:24 PM, Kariuki, John K <john.k.kariuki(a)intel.com<mailto:john.k.kariuki(a)intel.com>> wrote:
Mayank
That average latency (4K, random read, QD=1) your showing is very low. For a P3700 SSD that number should be in the low 80 usecs. I usually see numbers that low when my SSD is empty which usually indicates that the controller is returning the data without actually going to the NAND media to read data because it knows the media does not contain any data. Did you fill your SSD 2 times with sequential writes? What parameters did you use to fill your drive? In our presentation, we gave the time parameter based on an 800 GB P3700 SSD. Since your SSD is 2 TB you will need to multiply that by 2.5 so you should run with the following options ./perf -q 32 -s 131072 -w write -t 3000 to fill the SSD with sequential writes 2 times.
From: Mayank [mailto:mayank.patel(a)mayadata.io<mailto:mayank.patel(a)mayadata.io>]
Sent: Tuesday, December 19, 2017 10:12 PM
To: Kariuki, John K <john.k.kariuki(a)intel.com<mailto:john.k.kariuki(a)intel.com>>; Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] SPDK FIO poor 4K read performance
Hi John,
Thanks for sharing results.
In my latency data, QD=1.
Can you run perf with the following options(./perf -q 1 -s 4096 -t 600 -w randread) and share the output?
results.
[root(a)localhost spdk]# ./examples/nvme/perf/perf -q 1 -s 4096 -t 600 -w randread
Starting DPDK 17.11.0 initialization...
[ DPDK EAL parameters: perf -c 0x1 --file-prefix=spdk_pid24378 ]
EAL: Detected 112 lcore(s)
EAL: Probing VFIO support...
EAL: WARNING: Master core has no memory on local socket!
Initializing NVMe Controllers
EAL: PCI device 0000:18:00.0 on NUMA socket 0
EAL: probe driver: 8086:953 spdk_nvme
Attaching to NVMe Controller at 0000:18:00.0 [8086:0953]
Attached to NVMe Controller at 0000:18:00.0 [8086:0953]
Associating INTEL SSDPEDMD020T4 (CVFT4415009B2P0EGN ) with lcore 0
Initialization complete. Launching workers.
Starting thread on core 0
========================================================
Latency(us)
Device Information : IOPS MB/s Average min max
INTEL SSDPEDMD020T4 (CVFT4415009B2P0EGN ) from core 0: 158706.03 619.95 6.28 4.37 831.51
========================================================
Total : 158706.03 619.95 6.28 4.37 831.51
*** in this, latency is very less compared to fio!
Regarding PCI speed, snapshot from lspci for NVME SSD
LnkSta: Speed 8GT/s, Width x4, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
Can you perform randwrite with your setup, if possible?
On Wednesday 20 December 2017 03:17 AM, Kariuki, John K wrote:
Mayank
Thank you for sharing this information.
On the latency data (rand read 11.2k 45.9MB/s 88.83 usec), can you confirm that the QD=1?
I have a system with P3700 SSD (U.2 form factor) and I am getting a latency number of around 81 usecs with both perf and FIO.
SPDK PERF:
./perf -q 1 -s 4096 -t 600 -w randread -r 'trtype:PCIe traddr:0000:62:00.0'
Starting DPDK 17.08.0 initialization...
[ DPDK EAL parameters: perf -c 0x1 --file-prefix=spdk_pid43053 ]
EAL: Detected 112 lcore(s)
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
Initializing NVMe Controllers
EAL: PCI device 0000:62:00.0 on NUMA socket 0
EAL: probe driver: 8086:953 spdk_nvme
Attaching to NVMe Controller at 0000:62:00.0 [8086:0953]
Attached to NVMe Controller at 0000:62:00.0 [8086:0953]
Associating INTEL SSDPE2MD400G4 (CVFT6175001C400GGN ) with lcore 0
Initialization complete. Launching workers.
Starting thread on core 0
========================================================
Latency(us)
Device Information : IOPS MB/s Average min max
INTEL SSDPE2MD400G4 (CVFT6175001C400GGN ) from core 0: 12241.53 47.82 81.66 34.86 2891.90
========================================================
Total : 12241.53 47.82 81.66 34.86 2891.90
FIO:
RW=randrw MIX=100 BLK_SIZE=4096 IODEPTH=1 NUMJOBS=1 LD_PRELOAD=/home/john/spdk/examples/nvme/fio_plugin/fio_plugin fio fio.spdk_1ssd
filename1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk, iodepth=1
fio-3.2-84-g9b509
Starting 1 thread
Starting DPDK 17.08.0 initialization...
[ DPDK EAL parameters: fio -c 0x1 -m 512 --file-prefix=spdk_pid43241 ]
EAL: Detected 112 lcore(s)
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: PCI device 0000:62:00.0 on NUMA socket 0
EAL: probe driver: 8086:953 spdk_nvme
Jobs: 1 (f=0): [f(1)][100.0%][r=0KiB/s,w=0KiB/s][r=0,w=0 IOPS][eta 00m:00s]
filename1: (groupid=0, jobs=1): err= 0: pid=43356: Tue Dec 19 06:23:29 2017
read: IOPS=12.2k, BW=47.5MiB/s (49.8MB/s)(27.8GiB/600001msec)
slat (nsec): min=94, max=87632, avg=108.13, stdev=40.75
clat (nsec): min=196, max=3008.9k, avg=81777.09, stdev=38852.81
lat (usec): min=34, max=3009, avg=81.89, stdev=38.85
clat percentiles (usec):
| 50.000th=[ 78], 99.000th=[ 94], 99.900th=[ 241], 99.990th=[ 2278],
| 99.999th=[ 2671]
bw ( KiB/s): min=42738, max=50248, per=100.00%, avg=48802.24, stdev=1522.91, samples=1199
iops : min=10684, max=12562, avg=12200.43, stdev=380.76, samples=1199
lat (nsec) : 250=0.01%
lat (usec) : 50=0.01%, 100=99.78%, 250=0.12%, 500=0.04%, 750=0.01%
lat (usec) : 1000=0.01%
lat (msec) : 2=0.02%, 4=0.02%
cpu : usr=99.98%, sys=0.00%, ctx=51300, majf=0, minf=266
IO depths : 1=110.1%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwt: total=7298991,0,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=47.5MiB/s (49.8MB/s), 47.5MiB/s-47.5MiB/s (49.8MB/s-49.8MB/s), io=27.8GiB (29.9GB), run=600001-600001msec
Here is the FIO configuration, I usually use in my test environment
[global]
ioengine=spdk
direct=1
time_based=1
norandommap=1
ramp_time=60s
runtime=600s
thread=1
group_reporting=1
percentile_list=50:99:99.9:99.99:99.999
rw=${RW}
rwmixread=${MIX}
bs=${BLK_SIZE}
iodepth=${IODEPTH}
numjobs=${NUMJOBS}
[filename1]
filename=trtype=PCIe traddr=0000.62.00.0 ns=1
cpumask=0x1
I ran the test with your fio options and the latency is pretty consistent with perf
fio --time_based --name=benchmark --size=10G --runtime=600 '--filename=trtype=PCIe traddr=0000.62.00.0 ns=1' --ioengine=/home/john/spdk/examples/nvme/fio_plugin/fio_plugin --randrepeat=0 --iodepth=1 --thread --direct=1 --invalidate=1 --verify=0 --verify_fatal=0 --numjobs=1 --rw=randread --sync=1 --blocksize=4k --norandommap=1 --group_reporting
benchmark: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk, iodepth=1
fio-3.2-84-g9b509
Starting 1 thread
Starting DPDK 17.08.0 initialization...
[ DPDK EAL parameters: fio -c 0x1 -m 512 --file-prefix=spdk_pid43874 ]
EAL: Detected 112 lcore(s)
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: PCI device 0000:62:00.0 on NUMA socket 0
EAL: probe driver: 8086:953 spdk_nvme
Jobs: 1 (f=0): [f(1)][100.0%][r=4412KiB/s,w=0KiB/s][r=1103,w=0 IOPS][eta 00m:00s]
benchmark: (groupid=0, jobs=1): err= 0: pid=43989: Tue Dec 19 07:43:48 2017
read: IOPS=12.2k, BW=47.5MiB/s (49.8MB/s)(27.8GiB/600001msec)
slat (nsec): min=94, max=15837, avg=105.59, stdev=20.51
clat (usec): min=23, max=2947, avg=81.73, stdev=37.37
lat (usec): min=24, max=2947, avg=81.84, stdev=37.37
clat percentiles (usec):
| 1.00th=[ 63], 5.00th=[ 68], 10.00th=[ 72], 20.00th=[ 74],
| 30.00th=[ 75], 40.00th=[ 76], 50.00th=[ 78], 60.00th=[ 87],
| 70.00th=[ 89], 80.00th=[ 91], 90.00th=[ 92], 95.00th=[ 93],
| 99.00th=[ 94], 99.50th=[ 94], 99.90th=[ 221], 99.95th=[ 553],
| 99.99th=[ 2278]
bw ( KiB/s): min=42618, max=49384, per=99.98%, avg=48644.25, stdev=1407.57, samples=1199
iops : min=10654, max=12346, avg=12161.03, stdev=351.90, samples=1199
lat (usec) : 50=0.01%, 100=99.79%, 250=0.11%, 500=0.04%, 750=0.01%
lat (usec) : 1000=0.01%
lat (msec) : 2=0.02%, 4=0.02%
cpu : usr=100.00%, sys=0.00%, ctx=50106, majf=0, minf=280
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwt: total=7298286,0,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=47.5MiB/s (49.8MB/s), 47.5MiB/s-47.5MiB/s (49.8MB/s-49.8MB/s), io=27.8GiB (29.9GB), run=600001-600001msec
Can you run perf with the following options(./perf -q 1 -s 4096 -t 600 -w randread) and share the output?
Have you checked the PCI link capability and status to verify the Speed is 8GT/s and the width is x4?
On the sequential reads, you need to use a larger block size of 128K to reach the peak throughput.
From: Mayank [mailto:mayank.patel(a)mayadata.io]
Sent: Monday, December 18, 2017 11:33 PM
To: Kariuki, John K <john.k.kariuki(a)intel.com><mailto:john.k.kariuki(a)intel.com>; Storage Performance Development Kit <spdk(a)lists.01.org><mailto:spdk(a)lists.01.org>
Subject: Re: [SPDK] SPDK FIO poor 4K read performance
Hi John,
On Tuesday 19 December 2017 04:54 AM, Kariuki, John K wrote:
Mayank
Can you share your fio configuration file?
i used following fio command :
fio --time_based --time_based --name=benchmark --size=10G --runtime=600 '--filename=trtype=PCIe traddr=0000.18.00.0 ns=1' --ioengine=spdk/examples/nvme/fio_plugin/fio_plugin --randrepeat=0 --iodepth=32 --thread --direct=1 --invalidate=1 --verify=0 --verify_fatal=0 --numjobs=4 --rw=randwrite --sync=1 --blocksize=4k --norandommap=1 --group_reportin
Have you tried the latency performance test with the SPDK perf tool?
Yes, i tried it. Results for write with run-time 300sec is as below.
./examples/nvme/perf/perf -q 32 -s 4096 -t 300 -w write
========================================================
Latency(us)
Device Information : IOPS MB/s Average min max
INTEL SSDPEDMD020T4 (CVFT4415009B2P0EGN ) from core 0: 436776.09 1706.16 73.26 4.69 10563.94
========================================================
Total : 436776.09 1706.16 73.26 4.69 10563.94
################
+ ./examples/nvme/perf/perf -q 32 -s 4096 -t 300 -w read
========================================================
Latency(us)
Device Information : IOPS MB/s Average min max
INTEL SSDPEDMD020T4 (CVFT4415009B2P0EGN ) from core 0: 213554.89 834.20 149.84 4.05 9111.46
========================================================
Total : 213554.89 834.20 149.84 4.05 9111.46
Your random writes are pretty high. Did you precondition the SSD before the test? For the random write test you need to precondition the SSD by doing 90 minutes of random writes before testing.
Yes, I preconditioned SSD before test.
Here is a link to a video<https://www.youtube.com/watch?v=tkGE3pq5eIU&index=8&list=PLj-81kG3zG5ZIE-4CvqsvlFEHoOoWRIHZ> that highlights some performance best practices that you might want to check out.
Thanks for sharing video. I have already gone through this presentation and followed necessary steps mentioned in this presentation.
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Andrey Kuzmin
Sent: Monday, December 18, 2017 3:40 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org><mailto:spdk(a)lists.01.org>
Cc: mayank.patel(a)mayadata.io<mailto:mayank.patel(a)mayadata.io>
Subject: Re: [SPDK] SPDK FIO poor 4K read performance
On Tue, Dec 19, 2017, 01:27 Isaac Otsiabah <IOtsiabah(a)us.fujitsu.com<mailto:IOtsiabah(a)us.fujitsu.com>> wrote:
Hi Audrey,
Hi Isaac,
I guess you meant to address the original submission author, Mayank.
Regards,
Andrey
which
plugin are you running? examples/nvme/fio_plugin/fio_plugin or examples/bdev/fio_plugin/fio_plugin ? Is it on Fedora or Centos?
Isaac
From: SPDK [mailto:spdk-bounces(a)lists.01.org<mailto:spdk-bounces(a)lists.01.org>] On Behalf Of Andrey Kuzmin
Sent: Monday, December 18, 2017 5:59 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] SPDK FIO poor 4K read performance
On Mon, Dec 18, 2017, 16:08 Mayank <mayank.patel(a)mayadata.io<mailto:mayank.patel(a)mayadata.io>> wrote:
Hi All,
I have performed fio benchmark test with SPDK fio plugin for throughput test with iodepth = 32 and numjobs=4.
Latency results are as below:
rand read
11.2k
45.9MB/s
88.83 usec
rand write
126k
518MB/s
7.56 usec
Throughput results are as below:
Throughput (io:32 thr:4)
IOPS
Throuhgput
Latency (usec)
seq read
406k
1665MB/s
314.71
seq write
455k
1865MB/s
280.87
rand read
478k
1959MB/s
267.29
rand write
442k
1812MB/s
289.03
In results, 'm getting sequential read IOPS lesser than sequential write IOPS.
I am using NVME model number : INTEL SSDPEDMD020T4.
kernel : 4.13.13
fio version : 2.21
BTW, your fio version is couple of years behind.
can anybody tell me what could be the issue?
The latency data above clearly shows the reason: writes are being handled by the internal SSD cache of few usec latency, while reads are served off the real media, with typical (high-end) SSD latency of about 100 usec.
Regards,
Andrey
Thanks,
Mayank
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org<mailto:SPDK(a)lists.01.org>
https://lists.01.org/mailman/listinfo/spdk
--
Regards,
Andrey
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org<mailto:SPDK(a)lists.01.org>
https://lists.01.org/mailman/listinfo/spdk
--
Regards,
Andrey
[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 71231 bytes --]
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [SPDK] SPDK FIO poor 4K read performance
@ 2018-01-08 9:00 Mayank Patel
0 siblings, 0 replies; 15+ messages in thread
From: Mayank Patel @ 2018-01-08 9:00 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 19606 bytes --]
Hi John,
Thanks for suggesting preconditioning changes.
I ran precondition test with your changes for 2 times. Now, i am getting
read latency same as your results.
Sorry i couldn't reply you on time.
After filling up SSD, random read results with perf is as follow:
Starting DPDK 17.11.0 initialization...
[ DPDK EAL parameters: perf -c 0x1 --file-prefix=spdk_pid178910 ]
EAL: Detected 112 lcore(s)
EAL: Probing VFIO support...
Initializing NVMe Controllers
EAL: PCI device 0000:18:00.0 on NUMA socket 0
EAL: probe driver: 8086:953 spdk_nvme
Attaching to NVMe Controller at 0000:18:00.0 [8086:0953]
Attached to NVMe Controller at 0000:18:00.0 [8086:0953]
Associating INTEL SSDPEDMD020T4 (CVFT4415009B2P0EGN ) with lcore 0
Initialization complete. Launching workers.
Starting thread on core 0
========================================================
Latency(us)
Device Information : IOPS
MB/s Average min max
INTEL SSDPEDMD020T4 (CVFT4415009B2P0EGN ) from core 0: 11014.63
43.03 90.77 34.15 3730.32
========================================================
Total : 11014.63
43.03 90.77 34.15 3730.32
I've also ran fio test with QD=1
benchmark: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T)
4096B-4096B, ioengine=spdk, iodepth=1
fio-3.2
Starting 1 thread
Starting DPDK 17.11.0 initialization...
[ DPDK EAL parameters: fio -c 0x1 -m 512 --file-prefix=spdk_pid225681 ]
EAL: Detected 112 lcore(s)
EAL: Probing VFIO support...
EAL: PCI device 0000:18:00.0 on NUMA socket 0
EAL: probe driver: 8086:953 spdk_nvme
benchmark: (groupid=0, jobs=1): err= 0: pid=225812: Mon Jan 8 03:34:46 2018
read: IOPS=11.2k, BW=43.8MiB/s (45.9MB/s)(25.6GiB/600001msec)
slat (nsec): min=178, max=20412, avg=213.32, stdev=66.07
clat (usec): min=4, max=3226, avg=88.66, stdev=46.20
lat (usec): min=5, max=3226, avg=88.87, stdev=46.20
clat percentiles (usec):
| 1.00th=[ 69], 5.00th=[ 74], 10.00th=[ 78], 20.00th=[ 80],
| 30.00th=[ 82], 40.00th=[ 83], 50.00th=[ 84], 60.00th=[ 94],
| 70.00th=[ 96], 80.00th=[ 98], 90.00th=[ 99], 95.00th=[ 100],
| 99.00th=[ 101], 99.50th=[ 109], 99.90th=[ 297], 99.95th=[ 889],
| 99.99th=[ 2573]
bw ( KiB/s): min=39799, max=55815, per=100.00%, avg=45083.50,
stdev=1636.36, samples=1200
iops : min= 9949, max=13953, avg=11270.50, stdev=409.05,
samples=1200
lat (usec) : 10=0.01%, 20=0.01%, 50=0.54%, 100=95.61%, 250=3.69%
lat (usec) : 500=0.09%, 750=0.02%, 1000=0.01%
lat (msec) : 2=0.02%, 4=0.02%
cpu : usr=99.75%, sys=0.03%, ctx=157, majf=0, minf=2304
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%,
>=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
issued rwt: total=6720907,0,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=43.8MiB/s (45.9MB/s), 43.8MiB/s-43.8MiB/s (45.9MB/s-45.9MB/s),
io=25.6GiB (27.5GB), run=600001-600001msec
Now, random read results are same as yours, but write results are still
bothering me... Since latency in random write is very low compare to random
read.
Regards,
Mayank
On Thu, Dec 21, 2017 at 11:24 PM, Kariuki, John K <john.k.kariuki(a)intel.com>
wrote:
> Mayank
>
> That average latency (4K, random read, QD=1) your showing is very low. For
> a P3700 SSD that number should be in the low 80 usecs. I usually see
> numbers that low when my SSD is empty which usually indicates that the
> controller is returning the data without actually going to the NAND media
> to read data because it knows the media does not contain any data. Did you
> fill your SSD 2 times with sequential writes? What parameters did you use
> to fill your drive? In our presentation, we gave the time parameter based
> on an 800 GB P3700 SSD. Since your SSD is 2 TB you will need to multiply
> that by 2.5 so you should run with the following options ./perf -q 32 -s
> 131072 -w write -t 3000 to fill the SSD with sequential writes 2 times.
>
>
>
> *From:* Mayank [mailto:mayank.patel(a)mayadata.io]
> *Sent:* Tuesday, December 19, 2017 10:12 PM
>
> *To:* Kariuki, John K <john.k.kariuki(a)intel.com>; Storage Performance
> Development Kit <spdk(a)lists.01.org>
> *Subject:* Re: [SPDK] SPDK FIO poor 4K read performance
>
>
>
> Hi John,
>
> Thanks for sharing results.
>
> In my latency data, QD=1.
>
> Can you run perf with the following options(./perf -q 1 -s 4096 -t 600 -w
> randread) and share the output?
>
> results.
>
> [root(a)localhost spdk]# ./examples/nvme/perf/perf -q 1 -s 4096 -t 600 -w
> randread
> Starting DPDK 17.11.0 initialization...
> [ DPDK EAL parameters: perf -c 0x1 --file-prefix=spdk_pid24378 ]
> EAL: Detected 112 lcore(s)
> EAL: Probing VFIO support...
> EAL: WARNING: Master core has no memory on local socket!
> Initializing NVMe Controllers
> EAL: PCI device 0000:18:00.0 on NUMA socket 0
> EAL: probe driver: 8086:953 spdk_nvme
> Attaching to NVMe Controller at 0000:18:00.0 [8086:0953]
> Attached to NVMe Controller at 0000:18:00.0 [8086:0953]
> Associating INTEL SSDPEDMD020T4 (CVFT4415009B2P0EGN ) with lcore 0
> Initialization complete. Launching workers.
> Starting thread on core 0
> ========================================================
>
> Latency(us)
> Device Information : IOPS
> MB/s Average min max
> INTEL SSDPEDMD020T4 (CVFT4415009B2P0EGN ) from core 0: 158706.03
> 619.95 6.28 4.37 831.51
> ========================================================
> Total : 158706.03
> 619.95 6.28 4.37 831.51
>
> *** in this, latency is very less compared to fio!
>
> Regarding PCI speed, snapshot from lspci for NVME SSD
>
> LnkSta: Speed 8GT/s, Width x4, TrErr- Train- SlotClk- DLActive-
> BWMgmt- ABWMgmt-
>
> Can you perform randwrite with your setup, if possible?
>
>
>
> On Wednesday 20 December 2017 03:17 AM, Kariuki, John K wrote:
>
> Mayank
>
> Thank you for sharing this information.
>
> On the latency data (rand read 11.2k 45.9MB/s 88.83 usec), can you
> confirm that the QD=1?
>
> I have a system with P3700 SSD (U.2 form factor) and I am getting a
> latency number of around 81 usecs with both perf and FIO.
>
> SPDK PERF:
>
> ./perf -q 1 -s 4096 -t 600 -w randread -r 'trtype:PCIe traddr:0000:62:00.0'
>
> Starting DPDK 17.08.0 initialization...
>
> [ DPDK EAL parameters: perf -c 0x1 --file-prefix=spdk_pid43053 ]
>
> EAL: Detected 112 lcore(s)
>
> EAL: No free hugepages reported in hugepages-1048576kB
>
> EAL: Probing VFIO support...
>
> Initializing NVMe Controllers
>
> EAL: PCI device 0000:62:00.0 on NUMA socket 0
>
> EAL: probe driver: 8086:953 spdk_nvme
>
> Attaching to NVMe Controller at 0000:62:00.0 [8086:0953]
>
> Attached to NVMe Controller at 0000:62:00.0 [8086:0953]
>
> Associating INTEL SSDPE2MD400G4 (CVFT6175001C400GGN ) with lcore 0
>
> Initialization complete. Launching workers.
>
> Starting thread on core 0
>
> ========================================================
>
>
> Latency(us)
>
> Device Information : IOPS
> MB/s Average min max
>
> INTEL SSDPE2MD400G4 (CVFT6175001C400GGN ) from core 0: 12241.53
> 47.82 81.66 34.86 2891.90
>
> ========================================================
>
> Total : 12241.53
> 47.82 81.66 34.86 2891.90
>
>
>
> FIO:
>
> RW=randrw MIX=100 BLK_SIZE=4096 IODEPTH=1 NUMJOBS=1
> LD_PRELOAD=/home/john/spdk/examples/nvme/fio_plugin/fio_plugin fio
> fio.spdk_1ssd
>
> filename1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T)
> 4096B-4096B, ioengine=spdk, iodepth=1
>
> fio-3.2-84-g9b509
>
> Starting 1 thread
>
> Starting DPDK 17.08.0 initialization...
>
> [ DPDK EAL parameters: fio -c 0x1 -m 512 --file-prefix=spdk_pid43241 ]
>
> EAL: Detected 112 lcore(s)
>
> EAL: No free hugepages reported in hugepages-1048576kB
>
> EAL: Probing VFIO support...
>
> EAL: PCI device 0000:62:00.0 on NUMA socket 0
>
> EAL: probe driver: 8086:953 spdk_nvme
>
> Jobs: 1 (f=0): [f(1)][100.0%][r=0KiB/s,w=0KiB/s][r=0,w=0 IOPS][eta
> 00m:00s]
>
> filename1: (groupid=0, jobs=1): err= 0: pid=43356: Tue Dec 19 06:23:29 2017
>
> read: IOPS=12.2k, BW=47.5MiB/s (49.8MB/s)(27.8GiB/600001msec)
>
> slat (nsec): min=94, max=87632, avg=108.13, stdev=40.75
>
> clat (nsec): min=196, max=3008.9k, avg=81777.09, stdev=38852.81
>
> lat (usec): min=34, max=3009, avg=81.89, stdev=38.85
>
> clat percentiles (usec):
>
> | 50.000th=[ 78], 99.000th=[ 94], 99.900th=[ 241], 99.990th=[
> 2278],
>
> | 99.999th=[ 2671]
>
> bw ( KiB/s): min=42738, max=50248, per=100.00%, avg=48802.24,
> stdev=1522.91, samples=1199
>
> iops : min=10684, max=12562, avg=12200.43, stdev=380.76,
> samples=1199
>
> lat (nsec) : 250=0.01%
>
> lat (usec) : 50=0.01%, 100=99.78%, 250=0.12%, 500=0.04%, 750=0.01%
>
> lat (usec) : 1000=0.01%
>
> lat (msec) : 2=0.02%, 4=0.02%
>
> cpu : usr=99.98%, sys=0.00%, ctx=51300, majf=0, minf=266
>
> IO depths : 1=110.1%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%,
> >=64=0.0%
>
> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
> >=64=0.0%
>
> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
> >=64=0.0%
>
> issued rwt: total=7298991,0,0, short=0,0,0, dropped=0,0,0
>
> latency : target=0, window=0, percentile=100.00%, depth=1
>
>
>
> Run status group 0 (all jobs):
>
> READ: bw=47.5MiB/s (49.8MB/s), 47.5MiB/s-47.5MiB/s (49.8MB/s-49.8MB/s),
> io=27.8GiB (29.9GB), run=600001-600001msec
>
>
>
> Here is the FIO configuration, I usually use in my test environment
>
> [global]
>
> ioengine=spdk
>
> direct=1
>
>
>
> time_based=1
>
> norandommap=1
>
> ramp_time=60s
>
> runtime=600s
>
> thread=1
>
> group_reporting=1
>
> percentile_list=50:99:99.9:99.99:99.999
>
>
>
> rw=${RW}
>
> rwmixread=${MIX}
>
> bs=${BLK_SIZE}
>
> iodepth=${IODEPTH}
>
> numjobs=${NUMJOBS}
>
>
>
> [filename1]
>
> filename=trtype=PCIe traddr=0000.62.00.0 ns=1
>
> cpumask=0x1
>
>
>
> I ran the test with your fio options and the latency is pretty consistent
> with perf
>
> fio --time_based --name=benchmark --size=10G --runtime=600
> '--filename=trtype=PCIe traddr=0000.62.00.0 ns=1'
> --ioengine=/home/john/spdk/examples/nvme/fio_plugin/fio_plugin
> --randrepeat=0 --iodepth=1 --thread --direct=1 --invalidate=1 --verify=0
> --verify_fatal=0 --numjobs=1 --rw=randread --sync=1 --blocksize=4k
> --norandommap=1 --group_reporting
>
> benchmark: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T)
> 4096B-4096B, ioengine=spdk, iodepth=1
>
> fio-3.2-84-g9b509
>
> Starting 1 thread
>
> Starting DPDK 17.08.0 initialization...
>
> [ DPDK EAL parameters: fio -c 0x1 -m 512 --file-prefix=spdk_pid43874 ]
>
> EAL: Detected 112 lcore(s)
>
> EAL: No free hugepages reported in hugepages-1048576kB
>
> EAL: Probing VFIO support...
>
> EAL: PCI device 0000:62:00.0 on NUMA socket 0
>
> EAL: probe driver: 8086:953 spdk_nvme
>
> Jobs: 1 (f=0): [f(1)][100.0%][r=4412KiB/s,w=0KiB/s][r=1103,w=0 IOPS][eta
> 00m:00s]
>
> benchmark: (groupid=0, jobs=1): err= 0: pid=43989: Tue Dec 19 07:43:48 2017
>
> read: IOPS=12.2k, BW=47.5MiB/s (49.8MB/s)(27.8GiB/600001msec)
>
> slat (nsec): min=94, max=15837, avg=105.59, stdev=20.51
>
> clat (usec): min=23, max=2947, avg=81.73, stdev=37.37
>
> lat (usec): min=24, max=2947, avg=81.84, stdev=37.37
>
> clat percentiles (usec):
>
> | 1.00th=[ 63], 5.00th=[ 68], 10.00th=[ 72], 20.00th=[ 74],
>
> | 30.00th=[ 75], 40.00th=[ 76], 50.00th=[ 78], 60.00th=[ 87],
>
> | 70.00th=[ 89], 80.00th=[ 91], 90.00th=[ 92], 95.00th=[ 93],
>
> | 99.00th=[ 94], 99.50th=[ 94], 99.90th=[ 221], 99.95th=[ 553],
>
> | 99.99th=[ 2278]
>
> bw ( KiB/s): min=42618, max=49384, per=99.98%, avg=48644.25,
> stdev=1407.57, samples=1199
>
> iops : min=10654, max=12346, avg=12161.03, stdev=351.90,
> samples=1199
>
> lat (usec) : 50=0.01%, 100=99.79%, 250=0.11%, 500=0.04%, 750=0.01%
>
> lat (usec) : 1000=0.01%
>
> lat (msec) : 2=0.02%, 4=0.02%
>
> cpu : usr=100.00%, sys=0.00%, ctx=50106, majf=0, minf=280
>
> IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%,
> >=64=0.0%
>
> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
> >=64=0.0%
>
> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
> >=64=0.0%
>
> issued rwt: total=7298286,0,0, short=0,0,0, dropped=0,0,0
>
> latency : target=0, window=0, percentile=100.00%, depth=1
>
>
>
> Run status group 0 (all jobs):
>
> READ: bw=47.5MiB/s (49.8MB/s), 47.5MiB/s-47.5MiB/s (49.8MB/s-49.8MB/s),
> io=27.8GiB (29.9GB), run=600001-600001msec
>
>
>
> Can you run perf with the following options(./perf -q 1 -s 4096 -t 600 -w
> randread) and share the output?
>
> Have you checked the PCI link capability and status to verify the Speed is
> 8GT/s and the width is x4?
>
> On the sequential reads, you need to use a larger block size of 128K to
> reach the peak throughput.
>
>
>
> *From:* Mayank [mailto:mayank.patel(a)mayadata.io <mayank.patel(a)mayadata.io>]
>
> *Sent:* Monday, December 18, 2017 11:33 PM
> *To:* Kariuki, John K <john.k.kariuki(a)intel.com>
> <john.k.kariuki(a)intel.com>; Storage Performance Development Kit
> <spdk(a)lists.01.org> <spdk(a)lists.01.org>
> *Subject:* Re: [SPDK] SPDK FIO poor 4K read performance
>
>
>
> Hi John,
>
>
>
> On Tuesday 19 December 2017 04:54 AM, Kariuki, John K wrote:
>
> Mayank
>
> Can you share your fio configuration file?
>
> i used following fio command :
>
> fio --time_based --time_based --name=benchmark --size=10G --runtime=600
> '--filename=trtype=PCIe traddr=0000.18.00.0 ns=1'
> --ioengine=spdk/examples/nvme/fio_plugin/fio_plugin --randrepeat=0
> --iodepth=32 --thread --direct=1 --invalidate=1 --verify=0 --verify_fatal=0
> --numjobs=4 --rw=randwrite --sync=1 --blocksize=4k --norandommap=1
> --group_reportin
>
> Have you tried the latency performance test with the SPDK perf tool?
>
> Yes, i tried it. Results for write with run-time 300sec is as below.
> ./examples/nvme/perf/perf -q 32 -s 4096 -t 300 -w write
> ========================================================
>
> Latency(us)
> Device Information : IOPS
> MB/s Average min max
> INTEL SSDPEDMD020T4 (CVFT4415009B2P0EGN ) from core 0: 436776.09
> 1706.16 73.26 4.69 10563.94
> ========================================================
> Total : 436776.09
> 1706.16 73.26 4.69 10563.94
> ################
> + ./examples/nvme/perf/perf -q 32 -s 4096 -t 300 -w read
> ========================================================
>
> Latency(us)
> Device Information : IOPS
> MB/s Average min max
> INTEL SSDPEDMD020T4 (CVFT4415009B2P0EGN ) from core 0: 213554.89
> 834.20 149.84 4.05 9111.46
> ========================================================
> Total : 213554.89
> 834.20 149.84 4.05 9111.46
>
>
>
>
> Your random writes are pretty high. Did you precondition the SSD before
> the test? For the random write test you need to precondition the SSD by
> doing 90 minutes of random writes before testing.
>
> Yes, I preconditioned SSD before test.
>
>
> Here is a link to a video
> <https://www.youtube.com/watch?v=tkGE3pq5eIU&index=8&list=PLj-81kG3zG5ZIE-4CvqsvlFEHoOoWRIHZ>
> that highlights some performance best practices that you might want to
> check out.
>
> Thanks for sharing video. I have already gone through this presentation
> and followed necessary steps mentioned in this presentation.
>
>
>
>
> *From:* SPDK [mailto:spdk-bounces(a)lists.01.org <spdk-bounces(a)lists.01.org>]
> *On Behalf Of *Andrey Kuzmin
> *Sent:* Monday, December 18, 2017 3:40 PM
> *To:* Storage Performance Development Kit <spdk(a)lists.01.org>
> <spdk(a)lists.01.org>
> *Cc:* mayank.patel(a)mayadata.io
> *Subject:* Re: [SPDK] SPDK FIO poor 4K read performance
>
>
>
>
>
> On Tue, Dec 19, 2017, 01:27 Isaac Otsiabah <IOtsiabah(a)us.fujitsu.com>
> wrote:
>
> Hi Audrey,
>
>
>
> Hi Isaac,
>
>
>
> I guess you meant to address the original submission author, Mayank.
>
>
>
> Regards,
>
> Andrey
>
>
>
> which
>
> plugin are you running? examples/nvme/fio_plugin/fio_plugin or
> examples/bdev/fio_plugin/fio_plugin ? Is it on Fedora or Centos?
>
>
>
> Isaac
>
> *From:* SPDK [mailto:spdk-bounces(a)lists.01.org] *On Behalf Of *Andrey
> Kuzmin
> *Sent:* Monday, December 18, 2017 5:59 AM
> *To:* Storage Performance Development Kit <spdk(a)lists.01.org>
> *Subject:* Re: [SPDK] SPDK FIO poor 4K read performance
>
>
>
>
>
> On Mon, Dec 18, 2017, 16:08 Mayank <mayank.patel(a)mayadata.io> wrote:
>
> Hi All,
>
> I have performed fio benchmark test with SPDK fio plugin for throughput
> test with iodepth = 32 and numjobs=4.
>
> Latency results are as below:
>
>
>
> rand read
>
> 11.2k
>
> 45.9MB/s
>
> 88.83 usec
>
> rand write
>
> 126k
>
> 518MB/s
>
> 7.56 usec
>
> Throughput results are as below:
>
>
>
> Throughput (io:32 thr:4)
>
> IOPS
>
> Throuhgput
>
> Latency (usec)
>
> seq read
>
> 406k
>
> 1665MB/s
>
> 314.71
>
> seq write
>
> 455k
>
> 1865MB/s
>
> 280.87
>
> rand read
>
> 478k
>
> 1959MB/s
>
> 267.29
>
> rand write
>
> 442k
>
> 1812MB/s
>
> 289.03
>
>
>
> In results, 'm getting sequential read IOPS lesser than sequential write
> IOPS.
>
> I am using NVME model number : INTEL SSDPEDMD020T4.
> kernel : 4.13.13
> fio version : 2.21
>
>
>
> BTW, your fio version is couple of years behind.
>
>
> can anybody tell me what could be the issue?
>
>
>
> The latency data above clearly shows the reason: writes are being handled
> by the internal SSD cache of few usec latency, while reads are served off
> the real media, with typical (high-end) SSD latency of about 100 usec.
>
>
>
> Regards,
>
> Andrey
>
>
>
> Thanks,
> Mayank
>
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
>
> --
>
> Regards,
> Andrey
>
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
>
> --
>
> Regards,
> Andrey
>
>
>
>
>
[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 48606 bytes --]
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [SPDK] SPDK FIO poor 4K read performance
@ 2017-12-20 5:12 Mayank
0 siblings, 0 replies; 15+ messages in thread
From: Mayank @ 2017-12-20 5:12 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 16653 bytes --]
Hi John,
Thanks for sharing results.
In my latency data, QD=1.
Can you run perf with the following options(./perf -q 1 -s 4096 -t 600
-w randread) and share the output?
results.
[root(a)localhost spdk]# ./examples/nvme/perf/perf -q 1 -s 4096 -t 600 -w
randread
Starting DPDK 17.11.0 initialization...
[ DPDK EAL parameters: perf -c 0x1 --file-prefix=spdk_pid24378 ]
EAL: Detected 112 lcore(s)
EAL: Probing VFIO support...
EAL: WARNING: Master core has no memory on local socket!
Initializing NVMe Controllers
EAL: PCI device 0000:18:00.0 on NUMA socket 0
EAL: probe driver: 8086:953 spdk_nvme
Attaching to NVMe Controller at 0000:18:00.0 [8086:0953]
Attached to NVMe Controller at 0000:18:00.0 [8086:0953]
Associating INTEL SSDPEDMD020T4 (CVFT4415009B2P0EGN ) with lcore 0
Initialization complete. Launching workers.
Starting thread on core 0
========================================================
Latency(us)
Device Information : IOPS
MB/s Average min max
INTEL SSDPEDMD020T4 (CVFT4415009B2P0EGN ) from core 0: 158706.03
619.95 6.28 4.37 831.51
========================================================
Total : 158706.03 619.95 6.28 4.37 831.51
*** in this, latency is very less compared to fio!
Regarding PCI speed, snapshot from lspci for NVME SSD
LnkSta: Speed 8GT/s, Width x4, TrErr- Train- SlotClk- DLActive-
BWMgmt- ABWMgmt-
Can you perform randwrite with your setup, if possible?
On Wednesday 20 December 2017 03:17 AM, Kariuki, John K wrote:
>
> Mayank
>
> Thank you for sharing this information.
>
> On the latency data (rand read 11.2k 45.9MB/s 88.83 usec), can you
> confirm that the QD=1?
>
> I have a system with P3700 SSD (U.2 form factor) and I am getting a
> latency number of around 81 usecs with both perf and FIO.
>
> SPDK PERF:
>
> ./perf -q 1 -s 4096 -t 600 -w randread -r 'trtype:PCIe
> traddr:0000:62:00.0'
>
> Starting DPDK 17.08.0 initialization...
>
> [ DPDK EAL parameters: perf -c 0x1 --file-prefix=spdk_pid43053 ]
>
> EAL: Detected 112 lcore(s)
>
> EAL: No free hugepages reported in hugepages-1048576kB
>
> EAL: Probing VFIO support...
>
> Initializing NVMe Controllers
>
> EAL: PCI device 0000:62:00.0 on NUMA socket 0
>
> EAL: probe driver: 8086:953 spdk_nvme
>
> Attaching to NVMe Controller at 0000:62:00.0 [8086:0953]
>
> Attached to NVMe Controller at 0000:62:00.0 [8086:0953]
>
> Associating INTEL SSDPE2MD400G4 (CVFT6175001C400GGN ) with lcore 0
>
> Initialization complete. Launching workers.
>
> Starting thread on core 0
>
> ========================================================
>
> Latency(us)
>
> Device Information : IOPS
> MB/s Average min max
>
> INTEL SSDPE2MD400G4 (CVFT6175001C400GGN ) from core 0: 12241.53
> 47.82 81.66 34.86 2891.90
>
> ========================================================
>
> Total : 12241.53 47.82 81.66 34.86 2891.90
>
> FIO:
>
> RW=randrw MIX=100 BLK_SIZE=4096 IODEPTH=1 NUMJOBS=1
> LD_PRELOAD=/home/john/spdk/examples/nvme/fio_plugin/fio_plugin fio
> fio.spdk_1ssd
>
> filename1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T)
> 4096B-4096B, ioengine=spdk, iodepth=1
>
> fio-3.2-84-g9b509
>
> Starting 1 thread
>
> Starting DPDK 17.08.0 initialization...
>
> [ DPDK EAL parameters: fio -c 0x1 -m 512 --file-prefix=spdk_pid43241 ]
>
> EAL: Detected 112 lcore(s)
>
> EAL: No free hugepages reported in hugepages-1048576kB
>
> EAL: Probing VFIO support...
>
> EAL: PCI device 0000:62:00.0 on NUMA socket 0
>
> EAL: probe driver: 8086:953 spdk_nvme
>
> Jobs: 1 (f=0): [f(1)][100.0%][r=0KiB/s,w=0KiB/s][r=0,w=0 IOPS][eta
> 00m:00s]
>
> filename1: (groupid=0, jobs=1): err= 0: pid=43356: Tue Dec 19 06:23:29
> 2017
>
> read: IOPS=12.2k, BW=47.5MiB/s (49.8MB/s)(27.8GiB/600001msec)
>
> slat (nsec): min=94, max=87632, avg=108.13, stdev=40.75
>
> clat (nsec): min=196, max=3008.9k, avg=81777.09, stdev=38852.81
>
> lat (usec): min=34, max=3009, avg=81.89, stdev=38.85
>
> clat percentiles (usec):
>
> | 50.000th=[ 78], 99.000th=[ 94], 99.900th=[ 241],
> 99.990th=[ 2278],
>
> | 99.999th=[ 2671]
>
> bw ( KiB/s): min=42738, max=50248, per=100.00%, avg=48802.24,
> stdev=1522.91, samples=1199
>
> iops : min=10684, max=12562, avg=12200.43, stdev=380.76,
> samples=1199
>
> lat (nsec) : 250=0.01%
>
> lat (usec) : 50=0.01%, 100=99.78%, 250=0.12%, 500=0.04%, 750=0.01%
>
> lat (usec) : 1000=0.01%
>
> lat (msec) : 2=0.02%, 4=0.02%
>
> cpu : usr=99.98%, sys=0.00%, ctx=51300, majf=0, minf=266
>
> IO depths : 1=110.1%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%,
> >=64=0.0%
>
> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
> >=64=0.0%
>
> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
> >=64=0.0%
>
> issued rwt: total=7298991,0,0, short=0,0,0, dropped=0,0,0
>
> latency : target=0, window=0, percentile=100.00%, depth=1
>
> Run status group 0 (all jobs):
>
> READ: bw=47.5MiB/s (49.8MB/s), 47.5MiB/s-47.5MiB/s
> (49.8MB/s-49.8MB/s), io=27.8GiB (29.9GB), run=600001-600001msec
>
> Here is the FIO configuration, I usually use in my test environment
>
> [global]
>
> ioengine=spdk
>
> direct=1
>
> time_based=1
>
> norandommap=1
>
> ramp_time=60s
>
> runtime=600s
>
> thread=1
>
> group_reporting=1
>
> percentile_list=50:99:99.9:99.99:99.999
>
> rw=${RW}
>
> rwmixread=${MIX}
>
> bs=${BLK_SIZE}
>
> iodepth=${IODEPTH}
>
> numjobs=${NUMJOBS}
>
> [filename1]
>
> filename=trtype=PCIe traddr=0000.62.00.0 ns=1
>
> cpumask=0x1
>
> I ran the test with your fio options and the latency is pretty
> consistent with perf
>
> fio --time_based --name=benchmark --size=10G --runtime=600
> '--filename=trtype=PCIe traddr=0000.62.00.0 ns=1'
> --ioengine=/home/john/spdk/examples/nvme/fio_plugin/fio_plugin
> --randrepeat=0 --iodepth=1 --thread --direct=1 --invalidate=1
> --verify=0 --verify_fatal=0 --numjobs=1 --rw=randread --sync=1
> --blocksize=4k --norandommap=1 --group_reporting
>
> benchmark: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B,
> (T) 4096B-4096B, ioengine=spdk, iodepth=1
>
> fio-3.2-84-g9b509
>
> Starting 1 thread
>
> Starting DPDK 17.08.0 initialization...
>
> [ DPDK EAL parameters: fio -c 0x1 -m 512 --file-prefix=spdk_pid43874 ]
>
> EAL: Detected 112 lcore(s)
>
> EAL: No free hugepages reported in hugepages-1048576kB
>
> EAL: Probing VFIO support...
>
> EAL: PCI device 0000:62:00.0 on NUMA socket 0
>
> EAL: probe driver: 8086:953 spdk_nvme
>
> Jobs: 1 (f=0): [f(1)][100.0%][r=4412KiB/s,w=0KiB/s][r=1103,w=0
> IOPS][eta 00m:00s]
>
> benchmark: (groupid=0, jobs=1): err= 0: pid=43989: Tue Dec 19 07:43:48
> 2017
>
> read: IOPS=12.2k, BW=47.5MiB/s (49.8MB/s)(27.8GiB/600001msec)
>
> slat (nsec): min=94, max=15837, avg=105.59, stdev=20.51
>
> clat (usec): min=23, max=2947, avg=81.73, stdev=37.37
>
> lat (usec): min=24, max=2947, avg=81.84, stdev=37.37
>
> clat percentiles (usec):
>
> | 1.00th=[ 63], 5.00th=[ 68], 10.00th=[ 72], 20.00th=[ 74],
>
> | 30.00th=[ 75], 40.00th=[ 76], 50.00th=[ 78], 60.00th=[ 87],
>
> | 70.00th=[ 89], 80.00th=[ 91], 90.00th=[ 92], 95.00th=[ 93],
>
> | 99.00th=[ 94], 99.50th=[ 94], 99.90th=[ 221], 99.95th=[ 553],
>
> | 99.99th=[ 2278]
>
> bw ( KiB/s): min=42618, max=49384, per=99.98%, avg=48644.25,
> stdev=1407.57, samples=1199
>
> iops : min=10654, max=12346, avg=12161.03, stdev=351.90,
> samples=1199
>
> lat (usec) : 50=0.01%, 100=99.79%, 250=0.11%, 500=0.04%, 750=0.01%
>
> lat (usec) : 1000=0.01%
>
> lat (msec) : 2=0.02%, 4=0.02%
>
> cpu : usr=100.00%, sys=0.00%, ctx=50106, majf=0, minf=280
>
> IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%,
> >=64=0.0%
>
> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
> >=64=0.0%
>
> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
> >=64=0.0%
>
> issued rwt: total=7298286,0,0, short=0,0,0, dropped=0,0,0
>
> latency : target=0, window=0, percentile=100.00%, depth=1
>
> Run status group 0 (all jobs):
>
> READ: bw=47.5MiB/s (49.8MB/s), 47.5MiB/s-47.5MiB/s
> (49.8MB/s-49.8MB/s), io=27.8GiB (29.9GB), run=600001-600001msec
>
> Can you run perf with the following options(./perf -q 1 -s 4096 -t 600
> -w randread) and share the output?
>
> Have you checked the PCI link capability and status to verify the
> Speed is 8GT/s and the width is x4?
>
> On the sequential reads, you need to use a larger block size of 128K
> to reach the peak throughput.
>
> *From:*Mayank [mailto:mayank.patel(a)mayadata.io]
> *Sent:* Monday, December 18, 2017 11:33 PM
> *To:* Kariuki, John K <john.k.kariuki(a)intel.com>; Storage Performance
> Development Kit <spdk(a)lists.01.org>
> *Subject:* Re: [SPDK] SPDK FIO poor 4K read performance
>
> Hi John,
>
> On Tuesday 19 December 2017 04:54 AM, Kariuki, John K wrote:
>
> Mayank
>
> Can you share your fio configuration file?
>
> i used following fio command :
>
> fio --time_based --time_based --name=benchmark --size=10G
> --runtime=600 '--filename=trtype=PCIe traddr=0000.18.00.0 ns=1'
> --ioengine=spdk/examples/nvme/fio_plugin/fio_plugin --randrepeat=0
> --iodepth=32 --thread --direct=1 --invalidate=1 --verify=0
> --verify_fatal=0 --numjobs=4 --rw=randwrite --sync=1 --blocksize=4k
> --norandommap=1 --group_reportin
>
> Have you tried the latency performance test with the SPDK perf tool?
>
> Yes, i tried it. Results for write with run-time 300sec is as below.
> ./examples/nvme/perf/perf -q 32 -s 4096 -t 300 -w write
> ========================================================
> Latency(us)
> Device Information : IOPS
> MB/s Average min max
> INTEL SSDPEDMD020T4 (CVFT4415009B2P0EGN ) from core 0: 436776.09
> 1706.16 73.26 4.69 10563.94
> ========================================================
> Total : 436776.09
> 1706.16 73.26 4.69 10563.94
> ################
> + ./examples/nvme/perf/perf -q 32 -s 4096 -t 300 -w read
> ========================================================
> Latency(us)
> Device Information : IOPS
> MB/s Average min max
> INTEL SSDPEDMD020T4 (CVFT4415009B2P0EGN ) from core 0: 213554.89
> 834.20 149.84 4.05 9111.46
> ========================================================
> Total : 213554.89
> 834.20 149.84 4.05 9111.46
>
>
>
> Your random writes are pretty high. Did you precondition the SSD
> before the test? For the random write test you need to
> precondition the SSD by doing 90 minutes of random writes before
> testing.
>
> Yes, I preconditioned SSD before test.
>
> Here is a link to a video
> <https://www.youtube.com/watch?v=tkGE3pq5eIU&index=8&list=PLj-81kG3zG5ZIE-4CvqsvlFEHoOoWRIHZ>
> that highlights some performance best practices that you might
> want to check out.
>
> Thanks for sharing video. I have already gone through this
> presentation and followed necessary steps mentioned in this presentation.
>
> *From:*SPDK [mailto:spdk-bounces(a)lists.01.org] *On Behalf Of
> *Andrey Kuzmin
> *Sent:* Monday, December 18, 2017 3:40 PM
> *To:* Storage Performance Development Kit <spdk(a)lists.01.org>
> <mailto:spdk(a)lists.01.org>
> *Cc:* mayank.patel(a)mayadata.io <mailto:mayank.patel(a)mayadata.io>
> *Subject:* Re: [SPDK] SPDK FIO poor 4K read performance
>
> On Tue, Dec 19, 2017, 01:27 Isaac Otsiabah
> <IOtsiabah(a)us.fujitsu.com <mailto:IOtsiabah(a)us.fujitsu.com>> wrote:
>
> Hi Audrey,
>
> Hi Isaac,
>
> I guess you meant to address the original submission author, Mayank.
>
> Regards,
>
> Andrey
>
> which
>
> plugin are you running? examples/nvme/fio_plugin/fio_plugin
> or examples/bdev/fio_plugin/fio_plugin ? Is it on Fedora or
> Centos?
>
> Isaac
>
> *From:*SPDK [mailto:spdk-bounces(a)lists.01.org
> <mailto:spdk-bounces(a)lists.01.org>] *On Behalf Of *Andrey Kuzmin
> *Sent:* Monday, December 18, 2017 5:59 AM
> *To:* Storage Performance Development Kit <spdk(a)lists.01.org
> <mailto:spdk(a)lists.01.org>>
> *Subject:* Re: [SPDK] SPDK FIO poor 4K read performance
>
> On Mon, Dec 18, 2017, 16:08 Mayank <mayank.patel(a)mayadata.io
> <mailto:mayank.patel(a)mayadata.io>> wrote:
>
> Hi All,
>
> I have performed fio benchmark test with SPDK fio plugin
> for throughput test with iodepth = 32 and numjobs=4.
>
> Latency results are as below:
>
> rand read
>
>
>
> 11.2k
>
>
>
> 45.9MB/s
>
>
>
> 88.83 usec
>
> rand write
>
>
>
> 126k
>
>
>
> 518MB/s
>
>
>
> 7.56 usec
>
> Throughput results are as below:
>
> Throughput (io:32 thr:4)
>
>
>
> IOPS
>
>
>
> Throuhgput
>
>
>
> Latency (usec)
>
> seq read
>
>
>
> 406k
>
>
>
> 1665MB/s
>
>
>
> 314.71
>
> seq write
>
>
>
> 455k
>
>
>
> 1865MB/s
>
>
>
> 280.87
>
> rand read
>
>
>
> 478k
>
>
>
> 1959MB/s
>
>
>
> 267.29
>
> rand write
>
>
>
> 442k
>
>
>
> 1812MB/s
>
>
>
> 289.03
>
>
>
> In results, 'm getting sequential read IOPS lesser than
> sequential write IOPS.
>
> I am using NVME model number : INTEL SSDPEDMD020T4.
> kernel : 4.13.13
> fio version : 2.21
>
> BTW, your fio version is couple of years behind.
>
>
> can anybody tell me what could be the issue?
>
> The latency data above clearly shows the reason: writes are
> being handled by the internal SSD cache of few usec latency,
> while reads are served off the real media, with typical
> (high-end) SSD latency of about 100 usec.
>
> Regards,
>
> Andrey
>
>
>
> Thanks,
> Mayank
>
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org <mailto:SPDK(a)lists.01.org>
> https://lists.01.org/mailman/listinfo/spdk
>
> --
>
> Regards,
> Andrey
>
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org <mailto:SPDK(a)lists.01.org>
> https://lists.01.org/mailman/listinfo/spdk
>
> --
>
> Regards,
> Andrey
>
[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 71101 bytes --]
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [SPDK] SPDK FIO poor 4K read performance
@ 2017-12-19 21:47 Kariuki, John K
0 siblings, 0 replies; 15+ messages in thread
From: Kariuki, John K @ 2017-12-19 21:47 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 12193 bytes --]
Mayank
Thank you for sharing this information.
On the latency data (rand read 11.2k 45.9MB/s 88.83 usec), can you confirm that the QD=1?
I have a system with P3700 SSD (U.2 form factor) and I am getting a latency number of around 81 usecs with both perf and FIO.
SPDK PERF:
./perf -q 1 -s 4096 -t 600 -w randread -r 'trtype:PCIe traddr:0000:62:00.0'
Starting DPDK 17.08.0 initialization...
[ DPDK EAL parameters: perf -c 0x1 --file-prefix=spdk_pid43053 ]
EAL: Detected 112 lcore(s)
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
Initializing NVMe Controllers
EAL: PCI device 0000:62:00.0 on NUMA socket 0
EAL: probe driver: 8086:953 spdk_nvme
Attaching to NVMe Controller at 0000:62:00.0 [8086:0953]
Attached to NVMe Controller at 0000:62:00.0 [8086:0953]
Associating INTEL SSDPE2MD400G4 (CVFT6175001C400GGN ) with lcore 0
Initialization complete. Launching workers.
Starting thread on core 0
========================================================
Latency(us)
Device Information : IOPS MB/s Average min max
INTEL SSDPE2MD400G4 (CVFT6175001C400GGN ) from core 0: 12241.53 47.82 81.66 34.86 2891.90
========================================================
Total : 12241.53 47.82 81.66 34.86 2891.90
FIO:
RW=randrw MIX=100 BLK_SIZE=4096 IODEPTH=1 NUMJOBS=1 LD_PRELOAD=/home/john/spdk/examples/nvme/fio_plugin/fio_plugin fio fio.spdk_1ssd
filename1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk, iodepth=1
fio-3.2-84-g9b509
Starting 1 thread
Starting DPDK 17.08.0 initialization...
[ DPDK EAL parameters: fio -c 0x1 -m 512 --file-prefix=spdk_pid43241 ]
EAL: Detected 112 lcore(s)
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: PCI device 0000:62:00.0 on NUMA socket 0
EAL: probe driver: 8086:953 spdk_nvme
Jobs: 1 (f=0): [f(1)][100.0%][r=0KiB/s,w=0KiB/s][r=0,w=0 IOPS][eta 00m:00s]
filename1: (groupid=0, jobs=1): err= 0: pid=43356: Tue Dec 19 06:23:29 2017
read: IOPS=12.2k, BW=47.5MiB/s (49.8MB/s)(27.8GiB/600001msec)
slat (nsec): min=94, max=87632, avg=108.13, stdev=40.75
clat (nsec): min=196, max=3008.9k, avg=81777.09, stdev=38852.81
lat (usec): min=34, max=3009, avg=81.89, stdev=38.85
clat percentiles (usec):
| 50.000th=[ 78], 99.000th=[ 94], 99.900th=[ 241], 99.990th=[ 2278],
| 99.999th=[ 2671]
bw ( KiB/s): min=42738, max=50248, per=100.00%, avg=48802.24, stdev=1522.91, samples=1199
iops : min=10684, max=12562, avg=12200.43, stdev=380.76, samples=1199
lat (nsec) : 250=0.01%
lat (usec) : 50=0.01%, 100=99.78%, 250=0.12%, 500=0.04%, 750=0.01%
lat (usec) : 1000=0.01%
lat (msec) : 2=0.02%, 4=0.02%
cpu : usr=99.98%, sys=0.00%, ctx=51300, majf=0, minf=266
IO depths : 1=110.1%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwt: total=7298991,0,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=47.5MiB/s (49.8MB/s), 47.5MiB/s-47.5MiB/s (49.8MB/s-49.8MB/s), io=27.8GiB (29.9GB), run=600001-600001msec
Here is the FIO configuration, I usually use in my test environment
[global]
ioengine=spdk
direct=1
time_based=1
norandommap=1
ramp_time=60s
runtime=600s
thread=1
group_reporting=1
percentile_list=50:99:99.9:99.99:99.999
rw=${RW}
rwmixread=${MIX}
bs=${BLK_SIZE}
iodepth=${IODEPTH}
numjobs=${NUMJOBS}
[filename1]
filename=trtype=PCIe traddr=0000.62.00.0 ns=1
cpumask=0x1
I ran the test with your fio options and the latency is pretty consistent with perf
fio --time_based --name=benchmark --size=10G --runtime=600 '--filename=trtype=PCIe traddr=0000.62.00.0 ns=1' --ioengine=/home/john/spdk/examples/nvme/fio_plugin/fio_plugin --randrepeat=0 --iodepth=1 --thread --direct=1 --invalidate=1 --verify=0 --verify_fatal=0 --numjobs=1 --rw=randread --sync=1 --blocksize=4k --norandommap=1 --group_reporting
benchmark: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk, iodepth=1
fio-3.2-84-g9b509
Starting 1 thread
Starting DPDK 17.08.0 initialization...
[ DPDK EAL parameters: fio -c 0x1 -m 512 --file-prefix=spdk_pid43874 ]
EAL: Detected 112 lcore(s)
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: PCI device 0000:62:00.0 on NUMA socket 0
EAL: probe driver: 8086:953 spdk_nvme
Jobs: 1 (f=0): [f(1)][100.0%][r=4412KiB/s,w=0KiB/s][r=1103,w=0 IOPS][eta 00m:00s]
benchmark: (groupid=0, jobs=1): err= 0: pid=43989: Tue Dec 19 07:43:48 2017
read: IOPS=12.2k, BW=47.5MiB/s (49.8MB/s)(27.8GiB/600001msec)
slat (nsec): min=94, max=15837, avg=105.59, stdev=20.51
clat (usec): min=23, max=2947, avg=81.73, stdev=37.37
lat (usec): min=24, max=2947, avg=81.84, stdev=37.37
clat percentiles (usec):
| 1.00th=[ 63], 5.00th=[ 68], 10.00th=[ 72], 20.00th=[ 74],
| 30.00th=[ 75], 40.00th=[ 76], 50.00th=[ 78], 60.00th=[ 87],
| 70.00th=[ 89], 80.00th=[ 91], 90.00th=[ 92], 95.00th=[ 93],
| 99.00th=[ 94], 99.50th=[ 94], 99.90th=[ 221], 99.95th=[ 553],
| 99.99th=[ 2278]
bw ( KiB/s): min=42618, max=49384, per=99.98%, avg=48644.25, stdev=1407.57, samples=1199
iops : min=10654, max=12346, avg=12161.03, stdev=351.90, samples=1199
lat (usec) : 50=0.01%, 100=99.79%, 250=0.11%, 500=0.04%, 750=0.01%
lat (usec) : 1000=0.01%
lat (msec) : 2=0.02%, 4=0.02%
cpu : usr=100.00%, sys=0.00%, ctx=50106, majf=0, minf=280
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwt: total=7298286,0,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=47.5MiB/s (49.8MB/s), 47.5MiB/s-47.5MiB/s (49.8MB/s-49.8MB/s), io=27.8GiB (29.9GB), run=600001-600001msec
Can you run perf with the following options(./perf -q 1 -s 4096 -t 600 -w randread) and share the output?
Have you checked the PCI link capability and status to verify the Speed is 8GT/s and the width is x4?
On the sequential reads, you need to use a larger block size of 128K to reach the peak throughput.
From: Mayank [mailto:mayank.patel(a)mayadata.io]
Sent: Monday, December 18, 2017 11:33 PM
To: Kariuki, John K <john.k.kariuki(a)intel.com>; Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] SPDK FIO poor 4K read performance
Hi John,
On Tuesday 19 December 2017 04:54 AM, Kariuki, John K wrote:
Mayank
Can you share your fio configuration file?
i used following fio command :
fio --time_based --time_based --name=benchmark --size=10G --runtime=600 '--filename=trtype=PCIe traddr=0000.18.00.0 ns=1' --ioengine=spdk/examples/nvme/fio_plugin/fio_plugin --randrepeat=0 --iodepth=32 --thread --direct=1 --invalidate=1 --verify=0 --verify_fatal=0 --numjobs=4 --rw=randwrite --sync=1 --blocksize=4k --norandommap=1 --group_reportin
Have you tried the latency performance test with the SPDK perf tool?
Yes, i tried it. Results for write with run-time 300sec is as below.
./examples/nvme/perf/perf -q 32 -s 4096 -t 300 -w write
========================================================
Latency(us)
Device Information : IOPS MB/s Average min max
INTEL SSDPEDMD020T4 (CVFT4415009B2P0EGN ) from core 0: 436776.09 1706.16 73.26 4.69 10563.94
========================================================
Total : 436776.09 1706.16 73.26 4.69 10563.94
################
+ ./examples/nvme/perf/perf -q 32 -s 4096 -t 300 -w read
========================================================
Latency(us)
Device Information : IOPS MB/s Average min max
INTEL SSDPEDMD020T4 (CVFT4415009B2P0EGN ) from core 0: 213554.89 834.20 149.84 4.05 9111.46
========================================================
Total : 213554.89 834.20 149.84 4.05 9111.46
Your random writes are pretty high. Did you precondition the SSD before the test? For the random write test you need to precondition the SSD by doing 90 minutes of random writes before testing.
Yes, I preconditioned SSD before test.
Here is a link to a video<https://www.youtube.com/watch?v=tkGE3pq5eIU&index=8&list=PLj-81kG3zG5ZIE-4CvqsvlFEHoOoWRIHZ> that highlights some performance best practices that you might want to check out.
Thanks for sharing video. I have already gone through this presentation and followed necessary steps mentioned in this presentation.
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Andrey Kuzmin
Sent: Monday, December 18, 2017 3:40 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org><mailto:spdk(a)lists.01.org>
Cc: mayank.patel(a)mayadata.io<mailto:mayank.patel(a)mayadata.io>
Subject: Re: [SPDK] SPDK FIO poor 4K read performance
On Tue, Dec 19, 2017, 01:27 Isaac Otsiabah <IOtsiabah(a)us.fujitsu.com<mailto:IOtsiabah(a)us.fujitsu.com>> wrote:
Hi Audrey,
Hi Isaac,
I guess you meant to address the original submission author, Mayank.
Regards,
Andrey
which
plugin are you running? examples/nvme/fio_plugin/fio_plugin or examples/bdev/fio_plugin/fio_plugin ? Is it on Fedora or Centos?
Isaac
From: SPDK [mailto:spdk-bounces(a)lists.01.org<mailto:spdk-bounces(a)lists.01.org>] On Behalf Of Andrey Kuzmin
Sent: Monday, December 18, 2017 5:59 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] SPDK FIO poor 4K read performance
On Mon, Dec 18, 2017, 16:08 Mayank <mayank.patel(a)mayadata.io<mailto:mayank.patel(a)mayadata.io>> wrote:
Hi All,
I have performed fio benchmark test with SPDK fio plugin for throughput test with iodepth = 32 and numjobs=4.
Latency results are as below:
rand read
11.2k
45.9MB/s
88.83 usec
rand write
126k
518MB/s
7.56 usec
Throughput results are as below:
Throughput (io:32 thr:4)
IOPS
Throuhgput
Latency (usec)
seq read
406k
1665MB/s
314.71
seq write
455k
1865MB/s
280.87
rand read
478k
1959MB/s
267.29
rand write
442k
1812MB/s
289.03
In results, 'm getting sequential read IOPS lesser than sequential write IOPS.
I am using NVME model number : INTEL SSDPEDMD020T4.
kernel : 4.13.13
fio version : 2.21
BTW, your fio version is couple of years behind.
can anybody tell me what could be the issue?
The latency data above clearly shows the reason: writes are being handled by the internal SSD cache of few usec latency, while reads are served off the real media, with typical (high-end) SSD latency of about 100 usec.
Regards,
Andrey
Thanks,
Mayank
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org<mailto:SPDK(a)lists.01.org>
https://lists.01.org/mailman/listinfo/spdk
--
Regards,
Andrey
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org<mailto:SPDK(a)lists.01.org>
https://lists.01.org/mailman/listinfo/spdk
--
Regards,
Andrey
[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 54353 bytes --]
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [SPDK] SPDK FIO poor 4K read performance
@ 2017-12-19 6:33 Mayank
0 siblings, 0 replies; 15+ messages in thread
From: Mayank @ 2017-12-19 6:33 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 6251 bytes --]
Hi John,
On Tuesday 19 December 2017 04:54 AM, Kariuki, John K wrote:
>
> Mayank
>
> Can you share your fio configuration file?
>
i used following fio command :
fio --time_based --time_based --name=benchmark --size=10G --runtime=600
'--filename=trtype=PCIe traddr=0000.18.00.0 ns=1'
--ioengine=spdk/examples/nvme/fio_plugin/fio_plugin --randrepeat=0
--iodepth=32 --thread --direct=1 --invalidate=1 --verify=0
--verify_fatal=0 --numjobs=4 --rw=randwrite --sync=1 --blocksize=4k
--norandommap=1 --group_reportin
> Have you tried the latency performance test with the SPDK perf tool?
>
Yes, i tried it. Results for write with run-time 300sec is as below.
./examples/nvme/perf/perf -q 32 -s 4096 -t 300 -w write
========================================================
Latency(us)
Device Information : IOPS
MB/s Average min max
INTEL SSDPEDMD020T4 (CVFT4415009B2P0EGN ) from core 0: 436776.09
1706.16 73.26 4.69 10563.94
========================================================
Total : 436776.09
1706.16 73.26 4.69 10563.94
################
+ ./examples/nvme/perf/perf -q 32 -s 4096 -t 300 -w read
========================================================
Latency(us)
Device Information : IOPS
MB/s Average min max
INTEL SSDPEDMD020T4 (CVFT4415009B2P0EGN ) from core 0: 213554.89
834.20 149.84 4.05 9111.46
========================================================
Total : 213554.89
834.20 149.84 4.05 9111.46
> Your random writes are pretty high. Did you precondition the SSD
> before the test? For the random write test you need to precondition
> the SSD by doing 90 minutes of random writes before testing.
>
Yes, I preconditioned SSD before test.
>
> Here is a link to a video
> <https://www.youtube.com/watch?v=tkGE3pq5eIU&index=8&list=PLj-81kG3zG5ZIE-4CvqsvlFEHoOoWRIHZ>
> that highlights some performance best practices that you might want to
> check out.
>
Thanks for sharing video. I have already gone through this presentation
and followed necessary steps mentioned in this presentation.
>
> *From:*SPDK [mailto:spdk-bounces(a)lists.01.org] *On Behalf Of *Andrey
> Kuzmin
> *Sent:* Monday, December 18, 2017 3:40 PM
> *To:* Storage Performance Development Kit <spdk(a)lists.01.org>
> *Cc:* mayank.patel(a)mayadata.io
> *Subject:* Re: [SPDK] SPDK FIO poor 4K read performance
>
> On Tue, Dec 19, 2017, 01:27 Isaac Otsiabah <IOtsiabah(a)us.fujitsu.com
> <mailto:IOtsiabah(a)us.fujitsu.com>> wrote:
>
> Hi Audrey,
>
> Hi Isaac,
>
> I guess you meant to address the original submission author, Mayank.
>
> Regards,
>
> Andrey
>
> which
>
> plugin are you running? examples/nvme/fio_plugin/fio_plugin or
> examples/bdev/fio_plugin/fio_plugin ? Is it on Fedora or Centos?
>
> Isaac
>
> *From:*SPDK [mailto:spdk-bounces(a)lists.01.org
> <mailto:spdk-bounces(a)lists.01.org>] *On Behalf Of *Andrey Kuzmin
> *Sent:* Monday, December 18, 2017 5:59 AM
> *To:* Storage Performance Development Kit <spdk(a)lists.01.org
> <mailto:spdk(a)lists.01.org>>
> *Subject:* Re: [SPDK] SPDK FIO poor 4K read performance
>
> On Mon, Dec 18, 2017, 16:08 Mayank <mayank.patel(a)mayadata.io
> <mailto:mayank.patel(a)mayadata.io>> wrote:
>
> Hi All,
>
> I have performed fio benchmark test with SPDK fio plugin for
> throughput test with iodepth = 32 and numjobs=4.
>
> Latency results are as below:
>
> rand read
>
>
>
> 11.2k
>
>
>
> 45.9MB/s
>
>
>
> 88.83 usec
>
> rand write
>
>
>
> 126k
>
>
>
> 518MB/s
>
>
>
> 7.56 usec
>
> Throughput results are as below:
>
> Throughput (io:32 thr:4)
>
>
>
> IOPS
>
>
>
> Throuhgput
>
>
>
> Latency (usec)
>
> seq read
>
>
>
> 406k
>
>
>
> 1665MB/s
>
>
>
> 314.71
>
> seq write
>
>
>
> 455k
>
>
>
> 1865MB/s
>
>
>
> 280.87
>
> rand read
>
>
>
> 478k
>
>
>
> 1959MB/s
>
>
>
> 267.29
>
> rand write
>
>
>
> 442k
>
>
>
> 1812MB/s
>
>
>
> 289.03
>
>
>
> In results, 'm getting sequential read IOPS lesser than
> sequential write IOPS.
>
> I am using NVME model number : INTEL SSDPEDMD020T4.
> kernel : 4.13.13
> fio version : 2.21
>
> BTW, your fio version is couple of years behind.
>
>
> can anybody tell me what could be the issue?
>
> The latency data above clearly shows the reason: writes are being
> handled by the internal SSD cache of few usec latency, while reads
> are served off the real media, with typical (high-end) SSD latency
> of about 100 usec.
>
> Regards,
>
> Andrey
>
>
>
> Thanks,
> Mayank
>
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org <mailto:SPDK(a)lists.01.org>
> https://lists.01.org/mailman/listinfo/spdk
>
> --
>
> Regards,
> Andrey
>
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org <mailto:SPDK(a)lists.01.org>
> https://lists.01.org/mailman/listinfo/spdk
>
> --
>
> Regards,
> Andrey
>
[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 29606 bytes --]
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [SPDK] SPDK FIO poor 4K read performance
@ 2017-12-19 0:12 Isaac Otsiabah
0 siblings, 0 replies; 15+ messages in thread
From: Isaac Otsiabah @ 2017-12-19 0:12 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 2107 bytes --]
Yes, I meant Mayank. Thank you.
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Isaac Otsiabah
Sent: Monday, December 18, 2017 2:28 PM
To: 'mayank.patel(a)mayadata.io' <mayank.patel(a)mayadata.io>; 'spdk(a)lists.01.org' <spdk(a)lists.01.org>
Subject: Re: [SPDK] SPDK FIO poor 4K read performance
Hi Audrey, which plugin are you running? examples/nvme/fio_plugin/fio_plugin or examples/bdev/fio_plugin/fio_plugin ? Is it on Fedora or Centos?
Isaac
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Andrey Kuzmin
Sent: Monday, December 18, 2017 5:59 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] SPDK FIO poor 4K read performance
On Mon, Dec 18, 2017, 16:08 Mayank <mayank.patel(a)mayadata.io<mailto:mayank.patel(a)mayadata.io>> wrote:
Hi All,
I have performed fio benchmark test with SPDK fio plugin for throughput test with iodepth = 32 and numjobs=4.
Latency results are as below:
rand read
11.2k
45.9MB/s
88.83 usec
rand write
126k
518MB/s
7.56 usec
Throughput results are as below:
Throughput (io:32 thr:4)
IOPS
Throuhgput
Latency (usec)
seq read
406k
1665MB/s
314.71
seq write
455k
1865MB/s
280.87
rand read
478k
1959MB/s
267.29
rand write
442k
1812MB/s
289.03
In results, 'm getting sequential read IOPS lesser than sequential write IOPS.
I am using NVME model number : INTEL SSDPEDMD020T4.
kernel : 4.13.13
fio version : 2.21
BTW, your fio version is couple of years behind.
can anybody tell me what could be the issue?
The latency data above clearly shows the reason: writes are being handled by the internal SSD cache of few usec latency, while reads are served off the real media, with typical (high-end) SSD latency of about 100 usec.
Regards,
Andrey
Thanks,
Mayank
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org<mailto:SPDK(a)lists.01.org>
https://lists.01.org/mailman/listinfo/spdk
--
Regards,
Andrey
[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 10189 bytes --]
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [SPDK] SPDK FIO poor 4K read performance
@ 2017-12-18 23:24 Kariuki, John K
0 siblings, 0 replies; 15+ messages in thread
From: Kariuki, John K @ 2017-12-18 23:24 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 3026 bytes --]
Mayank
Can you share your fio configuration file?
Have you tried the latency performance test with the SPDK perf tool?
Your random writes are pretty high. Did you precondition the SSD before the test? For the random write test you need to precondition the SSD by doing 90 minutes of random writes before testing.
Here is a link to a video<https://www.youtube.com/watch?v=tkGE3pq5eIU&index=8&list=PLj-81kG3zG5ZIE-4CvqsvlFEHoOoWRIHZ> that highlights some performance best practices that you might want to check out.
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Andrey Kuzmin
Sent: Monday, December 18, 2017 3:40 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Cc: mayank.patel(a)mayadata.io
Subject: Re: [SPDK] SPDK FIO poor 4K read performance
On Tue, Dec 19, 2017, 01:27 Isaac Otsiabah <IOtsiabah(a)us.fujitsu.com<mailto:IOtsiabah(a)us.fujitsu.com>> wrote:
Hi Audrey,
Hi Isaac,
I guess you meant to address the original submission author, Mayank.
Regards,
Andrey
which
plugin are you running? examples/nvme/fio_plugin/fio_plugin or examples/bdev/fio_plugin/fio_plugin ? Is it on Fedora or Centos?
Isaac
From: SPDK [mailto:spdk-bounces(a)lists.01.org<mailto:spdk-bounces(a)lists.01.org>] On Behalf Of Andrey Kuzmin
Sent: Monday, December 18, 2017 5:59 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] SPDK FIO poor 4K read performance
On Mon, Dec 18, 2017, 16:08 Mayank <mayank.patel(a)mayadata.io<mailto:mayank.patel(a)mayadata.io>> wrote:
Hi All,
I have performed fio benchmark test with SPDK fio plugin for throughput test with iodepth = 32 and numjobs=4.
Latency results are as below:
rand read
11.2k
45.9MB/s
88.83 usec
rand write
126k
518MB/s
7.56 usec
Throughput results are as below:
Throughput (io:32 thr:4)
IOPS
Throuhgput
Latency (usec)
seq read
406k
1665MB/s
314.71
seq write
455k
1865MB/s
280.87
rand read
478k
1959MB/s
267.29
rand write
442k
1812MB/s
289.03
In results, 'm getting sequential read IOPS lesser than sequential write IOPS.
I am using NVME model number : INTEL SSDPEDMD020T4.
kernel : 4.13.13
fio version : 2.21
BTW, your fio version is couple of years behind.
can anybody tell me what could be the issue?
The latency data above clearly shows the reason: writes are being handled by the internal SSD cache of few usec latency, while reads are served off the real media, with typical (high-end) SSD latency of about 100 usec.
Regards,
Andrey
Thanks,
Mayank
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org<mailto:SPDK(a)lists.01.org>
https://lists.01.org/mailman/listinfo/spdk
--
Regards,
Andrey
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org<mailto:SPDK(a)lists.01.org>
https://lists.01.org/mailman/listinfo/spdk
--
Regards,
Andrey
[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 16544 bytes --]
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [SPDK] SPDK FIO poor 4K read performance
@ 2017-12-18 22:40 Andrey Kuzmin
0 siblings, 0 replies; 15+ messages in thread
From: Andrey Kuzmin @ 2017-12-18 22:40 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 2312 bytes --]
On Tue, Dec 19, 2017, 01:27 Isaac Otsiabah <IOtsiabah(a)us.fujitsu.com> wrote:
> Hi Audrey,
>
Hi Isaac,
I guess you meant to address the original submission author, Mayank.
Regards,
Andrey
which
> plugin are you running? examples/nvme/fio_plugin/fio_plugin or
> examples/bdev/fio_plugin/fio_plugin ? Is it on Fedora or Centos?
>
>
>
> Isaac
>
> *From:* SPDK [mailto:spdk-bounces(a)lists.01.org] *On Behalf Of *Andrey
> Kuzmin
> *Sent:* Monday, December 18, 2017 5:59 AM
> *To:* Storage Performance Development Kit <spdk(a)lists.01.org>
> *Subject:* Re: [SPDK] SPDK FIO poor 4K read performance
>
>
>
>
>
> On Mon, Dec 18, 2017, 16:08 Mayank <mayank.patel(a)mayadata.io> wrote:
>
> Hi All,
>
> I have performed fio benchmark test with SPDK fio plugin for throughput
> test with iodepth = 32 and numjobs=4.
>
> Latency results are as below:
>
>
>
> rand read
>
> 11.2k
>
> 45.9MB/s
>
> 88.83 usec
>
> rand write
>
> 126k
>
> 518MB/s
>
> 7.56 usec
>
> Throughput results are as below:
>
>
>
> Throughput (io:32 thr:4)
>
> IOPS
>
> Throuhgput
>
> Latency (usec)
>
> seq read
>
> 406k
>
> 1665MB/s
>
> 314.71
>
> seq write
>
> 455k
>
> 1865MB/s
>
> 280.87
>
> rand read
>
> 478k
>
> 1959MB/s
>
> 267.29
>
> rand write
>
> 442k
>
> 1812MB/s
>
> 289.03
>
>
>
> In results, 'm getting sequential read IOPS lesser than sequential write
> IOPS.
>
> I am using NVME model number : INTEL SSDPEDMD020T4.
> kernel : 4.13.13
> fio version : 2.21
>
>
>
> BTW, your fio version is couple of years behind.
>
>
> can anybody tell me what could be the issue?
>
>
>
> The latency data above clearly shows the reason: writes are being handled
> by the internal SSD cache of few usec latency, while reads are served off
> the real media, with typical (high-end) SSD latency of about 100 usec.
>
>
>
> Regards,
>
> Andrey
>
>
>
> Thanks,
> Mayank
>
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
>
> --
>
> Regards,
> Andrey
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
>
--
Regards,
Andrey
[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 9107 bytes --]
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [SPDK] SPDK FIO poor 4K read performance
@ 2017-12-18 22:27 Isaac Otsiabah
0 siblings, 0 replies; 15+ messages in thread
From: Isaac Otsiabah @ 2017-12-18 22:27 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 1763 bytes --]
Hi Audrey, which plugin are you running? examples/nvme/fio_plugin/fio_plugin or examples/bdev/fio_plugin/fio_plugin ? Is it on Fedora or Centos?
Isaac
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Andrey Kuzmin
Sent: Monday, December 18, 2017 5:59 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] SPDK FIO poor 4K read performance
On Mon, Dec 18, 2017, 16:08 Mayank <mayank.patel(a)mayadata.io<mailto:mayank.patel(a)mayadata.io>> wrote:
Hi All,
I have performed fio benchmark test with SPDK fio plugin for throughput test with iodepth = 32 and numjobs=4.
Latency results are as below:
rand read
11.2k
45.9MB/s
88.83 usec
rand write
126k
518MB/s
7.56 usec
Throughput results are as below:
Throughput (io:32 thr:4)
IOPS
Throuhgput
Latency (usec)
seq read
406k
1665MB/s
314.71
seq write
455k
1865MB/s
280.87
rand read
478k
1959MB/s
267.29
rand write
442k
1812MB/s
289.03
In results, 'm getting sequential read IOPS lesser than sequential write IOPS.
I am using NVME model number : INTEL SSDPEDMD020T4.
kernel : 4.13.13
fio version : 2.21
BTW, your fio version is couple of years behind.
can anybody tell me what could be the issue?
The latency data above clearly shows the reason: writes are being handled by the internal SSD cache of few usec latency, while reads are served off the real media, with typical (high-end) SSD latency of about 100 usec.
Regards,
Andrey
Thanks,
Mayank
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org<mailto:SPDK(a)lists.01.org>
https://lists.01.org/mailman/listinfo/spdk
--
Regards,
Andrey
[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 8895 bytes --]
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [SPDK] SPDK FIO poor 4K read performance
@ 2017-12-18 13:58 Andrey Kuzmin
0 siblings, 0 replies; 15+ messages in thread
From: Andrey Kuzmin @ 2017-12-18 13:58 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 1307 bytes --]
On Mon, Dec 18, 2017, 16:08 Mayank <mayank.patel(a)mayadata.io> wrote:
> Hi All,
>
> I have performed fio benchmark test with SPDK fio plugin for throughput
> test with iodepth = 32 and numjobs=4.
>
> Latency results are as below:
>
>
> rand read 11.2k 45.9MB/s 88.83 usec
> rand write 126k 518MB/s 7.56 usec
>
> Throughput results are as below:
>
> Throughput (io:32 thr:4) IOPS Throuhgput Latency (usec)
> seq read 406k 1665MB/s 314.71
> seq write 455k 1865MB/s 280.87
> rand read 478k 1959MB/s 267.29
> rand write 442k 1812MB/s 289.03
>
> In results, 'm getting sequential read IOPS lesser than sequential write
> IOPS.
>
> I am using NVME model number : INTEL SSDPEDMD020T4.
> kernel : 4.13.13
> fio version : 2.21
>
BTW, your fio version is couple of years behind.
>
> can anybody tell me what could be the issue?
>
The latency data above clearly shows the reason: writes are being handled
by the internal SSD cache of few usec latency, while reads are served off
the real media, with typical (high-end) SSD latency of about 100 usec.
Regards,
Andrey
>
>
> Thanks,
> Mayank
>
>
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
>
--
Regards,
Andrey
[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 4064 bytes --]
^ permalink raw reply [flat|nested] 15+ messages in thread
* [SPDK] SPDK FIO poor 4K read performance
@ 2017-12-18 13:08 Mayank
0 siblings, 0 replies; 15+ messages in thread
From: Mayank @ 2017-12-18 13:08 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 735 bytes --]
Hi All,
I have performed fio benchmark test with SPDK fio plugin for throughput
test with iodepth = 32 and numjobs=4.
Latency results are as below:
rand read 11.2k 45.9MB/s 88.83 usec
rand write 126k 518MB/s 7.56 usec
Throughput results are as below:
Throughput (io:32 thr:4) IOPS Throuhgput Latency (usec)
seq read 406k 1665MB/s 314.71
seq write 455k 1865MB/s 280.87
rand read 478k 1959MB/s 267.29
rand write 442k 1812MB/s 289.03
In results, 'm getting sequential read IOPS lesser than sequential write
IOPS.
I am using NVME model number : INTEL SSDPEDMD020T4.
kernel : 4.13.13
fio version : 2.21
can anybody tell me what could be the issue?
Thanks,
Mayank
[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 3439 bytes --]
^ permalink raw reply [flat|nested] 15+ messages in thread
end of thread, other threads:[~2018-01-11 20:30 UTC | newest]
Thread overview: 15+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-12-21 17:54 [SPDK] SPDK FIO poor 4K read performance Kariuki, John K
-- strict thread matches above, loose matches on Subject: below --
2018-01-11 20:30 Mayank Patel
2018-01-10 0:07 Kariuki, John K
2018-01-09 7:10 Mayank
2018-01-08 16:14 Kariuki, John K
2018-01-08 9:00 Mayank Patel
2017-12-20 5:12 Mayank
2017-12-19 21:47 Kariuki, John K
2017-12-19 6:33 Mayank
2017-12-19 0:12 Isaac Otsiabah
2017-12-18 23:24 Kariuki, John K
2017-12-18 22:40 Andrey Kuzmin
2017-12-18 22:27 Isaac Otsiabah
2017-12-18 13:58 Andrey Kuzmin
2017-12-18 13:08 Mayank
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.