* Re: [SPDK] FIO NVMe Performance Results
@ 2018-11-05 4:40 Cao, Gang
0 siblings, 0 replies; 25+ messages in thread
From: Cao, Gang @ 2018-11-05 4:40 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 8478 bytes --]
For the SPDK plugin, from the below result, looks like only 1 job started with 32 queue depth:
Jobs: 1 (f=0): [f(1),_(3)][100.0%][eta 00m:00s] <<<<
Try to update the FIO configuration file as
[job1]
filename=trtype=PCIe traddr=0000.1c.00.0 ns=1
[job2]
filename=trtype=PCIe traddr=0000.1c.00.0 ns=1
....
The kernel FIO is with 4 jobs and each job of 32 queue depth. For SPDK, you can configure as the same, or just configure two jobs and each job of 64 queue depth. The overall queue depth to the device is same of 128.
Thanks,
Gang
-----Original Message-----
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Gruher, Joseph R
Sent: Monday, November 5, 2018 10:49 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: [SPDK] FIO NVMe Performance Results
Hi folks-
I'm testing SPDK 18.10 on Ubuntu 18.04 with kernel 4.18.16. I used FIO with the kernel NVMe driver to measure the performance of a local (PCIe attached) Intel P4500 NVMe device on a 4KB random read workload and obtained 477K IOPS, roughly in line with the drive spec. Then I tested the same drive with the SPDK FIO plugin and only achieved 13K IOPS. The FIO test files and the results are pasted below. Any ideas where I'm going wrong here?
Thanks!
don(a)donst201:~/fio/single/rr$ ls /sys/devices/pci0000:17/0000:17:02.0/0000:1c:00.0/nvme/nvme1/
address cntlid dev device firmware_rev model nvme2n1 power rescan_controller reset_controller serial state subsysnqn subsystem transport uevent
don(a)donst201:~/fio/single/rr$ cat nvme2n1.ini |grep -v '#'
[global]
rw=randrw
rwmixread=100
numjobs=4
iodepth=32
bs=4k
direct=1
thread=1
time_based=1
ramp_time=0
runtime=10
ioengine=libaio
group_reporting=1
unified_rw_reporting=1
exitall=1
randrepeat=0
norandommap=1
[nvme2n1]
filename=/dev/nvme2n1
don(a)donst201:~/fio/single/rr$ sudo fio nvme2n1.ini
nvme2n1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32 ...
fio-3.11
Starting 4 threads
Jobs: 4 (f=4): [r(4)][100.0%][r=1874MiB/s][r=480k IOPS][eta 00m:00s]
nvme2n1: (groupid=0, jobs=4): err= 0: pid=2575: Mon Nov 5 02:26:25 2018
mixed: IOPS=477k, BW=1862MiB/s (1952MB/s)(18.2GiB/10001msec)
slat (nsec): min=1295, max=333088, avg=2216.35, stdev=927.89
clat (nsec): min=506, max=5818.8k, avg=265924.16, stdev=223761.59
lat (usec): min=6, max=5821, avg=268.19, stdev=223.76
clat percentiles (usec):
| 1.00th=[ 13], 5.00th=[ 74], 10.00th=[ 87], 20.00th=[ 115],
| 30.00th=[ 139], 40.00th=[ 167], 50.00th=[ 204], 60.00th=[ 247],
| 70.00th=[ 306], 80.00th=[ 388], 90.00th=[ 523], 95.00th=[ 660],
| 99.00th=[ 988], 99.50th=[ 1156], 99.90th=[ 2343], 99.95th=[ 3195],
| 99.99th=[ 4424]
bw ( KiB/s): min=448192, max=483104, per=25.00%, avg=476597.10, stdev=6748.78, samples=80
iops : min=112048, max=120776, avg=119149.27, stdev=1687.20, samples=80
lat (nsec) : 750=0.01%
lat (usec) : 10=0.52%, 20=1.25%, 50=0.25%, 100=13.38%, 250=45.13%
lat (usec) : 500=28.23%, 750=8.08%, 1000=2.22%
lat (msec) : 2=0.82%, 4=0.10%, 10=0.02%
cpu : usr=14.08%, sys=36.02%, ctx=1930207, majf=0, minf=697
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued rwts: total=4766095,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
Run status group 0 (all jobs):
MIXED: bw=1862MiB/s (1952MB/s), 1862MiB/s-1862MiB/s (1952MB/s-1952MB/s), io=18.2GiB (19.5GB), run=10001-10001msec Disk stats (read/write):
nvme2n1: ios=4709190/0, merge=0/0, ticks=4096/0, in_queue=1388752, util=100.00%
don(a)donst201:~/fio/single/rr$ sudo /home/don/install/spdk/spdk/scripts/setup.sh
Active mountpoints on /dev/nvme0n1, so not binding PCI dev 0000:03:00.0
0000:1c:00.0 (8086 0a54): nvme -> uio_pci_generic
0000:1d:00.0 (8086 0a54): nvme -> uio_pci_generic
0000:5e:00.0 (8086 0a54): nvme -> uio_pci_generic
0000:5f:00.0 (8086 0a54): nvme -> uio_pci_generic
0000:62:00.0 (8086 0a54): nvme -> uio_pci_generic
0000:63:00.0 (8086 0a54): nvme -> uio_pci_generic
0000:64:00.0 (8086 0a54): nvme -> uio_pci_generic
0000:65:00.0 (8086 0a54): nvme -> uio_pci_generic
0000:da:00.0 (8086 0a54): nvme -> uio_pci_generic
0000:db:00.0 (8086 0a54): nvme -> uio_pci_generic
0000:dc:00.0 (8086 0a54): nvme -> uio_pci_generic
0000:dd:00.0 (8086 0a54): nvme -> uio_pci_generic
0000:e0:00.0 (8086 0a54): nvme -> uio_pci_generic
0000:e1:00.0 (8086 0a54): nvme -> uio_pci_generic
0000:e2:00.0 (8086 0a54): nvme -> uio_pci_generic
0000:e3:00.0 (8086 0a54): nvme -> uio_pci_generic
0000:00:04.0 (8086 2021): ioatdma -> uio_pci_generic
0000:00:04.1 (8086 2021): ioatdma -> uio_pci_generic
0000:00:04.2 (8086 2021): ioatdma -> uio_pci_generic
0000:00:04.3 (8086 2021): ioatdma -> uio_pci_generic
0000:00:04.4 (8086 2021): ioatdma -> uio_pci_generic
0000:00:04.5 (8086 2021): ioatdma -> uio_pci_generic
0000:00:04.6 (8086 2021): ioatdma -> uio_pci_generic
0000:00:04.7 (8086 2021): ioatdma -> uio_pci_generic
0000:80:04.0 (8086 2021): ioatdma -> uio_pci_generic
0000:80:04.1 (8086 2021): ioatdma -> uio_pci_generic
0000:80:04.2 (8086 2021): ioatdma -> uio_pci_generic
0000:80:04.3 (8086 2021): ioatdma -> uio_pci_generic
0000:80:04.4 (8086 2021): ioatdma -> uio_pci_generic
0000:80:04.5 (8086 2021): ioatdma -> uio_pci_generic
0000:80:04.6 (8086 2021): ioatdma -> uio_pci_generic
0000:80:04.7 (8086 2021): ioatdma -> uio_pci_generic
don(a)donst201:~/fio/single/rr$ cat 1c.ini |grep -v '#'
[global]
rw=randrw
rwmixread=100
numjobs=4
iodepth=32
bs=4k
direct=1
thread=1
time_based=1
ramp_time=0
runtime=10
ioengine=/home/don/install/spdk/spdk/examples/nvme/fio_plugin/fio_plugin
group_reporting=1
unified_rw_reporting=1
exitall=1
randrepeat=0
norandommap=1
[0000.1c.00.0]
filename=trtype=PCIe traddr=0000.1c.00.0 ns=1
don(a)donst201:~/fio/single/rr$ sudo fio 1c.ini
0000.1c.00.0: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk, iodepth=32 ...
fio-3.11
Starting 4 threads
Starting SPDK v18.10 / DPDK 18.08.0 initialization...
[ DPDK EAL parameters: fio --no-shconf -c 0x1 -m 512 --file-prefix=spdk_pid3668 ]
EAL: Detected 36 lcore(s)
EAL: Detected 2 NUMA nodes
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: PCI device 0000:1c:00.0 on NUMA socket 0
EAL: probe driver: 8086:a54 spdk_nvme
Jobs: 1 (f=0): [f(1),_(3)][100.0%][eta 00m:00s]
0000.1c.00.0: (groupid=0, jobs=4): err= 0: pid=3709: Mon Nov 5 02:28:29 2018
mixed: IOPS=13.2k, BW=51.6MiB/s (54.1MB/s)(517MiB/10011msec)
slat (nsec): min=109, max=15344, avg=127.91, stdev=73.44
clat (usec): min=187, max=18715, avg=9683.80, stdev=650.54
lat (usec): min=193, max=18715, avg=9683.93, stdev=650.53
clat percentiles (usec):
| 1.00th=[ 8717], 5.00th=[ 8848], 10.00th=[ 8979], 20.00th=[ 9110],
| 30.00th=[ 9241], 40.00th=[ 9372], 50.00th=[ 9503], 60.00th=[ 9765],
| 70.00th=[10028], 80.00th=[10159], 90.00th=[10552], 95.00th=[10945],
| 99.00th=[11338], 99.50th=[11338], 99.90th=[11731], 99.95th=[13960],
| 99.99th=[17695]
bw ( KiB/s): min=13080, max=13304, per=25.00%, avg=13214.70, stdev=55.30, samples=80
iops : min= 3270, max= 3326, avg=3303.65, stdev=13.84, samples=80
lat (usec) : 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.02%, 10=70.85%, 20=29.11%
cpu : usr=100.01%, sys=0.00%, ctx=23, majf=0, minf=0
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.9%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued rwts: total=132300,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
Run status group 0 (all jobs):
MIXED: bw=51.6MiB/s (54.1MB/s), 51.6MiB/s-51.6MiB/s (54.1MB/s-54.1MB/s), io=517MiB (542MB), run=10011-10011msec
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [SPDK] FIO NVMe Performance Results
@ 2018-11-06 10:31 Andrey Kuzmin
0 siblings, 0 replies; 25+ messages in thread
From: Andrey Kuzmin @ 2018-11-06 10:31 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 53725 bytes --]
Also, it might be helpful to run the same workload against all of your
drives (to simplify, you can set up a single qd=128 job - meaning fio
thread, not a separate jib file - per drive). This has a potential to
pinpoint any issue with the specific drive/pcie slot.
Regards,
Andrey
On Tue, Nov 6, 2018, 04:40 Cao, Gang <gang.cao(a)intel.com> wrote:
> Hi Joe,
>
> Have you tried the SPDK perf tool? We also tried locally and does not see
> this symptom with FIO.
>
> Have a try on SPDK perf tool may give some ideas on the FIO related
> configuration. You can also pick two or four cores through the core_mask to
> ensure these cores are not used.
>
> Thanks,
> Gang
>
> -----Original Message-----
> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Gruher, Joseph
> R
> Sent: Tuesday, November 6, 2018 6:19 AM
> To: Storage Performance Development Kit <spdk(a)lists.01.org>
> Subject: Re: [SPDK] FIO NVMe Performance Results
>
> Hi Jim-
>
> Yes, this is all local PCIe attached NVMe drives for now - the goal is to
> baseline the local NVMe on the system with SPDK and then move on to NVMeoF
> from there.
>
> The basic test process (once SPDK has been built) is to reboot the system,
> run the included SPDK scripts/setup.sh script, and then run FIO with the
> SPDK user or bdev plugin. We're not starting app/nvmf_tgt/nvmf_tgt or
> anything like that so I don't think SPDK should be running anywhere outside
> of what is launched through FIO. All cores look idle until FIO is
> launched. If that doesn't sound like the right test process, let us know!
>
> Thanks,
> Joe
>
> > -----Original Message-----
> > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Harris,
> > James R
> > Sent: Monday, November 5, 2018 1:56 PM
> > To: Storage Performance Development Kit <spdk(a)lists.01.org>
> > Subject: Re: [SPDK] FIO NVMe Performance Results
> >
> > Hi Joe,
> >
> > This is all just local PCIe NVMe – no NVMe-oF? Can you make sure you
> > don’t have any other SPDK processes running while you’re doing this?
> > I’m concerned that fio is polling on the same CPU core as an SPDK
> > process, and your performance degradation is due to timeslice sharing.
> >
> > -Jim
> >
> >
> > On 11/5/18, 2:42 PM, "SPDK on behalf of Howell, Seth" <spdk-
> > bounces(a)lists.01.org on behalf of seth.howell(a)intel.com> wrote:
> >
> > I think that the uio_pci_generic vs. vfio thing is unrelated. The
> > only reason I brought it up was because it could have been indicative
> > of a larger configuration difference. However, I ran my test with and
> > without vfio support just to be sure and got the higher performance
> numbers both times.
> >
> > As a side note:
> > It's probably a good thing to have enabled in general when running
> > spdk benchmarks because that is the recommended configuration on any
> > physical machine.
> > You can enable it in grub by adding the following to the
> > GRUB_CMDLINE_LINUX line of /etc/default.grub: intel_iommu=on
> > and then running grub2-mkconfig.
> >
> > -----Original Message-----
> > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Gruher,
> > Joseph R
> > Sent: Monday, November 5, 2018 1:39 PM
> > To: Storage Performance Development Kit <spdk(a)lists.01.org>
> > Subject: Re: [SPDK] FIO NVMe Performance Results
> >
> > Attempting to fall back to earlier revisions has not yielded any
> > improvements for me. All tests use the same P4500 physical disk. FIO
> > files are as shown before, running a 4 thread, queue depth 32 per
> > thread, 4KB random read workload.
> >
> > SPDK 18.10, kernel 4.18.16, FIO 3.11 = 12.9k IOPS SPDK 18.10,
> > kernel 4.16.14, FIO 3.11 = 11.9k IOPS SPDK 18.04, kernel 4.18.16, FIO
> > 3.11 = 12.9k IOPS SPDK 18.04, kernel 4.16.14, FIO 3.11 = 13.0k IOPS
> > SPDK 18.04, kernel 4.16.16, FIO 3.3 = 13.1k IOPS
> >
> > Kernel NVMe driver, kernel 4.16.16, FIO 3.3 = 477k IOPS Kernel
> > NVMe driver, kernel 4.18.16, FIO 3.11 = 477k IOPS
> >
> > I am curious if the comment below about uio_pci_generic instead of
> > vfio can lead us anywhere, or maybe there's some OS setting or BIOS
> > setting that needs to be toggled...
> >
> > For the folks working inside Intel, I can also give you direct
> > access to the test system, if that would help.
> >
> > Thanks,
> > Joe
> >
> > > -----Original Message-----
> > > From: Gruher, Joseph R
> > > Sent: Monday, November 5, 2018 11:05 AM
> > > To: Storage Performance Development Kit <spdk(a)lists.01.org>
> > > Subject: RE: [SPDK] FIO NVMe Performance Results
> > >
> > > I've had good FIO NVMe results in the past we well, with SPDK
> 18.04 on
> > > Ubuntu 18.04, with kernel 4.16.14. That was on a different, but
> > > similar, hardware platform.
> > >
> > > I suspect something has changed now in SPDK 18.10 and/or kernel
> > > 4.18.16. I can try going back to earlier versions of each to see
> if I
> > > can recreate earlier good results and find where things went wrong.
> > >
> > > We're not running in a VM - this is a bare metal Xeon server with
> > > Skylake CPUs.
> > >
> > > Thanks,
> > > Joe
> > >
> > > > -----Original Message-----
> > > > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of
> Howell,
> > > > Seth
> > > > Sent: Monday, November 5, 2018 10:32 AM
> > > > To: Storage Performance Development Kit <spdk(a)lists.01.org>
> > > > Subject: Re: [SPDK] FIO NVMe Performance Results
> > > >
> > > > Hi Joseph,
> > > > I attempted to verify your results on my system. I am using
> fedora
> > > > 27 with kernel version 4.17.3 with a p3700 drive. I cloned fio
> from
> > > > https://github.com/axboe/fio and checked out the appropriate
> tag. I
> > > > got appx 408k iops.
> > > > Typically we run spdk with the iommu enabled. I noticed that your
> > > > devices are binding to uio_pci_generic instead of vfio. This
> > > > shouldn't make any difference at all, but I just want to make
> sure
> > > > you aren't running in a VM or something like that and passing
> > > > through your NVMe drives and that's why you aren't using the
> IOMMU.
> > > > That could explain a deeper configuration difference that could
> help
> > > > us know where to look for
> > > the disparity.
> > > >
> > > > Please see the contents of my configuration files and output
> below:
> > > >
> > > > Test.ini:
> > > > [global]
> > > > rw=randrw
> > > > rwmixread=100
> > > > numjobs=4
> > > > iodepth=32
> > > > bs=4k
> > > > direct=1
> > > > thread=1
> > > > time_based=1
> > > > ramp_time=0
> > > > runtime=10
> > > > ioengine=spdk_bdev
> > > > spdk_conf=./nvme.conf
> > > > group_reporting=1
> > > > unified_rw_reporting=1
> > > > exitall=1
> > > > randrepeat=0
> > > > norandommap=1
> > > > [0000.81.00.0]
> > > > filename=Nvme0n1
> > > >
> > > > nvme.conf:
> > > > [Nvme]
> > > > TransportId "trtype:PCIe traddr:0000.81.00.0" Nvme0
> > > >
> > > > Sudo
> > > >
> > >
> > LD_PRELOAD=/home/sethhowe/Desktop/Development_Folders/spdk/exam
> > > > ples/bdev/fio_plugin/fio_plugin ./fio ./test.ini >./out,txt 2>&1
> > > >
> > > > Out.txt:
> > > > 0000.81.00.0: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-
> > 4096B,
> > > > (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=32 ...
> > > > fio-3.11
> > > > Starting 4 threads
> > > > Starting SPDK v18.10 / DPDK 18.08.0 initialization...
> > > > [ DPDK EAL parameters: fio --no-shconf -c 0x1
> > > > --file-prefix=spdk_pid782339 ]
> > > > EAL: Detected 16 lcore(s)
> > > > EAL: Detected 2 NUMA nodes
> > > > EAL: No free hugepages reported in hugepages-1048576kB
> > > > EAL: Probing VFIO support...
> > > > EAL: VFIO support initialized
> > > > EAL: PCI device 0000:81:00.0 on NUMA socket 1
> > > > EAL: probe driver: 8086:953 spdk_nvme
> > > >
> > > > 0000.81.00.0: (groupid=0, jobs=4): err= 0: pid=782359: Mon Nov
> 5
> > > > 10:58:59
> > > > 2018
> > > > mixed: IOPS=410k, BW=1600MiB/s (1678MB/s)(15.6GiB/10001msec)
> > > > slat (nsec): min=179, max=19985, avg=245.97, stdev=122.96
> > > > clat (usec): min=8, max=1821, avg=311.82, stdev=167.14
> > > > lat (usec): min=8, max=1822, avg=312.06, stdev=167.14
> > > > clat percentiles (usec):
> > > > | 1.00th=[ 24], 5.00th=[ 48], 10.00th=[ 78],
> 20.00th=[ 141],
> > > > | 30.00th=[ 200], 40.00th=[ 258], 50.00th=[ 314],
> 60.00th=[ 371],
> > > > | 70.00th=[ 429], 80.00th=[ 486], 90.00th=[ 545],
> 95.00th=[ 570],
> > > > | 99.00th=[ 594], 99.50th=[ 594], 99.90th=[ 627],
> 99.95th=[ 652],
> > > > | 99.99th=[ 676]
> > > > bw ( KiB/s): min=407264, max=416040, per=25.01%,
> avg=409742.55,
> > > > stdev=1337.38, samples=76
> > > > iops : min=101816, max=104010, avg=102435.61,
> stdev=334.41,
> > > > samples=76
> > > > lat (usec) : 10=0.01%, 20=0.46%, 50=4.88%, 100=8.41%,
> 250=25.29%
> > > > lat (usec) : 500=43.15%, 750=17.81%, 1000=0.01%
> > > > lat (msec) : 2=0.01%
> > > > cpu : usr=99.78%, sys=0.00%, ctx=26, majf=0, minf=0
> > > > IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%,
> 32=99.9%,
> > > > >=64=0.0%
> > > > submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%,
> 64=0.0%,
> > > > >=64=0.0%
> > > > complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.1%, 32=0.1%,
> > > > 64=0.0%,
> > > > >=64=0.0%
> > > > issued rwts: total=4096867,0,0,0 short=0,0,0,0
> dropped=0,0,0,0
> > > > latency : target=0, window=0, percentile=100.00%, depth=32
> > > >
> > > > Run status group 0 (all jobs):
> > > > MIXED: bw=1600MiB/s (1678MB/s), 1600MiB/s-1600MiB/s
> > (1678MB/s-
> > > > 1678MB/s), io=15.6GiB (16.8GB), run=10001-10001msec
> > > >
> > > > Thanks,
> > > >
> > > > Seth
> > > >
> > > >
> > > >
> > > > -----Original Message-----
> > > > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of
> Andrey
> > > > Kuzmin
> > > > Sent: Monday, November 5, 2018 9:48 AM
> > > > To: Storage Performance Development Kit <spdk(a)lists.01.org>
> > > > Subject: Re: [SPDK] FIO NVMe Performance Results
> > > >
> > > > On Mon, Nov 5, 2018, 18:29 Gruher, Joseph R
> > > > <joseph.r.gruher(a)intel.com>
> > > > wrote:
> > > >
> > > > > My FIO version is v3.11.
> > > > >
> > > > > I tried the bdev FIO plugin as suggested, but performance is
> about
> > > > > the same as the other SPDK FIO plugin, I observed only 10.4K
> IOPS
> > > > > in a quick 4KB random read test. See configuration and
> > results below.
> > > > >
> > > >
> > > > As 12 ms median read latency for an nvme ssd is far beyond
> anything
> > > > reasonable, why don't you just perf this run, to see where the
> time
> > > > is being spent?
> > > >
> > > > Regards,
> > > > Andrey
> > > >
> > > > >
> > > > >
> > > > > don(a)donst201:~/fio/single/rr$ cat nvme.conf [Nvme] TransportId
> > > > > "trtype:PCIe traddr:0000.1c.00.0" Nvme0
> > > > >
> > > > >
> > > > > don(a)donst201:~/fio/single/rr$ cat 1c-bdev.ini |grep -v '#'
> > > > > [global]
> > > > > rw=randrw
> > > > > rwmixread=100
> > > > > numjobs=4
> > > > > iodepth=32
> > > > > bs=4k
> > > > > direct=1
> > > > > thread=1
> > > > > time_based=1
> > > > > ramp_time=0
> > > > > runtime=10
> > > > > ioengine=spdk_bdev
> > > > > spdk_conf=./nvme.conf
> > > > > group_reporting=1
> > > > > unified_rw_reporting=1
> > > > > exitall=1
> > > > > randrepeat=0
> > > > > norandommap=1
> > > > > [0000.1c.00.0]
> > > > > filename=Nvme0n1
> > > > >
> > > > >
> > > > > don(a)donst201:~/fio/single/rr$ sudo
> > > > >
> > > >
> > >
> > LD_PRELOAD=/home/don/install/spdk/spdk/examples/bdev/fio_plugin/fio_p
> > > > l
> > > > > ugin
> > > > > fio 1c-bdev.ini
> > > > > 0000.1c.00.0: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W)
> > > > > 4096B-4096B,
> > > > > (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=32 ...
> > > > > fio-3.11
> > > > > Starting 4 threads
> > > > > Starting SPDK v18.10 / DPDK 18.08.0 initialization...
> > > > > [ DPDK EAL parameters: fio --no-shconf -c 0x1
> > > > > --file-prefix=spdk_pid4865 ]
> > > > > EAL: Detected 36 lcore(s)
> > > > > EAL: Detected 2 NUMA nodes
> > > > > EAL: No free hugepages reported in hugepages-1048576kB
> > > > > EAL: Probing VFIO support...
> > > > > EAL: PCI device 0000:03:00.0 on NUMA socket 0
> > > > > EAL: probe driver: 8086:f1a5 spdk_nvme
> > > > > EAL: PCI device 0000:1c:00.0 on NUMA socket 0
> > > > > EAL: probe driver: 8086:a54 spdk_nvme
> > > > > EAL: PCI device 0000:1d:00.0 on NUMA socket 0
> > > > > EAL: probe driver: 8086:a54 spdk_nvme
> > > > > EAL: PCI device 0000:5e:00.0 on NUMA socket 0
> > > > > EAL: probe driver: 8086:a54 spdk_nvme
> > > > > EAL: PCI device 0000:5f:00.0 on NUMA socket 0
> > > > > EAL: probe driver: 8086:a54 spdk_nvme
> > > > > EAL: PCI device 0000:62:00.0 on NUMA socket 0
> > > > > EAL: probe driver: 8086:a54 spdk_nvme
> > > > > EAL: PCI device 0000:63:00.0 on NUMA socket 0
> > > > > EAL: probe driver: 8086:a54 spdk_nvme
> > > > > EAL: PCI device 0000:64:00.0 on NUMA socket 0
> > > > > EAL: probe driver: 8086:a54 spdk_nvme
> > > > > EAL: PCI device 0000:65:00.0 on NUMA socket 0
> > > > > EAL: probe driver: 8086:a54 spdk_nvme
> > > > > EAL: PCI device 0000:da:00.0 on NUMA socket 1
> > > > > EAL: probe driver: 8086:a54 spdk_nvme
> > > > > EAL: PCI device 0000:db:00.0 on NUMA socket 1
> > > > > EAL: probe driver: 8086:a54 spdk_nvme
> > > > > EAL: PCI device 0000:dc:00.0 on NUMA socket 1
> > > > > EAL: probe driver: 8086:a54 spdk_nvme
> > > > > EAL: PCI device 0000:dd:00.0 on NUMA socket 1
> > > > > EAL: probe driver: 8086:a54 spdk_nvme
> > > > > EAL: PCI device 0000:e0:00.0 on NUMA socket 1
> > > > > EAL: probe driver: 8086:a54 spdk_nvme
> > > > > EAL: PCI device 0000:e1:00.0 on NUMA socket 1
> > > > > EAL: probe driver: 8086:a54 spdk_nvme
> > > > > EAL: PCI device 0000:e2:00.0 on NUMA socket 1
> > > > > EAL: probe driver: 8086:a54 spdk_nvme
> > > > > EAL: PCI device 0000:e3:00.0 on NUMA socket 1
> > > > > EAL: probe driver: 8086:a54 spdk_nvme
> > > > > Jobs: 4 (f=4): [r(4)][100.0%][r=45.7MiB/s][r=11.7k IOPS][eta
> > > > > 00m:00s]
> > > > > 0000.1c.00.0: (groupid=0, jobs=4): err= 0: pid=4906: Mon Nov 5
> > > > > 15:23:47
> > > > > 2018
> > > > > mixed: IOPS=10.4k, BW=40.5MiB/s (42.5MB/s)(406MiB/10012msec)
> > > > > slat (nsec): min=201, max=15831, avg=238.11, stdev=148.78
> > > > > clat (usec): min=124, max=31244, avg=12330.69,
> stdev=1697.98
> > > > > lat (usec): min=130, max=31244, avg=12330.93,
> stdev=1697.98
> > > > > clat percentiles (usec):
> > > > > | 1.00th=[ 9634], 5.00th=[10159], 10.00th=[10552],
> > 20.00th=[10945],
> > > > > | 30.00th=[11338], 40.00th=[11731], 50.00th=[12125],
> > > > 60.00th=[12518],
> > > > > | 70.00th=[13042], 80.00th=[13566], 90.00th=[14353],
> > > > 95.00th=[15008],
> > > > > | 99.00th=[16712], 99.50th=[18744], 99.90th=[27395],
> > > > 99.95th=[29754],
> > > > > | 99.99th=[30278]
> > > > > bw ( KiB/s): min= 8784, max=11952, per=24.99%,
> avg=10375.53,
> > > > > stdev=929.68, samples=80
> > > > > iops : min= 2196, max= 2988, avg=2593.85,
> stdev=232.41,
> > > > > samples=80
> > > > > lat (usec) : 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
> > > > > lat (msec) : 2=0.01%, 4=0.02%, 10=2.71%, 20=96.91%,
> 50=0.33%
> > > > > cpu : usr=99.99%, sys=0.01%, ctx=28, majf=0, minf=0
> > > > > IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%,
> 32=99.9%,
> > > > > >=64=0.0%
> > > > > submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%,
> > 64=0.0%,
> > > > > >=64=0.0%
> > > > > complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%,
> > > > > 64=0.0%,
> > > > > >=64=0.0%
> > > > > issued rwts: total=103907,0,0,0 short=0,0,0,0
> dropped=0,0,0,0
> > > > > latency : target=0, window=0, percentile=100.00%,
> depth=32
> > > > > Run status group 0 (all jobs):
> > > > > MIXED: bw=40.5MiB/s (42.5MB/s), 40.5MiB/s-40.5MiB/s
> > > > > (42.5MB/s-42.5MB/s), io=406MiB (426MB), run=10012-10012msec
> > > > >
> > > > >
> > > > > Thanks,
> > > > > Joe
> > > > >
> > > > > > -----Original Message-----
> > > > > > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of
> ?????
> > > > ???????
> > > > > > Sent: Monday, November 5, 2018 12:45 AM
> > > > > > To: Storage Performance Development Kit <spdk(a)lists.01.org>
> > > > > > Subject: Re: [SPDK] FIO NVMe Performance Results
> > > > > >
> > > > > > Ok, Do you see some performance issues using general bdev
> fio
> > > > > > plugin
> > > > > with
> > > > > > you configuration ?
> > > > > >
> > > > > > Best Regards
> > > > > > Alex
> > > > > >
> > > > > >
> > > > > > >Понедельник, 5 ноября 2018, 11:39 +03:00 от "Cao, Gang"
> > > > > > <gang.cao(a)intel.com>:
> > > > > > >
> > > > > > >Yes. This is for the bdev version of SPDK FIO plugin and you
> > > > > > >can assign
> > > > > the
> > > > > > conf file as the parameter in the FIO configuration.
> > > > > > >
> > > > > > >Previously, it's another SPDK FIO plugin for the user space
> > > > > > >NVMe driver
> > > > > > only.
> > > > > > >
> > > > > > >These are two different FIO plugins.
> > > > > > >
> > > > > > >Thanks,
> > > > > > >Gang
> > > > > > >
> > > > > > >-----Original Message-----
> > > > > > >From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of
> > > > > > >Маров Алексей
> > > > > > >Sent: Monday, November 5, 2018 4:32 PM
> > > > > > >To: Storage Performance Development Kit < spdk(a)lists.01.org
> >
> > > > > > >Subject: Re: [SPDK] FIO NVMe Performance Results
> > > > > > >
> > > > > > >Sorry for incomplete message I have sent before
> > > > > > >
> > > > > > >Hello Gang.
> > > > > > >
> > > > > > >I just curious, why would you run fio with SPDK using this
> conf
> > > > > > >file,
> > > > > with is a
> > > > > > bit different from what is suggested in here
> > > >
> https://github.com/spdk/spdk/tree/master/examples/bdev/fio_plugin
> > > > > > .
> > > > > > >
> > > > > > >I mean may be you should create conf file for spdk like
> this one :
> > > > > > >
> > > > > > >nvme.conf :
> > > > > > >[Nvme]
> > > > > > > TransportId "trtype:PCIe traddr:0000.1c.00.0" Nvme0
> > > > > > >
> > > > > > >and provide it in fio config.
> > > > > > >
> > > > > > >fio.ini :
> > > > > > >[global]
> > > > > > >ioengine=spdk_bdev
> > > > > > >spdk_conf=....../nvme.conf
> > > > > > >rw=randrw
> > > > > > >rwmixread=100
> > > > > > >numjobs=4
> > > > > > >iodepth=32
> > > > > > >bs=4k
> > > > > > >direct=1
> > > > > > >thread=1
> > > > > > >time_based=1
> > > > > > >ramp_time=0
> > > > > > >runtime=10
> > > > > > >group_reporting=1
> > > > > > >unified_rw_reporting=1
> > > > > > >exitall=1
> > > > > > >randrepeat=0
> > > > > > >norandommap=1
> > > > > > >
> > > > > > >[0000:06:00.0]
> > > > > > >filename=Nvme0n1
> > > > > > >
> > > > > > >and run like this
> > > > > > >
> > > > > > ># LD_PRELOAD=examples/bdev/fio_plugin/fio_plugin fio
> > > > > > >examples/bdev/fio_plugin/fio.ini
> > > > > > >
> > > > > > >Just to note. The recommended version of fio for SPDK is 3.3
> > > > > > >
> > > > > > >Best Regards
> > > > > > >Alex
> > > > > > >
> > > > > > >
> > > > > > >>Понедельник, 5 ноября 2018, 9:53 +03:00 от "Cao, Gang" <
> > > > > > gang.cao(a)intel.com >:
> > > > > > >>
> > > > > > >>Thanks. This is interesting especially for your finding on
> > > > > > >>queue depth
> > > > > 1
> > > > > > case. We are trying to reproduce this issue locally same as
> your
> > > > > > configuration.
> > > > > > >>
> > > > > > >>Thanks,
> > > > > > >>Gang
> > > > > > >>
> > > > > > >>-----Original Message-----
> > > > > > >>From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of
> > > > > > >>Gruher, Joseph R
> > > > > > >>Sent: Monday, November 5, 2018 1:32 PM
> > > > > > >>To: Storage Performance Development Kit <
> spdk(a)lists.01.org >
> > > > > > >>Subject: Re: [SPDK] FIO NVMe Performance Results
> > > > > > >>
> > > > > > >>Interesting catch. It seems like FIO is actually not
> > > > > > >>accurately
> > > > > reporting the
> > > > > > number of jobs in the report out for the SPDK case. If I
> > > > > > disable the group_reporting parameter I do get a printout for
> > > > > > four separate jobs in
> > > > > the
> > > > > > SPDK case, so I think it is really running four jobs there,
> and
> > > > > > just
> > > > > summarizing
> > > > > > incorrectly. I also see four separate CPU cores go to 100%
> when
> > > > > > I run
> > > > > that
> > > > > > FIO test case, indicating four jobs are active.
> > > > > > >>
> > > > > > >>Also, I don't think would explain the difference in
> performance.
> > > > > Using an
> > > > > > aggregate queue depth of 32 versus 128 shouldn't lead to a
> drop
> > > > > > in performance from 377K to 13K.
> > > > > > >>
> > > > > > >>I retested SPDK with multiple separate job descriptions as
> you
> > > > > suggested
> > > > > > anyway, using four separate jobs instead of relying on the
> > > > > > numjobs=4 parameter, and the results are below. In that SPDK
> > > > > > test with four
> > > > > separate
> > > > > > FIO job statements, each with queue depth 32, I still only
> > > > > > achieved 12.5K IOPS.
> > > > > > >>
> > > > > > >>I then re-ran the kernel NVMe driver test case with only a
> > > > > > >>single job
> > > > > for
> > > > > > comparison, and even with only 1 job and queue depth 32 it
> still
> > > > > > achieves 254K IOPS.
> > > > > > >>
> > > > > > >>We might be on the right track with a problem around queue
> > > > > > >>depth
> > > > > > however. If I run the kernel NVMe driver case with just 1
> job
> > > > > > and queue depth of only 1, I then get 11.2K IOPS, which is
> > > > > > around the performance
> > > > > I get
> > > > > > in the SPDK test case. Perhaps the SPDK case is not actually
> > > > > > queuing
> > > > > IOs on
> > > > > > the device for some reason?
> > > > > > >>
> > > > > > >>
> > > > > > >>don(a)donst201:~/fio/single/rr$ cat nvme2n1.ini |grep -v '#'
> > > > > > >>[global]
> > > > > > >>rw=randrw
> > > > > > >>rwmixread=100
> > > > > > >>numjobs=1
> > > > > > >>iodepth=32
> > > > > > >>bs=4k
> > > > > > >>direct=1
> > > > > > >>thread=1
> > > > > > >>time_based=1
> > > > > > >>ramp_time=0
> > > > > > >>runtime=10
> > > > > > >>ioengine=libaio
> > > > > > >>group_reporting=1
> > > > > > >>unified_rw_reporting=1
> > > > > > >>exitall=1
> > > > > > >>randrepeat=0
> > > > > > >>norandommap=1
> > > > > > >>[nvme2n1]
> > > > > > >>filename=/dev/nvme2n1
> > > > > > >>
> > > > > > >>
> > > > > > >>don(a)donst201:~/fio/single/rr$ sudo fio nvme2n1.ini
> > > > > > >>nvme2n1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W)
> > > > > > >>4096B-4096B,
> > > > > > >>(T) 4096B-4096B, ioengine=libaio, iodepth=32
> > > > > > >>fio-3.11
> > > > > > >>Starting 1 thread
> > > > > > >>Jobs: 1 (f=1): [r(1)][100.0%][r=1025MiB/s][r=262k IOPS][eta
> > > > > > >>00m:00s]
> > > > > > >>nvme2n1: (groupid=0, jobs=1): err= 0: pid=2422: Mon Nov 5
> > > > > > >>05:19:13
> > > > > > >>2018
> > > > > > >> mixed: IOPS=254k, BW=992MiB/s
> > > (1040MB/s)(9922MiB/10001msec)
> > > > > > >> slat (nsec): min=1185, max=319052, avg=1611.64,
> > stdev=808.40
> > > > > > >> clat (usec): min=2, max=4743, avg=124.02, stdev=121.00
> > > > > > >> lat (usec): min=6, max=4744, avg=125.68, stdev=121.02
> > > > > > >> clat percentiles (usec):
> > > > > > >> | 1.00th=[ 9], 5.00th=[ 56], 10.00th=[ 61],
> 20.00th=[
> > > > > 73],
> > > > > > >> | 30.00th=[ 80], 40.00th=[ 91], 50.00th=[ 110],
> 60.00th=[
> > > > > 118],
> > > > > > >> | 70.00th=[ 130], 80.00th=[ 157], 90.00th=[ 202],
> > > > > > >>95.00th=[ 253],
> > > > > > >> | 99.00th=[ 408], 99.50th=[ 498], 99.90th=[ 1795],
> > > > > > >>99.95th=[ 3064],
> > > > > > >> | 99.99th=[ 3785]
> > > > > > >> bw ( KiB/s): min=612240, max=1078792, per=100.00%,
> > > > > > >>avg=1015996.40,
> > > > > > stdev=106032.96, samples=20
> > > > > > >> iops : min=153060, max=269698, avg=253998.90,
> > > > stdev=26508.40,
> > > > > > samples=20
> > > > > > >> lat (usec) : 4=0.01%, 10=1.47%, 20=1.19%, 50=0.17%,
> > 100=42.03%
> > > > > > >> lat (usec) : 250=49.95%, 500=4.70%, 750=0.28%,
> 1000=0.06%
> > > > > > >> lat (msec) : 2=0.06%, 4=0.09%, 10=0.01%
> > > > > > >> cpu : usr=21.45%, sys=51.11%, ctx=530348,
> majf=0,
> > minf=139
> > > > > > >> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%,
> > > 32=100.0%,
> > > > > > >=64=0.0%
> > > > > > >> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%,
> 32=0.0%,
> > > 64=0.0%,
> > > > > > >=64=0.0%
> > > > > > >> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%,
> 32=0.1%,
> > > > > > >> 64=0.0%,
> > > > > > >>>=64=0.0%
> > > > > > >> issued rwts: total=2540023,0,0,0 short=0,0,0,0
> > dropped=0,0,0,0
> > > > > > >> latency : target=0, window=0, percentile=100.00%,
> depth=32
> > > > > > >>Run status group 0 (all jobs):
> > > > > > >> MIXED: bw=992MiB/s (1040MB/s), 992MiB/s-992MiB/s
> > (1040MB/s-
> > > > > > 1040MB/s), io=9922MiB (10.4GB), run=10001-10001msec Disk
> stats
> > > > > > (read/write):
> > > > > > >> nvme2n1: ios=2514356/0, merge=0/0, ticks=400/0,
> > > > in_queue=297612,
> > > > > > >>util=99.06%
> > > > > > >>
> > > > > > >>
> > > > > > >>don(a)donst201:~/fio/single/rr$ cat 1c.ini |grep -v '#'
> > > > > > >>[global]
> > > > > > >>rw=randrw
> > > > > > >>rwmixread=100
> > > > > > >>numjobs=1
> > > > > > >>iodepth=32
> > > > > > >>bs=4k
> > > > > > >>direct=1
> > > > > > >>thread=1
> > > > > > >>time_based=1
> > > > > > >>ramp_time=0
> > > > > > >>runtime=10
> > > > > >
> > > >
> > >>ioengine=/home/don/install/spdk/spdk/examples/nvme/fio_plugin/fio_
> > > > > > >>pl
> > > > > > ug
> > > > > > >>in
> > > > > > >>group_reporting=1
> > > > > > >>unified_rw_reporting=1
> > > > > > >>exitall=1
> > > > > > >>randrepeat=0
> > > > > > >>norandommap=1
> > > > > > >>[0000.1c.00.0-1]
> > > > > > >>filename=trtype=PCIe traddr=0000.1c.00.0 ns=1
> [0000.1c.00.0-2]
> > > > > > >>filename=trtype=PCIe traddr=0000.1c.00.0 ns=1
> [0000.1c.00.0-3]
> > > > > > >>filename=trtype=PCIe traddr=0000.1c.00.0 ns=1
> [0000.1c.00.0-4]
> > > > > > >>filename=trtype=PCIe traddr=0000.1c.00.0 ns=1
> > > > > > >>
> > > > > > >>
> > > > > > >>don(a)donst201:~/fio/single/rr$ sudo fio 1c.ini
> > > > > > >>0000.1c.00.0-1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W)
> > > > > > >>4096B-4096B,
> > > > > > >>(T) 4096B-4096B, ioengine=spdk, iodepth=32
> > > > > > >>0000.1c.00.0-2: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W)
> > > > > > >>4096B-4096B,
> > > > > > >>(T) 4096B-4096B, ioengine=spdk, iodepth=32
> > > > > > >>0000.1c.00.0-3: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W)
> > > > > > >>4096B-4096B,
> > > > > > >>(T) 4096B-4096B, ioengine=spdk, iodepth=32
> > > > > > >>0000.1c.00.0-4: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W)
> > > > > > >>4096B-4096B,
> > > > > > >>(T) 4096B-4096B, ioengine=spdk, iodepth=32
> > > > > > >>fio-3.11
> > > > > > >>Starting 4 threads
> > > > > > >>Starting SPDK v18.10 / DPDK 18.08.0 initialization...
> > > > > > >>[ DPDK EAL parameters: fio --no-shconf -c 0x1 -m 512
> > > > > > >>--file-prefix=spdk_pid4270 ]
> > > > > > >>EAL: Detected 36 lcore(s)
> > > > > > >>EAL: Detected 2 NUMA nodes
> > > > > > >>EAL: No free hugepages reported in hugepages-1048576kB
> > > > > > >>EAL: Probing VFIO support...
> > > > > > >>EAL: PCI device 0000:1c:00.0 on NUMA socket 0
> > > > > > >>EAL: probe driver: 8086:a54 spdk_nvme
> > > > > > >>Jobs: 1 (f=0): [_(3),f(1)][100.0%][eta 00m:00s]
> > > > > > >>0000.1c.00.0-1: (groupid=0, jobs=4): err= 0: pid=4310: Mon
> Nov
> > > > > > >>5
> > > > > > >>05:16:03 2018
> > > > > > >> mixed: IOPS=12.5k, BW=48.0MiB/s
> > (51.4MB/s)(490MiB/10010msec)
> > > > > > >> slat (nsec): min=98, max=15092, avg=120.02, stdev=90.07
> > > > > > >> clat (usec): min=55, max=19757, avg=10207.22,
> stdev=884.06
> > > > > > >> lat (usec): min=60, max=19757, avg=10207.34,
> stdev=884.06
> > > > > > >> clat percentiles (usec):
> > > > > > >> | 1.00th=[ 8979], 5.00th=[ 9110], 10.00th=[ 9372],
> > > > > > >>20.00th=[ 9503],
> > > > > > >> | 30.00th=[ 9765], 40.00th=[ 9896], 50.00th=[10159],
> > > > > > >>60.00th=[10290],
> > > > > > >> | 70.00th=[10552], 80.00th=[10814], 90.00th=[11207],
> > > > > > >>95.00th=[11600],
> > > > > > >> | 99.00th=[12911], 99.50th=[14353], 99.90th=[17695],
> > > > > > >>99.95th=[18744],
> > > > > > >> | 99.99th=[19006]
> > > > > > >> bw ( KiB/s): min=12008, max=12976, per=25.00%,
> > > > > > >>avg=12535.70,
> > > > > > stdev=306.81, samples=80
> > > > > > >> iops : min= 3002, max= 3244, avg=3133.85,
> stdev=76.77,
> > > > > samples=80
> > > > > > >> lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%,
> 750=0.01%,
> > > > 1000=0.01%
> > > > > > >> lat (msec) : 2=0.01%, 4=0.02%, 10=45.13%, 20=54.83%
> > > > > > >> cpu : usr=100.01%, sys=0.00%, ctx=27, majf=0,
> minf=0
> > > > > > >> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%,
> > 32=99.9%,
> > > > > > >=64=0.0%
> > > > > > >> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%,
> 32=0.0%,
> > > 64=0.0%,
> > > > > > >=64=0.0%
> > > > > > >> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%,
> 32=0.1%,
> > > > > > >> 64=0.0%,
> > > > > > >>>=64=0.0%
> > > > > > >> issued rwts: total=125506,0,0,0 short=0,0,0,0
> dropped=0,0,0,0
> > > > > > >> latency : target=0, window=0, percentile=100.00%,
> depth=32
> > > > > > >>Run status group 0 (all jobs):
> > > > > > >> MIXED: bw=48.0MiB/s (51.4MB/s), 48.0MiB/s-48.0MiB/s
> > > > > > >>(51.4MB/s-51.4MB/s), io=490MiB (514MB), run=10010-
> > 10010msec
> > > > > > >>
> > > > > > >>
> > > > > > >>don(a)donst201:~/fio/single/rr$ cat nvme2n1.ini |grep -v '#'
> > > > > > >>[global]
> > > > > > >>rw=randrw
> > > > > > >>rwmixread=100
> > > > > > >>numjobs=1
> > > > > > >>iodepth=1
> > > > > > >>bs=4k
> > > > > > >>direct=1
> > > > > > >>thread=1
> > > > > > >>time_based=1
> > > > > > >>ramp_time=0
> > > > > > >>runtime=10
> > > > > > >>ioengine=libaio
> > > > > > >>group_reporting=1
> > > > > > >>unified_rw_reporting=1
> > > > > > >>exitall=1
> > > > > > >>randrepeat=0
> > > > > > >>norandommap=1
> > > > > > >>[nvme2n1]
> > > > > > >>filename=/dev/nvme2n1
> > > > > > >>
> > > > > > >>
> > > > > > >>don(a)donst201:~/fio/single/rr$ sudo fio nvme2n1.ini
> > > > > > >>nvme2n1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W)
> > > > > > >>4096B-4096B,
> > > > > > >>(T) 4096B-4096B, ioengine=libaio, iodepth=1
> > > > > > >>fio-3.11
> > > > > > >>Starting 1 thread
> > > > > > >>Jobs: 1 (f=1): [r(1)][100.0%][r=43.6MiB/s][r=11.2k
> IOPS][eta
> > > > > > >>00m:00s]
> > > > > > >>nvme2n1: (groupid=0, jobs=1): err= 0: pid=2525: Mon Nov 5
> > > > > > >>05:22:57
> > > > > > >>2018
> > > > > > >> mixed: IOPS=11.2k, BW=43.6MiB/s
> > (45.7MB/s)(436MiB/10001msec)
> > > > > > >> slat (nsec): min=4848, max=208595, avg=5126.33,
> > stdev=771.27
> > > > > > >> clat (nsec): min=1845, max=3405.8k, avg=83004.99,
> > > stdev=33577.36
> > > > > > >> lat (usec): min=15, max=3411, avg=88.26, stdev=33.59
> > > > > > >> clat percentiles (usec):
> > > > > > >> | 1.00th=[ 15], 5.00th=[ 58], 10.00th=[ 59],
> 20.00th=[
> > > > > 59],
> > > > > > >> | 30.00th=[ 60], 40.00th=[ 77], 50.00th=[ 77],
> 60.00th=[
> > > > > 78],
> > > > > > >> | 70.00th=[ 111], 80.00th=[ 113], 90.00th=[ 113],
> > > > > > >>95.00th=[ 115],
> > > > > > >> | 99.00th=[ 186], 99.50th=[ 186], 99.90th=[ 190],
> > > > > > >>99.95th=[ 192],
> > > > > > >> | 99.99th=[ 375]
> > > > > > >> bw ( KiB/s): min=43816, max=45016, per=99.98%,
> > > > > > >>avg=44639.58,
> > > > > > stdev=312.88, samples=19
> > > > > > >> iops : min=10954, max=11254, avg=11159.89,
> stdev=78.22,
> > > > > > samples=19
> > > > > > >> lat (usec) : 2=0.01%, 20=2.83%, 50=0.01%, 100=64.57%,
> > > 250=32.59%
> > > > > > >> lat (usec) : 500=0.01%, 750=0.01%
> > > > > > >> lat (msec) : 2=0.01%, 4=0.01%
> > > > > > >> cpu : usr=4.02%, sys=10.33%, ctx=111630,
> majf=0, minf=12
> > > > > > >> IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%,
> > > 32=0.0%,
> > > > > > >=64=0.0%
> > > > > > >> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%,
> 32=0.0%,
> > > 64=0.0%,
> > > > > > >=64=0.0%
> > > > > > >> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%,
> 32=0.0%,
> > > > > > >> 64=0.0%,
> > > > > > >>>=64=0.0%
> > > > > > >> issued rwts: total=111631,0,0,0 short=0,0,0,0
> dropped=0,0,0,0
> > > > > > >> latency : target=0, window=0, percentile=100.00%,
> depth=1
> > > > > > >>Run status group 0 (all jobs):
> > > > > > >> MIXED: bw=43.6MiB/s (45.7MB/s), 43.6MiB/s-43.6MiB/s
> > > (45.7MB/s-
> > > > > > 45.7MB/s), io=436MiB (457MB), run=10001-10001msec Disk stats
> > > > > > (read/write):
> > > > > > >> nvme2n1: ios=110275/0, merge=0/0, ticks=0/0,
> in_queue=8596,
> > > > > > >>util=86.22%
> > > > > > >>
> > > > > > >>
> > > > > > >>Thanks,
> > > > > > >>Joe
> > > > > > >>
> > > > > > >>> -----Original Message-----
> > > > > > >>> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf
> Of
> > > > > > >>> Cao, Gang
> > > > > > >>> Sent: Sunday, November 4, 2018 8:40 PM
> > > > > > >>> To: Storage Performance Development Kit <
> spdk(a)lists.01.org
> > > > > > >>> >
> > > > > > >>> Subject: Re: [SPDK] FIO NVMe Performance Results
> > > > > > >>>
> > > > > > >>> For the SPDK plugin, from the below result, looks like
> only
> > > > > > >>> 1 job started with
> > > > > > >>> 32 queue depth:
> > > > > > >>>
> > > > > > >>> Jobs: 1 (f=0): [f(1),_(3)][100.0%][eta 00m:00s] <<<<
> > > > > > >>>
> > > > > > >>> Try to update the FIO configuration file as
> > > > > > >>>
> > > > > > >>> [job1]
> > > > > > >>> filename=trtype=PCIe traddr=0000.1c.00.0 ns=1 [job2]
> > > > > > >>> filename=trtype=PCIe
> > > > > > >>> traddr=0000.1c.00.0 ns=1 ....
> > > > > > >>>
> > > > > > >>> The kernel FIO is with 4 jobs and each job of 32 queue
> depth.
> > > > > > >>> For SPDK, you can configure as the same, or just
> configure
> > > > > > >>> two jobs and each job of 64 queue depth. The overall
> queue
> > > > > > >>> depth to the device is
> > > > > > same of 128.
> > > > > > >>>
> > > > > > >>> Thanks,
> > > > > > >>> Gang
> > > > > > >>>
> > > > > > >>> -----Original Message-----
> > > > > > >>> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf
> Of
> > > > > > >>> Gruher, Joseph R
> > > > > > >>> Sent: Monday, November 5, 2018 10:49 AM
> > > > > > >>> To: Storage Performance Development Kit <
> spdk(a)lists.01.org
> > > > > > >>> >
> > > > > > >>> Subject: [SPDK] FIO NVMe Performance Results
> > > > > > >>>
> > > > > > >>> Hi folks-
> > > > > > >>>
> > > > > > >>> I'm testing SPDK 18.10 on Ubuntu 18.04 with kernel
> 4.18.16.
> > > > > > >>> I used FIO with the kernel NVMe driver to measure the
> > > > > > >>> performance of a local (PCIe
> > > > > > >>> attached) Intel P4500 NVMe device on a 4KB random read
> > > > > > >>> workload and obtained 477K IOPS, roughly in line with the
> > > > > > >>> drive
> > > spec.
> > > > > > >>> Then I tested the same drive with the SPDK FIO plugin and
> > > > > > >>> only achieved 13K IOPS. The FIO test files and the
> results
> > > > > > >>> are pasted below. Any ideas where I'm going wrong here?
> > > > > > >>>
> > > > > > >>> Thanks!
> > > > > > >>>
> > > > > > >>>
> > > > > > >>> don(a)donst201:~/fio/single/rr$ ls
> > > > > > >>>
> > /sys/devices/pci0000:17/0000:17:02.0/0000:1c:00.0/nvme/nvme1
> > > > > > >>> / address cntlid dev device firmware_rev model
> nvme2n1
> > > > > > >>> power rescan_controller reset_controller serial state
> > > > > > >>> subsysnqn subsystem transport uevent
> > > > > > >>>
> > > > > > >>>
> > > > > > >>> don(a)donst201:~/fio/single/rr$ cat nvme2n1.ini |grep -v
> '#'
> > > > > > >>> [global]
> > > > > > >>> rw=randrw
> > > > > > >>> rwmixread=100
> > > > > > >>> numjobs=4
> > > > > > >>> iodepth=32
> > > > > > >>> bs=4k
> > > > > > >>> direct=1
> > > > > > >>> thread=1
> > > > > > >>> time_based=1
> > > > > > >>> ramp_time=0
> > > > > > >>> runtime=10
> > > > > > >>> ioengine=libaio
> > > > > > >>> group_reporting=1
> > > > > > >>> unified_rw_reporting=1
> > > > > > >>> exitall=1
> > > > > > >>> randrepeat=0
> > > > > > >>> norandommap=1
> > > > > > >>> [nvme2n1]
> > > > > > >>> filename=/dev/nvme2n1
> > > > > > >>>
> > > > > > >>>
> > > > > > >>> don(a)donst201:~/fio/single/rr$ sudo fio nvme2n1.ini
> > > > > > >>> nvme2n1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W)
> > > > > > >>> 4096B-4096B,
> > > > > > >>> (T) 4096B-4096B, ioengine=libaio, iodepth=32 ...
> > > > > > >>> fio-3.11
> > > > > > >>> Starting 4 threads
> > > > > > >>> Jobs: 4 (f=4): [r(4)][100.0%][r=1874MiB/s][r=480k
> IOPS][eta
> > > > > > >>> 00m:00s]
> > > > > > >>> nvme2n1: (groupid=0, jobs=4): err= 0: pid=2575: Mon Nov
> 5
> > > > > > >>> 02:26:25
> > > > > > 2018
> > > > > > >>> mixed: IOPS=477k, BW=1862MiB/s
> > > > (1952MB/s)(18.2GiB/10001msec)
> > > > > > >>> slat (nsec): min=1295, max=333088, avg=2216.35,
> > stdev=927.89
> > > > > > >>> clat (nsec): min=506, max=5818.8k, avg=265924.16,
> > > > stdev=223761.59
> > > > > > >>> lat (usec): min=6, max=5821, avg=268.19,
> stdev=223.76
> > > > > > >>> clat percentiles (usec):
> > > > > > >>> | 1.00th=[ 13], 5.00th=[ 74], 10.00th=[
> 87], 20.00th=[
> > > > > 115],
> > > > > > >>> | 30.00th=[ 139], 40.00th=[ 167], 50.00th=[ 204],
> > > > > > >>> 60.00th=[
> > > > > 247],
> > > > > > >>> | 70.00th=[ 306], 80.00th=[ 388], 90.00th=[ 523],
> > > > > > >>> 95.00th=[
> > > > > 660],
> > > > > > >>> | 99.00th=[ 988], 99.50th=[ 1156], 99.90th=[ 2343],
> > > > > > >>> 99.95th=[
> > > > > 3195],
> > > > > > >>> | 99.99th=[ 4424]
> > > > > > >>> bw ( KiB/s): min=448192, max=483104, per=25.00%,
> > > > > > >>> avg=476597.10, stdev=6748.78, samples=80
> > > > > > >>> iops : min=112048, max=120776, avg=119149.27,
> > > > > stdev=1687.20,
> > > > > > >>> samples=80
> > > > > > >>> lat (nsec) : 750=0.01%
> > > > > > >>> lat (usec) : 10=0.52%, 20=1.25%, 50=0.25%,
> 100=13.38%,
> > > > 250=45.13%
> > > > > > >>> lat (usec) : 500=28.23%, 750=8.08%, 1000=2.22%
> > > > > > >>> lat (msec) : 2=0.82%, 4=0.10%, 10=0.02%
> > > > > > >>> cpu : usr=14.08%, sys=36.02%, ctx=1930207,
> majf=0,
> > > > > minf=697
> > > > > > >>> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%,
> > > > 32=100.0%,
> > > > > > >>> >=64=0.0%
> > > > > > >>> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%,
> 32=0.0%,
> > > > 64=0.0%,
> > > > > > >>> >=64=0.0%
> > > > > > >>> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%,
> 32=0.1%,
> > > > > > >>> 64=0.0%,
> > > > > > >>> >=64=0.0%
> > > > > > >>> issued rwts: total=4766095,0,0,0 short=0,0,0,0
> > dropped=0,0,0,0
> > > > > > >>> latency : target=0, window=0, percentile=100.00%,
> > depth=32
> > > > > > >>> Run status group 0 (all jobs):
> > > > > > >>> MIXED: bw=1862MiB/s (1952MB/s), 1862MiB/s-1862MiB/s
> > > > (1952MB/s-
> > > > > > >>> 1952MB/s), io=18.2GiB (19.5GB), run=10001-10001msec Disk
> > > > > > >>> stats
> > > > > > >>> (read/write):
> > > > > > >>> nvme2n1: ios=4709190/0, merge=0/0, ticks=4096/0,
> > > > > > in_queue=1388752,
> > > > > > >>> util=100.00%
> > > > > > >>>
> > > > > > >>>
> > > > > > >>> don(a)donst201:~/fio/single/rr$ sudo
> > > > > > >>> /home/don/install/spdk/spdk/scripts/setup.sh
> > > > > > >>> Active mountpoints on /dev/nvme0n1, so not binding PCI
> dev
> > > > > > >>> 0000:03:00.0
> > > > > > >>> 0000:1c:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > > > > >>> 0000:1d:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > > > > >>> 0000:5e:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > > > > >>> 0000:5f:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > > > > >>> 0000:62:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > > > > >>> 0000:63:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > > > > >>> 0000:64:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > > > > >>> 0000:65:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > > > > >>> 0000:da:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > > > > >>> 0000:db:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > > > > >>> 0000:dc:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > > > > >>> 0000:dd:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > > > > >>> 0000:e0:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > > > > >>> 0000:e1:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > > > > >>> 0000:e2:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > > > > >>> 0000:e3:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > > > > >>> 0000:00:04.0 (8086 2021): ioatdma -> uio_pci_generic
> > > > > > >>> 0000:00:04.1 (8086 2021): ioatdma -> uio_pci_generic
> > > > > > >>> 0000:00:04.2 (8086 2021): ioatdma -> uio_pci_generic
> > > > > > >>> 0000:00:04.3 (8086 2021): ioatdma -> uio_pci_generic
> > > > > > >>> 0000:00:04.4 (8086 2021): ioatdma -> uio_pci_generic
> > > > > > >>> 0000:00:04.5 (8086 2021): ioatdma -> uio_pci_generic
> > > > > > >>> 0000:00:04.6 (8086 2021): ioatdma -> uio_pci_generic
> > > > > > >>> 0000:00:04.7 (8086 2021): ioatdma -> uio_pci_generic
> > > > > > >>> 0000:80:04.0 (8086 2021): ioatdma -> uio_pci_generic
> > > > > > >>> 0000:80:04.1 (8086 2021): ioatdma -> uio_pci_generic
> > > > > > >>> 0000:80:04.2 (8086 2021): ioatdma -> uio_pci_generic
> > > > > > >>> 0000:80:04.3 (8086 2021): ioatdma -> uio_pci_generic
> > > > > > >>> 0000:80:04.4 (8086 2021): ioatdma -> uio_pci_generic
> > > > > > >>> 0000:80:04.5 (8086 2021): ioatdma -> uio_pci_generic
> > > > > > >>> 0000:80:04.6 (8086 2021): ioatdma -> uio_pci_generic
> > > > > > >>> 0000:80:04.7 (8086 2021): ioatdma -> uio_pci_generic
> > > > > > >>>
> > > > > > >>>
> > > > > > >>> don(a)donst201:~/fio/single/rr$ cat 1c.ini |grep -v '#'
> > > > > > >>> [global]
> > > > > > >>> rw=randrw
> > > > > > >>> rwmixread=100
> > > > > > >>> numjobs=4
> > > > > > >>> iodepth=32
> > > > > > >>> bs=4k
> > > > > > >>> direct=1
> > > > > > >>> thread=1
> > > > > > >>> time_based=1
> > > > > > >>> ramp_time=0
> > > > > > >>> runtime=10
> > > > > > >>>
> > > > > >
> > > >
> > ioengine=/home/don/install/spdk/spdk/examples/nvme/fio_plugin/fio_pl
> > > > > > >>> ug
> > > > > > >>> in
> > > > > > >>> group_reporting=1
> > > > > > >>> unified_rw_reporting=1
> > > > > > >>> exitall=1
> > > > > > >>> randrepeat=0
> > > > > > >>> norandommap=1
> > > > > > >>> [0000.1c.00.0]
> > > > > > >>> filename=trtype=PCIe traddr=0000.1c.00.0 ns=1
> > > > > > >>>
> > > > > > >>>
> > > > > > >>> don(a)donst201:~/fio/single/rr$ sudo fio 1c.ini
> > > > > > >>> 0000.1c.00.0: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W)
> > > > > > >>> 4096B-4096B,
> > > > > > >>> (T) 4096B-4096B, ioengine=spdk, iodepth=32 ...
> > > > > > >>> fio-3.11
> > > > > > >>> Starting 4 threads
> > > > > > >>> Starting SPDK v18.10 / DPDK 18.08.0 initialization...
> > > > > > >>> [ DPDK EAL parameters: fio --no-shconf -c 0x1 -m 512
> --file-
> > > > > > >>> prefix=spdk_pid3668 ]
> > > > > > >>> EAL: Detected 36 lcore(s)
> > > > > > >>> EAL: Detected 2 NUMA nodes
> > > > > > >>> EAL: No free hugepages reported in hugepages-1048576kB
> > > > > > >>> EAL: Probing VFIO support...
> > > > > > >>> EAL: PCI device 0000:1c:00.0 on NUMA socket 0
> > > > > > >>> EAL: probe driver: 8086:a54 spdk_nvme
> > > > > > >>> Jobs: 1 (f=0): [f(1),_(3)][100.0%][eta 00m:00s]
> > > > > > >>> 0000.1c.00.0: (groupid=0, jobs=4): err= 0: pid=3709: Mon
> Nov
> > > > > > >>> 5
> > > > > > >>> 02:28:29
> > > > > > >>> 2018
> > > > > > >>> mixed: IOPS=13.2k, BW=51.6MiB/s
> > > (54.1MB/s)(517MiB/10011msec)
> > > > > > >>> slat (nsec): min=109, max=15344, avg=127.91,
> stdev=73.44
> > > > > > >>> clat (usec): min=187, max=18715, avg=9683.80,
> > stdev=650.54
> > > > > > >>> lat (usec): min=193, max=18715, avg=9683.93,
> stdev=650.53
> > > > > > >>> clat percentiles (usec):
> > > > > > >>> | 1.00th=[ 8717], 5.00th=[ 8848], 10.00th=[ 8979],
> > > > > > >>> 20.00th=[
> > > > > 9110],
> > > > > > >>> | 30.00th=[ 9241], 40.00th=[ 9372], 50.00th=[ 9503],
> > > > > > >>> 60.00th=[
> > > > > 9765],
> > > > > > >>> | 70.00th=[10028], 80.00th=[10159], 90.00th=[10552],
> > > > > > 95.00th=[10945],
> > > > > > >>> | 99.00th=[11338], 99.50th=[11338], 99.90th=[11731],
> > > > > > 99.95th=[13960],
> > > > > > >>> | 99.99th=[17695]
> > > > > > >>> bw ( KiB/s): min=13080, max=13304, per=25.00%,
> > > > > > >>> avg=13214.70, stdev=55.30, samples=80
> > > > > > >>> iops : min= 3270, max= 3326, avg=3303.65,
> stdev=13.84,
> > > > > > samples=80
> > > > > > >>> lat (usec) : 250=0.01%, 500=0.01%, 750=0.01%,
> 1000=0.01%
> > > > > > >>> lat (msec) : 2=0.01%, 4=0.02%, 10=70.85%, 20=29.11%
> > > > > > >>> cpu : usr=100.01%, sys=0.00%, ctx=23, majf=0,
> minf=0
> > > > > > >>> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%,
> > > 32=99.9%,
> > > > > > >>> >=64=0.0%
> > > > > > >>> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%,
> 32=0.0%,
> > > > 64=0.0%,
> > > > > > >>> >=64=0.0%
> > > > > > >>> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%,
> 32=0.1%,
> > > > > > >>> 64=0.0%,
> > > > > > >>> >=64=0.0%
> > > > > > >>> issued rwts: total=132300,0,0,0 short=0,0,0,0
> > dropped=0,0,0,0
> > > > > > >>> latency : target=0, window=0, percentile=100.00%,
> > depth=32
> > > > > > >>> Run status group 0 (all jobs):
> > > > > > >>> MIXED: bw=51.6MiB/s (54.1MB/s), 51.6MiB/s-51.6MiB/s
> > > > (54.1MB/s-
> > > > > > >>> 54.1MB/s), io=517MiB (542MB), run=10011-10011msec
> > > > > > >>>
> > > > > > >>> _______________________________________________
> > > > > > >>> SPDK mailing list
> > > > > > >>> SPDK(a)lists.01.org
> > > > > > >>> https://lists.01.org/mailman/listinfo/spdk
> > > > > > >>> _______________________________________________
> > > > > > >>> SPDK mailing list
> > > > > > >>> SPDK(a)lists.01.org
> > > > > > >>> https://lists.01.org/mailman/listinfo/spdk
> > > > > > >>_______________________________________________
> > > > > > >>SPDK mailing list
> > > > > > >> SPDK(a)lists.01.org
> > > > > > >> https://lists.01.org/mailman/listinfo/spdk
> > > > > > >>_______________________________________________
> > > > > > >>SPDK mailing list
> > > > > > >> SPDK(a)lists.01.org
> > > > > > >> https://lists.01.org/mailman/listinfo/spdk
> > > > > > >
> > > > > > >
> > > > > > >--
> > > > > > >Маров Алексей
> > > > > > >_______________________________________________
> > > > > > >SPDK mailing list
> > > > > > >SPDK(a)lists.01.org
> > > > > > >https://lists.01.org/mailman/listinfo/spdk
> > > > > > >_______________________________________________
> > > > > > >SPDK mailing list
> > > > > > >SPDK(a)lists.01.org
> > > > > > >https://lists.01.org/mailman/listinfo/spdk
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Маров Алексей
> > > > > > _______________________________________________
> > > > > > SPDK mailing list
> > > > > > SPDK(a)lists.01.org
> > > > > > https://lists.01.org/mailman/listinfo/spdk
> > > > > _______________________________________________
> > > > > SPDK mailing list
> > > > > SPDK(a)lists.01.org
> > > > > https://lists.01.org/mailman/listinfo/spdk
> > > > >
> > > > --
> > > >
> > > > Regards,
> > > > Andrey
> > > > _______________________________________________
> > > > SPDK mailing list
> > > > SPDK(a)lists.01.org
> > > > https://lists.01.org/mailman/listinfo/spdk
> > > > _______________________________________________
> > > > SPDK mailing list
> > > > SPDK(a)lists.01.org
> > > > https://lists.01.org/mailman/listinfo/spdk
> > _______________________________________________
> > SPDK mailing list
> > SPDK(a)lists.01.org
> > https://lists.01.org/mailman/listinfo/spdk
> > _______________________________________________
> > SPDK mailing list
> > SPDK(a)lists.01.org
> > https://lists.01.org/mailman/listinfo/spdk
> >
> >
> > _______________________________________________
> > SPDK mailing list
> > SPDK(a)lists.01.org
> > https://lists.01.org/mailman/listinfo/spdk
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
>
--
Regards,
Andrey
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [SPDK] FIO NVMe Performance Results
@ 2018-11-06 1:40 Cao, Gang
0 siblings, 0 replies; 25+ messages in thread
From: Cao, Gang @ 2018-11-06 1:40 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 50592 bytes --]
Hi Joe,
Have you tried the SPDK perf tool? We also tried locally and does not see this symptom with FIO.
Have a try on SPDK perf tool may give some ideas on the FIO related configuration. You can also pick two or four cores through the core_mask to ensure these cores are not used.
Thanks,
Gang
-----Original Message-----
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Gruher, Joseph R
Sent: Tuesday, November 6, 2018 6:19 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] FIO NVMe Performance Results
Hi Jim-
Yes, this is all local PCIe attached NVMe drives for now - the goal is to baseline the local NVMe on the system with SPDK and then move on to NVMeoF from there.
The basic test process (once SPDK has been built) is to reboot the system, run the included SPDK scripts/setup.sh script, and then run FIO with the SPDK user or bdev plugin. We're not starting app/nvmf_tgt/nvmf_tgt or anything like that so I don't think SPDK should be running anywhere outside of what is launched through FIO. All cores look idle until FIO is launched. If that doesn't sound like the right test process, let us know!
Thanks,
Joe
> -----Original Message-----
> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Harris,
> James R
> Sent: Monday, November 5, 2018 1:56 PM
> To: Storage Performance Development Kit <spdk(a)lists.01.org>
> Subject: Re: [SPDK] FIO NVMe Performance Results
>
> Hi Joe,
>
> This is all just local PCIe NVMe – no NVMe-oF? Can you make sure you
> don’t have any other SPDK processes running while you’re doing this?
> I’m concerned that fio is polling on the same CPU core as an SPDK
> process, and your performance degradation is due to timeslice sharing.
>
> -Jim
>
>
> On 11/5/18, 2:42 PM, "SPDK on behalf of Howell, Seth" <spdk-
> bounces(a)lists.01.org on behalf of seth.howell(a)intel.com> wrote:
>
> I think that the uio_pci_generic vs. vfio thing is unrelated. The
> only reason I brought it up was because it could have been indicative
> of a larger configuration difference. However, I ran my test with and
> without vfio support just to be sure and got the higher performance numbers both times.
>
> As a side note:
> It's probably a good thing to have enabled in general when running
> spdk benchmarks because that is the recommended configuration on any
> physical machine.
> You can enable it in grub by adding the following to the
> GRUB_CMDLINE_LINUX line of /etc/default.grub: intel_iommu=on
> and then running grub2-mkconfig.
>
> -----Original Message-----
> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Gruher,
> Joseph R
> Sent: Monday, November 5, 2018 1:39 PM
> To: Storage Performance Development Kit <spdk(a)lists.01.org>
> Subject: Re: [SPDK] FIO NVMe Performance Results
>
> Attempting to fall back to earlier revisions has not yielded any
> improvements for me. All tests use the same P4500 physical disk. FIO
> files are as shown before, running a 4 thread, queue depth 32 per
> thread, 4KB random read workload.
>
> SPDK 18.10, kernel 4.18.16, FIO 3.11 = 12.9k IOPS SPDK 18.10,
> kernel 4.16.14, FIO 3.11 = 11.9k IOPS SPDK 18.04, kernel 4.18.16, FIO
> 3.11 = 12.9k IOPS SPDK 18.04, kernel 4.16.14, FIO 3.11 = 13.0k IOPS
> SPDK 18.04, kernel 4.16.16, FIO 3.3 = 13.1k IOPS
>
> Kernel NVMe driver, kernel 4.16.16, FIO 3.3 = 477k IOPS Kernel
> NVMe driver, kernel 4.18.16, FIO 3.11 = 477k IOPS
>
> I am curious if the comment below about uio_pci_generic instead of
> vfio can lead us anywhere, or maybe there's some OS setting or BIOS
> setting that needs to be toggled...
>
> For the folks working inside Intel, I can also give you direct
> access to the test system, if that would help.
>
> Thanks,
> Joe
>
> > -----Original Message-----
> > From: Gruher, Joseph R
> > Sent: Monday, November 5, 2018 11:05 AM
> > To: Storage Performance Development Kit <spdk(a)lists.01.org>
> > Subject: RE: [SPDK] FIO NVMe Performance Results
> >
> > I've had good FIO NVMe results in the past we well, with SPDK 18.04 on
> > Ubuntu 18.04, with kernel 4.16.14. That was on a different, but
> > similar, hardware platform.
> >
> > I suspect something has changed now in SPDK 18.10 and/or kernel
> > 4.18.16. I can try going back to earlier versions of each to see if I
> > can recreate earlier good results and find where things went wrong.
> >
> > We're not running in a VM - this is a bare metal Xeon server with
> > Skylake CPUs.
> >
> > Thanks,
> > Joe
> >
> > > -----Original Message-----
> > > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Howell,
> > > Seth
> > > Sent: Monday, November 5, 2018 10:32 AM
> > > To: Storage Performance Development Kit <spdk(a)lists.01.org>
> > > Subject: Re: [SPDK] FIO NVMe Performance Results
> > >
> > > Hi Joseph,
> > > I attempted to verify your results on my system. I am using fedora
> > > 27 with kernel version 4.17.3 with a p3700 drive. I cloned fio from
> > > https://github.com/axboe/fio and checked out the appropriate tag. I
> > > got appx 408k iops.
> > > Typically we run spdk with the iommu enabled. I noticed that your
> > > devices are binding to uio_pci_generic instead of vfio. This
> > > shouldn't make any difference at all, but I just want to make sure
> > > you aren't running in a VM or something like that and passing
> > > through your NVMe drives and that's why you aren't using the IOMMU.
> > > That could explain a deeper configuration difference that could help
> > > us know where to look for
> > the disparity.
> > >
> > > Please see the contents of my configuration files and output below:
> > >
> > > Test.ini:
> > > [global]
> > > rw=randrw
> > > rwmixread=100
> > > numjobs=4
> > > iodepth=32
> > > bs=4k
> > > direct=1
> > > thread=1
> > > time_based=1
> > > ramp_time=0
> > > runtime=10
> > > ioengine=spdk_bdev
> > > spdk_conf=./nvme.conf
> > > group_reporting=1
> > > unified_rw_reporting=1
> > > exitall=1
> > > randrepeat=0
> > > norandommap=1
> > > [0000.81.00.0]
> > > filename=Nvme0n1
> > >
> > > nvme.conf:
> > > [Nvme]
> > > TransportId "trtype:PCIe traddr:0000.81.00.0" Nvme0
> > >
> > > Sudo
> > >
> >
> LD_PRELOAD=/home/sethhowe/Desktop/Development_Folders/spdk/exam
> > > ples/bdev/fio_plugin/fio_plugin ./fio ./test.ini >./out,txt 2>&1
> > >
> > > Out.txt:
> > > 0000.81.00.0: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-
> 4096B,
> > > (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=32 ...
> > > fio-3.11
> > > Starting 4 threads
> > > Starting SPDK v18.10 / DPDK 18.08.0 initialization...
> > > [ DPDK EAL parameters: fio --no-shconf -c 0x1
> > > --file-prefix=spdk_pid782339 ]
> > > EAL: Detected 16 lcore(s)
> > > EAL: Detected 2 NUMA nodes
> > > EAL: No free hugepages reported in hugepages-1048576kB
> > > EAL: Probing VFIO support...
> > > EAL: VFIO support initialized
> > > EAL: PCI device 0000:81:00.0 on NUMA socket 1
> > > EAL: probe driver: 8086:953 spdk_nvme
> > >
> > > 0000.81.00.0: (groupid=0, jobs=4): err= 0: pid=782359: Mon Nov 5
> > > 10:58:59
> > > 2018
> > > mixed: IOPS=410k, BW=1600MiB/s (1678MB/s)(15.6GiB/10001msec)
> > > slat (nsec): min=179, max=19985, avg=245.97, stdev=122.96
> > > clat (usec): min=8, max=1821, avg=311.82, stdev=167.14
> > > lat (usec): min=8, max=1822, avg=312.06, stdev=167.14
> > > clat percentiles (usec):
> > > | 1.00th=[ 24], 5.00th=[ 48], 10.00th=[ 78], 20.00th=[ 141],
> > > | 30.00th=[ 200], 40.00th=[ 258], 50.00th=[ 314], 60.00th=[ 371],
> > > | 70.00th=[ 429], 80.00th=[ 486], 90.00th=[ 545], 95.00th=[ 570],
> > > | 99.00th=[ 594], 99.50th=[ 594], 99.90th=[ 627], 99.95th=[ 652],
> > > | 99.99th=[ 676]
> > > bw ( KiB/s): min=407264, max=416040, per=25.01%, avg=409742.55,
> > > stdev=1337.38, samples=76
> > > iops : min=101816, max=104010, avg=102435.61, stdev=334.41,
> > > samples=76
> > > lat (usec) : 10=0.01%, 20=0.46%, 50=4.88%, 100=8.41%, 250=25.29%
> > > lat (usec) : 500=43.15%, 750=17.81%, 1000=0.01%
> > > lat (msec) : 2=0.01%
> > > cpu : usr=99.78%, sys=0.00%, ctx=26, majf=0, minf=0
> > > IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.9%,
> > > >=64=0.0%
> > > submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
> > > >=64=0.0%
> > > complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.1%, 32=0.1%,
> > > 64=0.0%,
> > > >=64=0.0%
> > > issued rwts: total=4096867,0,0,0 short=0,0,0,0 dropped=0,0,0,0
> > > latency : target=0, window=0, percentile=100.00%, depth=32
> > >
> > > Run status group 0 (all jobs):
> > > MIXED: bw=1600MiB/s (1678MB/s), 1600MiB/s-1600MiB/s
> (1678MB/s-
> > > 1678MB/s), io=15.6GiB (16.8GB), run=10001-10001msec
> > >
> > > Thanks,
> > >
> > > Seth
> > >
> > >
> > >
> > > -----Original Message-----
> > > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Andrey
> > > Kuzmin
> > > Sent: Monday, November 5, 2018 9:48 AM
> > > To: Storage Performance Development Kit <spdk(a)lists.01.org>
> > > Subject: Re: [SPDK] FIO NVMe Performance Results
> > >
> > > On Mon, Nov 5, 2018, 18:29 Gruher, Joseph R
> > > <joseph.r.gruher(a)intel.com>
> > > wrote:
> > >
> > > > My FIO version is v3.11.
> > > >
> > > > I tried the bdev FIO plugin as suggested, but performance is about
> > > > the same as the other SPDK FIO plugin, I observed only 10.4K IOPS
> > > > in a quick 4KB random read test. See configuration and
> results below.
> > > >
> > >
> > > As 12 ms median read latency for an nvme ssd is far beyond anything
> > > reasonable, why don't you just perf this run, to see where the time
> > > is being spent?
> > >
> > > Regards,
> > > Andrey
> > >
> > > >
> > > >
> > > > don(a)donst201:~/fio/single/rr$ cat nvme.conf [Nvme] TransportId
> > > > "trtype:PCIe traddr:0000.1c.00.0" Nvme0
> > > >
> > > >
> > > > don(a)donst201:~/fio/single/rr$ cat 1c-bdev.ini |grep -v '#'
> > > > [global]
> > > > rw=randrw
> > > > rwmixread=100
> > > > numjobs=4
> > > > iodepth=32
> > > > bs=4k
> > > > direct=1
> > > > thread=1
> > > > time_based=1
> > > > ramp_time=0
> > > > runtime=10
> > > > ioengine=spdk_bdev
> > > > spdk_conf=./nvme.conf
> > > > group_reporting=1
> > > > unified_rw_reporting=1
> > > > exitall=1
> > > > randrepeat=0
> > > > norandommap=1
> > > > [0000.1c.00.0]
> > > > filename=Nvme0n1
> > > >
> > > >
> > > > don(a)donst201:~/fio/single/rr$ sudo
> > > >
> > >
> >
> LD_PRELOAD=/home/don/install/spdk/spdk/examples/bdev/fio_plugin/fio_p
> > > l
> > > > ugin
> > > > fio 1c-bdev.ini
> > > > 0000.1c.00.0: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W)
> > > > 4096B-4096B,
> > > > (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=32 ...
> > > > fio-3.11
> > > > Starting 4 threads
> > > > Starting SPDK v18.10 / DPDK 18.08.0 initialization...
> > > > [ DPDK EAL parameters: fio --no-shconf -c 0x1
> > > > --file-prefix=spdk_pid4865 ]
> > > > EAL: Detected 36 lcore(s)
> > > > EAL: Detected 2 NUMA nodes
> > > > EAL: No free hugepages reported in hugepages-1048576kB
> > > > EAL: Probing VFIO support...
> > > > EAL: PCI device 0000:03:00.0 on NUMA socket 0
> > > > EAL: probe driver: 8086:f1a5 spdk_nvme
> > > > EAL: PCI device 0000:1c:00.0 on NUMA socket 0
> > > > EAL: probe driver: 8086:a54 spdk_nvme
> > > > EAL: PCI device 0000:1d:00.0 on NUMA socket 0
> > > > EAL: probe driver: 8086:a54 spdk_nvme
> > > > EAL: PCI device 0000:5e:00.0 on NUMA socket 0
> > > > EAL: probe driver: 8086:a54 spdk_nvme
> > > > EAL: PCI device 0000:5f:00.0 on NUMA socket 0
> > > > EAL: probe driver: 8086:a54 spdk_nvme
> > > > EAL: PCI device 0000:62:00.0 on NUMA socket 0
> > > > EAL: probe driver: 8086:a54 spdk_nvme
> > > > EAL: PCI device 0000:63:00.0 on NUMA socket 0
> > > > EAL: probe driver: 8086:a54 spdk_nvme
> > > > EAL: PCI device 0000:64:00.0 on NUMA socket 0
> > > > EAL: probe driver: 8086:a54 spdk_nvme
> > > > EAL: PCI device 0000:65:00.0 on NUMA socket 0
> > > > EAL: probe driver: 8086:a54 spdk_nvme
> > > > EAL: PCI device 0000:da:00.0 on NUMA socket 1
> > > > EAL: probe driver: 8086:a54 spdk_nvme
> > > > EAL: PCI device 0000:db:00.0 on NUMA socket 1
> > > > EAL: probe driver: 8086:a54 spdk_nvme
> > > > EAL: PCI device 0000:dc:00.0 on NUMA socket 1
> > > > EAL: probe driver: 8086:a54 spdk_nvme
> > > > EAL: PCI device 0000:dd:00.0 on NUMA socket 1
> > > > EAL: probe driver: 8086:a54 spdk_nvme
> > > > EAL: PCI device 0000:e0:00.0 on NUMA socket 1
> > > > EAL: probe driver: 8086:a54 spdk_nvme
> > > > EAL: PCI device 0000:e1:00.0 on NUMA socket 1
> > > > EAL: probe driver: 8086:a54 spdk_nvme
> > > > EAL: PCI device 0000:e2:00.0 on NUMA socket 1
> > > > EAL: probe driver: 8086:a54 spdk_nvme
> > > > EAL: PCI device 0000:e3:00.0 on NUMA socket 1
> > > > EAL: probe driver: 8086:a54 spdk_nvme
> > > > Jobs: 4 (f=4): [r(4)][100.0%][r=45.7MiB/s][r=11.7k IOPS][eta
> > > > 00m:00s]
> > > > 0000.1c.00.0: (groupid=0, jobs=4): err= 0: pid=4906: Mon Nov 5
> > > > 15:23:47
> > > > 2018
> > > > mixed: IOPS=10.4k, BW=40.5MiB/s (42.5MB/s)(406MiB/10012msec)
> > > > slat (nsec): min=201, max=15831, avg=238.11, stdev=148.78
> > > > clat (usec): min=124, max=31244, avg=12330.69, stdev=1697.98
> > > > lat (usec): min=130, max=31244, avg=12330.93, stdev=1697.98
> > > > clat percentiles (usec):
> > > > | 1.00th=[ 9634], 5.00th=[10159], 10.00th=[10552],
> 20.00th=[10945],
> > > > | 30.00th=[11338], 40.00th=[11731], 50.00th=[12125],
> > > 60.00th=[12518],
> > > > | 70.00th=[13042], 80.00th=[13566], 90.00th=[14353],
> > > 95.00th=[15008],
> > > > | 99.00th=[16712], 99.50th=[18744], 99.90th=[27395],
> > > 99.95th=[29754],
> > > > | 99.99th=[30278]
> > > > bw ( KiB/s): min= 8784, max=11952, per=24.99%, avg=10375.53,
> > > > stdev=929.68, samples=80
> > > > iops : min= 2196, max= 2988, avg=2593.85, stdev=232.41,
> > > > samples=80
> > > > lat (usec) : 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
> > > > lat (msec) : 2=0.01%, 4=0.02%, 10=2.71%, 20=96.91%, 50=0.33%
> > > > cpu : usr=99.99%, sys=0.01%, ctx=28, majf=0, minf=0
> > > > IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.9%,
> > > > >=64=0.0%
> > > > submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%,
> 64=0.0%,
> > > > >=64=0.0%
> > > > complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%,
> > > > 64=0.0%,
> > > > >=64=0.0%
> > > > issued rwts: total=103907,0,0,0 short=0,0,0,0 dropped=0,0,0,0
> > > > latency : target=0, window=0, percentile=100.00%, depth=32
> > > > Run status group 0 (all jobs):
> > > > MIXED: bw=40.5MiB/s (42.5MB/s), 40.5MiB/s-40.5MiB/s
> > > > (42.5MB/s-42.5MB/s), io=406MiB (426MB), run=10012-10012msec
> > > >
> > > >
> > > > Thanks,
> > > > Joe
> > > >
> > > > > -----Original Message-----
> > > > > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of ?????
> > > ???????
> > > > > Sent: Monday, November 5, 2018 12:45 AM
> > > > > To: Storage Performance Development Kit <spdk(a)lists.01.org>
> > > > > Subject: Re: [SPDK] FIO NVMe Performance Results
> > > > >
> > > > > Ok, Do you see some performance issues using general bdev fio
> > > > > plugin
> > > > with
> > > > > you configuration ?
> > > > >
> > > > > Best Regards
> > > > > Alex
> > > > >
> > > > >
> > > > > >Понедельник, 5 ноября 2018, 11:39 +03:00 от "Cao, Gang"
> > > > > <gang.cao(a)intel.com>:
> > > > > >
> > > > > >Yes. This is for the bdev version of SPDK FIO plugin and you
> > > > > >can assign
> > > > the
> > > > > conf file as the parameter in the FIO configuration.
> > > > > >
> > > > > >Previously, it's another SPDK FIO plugin for the user space
> > > > > >NVMe driver
> > > > > only.
> > > > > >
> > > > > >These are two different FIO plugins.
> > > > > >
> > > > > >Thanks,
> > > > > >Gang
> > > > > >
> > > > > >-----Original Message-----
> > > > > >From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of
> > > > > >Маров Алексей
> > > > > >Sent: Monday, November 5, 2018 4:32 PM
> > > > > >To: Storage Performance Development Kit < spdk(a)lists.01.org >
> > > > > >Subject: Re: [SPDK] FIO NVMe Performance Results
> > > > > >
> > > > > >Sorry for incomplete message I have sent before
> > > > > >
> > > > > >Hello Gang.
> > > > > >
> > > > > >I just curious, why would you run fio with SPDK using this conf
> > > > > >file,
> > > > with is a
> > > > > bit different from what is suggested in here
> > > https://github.com/spdk/spdk/tree/master/examples/bdev/fio_plugin
> > > > > .
> > > > > >
> > > > > >I mean may be you should create conf file for spdk like this one :
> > > > > >
> > > > > >nvme.conf :
> > > > > >[Nvme]
> > > > > > TransportId "trtype:PCIe traddr:0000.1c.00.0" Nvme0
> > > > > >
> > > > > >and provide it in fio config.
> > > > > >
> > > > > >fio.ini :
> > > > > >[global]
> > > > > >ioengine=spdk_bdev
> > > > > >spdk_conf=....../nvme.conf
> > > > > >rw=randrw
> > > > > >rwmixread=100
> > > > > >numjobs=4
> > > > > >iodepth=32
> > > > > >bs=4k
> > > > > >direct=1
> > > > > >thread=1
> > > > > >time_based=1
> > > > > >ramp_time=0
> > > > > >runtime=10
> > > > > >group_reporting=1
> > > > > >unified_rw_reporting=1
> > > > > >exitall=1
> > > > > >randrepeat=0
> > > > > >norandommap=1
> > > > > >
> > > > > >[0000:06:00.0]
> > > > > >filename=Nvme0n1
> > > > > >
> > > > > >and run like this
> > > > > >
> > > > > ># LD_PRELOAD=examples/bdev/fio_plugin/fio_plugin fio
> > > > > >examples/bdev/fio_plugin/fio.ini
> > > > > >
> > > > > >Just to note. The recommended version of fio for SPDK is 3.3
> > > > > >
> > > > > >Best Regards
> > > > > >Alex
> > > > > >
> > > > > >
> > > > > >>Понедельник, 5 ноября 2018, 9:53 +03:00 от "Cao, Gang" <
> > > > > gang.cao(a)intel.com >:
> > > > > >>
> > > > > >>Thanks. This is interesting especially for your finding on
> > > > > >>queue depth
> > > > 1
> > > > > case. We are trying to reproduce this issue locally same as your
> > > > > configuration.
> > > > > >>
> > > > > >>Thanks,
> > > > > >>Gang
> > > > > >>
> > > > > >>-----Original Message-----
> > > > > >>From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of
> > > > > >>Gruher, Joseph R
> > > > > >>Sent: Monday, November 5, 2018 1:32 PM
> > > > > >>To: Storage Performance Development Kit < spdk(a)lists.01.org >
> > > > > >>Subject: Re: [SPDK] FIO NVMe Performance Results
> > > > > >>
> > > > > >>Interesting catch. It seems like FIO is actually not
> > > > > >>accurately
> > > > reporting the
> > > > > number of jobs in the report out for the SPDK case. If I
> > > > > disable the group_reporting parameter I do get a printout for
> > > > > four separate jobs in
> > > > the
> > > > > SPDK case, so I think it is really running four jobs there, and
> > > > > just
> > > > summarizing
> > > > > incorrectly. I also see four separate CPU cores go to 100% when
> > > > > I run
> > > > that
> > > > > FIO test case, indicating four jobs are active.
> > > > > >>
> > > > > >>Also, I don't think would explain the difference in performance.
> > > > Using an
> > > > > aggregate queue depth of 32 versus 128 shouldn't lead to a drop
> > > > > in performance from 377K to 13K.
> > > > > >>
> > > > > >>I retested SPDK with multiple separate job descriptions as you
> > > > suggested
> > > > > anyway, using four separate jobs instead of relying on the
> > > > > numjobs=4 parameter, and the results are below. In that SPDK
> > > > > test with four
> > > > separate
> > > > > FIO job statements, each with queue depth 32, I still only
> > > > > achieved 12.5K IOPS.
> > > > > >>
> > > > > >>I then re-ran the kernel NVMe driver test case with only a
> > > > > >>single job
> > > > for
> > > > > comparison, and even with only 1 job and queue depth 32 it still
> > > > > achieves 254K IOPS.
> > > > > >>
> > > > > >>We might be on the right track with a problem around queue
> > > > > >>depth
> > > > > however. If I run the kernel NVMe driver case with just 1 job
> > > > > and queue depth of only 1, I then get 11.2K IOPS, which is
> > > > > around the performance
> > > > I get
> > > > > in the SPDK test case. Perhaps the SPDK case is not actually
> > > > > queuing
> > > > IOs on
> > > > > the device for some reason?
> > > > > >>
> > > > > >>
> > > > > >>don(a)donst201:~/fio/single/rr$ cat nvme2n1.ini |grep -v '#'
> > > > > >>[global]
> > > > > >>rw=randrw
> > > > > >>rwmixread=100
> > > > > >>numjobs=1
> > > > > >>iodepth=32
> > > > > >>bs=4k
> > > > > >>direct=1
> > > > > >>thread=1
> > > > > >>time_based=1
> > > > > >>ramp_time=0
> > > > > >>runtime=10
> > > > > >>ioengine=libaio
> > > > > >>group_reporting=1
> > > > > >>unified_rw_reporting=1
> > > > > >>exitall=1
> > > > > >>randrepeat=0
> > > > > >>norandommap=1
> > > > > >>[nvme2n1]
> > > > > >>filename=/dev/nvme2n1
> > > > > >>
> > > > > >>
> > > > > >>don(a)donst201:~/fio/single/rr$ sudo fio nvme2n1.ini
> > > > > >>nvme2n1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W)
> > > > > >>4096B-4096B,
> > > > > >>(T) 4096B-4096B, ioengine=libaio, iodepth=32
> > > > > >>fio-3.11
> > > > > >>Starting 1 thread
> > > > > >>Jobs: 1 (f=1): [r(1)][100.0%][r=1025MiB/s][r=262k IOPS][eta
> > > > > >>00m:00s]
> > > > > >>nvme2n1: (groupid=0, jobs=1): err= 0: pid=2422: Mon Nov 5
> > > > > >>05:19:13
> > > > > >>2018
> > > > > >> mixed: IOPS=254k, BW=992MiB/s
> > (1040MB/s)(9922MiB/10001msec)
> > > > > >> slat (nsec): min=1185, max=319052, avg=1611.64,
> stdev=808.40
> > > > > >> clat (usec): min=2, max=4743, avg=124.02, stdev=121.00
> > > > > >> lat (usec): min=6, max=4744, avg=125.68, stdev=121.02
> > > > > >> clat percentiles (usec):
> > > > > >> | 1.00th=[ 9], 5.00th=[ 56], 10.00th=[ 61], 20.00th=[
> > > > 73],
> > > > > >> | 30.00th=[ 80], 40.00th=[ 91], 50.00th=[ 110], 60.00th=[
> > > > 118],
> > > > > >> | 70.00th=[ 130], 80.00th=[ 157], 90.00th=[ 202],
> > > > > >>95.00th=[ 253],
> > > > > >> | 99.00th=[ 408], 99.50th=[ 498], 99.90th=[ 1795],
> > > > > >>99.95th=[ 3064],
> > > > > >> | 99.99th=[ 3785]
> > > > > >> bw ( KiB/s): min=612240, max=1078792, per=100.00%,
> > > > > >>avg=1015996.40,
> > > > > stdev=106032.96, samples=20
> > > > > >> iops : min=153060, max=269698, avg=253998.90,
> > > stdev=26508.40,
> > > > > samples=20
> > > > > >> lat (usec) : 4=0.01%, 10=1.47%, 20=1.19%, 50=0.17%,
> 100=42.03%
> > > > > >> lat (usec) : 250=49.95%, 500=4.70%, 750=0.28%, 1000=0.06%
> > > > > >> lat (msec) : 2=0.06%, 4=0.09%, 10=0.01%
> > > > > >> cpu : usr=21.45%, sys=51.11%, ctx=530348, majf=0,
> minf=139
> > > > > >> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%,
> > 32=100.0%,
> > > > > >=64=0.0%
> > > > > >> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%,
> > 64=0.0%,
> > > > > >=64=0.0%
> > > > > >> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%,
> > > > > >> 64=0.0%,
> > > > > >>>=64=0.0%
> > > > > >> issued rwts: total=2540023,0,0,0 short=0,0,0,0
> dropped=0,0,0,0
> > > > > >> latency : target=0, window=0, percentile=100.00%, depth=32
> > > > > >>Run status group 0 (all jobs):
> > > > > >> MIXED: bw=992MiB/s (1040MB/s), 992MiB/s-992MiB/s
> (1040MB/s-
> > > > > 1040MB/s), io=9922MiB (10.4GB), run=10001-10001msec Disk stats
> > > > > (read/write):
> > > > > >> nvme2n1: ios=2514356/0, merge=0/0, ticks=400/0,
> > > in_queue=297612,
> > > > > >>util=99.06%
> > > > > >>
> > > > > >>
> > > > > >>don(a)donst201:~/fio/single/rr$ cat 1c.ini |grep -v '#'
> > > > > >>[global]
> > > > > >>rw=randrw
> > > > > >>rwmixread=100
> > > > > >>numjobs=1
> > > > > >>iodepth=32
> > > > > >>bs=4k
> > > > > >>direct=1
> > > > > >>thread=1
> > > > > >>time_based=1
> > > > > >>ramp_time=0
> > > > > >>runtime=10
> > > > >
> > >
> >>ioengine=/home/don/install/spdk/spdk/examples/nvme/fio_plugin/fio_
> > > > > >>pl
> > > > > ug
> > > > > >>in
> > > > > >>group_reporting=1
> > > > > >>unified_rw_reporting=1
> > > > > >>exitall=1
> > > > > >>randrepeat=0
> > > > > >>norandommap=1
> > > > > >>[0000.1c.00.0-1]
> > > > > >>filename=trtype=PCIe traddr=0000.1c.00.0 ns=1 [0000.1c.00.0-2]
> > > > > >>filename=trtype=PCIe traddr=0000.1c.00.0 ns=1 [0000.1c.00.0-3]
> > > > > >>filename=trtype=PCIe traddr=0000.1c.00.0 ns=1 [0000.1c.00.0-4]
> > > > > >>filename=trtype=PCIe traddr=0000.1c.00.0 ns=1
> > > > > >>
> > > > > >>
> > > > > >>don(a)donst201:~/fio/single/rr$ sudo fio 1c.ini
> > > > > >>0000.1c.00.0-1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W)
> > > > > >>4096B-4096B,
> > > > > >>(T) 4096B-4096B, ioengine=spdk, iodepth=32
> > > > > >>0000.1c.00.0-2: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W)
> > > > > >>4096B-4096B,
> > > > > >>(T) 4096B-4096B, ioengine=spdk, iodepth=32
> > > > > >>0000.1c.00.0-3: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W)
> > > > > >>4096B-4096B,
> > > > > >>(T) 4096B-4096B, ioengine=spdk, iodepth=32
> > > > > >>0000.1c.00.0-4: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W)
> > > > > >>4096B-4096B,
> > > > > >>(T) 4096B-4096B, ioengine=spdk, iodepth=32
> > > > > >>fio-3.11
> > > > > >>Starting 4 threads
> > > > > >>Starting SPDK v18.10 / DPDK 18.08.0 initialization...
> > > > > >>[ DPDK EAL parameters: fio --no-shconf -c 0x1 -m 512
> > > > > >>--file-prefix=spdk_pid4270 ]
> > > > > >>EAL: Detected 36 lcore(s)
> > > > > >>EAL: Detected 2 NUMA nodes
> > > > > >>EAL: No free hugepages reported in hugepages-1048576kB
> > > > > >>EAL: Probing VFIO support...
> > > > > >>EAL: PCI device 0000:1c:00.0 on NUMA socket 0
> > > > > >>EAL: probe driver: 8086:a54 spdk_nvme
> > > > > >>Jobs: 1 (f=0): [_(3),f(1)][100.0%][eta 00m:00s]
> > > > > >>0000.1c.00.0-1: (groupid=0, jobs=4): err= 0: pid=4310: Mon Nov
> > > > > >>5
> > > > > >>05:16:03 2018
> > > > > >> mixed: IOPS=12.5k, BW=48.0MiB/s
> (51.4MB/s)(490MiB/10010msec)
> > > > > >> slat (nsec): min=98, max=15092, avg=120.02, stdev=90.07
> > > > > >> clat (usec): min=55, max=19757, avg=10207.22, stdev=884.06
> > > > > >> lat (usec): min=60, max=19757, avg=10207.34, stdev=884.06
> > > > > >> clat percentiles (usec):
> > > > > >> | 1.00th=[ 8979], 5.00th=[ 9110], 10.00th=[ 9372],
> > > > > >>20.00th=[ 9503],
> > > > > >> | 30.00th=[ 9765], 40.00th=[ 9896], 50.00th=[10159],
> > > > > >>60.00th=[10290],
> > > > > >> | 70.00th=[10552], 80.00th=[10814], 90.00th=[11207],
> > > > > >>95.00th=[11600],
> > > > > >> | 99.00th=[12911], 99.50th=[14353], 99.90th=[17695],
> > > > > >>99.95th=[18744],
> > > > > >> | 99.99th=[19006]
> > > > > >> bw ( KiB/s): min=12008, max=12976, per=25.00%,
> > > > > >>avg=12535.70,
> > > > > stdev=306.81, samples=80
> > > > > >> iops : min= 3002, max= 3244, avg=3133.85, stdev=76.77,
> > > > samples=80
> > > > > >> lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%,
> > > 1000=0.01%
> > > > > >> lat (msec) : 2=0.01%, 4=0.02%, 10=45.13%, 20=54.83%
> > > > > >> cpu : usr=100.01%, sys=0.00%, ctx=27, majf=0, minf=0
> > > > > >> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%,
> 32=99.9%,
> > > > > >=64=0.0%
> > > > > >> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%,
> > 64=0.0%,
> > > > > >=64=0.0%
> > > > > >> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%,
> > > > > >> 64=0.0%,
> > > > > >>>=64=0.0%
> > > > > >> issued rwts: total=125506,0,0,0 short=0,0,0,0 dropped=0,0,0,0
> > > > > >> latency : target=0, window=0, percentile=100.00%, depth=32
> > > > > >>Run status group 0 (all jobs):
> > > > > >> MIXED: bw=48.0MiB/s (51.4MB/s), 48.0MiB/s-48.0MiB/s
> > > > > >>(51.4MB/s-51.4MB/s), io=490MiB (514MB), run=10010-
> 10010msec
> > > > > >>
> > > > > >>
> > > > > >>don(a)donst201:~/fio/single/rr$ cat nvme2n1.ini |grep -v '#'
> > > > > >>[global]
> > > > > >>rw=randrw
> > > > > >>rwmixread=100
> > > > > >>numjobs=1
> > > > > >>iodepth=1
> > > > > >>bs=4k
> > > > > >>direct=1
> > > > > >>thread=1
> > > > > >>time_based=1
> > > > > >>ramp_time=0
> > > > > >>runtime=10
> > > > > >>ioengine=libaio
> > > > > >>group_reporting=1
> > > > > >>unified_rw_reporting=1
> > > > > >>exitall=1
> > > > > >>randrepeat=0
> > > > > >>norandommap=1
> > > > > >>[nvme2n1]
> > > > > >>filename=/dev/nvme2n1
> > > > > >>
> > > > > >>
> > > > > >>don(a)donst201:~/fio/single/rr$ sudo fio nvme2n1.ini
> > > > > >>nvme2n1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W)
> > > > > >>4096B-4096B,
> > > > > >>(T) 4096B-4096B, ioengine=libaio, iodepth=1
> > > > > >>fio-3.11
> > > > > >>Starting 1 thread
> > > > > >>Jobs: 1 (f=1): [r(1)][100.0%][r=43.6MiB/s][r=11.2k IOPS][eta
> > > > > >>00m:00s]
> > > > > >>nvme2n1: (groupid=0, jobs=1): err= 0: pid=2525: Mon Nov 5
> > > > > >>05:22:57
> > > > > >>2018
> > > > > >> mixed: IOPS=11.2k, BW=43.6MiB/s
> (45.7MB/s)(436MiB/10001msec)
> > > > > >> slat (nsec): min=4848, max=208595, avg=5126.33,
> stdev=771.27
> > > > > >> clat (nsec): min=1845, max=3405.8k, avg=83004.99,
> > stdev=33577.36
> > > > > >> lat (usec): min=15, max=3411, avg=88.26, stdev=33.59
> > > > > >> clat percentiles (usec):
> > > > > >> | 1.00th=[ 15], 5.00th=[ 58], 10.00th=[ 59], 20.00th=[
> > > > 59],
> > > > > >> | 30.00th=[ 60], 40.00th=[ 77], 50.00th=[ 77], 60.00th=[
> > > > 78],
> > > > > >> | 70.00th=[ 111], 80.00th=[ 113], 90.00th=[ 113],
> > > > > >>95.00th=[ 115],
> > > > > >> | 99.00th=[ 186], 99.50th=[ 186], 99.90th=[ 190],
> > > > > >>99.95th=[ 192],
> > > > > >> | 99.99th=[ 375]
> > > > > >> bw ( KiB/s): min=43816, max=45016, per=99.98%,
> > > > > >>avg=44639.58,
> > > > > stdev=312.88, samples=19
> > > > > >> iops : min=10954, max=11254, avg=11159.89, stdev=78.22,
> > > > > samples=19
> > > > > >> lat (usec) : 2=0.01%, 20=2.83%, 50=0.01%, 100=64.57%,
> > 250=32.59%
> > > > > >> lat (usec) : 500=0.01%, 750=0.01%
> > > > > >> lat (msec) : 2=0.01%, 4=0.01%
> > > > > >> cpu : usr=4.02%, sys=10.33%, ctx=111630, majf=0, minf=12
> > > > > >> IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%,
> > 32=0.0%,
> > > > > >=64=0.0%
> > > > > >> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%,
> > 64=0.0%,
> > > > > >=64=0.0%
> > > > > >> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%,
> > > > > >> 64=0.0%,
> > > > > >>>=64=0.0%
> > > > > >> issued rwts: total=111631,0,0,0 short=0,0,0,0 dropped=0,0,0,0
> > > > > >> latency : target=0, window=0, percentile=100.00%, depth=1
> > > > > >>Run status group 0 (all jobs):
> > > > > >> MIXED: bw=43.6MiB/s (45.7MB/s), 43.6MiB/s-43.6MiB/s
> > (45.7MB/s-
> > > > > 45.7MB/s), io=436MiB (457MB), run=10001-10001msec Disk stats
> > > > > (read/write):
> > > > > >> nvme2n1: ios=110275/0, merge=0/0, ticks=0/0, in_queue=8596,
> > > > > >>util=86.22%
> > > > > >>
> > > > > >>
> > > > > >>Thanks,
> > > > > >>Joe
> > > > > >>
> > > > > >>> -----Original Message-----
> > > > > >>> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of
> > > > > >>> Cao, Gang
> > > > > >>> Sent: Sunday, November 4, 2018 8:40 PM
> > > > > >>> To: Storage Performance Development Kit < spdk(a)lists.01.org
> > > > > >>> >
> > > > > >>> Subject: Re: [SPDK] FIO NVMe Performance Results
> > > > > >>>
> > > > > >>> For the SPDK plugin, from the below result, looks like only
> > > > > >>> 1 job started with
> > > > > >>> 32 queue depth:
> > > > > >>>
> > > > > >>> Jobs: 1 (f=0): [f(1),_(3)][100.0%][eta 00m:00s] <<<<
> > > > > >>>
> > > > > >>> Try to update the FIO configuration file as
> > > > > >>>
> > > > > >>> [job1]
> > > > > >>> filename=trtype=PCIe traddr=0000.1c.00.0 ns=1 [job2]
> > > > > >>> filename=trtype=PCIe
> > > > > >>> traddr=0000.1c.00.0 ns=1 ....
> > > > > >>>
> > > > > >>> The kernel FIO is with 4 jobs and each job of 32 queue depth.
> > > > > >>> For SPDK, you can configure as the same, or just configure
> > > > > >>> two jobs and each job of 64 queue depth. The overall queue
> > > > > >>> depth to the device is
> > > > > same of 128.
> > > > > >>>
> > > > > >>> Thanks,
> > > > > >>> Gang
> > > > > >>>
> > > > > >>> -----Original Message-----
> > > > > >>> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of
> > > > > >>> Gruher, Joseph R
> > > > > >>> Sent: Monday, November 5, 2018 10:49 AM
> > > > > >>> To: Storage Performance Development Kit < spdk(a)lists.01.org
> > > > > >>> >
> > > > > >>> Subject: [SPDK] FIO NVMe Performance Results
> > > > > >>>
> > > > > >>> Hi folks-
> > > > > >>>
> > > > > >>> I'm testing SPDK 18.10 on Ubuntu 18.04 with kernel 4.18.16.
> > > > > >>> I used FIO with the kernel NVMe driver to measure the
> > > > > >>> performance of a local (PCIe
> > > > > >>> attached) Intel P4500 NVMe device on a 4KB random read
> > > > > >>> workload and obtained 477K IOPS, roughly in line with the
> > > > > >>> drive
> > spec.
> > > > > >>> Then I tested the same drive with the SPDK FIO plugin and
> > > > > >>> only achieved 13K IOPS. The FIO test files and the results
> > > > > >>> are pasted below. Any ideas where I'm going wrong here?
> > > > > >>>
> > > > > >>> Thanks!
> > > > > >>>
> > > > > >>>
> > > > > >>> don(a)donst201:~/fio/single/rr$ ls
> > > > > >>>
> /sys/devices/pci0000:17/0000:17:02.0/0000:1c:00.0/nvme/nvme1
> > > > > >>> / address cntlid dev device firmware_rev model nvme2n1
> > > > > >>> power rescan_controller reset_controller serial state
> > > > > >>> subsysnqn subsystem transport uevent
> > > > > >>>
> > > > > >>>
> > > > > >>> don(a)donst201:~/fio/single/rr$ cat nvme2n1.ini |grep -v '#'
> > > > > >>> [global]
> > > > > >>> rw=randrw
> > > > > >>> rwmixread=100
> > > > > >>> numjobs=4
> > > > > >>> iodepth=32
> > > > > >>> bs=4k
> > > > > >>> direct=1
> > > > > >>> thread=1
> > > > > >>> time_based=1
> > > > > >>> ramp_time=0
> > > > > >>> runtime=10
> > > > > >>> ioengine=libaio
> > > > > >>> group_reporting=1
> > > > > >>> unified_rw_reporting=1
> > > > > >>> exitall=1
> > > > > >>> randrepeat=0
> > > > > >>> norandommap=1
> > > > > >>> [nvme2n1]
> > > > > >>> filename=/dev/nvme2n1
> > > > > >>>
> > > > > >>>
> > > > > >>> don(a)donst201:~/fio/single/rr$ sudo fio nvme2n1.ini
> > > > > >>> nvme2n1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W)
> > > > > >>> 4096B-4096B,
> > > > > >>> (T) 4096B-4096B, ioengine=libaio, iodepth=32 ...
> > > > > >>> fio-3.11
> > > > > >>> Starting 4 threads
> > > > > >>> Jobs: 4 (f=4): [r(4)][100.0%][r=1874MiB/s][r=480k IOPS][eta
> > > > > >>> 00m:00s]
> > > > > >>> nvme2n1: (groupid=0, jobs=4): err= 0: pid=2575: Mon Nov 5
> > > > > >>> 02:26:25
> > > > > 2018
> > > > > >>> mixed: IOPS=477k, BW=1862MiB/s
> > > (1952MB/s)(18.2GiB/10001msec)
> > > > > >>> slat (nsec): min=1295, max=333088, avg=2216.35,
> stdev=927.89
> > > > > >>> clat (nsec): min=506, max=5818.8k, avg=265924.16,
> > > stdev=223761.59
> > > > > >>> lat (usec): min=6, max=5821, avg=268.19, stdev=223.76
> > > > > >>> clat percentiles (usec):
> > > > > >>> | 1.00th=[ 13], 5.00th=[ 74], 10.00th=[ 87], 20.00th=[
> > > > 115],
> > > > > >>> | 30.00th=[ 139], 40.00th=[ 167], 50.00th=[ 204],
> > > > > >>> 60.00th=[
> > > > 247],
> > > > > >>> | 70.00th=[ 306], 80.00th=[ 388], 90.00th=[ 523],
> > > > > >>> 95.00th=[
> > > > 660],
> > > > > >>> | 99.00th=[ 988], 99.50th=[ 1156], 99.90th=[ 2343],
> > > > > >>> 99.95th=[
> > > > 3195],
> > > > > >>> | 99.99th=[ 4424]
> > > > > >>> bw ( KiB/s): min=448192, max=483104, per=25.00%,
> > > > > >>> avg=476597.10, stdev=6748.78, samples=80
> > > > > >>> iops : min=112048, max=120776, avg=119149.27,
> > > > stdev=1687.20,
> > > > > >>> samples=80
> > > > > >>> lat (nsec) : 750=0.01%
> > > > > >>> lat (usec) : 10=0.52%, 20=1.25%, 50=0.25%, 100=13.38%,
> > > 250=45.13%
> > > > > >>> lat (usec) : 500=28.23%, 750=8.08%, 1000=2.22%
> > > > > >>> lat (msec) : 2=0.82%, 4=0.10%, 10=0.02%
> > > > > >>> cpu : usr=14.08%, sys=36.02%, ctx=1930207, majf=0,
> > > > minf=697
> > > > > >>> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%,
> > > 32=100.0%,
> > > > > >>> >=64=0.0%
> > > > > >>> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%,
> > > 64=0.0%,
> > > > > >>> >=64=0.0%
> > > > > >>> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%,
> > > > > >>> 64=0.0%,
> > > > > >>> >=64=0.0%
> > > > > >>> issued rwts: total=4766095,0,0,0 short=0,0,0,0
> dropped=0,0,0,0
> > > > > >>> latency : target=0, window=0, percentile=100.00%,
> depth=32
> > > > > >>> Run status group 0 (all jobs):
> > > > > >>> MIXED: bw=1862MiB/s (1952MB/s), 1862MiB/s-1862MiB/s
> > > (1952MB/s-
> > > > > >>> 1952MB/s), io=18.2GiB (19.5GB), run=10001-10001msec Disk
> > > > > >>> stats
> > > > > >>> (read/write):
> > > > > >>> nvme2n1: ios=4709190/0, merge=0/0, ticks=4096/0,
> > > > > in_queue=1388752,
> > > > > >>> util=100.00%
> > > > > >>>
> > > > > >>>
> > > > > >>> don(a)donst201:~/fio/single/rr$ sudo
> > > > > >>> /home/don/install/spdk/spdk/scripts/setup.sh
> > > > > >>> Active mountpoints on /dev/nvme0n1, so not binding PCI dev
> > > > > >>> 0000:03:00.0
> > > > > >>> 0000:1c:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > > > >>> 0000:1d:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > > > >>> 0000:5e:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > > > >>> 0000:5f:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > > > >>> 0000:62:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > > > >>> 0000:63:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > > > >>> 0000:64:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > > > >>> 0000:65:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > > > >>> 0000:da:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > > > >>> 0000:db:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > > > >>> 0000:dc:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > > > >>> 0000:dd:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > > > >>> 0000:e0:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > > > >>> 0000:e1:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > > > >>> 0000:e2:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > > > >>> 0000:e3:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > > > >>> 0000:00:04.0 (8086 2021): ioatdma -> uio_pci_generic
> > > > > >>> 0000:00:04.1 (8086 2021): ioatdma -> uio_pci_generic
> > > > > >>> 0000:00:04.2 (8086 2021): ioatdma -> uio_pci_generic
> > > > > >>> 0000:00:04.3 (8086 2021): ioatdma -> uio_pci_generic
> > > > > >>> 0000:00:04.4 (8086 2021): ioatdma -> uio_pci_generic
> > > > > >>> 0000:00:04.5 (8086 2021): ioatdma -> uio_pci_generic
> > > > > >>> 0000:00:04.6 (8086 2021): ioatdma -> uio_pci_generic
> > > > > >>> 0000:00:04.7 (8086 2021): ioatdma -> uio_pci_generic
> > > > > >>> 0000:80:04.0 (8086 2021): ioatdma -> uio_pci_generic
> > > > > >>> 0000:80:04.1 (8086 2021): ioatdma -> uio_pci_generic
> > > > > >>> 0000:80:04.2 (8086 2021): ioatdma -> uio_pci_generic
> > > > > >>> 0000:80:04.3 (8086 2021): ioatdma -> uio_pci_generic
> > > > > >>> 0000:80:04.4 (8086 2021): ioatdma -> uio_pci_generic
> > > > > >>> 0000:80:04.5 (8086 2021): ioatdma -> uio_pci_generic
> > > > > >>> 0000:80:04.6 (8086 2021): ioatdma -> uio_pci_generic
> > > > > >>> 0000:80:04.7 (8086 2021): ioatdma -> uio_pci_generic
> > > > > >>>
> > > > > >>>
> > > > > >>> don(a)donst201:~/fio/single/rr$ cat 1c.ini |grep -v '#'
> > > > > >>> [global]
> > > > > >>> rw=randrw
> > > > > >>> rwmixread=100
> > > > > >>> numjobs=4
> > > > > >>> iodepth=32
> > > > > >>> bs=4k
> > > > > >>> direct=1
> > > > > >>> thread=1
> > > > > >>> time_based=1
> > > > > >>> ramp_time=0
> > > > > >>> runtime=10
> > > > > >>>
> > > > >
> > >
> ioengine=/home/don/install/spdk/spdk/examples/nvme/fio_plugin/fio_pl
> > > > > >>> ug
> > > > > >>> in
> > > > > >>> group_reporting=1
> > > > > >>> unified_rw_reporting=1
> > > > > >>> exitall=1
> > > > > >>> randrepeat=0
> > > > > >>> norandommap=1
> > > > > >>> [0000.1c.00.0]
> > > > > >>> filename=trtype=PCIe traddr=0000.1c.00.0 ns=1
> > > > > >>>
> > > > > >>>
> > > > > >>> don(a)donst201:~/fio/single/rr$ sudo fio 1c.ini
> > > > > >>> 0000.1c.00.0: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W)
> > > > > >>> 4096B-4096B,
> > > > > >>> (T) 4096B-4096B, ioengine=spdk, iodepth=32 ...
> > > > > >>> fio-3.11
> > > > > >>> Starting 4 threads
> > > > > >>> Starting SPDK v18.10 / DPDK 18.08.0 initialization...
> > > > > >>> [ DPDK EAL parameters: fio --no-shconf -c 0x1 -m 512 --file-
> > > > > >>> prefix=spdk_pid3668 ]
> > > > > >>> EAL: Detected 36 lcore(s)
> > > > > >>> EAL: Detected 2 NUMA nodes
> > > > > >>> EAL: No free hugepages reported in hugepages-1048576kB
> > > > > >>> EAL: Probing VFIO support...
> > > > > >>> EAL: PCI device 0000:1c:00.0 on NUMA socket 0
> > > > > >>> EAL: probe driver: 8086:a54 spdk_nvme
> > > > > >>> Jobs: 1 (f=0): [f(1),_(3)][100.0%][eta 00m:00s]
> > > > > >>> 0000.1c.00.0: (groupid=0, jobs=4): err= 0: pid=3709: Mon Nov
> > > > > >>> 5
> > > > > >>> 02:28:29
> > > > > >>> 2018
> > > > > >>> mixed: IOPS=13.2k, BW=51.6MiB/s
> > (54.1MB/s)(517MiB/10011msec)
> > > > > >>> slat (nsec): min=109, max=15344, avg=127.91, stdev=73.44
> > > > > >>> clat (usec): min=187, max=18715, avg=9683.80,
> stdev=650.54
> > > > > >>> lat (usec): min=193, max=18715, avg=9683.93, stdev=650.53
> > > > > >>> clat percentiles (usec):
> > > > > >>> | 1.00th=[ 8717], 5.00th=[ 8848], 10.00th=[ 8979],
> > > > > >>> 20.00th=[
> > > > 9110],
> > > > > >>> | 30.00th=[ 9241], 40.00th=[ 9372], 50.00th=[ 9503],
> > > > > >>> 60.00th=[
> > > > 9765],
> > > > > >>> | 70.00th=[10028], 80.00th=[10159], 90.00th=[10552],
> > > > > 95.00th=[10945],
> > > > > >>> | 99.00th=[11338], 99.50th=[11338], 99.90th=[11731],
> > > > > 99.95th=[13960],
> > > > > >>> | 99.99th=[17695]
> > > > > >>> bw ( KiB/s): min=13080, max=13304, per=25.00%,
> > > > > >>> avg=13214.70, stdev=55.30, samples=80
> > > > > >>> iops : min= 3270, max= 3326, avg=3303.65, stdev=13.84,
> > > > > samples=80
> > > > > >>> lat (usec) : 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
> > > > > >>> lat (msec) : 2=0.01%, 4=0.02%, 10=70.85%, 20=29.11%
> > > > > >>> cpu : usr=100.01%, sys=0.00%, ctx=23, majf=0, minf=0
> > > > > >>> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%,
> > 32=99.9%,
> > > > > >>> >=64=0.0%
> > > > > >>> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%,
> > > 64=0.0%,
> > > > > >>> >=64=0.0%
> > > > > >>> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%,
> > > > > >>> 64=0.0%,
> > > > > >>> >=64=0.0%
> > > > > >>> issued rwts: total=132300,0,0,0 short=0,0,0,0
> dropped=0,0,0,0
> > > > > >>> latency : target=0, window=0, percentile=100.00%,
> depth=32
> > > > > >>> Run status group 0 (all jobs):
> > > > > >>> MIXED: bw=51.6MiB/s (54.1MB/s), 51.6MiB/s-51.6MiB/s
> > > (54.1MB/s-
> > > > > >>> 54.1MB/s), io=517MiB (542MB), run=10011-10011msec
> > > > > >>>
> > > > > >>> _______________________________________________
> > > > > >>> SPDK mailing list
> > > > > >>> SPDK(a)lists.01.org
> > > > > >>> https://lists.01.org/mailman/listinfo/spdk
> > > > > >>> _______________________________________________
> > > > > >>> SPDK mailing list
> > > > > >>> SPDK(a)lists.01.org
> > > > > >>> https://lists.01.org/mailman/listinfo/spdk
> > > > > >>_______________________________________________
> > > > > >>SPDK mailing list
> > > > > >> SPDK(a)lists.01.org
> > > > > >> https://lists.01.org/mailman/listinfo/spdk
> > > > > >>_______________________________________________
> > > > > >>SPDK mailing list
> > > > > >> SPDK(a)lists.01.org
> > > > > >> https://lists.01.org/mailman/listinfo/spdk
> > > > > >
> > > > > >
> > > > > >--
> > > > > >Маров Алексей
> > > > > >_______________________________________________
> > > > > >SPDK mailing list
> > > > > >SPDK(a)lists.01.org
> > > > > >https://lists.01.org/mailman/listinfo/spdk
> > > > > >_______________________________________________
> > > > > >SPDK mailing list
> > > > > >SPDK(a)lists.01.org
> > > > > >https://lists.01.org/mailman/listinfo/spdk
> > > > >
> > > > >
> > > > > --
> > > > > Маров Алексей
> > > > > _______________________________________________
> > > > > SPDK mailing list
> > > > > SPDK(a)lists.01.org
> > > > > https://lists.01.org/mailman/listinfo/spdk
> > > > _______________________________________________
> > > > SPDK mailing list
> > > > SPDK(a)lists.01.org
> > > > https://lists.01.org/mailman/listinfo/spdk
> > > >
> > > --
> > >
> > > Regards,
> > > Andrey
> > > _______________________________________________
> > > SPDK mailing list
> > > SPDK(a)lists.01.org
> > > https://lists.01.org/mailman/listinfo/spdk
> > > _______________________________________________
> > > SPDK mailing list
> > > SPDK(a)lists.01.org
> > > https://lists.01.org/mailman/listinfo/spdk
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
>
>
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [SPDK] FIO NVMe Performance Results
@ 2018-11-05 22:19 Gruher, Joseph R
0 siblings, 0 replies; 25+ messages in thread
From: Gruher, Joseph R @ 2018-11-05 22:19 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 49860 bytes --]
Hi Jim-
Yes, this is all local PCIe attached NVMe drives for now - the goal is to baseline the local NVMe on the system with SPDK and then move on to NVMeoF from there.
The basic test process (once SPDK has been built) is to reboot the system, run the included SPDK scripts/setup.sh script, and then run FIO with the SPDK user or bdev plugin. We're not starting app/nvmf_tgt/nvmf_tgt or anything like that so I don't think SPDK should be running anywhere outside of what is launched through FIO. All cores look idle until FIO is launched. If that doesn't sound like the right test process, let us know!
Thanks,
Joe
> -----Original Message-----
> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Harris, James R
> Sent: Monday, November 5, 2018 1:56 PM
> To: Storage Performance Development Kit <spdk(a)lists.01.org>
> Subject: Re: [SPDK] FIO NVMe Performance Results
>
> Hi Joe,
>
> This is all just local PCIe NVMe – no NVMe-oF? Can you make sure you don’t
> have any other SPDK processes running while you’re doing this? I’m
> concerned that fio is polling on the same CPU core as an SPDK process, and
> your performance degradation is due to timeslice sharing.
>
> -Jim
>
>
> On 11/5/18, 2:42 PM, "SPDK on behalf of Howell, Seth" <spdk-
> bounces(a)lists.01.org on behalf of seth.howell(a)intel.com> wrote:
>
> I think that the uio_pci_generic vs. vfio thing is unrelated. The only reason I
> brought it up was because it could have been indicative of a larger
> configuration difference. However, I ran my test with and without vfio
> support just to be sure and got the higher performance numbers both times.
>
> As a side note:
> It's probably a good thing to have enabled in general when running spdk
> benchmarks because that is the recommended configuration on any physical
> machine.
> You can enable it in grub by adding the following to the
> GRUB_CMDLINE_LINUX line of /etc/default.grub: intel_iommu=on
> and then running grub2-mkconfig.
>
> -----Original Message-----
> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Gruher,
> Joseph R
> Sent: Monday, November 5, 2018 1:39 PM
> To: Storage Performance Development Kit <spdk(a)lists.01.org>
> Subject: Re: [SPDK] FIO NVMe Performance Results
>
> Attempting to fall back to earlier revisions has not yielded any
> improvements for me. All tests use the same P4500 physical disk. FIO files
> are as shown before, running a 4 thread, queue depth 32 per thread, 4KB
> random read workload.
>
> SPDK 18.10, kernel 4.18.16, FIO 3.11 = 12.9k IOPS SPDK 18.10, kernel
> 4.16.14, FIO 3.11 = 11.9k IOPS SPDK 18.04, kernel 4.18.16, FIO 3.11 = 12.9k
> IOPS SPDK 18.04, kernel 4.16.14, FIO 3.11 = 13.0k IOPS SPDK 18.04, kernel
> 4.16.16, FIO 3.3 = 13.1k IOPS
>
> Kernel NVMe driver, kernel 4.16.16, FIO 3.3 = 477k IOPS Kernel NVMe
> driver, kernel 4.18.16, FIO 3.11 = 477k IOPS
>
> I am curious if the comment below about uio_pci_generic instead of vfio
> can lead us anywhere, or maybe there's some OS setting or BIOS setting that
> needs to be toggled...
>
> For the folks working inside Intel, I can also give you direct access to the
> test system, if that would help.
>
> Thanks,
> Joe
>
> > -----Original Message-----
> > From: Gruher, Joseph R
> > Sent: Monday, November 5, 2018 11:05 AM
> > To: Storage Performance Development Kit <spdk(a)lists.01.org>
> > Subject: RE: [SPDK] FIO NVMe Performance Results
> >
> > I've had good FIO NVMe results in the past we well, with SPDK 18.04 on
> > Ubuntu 18.04, with kernel 4.16.14. That was on a different, but
> > similar, hardware platform.
> >
> > I suspect something has changed now in SPDK 18.10 and/or kernel
> > 4.18.16. I can try going back to earlier versions of each to see if I
> > can recreate earlier good results and find where things went wrong.
> >
> > We're not running in a VM - this is a bare metal Xeon server with
> > Skylake CPUs.
> >
> > Thanks,
> > Joe
> >
> > > -----Original Message-----
> > > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Howell,
> > > Seth
> > > Sent: Monday, November 5, 2018 10:32 AM
> > > To: Storage Performance Development Kit <spdk(a)lists.01.org>
> > > Subject: Re: [SPDK] FIO NVMe Performance Results
> > >
> > > Hi Joseph,
> > > I attempted to verify your results on my system. I am using fedora
> > > 27 with kernel version 4.17.3 with a p3700 drive. I cloned fio from
> > > https://github.com/axboe/fio and checked out the appropriate tag. I
> > > got appx 408k iops.
> > > Typically we run spdk with the iommu enabled. I noticed that your
> > > devices are binding to uio_pci_generic instead of vfio. This
> > > shouldn't make any difference at all, but I just want to make sure
> > > you aren't running in a VM or something like that and passing
> > > through your NVMe drives and that's why you aren't using the IOMMU.
> > > That could explain a deeper configuration difference that could help
> > > us know where to look for
> > the disparity.
> > >
> > > Please see the contents of my configuration files and output below:
> > >
> > > Test.ini:
> > > [global]
> > > rw=randrw
> > > rwmixread=100
> > > numjobs=4
> > > iodepth=32
> > > bs=4k
> > > direct=1
> > > thread=1
> > > time_based=1
> > > ramp_time=0
> > > runtime=10
> > > ioengine=spdk_bdev
> > > spdk_conf=./nvme.conf
> > > group_reporting=1
> > > unified_rw_reporting=1
> > > exitall=1
> > > randrepeat=0
> > > norandommap=1
> > > [0000.81.00.0]
> > > filename=Nvme0n1
> > >
> > > nvme.conf:
> > > [Nvme]
> > > TransportId "trtype:PCIe traddr:0000.81.00.0" Nvme0
> > >
> > > Sudo
> > >
> >
> LD_PRELOAD=/home/sethhowe/Desktop/Development_Folders/spdk/exam
> > > ples/bdev/fio_plugin/fio_plugin ./fio ./test.ini >./out,txt 2>&1
> > >
> > > Out.txt:
> > > 0000.81.00.0: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-
> 4096B,
> > > (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=32 ...
> > > fio-3.11
> > > Starting 4 threads
> > > Starting SPDK v18.10 / DPDK 18.08.0 initialization...
> > > [ DPDK EAL parameters: fio --no-shconf -c 0x1
> > > --file-prefix=spdk_pid782339 ]
> > > EAL: Detected 16 lcore(s)
> > > EAL: Detected 2 NUMA nodes
> > > EAL: No free hugepages reported in hugepages-1048576kB
> > > EAL: Probing VFIO support...
> > > EAL: VFIO support initialized
> > > EAL: PCI device 0000:81:00.0 on NUMA socket 1
> > > EAL: probe driver: 8086:953 spdk_nvme
> > >
> > > 0000.81.00.0: (groupid=0, jobs=4): err= 0: pid=782359: Mon Nov 5
> > > 10:58:59
> > > 2018
> > > mixed: IOPS=410k, BW=1600MiB/s (1678MB/s)(15.6GiB/10001msec)
> > > slat (nsec): min=179, max=19985, avg=245.97, stdev=122.96
> > > clat (usec): min=8, max=1821, avg=311.82, stdev=167.14
> > > lat (usec): min=8, max=1822, avg=312.06, stdev=167.14
> > > clat percentiles (usec):
> > > | 1.00th=[ 24], 5.00th=[ 48], 10.00th=[ 78], 20.00th=[ 141],
> > > | 30.00th=[ 200], 40.00th=[ 258], 50.00th=[ 314], 60.00th=[ 371],
> > > | 70.00th=[ 429], 80.00th=[ 486], 90.00th=[ 545], 95.00th=[ 570],
> > > | 99.00th=[ 594], 99.50th=[ 594], 99.90th=[ 627], 99.95th=[ 652],
> > > | 99.99th=[ 676]
> > > bw ( KiB/s): min=407264, max=416040, per=25.01%, avg=409742.55,
> > > stdev=1337.38, samples=76
> > > iops : min=101816, max=104010, avg=102435.61, stdev=334.41,
> > > samples=76
> > > lat (usec) : 10=0.01%, 20=0.46%, 50=4.88%, 100=8.41%, 250=25.29%
> > > lat (usec) : 500=43.15%, 750=17.81%, 1000=0.01%
> > > lat (msec) : 2=0.01%
> > > cpu : usr=99.78%, sys=0.00%, ctx=26, majf=0, minf=0
> > > IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.9%,
> > > >=64=0.0%
> > > submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
> > > >=64=0.0%
> > > complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.1%, 32=0.1%,
> > > 64=0.0%,
> > > >=64=0.0%
> > > issued rwts: total=4096867,0,0,0 short=0,0,0,0 dropped=0,0,0,0
> > > latency : target=0, window=0, percentile=100.00%, depth=32
> > >
> > > Run status group 0 (all jobs):
> > > MIXED: bw=1600MiB/s (1678MB/s), 1600MiB/s-1600MiB/s
> (1678MB/s-
> > > 1678MB/s), io=15.6GiB (16.8GB), run=10001-10001msec
> > >
> > > Thanks,
> > >
> > > Seth
> > >
> > >
> > >
> > > -----Original Message-----
> > > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Andrey
> > > Kuzmin
> > > Sent: Monday, November 5, 2018 9:48 AM
> > > To: Storage Performance Development Kit <spdk(a)lists.01.org>
> > > Subject: Re: [SPDK] FIO NVMe Performance Results
> > >
> > > On Mon, Nov 5, 2018, 18:29 Gruher, Joseph R
> > > <joseph.r.gruher(a)intel.com>
> > > wrote:
> > >
> > > > My FIO version is v3.11.
> > > >
> > > > I tried the bdev FIO plugin as suggested, but performance is about
> > > > the same as the other SPDK FIO plugin, I observed only 10.4K IOPS
> > > > in a quick 4KB random read test. See configuration and results
> below.
> > > >
> > >
> > > As 12 ms median read latency for an nvme ssd is far beyond anything
> > > reasonable, why don't you just perf this run, to see where the time
> > > is being spent?
> > >
> > > Regards,
> > > Andrey
> > >
> > > >
> > > >
> > > > don(a)donst201:~/fio/single/rr$ cat nvme.conf [Nvme] TransportId
> > > > "trtype:PCIe traddr:0000.1c.00.0" Nvme0
> > > >
> > > >
> > > > don(a)donst201:~/fio/single/rr$ cat 1c-bdev.ini |grep -v '#'
> > > > [global]
> > > > rw=randrw
> > > > rwmixread=100
> > > > numjobs=4
> > > > iodepth=32
> > > > bs=4k
> > > > direct=1
> > > > thread=1
> > > > time_based=1
> > > > ramp_time=0
> > > > runtime=10
> > > > ioengine=spdk_bdev
> > > > spdk_conf=./nvme.conf
> > > > group_reporting=1
> > > > unified_rw_reporting=1
> > > > exitall=1
> > > > randrepeat=0
> > > > norandommap=1
> > > > [0000.1c.00.0]
> > > > filename=Nvme0n1
> > > >
> > > >
> > > > don(a)donst201:~/fio/single/rr$ sudo
> > > >
> > >
> >
> LD_PRELOAD=/home/don/install/spdk/spdk/examples/bdev/fio_plugin/fio_p
> > > l
> > > > ugin
> > > > fio 1c-bdev.ini
> > > > 0000.1c.00.0: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W)
> > > > 4096B-4096B,
> > > > (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=32 ...
> > > > fio-3.11
> > > > Starting 4 threads
> > > > Starting SPDK v18.10 / DPDK 18.08.0 initialization...
> > > > [ DPDK EAL parameters: fio --no-shconf -c 0x1
> > > > --file-prefix=spdk_pid4865 ]
> > > > EAL: Detected 36 lcore(s)
> > > > EAL: Detected 2 NUMA nodes
> > > > EAL: No free hugepages reported in hugepages-1048576kB
> > > > EAL: Probing VFIO support...
> > > > EAL: PCI device 0000:03:00.0 on NUMA socket 0
> > > > EAL: probe driver: 8086:f1a5 spdk_nvme
> > > > EAL: PCI device 0000:1c:00.0 on NUMA socket 0
> > > > EAL: probe driver: 8086:a54 spdk_nvme
> > > > EAL: PCI device 0000:1d:00.0 on NUMA socket 0
> > > > EAL: probe driver: 8086:a54 spdk_nvme
> > > > EAL: PCI device 0000:5e:00.0 on NUMA socket 0
> > > > EAL: probe driver: 8086:a54 spdk_nvme
> > > > EAL: PCI device 0000:5f:00.0 on NUMA socket 0
> > > > EAL: probe driver: 8086:a54 spdk_nvme
> > > > EAL: PCI device 0000:62:00.0 on NUMA socket 0
> > > > EAL: probe driver: 8086:a54 spdk_nvme
> > > > EAL: PCI device 0000:63:00.0 on NUMA socket 0
> > > > EAL: probe driver: 8086:a54 spdk_nvme
> > > > EAL: PCI device 0000:64:00.0 on NUMA socket 0
> > > > EAL: probe driver: 8086:a54 spdk_nvme
> > > > EAL: PCI device 0000:65:00.0 on NUMA socket 0
> > > > EAL: probe driver: 8086:a54 spdk_nvme
> > > > EAL: PCI device 0000:da:00.0 on NUMA socket 1
> > > > EAL: probe driver: 8086:a54 spdk_nvme
> > > > EAL: PCI device 0000:db:00.0 on NUMA socket 1
> > > > EAL: probe driver: 8086:a54 spdk_nvme
> > > > EAL: PCI device 0000:dc:00.0 on NUMA socket 1
> > > > EAL: probe driver: 8086:a54 spdk_nvme
> > > > EAL: PCI device 0000:dd:00.0 on NUMA socket 1
> > > > EAL: probe driver: 8086:a54 spdk_nvme
> > > > EAL: PCI device 0000:e0:00.0 on NUMA socket 1
> > > > EAL: probe driver: 8086:a54 spdk_nvme
> > > > EAL: PCI device 0000:e1:00.0 on NUMA socket 1
> > > > EAL: probe driver: 8086:a54 spdk_nvme
> > > > EAL: PCI device 0000:e2:00.0 on NUMA socket 1
> > > > EAL: probe driver: 8086:a54 spdk_nvme
> > > > EAL: PCI device 0000:e3:00.0 on NUMA socket 1
> > > > EAL: probe driver: 8086:a54 spdk_nvme
> > > > Jobs: 4 (f=4): [r(4)][100.0%][r=45.7MiB/s][r=11.7k IOPS][eta
> > > > 00m:00s]
> > > > 0000.1c.00.0: (groupid=0, jobs=4): err= 0: pid=4906: Mon Nov 5
> > > > 15:23:47
> > > > 2018
> > > > mixed: IOPS=10.4k, BW=40.5MiB/s (42.5MB/s)(406MiB/10012msec)
> > > > slat (nsec): min=201, max=15831, avg=238.11, stdev=148.78
> > > > clat (usec): min=124, max=31244, avg=12330.69, stdev=1697.98
> > > > lat (usec): min=130, max=31244, avg=12330.93, stdev=1697.98
> > > > clat percentiles (usec):
> > > > | 1.00th=[ 9634], 5.00th=[10159], 10.00th=[10552],
> 20.00th=[10945],
> > > > | 30.00th=[11338], 40.00th=[11731], 50.00th=[12125],
> > > 60.00th=[12518],
> > > > | 70.00th=[13042], 80.00th=[13566], 90.00th=[14353],
> > > 95.00th=[15008],
> > > > | 99.00th=[16712], 99.50th=[18744], 99.90th=[27395],
> > > 99.95th=[29754],
> > > > | 99.99th=[30278]
> > > > bw ( KiB/s): min= 8784, max=11952, per=24.99%, avg=10375.53,
> > > > stdev=929.68, samples=80
> > > > iops : min= 2196, max= 2988, avg=2593.85, stdev=232.41,
> > > > samples=80
> > > > lat (usec) : 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
> > > > lat (msec) : 2=0.01%, 4=0.02%, 10=2.71%, 20=96.91%, 50=0.33%
> > > > cpu : usr=99.99%, sys=0.01%, ctx=28, majf=0, minf=0
> > > > IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.9%,
> > > > >=64=0.0%
> > > > submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%,
> 64=0.0%,
> > > > >=64=0.0%
> > > > complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%,
> > > > 64=0.0%,
> > > > >=64=0.0%
> > > > issued rwts: total=103907,0,0,0 short=0,0,0,0 dropped=0,0,0,0
> > > > latency : target=0, window=0, percentile=100.00%, depth=32
> > > > Run status group 0 (all jobs):
> > > > MIXED: bw=40.5MiB/s (42.5MB/s), 40.5MiB/s-40.5MiB/s
> > > > (42.5MB/s-42.5MB/s), io=406MiB (426MB), run=10012-10012msec
> > > >
> > > >
> > > > Thanks,
> > > > Joe
> > > >
> > > > > -----Original Message-----
> > > > > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of ?????
> > > ???????
> > > > > Sent: Monday, November 5, 2018 12:45 AM
> > > > > To: Storage Performance Development Kit <spdk(a)lists.01.org>
> > > > > Subject: Re: [SPDK] FIO NVMe Performance Results
> > > > >
> > > > > Ok, Do you see some performance issues using general bdev fio
> > > > > plugin
> > > > with
> > > > > you configuration ?
> > > > >
> > > > > Best Regards
> > > > > Alex
> > > > >
> > > > >
> > > > > >Понедельник, 5 ноября 2018, 11:39 +03:00 от "Cao, Gang"
> > > > > <gang.cao(a)intel.com>:
> > > > > >
> > > > > >Yes. This is for the bdev version of SPDK FIO plugin and you
> > > > > >can assign
> > > > the
> > > > > conf file as the parameter in the FIO configuration.
> > > > > >
> > > > > >Previously, it's another SPDK FIO plugin for the user space
> > > > > >NVMe driver
> > > > > only.
> > > > > >
> > > > > >These are two different FIO plugins.
> > > > > >
> > > > > >Thanks,
> > > > > >Gang
> > > > > >
> > > > > >-----Original Message-----
> > > > > >From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of
> > > > > >Маров Алексей
> > > > > >Sent: Monday, November 5, 2018 4:32 PM
> > > > > >To: Storage Performance Development Kit < spdk(a)lists.01.org >
> > > > > >Subject: Re: [SPDK] FIO NVMe Performance Results
> > > > > >
> > > > > >Sorry for incomplete message I have sent before
> > > > > >
> > > > > >Hello Gang.
> > > > > >
> > > > > >I just curious, why would you run fio with SPDK using this conf
> > > > > >file,
> > > > with is a
> > > > > bit different from what is suggested in here
> > > https://github.com/spdk/spdk/tree/master/examples/bdev/fio_plugin
> > > > > .
> > > > > >
> > > > > >I mean may be you should create conf file for spdk like this one :
> > > > > >
> > > > > >nvme.conf :
> > > > > >[Nvme]
> > > > > > TransportId "trtype:PCIe traddr:0000.1c.00.0" Nvme0
> > > > > >
> > > > > >and provide it in fio config.
> > > > > >
> > > > > >fio.ini :
> > > > > >[global]
> > > > > >ioengine=spdk_bdev
> > > > > >spdk_conf=....../nvme.conf
> > > > > >rw=randrw
> > > > > >rwmixread=100
> > > > > >numjobs=4
> > > > > >iodepth=32
> > > > > >bs=4k
> > > > > >direct=1
> > > > > >thread=1
> > > > > >time_based=1
> > > > > >ramp_time=0
> > > > > >runtime=10
> > > > > >group_reporting=1
> > > > > >unified_rw_reporting=1
> > > > > >exitall=1
> > > > > >randrepeat=0
> > > > > >norandommap=1
> > > > > >
> > > > > >[0000:06:00.0]
> > > > > >filename=Nvme0n1
> > > > > >
> > > > > >and run like this
> > > > > >
> > > > > ># LD_PRELOAD=examples/bdev/fio_plugin/fio_plugin fio
> > > > > >examples/bdev/fio_plugin/fio.ini
> > > > > >
> > > > > >Just to note. The recommended version of fio for SPDK is 3.3
> > > > > >
> > > > > >Best Regards
> > > > > >Alex
> > > > > >
> > > > > >
> > > > > >>Понедельник, 5 ноября 2018, 9:53 +03:00 от "Cao, Gang" <
> > > > > gang.cao(a)intel.com >:
> > > > > >>
> > > > > >>Thanks. This is interesting especially for your finding on
> > > > > >>queue depth
> > > > 1
> > > > > case. We are trying to reproduce this issue locally same as your
> > > > > configuration.
> > > > > >>
> > > > > >>Thanks,
> > > > > >>Gang
> > > > > >>
> > > > > >>-----Original Message-----
> > > > > >>From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of
> > > > > >>Gruher, Joseph R
> > > > > >>Sent: Monday, November 5, 2018 1:32 PM
> > > > > >>To: Storage Performance Development Kit < spdk(a)lists.01.org >
> > > > > >>Subject: Re: [SPDK] FIO NVMe Performance Results
> > > > > >>
> > > > > >>Interesting catch. It seems like FIO is actually not
> > > > > >>accurately
> > > > reporting the
> > > > > number of jobs in the report out for the SPDK case. If I
> > > > > disable the group_reporting parameter I do get a printout for
> > > > > four separate jobs in
> > > > the
> > > > > SPDK case, so I think it is really running four jobs there, and
> > > > > just
> > > > summarizing
> > > > > incorrectly. I also see four separate CPU cores go to 100% when
> > > > > I run
> > > > that
> > > > > FIO test case, indicating four jobs are active.
> > > > > >>
> > > > > >>Also, I don't think would explain the difference in performance.
> > > > Using an
> > > > > aggregate queue depth of 32 versus 128 shouldn't lead to a drop
> > > > > in performance from 377K to 13K.
> > > > > >>
> > > > > >>I retested SPDK with multiple separate job descriptions as you
> > > > suggested
> > > > > anyway, using four separate jobs instead of relying on the
> > > > > numjobs=4 parameter, and the results are below. In that SPDK
> > > > > test with four
> > > > separate
> > > > > FIO job statements, each with queue depth 32, I still only
> > > > > achieved 12.5K IOPS.
> > > > > >>
> > > > > >>I then re-ran the kernel NVMe driver test case with only a
> > > > > >>single job
> > > > for
> > > > > comparison, and even with only 1 job and queue depth 32 it still
> > > > > achieves 254K IOPS.
> > > > > >>
> > > > > >>We might be on the right track with a problem around queue
> > > > > >>depth
> > > > > however. If I run the kernel NVMe driver case with just 1 job
> > > > > and queue depth of only 1, I then get 11.2K IOPS, which is
> > > > > around the performance
> > > > I get
> > > > > in the SPDK test case. Perhaps the SPDK case is not actually
> > > > > queuing
> > > > IOs on
> > > > > the device for some reason?
> > > > > >>
> > > > > >>
> > > > > >>don(a)donst201:~/fio/single/rr$ cat nvme2n1.ini |grep -v '#'
> > > > > >>[global]
> > > > > >>rw=randrw
> > > > > >>rwmixread=100
> > > > > >>numjobs=1
> > > > > >>iodepth=32
> > > > > >>bs=4k
> > > > > >>direct=1
> > > > > >>thread=1
> > > > > >>time_based=1
> > > > > >>ramp_time=0
> > > > > >>runtime=10
> > > > > >>ioengine=libaio
> > > > > >>group_reporting=1
> > > > > >>unified_rw_reporting=1
> > > > > >>exitall=1
> > > > > >>randrepeat=0
> > > > > >>norandommap=1
> > > > > >>[nvme2n1]
> > > > > >>filename=/dev/nvme2n1
> > > > > >>
> > > > > >>
> > > > > >>don(a)donst201:~/fio/single/rr$ sudo fio nvme2n1.ini
> > > > > >>nvme2n1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W)
> > > > > >>4096B-4096B,
> > > > > >>(T) 4096B-4096B, ioengine=libaio, iodepth=32
> > > > > >>fio-3.11
> > > > > >>Starting 1 thread
> > > > > >>Jobs: 1 (f=1): [r(1)][100.0%][r=1025MiB/s][r=262k IOPS][eta
> > > > > >>00m:00s]
> > > > > >>nvme2n1: (groupid=0, jobs=1): err= 0: pid=2422: Mon Nov 5
> > > > > >>05:19:13
> > > > > >>2018
> > > > > >> mixed: IOPS=254k, BW=992MiB/s
> > (1040MB/s)(9922MiB/10001msec)
> > > > > >> slat (nsec): min=1185, max=319052, avg=1611.64,
> stdev=808.40
> > > > > >> clat (usec): min=2, max=4743, avg=124.02, stdev=121.00
> > > > > >> lat (usec): min=6, max=4744, avg=125.68, stdev=121.02
> > > > > >> clat percentiles (usec):
> > > > > >> | 1.00th=[ 9], 5.00th=[ 56], 10.00th=[ 61], 20.00th=[
> > > > 73],
> > > > > >> | 30.00th=[ 80], 40.00th=[ 91], 50.00th=[ 110], 60.00th=[
> > > > 118],
> > > > > >> | 70.00th=[ 130], 80.00th=[ 157], 90.00th=[ 202],
> > > > > >>95.00th=[ 253],
> > > > > >> | 99.00th=[ 408], 99.50th=[ 498], 99.90th=[ 1795],
> > > > > >>99.95th=[ 3064],
> > > > > >> | 99.99th=[ 3785]
> > > > > >> bw ( KiB/s): min=612240, max=1078792, per=100.00%,
> > > > > >>avg=1015996.40,
> > > > > stdev=106032.96, samples=20
> > > > > >> iops : min=153060, max=269698, avg=253998.90,
> > > stdev=26508.40,
> > > > > samples=20
> > > > > >> lat (usec) : 4=0.01%, 10=1.47%, 20=1.19%, 50=0.17%,
> 100=42.03%
> > > > > >> lat (usec) : 250=49.95%, 500=4.70%, 750=0.28%, 1000=0.06%
> > > > > >> lat (msec) : 2=0.06%, 4=0.09%, 10=0.01%
> > > > > >> cpu : usr=21.45%, sys=51.11%, ctx=530348, majf=0,
> minf=139
> > > > > >> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%,
> > 32=100.0%,
> > > > > >=64=0.0%
> > > > > >> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%,
> > 64=0.0%,
> > > > > >=64=0.0%
> > > > > >> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%,
> > > > > >> 64=0.0%,
> > > > > >>>=64=0.0%
> > > > > >> issued rwts: total=2540023,0,0,0 short=0,0,0,0
> dropped=0,0,0,0
> > > > > >> latency : target=0, window=0, percentile=100.00%, depth=32
> > > > > >>Run status group 0 (all jobs):
> > > > > >> MIXED: bw=992MiB/s (1040MB/s), 992MiB/s-992MiB/s
> (1040MB/s-
> > > > > 1040MB/s), io=9922MiB (10.4GB), run=10001-10001msec Disk stats
> > > > > (read/write):
> > > > > >> nvme2n1: ios=2514356/0, merge=0/0, ticks=400/0,
> > > in_queue=297612,
> > > > > >>util=99.06%
> > > > > >>
> > > > > >>
> > > > > >>don(a)donst201:~/fio/single/rr$ cat 1c.ini |grep -v '#'
> > > > > >>[global]
> > > > > >>rw=randrw
> > > > > >>rwmixread=100
> > > > > >>numjobs=1
> > > > > >>iodepth=32
> > > > > >>bs=4k
> > > > > >>direct=1
> > > > > >>thread=1
> > > > > >>time_based=1
> > > > > >>ramp_time=0
> > > > > >>runtime=10
> > > > >
> > >
> >>ioengine=/home/don/install/spdk/spdk/examples/nvme/fio_plugin/fio_
> > > > > >>pl
> > > > > ug
> > > > > >>in
> > > > > >>group_reporting=1
> > > > > >>unified_rw_reporting=1
> > > > > >>exitall=1
> > > > > >>randrepeat=0
> > > > > >>norandommap=1
> > > > > >>[0000.1c.00.0-1]
> > > > > >>filename=trtype=PCIe traddr=0000.1c.00.0 ns=1 [0000.1c.00.0-2]
> > > > > >>filename=trtype=PCIe traddr=0000.1c.00.0 ns=1 [0000.1c.00.0-3]
> > > > > >>filename=trtype=PCIe traddr=0000.1c.00.0 ns=1 [0000.1c.00.0-4]
> > > > > >>filename=trtype=PCIe traddr=0000.1c.00.0 ns=1
> > > > > >>
> > > > > >>
> > > > > >>don(a)donst201:~/fio/single/rr$ sudo fio 1c.ini
> > > > > >>0000.1c.00.0-1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W)
> > > > > >>4096B-4096B,
> > > > > >>(T) 4096B-4096B, ioengine=spdk, iodepth=32
> > > > > >>0000.1c.00.0-2: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W)
> > > > > >>4096B-4096B,
> > > > > >>(T) 4096B-4096B, ioengine=spdk, iodepth=32
> > > > > >>0000.1c.00.0-3: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W)
> > > > > >>4096B-4096B,
> > > > > >>(T) 4096B-4096B, ioengine=spdk, iodepth=32
> > > > > >>0000.1c.00.0-4: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W)
> > > > > >>4096B-4096B,
> > > > > >>(T) 4096B-4096B, ioengine=spdk, iodepth=32
> > > > > >>fio-3.11
> > > > > >>Starting 4 threads
> > > > > >>Starting SPDK v18.10 / DPDK 18.08.0 initialization...
> > > > > >>[ DPDK EAL parameters: fio --no-shconf -c 0x1 -m 512
> > > > > >>--file-prefix=spdk_pid4270 ]
> > > > > >>EAL: Detected 36 lcore(s)
> > > > > >>EAL: Detected 2 NUMA nodes
> > > > > >>EAL: No free hugepages reported in hugepages-1048576kB
> > > > > >>EAL: Probing VFIO support...
> > > > > >>EAL: PCI device 0000:1c:00.0 on NUMA socket 0
> > > > > >>EAL: probe driver: 8086:a54 spdk_nvme
> > > > > >>Jobs: 1 (f=0): [_(3),f(1)][100.0%][eta 00m:00s]
> > > > > >>0000.1c.00.0-1: (groupid=0, jobs=4): err= 0: pid=4310: Mon Nov
> > > > > >>5
> > > > > >>05:16:03 2018
> > > > > >> mixed: IOPS=12.5k, BW=48.0MiB/s
> (51.4MB/s)(490MiB/10010msec)
> > > > > >> slat (nsec): min=98, max=15092, avg=120.02, stdev=90.07
> > > > > >> clat (usec): min=55, max=19757, avg=10207.22, stdev=884.06
> > > > > >> lat (usec): min=60, max=19757, avg=10207.34, stdev=884.06
> > > > > >> clat percentiles (usec):
> > > > > >> | 1.00th=[ 8979], 5.00th=[ 9110], 10.00th=[ 9372],
> > > > > >>20.00th=[ 9503],
> > > > > >> | 30.00th=[ 9765], 40.00th=[ 9896], 50.00th=[10159],
> > > > > >>60.00th=[10290],
> > > > > >> | 70.00th=[10552], 80.00th=[10814], 90.00th=[11207],
> > > > > >>95.00th=[11600],
> > > > > >> | 99.00th=[12911], 99.50th=[14353], 99.90th=[17695],
> > > > > >>99.95th=[18744],
> > > > > >> | 99.99th=[19006]
> > > > > >> bw ( KiB/s): min=12008, max=12976, per=25.00%,
> > > > > >>avg=12535.70,
> > > > > stdev=306.81, samples=80
> > > > > >> iops : min= 3002, max= 3244, avg=3133.85, stdev=76.77,
> > > > samples=80
> > > > > >> lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%,
> > > 1000=0.01%
> > > > > >> lat (msec) : 2=0.01%, 4=0.02%, 10=45.13%, 20=54.83%
> > > > > >> cpu : usr=100.01%, sys=0.00%, ctx=27, majf=0, minf=0
> > > > > >> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%,
> 32=99.9%,
> > > > > >=64=0.0%
> > > > > >> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%,
> > 64=0.0%,
> > > > > >=64=0.0%
> > > > > >> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%,
> > > > > >> 64=0.0%,
> > > > > >>>=64=0.0%
> > > > > >> issued rwts: total=125506,0,0,0 short=0,0,0,0 dropped=0,0,0,0
> > > > > >> latency : target=0, window=0, percentile=100.00%, depth=32
> > > > > >>Run status group 0 (all jobs):
> > > > > >> MIXED: bw=48.0MiB/s (51.4MB/s), 48.0MiB/s-48.0MiB/s
> > > > > >>(51.4MB/s-51.4MB/s), io=490MiB (514MB), run=10010-
> 10010msec
> > > > > >>
> > > > > >>
> > > > > >>don(a)donst201:~/fio/single/rr$ cat nvme2n1.ini |grep -v '#'
> > > > > >>[global]
> > > > > >>rw=randrw
> > > > > >>rwmixread=100
> > > > > >>numjobs=1
> > > > > >>iodepth=1
> > > > > >>bs=4k
> > > > > >>direct=1
> > > > > >>thread=1
> > > > > >>time_based=1
> > > > > >>ramp_time=0
> > > > > >>runtime=10
> > > > > >>ioengine=libaio
> > > > > >>group_reporting=1
> > > > > >>unified_rw_reporting=1
> > > > > >>exitall=1
> > > > > >>randrepeat=0
> > > > > >>norandommap=1
> > > > > >>[nvme2n1]
> > > > > >>filename=/dev/nvme2n1
> > > > > >>
> > > > > >>
> > > > > >>don(a)donst201:~/fio/single/rr$ sudo fio nvme2n1.ini
> > > > > >>nvme2n1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W)
> > > > > >>4096B-4096B,
> > > > > >>(T) 4096B-4096B, ioengine=libaio, iodepth=1
> > > > > >>fio-3.11
> > > > > >>Starting 1 thread
> > > > > >>Jobs: 1 (f=1): [r(1)][100.0%][r=43.6MiB/s][r=11.2k IOPS][eta
> > > > > >>00m:00s]
> > > > > >>nvme2n1: (groupid=0, jobs=1): err= 0: pid=2525: Mon Nov 5
> > > > > >>05:22:57
> > > > > >>2018
> > > > > >> mixed: IOPS=11.2k, BW=43.6MiB/s
> (45.7MB/s)(436MiB/10001msec)
> > > > > >> slat (nsec): min=4848, max=208595, avg=5126.33,
> stdev=771.27
> > > > > >> clat (nsec): min=1845, max=3405.8k, avg=83004.99,
> > stdev=33577.36
> > > > > >> lat (usec): min=15, max=3411, avg=88.26, stdev=33.59
> > > > > >> clat percentiles (usec):
> > > > > >> | 1.00th=[ 15], 5.00th=[ 58], 10.00th=[ 59], 20.00th=[
> > > > 59],
> > > > > >> | 30.00th=[ 60], 40.00th=[ 77], 50.00th=[ 77], 60.00th=[
> > > > 78],
> > > > > >> | 70.00th=[ 111], 80.00th=[ 113], 90.00th=[ 113],
> > > > > >>95.00th=[ 115],
> > > > > >> | 99.00th=[ 186], 99.50th=[ 186], 99.90th=[ 190],
> > > > > >>99.95th=[ 192],
> > > > > >> | 99.99th=[ 375]
> > > > > >> bw ( KiB/s): min=43816, max=45016, per=99.98%,
> > > > > >>avg=44639.58,
> > > > > stdev=312.88, samples=19
> > > > > >> iops : min=10954, max=11254, avg=11159.89, stdev=78.22,
> > > > > samples=19
> > > > > >> lat (usec) : 2=0.01%, 20=2.83%, 50=0.01%, 100=64.57%,
> > 250=32.59%
> > > > > >> lat (usec) : 500=0.01%, 750=0.01%
> > > > > >> lat (msec) : 2=0.01%, 4=0.01%
> > > > > >> cpu : usr=4.02%, sys=10.33%, ctx=111630, majf=0, minf=12
> > > > > >> IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%,
> > 32=0.0%,
> > > > > >=64=0.0%
> > > > > >> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%,
> > 64=0.0%,
> > > > > >=64=0.0%
> > > > > >> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%,
> > > > > >> 64=0.0%,
> > > > > >>>=64=0.0%
> > > > > >> issued rwts: total=111631,0,0,0 short=0,0,0,0 dropped=0,0,0,0
> > > > > >> latency : target=0, window=0, percentile=100.00%, depth=1
> > > > > >>Run status group 0 (all jobs):
> > > > > >> MIXED: bw=43.6MiB/s (45.7MB/s), 43.6MiB/s-43.6MiB/s
> > (45.7MB/s-
> > > > > 45.7MB/s), io=436MiB (457MB), run=10001-10001msec Disk stats
> > > > > (read/write):
> > > > > >> nvme2n1: ios=110275/0, merge=0/0, ticks=0/0, in_queue=8596,
> > > > > >>util=86.22%
> > > > > >>
> > > > > >>
> > > > > >>Thanks,
> > > > > >>Joe
> > > > > >>
> > > > > >>> -----Original Message-----
> > > > > >>> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of
> > > > > >>> Cao, Gang
> > > > > >>> Sent: Sunday, November 4, 2018 8:40 PM
> > > > > >>> To: Storage Performance Development Kit < spdk(a)lists.01.org
> > > > > >>> >
> > > > > >>> Subject: Re: [SPDK] FIO NVMe Performance Results
> > > > > >>>
> > > > > >>> For the SPDK plugin, from the below result, looks like only
> > > > > >>> 1 job started with
> > > > > >>> 32 queue depth:
> > > > > >>>
> > > > > >>> Jobs: 1 (f=0): [f(1),_(3)][100.0%][eta 00m:00s] <<<<
> > > > > >>>
> > > > > >>> Try to update the FIO configuration file as
> > > > > >>>
> > > > > >>> [job1]
> > > > > >>> filename=trtype=PCIe traddr=0000.1c.00.0 ns=1 [job2]
> > > > > >>> filename=trtype=PCIe
> > > > > >>> traddr=0000.1c.00.0 ns=1 ....
> > > > > >>>
> > > > > >>> The kernel FIO is with 4 jobs and each job of 32 queue depth.
> > > > > >>> For SPDK, you can configure as the same, or just configure
> > > > > >>> two jobs and each job of 64 queue depth. The overall queue
> > > > > >>> depth to the device is
> > > > > same of 128.
> > > > > >>>
> > > > > >>> Thanks,
> > > > > >>> Gang
> > > > > >>>
> > > > > >>> -----Original Message-----
> > > > > >>> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of
> > > > > >>> Gruher, Joseph R
> > > > > >>> Sent: Monday, November 5, 2018 10:49 AM
> > > > > >>> To: Storage Performance Development Kit < spdk(a)lists.01.org
> > > > > >>> >
> > > > > >>> Subject: [SPDK] FIO NVMe Performance Results
> > > > > >>>
> > > > > >>> Hi folks-
> > > > > >>>
> > > > > >>> I'm testing SPDK 18.10 on Ubuntu 18.04 with kernel 4.18.16.
> > > > > >>> I used FIO with the kernel NVMe driver to measure the
> > > > > >>> performance of a local (PCIe
> > > > > >>> attached) Intel P4500 NVMe device on a 4KB random read
> > > > > >>> workload and obtained 477K IOPS, roughly in line with the
> > > > > >>> drive
> > spec.
> > > > > >>> Then I tested the same drive with the SPDK FIO plugin and
> > > > > >>> only achieved 13K IOPS. The FIO test files and the results
> > > > > >>> are pasted below. Any ideas where I'm going wrong here?
> > > > > >>>
> > > > > >>> Thanks!
> > > > > >>>
> > > > > >>>
> > > > > >>> don(a)donst201:~/fio/single/rr$ ls
> > > > > >>>
> /sys/devices/pci0000:17/0000:17:02.0/0000:1c:00.0/nvme/nvme1
> > > > > >>> / address cntlid dev device firmware_rev model nvme2n1
> > > > > >>> power rescan_controller reset_controller serial state
> > > > > >>> subsysnqn subsystem transport uevent
> > > > > >>>
> > > > > >>>
> > > > > >>> don(a)donst201:~/fio/single/rr$ cat nvme2n1.ini |grep -v '#'
> > > > > >>> [global]
> > > > > >>> rw=randrw
> > > > > >>> rwmixread=100
> > > > > >>> numjobs=4
> > > > > >>> iodepth=32
> > > > > >>> bs=4k
> > > > > >>> direct=1
> > > > > >>> thread=1
> > > > > >>> time_based=1
> > > > > >>> ramp_time=0
> > > > > >>> runtime=10
> > > > > >>> ioengine=libaio
> > > > > >>> group_reporting=1
> > > > > >>> unified_rw_reporting=1
> > > > > >>> exitall=1
> > > > > >>> randrepeat=0
> > > > > >>> norandommap=1
> > > > > >>> [nvme2n1]
> > > > > >>> filename=/dev/nvme2n1
> > > > > >>>
> > > > > >>>
> > > > > >>> don(a)donst201:~/fio/single/rr$ sudo fio nvme2n1.ini
> > > > > >>> nvme2n1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W)
> > > > > >>> 4096B-4096B,
> > > > > >>> (T) 4096B-4096B, ioengine=libaio, iodepth=32 ...
> > > > > >>> fio-3.11
> > > > > >>> Starting 4 threads
> > > > > >>> Jobs: 4 (f=4): [r(4)][100.0%][r=1874MiB/s][r=480k IOPS][eta
> > > > > >>> 00m:00s]
> > > > > >>> nvme2n1: (groupid=0, jobs=4): err= 0: pid=2575: Mon Nov 5
> > > > > >>> 02:26:25
> > > > > 2018
> > > > > >>> mixed: IOPS=477k, BW=1862MiB/s
> > > (1952MB/s)(18.2GiB/10001msec)
> > > > > >>> slat (nsec): min=1295, max=333088, avg=2216.35,
> stdev=927.89
> > > > > >>> clat (nsec): min=506, max=5818.8k, avg=265924.16,
> > > stdev=223761.59
> > > > > >>> lat (usec): min=6, max=5821, avg=268.19, stdev=223.76
> > > > > >>> clat percentiles (usec):
> > > > > >>> | 1.00th=[ 13], 5.00th=[ 74], 10.00th=[ 87], 20.00th=[
> > > > 115],
> > > > > >>> | 30.00th=[ 139], 40.00th=[ 167], 50.00th=[ 204],
> > > > > >>> 60.00th=[
> > > > 247],
> > > > > >>> | 70.00th=[ 306], 80.00th=[ 388], 90.00th=[ 523],
> > > > > >>> 95.00th=[
> > > > 660],
> > > > > >>> | 99.00th=[ 988], 99.50th=[ 1156], 99.90th=[ 2343],
> > > > > >>> 99.95th=[
> > > > 3195],
> > > > > >>> | 99.99th=[ 4424]
> > > > > >>> bw ( KiB/s): min=448192, max=483104, per=25.00%,
> > > > > >>> avg=476597.10, stdev=6748.78, samples=80
> > > > > >>> iops : min=112048, max=120776, avg=119149.27,
> > > > stdev=1687.20,
> > > > > >>> samples=80
> > > > > >>> lat (nsec) : 750=0.01%
> > > > > >>> lat (usec) : 10=0.52%, 20=1.25%, 50=0.25%, 100=13.38%,
> > > 250=45.13%
> > > > > >>> lat (usec) : 500=28.23%, 750=8.08%, 1000=2.22%
> > > > > >>> lat (msec) : 2=0.82%, 4=0.10%, 10=0.02%
> > > > > >>> cpu : usr=14.08%, sys=36.02%, ctx=1930207, majf=0,
> > > > minf=697
> > > > > >>> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%,
> > > 32=100.0%,
> > > > > >>> >=64=0.0%
> > > > > >>> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%,
> > > 64=0.0%,
> > > > > >>> >=64=0.0%
> > > > > >>> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%,
> > > > > >>> 64=0.0%,
> > > > > >>> >=64=0.0%
> > > > > >>> issued rwts: total=4766095,0,0,0 short=0,0,0,0
> dropped=0,0,0,0
> > > > > >>> latency : target=0, window=0, percentile=100.00%,
> depth=32
> > > > > >>> Run status group 0 (all jobs):
> > > > > >>> MIXED: bw=1862MiB/s (1952MB/s), 1862MiB/s-1862MiB/s
> > > (1952MB/s-
> > > > > >>> 1952MB/s), io=18.2GiB (19.5GB), run=10001-10001msec Disk
> > > > > >>> stats
> > > > > >>> (read/write):
> > > > > >>> nvme2n1: ios=4709190/0, merge=0/0, ticks=4096/0,
> > > > > in_queue=1388752,
> > > > > >>> util=100.00%
> > > > > >>>
> > > > > >>>
> > > > > >>> don(a)donst201:~/fio/single/rr$ sudo
> > > > > >>> /home/don/install/spdk/spdk/scripts/setup.sh
> > > > > >>> Active mountpoints on /dev/nvme0n1, so not binding PCI dev
> > > > > >>> 0000:03:00.0
> > > > > >>> 0000:1c:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > > > >>> 0000:1d:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > > > >>> 0000:5e:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > > > >>> 0000:5f:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > > > >>> 0000:62:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > > > >>> 0000:63:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > > > >>> 0000:64:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > > > >>> 0000:65:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > > > >>> 0000:da:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > > > >>> 0000:db:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > > > >>> 0000:dc:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > > > >>> 0000:dd:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > > > >>> 0000:e0:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > > > >>> 0000:e1:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > > > >>> 0000:e2:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > > > >>> 0000:e3:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > > > >>> 0000:00:04.0 (8086 2021): ioatdma -> uio_pci_generic
> > > > > >>> 0000:00:04.1 (8086 2021): ioatdma -> uio_pci_generic
> > > > > >>> 0000:00:04.2 (8086 2021): ioatdma -> uio_pci_generic
> > > > > >>> 0000:00:04.3 (8086 2021): ioatdma -> uio_pci_generic
> > > > > >>> 0000:00:04.4 (8086 2021): ioatdma -> uio_pci_generic
> > > > > >>> 0000:00:04.5 (8086 2021): ioatdma -> uio_pci_generic
> > > > > >>> 0000:00:04.6 (8086 2021): ioatdma -> uio_pci_generic
> > > > > >>> 0000:00:04.7 (8086 2021): ioatdma -> uio_pci_generic
> > > > > >>> 0000:80:04.0 (8086 2021): ioatdma -> uio_pci_generic
> > > > > >>> 0000:80:04.1 (8086 2021): ioatdma -> uio_pci_generic
> > > > > >>> 0000:80:04.2 (8086 2021): ioatdma -> uio_pci_generic
> > > > > >>> 0000:80:04.3 (8086 2021): ioatdma -> uio_pci_generic
> > > > > >>> 0000:80:04.4 (8086 2021): ioatdma -> uio_pci_generic
> > > > > >>> 0000:80:04.5 (8086 2021): ioatdma -> uio_pci_generic
> > > > > >>> 0000:80:04.6 (8086 2021): ioatdma -> uio_pci_generic
> > > > > >>> 0000:80:04.7 (8086 2021): ioatdma -> uio_pci_generic
> > > > > >>>
> > > > > >>>
> > > > > >>> don(a)donst201:~/fio/single/rr$ cat 1c.ini |grep -v '#'
> > > > > >>> [global]
> > > > > >>> rw=randrw
> > > > > >>> rwmixread=100
> > > > > >>> numjobs=4
> > > > > >>> iodepth=32
> > > > > >>> bs=4k
> > > > > >>> direct=1
> > > > > >>> thread=1
> > > > > >>> time_based=1
> > > > > >>> ramp_time=0
> > > > > >>> runtime=10
> > > > > >>>
> > > > >
> > >
> ioengine=/home/don/install/spdk/spdk/examples/nvme/fio_plugin/fio_pl
> > > > > >>> ug
> > > > > >>> in
> > > > > >>> group_reporting=1
> > > > > >>> unified_rw_reporting=1
> > > > > >>> exitall=1
> > > > > >>> randrepeat=0
> > > > > >>> norandommap=1
> > > > > >>> [0000.1c.00.0]
> > > > > >>> filename=trtype=PCIe traddr=0000.1c.00.0 ns=1
> > > > > >>>
> > > > > >>>
> > > > > >>> don(a)donst201:~/fio/single/rr$ sudo fio 1c.ini
> > > > > >>> 0000.1c.00.0: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W)
> > > > > >>> 4096B-4096B,
> > > > > >>> (T) 4096B-4096B, ioengine=spdk, iodepth=32 ...
> > > > > >>> fio-3.11
> > > > > >>> Starting 4 threads
> > > > > >>> Starting SPDK v18.10 / DPDK 18.08.0 initialization...
> > > > > >>> [ DPDK EAL parameters: fio --no-shconf -c 0x1 -m 512 --file-
> > > > > >>> prefix=spdk_pid3668 ]
> > > > > >>> EAL: Detected 36 lcore(s)
> > > > > >>> EAL: Detected 2 NUMA nodes
> > > > > >>> EAL: No free hugepages reported in hugepages-1048576kB
> > > > > >>> EAL: Probing VFIO support...
> > > > > >>> EAL: PCI device 0000:1c:00.0 on NUMA socket 0
> > > > > >>> EAL: probe driver: 8086:a54 spdk_nvme
> > > > > >>> Jobs: 1 (f=0): [f(1),_(3)][100.0%][eta 00m:00s]
> > > > > >>> 0000.1c.00.0: (groupid=0, jobs=4): err= 0: pid=3709: Mon Nov
> > > > > >>> 5
> > > > > >>> 02:28:29
> > > > > >>> 2018
> > > > > >>> mixed: IOPS=13.2k, BW=51.6MiB/s
> > (54.1MB/s)(517MiB/10011msec)
> > > > > >>> slat (nsec): min=109, max=15344, avg=127.91, stdev=73.44
> > > > > >>> clat (usec): min=187, max=18715, avg=9683.80,
> stdev=650.54
> > > > > >>> lat (usec): min=193, max=18715, avg=9683.93, stdev=650.53
> > > > > >>> clat percentiles (usec):
> > > > > >>> | 1.00th=[ 8717], 5.00th=[ 8848], 10.00th=[ 8979],
> > > > > >>> 20.00th=[
> > > > 9110],
> > > > > >>> | 30.00th=[ 9241], 40.00th=[ 9372], 50.00th=[ 9503],
> > > > > >>> 60.00th=[
> > > > 9765],
> > > > > >>> | 70.00th=[10028], 80.00th=[10159], 90.00th=[10552],
> > > > > 95.00th=[10945],
> > > > > >>> | 99.00th=[11338], 99.50th=[11338], 99.90th=[11731],
> > > > > 99.95th=[13960],
> > > > > >>> | 99.99th=[17695]
> > > > > >>> bw ( KiB/s): min=13080, max=13304, per=25.00%,
> > > > > >>> avg=13214.70, stdev=55.30, samples=80
> > > > > >>> iops : min= 3270, max= 3326, avg=3303.65, stdev=13.84,
> > > > > samples=80
> > > > > >>> lat (usec) : 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
> > > > > >>> lat (msec) : 2=0.01%, 4=0.02%, 10=70.85%, 20=29.11%
> > > > > >>> cpu : usr=100.01%, sys=0.00%, ctx=23, majf=0, minf=0
> > > > > >>> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%,
> > 32=99.9%,
> > > > > >>> >=64=0.0%
> > > > > >>> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%,
> > > 64=0.0%,
> > > > > >>> >=64=0.0%
> > > > > >>> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%,
> > > > > >>> 64=0.0%,
> > > > > >>> >=64=0.0%
> > > > > >>> issued rwts: total=132300,0,0,0 short=0,0,0,0
> dropped=0,0,0,0
> > > > > >>> latency : target=0, window=0, percentile=100.00%,
> depth=32
> > > > > >>> Run status group 0 (all jobs):
> > > > > >>> MIXED: bw=51.6MiB/s (54.1MB/s), 51.6MiB/s-51.6MiB/s
> > > (54.1MB/s-
> > > > > >>> 54.1MB/s), io=517MiB (542MB), run=10011-10011msec
> > > > > >>>
> > > > > >>> _______________________________________________
> > > > > >>> SPDK mailing list
> > > > > >>> SPDK(a)lists.01.org
> > > > > >>> https://lists.01.org/mailman/listinfo/spdk
> > > > > >>> _______________________________________________
> > > > > >>> SPDK mailing list
> > > > > >>> SPDK(a)lists.01.org
> > > > > >>> https://lists.01.org/mailman/listinfo/spdk
> > > > > >>_______________________________________________
> > > > > >>SPDK mailing list
> > > > > >> SPDK(a)lists.01.org
> > > > > >> https://lists.01.org/mailman/listinfo/spdk
> > > > > >>_______________________________________________
> > > > > >>SPDK mailing list
> > > > > >> SPDK(a)lists.01.org
> > > > > >> https://lists.01.org/mailman/listinfo/spdk
> > > > > >
> > > > > >
> > > > > >--
> > > > > >Маров Алексей
> > > > > >_______________________________________________
> > > > > >SPDK mailing list
> > > > > >SPDK(a)lists.01.org
> > > > > >https://lists.01.org/mailman/listinfo/spdk
> > > > > >_______________________________________________
> > > > > >SPDK mailing list
> > > > > >SPDK(a)lists.01.org
> > > > > >https://lists.01.org/mailman/listinfo/spdk
> > > > >
> > > > >
> > > > > --
> > > > > Маров Алексей
> > > > > _______________________________________________
> > > > > SPDK mailing list
> > > > > SPDK(a)lists.01.org
> > > > > https://lists.01.org/mailman/listinfo/spdk
> > > > _______________________________________________
> > > > SPDK mailing list
> > > > SPDK(a)lists.01.org
> > > > https://lists.01.org/mailman/listinfo/spdk
> > > >
> > > --
> > >
> > > Regards,
> > > Andrey
> > > _______________________________________________
> > > SPDK mailing list
> > > SPDK(a)lists.01.org
> > > https://lists.01.org/mailman/listinfo/spdk
> > > _______________________________________________
> > > SPDK mailing list
> > > SPDK(a)lists.01.org
> > > https://lists.01.org/mailman/listinfo/spdk
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
>
>
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [SPDK] FIO NVMe Performance Results
@ 2018-11-05 21:56 Harris, James R
0 siblings, 0 replies; 25+ messages in thread
From: Harris, James R @ 2018-11-05 21:56 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 46705 bytes --]
Hi Joe,
This is all just local PCIe NVMe – no NVMe-oF? Can you make sure you don’t have any other SPDK processes running while you’re doing this? I’m concerned that fio is polling on the same CPU core as an SPDK process, and your performance degradation is due to timeslice sharing.
-Jim
On 11/5/18, 2:42 PM, "SPDK on behalf of Howell, Seth" <spdk-bounces(a)lists.01.org on behalf of seth.howell(a)intel.com> wrote:
I think that the uio_pci_generic vs. vfio thing is unrelated. The only reason I brought it up was because it could have been indicative of a larger configuration difference. However, I ran my test with and without vfio support just to be sure and got the higher performance numbers both times.
As a side note:
It's probably a good thing to have enabled in general when running spdk benchmarks because that is the recommended configuration on any physical machine.
You can enable it in grub by adding the following to the GRUB_CMDLINE_LINUX line of /etc/default.grub: intel_iommu=on
and then running grub2-mkconfig.
-----Original Message-----
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Gruher, Joseph R
Sent: Monday, November 5, 2018 1:39 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] FIO NVMe Performance Results
Attempting to fall back to earlier revisions has not yielded any improvements for me. All tests use the same P4500 physical disk. FIO files are as shown before, running a 4 thread, queue depth 32 per thread, 4KB random read workload.
SPDK 18.10, kernel 4.18.16, FIO 3.11 = 12.9k IOPS SPDK 18.10, kernel 4.16.14, FIO 3.11 = 11.9k IOPS SPDK 18.04, kernel 4.18.16, FIO 3.11 = 12.9k IOPS SPDK 18.04, kernel 4.16.14, FIO 3.11 = 13.0k IOPS SPDK 18.04, kernel 4.16.16, FIO 3.3 = 13.1k IOPS
Kernel NVMe driver, kernel 4.16.16, FIO 3.3 = 477k IOPS Kernel NVMe driver, kernel 4.18.16, FIO 3.11 = 477k IOPS
I am curious if the comment below about uio_pci_generic instead of vfio can lead us anywhere, or maybe there's some OS setting or BIOS setting that needs to be toggled...
For the folks working inside Intel, I can also give you direct access to the test system, if that would help.
Thanks,
Joe
> -----Original Message-----
> From: Gruher, Joseph R
> Sent: Monday, November 5, 2018 11:05 AM
> To: Storage Performance Development Kit <spdk(a)lists.01.org>
> Subject: RE: [SPDK] FIO NVMe Performance Results
>
> I've had good FIO NVMe results in the past we well, with SPDK 18.04 on
> Ubuntu 18.04, with kernel 4.16.14. That was on a different, but
> similar, hardware platform.
>
> I suspect something has changed now in SPDK 18.10 and/or kernel
> 4.18.16. I can try going back to earlier versions of each to see if I
> can recreate earlier good results and find where things went wrong.
>
> We're not running in a VM - this is a bare metal Xeon server with
> Skylake CPUs.
>
> Thanks,
> Joe
>
> > -----Original Message-----
> > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Howell,
> > Seth
> > Sent: Monday, November 5, 2018 10:32 AM
> > To: Storage Performance Development Kit <spdk(a)lists.01.org>
> > Subject: Re: [SPDK] FIO NVMe Performance Results
> >
> > Hi Joseph,
> > I attempted to verify your results on my system. I am using fedora
> > 27 with kernel version 4.17.3 with a p3700 drive. I cloned fio from
> > https://github.com/axboe/fio and checked out the appropriate tag. I
> > got appx 408k iops.
> > Typically we run spdk with the iommu enabled. I noticed that your
> > devices are binding to uio_pci_generic instead of vfio. This
> > shouldn't make any difference at all, but I just want to make sure
> > you aren't running in a VM or something like that and passing
> > through your NVMe drives and that's why you aren't using the IOMMU.
> > That could explain a deeper configuration difference that could help
> > us know where to look for
> the disparity.
> >
> > Please see the contents of my configuration files and output below:
> >
> > Test.ini:
> > [global]
> > rw=randrw
> > rwmixread=100
> > numjobs=4
> > iodepth=32
> > bs=4k
> > direct=1
> > thread=1
> > time_based=1
> > ramp_time=0
> > runtime=10
> > ioengine=spdk_bdev
> > spdk_conf=./nvme.conf
> > group_reporting=1
> > unified_rw_reporting=1
> > exitall=1
> > randrepeat=0
> > norandommap=1
> > [0000.81.00.0]
> > filename=Nvme0n1
> >
> > nvme.conf:
> > [Nvme]
> > TransportId "trtype:PCIe traddr:0000.81.00.0" Nvme0
> >
> > Sudo
> >
> LD_PRELOAD=/home/sethhowe/Desktop/Development_Folders/spdk/exam
> > ples/bdev/fio_plugin/fio_plugin ./fio ./test.ini >./out,txt 2>&1
> >
> > Out.txt:
> > 0000.81.00.0: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B,
> > (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=32 ...
> > fio-3.11
> > Starting 4 threads
> > Starting SPDK v18.10 / DPDK 18.08.0 initialization...
> > [ DPDK EAL parameters: fio --no-shconf -c 0x1
> > --file-prefix=spdk_pid782339 ]
> > EAL: Detected 16 lcore(s)
> > EAL: Detected 2 NUMA nodes
> > EAL: No free hugepages reported in hugepages-1048576kB
> > EAL: Probing VFIO support...
> > EAL: VFIO support initialized
> > EAL: PCI device 0000:81:00.0 on NUMA socket 1
> > EAL: probe driver: 8086:953 spdk_nvme
> >
> > 0000.81.00.0: (groupid=0, jobs=4): err= 0: pid=782359: Mon Nov 5
> > 10:58:59
> > 2018
> > mixed: IOPS=410k, BW=1600MiB/s (1678MB/s)(15.6GiB/10001msec)
> > slat (nsec): min=179, max=19985, avg=245.97, stdev=122.96
> > clat (usec): min=8, max=1821, avg=311.82, stdev=167.14
> > lat (usec): min=8, max=1822, avg=312.06, stdev=167.14
> > clat percentiles (usec):
> > | 1.00th=[ 24], 5.00th=[ 48], 10.00th=[ 78], 20.00th=[ 141],
> > | 30.00th=[ 200], 40.00th=[ 258], 50.00th=[ 314], 60.00th=[ 371],
> > | 70.00th=[ 429], 80.00th=[ 486], 90.00th=[ 545], 95.00th=[ 570],
> > | 99.00th=[ 594], 99.50th=[ 594], 99.90th=[ 627], 99.95th=[ 652],
> > | 99.99th=[ 676]
> > bw ( KiB/s): min=407264, max=416040, per=25.01%, avg=409742.55,
> > stdev=1337.38, samples=76
> > iops : min=101816, max=104010, avg=102435.61, stdev=334.41,
> > samples=76
> > lat (usec) : 10=0.01%, 20=0.46%, 50=4.88%, 100=8.41%, 250=25.29%
> > lat (usec) : 500=43.15%, 750=17.81%, 1000=0.01%
> > lat (msec) : 2=0.01%
> > cpu : usr=99.78%, sys=0.00%, ctx=26, majf=0, minf=0
> > IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.9%,
> > >=64=0.0%
> > submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
> > >=64=0.0%
> > complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.1%, 32=0.1%,
> > 64=0.0%,
> > >=64=0.0%
> > issued rwts: total=4096867,0,0,0 short=0,0,0,0 dropped=0,0,0,0
> > latency : target=0, window=0, percentile=100.00%, depth=32
> >
> > Run status group 0 (all jobs):
> > MIXED: bw=1600MiB/s (1678MB/s), 1600MiB/s-1600MiB/s (1678MB/s-
> > 1678MB/s), io=15.6GiB (16.8GB), run=10001-10001msec
> >
> > Thanks,
> >
> > Seth
> >
> >
> >
> > -----Original Message-----
> > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Andrey
> > Kuzmin
> > Sent: Monday, November 5, 2018 9:48 AM
> > To: Storage Performance Development Kit <spdk(a)lists.01.org>
> > Subject: Re: [SPDK] FIO NVMe Performance Results
> >
> > On Mon, Nov 5, 2018, 18:29 Gruher, Joseph R
> > <joseph.r.gruher(a)intel.com>
> > wrote:
> >
> > > My FIO version is v3.11.
> > >
> > > I tried the bdev FIO plugin as suggested, but performance is about
> > > the same as the other SPDK FIO plugin, I observed only 10.4K IOPS
> > > in a quick 4KB random read test. See configuration and results below.
> > >
> >
> > As 12 ms median read latency for an nvme ssd is far beyond anything
> > reasonable, why don't you just perf this run, to see where the time
> > is being spent?
> >
> > Regards,
> > Andrey
> >
> > >
> > >
> > > don(a)donst201:~/fio/single/rr$ cat nvme.conf [Nvme] TransportId
> > > "trtype:PCIe traddr:0000.1c.00.0" Nvme0
> > >
> > >
> > > don(a)donst201:~/fio/single/rr$ cat 1c-bdev.ini |grep -v '#'
> > > [global]
> > > rw=randrw
> > > rwmixread=100
> > > numjobs=4
> > > iodepth=32
> > > bs=4k
> > > direct=1
> > > thread=1
> > > time_based=1
> > > ramp_time=0
> > > runtime=10
> > > ioengine=spdk_bdev
> > > spdk_conf=./nvme.conf
> > > group_reporting=1
> > > unified_rw_reporting=1
> > > exitall=1
> > > randrepeat=0
> > > norandommap=1
> > > [0000.1c.00.0]
> > > filename=Nvme0n1
> > >
> > >
> > > don(a)donst201:~/fio/single/rr$ sudo
> > >
> >
> LD_PRELOAD=/home/don/install/spdk/spdk/examples/bdev/fio_plugin/fio_p
> > l
> > > ugin
> > > fio 1c-bdev.ini
> > > 0000.1c.00.0: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W)
> > > 4096B-4096B,
> > > (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=32 ...
> > > fio-3.11
> > > Starting 4 threads
> > > Starting SPDK v18.10 / DPDK 18.08.0 initialization...
> > > [ DPDK EAL parameters: fio --no-shconf -c 0x1
> > > --file-prefix=spdk_pid4865 ]
> > > EAL: Detected 36 lcore(s)
> > > EAL: Detected 2 NUMA nodes
> > > EAL: No free hugepages reported in hugepages-1048576kB
> > > EAL: Probing VFIO support...
> > > EAL: PCI device 0000:03:00.0 on NUMA socket 0
> > > EAL: probe driver: 8086:f1a5 spdk_nvme
> > > EAL: PCI device 0000:1c:00.0 on NUMA socket 0
> > > EAL: probe driver: 8086:a54 spdk_nvme
> > > EAL: PCI device 0000:1d:00.0 on NUMA socket 0
> > > EAL: probe driver: 8086:a54 spdk_nvme
> > > EAL: PCI device 0000:5e:00.0 on NUMA socket 0
> > > EAL: probe driver: 8086:a54 spdk_nvme
> > > EAL: PCI device 0000:5f:00.0 on NUMA socket 0
> > > EAL: probe driver: 8086:a54 spdk_nvme
> > > EAL: PCI device 0000:62:00.0 on NUMA socket 0
> > > EAL: probe driver: 8086:a54 spdk_nvme
> > > EAL: PCI device 0000:63:00.0 on NUMA socket 0
> > > EAL: probe driver: 8086:a54 spdk_nvme
> > > EAL: PCI device 0000:64:00.0 on NUMA socket 0
> > > EAL: probe driver: 8086:a54 spdk_nvme
> > > EAL: PCI device 0000:65:00.0 on NUMA socket 0
> > > EAL: probe driver: 8086:a54 spdk_nvme
> > > EAL: PCI device 0000:da:00.0 on NUMA socket 1
> > > EAL: probe driver: 8086:a54 spdk_nvme
> > > EAL: PCI device 0000:db:00.0 on NUMA socket 1
> > > EAL: probe driver: 8086:a54 spdk_nvme
> > > EAL: PCI device 0000:dc:00.0 on NUMA socket 1
> > > EAL: probe driver: 8086:a54 spdk_nvme
> > > EAL: PCI device 0000:dd:00.0 on NUMA socket 1
> > > EAL: probe driver: 8086:a54 spdk_nvme
> > > EAL: PCI device 0000:e0:00.0 on NUMA socket 1
> > > EAL: probe driver: 8086:a54 spdk_nvme
> > > EAL: PCI device 0000:e1:00.0 on NUMA socket 1
> > > EAL: probe driver: 8086:a54 spdk_nvme
> > > EAL: PCI device 0000:e2:00.0 on NUMA socket 1
> > > EAL: probe driver: 8086:a54 spdk_nvme
> > > EAL: PCI device 0000:e3:00.0 on NUMA socket 1
> > > EAL: probe driver: 8086:a54 spdk_nvme
> > > Jobs: 4 (f=4): [r(4)][100.0%][r=45.7MiB/s][r=11.7k IOPS][eta
> > > 00m:00s]
> > > 0000.1c.00.0: (groupid=0, jobs=4): err= 0: pid=4906: Mon Nov 5
> > > 15:23:47
> > > 2018
> > > mixed: IOPS=10.4k, BW=40.5MiB/s (42.5MB/s)(406MiB/10012msec)
> > > slat (nsec): min=201, max=15831, avg=238.11, stdev=148.78
> > > clat (usec): min=124, max=31244, avg=12330.69, stdev=1697.98
> > > lat (usec): min=130, max=31244, avg=12330.93, stdev=1697.98
> > > clat percentiles (usec):
> > > | 1.00th=[ 9634], 5.00th=[10159], 10.00th=[10552], 20.00th=[10945],
> > > | 30.00th=[11338], 40.00th=[11731], 50.00th=[12125],
> > 60.00th=[12518],
> > > | 70.00th=[13042], 80.00th=[13566], 90.00th=[14353],
> > 95.00th=[15008],
> > > | 99.00th=[16712], 99.50th=[18744], 99.90th=[27395],
> > 99.95th=[29754],
> > > | 99.99th=[30278]
> > > bw ( KiB/s): min= 8784, max=11952, per=24.99%, avg=10375.53,
> > > stdev=929.68, samples=80
> > > iops : min= 2196, max= 2988, avg=2593.85, stdev=232.41,
> > > samples=80
> > > lat (usec) : 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
> > > lat (msec) : 2=0.01%, 4=0.02%, 10=2.71%, 20=96.91%, 50=0.33%
> > > cpu : usr=99.99%, sys=0.01%, ctx=28, majf=0, minf=0
> > > IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.9%,
> > > >=64=0.0%
> > > submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
> > > >=64=0.0%
> > > complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%,
> > > 64=0.0%,
> > > >=64=0.0%
> > > issued rwts: total=103907,0,0,0 short=0,0,0,0 dropped=0,0,0,0
> > > latency : target=0, window=0, percentile=100.00%, depth=32
> > > Run status group 0 (all jobs):
> > > MIXED: bw=40.5MiB/s (42.5MB/s), 40.5MiB/s-40.5MiB/s
> > > (42.5MB/s-42.5MB/s), io=406MiB (426MB), run=10012-10012msec
> > >
> > >
> > > Thanks,
> > > Joe
> > >
> > > > -----Original Message-----
> > > > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of ?????
> > ???????
> > > > Sent: Monday, November 5, 2018 12:45 AM
> > > > To: Storage Performance Development Kit <spdk(a)lists.01.org>
> > > > Subject: Re: [SPDK] FIO NVMe Performance Results
> > > >
> > > > Ok, Do you see some performance issues using general bdev fio
> > > > plugin
> > > with
> > > > you configuration ?
> > > >
> > > > Best Regards
> > > > Alex
> > > >
> > > >
> > > > >Понедельник, 5 ноября 2018, 11:39 +03:00 от "Cao, Gang"
> > > > <gang.cao(a)intel.com>:
> > > > >
> > > > >Yes. This is for the bdev version of SPDK FIO plugin and you
> > > > >can assign
> > > the
> > > > conf file as the parameter in the FIO configuration.
> > > > >
> > > > >Previously, it's another SPDK FIO plugin for the user space
> > > > >NVMe driver
> > > > only.
> > > > >
> > > > >These are two different FIO plugins.
> > > > >
> > > > >Thanks,
> > > > >Gang
> > > > >
> > > > >-----Original Message-----
> > > > >From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of
> > > > >Маров Алексей
> > > > >Sent: Monday, November 5, 2018 4:32 PM
> > > > >To: Storage Performance Development Kit < spdk(a)lists.01.org >
> > > > >Subject: Re: [SPDK] FIO NVMe Performance Results
> > > > >
> > > > >Sorry for incomplete message I have sent before
> > > > >
> > > > >Hello Gang.
> > > > >
> > > > >I just curious, why would you run fio with SPDK using this conf
> > > > >file,
> > > with is a
> > > > bit different from what is suggested in here
> > https://github.com/spdk/spdk/tree/master/examples/bdev/fio_plugin
> > > > .
> > > > >
> > > > >I mean may be you should create conf file for spdk like this one :
> > > > >
> > > > >nvme.conf :
> > > > >[Nvme]
> > > > > TransportId "trtype:PCIe traddr:0000.1c.00.0" Nvme0
> > > > >
> > > > >and provide it in fio config.
> > > > >
> > > > >fio.ini :
> > > > >[global]
> > > > >ioengine=spdk_bdev
> > > > >spdk_conf=....../nvme.conf
> > > > >rw=randrw
> > > > >rwmixread=100
> > > > >numjobs=4
> > > > >iodepth=32
> > > > >bs=4k
> > > > >direct=1
> > > > >thread=1
> > > > >time_based=1
> > > > >ramp_time=0
> > > > >runtime=10
> > > > >group_reporting=1
> > > > >unified_rw_reporting=1
> > > > >exitall=1
> > > > >randrepeat=0
> > > > >norandommap=1
> > > > >
> > > > >[0000:06:00.0]
> > > > >filename=Nvme0n1
> > > > >
> > > > >and run like this
> > > > >
> > > > ># LD_PRELOAD=examples/bdev/fio_plugin/fio_plugin fio
> > > > >examples/bdev/fio_plugin/fio.ini
> > > > >
> > > > >Just to note. The recommended version of fio for SPDK is 3.3
> > > > >
> > > > >Best Regards
> > > > >Alex
> > > > >
> > > > >
> > > > >>Понедельник, 5 ноября 2018, 9:53 +03:00 от "Cao, Gang" <
> > > > gang.cao(a)intel.com >:
> > > > >>
> > > > >>Thanks. This is interesting especially for your finding on
> > > > >>queue depth
> > > 1
> > > > case. We are trying to reproduce this issue locally same as your
> > > > configuration.
> > > > >>
> > > > >>Thanks,
> > > > >>Gang
> > > > >>
> > > > >>-----Original Message-----
> > > > >>From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of
> > > > >>Gruher, Joseph R
> > > > >>Sent: Monday, November 5, 2018 1:32 PM
> > > > >>To: Storage Performance Development Kit < spdk(a)lists.01.org >
> > > > >>Subject: Re: [SPDK] FIO NVMe Performance Results
> > > > >>
> > > > >>Interesting catch. It seems like FIO is actually not
> > > > >>accurately
> > > reporting the
> > > > number of jobs in the report out for the SPDK case. If I
> > > > disable the group_reporting parameter I do get a printout for
> > > > four separate jobs in
> > > the
> > > > SPDK case, so I think it is really running four jobs there, and
> > > > just
> > > summarizing
> > > > incorrectly. I also see four separate CPU cores go to 100% when
> > > > I run
> > > that
> > > > FIO test case, indicating four jobs are active.
> > > > >>
> > > > >>Also, I don't think would explain the difference in performance.
> > > Using an
> > > > aggregate queue depth of 32 versus 128 shouldn't lead to a drop
> > > > in performance from 377K to 13K.
> > > > >>
> > > > >>I retested SPDK with multiple separate job descriptions as you
> > > suggested
> > > > anyway, using four separate jobs instead of relying on the
> > > > numjobs=4 parameter, and the results are below. In that SPDK
> > > > test with four
> > > separate
> > > > FIO job statements, each with queue depth 32, I still only
> > > > achieved 12.5K IOPS.
> > > > >>
> > > > >>I then re-ran the kernel NVMe driver test case with only a
> > > > >>single job
> > > for
> > > > comparison, and even with only 1 job and queue depth 32 it still
> > > > achieves 254K IOPS.
> > > > >>
> > > > >>We might be on the right track with a problem around queue
> > > > >>depth
> > > > however. If I run the kernel NVMe driver case with just 1 job
> > > > and queue depth of only 1, I then get 11.2K IOPS, which is
> > > > around the performance
> > > I get
> > > > in the SPDK test case. Perhaps the SPDK case is not actually
> > > > queuing
> > > IOs on
> > > > the device for some reason?
> > > > >>
> > > > >>
> > > > >>don(a)donst201:~/fio/single/rr$ cat nvme2n1.ini |grep -v '#'
> > > > >>[global]
> > > > >>rw=randrw
> > > > >>rwmixread=100
> > > > >>numjobs=1
> > > > >>iodepth=32
> > > > >>bs=4k
> > > > >>direct=1
> > > > >>thread=1
> > > > >>time_based=1
> > > > >>ramp_time=0
> > > > >>runtime=10
> > > > >>ioengine=libaio
> > > > >>group_reporting=1
> > > > >>unified_rw_reporting=1
> > > > >>exitall=1
> > > > >>randrepeat=0
> > > > >>norandommap=1
> > > > >>[nvme2n1]
> > > > >>filename=/dev/nvme2n1
> > > > >>
> > > > >>
> > > > >>don(a)donst201:~/fio/single/rr$ sudo fio nvme2n1.ini
> > > > >>nvme2n1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W)
> > > > >>4096B-4096B,
> > > > >>(T) 4096B-4096B, ioengine=libaio, iodepth=32
> > > > >>fio-3.11
> > > > >>Starting 1 thread
> > > > >>Jobs: 1 (f=1): [r(1)][100.0%][r=1025MiB/s][r=262k IOPS][eta
> > > > >>00m:00s]
> > > > >>nvme2n1: (groupid=0, jobs=1): err= 0: pid=2422: Mon Nov 5
> > > > >>05:19:13
> > > > >>2018
> > > > >> mixed: IOPS=254k, BW=992MiB/s
> (1040MB/s)(9922MiB/10001msec)
> > > > >> slat (nsec): min=1185, max=319052, avg=1611.64, stdev=808.40
> > > > >> clat (usec): min=2, max=4743, avg=124.02, stdev=121.00
> > > > >> lat (usec): min=6, max=4744, avg=125.68, stdev=121.02
> > > > >> clat percentiles (usec):
> > > > >> | 1.00th=[ 9], 5.00th=[ 56], 10.00th=[ 61], 20.00th=[
> > > 73],
> > > > >> | 30.00th=[ 80], 40.00th=[ 91], 50.00th=[ 110], 60.00th=[
> > > 118],
> > > > >> | 70.00th=[ 130], 80.00th=[ 157], 90.00th=[ 202],
> > > > >>95.00th=[ 253],
> > > > >> | 99.00th=[ 408], 99.50th=[ 498], 99.90th=[ 1795],
> > > > >>99.95th=[ 3064],
> > > > >> | 99.99th=[ 3785]
> > > > >> bw ( KiB/s): min=612240, max=1078792, per=100.00%,
> > > > >>avg=1015996.40,
> > > > stdev=106032.96, samples=20
> > > > >> iops : min=153060, max=269698, avg=253998.90,
> > stdev=26508.40,
> > > > samples=20
> > > > >> lat (usec) : 4=0.01%, 10=1.47%, 20=1.19%, 50=0.17%, 100=42.03%
> > > > >> lat (usec) : 250=49.95%, 500=4.70%, 750=0.28%, 1000=0.06%
> > > > >> lat (msec) : 2=0.06%, 4=0.09%, 10=0.01%
> > > > >> cpu : usr=21.45%, sys=51.11%, ctx=530348, majf=0, minf=139
> > > > >> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%,
> 32=100.0%,
> > > > >=64=0.0%
> > > > >> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%,
> 64=0.0%,
> > > > >=64=0.0%
> > > > >> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%,
> > > > >> 64=0.0%,
> > > > >>>=64=0.0%
> > > > >> issued rwts: total=2540023,0,0,0 short=0,0,0,0 dropped=0,0,0,0
> > > > >> latency : target=0, window=0, percentile=100.00%, depth=32
> > > > >>Run status group 0 (all jobs):
> > > > >> MIXED: bw=992MiB/s (1040MB/s), 992MiB/s-992MiB/s (1040MB/s-
> > > > 1040MB/s), io=9922MiB (10.4GB), run=10001-10001msec Disk stats
> > > > (read/write):
> > > > >> nvme2n1: ios=2514356/0, merge=0/0, ticks=400/0,
> > in_queue=297612,
> > > > >>util=99.06%
> > > > >>
> > > > >>
> > > > >>don(a)donst201:~/fio/single/rr$ cat 1c.ini |grep -v '#'
> > > > >>[global]
> > > > >>rw=randrw
> > > > >>rwmixread=100
> > > > >>numjobs=1
> > > > >>iodepth=32
> > > > >>bs=4k
> > > > >>direct=1
> > > > >>thread=1
> > > > >>time_based=1
> > > > >>ramp_time=0
> > > > >>runtime=10
> > > >
> > >>ioengine=/home/don/install/spdk/spdk/examples/nvme/fio_plugin/fio_
> > > > >>pl
> > > > ug
> > > > >>in
> > > > >>group_reporting=1
> > > > >>unified_rw_reporting=1
> > > > >>exitall=1
> > > > >>randrepeat=0
> > > > >>norandommap=1
> > > > >>[0000.1c.00.0-1]
> > > > >>filename=trtype=PCIe traddr=0000.1c.00.0 ns=1 [0000.1c.00.0-2]
> > > > >>filename=trtype=PCIe traddr=0000.1c.00.0 ns=1 [0000.1c.00.0-3]
> > > > >>filename=trtype=PCIe traddr=0000.1c.00.0 ns=1 [0000.1c.00.0-4]
> > > > >>filename=trtype=PCIe traddr=0000.1c.00.0 ns=1
> > > > >>
> > > > >>
> > > > >>don(a)donst201:~/fio/single/rr$ sudo fio 1c.ini
> > > > >>0000.1c.00.0-1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W)
> > > > >>4096B-4096B,
> > > > >>(T) 4096B-4096B, ioengine=spdk, iodepth=32
> > > > >>0000.1c.00.0-2: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W)
> > > > >>4096B-4096B,
> > > > >>(T) 4096B-4096B, ioengine=spdk, iodepth=32
> > > > >>0000.1c.00.0-3: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W)
> > > > >>4096B-4096B,
> > > > >>(T) 4096B-4096B, ioengine=spdk, iodepth=32
> > > > >>0000.1c.00.0-4: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W)
> > > > >>4096B-4096B,
> > > > >>(T) 4096B-4096B, ioengine=spdk, iodepth=32
> > > > >>fio-3.11
> > > > >>Starting 4 threads
> > > > >>Starting SPDK v18.10 / DPDK 18.08.0 initialization...
> > > > >>[ DPDK EAL parameters: fio --no-shconf -c 0x1 -m 512
> > > > >>--file-prefix=spdk_pid4270 ]
> > > > >>EAL: Detected 36 lcore(s)
> > > > >>EAL: Detected 2 NUMA nodes
> > > > >>EAL: No free hugepages reported in hugepages-1048576kB
> > > > >>EAL: Probing VFIO support...
> > > > >>EAL: PCI device 0000:1c:00.0 on NUMA socket 0
> > > > >>EAL: probe driver: 8086:a54 spdk_nvme
> > > > >>Jobs: 1 (f=0): [_(3),f(1)][100.0%][eta 00m:00s]
> > > > >>0000.1c.00.0-1: (groupid=0, jobs=4): err= 0: pid=4310: Mon Nov
> > > > >>5
> > > > >>05:16:03 2018
> > > > >> mixed: IOPS=12.5k, BW=48.0MiB/s (51.4MB/s)(490MiB/10010msec)
> > > > >> slat (nsec): min=98, max=15092, avg=120.02, stdev=90.07
> > > > >> clat (usec): min=55, max=19757, avg=10207.22, stdev=884.06
> > > > >> lat (usec): min=60, max=19757, avg=10207.34, stdev=884.06
> > > > >> clat percentiles (usec):
> > > > >> | 1.00th=[ 8979], 5.00th=[ 9110], 10.00th=[ 9372],
> > > > >>20.00th=[ 9503],
> > > > >> | 30.00th=[ 9765], 40.00th=[ 9896], 50.00th=[10159],
> > > > >>60.00th=[10290],
> > > > >> | 70.00th=[10552], 80.00th=[10814], 90.00th=[11207],
> > > > >>95.00th=[11600],
> > > > >> | 99.00th=[12911], 99.50th=[14353], 99.90th=[17695],
> > > > >>99.95th=[18744],
> > > > >> | 99.99th=[19006]
> > > > >> bw ( KiB/s): min=12008, max=12976, per=25.00%,
> > > > >>avg=12535.70,
> > > > stdev=306.81, samples=80
> > > > >> iops : min= 3002, max= 3244, avg=3133.85, stdev=76.77,
> > > samples=80
> > > > >> lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%,
> > 1000=0.01%
> > > > >> lat (msec) : 2=0.01%, 4=0.02%, 10=45.13%, 20=54.83%
> > > > >> cpu : usr=100.01%, sys=0.00%, ctx=27, majf=0, minf=0
> > > > >> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.9%,
> > > > >=64=0.0%
> > > > >> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%,
> 64=0.0%,
> > > > >=64=0.0%
> > > > >> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%,
> > > > >> 64=0.0%,
> > > > >>>=64=0.0%
> > > > >> issued rwts: total=125506,0,0,0 short=0,0,0,0 dropped=0,0,0,0
> > > > >> latency : target=0, window=0, percentile=100.00%, depth=32
> > > > >>Run status group 0 (all jobs):
> > > > >> MIXED: bw=48.0MiB/s (51.4MB/s), 48.0MiB/s-48.0MiB/s
> > > > >>(51.4MB/s-51.4MB/s), io=490MiB (514MB), run=10010-10010msec
> > > > >>
> > > > >>
> > > > >>don(a)donst201:~/fio/single/rr$ cat nvme2n1.ini |grep -v '#'
> > > > >>[global]
> > > > >>rw=randrw
> > > > >>rwmixread=100
> > > > >>numjobs=1
> > > > >>iodepth=1
> > > > >>bs=4k
> > > > >>direct=1
> > > > >>thread=1
> > > > >>time_based=1
> > > > >>ramp_time=0
> > > > >>runtime=10
> > > > >>ioengine=libaio
> > > > >>group_reporting=1
> > > > >>unified_rw_reporting=1
> > > > >>exitall=1
> > > > >>randrepeat=0
> > > > >>norandommap=1
> > > > >>[nvme2n1]
> > > > >>filename=/dev/nvme2n1
> > > > >>
> > > > >>
> > > > >>don(a)donst201:~/fio/single/rr$ sudo fio nvme2n1.ini
> > > > >>nvme2n1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W)
> > > > >>4096B-4096B,
> > > > >>(T) 4096B-4096B, ioengine=libaio, iodepth=1
> > > > >>fio-3.11
> > > > >>Starting 1 thread
> > > > >>Jobs: 1 (f=1): [r(1)][100.0%][r=43.6MiB/s][r=11.2k IOPS][eta
> > > > >>00m:00s]
> > > > >>nvme2n1: (groupid=0, jobs=1): err= 0: pid=2525: Mon Nov 5
> > > > >>05:22:57
> > > > >>2018
> > > > >> mixed: IOPS=11.2k, BW=43.6MiB/s (45.7MB/s)(436MiB/10001msec)
> > > > >> slat (nsec): min=4848, max=208595, avg=5126.33, stdev=771.27
> > > > >> clat (nsec): min=1845, max=3405.8k, avg=83004.99,
> stdev=33577.36
> > > > >> lat (usec): min=15, max=3411, avg=88.26, stdev=33.59
> > > > >> clat percentiles (usec):
> > > > >> | 1.00th=[ 15], 5.00th=[ 58], 10.00th=[ 59], 20.00th=[
> > > 59],
> > > > >> | 30.00th=[ 60], 40.00th=[ 77], 50.00th=[ 77], 60.00th=[
> > > 78],
> > > > >> | 70.00th=[ 111], 80.00th=[ 113], 90.00th=[ 113],
> > > > >>95.00th=[ 115],
> > > > >> | 99.00th=[ 186], 99.50th=[ 186], 99.90th=[ 190],
> > > > >>99.95th=[ 192],
> > > > >> | 99.99th=[ 375]
> > > > >> bw ( KiB/s): min=43816, max=45016, per=99.98%,
> > > > >>avg=44639.58,
> > > > stdev=312.88, samples=19
> > > > >> iops : min=10954, max=11254, avg=11159.89, stdev=78.22,
> > > > samples=19
> > > > >> lat (usec) : 2=0.01%, 20=2.83%, 50=0.01%, 100=64.57%,
> 250=32.59%
> > > > >> lat (usec) : 500=0.01%, 750=0.01%
> > > > >> lat (msec) : 2=0.01%, 4=0.01%
> > > > >> cpu : usr=4.02%, sys=10.33%, ctx=111630, majf=0, minf=12
> > > > >> IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%,
> 32=0.0%,
> > > > >=64=0.0%
> > > > >> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%,
> 64=0.0%,
> > > > >=64=0.0%
> > > > >> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%,
> > > > >> 64=0.0%,
> > > > >>>=64=0.0%
> > > > >> issued rwts: total=111631,0,0,0 short=0,0,0,0 dropped=0,0,0,0
> > > > >> latency : target=0, window=0, percentile=100.00%, depth=1
> > > > >>Run status group 0 (all jobs):
> > > > >> MIXED: bw=43.6MiB/s (45.7MB/s), 43.6MiB/s-43.6MiB/s
> (45.7MB/s-
> > > > 45.7MB/s), io=436MiB (457MB), run=10001-10001msec Disk stats
> > > > (read/write):
> > > > >> nvme2n1: ios=110275/0, merge=0/0, ticks=0/0, in_queue=8596,
> > > > >>util=86.22%
> > > > >>
> > > > >>
> > > > >>Thanks,
> > > > >>Joe
> > > > >>
> > > > >>> -----Original Message-----
> > > > >>> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of
> > > > >>> Cao, Gang
> > > > >>> Sent: Sunday, November 4, 2018 8:40 PM
> > > > >>> To: Storage Performance Development Kit < spdk(a)lists.01.org
> > > > >>> >
> > > > >>> Subject: Re: [SPDK] FIO NVMe Performance Results
> > > > >>>
> > > > >>> For the SPDK plugin, from the below result, looks like only
> > > > >>> 1 job started with
> > > > >>> 32 queue depth:
> > > > >>>
> > > > >>> Jobs: 1 (f=0): [f(1),_(3)][100.0%][eta 00m:00s] <<<<
> > > > >>>
> > > > >>> Try to update the FIO configuration file as
> > > > >>>
> > > > >>> [job1]
> > > > >>> filename=trtype=PCIe traddr=0000.1c.00.0 ns=1 [job2]
> > > > >>> filename=trtype=PCIe
> > > > >>> traddr=0000.1c.00.0 ns=1 ....
> > > > >>>
> > > > >>> The kernel FIO is with 4 jobs and each job of 32 queue depth.
> > > > >>> For SPDK, you can configure as the same, or just configure
> > > > >>> two jobs and each job of 64 queue depth. The overall queue
> > > > >>> depth to the device is
> > > > same of 128.
> > > > >>>
> > > > >>> Thanks,
> > > > >>> Gang
> > > > >>>
> > > > >>> -----Original Message-----
> > > > >>> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of
> > > > >>> Gruher, Joseph R
> > > > >>> Sent: Monday, November 5, 2018 10:49 AM
> > > > >>> To: Storage Performance Development Kit < spdk(a)lists.01.org
> > > > >>> >
> > > > >>> Subject: [SPDK] FIO NVMe Performance Results
> > > > >>>
> > > > >>> Hi folks-
> > > > >>>
> > > > >>> I'm testing SPDK 18.10 on Ubuntu 18.04 with kernel 4.18.16.
> > > > >>> I used FIO with the kernel NVMe driver to measure the
> > > > >>> performance of a local (PCIe
> > > > >>> attached) Intel P4500 NVMe device on a 4KB random read
> > > > >>> workload and obtained 477K IOPS, roughly in line with the
> > > > >>> drive
> spec.
> > > > >>> Then I tested the same drive with the SPDK FIO plugin and
> > > > >>> only achieved 13K IOPS. The FIO test files and the results
> > > > >>> are pasted below. Any ideas where I'm going wrong here?
> > > > >>>
> > > > >>> Thanks!
> > > > >>>
> > > > >>>
> > > > >>> don(a)donst201:~/fio/single/rr$ ls
> > > > >>> /sys/devices/pci0000:17/0000:17:02.0/0000:1c:00.0/nvme/nvme1
> > > > >>> / address cntlid dev device firmware_rev model nvme2n1
> > > > >>> power rescan_controller reset_controller serial state
> > > > >>> subsysnqn subsystem transport uevent
> > > > >>>
> > > > >>>
> > > > >>> don(a)donst201:~/fio/single/rr$ cat nvme2n1.ini |grep -v '#'
> > > > >>> [global]
> > > > >>> rw=randrw
> > > > >>> rwmixread=100
> > > > >>> numjobs=4
> > > > >>> iodepth=32
> > > > >>> bs=4k
> > > > >>> direct=1
> > > > >>> thread=1
> > > > >>> time_based=1
> > > > >>> ramp_time=0
> > > > >>> runtime=10
> > > > >>> ioengine=libaio
> > > > >>> group_reporting=1
> > > > >>> unified_rw_reporting=1
> > > > >>> exitall=1
> > > > >>> randrepeat=0
> > > > >>> norandommap=1
> > > > >>> [nvme2n1]
> > > > >>> filename=/dev/nvme2n1
> > > > >>>
> > > > >>>
> > > > >>> don(a)donst201:~/fio/single/rr$ sudo fio nvme2n1.ini
> > > > >>> nvme2n1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W)
> > > > >>> 4096B-4096B,
> > > > >>> (T) 4096B-4096B, ioengine=libaio, iodepth=32 ...
> > > > >>> fio-3.11
> > > > >>> Starting 4 threads
> > > > >>> Jobs: 4 (f=4): [r(4)][100.0%][r=1874MiB/s][r=480k IOPS][eta
> > > > >>> 00m:00s]
> > > > >>> nvme2n1: (groupid=0, jobs=4): err= 0: pid=2575: Mon Nov 5
> > > > >>> 02:26:25
> > > > 2018
> > > > >>> mixed: IOPS=477k, BW=1862MiB/s
> > (1952MB/s)(18.2GiB/10001msec)
> > > > >>> slat (nsec): min=1295, max=333088, avg=2216.35, stdev=927.89
> > > > >>> clat (nsec): min=506, max=5818.8k, avg=265924.16,
> > stdev=223761.59
> > > > >>> lat (usec): min=6, max=5821, avg=268.19, stdev=223.76
> > > > >>> clat percentiles (usec):
> > > > >>> | 1.00th=[ 13], 5.00th=[ 74], 10.00th=[ 87], 20.00th=[
> > > 115],
> > > > >>> | 30.00th=[ 139], 40.00th=[ 167], 50.00th=[ 204],
> > > > >>> 60.00th=[
> > > 247],
> > > > >>> | 70.00th=[ 306], 80.00th=[ 388], 90.00th=[ 523],
> > > > >>> 95.00th=[
> > > 660],
> > > > >>> | 99.00th=[ 988], 99.50th=[ 1156], 99.90th=[ 2343],
> > > > >>> 99.95th=[
> > > 3195],
> > > > >>> | 99.99th=[ 4424]
> > > > >>> bw ( KiB/s): min=448192, max=483104, per=25.00%,
> > > > >>> avg=476597.10, stdev=6748.78, samples=80
> > > > >>> iops : min=112048, max=120776, avg=119149.27,
> > > stdev=1687.20,
> > > > >>> samples=80
> > > > >>> lat (nsec) : 750=0.01%
> > > > >>> lat (usec) : 10=0.52%, 20=1.25%, 50=0.25%, 100=13.38%,
> > 250=45.13%
> > > > >>> lat (usec) : 500=28.23%, 750=8.08%, 1000=2.22%
> > > > >>> lat (msec) : 2=0.82%, 4=0.10%, 10=0.02%
> > > > >>> cpu : usr=14.08%, sys=36.02%, ctx=1930207, majf=0,
> > > minf=697
> > > > >>> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%,
> > 32=100.0%,
> > > > >>> >=64=0.0%
> > > > >>> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%,
> > 64=0.0%,
> > > > >>> >=64=0.0%
> > > > >>> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%,
> > > > >>> 64=0.0%,
> > > > >>> >=64=0.0%
> > > > >>> issued rwts: total=4766095,0,0,0 short=0,0,0,0 dropped=0,0,0,0
> > > > >>> latency : target=0, window=0, percentile=100.00%, depth=32
> > > > >>> Run status group 0 (all jobs):
> > > > >>> MIXED: bw=1862MiB/s (1952MB/s), 1862MiB/s-1862MiB/s
> > (1952MB/s-
> > > > >>> 1952MB/s), io=18.2GiB (19.5GB), run=10001-10001msec Disk
> > > > >>> stats
> > > > >>> (read/write):
> > > > >>> nvme2n1: ios=4709190/0, merge=0/0, ticks=4096/0,
> > > > in_queue=1388752,
> > > > >>> util=100.00%
> > > > >>>
> > > > >>>
> > > > >>> don(a)donst201:~/fio/single/rr$ sudo
> > > > >>> /home/don/install/spdk/spdk/scripts/setup.sh
> > > > >>> Active mountpoints on /dev/nvme0n1, so not binding PCI dev
> > > > >>> 0000:03:00.0
> > > > >>> 0000:1c:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > > >>> 0000:1d:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > > >>> 0000:5e:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > > >>> 0000:5f:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > > >>> 0000:62:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > > >>> 0000:63:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > > >>> 0000:64:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > > >>> 0000:65:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > > >>> 0000:da:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > > >>> 0000:db:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > > >>> 0000:dc:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > > >>> 0000:dd:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > > >>> 0000:e0:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > > >>> 0000:e1:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > > >>> 0000:e2:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > > >>> 0000:e3:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > > >>> 0000:00:04.0 (8086 2021): ioatdma -> uio_pci_generic
> > > > >>> 0000:00:04.1 (8086 2021): ioatdma -> uio_pci_generic
> > > > >>> 0000:00:04.2 (8086 2021): ioatdma -> uio_pci_generic
> > > > >>> 0000:00:04.3 (8086 2021): ioatdma -> uio_pci_generic
> > > > >>> 0000:00:04.4 (8086 2021): ioatdma -> uio_pci_generic
> > > > >>> 0000:00:04.5 (8086 2021): ioatdma -> uio_pci_generic
> > > > >>> 0000:00:04.6 (8086 2021): ioatdma -> uio_pci_generic
> > > > >>> 0000:00:04.7 (8086 2021): ioatdma -> uio_pci_generic
> > > > >>> 0000:80:04.0 (8086 2021): ioatdma -> uio_pci_generic
> > > > >>> 0000:80:04.1 (8086 2021): ioatdma -> uio_pci_generic
> > > > >>> 0000:80:04.2 (8086 2021): ioatdma -> uio_pci_generic
> > > > >>> 0000:80:04.3 (8086 2021): ioatdma -> uio_pci_generic
> > > > >>> 0000:80:04.4 (8086 2021): ioatdma -> uio_pci_generic
> > > > >>> 0000:80:04.5 (8086 2021): ioatdma -> uio_pci_generic
> > > > >>> 0000:80:04.6 (8086 2021): ioatdma -> uio_pci_generic
> > > > >>> 0000:80:04.7 (8086 2021): ioatdma -> uio_pci_generic
> > > > >>>
> > > > >>>
> > > > >>> don(a)donst201:~/fio/single/rr$ cat 1c.ini |grep -v '#'
> > > > >>> [global]
> > > > >>> rw=randrw
> > > > >>> rwmixread=100
> > > > >>> numjobs=4
> > > > >>> iodepth=32
> > > > >>> bs=4k
> > > > >>> direct=1
> > > > >>> thread=1
> > > > >>> time_based=1
> > > > >>> ramp_time=0
> > > > >>> runtime=10
> > > > >>>
> > > >
> > ioengine=/home/don/install/spdk/spdk/examples/nvme/fio_plugin/fio_pl
> > > > >>> ug
> > > > >>> in
> > > > >>> group_reporting=1
> > > > >>> unified_rw_reporting=1
> > > > >>> exitall=1
> > > > >>> randrepeat=0
> > > > >>> norandommap=1
> > > > >>> [0000.1c.00.0]
> > > > >>> filename=trtype=PCIe traddr=0000.1c.00.0 ns=1
> > > > >>>
> > > > >>>
> > > > >>> don(a)donst201:~/fio/single/rr$ sudo fio 1c.ini
> > > > >>> 0000.1c.00.0: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W)
> > > > >>> 4096B-4096B,
> > > > >>> (T) 4096B-4096B, ioengine=spdk, iodepth=32 ...
> > > > >>> fio-3.11
> > > > >>> Starting 4 threads
> > > > >>> Starting SPDK v18.10 / DPDK 18.08.0 initialization...
> > > > >>> [ DPDK EAL parameters: fio --no-shconf -c 0x1 -m 512 --file-
> > > > >>> prefix=spdk_pid3668 ]
> > > > >>> EAL: Detected 36 lcore(s)
> > > > >>> EAL: Detected 2 NUMA nodes
> > > > >>> EAL: No free hugepages reported in hugepages-1048576kB
> > > > >>> EAL: Probing VFIO support...
> > > > >>> EAL: PCI device 0000:1c:00.0 on NUMA socket 0
> > > > >>> EAL: probe driver: 8086:a54 spdk_nvme
> > > > >>> Jobs: 1 (f=0): [f(1),_(3)][100.0%][eta 00m:00s]
> > > > >>> 0000.1c.00.0: (groupid=0, jobs=4): err= 0: pid=3709: Mon Nov
> > > > >>> 5
> > > > >>> 02:28:29
> > > > >>> 2018
> > > > >>> mixed: IOPS=13.2k, BW=51.6MiB/s
> (54.1MB/s)(517MiB/10011msec)
> > > > >>> slat (nsec): min=109, max=15344, avg=127.91, stdev=73.44
> > > > >>> clat (usec): min=187, max=18715, avg=9683.80, stdev=650.54
> > > > >>> lat (usec): min=193, max=18715, avg=9683.93, stdev=650.53
> > > > >>> clat percentiles (usec):
> > > > >>> | 1.00th=[ 8717], 5.00th=[ 8848], 10.00th=[ 8979],
> > > > >>> 20.00th=[
> > > 9110],
> > > > >>> | 30.00th=[ 9241], 40.00th=[ 9372], 50.00th=[ 9503],
> > > > >>> 60.00th=[
> > > 9765],
> > > > >>> | 70.00th=[10028], 80.00th=[10159], 90.00th=[10552],
> > > > 95.00th=[10945],
> > > > >>> | 99.00th=[11338], 99.50th=[11338], 99.90th=[11731],
> > > > 99.95th=[13960],
> > > > >>> | 99.99th=[17695]
> > > > >>> bw ( KiB/s): min=13080, max=13304, per=25.00%,
> > > > >>> avg=13214.70, stdev=55.30, samples=80
> > > > >>> iops : min= 3270, max= 3326, avg=3303.65, stdev=13.84,
> > > > samples=80
> > > > >>> lat (usec) : 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
> > > > >>> lat (msec) : 2=0.01%, 4=0.02%, 10=70.85%, 20=29.11%
> > > > >>> cpu : usr=100.01%, sys=0.00%, ctx=23, majf=0, minf=0
> > > > >>> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%,
> 32=99.9%,
> > > > >>> >=64=0.0%
> > > > >>> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%,
> > 64=0.0%,
> > > > >>> >=64=0.0%
> > > > >>> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%,
> > > > >>> 64=0.0%,
> > > > >>> >=64=0.0%
> > > > >>> issued rwts: total=132300,0,0,0 short=0,0,0,0 dropped=0,0,0,0
> > > > >>> latency : target=0, window=0, percentile=100.00%, depth=32
> > > > >>> Run status group 0 (all jobs):
> > > > >>> MIXED: bw=51.6MiB/s (54.1MB/s), 51.6MiB/s-51.6MiB/s
> > (54.1MB/s-
> > > > >>> 54.1MB/s), io=517MiB (542MB), run=10011-10011msec
> > > > >>>
> > > > >>> _______________________________________________
> > > > >>> SPDK mailing list
> > > > >>> SPDK(a)lists.01.org
> > > > >>> https://lists.01.org/mailman/listinfo/spdk
> > > > >>> _______________________________________________
> > > > >>> SPDK mailing list
> > > > >>> SPDK(a)lists.01.org
> > > > >>> https://lists.01.org/mailman/listinfo/spdk
> > > > >>_______________________________________________
> > > > >>SPDK mailing list
> > > > >> SPDK(a)lists.01.org
> > > > >> https://lists.01.org/mailman/listinfo/spdk
> > > > >>_______________________________________________
> > > > >>SPDK mailing list
> > > > >> SPDK(a)lists.01.org
> > > > >> https://lists.01.org/mailman/listinfo/spdk
> > > > >
> > > > >
> > > > >--
> > > > >Маров Алексей
> > > > >_______________________________________________
> > > > >SPDK mailing list
> > > > >SPDK(a)lists.01.org
> > > > >https://lists.01.org/mailman/listinfo/spdk
> > > > >_______________________________________________
> > > > >SPDK mailing list
> > > > >SPDK(a)lists.01.org
> > > > >https://lists.01.org/mailman/listinfo/spdk
> > > >
> > > >
> > > > --
> > > > Маров Алексей
> > > > _______________________________________________
> > > > SPDK mailing list
> > > > SPDK(a)lists.01.org
> > > > https://lists.01.org/mailman/listinfo/spdk
> > > _______________________________________________
> > > SPDK mailing list
> > > SPDK(a)lists.01.org
> > > https://lists.01.org/mailman/listinfo/spdk
> > >
> > --
> >
> > Regards,
> > Andrey
> > _______________________________________________
> > SPDK mailing list
> > SPDK(a)lists.01.org
> > https://lists.01.org/mailman/listinfo/spdk
> > _______________________________________________
> > SPDK mailing list
> > SPDK(a)lists.01.org
> > https://lists.01.org/mailman/listinfo/spdk
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [SPDK] FIO NVMe Performance Results
@ 2018-11-05 21:42 Howell, Seth
0 siblings, 0 replies; 25+ messages in thread
From: Howell, Seth @ 2018-11-05 21:42 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 41920 bytes --]
I think that the uio_pci_generic vs. vfio thing is unrelated. The only reason I brought it up was because it could have been indicative of a larger configuration difference. However, I ran my test with and without vfio support just to be sure and got the higher performance numbers both times.
As a side note:
It's probably a good thing to have enabled in general when running spdk benchmarks because that is the recommended configuration on any physical machine.
You can enable it in grub by adding the following to the GRUB_CMDLINE_LINUX line of /etc/default.grub: intel_iommu=on
and then running grub2-mkconfig.
-----Original Message-----
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Gruher, Joseph R
Sent: Monday, November 5, 2018 1:39 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] FIO NVMe Performance Results
Attempting to fall back to earlier revisions has not yielded any improvements for me. All tests use the same P4500 physical disk. FIO files are as shown before, running a 4 thread, queue depth 32 per thread, 4KB random read workload.
SPDK 18.10, kernel 4.18.16, FIO 3.11 = 12.9k IOPS SPDK 18.10, kernel 4.16.14, FIO 3.11 = 11.9k IOPS SPDK 18.04, kernel 4.18.16, FIO 3.11 = 12.9k IOPS SPDK 18.04, kernel 4.16.14, FIO 3.11 = 13.0k IOPS SPDK 18.04, kernel 4.16.16, FIO 3.3 = 13.1k IOPS
Kernel NVMe driver, kernel 4.16.16, FIO 3.3 = 477k IOPS Kernel NVMe driver, kernel 4.18.16, FIO 3.11 = 477k IOPS
I am curious if the comment below about uio_pci_generic instead of vfio can lead us anywhere, or maybe there's some OS setting or BIOS setting that needs to be toggled...
For the folks working inside Intel, I can also give you direct access to the test system, if that would help.
Thanks,
Joe
> -----Original Message-----
> From: Gruher, Joseph R
> Sent: Monday, November 5, 2018 11:05 AM
> To: Storage Performance Development Kit <spdk(a)lists.01.org>
> Subject: RE: [SPDK] FIO NVMe Performance Results
>
> I've had good FIO NVMe results in the past we well, with SPDK 18.04 on
> Ubuntu 18.04, with kernel 4.16.14. That was on a different, but
> similar, hardware platform.
>
> I suspect something has changed now in SPDK 18.10 and/or kernel
> 4.18.16. I can try going back to earlier versions of each to see if I
> can recreate earlier good results and find where things went wrong.
>
> We're not running in a VM - this is a bare metal Xeon server with
> Skylake CPUs.
>
> Thanks,
> Joe
>
> > -----Original Message-----
> > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Howell,
> > Seth
> > Sent: Monday, November 5, 2018 10:32 AM
> > To: Storage Performance Development Kit <spdk(a)lists.01.org>
> > Subject: Re: [SPDK] FIO NVMe Performance Results
> >
> > Hi Joseph,
> > I attempted to verify your results on my system. I am using fedora
> > 27 with kernel version 4.17.3 with a p3700 drive. I cloned fio from
> > https://github.com/axboe/fio and checked out the appropriate tag. I
> > got appx 408k iops.
> > Typically we run spdk with the iommu enabled. I noticed that your
> > devices are binding to uio_pci_generic instead of vfio. This
> > shouldn't make any difference at all, but I just want to make sure
> > you aren't running in a VM or something like that and passing
> > through your NVMe drives and that's why you aren't using the IOMMU.
> > That could explain a deeper configuration difference that could help
> > us know where to look for
> the disparity.
> >
> > Please see the contents of my configuration files and output below:
> >
> > Test.ini:
> > [global]
> > rw=randrw
> > rwmixread=100
> > numjobs=4
> > iodepth=32
> > bs=4k
> > direct=1
> > thread=1
> > time_based=1
> > ramp_time=0
> > runtime=10
> > ioengine=spdk_bdev
> > spdk_conf=./nvme.conf
> > group_reporting=1
> > unified_rw_reporting=1
> > exitall=1
> > randrepeat=0
> > norandommap=1
> > [0000.81.00.0]
> > filename=Nvme0n1
> >
> > nvme.conf:
> > [Nvme]
> > TransportId "trtype:PCIe traddr:0000.81.00.0" Nvme0
> >
> > Sudo
> >
> LD_PRELOAD=/home/sethhowe/Desktop/Development_Folders/spdk/exam
> > ples/bdev/fio_plugin/fio_plugin ./fio ./test.ini >./out,txt 2>&1
> >
> > Out.txt:
> > 0000.81.00.0: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B,
> > (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=32 ...
> > fio-3.11
> > Starting 4 threads
> > Starting SPDK v18.10 / DPDK 18.08.0 initialization...
> > [ DPDK EAL parameters: fio --no-shconf -c 0x1
> > --file-prefix=spdk_pid782339 ]
> > EAL: Detected 16 lcore(s)
> > EAL: Detected 2 NUMA nodes
> > EAL: No free hugepages reported in hugepages-1048576kB
> > EAL: Probing VFIO support...
> > EAL: VFIO support initialized
> > EAL: PCI device 0000:81:00.0 on NUMA socket 1
> > EAL: probe driver: 8086:953 spdk_nvme
> >
> > 0000.81.00.0: (groupid=0, jobs=4): err= 0: pid=782359: Mon Nov 5
> > 10:58:59
> > 2018
> > mixed: IOPS=410k, BW=1600MiB/s (1678MB/s)(15.6GiB/10001msec)
> > slat (nsec): min=179, max=19985, avg=245.97, stdev=122.96
> > clat (usec): min=8, max=1821, avg=311.82, stdev=167.14
> > lat (usec): min=8, max=1822, avg=312.06, stdev=167.14
> > clat percentiles (usec):
> > | 1.00th=[ 24], 5.00th=[ 48], 10.00th=[ 78], 20.00th=[ 141],
> > | 30.00th=[ 200], 40.00th=[ 258], 50.00th=[ 314], 60.00th=[ 371],
> > | 70.00th=[ 429], 80.00th=[ 486], 90.00th=[ 545], 95.00th=[ 570],
> > | 99.00th=[ 594], 99.50th=[ 594], 99.90th=[ 627], 99.95th=[ 652],
> > | 99.99th=[ 676]
> > bw ( KiB/s): min=407264, max=416040, per=25.01%, avg=409742.55,
> > stdev=1337.38, samples=76
> > iops : min=101816, max=104010, avg=102435.61, stdev=334.41,
> > samples=76
> > lat (usec) : 10=0.01%, 20=0.46%, 50=4.88%, 100=8.41%, 250=25.29%
> > lat (usec) : 500=43.15%, 750=17.81%, 1000=0.01%
> > lat (msec) : 2=0.01%
> > cpu : usr=99.78%, sys=0.00%, ctx=26, majf=0, minf=0
> > IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.9%,
> > >=64=0.0%
> > submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
> > >=64=0.0%
> > complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.1%, 32=0.1%,
> > 64=0.0%,
> > >=64=0.0%
> > issued rwts: total=4096867,0,0,0 short=0,0,0,0 dropped=0,0,0,0
> > latency : target=0, window=0, percentile=100.00%, depth=32
> >
> > Run status group 0 (all jobs):
> > MIXED: bw=1600MiB/s (1678MB/s), 1600MiB/s-1600MiB/s (1678MB/s-
> > 1678MB/s), io=15.6GiB (16.8GB), run=10001-10001msec
> >
> > Thanks,
> >
> > Seth
> >
> >
> >
> > -----Original Message-----
> > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Andrey
> > Kuzmin
> > Sent: Monday, November 5, 2018 9:48 AM
> > To: Storage Performance Development Kit <spdk(a)lists.01.org>
> > Subject: Re: [SPDK] FIO NVMe Performance Results
> >
> > On Mon, Nov 5, 2018, 18:29 Gruher, Joseph R
> > <joseph.r.gruher(a)intel.com>
> > wrote:
> >
> > > My FIO version is v3.11.
> > >
> > > I tried the bdev FIO plugin as suggested, but performance is about
> > > the same as the other SPDK FIO plugin, I observed only 10.4K IOPS
> > > in a quick 4KB random read test. See configuration and results below.
> > >
> >
> > As 12 ms median read latency for an nvme ssd is far beyond anything
> > reasonable, why don't you just perf this run, to see where the time
> > is being spent?
> >
> > Regards,
> > Andrey
> >
> > >
> > >
> > > don(a)donst201:~/fio/single/rr$ cat nvme.conf [Nvme] TransportId
> > > "trtype:PCIe traddr:0000.1c.00.0" Nvme0
> > >
> > >
> > > don(a)donst201:~/fio/single/rr$ cat 1c-bdev.ini |grep -v '#'
> > > [global]
> > > rw=randrw
> > > rwmixread=100
> > > numjobs=4
> > > iodepth=32
> > > bs=4k
> > > direct=1
> > > thread=1
> > > time_based=1
> > > ramp_time=0
> > > runtime=10
> > > ioengine=spdk_bdev
> > > spdk_conf=./nvme.conf
> > > group_reporting=1
> > > unified_rw_reporting=1
> > > exitall=1
> > > randrepeat=0
> > > norandommap=1
> > > [0000.1c.00.0]
> > > filename=Nvme0n1
> > >
> > >
> > > don(a)donst201:~/fio/single/rr$ sudo
> > >
> >
> LD_PRELOAD=/home/don/install/spdk/spdk/examples/bdev/fio_plugin/fio_p
> > l
> > > ugin
> > > fio 1c-bdev.ini
> > > 0000.1c.00.0: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W)
> > > 4096B-4096B,
> > > (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=32 ...
> > > fio-3.11
> > > Starting 4 threads
> > > Starting SPDK v18.10 / DPDK 18.08.0 initialization...
> > > [ DPDK EAL parameters: fio --no-shconf -c 0x1
> > > --file-prefix=spdk_pid4865 ]
> > > EAL: Detected 36 lcore(s)
> > > EAL: Detected 2 NUMA nodes
> > > EAL: No free hugepages reported in hugepages-1048576kB
> > > EAL: Probing VFIO support...
> > > EAL: PCI device 0000:03:00.0 on NUMA socket 0
> > > EAL: probe driver: 8086:f1a5 spdk_nvme
> > > EAL: PCI device 0000:1c:00.0 on NUMA socket 0
> > > EAL: probe driver: 8086:a54 spdk_nvme
> > > EAL: PCI device 0000:1d:00.0 on NUMA socket 0
> > > EAL: probe driver: 8086:a54 spdk_nvme
> > > EAL: PCI device 0000:5e:00.0 on NUMA socket 0
> > > EAL: probe driver: 8086:a54 spdk_nvme
> > > EAL: PCI device 0000:5f:00.0 on NUMA socket 0
> > > EAL: probe driver: 8086:a54 spdk_nvme
> > > EAL: PCI device 0000:62:00.0 on NUMA socket 0
> > > EAL: probe driver: 8086:a54 spdk_nvme
> > > EAL: PCI device 0000:63:00.0 on NUMA socket 0
> > > EAL: probe driver: 8086:a54 spdk_nvme
> > > EAL: PCI device 0000:64:00.0 on NUMA socket 0
> > > EAL: probe driver: 8086:a54 spdk_nvme
> > > EAL: PCI device 0000:65:00.0 on NUMA socket 0
> > > EAL: probe driver: 8086:a54 spdk_nvme
> > > EAL: PCI device 0000:da:00.0 on NUMA socket 1
> > > EAL: probe driver: 8086:a54 spdk_nvme
> > > EAL: PCI device 0000:db:00.0 on NUMA socket 1
> > > EAL: probe driver: 8086:a54 spdk_nvme
> > > EAL: PCI device 0000:dc:00.0 on NUMA socket 1
> > > EAL: probe driver: 8086:a54 spdk_nvme
> > > EAL: PCI device 0000:dd:00.0 on NUMA socket 1
> > > EAL: probe driver: 8086:a54 spdk_nvme
> > > EAL: PCI device 0000:e0:00.0 on NUMA socket 1
> > > EAL: probe driver: 8086:a54 spdk_nvme
> > > EAL: PCI device 0000:e1:00.0 on NUMA socket 1
> > > EAL: probe driver: 8086:a54 spdk_nvme
> > > EAL: PCI device 0000:e2:00.0 on NUMA socket 1
> > > EAL: probe driver: 8086:a54 spdk_nvme
> > > EAL: PCI device 0000:e3:00.0 on NUMA socket 1
> > > EAL: probe driver: 8086:a54 spdk_nvme
> > > Jobs: 4 (f=4): [r(4)][100.0%][r=45.7MiB/s][r=11.7k IOPS][eta
> > > 00m:00s]
> > > 0000.1c.00.0: (groupid=0, jobs=4): err= 0: pid=4906: Mon Nov 5
> > > 15:23:47
> > > 2018
> > > mixed: IOPS=10.4k, BW=40.5MiB/s (42.5MB/s)(406MiB/10012msec)
> > > slat (nsec): min=201, max=15831, avg=238.11, stdev=148.78
> > > clat (usec): min=124, max=31244, avg=12330.69, stdev=1697.98
> > > lat (usec): min=130, max=31244, avg=12330.93, stdev=1697.98
> > > clat percentiles (usec):
> > > | 1.00th=[ 9634], 5.00th=[10159], 10.00th=[10552], 20.00th=[10945],
> > > | 30.00th=[11338], 40.00th=[11731], 50.00th=[12125],
> > 60.00th=[12518],
> > > | 70.00th=[13042], 80.00th=[13566], 90.00th=[14353],
> > 95.00th=[15008],
> > > | 99.00th=[16712], 99.50th=[18744], 99.90th=[27395],
> > 99.95th=[29754],
> > > | 99.99th=[30278]
> > > bw ( KiB/s): min= 8784, max=11952, per=24.99%, avg=10375.53,
> > > stdev=929.68, samples=80
> > > iops : min= 2196, max= 2988, avg=2593.85, stdev=232.41,
> > > samples=80
> > > lat (usec) : 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
> > > lat (msec) : 2=0.01%, 4=0.02%, 10=2.71%, 20=96.91%, 50=0.33%
> > > cpu : usr=99.99%, sys=0.01%, ctx=28, majf=0, minf=0
> > > IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.9%,
> > > >=64=0.0%
> > > submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
> > > >=64=0.0%
> > > complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%,
> > > 64=0.0%,
> > > >=64=0.0%
> > > issued rwts: total=103907,0,0,0 short=0,0,0,0 dropped=0,0,0,0
> > > latency : target=0, window=0, percentile=100.00%, depth=32
> > > Run status group 0 (all jobs):
> > > MIXED: bw=40.5MiB/s (42.5MB/s), 40.5MiB/s-40.5MiB/s
> > > (42.5MB/s-42.5MB/s), io=406MiB (426MB), run=10012-10012msec
> > >
> > >
> > > Thanks,
> > > Joe
> > >
> > > > -----Original Message-----
> > > > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of ?????
> > ???????
> > > > Sent: Monday, November 5, 2018 12:45 AM
> > > > To: Storage Performance Development Kit <spdk(a)lists.01.org>
> > > > Subject: Re: [SPDK] FIO NVMe Performance Results
> > > >
> > > > Ok, Do you see some performance issues using general bdev fio
> > > > plugin
> > > with
> > > > you configuration ?
> > > >
> > > > Best Regards
> > > > Alex
> > > >
> > > >
> > > > >Понедельник, 5 ноября 2018, 11:39 +03:00 от "Cao, Gang"
> > > > <gang.cao(a)intel.com>:
> > > > >
> > > > >Yes. This is for the bdev version of SPDK FIO plugin and you
> > > > >can assign
> > > the
> > > > conf file as the parameter in the FIO configuration.
> > > > >
> > > > >Previously, it's another SPDK FIO plugin for the user space
> > > > >NVMe driver
> > > > only.
> > > > >
> > > > >These are two different FIO plugins.
> > > > >
> > > > >Thanks,
> > > > >Gang
> > > > >
> > > > >-----Original Message-----
> > > > >From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of
> > > > >Маров Алексей
> > > > >Sent: Monday, November 5, 2018 4:32 PM
> > > > >To: Storage Performance Development Kit < spdk(a)lists.01.org >
> > > > >Subject: Re: [SPDK] FIO NVMe Performance Results
> > > > >
> > > > >Sorry for incomplete message I have sent before
> > > > >
> > > > >Hello Gang.
> > > > >
> > > > >I just curious, why would you run fio with SPDK using this conf
> > > > >file,
> > > with is a
> > > > bit different from what is suggested in here
> > https://github.com/spdk/spdk/tree/master/examples/bdev/fio_plugin
> > > > .
> > > > >
> > > > >I mean may be you should create conf file for spdk like this one :
> > > > >
> > > > >nvme.conf :
> > > > >[Nvme]
> > > > > TransportId "trtype:PCIe traddr:0000.1c.00.0" Nvme0
> > > > >
> > > > >and provide it in fio config.
> > > > >
> > > > >fio.ini :
> > > > >[global]
> > > > >ioengine=spdk_bdev
> > > > >spdk_conf=....../nvme.conf
> > > > >rw=randrw
> > > > >rwmixread=100
> > > > >numjobs=4
> > > > >iodepth=32
> > > > >bs=4k
> > > > >direct=1
> > > > >thread=1
> > > > >time_based=1
> > > > >ramp_time=0
> > > > >runtime=10
> > > > >group_reporting=1
> > > > >unified_rw_reporting=1
> > > > >exitall=1
> > > > >randrepeat=0
> > > > >norandommap=1
> > > > >
> > > > >[0000:06:00.0]
> > > > >filename=Nvme0n1
> > > > >
> > > > >and run like this
> > > > >
> > > > ># LD_PRELOAD=examples/bdev/fio_plugin/fio_plugin fio
> > > > >examples/bdev/fio_plugin/fio.ini
> > > > >
> > > > >Just to note. The recommended version of fio for SPDK is 3.3
> > > > >
> > > > >Best Regards
> > > > >Alex
> > > > >
> > > > >
> > > > >>Понедельник, 5 ноября 2018, 9:53 +03:00 от "Cao, Gang" <
> > > > gang.cao(a)intel.com >:
> > > > >>
> > > > >>Thanks. This is interesting especially for your finding on
> > > > >>queue depth
> > > 1
> > > > case. We are trying to reproduce this issue locally same as your
> > > > configuration.
> > > > >>
> > > > >>Thanks,
> > > > >>Gang
> > > > >>
> > > > >>-----Original Message-----
> > > > >>From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of
> > > > >>Gruher, Joseph R
> > > > >>Sent: Monday, November 5, 2018 1:32 PM
> > > > >>To: Storage Performance Development Kit < spdk(a)lists.01.org >
> > > > >>Subject: Re: [SPDK] FIO NVMe Performance Results
> > > > >>
> > > > >>Interesting catch. It seems like FIO is actually not
> > > > >>accurately
> > > reporting the
> > > > number of jobs in the report out for the SPDK case. If I
> > > > disable the group_reporting parameter I do get a printout for
> > > > four separate jobs in
> > > the
> > > > SPDK case, so I think it is really running four jobs there, and
> > > > just
> > > summarizing
> > > > incorrectly. I also see four separate CPU cores go to 100% when
> > > > I run
> > > that
> > > > FIO test case, indicating four jobs are active.
> > > > >>
> > > > >>Also, I don't think would explain the difference in performance.
> > > Using an
> > > > aggregate queue depth of 32 versus 128 shouldn't lead to a drop
> > > > in performance from 377K to 13K.
> > > > >>
> > > > >>I retested SPDK with multiple separate job descriptions as you
> > > suggested
> > > > anyway, using four separate jobs instead of relying on the
> > > > numjobs=4 parameter, and the results are below. In that SPDK
> > > > test with four
> > > separate
> > > > FIO job statements, each with queue depth 32, I still only
> > > > achieved 12.5K IOPS.
> > > > >>
> > > > >>I then re-ran the kernel NVMe driver test case with only a
> > > > >>single job
> > > for
> > > > comparison, and even with only 1 job and queue depth 32 it still
> > > > achieves 254K IOPS.
> > > > >>
> > > > >>We might be on the right track with a problem around queue
> > > > >>depth
> > > > however. If I run the kernel NVMe driver case with just 1 job
> > > > and queue depth of only 1, I then get 11.2K IOPS, which is
> > > > around the performance
> > > I get
> > > > in the SPDK test case. Perhaps the SPDK case is not actually
> > > > queuing
> > > IOs on
> > > > the device for some reason?
> > > > >>
> > > > >>
> > > > >>don(a)donst201:~/fio/single/rr$ cat nvme2n1.ini |grep -v '#'
> > > > >>[global]
> > > > >>rw=randrw
> > > > >>rwmixread=100
> > > > >>numjobs=1
> > > > >>iodepth=32
> > > > >>bs=4k
> > > > >>direct=1
> > > > >>thread=1
> > > > >>time_based=1
> > > > >>ramp_time=0
> > > > >>runtime=10
> > > > >>ioengine=libaio
> > > > >>group_reporting=1
> > > > >>unified_rw_reporting=1
> > > > >>exitall=1
> > > > >>randrepeat=0
> > > > >>norandommap=1
> > > > >>[nvme2n1]
> > > > >>filename=/dev/nvme2n1
> > > > >>
> > > > >>
> > > > >>don(a)donst201:~/fio/single/rr$ sudo fio nvme2n1.ini
> > > > >>nvme2n1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W)
> > > > >>4096B-4096B,
> > > > >>(T) 4096B-4096B, ioengine=libaio, iodepth=32
> > > > >>fio-3.11
> > > > >>Starting 1 thread
> > > > >>Jobs: 1 (f=1): [r(1)][100.0%][r=1025MiB/s][r=262k IOPS][eta
> > > > >>00m:00s]
> > > > >>nvme2n1: (groupid=0, jobs=1): err= 0: pid=2422: Mon Nov 5
> > > > >>05:19:13
> > > > >>2018
> > > > >> mixed: IOPS=254k, BW=992MiB/s
> (1040MB/s)(9922MiB/10001msec)
> > > > >> slat (nsec): min=1185, max=319052, avg=1611.64, stdev=808.40
> > > > >> clat (usec): min=2, max=4743, avg=124.02, stdev=121.00
> > > > >> lat (usec): min=6, max=4744, avg=125.68, stdev=121.02
> > > > >> clat percentiles (usec):
> > > > >> | 1.00th=[ 9], 5.00th=[ 56], 10.00th=[ 61], 20.00th=[
> > > 73],
> > > > >> | 30.00th=[ 80], 40.00th=[ 91], 50.00th=[ 110], 60.00th=[
> > > 118],
> > > > >> | 70.00th=[ 130], 80.00th=[ 157], 90.00th=[ 202],
> > > > >>95.00th=[ 253],
> > > > >> | 99.00th=[ 408], 99.50th=[ 498], 99.90th=[ 1795],
> > > > >>99.95th=[ 3064],
> > > > >> | 99.99th=[ 3785]
> > > > >> bw ( KiB/s): min=612240, max=1078792, per=100.00%,
> > > > >>avg=1015996.40,
> > > > stdev=106032.96, samples=20
> > > > >> iops : min=153060, max=269698, avg=253998.90,
> > stdev=26508.40,
> > > > samples=20
> > > > >> lat (usec) : 4=0.01%, 10=1.47%, 20=1.19%, 50=0.17%, 100=42.03%
> > > > >> lat (usec) : 250=49.95%, 500=4.70%, 750=0.28%, 1000=0.06%
> > > > >> lat (msec) : 2=0.06%, 4=0.09%, 10=0.01%
> > > > >> cpu : usr=21.45%, sys=51.11%, ctx=530348, majf=0, minf=139
> > > > >> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%,
> 32=100.0%,
> > > > >=64=0.0%
> > > > >> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%,
> 64=0.0%,
> > > > >=64=0.0%
> > > > >> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%,
> > > > >> 64=0.0%,
> > > > >>>=64=0.0%
> > > > >> issued rwts: total=2540023,0,0,0 short=0,0,0,0 dropped=0,0,0,0
> > > > >> latency : target=0, window=0, percentile=100.00%, depth=32
> > > > >>Run status group 0 (all jobs):
> > > > >> MIXED: bw=992MiB/s (1040MB/s), 992MiB/s-992MiB/s (1040MB/s-
> > > > 1040MB/s), io=9922MiB (10.4GB), run=10001-10001msec Disk stats
> > > > (read/write):
> > > > >> nvme2n1: ios=2514356/0, merge=0/0, ticks=400/0,
> > in_queue=297612,
> > > > >>util=99.06%
> > > > >>
> > > > >>
> > > > >>don(a)donst201:~/fio/single/rr$ cat 1c.ini |grep -v '#'
> > > > >>[global]
> > > > >>rw=randrw
> > > > >>rwmixread=100
> > > > >>numjobs=1
> > > > >>iodepth=32
> > > > >>bs=4k
> > > > >>direct=1
> > > > >>thread=1
> > > > >>time_based=1
> > > > >>ramp_time=0
> > > > >>runtime=10
> > > >
> > >>ioengine=/home/don/install/spdk/spdk/examples/nvme/fio_plugin/fio_
> > > > >>pl
> > > > ug
> > > > >>in
> > > > >>group_reporting=1
> > > > >>unified_rw_reporting=1
> > > > >>exitall=1
> > > > >>randrepeat=0
> > > > >>norandommap=1
> > > > >>[0000.1c.00.0-1]
> > > > >>filename=trtype=PCIe traddr=0000.1c.00.0 ns=1 [0000.1c.00.0-2]
> > > > >>filename=trtype=PCIe traddr=0000.1c.00.0 ns=1 [0000.1c.00.0-3]
> > > > >>filename=trtype=PCIe traddr=0000.1c.00.0 ns=1 [0000.1c.00.0-4]
> > > > >>filename=trtype=PCIe traddr=0000.1c.00.0 ns=1
> > > > >>
> > > > >>
> > > > >>don(a)donst201:~/fio/single/rr$ sudo fio 1c.ini
> > > > >>0000.1c.00.0-1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W)
> > > > >>4096B-4096B,
> > > > >>(T) 4096B-4096B, ioengine=spdk, iodepth=32
> > > > >>0000.1c.00.0-2: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W)
> > > > >>4096B-4096B,
> > > > >>(T) 4096B-4096B, ioengine=spdk, iodepth=32
> > > > >>0000.1c.00.0-3: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W)
> > > > >>4096B-4096B,
> > > > >>(T) 4096B-4096B, ioengine=spdk, iodepth=32
> > > > >>0000.1c.00.0-4: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W)
> > > > >>4096B-4096B,
> > > > >>(T) 4096B-4096B, ioengine=spdk, iodepth=32
> > > > >>fio-3.11
> > > > >>Starting 4 threads
> > > > >>Starting SPDK v18.10 / DPDK 18.08.0 initialization...
> > > > >>[ DPDK EAL parameters: fio --no-shconf -c 0x1 -m 512
> > > > >>--file-prefix=spdk_pid4270 ]
> > > > >>EAL: Detected 36 lcore(s)
> > > > >>EAL: Detected 2 NUMA nodes
> > > > >>EAL: No free hugepages reported in hugepages-1048576kB
> > > > >>EAL: Probing VFIO support...
> > > > >>EAL: PCI device 0000:1c:00.0 on NUMA socket 0
> > > > >>EAL: probe driver: 8086:a54 spdk_nvme
> > > > >>Jobs: 1 (f=0): [_(3),f(1)][100.0%][eta 00m:00s]
> > > > >>0000.1c.00.0-1: (groupid=0, jobs=4): err= 0: pid=4310: Mon Nov
> > > > >>5
> > > > >>05:16:03 2018
> > > > >> mixed: IOPS=12.5k, BW=48.0MiB/s (51.4MB/s)(490MiB/10010msec)
> > > > >> slat (nsec): min=98, max=15092, avg=120.02, stdev=90.07
> > > > >> clat (usec): min=55, max=19757, avg=10207.22, stdev=884.06
> > > > >> lat (usec): min=60, max=19757, avg=10207.34, stdev=884.06
> > > > >> clat percentiles (usec):
> > > > >> | 1.00th=[ 8979], 5.00th=[ 9110], 10.00th=[ 9372],
> > > > >>20.00th=[ 9503],
> > > > >> | 30.00th=[ 9765], 40.00th=[ 9896], 50.00th=[10159],
> > > > >>60.00th=[10290],
> > > > >> | 70.00th=[10552], 80.00th=[10814], 90.00th=[11207],
> > > > >>95.00th=[11600],
> > > > >> | 99.00th=[12911], 99.50th=[14353], 99.90th=[17695],
> > > > >>99.95th=[18744],
> > > > >> | 99.99th=[19006]
> > > > >> bw ( KiB/s): min=12008, max=12976, per=25.00%,
> > > > >>avg=12535.70,
> > > > stdev=306.81, samples=80
> > > > >> iops : min= 3002, max= 3244, avg=3133.85, stdev=76.77,
> > > samples=80
> > > > >> lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%,
> > 1000=0.01%
> > > > >> lat (msec) : 2=0.01%, 4=0.02%, 10=45.13%, 20=54.83%
> > > > >> cpu : usr=100.01%, sys=0.00%, ctx=27, majf=0, minf=0
> > > > >> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.9%,
> > > > >=64=0.0%
> > > > >> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%,
> 64=0.0%,
> > > > >=64=0.0%
> > > > >> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%,
> > > > >> 64=0.0%,
> > > > >>>=64=0.0%
> > > > >> issued rwts: total=125506,0,0,0 short=0,0,0,0 dropped=0,0,0,0
> > > > >> latency : target=0, window=0, percentile=100.00%, depth=32
> > > > >>Run status group 0 (all jobs):
> > > > >> MIXED: bw=48.0MiB/s (51.4MB/s), 48.0MiB/s-48.0MiB/s
> > > > >>(51.4MB/s-51.4MB/s), io=490MiB (514MB), run=10010-10010msec
> > > > >>
> > > > >>
> > > > >>don(a)donst201:~/fio/single/rr$ cat nvme2n1.ini |grep -v '#'
> > > > >>[global]
> > > > >>rw=randrw
> > > > >>rwmixread=100
> > > > >>numjobs=1
> > > > >>iodepth=1
> > > > >>bs=4k
> > > > >>direct=1
> > > > >>thread=1
> > > > >>time_based=1
> > > > >>ramp_time=0
> > > > >>runtime=10
> > > > >>ioengine=libaio
> > > > >>group_reporting=1
> > > > >>unified_rw_reporting=1
> > > > >>exitall=1
> > > > >>randrepeat=0
> > > > >>norandommap=1
> > > > >>[nvme2n1]
> > > > >>filename=/dev/nvme2n1
> > > > >>
> > > > >>
> > > > >>don(a)donst201:~/fio/single/rr$ sudo fio nvme2n1.ini
> > > > >>nvme2n1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W)
> > > > >>4096B-4096B,
> > > > >>(T) 4096B-4096B, ioengine=libaio, iodepth=1
> > > > >>fio-3.11
> > > > >>Starting 1 thread
> > > > >>Jobs: 1 (f=1): [r(1)][100.0%][r=43.6MiB/s][r=11.2k IOPS][eta
> > > > >>00m:00s]
> > > > >>nvme2n1: (groupid=0, jobs=1): err= 0: pid=2525: Mon Nov 5
> > > > >>05:22:57
> > > > >>2018
> > > > >> mixed: IOPS=11.2k, BW=43.6MiB/s (45.7MB/s)(436MiB/10001msec)
> > > > >> slat (nsec): min=4848, max=208595, avg=5126.33, stdev=771.27
> > > > >> clat (nsec): min=1845, max=3405.8k, avg=83004.99,
> stdev=33577.36
> > > > >> lat (usec): min=15, max=3411, avg=88.26, stdev=33.59
> > > > >> clat percentiles (usec):
> > > > >> | 1.00th=[ 15], 5.00th=[ 58], 10.00th=[ 59], 20.00th=[
> > > 59],
> > > > >> | 30.00th=[ 60], 40.00th=[ 77], 50.00th=[ 77], 60.00th=[
> > > 78],
> > > > >> | 70.00th=[ 111], 80.00th=[ 113], 90.00th=[ 113],
> > > > >>95.00th=[ 115],
> > > > >> | 99.00th=[ 186], 99.50th=[ 186], 99.90th=[ 190],
> > > > >>99.95th=[ 192],
> > > > >> | 99.99th=[ 375]
> > > > >> bw ( KiB/s): min=43816, max=45016, per=99.98%,
> > > > >>avg=44639.58,
> > > > stdev=312.88, samples=19
> > > > >> iops : min=10954, max=11254, avg=11159.89, stdev=78.22,
> > > > samples=19
> > > > >> lat (usec) : 2=0.01%, 20=2.83%, 50=0.01%, 100=64.57%,
> 250=32.59%
> > > > >> lat (usec) : 500=0.01%, 750=0.01%
> > > > >> lat (msec) : 2=0.01%, 4=0.01%
> > > > >> cpu : usr=4.02%, sys=10.33%, ctx=111630, majf=0, minf=12
> > > > >> IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%,
> 32=0.0%,
> > > > >=64=0.0%
> > > > >> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%,
> 64=0.0%,
> > > > >=64=0.0%
> > > > >> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%,
> > > > >> 64=0.0%,
> > > > >>>=64=0.0%
> > > > >> issued rwts: total=111631,0,0,0 short=0,0,0,0 dropped=0,0,0,0
> > > > >> latency : target=0, window=0, percentile=100.00%, depth=1
> > > > >>Run status group 0 (all jobs):
> > > > >> MIXED: bw=43.6MiB/s (45.7MB/s), 43.6MiB/s-43.6MiB/s
> (45.7MB/s-
> > > > 45.7MB/s), io=436MiB (457MB), run=10001-10001msec Disk stats
> > > > (read/write):
> > > > >> nvme2n1: ios=110275/0, merge=0/0, ticks=0/0, in_queue=8596,
> > > > >>util=86.22%
> > > > >>
> > > > >>
> > > > >>Thanks,
> > > > >>Joe
> > > > >>
> > > > >>> -----Original Message-----
> > > > >>> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of
> > > > >>> Cao, Gang
> > > > >>> Sent: Sunday, November 4, 2018 8:40 PM
> > > > >>> To: Storage Performance Development Kit < spdk(a)lists.01.org
> > > > >>> >
> > > > >>> Subject: Re: [SPDK] FIO NVMe Performance Results
> > > > >>>
> > > > >>> For the SPDK plugin, from the below result, looks like only
> > > > >>> 1 job started with
> > > > >>> 32 queue depth:
> > > > >>>
> > > > >>> Jobs: 1 (f=0): [f(1),_(3)][100.0%][eta 00m:00s] <<<<
> > > > >>>
> > > > >>> Try to update the FIO configuration file as
> > > > >>>
> > > > >>> [job1]
> > > > >>> filename=trtype=PCIe traddr=0000.1c.00.0 ns=1 [job2]
> > > > >>> filename=trtype=PCIe
> > > > >>> traddr=0000.1c.00.0 ns=1 ....
> > > > >>>
> > > > >>> The kernel FIO is with 4 jobs and each job of 32 queue depth.
> > > > >>> For SPDK, you can configure as the same, or just configure
> > > > >>> two jobs and each job of 64 queue depth. The overall queue
> > > > >>> depth to the device is
> > > > same of 128.
> > > > >>>
> > > > >>> Thanks,
> > > > >>> Gang
> > > > >>>
> > > > >>> -----Original Message-----
> > > > >>> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of
> > > > >>> Gruher, Joseph R
> > > > >>> Sent: Monday, November 5, 2018 10:49 AM
> > > > >>> To: Storage Performance Development Kit < spdk(a)lists.01.org
> > > > >>> >
> > > > >>> Subject: [SPDK] FIO NVMe Performance Results
> > > > >>>
> > > > >>> Hi folks-
> > > > >>>
> > > > >>> I'm testing SPDK 18.10 on Ubuntu 18.04 with kernel 4.18.16.
> > > > >>> I used FIO with the kernel NVMe driver to measure the
> > > > >>> performance of a local (PCIe
> > > > >>> attached) Intel P4500 NVMe device on a 4KB random read
> > > > >>> workload and obtained 477K IOPS, roughly in line with the
> > > > >>> drive
> spec.
> > > > >>> Then I tested the same drive with the SPDK FIO plugin and
> > > > >>> only achieved 13K IOPS. The FIO test files and the results
> > > > >>> are pasted below. Any ideas where I'm going wrong here?
> > > > >>>
> > > > >>> Thanks!
> > > > >>>
> > > > >>>
> > > > >>> don(a)donst201:~/fio/single/rr$ ls
> > > > >>> /sys/devices/pci0000:17/0000:17:02.0/0000:1c:00.0/nvme/nvme1
> > > > >>> / address cntlid dev device firmware_rev model nvme2n1
> > > > >>> power rescan_controller reset_controller serial state
> > > > >>> subsysnqn subsystem transport uevent
> > > > >>>
> > > > >>>
> > > > >>> don(a)donst201:~/fio/single/rr$ cat nvme2n1.ini |grep -v '#'
> > > > >>> [global]
> > > > >>> rw=randrw
> > > > >>> rwmixread=100
> > > > >>> numjobs=4
> > > > >>> iodepth=32
> > > > >>> bs=4k
> > > > >>> direct=1
> > > > >>> thread=1
> > > > >>> time_based=1
> > > > >>> ramp_time=0
> > > > >>> runtime=10
> > > > >>> ioengine=libaio
> > > > >>> group_reporting=1
> > > > >>> unified_rw_reporting=1
> > > > >>> exitall=1
> > > > >>> randrepeat=0
> > > > >>> norandommap=1
> > > > >>> [nvme2n1]
> > > > >>> filename=/dev/nvme2n1
> > > > >>>
> > > > >>>
> > > > >>> don(a)donst201:~/fio/single/rr$ sudo fio nvme2n1.ini
> > > > >>> nvme2n1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W)
> > > > >>> 4096B-4096B,
> > > > >>> (T) 4096B-4096B, ioengine=libaio, iodepth=32 ...
> > > > >>> fio-3.11
> > > > >>> Starting 4 threads
> > > > >>> Jobs: 4 (f=4): [r(4)][100.0%][r=1874MiB/s][r=480k IOPS][eta
> > > > >>> 00m:00s]
> > > > >>> nvme2n1: (groupid=0, jobs=4): err= 0: pid=2575: Mon Nov 5
> > > > >>> 02:26:25
> > > > 2018
> > > > >>> mixed: IOPS=477k, BW=1862MiB/s
> > (1952MB/s)(18.2GiB/10001msec)
> > > > >>> slat (nsec): min=1295, max=333088, avg=2216.35, stdev=927.89
> > > > >>> clat (nsec): min=506, max=5818.8k, avg=265924.16,
> > stdev=223761.59
> > > > >>> lat (usec): min=6, max=5821, avg=268.19, stdev=223.76
> > > > >>> clat percentiles (usec):
> > > > >>> | 1.00th=[ 13], 5.00th=[ 74], 10.00th=[ 87], 20.00th=[
> > > 115],
> > > > >>> | 30.00th=[ 139], 40.00th=[ 167], 50.00th=[ 204],
> > > > >>> 60.00th=[
> > > 247],
> > > > >>> | 70.00th=[ 306], 80.00th=[ 388], 90.00th=[ 523],
> > > > >>> 95.00th=[
> > > 660],
> > > > >>> | 99.00th=[ 988], 99.50th=[ 1156], 99.90th=[ 2343],
> > > > >>> 99.95th=[
> > > 3195],
> > > > >>> | 99.99th=[ 4424]
> > > > >>> bw ( KiB/s): min=448192, max=483104, per=25.00%,
> > > > >>> avg=476597.10, stdev=6748.78, samples=80
> > > > >>> iops : min=112048, max=120776, avg=119149.27,
> > > stdev=1687.20,
> > > > >>> samples=80
> > > > >>> lat (nsec) : 750=0.01%
> > > > >>> lat (usec) : 10=0.52%, 20=1.25%, 50=0.25%, 100=13.38%,
> > 250=45.13%
> > > > >>> lat (usec) : 500=28.23%, 750=8.08%, 1000=2.22%
> > > > >>> lat (msec) : 2=0.82%, 4=0.10%, 10=0.02%
> > > > >>> cpu : usr=14.08%, sys=36.02%, ctx=1930207, majf=0,
> > > minf=697
> > > > >>> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%,
> > 32=100.0%,
> > > > >>> >=64=0.0%
> > > > >>> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%,
> > 64=0.0%,
> > > > >>> >=64=0.0%
> > > > >>> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%,
> > > > >>> 64=0.0%,
> > > > >>> >=64=0.0%
> > > > >>> issued rwts: total=4766095,0,0,0 short=0,0,0,0 dropped=0,0,0,0
> > > > >>> latency : target=0, window=0, percentile=100.00%, depth=32
> > > > >>> Run status group 0 (all jobs):
> > > > >>> MIXED: bw=1862MiB/s (1952MB/s), 1862MiB/s-1862MiB/s
> > (1952MB/s-
> > > > >>> 1952MB/s), io=18.2GiB (19.5GB), run=10001-10001msec Disk
> > > > >>> stats
> > > > >>> (read/write):
> > > > >>> nvme2n1: ios=4709190/0, merge=0/0, ticks=4096/0,
> > > > in_queue=1388752,
> > > > >>> util=100.00%
> > > > >>>
> > > > >>>
> > > > >>> don(a)donst201:~/fio/single/rr$ sudo
> > > > >>> /home/don/install/spdk/spdk/scripts/setup.sh
> > > > >>> Active mountpoints on /dev/nvme0n1, so not binding PCI dev
> > > > >>> 0000:03:00.0
> > > > >>> 0000:1c:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > > >>> 0000:1d:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > > >>> 0000:5e:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > > >>> 0000:5f:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > > >>> 0000:62:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > > >>> 0000:63:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > > >>> 0000:64:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > > >>> 0000:65:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > > >>> 0000:da:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > > >>> 0000:db:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > > >>> 0000:dc:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > > >>> 0000:dd:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > > >>> 0000:e0:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > > >>> 0000:e1:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > > >>> 0000:e2:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > > >>> 0000:e3:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > > >>> 0000:00:04.0 (8086 2021): ioatdma -> uio_pci_generic
> > > > >>> 0000:00:04.1 (8086 2021): ioatdma -> uio_pci_generic
> > > > >>> 0000:00:04.2 (8086 2021): ioatdma -> uio_pci_generic
> > > > >>> 0000:00:04.3 (8086 2021): ioatdma -> uio_pci_generic
> > > > >>> 0000:00:04.4 (8086 2021): ioatdma -> uio_pci_generic
> > > > >>> 0000:00:04.5 (8086 2021): ioatdma -> uio_pci_generic
> > > > >>> 0000:00:04.6 (8086 2021): ioatdma -> uio_pci_generic
> > > > >>> 0000:00:04.7 (8086 2021): ioatdma -> uio_pci_generic
> > > > >>> 0000:80:04.0 (8086 2021): ioatdma -> uio_pci_generic
> > > > >>> 0000:80:04.1 (8086 2021): ioatdma -> uio_pci_generic
> > > > >>> 0000:80:04.2 (8086 2021): ioatdma -> uio_pci_generic
> > > > >>> 0000:80:04.3 (8086 2021): ioatdma -> uio_pci_generic
> > > > >>> 0000:80:04.4 (8086 2021): ioatdma -> uio_pci_generic
> > > > >>> 0000:80:04.5 (8086 2021): ioatdma -> uio_pci_generic
> > > > >>> 0000:80:04.6 (8086 2021): ioatdma -> uio_pci_generic
> > > > >>> 0000:80:04.7 (8086 2021): ioatdma -> uio_pci_generic
> > > > >>>
> > > > >>>
> > > > >>> don(a)donst201:~/fio/single/rr$ cat 1c.ini |grep -v '#'
> > > > >>> [global]
> > > > >>> rw=randrw
> > > > >>> rwmixread=100
> > > > >>> numjobs=4
> > > > >>> iodepth=32
> > > > >>> bs=4k
> > > > >>> direct=1
> > > > >>> thread=1
> > > > >>> time_based=1
> > > > >>> ramp_time=0
> > > > >>> runtime=10
> > > > >>>
> > > >
> > ioengine=/home/don/install/spdk/spdk/examples/nvme/fio_plugin/fio_pl
> > > > >>> ug
> > > > >>> in
> > > > >>> group_reporting=1
> > > > >>> unified_rw_reporting=1
> > > > >>> exitall=1
> > > > >>> randrepeat=0
> > > > >>> norandommap=1
> > > > >>> [0000.1c.00.0]
> > > > >>> filename=trtype=PCIe traddr=0000.1c.00.0 ns=1
> > > > >>>
> > > > >>>
> > > > >>> don(a)donst201:~/fio/single/rr$ sudo fio 1c.ini
> > > > >>> 0000.1c.00.0: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W)
> > > > >>> 4096B-4096B,
> > > > >>> (T) 4096B-4096B, ioengine=spdk, iodepth=32 ...
> > > > >>> fio-3.11
> > > > >>> Starting 4 threads
> > > > >>> Starting SPDK v18.10 / DPDK 18.08.0 initialization...
> > > > >>> [ DPDK EAL parameters: fio --no-shconf -c 0x1 -m 512 --file-
> > > > >>> prefix=spdk_pid3668 ]
> > > > >>> EAL: Detected 36 lcore(s)
> > > > >>> EAL: Detected 2 NUMA nodes
> > > > >>> EAL: No free hugepages reported in hugepages-1048576kB
> > > > >>> EAL: Probing VFIO support...
> > > > >>> EAL: PCI device 0000:1c:00.0 on NUMA socket 0
> > > > >>> EAL: probe driver: 8086:a54 spdk_nvme
> > > > >>> Jobs: 1 (f=0): [f(1),_(3)][100.0%][eta 00m:00s]
> > > > >>> 0000.1c.00.0: (groupid=0, jobs=4): err= 0: pid=3709: Mon Nov
> > > > >>> 5
> > > > >>> 02:28:29
> > > > >>> 2018
> > > > >>> mixed: IOPS=13.2k, BW=51.6MiB/s
> (54.1MB/s)(517MiB/10011msec)
> > > > >>> slat (nsec): min=109, max=15344, avg=127.91, stdev=73.44
> > > > >>> clat (usec): min=187, max=18715, avg=9683.80, stdev=650.54
> > > > >>> lat (usec): min=193, max=18715, avg=9683.93, stdev=650.53
> > > > >>> clat percentiles (usec):
> > > > >>> | 1.00th=[ 8717], 5.00th=[ 8848], 10.00th=[ 8979],
> > > > >>> 20.00th=[
> > > 9110],
> > > > >>> | 30.00th=[ 9241], 40.00th=[ 9372], 50.00th=[ 9503],
> > > > >>> 60.00th=[
> > > 9765],
> > > > >>> | 70.00th=[10028], 80.00th=[10159], 90.00th=[10552],
> > > > 95.00th=[10945],
> > > > >>> | 99.00th=[11338], 99.50th=[11338], 99.90th=[11731],
> > > > 99.95th=[13960],
> > > > >>> | 99.99th=[17695]
> > > > >>> bw ( KiB/s): min=13080, max=13304, per=25.00%,
> > > > >>> avg=13214.70, stdev=55.30, samples=80
> > > > >>> iops : min= 3270, max= 3326, avg=3303.65, stdev=13.84,
> > > > samples=80
> > > > >>> lat (usec) : 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
> > > > >>> lat (msec) : 2=0.01%, 4=0.02%, 10=70.85%, 20=29.11%
> > > > >>> cpu : usr=100.01%, sys=0.00%, ctx=23, majf=0, minf=0
> > > > >>> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%,
> 32=99.9%,
> > > > >>> >=64=0.0%
> > > > >>> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%,
> > 64=0.0%,
> > > > >>> >=64=0.0%
> > > > >>> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%,
> > > > >>> 64=0.0%,
> > > > >>> >=64=0.0%
> > > > >>> issued rwts: total=132300,0,0,0 short=0,0,0,0 dropped=0,0,0,0
> > > > >>> latency : target=0, window=0, percentile=100.00%, depth=32
> > > > >>> Run status group 0 (all jobs):
> > > > >>> MIXED: bw=51.6MiB/s (54.1MB/s), 51.6MiB/s-51.6MiB/s
> > (54.1MB/s-
> > > > >>> 54.1MB/s), io=517MiB (542MB), run=10011-10011msec
> > > > >>>
> > > > >>> _______________________________________________
> > > > >>> SPDK mailing list
> > > > >>> SPDK(a)lists.01.org
> > > > >>> https://lists.01.org/mailman/listinfo/spdk
> > > > >>> _______________________________________________
> > > > >>> SPDK mailing list
> > > > >>> SPDK(a)lists.01.org
> > > > >>> https://lists.01.org/mailman/listinfo/spdk
> > > > >>_______________________________________________
> > > > >>SPDK mailing list
> > > > >> SPDK(a)lists.01.org
> > > > >> https://lists.01.org/mailman/listinfo/spdk
> > > > >>_______________________________________________
> > > > >>SPDK mailing list
> > > > >> SPDK(a)lists.01.org
> > > > >> https://lists.01.org/mailman/listinfo/spdk
> > > > >
> > > > >
> > > > >--
> > > > >Маров Алексей
> > > > >_______________________________________________
> > > > >SPDK mailing list
> > > > >SPDK(a)lists.01.org
> > > > >https://lists.01.org/mailman/listinfo/spdk
> > > > >_______________________________________________
> > > > >SPDK mailing list
> > > > >SPDK(a)lists.01.org
> > > > >https://lists.01.org/mailman/listinfo/spdk
> > > >
> > > >
> > > > --
> > > > Маров Алексей
> > > > _______________________________________________
> > > > SPDK mailing list
> > > > SPDK(a)lists.01.org
> > > > https://lists.01.org/mailman/listinfo/spdk
> > > _______________________________________________
> > > SPDK mailing list
> > > SPDK(a)lists.01.org
> > > https://lists.01.org/mailman/listinfo/spdk
> > >
> > --
> >
> > Regards,
> > Andrey
> > _______________________________________________
> > SPDK mailing list
> > SPDK(a)lists.01.org
> > https://lists.01.org/mailman/listinfo/spdk
> > _______________________________________________
> > SPDK mailing list
> > SPDK(a)lists.01.org
> > https://lists.01.org/mailman/listinfo/spdk
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [SPDK] FIO NVMe Performance Results
@ 2018-11-05 20:45 Gruher, Joseph R
0 siblings, 0 replies; 25+ messages in thread
From: Gruher, Joseph R @ 2018-11-05 20:45 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 962 bytes --]
> -----Original Message-----
> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Andrey Kuzmin
> Sent: Monday, November 5, 2018 12:18 PM
> To: Storage Performance Development Kit <spdk(a)lists.01.org>
> Subject: Re: [SPDK] FIO NVMe Performance Results
>
> And, to wrap this up, your p4500 results are what one would expect from a
> modern ssd, in half a million iops range, not 10K IOPS your original post
> claimed as reasonable.
If you read the thread, it states we can achieve an expected result of 477K IOPS on our P4500 2TB drive only when using FIO with the Linux kernel NVMe driver. When using the same FIO workload with SPDK and the FIO SPDK plugin we can only achieve around 12K IOPS on the same system with the same physical disk. This is the problem statement which started the thread and the issue we are attempting to debug. You're really muddying the waters here so it would be great if you could just not reply further.
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [SPDK] FIO NVMe Performance Results
@ 2018-11-05 20:39 Gruher, Joseph R
0 siblings, 0 replies; 25+ messages in thread
From: Gruher, Joseph R @ 2018-11-05 20:39 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 40658 bytes --]
Attempting to fall back to earlier revisions has not yielded any improvements for me. All tests use the same P4500 physical disk. FIO files are as shown before, running a 4 thread, queue depth 32 per thread, 4KB random read workload.
SPDK 18.10, kernel 4.18.16, FIO 3.11 = 12.9k IOPS
SPDK 18.10, kernel 4.16.14, FIO 3.11 = 11.9k IOPS
SPDK 18.04, kernel 4.18.16, FIO 3.11 = 12.9k IOPS
SPDK 18.04, kernel 4.16.14, FIO 3.11 = 13.0k IOPS
SPDK 18.04, kernel 4.16.16, FIO 3.3 = 13.1k IOPS
Kernel NVMe driver, kernel 4.16.16, FIO 3.3 = 477k IOPS
Kernel NVMe driver, kernel 4.18.16, FIO 3.11 = 477k IOPS
I am curious if the comment below about uio_pci_generic instead of vfio can lead us anywhere, or maybe there's some OS setting or BIOS setting that needs to be toggled...
For the folks working inside Intel, I can also give you direct access to the test system, if that would help.
Thanks,
Joe
> -----Original Message-----
> From: Gruher, Joseph R
> Sent: Monday, November 5, 2018 11:05 AM
> To: Storage Performance Development Kit <spdk(a)lists.01.org>
> Subject: RE: [SPDK] FIO NVMe Performance Results
>
> I've had good FIO NVMe results in the past we well, with SPDK 18.04 on
> Ubuntu 18.04, with kernel 4.16.14. That was on a different, but similar,
> hardware platform.
>
> I suspect something has changed now in SPDK 18.10 and/or kernel 4.18.16. I
> can try going back to earlier versions of each to see if I can recreate earlier
> good results and find where things went wrong.
>
> We're not running in a VM - this is a bare metal Xeon server with Skylake
> CPUs.
>
> Thanks,
> Joe
>
> > -----Original Message-----
> > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Howell,
> > Seth
> > Sent: Monday, November 5, 2018 10:32 AM
> > To: Storage Performance Development Kit <spdk(a)lists.01.org>
> > Subject: Re: [SPDK] FIO NVMe Performance Results
> >
> > Hi Joseph,
> > I attempted to verify your results on my system. I am using fedora 27
> > with kernel version 4.17.3 with a p3700 drive. I cloned fio from
> > https://github.com/axboe/fio and checked out the appropriate tag. I
> > got appx 408k iops.
> > Typically we run spdk with the iommu enabled. I noticed that your
> > devices are binding to uio_pci_generic instead of vfio. This shouldn't
> > make any difference at all, but I just want to make sure you aren't
> > running in a VM or something like that and passing through your NVMe
> > drives and that's why you aren't using the IOMMU. That could explain a
> > deeper configuration difference that could help us know where to look for
> the disparity.
> >
> > Please see the contents of my configuration files and output below:
> >
> > Test.ini:
> > [global]
> > rw=randrw
> > rwmixread=100
> > numjobs=4
> > iodepth=32
> > bs=4k
> > direct=1
> > thread=1
> > time_based=1
> > ramp_time=0
> > runtime=10
> > ioengine=spdk_bdev
> > spdk_conf=./nvme.conf
> > group_reporting=1
> > unified_rw_reporting=1
> > exitall=1
> > randrepeat=0
> > norandommap=1
> > [0000.81.00.0]
> > filename=Nvme0n1
> >
> > nvme.conf:
> > [Nvme]
> > TransportId "trtype:PCIe traddr:0000.81.00.0" Nvme0
> >
> > Sudo
> >
> LD_PRELOAD=/home/sethhowe/Desktop/Development_Folders/spdk/exam
> > ples/bdev/fio_plugin/fio_plugin ./fio ./test.ini >./out,txt 2>&1
> >
> > Out.txt:
> > 0000.81.00.0: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B,
> > (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=32 ...
> > fio-3.11
> > Starting 4 threads
> > Starting SPDK v18.10 / DPDK 18.08.0 initialization...
> > [ DPDK EAL parameters: fio --no-shconf -c 0x1
> > --file-prefix=spdk_pid782339 ]
> > EAL: Detected 16 lcore(s)
> > EAL: Detected 2 NUMA nodes
> > EAL: No free hugepages reported in hugepages-1048576kB
> > EAL: Probing VFIO support...
> > EAL: VFIO support initialized
> > EAL: PCI device 0000:81:00.0 on NUMA socket 1
> > EAL: probe driver: 8086:953 spdk_nvme
> >
> > 0000.81.00.0: (groupid=0, jobs=4): err= 0: pid=782359: Mon Nov 5
> > 10:58:59
> > 2018
> > mixed: IOPS=410k, BW=1600MiB/s (1678MB/s)(15.6GiB/10001msec)
> > slat (nsec): min=179, max=19985, avg=245.97, stdev=122.96
> > clat (usec): min=8, max=1821, avg=311.82, stdev=167.14
> > lat (usec): min=8, max=1822, avg=312.06, stdev=167.14
> > clat percentiles (usec):
> > | 1.00th=[ 24], 5.00th=[ 48], 10.00th=[ 78], 20.00th=[ 141],
> > | 30.00th=[ 200], 40.00th=[ 258], 50.00th=[ 314], 60.00th=[ 371],
> > | 70.00th=[ 429], 80.00th=[ 486], 90.00th=[ 545], 95.00th=[ 570],
> > | 99.00th=[ 594], 99.50th=[ 594], 99.90th=[ 627], 99.95th=[ 652],
> > | 99.99th=[ 676]
> > bw ( KiB/s): min=407264, max=416040, per=25.01%, avg=409742.55,
> > stdev=1337.38, samples=76
> > iops : min=101816, max=104010, avg=102435.61, stdev=334.41,
> > samples=76
> > lat (usec) : 10=0.01%, 20=0.46%, 50=4.88%, 100=8.41%, 250=25.29%
> > lat (usec) : 500=43.15%, 750=17.81%, 1000=0.01%
> > lat (msec) : 2=0.01%
> > cpu : usr=99.78%, sys=0.00%, ctx=26, majf=0, minf=0
> > IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.9%,
> > >=64=0.0%
> > submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
> > >=64=0.0%
> > complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.1%, 32=0.1%, 64=0.0%,
> > >=64=0.0%
> > issued rwts: total=4096867,0,0,0 short=0,0,0,0 dropped=0,0,0,0
> > latency : target=0, window=0, percentile=100.00%, depth=32
> >
> > Run status group 0 (all jobs):
> > MIXED: bw=1600MiB/s (1678MB/s), 1600MiB/s-1600MiB/s (1678MB/s-
> > 1678MB/s), io=15.6GiB (16.8GB), run=10001-10001msec
> >
> > Thanks,
> >
> > Seth
> >
> >
> >
> > -----Original Message-----
> > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Andrey
> > Kuzmin
> > Sent: Monday, November 5, 2018 9:48 AM
> > To: Storage Performance Development Kit <spdk(a)lists.01.org>
> > Subject: Re: [SPDK] FIO NVMe Performance Results
> >
> > On Mon, Nov 5, 2018, 18:29 Gruher, Joseph R
> > <joseph.r.gruher(a)intel.com>
> > wrote:
> >
> > > My FIO version is v3.11.
> > >
> > > I tried the bdev FIO plugin as suggested, but performance is about
> > > the same as the other SPDK FIO plugin, I observed only 10.4K IOPS in
> > > a quick 4KB random read test. See configuration and results below.
> > >
> >
> > As 12 ms median read latency for an nvme ssd is far beyond anything
> > reasonable, why don't you just perf this run, to see where the time is
> > being spent?
> >
> > Regards,
> > Andrey
> >
> > >
> > >
> > > don(a)donst201:~/fio/single/rr$ cat nvme.conf [Nvme] TransportId
> > > "trtype:PCIe traddr:0000.1c.00.0" Nvme0
> > >
> > >
> > > don(a)donst201:~/fio/single/rr$ cat 1c-bdev.ini |grep -v '#'
> > > [global]
> > > rw=randrw
> > > rwmixread=100
> > > numjobs=4
> > > iodepth=32
> > > bs=4k
> > > direct=1
> > > thread=1
> > > time_based=1
> > > ramp_time=0
> > > runtime=10
> > > ioengine=spdk_bdev
> > > spdk_conf=./nvme.conf
> > > group_reporting=1
> > > unified_rw_reporting=1
> > > exitall=1
> > > randrepeat=0
> > > norandommap=1
> > > [0000.1c.00.0]
> > > filename=Nvme0n1
> > >
> > >
> > > don(a)donst201:~/fio/single/rr$ sudo
> > >
> >
> LD_PRELOAD=/home/don/install/spdk/spdk/examples/bdev/fio_plugin/fio_p
> > l
> > > ugin
> > > fio 1c-bdev.ini
> > > 0000.1c.00.0: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B,
> > > (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=32 ...
> > > fio-3.11
> > > Starting 4 threads
> > > Starting SPDK v18.10 / DPDK 18.08.0 initialization...
> > > [ DPDK EAL parameters: fio --no-shconf -c 0x1
> > > --file-prefix=spdk_pid4865 ]
> > > EAL: Detected 36 lcore(s)
> > > EAL: Detected 2 NUMA nodes
> > > EAL: No free hugepages reported in hugepages-1048576kB
> > > EAL: Probing VFIO support...
> > > EAL: PCI device 0000:03:00.0 on NUMA socket 0
> > > EAL: probe driver: 8086:f1a5 spdk_nvme
> > > EAL: PCI device 0000:1c:00.0 on NUMA socket 0
> > > EAL: probe driver: 8086:a54 spdk_nvme
> > > EAL: PCI device 0000:1d:00.0 on NUMA socket 0
> > > EAL: probe driver: 8086:a54 spdk_nvme
> > > EAL: PCI device 0000:5e:00.0 on NUMA socket 0
> > > EAL: probe driver: 8086:a54 spdk_nvme
> > > EAL: PCI device 0000:5f:00.0 on NUMA socket 0
> > > EAL: probe driver: 8086:a54 spdk_nvme
> > > EAL: PCI device 0000:62:00.0 on NUMA socket 0
> > > EAL: probe driver: 8086:a54 spdk_nvme
> > > EAL: PCI device 0000:63:00.0 on NUMA socket 0
> > > EAL: probe driver: 8086:a54 spdk_nvme
> > > EAL: PCI device 0000:64:00.0 on NUMA socket 0
> > > EAL: probe driver: 8086:a54 spdk_nvme
> > > EAL: PCI device 0000:65:00.0 on NUMA socket 0
> > > EAL: probe driver: 8086:a54 spdk_nvme
> > > EAL: PCI device 0000:da:00.0 on NUMA socket 1
> > > EAL: probe driver: 8086:a54 spdk_nvme
> > > EAL: PCI device 0000:db:00.0 on NUMA socket 1
> > > EAL: probe driver: 8086:a54 spdk_nvme
> > > EAL: PCI device 0000:dc:00.0 on NUMA socket 1
> > > EAL: probe driver: 8086:a54 spdk_nvme
> > > EAL: PCI device 0000:dd:00.0 on NUMA socket 1
> > > EAL: probe driver: 8086:a54 spdk_nvme
> > > EAL: PCI device 0000:e0:00.0 on NUMA socket 1
> > > EAL: probe driver: 8086:a54 spdk_nvme
> > > EAL: PCI device 0000:e1:00.0 on NUMA socket 1
> > > EAL: probe driver: 8086:a54 spdk_nvme
> > > EAL: PCI device 0000:e2:00.0 on NUMA socket 1
> > > EAL: probe driver: 8086:a54 spdk_nvme
> > > EAL: PCI device 0000:e3:00.0 on NUMA socket 1
> > > EAL: probe driver: 8086:a54 spdk_nvme
> > > Jobs: 4 (f=4): [r(4)][100.0%][r=45.7MiB/s][r=11.7k IOPS][eta
> > > 00m:00s]
> > > 0000.1c.00.0: (groupid=0, jobs=4): err= 0: pid=4906: Mon Nov 5
> > > 15:23:47
> > > 2018
> > > mixed: IOPS=10.4k, BW=40.5MiB/s (42.5MB/s)(406MiB/10012msec)
> > > slat (nsec): min=201, max=15831, avg=238.11, stdev=148.78
> > > clat (usec): min=124, max=31244, avg=12330.69, stdev=1697.98
> > > lat (usec): min=130, max=31244, avg=12330.93, stdev=1697.98
> > > clat percentiles (usec):
> > > | 1.00th=[ 9634], 5.00th=[10159], 10.00th=[10552], 20.00th=[10945],
> > > | 30.00th=[11338], 40.00th=[11731], 50.00th=[12125],
> > 60.00th=[12518],
> > > | 70.00th=[13042], 80.00th=[13566], 90.00th=[14353],
> > 95.00th=[15008],
> > > | 99.00th=[16712], 99.50th=[18744], 99.90th=[27395],
> > 99.95th=[29754],
> > > | 99.99th=[30278]
> > > bw ( KiB/s): min= 8784, max=11952, per=24.99%, avg=10375.53,
> > > stdev=929.68, samples=80
> > > iops : min= 2196, max= 2988, avg=2593.85, stdev=232.41,
> > > samples=80
> > > lat (usec) : 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
> > > lat (msec) : 2=0.01%, 4=0.02%, 10=2.71%, 20=96.91%, 50=0.33%
> > > cpu : usr=99.99%, sys=0.01%, ctx=28, majf=0, minf=0
> > > IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.9%,
> > > >=64=0.0%
> > > submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
> > > >=64=0.0%
> > > complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%,
> > > 64=0.0%,
> > > >=64=0.0%
> > > issued rwts: total=103907,0,0,0 short=0,0,0,0 dropped=0,0,0,0
> > > latency : target=0, window=0, percentile=100.00%, depth=32
> > > Run status group 0 (all jobs):
> > > MIXED: bw=40.5MiB/s (42.5MB/s), 40.5MiB/s-40.5MiB/s
> > > (42.5MB/s-42.5MB/s), io=406MiB (426MB), run=10012-10012msec
> > >
> > >
> > > Thanks,
> > > Joe
> > >
> > > > -----Original Message-----
> > > > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of ?????
> > ???????
> > > > Sent: Monday, November 5, 2018 12:45 AM
> > > > To: Storage Performance Development Kit <spdk(a)lists.01.org>
> > > > Subject: Re: [SPDK] FIO NVMe Performance Results
> > > >
> > > > Ok, Do you see some performance issues using general bdev fio
> > > > plugin
> > > with
> > > > you configuration ?
> > > >
> > > > Best Regards
> > > > Alex
> > > >
> > > >
> > > > >Понедельник, 5 ноября 2018, 11:39 +03:00 от "Cao, Gang"
> > > > <gang.cao(a)intel.com>:
> > > > >
> > > > >Yes. This is for the bdev version of SPDK FIO plugin and you can
> > > > >assign
> > > the
> > > > conf file as the parameter in the FIO configuration.
> > > > >
> > > > >Previously, it's another SPDK FIO plugin for the user space NVMe
> > > > >driver
> > > > only.
> > > > >
> > > > >These are two different FIO plugins.
> > > > >
> > > > >Thanks,
> > > > >Gang
> > > > >
> > > > >-----Original Message-----
> > > > >From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Маров
> > > > >Алексей
> > > > >Sent: Monday, November 5, 2018 4:32 PM
> > > > >To: Storage Performance Development Kit < spdk(a)lists.01.org >
> > > > >Subject: Re: [SPDK] FIO NVMe Performance Results
> > > > >
> > > > >Sorry for incomplete message I have sent before
> > > > >
> > > > >Hello Gang.
> > > > >
> > > > >I just curious, why would you run fio with SPDK using this conf
> > > > >file,
> > > with is a
> > > > bit different from what is suggested in here
> > https://github.com/spdk/spdk/tree/master/examples/bdev/fio_plugin
> > > > .
> > > > >
> > > > >I mean may be you should create conf file for spdk like this one :
> > > > >
> > > > >nvme.conf :
> > > > >[Nvme]
> > > > > TransportId "trtype:PCIe traddr:0000.1c.00.0" Nvme0
> > > > >
> > > > >and provide it in fio config.
> > > > >
> > > > >fio.ini :
> > > > >[global]
> > > > >ioengine=spdk_bdev
> > > > >spdk_conf=....../nvme.conf
> > > > >rw=randrw
> > > > >rwmixread=100
> > > > >numjobs=4
> > > > >iodepth=32
> > > > >bs=4k
> > > > >direct=1
> > > > >thread=1
> > > > >time_based=1
> > > > >ramp_time=0
> > > > >runtime=10
> > > > >group_reporting=1
> > > > >unified_rw_reporting=1
> > > > >exitall=1
> > > > >randrepeat=0
> > > > >norandommap=1
> > > > >
> > > > >[0000:06:00.0]
> > > > >filename=Nvme0n1
> > > > >
> > > > >and run like this
> > > > >
> > > > ># LD_PRELOAD=examples/bdev/fio_plugin/fio_plugin fio
> > > > >examples/bdev/fio_plugin/fio.ini
> > > > >
> > > > >Just to note. The recommended version of fio for SPDK is 3.3
> > > > >
> > > > >Best Regards
> > > > >Alex
> > > > >
> > > > >
> > > > >>Понедельник, 5 ноября 2018, 9:53 +03:00 от "Cao, Gang" <
> > > > gang.cao(a)intel.com >:
> > > > >>
> > > > >>Thanks. This is interesting especially for your finding on queue
> > > > >>depth
> > > 1
> > > > case. We are trying to reproduce this issue locally same as your
> > > > configuration.
> > > > >>
> > > > >>Thanks,
> > > > >>Gang
> > > > >>
> > > > >>-----Original Message-----
> > > > >>From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of
> > > > >>Gruher, Joseph R
> > > > >>Sent: Monday, November 5, 2018 1:32 PM
> > > > >>To: Storage Performance Development Kit < spdk(a)lists.01.org >
> > > > >>Subject: Re: [SPDK] FIO NVMe Performance Results
> > > > >>
> > > > >>Interesting catch. It seems like FIO is actually not accurately
> > > reporting the
> > > > number of jobs in the report out for the SPDK case. If I disable
> > > > the group_reporting parameter I do get a printout for four
> > > > separate jobs in
> > > the
> > > > SPDK case, so I think it is really running four jobs there, and
> > > > just
> > > summarizing
> > > > incorrectly. I also see four separate CPU cores go to 100% when I
> > > > run
> > > that
> > > > FIO test case, indicating four jobs are active.
> > > > >>
> > > > >>Also, I don't think would explain the difference in performance.
> > > Using an
> > > > aggregate queue depth of 32 versus 128 shouldn't lead to a drop in
> > > > performance from 377K to 13K.
> > > > >>
> > > > >>I retested SPDK with multiple separate job descriptions as you
> > > suggested
> > > > anyway, using four separate jobs instead of relying on the
> > > > numjobs=4 parameter, and the results are below. In that SPDK test
> > > > with four
> > > separate
> > > > FIO job statements, each with queue depth 32, I still only
> > > > achieved 12.5K IOPS.
> > > > >>
> > > > >>I then re-ran the kernel NVMe driver test case with only a
> > > > >>single job
> > > for
> > > > comparison, and even with only 1 job and queue depth 32 it still
> > > > achieves 254K IOPS.
> > > > >>
> > > > >>We might be on the right track with a problem around queue depth
> > > > however. If I run the kernel NVMe driver case with just 1 job and
> > > > queue depth of only 1, I then get 11.2K IOPS, which is around the
> > > > performance
> > > I get
> > > > in the SPDK test case. Perhaps the SPDK case is not actually
> > > > queuing
> > > IOs on
> > > > the device for some reason?
> > > > >>
> > > > >>
> > > > >>don(a)donst201:~/fio/single/rr$ cat nvme2n1.ini |grep -v '#'
> > > > >>[global]
> > > > >>rw=randrw
> > > > >>rwmixread=100
> > > > >>numjobs=1
> > > > >>iodepth=32
> > > > >>bs=4k
> > > > >>direct=1
> > > > >>thread=1
> > > > >>time_based=1
> > > > >>ramp_time=0
> > > > >>runtime=10
> > > > >>ioengine=libaio
> > > > >>group_reporting=1
> > > > >>unified_rw_reporting=1
> > > > >>exitall=1
> > > > >>randrepeat=0
> > > > >>norandommap=1
> > > > >>[nvme2n1]
> > > > >>filename=/dev/nvme2n1
> > > > >>
> > > > >>
> > > > >>don(a)donst201:~/fio/single/rr$ sudo fio nvme2n1.ini
> > > > >>nvme2n1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B,
> > > > >>(T) 4096B-4096B, ioengine=libaio, iodepth=32
> > > > >>fio-3.11
> > > > >>Starting 1 thread
> > > > >>Jobs: 1 (f=1): [r(1)][100.0%][r=1025MiB/s][r=262k IOPS][eta
> > > > >>00m:00s]
> > > > >>nvme2n1: (groupid=0, jobs=1): err= 0: pid=2422: Mon Nov 5
> > > > >>05:19:13
> > > > >>2018
> > > > >> mixed: IOPS=254k, BW=992MiB/s
> (1040MB/s)(9922MiB/10001msec)
> > > > >> slat (nsec): min=1185, max=319052, avg=1611.64, stdev=808.40
> > > > >> clat (usec): min=2, max=4743, avg=124.02, stdev=121.00
> > > > >> lat (usec): min=6, max=4744, avg=125.68, stdev=121.02
> > > > >> clat percentiles (usec):
> > > > >> | 1.00th=[ 9], 5.00th=[ 56], 10.00th=[ 61], 20.00th=[
> > > 73],
> > > > >> | 30.00th=[ 80], 40.00th=[ 91], 50.00th=[ 110], 60.00th=[
> > > 118],
> > > > >> | 70.00th=[ 130], 80.00th=[ 157], 90.00th=[ 202],
> > > > >>95.00th=[ 253],
> > > > >> | 99.00th=[ 408], 99.50th=[ 498], 99.90th=[ 1795],
> > > > >>99.95th=[ 3064],
> > > > >> | 99.99th=[ 3785]
> > > > >> bw ( KiB/s): min=612240, max=1078792, per=100.00%,
> > > > >>avg=1015996.40,
> > > > stdev=106032.96, samples=20
> > > > >> iops : min=153060, max=269698, avg=253998.90,
> > stdev=26508.40,
> > > > samples=20
> > > > >> lat (usec) : 4=0.01%, 10=1.47%, 20=1.19%, 50=0.17%, 100=42.03%
> > > > >> lat (usec) : 250=49.95%, 500=4.70%, 750=0.28%, 1000=0.06%
> > > > >> lat (msec) : 2=0.06%, 4=0.09%, 10=0.01%
> > > > >> cpu : usr=21.45%, sys=51.11%, ctx=530348, majf=0, minf=139
> > > > >> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%,
> 32=100.0%,
> > > > >=64=0.0%
> > > > >> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%,
> 64=0.0%,
> > > > >=64=0.0%
> > > > >> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%,
> > > > >> 64=0.0%,
> > > > >>>=64=0.0%
> > > > >> issued rwts: total=2540023,0,0,0 short=0,0,0,0 dropped=0,0,0,0
> > > > >> latency : target=0, window=0, percentile=100.00%, depth=32
> > > > >>Run status group 0 (all jobs):
> > > > >> MIXED: bw=992MiB/s (1040MB/s), 992MiB/s-992MiB/s (1040MB/s-
> > > > 1040MB/s), io=9922MiB (10.4GB), run=10001-10001msec Disk stats
> > > > (read/write):
> > > > >> nvme2n1: ios=2514356/0, merge=0/0, ticks=400/0,
> > in_queue=297612,
> > > > >>util=99.06%
> > > > >>
> > > > >>
> > > > >>don(a)donst201:~/fio/single/rr$ cat 1c.ini |grep -v '#'
> > > > >>[global]
> > > > >>rw=randrw
> > > > >>rwmixread=100
> > > > >>numjobs=1
> > > > >>iodepth=32
> > > > >>bs=4k
> > > > >>direct=1
> > > > >>thread=1
> > > > >>time_based=1
> > > > >>ramp_time=0
> > > > >>runtime=10
> > > >
> > >>ioengine=/home/don/install/spdk/spdk/examples/nvme/fio_plugin/fio_
> > > > >>pl
> > > > ug
> > > > >>in
> > > > >>group_reporting=1
> > > > >>unified_rw_reporting=1
> > > > >>exitall=1
> > > > >>randrepeat=0
> > > > >>norandommap=1
> > > > >>[0000.1c.00.0-1]
> > > > >>filename=trtype=PCIe traddr=0000.1c.00.0 ns=1 [0000.1c.00.0-2]
> > > > >>filename=trtype=PCIe traddr=0000.1c.00.0 ns=1 [0000.1c.00.0-3]
> > > > >>filename=trtype=PCIe traddr=0000.1c.00.0 ns=1 [0000.1c.00.0-4]
> > > > >>filename=trtype=PCIe traddr=0000.1c.00.0 ns=1
> > > > >>
> > > > >>
> > > > >>don(a)donst201:~/fio/single/rr$ sudo fio 1c.ini
> > > > >>0000.1c.00.0-1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W)
> > > > >>4096B-4096B,
> > > > >>(T) 4096B-4096B, ioengine=spdk, iodepth=32
> > > > >>0000.1c.00.0-2: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W)
> > > > >>4096B-4096B,
> > > > >>(T) 4096B-4096B, ioengine=spdk, iodepth=32
> > > > >>0000.1c.00.0-3: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W)
> > > > >>4096B-4096B,
> > > > >>(T) 4096B-4096B, ioengine=spdk, iodepth=32
> > > > >>0000.1c.00.0-4: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W)
> > > > >>4096B-4096B,
> > > > >>(T) 4096B-4096B, ioengine=spdk, iodepth=32
> > > > >>fio-3.11
> > > > >>Starting 4 threads
> > > > >>Starting SPDK v18.10 / DPDK 18.08.0 initialization...
> > > > >>[ DPDK EAL parameters: fio --no-shconf -c 0x1 -m 512
> > > > >>--file-prefix=spdk_pid4270 ]
> > > > >>EAL: Detected 36 lcore(s)
> > > > >>EAL: Detected 2 NUMA nodes
> > > > >>EAL: No free hugepages reported in hugepages-1048576kB
> > > > >>EAL: Probing VFIO support...
> > > > >>EAL: PCI device 0000:1c:00.0 on NUMA socket 0
> > > > >>EAL: probe driver: 8086:a54 spdk_nvme
> > > > >>Jobs: 1 (f=0): [_(3),f(1)][100.0%][eta 00m:00s]
> > > > >>0000.1c.00.0-1: (groupid=0, jobs=4): err= 0: pid=4310: Mon Nov
> > > > >>5
> > > > >>05:16:03 2018
> > > > >> mixed: IOPS=12.5k, BW=48.0MiB/s (51.4MB/s)(490MiB/10010msec)
> > > > >> slat (nsec): min=98, max=15092, avg=120.02, stdev=90.07
> > > > >> clat (usec): min=55, max=19757, avg=10207.22, stdev=884.06
> > > > >> lat (usec): min=60, max=19757, avg=10207.34, stdev=884.06
> > > > >> clat percentiles (usec):
> > > > >> | 1.00th=[ 8979], 5.00th=[ 9110], 10.00th=[ 9372],
> > > > >>20.00th=[ 9503],
> > > > >> | 30.00th=[ 9765], 40.00th=[ 9896], 50.00th=[10159],
> > > > >>60.00th=[10290],
> > > > >> | 70.00th=[10552], 80.00th=[10814], 90.00th=[11207],
> > > > >>95.00th=[11600],
> > > > >> | 99.00th=[12911], 99.50th=[14353], 99.90th=[17695],
> > > > >>99.95th=[18744],
> > > > >> | 99.99th=[19006]
> > > > >> bw ( KiB/s): min=12008, max=12976, per=25.00%, avg=12535.70,
> > > > stdev=306.81, samples=80
> > > > >> iops : min= 3002, max= 3244, avg=3133.85, stdev=76.77,
> > > samples=80
> > > > >> lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%,
> > 1000=0.01%
> > > > >> lat (msec) : 2=0.01%, 4=0.02%, 10=45.13%, 20=54.83%
> > > > >> cpu : usr=100.01%, sys=0.00%, ctx=27, majf=0, minf=0
> > > > >> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.9%,
> > > > >=64=0.0%
> > > > >> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%,
> 64=0.0%,
> > > > >=64=0.0%
> > > > >> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%,
> > > > >> 64=0.0%,
> > > > >>>=64=0.0%
> > > > >> issued rwts: total=125506,0,0,0 short=0,0,0,0 dropped=0,0,0,0
> > > > >> latency : target=0, window=0, percentile=100.00%, depth=32
> > > > >>Run status group 0 (all jobs):
> > > > >> MIXED: bw=48.0MiB/s (51.4MB/s), 48.0MiB/s-48.0MiB/s
> > > > >>(51.4MB/s-51.4MB/s), io=490MiB (514MB), run=10010-10010msec
> > > > >>
> > > > >>
> > > > >>don(a)donst201:~/fio/single/rr$ cat nvme2n1.ini |grep -v '#'
> > > > >>[global]
> > > > >>rw=randrw
> > > > >>rwmixread=100
> > > > >>numjobs=1
> > > > >>iodepth=1
> > > > >>bs=4k
> > > > >>direct=1
> > > > >>thread=1
> > > > >>time_based=1
> > > > >>ramp_time=0
> > > > >>runtime=10
> > > > >>ioengine=libaio
> > > > >>group_reporting=1
> > > > >>unified_rw_reporting=1
> > > > >>exitall=1
> > > > >>randrepeat=0
> > > > >>norandommap=1
> > > > >>[nvme2n1]
> > > > >>filename=/dev/nvme2n1
> > > > >>
> > > > >>
> > > > >>don(a)donst201:~/fio/single/rr$ sudo fio nvme2n1.ini
> > > > >>nvme2n1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B,
> > > > >>(T) 4096B-4096B, ioengine=libaio, iodepth=1
> > > > >>fio-3.11
> > > > >>Starting 1 thread
> > > > >>Jobs: 1 (f=1): [r(1)][100.0%][r=43.6MiB/s][r=11.2k IOPS][eta
> > > > >>00m:00s]
> > > > >>nvme2n1: (groupid=0, jobs=1): err= 0: pid=2525: Mon Nov 5
> > > > >>05:22:57
> > > > >>2018
> > > > >> mixed: IOPS=11.2k, BW=43.6MiB/s (45.7MB/s)(436MiB/10001msec)
> > > > >> slat (nsec): min=4848, max=208595, avg=5126.33, stdev=771.27
> > > > >> clat (nsec): min=1845, max=3405.8k, avg=83004.99,
> stdev=33577.36
> > > > >> lat (usec): min=15, max=3411, avg=88.26, stdev=33.59
> > > > >> clat percentiles (usec):
> > > > >> | 1.00th=[ 15], 5.00th=[ 58], 10.00th=[ 59], 20.00th=[
> > > 59],
> > > > >> | 30.00th=[ 60], 40.00th=[ 77], 50.00th=[ 77], 60.00th=[
> > > 78],
> > > > >> | 70.00th=[ 111], 80.00th=[ 113], 90.00th=[ 113],
> > > > >>95.00th=[ 115],
> > > > >> | 99.00th=[ 186], 99.50th=[ 186], 99.90th=[ 190],
> > > > >>99.95th=[ 192],
> > > > >> | 99.99th=[ 375]
> > > > >> bw ( KiB/s): min=43816, max=45016, per=99.98%, avg=44639.58,
> > > > stdev=312.88, samples=19
> > > > >> iops : min=10954, max=11254, avg=11159.89, stdev=78.22,
> > > > samples=19
> > > > >> lat (usec) : 2=0.01%, 20=2.83%, 50=0.01%, 100=64.57%,
> 250=32.59%
> > > > >> lat (usec) : 500=0.01%, 750=0.01%
> > > > >> lat (msec) : 2=0.01%, 4=0.01%
> > > > >> cpu : usr=4.02%, sys=10.33%, ctx=111630, majf=0, minf=12
> > > > >> IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%,
> 32=0.0%,
> > > > >=64=0.0%
> > > > >> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%,
> 64=0.0%,
> > > > >=64=0.0%
> > > > >> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%,
> > > > >> 64=0.0%,
> > > > >>>=64=0.0%
> > > > >> issued rwts: total=111631,0,0,0 short=0,0,0,0 dropped=0,0,0,0
> > > > >> latency : target=0, window=0, percentile=100.00%, depth=1
> > > > >>Run status group 0 (all jobs):
> > > > >> MIXED: bw=43.6MiB/s (45.7MB/s), 43.6MiB/s-43.6MiB/s
> (45.7MB/s-
> > > > 45.7MB/s), io=436MiB (457MB), run=10001-10001msec Disk stats
> > > > (read/write):
> > > > >> nvme2n1: ios=110275/0, merge=0/0, ticks=0/0, in_queue=8596,
> > > > >>util=86.22%
> > > > >>
> > > > >>
> > > > >>Thanks,
> > > > >>Joe
> > > > >>
> > > > >>> -----Original Message-----
> > > > >>> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of
> > > > >>> Cao, Gang
> > > > >>> Sent: Sunday, November 4, 2018 8:40 PM
> > > > >>> To: Storage Performance Development Kit < spdk(a)lists.01.org >
> > > > >>> Subject: Re: [SPDK] FIO NVMe Performance Results
> > > > >>>
> > > > >>> For the SPDK plugin, from the below result, looks like only 1
> > > > >>> job started with
> > > > >>> 32 queue depth:
> > > > >>>
> > > > >>> Jobs: 1 (f=0): [f(1),_(3)][100.0%][eta 00m:00s] <<<<
> > > > >>>
> > > > >>> Try to update the FIO configuration file as
> > > > >>>
> > > > >>> [job1]
> > > > >>> filename=trtype=PCIe traddr=0000.1c.00.0 ns=1 [job2]
> > > > >>> filename=trtype=PCIe
> > > > >>> traddr=0000.1c.00.0 ns=1 ....
> > > > >>>
> > > > >>> The kernel FIO is with 4 jobs and each job of 32 queue depth.
> > > > >>> For SPDK, you can configure as the same, or just configure two
> > > > >>> jobs and each job of 64 queue depth. The overall queue depth
> > > > >>> to the device is
> > > > same of 128.
> > > > >>>
> > > > >>> Thanks,
> > > > >>> Gang
> > > > >>>
> > > > >>> -----Original Message-----
> > > > >>> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of
> > > > >>> Gruher, Joseph R
> > > > >>> Sent: Monday, November 5, 2018 10:49 AM
> > > > >>> To: Storage Performance Development Kit < spdk(a)lists.01.org >
> > > > >>> Subject: [SPDK] FIO NVMe Performance Results
> > > > >>>
> > > > >>> Hi folks-
> > > > >>>
> > > > >>> I'm testing SPDK 18.10 on Ubuntu 18.04 with kernel 4.18.16. I
> > > > >>> used FIO with the kernel NVMe driver to measure the
> > > > >>> performance of a local (PCIe
> > > > >>> attached) Intel P4500 NVMe device on a 4KB random read
> > > > >>> workload and obtained 477K IOPS, roughly in line with the drive
> spec.
> > > > >>> Then I tested the same drive with the SPDK FIO plugin and only
> > > > >>> achieved 13K IOPS. The FIO test files and the results are
> > > > >>> pasted below. Any ideas where I'm going wrong here?
> > > > >>>
> > > > >>> Thanks!
> > > > >>>
> > > > >>>
> > > > >>> don(a)donst201:~/fio/single/rr$ ls
> > > > >>> /sys/devices/pci0000:17/0000:17:02.0/0000:1c:00.0/nvme/nvme1/
> > > > >>> address cntlid dev device firmware_rev model nvme2n1
> > > > >>> power rescan_controller reset_controller serial state
> > > > >>> subsysnqn subsystem transport uevent
> > > > >>>
> > > > >>>
> > > > >>> don(a)donst201:~/fio/single/rr$ cat nvme2n1.ini |grep -v '#'
> > > > >>> [global]
> > > > >>> rw=randrw
> > > > >>> rwmixread=100
> > > > >>> numjobs=4
> > > > >>> iodepth=32
> > > > >>> bs=4k
> > > > >>> direct=1
> > > > >>> thread=1
> > > > >>> time_based=1
> > > > >>> ramp_time=0
> > > > >>> runtime=10
> > > > >>> ioengine=libaio
> > > > >>> group_reporting=1
> > > > >>> unified_rw_reporting=1
> > > > >>> exitall=1
> > > > >>> randrepeat=0
> > > > >>> norandommap=1
> > > > >>> [nvme2n1]
> > > > >>> filename=/dev/nvme2n1
> > > > >>>
> > > > >>>
> > > > >>> don(a)donst201:~/fio/single/rr$ sudo fio nvme2n1.ini
> > > > >>> nvme2n1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W)
> > > > >>> 4096B-4096B,
> > > > >>> (T) 4096B-4096B, ioengine=libaio, iodepth=32 ...
> > > > >>> fio-3.11
> > > > >>> Starting 4 threads
> > > > >>> Jobs: 4 (f=4): [r(4)][100.0%][r=1874MiB/s][r=480k IOPS][eta
> > > > >>> 00m:00s]
> > > > >>> nvme2n1: (groupid=0, jobs=4): err= 0: pid=2575: Mon Nov 5
> > > > >>> 02:26:25
> > > > 2018
> > > > >>> mixed: IOPS=477k, BW=1862MiB/s
> > (1952MB/s)(18.2GiB/10001msec)
> > > > >>> slat (nsec): min=1295, max=333088, avg=2216.35, stdev=927.89
> > > > >>> clat (nsec): min=506, max=5818.8k, avg=265924.16,
> > stdev=223761.59
> > > > >>> lat (usec): min=6, max=5821, avg=268.19, stdev=223.76
> > > > >>> clat percentiles (usec):
> > > > >>> | 1.00th=[ 13], 5.00th=[ 74], 10.00th=[ 87], 20.00th=[
> > > 115],
> > > > >>> | 30.00th=[ 139], 40.00th=[ 167], 50.00th=[ 204],
> > > > >>> 60.00th=[
> > > 247],
> > > > >>> | 70.00th=[ 306], 80.00th=[ 388], 90.00th=[ 523],
> > > > >>> 95.00th=[
> > > 660],
> > > > >>> | 99.00th=[ 988], 99.50th=[ 1156], 99.90th=[ 2343],
> > > > >>> 99.95th=[
> > > 3195],
> > > > >>> | 99.99th=[ 4424]
> > > > >>> bw ( KiB/s): min=448192, max=483104, per=25.00%,
> > > > >>> avg=476597.10, stdev=6748.78, samples=80
> > > > >>> iops : min=112048, max=120776, avg=119149.27,
> > > stdev=1687.20,
> > > > >>> samples=80
> > > > >>> lat (nsec) : 750=0.01%
> > > > >>> lat (usec) : 10=0.52%, 20=1.25%, 50=0.25%, 100=13.38%,
> > 250=45.13%
> > > > >>> lat (usec) : 500=28.23%, 750=8.08%, 1000=2.22%
> > > > >>> lat (msec) : 2=0.82%, 4=0.10%, 10=0.02%
> > > > >>> cpu : usr=14.08%, sys=36.02%, ctx=1930207, majf=0,
> > > minf=697
> > > > >>> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%,
> > 32=100.0%,
> > > > >>> >=64=0.0%
> > > > >>> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%,
> > 64=0.0%,
> > > > >>> >=64=0.0%
> > > > >>> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%,
> > > > >>> 64=0.0%,
> > > > >>> >=64=0.0%
> > > > >>> issued rwts: total=4766095,0,0,0 short=0,0,0,0 dropped=0,0,0,0
> > > > >>> latency : target=0, window=0, percentile=100.00%, depth=32
> > > > >>> Run status group 0 (all jobs):
> > > > >>> MIXED: bw=1862MiB/s (1952MB/s), 1862MiB/s-1862MiB/s
> > (1952MB/s-
> > > > >>> 1952MB/s), io=18.2GiB (19.5GB), run=10001-10001msec Disk stats
> > > > >>> (read/write):
> > > > >>> nvme2n1: ios=4709190/0, merge=0/0, ticks=4096/0,
> > > > in_queue=1388752,
> > > > >>> util=100.00%
> > > > >>>
> > > > >>>
> > > > >>> don(a)donst201:~/fio/single/rr$ sudo
> > > > >>> /home/don/install/spdk/spdk/scripts/setup.sh
> > > > >>> Active mountpoints on /dev/nvme0n1, so not binding PCI dev
> > > > >>> 0000:03:00.0
> > > > >>> 0000:1c:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > > >>> 0000:1d:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > > >>> 0000:5e:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > > >>> 0000:5f:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > > >>> 0000:62:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > > >>> 0000:63:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > > >>> 0000:64:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > > >>> 0000:65:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > > >>> 0000:da:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > > >>> 0000:db:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > > >>> 0000:dc:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > > >>> 0000:dd:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > > >>> 0000:e0:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > > >>> 0000:e1:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > > >>> 0000:e2:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > > >>> 0000:e3:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > > >>> 0000:00:04.0 (8086 2021): ioatdma -> uio_pci_generic
> > > > >>> 0000:00:04.1 (8086 2021): ioatdma -> uio_pci_generic
> > > > >>> 0000:00:04.2 (8086 2021): ioatdma -> uio_pci_generic
> > > > >>> 0000:00:04.3 (8086 2021): ioatdma -> uio_pci_generic
> > > > >>> 0000:00:04.4 (8086 2021): ioatdma -> uio_pci_generic
> > > > >>> 0000:00:04.5 (8086 2021): ioatdma -> uio_pci_generic
> > > > >>> 0000:00:04.6 (8086 2021): ioatdma -> uio_pci_generic
> > > > >>> 0000:00:04.7 (8086 2021): ioatdma -> uio_pci_generic
> > > > >>> 0000:80:04.0 (8086 2021): ioatdma -> uio_pci_generic
> > > > >>> 0000:80:04.1 (8086 2021): ioatdma -> uio_pci_generic
> > > > >>> 0000:80:04.2 (8086 2021): ioatdma -> uio_pci_generic
> > > > >>> 0000:80:04.3 (8086 2021): ioatdma -> uio_pci_generic
> > > > >>> 0000:80:04.4 (8086 2021): ioatdma -> uio_pci_generic
> > > > >>> 0000:80:04.5 (8086 2021): ioatdma -> uio_pci_generic
> > > > >>> 0000:80:04.6 (8086 2021): ioatdma -> uio_pci_generic
> > > > >>> 0000:80:04.7 (8086 2021): ioatdma -> uio_pci_generic
> > > > >>>
> > > > >>>
> > > > >>> don(a)donst201:~/fio/single/rr$ cat 1c.ini |grep -v '#'
> > > > >>> [global]
> > > > >>> rw=randrw
> > > > >>> rwmixread=100
> > > > >>> numjobs=4
> > > > >>> iodepth=32
> > > > >>> bs=4k
> > > > >>> direct=1
> > > > >>> thread=1
> > > > >>> time_based=1
> > > > >>> ramp_time=0
> > > > >>> runtime=10
> > > > >>>
> > > >
> > ioengine=/home/don/install/spdk/spdk/examples/nvme/fio_plugin/fio_pl
> > > > >>> ug
> > > > >>> in
> > > > >>> group_reporting=1
> > > > >>> unified_rw_reporting=1
> > > > >>> exitall=1
> > > > >>> randrepeat=0
> > > > >>> norandommap=1
> > > > >>> [0000.1c.00.0]
> > > > >>> filename=trtype=PCIe traddr=0000.1c.00.0 ns=1
> > > > >>>
> > > > >>>
> > > > >>> don(a)donst201:~/fio/single/rr$ sudo fio 1c.ini
> > > > >>> 0000.1c.00.0: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W)
> > > > >>> 4096B-4096B,
> > > > >>> (T) 4096B-4096B, ioengine=spdk, iodepth=32 ...
> > > > >>> fio-3.11
> > > > >>> Starting 4 threads
> > > > >>> Starting SPDK v18.10 / DPDK 18.08.0 initialization...
> > > > >>> [ DPDK EAL parameters: fio --no-shconf -c 0x1 -m 512 --file-
> > > > >>> prefix=spdk_pid3668 ]
> > > > >>> EAL: Detected 36 lcore(s)
> > > > >>> EAL: Detected 2 NUMA nodes
> > > > >>> EAL: No free hugepages reported in hugepages-1048576kB
> > > > >>> EAL: Probing VFIO support...
> > > > >>> EAL: PCI device 0000:1c:00.0 on NUMA socket 0
> > > > >>> EAL: probe driver: 8086:a54 spdk_nvme
> > > > >>> Jobs: 1 (f=0): [f(1),_(3)][100.0%][eta 00m:00s]
> > > > >>> 0000.1c.00.0: (groupid=0, jobs=4): err= 0: pid=3709: Mon Nov
> > > > >>> 5
> > > > >>> 02:28:29
> > > > >>> 2018
> > > > >>> mixed: IOPS=13.2k, BW=51.6MiB/s
> (54.1MB/s)(517MiB/10011msec)
> > > > >>> slat (nsec): min=109, max=15344, avg=127.91, stdev=73.44
> > > > >>> clat (usec): min=187, max=18715, avg=9683.80, stdev=650.54
> > > > >>> lat (usec): min=193, max=18715, avg=9683.93, stdev=650.53
> > > > >>> clat percentiles (usec):
> > > > >>> | 1.00th=[ 8717], 5.00th=[ 8848], 10.00th=[ 8979],
> > > > >>> 20.00th=[
> > > 9110],
> > > > >>> | 30.00th=[ 9241], 40.00th=[ 9372], 50.00th=[ 9503],
> > > > >>> 60.00th=[
> > > 9765],
> > > > >>> | 70.00th=[10028], 80.00th=[10159], 90.00th=[10552],
> > > > 95.00th=[10945],
> > > > >>> | 99.00th=[11338], 99.50th=[11338], 99.90th=[11731],
> > > > 99.95th=[13960],
> > > > >>> | 99.99th=[17695]
> > > > >>> bw ( KiB/s): min=13080, max=13304, per=25.00%,
> > > > >>> avg=13214.70, stdev=55.30, samples=80
> > > > >>> iops : min= 3270, max= 3326, avg=3303.65, stdev=13.84,
> > > > samples=80
> > > > >>> lat (usec) : 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
> > > > >>> lat (msec) : 2=0.01%, 4=0.02%, 10=70.85%, 20=29.11%
> > > > >>> cpu : usr=100.01%, sys=0.00%, ctx=23, majf=0, minf=0
> > > > >>> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%,
> 32=99.9%,
> > > > >>> >=64=0.0%
> > > > >>> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%,
> > 64=0.0%,
> > > > >>> >=64=0.0%
> > > > >>> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%,
> > > > >>> 64=0.0%,
> > > > >>> >=64=0.0%
> > > > >>> issued rwts: total=132300,0,0,0 short=0,0,0,0 dropped=0,0,0,0
> > > > >>> latency : target=0, window=0, percentile=100.00%, depth=32
> > > > >>> Run status group 0 (all jobs):
> > > > >>> MIXED: bw=51.6MiB/s (54.1MB/s), 51.6MiB/s-51.6MiB/s
> > (54.1MB/s-
> > > > >>> 54.1MB/s), io=517MiB (542MB), run=10011-10011msec
> > > > >>>
> > > > >>> _______________________________________________
> > > > >>> SPDK mailing list
> > > > >>> SPDK(a)lists.01.org
> > > > >>> https://lists.01.org/mailman/listinfo/spdk
> > > > >>> _______________________________________________
> > > > >>> SPDK mailing list
> > > > >>> SPDK(a)lists.01.org
> > > > >>> https://lists.01.org/mailman/listinfo/spdk
> > > > >>_______________________________________________
> > > > >>SPDK mailing list
> > > > >> SPDK(a)lists.01.org
> > > > >> https://lists.01.org/mailman/listinfo/spdk
> > > > >>_______________________________________________
> > > > >>SPDK mailing list
> > > > >> SPDK(a)lists.01.org
> > > > >> https://lists.01.org/mailman/listinfo/spdk
> > > > >
> > > > >
> > > > >--
> > > > >Маров Алексей
> > > > >_______________________________________________
> > > > >SPDK mailing list
> > > > >SPDK(a)lists.01.org
> > > > >https://lists.01.org/mailman/listinfo/spdk
> > > > >_______________________________________________
> > > > >SPDK mailing list
> > > > >SPDK(a)lists.01.org
> > > > >https://lists.01.org/mailman/listinfo/spdk
> > > >
> > > >
> > > > --
> > > > Маров Алексей
> > > > _______________________________________________
> > > > SPDK mailing list
> > > > SPDK(a)lists.01.org
> > > > https://lists.01.org/mailman/listinfo/spdk
> > > _______________________________________________
> > > SPDK mailing list
> > > SPDK(a)lists.01.org
> > > https://lists.01.org/mailman/listinfo/spdk
> > >
> > --
> >
> > Regards,
> > Andrey
> > _______________________________________________
> > SPDK mailing list
> > SPDK(a)lists.01.org
> > https://lists.01.org/mailman/listinfo/spdk
> > _______________________________________________
> > SPDK mailing list
> > SPDK(a)lists.01.org
> > https://lists.01.org/mailman/listinfo/spdk
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [SPDK] FIO NVMe Performance Results
@ 2018-11-05 20:18 Andrey Kuzmin
0 siblings, 0 replies; 25+ messages in thread
From: Andrey Kuzmin @ 2018-11-05 20:18 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 4124 bytes --]
On Mon, Nov 5, 2018, 22:59 Andrey Kuzmin <andrey.v.kuzmin(a)gmail.com> wrote:
>
>
> On Mon, Nov 5, 2018, 22:38 Gruher, Joseph R <joseph.r.gruher(a)intel.com>
> wrote:
>
>> > >> When running FIO with queue depth the latency number FIO reports is
>> > >> not just the latency of the device but also includes time spent in
>> > >> the FIO queue. We saw 10400 IOPS meaning we're processing an IO on
>> > >> average every 96usec. For a queue depth of 128 that means each IO
>> > >> will spend an average of 96 * 128 = 12308usec in the FIO queue before
>> > >> being submitted to the device.
>> > >
>> > >
>> > > No. You've got four threads with qd=32 each, so your multiplicator
>> > > above is 32, not 128. Furthermore, typically in calcs like this it's
>> > > half the queue depth, not the full depth.
>>
>> You can view it that way, but then each thread also had 1/4 of the
>> overall IOPS, so it's a wash in the calculation.
>>
>> > > So the FIO average clat number of 12330usec looks reasonable to me.
>> > >>
>> > >
>> > > No, it's way off. No modern ssd will run your 128 io's sequentially,
>> > > it will all be run (almost) concurrently, especially under random
>> > > read. So the expected end-to-end latency here should be close to the
>> > > physical one (around 100 usec per 4K), with IOPS in hundred thousands
>> > range.
>>
>> The same math holds up the same for the high performing test case with
>> the kernel NVMe driver, where we achieve 477K IOPS, a result which is
>> aligned with the P4500 drive spec. 477K IOPS is an average of an IO
>> completing every 2.1usec. Each individual IO doesn't complete that fast,
>> because we know the device latency is more like 85usec, but with
>> parallelism that's the average rate at which IOs are being completed by the
>> device overall (this is different from the individual IO latency). We also
>> know our overall aggregate queue depth is 32x4=128, so then the time to
>> complete the queue is going to be the length of the queue multiplied by the
>> average rate at which IOs are completing, which is 2.1 * 128 = 268.2usec,
>> which is almost exactly what we see FIO report (see below).
>
>
> The above has nothing to do with how high queue depth io works inside ssd.
> On the contrary, it's just a direct consequence of how fio
> submission/completion code path runs. You get a completion every 2 usec,
> and fio doesn't submit next io until it handles that completion. Thus you
> get the exact average latency vs. queue depth/completion rate match in the
> below math.
>
Just to give you an idea. You're running four fio threads, and each gets
its own hw queue from the ssd. Each hw queue, with depth 32, processes
requests concurrently as they - given random workload - address different
nand dies (so there's little if any wait in the queue). Judging by the
math, there are about 50 dies inside your ssd so that each read waits on
another read on the same die to complete (nand dies process commands
sequentially).
So that's why you're getting around 250us latency, twice the typical 4K
read latency of the modern SSDs. Fio queue wait time you quote is an
illusion, just another way to describe multiple queues/multiple dies
processing inside ssd.
And, to wrap this up, your p4500 results are what one would expect from a
modern ssd, in half a million iops range, not 10K IOPS your original post
claimed as reasonable.
Regards,
Andrey
> Regards,
> A.
>
> The relationship between number of IOs, IOPS, and latency is fixed, it's a
>> calculation analogous to distance/time=velocity.
>>
>> mixed: IOPS=477k, BW=1862MiB/s (1952MB/s)(18.2GiB/10001msec)
>> slat (nsec): min=1295, max=333088, avg=2216.35, stdev=927.89
>> clat (nsec): min=506, max=5818.8k, avg=265924.16, stdev=223761.59
>> lat (usec): min=6, max=5821, avg=268.19, stdev=223.76
>>
>> _______________________________________________
>> SPDK mailing list
>> SPDK(a)lists.01.org
>> https://lists.01.org/mailman/listinfo/spdk
>>
> --
>
> Regards,
> Andrey
>
--
Regards,
Andrey
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [SPDK] FIO NVMe Performance Results
@ 2018-11-05 19:59 Andrey Kuzmin
0 siblings, 0 replies; 25+ messages in thread
From: Andrey Kuzmin @ 2018-11-05 19:59 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 3035 bytes --]
On Mon, Nov 5, 2018, 22:38 Gruher, Joseph R <joseph.r.gruher(a)intel.com>
wrote:
> > >> When running FIO with queue depth the latency number FIO reports is
> > >> not just the latency of the device but also includes time spent in
> > >> the FIO queue. We saw 10400 IOPS meaning we're processing an IO on
> > >> average every 96usec. For a queue depth of 128 that means each IO
> > >> will spend an average of 96 * 128 = 12308usec in the FIO queue before
> > >> being submitted to the device.
> > >
> > >
> > > No. You've got four threads with qd=32 each, so your multiplicator
> > > above is 32, not 128. Furthermore, typically in calcs like this it's
> > > half the queue depth, not the full depth.
>
> You can view it that way, but then each thread also had 1/4 of the overall
> IOPS, so it's a wash in the calculation.
>
> > > So the FIO average clat number of 12330usec looks reasonable to me.
> > >>
> > >
> > > No, it's way off. No modern ssd will run your 128 io's sequentially,
> > > it will all be run (almost) concurrently, especially under random
> > > read. So the expected end-to-end latency here should be close to the
> > > physical one (around 100 usec per 4K), with IOPS in hundred thousands
> > range.
>
> The same math holds up the same for the high performing test case with the
> kernel NVMe driver, where we achieve 477K IOPS, a result which is aligned
> with the P4500 drive spec. 477K IOPS is an average of an IO completing
> every 2.1usec. Each individual IO doesn't complete that fast, because we
> know the device latency is more like 85usec, but with parallelism that's
> the average rate at which IOs are being completed by the device overall
> (this is different from the individual IO latency). We also know our
> overall aggregate queue depth is 32x4=128, so then the time to complete the
> queue is going to be the length of the queue multiplied by the average rate
> at which IOs are completing, which is 2.1 * 128 = 268.2usec, which is
> almost exactly what we see FIO report (see below).
The above has nothing to do with how high queue depth io works inside ssd.
On the contrary, it's just a direct consequence of how fio
submission/completion code path runs. You get a completion every 2 usec,
and fio doesn't submit next io until it handles that completion. Thus you
get the exact average latency vs. queue depth/completion rate match in the
below math.
Regards,
A.
The relationship between number of IOs, IOPS, and latency is fixed, it's a
> calculation analogous to distance/time=velocity.
>
> mixed: IOPS=477k, BW=1862MiB/s (1952MB/s)(18.2GiB/10001msec)
> slat (nsec): min=1295, max=333088, avg=2216.35, stdev=927.89
> clat (nsec): min=506, max=5818.8k, avg=265924.16, stdev=223761.59
> lat (usec): min=6, max=5821, avg=268.19, stdev=223.76
>
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
>
--
Regards,
Andrey
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [SPDK] FIO NVMe Performance Results
@ 2018-11-05 19:38 Gruher, Joseph R
0 siblings, 0 replies; 25+ messages in thread
From: Gruher, Joseph R @ 2018-11-05 19:38 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 2296 bytes --]
> >> When running FIO with queue depth the latency number FIO reports is
> >> not just the latency of the device but also includes time spent in
> >> the FIO queue. We saw 10400 IOPS meaning we're processing an IO on
> >> average every 96usec. For a queue depth of 128 that means each IO
> >> will spend an average of 96 * 128 = 12308usec in the FIO queue before
> >> being submitted to the device.
> >
> >
> > No. You've got four threads with qd=32 each, so your multiplicator
> > above is 32, not 128. Furthermore, typically in calcs like this it's
> > half the queue depth, not the full depth.
You can view it that way, but then each thread also had 1/4 of the overall IOPS, so it's a wash in the calculation.
> > So the FIO average clat number of 12330usec looks reasonable to me.
> >>
> >
> > No, it's way off. No modern ssd will run your 128 io's sequentially,
> > it will all be run (almost) concurrently, especially under random
> > read. So the expected end-to-end latency here should be close to the
> > physical one (around 100 usec per 4K), with IOPS in hundred thousands
> range.
The same math holds up the same for the high performing test case with the kernel NVMe driver, where we achieve 477K IOPS, a result which is aligned with the P4500 drive spec. 477K IOPS is an average of an IO completing every 2.1usec. Each individual IO doesn't complete that fast, because we know the device latency is more like 85usec, but with parallelism that's the average rate at which IOs are being completed by the device overall (this is different from the individual IO latency). We also know our overall aggregate queue depth is 32x4=128, so then the time to complete the queue is going to be the length of the queue multiplied by the average rate at which IOs are completing, which is 2.1 * 128 = 268.2usec, which is almost exactly what we see FIO report (see below). The relationship between number of IOs, IOPS, and latency is fixed, it's a calculation analogous to distance/time=velocity.
mixed: IOPS=477k, BW=1862MiB/s (1952MB/s)(18.2GiB/10001msec)
slat (nsec): min=1295, max=333088, avg=2216.35, stdev=927.89
clat (nsec): min=506, max=5818.8k, avg=265924.16, stdev=223761.59
lat (usec): min=6, max=5821, avg=268.19, stdev=223.76
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [SPDK] FIO NVMe Performance Results
@ 2018-11-05 19:04 Gruher, Joseph R
0 siblings, 0 replies; 25+ messages in thread
From: Gruher, Joseph R @ 2018-11-05 19:04 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 37465 bytes --]
I've had good FIO NVMe results in the past we well, with SPDK 18.04 on Ubuntu 18.04, with kernel 4.16.14. That was on a different, but similar, hardware platform.
I suspect something has changed now in SPDK 18.10 and/or kernel 4.18.16. I can try going back to earlier versions of each to see if I can recreate earlier good results and find where things went wrong.
We're not running in a VM - this is a bare metal Xeon server with Skylake CPUs.
Thanks,
Joe
> -----Original Message-----
> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Howell, Seth
> Sent: Monday, November 5, 2018 10:32 AM
> To: Storage Performance Development Kit <spdk(a)lists.01.org>
> Subject: Re: [SPDK] FIO NVMe Performance Results
>
> Hi Joseph,
> I attempted to verify your results on my system. I am using fedora 27 with
> kernel version 4.17.3 with a p3700 drive. I cloned fio from
> https://github.com/axboe/fio and checked out the appropriate tag. I got
> appx 408k iops.
> Typically we run spdk with the iommu enabled. I noticed that your devices
> are binding to uio_pci_generic instead of vfio. This shouldn't make any
> difference at all, but I just want to make sure you aren't running in a VM or
> something like that and passing through your NVMe drives and that's why
> you aren't using the IOMMU. That could explain a deeper configuration
> difference that could help us know where to look for the disparity.
>
> Please see the contents of my configuration files and output below:
>
> Test.ini:
> [global]
> rw=randrw
> rwmixread=100
> numjobs=4
> iodepth=32
> bs=4k
> direct=1
> thread=1
> time_based=1
> ramp_time=0
> runtime=10
> ioengine=spdk_bdev
> spdk_conf=./nvme.conf
> group_reporting=1
> unified_rw_reporting=1
> exitall=1
> randrepeat=0
> norandommap=1
> [0000.81.00.0]
> filename=Nvme0n1
>
> nvme.conf:
> [Nvme]
> TransportId "trtype:PCIe traddr:0000.81.00.0" Nvme0
>
> Sudo
> LD_PRELOAD=/home/sethhowe/Desktop/Development_Folders/spdk/exam
> ples/bdev/fio_plugin/fio_plugin ./fio ./test.ini >./out,txt 2>&1
>
> Out.txt:
> 0000.81.00.0: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T)
> 4096B-4096B, ioengine=spdk_bdev, iodepth=32 ...
> fio-3.11
> Starting 4 threads
> Starting SPDK v18.10 / DPDK 18.08.0 initialization...
> [ DPDK EAL parameters: fio --no-shconf -c 0x1 --file-prefix=spdk_pid782339 ]
> EAL: Detected 16 lcore(s)
> EAL: Detected 2 NUMA nodes
> EAL: No free hugepages reported in hugepages-1048576kB
> EAL: Probing VFIO support...
> EAL: VFIO support initialized
> EAL: PCI device 0000:81:00.0 on NUMA socket 1
> EAL: probe driver: 8086:953 spdk_nvme
>
> 0000.81.00.0: (groupid=0, jobs=4): err= 0: pid=782359: Mon Nov 5 10:58:59
> 2018
> mixed: IOPS=410k, BW=1600MiB/s (1678MB/s)(15.6GiB/10001msec)
> slat (nsec): min=179, max=19985, avg=245.97, stdev=122.96
> clat (usec): min=8, max=1821, avg=311.82, stdev=167.14
> lat (usec): min=8, max=1822, avg=312.06, stdev=167.14
> clat percentiles (usec):
> | 1.00th=[ 24], 5.00th=[ 48], 10.00th=[ 78], 20.00th=[ 141],
> | 30.00th=[ 200], 40.00th=[ 258], 50.00th=[ 314], 60.00th=[ 371],
> | 70.00th=[ 429], 80.00th=[ 486], 90.00th=[ 545], 95.00th=[ 570],
> | 99.00th=[ 594], 99.50th=[ 594], 99.90th=[ 627], 99.95th=[ 652],
> | 99.99th=[ 676]
> bw ( KiB/s): min=407264, max=416040, per=25.01%, avg=409742.55,
> stdev=1337.38, samples=76
> iops : min=101816, max=104010, avg=102435.61, stdev=334.41,
> samples=76
> lat (usec) : 10=0.01%, 20=0.46%, 50=4.88%, 100=8.41%, 250=25.29%
> lat (usec) : 500=43.15%, 750=17.81%, 1000=0.01%
> lat (msec) : 2=0.01%
> cpu : usr=99.78%, sys=0.00%, ctx=26, majf=0, minf=0
> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.9%,
> >=64=0.0%
> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
> >=64=0.0%
> complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.1%, 32=0.1%, 64=0.0%,
> >=64=0.0%
> issued rwts: total=4096867,0,0,0 short=0,0,0,0 dropped=0,0,0,0
> latency : target=0, window=0, percentile=100.00%, depth=32
>
> Run status group 0 (all jobs):
> MIXED: bw=1600MiB/s (1678MB/s), 1600MiB/s-1600MiB/s (1678MB/s-
> 1678MB/s), io=15.6GiB (16.8GB), run=10001-10001msec
>
> Thanks,
>
> Seth
>
>
>
> -----Original Message-----
> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Andrey Kuzmin
> Sent: Monday, November 5, 2018 9:48 AM
> To: Storage Performance Development Kit <spdk(a)lists.01.org>
> Subject: Re: [SPDK] FIO NVMe Performance Results
>
> On Mon, Nov 5, 2018, 18:29 Gruher, Joseph R <joseph.r.gruher(a)intel.com>
> wrote:
>
> > My FIO version is v3.11.
> >
> > I tried the bdev FIO plugin as suggested, but performance is about the
> > same as the other SPDK FIO plugin, I observed only 10.4K IOPS in a
> > quick 4KB random read test. See configuration and results below.
> >
>
> As 12 ms median read latency for an nvme ssd is far beyond anything
> reasonable, why don't you just perf this run, to see where the time is being
> spent?
>
> Regards,
> Andrey
>
> >
> >
> > don(a)donst201:~/fio/single/rr$ cat nvme.conf [Nvme] TransportId
> > "trtype:PCIe traddr:0000.1c.00.0" Nvme0
> >
> >
> > don(a)donst201:~/fio/single/rr$ cat 1c-bdev.ini |grep -v '#'
> > [global]
> > rw=randrw
> > rwmixread=100
> > numjobs=4
> > iodepth=32
> > bs=4k
> > direct=1
> > thread=1
> > time_based=1
> > ramp_time=0
> > runtime=10
> > ioengine=spdk_bdev
> > spdk_conf=./nvme.conf
> > group_reporting=1
> > unified_rw_reporting=1
> > exitall=1
> > randrepeat=0
> > norandommap=1
> > [0000.1c.00.0]
> > filename=Nvme0n1
> >
> >
> > don(a)donst201:~/fio/single/rr$ sudo
> >
> LD_PRELOAD=/home/don/install/spdk/spdk/examples/bdev/fio_plugin/fio_p
> l
> > ugin
> > fio 1c-bdev.ini
> > 0000.1c.00.0: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B,
> > (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=32 ...
> > fio-3.11
> > Starting 4 threads
> > Starting SPDK v18.10 / DPDK 18.08.0 initialization...
> > [ DPDK EAL parameters: fio --no-shconf -c 0x1
> > --file-prefix=spdk_pid4865 ]
> > EAL: Detected 36 lcore(s)
> > EAL: Detected 2 NUMA nodes
> > EAL: No free hugepages reported in hugepages-1048576kB
> > EAL: Probing VFIO support...
> > EAL: PCI device 0000:03:00.0 on NUMA socket 0
> > EAL: probe driver: 8086:f1a5 spdk_nvme
> > EAL: PCI device 0000:1c:00.0 on NUMA socket 0
> > EAL: probe driver: 8086:a54 spdk_nvme
> > EAL: PCI device 0000:1d:00.0 on NUMA socket 0
> > EAL: probe driver: 8086:a54 spdk_nvme
> > EAL: PCI device 0000:5e:00.0 on NUMA socket 0
> > EAL: probe driver: 8086:a54 spdk_nvme
> > EAL: PCI device 0000:5f:00.0 on NUMA socket 0
> > EAL: probe driver: 8086:a54 spdk_nvme
> > EAL: PCI device 0000:62:00.0 on NUMA socket 0
> > EAL: probe driver: 8086:a54 spdk_nvme
> > EAL: PCI device 0000:63:00.0 on NUMA socket 0
> > EAL: probe driver: 8086:a54 spdk_nvme
> > EAL: PCI device 0000:64:00.0 on NUMA socket 0
> > EAL: probe driver: 8086:a54 spdk_nvme
> > EAL: PCI device 0000:65:00.0 on NUMA socket 0
> > EAL: probe driver: 8086:a54 spdk_nvme
> > EAL: PCI device 0000:da:00.0 on NUMA socket 1
> > EAL: probe driver: 8086:a54 spdk_nvme
> > EAL: PCI device 0000:db:00.0 on NUMA socket 1
> > EAL: probe driver: 8086:a54 spdk_nvme
> > EAL: PCI device 0000:dc:00.0 on NUMA socket 1
> > EAL: probe driver: 8086:a54 spdk_nvme
> > EAL: PCI device 0000:dd:00.0 on NUMA socket 1
> > EAL: probe driver: 8086:a54 spdk_nvme
> > EAL: PCI device 0000:e0:00.0 on NUMA socket 1
> > EAL: probe driver: 8086:a54 spdk_nvme
> > EAL: PCI device 0000:e1:00.0 on NUMA socket 1
> > EAL: probe driver: 8086:a54 spdk_nvme
> > EAL: PCI device 0000:e2:00.0 on NUMA socket 1
> > EAL: probe driver: 8086:a54 spdk_nvme
> > EAL: PCI device 0000:e3:00.0 on NUMA socket 1
> > EAL: probe driver: 8086:a54 spdk_nvme
> > Jobs: 4 (f=4): [r(4)][100.0%][r=45.7MiB/s][r=11.7k IOPS][eta 00m:00s]
> > 0000.1c.00.0: (groupid=0, jobs=4): err= 0: pid=4906: Mon Nov 5
> > 15:23:47
> > 2018
> > mixed: IOPS=10.4k, BW=40.5MiB/s (42.5MB/s)(406MiB/10012msec)
> > slat (nsec): min=201, max=15831, avg=238.11, stdev=148.78
> > clat (usec): min=124, max=31244, avg=12330.69, stdev=1697.98
> > lat (usec): min=130, max=31244, avg=12330.93, stdev=1697.98
> > clat percentiles (usec):
> > | 1.00th=[ 9634], 5.00th=[10159], 10.00th=[10552], 20.00th=[10945],
> > | 30.00th=[11338], 40.00th=[11731], 50.00th=[12125],
> 60.00th=[12518],
> > | 70.00th=[13042], 80.00th=[13566], 90.00th=[14353],
> 95.00th=[15008],
> > | 99.00th=[16712], 99.50th=[18744], 99.90th=[27395],
> 99.95th=[29754],
> > | 99.99th=[30278]
> > bw ( KiB/s): min= 8784, max=11952, per=24.99%, avg=10375.53,
> > stdev=929.68, samples=80
> > iops : min= 2196, max= 2988, avg=2593.85, stdev=232.41,
> > samples=80
> > lat (usec) : 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
> > lat (msec) : 2=0.01%, 4=0.02%, 10=2.71%, 20=96.91%, 50=0.33%
> > cpu : usr=99.99%, sys=0.01%, ctx=28, majf=0, minf=0
> > IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.9%,
> > >=64=0.0%
> > submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
> > >=64=0.0%
> > complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%,
> > >=64=0.0%
> > issued rwts: total=103907,0,0,0 short=0,0,0,0 dropped=0,0,0,0
> > latency : target=0, window=0, percentile=100.00%, depth=32
> > Run status group 0 (all jobs):
> > MIXED: bw=40.5MiB/s (42.5MB/s), 40.5MiB/s-40.5MiB/s
> > (42.5MB/s-42.5MB/s), io=406MiB (426MB), run=10012-10012msec
> >
> >
> > Thanks,
> > Joe
> >
> > > -----Original Message-----
> > > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of ?????
> ???????
> > > Sent: Monday, November 5, 2018 12:45 AM
> > > To: Storage Performance Development Kit <spdk(a)lists.01.org>
> > > Subject: Re: [SPDK] FIO NVMe Performance Results
> > >
> > > Ok, Do you see some performance issues using general bdev fio
> > > plugin
> > with
> > > you configuration ?
> > >
> > > Best Regards
> > > Alex
> > >
> > >
> > > >Понедельник, 5 ноября 2018, 11:39 +03:00 от "Cao, Gang"
> > > <gang.cao(a)intel.com>:
> > > >
> > > >Yes. This is for the bdev version of SPDK FIO plugin and you can
> > > >assign
> > the
> > > conf file as the parameter in the FIO configuration.
> > > >
> > > >Previously, it's another SPDK FIO plugin for the user space NVMe
> > > >driver
> > > only.
> > > >
> > > >These are two different FIO plugins.
> > > >
> > > >Thanks,
> > > >Gang
> > > >
> > > >-----Original Message-----
> > > >From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Маров
> > > >Алексей
> > > >Sent: Monday, November 5, 2018 4:32 PM
> > > >To: Storage Performance Development Kit < spdk(a)lists.01.org >
> > > >Subject: Re: [SPDK] FIO NVMe Performance Results
> > > >
> > > >Sorry for incomplete message I have sent before
> > > >
> > > >Hello Gang.
> > > >
> > > >I just curious, why would you run fio with SPDK using this conf
> > > >file,
> > with is a
> > > bit different from what is suggested in
> > > here
> https://github.com/spdk/spdk/tree/master/examples/bdev/fio_plugin
> > > .
> > > >
> > > >I mean may be you should create conf file for spdk like this one :
> > > >
> > > >nvme.conf :
> > > >[Nvme]
> > > > TransportId "trtype:PCIe traddr:0000.1c.00.0" Nvme0
> > > >
> > > >and provide it in fio config.
> > > >
> > > >fio.ini :
> > > >[global]
> > > >ioengine=spdk_bdev
> > > >spdk_conf=....../nvme.conf
> > > >rw=randrw
> > > >rwmixread=100
> > > >numjobs=4
> > > >iodepth=32
> > > >bs=4k
> > > >direct=1
> > > >thread=1
> > > >time_based=1
> > > >ramp_time=0
> > > >runtime=10
> > > >group_reporting=1
> > > >unified_rw_reporting=1
> > > >exitall=1
> > > >randrepeat=0
> > > >norandommap=1
> > > >
> > > >[0000:06:00.0]
> > > >filename=Nvme0n1
> > > >
> > > >and run like this
> > > >
> > > ># LD_PRELOAD=examples/bdev/fio_plugin/fio_plugin fio
> > > >examples/bdev/fio_plugin/fio.ini
> > > >
> > > >Just to note. The recommended version of fio for SPDK is 3.3
> > > >
> > > >Best Regards
> > > >Alex
> > > >
> > > >
> > > >>Понедельник, 5 ноября 2018, 9:53 +03:00 от "Cao, Gang" <
> > > gang.cao(a)intel.com >:
> > > >>
> > > >>Thanks. This is interesting especially for your finding on queue
> > > >>depth
> > 1
> > > case. We are trying to reproduce this issue locally same as your
> > > configuration.
> > > >>
> > > >>Thanks,
> > > >>Gang
> > > >>
> > > >>-----Original Message-----
> > > >>From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Gruher,
> > > >>Joseph R
> > > >>Sent: Monday, November 5, 2018 1:32 PM
> > > >>To: Storage Performance Development Kit < spdk(a)lists.01.org >
> > > >>Subject: Re: [SPDK] FIO NVMe Performance Results
> > > >>
> > > >>Interesting catch. It seems like FIO is actually not accurately
> > reporting the
> > > number of jobs in the report out for the SPDK case. If I disable
> > > the group_reporting parameter I do get a printout for four separate
> > > jobs in
> > the
> > > SPDK case, so I think it is really running four jobs there, and just
> > summarizing
> > > incorrectly. I also see four separate CPU cores go to 100% when I
> > > run
> > that
> > > FIO test case, indicating four jobs are active.
> > > >>
> > > >>Also, I don't think would explain the difference in performance.
> > Using an
> > > aggregate queue depth of 32 versus 128 shouldn't lead to a drop in
> > > performance from 377K to 13K.
> > > >>
> > > >>I retested SPDK with multiple separate job descriptions as you
> > suggested
> > > anyway, using four separate jobs instead of relying on the numjobs=4
> > > parameter, and the results are below. In that SPDK test with four
> > separate
> > > FIO job statements, each with queue depth 32, I still only achieved
> > > 12.5K IOPS.
> > > >>
> > > >>I then re-ran the kernel NVMe driver test case with only a single
> > > >>job
> > for
> > > comparison, and even with only 1 job and queue depth 32 it still
> > > achieves 254K IOPS.
> > > >>
> > > >>We might be on the right track with a problem around queue depth
> > > however. If I run the kernel NVMe driver case with just 1 job and
> > > queue depth of only 1, I then get 11.2K IOPS, which is around the
> > > performance
> > I get
> > > in the SPDK test case. Perhaps the SPDK case is not actually
> > > queuing
> > IOs on
> > > the device for some reason?
> > > >>
> > > >>
> > > >>don(a)donst201:~/fio/single/rr$ cat nvme2n1.ini |grep -v '#'
> > > >>[global]
> > > >>rw=randrw
> > > >>rwmixread=100
> > > >>numjobs=1
> > > >>iodepth=32
> > > >>bs=4k
> > > >>direct=1
> > > >>thread=1
> > > >>time_based=1
> > > >>ramp_time=0
> > > >>runtime=10
> > > >>ioengine=libaio
> > > >>group_reporting=1
> > > >>unified_rw_reporting=1
> > > >>exitall=1
> > > >>randrepeat=0
> > > >>norandommap=1
> > > >>[nvme2n1]
> > > >>filename=/dev/nvme2n1
> > > >>
> > > >>
> > > >>don(a)donst201:~/fio/single/rr$ sudo fio nvme2n1.ini
> > > >>nvme2n1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B,
> > > >>(T) 4096B-4096B, ioengine=libaio, iodepth=32
> > > >>fio-3.11
> > > >>Starting 1 thread
> > > >>Jobs: 1 (f=1): [r(1)][100.0%][r=1025MiB/s][r=262k IOPS][eta
> > > >>00m:00s]
> > > >>nvme2n1: (groupid=0, jobs=1): err= 0: pid=2422: Mon Nov 5
> > > >>05:19:13
> > > >>2018
> > > >> mixed: IOPS=254k, BW=992MiB/s (1040MB/s)(9922MiB/10001msec)
> > > >> slat (nsec): min=1185, max=319052, avg=1611.64, stdev=808.40
> > > >> clat (usec): min=2, max=4743, avg=124.02, stdev=121.00
> > > >> lat (usec): min=6, max=4744, avg=125.68, stdev=121.02
> > > >> clat percentiles (usec):
> > > >> | 1.00th=[ 9], 5.00th=[ 56], 10.00th=[ 61], 20.00th=[
> > 73],
> > > >> | 30.00th=[ 80], 40.00th=[ 91], 50.00th=[ 110], 60.00th=[
> > 118],
> > > >> | 70.00th=[ 130], 80.00th=[ 157], 90.00th=[ 202],
> > > >>95.00th=[ 253],
> > > >> | 99.00th=[ 408], 99.50th=[ 498], 99.90th=[ 1795],
> > > >>99.95th=[ 3064],
> > > >> | 99.99th=[ 3785]
> > > >> bw ( KiB/s): min=612240, max=1078792, per=100.00%,
> > > >>avg=1015996.40,
> > > stdev=106032.96, samples=20
> > > >> iops : min=153060, max=269698, avg=253998.90,
> stdev=26508.40,
> > > samples=20
> > > >> lat (usec) : 4=0.01%, 10=1.47%, 20=1.19%, 50=0.17%, 100=42.03%
> > > >> lat (usec) : 250=49.95%, 500=4.70%, 750=0.28%, 1000=0.06%
> > > >> lat (msec) : 2=0.06%, 4=0.09%, 10=0.01%
> > > >> cpu : usr=21.45%, sys=51.11%, ctx=530348, majf=0, minf=139
> > > >> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%,
> > > >=64=0.0%
> > > >> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
> > > >=64=0.0%
> > > >> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%,
> > > >> 64=0.0%,
> > > >>>=64=0.0%
> > > >> issued rwts: total=2540023,0,0,0 short=0,0,0,0 dropped=0,0,0,0
> > > >> latency : target=0, window=0, percentile=100.00%, depth=32
> > > >>Run status group 0 (all jobs):
> > > >> MIXED: bw=992MiB/s (1040MB/s), 992MiB/s-992MiB/s (1040MB/s-
> > > 1040MB/s), io=9922MiB (10.4GB), run=10001-10001msec Disk stats
> > > (read/write):
> > > >> nvme2n1: ios=2514356/0, merge=0/0, ticks=400/0,
> in_queue=297612,
> > > >>util=99.06%
> > > >>
> > > >>
> > > >>don(a)donst201:~/fio/single/rr$ cat 1c.ini |grep -v '#'
> > > >>[global]
> > > >>rw=randrw
> > > >>rwmixread=100
> > > >>numjobs=1
> > > >>iodepth=32
> > > >>bs=4k
> > > >>direct=1
> > > >>thread=1
> > > >>time_based=1
> > > >>ramp_time=0
> > > >>runtime=10
> > >
> >>ioengine=/home/don/install/spdk/spdk/examples/nvme/fio_plugin/fio_
> > > >>pl
> > > ug
> > > >>in
> > > >>group_reporting=1
> > > >>unified_rw_reporting=1
> > > >>exitall=1
> > > >>randrepeat=0
> > > >>norandommap=1
> > > >>[0000.1c.00.0-1]
> > > >>filename=trtype=PCIe traddr=0000.1c.00.0 ns=1 [0000.1c.00.0-2]
> > > >>filename=trtype=PCIe traddr=0000.1c.00.0 ns=1 [0000.1c.00.0-3]
> > > >>filename=trtype=PCIe traddr=0000.1c.00.0 ns=1 [0000.1c.00.0-4]
> > > >>filename=trtype=PCIe traddr=0000.1c.00.0 ns=1
> > > >>
> > > >>
> > > >>don(a)donst201:~/fio/single/rr$ sudo fio 1c.ini
> > > >>0000.1c.00.0-1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W)
> > > >>4096B-4096B,
> > > >>(T) 4096B-4096B, ioengine=spdk, iodepth=32
> > > >>0000.1c.00.0-2: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W)
> > > >>4096B-4096B,
> > > >>(T) 4096B-4096B, ioengine=spdk, iodepth=32
> > > >>0000.1c.00.0-3: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W)
> > > >>4096B-4096B,
> > > >>(T) 4096B-4096B, ioengine=spdk, iodepth=32
> > > >>0000.1c.00.0-4: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W)
> > > >>4096B-4096B,
> > > >>(T) 4096B-4096B, ioengine=spdk, iodepth=32
> > > >>fio-3.11
> > > >>Starting 4 threads
> > > >>Starting SPDK v18.10 / DPDK 18.08.0 initialization...
> > > >>[ DPDK EAL parameters: fio --no-shconf -c 0x1 -m 512
> > > >>--file-prefix=spdk_pid4270 ]
> > > >>EAL: Detected 36 lcore(s)
> > > >>EAL: Detected 2 NUMA nodes
> > > >>EAL: No free hugepages reported in hugepages-1048576kB
> > > >>EAL: Probing VFIO support...
> > > >>EAL: PCI device 0000:1c:00.0 on NUMA socket 0
> > > >>EAL: probe driver: 8086:a54 spdk_nvme
> > > >>Jobs: 1 (f=0): [_(3),f(1)][100.0%][eta 00m:00s]
> > > >>0000.1c.00.0-1: (groupid=0, jobs=4): err= 0: pid=4310: Mon Nov 5
> > > >>05:16:03 2018
> > > >> mixed: IOPS=12.5k, BW=48.0MiB/s (51.4MB/s)(490MiB/10010msec)
> > > >> slat (nsec): min=98, max=15092, avg=120.02, stdev=90.07
> > > >> clat (usec): min=55, max=19757, avg=10207.22, stdev=884.06
> > > >> lat (usec): min=60, max=19757, avg=10207.34, stdev=884.06
> > > >> clat percentiles (usec):
> > > >> | 1.00th=[ 8979], 5.00th=[ 9110], 10.00th=[ 9372],
> > > >>20.00th=[ 9503],
> > > >> | 30.00th=[ 9765], 40.00th=[ 9896], 50.00th=[10159],
> > > >>60.00th=[10290],
> > > >> | 70.00th=[10552], 80.00th=[10814], 90.00th=[11207],
> > > >>95.00th=[11600],
> > > >> | 99.00th=[12911], 99.50th=[14353], 99.90th=[17695],
> > > >>99.95th=[18744],
> > > >> | 99.99th=[19006]
> > > >> bw ( KiB/s): min=12008, max=12976, per=25.00%, avg=12535.70,
> > > stdev=306.81, samples=80
> > > >> iops : min= 3002, max= 3244, avg=3133.85, stdev=76.77,
> > samples=80
> > > >> lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%,
> 1000=0.01%
> > > >> lat (msec) : 2=0.01%, 4=0.02%, 10=45.13%, 20=54.83%
> > > >> cpu : usr=100.01%, sys=0.00%, ctx=27, majf=0, minf=0
> > > >> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.9%,
> > > >=64=0.0%
> > > >> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
> > > >=64=0.0%
> > > >> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%,
> > > >> 64=0.0%,
> > > >>>=64=0.0%
> > > >> issued rwts: total=125506,0,0,0 short=0,0,0,0 dropped=0,0,0,0
> > > >> latency : target=0, window=0, percentile=100.00%, depth=32
> > > >>Run status group 0 (all jobs):
> > > >> MIXED: bw=48.0MiB/s (51.4MB/s), 48.0MiB/s-48.0MiB/s
> > > >>(51.4MB/s-51.4MB/s), io=490MiB (514MB), run=10010-10010msec
> > > >>
> > > >>
> > > >>don(a)donst201:~/fio/single/rr$ cat nvme2n1.ini |grep -v '#'
> > > >>[global]
> > > >>rw=randrw
> > > >>rwmixread=100
> > > >>numjobs=1
> > > >>iodepth=1
> > > >>bs=4k
> > > >>direct=1
> > > >>thread=1
> > > >>time_based=1
> > > >>ramp_time=0
> > > >>runtime=10
> > > >>ioengine=libaio
> > > >>group_reporting=1
> > > >>unified_rw_reporting=1
> > > >>exitall=1
> > > >>randrepeat=0
> > > >>norandommap=1
> > > >>[nvme2n1]
> > > >>filename=/dev/nvme2n1
> > > >>
> > > >>
> > > >>don(a)donst201:~/fio/single/rr$ sudo fio nvme2n1.ini
> > > >>nvme2n1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B,
> > > >>(T) 4096B-4096B, ioengine=libaio, iodepth=1
> > > >>fio-3.11
> > > >>Starting 1 thread
> > > >>Jobs: 1 (f=1): [r(1)][100.0%][r=43.6MiB/s][r=11.2k IOPS][eta
> > > >>00m:00s]
> > > >>nvme2n1: (groupid=0, jobs=1): err= 0: pid=2525: Mon Nov 5
> > > >>05:22:57
> > > >>2018
> > > >> mixed: IOPS=11.2k, BW=43.6MiB/s (45.7MB/s)(436MiB/10001msec)
> > > >> slat (nsec): min=4848, max=208595, avg=5126.33, stdev=771.27
> > > >> clat (nsec): min=1845, max=3405.8k, avg=83004.99, stdev=33577.36
> > > >> lat (usec): min=15, max=3411, avg=88.26, stdev=33.59
> > > >> clat percentiles (usec):
> > > >> | 1.00th=[ 15], 5.00th=[ 58], 10.00th=[ 59], 20.00th=[
> > 59],
> > > >> | 30.00th=[ 60], 40.00th=[ 77], 50.00th=[ 77], 60.00th=[
> > 78],
> > > >> | 70.00th=[ 111], 80.00th=[ 113], 90.00th=[ 113],
> > > >>95.00th=[ 115],
> > > >> | 99.00th=[ 186], 99.50th=[ 186], 99.90th=[ 190],
> > > >>99.95th=[ 192],
> > > >> | 99.99th=[ 375]
> > > >> bw ( KiB/s): min=43816, max=45016, per=99.98%, avg=44639.58,
> > > stdev=312.88, samples=19
> > > >> iops : min=10954, max=11254, avg=11159.89, stdev=78.22,
> > > samples=19
> > > >> lat (usec) : 2=0.01%, 20=2.83%, 50=0.01%, 100=64.57%, 250=32.59%
> > > >> lat (usec) : 500=0.01%, 750=0.01%
> > > >> lat (msec) : 2=0.01%, 4=0.01%
> > > >> cpu : usr=4.02%, sys=10.33%, ctx=111630, majf=0, minf=12
> > > >> IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%,
> > > >=64=0.0%
> > > >> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
> > > >=64=0.0%
> > > >> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%,
> > > >> 64=0.0%,
> > > >>>=64=0.0%
> > > >> issued rwts: total=111631,0,0,0 short=0,0,0,0 dropped=0,0,0,0
> > > >> latency : target=0, window=0, percentile=100.00%, depth=1
> > > >>Run status group 0 (all jobs):
> > > >> MIXED: bw=43.6MiB/s (45.7MB/s), 43.6MiB/s-43.6MiB/s (45.7MB/s-
> > > 45.7MB/s), io=436MiB (457MB), run=10001-10001msec Disk stats
> > > (read/write):
> > > >> nvme2n1: ios=110275/0, merge=0/0, ticks=0/0, in_queue=8596,
> > > >>util=86.22%
> > > >>
> > > >>
> > > >>Thanks,
> > > >>Joe
> > > >>
> > > >>> -----Original Message-----
> > > >>> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Cao,
> > > >>> Gang
> > > >>> Sent: Sunday, November 4, 2018 8:40 PM
> > > >>> To: Storage Performance Development Kit < spdk(a)lists.01.org >
> > > >>> Subject: Re: [SPDK] FIO NVMe Performance Results
> > > >>>
> > > >>> For the SPDK plugin, from the below result, looks like only 1
> > > >>> job started with
> > > >>> 32 queue depth:
> > > >>>
> > > >>> Jobs: 1 (f=0): [f(1),_(3)][100.0%][eta 00m:00s] <<<<
> > > >>>
> > > >>> Try to update the FIO configuration file as
> > > >>>
> > > >>> [job1]
> > > >>> filename=trtype=PCIe traddr=0000.1c.00.0 ns=1 [job2]
> > > >>> filename=trtype=PCIe
> > > >>> traddr=0000.1c.00.0 ns=1 ....
> > > >>>
> > > >>> The kernel FIO is with 4 jobs and each job of 32 queue depth.
> > > >>> For SPDK, you can configure as the same, or just configure two
> > > >>> jobs and each job of 64 queue depth. The overall queue depth to
> > > >>> the device is
> > > same of 128.
> > > >>>
> > > >>> Thanks,
> > > >>> Gang
> > > >>>
> > > >>> -----Original Message-----
> > > >>> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of
> > > >>> Gruher, Joseph R
> > > >>> Sent: Monday, November 5, 2018 10:49 AM
> > > >>> To: Storage Performance Development Kit < spdk(a)lists.01.org >
> > > >>> Subject: [SPDK] FIO NVMe Performance Results
> > > >>>
> > > >>> Hi folks-
> > > >>>
> > > >>> I'm testing SPDK 18.10 on Ubuntu 18.04 with kernel 4.18.16. I
> > > >>> used FIO with the kernel NVMe driver to measure the performance
> > > >>> of a local (PCIe
> > > >>> attached) Intel P4500 NVMe device on a 4KB random read workload
> > > >>> and obtained 477K IOPS, roughly in line with the drive spec.
> > > >>> Then I tested the same drive with the SPDK FIO plugin and only
> > > >>> achieved 13K IOPS. The FIO test files and the results are
> > > >>> pasted below. Any ideas where I'm going wrong here?
> > > >>>
> > > >>> Thanks!
> > > >>>
> > > >>>
> > > >>> don(a)donst201:~/fio/single/rr$ ls
> > > >>> /sys/devices/pci0000:17/0000:17:02.0/0000:1c:00.0/nvme/nvme1/
> > > >>> address cntlid dev device firmware_rev model nvme2n1 power
> > > >>> rescan_controller reset_controller serial state subsysnqn
> > > >>> subsystem transport uevent
> > > >>>
> > > >>>
> > > >>> don(a)donst201:~/fio/single/rr$ cat nvme2n1.ini |grep -v '#'
> > > >>> [global]
> > > >>> rw=randrw
> > > >>> rwmixread=100
> > > >>> numjobs=4
> > > >>> iodepth=32
> > > >>> bs=4k
> > > >>> direct=1
> > > >>> thread=1
> > > >>> time_based=1
> > > >>> ramp_time=0
> > > >>> runtime=10
> > > >>> ioengine=libaio
> > > >>> group_reporting=1
> > > >>> unified_rw_reporting=1
> > > >>> exitall=1
> > > >>> randrepeat=0
> > > >>> norandommap=1
> > > >>> [nvme2n1]
> > > >>> filename=/dev/nvme2n1
> > > >>>
> > > >>>
> > > >>> don(a)donst201:~/fio/single/rr$ sudo fio nvme2n1.ini
> > > >>> nvme2n1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B,
> > > >>> (T) 4096B-4096B, ioengine=libaio, iodepth=32 ...
> > > >>> fio-3.11
> > > >>> Starting 4 threads
> > > >>> Jobs: 4 (f=4): [r(4)][100.0%][r=1874MiB/s][r=480k IOPS][eta
> > > >>> 00m:00s]
> > > >>> nvme2n1: (groupid=0, jobs=4): err= 0: pid=2575: Mon Nov 5
> > > >>> 02:26:25
> > > 2018
> > > >>> mixed: IOPS=477k, BW=1862MiB/s
> (1952MB/s)(18.2GiB/10001msec)
> > > >>> slat (nsec): min=1295, max=333088, avg=2216.35, stdev=927.89
> > > >>> clat (nsec): min=506, max=5818.8k, avg=265924.16,
> stdev=223761.59
> > > >>> lat (usec): min=6, max=5821, avg=268.19, stdev=223.76
> > > >>> clat percentiles (usec):
> > > >>> | 1.00th=[ 13], 5.00th=[ 74], 10.00th=[ 87], 20.00th=[
> > 115],
> > > >>> | 30.00th=[ 139], 40.00th=[ 167], 50.00th=[ 204],
> > > >>> 60.00th=[
> > 247],
> > > >>> | 70.00th=[ 306], 80.00th=[ 388], 90.00th=[ 523],
> > > >>> 95.00th=[
> > 660],
> > > >>> | 99.00th=[ 988], 99.50th=[ 1156], 99.90th=[ 2343],
> > > >>> 99.95th=[
> > 3195],
> > > >>> | 99.99th=[ 4424]
> > > >>> bw ( KiB/s): min=448192, max=483104, per=25.00%,
> > > >>> avg=476597.10, stdev=6748.78, samples=80
> > > >>> iops : min=112048, max=120776, avg=119149.27,
> > stdev=1687.20,
> > > >>> samples=80
> > > >>> lat (nsec) : 750=0.01%
> > > >>> lat (usec) : 10=0.52%, 20=1.25%, 50=0.25%, 100=13.38%,
> 250=45.13%
> > > >>> lat (usec) : 500=28.23%, 750=8.08%, 1000=2.22%
> > > >>> lat (msec) : 2=0.82%, 4=0.10%, 10=0.02%
> > > >>> cpu : usr=14.08%, sys=36.02%, ctx=1930207, majf=0,
> > minf=697
> > > >>> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%,
> 32=100.0%,
> > > >>> >=64=0.0%
> > > >>> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%,
> 64=0.0%,
> > > >>> >=64=0.0%
> > > >>> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%,
> > > >>> 64=0.0%,
> > > >>> >=64=0.0%
> > > >>> issued rwts: total=4766095,0,0,0 short=0,0,0,0 dropped=0,0,0,0
> > > >>> latency : target=0, window=0, percentile=100.00%, depth=32
> > > >>> Run status group 0 (all jobs):
> > > >>> MIXED: bw=1862MiB/s (1952MB/s), 1862MiB/s-1862MiB/s
> (1952MB/s-
> > > >>> 1952MB/s), io=18.2GiB (19.5GB), run=10001-10001msec Disk stats
> > > >>> (read/write):
> > > >>> nvme2n1: ios=4709190/0, merge=0/0, ticks=4096/0,
> > > in_queue=1388752,
> > > >>> util=100.00%
> > > >>>
> > > >>>
> > > >>> don(a)donst201:~/fio/single/rr$ sudo
> > > >>> /home/don/install/spdk/spdk/scripts/setup.sh
> > > >>> Active mountpoints on /dev/nvme0n1, so not binding PCI dev
> > > >>> 0000:03:00.0
> > > >>> 0000:1c:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > >>> 0000:1d:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > >>> 0000:5e:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > >>> 0000:5f:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > >>> 0000:62:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > >>> 0000:63:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > >>> 0000:64:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > >>> 0000:65:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > >>> 0000:da:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > >>> 0000:db:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > >>> 0000:dc:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > >>> 0000:dd:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > >>> 0000:e0:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > >>> 0000:e1:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > >>> 0000:e2:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > >>> 0000:e3:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > >>> 0000:00:04.0 (8086 2021): ioatdma -> uio_pci_generic
> > > >>> 0000:00:04.1 (8086 2021): ioatdma -> uio_pci_generic
> > > >>> 0000:00:04.2 (8086 2021): ioatdma -> uio_pci_generic
> > > >>> 0000:00:04.3 (8086 2021): ioatdma -> uio_pci_generic
> > > >>> 0000:00:04.4 (8086 2021): ioatdma -> uio_pci_generic
> > > >>> 0000:00:04.5 (8086 2021): ioatdma -> uio_pci_generic
> > > >>> 0000:00:04.6 (8086 2021): ioatdma -> uio_pci_generic
> > > >>> 0000:00:04.7 (8086 2021): ioatdma -> uio_pci_generic
> > > >>> 0000:80:04.0 (8086 2021): ioatdma -> uio_pci_generic
> > > >>> 0000:80:04.1 (8086 2021): ioatdma -> uio_pci_generic
> > > >>> 0000:80:04.2 (8086 2021): ioatdma -> uio_pci_generic
> > > >>> 0000:80:04.3 (8086 2021): ioatdma -> uio_pci_generic
> > > >>> 0000:80:04.4 (8086 2021): ioatdma -> uio_pci_generic
> > > >>> 0000:80:04.5 (8086 2021): ioatdma -> uio_pci_generic
> > > >>> 0000:80:04.6 (8086 2021): ioatdma -> uio_pci_generic
> > > >>> 0000:80:04.7 (8086 2021): ioatdma -> uio_pci_generic
> > > >>>
> > > >>>
> > > >>> don(a)donst201:~/fio/single/rr$ cat 1c.ini |grep -v '#'
> > > >>> [global]
> > > >>> rw=randrw
> > > >>> rwmixread=100
> > > >>> numjobs=4
> > > >>> iodepth=32
> > > >>> bs=4k
> > > >>> direct=1
> > > >>> thread=1
> > > >>> time_based=1
> > > >>> ramp_time=0
> > > >>> runtime=10
> > > >>>
> > >
> ioengine=/home/don/install/spdk/spdk/examples/nvme/fio_plugin/fio_pl
> > > >>> ug
> > > >>> in
> > > >>> group_reporting=1
> > > >>> unified_rw_reporting=1
> > > >>> exitall=1
> > > >>> randrepeat=0
> > > >>> norandommap=1
> > > >>> [0000.1c.00.0]
> > > >>> filename=trtype=PCIe traddr=0000.1c.00.0 ns=1
> > > >>>
> > > >>>
> > > >>> don(a)donst201:~/fio/single/rr$ sudo fio 1c.ini
> > > >>> 0000.1c.00.0: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W)
> > > >>> 4096B-4096B,
> > > >>> (T) 4096B-4096B, ioengine=spdk, iodepth=32 ...
> > > >>> fio-3.11
> > > >>> Starting 4 threads
> > > >>> Starting SPDK v18.10 / DPDK 18.08.0 initialization...
> > > >>> [ DPDK EAL parameters: fio --no-shconf -c 0x1 -m 512 --file-
> > > >>> prefix=spdk_pid3668 ]
> > > >>> EAL: Detected 36 lcore(s)
> > > >>> EAL: Detected 2 NUMA nodes
> > > >>> EAL: No free hugepages reported in hugepages-1048576kB
> > > >>> EAL: Probing VFIO support...
> > > >>> EAL: PCI device 0000:1c:00.0 on NUMA socket 0
> > > >>> EAL: probe driver: 8086:a54 spdk_nvme
> > > >>> Jobs: 1 (f=0): [f(1),_(3)][100.0%][eta 00m:00s]
> > > >>> 0000.1c.00.0: (groupid=0, jobs=4): err= 0: pid=3709: Mon Nov 5
> > > >>> 02:28:29
> > > >>> 2018
> > > >>> mixed: IOPS=13.2k, BW=51.6MiB/s (54.1MB/s)(517MiB/10011msec)
> > > >>> slat (nsec): min=109, max=15344, avg=127.91, stdev=73.44
> > > >>> clat (usec): min=187, max=18715, avg=9683.80, stdev=650.54
> > > >>> lat (usec): min=193, max=18715, avg=9683.93, stdev=650.53
> > > >>> clat percentiles (usec):
> > > >>> | 1.00th=[ 8717], 5.00th=[ 8848], 10.00th=[ 8979],
> > > >>> 20.00th=[
> > 9110],
> > > >>> | 30.00th=[ 9241], 40.00th=[ 9372], 50.00th=[ 9503],
> > > >>> 60.00th=[
> > 9765],
> > > >>> | 70.00th=[10028], 80.00th=[10159], 90.00th=[10552],
> > > 95.00th=[10945],
> > > >>> | 99.00th=[11338], 99.50th=[11338], 99.90th=[11731],
> > > 99.95th=[13960],
> > > >>> | 99.99th=[17695]
> > > >>> bw ( KiB/s): min=13080, max=13304, per=25.00%, avg=13214.70,
> > > >>> stdev=55.30, samples=80
> > > >>> iops : min= 3270, max= 3326, avg=3303.65, stdev=13.84,
> > > samples=80
> > > >>> lat (usec) : 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
> > > >>> lat (msec) : 2=0.01%, 4=0.02%, 10=70.85%, 20=29.11%
> > > >>> cpu : usr=100.01%, sys=0.00%, ctx=23, majf=0, minf=0
> > > >>> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.9%,
> > > >>> >=64=0.0%
> > > >>> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%,
> 64=0.0%,
> > > >>> >=64=0.0%
> > > >>> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%,
> > > >>> 64=0.0%,
> > > >>> >=64=0.0%
> > > >>> issued rwts: total=132300,0,0,0 short=0,0,0,0 dropped=0,0,0,0
> > > >>> latency : target=0, window=0, percentile=100.00%, depth=32
> > > >>> Run status group 0 (all jobs):
> > > >>> MIXED: bw=51.6MiB/s (54.1MB/s), 51.6MiB/s-51.6MiB/s
> (54.1MB/s-
> > > >>> 54.1MB/s), io=517MiB (542MB), run=10011-10011msec
> > > >>>
> > > >>> _______________________________________________
> > > >>> SPDK mailing list
> > > >>> SPDK(a)lists.01.org
> > > >>> https://lists.01.org/mailman/listinfo/spdk
> > > >>> _______________________________________________
> > > >>> SPDK mailing list
> > > >>> SPDK(a)lists.01.org
> > > >>> https://lists.01.org/mailman/listinfo/spdk
> > > >>_______________________________________________
> > > >>SPDK mailing list
> > > >> SPDK(a)lists.01.org
> > > >> https://lists.01.org/mailman/listinfo/spdk
> > > >>_______________________________________________
> > > >>SPDK mailing list
> > > >> SPDK(a)lists.01.org
> > > >> https://lists.01.org/mailman/listinfo/spdk
> > > >
> > > >
> > > >--
> > > >Маров Алексей
> > > >_______________________________________________
> > > >SPDK mailing list
> > > >SPDK(a)lists.01.org
> > > >https://lists.01.org/mailman/listinfo/spdk
> > > >_______________________________________________
> > > >SPDK mailing list
> > > >SPDK(a)lists.01.org
> > > >https://lists.01.org/mailman/listinfo/spdk
> > >
> > >
> > > --
> > > Маров Алексей
> > > _______________________________________________
> > > SPDK mailing list
> > > SPDK(a)lists.01.org
> > > https://lists.01.org/mailman/listinfo/spdk
> > _______________________________________________
> > SPDK mailing list
> > SPDK(a)lists.01.org
> > https://lists.01.org/mailman/listinfo/spdk
> >
> --
>
> Regards,
> Andrey
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [SPDK] FIO NVMe Performance Results
@ 2018-11-05 18:40 Andrey Kuzmin
0 siblings, 0 replies; 25+ messages in thread
From: Andrey Kuzmin @ 2018-11-05 18:40 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 2815 bytes --]
The below, of course, assumes you're running a nand-based (highly
parallelized) ssd, not a piece of nvram behind the nvme interface. In that
latter case you'd be right.as memory controller will be runnig all your
queue depth of io's one by one.
Regards,
Andrey
On Mon, Nov 5, 2018, 21:33 Andrey Kuzmin <andrey.v.kuzmin(a)gmail.com> wrote:
>
>
> On Mon, Nov 5, 2018, 21:27 Gruher, Joseph R <joseph.r.gruher(a)intel.com>
> wrote:
>
>>
>>
>> > -----Original Message-----
>> > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Andrey
>> Kuzmin
>> > Sent: Monday, November 5, 2018 8:48 AM
>> > To: Storage Performance Development Kit <spdk(a)lists.01.org>
>> > Subject: Re: [SPDK] FIO NVMe Performance Results
>> >
>> > On Mon, Nov 5, 2018, 18:29 Gruher, Joseph R <joseph.r.gruher(a)intel.com>
>> > wrote:
>> >
>> > > My FIO version is v3.11.
>> > >
>> > > I tried the bdev FIO plugin as suggested, but performance is about the
>> > > same as the other SPDK FIO plugin, I observed only 10.4K IOPS in a
>> > > quick 4KB random read test. See configuration and results below.
>> > >
>> >
>> > As 12 ms median read latency for an nvme ssd is far beyond anything
>> > reasonable, why don't you just perf this run, to see where the time is
>> being
>> > spent?
>> >
>> > Regards,
>> > Andrey
>>
>> When running FIO with queue depth the latency number FIO reports is not
>> just the latency of the device but also includes time spent in the FIO
>> queue. We saw 10400 IOPS meaning we're processing an IO on average every
>> 96usec. For a queue depth of 128 that means each IO will spend an average
>> of 96 * 128 = 12308usec in the FIO queue before being submitted to the
>> device.
>
>
> No. You've got four threads with qd=32 each, so your multiplicator above
> is 32, not 128. Furthermore, typically in calcs like this it's half the
> queue depth, not the full depth.
>
> So the FIO average clat number of 12330usec looks reasonable to me.
>>
>
> No, it's way off. No modern ssd will run your 128 io's sequentially, it
> will all be run (almost) concurrently, especially under random read. So the
> expected end-to-end latency here should be close to the physical one
> (around 100 usec per 4K), with IOPS in hundred thousands range.
>
> Regards,
> Andrey
>
>>
>> We can get a better feel for the latency of the device by running FIO
>> with 1 job and queue depth = 1. If I try that with the bdev plugin here I
>> get an average latency of 88usec which is well aligned with the drive spec
>> of 85usec.
>>
>> Thanks,
>> Joe
>> _______________________________________________
>> SPDK mailing list
>> SPDK(a)lists.01.org
>> https://lists.01.org/mailman/listinfo/spdk
>>
> --
>
> Regards,
> Andrey
>
--
Regards,
Andrey
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [SPDK] FIO NVMe Performance Results
@ 2018-11-05 18:33 Andrey Kuzmin
0 siblings, 0 replies; 25+ messages in thread
From: Andrey Kuzmin @ 2018-11-05 18:33 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 2342 bytes --]
On Mon, Nov 5, 2018, 21:27 Gruher, Joseph R <joseph.r.gruher(a)intel.com>
wrote:
>
>
> > -----Original Message-----
> > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Andrey Kuzmin
> > Sent: Monday, November 5, 2018 8:48 AM
> > To: Storage Performance Development Kit <spdk(a)lists.01.org>
> > Subject: Re: [SPDK] FIO NVMe Performance Results
> >
> > On Mon, Nov 5, 2018, 18:29 Gruher, Joseph R <joseph.r.gruher(a)intel.com>
> > wrote:
> >
> > > My FIO version is v3.11.
> > >
> > > I tried the bdev FIO plugin as suggested, but performance is about the
> > > same as the other SPDK FIO plugin, I observed only 10.4K IOPS in a
> > > quick 4KB random read test. See configuration and results below.
> > >
> >
> > As 12 ms median read latency for an nvme ssd is far beyond anything
> > reasonable, why don't you just perf this run, to see where the time is
> being
> > spent?
> >
> > Regards,
> > Andrey
>
> When running FIO with queue depth the latency number FIO reports is not
> just the latency of the device but also includes time spent in the FIO
> queue. We saw 10400 IOPS meaning we're processing an IO on average every
> 96usec. For a queue depth of 128 that means each IO will spend an average
> of 96 * 128 = 12308usec in the FIO queue before being submitted to the
> device.
No. You've got four threads with qd=32 each, so your multiplicator above is
32, not 128. Furthermore, typically in calcs like this it's half the queue
depth, not the full depth.
So the FIO average clat number of 12330usec looks reasonable to me.
>
No, it's way off. No modern ssd will run your 128 io's sequentially, it
will all be run (almost) concurrently, especially under random read. So the
expected end-to-end latency here should be close to the physical one
(around 100 usec per 4K), with IOPS in hundred thousands range.
Regards,
Andrey
>
> We can get a better feel for the latency of the device by running FIO with
> 1 job and queue depth = 1. If I try that with the bdev plugin here I get
> an average latency of 88usec which is well aligned with the drive spec of
> 85usec.
>
> Thanks,
> Joe
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
>
--
Regards,
Andrey
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [SPDK] FIO NVMe Performance Results
@ 2018-11-05 18:32 Howell, Seth
0 siblings, 0 replies; 25+ messages in thread
From: Howell, Seth @ 2018-11-05 18:32 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 34737 bytes --]
Hi Joseph,
I attempted to verify your results on my system. I am using fedora 27 with kernel version 4.17.3 with a p3700 drive. I cloned fio from https://github.com/axboe/fio and checked out the appropriate tag. I got appx 408k iops.
Typically we run spdk with the iommu enabled. I noticed that your devices are binding to uio_pci_generic instead of vfio. This shouldn't make any difference at all, but I just want to make sure you aren't running in a VM or something like that and passing through your NVMe drives and that's why you aren't using the IOMMU. That could explain a deeper configuration difference that could help us know where to look for the disparity.
Please see the contents of my configuration files and output below:
Test.ini:
[global]
rw=randrw
rwmixread=100
numjobs=4
iodepth=32
bs=4k
direct=1
thread=1
time_based=1
ramp_time=0
runtime=10
ioengine=spdk_bdev
spdk_conf=./nvme.conf
group_reporting=1
unified_rw_reporting=1
exitall=1
randrepeat=0
norandommap=1
[0000.81.00.0]
filename=Nvme0n1
nvme.conf:
[Nvme]
TransportId "trtype:PCIe traddr:0000.81.00.0" Nvme0
Sudo LD_PRELOAD=/home/sethhowe/Desktop/Development_Folders/spdk/examples/bdev/fio_plugin/fio_plugin ./fio ./test.ini >./out,txt 2>&1
Out.txt:
0000.81.00.0: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=32
...
fio-3.11
Starting 4 threads
Starting SPDK v18.10 / DPDK 18.08.0 initialization...
[ DPDK EAL parameters: fio --no-shconf -c 0x1 --file-prefix=spdk_pid782339 ]
EAL: Detected 16 lcore(s)
EAL: Detected 2 NUMA nodes
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: PCI device 0000:81:00.0 on NUMA socket 1
EAL: probe driver: 8086:953 spdk_nvme
0000.81.00.0: (groupid=0, jobs=4): err= 0: pid=782359: Mon Nov 5 10:58:59 2018
mixed: IOPS=410k, BW=1600MiB/s (1678MB/s)(15.6GiB/10001msec)
slat (nsec): min=179, max=19985, avg=245.97, stdev=122.96
clat (usec): min=8, max=1821, avg=311.82, stdev=167.14
lat (usec): min=8, max=1822, avg=312.06, stdev=167.14
clat percentiles (usec):
| 1.00th=[ 24], 5.00th=[ 48], 10.00th=[ 78], 20.00th=[ 141],
| 30.00th=[ 200], 40.00th=[ 258], 50.00th=[ 314], 60.00th=[ 371],
| 70.00th=[ 429], 80.00th=[ 486], 90.00th=[ 545], 95.00th=[ 570],
| 99.00th=[ 594], 99.50th=[ 594], 99.90th=[ 627], 99.95th=[ 652],
| 99.99th=[ 676]
bw ( KiB/s): min=407264, max=416040, per=25.01%, avg=409742.55, stdev=1337.38, samples=76
iops : min=101816, max=104010, avg=102435.61, stdev=334.41, samples=76
lat (usec) : 10=0.01%, 20=0.46%, 50=4.88%, 100=8.41%, 250=25.29%
lat (usec) : 500=43.15%, 750=17.81%, 1000=0.01%
lat (msec) : 2=0.01%
cpu : usr=99.78%, sys=0.00%, ctx=26, majf=0, minf=0
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.9%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.1%, 32=0.1%, 64=0.0%, >=64=0.0%
issued rwts: total=4096867,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
Run status group 0 (all jobs):
MIXED: bw=1600MiB/s (1678MB/s), 1600MiB/s-1600MiB/s (1678MB/s-1678MB/s), io=15.6GiB (16.8GB), run=10001-10001msec
Thanks,
Seth
-----Original Message-----
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Andrey Kuzmin
Sent: Monday, November 5, 2018 9:48 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] FIO NVMe Performance Results
On Mon, Nov 5, 2018, 18:29 Gruher, Joseph R <joseph.r.gruher(a)intel.com>
wrote:
> My FIO version is v3.11.
>
> I tried the bdev FIO plugin as suggested, but performance is about the
> same as the other SPDK FIO plugin, I observed only 10.4K IOPS in a
> quick 4KB random read test. See configuration and results below.
>
As 12 ms median read latency for an nvme ssd is far beyond anything reasonable, why don't you just perf this run, to see where the time is being spent?
Regards,
Andrey
>
>
> don(a)donst201:~/fio/single/rr$ cat nvme.conf [Nvme] TransportId
> "trtype:PCIe traddr:0000.1c.00.0" Nvme0
>
>
> don(a)donst201:~/fio/single/rr$ cat 1c-bdev.ini |grep -v '#'
> [global]
> rw=randrw
> rwmixread=100
> numjobs=4
> iodepth=32
> bs=4k
> direct=1
> thread=1
> time_based=1
> ramp_time=0
> runtime=10
> ioengine=spdk_bdev
> spdk_conf=./nvme.conf
> group_reporting=1
> unified_rw_reporting=1
> exitall=1
> randrepeat=0
> norandommap=1
> [0000.1c.00.0]
> filename=Nvme0n1
>
>
> don(a)donst201:~/fio/single/rr$ sudo
> LD_PRELOAD=/home/don/install/spdk/spdk/examples/bdev/fio_plugin/fio_pl
> ugin
> fio 1c-bdev.ini
> 0000.1c.00.0: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B,
> (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=32 ...
> fio-3.11
> Starting 4 threads
> Starting SPDK v18.10 / DPDK 18.08.0 initialization...
> [ DPDK EAL parameters: fio --no-shconf -c 0x1
> --file-prefix=spdk_pid4865 ]
> EAL: Detected 36 lcore(s)
> EAL: Detected 2 NUMA nodes
> EAL: No free hugepages reported in hugepages-1048576kB
> EAL: Probing VFIO support...
> EAL: PCI device 0000:03:00.0 on NUMA socket 0
> EAL: probe driver: 8086:f1a5 spdk_nvme
> EAL: PCI device 0000:1c:00.0 on NUMA socket 0
> EAL: probe driver: 8086:a54 spdk_nvme
> EAL: PCI device 0000:1d:00.0 on NUMA socket 0
> EAL: probe driver: 8086:a54 spdk_nvme
> EAL: PCI device 0000:5e:00.0 on NUMA socket 0
> EAL: probe driver: 8086:a54 spdk_nvme
> EAL: PCI device 0000:5f:00.0 on NUMA socket 0
> EAL: probe driver: 8086:a54 spdk_nvme
> EAL: PCI device 0000:62:00.0 on NUMA socket 0
> EAL: probe driver: 8086:a54 spdk_nvme
> EAL: PCI device 0000:63:00.0 on NUMA socket 0
> EAL: probe driver: 8086:a54 spdk_nvme
> EAL: PCI device 0000:64:00.0 on NUMA socket 0
> EAL: probe driver: 8086:a54 spdk_nvme
> EAL: PCI device 0000:65:00.0 on NUMA socket 0
> EAL: probe driver: 8086:a54 spdk_nvme
> EAL: PCI device 0000:da:00.0 on NUMA socket 1
> EAL: probe driver: 8086:a54 spdk_nvme
> EAL: PCI device 0000:db:00.0 on NUMA socket 1
> EAL: probe driver: 8086:a54 spdk_nvme
> EAL: PCI device 0000:dc:00.0 on NUMA socket 1
> EAL: probe driver: 8086:a54 spdk_nvme
> EAL: PCI device 0000:dd:00.0 on NUMA socket 1
> EAL: probe driver: 8086:a54 spdk_nvme
> EAL: PCI device 0000:e0:00.0 on NUMA socket 1
> EAL: probe driver: 8086:a54 spdk_nvme
> EAL: PCI device 0000:e1:00.0 on NUMA socket 1
> EAL: probe driver: 8086:a54 spdk_nvme
> EAL: PCI device 0000:e2:00.0 on NUMA socket 1
> EAL: probe driver: 8086:a54 spdk_nvme
> EAL: PCI device 0000:e3:00.0 on NUMA socket 1
> EAL: probe driver: 8086:a54 spdk_nvme
> Jobs: 4 (f=4): [r(4)][100.0%][r=45.7MiB/s][r=11.7k IOPS][eta 00m:00s]
> 0000.1c.00.0: (groupid=0, jobs=4): err= 0: pid=4906: Mon Nov 5
> 15:23:47
> 2018
> mixed: IOPS=10.4k, BW=40.5MiB/s (42.5MB/s)(406MiB/10012msec)
> slat (nsec): min=201, max=15831, avg=238.11, stdev=148.78
> clat (usec): min=124, max=31244, avg=12330.69, stdev=1697.98
> lat (usec): min=130, max=31244, avg=12330.93, stdev=1697.98
> clat percentiles (usec):
> | 1.00th=[ 9634], 5.00th=[10159], 10.00th=[10552], 20.00th=[10945],
> | 30.00th=[11338], 40.00th=[11731], 50.00th=[12125], 60.00th=[12518],
> | 70.00th=[13042], 80.00th=[13566], 90.00th=[14353], 95.00th=[15008],
> | 99.00th=[16712], 99.50th=[18744], 99.90th=[27395], 99.95th=[29754],
> | 99.99th=[30278]
> bw ( KiB/s): min= 8784, max=11952, per=24.99%, avg=10375.53,
> stdev=929.68, samples=80
> iops : min= 2196, max= 2988, avg=2593.85, stdev=232.41,
> samples=80
> lat (usec) : 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
> lat (msec) : 2=0.01%, 4=0.02%, 10=2.71%, 20=96.91%, 50=0.33%
> cpu : usr=99.99%, sys=0.01%, ctx=28, majf=0, minf=0
> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.9%,
> >=64=0.0%
> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
> >=64=0.0%
> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%,
> >=64=0.0%
> issued rwts: total=103907,0,0,0 short=0,0,0,0 dropped=0,0,0,0
> latency : target=0, window=0, percentile=100.00%, depth=32
> Run status group 0 (all jobs):
> MIXED: bw=40.5MiB/s (42.5MB/s), 40.5MiB/s-40.5MiB/s
> (42.5MB/s-42.5MB/s), io=406MiB (426MB), run=10012-10012msec
>
>
> Thanks,
> Joe
>
> > -----Original Message-----
> > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of ????? ???????
> > Sent: Monday, November 5, 2018 12:45 AM
> > To: Storage Performance Development Kit <spdk(a)lists.01.org>
> > Subject: Re: [SPDK] FIO NVMe Performance Results
> >
> > Ok, Do you see some performance issues using general bdev fio
> > plugin
> with
> > you configuration ?
> >
> > Best Regards
> > Alex
> >
> >
> > >Понедельник, 5 ноября 2018, 11:39 +03:00 от "Cao, Gang"
> > <gang.cao(a)intel.com>:
> > >
> > >Yes. This is for the bdev version of SPDK FIO plugin and you can
> > >assign
> the
> > conf file as the parameter in the FIO configuration.
> > >
> > >Previously, it's another SPDK FIO plugin for the user space NVMe
> > >driver
> > only.
> > >
> > >These are two different FIO plugins.
> > >
> > >Thanks,
> > >Gang
> > >
> > >-----Original Message-----
> > >From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Маров
> > >Алексей
> > >Sent: Monday, November 5, 2018 4:32 PM
> > >To: Storage Performance Development Kit < spdk(a)lists.01.org >
> > >Subject: Re: [SPDK] FIO NVMe Performance Results
> > >
> > >Sorry for incomplete message I have sent before
> > >
> > >Hello Gang.
> > >
> > >I just curious, why would you run fio with SPDK using this conf
> > >file,
> with is a
> > bit different from what is suggested in
> > here https://github.com/spdk/spdk/tree/master/examples/bdev/fio_plugin
> > .
> > >
> > >I mean may be you should create conf file for spdk like this one :
> > >
> > >nvme.conf :
> > >[Nvme]
> > > TransportId "trtype:PCIe traddr:0000.1c.00.0" Nvme0
> > >
> > >and provide it in fio config.
> > >
> > >fio.ini :
> > >[global]
> > >ioengine=spdk_bdev
> > >spdk_conf=....../nvme.conf
> > >rw=randrw
> > >rwmixread=100
> > >numjobs=4
> > >iodepth=32
> > >bs=4k
> > >direct=1
> > >thread=1
> > >time_based=1
> > >ramp_time=0
> > >runtime=10
> > >group_reporting=1
> > >unified_rw_reporting=1
> > >exitall=1
> > >randrepeat=0
> > >norandommap=1
> > >
> > >[0000:06:00.0]
> > >filename=Nvme0n1
> > >
> > >and run like this
> > >
> > ># LD_PRELOAD=examples/bdev/fio_plugin/fio_plugin fio
> > >examples/bdev/fio_plugin/fio.ini
> > >
> > >Just to note. The recommended version of fio for SPDK is 3.3
> > >
> > >Best Regards
> > >Alex
> > >
> > >
> > >>Понедельник, 5 ноября 2018, 9:53 +03:00 от "Cao, Gang" <
> > gang.cao(a)intel.com >:
> > >>
> > >>Thanks. This is interesting especially for your finding on queue
> > >>depth
> 1
> > case. We are trying to reproduce this issue locally same as your
> > configuration.
> > >>
> > >>Thanks,
> > >>Gang
> > >>
> > >>-----Original Message-----
> > >>From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Gruher,
> > >>Joseph R
> > >>Sent: Monday, November 5, 2018 1:32 PM
> > >>To: Storage Performance Development Kit < spdk(a)lists.01.org >
> > >>Subject: Re: [SPDK] FIO NVMe Performance Results
> > >>
> > >>Interesting catch. It seems like FIO is actually not accurately
> reporting the
> > number of jobs in the report out for the SPDK case. If I disable
> > the group_reporting parameter I do get a printout for four separate
> > jobs in
> the
> > SPDK case, so I think it is really running four jobs there, and just
> summarizing
> > incorrectly. I also see four separate CPU cores go to 100% when I
> > run
> that
> > FIO test case, indicating four jobs are active.
> > >>
> > >>Also, I don't think would explain the difference in performance.
> Using an
> > aggregate queue depth of 32 versus 128 shouldn't lead to a drop in
> > performance from 377K to 13K.
> > >>
> > >>I retested SPDK with multiple separate job descriptions as you
> suggested
> > anyway, using four separate jobs instead of relying on the numjobs=4
> > parameter, and the results are below. In that SPDK test with four
> separate
> > FIO job statements, each with queue depth 32, I still only achieved
> > 12.5K IOPS.
> > >>
> > >>I then re-ran the kernel NVMe driver test case with only a single
> > >>job
> for
> > comparison, and even with only 1 job and queue depth 32 it still
> > achieves 254K IOPS.
> > >>
> > >>We might be on the right track with a problem around queue depth
> > however. If I run the kernel NVMe driver case with just 1 job and
> > queue depth of only 1, I then get 11.2K IOPS, which is around the
> > performance
> I get
> > in the SPDK test case. Perhaps the SPDK case is not actually
> > queuing
> IOs on
> > the device for some reason?
> > >>
> > >>
> > >>don(a)donst201:~/fio/single/rr$ cat nvme2n1.ini |grep -v '#'
> > >>[global]
> > >>rw=randrw
> > >>rwmixread=100
> > >>numjobs=1
> > >>iodepth=32
> > >>bs=4k
> > >>direct=1
> > >>thread=1
> > >>time_based=1
> > >>ramp_time=0
> > >>runtime=10
> > >>ioengine=libaio
> > >>group_reporting=1
> > >>unified_rw_reporting=1
> > >>exitall=1
> > >>randrepeat=0
> > >>norandommap=1
> > >>[nvme2n1]
> > >>filename=/dev/nvme2n1
> > >>
> > >>
> > >>don(a)donst201:~/fio/single/rr$ sudo fio nvme2n1.ini
> > >>nvme2n1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B,
> > >>(T) 4096B-4096B, ioengine=libaio, iodepth=32
> > >>fio-3.11
> > >>Starting 1 thread
> > >>Jobs: 1 (f=1): [r(1)][100.0%][r=1025MiB/s][r=262k IOPS][eta
> > >>00m:00s]
> > >>nvme2n1: (groupid=0, jobs=1): err= 0: pid=2422: Mon Nov 5
> > >>05:19:13
> > >>2018
> > >> mixed: IOPS=254k, BW=992MiB/s (1040MB/s)(9922MiB/10001msec)
> > >> slat (nsec): min=1185, max=319052, avg=1611.64, stdev=808.40
> > >> clat (usec): min=2, max=4743, avg=124.02, stdev=121.00
> > >> lat (usec): min=6, max=4744, avg=125.68, stdev=121.02
> > >> clat percentiles (usec):
> > >> | 1.00th=[ 9], 5.00th=[ 56], 10.00th=[ 61], 20.00th=[
> 73],
> > >> | 30.00th=[ 80], 40.00th=[ 91], 50.00th=[ 110], 60.00th=[
> 118],
> > >> | 70.00th=[ 130], 80.00th=[ 157], 90.00th=[ 202],
> > >>95.00th=[ 253],
> > >> | 99.00th=[ 408], 99.50th=[ 498], 99.90th=[ 1795],
> > >>99.95th=[ 3064],
> > >> | 99.99th=[ 3785]
> > >> bw ( KiB/s): min=612240, max=1078792, per=100.00%,
> > >>avg=1015996.40,
> > stdev=106032.96, samples=20
> > >> iops : min=153060, max=269698, avg=253998.90, stdev=26508.40,
> > samples=20
> > >> lat (usec) : 4=0.01%, 10=1.47%, 20=1.19%, 50=0.17%, 100=42.03%
> > >> lat (usec) : 250=49.95%, 500=4.70%, 750=0.28%, 1000=0.06%
> > >> lat (msec) : 2=0.06%, 4=0.09%, 10=0.01%
> > >> cpu : usr=21.45%, sys=51.11%, ctx=530348, majf=0, minf=139
> > >> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%,
> > >=64=0.0%
> > >> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
> > >=64=0.0%
> > >> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%,
> > >> 64=0.0%,
> > >>>=64=0.0%
> > >> issued rwts: total=2540023,0,0,0 short=0,0,0,0 dropped=0,0,0,0
> > >> latency : target=0, window=0, percentile=100.00%, depth=32
> > >>Run status group 0 (all jobs):
> > >> MIXED: bw=992MiB/s (1040MB/s), 992MiB/s-992MiB/s (1040MB/s-
> > 1040MB/s), io=9922MiB (10.4GB), run=10001-10001msec Disk stats
> > (read/write):
> > >> nvme2n1: ios=2514356/0, merge=0/0, ticks=400/0, in_queue=297612,
> > >>util=99.06%
> > >>
> > >>
> > >>don(a)donst201:~/fio/single/rr$ cat 1c.ini |grep -v '#'
> > >>[global]
> > >>rw=randrw
> > >>rwmixread=100
> > >>numjobs=1
> > >>iodepth=32
> > >>bs=4k
> > >>direct=1
> > >>thread=1
> > >>time_based=1
> > >>ramp_time=0
> > >>runtime=10
> > >>ioengine=/home/don/install/spdk/spdk/examples/nvme/fio_plugin/fio_
> > >>pl
> > ug
> > >>in
> > >>group_reporting=1
> > >>unified_rw_reporting=1
> > >>exitall=1
> > >>randrepeat=0
> > >>norandommap=1
> > >>[0000.1c.00.0-1]
> > >>filename=trtype=PCIe traddr=0000.1c.00.0 ns=1 [0000.1c.00.0-2]
> > >>filename=trtype=PCIe traddr=0000.1c.00.0 ns=1 [0000.1c.00.0-3]
> > >>filename=trtype=PCIe traddr=0000.1c.00.0 ns=1 [0000.1c.00.0-4]
> > >>filename=trtype=PCIe traddr=0000.1c.00.0 ns=1
> > >>
> > >>
> > >>don(a)donst201:~/fio/single/rr$ sudo fio 1c.ini
> > >>0000.1c.00.0-1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W)
> > >>4096B-4096B,
> > >>(T) 4096B-4096B, ioengine=spdk, iodepth=32
> > >>0000.1c.00.0-2: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W)
> > >>4096B-4096B,
> > >>(T) 4096B-4096B, ioengine=spdk, iodepth=32
> > >>0000.1c.00.0-3: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W)
> > >>4096B-4096B,
> > >>(T) 4096B-4096B, ioengine=spdk, iodepth=32
> > >>0000.1c.00.0-4: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W)
> > >>4096B-4096B,
> > >>(T) 4096B-4096B, ioengine=spdk, iodepth=32
> > >>fio-3.11
> > >>Starting 4 threads
> > >>Starting SPDK v18.10 / DPDK 18.08.0 initialization...
> > >>[ DPDK EAL parameters: fio --no-shconf -c 0x1 -m 512
> > >>--file-prefix=spdk_pid4270 ]
> > >>EAL: Detected 36 lcore(s)
> > >>EAL: Detected 2 NUMA nodes
> > >>EAL: No free hugepages reported in hugepages-1048576kB
> > >>EAL: Probing VFIO support...
> > >>EAL: PCI device 0000:1c:00.0 on NUMA socket 0
> > >>EAL: probe driver: 8086:a54 spdk_nvme
> > >>Jobs: 1 (f=0): [_(3),f(1)][100.0%][eta 00m:00s]
> > >>0000.1c.00.0-1: (groupid=0, jobs=4): err= 0: pid=4310: Mon Nov 5
> > >>05:16:03 2018
> > >> mixed: IOPS=12.5k, BW=48.0MiB/s (51.4MB/s)(490MiB/10010msec)
> > >> slat (nsec): min=98, max=15092, avg=120.02, stdev=90.07
> > >> clat (usec): min=55, max=19757, avg=10207.22, stdev=884.06
> > >> lat (usec): min=60, max=19757, avg=10207.34, stdev=884.06
> > >> clat percentiles (usec):
> > >> | 1.00th=[ 8979], 5.00th=[ 9110], 10.00th=[ 9372],
> > >>20.00th=[ 9503],
> > >> | 30.00th=[ 9765], 40.00th=[ 9896], 50.00th=[10159],
> > >>60.00th=[10290],
> > >> | 70.00th=[10552], 80.00th=[10814], 90.00th=[11207],
> > >>95.00th=[11600],
> > >> | 99.00th=[12911], 99.50th=[14353], 99.90th=[17695],
> > >>99.95th=[18744],
> > >> | 99.99th=[19006]
> > >> bw ( KiB/s): min=12008, max=12976, per=25.00%, avg=12535.70,
> > stdev=306.81, samples=80
> > >> iops : min= 3002, max= 3244, avg=3133.85, stdev=76.77,
> samples=80
> > >> lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
> > >> lat (msec) : 2=0.01%, 4=0.02%, 10=45.13%, 20=54.83%
> > >> cpu : usr=100.01%, sys=0.00%, ctx=27, majf=0, minf=0
> > >> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.9%,
> > >=64=0.0%
> > >> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
> > >=64=0.0%
> > >> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%,
> > >> 64=0.0%,
> > >>>=64=0.0%
> > >> issued rwts: total=125506,0,0,0 short=0,0,0,0 dropped=0,0,0,0
> > >> latency : target=0, window=0, percentile=100.00%, depth=32
> > >>Run status group 0 (all jobs):
> > >> MIXED: bw=48.0MiB/s (51.4MB/s), 48.0MiB/s-48.0MiB/s
> > >>(51.4MB/s-51.4MB/s), io=490MiB (514MB), run=10010-10010msec
> > >>
> > >>
> > >>don(a)donst201:~/fio/single/rr$ cat nvme2n1.ini |grep -v '#'
> > >>[global]
> > >>rw=randrw
> > >>rwmixread=100
> > >>numjobs=1
> > >>iodepth=1
> > >>bs=4k
> > >>direct=1
> > >>thread=1
> > >>time_based=1
> > >>ramp_time=0
> > >>runtime=10
> > >>ioengine=libaio
> > >>group_reporting=1
> > >>unified_rw_reporting=1
> > >>exitall=1
> > >>randrepeat=0
> > >>norandommap=1
> > >>[nvme2n1]
> > >>filename=/dev/nvme2n1
> > >>
> > >>
> > >>don(a)donst201:~/fio/single/rr$ sudo fio nvme2n1.ini
> > >>nvme2n1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B,
> > >>(T) 4096B-4096B, ioengine=libaio, iodepth=1
> > >>fio-3.11
> > >>Starting 1 thread
> > >>Jobs: 1 (f=1): [r(1)][100.0%][r=43.6MiB/s][r=11.2k IOPS][eta
> > >>00m:00s]
> > >>nvme2n1: (groupid=0, jobs=1): err= 0: pid=2525: Mon Nov 5
> > >>05:22:57
> > >>2018
> > >> mixed: IOPS=11.2k, BW=43.6MiB/s (45.7MB/s)(436MiB/10001msec)
> > >> slat (nsec): min=4848, max=208595, avg=5126.33, stdev=771.27
> > >> clat (nsec): min=1845, max=3405.8k, avg=83004.99, stdev=33577.36
> > >> lat (usec): min=15, max=3411, avg=88.26, stdev=33.59
> > >> clat percentiles (usec):
> > >> | 1.00th=[ 15], 5.00th=[ 58], 10.00th=[ 59], 20.00th=[
> 59],
> > >> | 30.00th=[ 60], 40.00th=[ 77], 50.00th=[ 77], 60.00th=[
> 78],
> > >> | 70.00th=[ 111], 80.00th=[ 113], 90.00th=[ 113],
> > >>95.00th=[ 115],
> > >> | 99.00th=[ 186], 99.50th=[ 186], 99.90th=[ 190],
> > >>99.95th=[ 192],
> > >> | 99.99th=[ 375]
> > >> bw ( KiB/s): min=43816, max=45016, per=99.98%, avg=44639.58,
> > stdev=312.88, samples=19
> > >> iops : min=10954, max=11254, avg=11159.89, stdev=78.22,
> > samples=19
> > >> lat (usec) : 2=0.01%, 20=2.83%, 50=0.01%, 100=64.57%, 250=32.59%
> > >> lat (usec) : 500=0.01%, 750=0.01%
> > >> lat (msec) : 2=0.01%, 4=0.01%
> > >> cpu : usr=4.02%, sys=10.33%, ctx=111630, majf=0, minf=12
> > >> IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%,
> > >=64=0.0%
> > >> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
> > >=64=0.0%
> > >> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%,
> > >> 64=0.0%,
> > >>>=64=0.0%
> > >> issued rwts: total=111631,0,0,0 short=0,0,0,0 dropped=0,0,0,0
> > >> latency : target=0, window=0, percentile=100.00%, depth=1
> > >>Run status group 0 (all jobs):
> > >> MIXED: bw=43.6MiB/s (45.7MB/s), 43.6MiB/s-43.6MiB/s (45.7MB/s-
> > 45.7MB/s), io=436MiB (457MB), run=10001-10001msec Disk stats
> > (read/write):
> > >> nvme2n1: ios=110275/0, merge=0/0, ticks=0/0, in_queue=8596,
> > >>util=86.22%
> > >>
> > >>
> > >>Thanks,
> > >>Joe
> > >>
> > >>> -----Original Message-----
> > >>> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Cao,
> > >>> Gang
> > >>> Sent: Sunday, November 4, 2018 8:40 PM
> > >>> To: Storage Performance Development Kit < spdk(a)lists.01.org >
> > >>> Subject: Re: [SPDK] FIO NVMe Performance Results
> > >>>
> > >>> For the SPDK plugin, from the below result, looks like only 1
> > >>> job started with
> > >>> 32 queue depth:
> > >>>
> > >>> Jobs: 1 (f=0): [f(1),_(3)][100.0%][eta 00m:00s] <<<<
> > >>>
> > >>> Try to update the FIO configuration file as
> > >>>
> > >>> [job1]
> > >>> filename=trtype=PCIe traddr=0000.1c.00.0 ns=1 [job2]
> > >>> filename=trtype=PCIe
> > >>> traddr=0000.1c.00.0 ns=1 ....
> > >>>
> > >>> The kernel FIO is with 4 jobs and each job of 32 queue depth.
> > >>> For SPDK, you can configure as the same, or just configure two
> > >>> jobs and each job of 64 queue depth. The overall queue depth to
> > >>> the device is
> > same of 128.
> > >>>
> > >>> Thanks,
> > >>> Gang
> > >>>
> > >>> -----Original Message-----
> > >>> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of
> > >>> Gruher, Joseph R
> > >>> Sent: Monday, November 5, 2018 10:49 AM
> > >>> To: Storage Performance Development Kit < spdk(a)lists.01.org >
> > >>> Subject: [SPDK] FIO NVMe Performance Results
> > >>>
> > >>> Hi folks-
> > >>>
> > >>> I'm testing SPDK 18.10 on Ubuntu 18.04 with kernel 4.18.16. I
> > >>> used FIO with the kernel NVMe driver to measure the performance
> > >>> of a local (PCIe
> > >>> attached) Intel P4500 NVMe device on a 4KB random read workload
> > >>> and obtained 477K IOPS, roughly in line with the drive spec.
> > >>> Then I tested the same drive with the SPDK FIO plugin and only
> > >>> achieved 13K IOPS. The FIO test files and the results are
> > >>> pasted below. Any ideas where I'm going wrong here?
> > >>>
> > >>> Thanks!
> > >>>
> > >>>
> > >>> don(a)donst201:~/fio/single/rr$ ls
> > >>> /sys/devices/pci0000:17/0000:17:02.0/0000:1c:00.0/nvme/nvme1/
> > >>> address cntlid dev device firmware_rev model nvme2n1
> > >>> power rescan_controller reset_controller serial state
> > >>> subsysnqn subsystem transport uevent
> > >>>
> > >>>
> > >>> don(a)donst201:~/fio/single/rr$ cat nvme2n1.ini |grep -v '#'
> > >>> [global]
> > >>> rw=randrw
> > >>> rwmixread=100
> > >>> numjobs=4
> > >>> iodepth=32
> > >>> bs=4k
> > >>> direct=1
> > >>> thread=1
> > >>> time_based=1
> > >>> ramp_time=0
> > >>> runtime=10
> > >>> ioengine=libaio
> > >>> group_reporting=1
> > >>> unified_rw_reporting=1
> > >>> exitall=1
> > >>> randrepeat=0
> > >>> norandommap=1
> > >>> [nvme2n1]
> > >>> filename=/dev/nvme2n1
> > >>>
> > >>>
> > >>> don(a)donst201:~/fio/single/rr$ sudo fio nvme2n1.ini
> > >>> nvme2n1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B,
> > >>> (T) 4096B-4096B, ioengine=libaio, iodepth=32 ...
> > >>> fio-3.11
> > >>> Starting 4 threads
> > >>> Jobs: 4 (f=4): [r(4)][100.0%][r=1874MiB/s][r=480k IOPS][eta
> > >>> 00m:00s]
> > >>> nvme2n1: (groupid=0, jobs=4): err= 0: pid=2575: Mon Nov 5
> > >>> 02:26:25
> > 2018
> > >>> mixed: IOPS=477k, BW=1862MiB/s (1952MB/s)(18.2GiB/10001msec)
> > >>> slat (nsec): min=1295, max=333088, avg=2216.35, stdev=927.89
> > >>> clat (nsec): min=506, max=5818.8k, avg=265924.16, stdev=223761.59
> > >>> lat (usec): min=6, max=5821, avg=268.19, stdev=223.76
> > >>> clat percentiles (usec):
> > >>> | 1.00th=[ 13], 5.00th=[ 74], 10.00th=[ 87], 20.00th=[
> 115],
> > >>> | 30.00th=[ 139], 40.00th=[ 167], 50.00th=[ 204],
> > >>> 60.00th=[
> 247],
> > >>> | 70.00th=[ 306], 80.00th=[ 388], 90.00th=[ 523],
> > >>> 95.00th=[
> 660],
> > >>> | 99.00th=[ 988], 99.50th=[ 1156], 99.90th=[ 2343],
> > >>> 99.95th=[
> 3195],
> > >>> | 99.99th=[ 4424]
> > >>> bw ( KiB/s): min=448192, max=483104, per=25.00%,
> > >>> avg=476597.10, stdev=6748.78, samples=80
> > >>> iops : min=112048, max=120776, avg=119149.27,
> stdev=1687.20,
> > >>> samples=80
> > >>> lat (nsec) : 750=0.01%
> > >>> lat (usec) : 10=0.52%, 20=1.25%, 50=0.25%, 100=13.38%, 250=45.13%
> > >>> lat (usec) : 500=28.23%, 750=8.08%, 1000=2.22%
> > >>> lat (msec) : 2=0.82%, 4=0.10%, 10=0.02%
> > >>> cpu : usr=14.08%, sys=36.02%, ctx=1930207, majf=0,
> minf=697
> > >>> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%,
> > >>> >=64=0.0%
> > >>> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
> > >>> >=64=0.0%
> > >>> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%,
> > >>> 64=0.0%,
> > >>> >=64=0.0%
> > >>> issued rwts: total=4766095,0,0,0 short=0,0,0,0 dropped=0,0,0,0
> > >>> latency : target=0, window=0, percentile=100.00%, depth=32
> > >>> Run status group 0 (all jobs):
> > >>> MIXED: bw=1862MiB/s (1952MB/s), 1862MiB/s-1862MiB/s (1952MB/s-
> > >>> 1952MB/s), io=18.2GiB (19.5GB), run=10001-10001msec Disk stats
> > >>> (read/write):
> > >>> nvme2n1: ios=4709190/0, merge=0/0, ticks=4096/0,
> > in_queue=1388752,
> > >>> util=100.00%
> > >>>
> > >>>
> > >>> don(a)donst201:~/fio/single/rr$ sudo
> > >>> /home/don/install/spdk/spdk/scripts/setup.sh
> > >>> Active mountpoints on /dev/nvme0n1, so not binding PCI dev
> > >>> 0000:03:00.0
> > >>> 0000:1c:00.0 (8086 0a54): nvme -> uio_pci_generic
> > >>> 0000:1d:00.0 (8086 0a54): nvme -> uio_pci_generic
> > >>> 0000:5e:00.0 (8086 0a54): nvme -> uio_pci_generic
> > >>> 0000:5f:00.0 (8086 0a54): nvme -> uio_pci_generic
> > >>> 0000:62:00.0 (8086 0a54): nvme -> uio_pci_generic
> > >>> 0000:63:00.0 (8086 0a54): nvme -> uio_pci_generic
> > >>> 0000:64:00.0 (8086 0a54): nvme -> uio_pci_generic
> > >>> 0000:65:00.0 (8086 0a54): nvme -> uio_pci_generic
> > >>> 0000:da:00.0 (8086 0a54): nvme -> uio_pci_generic
> > >>> 0000:db:00.0 (8086 0a54): nvme -> uio_pci_generic
> > >>> 0000:dc:00.0 (8086 0a54): nvme -> uio_pci_generic
> > >>> 0000:dd:00.0 (8086 0a54): nvme -> uio_pci_generic
> > >>> 0000:e0:00.0 (8086 0a54): nvme -> uio_pci_generic
> > >>> 0000:e1:00.0 (8086 0a54): nvme -> uio_pci_generic
> > >>> 0000:e2:00.0 (8086 0a54): nvme -> uio_pci_generic
> > >>> 0000:e3:00.0 (8086 0a54): nvme -> uio_pci_generic
> > >>> 0000:00:04.0 (8086 2021): ioatdma -> uio_pci_generic
> > >>> 0000:00:04.1 (8086 2021): ioatdma -> uio_pci_generic
> > >>> 0000:00:04.2 (8086 2021): ioatdma -> uio_pci_generic
> > >>> 0000:00:04.3 (8086 2021): ioatdma -> uio_pci_generic
> > >>> 0000:00:04.4 (8086 2021): ioatdma -> uio_pci_generic
> > >>> 0000:00:04.5 (8086 2021): ioatdma -> uio_pci_generic
> > >>> 0000:00:04.6 (8086 2021): ioatdma -> uio_pci_generic
> > >>> 0000:00:04.7 (8086 2021): ioatdma -> uio_pci_generic
> > >>> 0000:80:04.0 (8086 2021): ioatdma -> uio_pci_generic
> > >>> 0000:80:04.1 (8086 2021): ioatdma -> uio_pci_generic
> > >>> 0000:80:04.2 (8086 2021): ioatdma -> uio_pci_generic
> > >>> 0000:80:04.3 (8086 2021): ioatdma -> uio_pci_generic
> > >>> 0000:80:04.4 (8086 2021): ioatdma -> uio_pci_generic
> > >>> 0000:80:04.5 (8086 2021): ioatdma -> uio_pci_generic
> > >>> 0000:80:04.6 (8086 2021): ioatdma -> uio_pci_generic
> > >>> 0000:80:04.7 (8086 2021): ioatdma -> uio_pci_generic
> > >>>
> > >>>
> > >>> don(a)donst201:~/fio/single/rr$ cat 1c.ini |grep -v '#'
> > >>> [global]
> > >>> rw=randrw
> > >>> rwmixread=100
> > >>> numjobs=4
> > >>> iodepth=32
> > >>> bs=4k
> > >>> direct=1
> > >>> thread=1
> > >>> time_based=1
> > >>> ramp_time=0
> > >>> runtime=10
> > >>>
> > ioengine=/home/don/install/spdk/spdk/examples/nvme/fio_plugin/fio_pl
> > >>> ug
> > >>> in
> > >>> group_reporting=1
> > >>> unified_rw_reporting=1
> > >>> exitall=1
> > >>> randrepeat=0
> > >>> norandommap=1
> > >>> [0000.1c.00.0]
> > >>> filename=trtype=PCIe traddr=0000.1c.00.0 ns=1
> > >>>
> > >>>
> > >>> don(a)donst201:~/fio/single/rr$ sudo fio 1c.ini
> > >>> 0000.1c.00.0: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W)
> > >>> 4096B-4096B,
> > >>> (T) 4096B-4096B, ioengine=spdk, iodepth=32 ...
> > >>> fio-3.11
> > >>> Starting 4 threads
> > >>> Starting SPDK v18.10 / DPDK 18.08.0 initialization...
> > >>> [ DPDK EAL parameters: fio --no-shconf -c 0x1 -m 512 --file-
> > >>> prefix=spdk_pid3668 ]
> > >>> EAL: Detected 36 lcore(s)
> > >>> EAL: Detected 2 NUMA nodes
> > >>> EAL: No free hugepages reported in hugepages-1048576kB
> > >>> EAL: Probing VFIO support...
> > >>> EAL: PCI device 0000:1c:00.0 on NUMA socket 0
> > >>> EAL: probe driver: 8086:a54 spdk_nvme
> > >>> Jobs: 1 (f=0): [f(1),_(3)][100.0%][eta 00m:00s]
> > >>> 0000.1c.00.0: (groupid=0, jobs=4): err= 0: pid=3709: Mon Nov 5
> > >>> 02:28:29
> > >>> 2018
> > >>> mixed: IOPS=13.2k, BW=51.6MiB/s (54.1MB/s)(517MiB/10011msec)
> > >>> slat (nsec): min=109, max=15344, avg=127.91, stdev=73.44
> > >>> clat (usec): min=187, max=18715, avg=9683.80, stdev=650.54
> > >>> lat (usec): min=193, max=18715, avg=9683.93, stdev=650.53
> > >>> clat percentiles (usec):
> > >>> | 1.00th=[ 8717], 5.00th=[ 8848], 10.00th=[ 8979],
> > >>> 20.00th=[
> 9110],
> > >>> | 30.00th=[ 9241], 40.00th=[ 9372], 50.00th=[ 9503],
> > >>> 60.00th=[
> 9765],
> > >>> | 70.00th=[10028], 80.00th=[10159], 90.00th=[10552],
> > 95.00th=[10945],
> > >>> | 99.00th=[11338], 99.50th=[11338], 99.90th=[11731],
> > 99.95th=[13960],
> > >>> | 99.99th=[17695]
> > >>> bw ( KiB/s): min=13080, max=13304, per=25.00%, avg=13214.70,
> > >>> stdev=55.30, samples=80
> > >>> iops : min= 3270, max= 3326, avg=3303.65, stdev=13.84,
> > samples=80
> > >>> lat (usec) : 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
> > >>> lat (msec) : 2=0.01%, 4=0.02%, 10=70.85%, 20=29.11%
> > >>> cpu : usr=100.01%, sys=0.00%, ctx=23, majf=0, minf=0
> > >>> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.9%,
> > >>> >=64=0.0%
> > >>> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
> > >>> >=64=0.0%
> > >>> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%,
> > >>> 64=0.0%,
> > >>> >=64=0.0%
> > >>> issued rwts: total=132300,0,0,0 short=0,0,0,0 dropped=0,0,0,0
> > >>> latency : target=0, window=0, percentile=100.00%, depth=32
> > >>> Run status group 0 (all jobs):
> > >>> MIXED: bw=51.6MiB/s (54.1MB/s), 51.6MiB/s-51.6MiB/s (54.1MB/s-
> > >>> 54.1MB/s), io=517MiB (542MB), run=10011-10011msec
> > >>>
> > >>> _______________________________________________
> > >>> SPDK mailing list
> > >>> SPDK(a)lists.01.org
> > >>> https://lists.01.org/mailman/listinfo/spdk
> > >>> _______________________________________________
> > >>> SPDK mailing list
> > >>> SPDK(a)lists.01.org
> > >>> https://lists.01.org/mailman/listinfo/spdk
> > >>_______________________________________________
> > >>SPDK mailing list
> > >> SPDK(a)lists.01.org
> > >> https://lists.01.org/mailman/listinfo/spdk
> > >>_______________________________________________
> > >>SPDK mailing list
> > >> SPDK(a)lists.01.org
> > >> https://lists.01.org/mailman/listinfo/spdk
> > >
> > >
> > >--
> > >Маров Алексей
> > >_______________________________________________
> > >SPDK mailing list
> > >SPDK(a)lists.01.org
> > >https://lists.01.org/mailman/listinfo/spdk
> > >_______________________________________________
> > >SPDK mailing list
> > >SPDK(a)lists.01.org
> > >https://lists.01.org/mailman/listinfo/spdk
> >
> >
> > --
> > Маров Алексей
> > _______________________________________________
> > SPDK mailing list
> > SPDK(a)lists.01.org
> > https://lists.01.org/mailman/listinfo/spdk
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
>
--
Regards,
Andrey
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [SPDK] FIO NVMe Performance Results
@ 2018-11-05 18:27 Gruher, Joseph R
0 siblings, 0 replies; 25+ messages in thread
From: Gruher, Joseph R @ 2018-11-05 18:27 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 1509 bytes --]
> -----Original Message-----
> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Andrey Kuzmin
> Sent: Monday, November 5, 2018 8:48 AM
> To: Storage Performance Development Kit <spdk(a)lists.01.org>
> Subject: Re: [SPDK] FIO NVMe Performance Results
>
> On Mon, Nov 5, 2018, 18:29 Gruher, Joseph R <joseph.r.gruher(a)intel.com>
> wrote:
>
> > My FIO version is v3.11.
> >
> > I tried the bdev FIO plugin as suggested, but performance is about the
> > same as the other SPDK FIO plugin, I observed only 10.4K IOPS in a
> > quick 4KB random read test. See configuration and results below.
> >
>
> As 12 ms median read latency for an nvme ssd is far beyond anything
> reasonable, why don't you just perf this run, to see where the time is being
> spent?
>
> Regards,
> Andrey
When running FIO with queue depth the latency number FIO reports is not just the latency of the device but also includes time spent in the FIO queue. We saw 10400 IOPS meaning we're processing an IO on average every 96usec. For a queue depth of 128 that means each IO will spend an average of 96 * 128 = 12308usec in the FIO queue before being submitted to the device. So the FIO average clat number of 12330usec looks reasonable to me.
We can get a better feel for the latency of the device by running FIO with 1 job and queue depth = 1. If I try that with the bdev plugin here I get an average latency of 88usec which is well aligned with the drive spec of 85usec.
Thanks,
Joe
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [SPDK] FIO NVMe Performance Results
@ 2018-11-05 16:48 Andrey Kuzmin
0 siblings, 0 replies; 25+ messages in thread
From: Andrey Kuzmin @ 2018-11-05 16:48 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 30535 bytes --]
On Mon, Nov 5, 2018, 18:29 Gruher, Joseph R <joseph.r.gruher(a)intel.com>
wrote:
> My FIO version is v3.11.
>
> I tried the bdev FIO plugin as suggested, but performance is about the
> same as the other SPDK FIO plugin, I observed only 10.4K IOPS in a quick
> 4KB random read test. See configuration and results below.
>
As 12 ms median read latency for an nvme ssd is far beyond anything
reasonable, why don't you just perf this run, to see where the time is
being spent?
Regards,
Andrey
>
>
> don(a)donst201:~/fio/single/rr$ cat nvme.conf
> [Nvme]
> TransportId "trtype:PCIe traddr:0000.1c.00.0" Nvme0
>
>
> don(a)donst201:~/fio/single/rr$ cat 1c-bdev.ini |grep -v '#'
> [global]
> rw=randrw
> rwmixread=100
> numjobs=4
> iodepth=32
> bs=4k
> direct=1
> thread=1
> time_based=1
> ramp_time=0
> runtime=10
> ioengine=spdk_bdev
> spdk_conf=./nvme.conf
> group_reporting=1
> unified_rw_reporting=1
> exitall=1
> randrepeat=0
> norandommap=1
> [0000.1c.00.0]
> filename=Nvme0n1
>
>
> don(a)donst201:~/fio/single/rr$ sudo
> LD_PRELOAD=/home/don/install/spdk/spdk/examples/bdev/fio_plugin/fio_plugin
> fio 1c-bdev.ini
> 0000.1c.00.0: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T)
> 4096B-4096B, ioengine=spdk_bdev, iodepth=32
> ...
> fio-3.11
> Starting 4 threads
> Starting SPDK v18.10 / DPDK 18.08.0 initialization...
> [ DPDK EAL parameters: fio --no-shconf -c 0x1 --file-prefix=spdk_pid4865 ]
> EAL: Detected 36 lcore(s)
> EAL: Detected 2 NUMA nodes
> EAL: No free hugepages reported in hugepages-1048576kB
> EAL: Probing VFIO support...
> EAL: PCI device 0000:03:00.0 on NUMA socket 0
> EAL: probe driver: 8086:f1a5 spdk_nvme
> EAL: PCI device 0000:1c:00.0 on NUMA socket 0
> EAL: probe driver: 8086:a54 spdk_nvme
> EAL: PCI device 0000:1d:00.0 on NUMA socket 0
> EAL: probe driver: 8086:a54 spdk_nvme
> EAL: PCI device 0000:5e:00.0 on NUMA socket 0
> EAL: probe driver: 8086:a54 spdk_nvme
> EAL: PCI device 0000:5f:00.0 on NUMA socket 0
> EAL: probe driver: 8086:a54 spdk_nvme
> EAL: PCI device 0000:62:00.0 on NUMA socket 0
> EAL: probe driver: 8086:a54 spdk_nvme
> EAL: PCI device 0000:63:00.0 on NUMA socket 0
> EAL: probe driver: 8086:a54 spdk_nvme
> EAL: PCI device 0000:64:00.0 on NUMA socket 0
> EAL: probe driver: 8086:a54 spdk_nvme
> EAL: PCI device 0000:65:00.0 on NUMA socket 0
> EAL: probe driver: 8086:a54 spdk_nvme
> EAL: PCI device 0000:da:00.0 on NUMA socket 1
> EAL: probe driver: 8086:a54 spdk_nvme
> EAL: PCI device 0000:db:00.0 on NUMA socket 1
> EAL: probe driver: 8086:a54 spdk_nvme
> EAL: PCI device 0000:dc:00.0 on NUMA socket 1
> EAL: probe driver: 8086:a54 spdk_nvme
> EAL: PCI device 0000:dd:00.0 on NUMA socket 1
> EAL: probe driver: 8086:a54 spdk_nvme
> EAL: PCI device 0000:e0:00.0 on NUMA socket 1
> EAL: probe driver: 8086:a54 spdk_nvme
> EAL: PCI device 0000:e1:00.0 on NUMA socket 1
> EAL: probe driver: 8086:a54 spdk_nvme
> EAL: PCI device 0000:e2:00.0 on NUMA socket 1
> EAL: probe driver: 8086:a54 spdk_nvme
> EAL: PCI device 0000:e3:00.0 on NUMA socket 1
> EAL: probe driver: 8086:a54 spdk_nvme
> Jobs: 4 (f=4): [r(4)][100.0%][r=45.7MiB/s][r=11.7k IOPS][eta 00m:00s]
> 0000.1c.00.0: (groupid=0, jobs=4): err= 0: pid=4906: Mon Nov 5 15:23:47
> 2018
> mixed: IOPS=10.4k, BW=40.5MiB/s (42.5MB/s)(406MiB/10012msec)
> slat (nsec): min=201, max=15831, avg=238.11, stdev=148.78
> clat (usec): min=124, max=31244, avg=12330.69, stdev=1697.98
> lat (usec): min=130, max=31244, avg=12330.93, stdev=1697.98
> clat percentiles (usec):
> | 1.00th=[ 9634], 5.00th=[10159], 10.00th=[10552], 20.00th=[10945],
> | 30.00th=[11338], 40.00th=[11731], 50.00th=[12125], 60.00th=[12518],
> | 70.00th=[13042], 80.00th=[13566], 90.00th=[14353], 95.00th=[15008],
> | 99.00th=[16712], 99.50th=[18744], 99.90th=[27395], 99.95th=[29754],
> | 99.99th=[30278]
> bw ( KiB/s): min= 8784, max=11952, per=24.99%, avg=10375.53,
> stdev=929.68, samples=80
> iops : min= 2196, max= 2988, avg=2593.85, stdev=232.41,
> samples=80
> lat (usec) : 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
> lat (msec) : 2=0.01%, 4=0.02%, 10=2.71%, 20=96.91%, 50=0.33%
> cpu : usr=99.99%, sys=0.01%, ctx=28, majf=0, minf=0
> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.9%,
> >=64=0.0%
> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
> >=64=0.0%
> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%,
> >=64=0.0%
> issued rwts: total=103907,0,0,0 short=0,0,0,0 dropped=0,0,0,0
> latency : target=0, window=0, percentile=100.00%, depth=32
> Run status group 0 (all jobs):
> MIXED: bw=40.5MiB/s (42.5MB/s), 40.5MiB/s-40.5MiB/s (42.5MB/s-42.5MB/s),
> io=406MiB (426MB), run=10012-10012msec
>
>
> Thanks,
> Joe
>
> > -----Original Message-----
> > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of ????? ???????
> > Sent: Monday, November 5, 2018 12:45 AM
> > To: Storage Performance Development Kit <spdk(a)lists.01.org>
> > Subject: Re: [SPDK] FIO NVMe Performance Results
> >
> > Ok, Do you see some performance issues using general bdev fio plugin
> with
> > you configuration ?
> >
> > Best Regards
> > Alex
> >
> >
> > >Понедельник, 5 ноября 2018, 11:39 +03:00 от "Cao, Gang"
> > <gang.cao(a)intel.com>:
> > >
> > >Yes. This is for the bdev version of SPDK FIO plugin and you can assign
> the
> > conf file as the parameter in the FIO configuration.
> > >
> > >Previously, it's another SPDK FIO plugin for the user space NVMe driver
> > only.
> > >
> > >These are two different FIO plugins.
> > >
> > >Thanks,
> > >Gang
> > >
> > >-----Original Message-----
> > >From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Маров
> > >Алексей
> > >Sent: Monday, November 5, 2018 4:32 PM
> > >To: Storage Performance Development Kit < spdk(a)lists.01.org >
> > >Subject: Re: [SPDK] FIO NVMe Performance Results
> > >
> > >Sorry for incomplete message I have sent before
> > >
> > >Hello Gang.
> > >
> > >I just curious, why would you run fio with SPDK using this conf file,
> with is a
> > bit different from what is suggested in
> > here https://github.com/spdk/spdk/tree/master/examples/bdev/fio_plugin
> > .
> > >
> > >I mean may be you should create conf file for spdk like this one :
> > >
> > >nvme.conf :
> > >[Nvme]
> > > TransportId "trtype:PCIe traddr:0000.1c.00.0" Nvme0
> > >
> > >and provide it in fio config.
> > >
> > >fio.ini :
> > >[global]
> > >ioengine=spdk_bdev
> > >spdk_conf=....../nvme.conf
> > >rw=randrw
> > >rwmixread=100
> > >numjobs=4
> > >iodepth=32
> > >bs=4k
> > >direct=1
> > >thread=1
> > >time_based=1
> > >ramp_time=0
> > >runtime=10
> > >group_reporting=1
> > >unified_rw_reporting=1
> > >exitall=1
> > >randrepeat=0
> > >norandommap=1
> > >
> > >[0000:06:00.0]
> > >filename=Nvme0n1
> > >
> > >and run like this
> > >
> > ># LD_PRELOAD=examples/bdev/fio_plugin/fio_plugin fio
> > >examples/bdev/fio_plugin/fio.ini
> > >
> > >Just to note. The recommended version of fio for SPDK is 3.3
> > >
> > >Best Regards
> > >Alex
> > >
> > >
> > >>Понедельник, 5 ноября 2018, 9:53 +03:00 от "Cao, Gang" <
> > gang.cao(a)intel.com >:
> > >>
> > >>Thanks. This is interesting especially for your finding on queue depth
> 1
> > case. We are trying to reproduce this issue locally same as your
> > configuration.
> > >>
> > >>Thanks,
> > >>Gang
> > >>
> > >>-----Original Message-----
> > >>From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Gruher,
> > >>Joseph R
> > >>Sent: Monday, November 5, 2018 1:32 PM
> > >>To: Storage Performance Development Kit < spdk(a)lists.01.org >
> > >>Subject: Re: [SPDK] FIO NVMe Performance Results
> > >>
> > >>Interesting catch. It seems like FIO is actually not accurately
> reporting the
> > number of jobs in the report out for the SPDK case. If I disable the
> > group_reporting parameter I do get a printout for four separate jobs in
> the
> > SPDK case, so I think it is really running four jobs there, and just
> summarizing
> > incorrectly. I also see four separate CPU cores go to 100% when I run
> that
> > FIO test case, indicating four jobs are active.
> > >>
> > >>Also, I don't think would explain the difference in performance.
> Using an
> > aggregate queue depth of 32 versus 128 shouldn't lead to a drop in
> > performance from 377K to 13K.
> > >>
> > >>I retested SPDK with multiple separate job descriptions as you
> suggested
> > anyway, using four separate jobs instead of relying on the numjobs=4
> > parameter, and the results are below. In that SPDK test with four
> separate
> > FIO job statements, each with queue depth 32, I still only achieved 12.5K
> > IOPS.
> > >>
> > >>I then re-ran the kernel NVMe driver test case with only a single job
> for
> > comparison, and even with only 1 job and queue depth 32 it still achieves
> > 254K IOPS.
> > >>
> > >>We might be on the right track with a problem around queue depth
> > however. If I run the kernel NVMe driver case with just 1 job and queue
> > depth of only 1, I then get 11.2K IOPS, which is around the performance
> I get
> > in the SPDK test case. Perhaps the SPDK case is not actually queuing
> IOs on
> > the device for some reason?
> > >>
> > >>
> > >>don(a)donst201:~/fio/single/rr$ cat nvme2n1.ini |grep -v '#'
> > >>[global]
> > >>rw=randrw
> > >>rwmixread=100
> > >>numjobs=1
> > >>iodepth=32
> > >>bs=4k
> > >>direct=1
> > >>thread=1
> > >>time_based=1
> > >>ramp_time=0
> > >>runtime=10
> > >>ioengine=libaio
> > >>group_reporting=1
> > >>unified_rw_reporting=1
> > >>exitall=1
> > >>randrepeat=0
> > >>norandommap=1
> > >>[nvme2n1]
> > >>filename=/dev/nvme2n1
> > >>
> > >>
> > >>don(a)donst201:~/fio/single/rr$ sudo fio nvme2n1.ini
> > >>nvme2n1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T)
> > >>4096B-4096B, ioengine=libaio, iodepth=32
> > >>fio-3.11
> > >>Starting 1 thread
> > >>Jobs: 1 (f=1): [r(1)][100.0%][r=1025MiB/s][r=262k IOPS][eta 00m:00s]
> > >>nvme2n1: (groupid=0, jobs=1): err= 0: pid=2422: Mon Nov 5 05:19:13
> > >>2018
> > >> mixed: IOPS=254k, BW=992MiB/s (1040MB/s)(9922MiB/10001msec)
> > >> slat (nsec): min=1185, max=319052, avg=1611.64, stdev=808.40
> > >> clat (usec): min=2, max=4743, avg=124.02, stdev=121.00
> > >> lat (usec): min=6, max=4744, avg=125.68, stdev=121.02
> > >> clat percentiles (usec):
> > >> | 1.00th=[ 9], 5.00th=[ 56], 10.00th=[ 61], 20.00th=[
> 73],
> > >> | 30.00th=[ 80], 40.00th=[ 91], 50.00th=[ 110], 60.00th=[
> 118],
> > >> | 70.00th=[ 130], 80.00th=[ 157], 90.00th=[ 202], 95.00th=[
> > >>253],
> > >> | 99.00th=[ 408], 99.50th=[ 498], 99.90th=[ 1795], 99.95th=[
> > >>3064],
> > >> | 99.99th=[ 3785]
> > >> bw ( KiB/s): min=612240, max=1078792, per=100.00%, avg=1015996.40,
> > stdev=106032.96, samples=20
> > >> iops : min=153060, max=269698, avg=253998.90, stdev=26508.40,
> > samples=20
> > >> lat (usec) : 4=0.01%, 10=1.47%, 20=1.19%, 50=0.17%, 100=42.03%
> > >> lat (usec) : 250=49.95%, 500=4.70%, 750=0.28%, 1000=0.06%
> > >> lat (msec) : 2=0.06%, 4=0.09%, 10=0.01%
> > >> cpu : usr=21.45%, sys=51.11%, ctx=530348, majf=0, minf=139
> > >> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%,
> > >=64=0.0%
> > >> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
> > >=64=0.0%
> > >> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%,
> > >>>=64=0.0%
> > >> issued rwts: total=2540023,0,0,0 short=0,0,0,0 dropped=0,0,0,0
> > >> latency : target=0, window=0, percentile=100.00%, depth=32
> > >>Run status group 0 (all jobs):
> > >> MIXED: bw=992MiB/s (1040MB/s), 992MiB/s-992MiB/s (1040MB/s-
> > 1040MB/s), io=9922MiB (10.4GB), run=10001-10001msec Disk stats
> > (read/write):
> > >> nvme2n1: ios=2514356/0, merge=0/0, ticks=400/0, in_queue=297612,
> > >>util=99.06%
> > >>
> > >>
> > >>don(a)donst201:~/fio/single/rr$ cat 1c.ini |grep -v '#'
> > >>[global]
> > >>rw=randrw
> > >>rwmixread=100
> > >>numjobs=1
> > >>iodepth=32
> > >>bs=4k
> > >>direct=1
> > >>thread=1
> > >>time_based=1
> > >>ramp_time=0
> > >>runtime=10
> > >>ioengine=/home/don/install/spdk/spdk/examples/nvme/fio_plugin/fio_pl
> > ug
> > >>in
> > >>group_reporting=1
> > >>unified_rw_reporting=1
> > >>exitall=1
> > >>randrepeat=0
> > >>norandommap=1
> > >>[0000.1c.00.0-1]
> > >>filename=trtype=PCIe traddr=0000.1c.00.0 ns=1 [0000.1c.00.0-2]
> > >>filename=trtype=PCIe traddr=0000.1c.00.0 ns=1 [0000.1c.00.0-3]
> > >>filename=trtype=PCIe traddr=0000.1c.00.0 ns=1 [0000.1c.00.0-4]
> > >>filename=trtype=PCIe traddr=0000.1c.00.0 ns=1
> > >>
> > >>
> > >>don(a)donst201:~/fio/single/rr$ sudo fio 1c.ini
> > >>0000.1c.00.0-1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B,
> > >>(T) 4096B-4096B, ioengine=spdk, iodepth=32
> > >>0000.1c.00.0-2: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B,
> > >>(T) 4096B-4096B, ioengine=spdk, iodepth=32
> > >>0000.1c.00.0-3: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B,
> > >>(T) 4096B-4096B, ioengine=spdk, iodepth=32
> > >>0000.1c.00.0-4: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B,
> > >>(T) 4096B-4096B, ioengine=spdk, iodepth=32
> > >>fio-3.11
> > >>Starting 4 threads
> > >>Starting SPDK v18.10 / DPDK 18.08.0 initialization...
> > >>[ DPDK EAL parameters: fio --no-shconf -c 0x1 -m 512
> > >>--file-prefix=spdk_pid4270 ]
> > >>EAL: Detected 36 lcore(s)
> > >>EAL: Detected 2 NUMA nodes
> > >>EAL: No free hugepages reported in hugepages-1048576kB
> > >>EAL: Probing VFIO support...
> > >>EAL: PCI device 0000:1c:00.0 on NUMA socket 0
> > >>EAL: probe driver: 8086:a54 spdk_nvme
> > >>Jobs: 1 (f=0): [_(3),f(1)][100.0%][eta 00m:00s]
> > >>0000.1c.00.0-1: (groupid=0, jobs=4): err= 0: pid=4310: Mon Nov 5
> > >>05:16:03 2018
> > >> mixed: IOPS=12.5k, BW=48.0MiB/s (51.4MB/s)(490MiB/10010msec)
> > >> slat (nsec): min=98, max=15092, avg=120.02, stdev=90.07
> > >> clat (usec): min=55, max=19757, avg=10207.22, stdev=884.06
> > >> lat (usec): min=60, max=19757, avg=10207.34, stdev=884.06
> > >> clat percentiles (usec):
> > >> | 1.00th=[ 8979], 5.00th=[ 9110], 10.00th=[ 9372], 20.00th=[
> > >>9503],
> > >> | 30.00th=[ 9765], 40.00th=[ 9896], 50.00th=[10159],
> > >>60.00th=[10290],
> > >> | 70.00th=[10552], 80.00th=[10814], 90.00th=[11207],
> > >>95.00th=[11600],
> > >> | 99.00th=[12911], 99.50th=[14353], 99.90th=[17695],
> > >>99.95th=[18744],
> > >> | 99.99th=[19006]
> > >> bw ( KiB/s): min=12008, max=12976, per=25.00%, avg=12535.70,
> > stdev=306.81, samples=80
> > >> iops : min= 3002, max= 3244, avg=3133.85, stdev=76.77,
> samples=80
> > >> lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
> > >> lat (msec) : 2=0.01%, 4=0.02%, 10=45.13%, 20=54.83%
> > >> cpu : usr=100.01%, sys=0.00%, ctx=27, majf=0, minf=0
> > >> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.9%,
> > >=64=0.0%
> > >> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
> > >=64=0.0%
> > >> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%,
> > >>>=64=0.0%
> > >> issued rwts: total=125506,0,0,0 short=0,0,0,0 dropped=0,0,0,0
> > >> latency : target=0, window=0, percentile=100.00%, depth=32
> > >>Run status group 0 (all jobs):
> > >> MIXED: bw=48.0MiB/s (51.4MB/s), 48.0MiB/s-48.0MiB/s
> > >>(51.4MB/s-51.4MB/s), io=490MiB (514MB), run=10010-10010msec
> > >>
> > >>
> > >>don(a)donst201:~/fio/single/rr$ cat nvme2n1.ini |grep -v '#'
> > >>[global]
> > >>rw=randrw
> > >>rwmixread=100
> > >>numjobs=1
> > >>iodepth=1
> > >>bs=4k
> > >>direct=1
> > >>thread=1
> > >>time_based=1
> > >>ramp_time=0
> > >>runtime=10
> > >>ioengine=libaio
> > >>group_reporting=1
> > >>unified_rw_reporting=1
> > >>exitall=1
> > >>randrepeat=0
> > >>norandommap=1
> > >>[nvme2n1]
> > >>filename=/dev/nvme2n1
> > >>
> > >>
> > >>don(a)donst201:~/fio/single/rr$ sudo fio nvme2n1.ini
> > >>nvme2n1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T)
> > >>4096B-4096B, ioengine=libaio, iodepth=1
> > >>fio-3.11
> > >>Starting 1 thread
> > >>Jobs: 1 (f=1): [r(1)][100.0%][r=43.6MiB/s][r=11.2k IOPS][eta 00m:00s]
> > >>nvme2n1: (groupid=0, jobs=1): err= 0: pid=2525: Mon Nov 5 05:22:57
> > >>2018
> > >> mixed: IOPS=11.2k, BW=43.6MiB/s (45.7MB/s)(436MiB/10001msec)
> > >> slat (nsec): min=4848, max=208595, avg=5126.33, stdev=771.27
> > >> clat (nsec): min=1845, max=3405.8k, avg=83004.99, stdev=33577.36
> > >> lat (usec): min=15, max=3411, avg=88.26, stdev=33.59
> > >> clat percentiles (usec):
> > >> | 1.00th=[ 15], 5.00th=[ 58], 10.00th=[ 59], 20.00th=[
> 59],
> > >> | 30.00th=[ 60], 40.00th=[ 77], 50.00th=[ 77], 60.00th=[
> 78],
> > >> | 70.00th=[ 111], 80.00th=[ 113], 90.00th=[ 113], 95.00th=[
> > >>115],
> > >> | 99.00th=[ 186], 99.50th=[ 186], 99.90th=[ 190], 99.95th=[
> > >>192],
> > >> | 99.99th=[ 375]
> > >> bw ( KiB/s): min=43816, max=45016, per=99.98%, avg=44639.58,
> > stdev=312.88, samples=19
> > >> iops : min=10954, max=11254, avg=11159.89, stdev=78.22,
> > samples=19
> > >> lat (usec) : 2=0.01%, 20=2.83%, 50=0.01%, 100=64.57%, 250=32.59%
> > >> lat (usec) : 500=0.01%, 750=0.01%
> > >> lat (msec) : 2=0.01%, 4=0.01%
> > >> cpu : usr=4.02%, sys=10.33%, ctx=111630, majf=0, minf=12
> > >> IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%,
> > >=64=0.0%
> > >> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
> > >=64=0.0%
> > >> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
> > >>>=64=0.0%
> > >> issued rwts: total=111631,0,0,0 short=0,0,0,0 dropped=0,0,0,0
> > >> latency : target=0, window=0, percentile=100.00%, depth=1
> > >>Run status group 0 (all jobs):
> > >> MIXED: bw=43.6MiB/s (45.7MB/s), 43.6MiB/s-43.6MiB/s (45.7MB/s-
> > 45.7MB/s), io=436MiB (457MB), run=10001-10001msec Disk stats
> > (read/write):
> > >> nvme2n1: ios=110275/0, merge=0/0, ticks=0/0, in_queue=8596,
> > >>util=86.22%
> > >>
> > >>
> > >>Thanks,
> > >>Joe
> > >>
> > >>> -----Original Message-----
> > >>> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Cao, Gang
> > >>> Sent: Sunday, November 4, 2018 8:40 PM
> > >>> To: Storage Performance Development Kit < spdk(a)lists.01.org >
> > >>> Subject: Re: [SPDK] FIO NVMe Performance Results
> > >>>
> > >>> For the SPDK plugin, from the below result, looks like only 1 job
> > >>> started with
> > >>> 32 queue depth:
> > >>>
> > >>> Jobs: 1 (f=0): [f(1),_(3)][100.0%][eta 00m:00s] <<<<
> > >>>
> > >>> Try to update the FIO configuration file as
> > >>>
> > >>> [job1]
> > >>> filename=trtype=PCIe traddr=0000.1c.00.0 ns=1 [job2]
> > >>> filename=trtype=PCIe
> > >>> traddr=0000.1c.00.0 ns=1 ....
> > >>>
> > >>> The kernel FIO is with 4 jobs and each job of 32 queue depth. For
> > >>> SPDK, you can configure as the same, or just configure two jobs and
> > >>> each job of 64 queue depth. The overall queue depth to the device is
> > same of 128.
> > >>>
> > >>> Thanks,
> > >>> Gang
> > >>>
> > >>> -----Original Message-----
> > >>> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Gruher,
> > >>> Joseph R
> > >>> Sent: Monday, November 5, 2018 10:49 AM
> > >>> To: Storage Performance Development Kit < spdk(a)lists.01.org >
> > >>> Subject: [SPDK] FIO NVMe Performance Results
> > >>>
> > >>> Hi folks-
> > >>>
> > >>> I'm testing SPDK 18.10 on Ubuntu 18.04 with kernel 4.18.16. I used
> > >>> FIO with the kernel NVMe driver to measure the performance of a
> > >>> local (PCIe
> > >>> attached) Intel P4500 NVMe device on a 4KB random read workload and
> > >>> obtained 477K IOPS, roughly in line with the drive spec. Then I
> > >>> tested the same drive with the SPDK FIO plugin and only achieved 13K
> > >>> IOPS. The FIO test files and the results are pasted below. Any
> > >>> ideas where I'm going wrong here?
> > >>>
> > >>> Thanks!
> > >>>
> > >>>
> > >>> don(a)donst201:~/fio/single/rr$ ls
> > >>> /sys/devices/pci0000:17/0000:17:02.0/0000:1c:00.0/nvme/nvme1/
> > >>> address cntlid dev device firmware_rev model nvme2n1 power
> > >>> rescan_controller reset_controller serial state subsysnqn
> > >>> subsystem transport uevent
> > >>>
> > >>>
> > >>> don(a)donst201:~/fio/single/rr$ cat nvme2n1.ini |grep -v '#'
> > >>> [global]
> > >>> rw=randrw
> > >>> rwmixread=100
> > >>> numjobs=4
> > >>> iodepth=32
> > >>> bs=4k
> > >>> direct=1
> > >>> thread=1
> > >>> time_based=1
> > >>> ramp_time=0
> > >>> runtime=10
> > >>> ioengine=libaio
> > >>> group_reporting=1
> > >>> unified_rw_reporting=1
> > >>> exitall=1
> > >>> randrepeat=0
> > >>> norandommap=1
> > >>> [nvme2n1]
> > >>> filename=/dev/nvme2n1
> > >>>
> > >>>
> > >>> don(a)donst201:~/fio/single/rr$ sudo fio nvme2n1.ini
> > >>> nvme2n1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T)
> > >>> 4096B-4096B, ioengine=libaio, iodepth=32 ...
> > >>> fio-3.11
> > >>> Starting 4 threads
> > >>> Jobs: 4 (f=4): [r(4)][100.0%][r=1874MiB/s][r=480k IOPS][eta 00m:00s]
> > >>> nvme2n1: (groupid=0, jobs=4): err= 0: pid=2575: Mon Nov 5 02:26:25
> > 2018
> > >>> mixed: IOPS=477k, BW=1862MiB/s (1952MB/s)(18.2GiB/10001msec)
> > >>> slat (nsec): min=1295, max=333088, avg=2216.35, stdev=927.89
> > >>> clat (nsec): min=506, max=5818.8k, avg=265924.16, stdev=223761.59
> > >>> lat (usec): min=6, max=5821, avg=268.19, stdev=223.76
> > >>> clat percentiles (usec):
> > >>> | 1.00th=[ 13], 5.00th=[ 74], 10.00th=[ 87], 20.00th=[
> 115],
> > >>> | 30.00th=[ 139], 40.00th=[ 167], 50.00th=[ 204], 60.00th=[
> 247],
> > >>> | 70.00th=[ 306], 80.00th=[ 388], 90.00th=[ 523], 95.00th=[
> 660],
> > >>> | 99.00th=[ 988], 99.50th=[ 1156], 99.90th=[ 2343], 99.95th=[
> 3195],
> > >>> | 99.99th=[ 4424]
> > >>> bw ( KiB/s): min=448192, max=483104, per=25.00%, avg=476597.10,
> > >>> stdev=6748.78, samples=80
> > >>> iops : min=112048, max=120776, avg=119149.27,
> stdev=1687.20,
> > >>> samples=80
> > >>> lat (nsec) : 750=0.01%
> > >>> lat (usec) : 10=0.52%, 20=1.25%, 50=0.25%, 100=13.38%, 250=45.13%
> > >>> lat (usec) : 500=28.23%, 750=8.08%, 1000=2.22%
> > >>> lat (msec) : 2=0.82%, 4=0.10%, 10=0.02%
> > >>> cpu : usr=14.08%, sys=36.02%, ctx=1930207, majf=0,
> minf=697
> > >>> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%,
> > >>> >=64=0.0%
> > >>> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
> > >>> >=64=0.0%
> > >>> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%,
> > >>> 64=0.0%,
> > >>> >=64=0.0%
> > >>> issued rwts: total=4766095,0,0,0 short=0,0,0,0 dropped=0,0,0,0
> > >>> latency : target=0, window=0, percentile=100.00%, depth=32
> > >>> Run status group 0 (all jobs):
> > >>> MIXED: bw=1862MiB/s (1952MB/s), 1862MiB/s-1862MiB/s (1952MB/s-
> > >>> 1952MB/s), io=18.2GiB (19.5GB), run=10001-10001msec Disk stats
> > >>> (read/write):
> > >>> nvme2n1: ios=4709190/0, merge=0/0, ticks=4096/0,
> > in_queue=1388752,
> > >>> util=100.00%
> > >>>
> > >>>
> > >>> don(a)donst201:~/fio/single/rr$ sudo
> > >>> /home/don/install/spdk/spdk/scripts/setup.sh
> > >>> Active mountpoints on /dev/nvme0n1, so not binding PCI dev
> > >>> 0000:03:00.0
> > >>> 0000:1c:00.0 (8086 0a54): nvme -> uio_pci_generic
> > >>> 0000:1d:00.0 (8086 0a54): nvme -> uio_pci_generic
> > >>> 0000:5e:00.0 (8086 0a54): nvme -> uio_pci_generic
> > >>> 0000:5f:00.0 (8086 0a54): nvme -> uio_pci_generic
> > >>> 0000:62:00.0 (8086 0a54): nvme -> uio_pci_generic
> > >>> 0000:63:00.0 (8086 0a54): nvme -> uio_pci_generic
> > >>> 0000:64:00.0 (8086 0a54): nvme -> uio_pci_generic
> > >>> 0000:65:00.0 (8086 0a54): nvme -> uio_pci_generic
> > >>> 0000:da:00.0 (8086 0a54): nvme -> uio_pci_generic
> > >>> 0000:db:00.0 (8086 0a54): nvme -> uio_pci_generic
> > >>> 0000:dc:00.0 (8086 0a54): nvme -> uio_pci_generic
> > >>> 0000:dd:00.0 (8086 0a54): nvme -> uio_pci_generic
> > >>> 0000:e0:00.0 (8086 0a54): nvme -> uio_pci_generic
> > >>> 0000:e1:00.0 (8086 0a54): nvme -> uio_pci_generic
> > >>> 0000:e2:00.0 (8086 0a54): nvme -> uio_pci_generic
> > >>> 0000:e3:00.0 (8086 0a54): nvme -> uio_pci_generic
> > >>> 0000:00:04.0 (8086 2021): ioatdma -> uio_pci_generic
> > >>> 0000:00:04.1 (8086 2021): ioatdma -> uio_pci_generic
> > >>> 0000:00:04.2 (8086 2021): ioatdma -> uio_pci_generic
> > >>> 0000:00:04.3 (8086 2021): ioatdma -> uio_pci_generic
> > >>> 0000:00:04.4 (8086 2021): ioatdma -> uio_pci_generic
> > >>> 0000:00:04.5 (8086 2021): ioatdma -> uio_pci_generic
> > >>> 0000:00:04.6 (8086 2021): ioatdma -> uio_pci_generic
> > >>> 0000:00:04.7 (8086 2021): ioatdma -> uio_pci_generic
> > >>> 0000:80:04.0 (8086 2021): ioatdma -> uio_pci_generic
> > >>> 0000:80:04.1 (8086 2021): ioatdma -> uio_pci_generic
> > >>> 0000:80:04.2 (8086 2021): ioatdma -> uio_pci_generic
> > >>> 0000:80:04.3 (8086 2021): ioatdma -> uio_pci_generic
> > >>> 0000:80:04.4 (8086 2021): ioatdma -> uio_pci_generic
> > >>> 0000:80:04.5 (8086 2021): ioatdma -> uio_pci_generic
> > >>> 0000:80:04.6 (8086 2021): ioatdma -> uio_pci_generic
> > >>> 0000:80:04.7 (8086 2021): ioatdma -> uio_pci_generic
> > >>>
> > >>>
> > >>> don(a)donst201:~/fio/single/rr$ cat 1c.ini |grep -v '#'
> > >>> [global]
> > >>> rw=randrw
> > >>> rwmixread=100
> > >>> numjobs=4
> > >>> iodepth=32
> > >>> bs=4k
> > >>> direct=1
> > >>> thread=1
> > >>> time_based=1
> > >>> ramp_time=0
> > >>> runtime=10
> > >>>
> > ioengine=/home/don/install/spdk/spdk/examples/nvme/fio_plugin/fio_pl
> > >>> ug
> > >>> in
> > >>> group_reporting=1
> > >>> unified_rw_reporting=1
> > >>> exitall=1
> > >>> randrepeat=0
> > >>> norandommap=1
> > >>> [0000.1c.00.0]
> > >>> filename=trtype=PCIe traddr=0000.1c.00.0 ns=1
> > >>>
> > >>>
> > >>> don(a)donst201:~/fio/single/rr$ sudo fio 1c.ini
> > >>> 0000.1c.00.0: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B,
> > >>> (T) 4096B-4096B, ioengine=spdk, iodepth=32 ...
> > >>> fio-3.11
> > >>> Starting 4 threads
> > >>> Starting SPDK v18.10 / DPDK 18.08.0 initialization...
> > >>> [ DPDK EAL parameters: fio --no-shconf -c 0x1 -m 512 --file-
> > >>> prefix=spdk_pid3668 ]
> > >>> EAL: Detected 36 lcore(s)
> > >>> EAL: Detected 2 NUMA nodes
> > >>> EAL: No free hugepages reported in hugepages-1048576kB
> > >>> EAL: Probing VFIO support...
> > >>> EAL: PCI device 0000:1c:00.0 on NUMA socket 0
> > >>> EAL: probe driver: 8086:a54 spdk_nvme
> > >>> Jobs: 1 (f=0): [f(1),_(3)][100.0%][eta 00m:00s]
> > >>> 0000.1c.00.0: (groupid=0, jobs=4): err= 0: pid=3709: Mon Nov 5
> > >>> 02:28:29
> > >>> 2018
> > >>> mixed: IOPS=13.2k, BW=51.6MiB/s (54.1MB/s)(517MiB/10011msec)
> > >>> slat (nsec): min=109, max=15344, avg=127.91, stdev=73.44
> > >>> clat (usec): min=187, max=18715, avg=9683.80, stdev=650.54
> > >>> lat (usec): min=193, max=18715, avg=9683.93, stdev=650.53
> > >>> clat percentiles (usec):
> > >>> | 1.00th=[ 8717], 5.00th=[ 8848], 10.00th=[ 8979], 20.00th=[
> 9110],
> > >>> | 30.00th=[ 9241], 40.00th=[ 9372], 50.00th=[ 9503], 60.00th=[
> 9765],
> > >>> | 70.00th=[10028], 80.00th=[10159], 90.00th=[10552],
> > 95.00th=[10945],
> > >>> | 99.00th=[11338], 99.50th=[11338], 99.90th=[11731],
> > 99.95th=[13960],
> > >>> | 99.99th=[17695]
> > >>> bw ( KiB/s): min=13080, max=13304, per=25.00%, avg=13214.70,
> > >>> stdev=55.30, samples=80
> > >>> iops : min= 3270, max= 3326, avg=3303.65, stdev=13.84,
> > samples=80
> > >>> lat (usec) : 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
> > >>> lat (msec) : 2=0.01%, 4=0.02%, 10=70.85%, 20=29.11%
> > >>> cpu : usr=100.01%, sys=0.00%, ctx=23, majf=0, minf=0
> > >>> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.9%,
> > >>> >=64=0.0%
> > >>> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
> > >>> >=64=0.0%
> > >>> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%,
> > >>> 64=0.0%,
> > >>> >=64=0.0%
> > >>> issued rwts: total=132300,0,0,0 short=0,0,0,0 dropped=0,0,0,0
> > >>> latency : target=0, window=0, percentile=100.00%, depth=32
> > >>> Run status group 0 (all jobs):
> > >>> MIXED: bw=51.6MiB/s (54.1MB/s), 51.6MiB/s-51.6MiB/s (54.1MB/s-
> > >>> 54.1MB/s), io=517MiB (542MB), run=10011-10011msec
> > >>>
> > >>> _______________________________________________
> > >>> SPDK mailing list
> > >>> SPDK(a)lists.01.org
> > >>> https://lists.01.org/mailman/listinfo/spdk
> > >>> _______________________________________________
> > >>> SPDK mailing list
> > >>> SPDK(a)lists.01.org
> > >>> https://lists.01.org/mailman/listinfo/spdk
> > >>_______________________________________________
> > >>SPDK mailing list
> > >> SPDK(a)lists.01.org
> > >> https://lists.01.org/mailman/listinfo/spdk
> > >>_______________________________________________
> > >>SPDK mailing list
> > >> SPDK(a)lists.01.org
> > >> https://lists.01.org/mailman/listinfo/spdk
> > >
> > >
> > >--
> > >Маров Алексей
> > >_______________________________________________
> > >SPDK mailing list
> > >SPDK(a)lists.01.org
> > >https://lists.01.org/mailman/listinfo/spdk
> > >_______________________________________________
> > >SPDK mailing list
> > >SPDK(a)lists.01.org
> > >https://lists.01.org/mailman/listinfo/spdk
> >
> >
> > --
> > Маров Алексей
> > _______________________________________________
> > SPDK mailing list
> > SPDK(a)lists.01.org
> > https://lists.01.org/mailman/listinfo/spdk
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
>
--
Regards,
Andrey
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [SPDK] FIO NVMe Performance Results
@ 2018-11-05 15:29 Gruher, Joseph R
0 siblings, 0 replies; 25+ messages in thread
From: Gruher, Joseph R @ 2018-11-05 15:29 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 28808 bytes --]
My FIO version is v3.11.
I tried the bdev FIO plugin as suggested, but performance is about the same as the other SPDK FIO plugin, I observed only 10.4K IOPS in a quick 4KB random read test. See configuration and results below.
don(a)donst201:~/fio/single/rr$ cat nvme.conf
[Nvme]
TransportId "trtype:PCIe traddr:0000.1c.00.0" Nvme0
don(a)donst201:~/fio/single/rr$ cat 1c-bdev.ini |grep -v '#'
[global]
rw=randrw
rwmixread=100
numjobs=4
iodepth=32
bs=4k
direct=1
thread=1
time_based=1
ramp_time=0
runtime=10
ioengine=spdk_bdev
spdk_conf=./nvme.conf
group_reporting=1
unified_rw_reporting=1
exitall=1
randrepeat=0
norandommap=1
[0000.1c.00.0]
filename=Nvme0n1
don(a)donst201:~/fio/single/rr$ sudo LD_PRELOAD=/home/don/install/spdk/spdk/examples/bdev/fio_plugin/fio_plugin fio 1c-bdev.ini
0000.1c.00.0: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=32
...
fio-3.11
Starting 4 threads
Starting SPDK v18.10 / DPDK 18.08.0 initialization...
[ DPDK EAL parameters: fio --no-shconf -c 0x1 --file-prefix=spdk_pid4865 ]
EAL: Detected 36 lcore(s)
EAL: Detected 2 NUMA nodes
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: PCI device 0000:03:00.0 on NUMA socket 0
EAL: probe driver: 8086:f1a5 spdk_nvme
EAL: PCI device 0000:1c:00.0 on NUMA socket 0
EAL: probe driver: 8086:a54 spdk_nvme
EAL: PCI device 0000:1d:00.0 on NUMA socket 0
EAL: probe driver: 8086:a54 spdk_nvme
EAL: PCI device 0000:5e:00.0 on NUMA socket 0
EAL: probe driver: 8086:a54 spdk_nvme
EAL: PCI device 0000:5f:00.0 on NUMA socket 0
EAL: probe driver: 8086:a54 spdk_nvme
EAL: PCI device 0000:62:00.0 on NUMA socket 0
EAL: probe driver: 8086:a54 spdk_nvme
EAL: PCI device 0000:63:00.0 on NUMA socket 0
EAL: probe driver: 8086:a54 spdk_nvme
EAL: PCI device 0000:64:00.0 on NUMA socket 0
EAL: probe driver: 8086:a54 spdk_nvme
EAL: PCI device 0000:65:00.0 on NUMA socket 0
EAL: probe driver: 8086:a54 spdk_nvme
EAL: PCI device 0000:da:00.0 on NUMA socket 1
EAL: probe driver: 8086:a54 spdk_nvme
EAL: PCI device 0000:db:00.0 on NUMA socket 1
EAL: probe driver: 8086:a54 spdk_nvme
EAL: PCI device 0000:dc:00.0 on NUMA socket 1
EAL: probe driver: 8086:a54 spdk_nvme
EAL: PCI device 0000:dd:00.0 on NUMA socket 1
EAL: probe driver: 8086:a54 spdk_nvme
EAL: PCI device 0000:e0:00.0 on NUMA socket 1
EAL: probe driver: 8086:a54 spdk_nvme
EAL: PCI device 0000:e1:00.0 on NUMA socket 1
EAL: probe driver: 8086:a54 spdk_nvme
EAL: PCI device 0000:e2:00.0 on NUMA socket 1
EAL: probe driver: 8086:a54 spdk_nvme
EAL: PCI device 0000:e3:00.0 on NUMA socket 1
EAL: probe driver: 8086:a54 spdk_nvme
Jobs: 4 (f=4): [r(4)][100.0%][r=45.7MiB/s][r=11.7k IOPS][eta 00m:00s]
0000.1c.00.0: (groupid=0, jobs=4): err= 0: pid=4906: Mon Nov 5 15:23:47 2018
mixed: IOPS=10.4k, BW=40.5MiB/s (42.5MB/s)(406MiB/10012msec)
slat (nsec): min=201, max=15831, avg=238.11, stdev=148.78
clat (usec): min=124, max=31244, avg=12330.69, stdev=1697.98
lat (usec): min=130, max=31244, avg=12330.93, stdev=1697.98
clat percentiles (usec):
| 1.00th=[ 9634], 5.00th=[10159], 10.00th=[10552], 20.00th=[10945],
| 30.00th=[11338], 40.00th=[11731], 50.00th=[12125], 60.00th=[12518],
| 70.00th=[13042], 80.00th=[13566], 90.00th=[14353], 95.00th=[15008],
| 99.00th=[16712], 99.50th=[18744], 99.90th=[27395], 99.95th=[29754],
| 99.99th=[30278]
bw ( KiB/s): min= 8784, max=11952, per=24.99%, avg=10375.53, stdev=929.68, samples=80
iops : min= 2196, max= 2988, avg=2593.85, stdev=232.41, samples=80
lat (usec) : 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.02%, 10=2.71%, 20=96.91%, 50=0.33%
cpu : usr=99.99%, sys=0.01%, ctx=28, majf=0, minf=0
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.9%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued rwts: total=103907,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
Run status group 0 (all jobs):
MIXED: bw=40.5MiB/s (42.5MB/s), 40.5MiB/s-40.5MiB/s (42.5MB/s-42.5MB/s), io=406MiB (426MB), run=10012-10012msec
Thanks,
Joe
> -----Original Message-----
> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of ????? ???????
> Sent: Monday, November 5, 2018 12:45 AM
> To: Storage Performance Development Kit <spdk(a)lists.01.org>
> Subject: Re: [SPDK] FIO NVMe Performance Results
>
> Ok, Do you see some performance issues using general bdev fio plugin with
> you configuration ?
>
> Best Regards
> Alex
>
>
> >Понедельник, 5 ноября 2018, 11:39 +03:00 от "Cao, Gang"
> <gang.cao(a)intel.com>:
> >
> >Yes. This is for the bdev version of SPDK FIO plugin and you can assign the
> conf file as the parameter in the FIO configuration.
> >
> >Previously, it's another SPDK FIO plugin for the user space NVMe driver
> only.
> >
> >These are two different FIO plugins.
> >
> >Thanks,
> >Gang
> >
> >-----Original Message-----
> >From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Маров
> >Алексей
> >Sent: Monday, November 5, 2018 4:32 PM
> >To: Storage Performance Development Kit < spdk(a)lists.01.org >
> >Subject: Re: [SPDK] FIO NVMe Performance Results
> >
> >Sorry for incomplete message I have sent before
> >
> >Hello Gang.
> >
> >I just curious, why would you run fio with SPDK using this conf file, with is a
> bit different from what is suggested in
> here https://github.com/spdk/spdk/tree/master/examples/bdev/fio_plugin
> .
> >
> >I mean may be you should create conf file for spdk like this one :
> >
> >nvme.conf :
> >[Nvme]
> > TransportId "trtype:PCIe traddr:0000.1c.00.0" Nvme0
> >
> >and provide it in fio config.
> >
> >fio.ini :
> >[global]
> >ioengine=spdk_bdev
> >spdk_conf=....../nvme.conf
> >rw=randrw
> >rwmixread=100
> >numjobs=4
> >iodepth=32
> >bs=4k
> >direct=1
> >thread=1
> >time_based=1
> >ramp_time=0
> >runtime=10
> >group_reporting=1
> >unified_rw_reporting=1
> >exitall=1
> >randrepeat=0
> >norandommap=1
> >
> >[0000:06:00.0]
> >filename=Nvme0n1
> >
> >and run like this
> >
> ># LD_PRELOAD=examples/bdev/fio_plugin/fio_plugin fio
> >examples/bdev/fio_plugin/fio.ini
> >
> >Just to note. The recommended version of fio for SPDK is 3.3
> >
> >Best Regards
> >Alex
> >
> >
> >>Понедельник, 5 ноября 2018, 9:53 +03:00 от "Cao, Gang" <
> gang.cao(a)intel.com >:
> >>
> >>Thanks. This is interesting especially for your finding on queue depth 1
> case. We are trying to reproduce this issue locally same as your
> configuration.
> >>
> >>Thanks,
> >>Gang
> >>
> >>-----Original Message-----
> >>From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Gruher,
> >>Joseph R
> >>Sent: Monday, November 5, 2018 1:32 PM
> >>To: Storage Performance Development Kit < spdk(a)lists.01.org >
> >>Subject: Re: [SPDK] FIO NVMe Performance Results
> >>
> >>Interesting catch. It seems like FIO is actually not accurately reporting the
> number of jobs in the report out for the SPDK case. If I disable the
> group_reporting parameter I do get a printout for four separate jobs in the
> SPDK case, so I think it is really running four jobs there, and just summarizing
> incorrectly. I also see four separate CPU cores go to 100% when I run that
> FIO test case, indicating four jobs are active.
> >>
> >>Also, I don't think would explain the difference in performance. Using an
> aggregate queue depth of 32 versus 128 shouldn't lead to a drop in
> performance from 377K to 13K.
> >>
> >>I retested SPDK with multiple separate job descriptions as you suggested
> anyway, using four separate jobs instead of relying on the numjobs=4
> parameter, and the results are below. In that SPDK test with four separate
> FIO job statements, each with queue depth 32, I still only achieved 12.5K
> IOPS.
> >>
> >>I then re-ran the kernel NVMe driver test case with only a single job for
> comparison, and even with only 1 job and queue depth 32 it still achieves
> 254K IOPS.
> >>
> >>We might be on the right track with a problem around queue depth
> however. If I run the kernel NVMe driver case with just 1 job and queue
> depth of only 1, I then get 11.2K IOPS, which is around the performance I get
> in the SPDK test case. Perhaps the SPDK case is not actually queuing IOs on
> the device for some reason?
> >>
> >>
> >>don(a)donst201:~/fio/single/rr$ cat nvme2n1.ini |grep -v '#'
> >>[global]
> >>rw=randrw
> >>rwmixread=100
> >>numjobs=1
> >>iodepth=32
> >>bs=4k
> >>direct=1
> >>thread=1
> >>time_based=1
> >>ramp_time=0
> >>runtime=10
> >>ioengine=libaio
> >>group_reporting=1
> >>unified_rw_reporting=1
> >>exitall=1
> >>randrepeat=0
> >>norandommap=1
> >>[nvme2n1]
> >>filename=/dev/nvme2n1
> >>
> >>
> >>don(a)donst201:~/fio/single/rr$ sudo fio nvme2n1.ini
> >>nvme2n1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T)
> >>4096B-4096B, ioengine=libaio, iodepth=32
> >>fio-3.11
> >>Starting 1 thread
> >>Jobs: 1 (f=1): [r(1)][100.0%][r=1025MiB/s][r=262k IOPS][eta 00m:00s]
> >>nvme2n1: (groupid=0, jobs=1): err= 0: pid=2422: Mon Nov 5 05:19:13
> >>2018
> >> mixed: IOPS=254k, BW=992MiB/s (1040MB/s)(9922MiB/10001msec)
> >> slat (nsec): min=1185, max=319052, avg=1611.64, stdev=808.40
> >> clat (usec): min=2, max=4743, avg=124.02, stdev=121.00
> >> lat (usec): min=6, max=4744, avg=125.68, stdev=121.02
> >> clat percentiles (usec):
> >> | 1.00th=[ 9], 5.00th=[ 56], 10.00th=[ 61], 20.00th=[ 73],
> >> | 30.00th=[ 80], 40.00th=[ 91], 50.00th=[ 110], 60.00th=[ 118],
> >> | 70.00th=[ 130], 80.00th=[ 157], 90.00th=[ 202], 95.00th=[
> >>253],
> >> | 99.00th=[ 408], 99.50th=[ 498], 99.90th=[ 1795], 99.95th=[
> >>3064],
> >> | 99.99th=[ 3785]
> >> bw ( KiB/s): min=612240, max=1078792, per=100.00%, avg=1015996.40,
> stdev=106032.96, samples=20
> >> iops : min=153060, max=269698, avg=253998.90, stdev=26508.40,
> samples=20
> >> lat (usec) : 4=0.01%, 10=1.47%, 20=1.19%, 50=0.17%, 100=42.03%
> >> lat (usec) : 250=49.95%, 500=4.70%, 750=0.28%, 1000=0.06%
> >> lat (msec) : 2=0.06%, 4=0.09%, 10=0.01%
> >> cpu : usr=21.45%, sys=51.11%, ctx=530348, majf=0, minf=139
> >> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%,
> >=64=0.0%
> >> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
> >=64=0.0%
> >> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%,
> >>>=64=0.0%
> >> issued rwts: total=2540023,0,0,0 short=0,0,0,0 dropped=0,0,0,0
> >> latency : target=0, window=0, percentile=100.00%, depth=32
> >>Run status group 0 (all jobs):
> >> MIXED: bw=992MiB/s (1040MB/s), 992MiB/s-992MiB/s (1040MB/s-
> 1040MB/s), io=9922MiB (10.4GB), run=10001-10001msec Disk stats
> (read/write):
> >> nvme2n1: ios=2514356/0, merge=0/0, ticks=400/0, in_queue=297612,
> >>util=99.06%
> >>
> >>
> >>don(a)donst201:~/fio/single/rr$ cat 1c.ini |grep -v '#'
> >>[global]
> >>rw=randrw
> >>rwmixread=100
> >>numjobs=1
> >>iodepth=32
> >>bs=4k
> >>direct=1
> >>thread=1
> >>time_based=1
> >>ramp_time=0
> >>runtime=10
> >>ioengine=/home/don/install/spdk/spdk/examples/nvme/fio_plugin/fio_pl
> ug
> >>in
> >>group_reporting=1
> >>unified_rw_reporting=1
> >>exitall=1
> >>randrepeat=0
> >>norandommap=1
> >>[0000.1c.00.0-1]
> >>filename=trtype=PCIe traddr=0000.1c.00.0 ns=1 [0000.1c.00.0-2]
> >>filename=trtype=PCIe traddr=0000.1c.00.0 ns=1 [0000.1c.00.0-3]
> >>filename=trtype=PCIe traddr=0000.1c.00.0 ns=1 [0000.1c.00.0-4]
> >>filename=trtype=PCIe traddr=0000.1c.00.0 ns=1
> >>
> >>
> >>don(a)donst201:~/fio/single/rr$ sudo fio 1c.ini
> >>0000.1c.00.0-1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B,
> >>(T) 4096B-4096B, ioengine=spdk, iodepth=32
> >>0000.1c.00.0-2: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B,
> >>(T) 4096B-4096B, ioengine=spdk, iodepth=32
> >>0000.1c.00.0-3: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B,
> >>(T) 4096B-4096B, ioengine=spdk, iodepth=32
> >>0000.1c.00.0-4: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B,
> >>(T) 4096B-4096B, ioengine=spdk, iodepth=32
> >>fio-3.11
> >>Starting 4 threads
> >>Starting SPDK v18.10 / DPDK 18.08.0 initialization...
> >>[ DPDK EAL parameters: fio --no-shconf -c 0x1 -m 512
> >>--file-prefix=spdk_pid4270 ]
> >>EAL: Detected 36 lcore(s)
> >>EAL: Detected 2 NUMA nodes
> >>EAL: No free hugepages reported in hugepages-1048576kB
> >>EAL: Probing VFIO support...
> >>EAL: PCI device 0000:1c:00.0 on NUMA socket 0
> >>EAL: probe driver: 8086:a54 spdk_nvme
> >>Jobs: 1 (f=0): [_(3),f(1)][100.0%][eta 00m:00s]
> >>0000.1c.00.0-1: (groupid=0, jobs=4): err= 0: pid=4310: Mon Nov 5
> >>05:16:03 2018
> >> mixed: IOPS=12.5k, BW=48.0MiB/s (51.4MB/s)(490MiB/10010msec)
> >> slat (nsec): min=98, max=15092, avg=120.02, stdev=90.07
> >> clat (usec): min=55, max=19757, avg=10207.22, stdev=884.06
> >> lat (usec): min=60, max=19757, avg=10207.34, stdev=884.06
> >> clat percentiles (usec):
> >> | 1.00th=[ 8979], 5.00th=[ 9110], 10.00th=[ 9372], 20.00th=[
> >>9503],
> >> | 30.00th=[ 9765], 40.00th=[ 9896], 50.00th=[10159],
> >>60.00th=[10290],
> >> | 70.00th=[10552], 80.00th=[10814], 90.00th=[11207],
> >>95.00th=[11600],
> >> | 99.00th=[12911], 99.50th=[14353], 99.90th=[17695],
> >>99.95th=[18744],
> >> | 99.99th=[19006]
> >> bw ( KiB/s): min=12008, max=12976, per=25.00%, avg=12535.70,
> stdev=306.81, samples=80
> >> iops : min= 3002, max= 3244, avg=3133.85, stdev=76.77, samples=80
> >> lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
> >> lat (msec) : 2=0.01%, 4=0.02%, 10=45.13%, 20=54.83%
> >> cpu : usr=100.01%, sys=0.00%, ctx=27, majf=0, minf=0
> >> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.9%,
> >=64=0.0%
> >> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
> >=64=0.0%
> >> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%,
> >>>=64=0.0%
> >> issued rwts: total=125506,0,0,0 short=0,0,0,0 dropped=0,0,0,0
> >> latency : target=0, window=0, percentile=100.00%, depth=32
> >>Run status group 0 (all jobs):
> >> MIXED: bw=48.0MiB/s (51.4MB/s), 48.0MiB/s-48.0MiB/s
> >>(51.4MB/s-51.4MB/s), io=490MiB (514MB), run=10010-10010msec
> >>
> >>
> >>don(a)donst201:~/fio/single/rr$ cat nvme2n1.ini |grep -v '#'
> >>[global]
> >>rw=randrw
> >>rwmixread=100
> >>numjobs=1
> >>iodepth=1
> >>bs=4k
> >>direct=1
> >>thread=1
> >>time_based=1
> >>ramp_time=0
> >>runtime=10
> >>ioengine=libaio
> >>group_reporting=1
> >>unified_rw_reporting=1
> >>exitall=1
> >>randrepeat=0
> >>norandommap=1
> >>[nvme2n1]
> >>filename=/dev/nvme2n1
> >>
> >>
> >>don(a)donst201:~/fio/single/rr$ sudo fio nvme2n1.ini
> >>nvme2n1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T)
> >>4096B-4096B, ioengine=libaio, iodepth=1
> >>fio-3.11
> >>Starting 1 thread
> >>Jobs: 1 (f=1): [r(1)][100.0%][r=43.6MiB/s][r=11.2k IOPS][eta 00m:00s]
> >>nvme2n1: (groupid=0, jobs=1): err= 0: pid=2525: Mon Nov 5 05:22:57
> >>2018
> >> mixed: IOPS=11.2k, BW=43.6MiB/s (45.7MB/s)(436MiB/10001msec)
> >> slat (nsec): min=4848, max=208595, avg=5126.33, stdev=771.27
> >> clat (nsec): min=1845, max=3405.8k, avg=83004.99, stdev=33577.36
> >> lat (usec): min=15, max=3411, avg=88.26, stdev=33.59
> >> clat percentiles (usec):
> >> | 1.00th=[ 15], 5.00th=[ 58], 10.00th=[ 59], 20.00th=[ 59],
> >> | 30.00th=[ 60], 40.00th=[ 77], 50.00th=[ 77], 60.00th=[ 78],
> >> | 70.00th=[ 111], 80.00th=[ 113], 90.00th=[ 113], 95.00th=[
> >>115],
> >> | 99.00th=[ 186], 99.50th=[ 186], 99.90th=[ 190], 99.95th=[
> >>192],
> >> | 99.99th=[ 375]
> >> bw ( KiB/s): min=43816, max=45016, per=99.98%, avg=44639.58,
> stdev=312.88, samples=19
> >> iops : min=10954, max=11254, avg=11159.89, stdev=78.22,
> samples=19
> >> lat (usec) : 2=0.01%, 20=2.83%, 50=0.01%, 100=64.57%, 250=32.59%
> >> lat (usec) : 500=0.01%, 750=0.01%
> >> lat (msec) : 2=0.01%, 4=0.01%
> >> cpu : usr=4.02%, sys=10.33%, ctx=111630, majf=0, minf=12
> >> IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%,
> >=64=0.0%
> >> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
> >=64=0.0%
> >> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
> >>>=64=0.0%
> >> issued rwts: total=111631,0,0,0 short=0,0,0,0 dropped=0,0,0,0
> >> latency : target=0, window=0, percentile=100.00%, depth=1
> >>Run status group 0 (all jobs):
> >> MIXED: bw=43.6MiB/s (45.7MB/s), 43.6MiB/s-43.6MiB/s (45.7MB/s-
> 45.7MB/s), io=436MiB (457MB), run=10001-10001msec Disk stats
> (read/write):
> >> nvme2n1: ios=110275/0, merge=0/0, ticks=0/0, in_queue=8596,
> >>util=86.22%
> >>
> >>
> >>Thanks,
> >>Joe
> >>
> >>> -----Original Message-----
> >>> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Cao, Gang
> >>> Sent: Sunday, November 4, 2018 8:40 PM
> >>> To: Storage Performance Development Kit < spdk(a)lists.01.org >
> >>> Subject: Re: [SPDK] FIO NVMe Performance Results
> >>>
> >>> For the SPDK plugin, from the below result, looks like only 1 job
> >>> started with
> >>> 32 queue depth:
> >>>
> >>> Jobs: 1 (f=0): [f(1),_(3)][100.0%][eta 00m:00s] <<<<
> >>>
> >>> Try to update the FIO configuration file as
> >>>
> >>> [job1]
> >>> filename=trtype=PCIe traddr=0000.1c.00.0 ns=1 [job2]
> >>> filename=trtype=PCIe
> >>> traddr=0000.1c.00.0 ns=1 ....
> >>>
> >>> The kernel FIO is with 4 jobs and each job of 32 queue depth. For
> >>> SPDK, you can configure as the same, or just configure two jobs and
> >>> each job of 64 queue depth. The overall queue depth to the device is
> same of 128.
> >>>
> >>> Thanks,
> >>> Gang
> >>>
> >>> -----Original Message-----
> >>> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Gruher,
> >>> Joseph R
> >>> Sent: Monday, November 5, 2018 10:49 AM
> >>> To: Storage Performance Development Kit < spdk(a)lists.01.org >
> >>> Subject: [SPDK] FIO NVMe Performance Results
> >>>
> >>> Hi folks-
> >>>
> >>> I'm testing SPDK 18.10 on Ubuntu 18.04 with kernel 4.18.16. I used
> >>> FIO with the kernel NVMe driver to measure the performance of a
> >>> local (PCIe
> >>> attached) Intel P4500 NVMe device on a 4KB random read workload and
> >>> obtained 477K IOPS, roughly in line with the drive spec. Then I
> >>> tested the same drive with the SPDK FIO plugin and only achieved 13K
> >>> IOPS. The FIO test files and the results are pasted below. Any
> >>> ideas where I'm going wrong here?
> >>>
> >>> Thanks!
> >>>
> >>>
> >>> don(a)donst201:~/fio/single/rr$ ls
> >>> /sys/devices/pci0000:17/0000:17:02.0/0000:1c:00.0/nvme/nvme1/
> >>> address cntlid dev device firmware_rev model nvme2n1 power
> >>> rescan_controller reset_controller serial state subsysnqn
> >>> subsystem transport uevent
> >>>
> >>>
> >>> don(a)donst201:~/fio/single/rr$ cat nvme2n1.ini |grep -v '#'
> >>> [global]
> >>> rw=randrw
> >>> rwmixread=100
> >>> numjobs=4
> >>> iodepth=32
> >>> bs=4k
> >>> direct=1
> >>> thread=1
> >>> time_based=1
> >>> ramp_time=0
> >>> runtime=10
> >>> ioengine=libaio
> >>> group_reporting=1
> >>> unified_rw_reporting=1
> >>> exitall=1
> >>> randrepeat=0
> >>> norandommap=1
> >>> [nvme2n1]
> >>> filename=/dev/nvme2n1
> >>>
> >>>
> >>> don(a)donst201:~/fio/single/rr$ sudo fio nvme2n1.ini
> >>> nvme2n1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T)
> >>> 4096B-4096B, ioengine=libaio, iodepth=32 ...
> >>> fio-3.11
> >>> Starting 4 threads
> >>> Jobs: 4 (f=4): [r(4)][100.0%][r=1874MiB/s][r=480k IOPS][eta 00m:00s]
> >>> nvme2n1: (groupid=0, jobs=4): err= 0: pid=2575: Mon Nov 5 02:26:25
> 2018
> >>> mixed: IOPS=477k, BW=1862MiB/s (1952MB/s)(18.2GiB/10001msec)
> >>> slat (nsec): min=1295, max=333088, avg=2216.35, stdev=927.89
> >>> clat (nsec): min=506, max=5818.8k, avg=265924.16, stdev=223761.59
> >>> lat (usec): min=6, max=5821, avg=268.19, stdev=223.76
> >>> clat percentiles (usec):
> >>> | 1.00th=[ 13], 5.00th=[ 74], 10.00th=[ 87], 20.00th=[ 115],
> >>> | 30.00th=[ 139], 40.00th=[ 167], 50.00th=[ 204], 60.00th=[ 247],
> >>> | 70.00th=[ 306], 80.00th=[ 388], 90.00th=[ 523], 95.00th=[ 660],
> >>> | 99.00th=[ 988], 99.50th=[ 1156], 99.90th=[ 2343], 99.95th=[ 3195],
> >>> | 99.99th=[ 4424]
> >>> bw ( KiB/s): min=448192, max=483104, per=25.00%, avg=476597.10,
> >>> stdev=6748.78, samples=80
> >>> iops : min=112048, max=120776, avg=119149.27, stdev=1687.20,
> >>> samples=80
> >>> lat (nsec) : 750=0.01%
> >>> lat (usec) : 10=0.52%, 20=1.25%, 50=0.25%, 100=13.38%, 250=45.13%
> >>> lat (usec) : 500=28.23%, 750=8.08%, 1000=2.22%
> >>> lat (msec) : 2=0.82%, 4=0.10%, 10=0.02%
> >>> cpu : usr=14.08%, sys=36.02%, ctx=1930207, majf=0, minf=697
> >>> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%,
> >>> >=64=0.0%
> >>> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
> >>> >=64=0.0%
> >>> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%,
> >>> 64=0.0%,
> >>> >=64=0.0%
> >>> issued rwts: total=4766095,0,0,0 short=0,0,0,0 dropped=0,0,0,0
> >>> latency : target=0, window=0, percentile=100.00%, depth=32
> >>> Run status group 0 (all jobs):
> >>> MIXED: bw=1862MiB/s (1952MB/s), 1862MiB/s-1862MiB/s (1952MB/s-
> >>> 1952MB/s), io=18.2GiB (19.5GB), run=10001-10001msec Disk stats
> >>> (read/write):
> >>> nvme2n1: ios=4709190/0, merge=0/0, ticks=4096/0,
> in_queue=1388752,
> >>> util=100.00%
> >>>
> >>>
> >>> don(a)donst201:~/fio/single/rr$ sudo
> >>> /home/don/install/spdk/spdk/scripts/setup.sh
> >>> Active mountpoints on /dev/nvme0n1, so not binding PCI dev
> >>> 0000:03:00.0
> >>> 0000:1c:00.0 (8086 0a54): nvme -> uio_pci_generic
> >>> 0000:1d:00.0 (8086 0a54): nvme -> uio_pci_generic
> >>> 0000:5e:00.0 (8086 0a54): nvme -> uio_pci_generic
> >>> 0000:5f:00.0 (8086 0a54): nvme -> uio_pci_generic
> >>> 0000:62:00.0 (8086 0a54): nvme -> uio_pci_generic
> >>> 0000:63:00.0 (8086 0a54): nvme -> uio_pci_generic
> >>> 0000:64:00.0 (8086 0a54): nvme -> uio_pci_generic
> >>> 0000:65:00.0 (8086 0a54): nvme -> uio_pci_generic
> >>> 0000:da:00.0 (8086 0a54): nvme -> uio_pci_generic
> >>> 0000:db:00.0 (8086 0a54): nvme -> uio_pci_generic
> >>> 0000:dc:00.0 (8086 0a54): nvme -> uio_pci_generic
> >>> 0000:dd:00.0 (8086 0a54): nvme -> uio_pci_generic
> >>> 0000:e0:00.0 (8086 0a54): nvme -> uio_pci_generic
> >>> 0000:e1:00.0 (8086 0a54): nvme -> uio_pci_generic
> >>> 0000:e2:00.0 (8086 0a54): nvme -> uio_pci_generic
> >>> 0000:e3:00.0 (8086 0a54): nvme -> uio_pci_generic
> >>> 0000:00:04.0 (8086 2021): ioatdma -> uio_pci_generic
> >>> 0000:00:04.1 (8086 2021): ioatdma -> uio_pci_generic
> >>> 0000:00:04.2 (8086 2021): ioatdma -> uio_pci_generic
> >>> 0000:00:04.3 (8086 2021): ioatdma -> uio_pci_generic
> >>> 0000:00:04.4 (8086 2021): ioatdma -> uio_pci_generic
> >>> 0000:00:04.5 (8086 2021): ioatdma -> uio_pci_generic
> >>> 0000:00:04.6 (8086 2021): ioatdma -> uio_pci_generic
> >>> 0000:00:04.7 (8086 2021): ioatdma -> uio_pci_generic
> >>> 0000:80:04.0 (8086 2021): ioatdma -> uio_pci_generic
> >>> 0000:80:04.1 (8086 2021): ioatdma -> uio_pci_generic
> >>> 0000:80:04.2 (8086 2021): ioatdma -> uio_pci_generic
> >>> 0000:80:04.3 (8086 2021): ioatdma -> uio_pci_generic
> >>> 0000:80:04.4 (8086 2021): ioatdma -> uio_pci_generic
> >>> 0000:80:04.5 (8086 2021): ioatdma -> uio_pci_generic
> >>> 0000:80:04.6 (8086 2021): ioatdma -> uio_pci_generic
> >>> 0000:80:04.7 (8086 2021): ioatdma -> uio_pci_generic
> >>>
> >>>
> >>> don(a)donst201:~/fio/single/rr$ cat 1c.ini |grep -v '#'
> >>> [global]
> >>> rw=randrw
> >>> rwmixread=100
> >>> numjobs=4
> >>> iodepth=32
> >>> bs=4k
> >>> direct=1
> >>> thread=1
> >>> time_based=1
> >>> ramp_time=0
> >>> runtime=10
> >>>
> ioengine=/home/don/install/spdk/spdk/examples/nvme/fio_plugin/fio_pl
> >>> ug
> >>> in
> >>> group_reporting=1
> >>> unified_rw_reporting=1
> >>> exitall=1
> >>> randrepeat=0
> >>> norandommap=1
> >>> [0000.1c.00.0]
> >>> filename=trtype=PCIe traddr=0000.1c.00.0 ns=1
> >>>
> >>>
> >>> don(a)donst201:~/fio/single/rr$ sudo fio 1c.ini
> >>> 0000.1c.00.0: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B,
> >>> (T) 4096B-4096B, ioengine=spdk, iodepth=32 ...
> >>> fio-3.11
> >>> Starting 4 threads
> >>> Starting SPDK v18.10 / DPDK 18.08.0 initialization...
> >>> [ DPDK EAL parameters: fio --no-shconf -c 0x1 -m 512 --file-
> >>> prefix=spdk_pid3668 ]
> >>> EAL: Detected 36 lcore(s)
> >>> EAL: Detected 2 NUMA nodes
> >>> EAL: No free hugepages reported in hugepages-1048576kB
> >>> EAL: Probing VFIO support...
> >>> EAL: PCI device 0000:1c:00.0 on NUMA socket 0
> >>> EAL: probe driver: 8086:a54 spdk_nvme
> >>> Jobs: 1 (f=0): [f(1),_(3)][100.0%][eta 00m:00s]
> >>> 0000.1c.00.0: (groupid=0, jobs=4): err= 0: pid=3709: Mon Nov 5
> >>> 02:28:29
> >>> 2018
> >>> mixed: IOPS=13.2k, BW=51.6MiB/s (54.1MB/s)(517MiB/10011msec)
> >>> slat (nsec): min=109, max=15344, avg=127.91, stdev=73.44
> >>> clat (usec): min=187, max=18715, avg=9683.80, stdev=650.54
> >>> lat (usec): min=193, max=18715, avg=9683.93, stdev=650.53
> >>> clat percentiles (usec):
> >>> | 1.00th=[ 8717], 5.00th=[ 8848], 10.00th=[ 8979], 20.00th=[ 9110],
> >>> | 30.00th=[ 9241], 40.00th=[ 9372], 50.00th=[ 9503], 60.00th=[ 9765],
> >>> | 70.00th=[10028], 80.00th=[10159], 90.00th=[10552],
> 95.00th=[10945],
> >>> | 99.00th=[11338], 99.50th=[11338], 99.90th=[11731],
> 99.95th=[13960],
> >>> | 99.99th=[17695]
> >>> bw ( KiB/s): min=13080, max=13304, per=25.00%, avg=13214.70,
> >>> stdev=55.30, samples=80
> >>> iops : min= 3270, max= 3326, avg=3303.65, stdev=13.84,
> samples=80
> >>> lat (usec) : 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
> >>> lat (msec) : 2=0.01%, 4=0.02%, 10=70.85%, 20=29.11%
> >>> cpu : usr=100.01%, sys=0.00%, ctx=23, majf=0, minf=0
> >>> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.9%,
> >>> >=64=0.0%
> >>> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
> >>> >=64=0.0%
> >>> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%,
> >>> 64=0.0%,
> >>> >=64=0.0%
> >>> issued rwts: total=132300,0,0,0 short=0,0,0,0 dropped=0,0,0,0
> >>> latency : target=0, window=0, percentile=100.00%, depth=32
> >>> Run status group 0 (all jobs):
> >>> MIXED: bw=51.6MiB/s (54.1MB/s), 51.6MiB/s-51.6MiB/s (54.1MB/s-
> >>> 54.1MB/s), io=517MiB (542MB), run=10011-10011msec
> >>>
> >>> _______________________________________________
> >>> SPDK mailing list
> >>> SPDK(a)lists.01.org
> >>> https://lists.01.org/mailman/listinfo/spdk
> >>> _______________________________________________
> >>> SPDK mailing list
> >>> SPDK(a)lists.01.org
> >>> https://lists.01.org/mailman/listinfo/spdk
> >>_______________________________________________
> >>SPDK mailing list
> >> SPDK(a)lists.01.org
> >> https://lists.01.org/mailman/listinfo/spdk
> >>_______________________________________________
> >>SPDK mailing list
> >> SPDK(a)lists.01.org
> >> https://lists.01.org/mailman/listinfo/spdk
> >
> >
> >--
> >Маров Алексей
> >_______________________________________________
> >SPDK mailing list
> >SPDK(a)lists.01.org
> >https://lists.01.org/mailman/listinfo/spdk
> >_______________________________________________
> >SPDK mailing list
> >SPDK(a)lists.01.org
> >https://lists.01.org/mailman/listinfo/spdk
>
>
> --
> Маров Алексей
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [SPDK] FIO NVMe Performance Results
@ 2018-11-05 8:45
0 siblings, 0 replies; 25+ messages in thread
From: @ 2018-11-05 8:45 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 22647 bytes --]
Ok, Do you see some performance issues using general bdev fio plugin with you configuration ?
Best Regards
Alex
>Понедельник, 5 ноября 2018, 11:39 +03:00 от "Cao, Gang" <gang.cao(a)intel.com>:
>
>Yes. This is for the bdev version of SPDK FIO plugin and you can assign the conf file as the parameter in the FIO configuration.
>
>Previously, it's another SPDK FIO plugin for the user space NVMe driver only.
>
>These are two different FIO plugins.
>
>Thanks,
>Gang
>
>-----Original Message-----
>From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Маров Алексей
>Sent: Monday, November 5, 2018 4:32 PM
>To: Storage Performance Development Kit < spdk(a)lists.01.org >
>Subject: Re: [SPDK] FIO NVMe Performance Results
>
>Sorry for incomplete message I have sent before
>
>Hello Gang.
>
>I just curious, why would you run fio with SPDK using this conf file, with is a bit different from what is suggested in here https://github.com/spdk/spdk/tree/master/examples/bdev/fio_plugin .
>
>I mean may be you should create conf file for spdk like this one :
>
>nvme.conf :
>[Nvme]
> TransportId "trtype:PCIe traddr:0000.1c.00.0" Nvme0
>
>and provide it in fio config.
>
>fio.ini :
>[global]
>ioengine=spdk_bdev
>spdk_conf=....../nvme.conf
>rw=randrw
>rwmixread=100
>numjobs=4
>iodepth=32
>bs=4k
>direct=1
>thread=1
>time_based=1
>ramp_time=0
>runtime=10
>group_reporting=1
>unified_rw_reporting=1
>exitall=1
>randrepeat=0
>norandommap=1
>
>[0000:06:00.0]
>filename=Nvme0n1
>
>and run like this
>
># LD_PRELOAD=examples/bdev/fio_plugin/fio_plugin fio examples/bdev/fio_plugin/fio.ini
>
>Just to note. The recommended version of fio for SPDK is 3.3
>
>Best Regards
>Alex
>
>
>>Понедельник, 5 ноября 2018, 9:53 +03:00 от "Cao, Gang" < gang.cao(a)intel.com >:
>>
>>Thanks. This is interesting especially for your finding on queue depth 1 case. We are trying to reproduce this issue locally same as your configuration.
>>
>>Thanks,
>>Gang
>>
>>-----Original Message-----
>>From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Gruher, Joseph R
>>Sent: Monday, November 5, 2018 1:32 PM
>>To: Storage Performance Development Kit < spdk(a)lists.01.org >
>>Subject: Re: [SPDK] FIO NVMe Performance Results
>>
>>Interesting catch. It seems like FIO is actually not accurately reporting the number of jobs in the report out for the SPDK case. If I disable the group_reporting parameter I do get a printout for four separate jobs in the SPDK case, so I think it is really running four jobs there, and just summarizing incorrectly. I also see four separate CPU cores go to 100% when I run that FIO test case, indicating four jobs are active.
>>
>>Also, I don't think would explain the difference in performance. Using an aggregate queue depth of 32 versus 128 shouldn't lead to a drop in performance from 377K to 13K.
>>
>>I retested SPDK with multiple separate job descriptions as you suggested anyway, using four separate jobs instead of relying on the numjobs=4 parameter, and the results are below. In that SPDK test with four separate FIO job statements, each with queue depth 32, I still only achieved 12.5K IOPS.
>>
>>I then re-ran the kernel NVMe driver test case with only a single job for comparison, and even with only 1 job and queue depth 32 it still achieves 254K IOPS.
>>
>>We might be on the right track with a problem around queue depth however. If I run the kernel NVMe driver case with just 1 job and queue depth of only 1, I then get 11.2K IOPS, which is around the performance I get in the SPDK test case. Perhaps the SPDK case is not actually queuing IOs on the device for some reason?
>>
>>
>>don(a)donst201:~/fio/single/rr$ cat nvme2n1.ini |grep -v '#'
>>[global]
>>rw=randrw
>>rwmixread=100
>>numjobs=1
>>iodepth=32
>>bs=4k
>>direct=1
>>thread=1
>>time_based=1
>>ramp_time=0
>>runtime=10
>>ioengine=libaio
>>group_reporting=1
>>unified_rw_reporting=1
>>exitall=1
>>randrepeat=0
>>norandommap=1
>>[nvme2n1]
>>filename=/dev/nvme2n1
>>
>>
>>don(a)donst201:~/fio/single/rr$ sudo fio nvme2n1.ini
>>nvme2n1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32
>>fio-3.11
>>Starting 1 thread
>>Jobs: 1 (f=1): [r(1)][100.0%][r=1025MiB/s][r=262k IOPS][eta 00m:00s]
>>nvme2n1: (groupid=0, jobs=1): err= 0: pid=2422: Mon Nov 5 05:19:13 2018
>> mixed: IOPS=254k, BW=992MiB/s (1040MB/s)(9922MiB/10001msec)
>> slat (nsec): min=1185, max=319052, avg=1611.64, stdev=808.40
>> clat (usec): min=2, max=4743, avg=124.02, stdev=121.00
>> lat (usec): min=6, max=4744, avg=125.68, stdev=121.02
>> clat percentiles (usec):
>> | 1.00th=[ 9], 5.00th=[ 56], 10.00th=[ 61], 20.00th=[ 73],
>> | 30.00th=[ 80], 40.00th=[ 91], 50.00th=[ 110], 60.00th=[ 118],
>> | 70.00th=[ 130], 80.00th=[ 157], 90.00th=[ 202], 95.00th=[ 253],
>> | 99.00th=[ 408], 99.50th=[ 498], 99.90th=[ 1795], 99.95th=[ 3064],
>> | 99.99th=[ 3785]
>> bw ( KiB/s): min=612240, max=1078792, per=100.00%, avg=1015996.40, stdev=106032.96, samples=20
>> iops : min=153060, max=269698, avg=253998.90, stdev=26508.40, samples=20
>> lat (usec) : 4=0.01%, 10=1.47%, 20=1.19%, 50=0.17%, 100=42.03%
>> lat (usec) : 250=49.95%, 500=4.70%, 750=0.28%, 1000=0.06%
>> lat (msec) : 2=0.06%, 4=0.09%, 10=0.01%
>> cpu : usr=21.45%, sys=51.11%, ctx=530348, majf=0, minf=139
>> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
>> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
>> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
>> issued rwts: total=2540023,0,0,0 short=0,0,0,0 dropped=0,0,0,0
>> latency : target=0, window=0, percentile=100.00%, depth=32
>>Run status group 0 (all jobs):
>> MIXED: bw=992MiB/s (1040MB/s), 992MiB/s-992MiB/s (1040MB/s-1040MB/s), io=9922MiB (10.4GB), run=10001-10001msec Disk stats (read/write):
>> nvme2n1: ios=2514356/0, merge=0/0, ticks=400/0, in_queue=297612, util=99.06%
>>
>>
>>don(a)donst201:~/fio/single/rr$ cat 1c.ini |grep -v '#'
>>[global]
>>rw=randrw
>>rwmixread=100
>>numjobs=1
>>iodepth=32
>>bs=4k
>>direct=1
>>thread=1
>>time_based=1
>>ramp_time=0
>>runtime=10
>>ioengine=/home/don/install/spdk/spdk/examples/nvme/fio_plugin/fio_plugin
>>group_reporting=1
>>unified_rw_reporting=1
>>exitall=1
>>randrepeat=0
>>norandommap=1
>>[0000.1c.00.0-1]
>>filename=trtype=PCIe traddr=0000.1c.00.0 ns=1 [0000.1c.00.0-2] filename=trtype=PCIe traddr=0000.1c.00.0 ns=1 [0000.1c.00.0-3] filename=trtype=PCIe traddr=0000.1c.00.0 ns=1 [0000.1c.00.0-4] filename=trtype=PCIe traddr=0000.1c.00.0 ns=1
>>
>>
>>don(a)donst201:~/fio/single/rr$ sudo fio 1c.ini
>>0000.1c.00.0-1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk, iodepth=32
>>0000.1c.00.0-2: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk, iodepth=32
>>0000.1c.00.0-3: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk, iodepth=32
>>0000.1c.00.0-4: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk, iodepth=32
>>fio-3.11
>>Starting 4 threads
>>Starting SPDK v18.10 / DPDK 18.08.0 initialization...
>>[ DPDK EAL parameters: fio --no-shconf -c 0x1 -m 512 --file-prefix=spdk_pid4270 ]
>>EAL: Detected 36 lcore(s)
>>EAL: Detected 2 NUMA nodes
>>EAL: No free hugepages reported in hugepages-1048576kB
>>EAL: Probing VFIO support...
>>EAL: PCI device 0000:1c:00.0 on NUMA socket 0
>>EAL: probe driver: 8086:a54 spdk_nvme
>>Jobs: 1 (f=0): [_(3),f(1)][100.0%][eta 00m:00s]
>>0000.1c.00.0-1: (groupid=0, jobs=4): err= 0: pid=4310: Mon Nov 5 05:16:03 2018
>> mixed: IOPS=12.5k, BW=48.0MiB/s (51.4MB/s)(490MiB/10010msec)
>> slat (nsec): min=98, max=15092, avg=120.02, stdev=90.07
>> clat (usec): min=55, max=19757, avg=10207.22, stdev=884.06
>> lat (usec): min=60, max=19757, avg=10207.34, stdev=884.06
>> clat percentiles (usec):
>> | 1.00th=[ 8979], 5.00th=[ 9110], 10.00th=[ 9372], 20.00th=[ 9503],
>> | 30.00th=[ 9765], 40.00th=[ 9896], 50.00th=[10159], 60.00th=[10290],
>> | 70.00th=[10552], 80.00th=[10814], 90.00th=[11207], 95.00th=[11600],
>> | 99.00th=[12911], 99.50th=[14353], 99.90th=[17695], 99.95th=[18744],
>> | 99.99th=[19006]
>> bw ( KiB/s): min=12008, max=12976, per=25.00%, avg=12535.70, stdev=306.81, samples=80
>> iops : min= 3002, max= 3244, avg=3133.85, stdev=76.77, samples=80
>> lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
>> lat (msec) : 2=0.01%, 4=0.02%, 10=45.13%, 20=54.83%
>> cpu : usr=100.01%, sys=0.00%, ctx=27, majf=0, minf=0
>> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.9%, >=64=0.0%
>> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
>> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
>> issued rwts: total=125506,0,0,0 short=0,0,0,0 dropped=0,0,0,0
>> latency : target=0, window=0, percentile=100.00%, depth=32
>>Run status group 0 (all jobs):
>> MIXED: bw=48.0MiB/s (51.4MB/s), 48.0MiB/s-48.0MiB/s (51.4MB/s-51.4MB/s), io=490MiB (514MB), run=10010-10010msec
>>
>>
>>don(a)donst201:~/fio/single/rr$ cat nvme2n1.ini |grep -v '#'
>>[global]
>>rw=randrw
>>rwmixread=100
>>numjobs=1
>>iodepth=1
>>bs=4k
>>direct=1
>>thread=1
>>time_based=1
>>ramp_time=0
>>runtime=10
>>ioengine=libaio
>>group_reporting=1
>>unified_rw_reporting=1
>>exitall=1
>>randrepeat=0
>>norandommap=1
>>[nvme2n1]
>>filename=/dev/nvme2n1
>>
>>
>>don(a)donst201:~/fio/single/rr$ sudo fio nvme2n1.ini
>>nvme2n1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
>>fio-3.11
>>Starting 1 thread
>>Jobs: 1 (f=1): [r(1)][100.0%][r=43.6MiB/s][r=11.2k IOPS][eta 00m:00s]
>>nvme2n1: (groupid=0, jobs=1): err= 0: pid=2525: Mon Nov 5 05:22:57 2018
>> mixed: IOPS=11.2k, BW=43.6MiB/s (45.7MB/s)(436MiB/10001msec)
>> slat (nsec): min=4848, max=208595, avg=5126.33, stdev=771.27
>> clat (nsec): min=1845, max=3405.8k, avg=83004.99, stdev=33577.36
>> lat (usec): min=15, max=3411, avg=88.26, stdev=33.59
>> clat percentiles (usec):
>> | 1.00th=[ 15], 5.00th=[ 58], 10.00th=[ 59], 20.00th=[ 59],
>> | 30.00th=[ 60], 40.00th=[ 77], 50.00th=[ 77], 60.00th=[ 78],
>> | 70.00th=[ 111], 80.00th=[ 113], 90.00th=[ 113], 95.00th=[ 115],
>> | 99.00th=[ 186], 99.50th=[ 186], 99.90th=[ 190], 99.95th=[ 192],
>> | 99.99th=[ 375]
>> bw ( KiB/s): min=43816, max=45016, per=99.98%, avg=44639.58, stdev=312.88, samples=19
>> iops : min=10954, max=11254, avg=11159.89, stdev=78.22, samples=19
>> lat (usec) : 2=0.01%, 20=2.83%, 50=0.01%, 100=64.57%, 250=32.59%
>> lat (usec) : 500=0.01%, 750=0.01%
>> lat (msec) : 2=0.01%, 4=0.01%
>> cpu : usr=4.02%, sys=10.33%, ctx=111630, majf=0, minf=12
>> IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
>> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
>> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
>> issued rwts: total=111631,0,0,0 short=0,0,0,0 dropped=0,0,0,0
>> latency : target=0, window=0, percentile=100.00%, depth=1
>>Run status group 0 (all jobs):
>> MIXED: bw=43.6MiB/s (45.7MB/s), 43.6MiB/s-43.6MiB/s (45.7MB/s-45.7MB/s), io=436MiB (457MB), run=10001-10001msec Disk stats (read/write):
>> nvme2n1: ios=110275/0, merge=0/0, ticks=0/0, in_queue=8596, util=86.22%
>>
>>
>>Thanks,
>>Joe
>>
>>> -----Original Message-----
>>> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Cao, Gang
>>> Sent: Sunday, November 4, 2018 8:40 PM
>>> To: Storage Performance Development Kit < spdk(a)lists.01.org >
>>> Subject: Re: [SPDK] FIO NVMe Performance Results
>>>
>>> For the SPDK plugin, from the below result, looks like only 1 job
>>> started with
>>> 32 queue depth:
>>>
>>> Jobs: 1 (f=0): [f(1),_(3)][100.0%][eta 00m:00s] <<<<
>>>
>>> Try to update the FIO configuration file as
>>>
>>> [job1]
>>> filename=trtype=PCIe traddr=0000.1c.00.0 ns=1 [job2]
>>> filename=trtype=PCIe
>>> traddr=0000.1c.00.0 ns=1 ....
>>>
>>> The kernel FIO is with 4 jobs and each job of 32 queue depth. For
>>> SPDK, you can configure as the same, or just configure two jobs and
>>> each job of 64 queue depth. The overall queue depth to the device is same of 128.
>>>
>>> Thanks,
>>> Gang
>>>
>>> -----Original Message-----
>>> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Gruher,
>>> Joseph R
>>> Sent: Monday, November 5, 2018 10:49 AM
>>> To: Storage Performance Development Kit < spdk(a)lists.01.org >
>>> Subject: [SPDK] FIO NVMe Performance Results
>>>
>>> Hi folks-
>>>
>>> I'm testing SPDK 18.10 on Ubuntu 18.04 with kernel 4.18.16. I used
>>> FIO with the kernel NVMe driver to measure the performance of a local
>>> (PCIe
>>> attached) Intel P4500 NVMe device on a 4KB random read workload and
>>> obtained 477K IOPS, roughly in line with the drive spec. Then I
>>> tested the same drive with the SPDK FIO plugin and only achieved 13K
>>> IOPS. The FIO test files and the results are pasted below. Any ideas
>>> where I'm going wrong here?
>>>
>>> Thanks!
>>>
>>>
>>> don(a)donst201:~/fio/single/rr$ ls
>>> /sys/devices/pci0000:17/0000:17:02.0/0000:1c:00.0/nvme/nvme1/
>>> address cntlid dev device firmware_rev model nvme2n1 power
>>> rescan_controller reset_controller serial state subsysnqn
>>> subsystem transport uevent
>>>
>>>
>>> don(a)donst201:~/fio/single/rr$ cat nvme2n1.ini |grep -v '#'
>>> [global]
>>> rw=randrw
>>> rwmixread=100
>>> numjobs=4
>>> iodepth=32
>>> bs=4k
>>> direct=1
>>> thread=1
>>> time_based=1
>>> ramp_time=0
>>> runtime=10
>>> ioengine=libaio
>>> group_reporting=1
>>> unified_rw_reporting=1
>>> exitall=1
>>> randrepeat=0
>>> norandommap=1
>>> [nvme2n1]
>>> filename=/dev/nvme2n1
>>>
>>>
>>> don(a)donst201:~/fio/single/rr$ sudo fio nvme2n1.ini
>>> nvme2n1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T)
>>> 4096B-4096B, ioengine=libaio, iodepth=32 ...
>>> fio-3.11
>>> Starting 4 threads
>>> Jobs: 4 (f=4): [r(4)][100.0%][r=1874MiB/s][r=480k IOPS][eta 00m:00s]
>>> nvme2n1: (groupid=0, jobs=4): err= 0: pid=2575: Mon Nov 5 02:26:25 2018
>>> mixed: IOPS=477k, BW=1862MiB/s (1952MB/s)(18.2GiB/10001msec)
>>> slat (nsec): min=1295, max=333088, avg=2216.35, stdev=927.89
>>> clat (nsec): min=506, max=5818.8k, avg=265924.16, stdev=223761.59
>>> lat (usec): min=6, max=5821, avg=268.19, stdev=223.76
>>> clat percentiles (usec):
>>> | 1.00th=[ 13], 5.00th=[ 74], 10.00th=[ 87], 20.00th=[ 115],
>>> | 30.00th=[ 139], 40.00th=[ 167], 50.00th=[ 204], 60.00th=[ 247],
>>> | 70.00th=[ 306], 80.00th=[ 388], 90.00th=[ 523], 95.00th=[ 660],
>>> | 99.00th=[ 988], 99.50th=[ 1156], 99.90th=[ 2343], 99.95th=[ 3195],
>>> | 99.99th=[ 4424]
>>> bw ( KiB/s): min=448192, max=483104, per=25.00%, avg=476597.10,
>>> stdev=6748.78, samples=80
>>> iops : min=112048, max=120776, avg=119149.27, stdev=1687.20,
>>> samples=80
>>> lat (nsec) : 750=0.01%
>>> lat (usec) : 10=0.52%, 20=1.25%, 50=0.25%, 100=13.38%, 250=45.13%
>>> lat (usec) : 500=28.23%, 750=8.08%, 1000=2.22%
>>> lat (msec) : 2=0.82%, 4=0.10%, 10=0.02%
>>> cpu : usr=14.08%, sys=36.02%, ctx=1930207, majf=0, minf=697
>>> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%,
>>> >=64=0.0%
>>> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>>> >=64=0.0%
>>> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%,
>>> >=64=0.0%
>>> issued rwts: total=4766095,0,0,0 short=0,0,0,0 dropped=0,0,0,0
>>> latency : target=0, window=0, percentile=100.00%, depth=32
>>> Run status group 0 (all jobs):
>>> MIXED: bw=1862MiB/s (1952MB/s), 1862MiB/s-1862MiB/s (1952MB/s-
>>> 1952MB/s), io=18.2GiB (19.5GB), run=10001-10001msec Disk stats
>>> (read/write):
>>> nvme2n1: ios=4709190/0, merge=0/0, ticks=4096/0, in_queue=1388752,
>>> util=100.00%
>>>
>>>
>>> don(a)donst201:~/fio/single/rr$ sudo
>>> /home/don/install/spdk/spdk/scripts/setup.sh
>>> Active mountpoints on /dev/nvme0n1, so not binding PCI dev
>>> 0000:03:00.0
>>> 0000:1c:00.0 (8086 0a54): nvme -> uio_pci_generic
>>> 0000:1d:00.0 (8086 0a54): nvme -> uio_pci_generic
>>> 0000:5e:00.0 (8086 0a54): nvme -> uio_pci_generic
>>> 0000:5f:00.0 (8086 0a54): nvme -> uio_pci_generic
>>> 0000:62:00.0 (8086 0a54): nvme -> uio_pci_generic
>>> 0000:63:00.0 (8086 0a54): nvme -> uio_pci_generic
>>> 0000:64:00.0 (8086 0a54): nvme -> uio_pci_generic
>>> 0000:65:00.0 (8086 0a54): nvme -> uio_pci_generic
>>> 0000:da:00.0 (8086 0a54): nvme -> uio_pci_generic
>>> 0000:db:00.0 (8086 0a54): nvme -> uio_pci_generic
>>> 0000:dc:00.0 (8086 0a54): nvme -> uio_pci_generic
>>> 0000:dd:00.0 (8086 0a54): nvme -> uio_pci_generic
>>> 0000:e0:00.0 (8086 0a54): nvme -> uio_pci_generic
>>> 0000:e1:00.0 (8086 0a54): nvme -> uio_pci_generic
>>> 0000:e2:00.0 (8086 0a54): nvme -> uio_pci_generic
>>> 0000:e3:00.0 (8086 0a54): nvme -> uio_pci_generic
>>> 0000:00:04.0 (8086 2021): ioatdma -> uio_pci_generic
>>> 0000:00:04.1 (8086 2021): ioatdma -> uio_pci_generic
>>> 0000:00:04.2 (8086 2021): ioatdma -> uio_pci_generic
>>> 0000:00:04.3 (8086 2021): ioatdma -> uio_pci_generic
>>> 0000:00:04.4 (8086 2021): ioatdma -> uio_pci_generic
>>> 0000:00:04.5 (8086 2021): ioatdma -> uio_pci_generic
>>> 0000:00:04.6 (8086 2021): ioatdma -> uio_pci_generic
>>> 0000:00:04.7 (8086 2021): ioatdma -> uio_pci_generic
>>> 0000:80:04.0 (8086 2021): ioatdma -> uio_pci_generic
>>> 0000:80:04.1 (8086 2021): ioatdma -> uio_pci_generic
>>> 0000:80:04.2 (8086 2021): ioatdma -> uio_pci_generic
>>> 0000:80:04.3 (8086 2021): ioatdma -> uio_pci_generic
>>> 0000:80:04.4 (8086 2021): ioatdma -> uio_pci_generic
>>> 0000:80:04.5 (8086 2021): ioatdma -> uio_pci_generic
>>> 0000:80:04.6 (8086 2021): ioatdma -> uio_pci_generic
>>> 0000:80:04.7 (8086 2021): ioatdma -> uio_pci_generic
>>>
>>>
>>> don(a)donst201:~/fio/single/rr$ cat 1c.ini |grep -v '#'
>>> [global]
>>> rw=randrw
>>> rwmixread=100
>>> numjobs=4
>>> iodepth=32
>>> bs=4k
>>> direct=1
>>> thread=1
>>> time_based=1
>>> ramp_time=0
>>> runtime=10
>>> ioengine=/home/don/install/spdk/spdk/examples/nvme/fio_plugin/fio_plug
>>> in
>>> group_reporting=1
>>> unified_rw_reporting=1
>>> exitall=1
>>> randrepeat=0
>>> norandommap=1
>>> [0000.1c.00.0]
>>> filename=trtype=PCIe traddr=0000.1c.00.0 ns=1
>>>
>>>
>>> don(a)donst201:~/fio/single/rr$ sudo fio 1c.ini
>>> 0000.1c.00.0: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B,
>>> (T) 4096B-4096B, ioengine=spdk, iodepth=32 ...
>>> fio-3.11
>>> Starting 4 threads
>>> Starting SPDK v18.10 / DPDK 18.08.0 initialization...
>>> [ DPDK EAL parameters: fio --no-shconf -c 0x1 -m 512 --file-
>>> prefix=spdk_pid3668 ]
>>> EAL: Detected 36 lcore(s)
>>> EAL: Detected 2 NUMA nodes
>>> EAL: No free hugepages reported in hugepages-1048576kB
>>> EAL: Probing VFIO support...
>>> EAL: PCI device 0000:1c:00.0 on NUMA socket 0
>>> EAL: probe driver: 8086:a54 spdk_nvme
>>> Jobs: 1 (f=0): [f(1),_(3)][100.0%][eta 00m:00s]
>>> 0000.1c.00.0: (groupid=0, jobs=4): err= 0: pid=3709: Mon Nov 5
>>> 02:28:29
>>> 2018
>>> mixed: IOPS=13.2k, BW=51.6MiB/s (54.1MB/s)(517MiB/10011msec)
>>> slat (nsec): min=109, max=15344, avg=127.91, stdev=73.44
>>> clat (usec): min=187, max=18715, avg=9683.80, stdev=650.54
>>> lat (usec): min=193, max=18715, avg=9683.93, stdev=650.53
>>> clat percentiles (usec):
>>> | 1.00th=[ 8717], 5.00th=[ 8848], 10.00th=[ 8979], 20.00th=[ 9110],
>>> | 30.00th=[ 9241], 40.00th=[ 9372], 50.00th=[ 9503], 60.00th=[ 9765],
>>> | 70.00th=[10028], 80.00th=[10159], 90.00th=[10552], 95.00th=[10945],
>>> | 99.00th=[11338], 99.50th=[11338], 99.90th=[11731], 99.95th=[13960],
>>> | 99.99th=[17695]
>>> bw ( KiB/s): min=13080, max=13304, per=25.00%, avg=13214.70,
>>> stdev=55.30, samples=80
>>> iops : min= 3270, max= 3326, avg=3303.65, stdev=13.84, samples=80
>>> lat (usec) : 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
>>> lat (msec) : 2=0.01%, 4=0.02%, 10=70.85%, 20=29.11%
>>> cpu : usr=100.01%, sys=0.00%, ctx=23, majf=0, minf=0
>>> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.9%,
>>> >=64=0.0%
>>> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>>> >=64=0.0%
>>> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%,
>>> >=64=0.0%
>>> issued rwts: total=132300,0,0,0 short=0,0,0,0 dropped=0,0,0,0
>>> latency : target=0, window=0, percentile=100.00%, depth=32
>>> Run status group 0 (all jobs):
>>> MIXED: bw=51.6MiB/s (54.1MB/s), 51.6MiB/s-51.6MiB/s (54.1MB/s-
>>> 54.1MB/s), io=517MiB (542MB), run=10011-10011msec
>>>
>>> _______________________________________________
>>> SPDK mailing list
>>> SPDK(a)lists.01.org
>>> https://lists.01.org/mailman/listinfo/spdk
>>> _______________________________________________
>>> SPDK mailing list
>>> SPDK(a)lists.01.org
>>> https://lists.01.org/mailman/listinfo/spdk
>>_______________________________________________
>>SPDK mailing list
>> SPDK(a)lists.01.org
>> https://lists.01.org/mailman/listinfo/spdk
>>_______________________________________________
>>SPDK mailing list
>> SPDK(a)lists.01.org
>> https://lists.01.org/mailman/listinfo/spdk
>
>
>--
>Маров Алексей
>_______________________________________________
>SPDK mailing list
>SPDK(a)lists.01.org
>https://lists.01.org/mailman/listinfo/spdk
>_______________________________________________
>SPDK mailing list
>SPDK(a)lists.01.org
>https://lists.01.org/mailman/listinfo/spdk
--
Маров Алексей
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [SPDK] FIO NVMe Performance Results
@ 2018-11-05 8:39 Cao, Gang
0 siblings, 0 replies; 25+ messages in thread
From: Cao, Gang @ 2018-11-05 8:39 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 21725 bytes --]
Yes. This is for the bdev version of SPDK FIO plugin and you can assign the conf file as the parameter in the FIO configuration.
Previously, it's another SPDK FIO plugin for the user space NVMe driver only.
These are two different FIO plugins.
Thanks,
Gang
-----Original Message-----
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Маров Алексей
Sent: Monday, November 5, 2018 4:32 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] FIO NVMe Performance Results
Sorry for incomplete message I have sent before
Hello Gang.
I just curious, why would you run fio with SPDK using this conf file, with is a bit different from what is suggested in here https://github.com/spdk/spdk/tree/master/examples/bdev/fio_plugin .
I mean may be you should create conf file for spdk like this one :
nvme.conf :
[Nvme]
TransportId "trtype:PCIe traddr:0000.1c.00.0" Nvme0
and provide it in fio config.
fio.ini :
[global]
ioengine=spdk_bdev
spdk_conf=....../nvme.conf
rw=randrw
rwmixread=100
numjobs=4
iodepth=32
bs=4k
direct=1
thread=1
time_based=1
ramp_time=0
runtime=10
group_reporting=1
unified_rw_reporting=1
exitall=1
randrepeat=0
norandommap=1
[0000:06:00.0]
filename=Nvme0n1
and run like this
# LD_PRELOAD=examples/bdev/fio_plugin/fio_plugin fio examples/bdev/fio_plugin/fio.ini
Just to note. The recommended version of fio for SPDK is 3.3
Best Regards
Alex
>Понедельник, 5 ноября 2018, 9:53 +03:00 от "Cao, Gang" <gang.cao(a)intel.com>:
>
>Thanks. This is interesting especially for your finding on queue depth 1 case. We are trying to reproduce this issue locally same as your configuration.
>
>Thanks,
>Gang
>
>-----Original Message-----
>From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Gruher, Joseph R
>Sent: Monday, November 5, 2018 1:32 PM
>To: Storage Performance Development Kit < spdk(a)lists.01.org >
>Subject: Re: [SPDK] FIO NVMe Performance Results
>
>Interesting catch. It seems like FIO is actually not accurately reporting the number of jobs in the report out for the SPDK case. If I disable the group_reporting parameter I do get a printout for four separate jobs in the SPDK case, so I think it is really running four jobs there, and just summarizing incorrectly. I also see four separate CPU cores go to 100% when I run that FIO test case, indicating four jobs are active.
>
>Also, I don't think would explain the difference in performance. Using an aggregate queue depth of 32 versus 128 shouldn't lead to a drop in performance from 377K to 13K.
>
>I retested SPDK with multiple separate job descriptions as you suggested anyway, using four separate jobs instead of relying on the numjobs=4 parameter, and the results are below. In that SPDK test with four separate FIO job statements, each with queue depth 32, I still only achieved 12.5K IOPS.
>
>I then re-ran the kernel NVMe driver test case with only a single job for comparison, and even with only 1 job and queue depth 32 it still achieves 254K IOPS.
>
>We might be on the right track with a problem around queue depth however. If I run the kernel NVMe driver case with just 1 job and queue depth of only 1, I then get 11.2K IOPS, which is around the performance I get in the SPDK test case. Perhaps the SPDK case is not actually queuing IOs on the device for some reason?
>
>
>don(a)donst201:~/fio/single/rr$ cat nvme2n1.ini |grep -v '#'
>[global]
>rw=randrw
>rwmixread=100
>numjobs=1
>iodepth=32
>bs=4k
>direct=1
>thread=1
>time_based=1
>ramp_time=0
>runtime=10
>ioengine=libaio
>group_reporting=1
>unified_rw_reporting=1
>exitall=1
>randrepeat=0
>norandommap=1
>[nvme2n1]
>filename=/dev/nvme2n1
>
>
>don(a)donst201:~/fio/single/rr$ sudo fio nvme2n1.ini
>nvme2n1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32
>fio-3.11
>Starting 1 thread
>Jobs: 1 (f=1): [r(1)][100.0%][r=1025MiB/s][r=262k IOPS][eta 00m:00s]
>nvme2n1: (groupid=0, jobs=1): err= 0: pid=2422: Mon Nov 5 05:19:13 2018
> mixed: IOPS=254k, BW=992MiB/s (1040MB/s)(9922MiB/10001msec)
> slat (nsec): min=1185, max=319052, avg=1611.64, stdev=808.40
> clat (usec): min=2, max=4743, avg=124.02, stdev=121.00
> lat (usec): min=6, max=4744, avg=125.68, stdev=121.02
> clat percentiles (usec):
> | 1.00th=[ 9], 5.00th=[ 56], 10.00th=[ 61], 20.00th=[ 73],
> | 30.00th=[ 80], 40.00th=[ 91], 50.00th=[ 110], 60.00th=[ 118],
> | 70.00th=[ 130], 80.00th=[ 157], 90.00th=[ 202], 95.00th=[ 253],
> | 99.00th=[ 408], 99.50th=[ 498], 99.90th=[ 1795], 99.95th=[ 3064],
> | 99.99th=[ 3785]
> bw ( KiB/s): min=612240, max=1078792, per=100.00%, avg=1015996.40, stdev=106032.96, samples=20
> iops : min=153060, max=269698, avg=253998.90, stdev=26508.40, samples=20
> lat (usec) : 4=0.01%, 10=1.47%, 20=1.19%, 50=0.17%, 100=42.03%
> lat (usec) : 250=49.95%, 500=4.70%, 750=0.28%, 1000=0.06%
> lat (msec) : 2=0.06%, 4=0.09%, 10=0.01%
> cpu : usr=21.45%, sys=51.11%, ctx=530348, majf=0, minf=139
> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
> issued rwts: total=2540023,0,0,0 short=0,0,0,0 dropped=0,0,0,0
> latency : target=0, window=0, percentile=100.00%, depth=32
>Run status group 0 (all jobs):
> MIXED: bw=992MiB/s (1040MB/s), 992MiB/s-992MiB/s (1040MB/s-1040MB/s), io=9922MiB (10.4GB), run=10001-10001msec Disk stats (read/write):
> nvme2n1: ios=2514356/0, merge=0/0, ticks=400/0, in_queue=297612, util=99.06%
>
>
>don(a)donst201:~/fio/single/rr$ cat 1c.ini |grep -v '#'
>[global]
>rw=randrw
>rwmixread=100
>numjobs=1
>iodepth=32
>bs=4k
>direct=1
>thread=1
>time_based=1
>ramp_time=0
>runtime=10
>ioengine=/home/don/install/spdk/spdk/examples/nvme/fio_plugin/fio_plugin
>group_reporting=1
>unified_rw_reporting=1
>exitall=1
>randrepeat=0
>norandommap=1
>[0000.1c.00.0-1]
>filename=trtype=PCIe traddr=0000.1c.00.0 ns=1 [0000.1c.00.0-2] filename=trtype=PCIe traddr=0000.1c.00.0 ns=1 [0000.1c.00.0-3] filename=trtype=PCIe traddr=0000.1c.00.0 ns=1 [0000.1c.00.0-4] filename=trtype=PCIe traddr=0000.1c.00.0 ns=1
>
>
>don(a)donst201:~/fio/single/rr$ sudo fio 1c.ini
>0000.1c.00.0-1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk, iodepth=32
>0000.1c.00.0-2: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk, iodepth=32
>0000.1c.00.0-3: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk, iodepth=32
>0000.1c.00.0-4: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk, iodepth=32
>fio-3.11
>Starting 4 threads
>Starting SPDK v18.10 / DPDK 18.08.0 initialization...
>[ DPDK EAL parameters: fio --no-shconf -c 0x1 -m 512 --file-prefix=spdk_pid4270 ]
>EAL: Detected 36 lcore(s)
>EAL: Detected 2 NUMA nodes
>EAL: No free hugepages reported in hugepages-1048576kB
>EAL: Probing VFIO support...
>EAL: PCI device 0000:1c:00.0 on NUMA socket 0
>EAL: probe driver: 8086:a54 spdk_nvme
>Jobs: 1 (f=0): [_(3),f(1)][100.0%][eta 00m:00s]
>0000.1c.00.0-1: (groupid=0, jobs=4): err= 0: pid=4310: Mon Nov 5 05:16:03 2018
> mixed: IOPS=12.5k, BW=48.0MiB/s (51.4MB/s)(490MiB/10010msec)
> slat (nsec): min=98, max=15092, avg=120.02, stdev=90.07
> clat (usec): min=55, max=19757, avg=10207.22, stdev=884.06
> lat (usec): min=60, max=19757, avg=10207.34, stdev=884.06
> clat percentiles (usec):
> | 1.00th=[ 8979], 5.00th=[ 9110], 10.00th=[ 9372], 20.00th=[ 9503],
> | 30.00th=[ 9765], 40.00th=[ 9896], 50.00th=[10159], 60.00th=[10290],
> | 70.00th=[10552], 80.00th=[10814], 90.00th=[11207], 95.00th=[11600],
> | 99.00th=[12911], 99.50th=[14353], 99.90th=[17695], 99.95th=[18744],
> | 99.99th=[19006]
> bw ( KiB/s): min=12008, max=12976, per=25.00%, avg=12535.70, stdev=306.81, samples=80
> iops : min= 3002, max= 3244, avg=3133.85, stdev=76.77, samples=80
> lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
> lat (msec) : 2=0.01%, 4=0.02%, 10=45.13%, 20=54.83%
> cpu : usr=100.01%, sys=0.00%, ctx=27, majf=0, minf=0
> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.9%, >=64=0.0%
> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
> issued rwts: total=125506,0,0,0 short=0,0,0,0 dropped=0,0,0,0
> latency : target=0, window=0, percentile=100.00%, depth=32
>Run status group 0 (all jobs):
> MIXED: bw=48.0MiB/s (51.4MB/s), 48.0MiB/s-48.0MiB/s (51.4MB/s-51.4MB/s), io=490MiB (514MB), run=10010-10010msec
>
>
>don(a)donst201:~/fio/single/rr$ cat nvme2n1.ini |grep -v '#'
>[global]
>rw=randrw
>rwmixread=100
>numjobs=1
>iodepth=1
>bs=4k
>direct=1
>thread=1
>time_based=1
>ramp_time=0
>runtime=10
>ioengine=libaio
>group_reporting=1
>unified_rw_reporting=1
>exitall=1
>randrepeat=0
>norandommap=1
>[nvme2n1]
>filename=/dev/nvme2n1
>
>
>don(a)donst201:~/fio/single/rr$ sudo fio nvme2n1.ini
>nvme2n1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
>fio-3.11
>Starting 1 thread
>Jobs: 1 (f=1): [r(1)][100.0%][r=43.6MiB/s][r=11.2k IOPS][eta 00m:00s]
>nvme2n1: (groupid=0, jobs=1): err= 0: pid=2525: Mon Nov 5 05:22:57 2018
> mixed: IOPS=11.2k, BW=43.6MiB/s (45.7MB/s)(436MiB/10001msec)
> slat (nsec): min=4848, max=208595, avg=5126.33, stdev=771.27
> clat (nsec): min=1845, max=3405.8k, avg=83004.99, stdev=33577.36
> lat (usec): min=15, max=3411, avg=88.26, stdev=33.59
> clat percentiles (usec):
> | 1.00th=[ 15], 5.00th=[ 58], 10.00th=[ 59], 20.00th=[ 59],
> | 30.00th=[ 60], 40.00th=[ 77], 50.00th=[ 77], 60.00th=[ 78],
> | 70.00th=[ 111], 80.00th=[ 113], 90.00th=[ 113], 95.00th=[ 115],
> | 99.00th=[ 186], 99.50th=[ 186], 99.90th=[ 190], 99.95th=[ 192],
> | 99.99th=[ 375]
> bw ( KiB/s): min=43816, max=45016, per=99.98%, avg=44639.58, stdev=312.88, samples=19
> iops : min=10954, max=11254, avg=11159.89, stdev=78.22, samples=19
> lat (usec) : 2=0.01%, 20=2.83%, 50=0.01%, 100=64.57%, 250=32.59%
> lat (usec) : 500=0.01%, 750=0.01%
> lat (msec) : 2=0.01%, 4=0.01%
> cpu : usr=4.02%, sys=10.33%, ctx=111630, majf=0, minf=12
> IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
> issued rwts: total=111631,0,0,0 short=0,0,0,0 dropped=0,0,0,0
> latency : target=0, window=0, percentile=100.00%, depth=1
>Run status group 0 (all jobs):
> MIXED: bw=43.6MiB/s (45.7MB/s), 43.6MiB/s-43.6MiB/s (45.7MB/s-45.7MB/s), io=436MiB (457MB), run=10001-10001msec Disk stats (read/write):
> nvme2n1: ios=110275/0, merge=0/0, ticks=0/0, in_queue=8596, util=86.22%
>
>
>Thanks,
>Joe
>
>> -----Original Message-----
>> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Cao, Gang
>> Sent: Sunday, November 4, 2018 8:40 PM
>> To: Storage Performance Development Kit < spdk(a)lists.01.org >
>> Subject: Re: [SPDK] FIO NVMe Performance Results
>>
>> For the SPDK plugin, from the below result, looks like only 1 job
>> started with
>> 32 queue depth:
>>
>> Jobs: 1 (f=0): [f(1),_(3)][100.0%][eta 00m:00s] <<<<
>>
>> Try to update the FIO configuration file as
>>
>> [job1]
>> filename=trtype=PCIe traddr=0000.1c.00.0 ns=1 [job2]
>> filename=trtype=PCIe
>> traddr=0000.1c.00.0 ns=1 ....
>>
>> The kernel FIO is with 4 jobs and each job of 32 queue depth. For
>> SPDK, you can configure as the same, or just configure two jobs and
>> each job of 64 queue depth. The overall queue depth to the device is same of 128.
>>
>> Thanks,
>> Gang
>>
>> -----Original Message-----
>> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Gruher,
>> Joseph R
>> Sent: Monday, November 5, 2018 10:49 AM
>> To: Storage Performance Development Kit < spdk(a)lists.01.org >
>> Subject: [SPDK] FIO NVMe Performance Results
>>
>> Hi folks-
>>
>> I'm testing SPDK 18.10 on Ubuntu 18.04 with kernel 4.18.16. I used
>> FIO with the kernel NVMe driver to measure the performance of a local
>> (PCIe
>> attached) Intel P4500 NVMe device on a 4KB random read workload and
>> obtained 477K IOPS, roughly in line with the drive spec. Then I
>> tested the same drive with the SPDK FIO plugin and only achieved 13K
>> IOPS. The FIO test files and the results are pasted below. Any ideas
>> where I'm going wrong here?
>>
>> Thanks!
>>
>>
>> don(a)donst201:~/fio/single/rr$ ls
>> /sys/devices/pci0000:17/0000:17:02.0/0000:1c:00.0/nvme/nvme1/
>> address cntlid dev device firmware_rev model nvme2n1 power
>> rescan_controller reset_controller serial state subsysnqn
>> subsystem transport uevent
>>
>>
>> don(a)donst201:~/fio/single/rr$ cat nvme2n1.ini |grep -v '#'
>> [global]
>> rw=randrw
>> rwmixread=100
>> numjobs=4
>> iodepth=32
>> bs=4k
>> direct=1
>> thread=1
>> time_based=1
>> ramp_time=0
>> runtime=10
>> ioengine=libaio
>> group_reporting=1
>> unified_rw_reporting=1
>> exitall=1
>> randrepeat=0
>> norandommap=1
>> [nvme2n1]
>> filename=/dev/nvme2n1
>>
>>
>> don(a)donst201:~/fio/single/rr$ sudo fio nvme2n1.ini
>> nvme2n1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T)
>> 4096B-4096B, ioengine=libaio, iodepth=32 ...
>> fio-3.11
>> Starting 4 threads
>> Jobs: 4 (f=4): [r(4)][100.0%][r=1874MiB/s][r=480k IOPS][eta 00m:00s]
>> nvme2n1: (groupid=0, jobs=4): err= 0: pid=2575: Mon Nov 5 02:26:25 2018
>> mixed: IOPS=477k, BW=1862MiB/s (1952MB/s)(18.2GiB/10001msec)
>> slat (nsec): min=1295, max=333088, avg=2216.35, stdev=927.89
>> clat (nsec): min=506, max=5818.8k, avg=265924.16, stdev=223761.59
>> lat (usec): min=6, max=5821, avg=268.19, stdev=223.76
>> clat percentiles (usec):
>> | 1.00th=[ 13], 5.00th=[ 74], 10.00th=[ 87], 20.00th=[ 115],
>> | 30.00th=[ 139], 40.00th=[ 167], 50.00th=[ 204], 60.00th=[ 247],
>> | 70.00th=[ 306], 80.00th=[ 388], 90.00th=[ 523], 95.00th=[ 660],
>> | 99.00th=[ 988], 99.50th=[ 1156], 99.90th=[ 2343], 99.95th=[ 3195],
>> | 99.99th=[ 4424]
>> bw ( KiB/s): min=448192, max=483104, per=25.00%, avg=476597.10,
>> stdev=6748.78, samples=80
>> iops : min=112048, max=120776, avg=119149.27, stdev=1687.20,
>> samples=80
>> lat (nsec) : 750=0.01%
>> lat (usec) : 10=0.52%, 20=1.25%, 50=0.25%, 100=13.38%, 250=45.13%
>> lat (usec) : 500=28.23%, 750=8.08%, 1000=2.22%
>> lat (msec) : 2=0.82%, 4=0.10%, 10=0.02%
>> cpu : usr=14.08%, sys=36.02%, ctx=1930207, majf=0, minf=697
>> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%,
>> >=64=0.0%
>> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>> >=64=0.0%
>> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%,
>> >=64=0.0%
>> issued rwts: total=4766095,0,0,0 short=0,0,0,0 dropped=0,0,0,0
>> latency : target=0, window=0, percentile=100.00%, depth=32
>> Run status group 0 (all jobs):
>> MIXED: bw=1862MiB/s (1952MB/s), 1862MiB/s-1862MiB/s (1952MB/s-
>> 1952MB/s), io=18.2GiB (19.5GB), run=10001-10001msec Disk stats
>> (read/write):
>> nvme2n1: ios=4709190/0, merge=0/0, ticks=4096/0, in_queue=1388752,
>> util=100.00%
>>
>>
>> don(a)donst201:~/fio/single/rr$ sudo
>> /home/don/install/spdk/spdk/scripts/setup.sh
>> Active mountpoints on /dev/nvme0n1, so not binding PCI dev
>> 0000:03:00.0
>> 0000:1c:00.0 (8086 0a54): nvme -> uio_pci_generic
>> 0000:1d:00.0 (8086 0a54): nvme -> uio_pci_generic
>> 0000:5e:00.0 (8086 0a54): nvme -> uio_pci_generic
>> 0000:5f:00.0 (8086 0a54): nvme -> uio_pci_generic
>> 0000:62:00.0 (8086 0a54): nvme -> uio_pci_generic
>> 0000:63:00.0 (8086 0a54): nvme -> uio_pci_generic
>> 0000:64:00.0 (8086 0a54): nvme -> uio_pci_generic
>> 0000:65:00.0 (8086 0a54): nvme -> uio_pci_generic
>> 0000:da:00.0 (8086 0a54): nvme -> uio_pci_generic
>> 0000:db:00.0 (8086 0a54): nvme -> uio_pci_generic
>> 0000:dc:00.0 (8086 0a54): nvme -> uio_pci_generic
>> 0000:dd:00.0 (8086 0a54): nvme -> uio_pci_generic
>> 0000:e0:00.0 (8086 0a54): nvme -> uio_pci_generic
>> 0000:e1:00.0 (8086 0a54): nvme -> uio_pci_generic
>> 0000:e2:00.0 (8086 0a54): nvme -> uio_pci_generic
>> 0000:e3:00.0 (8086 0a54): nvme -> uio_pci_generic
>> 0000:00:04.0 (8086 2021): ioatdma -> uio_pci_generic
>> 0000:00:04.1 (8086 2021): ioatdma -> uio_pci_generic
>> 0000:00:04.2 (8086 2021): ioatdma -> uio_pci_generic
>> 0000:00:04.3 (8086 2021): ioatdma -> uio_pci_generic
>> 0000:00:04.4 (8086 2021): ioatdma -> uio_pci_generic
>> 0000:00:04.5 (8086 2021): ioatdma -> uio_pci_generic
>> 0000:00:04.6 (8086 2021): ioatdma -> uio_pci_generic
>> 0000:00:04.7 (8086 2021): ioatdma -> uio_pci_generic
>> 0000:80:04.0 (8086 2021): ioatdma -> uio_pci_generic
>> 0000:80:04.1 (8086 2021): ioatdma -> uio_pci_generic
>> 0000:80:04.2 (8086 2021): ioatdma -> uio_pci_generic
>> 0000:80:04.3 (8086 2021): ioatdma -> uio_pci_generic
>> 0000:80:04.4 (8086 2021): ioatdma -> uio_pci_generic
>> 0000:80:04.5 (8086 2021): ioatdma -> uio_pci_generic
>> 0000:80:04.6 (8086 2021): ioatdma -> uio_pci_generic
>> 0000:80:04.7 (8086 2021): ioatdma -> uio_pci_generic
>>
>>
>> don(a)donst201:~/fio/single/rr$ cat 1c.ini |grep -v '#'
>> [global]
>> rw=randrw
>> rwmixread=100
>> numjobs=4
>> iodepth=32
>> bs=4k
>> direct=1
>> thread=1
>> time_based=1
>> ramp_time=0
>> runtime=10
>> ioengine=/home/don/install/spdk/spdk/examples/nvme/fio_plugin/fio_plug
>> in
>> group_reporting=1
>> unified_rw_reporting=1
>> exitall=1
>> randrepeat=0
>> norandommap=1
>> [0000.1c.00.0]
>> filename=trtype=PCIe traddr=0000.1c.00.0 ns=1
>>
>>
>> don(a)donst201:~/fio/single/rr$ sudo fio 1c.ini
>> 0000.1c.00.0: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B,
>> (T) 4096B-4096B, ioengine=spdk, iodepth=32 ...
>> fio-3.11
>> Starting 4 threads
>> Starting SPDK v18.10 / DPDK 18.08.0 initialization...
>> [ DPDK EAL parameters: fio --no-shconf -c 0x1 -m 512 --file-
>> prefix=spdk_pid3668 ]
>> EAL: Detected 36 lcore(s)
>> EAL: Detected 2 NUMA nodes
>> EAL: No free hugepages reported in hugepages-1048576kB
>> EAL: Probing VFIO support...
>> EAL: PCI device 0000:1c:00.0 on NUMA socket 0
>> EAL: probe driver: 8086:a54 spdk_nvme
>> Jobs: 1 (f=0): [f(1),_(3)][100.0%][eta 00m:00s]
>> 0000.1c.00.0: (groupid=0, jobs=4): err= 0: pid=3709: Mon Nov 5
>> 02:28:29
>> 2018
>> mixed: IOPS=13.2k, BW=51.6MiB/s (54.1MB/s)(517MiB/10011msec)
>> slat (nsec): min=109, max=15344, avg=127.91, stdev=73.44
>> clat (usec): min=187, max=18715, avg=9683.80, stdev=650.54
>> lat (usec): min=193, max=18715, avg=9683.93, stdev=650.53
>> clat percentiles (usec):
>> | 1.00th=[ 8717], 5.00th=[ 8848], 10.00th=[ 8979], 20.00th=[ 9110],
>> | 30.00th=[ 9241], 40.00th=[ 9372], 50.00th=[ 9503], 60.00th=[ 9765],
>> | 70.00th=[10028], 80.00th=[10159], 90.00th=[10552], 95.00th=[10945],
>> | 99.00th=[11338], 99.50th=[11338], 99.90th=[11731], 99.95th=[13960],
>> | 99.99th=[17695]
>> bw ( KiB/s): min=13080, max=13304, per=25.00%, avg=13214.70,
>> stdev=55.30, samples=80
>> iops : min= 3270, max= 3326, avg=3303.65, stdev=13.84, samples=80
>> lat (usec) : 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
>> lat (msec) : 2=0.01%, 4=0.02%, 10=70.85%, 20=29.11%
>> cpu : usr=100.01%, sys=0.00%, ctx=23, majf=0, minf=0
>> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.9%,
>> >=64=0.0%
>> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>> >=64=0.0%
>> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%,
>> >=64=0.0%
>> issued rwts: total=132300,0,0,0 short=0,0,0,0 dropped=0,0,0,0
>> latency : target=0, window=0, percentile=100.00%, depth=32
>> Run status group 0 (all jobs):
>> MIXED: bw=51.6MiB/s (54.1MB/s), 51.6MiB/s-51.6MiB/s (54.1MB/s-
>> 54.1MB/s), io=517MiB (542MB), run=10011-10011msec
>>
>> _______________________________________________
>> SPDK mailing list
>> SPDK(a)lists.01.org
>> https://lists.01.org/mailman/listinfo/spdk
>> _______________________________________________
>> SPDK mailing list
>> SPDK(a)lists.01.org
>> https://lists.01.org/mailman/listinfo/spdk
>_______________________________________________
>SPDK mailing list
>SPDK(a)lists.01.org
>https://lists.01.org/mailman/listinfo/spdk
>_______________________________________________
>SPDK mailing list
>SPDK(a)lists.01.org
>https://lists.01.org/mailman/listinfo/spdk
--
Маров Алексей
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [SPDK] FIO NVMe Performance Results
@ 2018-11-05 8:32
0 siblings, 0 replies; 25+ messages in thread
From: @ 2018-11-05 8:32 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 21051 bytes --]
Sorry for incomplete message I have sent before
Hello Gang.
I just curious, why would you run fio with SPDK using this conf file, with is a bit different from what is suggested in here https://github.com/spdk/spdk/tree/master/examples/bdev/fio_plugin .
I mean may be you should create conf file for spdk like this one :
nvme.conf :
[Nvme]
TransportId "trtype:PCIe traddr:0000.1c.00.0" Nvme0
and provide it in fio config.
fio.ini :
[global]
ioengine=spdk_bdev
spdk_conf=....../nvme.conf
rw=randrw
rwmixread=100
numjobs=4
iodepth=32
bs=4k
direct=1
thread=1
time_based=1
ramp_time=0
runtime=10
group_reporting=1
unified_rw_reporting=1
exitall=1
randrepeat=0
norandommap=1
[0000:06:00.0]
filename=Nvme0n1
and run like this
# LD_PRELOAD=examples/bdev/fio_plugin/fio_plugin fio examples/bdev/fio_plugin/fio.ini
Just to note. The recommended version of fio for SPDK is 3.3
Best Regards
Alex
>Понедельник, 5 ноября 2018, 9:53 +03:00 от "Cao, Gang" <gang.cao(a)intel.com>:
>
>Thanks. This is interesting especially for your finding on queue depth 1 case. We are trying to reproduce this issue locally same as your configuration.
>
>Thanks,
>Gang
>
>-----Original Message-----
>From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Gruher, Joseph R
>Sent: Monday, November 5, 2018 1:32 PM
>To: Storage Performance Development Kit < spdk(a)lists.01.org >
>Subject: Re: [SPDK] FIO NVMe Performance Results
>
>Interesting catch. It seems like FIO is actually not accurately reporting the number of jobs in the report out for the SPDK case. If I disable the group_reporting parameter I do get a printout for four separate jobs in the SPDK case, so I think it is really running four jobs there, and just summarizing incorrectly. I also see four separate CPU cores go to 100% when I run that FIO test case, indicating four jobs are active.
>
>Also, I don't think would explain the difference in performance. Using an aggregate queue depth of 32 versus 128 shouldn't lead to a drop in performance from 377K to 13K.
>
>I retested SPDK with multiple separate job descriptions as you suggested anyway, using four separate jobs instead of relying on the numjobs=4 parameter, and the results are below. In that SPDK test with four separate FIO job statements, each with queue depth 32, I still only achieved 12.5K IOPS.
>
>I then re-ran the kernel NVMe driver test case with only a single job for comparison, and even with only 1 job and queue depth 32 it still achieves 254K IOPS.
>
>We might be on the right track with a problem around queue depth however. If I run the kernel NVMe driver case with just 1 job and queue depth of only 1, I then get 11.2K IOPS, which is around the performance I get in the SPDK test case. Perhaps the SPDK case is not actually queuing IOs on the device for some reason?
>
>
>don(a)donst201:~/fio/single/rr$ cat nvme2n1.ini |grep -v '#'
>[global]
>rw=randrw
>rwmixread=100
>numjobs=1
>iodepth=32
>bs=4k
>direct=1
>thread=1
>time_based=1
>ramp_time=0
>runtime=10
>ioengine=libaio
>group_reporting=1
>unified_rw_reporting=1
>exitall=1
>randrepeat=0
>norandommap=1
>[nvme2n1]
>filename=/dev/nvme2n1
>
>
>don(a)donst201:~/fio/single/rr$ sudo fio nvme2n1.ini
>nvme2n1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32
>fio-3.11
>Starting 1 thread
>Jobs: 1 (f=1): [r(1)][100.0%][r=1025MiB/s][r=262k IOPS][eta 00m:00s]
>nvme2n1: (groupid=0, jobs=1): err= 0: pid=2422: Mon Nov 5 05:19:13 2018
> mixed: IOPS=254k, BW=992MiB/s (1040MB/s)(9922MiB/10001msec)
> slat (nsec): min=1185, max=319052, avg=1611.64, stdev=808.40
> clat (usec): min=2, max=4743, avg=124.02, stdev=121.00
> lat (usec): min=6, max=4744, avg=125.68, stdev=121.02
> clat percentiles (usec):
> | 1.00th=[ 9], 5.00th=[ 56], 10.00th=[ 61], 20.00th=[ 73],
> | 30.00th=[ 80], 40.00th=[ 91], 50.00th=[ 110], 60.00th=[ 118],
> | 70.00th=[ 130], 80.00th=[ 157], 90.00th=[ 202], 95.00th=[ 253],
> | 99.00th=[ 408], 99.50th=[ 498], 99.90th=[ 1795], 99.95th=[ 3064],
> | 99.99th=[ 3785]
> bw ( KiB/s): min=612240, max=1078792, per=100.00%, avg=1015996.40, stdev=106032.96, samples=20
> iops : min=153060, max=269698, avg=253998.90, stdev=26508.40, samples=20
> lat (usec) : 4=0.01%, 10=1.47%, 20=1.19%, 50=0.17%, 100=42.03%
> lat (usec) : 250=49.95%, 500=4.70%, 750=0.28%, 1000=0.06%
> lat (msec) : 2=0.06%, 4=0.09%, 10=0.01%
> cpu : usr=21.45%, sys=51.11%, ctx=530348, majf=0, minf=139
> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
> issued rwts: total=2540023,0,0,0 short=0,0,0,0 dropped=0,0,0,0
> latency : target=0, window=0, percentile=100.00%, depth=32
>Run status group 0 (all jobs):
> MIXED: bw=992MiB/s (1040MB/s), 992MiB/s-992MiB/s (1040MB/s-1040MB/s), io=9922MiB (10.4GB), run=10001-10001msec Disk stats (read/write):
> nvme2n1: ios=2514356/0, merge=0/0, ticks=400/0, in_queue=297612, util=99.06%
>
>
>don(a)donst201:~/fio/single/rr$ cat 1c.ini |grep -v '#'
>[global]
>rw=randrw
>rwmixread=100
>numjobs=1
>iodepth=32
>bs=4k
>direct=1
>thread=1
>time_based=1
>ramp_time=0
>runtime=10
>ioengine=/home/don/install/spdk/spdk/examples/nvme/fio_plugin/fio_plugin
>group_reporting=1
>unified_rw_reporting=1
>exitall=1
>randrepeat=0
>norandommap=1
>[0000.1c.00.0-1]
>filename=trtype=PCIe traddr=0000.1c.00.0 ns=1 [0000.1c.00.0-2] filename=trtype=PCIe traddr=0000.1c.00.0 ns=1 [0000.1c.00.0-3] filename=trtype=PCIe traddr=0000.1c.00.0 ns=1 [0000.1c.00.0-4] filename=trtype=PCIe traddr=0000.1c.00.0 ns=1
>
>
>don(a)donst201:~/fio/single/rr$ sudo fio 1c.ini
>0000.1c.00.0-1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk, iodepth=32
>0000.1c.00.0-2: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk, iodepth=32
>0000.1c.00.0-3: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk, iodepth=32
>0000.1c.00.0-4: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk, iodepth=32
>fio-3.11
>Starting 4 threads
>Starting SPDK v18.10 / DPDK 18.08.0 initialization...
>[ DPDK EAL parameters: fio --no-shconf -c 0x1 -m 512 --file-prefix=spdk_pid4270 ]
>EAL: Detected 36 lcore(s)
>EAL: Detected 2 NUMA nodes
>EAL: No free hugepages reported in hugepages-1048576kB
>EAL: Probing VFIO support...
>EAL: PCI device 0000:1c:00.0 on NUMA socket 0
>EAL: probe driver: 8086:a54 spdk_nvme
>Jobs: 1 (f=0): [_(3),f(1)][100.0%][eta 00m:00s]
>0000.1c.00.0-1: (groupid=0, jobs=4): err= 0: pid=4310: Mon Nov 5 05:16:03 2018
> mixed: IOPS=12.5k, BW=48.0MiB/s (51.4MB/s)(490MiB/10010msec)
> slat (nsec): min=98, max=15092, avg=120.02, stdev=90.07
> clat (usec): min=55, max=19757, avg=10207.22, stdev=884.06
> lat (usec): min=60, max=19757, avg=10207.34, stdev=884.06
> clat percentiles (usec):
> | 1.00th=[ 8979], 5.00th=[ 9110], 10.00th=[ 9372], 20.00th=[ 9503],
> | 30.00th=[ 9765], 40.00th=[ 9896], 50.00th=[10159], 60.00th=[10290],
> | 70.00th=[10552], 80.00th=[10814], 90.00th=[11207], 95.00th=[11600],
> | 99.00th=[12911], 99.50th=[14353], 99.90th=[17695], 99.95th=[18744],
> | 99.99th=[19006]
> bw ( KiB/s): min=12008, max=12976, per=25.00%, avg=12535.70, stdev=306.81, samples=80
> iops : min= 3002, max= 3244, avg=3133.85, stdev=76.77, samples=80
> lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
> lat (msec) : 2=0.01%, 4=0.02%, 10=45.13%, 20=54.83%
> cpu : usr=100.01%, sys=0.00%, ctx=27, majf=0, minf=0
> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.9%, >=64=0.0%
> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
> issued rwts: total=125506,0,0,0 short=0,0,0,0 dropped=0,0,0,0
> latency : target=0, window=0, percentile=100.00%, depth=32
>Run status group 0 (all jobs):
> MIXED: bw=48.0MiB/s (51.4MB/s), 48.0MiB/s-48.0MiB/s (51.4MB/s-51.4MB/s), io=490MiB (514MB), run=10010-10010msec
>
>
>don(a)donst201:~/fio/single/rr$ cat nvme2n1.ini |grep -v '#'
>[global]
>rw=randrw
>rwmixread=100
>numjobs=1
>iodepth=1
>bs=4k
>direct=1
>thread=1
>time_based=1
>ramp_time=0
>runtime=10
>ioengine=libaio
>group_reporting=1
>unified_rw_reporting=1
>exitall=1
>randrepeat=0
>norandommap=1
>[nvme2n1]
>filename=/dev/nvme2n1
>
>
>don(a)donst201:~/fio/single/rr$ sudo fio nvme2n1.ini
>nvme2n1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
>fio-3.11
>Starting 1 thread
>Jobs: 1 (f=1): [r(1)][100.0%][r=43.6MiB/s][r=11.2k IOPS][eta 00m:00s]
>nvme2n1: (groupid=0, jobs=1): err= 0: pid=2525: Mon Nov 5 05:22:57 2018
> mixed: IOPS=11.2k, BW=43.6MiB/s (45.7MB/s)(436MiB/10001msec)
> slat (nsec): min=4848, max=208595, avg=5126.33, stdev=771.27
> clat (nsec): min=1845, max=3405.8k, avg=83004.99, stdev=33577.36
> lat (usec): min=15, max=3411, avg=88.26, stdev=33.59
> clat percentiles (usec):
> | 1.00th=[ 15], 5.00th=[ 58], 10.00th=[ 59], 20.00th=[ 59],
> | 30.00th=[ 60], 40.00th=[ 77], 50.00th=[ 77], 60.00th=[ 78],
> | 70.00th=[ 111], 80.00th=[ 113], 90.00th=[ 113], 95.00th=[ 115],
> | 99.00th=[ 186], 99.50th=[ 186], 99.90th=[ 190], 99.95th=[ 192],
> | 99.99th=[ 375]
> bw ( KiB/s): min=43816, max=45016, per=99.98%, avg=44639.58, stdev=312.88, samples=19
> iops : min=10954, max=11254, avg=11159.89, stdev=78.22, samples=19
> lat (usec) : 2=0.01%, 20=2.83%, 50=0.01%, 100=64.57%, 250=32.59%
> lat (usec) : 500=0.01%, 750=0.01%
> lat (msec) : 2=0.01%, 4=0.01%
> cpu : usr=4.02%, sys=10.33%, ctx=111630, majf=0, minf=12
> IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
> issued rwts: total=111631,0,0,0 short=0,0,0,0 dropped=0,0,0,0
> latency : target=0, window=0, percentile=100.00%, depth=1
>Run status group 0 (all jobs):
> MIXED: bw=43.6MiB/s (45.7MB/s), 43.6MiB/s-43.6MiB/s (45.7MB/s-45.7MB/s), io=436MiB (457MB), run=10001-10001msec Disk stats (read/write):
> nvme2n1: ios=110275/0, merge=0/0, ticks=0/0, in_queue=8596, util=86.22%
>
>
>Thanks,
>Joe
>
>> -----Original Message-----
>> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Cao, Gang
>> Sent: Sunday, November 4, 2018 8:40 PM
>> To: Storage Performance Development Kit < spdk(a)lists.01.org >
>> Subject: Re: [SPDK] FIO NVMe Performance Results
>>
>> For the SPDK plugin, from the below result, looks like only 1 job
>> started with
>> 32 queue depth:
>>
>> Jobs: 1 (f=0): [f(1),_(3)][100.0%][eta 00m:00s] <<<<
>>
>> Try to update the FIO configuration file as
>>
>> [job1]
>> filename=trtype=PCIe traddr=0000.1c.00.0 ns=1 [job2]
>> filename=trtype=PCIe
>> traddr=0000.1c.00.0 ns=1 ....
>>
>> The kernel FIO is with 4 jobs and each job of 32 queue depth. For
>> SPDK, you can configure as the same, or just configure two jobs and
>> each job of 64 queue depth. The overall queue depth to the device is same of 128.
>>
>> Thanks,
>> Gang
>>
>> -----Original Message-----
>> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Gruher,
>> Joseph R
>> Sent: Monday, November 5, 2018 10:49 AM
>> To: Storage Performance Development Kit < spdk(a)lists.01.org >
>> Subject: [SPDK] FIO NVMe Performance Results
>>
>> Hi folks-
>>
>> I'm testing SPDK 18.10 on Ubuntu 18.04 with kernel 4.18.16. I used
>> FIO with the kernel NVMe driver to measure the performance of a local
>> (PCIe
>> attached) Intel P4500 NVMe device on a 4KB random read workload and
>> obtained 477K IOPS, roughly in line with the drive spec. Then I
>> tested the same drive with the SPDK FIO plugin and only achieved 13K
>> IOPS. The FIO test files and the results are pasted below. Any ideas
>> where I'm going wrong here?
>>
>> Thanks!
>>
>>
>> don(a)donst201:~/fio/single/rr$ ls
>> /sys/devices/pci0000:17/0000:17:02.0/0000:1c:00.0/nvme/nvme1/
>> address cntlid dev device firmware_rev model nvme2n1 power
>> rescan_controller reset_controller serial state subsysnqn
>> subsystem transport uevent
>>
>>
>> don(a)donst201:~/fio/single/rr$ cat nvme2n1.ini |grep -v '#'
>> [global]
>> rw=randrw
>> rwmixread=100
>> numjobs=4
>> iodepth=32
>> bs=4k
>> direct=1
>> thread=1
>> time_based=1
>> ramp_time=0
>> runtime=10
>> ioengine=libaio
>> group_reporting=1
>> unified_rw_reporting=1
>> exitall=1
>> randrepeat=0
>> norandommap=1
>> [nvme2n1]
>> filename=/dev/nvme2n1
>>
>>
>> don(a)donst201:~/fio/single/rr$ sudo fio nvme2n1.ini
>> nvme2n1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T)
>> 4096B-4096B, ioengine=libaio, iodepth=32 ...
>> fio-3.11
>> Starting 4 threads
>> Jobs: 4 (f=4): [r(4)][100.0%][r=1874MiB/s][r=480k IOPS][eta 00m:00s]
>> nvme2n1: (groupid=0, jobs=4): err= 0: pid=2575: Mon Nov 5 02:26:25 2018
>> mixed: IOPS=477k, BW=1862MiB/s (1952MB/s)(18.2GiB/10001msec)
>> slat (nsec): min=1295, max=333088, avg=2216.35, stdev=927.89
>> clat (nsec): min=506, max=5818.8k, avg=265924.16, stdev=223761.59
>> lat (usec): min=6, max=5821, avg=268.19, stdev=223.76
>> clat percentiles (usec):
>> | 1.00th=[ 13], 5.00th=[ 74], 10.00th=[ 87], 20.00th=[ 115],
>> | 30.00th=[ 139], 40.00th=[ 167], 50.00th=[ 204], 60.00th=[ 247],
>> | 70.00th=[ 306], 80.00th=[ 388], 90.00th=[ 523], 95.00th=[ 660],
>> | 99.00th=[ 988], 99.50th=[ 1156], 99.90th=[ 2343], 99.95th=[ 3195],
>> | 99.99th=[ 4424]
>> bw ( KiB/s): min=448192, max=483104, per=25.00%, avg=476597.10,
>> stdev=6748.78, samples=80
>> iops : min=112048, max=120776, avg=119149.27, stdev=1687.20,
>> samples=80
>> lat (nsec) : 750=0.01%
>> lat (usec) : 10=0.52%, 20=1.25%, 50=0.25%, 100=13.38%, 250=45.13%
>> lat (usec) : 500=28.23%, 750=8.08%, 1000=2.22%
>> lat (msec) : 2=0.82%, 4=0.10%, 10=0.02%
>> cpu : usr=14.08%, sys=36.02%, ctx=1930207, majf=0, minf=697
>> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%,
>> >=64=0.0%
>> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>> >=64=0.0%
>> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%,
>> >=64=0.0%
>> issued rwts: total=4766095,0,0,0 short=0,0,0,0 dropped=0,0,0,0
>> latency : target=0, window=0, percentile=100.00%, depth=32
>> Run status group 0 (all jobs):
>> MIXED: bw=1862MiB/s (1952MB/s), 1862MiB/s-1862MiB/s (1952MB/s-
>> 1952MB/s), io=18.2GiB (19.5GB), run=10001-10001msec Disk stats
>> (read/write):
>> nvme2n1: ios=4709190/0, merge=0/0, ticks=4096/0, in_queue=1388752,
>> util=100.00%
>>
>>
>> don(a)donst201:~/fio/single/rr$ sudo
>> /home/don/install/spdk/spdk/scripts/setup.sh
>> Active mountpoints on /dev/nvme0n1, so not binding PCI dev
>> 0000:03:00.0
>> 0000:1c:00.0 (8086 0a54): nvme -> uio_pci_generic
>> 0000:1d:00.0 (8086 0a54): nvme -> uio_pci_generic
>> 0000:5e:00.0 (8086 0a54): nvme -> uio_pci_generic
>> 0000:5f:00.0 (8086 0a54): nvme -> uio_pci_generic
>> 0000:62:00.0 (8086 0a54): nvme -> uio_pci_generic
>> 0000:63:00.0 (8086 0a54): nvme -> uio_pci_generic
>> 0000:64:00.0 (8086 0a54): nvme -> uio_pci_generic
>> 0000:65:00.0 (8086 0a54): nvme -> uio_pci_generic
>> 0000:da:00.0 (8086 0a54): nvme -> uio_pci_generic
>> 0000:db:00.0 (8086 0a54): nvme -> uio_pci_generic
>> 0000:dc:00.0 (8086 0a54): nvme -> uio_pci_generic
>> 0000:dd:00.0 (8086 0a54): nvme -> uio_pci_generic
>> 0000:e0:00.0 (8086 0a54): nvme -> uio_pci_generic
>> 0000:e1:00.0 (8086 0a54): nvme -> uio_pci_generic
>> 0000:e2:00.0 (8086 0a54): nvme -> uio_pci_generic
>> 0000:e3:00.0 (8086 0a54): nvme -> uio_pci_generic
>> 0000:00:04.0 (8086 2021): ioatdma -> uio_pci_generic
>> 0000:00:04.1 (8086 2021): ioatdma -> uio_pci_generic
>> 0000:00:04.2 (8086 2021): ioatdma -> uio_pci_generic
>> 0000:00:04.3 (8086 2021): ioatdma -> uio_pci_generic
>> 0000:00:04.4 (8086 2021): ioatdma -> uio_pci_generic
>> 0000:00:04.5 (8086 2021): ioatdma -> uio_pci_generic
>> 0000:00:04.6 (8086 2021): ioatdma -> uio_pci_generic
>> 0000:00:04.7 (8086 2021): ioatdma -> uio_pci_generic
>> 0000:80:04.0 (8086 2021): ioatdma -> uio_pci_generic
>> 0000:80:04.1 (8086 2021): ioatdma -> uio_pci_generic
>> 0000:80:04.2 (8086 2021): ioatdma -> uio_pci_generic
>> 0000:80:04.3 (8086 2021): ioatdma -> uio_pci_generic
>> 0000:80:04.4 (8086 2021): ioatdma -> uio_pci_generic
>> 0000:80:04.5 (8086 2021): ioatdma -> uio_pci_generic
>> 0000:80:04.6 (8086 2021): ioatdma -> uio_pci_generic
>> 0000:80:04.7 (8086 2021): ioatdma -> uio_pci_generic
>>
>>
>> don(a)donst201:~/fio/single/rr$ cat 1c.ini |grep -v '#'
>> [global]
>> rw=randrw
>> rwmixread=100
>> numjobs=4
>> iodepth=32
>> bs=4k
>> direct=1
>> thread=1
>> time_based=1
>> ramp_time=0
>> runtime=10
>> ioengine=/home/don/install/spdk/spdk/examples/nvme/fio_plugin/fio_plug
>> in
>> group_reporting=1
>> unified_rw_reporting=1
>> exitall=1
>> randrepeat=0
>> norandommap=1
>> [0000.1c.00.0]
>> filename=trtype=PCIe traddr=0000.1c.00.0 ns=1
>>
>>
>> don(a)donst201:~/fio/single/rr$ sudo fio 1c.ini
>> 0000.1c.00.0: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B,
>> (T) 4096B-4096B, ioengine=spdk, iodepth=32 ...
>> fio-3.11
>> Starting 4 threads
>> Starting SPDK v18.10 / DPDK 18.08.0 initialization...
>> [ DPDK EAL parameters: fio --no-shconf -c 0x1 -m 512 --file-
>> prefix=spdk_pid3668 ]
>> EAL: Detected 36 lcore(s)
>> EAL: Detected 2 NUMA nodes
>> EAL: No free hugepages reported in hugepages-1048576kB
>> EAL: Probing VFIO support...
>> EAL: PCI device 0000:1c:00.0 on NUMA socket 0
>> EAL: probe driver: 8086:a54 spdk_nvme
>> Jobs: 1 (f=0): [f(1),_(3)][100.0%][eta 00m:00s]
>> 0000.1c.00.0: (groupid=0, jobs=4): err= 0: pid=3709: Mon Nov 5
>> 02:28:29
>> 2018
>> mixed: IOPS=13.2k, BW=51.6MiB/s (54.1MB/s)(517MiB/10011msec)
>> slat (nsec): min=109, max=15344, avg=127.91, stdev=73.44
>> clat (usec): min=187, max=18715, avg=9683.80, stdev=650.54
>> lat (usec): min=193, max=18715, avg=9683.93, stdev=650.53
>> clat percentiles (usec):
>> | 1.00th=[ 8717], 5.00th=[ 8848], 10.00th=[ 8979], 20.00th=[ 9110],
>> | 30.00th=[ 9241], 40.00th=[ 9372], 50.00th=[ 9503], 60.00th=[ 9765],
>> | 70.00th=[10028], 80.00th=[10159], 90.00th=[10552], 95.00th=[10945],
>> | 99.00th=[11338], 99.50th=[11338], 99.90th=[11731], 99.95th=[13960],
>> | 99.99th=[17695]
>> bw ( KiB/s): min=13080, max=13304, per=25.00%, avg=13214.70,
>> stdev=55.30, samples=80
>> iops : min= 3270, max= 3326, avg=3303.65, stdev=13.84, samples=80
>> lat (usec) : 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
>> lat (msec) : 2=0.01%, 4=0.02%, 10=70.85%, 20=29.11%
>> cpu : usr=100.01%, sys=0.00%, ctx=23, majf=0, minf=0
>> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.9%,
>> >=64=0.0%
>> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>> >=64=0.0%
>> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%,
>> >=64=0.0%
>> issued rwts: total=132300,0,0,0 short=0,0,0,0 dropped=0,0,0,0
>> latency : target=0, window=0, percentile=100.00%, depth=32
>> Run status group 0 (all jobs):
>> MIXED: bw=51.6MiB/s (54.1MB/s), 51.6MiB/s-51.6MiB/s (54.1MB/s-
>> 54.1MB/s), io=517MiB (542MB), run=10011-10011msec
>>
>> _______________________________________________
>> SPDK mailing list
>> SPDK(a)lists.01.org
>> https://lists.01.org/mailman/listinfo/spdk
>> _______________________________________________
>> SPDK mailing list
>> SPDK(a)lists.01.org
>> https://lists.01.org/mailman/listinfo/spdk
>_______________________________________________
>SPDK mailing list
>SPDK(a)lists.01.org
>https://lists.01.org/mailman/listinfo/spdk
>_______________________________________________
>SPDK mailing list
>SPDK(a)lists.01.org
>https://lists.01.org/mailman/listinfo/spdk
--
Маров Алексей
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [SPDK] FIO NVMe Performance Results
@ 2018-11-05 8:27
0 siblings, 0 replies; 25+ messages in thread
From: @ 2018-11-05 8:27 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 20513 bytes --]
Hello Gang.
I just curious, why would you run fio with SPDK using this conf file, with is a bit different from what is suggested in here https://github.com/spdk/spdk/tree/master/examples/bdev/fio_plugin .
I mean may be you should create conf file for spdk like this one :
nvme.conf :
[Nvme]
TransportId "trtype:PCIe traddr:0000.1c.00.0" Nvme0
and provide it in fio config.
fio.ini :
>Понедельник, 5 ноября 2018, 9:53 +03:00 от "Cao, Gang" <gang.cao(a)intel.com>:
>
>Thanks. This is interesting especially for your finding on queue depth 1 case. We are trying to reproduce this issue locally same as your configuration.
>
>Thanks,
>Gang
>
>-----Original Message-----
>From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Gruher, Joseph R
>Sent: Monday, November 5, 2018 1:32 PM
>To: Storage Performance Development Kit < spdk(a)lists.01.org >
>Subject: Re: [SPDK] FIO NVMe Performance Results
>
>Interesting catch. It seems like FIO is actually not accurately reporting the number of jobs in the report out for the SPDK case. If I disable the group_reporting parameter I do get a printout for four separate jobs in the SPDK case, so I think it is really running four jobs there, and just summarizing incorrectly. I also see four separate CPU cores go to 100% when I run that FIO test case, indicating four jobs are active.
>
>Also, I don't think would explain the difference in performance. Using an aggregate queue depth of 32 versus 128 shouldn't lead to a drop in performance from 377K to 13K.
>
>I retested SPDK with multiple separate job descriptions as you suggested anyway, using four separate jobs instead of relying on the numjobs=4 parameter, and the results are below. In that SPDK test with four separate FIO job statements, each with queue depth 32, I still only achieved 12.5K IOPS.
>
>I then re-ran the kernel NVMe driver test case with only a single job for comparison, and even with only 1 job and queue depth 32 it still achieves 254K IOPS.
>
>We might be on the right track with a problem around queue depth however. If I run the kernel NVMe driver case with just 1 job and queue depth of only 1, I then get 11.2K IOPS, which is around the performance I get in the SPDK test case. Perhaps the SPDK case is not actually queuing IOs on the device for some reason?
>
>
>don(a)donst201:~/fio/single/rr$ cat nvme2n1.ini |grep -v '#'
>[global]
>rw=randrw
>rwmixread=100
>numjobs=1
>iodepth=32
>bs=4k
>direct=1
>thread=1
>time_based=1
>ramp_time=0
>runtime=10
>ioengine=libaio
>group_reporting=1
>unified_rw_reporting=1
>exitall=1
>randrepeat=0
>norandommap=1
>[nvme2n1]
>filename=/dev/nvme2n1
>
>
>don(a)donst201:~/fio/single/rr$ sudo fio nvme2n1.ini
>nvme2n1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32
>fio-3.11
>Starting 1 thread
>Jobs: 1 (f=1): [r(1)][100.0%][r=1025MiB/s][r=262k IOPS][eta 00m:00s]
>nvme2n1: (groupid=0, jobs=1): err= 0: pid=2422: Mon Nov 5 05:19:13 2018
> mixed: IOPS=254k, BW=992MiB/s (1040MB/s)(9922MiB/10001msec)
> slat (nsec): min=1185, max=319052, avg=1611.64, stdev=808.40
> clat (usec): min=2, max=4743, avg=124.02, stdev=121.00
> lat (usec): min=6, max=4744, avg=125.68, stdev=121.02
> clat percentiles (usec):
> | 1.00th=[ 9], 5.00th=[ 56], 10.00th=[ 61], 20.00th=[ 73],
> | 30.00th=[ 80], 40.00th=[ 91], 50.00th=[ 110], 60.00th=[ 118],
> | 70.00th=[ 130], 80.00th=[ 157], 90.00th=[ 202], 95.00th=[ 253],
> | 99.00th=[ 408], 99.50th=[ 498], 99.90th=[ 1795], 99.95th=[ 3064],
> | 99.99th=[ 3785]
> bw ( KiB/s): min=612240, max=1078792, per=100.00%, avg=1015996.40, stdev=106032.96, samples=20
> iops : min=153060, max=269698, avg=253998.90, stdev=26508.40, samples=20
> lat (usec) : 4=0.01%, 10=1.47%, 20=1.19%, 50=0.17%, 100=42.03%
> lat (usec) : 250=49.95%, 500=4.70%, 750=0.28%, 1000=0.06%
> lat (msec) : 2=0.06%, 4=0.09%, 10=0.01%
> cpu : usr=21.45%, sys=51.11%, ctx=530348, majf=0, minf=139
> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
> issued rwts: total=2540023,0,0,0 short=0,0,0,0 dropped=0,0,0,0
> latency : target=0, window=0, percentile=100.00%, depth=32
>Run status group 0 (all jobs):
> MIXED: bw=992MiB/s (1040MB/s), 992MiB/s-992MiB/s (1040MB/s-1040MB/s), io=9922MiB (10.4GB), run=10001-10001msec Disk stats (read/write):
> nvme2n1: ios=2514356/0, merge=0/0, ticks=400/0, in_queue=297612, util=99.06%
>
>
>don(a)donst201:~/fio/single/rr$ cat 1c.ini |grep -v '#'
>[global]
>rw=randrw
>rwmixread=100
>numjobs=1
>iodepth=32
>bs=4k
>direct=1
>thread=1
>time_based=1
>ramp_time=0
>runtime=10
>ioengine=/home/don/install/spdk/spdk/examples/nvme/fio_plugin/fio_plugin
>group_reporting=1
>unified_rw_reporting=1
>exitall=1
>randrepeat=0
>norandommap=1
>[0000.1c.00.0-1]
>filename=trtype=PCIe traddr=0000.1c.00.0 ns=1 [0000.1c.00.0-2] filename=trtype=PCIe traddr=0000.1c.00.0 ns=1 [0000.1c.00.0-3] filename=trtype=PCIe traddr=0000.1c.00.0 ns=1 [0000.1c.00.0-4] filename=trtype=PCIe traddr=0000.1c.00.0 ns=1
>
>
>don(a)donst201:~/fio/single/rr$ sudo fio 1c.ini
>0000.1c.00.0-1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk, iodepth=32
>0000.1c.00.0-2: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk, iodepth=32
>0000.1c.00.0-3: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk, iodepth=32
>0000.1c.00.0-4: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk, iodepth=32
>fio-3.11
>Starting 4 threads
>Starting SPDK v18.10 / DPDK 18.08.0 initialization...
>[ DPDK EAL parameters: fio --no-shconf -c 0x1 -m 512 --file-prefix=spdk_pid4270 ]
>EAL: Detected 36 lcore(s)
>EAL: Detected 2 NUMA nodes
>EAL: No free hugepages reported in hugepages-1048576kB
>EAL: Probing VFIO support...
>EAL: PCI device 0000:1c:00.0 on NUMA socket 0
>EAL: probe driver: 8086:a54 spdk_nvme
>Jobs: 1 (f=0): [_(3),f(1)][100.0%][eta 00m:00s]
>0000.1c.00.0-1: (groupid=0, jobs=4): err= 0: pid=4310: Mon Nov 5 05:16:03 2018
> mixed: IOPS=12.5k, BW=48.0MiB/s (51.4MB/s)(490MiB/10010msec)
> slat (nsec): min=98, max=15092, avg=120.02, stdev=90.07
> clat (usec): min=55, max=19757, avg=10207.22, stdev=884.06
> lat (usec): min=60, max=19757, avg=10207.34, stdev=884.06
> clat percentiles (usec):
> | 1.00th=[ 8979], 5.00th=[ 9110], 10.00th=[ 9372], 20.00th=[ 9503],
> | 30.00th=[ 9765], 40.00th=[ 9896], 50.00th=[10159], 60.00th=[10290],
> | 70.00th=[10552], 80.00th=[10814], 90.00th=[11207], 95.00th=[11600],
> | 99.00th=[12911], 99.50th=[14353], 99.90th=[17695], 99.95th=[18744],
> | 99.99th=[19006]
> bw ( KiB/s): min=12008, max=12976, per=25.00%, avg=12535.70, stdev=306.81, samples=80
> iops : min= 3002, max= 3244, avg=3133.85, stdev=76.77, samples=80
> lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
> lat (msec) : 2=0.01%, 4=0.02%, 10=45.13%, 20=54.83%
> cpu : usr=100.01%, sys=0.00%, ctx=27, majf=0, minf=0
> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.9%, >=64=0.0%
> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
> issued rwts: total=125506,0,0,0 short=0,0,0,0 dropped=0,0,0,0
> latency : target=0, window=0, percentile=100.00%, depth=32
>Run status group 0 (all jobs):
> MIXED: bw=48.0MiB/s (51.4MB/s), 48.0MiB/s-48.0MiB/s (51.4MB/s-51.4MB/s), io=490MiB (514MB), run=10010-10010msec
>
>
>don(a)donst201:~/fio/single/rr$ cat nvme2n1.ini |grep -v '#'
>[global]
>rw=randrw
>rwmixread=100
>numjobs=1
>iodepth=1
>bs=4k
>direct=1
>thread=1
>time_based=1
>ramp_time=0
>runtime=10
>ioengine=libaio
>group_reporting=1
>unified_rw_reporting=1
>exitall=1
>randrepeat=0
>norandommap=1
>[nvme2n1]
>filename=/dev/nvme2n1
>
>
>don(a)donst201:~/fio/single/rr$ sudo fio nvme2n1.ini
>nvme2n1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
>fio-3.11
>Starting 1 thread
>Jobs: 1 (f=1): [r(1)][100.0%][r=43.6MiB/s][r=11.2k IOPS][eta 00m:00s]
>nvme2n1: (groupid=0, jobs=1): err= 0: pid=2525: Mon Nov 5 05:22:57 2018
> mixed: IOPS=11.2k, BW=43.6MiB/s (45.7MB/s)(436MiB/10001msec)
> slat (nsec): min=4848, max=208595, avg=5126.33, stdev=771.27
> clat (nsec): min=1845, max=3405.8k, avg=83004.99, stdev=33577.36
> lat (usec): min=15, max=3411, avg=88.26, stdev=33.59
> clat percentiles (usec):
> | 1.00th=[ 15], 5.00th=[ 58], 10.00th=[ 59], 20.00th=[ 59],
> | 30.00th=[ 60], 40.00th=[ 77], 50.00th=[ 77], 60.00th=[ 78],
> | 70.00th=[ 111], 80.00th=[ 113], 90.00th=[ 113], 95.00th=[ 115],
> | 99.00th=[ 186], 99.50th=[ 186], 99.90th=[ 190], 99.95th=[ 192],
> | 99.99th=[ 375]
> bw ( KiB/s): min=43816, max=45016, per=99.98%, avg=44639.58, stdev=312.88, samples=19
> iops : min=10954, max=11254, avg=11159.89, stdev=78.22, samples=19
> lat (usec) : 2=0.01%, 20=2.83%, 50=0.01%, 100=64.57%, 250=32.59%
> lat (usec) : 500=0.01%, 750=0.01%
> lat (msec) : 2=0.01%, 4=0.01%
> cpu : usr=4.02%, sys=10.33%, ctx=111630, majf=0, minf=12
> IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
> issued rwts: total=111631,0,0,0 short=0,0,0,0 dropped=0,0,0,0
> latency : target=0, window=0, percentile=100.00%, depth=1
>Run status group 0 (all jobs):
> MIXED: bw=43.6MiB/s (45.7MB/s), 43.6MiB/s-43.6MiB/s (45.7MB/s-45.7MB/s), io=436MiB (457MB), run=10001-10001msec Disk stats (read/write):
> nvme2n1: ios=110275/0, merge=0/0, ticks=0/0, in_queue=8596, util=86.22%
>
>
>Thanks,
>Joe
>
>> -----Original Message-----
>> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Cao, Gang
>> Sent: Sunday, November 4, 2018 8:40 PM
>> To: Storage Performance Development Kit < spdk(a)lists.01.org >
>> Subject: Re: [SPDK] FIO NVMe Performance Results
>>
>> For the SPDK plugin, from the below result, looks like only 1 job
>> started with
>> 32 queue depth:
>>
>> Jobs: 1 (f=0): [f(1),_(3)][100.0%][eta 00m:00s] <<<<
>>
>> Try to update the FIO configuration file as
>>
>> [job1]
>> filename=trtype=PCIe traddr=0000.1c.00.0 ns=1 [job2]
>> filename=trtype=PCIe
>> traddr=0000.1c.00.0 ns=1 ....
>>
>> The kernel FIO is with 4 jobs and each job of 32 queue depth. For
>> SPDK, you can configure as the same, or just configure two jobs and
>> each job of 64 queue depth. The overall queue depth to the device is same of 128.
>>
>> Thanks,
>> Gang
>>
>> -----Original Message-----
>> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Gruher,
>> Joseph R
>> Sent: Monday, November 5, 2018 10:49 AM
>> To: Storage Performance Development Kit < spdk(a)lists.01.org >
>> Subject: [SPDK] FIO NVMe Performance Results
>>
>> Hi folks-
>>
>> I'm testing SPDK 18.10 on Ubuntu 18.04 with kernel 4.18.16. I used
>> FIO with the kernel NVMe driver to measure the performance of a local
>> (PCIe
>> attached) Intel P4500 NVMe device on a 4KB random read workload and
>> obtained 477K IOPS, roughly in line with the drive spec. Then I
>> tested the same drive with the SPDK FIO plugin and only achieved 13K
>> IOPS. The FIO test files and the results are pasted below. Any ideas
>> where I'm going wrong here?
>>
>> Thanks!
>>
>>
>> don(a)donst201:~/fio/single/rr$ ls
>> /sys/devices/pci0000:17/0000:17:02.0/0000:1c:00.0/nvme/nvme1/
>> address cntlid dev device firmware_rev model nvme2n1 power
>> rescan_controller reset_controller serial state subsysnqn
>> subsystem transport uevent
>>
>>
>> don(a)donst201:~/fio/single/rr$ cat nvme2n1.ini |grep -v '#'
>> [global]
>> rw=randrw
>> rwmixread=100
>> numjobs=4
>> iodepth=32
>> bs=4k
>> direct=1
>> thread=1
>> time_based=1
>> ramp_time=0
>> runtime=10
>> ioengine=libaio
>> group_reporting=1
>> unified_rw_reporting=1
>> exitall=1
>> randrepeat=0
>> norandommap=1
>> [nvme2n1]
>> filename=/dev/nvme2n1
>>
>>
>> don(a)donst201:~/fio/single/rr$ sudo fio nvme2n1.ini
>> nvme2n1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T)
>> 4096B-4096B, ioengine=libaio, iodepth=32 ...
>> fio-3.11
>> Starting 4 threads
>> Jobs: 4 (f=4): [r(4)][100.0%][r=1874MiB/s][r=480k IOPS][eta 00m:00s]
>> nvme2n1: (groupid=0, jobs=4): err= 0: pid=2575: Mon Nov 5 02:26:25 2018
>> mixed: IOPS=477k, BW=1862MiB/s (1952MB/s)(18.2GiB/10001msec)
>> slat (nsec): min=1295, max=333088, avg=2216.35, stdev=927.89
>> clat (nsec): min=506, max=5818.8k, avg=265924.16, stdev=223761.59
>> lat (usec): min=6, max=5821, avg=268.19, stdev=223.76
>> clat percentiles (usec):
>> | 1.00th=[ 13], 5.00th=[ 74], 10.00th=[ 87], 20.00th=[ 115],
>> | 30.00th=[ 139], 40.00th=[ 167], 50.00th=[ 204], 60.00th=[ 247],
>> | 70.00th=[ 306], 80.00th=[ 388], 90.00th=[ 523], 95.00th=[ 660],
>> | 99.00th=[ 988], 99.50th=[ 1156], 99.90th=[ 2343], 99.95th=[ 3195],
>> | 99.99th=[ 4424]
>> bw ( KiB/s): min=448192, max=483104, per=25.00%, avg=476597.10,
>> stdev=6748.78, samples=80
>> iops : min=112048, max=120776, avg=119149.27, stdev=1687.20,
>> samples=80
>> lat (nsec) : 750=0.01%
>> lat (usec) : 10=0.52%, 20=1.25%, 50=0.25%, 100=13.38%, 250=45.13%
>> lat (usec) : 500=28.23%, 750=8.08%, 1000=2.22%
>> lat (msec) : 2=0.82%, 4=0.10%, 10=0.02%
>> cpu : usr=14.08%, sys=36.02%, ctx=1930207, majf=0, minf=697
>> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%,
>> >=64=0.0%
>> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>> >=64=0.0%
>> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%,
>> >=64=0.0%
>> issued rwts: total=4766095,0,0,0 short=0,0,0,0 dropped=0,0,0,0
>> latency : target=0, window=0, percentile=100.00%, depth=32
>> Run status group 0 (all jobs):
>> MIXED: bw=1862MiB/s (1952MB/s), 1862MiB/s-1862MiB/s (1952MB/s-
>> 1952MB/s), io=18.2GiB (19.5GB), run=10001-10001msec Disk stats
>> (read/write):
>> nvme2n1: ios=4709190/0, merge=0/0, ticks=4096/0, in_queue=1388752,
>> util=100.00%
>>
>>
>> don(a)donst201:~/fio/single/rr$ sudo
>> /home/don/install/spdk/spdk/scripts/setup.sh
>> Active mountpoints on /dev/nvme0n1, so not binding PCI dev
>> 0000:03:00.0
>> 0000:1c:00.0 (8086 0a54): nvme -> uio_pci_generic
>> 0000:1d:00.0 (8086 0a54): nvme -> uio_pci_generic
>> 0000:5e:00.0 (8086 0a54): nvme -> uio_pci_generic
>> 0000:5f:00.0 (8086 0a54): nvme -> uio_pci_generic
>> 0000:62:00.0 (8086 0a54): nvme -> uio_pci_generic
>> 0000:63:00.0 (8086 0a54): nvme -> uio_pci_generic
>> 0000:64:00.0 (8086 0a54): nvme -> uio_pci_generic
>> 0000:65:00.0 (8086 0a54): nvme -> uio_pci_generic
>> 0000:da:00.0 (8086 0a54): nvme -> uio_pci_generic
>> 0000:db:00.0 (8086 0a54): nvme -> uio_pci_generic
>> 0000:dc:00.0 (8086 0a54): nvme -> uio_pci_generic
>> 0000:dd:00.0 (8086 0a54): nvme -> uio_pci_generic
>> 0000:e0:00.0 (8086 0a54): nvme -> uio_pci_generic
>> 0000:e1:00.0 (8086 0a54): nvme -> uio_pci_generic
>> 0000:e2:00.0 (8086 0a54): nvme -> uio_pci_generic
>> 0000:e3:00.0 (8086 0a54): nvme -> uio_pci_generic
>> 0000:00:04.0 (8086 2021): ioatdma -> uio_pci_generic
>> 0000:00:04.1 (8086 2021): ioatdma -> uio_pci_generic
>> 0000:00:04.2 (8086 2021): ioatdma -> uio_pci_generic
>> 0000:00:04.3 (8086 2021): ioatdma -> uio_pci_generic
>> 0000:00:04.4 (8086 2021): ioatdma -> uio_pci_generic
>> 0000:00:04.5 (8086 2021): ioatdma -> uio_pci_generic
>> 0000:00:04.6 (8086 2021): ioatdma -> uio_pci_generic
>> 0000:00:04.7 (8086 2021): ioatdma -> uio_pci_generic
>> 0000:80:04.0 (8086 2021): ioatdma -> uio_pci_generic
>> 0000:80:04.1 (8086 2021): ioatdma -> uio_pci_generic
>> 0000:80:04.2 (8086 2021): ioatdma -> uio_pci_generic
>> 0000:80:04.3 (8086 2021): ioatdma -> uio_pci_generic
>> 0000:80:04.4 (8086 2021): ioatdma -> uio_pci_generic
>> 0000:80:04.5 (8086 2021): ioatdma -> uio_pci_generic
>> 0000:80:04.6 (8086 2021): ioatdma -> uio_pci_generic
>> 0000:80:04.7 (8086 2021): ioatdma -> uio_pci_generic
>>
>>
>> don(a)donst201:~/fio/single/rr$ cat 1c.ini |grep -v '#'
>> [global]
>> rw=randrw
>> rwmixread=100
>> numjobs=4
>> iodepth=32
>> bs=4k
>> direct=1
>> thread=1
>> time_based=1
>> ramp_time=0
>> runtime=10
>> ioengine=/home/don/install/spdk/spdk/examples/nvme/fio_plugin/fio_plug
>> in
>> group_reporting=1
>> unified_rw_reporting=1
>> exitall=1
>> randrepeat=0
>> norandommap=1
>> [0000.1c.00.0]
>> filename=trtype=PCIe traddr=0000.1c.00.0 ns=1
>>
>>
>> don(a)donst201:~/fio/single/rr$ sudo fio 1c.ini
>> 0000.1c.00.0: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B,
>> (T) 4096B-4096B, ioengine=spdk, iodepth=32 ...
>> fio-3.11
>> Starting 4 threads
>> Starting SPDK v18.10 / DPDK 18.08.0 initialization...
>> [ DPDK EAL parameters: fio --no-shconf -c 0x1 -m 512 --file-
>> prefix=spdk_pid3668 ]
>> EAL: Detected 36 lcore(s)
>> EAL: Detected 2 NUMA nodes
>> EAL: No free hugepages reported in hugepages-1048576kB
>> EAL: Probing VFIO support...
>> EAL: PCI device 0000:1c:00.0 on NUMA socket 0
>> EAL: probe driver: 8086:a54 spdk_nvme
>> Jobs: 1 (f=0): [f(1),_(3)][100.0%][eta 00m:00s]
>> 0000.1c.00.0: (groupid=0, jobs=4): err= 0: pid=3709: Mon Nov 5
>> 02:28:29
>> 2018
>> mixed: IOPS=13.2k, BW=51.6MiB/s (54.1MB/s)(517MiB/10011msec)
>> slat (nsec): min=109, max=15344, avg=127.91, stdev=73.44
>> clat (usec): min=187, max=18715, avg=9683.80, stdev=650.54
>> lat (usec): min=193, max=18715, avg=9683.93, stdev=650.53
>> clat percentiles (usec):
>> | 1.00th=[ 8717], 5.00th=[ 8848], 10.00th=[ 8979], 20.00th=[ 9110],
>> | 30.00th=[ 9241], 40.00th=[ 9372], 50.00th=[ 9503], 60.00th=[ 9765],
>> | 70.00th=[10028], 80.00th=[10159], 90.00th=[10552], 95.00th=[10945],
>> | 99.00th=[11338], 99.50th=[11338], 99.90th=[11731], 99.95th=[13960],
>> | 99.99th=[17695]
>> bw ( KiB/s): min=13080, max=13304, per=25.00%, avg=13214.70,
>> stdev=55.30, samples=80
>> iops : min= 3270, max= 3326, avg=3303.65, stdev=13.84, samples=80
>> lat (usec) : 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
>> lat (msec) : 2=0.01%, 4=0.02%, 10=70.85%, 20=29.11%
>> cpu : usr=100.01%, sys=0.00%, ctx=23, majf=0, minf=0
>> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.9%,
>> >=64=0.0%
>> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>> >=64=0.0%
>> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%,
>> >=64=0.0%
>> issued rwts: total=132300,0,0,0 short=0,0,0,0 dropped=0,0,0,0
>> latency : target=0, window=0, percentile=100.00%, depth=32
>> Run status group 0 (all jobs):
>> MIXED: bw=51.6MiB/s (54.1MB/s), 51.6MiB/s-51.6MiB/s (54.1MB/s-
>> 54.1MB/s), io=517MiB (542MB), run=10011-10011msec
>>
>> _______________________________________________
>> SPDK mailing list
>> SPDK(a)lists.01.org
>> https://lists.01.org/mailman/listinfo/spdk
>> _______________________________________________
>> SPDK mailing list
>> SPDK(a)lists.01.org
>> https://lists.01.org/mailman/listinfo/spdk
>_______________________________________________
>SPDK mailing list
>SPDK(a)lists.01.org
>https://lists.01.org/mailman/listinfo/spdk
>_______________________________________________
>SPDK mailing list
>SPDK(a)lists.01.org
>https://lists.01.org/mailman/listinfo/spdk
--
Маров Алексей
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [SPDK] FIO NVMe Performance Results
@ 2018-11-05 6:53 Cao, Gang
0 siblings, 0 replies; 25+ messages in thread
From: Cao, Gang @ 2018-11-05 6:53 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 19130 bytes --]
Thanks. This is interesting especially for your finding on queue depth 1 case. We are trying to reproduce this issue locally same as your configuration.
Thanks,
Gang
-----Original Message-----
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Gruher, Joseph R
Sent: Monday, November 5, 2018 1:32 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] FIO NVMe Performance Results
Interesting catch. It seems like FIO is actually not accurately reporting the number of jobs in the report out for the SPDK case. If I disable the group_reporting parameter I do get a printout for four separate jobs in the SPDK case, so I think it is really running four jobs there, and just summarizing incorrectly. I also see four separate CPU cores go to 100% when I run that FIO test case, indicating four jobs are active.
Also, I don't think would explain the difference in performance. Using an aggregate queue depth of 32 versus 128 shouldn't lead to a drop in performance from 377K to 13K.
I retested SPDK with multiple separate job descriptions as you suggested anyway, using four separate jobs instead of relying on the numjobs=4 parameter, and the results are below. In that SPDK test with four separate FIO job statements, each with queue depth 32, I still only achieved 12.5K IOPS.
I then re-ran the kernel NVMe driver test case with only a single job for comparison, and even with only 1 job and queue depth 32 it still achieves 254K IOPS.
We might be on the right track with a problem around queue depth however. If I run the kernel NVMe driver case with just 1 job and queue depth of only 1, I then get 11.2K IOPS, which is around the performance I get in the SPDK test case. Perhaps the SPDK case is not actually queuing IOs on the device for some reason?
don(a)donst201:~/fio/single/rr$ cat nvme2n1.ini |grep -v '#'
[global]
rw=randrw
rwmixread=100
numjobs=1
iodepth=32
bs=4k
direct=1
thread=1
time_based=1
ramp_time=0
runtime=10
ioengine=libaio
group_reporting=1
unified_rw_reporting=1
exitall=1
randrepeat=0
norandommap=1
[nvme2n1]
filename=/dev/nvme2n1
don(a)donst201:~/fio/single/rr$ sudo fio nvme2n1.ini
nvme2n1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32
fio-3.11
Starting 1 thread
Jobs: 1 (f=1): [r(1)][100.0%][r=1025MiB/s][r=262k IOPS][eta 00m:00s]
nvme2n1: (groupid=0, jobs=1): err= 0: pid=2422: Mon Nov 5 05:19:13 2018
mixed: IOPS=254k, BW=992MiB/s (1040MB/s)(9922MiB/10001msec)
slat (nsec): min=1185, max=319052, avg=1611.64, stdev=808.40
clat (usec): min=2, max=4743, avg=124.02, stdev=121.00
lat (usec): min=6, max=4744, avg=125.68, stdev=121.02
clat percentiles (usec):
| 1.00th=[ 9], 5.00th=[ 56], 10.00th=[ 61], 20.00th=[ 73],
| 30.00th=[ 80], 40.00th=[ 91], 50.00th=[ 110], 60.00th=[ 118],
| 70.00th=[ 130], 80.00th=[ 157], 90.00th=[ 202], 95.00th=[ 253],
| 99.00th=[ 408], 99.50th=[ 498], 99.90th=[ 1795], 99.95th=[ 3064],
| 99.99th=[ 3785]
bw ( KiB/s): min=612240, max=1078792, per=100.00%, avg=1015996.40, stdev=106032.96, samples=20
iops : min=153060, max=269698, avg=253998.90, stdev=26508.40, samples=20
lat (usec) : 4=0.01%, 10=1.47%, 20=1.19%, 50=0.17%, 100=42.03%
lat (usec) : 250=49.95%, 500=4.70%, 750=0.28%, 1000=0.06%
lat (msec) : 2=0.06%, 4=0.09%, 10=0.01%
cpu : usr=21.45%, sys=51.11%, ctx=530348, majf=0, minf=139
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued rwts: total=2540023,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
Run status group 0 (all jobs):
MIXED: bw=992MiB/s (1040MB/s), 992MiB/s-992MiB/s (1040MB/s-1040MB/s), io=9922MiB (10.4GB), run=10001-10001msec Disk stats (read/write):
nvme2n1: ios=2514356/0, merge=0/0, ticks=400/0, in_queue=297612, util=99.06%
don(a)donst201:~/fio/single/rr$ cat 1c.ini |grep -v '#'
[global]
rw=randrw
rwmixread=100
numjobs=1
iodepth=32
bs=4k
direct=1
thread=1
time_based=1
ramp_time=0
runtime=10
ioengine=/home/don/install/spdk/spdk/examples/nvme/fio_plugin/fio_plugin
group_reporting=1
unified_rw_reporting=1
exitall=1
randrepeat=0
norandommap=1
[0000.1c.00.0-1]
filename=trtype=PCIe traddr=0000.1c.00.0 ns=1 [0000.1c.00.0-2] filename=trtype=PCIe traddr=0000.1c.00.0 ns=1 [0000.1c.00.0-3] filename=trtype=PCIe traddr=0000.1c.00.0 ns=1 [0000.1c.00.0-4] filename=trtype=PCIe traddr=0000.1c.00.0 ns=1
don(a)donst201:~/fio/single/rr$ sudo fio 1c.ini
0000.1c.00.0-1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk, iodepth=32
0000.1c.00.0-2: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk, iodepth=32
0000.1c.00.0-3: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk, iodepth=32
0000.1c.00.0-4: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk, iodepth=32
fio-3.11
Starting 4 threads
Starting SPDK v18.10 / DPDK 18.08.0 initialization...
[ DPDK EAL parameters: fio --no-shconf -c 0x1 -m 512 --file-prefix=spdk_pid4270 ]
EAL: Detected 36 lcore(s)
EAL: Detected 2 NUMA nodes
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: PCI device 0000:1c:00.0 on NUMA socket 0
EAL: probe driver: 8086:a54 spdk_nvme
Jobs: 1 (f=0): [_(3),f(1)][100.0%][eta 00m:00s]
0000.1c.00.0-1: (groupid=0, jobs=4): err= 0: pid=4310: Mon Nov 5 05:16:03 2018
mixed: IOPS=12.5k, BW=48.0MiB/s (51.4MB/s)(490MiB/10010msec)
slat (nsec): min=98, max=15092, avg=120.02, stdev=90.07
clat (usec): min=55, max=19757, avg=10207.22, stdev=884.06
lat (usec): min=60, max=19757, avg=10207.34, stdev=884.06
clat percentiles (usec):
| 1.00th=[ 8979], 5.00th=[ 9110], 10.00th=[ 9372], 20.00th=[ 9503],
| 30.00th=[ 9765], 40.00th=[ 9896], 50.00th=[10159], 60.00th=[10290],
| 70.00th=[10552], 80.00th=[10814], 90.00th=[11207], 95.00th=[11600],
| 99.00th=[12911], 99.50th=[14353], 99.90th=[17695], 99.95th=[18744],
| 99.99th=[19006]
bw ( KiB/s): min=12008, max=12976, per=25.00%, avg=12535.70, stdev=306.81, samples=80
iops : min= 3002, max= 3244, avg=3133.85, stdev=76.77, samples=80
lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.02%, 10=45.13%, 20=54.83%
cpu : usr=100.01%, sys=0.00%, ctx=27, majf=0, minf=0
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.9%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued rwts: total=125506,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
Run status group 0 (all jobs):
MIXED: bw=48.0MiB/s (51.4MB/s), 48.0MiB/s-48.0MiB/s (51.4MB/s-51.4MB/s), io=490MiB (514MB), run=10010-10010msec
don(a)donst201:~/fio/single/rr$ cat nvme2n1.ini |grep -v '#'
[global]
rw=randrw
rwmixread=100
numjobs=1
iodepth=1
bs=4k
direct=1
thread=1
time_based=1
ramp_time=0
runtime=10
ioengine=libaio
group_reporting=1
unified_rw_reporting=1
exitall=1
randrepeat=0
norandommap=1
[nvme2n1]
filename=/dev/nvme2n1
don(a)donst201:~/fio/single/rr$ sudo fio nvme2n1.ini
nvme2n1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.11
Starting 1 thread
Jobs: 1 (f=1): [r(1)][100.0%][r=43.6MiB/s][r=11.2k IOPS][eta 00m:00s]
nvme2n1: (groupid=0, jobs=1): err= 0: pid=2525: Mon Nov 5 05:22:57 2018
mixed: IOPS=11.2k, BW=43.6MiB/s (45.7MB/s)(436MiB/10001msec)
slat (nsec): min=4848, max=208595, avg=5126.33, stdev=771.27
clat (nsec): min=1845, max=3405.8k, avg=83004.99, stdev=33577.36
lat (usec): min=15, max=3411, avg=88.26, stdev=33.59
clat percentiles (usec):
| 1.00th=[ 15], 5.00th=[ 58], 10.00th=[ 59], 20.00th=[ 59],
| 30.00th=[ 60], 40.00th=[ 77], 50.00th=[ 77], 60.00th=[ 78],
| 70.00th=[ 111], 80.00th=[ 113], 90.00th=[ 113], 95.00th=[ 115],
| 99.00th=[ 186], 99.50th=[ 186], 99.90th=[ 190], 99.95th=[ 192],
| 99.99th=[ 375]
bw ( KiB/s): min=43816, max=45016, per=99.98%, avg=44639.58, stdev=312.88, samples=19
iops : min=10954, max=11254, avg=11159.89, stdev=78.22, samples=19
lat (usec) : 2=0.01%, 20=2.83%, 50=0.01%, 100=64.57%, 250=32.59%
lat (usec) : 500=0.01%, 750=0.01%
lat (msec) : 2=0.01%, 4=0.01%
cpu : usr=4.02%, sys=10.33%, ctx=111630, majf=0, minf=12
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=111631,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
MIXED: bw=43.6MiB/s (45.7MB/s), 43.6MiB/s-43.6MiB/s (45.7MB/s-45.7MB/s), io=436MiB (457MB), run=10001-10001msec Disk stats (read/write):
nvme2n1: ios=110275/0, merge=0/0, ticks=0/0, in_queue=8596, util=86.22%
Thanks,
Joe
> -----Original Message-----
> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Cao, Gang
> Sent: Sunday, November 4, 2018 8:40 PM
> To: Storage Performance Development Kit <spdk(a)lists.01.org>
> Subject: Re: [SPDK] FIO NVMe Performance Results
>
> For the SPDK plugin, from the below result, looks like only 1 job
> started with
> 32 queue depth:
>
> Jobs: 1 (f=0): [f(1),_(3)][100.0%][eta 00m:00s] <<<<
>
> Try to update the FIO configuration file as
>
> [job1]
> filename=trtype=PCIe traddr=0000.1c.00.0 ns=1 [job2]
> filename=trtype=PCIe
> traddr=0000.1c.00.0 ns=1 ....
>
> The kernel FIO is with 4 jobs and each job of 32 queue depth. For
> SPDK, you can configure as the same, or just configure two jobs and
> each job of 64 queue depth. The overall queue depth to the device is same of 128.
>
> Thanks,
> Gang
>
> -----Original Message-----
> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Gruher,
> Joseph R
> Sent: Monday, November 5, 2018 10:49 AM
> To: Storage Performance Development Kit <spdk(a)lists.01.org>
> Subject: [SPDK] FIO NVMe Performance Results
>
> Hi folks-
>
> I'm testing SPDK 18.10 on Ubuntu 18.04 with kernel 4.18.16. I used
> FIO with the kernel NVMe driver to measure the performance of a local
> (PCIe
> attached) Intel P4500 NVMe device on a 4KB random read workload and
> obtained 477K IOPS, roughly in line with the drive spec. Then I
> tested the same drive with the SPDK FIO plugin and only achieved 13K
> IOPS. The FIO test files and the results are pasted below. Any ideas
> where I'm going wrong here?
>
> Thanks!
>
>
> don(a)donst201:~/fio/single/rr$ ls
> /sys/devices/pci0000:17/0000:17:02.0/0000:1c:00.0/nvme/nvme1/
> address cntlid dev device firmware_rev model nvme2n1 power
> rescan_controller reset_controller serial state subsysnqn
> subsystem transport uevent
>
>
> don(a)donst201:~/fio/single/rr$ cat nvme2n1.ini |grep -v '#'
> [global]
> rw=randrw
> rwmixread=100
> numjobs=4
> iodepth=32
> bs=4k
> direct=1
> thread=1
> time_based=1
> ramp_time=0
> runtime=10
> ioengine=libaio
> group_reporting=1
> unified_rw_reporting=1
> exitall=1
> randrepeat=0
> norandommap=1
> [nvme2n1]
> filename=/dev/nvme2n1
>
>
> don(a)donst201:~/fio/single/rr$ sudo fio nvme2n1.ini
> nvme2n1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T)
> 4096B-4096B, ioengine=libaio, iodepth=32 ...
> fio-3.11
> Starting 4 threads
> Jobs: 4 (f=4): [r(4)][100.0%][r=1874MiB/s][r=480k IOPS][eta 00m:00s]
> nvme2n1: (groupid=0, jobs=4): err= 0: pid=2575: Mon Nov 5 02:26:25 2018
> mixed: IOPS=477k, BW=1862MiB/s (1952MB/s)(18.2GiB/10001msec)
> slat (nsec): min=1295, max=333088, avg=2216.35, stdev=927.89
> clat (nsec): min=506, max=5818.8k, avg=265924.16, stdev=223761.59
> lat (usec): min=6, max=5821, avg=268.19, stdev=223.76
> clat percentiles (usec):
> | 1.00th=[ 13], 5.00th=[ 74], 10.00th=[ 87], 20.00th=[ 115],
> | 30.00th=[ 139], 40.00th=[ 167], 50.00th=[ 204], 60.00th=[ 247],
> | 70.00th=[ 306], 80.00th=[ 388], 90.00th=[ 523], 95.00th=[ 660],
> | 99.00th=[ 988], 99.50th=[ 1156], 99.90th=[ 2343], 99.95th=[ 3195],
> | 99.99th=[ 4424]
> bw ( KiB/s): min=448192, max=483104, per=25.00%, avg=476597.10,
> stdev=6748.78, samples=80
> iops : min=112048, max=120776, avg=119149.27, stdev=1687.20,
> samples=80
> lat (nsec) : 750=0.01%
> lat (usec) : 10=0.52%, 20=1.25%, 50=0.25%, 100=13.38%, 250=45.13%
> lat (usec) : 500=28.23%, 750=8.08%, 1000=2.22%
> lat (msec) : 2=0.82%, 4=0.10%, 10=0.02%
> cpu : usr=14.08%, sys=36.02%, ctx=1930207, majf=0, minf=697
> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%,
> >=64=0.0%
> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
> >=64=0.0%
> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%,
> >=64=0.0%
> issued rwts: total=4766095,0,0,0 short=0,0,0,0 dropped=0,0,0,0
> latency : target=0, window=0, percentile=100.00%, depth=32
> Run status group 0 (all jobs):
> MIXED: bw=1862MiB/s (1952MB/s), 1862MiB/s-1862MiB/s (1952MB/s-
> 1952MB/s), io=18.2GiB (19.5GB), run=10001-10001msec Disk stats
> (read/write):
> nvme2n1: ios=4709190/0, merge=0/0, ticks=4096/0, in_queue=1388752,
> util=100.00%
>
>
> don(a)donst201:~/fio/single/rr$ sudo
> /home/don/install/spdk/spdk/scripts/setup.sh
> Active mountpoints on /dev/nvme0n1, so not binding PCI dev
> 0000:03:00.0
> 0000:1c:00.0 (8086 0a54): nvme -> uio_pci_generic
> 0000:1d:00.0 (8086 0a54): nvme -> uio_pci_generic
> 0000:5e:00.0 (8086 0a54): nvme -> uio_pci_generic
> 0000:5f:00.0 (8086 0a54): nvme -> uio_pci_generic
> 0000:62:00.0 (8086 0a54): nvme -> uio_pci_generic
> 0000:63:00.0 (8086 0a54): nvme -> uio_pci_generic
> 0000:64:00.0 (8086 0a54): nvme -> uio_pci_generic
> 0000:65:00.0 (8086 0a54): nvme -> uio_pci_generic
> 0000:da:00.0 (8086 0a54): nvme -> uio_pci_generic
> 0000:db:00.0 (8086 0a54): nvme -> uio_pci_generic
> 0000:dc:00.0 (8086 0a54): nvme -> uio_pci_generic
> 0000:dd:00.0 (8086 0a54): nvme -> uio_pci_generic
> 0000:e0:00.0 (8086 0a54): nvme -> uio_pci_generic
> 0000:e1:00.0 (8086 0a54): nvme -> uio_pci_generic
> 0000:e2:00.0 (8086 0a54): nvme -> uio_pci_generic
> 0000:e3:00.0 (8086 0a54): nvme -> uio_pci_generic
> 0000:00:04.0 (8086 2021): ioatdma -> uio_pci_generic
> 0000:00:04.1 (8086 2021): ioatdma -> uio_pci_generic
> 0000:00:04.2 (8086 2021): ioatdma -> uio_pci_generic
> 0000:00:04.3 (8086 2021): ioatdma -> uio_pci_generic
> 0000:00:04.4 (8086 2021): ioatdma -> uio_pci_generic
> 0000:00:04.5 (8086 2021): ioatdma -> uio_pci_generic
> 0000:00:04.6 (8086 2021): ioatdma -> uio_pci_generic
> 0000:00:04.7 (8086 2021): ioatdma -> uio_pci_generic
> 0000:80:04.0 (8086 2021): ioatdma -> uio_pci_generic
> 0000:80:04.1 (8086 2021): ioatdma -> uio_pci_generic
> 0000:80:04.2 (8086 2021): ioatdma -> uio_pci_generic
> 0000:80:04.3 (8086 2021): ioatdma -> uio_pci_generic
> 0000:80:04.4 (8086 2021): ioatdma -> uio_pci_generic
> 0000:80:04.5 (8086 2021): ioatdma -> uio_pci_generic
> 0000:80:04.6 (8086 2021): ioatdma -> uio_pci_generic
> 0000:80:04.7 (8086 2021): ioatdma -> uio_pci_generic
>
>
> don(a)donst201:~/fio/single/rr$ cat 1c.ini |grep -v '#'
> [global]
> rw=randrw
> rwmixread=100
> numjobs=4
> iodepth=32
> bs=4k
> direct=1
> thread=1
> time_based=1
> ramp_time=0
> runtime=10
> ioengine=/home/don/install/spdk/spdk/examples/nvme/fio_plugin/fio_plug
> in
> group_reporting=1
> unified_rw_reporting=1
> exitall=1
> randrepeat=0
> norandommap=1
> [0000.1c.00.0]
> filename=trtype=PCIe traddr=0000.1c.00.0 ns=1
>
>
> don(a)donst201:~/fio/single/rr$ sudo fio 1c.ini
> 0000.1c.00.0: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B,
> (T) 4096B-4096B, ioengine=spdk, iodepth=32 ...
> fio-3.11
> Starting 4 threads
> Starting SPDK v18.10 / DPDK 18.08.0 initialization...
> [ DPDK EAL parameters: fio --no-shconf -c 0x1 -m 512 --file-
> prefix=spdk_pid3668 ]
> EAL: Detected 36 lcore(s)
> EAL: Detected 2 NUMA nodes
> EAL: No free hugepages reported in hugepages-1048576kB
> EAL: Probing VFIO support...
> EAL: PCI device 0000:1c:00.0 on NUMA socket 0
> EAL: probe driver: 8086:a54 spdk_nvme
> Jobs: 1 (f=0): [f(1),_(3)][100.0%][eta 00m:00s]
> 0000.1c.00.0: (groupid=0, jobs=4): err= 0: pid=3709: Mon Nov 5
> 02:28:29
> 2018
> mixed: IOPS=13.2k, BW=51.6MiB/s (54.1MB/s)(517MiB/10011msec)
> slat (nsec): min=109, max=15344, avg=127.91, stdev=73.44
> clat (usec): min=187, max=18715, avg=9683.80, stdev=650.54
> lat (usec): min=193, max=18715, avg=9683.93, stdev=650.53
> clat percentiles (usec):
> | 1.00th=[ 8717], 5.00th=[ 8848], 10.00th=[ 8979], 20.00th=[ 9110],
> | 30.00th=[ 9241], 40.00th=[ 9372], 50.00th=[ 9503], 60.00th=[ 9765],
> | 70.00th=[10028], 80.00th=[10159], 90.00th=[10552], 95.00th=[10945],
> | 99.00th=[11338], 99.50th=[11338], 99.90th=[11731], 99.95th=[13960],
> | 99.99th=[17695]
> bw ( KiB/s): min=13080, max=13304, per=25.00%, avg=13214.70,
> stdev=55.30, samples=80
> iops : min= 3270, max= 3326, avg=3303.65, stdev=13.84, samples=80
> lat (usec) : 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
> lat (msec) : 2=0.01%, 4=0.02%, 10=70.85%, 20=29.11%
> cpu : usr=100.01%, sys=0.00%, ctx=23, majf=0, minf=0
> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.9%,
> >=64=0.0%
> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
> >=64=0.0%
> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%,
> >=64=0.0%
> issued rwts: total=132300,0,0,0 short=0,0,0,0 dropped=0,0,0,0
> latency : target=0, window=0, percentile=100.00%, depth=32
> Run status group 0 (all jobs):
> MIXED: bw=51.6MiB/s (54.1MB/s), 51.6MiB/s-51.6MiB/s (54.1MB/s-
> 54.1MB/s), io=517MiB (542MB), run=10011-10011msec
>
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [SPDK] FIO NVMe Performance Results
@ 2018-11-05 5:32 Gruher, Joseph R
0 siblings, 0 replies; 25+ messages in thread
From: Gruher, Joseph R @ 2018-11-05 5:32 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 18533 bytes --]
Interesting catch. It seems like FIO is actually not accurately reporting the number of jobs in the report out for the SPDK case. If I disable the group_reporting parameter I do get a printout for four separate jobs in the SPDK case, so I think it is really running four jobs there, and just summarizing incorrectly. I also see four separate CPU cores go to 100% when I run that FIO test case, indicating four jobs are active.
Also, I don't think would explain the difference in performance. Using an aggregate queue depth of 32 versus 128 shouldn't lead to a drop in performance from 377K to 13K.
I retested SPDK with multiple separate job descriptions as you suggested anyway, using four separate jobs instead of relying on the numjobs=4 parameter, and the results are below. In that SPDK test with four separate FIO job statements, each with queue depth 32, I still only achieved 12.5K IOPS.
I then re-ran the kernel NVMe driver test case with only a single job for comparison, and even with only 1 job and queue depth 32 it still achieves 254K IOPS.
We might be on the right track with a problem around queue depth however. If I run the kernel NVMe driver case with just 1 job and queue depth of only 1, I then get 11.2K IOPS, which is around the performance I get in the SPDK test case. Perhaps the SPDK case is not actually queuing IOs on the device for some reason?
don(a)donst201:~/fio/single/rr$ cat nvme2n1.ini |grep -v '#'
[global]
rw=randrw
rwmixread=100
numjobs=1
iodepth=32
bs=4k
direct=1
thread=1
time_based=1
ramp_time=0
runtime=10
ioengine=libaio
group_reporting=1
unified_rw_reporting=1
exitall=1
randrepeat=0
norandommap=1
[nvme2n1]
filename=/dev/nvme2n1
don(a)donst201:~/fio/single/rr$ sudo fio nvme2n1.ini
nvme2n1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32
fio-3.11
Starting 1 thread
Jobs: 1 (f=1): [r(1)][100.0%][r=1025MiB/s][r=262k IOPS][eta 00m:00s]
nvme2n1: (groupid=0, jobs=1): err= 0: pid=2422: Mon Nov 5 05:19:13 2018
mixed: IOPS=254k, BW=992MiB/s (1040MB/s)(9922MiB/10001msec)
slat (nsec): min=1185, max=319052, avg=1611.64, stdev=808.40
clat (usec): min=2, max=4743, avg=124.02, stdev=121.00
lat (usec): min=6, max=4744, avg=125.68, stdev=121.02
clat percentiles (usec):
| 1.00th=[ 9], 5.00th=[ 56], 10.00th=[ 61], 20.00th=[ 73],
| 30.00th=[ 80], 40.00th=[ 91], 50.00th=[ 110], 60.00th=[ 118],
| 70.00th=[ 130], 80.00th=[ 157], 90.00th=[ 202], 95.00th=[ 253],
| 99.00th=[ 408], 99.50th=[ 498], 99.90th=[ 1795], 99.95th=[ 3064],
| 99.99th=[ 3785]
bw ( KiB/s): min=612240, max=1078792, per=100.00%, avg=1015996.40, stdev=106032.96, samples=20
iops : min=153060, max=269698, avg=253998.90, stdev=26508.40, samples=20
lat (usec) : 4=0.01%, 10=1.47%, 20=1.19%, 50=0.17%, 100=42.03%
lat (usec) : 250=49.95%, 500=4.70%, 750=0.28%, 1000=0.06%
lat (msec) : 2=0.06%, 4=0.09%, 10=0.01%
cpu : usr=21.45%, sys=51.11%, ctx=530348, majf=0, minf=139
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued rwts: total=2540023,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
Run status group 0 (all jobs):
MIXED: bw=992MiB/s (1040MB/s), 992MiB/s-992MiB/s (1040MB/s-1040MB/s), io=9922MiB (10.4GB), run=10001-10001msec
Disk stats (read/write):
nvme2n1: ios=2514356/0, merge=0/0, ticks=400/0, in_queue=297612, util=99.06%
don(a)donst201:~/fio/single/rr$ cat 1c.ini |grep -v '#'
[global]
rw=randrw
rwmixread=100
numjobs=1
iodepth=32
bs=4k
direct=1
thread=1
time_based=1
ramp_time=0
runtime=10
ioengine=/home/don/install/spdk/spdk/examples/nvme/fio_plugin/fio_plugin
group_reporting=1
unified_rw_reporting=1
exitall=1
randrepeat=0
norandommap=1
[0000.1c.00.0-1]
filename=trtype=PCIe traddr=0000.1c.00.0 ns=1
[0000.1c.00.0-2]
filename=trtype=PCIe traddr=0000.1c.00.0 ns=1
[0000.1c.00.0-3]
filename=trtype=PCIe traddr=0000.1c.00.0 ns=1
[0000.1c.00.0-4]
filename=trtype=PCIe traddr=0000.1c.00.0 ns=1
don(a)donst201:~/fio/single/rr$ sudo fio 1c.ini
0000.1c.00.0-1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk, iodepth=32
0000.1c.00.0-2: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk, iodepth=32
0000.1c.00.0-3: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk, iodepth=32
0000.1c.00.0-4: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk, iodepth=32
fio-3.11
Starting 4 threads
Starting SPDK v18.10 / DPDK 18.08.0 initialization...
[ DPDK EAL parameters: fio --no-shconf -c 0x1 -m 512 --file-prefix=spdk_pid4270 ]
EAL: Detected 36 lcore(s)
EAL: Detected 2 NUMA nodes
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: PCI device 0000:1c:00.0 on NUMA socket 0
EAL: probe driver: 8086:a54 spdk_nvme
Jobs: 1 (f=0): [_(3),f(1)][100.0%][eta 00m:00s]
0000.1c.00.0-1: (groupid=0, jobs=4): err= 0: pid=4310: Mon Nov 5 05:16:03 2018
mixed: IOPS=12.5k, BW=48.0MiB/s (51.4MB/s)(490MiB/10010msec)
slat (nsec): min=98, max=15092, avg=120.02, stdev=90.07
clat (usec): min=55, max=19757, avg=10207.22, stdev=884.06
lat (usec): min=60, max=19757, avg=10207.34, stdev=884.06
clat percentiles (usec):
| 1.00th=[ 8979], 5.00th=[ 9110], 10.00th=[ 9372], 20.00th=[ 9503],
| 30.00th=[ 9765], 40.00th=[ 9896], 50.00th=[10159], 60.00th=[10290],
| 70.00th=[10552], 80.00th=[10814], 90.00th=[11207], 95.00th=[11600],
| 99.00th=[12911], 99.50th=[14353], 99.90th=[17695], 99.95th=[18744],
| 99.99th=[19006]
bw ( KiB/s): min=12008, max=12976, per=25.00%, avg=12535.70, stdev=306.81, samples=80
iops : min= 3002, max= 3244, avg=3133.85, stdev=76.77, samples=80
lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.02%, 10=45.13%, 20=54.83%
cpu : usr=100.01%, sys=0.00%, ctx=27, majf=0, minf=0
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.9%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued rwts: total=125506,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
Run status group 0 (all jobs):
MIXED: bw=48.0MiB/s (51.4MB/s), 48.0MiB/s-48.0MiB/s (51.4MB/s-51.4MB/s), io=490MiB (514MB), run=10010-10010msec
don(a)donst201:~/fio/single/rr$ cat nvme2n1.ini |grep -v '#'
[global]
rw=randrw
rwmixread=100
numjobs=1
iodepth=1
bs=4k
direct=1
thread=1
time_based=1
ramp_time=0
runtime=10
ioengine=libaio
group_reporting=1
unified_rw_reporting=1
exitall=1
randrepeat=0
norandommap=1
[nvme2n1]
filename=/dev/nvme2n1
don(a)donst201:~/fio/single/rr$ sudo fio nvme2n1.ini
nvme2n1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.11
Starting 1 thread
Jobs: 1 (f=1): [r(1)][100.0%][r=43.6MiB/s][r=11.2k IOPS][eta 00m:00s]
nvme2n1: (groupid=0, jobs=1): err= 0: pid=2525: Mon Nov 5 05:22:57 2018
mixed: IOPS=11.2k, BW=43.6MiB/s (45.7MB/s)(436MiB/10001msec)
slat (nsec): min=4848, max=208595, avg=5126.33, stdev=771.27
clat (nsec): min=1845, max=3405.8k, avg=83004.99, stdev=33577.36
lat (usec): min=15, max=3411, avg=88.26, stdev=33.59
clat percentiles (usec):
| 1.00th=[ 15], 5.00th=[ 58], 10.00th=[ 59], 20.00th=[ 59],
| 30.00th=[ 60], 40.00th=[ 77], 50.00th=[ 77], 60.00th=[ 78],
| 70.00th=[ 111], 80.00th=[ 113], 90.00th=[ 113], 95.00th=[ 115],
| 99.00th=[ 186], 99.50th=[ 186], 99.90th=[ 190], 99.95th=[ 192],
| 99.99th=[ 375]
bw ( KiB/s): min=43816, max=45016, per=99.98%, avg=44639.58, stdev=312.88, samples=19
iops : min=10954, max=11254, avg=11159.89, stdev=78.22, samples=19
lat (usec) : 2=0.01%, 20=2.83%, 50=0.01%, 100=64.57%, 250=32.59%
lat (usec) : 500=0.01%, 750=0.01%
lat (msec) : 2=0.01%, 4=0.01%
cpu : usr=4.02%, sys=10.33%, ctx=111630, majf=0, minf=12
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=111631,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
MIXED: bw=43.6MiB/s (45.7MB/s), 43.6MiB/s-43.6MiB/s (45.7MB/s-45.7MB/s), io=436MiB (457MB), run=10001-10001msec
Disk stats (read/write):
nvme2n1: ios=110275/0, merge=0/0, ticks=0/0, in_queue=8596, util=86.22%
Thanks,
Joe
> -----Original Message-----
> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Cao, Gang
> Sent: Sunday, November 4, 2018 8:40 PM
> To: Storage Performance Development Kit <spdk(a)lists.01.org>
> Subject: Re: [SPDK] FIO NVMe Performance Results
>
> For the SPDK plugin, from the below result, looks like only 1 job started with
> 32 queue depth:
>
> Jobs: 1 (f=0): [f(1),_(3)][100.0%][eta 00m:00s] <<<<
>
> Try to update the FIO configuration file as
>
> [job1]
> filename=trtype=PCIe traddr=0000.1c.00.0 ns=1 [job2] filename=trtype=PCIe
> traddr=0000.1c.00.0 ns=1 ....
>
> The kernel FIO is with 4 jobs and each job of 32 queue depth. For SPDK, you
> can configure as the same, or just configure two jobs and each job of 64
> queue depth. The overall queue depth to the device is same of 128.
>
> Thanks,
> Gang
>
> -----Original Message-----
> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Gruher, Joseph
> R
> Sent: Monday, November 5, 2018 10:49 AM
> To: Storage Performance Development Kit <spdk(a)lists.01.org>
> Subject: [SPDK] FIO NVMe Performance Results
>
> Hi folks-
>
> I'm testing SPDK 18.10 on Ubuntu 18.04 with kernel 4.18.16. I used FIO with
> the kernel NVMe driver to measure the performance of a local (PCIe
> attached) Intel P4500 NVMe device on a 4KB random read workload and
> obtained 477K IOPS, roughly in line with the drive spec. Then I tested the
> same drive with the SPDK FIO plugin and only achieved 13K IOPS. The FIO
> test files and the results are pasted below. Any ideas where I'm going wrong
> here?
>
> Thanks!
>
>
> don(a)donst201:~/fio/single/rr$ ls
> /sys/devices/pci0000:17/0000:17:02.0/0000:1c:00.0/nvme/nvme1/
> address cntlid dev device firmware_rev model nvme2n1 power
> rescan_controller reset_controller serial state subsysnqn subsystem
> transport uevent
>
>
> don(a)donst201:~/fio/single/rr$ cat nvme2n1.ini |grep -v '#'
> [global]
> rw=randrw
> rwmixread=100
> numjobs=4
> iodepth=32
> bs=4k
> direct=1
> thread=1
> time_based=1
> ramp_time=0
> runtime=10
> ioengine=libaio
> group_reporting=1
> unified_rw_reporting=1
> exitall=1
> randrepeat=0
> norandommap=1
> [nvme2n1]
> filename=/dev/nvme2n1
>
>
> don(a)donst201:~/fio/single/rr$ sudo fio nvme2n1.ini
> nvme2n1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T)
> 4096B-4096B, ioengine=libaio, iodepth=32 ...
> fio-3.11
> Starting 4 threads
> Jobs: 4 (f=4): [r(4)][100.0%][r=1874MiB/s][r=480k IOPS][eta 00m:00s]
> nvme2n1: (groupid=0, jobs=4): err= 0: pid=2575: Mon Nov 5 02:26:25 2018
> mixed: IOPS=477k, BW=1862MiB/s (1952MB/s)(18.2GiB/10001msec)
> slat (nsec): min=1295, max=333088, avg=2216.35, stdev=927.89
> clat (nsec): min=506, max=5818.8k, avg=265924.16, stdev=223761.59
> lat (usec): min=6, max=5821, avg=268.19, stdev=223.76
> clat percentiles (usec):
> | 1.00th=[ 13], 5.00th=[ 74], 10.00th=[ 87], 20.00th=[ 115],
> | 30.00th=[ 139], 40.00th=[ 167], 50.00th=[ 204], 60.00th=[ 247],
> | 70.00th=[ 306], 80.00th=[ 388], 90.00th=[ 523], 95.00th=[ 660],
> | 99.00th=[ 988], 99.50th=[ 1156], 99.90th=[ 2343], 99.95th=[ 3195],
> | 99.99th=[ 4424]
> bw ( KiB/s): min=448192, max=483104, per=25.00%, avg=476597.10,
> stdev=6748.78, samples=80
> iops : min=112048, max=120776, avg=119149.27, stdev=1687.20,
> samples=80
> lat (nsec) : 750=0.01%
> lat (usec) : 10=0.52%, 20=1.25%, 50=0.25%, 100=13.38%, 250=45.13%
> lat (usec) : 500=28.23%, 750=8.08%, 1000=2.22%
> lat (msec) : 2=0.82%, 4=0.10%, 10=0.02%
> cpu : usr=14.08%, sys=36.02%, ctx=1930207, majf=0, minf=697
> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%,
> >=64=0.0%
> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
> >=64=0.0%
> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%,
> >=64=0.0%
> issued rwts: total=4766095,0,0,0 short=0,0,0,0 dropped=0,0,0,0
> latency : target=0, window=0, percentile=100.00%, depth=32
> Run status group 0 (all jobs):
> MIXED: bw=1862MiB/s (1952MB/s), 1862MiB/s-1862MiB/s (1952MB/s-
> 1952MB/s), io=18.2GiB (19.5GB), run=10001-10001msec Disk stats
> (read/write):
> nvme2n1: ios=4709190/0, merge=0/0, ticks=4096/0, in_queue=1388752,
> util=100.00%
>
>
> don(a)donst201:~/fio/single/rr$ sudo
> /home/don/install/spdk/spdk/scripts/setup.sh
> Active mountpoints on /dev/nvme0n1, so not binding PCI dev 0000:03:00.0
> 0000:1c:00.0 (8086 0a54): nvme -> uio_pci_generic
> 0000:1d:00.0 (8086 0a54): nvme -> uio_pci_generic
> 0000:5e:00.0 (8086 0a54): nvme -> uio_pci_generic
> 0000:5f:00.0 (8086 0a54): nvme -> uio_pci_generic
> 0000:62:00.0 (8086 0a54): nvme -> uio_pci_generic
> 0000:63:00.0 (8086 0a54): nvme -> uio_pci_generic
> 0000:64:00.0 (8086 0a54): nvme -> uio_pci_generic
> 0000:65:00.0 (8086 0a54): nvme -> uio_pci_generic
> 0000:da:00.0 (8086 0a54): nvme -> uio_pci_generic
> 0000:db:00.0 (8086 0a54): nvme -> uio_pci_generic
> 0000:dc:00.0 (8086 0a54): nvme -> uio_pci_generic
> 0000:dd:00.0 (8086 0a54): nvme -> uio_pci_generic
> 0000:e0:00.0 (8086 0a54): nvme -> uio_pci_generic
> 0000:e1:00.0 (8086 0a54): nvme -> uio_pci_generic
> 0000:e2:00.0 (8086 0a54): nvme -> uio_pci_generic
> 0000:e3:00.0 (8086 0a54): nvme -> uio_pci_generic
> 0000:00:04.0 (8086 2021): ioatdma -> uio_pci_generic
> 0000:00:04.1 (8086 2021): ioatdma -> uio_pci_generic
> 0000:00:04.2 (8086 2021): ioatdma -> uio_pci_generic
> 0000:00:04.3 (8086 2021): ioatdma -> uio_pci_generic
> 0000:00:04.4 (8086 2021): ioatdma -> uio_pci_generic
> 0000:00:04.5 (8086 2021): ioatdma -> uio_pci_generic
> 0000:00:04.6 (8086 2021): ioatdma -> uio_pci_generic
> 0000:00:04.7 (8086 2021): ioatdma -> uio_pci_generic
> 0000:80:04.0 (8086 2021): ioatdma -> uio_pci_generic
> 0000:80:04.1 (8086 2021): ioatdma -> uio_pci_generic
> 0000:80:04.2 (8086 2021): ioatdma -> uio_pci_generic
> 0000:80:04.3 (8086 2021): ioatdma -> uio_pci_generic
> 0000:80:04.4 (8086 2021): ioatdma -> uio_pci_generic
> 0000:80:04.5 (8086 2021): ioatdma -> uio_pci_generic
> 0000:80:04.6 (8086 2021): ioatdma -> uio_pci_generic
> 0000:80:04.7 (8086 2021): ioatdma -> uio_pci_generic
>
>
> don(a)donst201:~/fio/single/rr$ cat 1c.ini |grep -v '#'
> [global]
> rw=randrw
> rwmixread=100
> numjobs=4
> iodepth=32
> bs=4k
> direct=1
> thread=1
> time_based=1
> ramp_time=0
> runtime=10
> ioengine=/home/don/install/spdk/spdk/examples/nvme/fio_plugin/fio_plug
> in
> group_reporting=1
> unified_rw_reporting=1
> exitall=1
> randrepeat=0
> norandommap=1
> [0000.1c.00.0]
> filename=trtype=PCIe traddr=0000.1c.00.0 ns=1
>
>
> don(a)donst201:~/fio/single/rr$ sudo fio 1c.ini
> 0000.1c.00.0: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T)
> 4096B-4096B, ioengine=spdk, iodepth=32 ...
> fio-3.11
> Starting 4 threads
> Starting SPDK v18.10 / DPDK 18.08.0 initialization...
> [ DPDK EAL parameters: fio --no-shconf -c 0x1 -m 512 --file-
> prefix=spdk_pid3668 ]
> EAL: Detected 36 lcore(s)
> EAL: Detected 2 NUMA nodes
> EAL: No free hugepages reported in hugepages-1048576kB
> EAL: Probing VFIO support...
> EAL: PCI device 0000:1c:00.0 on NUMA socket 0
> EAL: probe driver: 8086:a54 spdk_nvme
> Jobs: 1 (f=0): [f(1),_(3)][100.0%][eta 00m:00s]
> 0000.1c.00.0: (groupid=0, jobs=4): err= 0: pid=3709: Mon Nov 5 02:28:29
> 2018
> mixed: IOPS=13.2k, BW=51.6MiB/s (54.1MB/s)(517MiB/10011msec)
> slat (nsec): min=109, max=15344, avg=127.91, stdev=73.44
> clat (usec): min=187, max=18715, avg=9683.80, stdev=650.54
> lat (usec): min=193, max=18715, avg=9683.93, stdev=650.53
> clat percentiles (usec):
> | 1.00th=[ 8717], 5.00th=[ 8848], 10.00th=[ 8979], 20.00th=[ 9110],
> | 30.00th=[ 9241], 40.00th=[ 9372], 50.00th=[ 9503], 60.00th=[ 9765],
> | 70.00th=[10028], 80.00th=[10159], 90.00th=[10552], 95.00th=[10945],
> | 99.00th=[11338], 99.50th=[11338], 99.90th=[11731], 99.95th=[13960],
> | 99.99th=[17695]
> bw ( KiB/s): min=13080, max=13304, per=25.00%, avg=13214.70,
> stdev=55.30, samples=80
> iops : min= 3270, max= 3326, avg=3303.65, stdev=13.84, samples=80
> lat (usec) : 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
> lat (msec) : 2=0.01%, 4=0.02%, 10=70.85%, 20=29.11%
> cpu : usr=100.01%, sys=0.00%, ctx=23, majf=0, minf=0
> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.9%,
> >=64=0.0%
> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
> >=64=0.0%
> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%,
> >=64=0.0%
> issued rwts: total=132300,0,0,0 short=0,0,0,0 dropped=0,0,0,0
> latency : target=0, window=0, percentile=100.00%, depth=32
> Run status group 0 (all jobs):
> MIXED: bw=51.6MiB/s (54.1MB/s), 51.6MiB/s-51.6MiB/s (54.1MB/s-
> 54.1MB/s), io=517MiB (542MB), run=10011-10011msec
>
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
^ permalink raw reply [flat|nested] 25+ messages in thread
* [SPDK] FIO NVMe Performance Results
@ 2018-11-05 2:49 Gruher, Joseph R
0 siblings, 0 replies; 25+ messages in thread
From: Gruher, Joseph R @ 2018-11-05 2:49 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 7534 bytes --]
Hi folks-
I'm testing SPDK 18.10 on Ubuntu 18.04 with kernel 4.18.16. I used FIO with the kernel NVMe driver to measure the performance of a local (PCIe attached) Intel P4500 NVMe device on a 4KB random read workload and obtained 477K IOPS, roughly in line with the drive spec. Then I tested the same drive with the SPDK FIO plugin and only achieved 13K IOPS. The FIO test files and the results are pasted below. Any ideas where I'm going wrong here?
Thanks!
don(a)donst201:~/fio/single/rr$ ls /sys/devices/pci0000:17/0000:17:02.0/0000:1c:00.0/nvme/nvme1/
address cntlid dev device firmware_rev model nvme2n1 power rescan_controller reset_controller serial state subsysnqn subsystem transport uevent
don(a)donst201:~/fio/single/rr$ cat nvme2n1.ini |grep -v '#'
[global]
rw=randrw
rwmixread=100
numjobs=4
iodepth=32
bs=4k
direct=1
thread=1
time_based=1
ramp_time=0
runtime=10
ioengine=libaio
group_reporting=1
unified_rw_reporting=1
exitall=1
randrepeat=0
norandommap=1
[nvme2n1]
filename=/dev/nvme2n1
don(a)donst201:~/fio/single/rr$ sudo fio nvme2n1.ini
nvme2n1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32
...
fio-3.11
Starting 4 threads
Jobs: 4 (f=4): [r(4)][100.0%][r=1874MiB/s][r=480k IOPS][eta 00m:00s]
nvme2n1: (groupid=0, jobs=4): err= 0: pid=2575: Mon Nov 5 02:26:25 2018
mixed: IOPS=477k, BW=1862MiB/s (1952MB/s)(18.2GiB/10001msec)
slat (nsec): min=1295, max=333088, avg=2216.35, stdev=927.89
clat (nsec): min=506, max=5818.8k, avg=265924.16, stdev=223761.59
lat (usec): min=6, max=5821, avg=268.19, stdev=223.76
clat percentiles (usec):
| 1.00th=[ 13], 5.00th=[ 74], 10.00th=[ 87], 20.00th=[ 115],
| 30.00th=[ 139], 40.00th=[ 167], 50.00th=[ 204], 60.00th=[ 247],
| 70.00th=[ 306], 80.00th=[ 388], 90.00th=[ 523], 95.00th=[ 660],
| 99.00th=[ 988], 99.50th=[ 1156], 99.90th=[ 2343], 99.95th=[ 3195],
| 99.99th=[ 4424]
bw ( KiB/s): min=448192, max=483104, per=25.00%, avg=476597.10, stdev=6748.78, samples=80
iops : min=112048, max=120776, avg=119149.27, stdev=1687.20, samples=80
lat (nsec) : 750=0.01%
lat (usec) : 10=0.52%, 20=1.25%, 50=0.25%, 100=13.38%, 250=45.13%
lat (usec) : 500=28.23%, 750=8.08%, 1000=2.22%
lat (msec) : 2=0.82%, 4=0.10%, 10=0.02%
cpu : usr=14.08%, sys=36.02%, ctx=1930207, majf=0, minf=697
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued rwts: total=4766095,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
Run status group 0 (all jobs):
MIXED: bw=1862MiB/s (1952MB/s), 1862MiB/s-1862MiB/s (1952MB/s-1952MB/s), io=18.2GiB (19.5GB), run=10001-10001msec
Disk stats (read/write):
nvme2n1: ios=4709190/0, merge=0/0, ticks=4096/0, in_queue=1388752, util=100.00%
don(a)donst201:~/fio/single/rr$ sudo /home/don/install/spdk/spdk/scripts/setup.sh
Active mountpoints on /dev/nvme0n1, so not binding PCI dev 0000:03:00.0
0000:1c:00.0 (8086 0a54): nvme -> uio_pci_generic
0000:1d:00.0 (8086 0a54): nvme -> uio_pci_generic
0000:5e:00.0 (8086 0a54): nvme -> uio_pci_generic
0000:5f:00.0 (8086 0a54): nvme -> uio_pci_generic
0000:62:00.0 (8086 0a54): nvme -> uio_pci_generic
0000:63:00.0 (8086 0a54): nvme -> uio_pci_generic
0000:64:00.0 (8086 0a54): nvme -> uio_pci_generic
0000:65:00.0 (8086 0a54): nvme -> uio_pci_generic
0000:da:00.0 (8086 0a54): nvme -> uio_pci_generic
0000:db:00.0 (8086 0a54): nvme -> uio_pci_generic
0000:dc:00.0 (8086 0a54): nvme -> uio_pci_generic
0000:dd:00.0 (8086 0a54): nvme -> uio_pci_generic
0000:e0:00.0 (8086 0a54): nvme -> uio_pci_generic
0000:e1:00.0 (8086 0a54): nvme -> uio_pci_generic
0000:e2:00.0 (8086 0a54): nvme -> uio_pci_generic
0000:e3:00.0 (8086 0a54): nvme -> uio_pci_generic
0000:00:04.0 (8086 2021): ioatdma -> uio_pci_generic
0000:00:04.1 (8086 2021): ioatdma -> uio_pci_generic
0000:00:04.2 (8086 2021): ioatdma -> uio_pci_generic
0000:00:04.3 (8086 2021): ioatdma -> uio_pci_generic
0000:00:04.4 (8086 2021): ioatdma -> uio_pci_generic
0000:00:04.5 (8086 2021): ioatdma -> uio_pci_generic
0000:00:04.6 (8086 2021): ioatdma -> uio_pci_generic
0000:00:04.7 (8086 2021): ioatdma -> uio_pci_generic
0000:80:04.0 (8086 2021): ioatdma -> uio_pci_generic
0000:80:04.1 (8086 2021): ioatdma -> uio_pci_generic
0000:80:04.2 (8086 2021): ioatdma -> uio_pci_generic
0000:80:04.3 (8086 2021): ioatdma -> uio_pci_generic
0000:80:04.4 (8086 2021): ioatdma -> uio_pci_generic
0000:80:04.5 (8086 2021): ioatdma -> uio_pci_generic
0000:80:04.6 (8086 2021): ioatdma -> uio_pci_generic
0000:80:04.7 (8086 2021): ioatdma -> uio_pci_generic
don(a)donst201:~/fio/single/rr$ cat 1c.ini |grep -v '#'
[global]
rw=randrw
rwmixread=100
numjobs=4
iodepth=32
bs=4k
direct=1
thread=1
time_based=1
ramp_time=0
runtime=10
ioengine=/home/don/install/spdk/spdk/examples/nvme/fio_plugin/fio_plugin
group_reporting=1
unified_rw_reporting=1
exitall=1
randrepeat=0
norandommap=1
[0000.1c.00.0]
filename=trtype=PCIe traddr=0000.1c.00.0 ns=1
don(a)donst201:~/fio/single/rr$ sudo fio 1c.ini
0000.1c.00.0: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk, iodepth=32
...
fio-3.11
Starting 4 threads
Starting SPDK v18.10 / DPDK 18.08.0 initialization...
[ DPDK EAL parameters: fio --no-shconf -c 0x1 -m 512 --file-prefix=spdk_pid3668 ]
EAL: Detected 36 lcore(s)
EAL: Detected 2 NUMA nodes
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: PCI device 0000:1c:00.0 on NUMA socket 0
EAL: probe driver: 8086:a54 spdk_nvme
Jobs: 1 (f=0): [f(1),_(3)][100.0%][eta 00m:00s]
0000.1c.00.0: (groupid=0, jobs=4): err= 0: pid=3709: Mon Nov 5 02:28:29 2018
mixed: IOPS=13.2k, BW=51.6MiB/s (54.1MB/s)(517MiB/10011msec)
slat (nsec): min=109, max=15344, avg=127.91, stdev=73.44
clat (usec): min=187, max=18715, avg=9683.80, stdev=650.54
lat (usec): min=193, max=18715, avg=9683.93, stdev=650.53
clat percentiles (usec):
| 1.00th=[ 8717], 5.00th=[ 8848], 10.00th=[ 8979], 20.00th=[ 9110],
| 30.00th=[ 9241], 40.00th=[ 9372], 50.00th=[ 9503], 60.00th=[ 9765],
| 70.00th=[10028], 80.00th=[10159], 90.00th=[10552], 95.00th=[10945],
| 99.00th=[11338], 99.50th=[11338], 99.90th=[11731], 99.95th=[13960],
| 99.99th=[17695]
bw ( KiB/s): min=13080, max=13304, per=25.00%, avg=13214.70, stdev=55.30, samples=80
iops : min= 3270, max= 3326, avg=3303.65, stdev=13.84, samples=80
lat (usec) : 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.02%, 10=70.85%, 20=29.11%
cpu : usr=100.01%, sys=0.00%, ctx=23, majf=0, minf=0
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.9%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued rwts: total=132300,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
Run status group 0 (all jobs):
MIXED: bw=51.6MiB/s (54.1MB/s), 51.6MiB/s-51.6MiB/s (54.1MB/s-54.1MB/s), io=517MiB (542MB), run=10011-10011msec
^ permalink raw reply [flat|nested] 25+ messages in thread
end of thread, other threads:[~2018-11-06 10:31 UTC | newest]
Thread overview: 25+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-11-05 4:40 [SPDK] FIO NVMe Performance Results Cao, Gang
-- strict thread matches above, loose matches on Subject: below --
2018-11-06 10:31 Andrey Kuzmin
2018-11-06 1:40 Cao, Gang
2018-11-05 22:19 Gruher, Joseph R
2018-11-05 21:56 Harris, James R
2018-11-05 21:42 Howell, Seth
2018-11-05 20:45 Gruher, Joseph R
2018-11-05 20:39 Gruher, Joseph R
2018-11-05 20:18 Andrey Kuzmin
2018-11-05 19:59 Andrey Kuzmin
2018-11-05 19:38 Gruher, Joseph R
2018-11-05 19:04 Gruher, Joseph R
2018-11-05 18:40 Andrey Kuzmin
2018-11-05 18:33 Andrey Kuzmin
2018-11-05 18:32 Howell, Seth
2018-11-05 18:27 Gruher, Joseph R
2018-11-05 16:48 Andrey Kuzmin
2018-11-05 15:29 Gruher, Joseph R
2018-11-05 8:45
2018-11-05 8:39 Cao, Gang
2018-11-05 8:32
2018-11-05 8:27
2018-11-05 6:53 Cao, Gang
2018-11-05 5:32 Gruher, Joseph R
2018-11-05 2:49 Gruher, Joseph R
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.