linux-fsdevel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Tom Talpey <tom@talpey.com>
To: John Hubbard <jhubbard@nvidia.com>,
	john.hubbard@gmail.com, linux-mm@kvack.org
Cc: Andrew Morton <akpm@linux-foundation.org>,
	LKML <linux-kernel@vger.kernel.org>,
	linux-rdma <linux-rdma@vger.kernel.org>,
	linux-fsdevel@vger.kernel.org
Subject: Re: [PATCH v2 0/6] RFC: gup+dma: tracking dma-pinned pages
Date: Thu, 29 Nov 2018 21:18:05 -0500	[thread overview]
Message-ID: <1939f47a-eaec-3f2c-4ae7-f92d9fba7693@talpey.com> (raw)
In-Reply-To: <7a68b7fc-ff9d-381e-2444-909c9c2f6679@nvidia.com>

On 11/29/2018 8:39 PM, John Hubbard wrote:
> On 11/28/18 5:59 AM, Tom Talpey wrote:
>> On 11/27/2018 9:52 PM, John Hubbard wrote:
>>> On 11/27/18 5:21 PM, Tom Talpey wrote:
>>>> On 11/21/2018 5:06 PM, John Hubbard wrote:
>>>>> On 11/21/18 8:49 AM, Tom Talpey wrote:
>>>>>> On 11/21/2018 1:09 AM, John Hubbard wrote:
>>>>>>> On 11/19/18 10:57 AM, Tom Talpey wrote:
>>> [...]
>>>> I'm super-limited here this week hardware-wise and have not been able
>>>> to try testing with the patched kernel.
>>>>
>>>> I was able to compare my earlier quick test with a Bionic 4.15 kernel
>>>> (400K IOPS) against a similar 4.20rc3 kernel, and the rate dropped to
>>>> ~_375K_ IOPS. Which I found perhaps troubling. But it was only a quick
>>>> test, and without your change.
>>>>
>>>
>>> So just to double check (again): you are running fio with these parameters,
>>> right?
>>>
>>> [reader]
>>> direct=1
>>> ioengine=libaio
>>> blocksize=4096
>>> size=1g
>>> numjobs=1
>>> rw=read
>>> iodepth=64
>>
>> Correct, I copy/pasted these directly. I also ran with size=10g because
>> the 1g provides a really small sample set.
>>
>> There was one other difference, your results indicated fio 3.3 was used.
>> My Bionic install has fio 3.1. I don't find that relevant because our
>> goal is to compare before/after, which I haven't done yet.
>>
> 
> OK, the 50 MB/s was due to my particular .config. I had some expensive debug options
> set in mm, fs and locking subsystems. Turning those off, I'm back up to the rated
> speed of the Samsung NVMe device, so now we should have a clearer picture of the
> performance that real users will see.

Oh, good! I'm especially glad because I was having a heck of a time
reconfiguring the one machine I have available for this.

> Continuing on, then: running a before and after test, I don't see any significant
> difference in the fio results:

Excerpting from below:

 > Baseline 4.20.0-rc3 (commit f2ce1065e767), as before:
 >     read: IOPS=193k, BW=753MiB/s (790MB/s)(1024MiB/1360msec)
 >    cpu          : usr=16.26%, sys=48.05%, ctx=251258, majf=0, minf=73

vs

 > With patches applied:
 >     read: IOPS=193k, BW=753MiB/s (790MB/s)(1024MiB/1360msec)
 >    cpu          : usr=16.26%, sys=48.05%, ctx=251258, majf=0, minf=73

Perfect results, not CPU limited, and full IOPS.

Curiously identical, so I trust you've checked that you measured
both targets, but if so, I say it's good.

Tom.

> 
> fio.conf:
> 
> [reader]
> direct=1
> ioengine=libaio
> blocksize=4096
> size=1g
> numjobs=1
> rw=read
> iodepth=64
> 
> ---------------------------------------------------------
> Baseline 4.20.0-rc3 (commit f2ce1065e767), as before:
> 
> $ fio ./experimental-fio.conf
> reader: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
> fio-3.3
> Starting 1 process
> Jobs: 1 (f=1)
> reader: (groupid=0, jobs=1): err= 0: pid=1738: Thu Nov 29 17:20:07 2018
>     read: IOPS=193k, BW=753MiB/s (790MB/s)(1024MiB/1360msec)
>      slat (nsec): min=1381, max=46469, avg=1649.48, stdev=594.46
>      clat (usec): min=162, max=12247, avg=330.00, stdev=185.55
>       lat (usec): min=165, max=12253, avg=331.68, stdev=185.69
>      clat percentiles (usec):
>       |  1.00th=[  322],  5.00th=[  326], 10.00th=[  326], 20.00th=[  326],
>       | 30.00th=[  326], 40.00th=[  326], 50.00th=[  326], 60.00th=[  326],
>       | 70.00th=[  326], 80.00th=[  326], 90.00th=[  326], 95.00th=[  326],
>       | 99.00th=[  379], 99.50th=[  594], 99.90th=[  603], 99.95th=[  611],
>       | 99.99th=[12125]
>     bw (  KiB/s): min=751640, max=782912, per=99.52%, avg=767276.00, stdev=22112.64, samples=2
>     iops        : min=187910, max=195728, avg=191819.00, stdev=5528.16, samples=2
>    lat (usec)   : 250=0.08%, 500=99.30%, 750=0.59%
>    lat (msec)   : 20=0.02%
>    cpu          : usr=16.26%, sys=48.05%, ctx=251258, majf=0, minf=73
>    IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
>       submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
>       complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
>       issued rwts: total=262144,0,0,0 short=0,0,0,0 dropped=0,0,0,0
>       latency   : target=0, window=0, percentile=100.00%, depth=64
> 
> Run status group 0 (all jobs):
>     READ: bw=753MiB/s (790MB/s), 753MiB/s-753MiB/s (790MB/s-790MB/s), io=1024MiB (1074MB), run=1360-1360msec
> 
> Disk stats (read/write):
>    nvme0n1: ios=220798/0, merge=0/0, ticks=71481/0, in_queue=71966, util=100.00%
> 
> ---------------------------------------------------------
> With patches applied:
> 
> <redforge> fast_256GB $ fio ./experimental-fio.conf
> reader: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
> fio-3.3
> Starting 1 process
> Jobs: 1 (f=1)
> reader: (groupid=0, jobs=1): err= 0: pid=1738: Thu Nov 29 17:20:07 2018
>     read: IOPS=193k, BW=753MiB/s (790MB/s)(1024MiB/1360msec)
>      slat (nsec): min=1381, max=46469, avg=1649.48, stdev=594.46
>      clat (usec): min=162, max=12247, avg=330.00, stdev=185.55
>       lat (usec): min=165, max=12253, avg=331.68, stdev=185.69
>      clat percentiles (usec):
>       |  1.00th=[  322],  5.00th=[  326], 10.00th=[  326], 20.00th=[  326],
>       | 30.00th=[  326], 40.00th=[  326], 50.00th=[  326], 60.00th=[  326],
>       | 70.00th=[  326], 80.00th=[  326], 90.00th=[  326], 95.00th=[  326],
>       | 99.00th=[  379], 99.50th=[  594], 99.90th=[  603], 99.95th=[  611],
>       | 99.99th=[12125]
>     bw (  KiB/s): min=751640, max=782912, per=99.52%, avg=767276.00, stdev=22112.64, samples=2
>     iops        : min=187910, max=195728, avg=191819.00, stdev=5528.16, samples=2
>    lat (usec)   : 250=0.08%, 500=99.30%, 750=0.59%
>    lat (msec)   : 20=0.02%
>    cpu          : usr=16.26%, sys=48.05%, ctx=251258, majf=0, minf=73
>    IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
>       submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
>       complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
>       issued rwts: total=262144,0,0,0 short=0,0,0,0 dropped=0,0,0,0
>       latency   : target=0, window=0, percentile=100.00%, depth=64
> 
> Run status group 0 (all jobs):
>     READ: bw=753MiB/s (790MB/s), 753MiB/s-753MiB/s (790MB/s-790MB/s), io=1024MiB (1074MB), run=1360-1360msec
> 
> Disk stats (read/write):
>    nvme0n1: ios=220798/0, merge=0/0, ticks=71481/0, in_queue=71966, util=100.00%
> 
> 
> thanks,
> 

  reply	other threads:[~2018-11-30  2:18 UTC|newest]

Thread overview: 26+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2018-11-10  8:50 [PATCH v2 0/6] RFC: gup+dma: tracking dma-pinned pages john.hubbard
2018-11-10  8:50 ` [PATCH v2 1/6] mm/gup: finish consolidating error handling john.hubbard
2018-11-12 15:41   ` Keith Busch
2018-11-12 16:14     ` Dan Williams
2018-11-15  0:45       ` John Hubbard
2018-11-10  8:50 ` [PATCH v2 2/6] mm: introduce put_user_page*(), placeholder versions john.hubbard
2018-11-11 14:10   ` Mike Rapoport
2018-11-10  8:50 ` [PATCH v2 3/6] infiniband/mm: convert put_page() to put_user_page*() john.hubbard
2018-11-10  8:50 ` [PATCH v2 4/6] mm: introduce page->dma_pinned_flags, _count john.hubbard
2018-11-10  8:50 ` [PATCH v2 5/6] mm: introduce zone_gup_lock, for dma-pinned pages john.hubbard
2018-11-10  8:50 ` [PATCH v2 6/6] mm: track gup pages with page->dma_pinned_* fields john.hubbard
2018-11-12 13:58   ` Jan Kara
2018-11-15  6:28   ` [LKP] [mm] 0e9755bfa2: kernel_BUG_at_include/linux/mm.h kernel test robot
2018-11-19 18:57 ` [PATCH v2 0/6] RFC: gup+dma: tracking dma-pinned pages Tom Talpey
2018-11-21  6:09   ` John Hubbard
2018-11-21 16:49     ` Tom Talpey
2018-11-21 22:06       ` John Hubbard
2018-11-28  1:21         ` Tom Talpey
2018-11-28  2:52           ` John Hubbard
2018-11-28 13:59             ` Tom Talpey
2018-11-30  1:39               ` John Hubbard
2018-11-30  2:18                 ` Tom Talpey [this message]
2018-11-30  2:21                   ` John Hubbard
2018-11-30  2:30                     ` Tom Talpey
2018-11-30  3:00                       ` John Hubbard
2018-11-30  3:14                         ` Tom Talpey

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=1939f47a-eaec-3f2c-4ae7-f92d9fba7693@talpey.com \
    --to=tom@talpey.com \
    --cc=akpm@linux-foundation.org \
    --cc=jhubbard@nvidia.com \
    --cc=john.hubbard@gmail.com \
    --cc=linux-fsdevel@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-mm@kvack.org \
    --cc=linux-rdma@vger.kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body. 
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).