From: Tom Talpey <tom@talpey.com>
To: john.hubbard@gmail.com, Andrew Morton <akpm@linux-foundation.org>,
linux-mm@kvack.org
Cc: Al Viro <viro@zeniv.linux.org.uk>,
Christian Benvenuti <benve@cisco.com>,
Christoph Hellwig <hch@infradead.org>,
Christopher Lameter <cl@linux.com>,
Dan Williams <dan.j.williams@intel.com>,
Dave Chinner <david@fromorbit.com>,
Dennis Dalessandro <dennis.dalessandro@intel.com>,
Doug Ledford <dledford@redhat.com>, Jan Kara <jack@suse.cz>,
Jason Gunthorpe <jgg@ziepe.ca>,
Jerome Glisse <jglisse@redhat.com>,
Matthew Wilcox <willy@infradead.org>,
Michal Hocko <mhocko@kernel.org>,
Mike Rapoport <rppt@linux.ibm.com>,
Mike Marciniszyn <mike.marciniszyn@intel.com>,
Ralph Campbell <rcampbell@nvidia.com>,
LKML <linux-kernel@vger.kernel.org>,
linux-fsdevel@vger.kernel.org, John Hubbard <jhubbard@nvidia.com>
Subject: Re: [PATCH 0/6] RFC v2: mm: gup/dma tracking
Date: Mon, 4 Feb 2019 20:41:50 -0500 [thread overview]
Message-ID: <dbca5400-b0c0-0958-c3ba-ff672f301799@talpey.com> (raw)
In-Reply-To: <20190204052135.25784-1-jhubbard@nvidia.com>
On 2/4/2019 12:21 AM, john.hubbard@gmail.com wrote:
> From: John Hubbard <jhubbard@nvidia.com>
>
>
> Performance: here is an fio run on an NVMe drive, using this for the fio
> configuration file:
>
> [reader]
> direct=1
> ioengine=libaio
> blocksize=4096
> size=1g
> numjobs=1
> rw=read
> iodepth=64
>
> reader: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
> fio-3.3
> Starting 1 process
> Jobs: 1 (f=1)
> reader: (groupid=0, jobs=1): err= 0: pid=7011: Sun Feb 3 20:36:51 2019
> read: IOPS=190k, BW=741MiB/s (778MB/s)(1024MiB/1381msec)
> slat (nsec): min=2716, max=57255, avg=4048.14, stdev=1084.10
> clat (usec): min=20, max=12485, avg=332.63, stdev=191.77
> lat (usec): min=22, max=12498, avg=336.72, stdev=192.07
> clat percentiles (usec):
> | 1.00th=[ 322], 5.00th=[ 322], 10.00th=[ 322], 20.00th=[ 326],
> | 30.00th=[ 326], 40.00th=[ 326], 50.00th=[ 326], 60.00th=[ 326],
> | 70.00th=[ 326], 80.00th=[ 330], 90.00th=[ 330], 95.00th=[ 330],
> | 99.00th=[ 478], 99.50th=[ 717], 99.90th=[ 1074], 99.95th=[ 1090],
> | 99.99th=[12256]
These latencies are concerning. The best results we saw at the end of
November (previous approach) were MUCH flatter. These really start
spiking at three 9's, and are sky-high at four 9's. The "stdev" values
for clat and lat are about 10 times the previous. There's some kind
of serious queuing contention here, that wasn't there in November.
> bw ( KiB/s): min=730152, max=776512, per=99.22%, avg=753332.00, stdev=32781.47, samples=2
> iops : min=182538, max=194128, avg=188333.00, stdev=8195.37, samples=2
> lat (usec) : 50=0.01%, 100=0.01%, 250=0.07%, 500=99.26%, 750=0.38%
> lat (usec) : 1000=0.02%
> lat (msec) : 2=0.24%, 20=0.02%
> cpu : usr=15.07%, sys=84.13%, ctx=10, majf=0, minf=74
System CPU 84% is roughly double the November results of 45%. Ouch.
Did you re-run the baseline on the new unpatched base kernel and can
we see the before/after?
Tom.
> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
> issued rwts: total=262144,0,0,0 short=0,0,0,0 dropped=0,0,0,0
> latency : target=0, window=0, percentile=100.00%, depth=64
>
> Run status group 0 (all jobs):
> READ: bw=741MiB/s (778MB/s), 741MiB/s-741MiB/s (778MB/s-778MB/s), io=1024MiB (1074MB), run=1381-1381msec
>
> Disk stats (read/write):
> nvme0n1: ios=216966/0, merge=0/0, ticks=6112/0, in_queue=704, util=91.34%
next prev parent reply other threads:[~2019-02-05 1:49 UTC|newest]
Thread overview: 29+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-02-04 5:21 [PATCH 0/6] RFC v2: mm: gup/dma tracking john.hubbard
2019-02-04 5:21 ` [PATCH 1/6] mm: introduce put_user_page*(), placeholder versions john.hubbard
2019-02-04 5:21 ` [PATCH 2/6] infiniband/mm: convert put_page() to put_user_page*() john.hubbard
2019-02-04 5:21 ` [PATCH 3/6] mm: page_cache_add_speculative(): refactoring john.hubbard
2019-02-04 5:21 ` [PATCH 4/6] mm/gup: track gup-pinned pages john.hubbard
2019-02-04 18:19 ` Matthew Wilcox
2019-02-04 19:11 ` John Hubbard
2019-02-11 9:51 ` [LKP] [mm/gup] cdaa813278: kvm-unit-tests.vmx_ept_access_test_paddr_read_write.fail kernel test robot
2019-02-20 19:24 ` [PATCH 4/6] mm/gup: track gup-pinned pages Ira Weiny
2019-02-20 20:22 ` John Hubbard
2019-02-28 12:15 ` [LKP] [mm/gup] cdaa813278: stress-ng.numa.ops_per_sec 4671.0% improvement kernel test robot
2019-02-04 5:21 ` [PATCH 5/6] mm/gup: /proc/vmstat support for get/put user pages john.hubbard
2019-02-18 2:16 ` [LKP] [mm/gup] e7ae097b0b: will-it-scale.per_process_ops -5.0% regression kernel test robot
2019-02-04 5:21 ` [PATCH 6/6] mm/gup: Documentation/vm/get_user_pages.rst, MAINTAINERS john.hubbard
2019-02-05 16:40 ` Mike Rapoport
2019-02-05 21:53 ` John Hubbard
2019-02-04 16:08 ` [PATCH 0/6] RFC v2: mm: gup/dma tracking Christopher Lameter
2019-02-04 16:12 ` Christoph Hellwig
2019-02-04 16:59 ` Christopher Lameter
2019-02-04 17:14 ` Christopher Lameter
2019-02-04 17:51 ` Jason Gunthorpe
2019-02-04 18:21 ` Christopher Lameter
2019-02-04 19:09 ` Matthew Wilcox
2019-02-04 23:35 ` Ira Weiny
2019-02-05 19:30 ` Christopher Lameter
2019-02-05 1:41 ` Tom Talpey [this message]
2019-02-05 8:22 ` John Hubbard
2019-02-05 13:38 ` Tom Talpey
2019-02-05 21:55 ` John Hubbard
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=dbca5400-b0c0-0958-c3ba-ff672f301799@talpey.com \
--to=tom@talpey.com \
--cc=akpm@linux-foundation.org \
--cc=benve@cisco.com \
--cc=cl@linux.com \
--cc=dan.j.williams@intel.com \
--cc=david@fromorbit.com \
--cc=dennis.dalessandro@intel.com \
--cc=dledford@redhat.com \
--cc=hch@infradead.org \
--cc=jack@suse.cz \
--cc=jgg@ziepe.ca \
--cc=jglisse@redhat.com \
--cc=jhubbard@nvidia.com \
--cc=john.hubbard@gmail.com \
--cc=linux-fsdevel@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=mhocko@kernel.org \
--cc=mike.marciniszyn@intel.com \
--cc=rcampbell@nvidia.com \
--cc=rppt@linux.ibm.com \
--cc=viro@zeniv.linux.org.uk \
--cc=willy@infradead.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).