From: Dominique Martinet <asmadeus@codewreck.org>
To: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Dave Chinner <david@fromorbit.com>,
Jiri Kosina <jikos@kernel.org>,
Matthew Wilcox <willy@infradead.org>,
Jann Horn <jannh@google.com>,
Andrew Morton <akpm@linux-foundation.org>,
Greg KH <gregkh@linuxfoundation.org>,
Peter Zijlstra <peterz@infradead.org>,
Michal Hocko <mhocko@suse.com>, Linux-MM <linux-mm@kvack.org>,
kernel list <linux-kernel@vger.kernel.org>,
Linux API <linux-api@vger.kernel.org>
Subject: Re: [PATCH] mm/mincore: allow for making sys_mincore() privileged
Date: Fri, 11 Jan 2019 05:57:50 +0100 [thread overview]
Message-ID: <20190111045750.GA27333@nautica> (raw)
In-Reply-To: <CAHk-=wip2CPrdOwgF0z4n2tsdW7uu+Egtcx9Mxxe3gPfPW_JmQ@mail.gmail.com>
Linus Torvalds wrote on Thu, Jan 10, 2019:
> On Thu, Jan 10, 2019 at 4:25 AM Dominique Martinet
> <asmadeus@codewreck.org> wrote:
> > Linus Torvalds wrote on Thu, Jan 10, 2019:
> > > (Except, of course, if somebody actually notices outside of tests.
> > > Which may well happen and just force us to revert that commit. But
> > > that's a separate issue entirely).
> >
> > Both Dave and I pointed at a couple of utilities that break with
> > this. nocache can arguably work with the new behaviour but will behave
> > differently; vmtouch on the other hand is no longer able to display
> > what's in cache or not - people use that for example to "warm up" a
> > container in page cache based on how it appears after it had been
> > running for a while is a pretty valid usecase to me.
>
> So honestly, the main reason I'm loath to revert is that yes, we know
> of theoretical differences, but they seem to all be
> performance-related.
I don't see what other use mincore could have, yes - even the
"debugging" use I gave is performance investigations and not hard
problems (and I probably would go straight to perf nowadays, you'd get
the info that the program doesn't use cache from the call graphs)
> It would be really good to hear numbers. Is the warm-up optimization
> something that changes things from 3ms to 3.5ms? Or does it change
> things from 3ms to half a second?
This is heavily workload and storage hardware dependant, so hard to give
some absolute value.
Trying with some big server, fast SSD, mysql and doing:
# echo 3 > /proc/sys/vm/drop_caches
# (optional) prefetch table and innodb files
# systemctl restart mariadb
# time mysql -q db "select * from mytable where id in $ENTRIES" > /dev/null
# time mysql -q db "select * from mytable where id in $ENTRIES2" > /dev/null
# time mysql -q db "select * from mytable where id in $ENTRIES3" > /dev/null
(where ENTRIES* are lists of 1000 id, and id is indexed; the table is 8GB
for 62590661 entries so 1000 entries is approx 128KB of data out of that
file)
I get on average over a few queries approximately a real time of 350ms,
230ms and 220ms immediately after drop cache and service restart, and
150ms, 60ms and 60ms after a prefetch (hand-wavy average over 3 runs, I
didn't have the patience to do proper testing).
(In both cases, user/sys are less than 10ms; I don't see much difference
there)
If I restart the service without dropping caches and redo the query I
get 60ms from the first query onwards so I must not be preloading
everything properly, some real script that would look all over a
container to properly restore the page cache would do better than me
blindly preloading a few files.
Either way, we're talking about a factor of 2-3 until the application has
been looking at most of the entries, and I didn't try to see how that
would look like on spinning disks or the kind of slow storage one would
get on VPS somewhere in the cloud - I'm sure someone with time to waste
could get much more impressive figures, but this already look pretty
worthwhile to me.
--
Dominique Martinet | Asmadeus
next prev parent reply other threads:[~2019-01-11 4:57 UTC|newest]
Thread overview: 161+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-01-05 17:27 [PATCH] mm/mincore: allow for making sys_mincore() privileged Jiri Kosina
2019-01-05 19:14 ` Vlastimil Babka
2019-01-05 19:24 ` Jiri Kosina
2019-01-05 19:38 ` Vlastimil Babka
2019-01-08 9:14 ` Bernd Petrovitsch
2019-01-08 11:37 ` Jiri Kosina
2019-01-08 13:53 ` Bernd Petrovitsch
2019-01-08 14:08 ` Kirill A. Shutemov
2019-01-05 19:44 ` kbuild test robot
2019-01-05 19:46 ` Linus Torvalds
2019-01-05 20:12 ` Jiri Kosina
2019-01-05 20:17 ` Linus Torvalds
2019-01-05 20:43 ` Jiri Kosina
2019-01-05 21:54 ` Linus Torvalds
2019-01-06 11:33 ` Kevin Easton
2019-01-08 8:50 ` Kevin Easton
2019-01-18 14:23 ` Tejun Heo
2019-01-05 20:13 ` Linus Torvalds
2019-01-05 19:56 ` kbuild test robot
2019-01-05 22:54 ` Jann Horn
2019-01-05 23:05 ` Linus Torvalds
2019-01-05 23:16 ` Linus Torvalds
2019-01-05 23:28 ` Linus Torvalds
2019-01-05 23:39 ` Linus Torvalds
2019-01-06 0:11 ` Matthew Wilcox
2019-01-06 0:22 ` Linus Torvalds
2019-01-06 1:50 ` Linus Torvalds
2019-01-06 21:46 ` Linus Torvalds
2019-01-08 4:43 ` Dave Chinner
2019-01-08 17:57 ` Linus Torvalds
2019-01-09 2:24 ` Dave Chinner
2019-01-09 2:31 ` Jiri Kosina
2019-01-09 4:39 ` Dave Chinner
2019-01-09 10:08 ` Jiri Kosina
2019-01-10 1:15 ` Dave Chinner
2019-01-10 7:54 ` Jiri Kosina
2019-01-09 18:25 ` Linus Torvalds
2019-01-10 0:44 ` Dave Chinner
2019-01-10 1:18 ` Linus Torvalds
2019-01-10 5:26 ` Andy Lutomirski
2019-01-10 14:47 ` Matthew Wilcox
2019-01-10 21:44 ` Dave Chinner
2019-01-10 21:59 ` Linus Torvalds
2019-01-11 1:47 ` Dave Chinner
2019-01-10 7:03 ` Dave Chinner
2019-01-10 11:47 ` Linus Torvalds
2019-01-10 12:24 ` Dominique Martinet
2019-01-10 22:11 ` Linus Torvalds
2019-01-11 2:03 ` Dave Chinner
2019-01-11 2:18 ` Linus Torvalds
2019-01-11 4:04 ` Dave Chinner
2019-01-11 4:08 ` Andy Lutomirski
2019-01-11 7:20 ` Dave Chinner
2019-01-11 7:08 ` Linus Torvalds
2019-01-11 7:36 ` Dave Chinner
2019-01-11 16:26 ` Linus Torvalds
2019-01-15 23:45 ` Dave Chinner
2019-01-16 4:54 ` Linus Torvalds
2019-01-16 5:49 ` Linus Torvalds
2019-01-17 1:26 ` Dave Chinner
2019-02-20 15:49 ` Nicolai Stange
2019-01-11 4:57 ` Dominique Martinet [this message]
2019-01-11 7:11 ` Linus Torvalds
2019-01-11 7:32 ` Dominique Martinet
2019-01-16 0:42 ` Josh Snyder
2019-01-16 5:00 ` Linus Torvalds
2019-01-16 5:25 ` Andy Lutomirski
2019-01-16 5:34 ` Linus Torvalds
2019-01-16 5:46 ` Dominique Martinet
2019-01-16 5:58 ` Linus Torvalds
2019-01-16 6:34 ` Dominique Martinet
2019-01-16 7:52 ` Josh Snyder
2019-01-16 12:18 ` Kevin Easton
2019-01-17 21:45 ` Vlastimil Babka
2019-01-18 4:49 ` Linus Torvalds
2019-01-18 18:58 ` Vlastimil Babka
2019-01-16 16:12 ` Jiri Kosina
2019-01-16 17:48 ` Linus Torvalds
2019-01-16 20:23 ` Jiri Kosina
2019-01-16 21:37 ` Matthew Wilcox
2019-01-16 21:41 ` Jiri Kosina
2019-01-17 9:52 ` Cyril Hrubis
2019-01-28 13:49 ` Cyril Hrubis
2019-01-17 4:51 ` Linus Torvalds
2019-01-18 4:54 ` Linus Torvalds
2019-01-17 1:49 ` Dominique Martinet
2019-01-23 20:27 ` Linus Torvalds
2019-01-23 20:35 ` Linus Torvalds
2019-01-23 23:12 ` Jiri Kosina
2019-01-24 0:20 ` Linus Torvalds
2019-01-24 0:24 ` Dominique Martinet
2019-01-24 12:45 ` Dominique Martinet
2019-01-24 14:25 ` Jiri Kosina
2019-01-27 22:35 ` Jiri Kosina
2019-01-28 0:05 ` Dominique Martinet
2019-01-29 23:52 ` Jiri Kosina
2019-01-30 9:09 ` Michal Hocko
2019-01-30 12:29 ` Jiri Kosina
2019-01-16 12:36 ` Matthew Wilcox
2019-01-10 14:50 ` Matthew Wilcox
2019-01-11 7:36 ` Jiri Kosina
2019-01-17 2:22 ` Dave Chinner
2019-01-17 8:18 ` Jiri Kosina
2019-01-17 21:06 ` Dave Chinner
2019-01-07 4:32 ` Dominique Martinet
2019-01-07 10:33 ` Vlastimil Babka
2019-01-07 11:08 ` Dominique Martinet
2019-01-07 11:59 ` Vlastimil Babka
2019-01-07 13:29 ` Daniel Gruss
2019-01-07 10:10 ` Michael Ellerman
2019-01-05 23:09 ` Jiri Kosina
2019-01-30 12:44 ` [PATCH 0/3] mincore() and IOCB_NOWAIT adjustments Vlastimil Babka
2019-01-30 12:44 ` [PATCH 1/3] mm/mincore: make mincore() more conservative Vlastimil Babka
2019-01-31 9:43 ` Michal Hocko
2019-01-31 9:51 ` Dominique Martinet
2019-01-31 17:46 ` Josh Snyder
2019-02-01 8:56 ` Vlastimil Babka
2019-03-06 23:13 ` Andrew Morton
2019-03-07 0:01 ` Jiri Kosina
2019-03-07 0:40 ` Dominique Martinet
2019-03-07 5:46 ` Jiri Kosina
2019-01-30 12:44 ` [PATCH 2/3] mm/filemap: initiate readahead even if IOCB_NOWAIT is set for the I/O Vlastimil Babka
2019-01-30 15:04 ` Florian Weimer
2019-01-30 15:15 ` Jiri Kosina
2019-01-31 10:47 ` Florian Weimer
2019-01-31 11:34 ` Jiri Kosina
2019-01-31 9:56 ` Michal Hocko
2019-01-31 10:15 ` Jiri Kosina
2019-01-31 10:23 ` Michal Hocko
2019-01-31 10:30 ` Jiri Kosina
2019-01-31 11:32 ` Michal Hocko
2019-01-31 17:54 ` Linus Torvalds
2019-02-01 5:13 ` Dave Chinner
2019-02-01 7:05 ` Linus Torvalds
2019-02-01 7:21 ` Linus Torvalds
2019-02-01 1:44 ` Dave Chinner
2019-02-12 15:48 ` Jiri Kosina
2019-01-31 12:04 ` Daniel Gruss
2019-01-31 12:06 ` Vlastimil Babka
2019-01-31 12:08 ` Jiri Kosina
2019-01-31 12:57 ` Daniel Gruss
2019-01-30 12:44 ` [PATCH 3/3] mm/mincore: provide mapped status when cached status is not allowed Vlastimil Babka
2019-01-31 10:09 ` Michal Hocko
2019-02-01 9:04 ` Vlastimil Babka
2019-02-01 9:11 ` Michal Hocko
2019-02-01 9:27 ` Vlastimil Babka
2019-02-06 20:14 ` Jiri Kosina
2019-02-12 3:44 ` Jiri Kosina
2019-02-12 6:36 ` Michal Hocko
2019-02-12 13:09 ` Jiri Kosina
2019-02-12 14:01 ` Michal Hocko
2019-03-06 12:11 ` [PATCH 0/3] mincore() and IOCB_NOWAIT adjustments Jiri Kosina
2019-03-06 22:35 ` Andrew Morton
2019-03-06 22:48 ` Jiri Kosina
2019-03-06 23:23 ` Andrew Morton
2019-03-06 23:32 ` Dominique Martinet
2019-03-06 23:38 ` Andrew Morton
2019-03-09 16:53 ` Linus Torvalds
2019-03-12 14:17 ` [PATCH v2 0/2] prevent mincore() page cache leaks Vlastimil Babka
2019-03-12 14:17 ` [PATCH v2 1/2] mm/mincore: make mincore() more conservative Vlastimil Babka
2019-03-12 14:17 ` [PATCH v2 2/2] mm/mincore: provide mapped status when cached status is not allowed Vlastimil Babka
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20190111045750.GA27333@nautica \
--to=asmadeus@codewreck.org \
--cc=akpm@linux-foundation.org \
--cc=david@fromorbit.com \
--cc=gregkh@linuxfoundation.org \
--cc=jannh@google.com \
--cc=jikos@kernel.org \
--cc=linux-api@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=mhocko@suse.com \
--cc=peterz@infradead.org \
--cc=torvalds@linux-foundation.org \
--cc=willy@infradead.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).