From: Dave Chinner <david@fromorbit.com>
To: Kent Overstreet <kent.overstreet@gmail.com>
Cc: Andreas Dilger <adilger@dilger.ca>,
Linus Torvalds <torvalds@linux-foundation.org>,
Dave Chinner <dchinner@redhat.com>,
"Darrick J . Wong" <darrick.wong@oracle.com>,
Christoph Hellwig <hch@lst.de>,
Matthew Wilcox <willy@infradead.org>,
Amir Goldstein <amir73il@gmail.com>, Jan Kara <jack@suse.cz>,
Linux List Kernel Mailing <linux-kernel@vger.kernel.org>,
linux-xfs <linux-xfs@vger.kernel.org>,
linux-fsdevel <linux-fsdevel@vger.kernel.org>,
Josef Bacik <josef@toxicpanda.com>,
Alexander Viro <viro@zeniv.linux.org.uk>,
Andrew Morton <akpm@linux-foundation.org>
Subject: Re: pagecache locking (was: bcachefs status update) merged)
Date: Fri, 14 Jun 2019 10:35:18 +1000 [thread overview]
Message-ID: <20190614003518.GL14363@dread.disaster.area> (raw)
In-Reply-To: <20190613212112.GB28171@kmo-pixel>
On Thu, Jun 13, 2019 at 05:21:12PM -0400, Kent Overstreet wrote:
> On Thu, Jun 13, 2019 at 03:13:40PM -0600, Andreas Dilger wrote:
> > There are definitely workloads that require multiple threads doing non-overlapping
> > writes to a single file in HPC. This is becoming an increasingly common problem
> > as the number of cores on a single client increase, since there is typically one
> > thread per core trying to write to a shared file. Using multiple files (one per
> > core) is possible, but that has file management issues for users when there are a
> > million cores running on the same job/file (obviously not on the same client node)
> > dumping data every hour.
>
> Mixed buffered and O_DIRECT though? That profile looks like just buffered IO to
> me.
>
> > We were just looking at this exact problem last week, and most of the threads are
> > spinning in grab_cache_page_nowait->add_to_page_cache_lru() and set_page_dirty()
> > when writing at 1.9GB/s when they could be writing at 5.8GB/s (when threads are
> > writing O_DIRECT instead of buffered). Flame graph is attached for 16-thread case,
> > but high-end systems today easily have 2-4x that many cores.
>
> Yeah I've been spending some time on buffered IO performance too - 4k page
> overhead is a killer.
>
> bcachefs has a buffered write path that looks up multiple pages at a time and
> locks them, and then copies the data to all the pages at once (I stole the idea
> from btrfs). It was a very significant performance increase.
Careful with that - locking multiple pages is also a deadlock vector
that triggers unexpectedly when something conspires to lock pages in
non-ascending order. e.g.
64081362e8ff mm/page-writeback.c: fix range_cyclic writeback vs writepages deadlock
The fs/iomap.c code avoids this problem by mapping the IO first,
then iterating pages one at a time until the mapping is consumed,
then it gets another mapping. It also avoids needing to put a page
array on stack....
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
next prev parent reply other threads:[~2019-06-14 0:36 UTC|newest]
Thread overview: 62+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-06-10 19:14 bcachefs status update (it's done cooking; let's get this sucker merged) Kent Overstreet
2019-06-10 19:14 ` [PATCH 01/12] Compiler Attributes: add __flatten Kent Overstreet
2019-06-12 17:16 ` Greg KH
2019-06-10 19:14 ` [PATCH 02/12] locking: SIX locks (shared/intent/exclusive) Kent Overstreet
2019-06-10 19:14 ` [PATCH 03/12] mm: pagecache add lock Kent Overstreet
2019-06-10 19:14 ` [PATCH 04/12] mm: export find_get_pages() Kent Overstreet
2019-06-10 19:14 ` [PATCH 05/12] fs: insert_inode_locked2() Kent Overstreet
2019-06-10 19:14 ` [PATCH 06/12] fs: factor out d_mark_tmpfile() Kent Overstreet
2019-06-10 19:14 ` [PATCH 07/12] Propagate gfp_t when allocating pte entries from __vmalloc Kent Overstreet
2019-06-10 19:14 ` [PATCH 08/12] block: Add some exports for bcachefs Kent Overstreet
2019-06-10 19:14 ` [PATCH 09/12] bcache: optimize continue_at_nobarrier() Kent Overstreet
2019-06-10 19:14 ` [PATCH 10/12] bcache: move closures to lib/ Kent Overstreet
2019-06-11 10:25 ` Coly Li
2019-06-13 7:28 ` Christoph Hellwig
2019-06-13 11:04 ` Kent Overstreet
2019-06-10 19:14 ` [PATCH 11/12] closures: closure_wait_event() Kent Overstreet
2019-06-11 10:25 ` Coly Li
2019-06-12 17:17 ` Greg KH
2019-06-10 19:14 ` [PATCH 12/12] closures: fix a race on wakeup from closure_sync Kent Overstreet
2019-07-16 10:47 ` Coly Li
2019-07-18 7:46 ` Coly Li
2019-07-22 17:22 ` Kent Overstreet
2019-06-10 20:46 ` bcachefs status update (it's done cooking; let's get this sucker merged) Linus Torvalds
2019-06-11 1:17 ` Kent Overstreet
2019-06-11 4:33 ` Dave Chinner
2019-06-12 16:21 ` Kent Overstreet
2019-06-12 23:02 ` Dave Chinner
2019-06-13 18:36 ` pagecache locking (was: bcachefs status update) merged) Kent Overstreet
2019-06-13 21:13 ` Andreas Dilger
2019-06-13 21:21 ` Kent Overstreet
2019-06-14 0:35 ` Dave Chinner [this message]
2019-06-13 23:55 ` Dave Chinner
2019-06-14 2:30 ` Linus Torvalds
2019-06-14 7:30 ` Dave Chinner
2019-06-15 1:15 ` Linus Torvalds
2019-06-14 3:08 ` Linus Torvalds
2019-06-15 4:01 ` Linus Torvalds
2019-06-17 22:47 ` Dave Chinner
2019-06-17 23:38 ` Linus Torvalds
2019-06-18 4:21 ` Amir Goldstein
2019-06-19 10:38 ` Jan Kara
2019-06-19 22:37 ` Dave Chinner
2019-07-03 0:04 ` pagecache locking Boaz Harrosh
[not found] ` <DM6PR19MB250857CB8A3A1C8279D6F2F3C5FB0@DM6PR19MB2508.namprd19.prod.outlook.com>
2019-07-03 1:25 ` Boaz Harrosh
2019-07-05 23:31 ` Dave Chinner
2019-07-07 15:05 ` Boaz Harrosh
2019-07-07 23:55 ` Dave Chinner
2019-07-08 13:31 ` Jan Kara
2019-07-09 23:47 ` Dave Chinner
2019-07-10 8:41 ` Jan Kara
2019-06-14 17:08 ` pagecache locking (was: bcachefs status update) merged) Kent Overstreet
2019-06-19 8:21 ` bcachefs status update (it's done cooking; let's get this sucker merged) Jan Kara
2019-07-03 1:04 ` [PATCH] mm: Support madvise_willneed override by Filesystems Boaz Harrosh
2019-07-03 17:21 ` Jan Kara
2019-07-03 18:03 ` Boaz Harrosh
2019-06-11 4:55 ` bcachefs status update (it's done cooking; let's get this sucker merged) Linus Torvalds
2019-06-11 14:26 ` Matthew Wilcox
2019-06-11 4:10 ` Dave Chinner
2019-06-11 4:39 ` Linus Torvalds
2019-06-11 7:10 ` Dave Chinner
2019-06-12 2:07 ` Linus Torvalds
2019-07-03 5:59 ` Stefan K
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20190614003518.GL14363@dread.disaster.area \
--to=david@fromorbit.com \
--cc=adilger@dilger.ca \
--cc=akpm@linux-foundation.org \
--cc=amir73il@gmail.com \
--cc=darrick.wong@oracle.com \
--cc=dchinner@redhat.com \
--cc=hch@lst.de \
--cc=jack@suse.cz \
--cc=josef@toxicpanda.com \
--cc=kent.overstreet@gmail.com \
--cc=linux-fsdevel@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-xfs@vger.kernel.org \
--cc=torvalds@linux-foundation.org \
--cc=viro@zeniv.linux.org.uk \
--cc=willy@infradead.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).