From: Dave Chinner <david@fromorbit.com>
To: Kent Overstreet <kent.overstreet@gmail.com>
Cc: Andreas Dilger <adilger@dilger.ca>,
Linus Torvalds <torvalds@linux-foundation.org>,
Dave Chinner <dchinner@redhat.com>,
"Darrick J . Wong" <darrick.wong@oracle.com>,
Christoph Hellwig <hch@lst.de>,
Matthew Wilcox <willy@infradead.org>,
Amir Goldstein <amir73il@gmail.com>, Jan Kara <jack@suse.cz>,
Linux List Kernel Mailing <linux-kernel@vger.kernel.org>,
linux-xfs <linux-xfs@vger.kernel.org>,
linux-fsdevel <linux-fsdevel@vger.kernel.org>,
Josef Bacik <josef@toxicpanda.com>,
Alexander Viro <viro@zeniv.linux.org.uk>,
Andrew Morton <akpm@linux-foundation.org>
Subject: Re: pagecache locking (was: bcachefs status update) merged)
Date: Fri, 14 Jun 2019 10:35:18 +1000 [thread overview]
Message-ID: <20190614003518.GL14363@dread.disaster.area> (raw)
In-Reply-To: <20190613212112.GB28171@kmo-pixel>
On Thu, Jun 13, 2019 at 05:21:12PM -0400, Kent Overstreet wrote:
> On Thu, Jun 13, 2019 at 03:13:40PM -0600, Andreas Dilger wrote:
> > There are definitely workloads that require multiple threads doing non-overlapping
> > writes to a single file in HPC. This is becoming an increasingly common problem
> > as the number of cores on a single client increase, since there is typically one
> > thread per core trying to write to a shared file. Using multiple files (one per
> > core) is possible, but that has file management issues for users when there are a
> > million cores running on the same job/file (obviously not on the same client node)
> > dumping data every hour.
>
> Mixed buffered and O_DIRECT though? That profile looks like just buffered IO to
> me.
>
> > We were just looking at this exact problem last week, and most of the threads are
> > spinning in grab_cache_page_nowait->add_to_page_cache_lru() and set_page_dirty()
> > when writing at 1.9GB/s when they could be writing at 5.8GB/s (when threads are
> > writing O_DIRECT instead of buffered). Flame graph is attached for 16-thread case,
> > but high-end systems today easily have 2-4x that many cores.
>
> Yeah I've been spending some time on buffered IO performance too - 4k page
> overhead is a killer.
>
> bcachefs has a buffered write path that looks up multiple pages at a time and
> locks them, and then copies the data to all the pages at once (I stole the idea
> from btrfs). It was a very significant performance increase.
Careful with that - locking multiple pages is also a deadlock vector
that triggers unexpectedly when something conspires to lock pages in
non-ascending order. e.g.
64081362e8ff mm/page-writeback.c: fix range_cyclic writeback vs writepages deadlock
The fs/iomap.c code avoids this problem by mapping the IO first,
then iterating pages one at a time until the mapping is consumed,
then it gets another mapping. It also avoids needing to put a page
array on stack....
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
next prev parent reply other threads:[~2019-06-14 0:36 UTC|newest]
Thread overview: 66+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-06-10 19:14 bcachefs status update (it's done cooking; let's get this sucker merged) Kent Overstreet
2019-06-10 19:14 ` [PATCH 01/12] Compiler Attributes: add __flatten Kent Overstreet
2019-06-12 17:16 ` Greg KH
2019-06-10 19:14 ` [PATCH 02/12] locking: SIX locks (shared/intent/exclusive) Kent Overstreet
2019-06-10 19:14 ` [PATCH 03/12] mm: pagecache add lock Kent Overstreet
2019-06-10 19:14 ` [PATCH 04/12] mm: export find_get_pages() Kent Overstreet
2019-06-10 19:14 ` [PATCH 05/12] fs: insert_inode_locked2() Kent Overstreet
2019-06-10 19:14 ` [PATCH 06/12] fs: factor out d_mark_tmpfile() Kent Overstreet
2019-06-10 19:14 ` [PATCH 07/12] Propagate gfp_t when allocating pte entries from __vmalloc Kent Overstreet
2019-06-10 19:14 ` [PATCH 08/12] block: Add some exports for bcachefs Kent Overstreet
2019-06-10 19:14 ` [PATCH 09/12] bcache: optimize continue_at_nobarrier() Kent Overstreet
2019-06-10 19:14 ` [PATCH 10/12] bcache: move closures to lib/ Kent Overstreet
2019-06-11 10:25 ` Coly Li
2019-06-13 7:28 ` Christoph Hellwig
2019-06-13 11:04 ` Kent Overstreet
2019-06-10 19:14 ` [PATCH 11/12] closures: closure_wait_event() Kent Overstreet
2019-06-11 10:25 ` Coly Li
2019-06-12 17:17 ` Greg KH
2019-06-10 19:14 ` [PATCH 12/12] closures: fix a race on wakeup from closure_sync Kent Overstreet
2019-07-16 10:47 ` Coly Li
2019-07-18 7:46 ` Coly Li
2019-07-22 17:22 ` Kent Overstreet
2020-11-24 23:07 ` Marc Smith
2020-11-25 18:10 ` Marc Smith
2019-06-10 20:46 ` bcachefs status update (it's done cooking; let's get this sucker merged) Linus Torvalds
2019-06-11 1:17 ` Kent Overstreet
2019-06-11 4:33 ` Dave Chinner
2019-06-12 16:21 ` Kent Overstreet
2019-06-12 23:02 ` Dave Chinner
2019-06-13 18:36 ` pagecache locking (was: bcachefs status update) merged) Kent Overstreet
2019-06-13 21:13 ` Andreas Dilger
2019-06-13 21:21 ` Kent Overstreet
2019-06-14 0:35 ` Dave Chinner [this message]
2019-06-13 23:55 ` Dave Chinner
2019-06-14 2:30 ` Linus Torvalds
2019-06-14 7:30 ` Dave Chinner
2019-06-15 1:15 ` Linus Torvalds
2019-06-14 3:08 ` Linus Torvalds
2019-06-15 4:01 ` Linus Torvalds
2019-06-17 22:47 ` Dave Chinner
2019-06-17 23:38 ` Linus Torvalds
2019-06-18 4:21 ` Amir Goldstein
2019-06-19 10:38 ` Jan Kara
2019-06-19 22:37 ` Dave Chinner
2019-07-03 0:04 ` pagecache locking Boaz Harrosh
[not found] ` <DM6PR19MB250857CB8A3A1C8279D6F2F3C5FB0@DM6PR19MB2508.namprd19.prod.outlook.com>
2019-07-03 1:25 ` Boaz Harrosh
2019-07-05 23:31 ` Dave Chinner
2019-07-07 15:05 ` Boaz Harrosh
2019-07-07 23:55 ` Dave Chinner
2019-07-08 13:31 ` Jan Kara
2019-07-09 23:47 ` Dave Chinner
2019-07-10 8:41 ` Jan Kara
2019-06-14 17:08 ` pagecache locking (was: bcachefs status update) merged) Kent Overstreet
2019-06-19 8:21 ` bcachefs status update (it's done cooking; let's get this sucker merged) Jan Kara
2019-07-03 1:04 ` [PATCH] mm: Support madvise_willneed override by Filesystems Boaz Harrosh
2019-07-03 17:21 ` Jan Kara
2019-07-03 18:03 ` Boaz Harrosh
2019-06-11 4:55 ` bcachefs status update (it's done cooking; let's get this sucker merged) Linus Torvalds
2019-06-11 14:26 ` Matthew Wilcox
2019-06-11 4:10 ` Dave Chinner
2019-06-11 4:39 ` Linus Torvalds
2019-06-11 7:10 ` Dave Chinner
2019-06-12 2:07 ` Linus Torvalds
2019-07-03 5:59 ` Stefan K
2020-06-10 11:02 ` Stefan K
2020-06-10 11:32 ` Kent Overstreet
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20190614003518.GL14363@dread.disaster.area \
--to=david@fromorbit.com \
--cc=adilger@dilger.ca \
--cc=akpm@linux-foundation.org \
--cc=amir73il@gmail.com \
--cc=darrick.wong@oracle.com \
--cc=dchinner@redhat.com \
--cc=hch@lst.de \
--cc=jack@suse.cz \
--cc=josef@toxicpanda.com \
--cc=kent.overstreet@gmail.com \
--cc=linux-fsdevel@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-xfs@vger.kernel.org \
--cc=torvalds@linux-foundation.org \
--cc=viro@zeniv.linux.org.uk \
--cc=willy@infradead.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.