From: Dave Chinner <david@fromorbit.com>
To: "J. Bruce Fields" <bfields@fieldses.org>
Cc: Masayoshi Mizuma <msys.mizuma@gmail.com>,
Eric Sandeen <sandeen@sandeen.net>,
"Darrick J. Wong" <darrick.wong@oracle.com>,
Christoph Hellwig <hch@infradead.org>,
Theodore Ts'o <tytso@mit.edu>,
Andreas Dilger <adilger.kernel@dilger.ca>,
Alexander Viro <viro@zeniv.linux.org.uk>,
Masayoshi Mizuma <m.mizuma@jp.fujitsu.com>,
linux-ext4@vger.kernel.org, linux-fsdevel@vger.kernel.org,
linux-xfs <linux-xfs@vger.kernel.org>,
jlayton@redhat.com
Subject: Re: [PATCH] fs: i_version mntopt gets visible through /proc/mounts
Date: Tue, 23 Jun 2020 08:03:54 +1000 [thread overview]
Message-ID: <20200622220354.GU2005@dread.disaster.area> (raw)
In-Reply-To: <20200622212612.GA11051@fieldses.org>
On Mon, Jun 22, 2020 at 05:26:12PM -0400, J. Bruce Fields wrote:
> On Sun, Jun 21, 2020 at 09:54:08AM +1000, Dave Chinner wrote:
> > On Fri, Jun 19, 2020 at 09:56:33PM -0400, J. Bruce Fields wrote:
> > > On Sat, Jun 20, 2020 at 11:49:57AM +1000, Dave Chinner wrote:
> > > > However, other people have different opinions on this matter (and we
> > > > know that from the people who considered XFS v4 -> v5 going slower
> > > > because iversion a major regression), and so we must acknowledge
> > > > those opinions even if we don't agree with them.
> > >
> > > Do you have any of those reports handy? Were there numbers?
> >
> > e.g. RH BZ #1355813 when v5 format was enabled by default in RHEL7.
> > Numbers were 40-47% performance degradation for in-cache writes
> > caused by the original IVERSION implementation using iozone. There
> > were others I recall, all realted to similar high-IOP small random
> > writes workloads typical of databases....
>
> Thanks, that's an interesting bug! Though a bit tangled. This is where
> you identified the change attribute as the main culprit:
>
> https://bugzilla.redhat.com/show_bug.cgi?id=1355813#c42
>
> The test was running at 70,000 writes/s (2.2GB/s), so it was one
> transaction per write() syscall: timestamp updates. On CRC
> enabled filesystems, we have a change counter for NFSv4 - it
> gets incremented on every write() syscall, even when the
> timestamp doesn't change. That's the difference in behaviour and
> hence performance in this test.
>
> In RHEL8, or anything post-v4.16, the frequency of change attribute
> updates should be back down to that of timestamp updates on this
> workload. So it'd be interesting to repeat that experiment now.
Yup, which in itself has been a problem for similar workloads.
There's a reason we now recommend the use of lazytime for high
performance database workloads that can do hundreds of thousands of
small write IOs a second...
> The bug was reporting in-house testing, and doesn't show any evidence
> that particular regression was encountered by users; Eric said:
>
> https://bugzilla.redhat.com/show_bug.cgi?id=1355813#c52
>
> Root cause of this minor in-memory regression was inode
> versioning behavior; as it's unlikely to have real-world effects
> (and has been open for years with no customer complaints) I'm
> closing this WONTFIX to get it off the radar.
It's just the first I found because bugzilla has a slow, less than
useful search engine. We know that real applications have
hit this, and we know even the overhead of timestamp updates on
writes is way too high for them.
> The typical user may just skip an upgrade or otherwise work around the
> problem rather than root-causing it like this, so absence of reports
> isn't conclusive. I understand wanting to err on the side of caution.
Yup, it's a generic problem - just because we've worked around or
mitigated the most common situations it impacts performance, that
doesn't mean they work for everyone....
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
next prev parent reply other threads:[~2020-06-22 22:04 UTC|newest]
Thread overview: 33+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-06-16 20:21 [PATCH] fs: i_version mntopt gets visible through /proc/mounts Masayoshi Mizuma
2020-06-17 8:03 ` Christoph Hellwig
2020-06-17 13:33 ` Masayoshi Mizuma
2020-06-17 15:58 ` J. Bruce Fields
2020-06-17 17:14 ` Eric Sandeen
2020-06-17 17:24 ` Darrick J. Wong
2020-06-17 17:55 ` Eric Sandeen
2020-06-17 18:18 ` J. Bruce Fields
2020-06-17 18:28 ` Eric Sandeen
2020-06-17 18:45 ` J. Bruce Fields
2020-06-18 1:30 ` Masayoshi Mizuma
2020-06-18 1:44 ` Darrick J. Wong
2020-06-18 3:33 ` Masayoshi Mizuma
2020-06-18 3:05 ` Dave Chinner
2020-06-18 3:45 ` Masayoshi Mizuma
2020-06-18 22:39 ` Dave Chinner
2020-06-19 2:20 ` J. Bruce Fields
2020-06-19 2:44 ` Dave Chinner
2020-06-19 12:04 ` Jeff Layton
2020-06-19 20:40 ` J. Bruce Fields
2020-06-19 22:10 ` Dave Chinner
2020-06-19 22:28 ` J. Bruce Fields
2020-06-20 1:49 ` Dave Chinner
2020-06-20 1:56 ` J. Bruce Fields
2020-06-20 17:00 ` Eric Sandeen
2020-06-20 17:09 ` J. Bruce Fields
2020-06-20 23:54 ` Dave Chinner
2020-06-22 21:26 ` J. Bruce Fields
2020-06-22 22:03 ` Dave Chinner [this message]
2020-06-19 13:17 ` Christoph Hellwig
2020-07-13 23:45 ` Eric Sandeen
2020-07-14 8:30 ` Christoph Hellwig
2020-07-14 20:26 ` Eric Sandeen
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20200622220354.GU2005@dread.disaster.area \
--to=david@fromorbit.com \
--cc=adilger.kernel@dilger.ca \
--cc=bfields@fieldses.org \
--cc=darrick.wong@oracle.com \
--cc=hch@infradead.org \
--cc=jlayton@redhat.com \
--cc=linux-ext4@vger.kernel.org \
--cc=linux-fsdevel@vger.kernel.org \
--cc=linux-xfs@vger.kernel.org \
--cc=m.mizuma@jp.fujitsu.com \
--cc=msys.mizuma@gmail.com \
--cc=sandeen@sandeen.net \
--cc=tytso@mit.edu \
--cc=viro@zeniv.linux.org.uk \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).