From: Daniel Phillips <daniel@phunq.net>
To: Dave Chinner <david@fromorbit.com>, "Theodore Y. Ts'o" <tytso@mit.edu>
Cc: Andreas Dilger <adilger@dilger.ca>,
linux-ext4@vger.kernel.org, linux-kernel@vger.kernel.org,
linux-fsdevel@vger.kernel.org,
OGAWA Hirofumi <hirofumi@mail.parknet.co.jp>
Subject: Re: [RFC] Thing 1: Shardmap for Ext4
Date: Thu, 5 Dec 2019 21:09:28 -0800 [thread overview]
Message-ID: <1dd1f9f6-89a4-e73a-d7b9-94a12412876c@phunq.net> (raw)
In-Reply-To: <20191206011640.GQ2695@dread.disaster.area>
On 2019-12-05 5:16 p.m., Dave Chinner wrote:
> On Wed, Dec 04, 2019 at 06:41:06PM -0500, Theodore Y. Ts'o wrote:
>> On Wed, Dec 04, 2019 at 11:31:50AM -0700, Andreas Dilger wrote:
>>> One important use case that we have for Lustre that is not yet in the
>>> upstream ext4[*] is the ability to do parallel directory operations.
>>> This means we can create, lookup, and/or unlink entries in the same
>>> directory concurrently, to increase parallelism for large directories.
>>>
>>> [*] we've tried to submit the pdirops patch a couple of times, but the
>>> main blocker is that the VFS has a single directory mutex and couldn't
>>> use the added functionality without significant VFS changes.
>>> Patch at https://git.whamcloud.com/?p=fs/lustre-release.git;f=ldiskfs/kernel_patches/patches/rhel8/ext4-pdirop.patch;hb=HEAD
>>>
>>
>> The XFS folks recently added support for parallel directory operations
>> into the VFS, for the benefit of XFS has this feature.
>
> The use of shared i_rwsem locking on the directory inode during
> lookup/pathwalk allows for concurrent lookup/readdir operations on
> a single directory. However, the parent dir i_rwsem is still held
> exclusive for directory modifications like create, unlink, etc.
>
> IOWs, the VFS doesn't allow for concurrent directory modification
> right now, and that's going to be the limiting factor no matter what
> you do with internal filesystem locking.
On a scale of 0 to 10, how hard do you think that would be to relax
in VFS, given the restriction of no concurrent inter-directory moves?
next prev parent reply other threads:[~2019-12-06 5:09 UTC|newest]
Thread overview: 32+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-11-27 1:47 [RFC] Thing 1: Shardmap fox Ext4 Daniel Phillips
2019-11-27 7:40 ` Vyacheslav Dubeyko
2019-11-27 8:28 ` Daniel Phillips
2019-11-27 19:35 ` Viacheslav Dubeyko
2019-11-28 2:54 ` Daniel Phillips
2019-11-28 9:15 ` Andreas Dilger
2019-11-28 10:03 ` Daniel Phillips
2019-11-27 14:25 ` Theodore Y. Ts'o
2019-11-27 22:27 ` Daniel Phillips
2019-11-28 2:28 ` Theodore Y. Ts'o
2019-11-28 4:27 ` Daniel Phillips
2019-11-30 17:50 ` Theodore Y. Ts'o
2019-12-01 8:21 ` Daniel Phillips
2019-12-04 18:31 ` Andreas Dilger
2019-12-04 21:44 ` Daniel Phillips
2019-12-05 0:36 ` Andreas Dilger
2019-12-05 2:27 ` [RFC] Thing 1: Shardmap for Ext4 Daniel Phillips
2019-12-04 23:41 ` [RFC] Thing 1: Shardmap fox Ext4 Theodore Y. Ts'o
2019-12-06 1:16 ` Dave Chinner
2019-12-06 5:09 ` Daniel Phillips [this message]
2019-12-08 22:42 ` [RFC] Thing 1: Shardmap for Ext4 Dave Chinner
2019-11-28 21:17 ` [RFC] Thing 1: Shardmap fox Ext4 Daniel Phillips
2019-12-08 10:25 ` Daniel Phillips
2019-12-02 1:45 ` Daniel Phillips
2019-12-04 15:55 ` Vyacheslav Dubeyko
2019-12-05 9:46 ` Daniel Phillips
2019-12-06 11:47 ` Vyacheslav Dubeyko
2019-12-07 0:46 ` [RFC] Thing 1: Shardmap for Ext4 Daniel Phillips
2019-12-04 18:03 ` [RFC] Thing 1: Shardmap fox Ext4 Andreas Dilger
2019-12-04 20:47 ` Daniel Phillips
2019-12-04 20:53 ` Daniel Phillips
2019-12-05 5:59 ` Daniel Phillips
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1dd1f9f6-89a4-e73a-d7b9-94a12412876c@phunq.net \
--to=daniel@phunq.net \
--cc=adilger@dilger.ca \
--cc=david@fromorbit.com \
--cc=hirofumi@mail.parknet.co.jp \
--cc=linux-ext4@vger.kernel.org \
--cc=linux-fsdevel@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=tytso@mit.edu \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).