Linux-ext4 Archive on lore.kernel.org
 help / color / Atom feed
From: Ira Weiny <ira.weiny@intel.com>
To: Jan Kara <jack@suse.cz>
Cc: Dave Chinner <david@fromorbit.com>,
	linux-kernel@vger.kernel.org,
	Alexander Viro <viro@zeniv.linux.org.uk>,
	"Darrick J. Wong" <darrick.wong@oracle.com>,
	Dan Williams <dan.j.williams@intel.com>,
	Christoph Hellwig <hch@lst.de>,
	"Theodore Y. Ts'o" <tytso@mit.edu>,
	linux-ext4@vger.kernel.org, linux-xfs@vger.kernel.org,
	linux-fsdevel@vger.kernel.org
Subject: Re: [PATCH 5/5] fs/xfs: Allow toggle of physical DAX flag
Date: Fri, 8 Nov 2019 11:36:13 -0800
Message-ID: <20191108193612.GA4800@iweiny-DESK2.sc.intel.com> (raw)
In-Reply-To: <20191108134606.GL20863@quack2.suse.cz>

On Fri, Nov 08, 2019 at 02:46:06PM +0100, Jan Kara wrote:
> On Fri 08-11-19 14:12:38, Jan Kara wrote:
> > On Mon 21-10-19 15:49:31, Ira Weiny wrote:
> > > On Mon, Oct 21, 2019 at 11:45:36AM +1100, Dave Chinner wrote:
> > > > On Sun, Oct 20, 2019 at 08:59:35AM -0700, ira.weiny@intel.com wrote:
> > > > That, fundamentally, is the issue here - it's not setting/clearing
> > > > the DAX flag that is the issue, it's doing a swap of the
> > > > mapping->a_ops while there may be other code using that ops
> > > > structure.
> > > > 
> > > > IOWs, if there is any code anywhere in the kernel that
> > > > calls an address space op without holding one of the three locks we
> > > > hold here (i_rwsem, MMAPLOCK, ILOCK) then it can race with the swap
> > > > of the address space operations.
> > > > 
> > > > By limiting the address space swap to file sizes of zero, we rule
> > > > out the page fault path (mmap of a zero length file segv's with an
> > > > access beyond EOF on the first read/write page fault, right?).
> > > 
> > > Yes I checked that and thought we were safe here...
> > > 
> > > > However, other aops callers that might run unlocked and do the wrong
> > > > thing if the aops pointer is swapped between check of the aop method
> > > > existing and actually calling it even if the file size is zero?
> > > > 
> > > > A quick look shows that FIBMAP (ioctl_fibmap())) looks susceptible
> > > > to such a race condition with the current definitions of the XFS DAX
> > > > aops. I'm guessing there will be others, but I haven't looked
> > > > further than this...
> > > 
> > > I'll check for others and think on what to do about this.  ext4 will have the
> > > same problem I think.  :-(
> > 
> > Just as a datapoint, ext4 is bold and sets inode->i_mapping->a_ops on
> > existing inodes when switching journal data flag and so far it has not
> > blown up. What we did to deal with issues Dave describes is that we
> > introduced percpu rw-semaphore guarding switching of aops and then inside
> > problematic functions redirect callbacks in the right direction under this
> > semaphore. Somewhat ugly but it seems to work.

Ah I am glad you brought this up.  I had not seen this before.

Is that s_journal_flag_rwsem?

In the general case I don't think that correctly protects against:

	if (a_ops->call)
		a_ops->call();

Because not all operations are defined in both ext4_aops and
ext4_journalled_aops.  Specifically migratepage.

move_to_new_page() specifically follows the pattern above with migratepage.  So
is there a bug here?

> 
> Thinking about this some more, perhaps this scheme could be actually
> transformed in something workable. We could have a global (or maybe per-sb
> but I'm not sure it's worth it) percpu rwsem and we could transform aops
> calls into:
> 
> percpu_down_read(aops_rwsem);
> do_call();
> percpu_up_read(aops_rwsem);
> 
> With some macro magic it needn't be even that ugly.

I think this is safer.  And what I have been investigating/coding up.  Because
that also would protect against the above with:

percpu_down_read(aops_rwsem);
	if (a_ops->call)
		a_ops->call();
percpu_up_read(aops_rwsem);

However I have been looking at SRCU because we also have patterns like:


	generic_file_buffered_read
		if (a_ops->is_partially_uptodate)
			a_ops->is_partially_uptodate()
		page_cache_sync_readahead
			force_page_cache_readahead
				if (!a_ops->readpage && !a_ops->readpages)
					return;
				__do_page_cache_readahead
					read_pages
						if (a_ops->readpages)
							a_ops->readpages()
						a_ops->readpage


So we would have to pass the a_ops through to use a rwsem.  Where SRCU I think
would be fine to just take the SRCU read lock multiple times.  Am I wrong?


We also have a 3rd (2nd?) issue.  There are callers who check for the presence
of an operation to be used later.  For example do_dentry_open():

do_dentry_open()
{
...
	if (<flags> & O_DIRECT)
		if (!<a_ops> || !<a_ops>->direct_IO)
			return -EINVAL;
...
}

After this open direct_IO better be there AFAICT so changing the a_ops later
would not be good.  For ext4 direct_IO is defined for all the a_ops...  so I
guess that is not a big deal.  However, is the user really getting the behavior
they expect in this case?

I'm afraid of requiring FSs to have to follow rules in defining their a_ops.
Because I'm afraid maintaining those rules would be hard and would eventually
lead to crashes when someone did it wrong.

:-/

So for this 3rd (2nd) case I think we should simply take a reference to the
a_ops and fail changing the mode.  For the DAX case that means the user is best
served by taking a write lease on the file to ensure there are no other opens
which could cause issues.

Would that work for changing the journaling mode?

And I _think_ this is the only issue we have with this right now. But if other
callers of a_ops needed the pattern of using the a_ops at a time across context
changes they would need to ensure this reference was taken.

What I have come up with thus far is an interface like:

/*
 * as_get_a_ops() -- safely get the a_ops from the address_space specified
 *
 * @as: address space to get a_ops from
 * @ref: used to indicate if a reference is required on this a_ops
 * @tok: srcu token to be returned in as_put_a_ops()
 *
 * The a_ops returned is protected from changing until as_put_a_ops().
 *
 * If ref is specified then ref must also be specified in as_put_a_ops() to
 * release this reference.  In this case a reference is taken on the a_ops
 * which will prevent it from changing until the reference is released.
 *
 * References should _ONLY_ be taken when the a_ops needs to be constant
 * across a user context switch because doing so will block changing the a_ops
 * until that reference is released.
 *
 * Examples of using a reference are checks for specific a_ops pointers which
 * are expected to support functionality at a later date (example direct_IO)
 */
static inline const struct address_space_operations *
as_get_a_ops(struct address_space *as, bool ref, int *tok)
{
	...
}

static inline void
as_assign_a_ops(struct address_space *as,
                const struct address_space_operations *a_ops)
{
	...
}

static inline void as_put_a_ops(struct address_space *as, int tok, bool ref)
{
	...
}


I'm still working out the details of using SRCU and a ref count.  I have made
at least 1 complete pass of all the a_ops users and I think this would cover
them all.

Thoughts?
Ira

> 
> 								Honza
> -- 
> Jan Kara <jack@suse.com>
> SUSE Labs, CR

  reply index

Thread overview: 33+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-10-20 15:59 [PATCH 0/5] Enable per-file/directory DAX operations ira.weiny
2019-10-20 15:59 ` [PATCH 1/5] fs/stat: Define DAX statx attribute ira.weiny
2019-10-22 11:32   ` Boaz Harrosh
2019-10-22 16:51     ` Ira Weiny
2019-10-20 15:59 ` [PATCH 2/5] fs/xfs: Isolate the physical DAX flag from effective ira.weiny
2019-10-21  0:26   ` Dave Chinner
2019-10-21 17:40     ` Ira Weiny
2019-10-20 15:59 ` [PATCH 3/5] fs/xfs: Separate functionality of xfs_inode_supports_dax() ira.weiny
2019-10-20 15:59 ` [PATCH 4/5] fs/xfs: Clean up DAX support check ira.weiny
2019-10-20 15:59 ` [PATCH 5/5] fs/xfs: Allow toggle of physical DAX flag ira.weiny
2019-10-21  0:45   ` Dave Chinner
2019-10-21 22:49     ` Ira Weiny
2019-10-21 23:46       ` Dave Chinner
2019-11-08 13:12       ` Jan Kara
2019-11-08 13:46         ` Jan Kara
2019-11-08 19:36           ` Ira Weiny [this message]
2019-11-11 16:07             ` Jan Kara
2019-11-11 23:54               ` Ira Weiny
2019-10-22 11:21 ` [PATCH 0/5] Enable per-file/directory DAX operations Boaz Harrosh
2019-10-23 13:09   ` Boaz Harrosh
2019-10-23 22:13     ` Dave Chinner
2019-10-24  2:31       ` Boaz Harrosh
2019-10-24  7:34         ` Dave Chinner
2019-10-24 14:05           ` Boaz Harrosh
2019-10-24 21:35             ` Dave Chinner
2019-10-24 23:29               ` Boaz Harrosh
2019-10-25  0:36                 ` Dave Chinner
2019-10-25  1:15                   ` Boaz Harrosh
2019-10-25 20:49                   ` Ira Weiny
2019-10-27 22:10                     ` Dave Chinner
2019-10-31 16:17                       ` Ira Weiny
2019-11-01 22:47                         ` Dave Chinner
2019-11-02  4:25                           ` Dan Williams

Reply instructions:

You may reply publically to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20191108193612.GA4800@iweiny-DESK2.sc.intel.com \
    --to=ira.weiny@intel.com \
    --cc=dan.j.williams@intel.com \
    --cc=darrick.wong@oracle.com \
    --cc=david@fromorbit.com \
    --cc=hch@lst.de \
    --cc=jack@suse.cz \
    --cc=linux-ext4@vger.kernel.org \
    --cc=linux-fsdevel@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-xfs@vger.kernel.org \
    --cc=tytso@mit.edu \
    --cc=viro@zeniv.linux.org.uk \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

Linux-ext4 Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/linux-ext4/0 linux-ext4/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 linux-ext4 linux-ext4/ https://lore.kernel.org/linux-ext4 \
		linux-ext4@vger.kernel.org
	public-inbox-index linux-ext4

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.linux-ext4


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git