Re: [PATCH v13 02/10] xfs: Add delay ready attr remove routines

[Allison Henderson <allison.henderson@oracle.com>]

On 10/28/20 5:28 AM, Brian Foster wrote:
> On Tue, Oct 27, 2020 at 03:27:20PM -0700, Allison Henderson wrote:
>>
>>
>> On 10/27/20 5:16 AM, Brian Foster wrote:
>>> On Thu, Oct 22, 2020 at 11:34:27PM -0700, Allison Henderson wrote:
>>>> This patch modifies the attr remove routines to be delay ready. This
>>>> means they no longer roll or commit transactions, but instead return
>>>> -EAGAIN to have the calling routine roll and refresh the transaction. In
>>>> this series, xfs_attr_remove_args has become xfs_attr_remove_iter, which
>>>> uses a sort of state machine like switch to keep track of where it was
>>>> when EAGAIN was returned. xfs_attr_node_removename has also been
>>>> modified to use the switch, and a new version of xfs_attr_remove_args
>>>> consists of a simple loop to refresh the transaction until the operation
>>>> is completed.  A new XFS_DAC_DEFER_FINISH flag is used to finish the
>>>> transaction where ever the existing code used to.
>>>>
>>>> Calls to xfs_attr_rmtval_remove are replaced with the delay ready
>>>> version __xfs_attr_rmtval_remove. We will rename
>>>> __xfs_attr_rmtval_remove back to xfs_attr_rmtval_remove when we are
>>>> done.
>>>>
>>>> xfs_attr_rmtval_remove itself is still in use by the set routines (used
>>>> during a rename).  For reasons of preserving existing function, we
>>>> modify xfs_attr_rmtval_remove to call xfs_defer_finish when the flag is
>>>> set.  Similar to how xfs_attr_remove_args does here.  Once we transition
>>>> the set routines to be delay ready, xfs_attr_rmtval_remove is no longer
>>>> used and will be removed.
>>>>
>>>> This patch also adds a new struct xfs_delattr_context, which we will use
>>>> to keep track of the current state of an attribute operation. The new
>>>> xfs_delattr_state enum is used to track various operations that are in
>>>> progress so that we know not to repeat them, and resume where we left
>>>> off before EAGAIN was returned to cycle out the transaction. Other
>>>> members take the place of local variables that need to retain their
>>>> values across multiple function recalls.  See xfs_attr.h for a more
>>>> detailed diagram of the states.
>>>>
>>>> Signed-off-by: Allison Henderson <allison.henderson@oracle.com>
>>>> ---
>>>>    fs/xfs/libxfs/xfs_attr.c        | 200 +++++++++++++++++++++++++++++-----------
>>>>    fs/xfs/libxfs/xfs_attr.h        |  72 +++++++++++++++
>>>>    fs/xfs/libxfs/xfs_attr_leaf.c   |   2 +-
>>>>    fs/xfs/libxfs/xfs_attr_remote.c |  37 ++++----
>>>>    fs/xfs/libxfs/xfs_attr_remote.h |   2 +-
>>>>    fs/xfs/xfs_attr_inactive.c      |   2 +-
>>>>    6 files changed, 241 insertions(+), 74 deletions(-)
>>>>
>>>> diff --git a/fs/xfs/libxfs/xfs_attr.c b/fs/xfs/libxfs/xfs_attr.c
>>>> index f4d39bf..6ca94cb 100644
>>>> --- a/fs/xfs/libxfs/xfs_attr.c
>>>> +++ b/fs/xfs/libxfs/xfs_attr.c
> ...
>>>> @@ -1282,31 +1356,53 @@ xfs_attr_node_remove_step(
>>>>     *
>>>>     * This routine will find the blocks of the name to remove, remove them and
>>>>     * shirnk the tree if needed.
>>>> + *
>>>> + * This routine is meant to function as either an inline or delayed operation,
>>>> + * and may return -EAGAIN when the transaction needs to be rolled.  Calling
>>>> + * functions will need to handle this, and recall the function until a
>>>> + * successful error code is returned.
>>>>     */
>>>>    STATIC int
>>>> -xfs_attr_node_removename(
>>>> -	struct xfs_da_args	*args)
>>>> +xfs_attr_node_removename_iter(
>>>> +	struct xfs_delattr_context	*dac)
>>>>    {
>>>> -	struct xfs_da_state	*state;
>>>> -	int			error;
>>>> -	struct xfs_inode	*dp = args->dp;
>>>> +	struct xfs_da_args		*args = dac->da_args;
>>>> +	struct xfs_da_state		*state;
>>>> +	int				error;
>>>> +	struct xfs_inode		*dp = args->dp;
>>>>    	trace_xfs_attr_node_removename(args);
>>>> +	state = dac->da_state;
>>>> -	error = xfs_attr_node_removename_setup(args, &state);
>>>> -	if (error)
>>>> -		goto out;
>>>> +	if ((dac->flags & XFS_DAC_NODE_RMVNAME_INIT) == 0) {
>>>> +		dac->flags |= XFS_DAC_NODE_RMVNAME_INIT;
>>>> +		error = xfs_attr_node_removename_setup(dac, &state);
>>>> +		if (error)
>>>> +			goto out;
>>>> +	}
>>>> -	error = xfs_attr_node_remove_step(args, state);
>>>> -	if (error)
>>>> -		goto out;
>>>> +	switch (dac->dela_state) {
>>>> +	case XFS_DAS_UNINIT:
>>>> +		error = xfs_attr_node_remove_step(dac);
>>>> +		if (error)
>>>> +			break;
>>>
>>> I think there's a bit more preliminary refactoring to do here to isolate
>>> the state management to this one function. I.e., from the discussion on
>>> the previous version, we'd ideally pull the logic that checks for the
>>> subsequent shrink state out of xfs_attr_node_remove_step() and lift it
>>> into this branch. See the pseudocode in the previous discussion for an
>>> example of what I mean:
>>>
>>>     https://urldefense.com/v3/__https://lore.kernel.org/linux-xfs/20200901170020.GC174813@bfoster/__;!!GqivPVa7Brio!JKxU3Z07HVj0V1YFesXrveRWnoGuWqiTZuaIDiG9UFmSxz-aFGsZPpLtjIOjSht7WL_h$
>>>
>>> The general goal of that is to refactor the existing code such that all
>>> of the state transitions and whatnot are shown in one place and the rest
>>> is broken down into smaller functional helpers.
>>>
>>> Brian
>>
>> Yes, I did see the pseudo code, though I wasnt able to get it through the
>> test cases quite the way it appears.  Because whether or not we proceed to
>> shrink is determined by the leaf remove that's burried in the step function.
>> Otherwise we run into some failed asserts in the shrink routines for
>> "state->path.active == 1". Alternatly we can add another helper to pull it
>> up to this scope.  Then the switch ends up looking like this:
>>
> 
> Ok. That was more focused on just showing an approach that collects all
> of the state transition logic in one place. The factoring below seems
> generally reasonable to me..
> 
>>          switch (dac->dela_state) {
>>
>>          case XFS_DAS_UNINIT:
>>
>>                  /*
>>
>>                   * repeatedly remove remote blocks, remove the entry and
>>                   * join. returns -EAGAIN or 0 for completion of the step.
>>                   */
>>
>>                  error = xfs_attr_node_remove_step(dac);
>>
>>                  if (error)
>>
>>                          break;
>>
>>
>>
>>                  retval = xfs_attr_node_remove_cleanup(dac);
>>
>>
>>
>>                  /*
>>
>>                   * Check to see if the tree needs to be collapsed.  Set the
>> flag to
>>                   * indicate that the calling function needs to move the to
>> shrink
>>                   * operation
>>
>>                   */
>>
>>                  if (retval && (state->path.active > 1)) {
>>
>>                          error = xfs_da3_join(state);
>>
>>                          if (error)
>>
>>                                  return error;
>>
>>
>>
>>                          dac->dela_state = XFS_DAS_RM_SHRINK;
>>
>>                          return -EAGAIN;
>>
>>                  }
>>
>>
>>
>>                  /* check whether to shrink or return success */
>>
>>                  if (!error && xfs_bmap_one_block(dp, XFS_ATTR_FORK)) {
>>
>>                          dac->dela_state = XFS_DAS_RM_SHRINK;
>>
>>                          error = -EAGAIN;
>>
>>                  }
> 
> ... though what is the purpose of this hunk? This seems to diverge from
> the current logic, but maybe I'm missing something.
> 
That's just me trying to get as close to the psuedo code as possible, 
but if the concern is just to keep the state management in this scope, I 
may move it back down into the the XFS_DAS_RM_SHRINK case where it was, 
I think it looks a little cleaner there.


>>
>>                  break;
>>
>>          case XFS_DAS_RM_SHRINK:
>>
>>                  /*
>>
>>                   * If the result is small enough, push it all into the
>> inode.
>>                   */
>>
So it would go back down here like this, which is more akin to the 
original code flow:


		if (xfs_bmap_one_block(dp, XFS_ATTR_FORK))
>>                  error = xfs_attr_node_shrink(args, state);
>>
>>
>>
>>                  break;
>>
>>          default:
>>
>>                  ASSERT(0);
>>
>>                  return -EINVAL;
>>
>>          }
>>
>>
>>
>>          if (error == -EAGAIN)
>>
>>                  return error;
>>
>>
>>
>>
>>
>> And then we have this little clean up helper that we pull out of
>> remove_step:
>>
>>
>>
>> STATIC int
>>
>> xfs_attr_node_remove_cleanup(
>>
>>          struct xfs_delattr_context      *dac)
>>
>> {
>>
>>          struct xfs_da_args              *args = dac->da_args;
>>
>>          struct xfs_da_state_blk         *blk;
>>
>>          int                             retval;
>>
>>
>>
>>          /*
>>
>>           * Remove the name and update the hashvals in the tree.
>>
>>           */
>>
>>          blk = &dac->da_state->path.blk[ dac->da_state->path.active-1 ];
>>
>>          ASSERT(blk->magic == XFS_ATTR_LEAF_MAGIC);
>>
>>          retval = xfs_attr3_leaf_remove(blk->bp, args);
>>
>>          xfs_da3_fixhashpath(dac->da_state, &dac->da_state->path);
>>
>>
>>
>>          return retval;
>>
>> }
>>
>>
>> This configuration seems to get through the test cases.  It's not quite as
>> tidy, but it does get all the state handling into this scope.  If people
>> prefer it this way I can add in the extra helper and make these adjustments?
>>
> 
> Makes sense. I think the goal should be to pull the state management
> into one (or as few) place(s) as technically possible and we should
> refactor however necessary to accomplish that (please just pull the
> refactoring out into preliminary patches to facilitate review). Thanks.
> 
> Brian
Sure, I'll add in another patch for that extra helper in the next set. 
Thanks for the review!

Allison

> 
>> Allison
>>
>>>
>>>> -	/*
>>>> -	 * If the result is small enough, push it all into the inode.
>>>> -	 */
>>>> -	if (xfs_bmap_one_block(dp, XFS_ATTR_FORK))
>>>> -		error = xfs_attr_node_shrink(args, state);
>>>> +		/* do not break, proceed to shrink if needed */
>>>> +	case XFS_DAS_RM_SHRINK:
>>>> +		/*
>>>> +		 * If the result is small enough, push it all into the inode.
>>>> +		 */
>>>> +		if (xfs_bmap_one_block(dp, XFS_ATTR_FORK))
>>>> +			error = xfs_attr_node_shrink(args, state);
>>>> +		break;
>>>> +	default:
>>>> +		ASSERT(0);
>>>> +		return -EINVAL;
>>>> +	}
>>>> +
>>>> +	if (error == -EAGAIN)
>>>> +		return error;
>>>>    out:
>>>>    	if (state)
>>>>    		xfs_da_state_free(state);
>>>> diff --git a/fs/xfs/libxfs/xfs_attr.h b/fs/xfs/libxfs/xfs_attr.h
>>>> index 3e97a93..64dcf0f 100644
>>>> --- a/fs/xfs/libxfs/xfs_attr.h
>>>> +++ b/fs/xfs/libxfs/xfs_attr.h
>>>> @@ -74,6 +74,74 @@ struct xfs_attr_list_context {
>>>>    };
>>>> +/*
>>>> + * ========================================================================
>>>> + * Structure used to pass context around among the delayed routines.
>>>> + * ========================================================================
>>>> + */
>>>> +
>>>> +/*
>>>> + * Below is a state machine diagram for attr remove operations. The  XFS_DAS_*
>>>> + * states indicate places where the function would return -EAGAIN, and then
>>>> + * immediately resume from after being recalled by the calling function. States
>>>> + * marked as a "subroutine state" indicate that they belong to a subroutine, and
>>>> + * so the calling function needs to pass them back to that subroutine to allow
>>>> + * it to finish where it left off. But they otherwise do not have a role in the
>>>> + * calling function other than just passing through.
>>>> + *
>>>> + * xfs_attr_remove_iter()
>>>> + *	  XFS_DAS_RM_SHRINK ─┐
>>>> + *	  (subroutine state) │
>>>> + *	                     └─>xfs_attr_node_removename()
>>>> + *	                                      │
>>>> + *	                                      v
>>>> + *	                                   need to
>>>> + *	                                shrink tree? ─n─┐
>>>> + *	                                      │         │
>>>> + *	                                      y         │
>>>> + *	                                      │         │
>>>> + *	                                      v         │
>>>> + *	                              XFS_DAS_RM_SHRINK │
>>>> + *	                                      │         │
>>>> + *	                                      v         │
>>>> + *	                                     done <─────┘
>>>> + *
>>>> + */
>>>> +
>>>> +/*
>>>> + * Enum values for xfs_delattr_context.da_state
>>>> + *
>>>> + * These values are used by delayed attribute operations to keep track  of where
>>>> + * they were before they returned -EAGAIN.  A return code of -EAGAIN signals the
>>>> + * calling function to roll the transaction, and then recall the subroutine to
>>>> + * finish the operation.  The enum is then used by the subroutine to jump back
>>>> + * to where it was and resume executing where it left off.
>>>> + */
>>>> +enum xfs_delattr_state {
>>>> +	XFS_DAS_UNINIT		= 0,  /* No state has been set yet */
>>>> +	XFS_DAS_RM_SHRINK,	      /* We are shrinking the tree */
>>>> +};
>>>> +
>>>> +/*
>>>> + * Defines for xfs_delattr_context.flags
>>>> + */
>>>> +#define XFS_DAC_DEFER_FINISH		0x01 /* finish the transaction */
>>>> +#define XFS_DAC_NODE_RMVNAME_INIT	0x02 /* xfs_attr_node_removename init */
>>>> +
>>>> +/*
>>>> + * Context used for keeping track of delayed attribute operations
>>>> + */
>>>> +struct xfs_delattr_context {
>>>> +	struct xfs_da_args      *da_args;
>>>> +
>>>> +	/* Used in xfs_attr_node_removename to roll through removing blocks */
>>>> +	struct xfs_da_state     *da_state;
>>>> +
>>>> +	/* Used to keep track of current state of delayed operation */
>>>> +	unsigned int            flags;
>>>> +	enum xfs_delattr_state  dela_state;
>>>> +};
>>>> +
>>>>    /*========================================================================
>>>>     * Function prototypes for the kernel.
>>>>     *========================================================================*/
>>>> @@ -91,6 +159,10 @@ int xfs_attr_set(struct xfs_da_args *args);
>>>>    int xfs_attr_set_args(struct xfs_da_args *args);
>>>>    int xfs_has_attr(struct xfs_da_args *args);
>>>>    int xfs_attr_remove_args(struct xfs_da_args *args);
>>>> +int xfs_attr_remove_iter(struct xfs_delattr_context *dac);
>>>> +int xfs_attr_trans_roll(struct xfs_delattr_context *dac);
>>>>    bool xfs_attr_namecheck(const void *name, size_t length);
>>>> +void xfs_delattr_context_init(struct xfs_delattr_context *dac,
>>>> +			      struct xfs_da_args *args);
>>>>    #endif	/* __XFS_ATTR_H__ */
>>>> diff --git a/fs/xfs/libxfs/xfs_attr_leaf.c b/fs/xfs/libxfs/xfs_attr_leaf.c
>>>> index bb128db..338377e 100644
>>>> --- a/fs/xfs/libxfs/xfs_attr_leaf.c
>>>> +++ b/fs/xfs/libxfs/xfs_attr_leaf.c
>>>> @@ -19,8 +19,8 @@
>>>>    #include "xfs_bmap_btree.h"
>>>>    #include "xfs_bmap.h"
>>>>    #include "xfs_attr_sf.h"
>>>> -#include "xfs_attr_remote.h"
>>>>    #include "xfs_attr.h"
>>>> +#include "xfs_attr_remote.h"
>>>>    #include "xfs_attr_leaf.h"
>>>>    #include "xfs_error.h"
>>>>    #include "xfs_trace.h"
>>>> diff --git a/fs/xfs/libxfs/xfs_attr_remote.c b/fs/xfs/libxfs/xfs_attr_remote.c
>>>> index 48d8e9c..1426c15 100644
>>>> --- a/fs/xfs/libxfs/xfs_attr_remote.c
>>>> +++ b/fs/xfs/libxfs/xfs_attr_remote.c
>>>> @@ -674,10 +674,12 @@ xfs_attr_rmtval_invalidate(
>>>>     */
>>>>    int
>>>>    xfs_attr_rmtval_remove(
>>>> -	struct xfs_da_args      *args)
>>>> +	struct xfs_da_args		*args)
>>>>    {
>>>> -	int			error;
>>>> -	int			retval;
>>>> +	int				error;
>>>> +	struct xfs_delattr_context	dac  = {
>>>> +		.da_args	= args,
>>>> +	};
>>>>    	trace_xfs_attr_rmtval_remove(args);
>>>> @@ -685,19 +687,17 @@ xfs_attr_rmtval_remove(
>>>>    	 * Keep de-allocating extents until the remote-value region is gone.
>>>>    	 */
>>>>    	do {
>>>> -		retval = __xfs_attr_rmtval_remove(args);
>>>> -		if (retval && retval != -EAGAIN)
>>>> -			return retval;
>>>> +		error = __xfs_attr_rmtval_remove(&dac);
>>>> +		if (error != -EAGAIN)
>>>> +			break;
>>>> -		/*
>>>> -		 * Close out trans and start the next one in the chain.
>>>> -		 */
>>>> -		error = xfs_trans_roll_inode(&args->trans, args->dp);
>>>> +		error = xfs_attr_trans_roll(&dac);
>>>>    		if (error)
>>>>    			return error;
>>>> -	} while (retval == -EAGAIN);
>>>> -	return 0;
>>>> +	} while (true);
>>>> +
>>>> +	return error;
>>>>    }
>>>>    /*
>>>> @@ -707,9 +707,10 @@ xfs_attr_rmtval_remove(
>>>>     */
>>>>    int
>>>>    __xfs_attr_rmtval_remove(
>>>> -	struct xfs_da_args	*args)
>>>> +	struct xfs_delattr_context	*dac)
>>>>    {
>>>> -	int			error, done;
>>>> +	struct xfs_da_args		*args = dac->da_args;
>>>> +	int				error, done;
>>>>    	/*
>>>>    	 * Unmap value blocks for this attr.
>>>> @@ -719,12 +720,10 @@ __xfs_attr_rmtval_remove(
>>>>    	if (error)
>>>>    		return error;
>>>> -	error = xfs_defer_finish(&args->trans);
>>>> -	if (error)
>>>> -		return error;
>>>> -
>>>> -	if (!done)
>>>> +	if (!done) {
>>>> +		dac->flags |= XFS_DAC_DEFER_FINISH;
>>>>    		return -EAGAIN;
>>>> +	}
>>>>    	return error;
>>>>    }
>>>> diff --git a/fs/xfs/libxfs/xfs_attr_remote.h b/fs/xfs/libxfs/xfs_attr_remote.h
>>>> index 9eee615..002fd30 100644
>>>> --- a/fs/xfs/libxfs/xfs_attr_remote.h
>>>> +++ b/fs/xfs/libxfs/xfs_attr_remote.h
>>>> @@ -14,5 +14,5 @@ int xfs_attr_rmtval_remove(struct xfs_da_args *args);
>>>>    int xfs_attr_rmtval_stale(struct xfs_inode *ip, struct xfs_bmbt_irec *map,
>>>>    		xfs_buf_flags_t incore_flags);
>>>>    int xfs_attr_rmtval_invalidate(struct xfs_da_args *args);
>>>> -int __xfs_attr_rmtval_remove(struct xfs_da_args *args);
>>>> +int __xfs_attr_rmtval_remove(struct xfs_delattr_context *dac);
>>>>    #endif /* __XFS_ATTR_REMOTE_H__ */
>>>> diff --git a/fs/xfs/xfs_attr_inactive.c b/fs/xfs/xfs_attr_inactive.c
>>>> index bfad669..aaa7e66 100644
>>>> --- a/fs/xfs/xfs_attr_inactive.c
>>>> +++ b/fs/xfs/xfs_attr_inactive.c
>>>> @@ -15,10 +15,10 @@
>>>>    #include "xfs_da_format.h"
>>>>    #include "xfs_da_btree.h"
>>>>    #include "xfs_inode.h"
>>>> +#include "xfs_attr.h"
>>>>    #include "xfs_attr_remote.h"
>>>>    #include "xfs_trans.h"
>>>>    #include "xfs_bmap.h"
>>>> -#include "xfs_attr.h"
>>>>    #include "xfs_attr_leaf.h"
>>>>    #include "xfs_quota.h"
>>>>    #include "xfs_dir2.h"
>>>> -- 
>>>> 2.7.4
>>>>
>>>
>>
>