Re: [PATCH 05/10] xfs: automatic log item relog mechanism

[Brian Foster <bfoster@redhat.com>]

On Fri, Jul 03, 2020 at 04:08:23PM +1000, Dave Chinner wrote:
> On Wed, Jul 01, 2020 at 12:51:11PM -0400, Brian Foster wrote:
> > Now that relog reservation is available and relog state tracking is
> > in place, all that remains to automatically relog items is the relog
> > mechanism itself. An item with relogging enabled is basically pinned
> > from writeback until relog is disabled. Instead of being written
> > back, the item must instead be periodically committed in a new
> > transaction to move it forward in the physical log. The purpose of
> > moving the item is to avoid long term tail pinning and thus avoid
> > log deadlocks for long running operations.
> > 
> > The ideal time to relog an item is in response to tail pushing
> > pressure. This accommodates the current workload at any given time
> > as opposed to a fixed time interval or log reservation heuristic,
> > which risks performance regression. This is essentially the same
> > heuristic that drives metadata writeback. XFS already implements
> > various log tail pushing heuristics that attempt to keep the log
> > progressing on an active fileystem under various workloads.
> > 
> > The act of relogging an item simply requires to add it to a
> > transaction and commit. This pushes the already dirty item into a
> > subsequent log checkpoint and frees up its previous location in the
> > on-disk log. Joining an item to a transaction of course requires
> > locking the item first, which means we have to be aware of
> > type-specific locks and lock ordering wherever the relog takes
> > place.
> > 
> > Fundamentally, this points to xfsaild as the ideal location to
> > process relog enabled items. xfsaild already processes log resident
> > items, is driven by log tail pushing pressure, processes arbitrary
> > log item types through callbacks, and is sensitive to type-specific
> > locking rules by design. The fact that automatic relogging
> > essentially diverts items between writeback or relog also suggests
> > xfsaild as an ideal location to process items one way or the other.
> > 
> > Of course, we don't want xfsaild to process transactions as it is a
> > critical component of the log subsystem for driving metadata
> > writeback and freeing up log space. Therefore, similar to how
> > xfsaild builds up a writeback queue of dirty items and queues writes
> > asynchronously, make xfsaild responsible only for directing pending
> > relog items into an appropriate queue and create an async
> > (workqueue) context for processing the queue. The workqueue context
> > utilizes the pre-reserved log reservation to drain the queue by
> > rolling a permanent transaction.
> > 
> > Update the AIL pushing infrastructure to support a new RELOG item
> > state. If a log item push returns the relog state, queue the item
> > for relog instead of writeback. On completion of a push cycle,
> > schedule the relog task at the same point metadata buffer I/O is
> > submitted. This allows items to be relogged automatically under the
> > same locking rules and pressure heuristics that govern metadata
> > writeback.
> > 
> > Signed-off-by: Brian Foster <bfoster@redhat.com>
> 
> A note while it's still fresh in my mind: memory reclaim is going to
> force relogging of items whether they need it or not. The inode
> shrinker pushes the AIL to it's highest current LSN, which means the
> first shrinker invocation will relog the items. Sustained memory
> pressure will result in this sort of behaviour
> 
...
> 
> So it looks like when there is memory pressure we are going to
> trigger a relog every second AIL push cycle, and a synchronous log
> force every other log cycle.
> 

Indeed. I went back and forth on how to report status of already
relogged items so that bit is somewhat accidental. I could probably
remove the log force increment from the RELOG_QUEUED condition and let
the item fall back to the whatever state is most appropriate since I
didn't have an explicit reason for that other than trying to preserve
behavior from previous versions.

Ultimately even though much of this code is implemented around the AIL,
the AIL fundamentally serves as a notification mechanism to identify
when to relog items. All we really need is some indication that an item
is being pushed due to reservation pressure, whether it be a log item
state bit or a function callout, etc., and much of the rest of the
implementation could be lifted out into a separate mechanism. IOW, if
the push frequency of an item is too crude to drive relogs by itself,
that could probably be addressed by filtering the feedback mechanism to
exclude non head/tail related pressure. For example, the push code could
consider the heuristic implemented in xlog_grant_push_ail() to determine
whether to relog items at the tail, whether to force the log for already
relogged items, or just fall back to a traditional state.

Brian

> Cheers,
> 
> Dave.
> -- 
> Dave Chinner
> david@fromorbit.com
>