[XFS SUMMIT] AIL Behavioural Issues

[XFS SUMMIT] AIL Behavioural Issues

[Dave Chinner]

Topic:	AIL Behavioural Issues

Scope:
	Performance
	Efficiency
	Scalability

Proposal:

The XFS AIL numerous issues on high performance storage and
highly concurrent workloads. These can be broken down into several
categories:

1. Tracks too many redundant objects

We track inodes and dquots individually in the AIL, but we push
them as groups based on the backing buffer for the objects. For
inodes, this means we can have 32 inode items in the AIL for a
single buffer. We push the first inode and queue the buffer to the
delwri list, then we have to skip the other 31 inodes on that buffer
in the AIL.

This typically shows up as having a "success to flushing" ratio of
~30:1 in the AIL stats. We are doing 30x more work that we need to
inserting, iterating, and removing inodes in the AIL than required
to flush them. dquots have a similar problem.

So the question becomes this: how do we efficiently track related
objects that share a common backing object through the AIL? For
inodes, I'm attempting to do this with ordered buffers, but that is
proving to have a lot of lock ordering issues associated with it.

Perhaps there is a better way? Discuss.

2. Creates massive IO lists and so iteration latency can be multiple
  seconds as IO submission blocks

When tail pushing, the AIL will just keep building a delayed write
buffer list as long as it doesn't get stuck or hit it's target. This
can result in a submission list tens of thousands of buffers long
and hence the IO submission process ends up blocking on full request
queues and being throttled to the rate at which IO completions
occur. On slow storage, this can result in the AIL being blocked for
seconds (or more) while it works through the submission list.

If the item that pins the tail of the log is at the end of the
submission list, then the transaction reservations also block
waiting for the tail of the log to move, and hence everything in the
filesystem stops until submissions and IO completes. Then the AIL
does another scan, creates another huge list, and blocks again....

The long lists have the advantage that sorting them allows good IO
merging to take place. This reduces the IO time for slow disks
because it reduces seeks, but then the long list still takes a long
time to work through. It's a catch-22 situation.

Several things to discuss:
	- shorter lists to reduce submission latency
	- 2-dimensional AIL lists
		- existing LSN ordered list
		- additional daddr ordered list
	- no sorting?
	- immediate dispatch rather than delwri lists?
	- IO submission in a separate thread


3. Can block on CIL pushing

When the AIL hits pinned items, it will eventually issue a log force
to try to get them unpinned and so make progress. It does this when
it reaches it's target and there pinned objects that couldn't be
pushed prior to reaching the target.

The log force, while not a synchronous log force, still ends up
blocking on the CIL push to flush everything to the iclogs. This,
unfortunately, typically results in the CIL push blocking waiting
for iclog submission and IO completion as the amount of dirty
metadata in the CIL is greater than iclogs * iclogs_size. Hence tail
pushing can actually block behind log writes rather than just
triggering a CIL push to occur in the background and continuing to
scan for items to push.

discussion points:
	- do we even care if we hit pinned buffers in the AIL push
	  anymore?
	- do we actually need to wait for the CIL flush in the
	  AIL code?
	- do any async log forces need to wait for the CIL flush?
	- do we need a separate "background CIL push" API for
	  the AIL to use?

4. Single threaded

Because the AIL is single threaded, there is a hard limit on the
amount of scanning and IO submission work it can do, and any place
it blocks takes away from it's capacity to flush dirty metadata to
disk.

Discussion points:
	- how long can we get away with efficiency improvements to
	  keep ahead of tens to hundreds of CPUs dirtying metadata?
	- how do break up multi-threaded pushing?
	- can we use multiple AILs (e.g. 1 per AG) and co-ordinate
	  flush targets across them?
	- if we can parallelise the AIL, how much do we care about
	  other improvements?


5. Does not control it's own activity

Typically the AIL is invoked to push on dirty metadata items when
the log reservations pass 75% of the size of the log. This works
great when the log is small because we hit the threshold quickly and
frequently and so the AIL is essentially being told to "just keep
working".

However, when we have large journals, 75% of the log can be 1.5GB of
space, and so it can take some time for the AIL to be woken to do
work. If lots of modifications are taking place, then the log can
run out of space before the AIL has taken any significant action to
start writeback, and because there are so many objects in the AIL it
struggles initially to reach a steady state where tail pushing keeps
enough log space free that incoming transaction reservations don't
have to use the slow wait path.

To make matters more complex, other events will cause the AIL to try
to push everything to disk. Every 30s the log worker comes along and
tells the AIL to "push everything". And every time memory reclaim
runs the shrinker tells the AIL to "push everything". This often
cause the AIL to walk into recently committed and relogged objects
to try to push them, often resulting finding pinned objects and
hence issuing frequent (and unnecessary) log forces to try to unpin
them.

Hence under heavy loads, the AIL really ends up just trying to push
everything all the time, and that ends up exacerbating the
inefficiencies and blocking issues that result from long IO lists
and log forces.

The question I'm asking here is whether this should be turned upside
down. That is, rather than have random callers tell the AIL what it
should do, the AIL itself should decide when are where to push
items. The AIL knows exactly how much space is active in the log -
the head and tail of the AIL define the on-disk log head and tail -
nad when combined with visibility of the amount of unused
reservation space we have a pretty good idea of:

	a) how quickly pushing and IO completions is moving the
	   tail forwards;
	b) how quicly the head is being moved forwards by the
	   workload committing transactions; and
	c) how much demand there is for log space by the workload
	   transaction reservation rate.

From this information we should be able to determine how much work
we need to do, and how long we can leave items in the AIL before we
need to flush them. If there is no reservation pressure, we don't
need to rush to push the tail of the log. If we are moving the tail
of the log faster than the workload is moving the head, then we
don't have to flush the AIL flat out all the time. And so on.

So this is a discussion opportunity to design a better tail pushing
algorithm that is not dependent on being told what to do, but
instead use observation of the rate of change of critical metrics to
determine the rate at which we need to do push work....

-- 
Dave Chinner
david@fromorbit.com