[XFS SUMMIT] Data writeback scalability

[XFS SUMMIT] Data writeback scalability

[Dave Chinner]

Topic:	Data writeback scalability

Scope:	Performance
	Generic writeback infrastructure limitations
	Buffered IO sucks.

Proposal:

This is largely a "how do we solve this" discussion topic.

In running small file writeback tests recently, I've realised that
the single biggest performance limitation is the single threaded BDI
flush worker that builds data IOs from the page cache of dirty
inodes. XFS does delayed allocation, so writeback overhead is
largely determined by how many physical allocations need to be done
to write back the dirty data.

It turns out that it's not that many. The BDI flusher becomes
completely CPU bound at about 80-90k allocations per second, which
means we are limited to writing back data to that many spearate
regions per second. If we are writing back small files (say 4kB
each), then the generic data writeback infrastructure cannot clean
more than 100k files/s. I've got a device that can do 1.6M random
4kB write IOPS every second with aio+dio, yet I can't use more than
about 5% of that capacity through the page cache....

SO I gamed the system: I set an extent size hint of the root
directory of 4KB so that delayed allocation is never used. That got
me to 160k files/s in about 40k IOPS and the flusher thread about
70% busy. Everything was still blocking in
balance_dirty_pages_ratelimited(), so there's still a huge amount of
IO performance being left on the table because we just can't flush
dirty pages fast enough to keep modern SSDs busy from a single
thread.

IOWs, this is a discussion topic for how we might work towards using
multiple data flushing threads efficiently for XFS. Most efficient
would be a flusher thread per AG, but that is unrealistic for high
AG count filesystems. Similarly, per-CPU flushers does no good if
we've only got 4 AGs in the filesystem.

This is made more difficult because the high speed collision that
occurred years ago between the BDI infrastructure, dirty inode
tracking, dirty inode writeback and cgroups has left this code a
complex, fragile tangle of esoteric, untestable code. There are
enough subtle race conditions between a single BDI flusher thread,
writeback, mounts and the block device life cycle that everything is
likely to break if we try to add concurrency into this code.

So there's a big architectural question here: do we start again and
try to engineer something for XFS that does everything we need and
then push that towards being a generic solution (like we did with
iomap to replace bufferheads), or do we pull the loose string on the
existing code and try to add IO concurrency into that code without
making the mess worse?

What other options do we have? What other approaches to the problem
are there? Does this interact with SSD specific allocation policies
in some way? Is delayed allocation even relevant anymore with SSDs
that can do millions of IOPS?

Food for thought.

-- 
Dave Chinner
david@fromorbit.com