Re: file streams allocator behavior

[Dave Chinner <david@fromorbit.com>]

On Sat, Oct 25, 2014 at 01:12:48PM -0500, Stan Hoeppner wrote:
> I recall reading a while back something about disabling the filestreams
> allocator, or at least changing its behavior, but I'm unable to find that.
> 
> What I'm trying to do is use parallel dd w/O_DIRECT to write 44 files in
> parallel to 44 directories, thus all 44 AGs, in one test, then write 44
> files to one dir, one AG, in another test.  The purpose of this
> quick/dirty exercise is to demonstrate throughput differences due to
> full platter seeking in the former case and localized seeking in the
> latter case.
> 
> But of course the problem I'm running into in the single directory case
> is that the filestreams allocator starts writing all of the 44 files
> into the appropriate AG, but then begins allocating extents for each
> file in other AGs.  This is of course defeating the purpose of the tests.

That's caused by allocator contention. When you try to write 44
files to the same dir in parallel, they'll all start with the same
target AG, but then when one thread is allocating into AG 43 and has
the AG locked, a second attempt to allocate to than AG will see the
AG locked and so it will move to find the next AG that is not
locked.

Remember, AGs were not originally designed for confining physical
locality - they are designed to allow allocator parallelism. Hence
once the file has jumped to a new AG it will try to allocate
sequentially from that point onwards in that same AG, until either
ENOSPC or further contention.

Hence with a workload like this, if the writes continue for long
enough each file will end up finding it's own uncontended AG and
hence mostly end up contiguous on disk and not getting blocked
waiting for allocation on other files. When you have as many writers
as there are AGs, however, such a steady state is generally not
possible as there will always be files trying to write into the same
AG.

As it is, filestreams is not designed for this sort of parallel
workload. filestreams is designed to separate single threaded
streams of IO into different locations, not handle concurrent writes
into multiple files in the same directory.

As it is, the inode64 will probably demonstrate exactly the same
behaviour because it will start by trying to write all the files to
the same AG and hence hit allocator contention, too.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs