Re: getdents - ext4 vs btrfs performance

[Chris Mason <chris.mason@oracle.com>]

On Wed, Mar 14, 2012 at 08:50:02AM -0400, Ted Ts'o wrote:
> On Wed, Mar 14, 2012 at 09:12:02AM +0100, Lukas Czerner wrote:
> > I kind of like the idea about having the separate btree with inode
> > numbers for the directory reading, just because it does not affect
> > allocation policy nor the write performance which is a good thing. Also
> > it has been done before in btrfs and it works very well for them. The
> > only downside (aside from format change) is the extra step when removing
> > the directory entry, but the positives outperform the negatives.
> 
> Well, there will be extra (journaled!) block reads and writes involved
> when adding or removing directory entries.  So the performance on
> workloads that do a large number of directory adds and removed will
> have to be impacted.  By how much is not obvious, but it will be
> something which is measurable.

Most of the difference in btrfs rm times comes from our double indexing.
We store the directory name 3 times, once for each dir index and once
in the inode backref.  But the inode backref doesn't generate seeks the
same way the extra directory index does, so it is probably only 10% of
the cost.

We also delete file data crcs during rm, so if you're benchmarking
things to see how much it will hurt, benchmark btrfs on zero length
files.

In terms of other costs, most users of readdir will eventually wander
off and stat or open the files in the dir.  With your current indexes, the stat
reads the same directory blocks that readdir used.

With the double index, the stat reads the name index instead of the
readdir index.  For the common case of normal directories, today's ext4
will have the directory blocks in cache when stat is called.  Btrfs has
to do extra reads to pull in the name index.

Josef worked around this by prefilling a dentry during readdir with
details on where to find the inode.

Either way, I'd definitely recommend sitting down with the xfs directory
indexes and look at how they manage locality without the double
indexing.  It's faster than btrfs and without the penalty.

It's also worth mentioning that on an SSD, there is no benefit at all to
the extra index.  So when the industry finally floods the markets with
hybrid PCM shingled drives, this might all be moot.

-chris