newstore performance update

[Mark Nelson <mnelson@redhat.com>]

Hi Guys,

Sage has been furiously working away at fixing bugs in newstore and 
improving performance.  Specifically we've been focused on write 
performance as newstore was lagging filestore but quite a bit 
previously.  A lot of work has gone into implementing libaio behind the 
scenes and as a result performance on spinning disks with SSD WAL (and 
SSD backed rocksdb) has improved pretty dramatically. It's now often 
beating filestore:

http://nhm.ceph.com/newstore/newstore-5d96fe6-no_overlay.pdf

On the other hand, sequential writes are slower than random writes when 
the OSD, DB, and WAL are all on the same device be it a spinning disk or 
SSD.  In this situation newstore does better with random writes and 
sometimes beats filestore (such as in the everything-on-spinning disk 
tests, and when IO sizes are small in the everything-on-ssd tests).

Newstore is changing daily so keep in mind that these results are almost 
assuredly going to change.  An interesting area of investigation will be 
why sequential writes are slower than random writes, and whether or not 
we are being limited by rocksdb ingest speed and how.

I've also uploaded a quick perf call-graph I grabbed during the 
"all-SSD" 32KB sequential write test to see if rocksdb was starving one 
of the cores, but found something that looks quite a bit different:

http://nhm.ceph.com/newstore/newstore-5d96fe6-no_overlay.pdf

Mark