Re: dm stripe: add DAX support

From: Mike Snitzer <snitzer-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
To: "Kani, Toshimitsu" <toshi.kani-ZPxbGqLxI0U@public.gmane.org>,
	axboe-b10kYP2dOMg@public.gmane.org
Cc: "linux-nvdimm-y27Ovi1pjclAfugRpC6u6w@public.gmane.org"
	<linux-nvdimm-y27Ovi1pjclAfugRpC6u6w@public.gmane.org>,
	"linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org"
	<linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org>,
	"linux-raid-u79uwXL29TY76Z2rM5mHXA@public.gmane.org"
	<linux-raid-u79uwXL29TY76Z2rM5mHXA@public.gmane.org>,
	"dm-devel-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org"
	<dm-devel-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>,
	"agk-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org"
	<agk-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
Subject: Re: dm stripe: add DAX support
Date: Tue, 12 Jul 2016 22:01:01 -0400	[thread overview]
Message-ID: <20160713020100.GA5872@redhat.com> (raw)
In-Reply-To: <1468362104.8908.43.camel-ZPxbGqLxI0U@public.gmane.org>

On Tue, Jul 12 2016 at  6:22pm -0400,
Kani, Toshimitsu <toshi.kani-ZPxbGqLxI0U@public.gmane.org> wrote:

> On Fri, 2016-06-24 at 14:29 -0400, Mike Snitzer wrote:
> > 
> > BTW, if in your testing you could evaluate/quantify any extra overhead
> > from DM that'd be useful to share.  It could be there are bottlenecks
> > that need to be fixed, etc.
> 
> Here are some results from fio benchmark.  The test is single-threaded and is
> bound to one CPU.
> 
>  DAX  LVM   IOPS   NOTE
>  ---------------------------------------
>   Y    N    790K
>   Y    Y    754K   5% overhead with LVM
>   N    N    567K
>   N    Y    457K   20% overhead with LVM
> 
>  DAX: Y: mount -o dax,noatime, N: mount -o noatime
>  LVM: Y: dm-linear on pmem0 device, N: pmem0 device
>  fio: bs=4k, size=2G, direct=1, rw=randread, numjobs=1
> 
> Among the 5% overhead with DAX/LVM, the new DM direct_access interfaces
> account for less than 0.5%.
> 
>  dm_blk_direct_access 0.28%
>  linear_direct_access 0.17%
> 
> The average latency increases slightly from 0.93us to 0.95us.  I think most of
> the overhead comes from the submit_bio() path, which is used only for
> accessing metadata with DAX.  I believe this is due to cloning bio for each
> request in DM.  There is 12% more L2 miss in total.
> 
> Without DAX, 20% overhead is observed with LVM.  Average latency increases
> from 1.39us to 1.82us.  Without DAX, bio is cloned for both data and metadata.

Thanks for putting this summary together.  Unfortunately none of the DM
changes can be queued for 4.8 until Jens takes the 2 block core patches:
https://patchwork.kernel.org/patch/9196021/
https://patchwork.kernel.org/patch/9196019/

Not sure what the hold up and/or issue is with them.  But I've asked
twice (and implicilty a 3rd time here).  Hopefully they land in time for
4.8.

Mike