From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Date: Mon, 16 Nov 2015 13:01:02 -0700 From: Ross Zwisler Subject: Re: [PATCH v2 00/11] DAX fsynx/msync support Message-ID: <20151116200102.GA9737@linux.intel.com> References: <1447459610-14259-1-git-send-email-ross.zwisler@linux.intel.com> <20151116144130.GD3443@quack.suse.cz> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: Sender: owner-linux-mm@kvack.org To: Dan Williams Cc: Jan Kara , Ross Zwisler , "linux-kernel@vger.kernel.org" , "H. Peter Anvin" , "J. Bruce Fields" , Theodore Ts'o , Alexander Viro , Andreas Dilger , Dave Chinner , Ingo Molnar , Jan Kara , Jeff Layton , Matthew Wilcox , Thomas Gleixner , linux-ext4 , linux-fsdevel , Linux MM , "linux-nvdimm@lists.01.org" , X86 ML , XFS Developers , Andrew Morton , Matthew Wilcox , Dave Hansen List-ID: On Mon, Nov 16, 2015 at 08:58:11AM -0800, Dan Williams wrote: > On Mon, Nov 16, 2015 at 6:41 AM, Jan Kara wrote: > > On Fri 13-11-15 17:06:39, Ross Zwisler wrote: > >> This patch series adds support for fsync/msync to DAX. > >> > >> Patches 1 through 7 add various utilities that the DAX code will eventually > >> need, and the DAX code itself is added by patch 8. Patches 9-11 update the > >> three filesystems that currently support DAX, ext2, ext4 and XFS, to use > >> the new DAX fsync/msync code. > >> > >> These patches build on the recent DAX locking changes from Dave Chinner, > >> Jan Kara and myself. Dave's changes for XFS and my changes for ext2 have > >> been merged in the v4.4 window, but Jan's are still unmerged. You can grab > >> them here: > >> > >> http://www.spinics.net/lists/linux-ext4/msg49951.html > > > > I had a quick look and the patches look sane to me. I'll try to give them > > more detailed look later this week. When thinking about the general design > > I was wondering: When we have this infrastructure to track data potentially > > lingering in CPU caches, would not it be a performance win to use standard > > cached stores in dax_io() and mark corresponding pages as dirty in page > > cache the same way as this patch set does it for mmaped writes? I have no > > idea how costly are non-temporal stores compared to cached ones and how > > would this compare to the cost of dirty tracking so this may be just > > completely bogus... > > Keep in mind that this approach will flush every virtual address that > may be dirty. For example, if you touch 1byte in a 2MB page we'll end > up looping through the entire 2MB range. At some point the dirty size > becomes large enough that is cheaper to flush the entire cache, we > have not measured where that crossover point is. Yep, I expect there will be a crossover point where flushing the entire processor cache will be beneficial. I agree with Dan that we'll need to figure this out via measurement, and that we'd similarly need measurements to justify the decision to write dirty data at the DAX level without flushing and mark entries as dirty for fsync/msync to clean up later. It could turn out to be great, but we'll have to see. :) -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752600AbbKPUBM (ORCPT ); Mon, 16 Nov 2015 15:01:12 -0500 Received: from mga03.intel.com ([134.134.136.65]:55278 "EHLO mga03.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751935AbbKPUBH (ORCPT ); Mon, 16 Nov 2015 15:01:07 -0500 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.20,304,1444719600"; d="scan'208";a="839932371" Date: Mon, 16 Nov 2015 13:01:02 -0700 From: Ross Zwisler To: Dan Williams Cc: Jan Kara , Ross Zwisler , "linux-kernel@vger.kernel.org" , "H. Peter Anvin" , "J. Bruce Fields" , "Theodore Ts'o" , Alexander Viro , Andreas Dilger , Dave Chinner , Ingo Molnar , Jan Kara , Jeff Layton , Matthew Wilcox , Thomas Gleixner , linux-ext4 , linux-fsdevel , Linux MM , "linux-nvdimm@lists.01.org" , X86 ML , XFS Developers , Andrew Morton , Matthew Wilcox , Dave Hansen Subject: Re: [PATCH v2 00/11] DAX fsynx/msync support Message-ID: <20151116200102.GA9737@linux.intel.com> Mail-Followup-To: Ross Zwisler , Dan Williams , Jan Kara , "linux-kernel@vger.kernel.org" , "H. Peter Anvin" , "J. Bruce Fields" , Theodore Ts'o , Alexander Viro , Andreas Dilger , Dave Chinner , Ingo Molnar , Jan Kara , Jeff Layton , Matthew Wilcox , Thomas Gleixner , linux-ext4 , linux-fsdevel , Linux MM , "linux-nvdimm@lists.01.org" , X86 ML , XFS Developers , Andrew Morton , Matthew Wilcox , Dave Hansen References: <1447459610-14259-1-git-send-email-ross.zwisler@linux.intel.com> <20151116144130.GD3443@quack.suse.cz> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.23 (2014-03-12) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Mon, Nov 16, 2015 at 08:58:11AM -0800, Dan Williams wrote: > On Mon, Nov 16, 2015 at 6:41 AM, Jan Kara wrote: > > On Fri 13-11-15 17:06:39, Ross Zwisler wrote: > >> This patch series adds support for fsync/msync to DAX. > >> > >> Patches 1 through 7 add various utilities that the DAX code will eventually > >> need, and the DAX code itself is added by patch 8. Patches 9-11 update the > >> three filesystems that currently support DAX, ext2, ext4 and XFS, to use > >> the new DAX fsync/msync code. > >> > >> These patches build on the recent DAX locking changes from Dave Chinner, > >> Jan Kara and myself. Dave's changes for XFS and my changes for ext2 have > >> been merged in the v4.4 window, but Jan's are still unmerged. You can grab > >> them here: > >> > >> http://www.spinics.net/lists/linux-ext4/msg49951.html > > > > I had a quick look and the patches look sane to me. I'll try to give them > > more detailed look later this week. When thinking about the general design > > I was wondering: When we have this infrastructure to track data potentially > > lingering in CPU caches, would not it be a performance win to use standard > > cached stores in dax_io() and mark corresponding pages as dirty in page > > cache the same way as this patch set does it for mmaped writes? I have no > > idea how costly are non-temporal stores compared to cached ones and how > > would this compare to the cost of dirty tracking so this may be just > > completely bogus... > > Keep in mind that this approach will flush every virtual address that > may be dirty. For example, if you touch 1byte in a 2MB page we'll end > up looping through the entire 2MB range. At some point the dirty size > becomes large enough that is cheaper to flush the entire cache, we > have not measured where that crossover point is. Yep, I expect there will be a crossover point where flushing the entire processor cache will be beneficial. I agree with Dan that we'll need to figure this out via measurement, and that we'd similarly need measurements to justify the decision to write dirty data at the DAX level without flushing and mark entries as dirty for fsync/msync to clean up later. It could turn out to be great, but we'll have to see. :) From mboxrd@z Thu Jan 1 00:00:00 1970 From: Ross Zwisler Subject: Re: [PATCH v2 00/11] DAX fsynx/msync support Date: Mon, 16 Nov 2015 13:01:02 -0700 Message-ID: <20151116200102.GA9737@linux.intel.com> References: <1447459610-14259-1-git-send-email-ross.zwisler@linux.intel.com> <20151116144130.GD3443@quack.suse.cz> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Cc: Jan Kara , Ross Zwisler , "linux-kernel@vger.kernel.org" , "H. Peter Anvin" , "J. Bruce Fields" , Theodore Ts'o , Alexander Viro , Andreas Dilger , Dave Chinner , Ingo Molnar , Jan Kara , Jeff Layton , Matthew Wilcox , Thomas Gleixner , linux-ext4 , linux-fsdevel , Linux MM , "linux-nvdimm@lists.01.org" , X86 ML , XFS Developers , Andrew Morton , Matthew Wilcox , Dave Hansen Return-path: Content-Disposition: inline In-Reply-To: Sender: linux-kernel-owner@vger.kernel.org List-Id: linux-ext4.vger.kernel.org On Mon, Nov 16, 2015 at 08:58:11AM -0800, Dan Williams wrote: > On Mon, Nov 16, 2015 at 6:41 AM, Jan Kara wrote: > > On Fri 13-11-15 17:06:39, Ross Zwisler wrote: > >> This patch series adds support for fsync/msync to DAX. > >> > >> Patches 1 through 7 add various utilities that the DAX code will eventually > >> need, and the DAX code itself is added by patch 8. Patches 9-11 update the > >> three filesystems that currently support DAX, ext2, ext4 and XFS, to use > >> the new DAX fsync/msync code. > >> > >> These patches build on the recent DAX locking changes from Dave Chinner, > >> Jan Kara and myself. Dave's changes for XFS and my changes for ext2 have > >> been merged in the v4.4 window, but Jan's are still unmerged. You can grab > >> them here: > >> > >> http://www.spinics.net/lists/linux-ext4/msg49951.html > > > > I had a quick look and the patches look sane to me. I'll try to give them > > more detailed look later this week. When thinking about the general design > > I was wondering: When we have this infrastructure to track data potentially > > lingering in CPU caches, would not it be a performance win to use standard > > cached stores in dax_io() and mark corresponding pages as dirty in page > > cache the same way as this patch set does it for mmaped writes? I have no > > idea how costly are non-temporal stores compared to cached ones and how > > would this compare to the cost of dirty tracking so this may be just > > completely bogus... > > Keep in mind that this approach will flush every virtual address that > may be dirty. For example, if you touch 1byte in a 2MB page we'll end > up looping through the entire 2MB range. At some point the dirty size > becomes large enough that is cheaper to flush the entire cache, we > have not measured where that crossover point is. Yep, I expect there will be a crossover point where flushing the entire processor cache will be beneficial. I agree with Dan that we'll need to figure this out via measurement, and that we'd similarly need measurements to justify the decision to write dirty data at the DAX level without flushing and mark entries as dirty for fsync/msync to clean up later. It could turn out to be great, but we'll have to see. :) From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 40CEC7CBF for ; Mon, 16 Nov 2015 14:01:18 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay1.corp.sgi.com (Postfix) with ESMTP id 169178F8059 for ; Mon, 16 Nov 2015 12:01:17 -0800 (PST) Received: from mga14.intel.com ([192.55.52.115]) by cuda.sgi.com with ESMTP id cK3cA1W0jo48OBsJ for ; Mon, 16 Nov 2015 12:01:10 -0800 (PST) Date: Mon, 16 Nov 2015 13:01:02 -0700 From: Ross Zwisler Subject: Re: [PATCH v2 00/11] DAX fsynx/msync support Message-ID: <20151116200102.GA9737@linux.intel.com> References: <1447459610-14259-1-git-send-email-ross.zwisler@linux.intel.com> <20151116144130.GD3443@quack.suse.cz> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: List-Id: XFS Filesystem from SGI List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: xfs-bounces@oss.sgi.com Sender: xfs-bounces@oss.sgi.com To: Dan Williams Cc: Jan Kara , Dave Hansen , "J. Bruce Fields" , Linux MM , Andreas Dilger , "H. Peter Anvin" , Jeff Layton , "linux-nvdimm@lists.01.org" , X86 ML , Ingo Molnar , Matthew Wilcox , Ross Zwisler , linux-ext4 , XFS Developers , Alexander Viro , Thomas Gleixner , Theodore Ts'o , "linux-kernel@vger.kernel.org" , Jan Kara , linux-fsdevel , Andrew Morton , Matthew Wilcox On Mon, Nov 16, 2015 at 08:58:11AM -0800, Dan Williams wrote: > On Mon, Nov 16, 2015 at 6:41 AM, Jan Kara wrote: > > On Fri 13-11-15 17:06:39, Ross Zwisler wrote: > >> This patch series adds support for fsync/msync to DAX. > >> > >> Patches 1 through 7 add various utilities that the DAX code will eventually > >> need, and the DAX code itself is added by patch 8. Patches 9-11 update the > >> three filesystems that currently support DAX, ext2, ext4 and XFS, to use > >> the new DAX fsync/msync code. > >> > >> These patches build on the recent DAX locking changes from Dave Chinner, > >> Jan Kara and myself. Dave's changes for XFS and my changes for ext2 have > >> been merged in the v4.4 window, but Jan's are still unmerged. You can grab > >> them here: > >> > >> http://www.spinics.net/lists/linux-ext4/msg49951.html > > > > I had a quick look and the patches look sane to me. I'll try to give them > > more detailed look later this week. When thinking about the general design > > I was wondering: When we have this infrastructure to track data potentially > > lingering in CPU caches, would not it be a performance win to use standard > > cached stores in dax_io() and mark corresponding pages as dirty in page > > cache the same way as this patch set does it for mmaped writes? I have no > > idea how costly are non-temporal stores compared to cached ones and how > > would this compare to the cost of dirty tracking so this may be just > > completely bogus... > > Keep in mind that this approach will flush every virtual address that > may be dirty. For example, if you touch 1byte in a 2MB page we'll end > up looping through the entire 2MB range. At some point the dirty size > becomes large enough that is cheaper to flush the entire cache, we > have not measured where that crossover point is. Yep, I expect there will be a crossover point where flushing the entire processor cache will be beneficial. I agree with Dan that we'll need to figure this out via measurement, and that we'd similarly need measurements to justify the decision to write dirty data at the DAX level without flushing and mark entries as dirty for fsync/msync to clean up later. It could turn out to be great, but we'll have to see. :) _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs