From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1756058AbbKCVAE (ORCPT ); Tue, 3 Nov 2015 16:00:04 -0500 Received: from ipmail07.adl2.internode.on.net ([150.101.137.131]:52495 "EHLO ipmail07.adl2.internode.on.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1755291AbbKCVAB (ORCPT ); Tue, 3 Nov 2015 16:00:01 -0500 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A2D+BgDnHzlW/+rW03ZegzuBQqpNAQEGiy6FJYQGhg0CAgEBAoFBTQEBAQEBAYELhDUBAQEDATocIwULCAMYCSUPBSUDIROIJgfCGQEBCAIBIBmFdYVFiTgFlkaNGo8xjRVjhBgqNIU0AQEB Date: Wed, 4 Nov 2015 07:59:33 +1100 From: Dave Chinner To: Ross Zwisler Cc: Dan Williams , Jens Axboe , Jan Kara , "linux-nvdimm@lists.01.org" , "linux-kernel@vger.kernel.org" , Jeff Moyer , Jan Kara , Christoph Hellwig Subject: Re: [PATCH v3 02/15] dax: increase granularity of dax_clear_blocks() operations Message-ID: <20151103205933.GI19199@dastard> References: <20151102042941.6610.27784.stgit@dwillia2-desk3.amr.corp.intel.com> <20151102042952.6610.7185.stgit@dwillia2-desk3.amr.corp.intel.com> <20151103005113.GN10656@dastard> <20151103044802.GP10656@dastard> <20151103175757.GA23366@linux.intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20151103175757.GA23366@linux.intel.com> User-Agent: Mutt/1.5.21 (2010-09-15) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Tue, Nov 03, 2015 at 10:57:57AM -0700, Ross Zwisler wrote: > On Mon, Nov 02, 2015 at 09:31:11PM -0800, Dan Williams wrote: > > On Mon, Nov 2, 2015 at 8:48 PM, Dave Chinner wrote: > > > On Mon, Nov 02, 2015 at 07:27:26PM -0800, Dan Williams wrote: > > >> On Mon, Nov 2, 2015 at 4:51 PM, Dave Chinner wrote: > > >> > On Sun, Nov 01, 2015 at 11:29:53PM -0500, Dan Williams wrote: > > >> > The zeroing (and the data, for that matter) doesn't need to be > > >> > committed to persistent store until the allocation is written and > > >> > committed to the journal - that will happen with a REQ_FLUSH|REQ_FUA > > >> > write, so it makes sense to deploy the big hammer and delay the > > >> > blocking CPU cache flushes until the last possible moment in cases > > >> > like this. > > >> > > >> In pmem terms that would be a non-temporal memset plus a delayed > > >> wmb_pmem at REQ_FLUSH time. Better to write around the cache than > > >> loop over the dirty-data issuing flushes after the fact. We'll bump > > >> the priority of the non-temporal memset implementation. > > > > > > Why is it better to do two synchronous physical writes to memory > > > within a couple of microseconds of CPU time rather than writing them > > > through the cache and, in most cases, only doing one physical write > > > to memory in a separate context that expects to wait for a flush > > > to complete? > > > > With a switch to non-temporal writes they wouldn't be synchronous, > > although it's doubtful that the subsequent writes after zeroing would > > also hit the store buffer. > > > > If we had a method to flush by physical-cache-way rather than a > > virtual address then it would indeed be better to save up for one > > final flush, but when we need to resort to looping through all the > > virtual addresses that might have touched it gets expensive. > > I agree with the idea that we should avoid the "big hammer" flushing in > response to REQ_FLUSH. Here are the steps that are needed to make sure that > something is durable on media with PMEM/DAX: > > 1) Write, either with non-temporal stores or with stores that use the > processor cache > > 2) If you wrote using the processor cache, flush or write back the processor > cache > > 3) wmb_pmem(), synchronizing all non-temporal writes and flushes durably to > media. Right, and when you look at buffered IO, we have: 1) write to page cache, mark page dirty 2) if you have dirty cached pages, flush dirty pages to device 3) REQ_FLUSH causes everything to be durable. > PMEM does all I/O using 1 and 3 with non-temporal stores, and mmaps that go to > userspace can used cached writes, so on fsync/msync we do a bunch of flushes > for step 2. In either case I think we should have the PMEM driver just do > step 3, the wmb_pmem(), in response to REQ_FLUSH. This allows the zeroing > code to just do non-temporal writes of zeros, the DAX fsync/msync code to just > do flushes (which is what my patch set already does), and just leave the > wmb_pmem() to the PMEM driver at REQ_FLUSH time. > > This just means that the layers above the PMEM code either need to use > non-temporal writes for their I/Os, or do flushing, which I don't think is too > onerous. Agreed - it fits neatly into the existing infrastructure and algorithms and there's no evidence to suggest that using the existing infrastructure is going to cause undue burden on PMEM based workloads. Hence I really think this is the right way to proceed... Cheers, Dave. -- Dave Chinner david@fromorbit.com