From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Date: Mon, 16 Oct 2017 09:26:44 +0200 From: Christoph Hellwig Subject: Re: [PATCH v9 0/6] MAP_DIRECT for DAX userspace flush Message-ID: <20171016072644.GB28270@lst.de> References: <150776922692.9144.16963640112710410217.stgit@dwillia2-desk3.amr.corp.intel.com> <20171012142319.GA11254@lst.de> <20171013065716.GB26461@lst.de> <20171013163822.GA17411@obsidianresearch.com> <20171013173145.GA18702@obsidianresearch.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20171013173145.GA18702@obsidianresearch.com> Sender: owner-linux-mm@kvack.org To: Jason Gunthorpe Cc: Dan Williams , Christoph Hellwig , "linux-nvdimm@lists.01.org" , linux-xfs@vger.kernel.org, Jan Kara , Arnd Bergmann , "Darrick J. Wong" , Linux API , Dave Chinner , "J. Bruce Fields" , Linux MM , Jeff Moyer , Al Viro , Andy Lutomirski , Ross Zwisler , linux-fsdevel , Jeff Layton , Linus Torvalds , Andrew Morton List-ID: On Fri, Oct 13, 2017 at 11:31:45AM -0600, Jason Gunthorpe wrote: > I don't think that really represents how lots of apps actually use > RDMA. > > RDMA is often buried down in the software stack (eg in a MPI), and by > the time a mapping gets used for RDMA transfer the link between the > FD, mmap and the MR is totally opaque. > > Having a MR specific notification means the low level RDMA libraries > have a chance to deal with everything for the app. > > Eg consider a HPC app using MPI that uses some DAX aware library to > get DAX backed mmap's. It then passes memory in those mmaps to the > MPI library to do transfers. The MPI creates the MR on demand. > I suspect one of the more interesting use cases might be a file server, for which that's not the case. But otherwise I agree with the above, and also thing that notifying the MR handle is the only way to go for another very important reason: fencing. What if the application/library does not react on the notification? With a per-MR notification we can unregister the MR in kernel space and have a rock solid fencing mechanism. And that is the most important bit here. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from verein.lst.de ([213.95.11.211]:53468 "EHLO newverein.lst.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750880AbdJPH0q (ORCPT ); Mon, 16 Oct 2017 03:26:46 -0400 Date: Mon, 16 Oct 2017 09:26:44 +0200 From: Christoph Hellwig Subject: Re: [PATCH v9 0/6] MAP_DIRECT for DAX userspace flush Message-ID: <20171016072644.GB28270@lst.de> References: <150776922692.9144.16963640112710410217.stgit@dwillia2-desk3.amr.corp.intel.com> <20171012142319.GA11254@lst.de> <20171013065716.GB26461@lst.de> <20171013163822.GA17411@obsidianresearch.com> <20171013173145.GA18702@obsidianresearch.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20171013173145.GA18702@obsidianresearch.com> Sender: linux-xfs-owner@vger.kernel.org List-ID: List-Id: xfs To: Jason Gunthorpe Cc: Dan Williams , Christoph Hellwig , "linux-nvdimm@lists.01.org" , linux-xfs@vger.kernel.org, Jan Kara , Arnd Bergmann , "Darrick J. Wong" , Linux API , Dave Chinner , "J. Bruce Fields" , Linux MM , Jeff Moyer , Al Viro , Andy Lutomirski , Ross Zwisler , linux-fsdevel , Jeff Layton , Linus Torvalds , Andrew Morton On Fri, Oct 13, 2017 at 11:31:45AM -0600, Jason Gunthorpe wrote: > I don't think that really represents how lots of apps actually use > RDMA. > > RDMA is often buried down in the software stack (eg in a MPI), and by > the time a mapping gets used for RDMA transfer the link between the > FD, mmap and the MR is totally opaque. > > Having a MR specific notification means the low level RDMA libraries > have a chance to deal with everything for the app. > > Eg consider a HPC app using MPI that uses some DAX aware library to > get DAX backed mmap's. It then passes memory in those mmaps to the > MPI library to do transfers. The MPI creates the MR on demand. > I suspect one of the more interesting use cases might be a file server, for which that's not the case. But otherwise I agree with the above, and also thing that notifying the MR handle is the only way to go for another very important reason: fencing. What if the application/library does not react on the notification? With a per-MR notification we can unregister the MR in kernel space and have a rock solid fencing mechanism. And that is the most important bit here. From mboxrd@z Thu Jan 1 00:00:00 1970 From: Christoph Hellwig Subject: Re: [PATCH v9 0/6] MAP_DIRECT for DAX userspace flush Date: Mon, 16 Oct 2017 09:26:44 +0200 Message-ID: <20171016072644.GB28270@lst.de> References: <150776922692.9144.16963640112710410217.stgit@dwillia2-desk3.amr.corp.intel.com> <20171012142319.GA11254@lst.de> <20171013065716.GB26461@lst.de> <20171013163822.GA17411@obsidianresearch.com> <20171013173145.GA18702@obsidianresearch.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Return-path: Content-Disposition: inline In-Reply-To: <20171013173145.GA18702-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> Sender: linux-api-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org To: Jason Gunthorpe Cc: Dan Williams , Christoph Hellwig , "linux-nvdimm-hn68Rpc1hR1g9hUCZPvPmw@public.gmane.org" , linux-xfs-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, Jan Kara , Arnd Bergmann , "Darrick J. Wong" , Linux API , Dave Chinner , "J. Bruce Fields" , Linux MM , Jeff Moyer , Al Viro , Andy Lutomirski , Ross Zwisler , linux-fsdevel , Jeff Layton , Linus Torvalds , Andrew Morton List-Id: linux-api@vger.kernel.org On Fri, Oct 13, 2017 at 11:31:45AM -0600, Jason Gunthorpe wrote: > I don't think that really represents how lots of apps actually use > RDMA. > > RDMA is often buried down in the software stack (eg in a MPI), and by > the time a mapping gets used for RDMA transfer the link between the > FD, mmap and the MR is totally opaque. > > Having a MR specific notification means the low level RDMA libraries > have a chance to deal with everything for the app. > > Eg consider a HPC app using MPI that uses some DAX aware library to > get DAX backed mmap's. It then passes memory in those mmaps to the > MPI library to do transfers. The MPI creates the MR on demand. > I suspect one of the more interesting use cases might be a file server, for which that's not the case. But otherwise I agree with the above, and also thing that notifying the MR handle is the only way to go for another very important reason: fencing. What if the application/library does not react on the notification? With a per-MR notification we can unregister the MR in kernel space and have a rock solid fencing mechanism. And that is the most important bit here.