From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Date: Fri, 25 Nov 2016 12:41:33 -0700 From: Jason Gunthorpe Subject: Re: Enabling peer to peer device transactions for PCIe devices Message-ID: <20161125194133.GF16504@obsidianresearch.com> References: <7bc38037-b6ab-943f-59db-6280e16901ab@amd.com> <20161123193228.GC12146@obsidianresearch.com> <20161123203332.GA15062@obsidianresearch.com> <20161123215510.GA16311@obsidianresearch.com> <91d28749-bc64-622f-56a1-26c00e6b462a@deltatee.com> <20161124164249.GD20818@obsidianresearch.com> <9cc22068-ede8-c1bc-5d8b-cf6224a7ce05@deltatee.com> <20161125075817.GA18428@infradead.org> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20161125075817.GA18428@infradead.org> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: linux-nvdimm-bounces@lists.01.org Sender: "Linux-nvdimm" To: Christoph Hellwig Cc: Haggai Eran , "Bridgman, John , linux-rdma@vger.kernel.org" , "linux-nvdimm@lists.01.org" , "Kuehling, Felix , Serguei Sagalovitch" , "Blinzer, Paul , linux-kernel@vger.kernel.org" , "dri-devel@lists.freedesktop.org" , Sander,, "Suravee , linux-pci@vger.kernel.org" , Deucher,, "Christian , Linux-media@vger.kernel.org" List-ID: On Thu, Nov 24, 2016 at 11:58:17PM -0800, Christoph Hellwig wrote: > On Thu, Nov 24, 2016 at 11:11:34AM -0700, Logan Gunthorpe wrote: > > * Regular DAX in the FS doesn't work at this time because the FS can > > move the file you think your transfer to out from under you. Though I > > understand there's been some work with XFS to solve that issue. > > The file system will never move anything under locked down pages, > locking down pages is used exactly to protect against that. .. And ODP style mmu notifiers work correctly as well, I'd assume. So this should work with ZONE_DEVICE, if it doesn't it is a filesystem bug? > really want a notification to the consumer if the file systems wants > to remove the mapping. We have implemented that using FL_LAYOUTS locks > for NFSD, but only XFS supports it so far. Without that a long term > locked down region of memory (e.g. a kernel MR) would prevent various > file operations that would simply hang. So you imagine a signal back to user space asking user space to drop any RDMA MRS so the FS can relocate things? Do we need that, or should we encourage people to use either short lived MRs or ODP MRs when working with scenarios that need FS relocation? Jason _______________________________________________ Linux-nvdimm mailing list Linux-nvdimm@lists.01.org https://lists.01.org/mailman/listinfo/linux-nvdimm From mboxrd@z Thu Jan 1 00:00:00 1970 From: Jason Gunthorpe Subject: Re: Enabling peer to peer device transactions for PCIe devices Date: Fri, 25 Nov 2016 12:41:33 -0700 Message-ID: <20161125194133.GF16504@obsidianresearch.com> References: <7bc38037-b6ab-943f-59db-6280e16901ab@amd.com> <20161123193228.GC12146@obsidianresearch.com> <20161123203332.GA15062@obsidianresearch.com> <20161123215510.GA16311@obsidianresearch.com> <91d28749-bc64-622f-56a1-26c00e6b462a@deltatee.com> <20161124164249.GD20818@obsidianresearch.com> <9cc22068-ede8-c1bc-5d8b-cf6224a7ce05@deltatee.com> <20161125075817.GA18428@infradead.org> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Return-path: Content-Disposition: inline In-Reply-To: <20161125075817.GA18428-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: linux-nvdimm-bounces-hn68Rpc1hR1g9hUCZPvPmw@public.gmane.org Sender: "Linux-nvdimm" To: Christoph Hellwig Cc: Haggai Eran , "Bridgman, John" , "linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org" , "linux-nvdimm-hn68Rpc1hR1g9hUCZPvPmw@public.gmane.org" , "Kuehling, Felix" , Serguei Sagalovitch , "Blinzer, Paul" , "linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org" , "dri-devel-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW@public.gmane.org" , "Sander, Ben" , "Suthikulpanit, Suravee" , "linux-pci-u79uwXL29TY76Z2rM5mHXA@public.gmane.org" , "Deucher, Alexander" , "Koenig, Christian" , "Linux-media-u79uwXL29TY76Z2rM5mHXA@public.gmane.org" List-Id: linux-rdma@vger.kernel.org On Thu, Nov 24, 2016 at 11:58:17PM -0800, Christoph Hellwig wrote: > On Thu, Nov 24, 2016 at 11:11:34AM -0700, Logan Gunthorpe wrote: > > * Regular DAX in the FS doesn't work at this time because the FS can > > move the file you think your transfer to out from under you. Though I > > understand there's been some work with XFS to solve that issue. > > The file system will never move anything under locked down pages, > locking down pages is used exactly to protect against that. .. And ODP style mmu notifiers work correctly as well, I'd assume. So this should work with ZONE_DEVICE, if it doesn't it is a filesystem bug? > really want a notification to the consumer if the file systems wants > to remove the mapping. We have implemented that using FL_LAYOUTS locks > for NFSD, but only XFS supports it so far. Without that a long term > locked down region of memory (e.g. a kernel MR) would prevent various > file operations that would simply hang. So you imagine a signal back to user space asking user space to drop any RDMA MRS so the FS can relocate things? Do we need that, or should we encourage people to use either short lived MRs or ODP MRs when working with scenarios that need FS relocation? Jason From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S932855AbcKYTlz (ORCPT ); Fri, 25 Nov 2016 14:41:55 -0500 Received: from quartz.orcorp.ca ([184.70.90.242]:38826 "EHLO quartz.orcorp.ca" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S932342AbcKYTlr (ORCPT ); Fri, 25 Nov 2016 14:41:47 -0500 Date: Fri, 25 Nov 2016 12:41:33 -0700 From: Jason Gunthorpe To: Christoph Hellwig Cc: Logan Gunthorpe , Serguei Sagalovitch , Dan Williams , "Deucher, Alexander" , "linux-nvdimm@lists.01.org" , "linux-rdma@vger.kernel.org" , "linux-pci@vger.kernel.org" , "Kuehling, Felix" , "Bridgman, John" , "linux-kernel@vger.kernel.org" , "dri-devel@lists.freedesktop.org" , "Koenig, Christian" , "Sander, Ben" , "Suthikulpanit, Suravee" , "Blinzer, Paul" , "Linux-media@vger.kernel.org" , Haggai Eran Subject: Re: Enabling peer to peer device transactions for PCIe devices Message-ID: <20161125194133.GF16504@obsidianresearch.com> References: <7bc38037-b6ab-943f-59db-6280e16901ab@amd.com> <20161123193228.GC12146@obsidianresearch.com> <20161123203332.GA15062@obsidianresearch.com> <20161123215510.GA16311@obsidianresearch.com> <91d28749-bc64-622f-56a1-26c00e6b462a@deltatee.com> <20161124164249.GD20818@obsidianresearch.com> <9cc22068-ede8-c1bc-5d8b-cf6224a7ce05@deltatee.com> <20161125075817.GA18428@infradead.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20161125075817.GA18428@infradead.org> User-Agent: Mutt/1.5.23 (2014-03-12) X-Broken-Reverse-DNS: no host name found for IP address 10.0.0.151 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Thu, Nov 24, 2016 at 11:58:17PM -0800, Christoph Hellwig wrote: > On Thu, Nov 24, 2016 at 11:11:34AM -0700, Logan Gunthorpe wrote: > > * Regular DAX in the FS doesn't work at this time because the FS can > > move the file you think your transfer to out from under you. Though I > > understand there's been some work with XFS to solve that issue. > > The file system will never move anything under locked down pages, > locking down pages is used exactly to protect against that. .. And ODP style mmu notifiers work correctly as well, I'd assume. So this should work with ZONE_DEVICE, if it doesn't it is a filesystem bug? > really want a notification to the consumer if the file systems wants > to remove the mapping. We have implemented that using FL_LAYOUTS locks > for NFSD, but only XFS supports it so far. Without that a long term > locked down region of memory (e.g. a kernel MR) would prevent various > file operations that would simply hang. So you imagine a signal back to user space asking user space to drop any RDMA MRS so the FS can relocate things? Do we need that, or should we encourage people to use either short lived MRs or ODP MRs when working with scenarios that need FS relocation? Jason