From mboxrd@z Thu Jan 1 00:00:00 1970 From: Benjamin Herrenschmidt Subject: Re: [RFC 0/8] Copy Offload with Peer-to-Peer PCI Memory Date: Fri, 14 Apr 2017 08:37:29 +1000 Message-ID: <1492123049.7236.123.camel@kernel.crashing.org> References: <1490911959-5146-1-git-send-email-logang@deltatee.com> <1491974532.7236.43.camel@kernel.crashing.org> <5ac22496-56ec-025d-f153-140001d2a7f9@deltatee.com> <1492034124.7236.77.camel@kernel.crashing.org> <81888a1e-eb0d-cbbc-dc66-0a09c32e4ea2@deltatee.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <81888a1e-eb0d-cbbc-dc66-0a09c32e4ea2-OTvnGxWRz7hWk0Htik3J/w@public.gmane.org> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: linux-nvdimm-bounces-hn68Rpc1hR1g9hUCZPvPmw@public.gmane.org Sender: "Linux-nvdimm" To: Logan Gunthorpe , Christoph Hellwig , Sagi Grimberg , "James E.J. Bottomley" , "Martin K. Petersen" , Jens Axboe , Steve Wise , Stephen Bates , Max Gurtovoy , Dan Williams , Keith Busch , Jason Gunthorpe Cc: linux-scsi-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, linux-pci-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, linux-nvme-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r@public.gmane.org, Jerome Glisse , linux-nvdimm-y27Ovi1pjclAfugRpC6u6w@public.gmane.org List-Id: linux-nvdimm@lists.01.org On Thu, 2017-04-13 at 15:22 -0600, Logan Gunthorpe wrote: > > On 12/04/17 03:55 PM, Benjamin Herrenschmidt wrote: > > Look at pcibios_resource_to_bus() and pcibios_bus_to_resource(). They > > will perform the conversion between the struct resource content (CPU > > physical address) and the actual PCI bus side address. > > Ah, thanks for the tip! On my system, this translation returns the same > address so it was not necessary. And, yes, that means this would have to > find its way into the dma mapping routine somehow. This means we'll > eventually need a way to look-up the p2pmem device from the struct page. > Which means we will likely need a new flag bit in the struct page or > something. The big difficulty I see is testing. Do you know what > architectures or in what circumstances are these translations used? I think a bunch of non-x86 architectures but I don't know which ones outside of powerpc. > > When behind the same switch you need to use PCI addresses. If one tries > > later to do P2P between host bridges (via the CPU fabric) things get > > more complex and one will have to use either CPU addresses or something > > else alltogether (probably would have to teach the arch DMA mapping > > routines to work with those struct pages you create and return the > > right thing). > > Probably for starters we'd want to explicitly deny cases between host > bridges and add that later if someone wants to do the testing. Cheers, Ben. > Thanks, > > Logan From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752368AbdDMWjH (ORCPT ); Thu, 13 Apr 2017 18:39:07 -0400 Received: from gate.crashing.org ([63.228.1.57]:33951 "EHLO gate.crashing.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751808AbdDMWjE (ORCPT ); Thu, 13 Apr 2017 18:39:04 -0400 Message-ID: <1492123049.7236.123.camel@kernel.crashing.org> Subject: Re: [RFC 0/8] Copy Offload with Peer-to-Peer PCI Memory From: Benjamin Herrenschmidt To: Logan Gunthorpe , Christoph Hellwig , Sagi Grimberg , "James E.J. Bottomley" , "Martin K. Petersen" , Jens Axboe , Steve Wise , Stephen Bates , Max Gurtovoy , Dan Williams , Keith Busch , Jason Gunthorpe Cc: linux-pci@vger.kernel.org, linux-scsi@vger.kernel.org, linux-nvme@lists.infradead.org, linux-rdma@vger.kernel.org, linux-nvdimm@ml01.01.org, linux-kernel@vger.kernel.org, Jerome Glisse Date: Fri, 14 Apr 2017 08:37:29 +1000 In-Reply-To: <81888a1e-eb0d-cbbc-dc66-0a09c32e4ea2@deltatee.com> References: <1490911959-5146-1-git-send-email-logang@deltatee.com> <1491974532.7236.43.camel@kernel.crashing.org> <5ac22496-56ec-025d-f153-140001d2a7f9@deltatee.com> <1492034124.7236.77.camel@kernel.crashing.org> <81888a1e-eb0d-cbbc-dc66-0a09c32e4ea2@deltatee.com> Content-Type: text/plain; charset="UTF-8" X-Mailer: Evolution 3.22.6 (3.22.6-1.fc25) Mime-Version: 1.0 Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Thu, 2017-04-13 at 15:22 -0600, Logan Gunthorpe wrote: > > On 12/04/17 03:55 PM, Benjamin Herrenschmidt wrote: > > Look at pcibios_resource_to_bus() and pcibios_bus_to_resource(). They > > will perform the conversion between the struct resource content (CPU > > physical address) and the actual PCI bus side address. > > Ah, thanks for the tip! On my system, this translation returns the same > address so it was not necessary. And, yes, that means this would have to > find its way into the dma mapping routine somehow. This means we'll > eventually need a way to look-up the p2pmem device from the struct page. > Which means we will likely need a new flag bit in the struct page or > something. The big difficulty I see is testing. Do you know what > architectures or in what circumstances are these translations used? I think a bunch of non-x86 architectures but I don't know which ones outside of powerpc. > > When behind the same switch you need to use PCI addresses. If one tries > > later to do P2P between host bridges (via the CPU fabric) things get > > more complex and one will have to use either CPU addresses or something > > else alltogether (probably would have to teach the arch DMA mapping > > routines to work with those struct pages you create and return the > > right thing). > > Probably for starters we'd want to explicitly deny cases between host > bridges and add that later if someone wants to do the testing. Cheers, Ben. > Thanks, > > Logan From mboxrd@z Thu Jan 1 00:00:00 1970 From: benh@kernel.crashing.org (Benjamin Herrenschmidt) Date: Fri, 14 Apr 2017 08:37:29 +1000 Subject: [RFC 0/8] Copy Offload with Peer-to-Peer PCI Memory In-Reply-To: <81888a1e-eb0d-cbbc-dc66-0a09c32e4ea2@deltatee.com> References: <1490911959-5146-1-git-send-email-logang@deltatee.com> <1491974532.7236.43.camel@kernel.crashing.org> <5ac22496-56ec-025d-f153-140001d2a7f9@deltatee.com> <1492034124.7236.77.camel@kernel.crashing.org> <81888a1e-eb0d-cbbc-dc66-0a09c32e4ea2@deltatee.com> Message-ID: <1492123049.7236.123.camel@kernel.crashing.org> On Thu, 2017-04-13@15:22 -0600, Logan Gunthorpe wrote: > > On 12/04/17 03:55 PM, Benjamin Herrenschmidt wrote: > > Look at pcibios_resource_to_bus() and pcibios_bus_to_resource(). They > > will perform the conversion between the struct resource content (CPU > > physical address) and the actual PCI bus side address. > > Ah, thanks for the tip! On my system, this translation returns the same > address so it was not necessary. And, yes, that means this would have to > find its way into the dma mapping routine somehow. This means we'll > eventually need a way to look-up the p2pmem device from the struct page. > Which means we will likely need a new flag bit in the struct page or > something. The big difficulty I see is testing. Do you know what > architectures or in what circumstances are these translations used? I think a bunch of non-x86 architectures but I don't know which ones outside of powerpc. > > When behind the same switch you need to use PCI addresses. If one tries > > later to do P2P between host bridges (via the CPU fabric) things get > > more complex and one will have to use either CPU addresses or something > > else alltogether (probably would have to teach the arch DMA mapping > > routines to work with those struct pages you create and return the > > right thing). > > Probably for starters we'd want to explicitly deny cases between host > bridges and add that later if someone wants to do the testing. Cheers, Ben. > Thanks, > > Logan