From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-oi0-x244.google.com (mail-oi0-x244.google.com [IPv6:2607:f8b0:4003:c06::244]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ml01.01.org (Postfix) with ESMTPS id 2A63F224C0F4A for ; Thu, 1 Mar 2018 13:30:55 -0800 (PST) Received: by mail-oi0-x244.google.com with SMTP id j81so5641947oia.0 for ; Thu, 01 Mar 2018 13:37:04 -0800 (PST) MIME-Version: 1.0 In-Reply-To: <1519936477.4592.23.camel@au1.ibm.com> References: <20180228234006.21093-1-logang@deltatee.com> <1519876489.4592.3.camel@kernel.crashing.org> <1519876569.4592.4.camel@au1.ibm.com> <1519936477.4592.23.camel@au1.ibm.com> From: Dan Williams Date: Thu, 1 Mar 2018 13:37:03 -0800 Message-ID: Subject: Re: [PATCH v2 00/10] Copy Offload in NVMe Fabrics with P2P PCI Memory List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: linux-nvdimm-bounces@lists.01.org Sender: "Linux-nvdimm" To: benh@au1.ibm.com Cc: Jens Axboe , Keith Busch , Oliver OHalloran , Alex Williamson , linux-nvdimm , linux-rdma , linux-pci@vger.kernel.org, Linux Kernel Mailing List , linux-nvme@lists.infradead.org, linux-block@vger.kernel.org, =?UTF-8?B?SsOpcsO0bWUgR2xpc3Nl?= , Jason Gunthorpe , Bjorn Helgaas , Max Gurtovoy , Christoph Hellwig List-ID: On Thu, Mar 1, 2018 at 12:34 PM, Benjamin Herrenschmidt wrote: > On Thu, 2018-03-01 at 11:21 -0800, Dan Williams wrote: >> On Wed, Feb 28, 2018 at 7:56 PM, Benjamin Herrenschmidt >> wrote: >> > On Thu, 2018-03-01 at 14:54 +1100, Benjamin Herrenschmidt wrote: >> > > On Wed, 2018-02-28 at 16:39 -0700, Logan Gunthorpe wrote: >> > > > Hi Everyone, >> > > >> > > >> > > So Oliver (CC) was having issues getting any of that to work for us. >> > > >> > > The problem is that acccording to him (I didn't double check the latest >> > > patches) you effectively hotplug the PCIe memory into the system when >> > > creating struct pages. >> > > >> > > This cannot possibly work for us. First we cannot map PCIe memory as >> > > cachable. (Note that doing so is a bad idea if you are behind a PLX >> > > switch anyway since you'd ahve to manage cache coherency in SW). >> > >> > Note: I think the above means it won't work behind a switch on x86 >> > either, will it ? >> >> The devm_memremap_pages() infrastructure allows placing the memmap in >> "System-RAM" even if the hotplugged range is in PCI space. So, even if >> it is an issue on some configurations, it's just a simple adjustment >> to where the memmap is placed. > > But what happens with that PCI memory ? Is it effectively turned into > nromal memory (ie, usable for normal allocations, potentially used to > populate user pages etc...) or is it kept aside ? > > Also on ppc64, the physical addresses of PCIe make it so far appart > that there's no way we can map them into the linear mapping at the > normal offset of PAGE_OFFSET + (pfn << PAGE_SHIFT), so things like > page_address or virt_to_page cannot work as-is on PCIe addresses. Ah ok, I'd need to look at the details. I had been assuming that sparse-vmemmap could handle such a situation, but that could indeed be a broken assumption. _______________________________________________ Linux-nvdimm mailing list Linux-nvdimm@lists.01.org https://lists.01.org/mailman/listinfo/linux-nvdimm From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: MIME-Version: 1.0 In-Reply-To: <1519936477.4592.23.camel@au1.ibm.com> References: <20180228234006.21093-1-logang@deltatee.com> <1519876489.4592.3.camel@kernel.crashing.org> <1519876569.4592.4.camel@au1.ibm.com> <1519936477.4592.23.camel@au1.ibm.com> From: Dan Williams Date: Thu, 1 Mar 2018 13:37:03 -0800 Message-ID: Subject: Re: [PATCH v2 00/10] Copy Offload in NVMe Fabrics with P2P PCI Memory To: benh@au1.ibm.com Cc: Logan Gunthorpe , Linux Kernel Mailing List , linux-pci@vger.kernel.org, linux-nvme@lists.infradead.org, linux-rdma , linux-nvdimm , linux-block@vger.kernel.org, Stephen Bates , Christoph Hellwig , Jens Axboe , Keith Busch , Sagi Grimberg , Bjorn Helgaas , Jason Gunthorpe , Max Gurtovoy , =?UTF-8?B?SsOpcsO0bWUgR2xpc3Nl?= , Alex Williamson , Oliver OHalloran Content-Type: text/plain; charset="UTF-8" List-ID: On Thu, Mar 1, 2018 at 12:34 PM, Benjamin Herrenschmidt wrote: > On Thu, 2018-03-01 at 11:21 -0800, Dan Williams wrote: >> On Wed, Feb 28, 2018 at 7:56 PM, Benjamin Herrenschmidt >> wrote: >> > On Thu, 2018-03-01 at 14:54 +1100, Benjamin Herrenschmidt wrote: >> > > On Wed, 2018-02-28 at 16:39 -0700, Logan Gunthorpe wrote: >> > > > Hi Everyone, >> > > >> > > >> > > So Oliver (CC) was having issues getting any of that to work for us. >> > > >> > > The problem is that acccording to him (I didn't double check the latest >> > > patches) you effectively hotplug the PCIe memory into the system when >> > > creating struct pages. >> > > >> > > This cannot possibly work for us. First we cannot map PCIe memory as >> > > cachable. (Note that doing so is a bad idea if you are behind a PLX >> > > switch anyway since you'd ahve to manage cache coherency in SW). >> > >> > Note: I think the above means it won't work behind a switch on x86 >> > either, will it ? >> >> The devm_memremap_pages() infrastructure allows placing the memmap in >> "System-RAM" even if the hotplugged range is in PCI space. So, even if >> it is an issue on some configurations, it's just a simple adjustment >> to where the memmap is placed. > > But what happens with that PCI memory ? Is it effectively turned into > nromal memory (ie, usable for normal allocations, potentially used to > populate user pages etc...) or is it kept aside ? > > Also on ppc64, the physical addresses of PCIe make it so far appart > that there's no way we can map them into the linear mapping at the > normal offset of PAGE_OFFSET + (pfn << PAGE_SHIFT), so things like > page_address or virt_to_page cannot work as-is on PCIe addresses. Ah ok, I'd need to look at the details. I had been assuming that sparse-vmemmap could handle such a situation, but that could indeed be a broken assumption. From mboxrd@z Thu Jan 1 00:00:00 1970 From: Dan Williams Subject: Re: [PATCH v2 00/10] Copy Offload in NVMe Fabrics with P2P PCI Memory Date: Thu, 1 Mar 2018 13:37:03 -0800 Message-ID: References: <20180228234006.21093-1-logang@deltatee.com> <1519876489.4592.3.camel@kernel.crashing.org> <1519876569.4592.4.camel@au1.ibm.com> <1519936477.4592.23.camel@au1.ibm.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <1519936477.4592.23.camel-8fk3Idey6ehBDgjK7y7TUQ@public.gmane.org> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: linux-nvdimm-bounces-hn68Rpc1hR1g9hUCZPvPmw@public.gmane.org Sender: "Linux-nvdimm" To: benh-8fk3Idey6ehBDgjK7y7TUQ@public.gmane.org Cc: Jens Axboe , Keith Busch , Oliver OHalloran , Alex Williamson , linux-nvdimm , linux-rdma , linux-pci-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, Linux Kernel Mailing List , linux-nvme-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r@public.gmane.org, linux-block-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, =?UTF-8?B?SsOpcsO0bWUgR2xpc3Nl?= , Jason Gunthorpe , Bjorn Helgaas , Max Gurtovoy , Christoph Hellwig List-Id: linux-rdma@vger.kernel.org On Thu, Mar 1, 2018 at 12:34 PM, Benjamin Herrenschmidt wrote: > On Thu, 2018-03-01 at 11:21 -0800, Dan Williams wrote: >> On Wed, Feb 28, 2018 at 7:56 PM, Benjamin Herrenschmidt >> wrote: >> > On Thu, 2018-03-01 at 14:54 +1100, Benjamin Herrenschmidt wrote: >> > > On Wed, 2018-02-28 at 16:39 -0700, Logan Gunthorpe wrote: >> > > > Hi Everyone, >> > > >> > > >> > > So Oliver (CC) was having issues getting any of that to work for us. >> > > >> > > The problem is that acccording to him (I didn't double check the latest >> > > patches) you effectively hotplug the PCIe memory into the system when >> > > creating struct pages. >> > > >> > > This cannot possibly work for us. First we cannot map PCIe memory as >> > > cachable. (Note that doing so is a bad idea if you are behind a PLX >> > > switch anyway since you'd ahve to manage cache coherency in SW). >> > >> > Note: I think the above means it won't work behind a switch on x86 >> > either, will it ? >> >> The devm_memremap_pages() infrastructure allows placing the memmap in >> "System-RAM" even if the hotplugged range is in PCI space. So, even if >> it is an issue on some configurations, it's just a simple adjustment >> to where the memmap is placed. > > But what happens with that PCI memory ? Is it effectively turned into > nromal memory (ie, usable for normal allocations, potentially used to > populate user pages etc...) or is it kept aside ? > > Also on ppc64, the physical addresses of PCIe make it so far appart > that there's no way we can map them into the linear mapping at the > normal offset of PAGE_OFFSET + (pfn << PAGE_SHIFT), so things like > page_address or virt_to_page cannot work as-is on PCIe addresses. Ah ok, I'd need to look at the details. I had been assuming that sparse-vmemmap could handle such a situation, but that could indeed be a broken assumption. From mboxrd@z Thu Jan 1 00:00:00 1970 From: dan.j.williams@intel.com (Dan Williams) Date: Thu, 1 Mar 2018 13:37:03 -0800 Subject: [PATCH v2 00/10] Copy Offload in NVMe Fabrics with P2P PCI Memory In-Reply-To: <1519936477.4592.23.camel@au1.ibm.com> References: <20180228234006.21093-1-logang@deltatee.com> <1519876489.4592.3.camel@kernel.crashing.org> <1519876569.4592.4.camel@au1.ibm.com> <1519936477.4592.23.camel@au1.ibm.com> Message-ID: On Thu, Mar 1, 2018 at 12:34 PM, Benjamin Herrenschmidt wrote: > On Thu, 2018-03-01@11:21 -0800, Dan Williams wrote: >> On Wed, Feb 28, 2018 at 7:56 PM, Benjamin Herrenschmidt >> wrote: >> > On Thu, 2018-03-01@14:54 +1100, Benjamin Herrenschmidt wrote: >> > > On Wed, 2018-02-28@16:39 -0700, Logan Gunthorpe wrote: >> > > > Hi Everyone, >> > > >> > > >> > > So Oliver (CC) was having issues getting any of that to work for us. >> > > >> > > The problem is that acccording to him (I didn't double check the latest >> > > patches) you effectively hotplug the PCIe memory into the system when >> > > creating struct pages. >> > > >> > > This cannot possibly work for us. First we cannot map PCIe memory as >> > > cachable. (Note that doing so is a bad idea if you are behind a PLX >> > > switch anyway since you'd ahve to manage cache coherency in SW). >> > >> > Note: I think the above means it won't work behind a switch on x86 >> > either, will it ? >> >> The devm_memremap_pages() infrastructure allows placing the memmap in >> "System-RAM" even if the hotplugged range is in PCI space. So, even if >> it is an issue on some configurations, it's just a simple adjustment >> to where the memmap is placed. > > But what happens with that PCI memory ? Is it effectively turned into > nromal memory (ie, usable for normal allocations, potentially used to > populate user pages etc...) or is it kept aside ? > > Also on ppc64, the physical addresses of PCIe make it so far appart > that there's no way we can map them into the linear mapping at the > normal offset of PAGE_OFFSET + (pfn << PAGE_SHIFT), so things like > page_address or virt_to_page cannot work as-is on PCIe addresses. Ah ok, I'd need to look at the details. I had been assuming that sparse-vmemmap could handle such a situation, but that could indeed be a broken assumption.