From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mx1.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ml01.01.org (Postfix) with ESMTPS id D9A7E224C0F49 for ; Thu, 1 Mar 2018 13:29:29 -0800 (PST) Date: Thu, 1 Mar 2018 16:35:33 -0500 From: Jerome Glisse Subject: Re: [PATCH v2 04/10] PCI/P2PDMA: Clear ACS P2P flags for all devices behind switches Message-ID: <20180301213533.GE6742@redhat.com> References: <20180228234006.21093-1-logang@deltatee.com> <20180228234006.21093-5-logang@deltatee.com> <20180301180257.GH13722@bhelgaas-glaptop.roam.corp.google.com> <0D05579B-789C-4A19-B3A2-C1A630BE31C0@raithlin.com> <20180301142155.5966c4c0@w520.home> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Errors-To: linux-nvdimm-bounces@lists.01.org Sender: "Linux-nvdimm" To: Stephen Bates Cc: Jens Axboe , Keith Busch , "linux-nvdimm@lists.01.org" , "linux-rdma@vger.kernel.org" , "linux-pci@vger.kernel.org" , "linux-kernel@vger.kernel.org" , "linux-nvme@lists.infradead.org" , "linux-block@vger.kernel.org" , Alex Williamson , Jason Gunthorpe , Bjorn Helgaas , Benjamin Herrenschmidt , Bjorn Helgaas , Max Gurtovoy , Christoph Hellwig List-ID: On Thu, Mar 01, 2018 at 09:32:20PM +0000, Stephen Bates wrote: > > your kernel provider needs to decide whether they favor device assignme= nt or p2p > = > Thanks Alex! The hardware requirements for P2P (switch, high performance = EPs) are such that we really only expect CONFIG_P2P_DMA to be enabled in sp= ecific instances and in those instances the users have made a decision to f= avor P2P over IOMMU isolation. Or they have setup their PCIe topology in a = way that gives them IOMMU isolation where they want it and P2P where they w= ant it. > = > = Note that they are usecase for P2P where IOMMU isolation matter and the traffic through root complex isn't see as an issue. For instance for GPU the idea is that you want to allow the RDMA device to directly read or write from GPU memory to avoid having to migrate memory to system memory. This isn't so much for performance than for ease of use. Cheers, J=E9r=F4me _______________________________________________ Linux-nvdimm mailing list Linux-nvdimm@lists.01.org https://lists.01.org/mailman/listinfo/linux-nvdimm From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Date: Thu, 1 Mar 2018 16:35:33 -0500 From: Jerome Glisse To: Stephen Bates Cc: Alex Williamson , Bjorn Helgaas , Logan Gunthorpe , "linux-kernel@vger.kernel.org" , "linux-pci@vger.kernel.org" , "linux-nvme@lists.infradead.org" , "linux-rdma@vger.kernel.org" , "linux-nvdimm@lists.01.org" , "linux-block@vger.kernel.org" , Christoph Hellwig , Jens Axboe , Keith Busch , Sagi Grimberg , Bjorn Helgaas , Jason Gunthorpe , Max Gurtovoy , Dan Williams , Benjamin Herrenschmidt Subject: Re: [PATCH v2 04/10] PCI/P2PDMA: Clear ACS P2P flags for all devices behind switches Message-ID: <20180301213533.GE6742@redhat.com> References: <20180228234006.21093-1-logang@deltatee.com> <20180228234006.21093-5-logang@deltatee.com> <20180301180257.GH13722@bhelgaas-glaptop.roam.corp.google.com> <0D05579B-789C-4A19-B3A2-C1A630BE31C0@raithlin.com> <20180301142155.5966c4c0@w520.home> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 In-Reply-To: List-ID: On Thu, Mar 01, 2018 at 09:32:20PM +0000, Stephen Bates wrote: > > your kernel provider needs to decide whether they favor device assignment or p2p > > Thanks Alex! The hardware requirements for P2P (switch, high performance EPs) are such that we really only expect CONFIG_P2P_DMA to be enabled in specific instances and in those instances the users have made a decision to favor P2P over IOMMU isolation. Or they have setup their PCIe topology in a way that gives them IOMMU isolation where they want it and P2P where they want it. > > Note that they are usecase for P2P where IOMMU isolation matter and the traffic through root complex isn't see as an issue. For instance for GPU the idea is that you want to allow the RDMA device to directly read or write from GPU memory to avoid having to migrate memory to system memory. This isn't so much for performance than for ease of use. Cheers, J�r�me From mboxrd@z Thu Jan 1 00:00:00 1970 From: Jerome Glisse Subject: Re: [PATCH v2 04/10] PCI/P2PDMA: Clear ACS P2P flags for all devices behind switches Date: Thu, 1 Mar 2018 16:35:33 -0500 Message-ID: <20180301213533.GE6742@redhat.com> References: <20180228234006.21093-1-logang@deltatee.com> <20180228234006.21093-5-logang@deltatee.com> <20180301180257.GH13722@bhelgaas-glaptop.roam.corp.google.com> <0D05579B-789C-4A19-B3A2-C1A630BE31C0@raithlin.com> <20180301142155.5966c4c0@w520.home> Mime-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Return-path: Content-Disposition: inline In-Reply-To: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: linux-nvdimm-bounces-hn68Rpc1hR1g9hUCZPvPmw@public.gmane.org Sender: "Linux-nvdimm" To: Stephen Bates Cc: Jens Axboe , Keith Busch , "linux-nvdimm-hn68Rpc1hR1g9hUCZPvPmw@public.gmane.org" , "linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org" , "linux-pci-u79uwXL29TY76Z2rM5mHXA@public.gmane.org" , "linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org" , "linux-nvme-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r@public.gmane.org" , "linux-block-u79uwXL29TY76Z2rM5mHXA@public.gmane.org" , Alex Williamson , Jason Gunthorpe , Bjorn Helgaas , Benjamin Herrenschmidt , Bjorn Helgaas , Max Gurtovoy , Christoph Hellwig List-Id: linux-rdma@vger.kernel.org On Thu, Mar 01, 2018 at 09:32:20PM +0000, Stephen Bates wrote: > > your kernel provider needs to decide whether they favor device assignme= nt or p2p > = > Thanks Alex! The hardware requirements for P2P (switch, high performance = EPs) are such that we really only expect CONFIG_P2P_DMA to be enabled in sp= ecific instances and in those instances the users have made a decision to f= avor P2P over IOMMU isolation. Or they have setup their PCIe topology in a = way that gives them IOMMU isolation where they want it and P2P where they w= ant it. > = > = Note that they are usecase for P2P where IOMMU isolation matter and the traffic through root complex isn't see as an issue. For instance for GPU the idea is that you want to allow the RDMA device to directly read or write from GPU memory to avoid having to migrate memory to system memory. This isn't so much for performance than for ease of use. Cheers, J=E9r=F4me From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1162251AbeCAVfx (ORCPT ); Thu, 1 Mar 2018 16:35:53 -0500 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:45198 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1162225AbeCAVfi (ORCPT ); Thu, 1 Mar 2018 16:35:38 -0500 Date: Thu, 1 Mar 2018 16:35:33 -0500 From: Jerome Glisse To: Stephen Bates Cc: Alex Williamson , Bjorn Helgaas , Logan Gunthorpe , "linux-kernel@vger.kernel.org" , "linux-pci@vger.kernel.org" , "linux-nvme@lists.infradead.org" , "linux-rdma@vger.kernel.org" , "linux-nvdimm@lists.01.org" , "linux-block@vger.kernel.org" , Christoph Hellwig , Jens Axboe , Keith Busch , Sagi Grimberg , Bjorn Helgaas , Jason Gunthorpe , Max Gurtovoy , Dan Williams , Benjamin Herrenschmidt Subject: Re: [PATCH v2 04/10] PCI/P2PDMA: Clear ACS P2P flags for all devices behind switches Message-ID: <20180301213533.GE6742@redhat.com> References: <20180228234006.21093-1-logang@deltatee.com> <20180228234006.21093-5-logang@deltatee.com> <20180301180257.GH13722@bhelgaas-glaptop.roam.corp.google.com> <0D05579B-789C-4A19-B3A2-C1A630BE31C0@raithlin.com> <20180301142155.5966c4c0@w520.home> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: User-Agent: Mutt/1.9.2 (2017-12-15) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Thu, Mar 01, 2018 at 09:32:20PM +0000, Stephen Bates wrote: > > your kernel provider needs to decide whether they favor device assignment or p2p > > Thanks Alex! The hardware requirements for P2P (switch, high performance EPs) are such that we really only expect CONFIG_P2P_DMA to be enabled in specific instances and in those instances the users have made a decision to favor P2P over IOMMU isolation. Or they have setup their PCIe topology in a way that gives them IOMMU isolation where they want it and P2P where they want it. > > Note that they are usecase for P2P where IOMMU isolation matter and the traffic through root complex isn't see as an issue. For instance for GPU the idea is that you want to allow the RDMA device to directly read or write from GPU memory to avoid having to migrate memory to system memory. This isn't so much for performance than for ease of use. Cheers, Jérôme From mboxrd@z Thu Jan 1 00:00:00 1970 From: jglisse@redhat.com (Jerome Glisse) Date: Thu, 1 Mar 2018 16:35:33 -0500 Subject: [PATCH v2 04/10] PCI/P2PDMA: Clear ACS P2P flags for all devices behind switches In-Reply-To: References: <20180228234006.21093-1-logang@deltatee.com> <20180228234006.21093-5-logang@deltatee.com> <20180301180257.GH13722@bhelgaas-glaptop.roam.corp.google.com> <0D05579B-789C-4A19-B3A2-C1A630BE31C0@raithlin.com> <20180301142155.5966c4c0@w520.home> Message-ID: <20180301213533.GE6742@redhat.com> On Thu, Mar 01, 2018@09:32:20PM +0000, Stephen Bates wrote: > > your kernel provider needs to decide whether they favor device assignment or p2p > > Thanks Alex! The hardware requirements for P2P (switch, high performance EPs) are such that we really only expect CONFIG_P2P_DMA to be enabled in specific instances and in those instances the users have made a decision to favor P2P over IOMMU isolation. Or they have setup their PCIe topology in a way that gives them IOMMU isolation where they want it and P2P where they want it. > > Note that they are usecase for P2P where IOMMU isolation matter and the traffic through root complex isn't see as an issue. For instance for GPU the idea is that you want to allow the RDMA device to directly read or write from GPU memory to avoid having to migrate memory to system memory. This isn't so much for performance than for ease of use. Cheers, J?r?me