From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from ale.deltatee.com (ale.deltatee.com [207.54.116.67]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ml01.01.org (Postfix) with ESMTPS id 448E02194D3AE for ; Thu, 30 Aug 2018 12:30:42 -0700 (PDT) References: <20180830185352.3369-1-logang@deltatee.com> <20180830192012.GG3529@redhat.com> From: Logan Gunthorpe Message-ID: Date: Thu, 30 Aug 2018 13:30:32 -0600 MIME-Version: 1.0 In-Reply-To: <20180830192012.GG3529@redhat.com> Content-Language: en-US Subject: Re: [PATCH v5 00/13] Copy Offload in NVMe Fabrics with P2P PCI Memory List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: linux-nvdimm-bounces@lists.01.org Sender: "Linux-nvdimm" To: Jerome Glisse Cc: linux-nvdimm@lists.01.org, linux-rdma@vger.kernel.org, linux-pci@vger.kernel.org, linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org, linux-block@vger.kernel.org, Alex Williamson , Jason Gunthorpe , =?UTF-8?Q?Christian_K=c3=b6nig?= , Benjamin Herrenschmidt , Bjorn Helgaas , Max Gurtovoy , Christoph Hellwig List-ID: On 30/08/18 01:20 PM, Jerome Glisse wrote: > On Thu, Aug 30, 2018 at 12:53:39PM -0600, Logan Gunthorpe wrote: > > [...] > >> >> When the PCI P2PDMA config option is selected the ACS bits in every >> bridge port in the system are turned off to allow traffic to >> pass freely behind the root port. At this time, the bit must be disabled >> at boot so the IOMMU subsystem can correctly create the groups, though >> this could be addressed in the future. There is no way to dynamically >> disable the bit and alter the groups. Oh, sorry this paragraph in the cover letter is wrong now. We now rely on the disable_acs_redir command line option introduced in aaca43fda742 ("PCI: Add "pci=disable_acs_redir=" parameter for peer-to-peer support") > Can you provide an example on how to test this ? Like kernel command > line option, the doc patch does not have any such example. It would be > nice to add. Do you mean to test the patchset or the ACS bits you quoted? Testing the patchset is a matter of having the right hardware (ie an RDMA NIC and CMB enabled NVMe behind a PCIe switch, with the ACS bits set correctly by the above command line option) and setting the p2pmem configfs attribute in an nvme-of port to 'yes'. Logan _______________________________________________ Linux-nvdimm mailing list Linux-nvdimm@lists.01.org https://lists.01.org/mailman/listinfo/linux-nvdimm From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from ale.deltatee.com ([207.54.116.67]:40724 "EHLO ale.deltatee.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725836AbeH3Xe0 (ORCPT ); Thu, 30 Aug 2018 19:34:26 -0400 To: Jerome Glisse Cc: linux-kernel@vger.kernel.org, linux-pci@vger.kernel.org, linux-nvme@lists.infradead.org, linux-rdma@vger.kernel.org, linux-nvdimm@lists.01.org, linux-block@vger.kernel.org, Stephen Bates , Christoph Hellwig , Keith Busch , Sagi Grimberg , Bjorn Helgaas , Jason Gunthorpe , Max Gurtovoy , Dan Williams , Benjamin Herrenschmidt , Alex Williamson , =?UTF-8?Q?Christian_K=c3=b6nig?= References: <20180830185352.3369-1-logang@deltatee.com> <20180830192012.GG3529@redhat.com> From: Logan Gunthorpe Message-ID: Date: Thu, 30 Aug 2018 13:30:32 -0600 MIME-Version: 1.0 In-Reply-To: <20180830192012.GG3529@redhat.com> Content-Type: text/plain; charset=utf-8 Subject: Re: [PATCH v5 00/13] Copy Offload in NVMe Fabrics with P2P PCI Memory Sender: linux-block-owner@vger.kernel.org List-Id: linux-block@vger.kernel.org On 30/08/18 01:20 PM, Jerome Glisse wrote: > On Thu, Aug 30, 2018 at 12:53:39PM -0600, Logan Gunthorpe wrote: > > [...] > >> >> When the PCI P2PDMA config option is selected the ACS bits in every >> bridge port in the system are turned off to allow traffic to >> pass freely behind the root port. At this time, the bit must be disabled >> at boot so the IOMMU subsystem can correctly create the groups, though >> this could be addressed in the future. There is no way to dynamically >> disable the bit and alter the groups. Oh, sorry this paragraph in the cover letter is wrong now. We now rely on the disable_acs_redir command line option introduced in aaca43fda742 ("PCI: Add "pci=disable_acs_redir=" parameter for peer-to-peer support") > Can you provide an example on how to test this ? Like kernel command > line option, the doc patch does not have any such example. It would be > nice to add. Do you mean to test the patchset or the ACS bits you quoted? Testing the patchset is a matter of having the right hardware (ie an RDMA NIC and CMB enabled NVMe behind a PCIe switch, with the ACS bits set correctly by the above command line option) and setting the p2pmem configfs attribute in an nvme-of port to 'yes'. Logan From mboxrd@z Thu Jan 1 00:00:00 1970 From: Logan Gunthorpe Subject: Re: [PATCH v5 00/13] Copy Offload in NVMe Fabrics with P2P PCI Memory Date: Thu, 30 Aug 2018 13:30:32 -0600 Message-ID: References: <20180830185352.3369-1-logang@deltatee.com> <20180830192012.GG3529@redhat.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <20180830192012.GG3529-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> Content-Language: en-US List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: linux-nvdimm-bounces-hn68Rpc1hR1g9hUCZPvPmw@public.gmane.org Sender: "Linux-nvdimm" To: Jerome Glisse Cc: linux-nvdimm-hn68Rpc1hR1g9hUCZPvPmw@public.gmane.org, linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, linux-pci-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, linux-nvme-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r@public.gmane.org, linux-block-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, Alex Williamson , Jason Gunthorpe , =?UTF-8?Q?Christian_K=c3=b6nig?= , Benjamin Herrenschmidt , Bjorn Helgaas , Max Gurtovoy , Christoph Hellwig List-Id: linux-rdma@vger.kernel.org On 30/08/18 01:20 PM, Jerome Glisse wrote: > On Thu, Aug 30, 2018 at 12:53:39PM -0600, Logan Gunthorpe wrote: > > [...] > >> >> When the PCI P2PDMA config option is selected the ACS bits in every >> bridge port in the system are turned off to allow traffic to >> pass freely behind the root port. At this time, the bit must be disabled >> at boot so the IOMMU subsystem can correctly create the groups, though >> this could be addressed in the future. There is no way to dynamically >> disable the bit and alter the groups. Oh, sorry this paragraph in the cover letter is wrong now. We now rely on the disable_acs_redir command line option introduced in aaca43fda742 ("PCI: Add "pci=disable_acs_redir=" parameter for peer-to-peer support") > Can you provide an example on how to test this ? Like kernel command > line option, the doc patch does not have any such example. It would be > nice to add. Do you mean to test the patchset or the ACS bits you quoted? Testing the patchset is a matter of having the right hardware (ie an RDMA NIC and CMB enabled NVMe behind a PCIe switch, with the ACS bits set correctly by the above command line option) and setting the p2pmem configfs attribute in an nvme-of port to 'yes'. Logan From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Return-Path: To: Jerome Glisse References: <20180830185352.3369-1-logang@deltatee.com> <20180830192012.GG3529@redhat.com> From: Logan Gunthorpe Message-ID: Date: Thu, 30 Aug 2018 13:30:32 -0600 MIME-Version: 1.0 In-Reply-To: <20180830192012.GG3529@redhat.com> Subject: Re: [PATCH v5 00/13] Copy Offload in NVMe Fabrics with P2P PCI Memory List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Keith Busch , Sagi Grimberg , linux-nvdimm@lists.01.org, linux-rdma@vger.kernel.org, linux-pci@vger.kernel.org, linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org, Stephen Bates , linux-block@vger.kernel.org, Alex Williamson , Jason Gunthorpe , =?UTF-8?Q?Christian_K=c3=b6nig?= , Benjamin Herrenschmidt , Bjorn Helgaas , Max Gurtovoy , Dan Williams , Christoph Hellwig Content-Type: text/plain; charset="us-ascii" Sender: "Linux-nvme" Errors-To: linux-nvme-bounces+bjorn=helgaas.com@lists.infradead.org List-ID: On 30/08/18 01:20 PM, Jerome Glisse wrote: > On Thu, Aug 30, 2018 at 12:53:39PM -0600, Logan Gunthorpe wrote: > > [...] > >> >> When the PCI P2PDMA config option is selected the ACS bits in every >> bridge port in the system are turned off to allow traffic to >> pass freely behind the root port. At this time, the bit must be disabled >> at boot so the IOMMU subsystem can correctly create the groups, though >> this could be addressed in the future. There is no way to dynamically >> disable the bit and alter the groups. Oh, sorry this paragraph in the cover letter is wrong now. We now rely on the disable_acs_redir command line option introduced in aaca43fda742 ("PCI: Add "pci=disable_acs_redir=" parameter for peer-to-peer support") > Can you provide an example on how to test this ? Like kernel command > line option, the doc patch does not have any such example. It would be > nice to add. Do you mean to test the patchset or the ACS bits you quoted? Testing the patchset is a matter of having the right hardware (ie an RDMA NIC and CMB enabled NVMe behind a PCIe switch, with the ACS bits set correctly by the above command line option) and setting the p2pmem configfs attribute in an nvme-of port to 'yes'. Logan _______________________________________________ Linux-nvme mailing list Linux-nvme@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-nvme From mboxrd@z Thu Jan 1 00:00:00 1970 From: logang@deltatee.com (Logan Gunthorpe) Date: Thu, 30 Aug 2018 13:30:32 -0600 Subject: [PATCH v5 00/13] Copy Offload in NVMe Fabrics with P2P PCI Memory In-Reply-To: <20180830192012.GG3529@redhat.com> References: <20180830185352.3369-1-logang@deltatee.com> <20180830192012.GG3529@redhat.com> Message-ID: On 30/08/18 01:20 PM, Jerome Glisse wrote: > On Thu, Aug 30, 2018@12:53:39PM -0600, Logan Gunthorpe wrote: > > [...] > >> >> When the PCI P2PDMA config option is selected the ACS bits in every >> bridge port in the system are turned off to allow traffic to >> pass freely behind the root port. At this time, the bit must be disabled >> at boot so the IOMMU subsystem can correctly create the groups, though >> this could be addressed in the future. There is no way to dynamically >> disable the bit and alter the groups. Oh, sorry this paragraph in the cover letter is wrong now. We now rely on the disable_acs_redir command line option introduced in aaca43fda742 ("PCI: Add "pci=disable_acs_redir=" parameter for peer-to-peer support") > Can you provide an example on how to test this ? Like kernel command > line option, the doc patch does not have any such example. It would be > nice to add. Do you mean to test the patchset or the ACS bits you quoted? Testing the patchset is a matter of having the right hardware (ie an RDMA NIC and CMB enabled NVMe behind a PCIe switch, with the ACS bits set correctly by the above command line option) and setting the p2pmem configfs attribute in an nvme-of port to 'yes'. Logan