From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from zimbra1.kalray.eu (zimbra1.kalray.eu [92.103.151.219]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ml01.01.org (Postfix) with ESMTPS id 0D7E721A0480E for ; Tue, 25 Apr 2017 04:58:56 -0700 (PDT) Date: Tue, 25 Apr 2017 13:58:53 +0200 (CEST) From: Marta Rybczynska Message-ID: <117239810.400362790.1493121533234.JavaMail.zimbra@kalray.eu> In-Reply-To: References: <1490911959-5146-1-git-send-email-logang@deltatee.com> <63d73995-4dfa-6270-ca13-972e0628011b@deltatee.com> Subject: Re: [RFC 1/8] Introduce Peer-to-Peer memory (p2pmem) device MIME-Version: 1.0 List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: linux-nvdimm-bounces@lists.01.org Sender: "Linux-nvdimm" To: Logan Gunthorpe Cc: Jens Axboe , Keith Busch , "James E.J. Bottomley" , "Martin K. Petersen" , linux-nvdimm@lists.01.org, linux-rdma@vger.kernel.org, linux-pci@vger.kernel.org, Steve Wise , linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org, Sinan Kaya , Alex Williamson , linux-scsi@vger.kernel.org, Bjorn Helgaas , Max Gurtovoy , Christoph Hellwig , Jason Gunthorpe List-ID: > On 02/04/17 03:03 PM, Sinan Kaya wrote: >> Push the decision all the way to the user. Let them decide whether they >> want this feature to work on a root port connected port or under the >> switch. > > Yes, I prefer this too. If other folks agree with that I'd be very happy > to go back to user chooses. I think Sagi was the most vocal proponent > for kernel chooses at LSF so hopefully he will read this thread and > offer some opinion. > >> I thought the issue was feature didn't work at all with some root ports >> or there was some kind of memory corruption issue that you were trying to >> avoid with the existing systems. > > I *think* there are some much older root ports where P2P TLPs don't even > get through. But it doesn't really change the situation: in the nvmet > case, the user would enable p2pmem and then be unable to connect and > thus choose to disable it going forward. Not a big difference from the > user seeing bad performance and not choosing to enable it. > >> I think you should get rid of all pci searching business in your code. > > Yes, my original proposal was when you configure the nvme target you > chose the specific p2pmem device to use. That code had no tie ins to PCI > code and could, in theory, work generically with any device and bus. > I would add one issue that doesn't seem to be addressed: in my experience P2P doesn't work when IOMMU activated. It works best with deactivation at the BIOS level, even the kernel options are not enough in some cases. This is another argument to leave the chose to user/integrator. Marta _______________________________________________ Linux-nvdimm mailing list Linux-nvdimm@lists.01.org https://lists.01.org/mailman/listinfo/linux-nvdimm From mboxrd@z Thu Jan 1 00:00:00 1970 From: Marta Rybczynska Subject: Re: [RFC 1/8] Introduce Peer-to-Peer memory (p2pmem) device Date: Tue, 25 Apr 2017 13:58:53 +0200 (CEST) Message-ID: <117239810.400362790.1493121533234.JavaMail.zimbra@kalray.eu> References: <1490911959-5146-1-git-send-email-logang@deltatee.com> <63d73995-4dfa-6270-ca13-972e0628011b@deltatee.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: linux-nvdimm-bounces-hn68Rpc1hR1g9hUCZPvPmw@public.gmane.org Sender: "Linux-nvdimm" To: Logan Gunthorpe Cc: Jens Axboe , Keith Busch , "James E.J. Bottomley" , "Martin K. Petersen" , linux-nvdimm-hn68Rpc1hR1g9hUCZPvPmw@public.gmane.org, linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, linux-pci-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, Steve Wise , linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, linux-nvme-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r@public.gmane.org, Sinan Kaya , Alex Williamson , linux-scsi-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, Bjorn Helgaas , Max Gurtovoy , Christoph Hellwig , Jason Gunthorpe List-Id: linux-rdma@vger.kernel.org > On 02/04/17 03:03 PM, Sinan Kaya wrote: >> Push the decision all the way to the user. Let them decide whether they >> want this feature to work on a root port connected port or under the >> switch. > > Yes, I prefer this too. If other folks agree with that I'd be very happy > to go back to user chooses. I think Sagi was the most vocal proponent > for kernel chooses at LSF so hopefully he will read this thread and > offer some opinion. > >> I thought the issue was feature didn't work at all with some root ports >> or there was some kind of memory corruption issue that you were trying to >> avoid with the existing systems. > > I *think* there are some much older root ports where P2P TLPs don't even > get through. But it doesn't really change the situation: in the nvmet > case, the user would enable p2pmem and then be unable to connect and > thus choose to disable it going forward. Not a big difference from the > user seeing bad performance and not choosing to enable it. > >> I think you should get rid of all pci searching business in your code. > > Yes, my original proposal was when you configure the nvme target you > chose the specific p2pmem device to use. That code had no tie ins to PCI > code and could, in theory, work generically with any device and bus. > I would add one issue that doesn't seem to be addressed: in my experience P2P doesn't work when IOMMU activated. It works best with deactivation at the BIOS level, even the kernel options are not enough in some cases. This is another argument to leave the chose to user/integrator. Marta From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1947273AbdDYMIh (ORCPT ); Tue, 25 Apr 2017 08:08:37 -0400 Received: from zimbra1.kalray.eu ([92.103.151.219]:54715 "EHLO zimbra1.kalray.eu" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1175922AbdDYMIb (ORCPT ); Tue, 25 Apr 2017 08:08:31 -0400 X-Greylist: delayed 575 seconds by postgrey-1.27 at vger.kernel.org; Tue, 25 Apr 2017 08:08:30 EDT DKIM-Filter: OpenDKIM Filter v2.9.2 zimbra1.kalray.eu 99EDE2819E2 Date: Tue, 25 Apr 2017 13:58:53 +0200 (CEST) From: Marta Rybczynska To: Logan Gunthorpe Cc: Sinan Kaya , Christoph Hellwig , Sagi Grimberg , "James E.J. Bottomley" , "Martin K. Petersen" , Jens Axboe , Steve Wise , Stephen Bates , Max Gurtovoy , Dan Williams , Keith Busch , Jason Gunthorpe , linux-pci@vger.kernel.org, linux-scsi@vger.kernel.org, linux-nvme@lists.infradead.org, linux-rdma@vger.kernel.org, linux-nvdimm@ml01.01.org, linux-kernel@vger.kernel.org, Alex Williamson , Bjorn Helgaas Message-ID: <117239810.400362790.1493121533234.JavaMail.zimbra@kalray.eu> In-Reply-To: References: <1490911959-5146-1-git-send-email-logang@deltatee.com> <63d73995-4dfa-6270-ca13-972e0628011b@deltatee.com> Subject: Re: [RFC 1/8] Introduce Peer-to-Peer memory (p2pmem) device MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [192.168.37.210] X-Mailer: Zimbra 8.6.0_GA_1182 (ZimbraWebClient - FF45 (Linux)/8.6.0_GA_1182) Thread-Topic: Introduce Peer-to-Peer memory (p2pmem) device Thread-Index: xERFZreghxIKUyfkKN1rIWRkU/X0aQ== Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org > On 02/04/17 03:03 PM, Sinan Kaya wrote: >> Push the decision all the way to the user. Let them decide whether they >> want this feature to work on a root port connected port or under the >> switch. > > Yes, I prefer this too. If other folks agree with that I'd be very happy > to go back to user chooses. I think Sagi was the most vocal proponent > for kernel chooses at LSF so hopefully he will read this thread and > offer some opinion. > >> I thought the issue was feature didn't work at all with some root ports >> or there was some kind of memory corruption issue that you were trying to >> avoid with the existing systems. > > I *think* there are some much older root ports where P2P TLPs don't even > get through. But it doesn't really change the situation: in the nvmet > case, the user would enable p2pmem and then be unable to connect and > thus choose to disable it going forward. Not a big difference from the > user seeing bad performance and not choosing to enable it. > >> I think you should get rid of all pci searching business in your code. > > Yes, my original proposal was when you configure the nvme target you > chose the specific p2pmem device to use. That code had no tie ins to PCI > code and could, in theory, work generically with any device and bus. > I would add one issue that doesn't seem to be addressed: in my experience P2P doesn't work when IOMMU activated. It works best with deactivation at the BIOS level, even the kernel options are not enough in some cases. This is another argument to leave the chose to user/integrator. Marta From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from zimbra1.kalray.eu ([92.103.151.219]:54715 "EHLO zimbra1.kalray.eu" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1175922AbdDYMIb (ORCPT ); Tue, 25 Apr 2017 08:08:31 -0400 Date: Tue, 25 Apr 2017 13:58:53 +0200 (CEST) From: Marta Rybczynska To: Logan Gunthorpe Cc: Sinan Kaya , Christoph Hellwig , Sagi Grimberg , "James E.J. Bottomley" , "Martin K. Petersen" , Jens Axboe , Steve Wise , Stephen Bates , Max Gurtovoy , Dan Williams , Keith Busch , Jason Gunthorpe , linux-pci@vger.kernel.org, linux-scsi@vger.kernel.org, linux-nvme@lists.infradead.org, linux-rdma@vger.kernel.org, linux-nvdimm@lists.01.org, linux-kernel@vger.kernel.org, Alex Williamson , Bjorn Helgaas Message-ID: <117239810.400362790.1493121533234.JavaMail.zimbra@kalray.eu> In-Reply-To: References: <1490911959-5146-1-git-send-email-logang@deltatee.com> <63d73995-4dfa-6270-ca13-972e0628011b@deltatee.com> Subject: Re: [RFC 1/8] Introduce Peer-to-Peer memory (p2pmem) device MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Sender: linux-pci-owner@vger.kernel.org List-ID: > On 02/04/17 03:03 PM, Sinan Kaya wrote: >> Push the decision all the way to the user. Let them decide whether they >> want this feature to work on a root port connected port or under the >> switch. > > Yes, I prefer this too. If other folks agree with that I'd be very happy > to go back to user chooses. I think Sagi was the most vocal proponent > for kernel chooses at LSF so hopefully he will read this thread and > offer some opinion. > >> I thought the issue was feature didn't work at all with some root ports >> or there was some kind of memory corruption issue that you were trying to >> avoid with the existing systems. > > I *think* there are some much older root ports where P2P TLPs don't even > get through. But it doesn't really change the situation: in the nvmet > case, the user would enable p2pmem and then be unable to connect and > thus choose to disable it going forward. Not a big difference from the > user seeing bad performance and not choosing to enable it. > >> I think you should get rid of all pci searching business in your code. > > Yes, my original proposal was when you configure the nvme target you > chose the specific p2pmem device to use. That code had no tie ins to PCI > code and could, in theory, work generically with any device and bus. > I would add one issue that doesn't seem to be addressed: in my experience P2P doesn't work when IOMMU activated. It works best with deactivation at the BIOS level, even the kernel options are not enough in some cases. This is another argument to leave the chose to user/integrator. Marta From mboxrd@z Thu Jan 1 00:00:00 1970 From: mrybczyn@kalray.eu (Marta Rybczynska) Date: Tue, 25 Apr 2017 13:58:53 +0200 (CEST) Subject: [RFC 1/8] Introduce Peer-to-Peer memory (p2pmem) device In-Reply-To: References: <1490911959-5146-1-git-send-email-logang@deltatee.com> <63d73995-4dfa-6270-ca13-972e0628011b@deltatee.com> Message-ID: <117239810.400362790.1493121533234.JavaMail.zimbra@kalray.eu> > On 02/04/17 03:03 PM, Sinan Kaya wrote: >> Push the decision all the way to the user. Let them decide whether they >> want this feature to work on a root port connected port or under the >> switch. > > Yes, I prefer this too. If other folks agree with that I'd be very happy > to go back to user chooses. I think Sagi was the most vocal proponent > for kernel chooses at LSF so hopefully he will read this thread and > offer some opinion. > >> I thought the issue was feature didn't work at all with some root ports >> or there was some kind of memory corruption issue that you were trying to >> avoid with the existing systems. > > I *think* there are some much older root ports where P2P TLPs don't even > get through. But it doesn't really change the situation: in the nvmet > case, the user would enable p2pmem and then be unable to connect and > thus choose to disable it going forward. Not a big difference from the > user seeing bad performance and not choosing to enable it. > >> I think you should get rid of all pci searching business in your code. > > Yes, my original proposal was when you configure the nvme target you > chose the specific p2pmem device to use. That code had no tie ins to PCI > code and could, in theory, work generically with any device and bus. > I would add one issue that doesn't seem to be addressed: in my experience P2P doesn't work when IOMMU activated. It works best with deactivation at the BIOS level, even the kernel options are not enough in some cases. This is another argument to leave the chose to user/integrator. Marta