From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from ale.deltatee.com (ale.deltatee.com [207.54.116.67]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ml01.01.org (Postfix) with ESMTPS id AB24421DFA7A9 for ; Sun, 2 Apr 2017 21:26:21 -0700 (PDT) References: <1490911959-5146-1-git-send-email-logang@deltatee.com> <1490911959-5146-2-git-send-email-logang@deltatee.com> <7158f2e8-2016-f398-e77f-0fcbe6cb41dd@deltatee.com> <0280fbb4-ba9e-ac64-6bb3-b72590a54e57@deltatee.com> <0ae27bca-21be-b89c-aba4-6cc9766ebd7b@codeaurora.org> <63d73995-4dfa-6270-ca13-972e0628011b@deltatee.com> From: Logan Gunthorpe Message-ID: Date: Sun, 2 Apr 2017 22:26:08 -0600 MIME-Version: 1.0 In-Reply-To: Subject: Re: [RFC 1/8] Introduce Peer-to-Peer memory (p2pmem) device List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: linux-nvdimm-bounces@lists.01.org Sender: "Linux-nvdimm" To: Sinan Kaya Cc: Jens Axboe , Jason Gunthorpe , "James E.J. Bottomley" , "Martin K. Petersen" , linux-nvdimm@lists.01.org, linux-rdma@vger.kernel.org, linux-pci@vger.kernel.org, Steve Wise , linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org, Keith Busch , Alex Williamson , linux-scsi@vger.kernel.org, Bjorn Helgaas , Max Gurtovoy , Christoph Hellwig List-ID: On 02/04/17 03:03 PM, Sinan Kaya wrote: > Push the decision all the way to the user. Let them decide whether they > want this feature to work on a root port connected port or under the > switch. Yes, I prefer this too. If other folks agree with that I'd be very happy to go back to user chooses. I think Sagi was the most vocal proponent for kernel chooses at LSF so hopefully he will read this thread and offer some opinion. > I thought the issue was feature didn't work at all with some root ports > or there was some kind of memory corruption issue that you were trying to > avoid with the existing systems. I *think* there are some much older root ports where P2P TLPs don't even get through. But it doesn't really change the situation: in the nvmet case, the user would enable p2pmem and then be unable to connect and thus choose to disable it going forward. Not a big difference from the user seeing bad performance and not choosing to enable it. > I think you should get rid of all pci searching business in your code. Yes, my original proposal was when you configure the nvme target you chose the specific p2pmem device to use. That code had no tie ins to PCI code and could, in theory, work generically with any device and bus. Logan _______________________________________________ Linux-nvdimm mailing list Linux-nvdimm@lists.01.org https://lists.01.org/mailman/listinfo/linux-nvdimm From mboxrd@z Thu Jan 1 00:00:00 1970 From: Logan Gunthorpe Subject: Re: [RFC 1/8] Introduce Peer-to-Peer memory (p2pmem) device Date: Sun, 2 Apr 2017 22:26:08 -0600 Message-ID: References: <1490911959-5146-1-git-send-email-logang@deltatee.com> <1490911959-5146-2-git-send-email-logang@deltatee.com> <7158f2e8-2016-f398-e77f-0fcbe6cb41dd@deltatee.com> <0280fbb4-ba9e-ac64-6bb3-b72590a54e57@deltatee.com> <0ae27bca-21be-b89c-aba4-6cc9766ebd7b@codeaurora.org> <63d73995-4dfa-6270-ca13-972e0628011b@deltatee.com> Mime-Version: 1.0 Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: Sender: linux-scsi-owner@vger.kernel.org To: Sinan Kaya Cc: Christoph Hellwig , Sagi Grimberg , "James E.J. Bottomley" , "Martin K. Petersen" , Jens Axboe , Steve Wise , Stephen Bates , Max Gurtovoy , Dan Williams , Keith Busch , Jason Gunthorpe , linux-pci@vger.kernel.org, linux-scsi@vger.kernel.org, linux-nvme@lists.infradead.org, linux-rdma@vger.kernel.org, linux-nvdimm@lists.01.org, linux-kernel@vger.kernel.org, Alex Williamson , Bjorn Helgaas List-Id: linux-rdma@vger.kernel.org On 02/04/17 03:03 PM, Sinan Kaya wrote: > Push the decision all the way to the user. Let them decide whether they > want this feature to work on a root port connected port or under the > switch. Yes, I prefer this too. If other folks agree with that I'd be very happy to go back to user chooses. I think Sagi was the most vocal proponent for kernel chooses at LSF so hopefully he will read this thread and offer some opinion. > I thought the issue was feature didn't work at all with some root ports > or there was some kind of memory corruption issue that you were trying to > avoid with the existing systems. I *think* there are some much older root ports where P2P TLPs don't even get through. But it doesn't really change the situation: in the nvmet case, the user would enable p2pmem and then be unable to connect and thus choose to disable it going forward. Not a big difference from the user seeing bad performance and not choosing to enable it. > I think you should get rid of all pci searching business in your code. Yes, my original proposal was when you configure the nvme target you chose the specific p2pmem device to use. That code had no tie ins to PCI code and could, in theory, work generically with any device and bus. Logan From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751912AbdDCE02 (ORCPT ); Mon, 3 Apr 2017 00:26:28 -0400 Received: from ale.deltatee.com ([207.54.116.67]:54427 "EHLO ale.deltatee.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751802AbdDCE0Y (ORCPT ); Mon, 3 Apr 2017 00:26:24 -0400 To: Sinan Kaya References: <1490911959-5146-1-git-send-email-logang@deltatee.com> <1490911959-5146-2-git-send-email-logang@deltatee.com> <7158f2e8-2016-f398-e77f-0fcbe6cb41dd@deltatee.com> <0280fbb4-ba9e-ac64-6bb3-b72590a54e57@deltatee.com> <0ae27bca-21be-b89c-aba4-6cc9766ebd7b@codeaurora.org> <63d73995-4dfa-6270-ca13-972e0628011b@deltatee.com> Cc: Christoph Hellwig , Sagi Grimberg , "James E.J. Bottomley" , "Martin K. Petersen" , Jens Axboe , Steve Wise , Stephen Bates , Max Gurtovoy , Dan Williams , Keith Busch , Jason Gunthorpe , linux-pci@vger.kernel.org, linux-scsi@vger.kernel.org, linux-nvme@lists.infradead.org, linux-rdma@vger.kernel.org, linux-nvdimm@ml01.01.org, linux-kernel@vger.kernel.org, Alex Williamson , Bjorn Helgaas From: Logan Gunthorpe Message-ID: Date: Sun, 2 Apr 2017 22:26:08 -0600 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Icedove/45.6.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-SA-Exim-Connect-IP: 50.66.97.235 X-SA-Exim-Rcpt-To: bhelgaas@google.com, alex.williamson@redhat.com, linux-kernel@vger.kernel.org, linux-nvdimm@lists.01.org, linux-rdma@vger.kernel.org, linux-nvme@lists.infradead.org, linux-scsi@vger.kernel.org, linux-pci@vger.kernel.org, jgunthorpe@obsidianresearch.com, keith.busch@intel.com, dan.j.williams@intel.com, maxg@mellanox.com, sbates@raithlin.com, swise@opengridcomputing.com, axboe@kernel.dk, martin.petersen@oracle.com, jejb@linux.vnet.ibm.com, sagi@grimberg.me, hch@lst.de, okaya@codeaurora.org X-SA-Exim-Mail-From: logang@deltatee.com Subject: Re: [RFC 1/8] Introduce Peer-to-Peer memory (p2pmem) device X-SA-Exim-Version: 4.2.1 (built Mon, 26 Dec 2011 16:24:06 +0000) X-SA-Exim-Scanned: Yes (on ale.deltatee.com) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 02/04/17 03:03 PM, Sinan Kaya wrote: > Push the decision all the way to the user. Let them decide whether they > want this feature to work on a root port connected port or under the > switch. Yes, I prefer this too. If other folks agree with that I'd be very happy to go back to user chooses. I think Sagi was the most vocal proponent for kernel chooses at LSF so hopefully he will read this thread and offer some opinion. > I thought the issue was feature didn't work at all with some root ports > or there was some kind of memory corruption issue that you were trying to > avoid with the existing systems. I *think* there are some much older root ports where P2P TLPs don't even get through. But it doesn't really change the situation: in the nvmet case, the user would enable p2pmem and then be unable to connect and thus choose to disable it going forward. Not a big difference from the user seeing bad performance and not choosing to enable it. > I think you should get rid of all pci searching business in your code. Yes, my original proposal was when you configure the nvme target you chose the specific p2pmem device to use. That code had no tie ins to PCI code and could, in theory, work generically with any device and bus. Logan From mboxrd@z Thu Jan 1 00:00:00 1970 From: logang@deltatee.com (Logan Gunthorpe) Date: Sun, 2 Apr 2017 22:26:08 -0600 Subject: [RFC 1/8] Introduce Peer-to-Peer memory (p2pmem) device In-Reply-To: References: <1490911959-5146-1-git-send-email-logang@deltatee.com> <1490911959-5146-2-git-send-email-logang@deltatee.com> <7158f2e8-2016-f398-e77f-0fcbe6cb41dd@deltatee.com> <0280fbb4-ba9e-ac64-6bb3-b72590a54e57@deltatee.com> <0ae27bca-21be-b89c-aba4-6cc9766ebd7b@codeaurora.org> <63d73995-4dfa-6270-ca13-972e0628011b@deltatee.com> Message-ID: On 02/04/17 03:03 PM, Sinan Kaya wrote: > Push the decision all the way to the user. Let them decide whether they > want this feature to work on a root port connected port or under the > switch. Yes, I prefer this too. If other folks agree with that I'd be very happy to go back to user chooses. I think Sagi was the most vocal proponent for kernel chooses at LSF so hopefully he will read this thread and offer some opinion. > I thought the issue was feature didn't work at all with some root ports > or there was some kind of memory corruption issue that you were trying to > avoid with the existing systems. I *think* there are some much older root ports where P2P TLPs don't even get through. But it doesn't really change the situation: in the nvmet case, the user would enable p2pmem and then be unable to connect and thus choose to disable it going forward. Not a big difference from the user seeing bad performance and not choosing to enable it. > I think you should get rid of all pci searching business in your code. Yes, my original proposal was when you configure the nvme target you chose the specific p2pmem device to use. That code had no tie ins to PCI code and could, in theory, work generically with any device and bus. Logan