From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1162796AbeCAW4z (ORCPT ); Thu, 1 Mar 2018 17:56:55 -0500 Received: from ale.deltatee.com ([207.54.116.67]:39342 "EHLO ale.deltatee.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1162589AbeCAW4x (ORCPT ); Thu, 1 Mar 2018 17:56:53 -0500 To: Jason Gunthorpe Cc: Sagi Grimberg , linux-kernel@vger.kernel.org, linux-pci@vger.kernel.org, linux-nvme@lists.infradead.org, linux-rdma@vger.kernel.org, linux-nvdimm@lists.01.org, linux-block@vger.kernel.org, Stephen Bates , Christoph Hellwig , Jens Axboe , Keith Busch , Bjorn Helgaas , Max Gurtovoy , Dan Williams , =?UTF-8?B?SsOpcsO0bWUgR2xpc3Nl?= , Benjamin Herrenschmidt , Alex Williamson , Steve Wise References: <20180228234006.21093-1-logang@deltatee.com> <20180228234006.21093-11-logang@deltatee.com> <749e3752-4349-0bdf-5243-3d510c2b26db@grimberg.me> <40d69074-31a8-d06a-ade9-90de7712c553@deltatee.com> <5649098f-b775-815b-8b9a-f34628873ff4@grimberg.me> <20180301184249.GI19007@ziepe.ca> <20180301224540.GL19007@ziepe.ca> From: Logan Gunthorpe Message-ID: Date: Thu, 1 Mar 2018 15:56:38 -0700 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.6.0 MIME-Version: 1.0 In-Reply-To: <20180301224540.GL19007@ziepe.ca> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-SA-Exim-Connect-IP: 172.16.1.162 X-SA-Exim-Rcpt-To: swise@opengridcomputing.com, alex.williamson@redhat.com, benh@kernel.crashing.org, jglisse@redhat.com, dan.j.williams@intel.com, maxg@mellanox.com, bhelgaas@google.com, keith.busch@intel.com, axboe@kernel.dk, hch@lst.de, sbates@raithlin.com, linux-block@vger.kernel.org, linux-nvdimm@lists.01.org, linux-rdma@vger.kernel.org, linux-nvme@lists.infradead.org, linux-pci@vger.kernel.org, linux-kernel@vger.kernel.org, sagi@grimberg.me, jgg@ziepe.ca X-SA-Exim-Mail-From: logang@deltatee.com Subject: Re: [PATCH v2 10/10] nvmet: Optionally use PCI P2P memory X-SA-Exim-Version: 4.2.1 (built Tue, 02 Aug 2016 21:08:31 +0000) X-SA-Exim-Scanned: Yes (on ale.deltatee.com) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 01/03/18 03:45 PM, Jason Gunthorpe wrote: > I can appreciate you might have some special use case for that, but it > absolutely should require special configuration and not just magically > happen. Well if driver doesn't want someone doing p2p transfers with the memory it shouldn't publish it to be used for exactly that purpose. > You bring up IB adaptor memory - if we put that into the allocator > then what is to stop the NMVe driver just using it instead of the CMB > buffer? That would be totally wrong in almost all cases. If you mean for SQEs in the NVMe driver, look at the code, it specifically allocates it from it's own device. If you mean the nvmet-rdma then it's using that memory exactly as it was meant to. Again, if the IB driver doesn't want someone to use that memory for P2P transfers it shouldn't publish it as such. > Seems like a very subtle and hard to debug performance trap to leave > for the users, and pretty much the only reason to use P2P is > performance... So why have such a dangerous interface? It's not at all dangerous, the code specifically only uses P2P memory that's local enough. And the majority of the code is there to make sure it will all work in all cases. Honestly, though, I'd love to go back to the case where the user selects which p2pmem device to use, but that was very unpopular last year. It would simplify a bunch of things though. Also, no, the reason to use P2P is not performance. Only if you have very specific hardware can you get a performance bump and it isn't all that significant. The reason to use P2P is so you can design performant systems with small CPUs, less or slower DRAM, and low lane counts to the CPU, etc. Logan