From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751481AbeCNQXz (ORCPT ); Wed, 14 Mar 2018 12:23:55 -0400 Received: from ale.deltatee.com ([207.54.116.67]:40802 "EHLO ale.deltatee.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750862AbeCNQXx (ORCPT ); Wed, 14 Mar 2018 12:23:53 -0400 To: David Laight , Sinan Kaya , "linux-kernel@vger.kernel.org" , "linux-pci@vger.kernel.org" , "linux-nvme@lists.infradead.org" , "linux-rdma@vger.kernel.org" , "linux-nvdimm@lists.01.org" , "linux-block@vger.kernel.org" Cc: Stephen Bates , Christoph Hellwig , Jens Axboe , Keith Busch , Sagi Grimberg , Bjorn Helgaas , Jason Gunthorpe , Max Gurtovoy , Dan Williams , =?UTF-8?B?SsOpcsO0bWUgR2xpc3Nl?= , Benjamin Herrenschmidt , Alex Williamson References: <20180312193525.2855-1-logang@deltatee.com> <20180312193525.2855-2-logang@deltatee.com> <59fd2f5d-177f-334a-a9c4-0f8a6ec7c303@codeaurora.org> <24d8e5c2-065d-8bde-3f5d-7f158be9c578@deltatee.com> <52cbbbc4-c488-f83f-8d02-14d455b4efd7@codeaurora.org> <3e738f95-d73c-4182-2fa1-8664aafb1ab7@deltatee.com> <703aa92c-0c1c-4852-5887-6f6e6ccde0fb@codeaurora.org> <3ea80992-a0fc-08f2-d93d-ae0ec4e3f4ce@codeaurora.org> <4eb6850c-df1b-fd44-3ee0-d43a50270b53@deltatee.com> <757fca36-dee4-e070-669e-f2788bd78e41@codeaurora.org> <4f761f55-4e9a-dccb-d12f-c59d2cd689db@deltatee.com> <016dc910-f96a-8a60-4bda-fa24eea98ea5@codeaurora.org> <2b152932-2f44-408b-e3ed-b4608d95f82e@deltatee.com> <156c24fb-6e27-28f6-0b36-7fd83311ce37@codeaurora.org> <932bbf48-9d86-97ec-17bb-052099aff99e@deltatee.com> <6e36d43b0493455a8547bc50c8ec77ba@AcuMS.aculab.com> From: Logan Gunthorpe Message-ID: Date: Wed, 14 Mar 2018 10:23:27 -0600 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.6.0 MIME-Version: 1.0 In-Reply-To: <6e36d43b0493455a8547bc50c8ec77ba@AcuMS.aculab.com> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-SA-Exim-Connect-IP: 172.16.1.162 X-SA-Exim-Rcpt-To: alex.williamson@redhat.com, benh@kernel.crashing.org, jglisse@redhat.com, dan.j.williams@intel.com, maxg@mellanox.com, jgg@mellanox.com, bhelgaas@google.com, sagi@grimberg.me, keith.busch@intel.com, axboe@kernel.dk, hch@lst.de, sbates@raithlin.com, linux-block@vger.kernel.org, linux-nvdimm@lists.01.org, linux-rdma@vger.kernel.org, linux-nvme@lists.infradead.org, linux-pci@vger.kernel.org, linux-kernel@vger.kernel.org, okaya@codeaurora.org, David.Laight@ACULAB.COM X-SA-Exim-Mail-From: logang@deltatee.com Subject: Re: [PATCH v3 01/11] PCI/P2PDMA: Support peer-to-peer memory X-SA-Exim-Version: 4.2.1 (built Tue, 02 Aug 2016 21:08:31 +0000) X-SA-Exim-Scanned: Yes (on ale.deltatee.com) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 14/03/18 06:16 AM, David Laight wrote: > That surprises me (unless I missed something last time I read the spec). > While P2P writes are relatively easy to handle, reads and any other TLP that > require acks are a completely different proposition. > There are no additional fields that can be set in the read TLP and will be > reflected back in the ack(s) than can be used to route the acks back to the > correct initiator. > > I'm pretty sure that to support P2P reads a switch would have to save > the received read TLP and (possibly later on) issue read TLP of its own > for the required data. > I'm not even sure it is easy to interleave the P2P reads with those > coming from the root. > That requires a potentially infinite queue of pending requests. This is wrong. A completion is a TLP just like any other and makes use of the Destination ID field in the header to route it back to the original requester. > Some x86 root ports support P2P writes (maybe with a bios option). > It would be a shame not to be able to do P2P writes on such systems > even though P2P reads won't work. Yes, and this has been discussed many times. It won't be changing in the near term. Logan