From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752112AbeCWVuv (ORCPT ); Fri, 23 Mar 2018 17:50:51 -0400 Received: from mail.kernel.org ([198.145.29.99]:57486 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751847AbeCWVut (ORCPT ); Fri, 23 Mar 2018 17:50:49 -0400 DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 4F0D721770 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=helgaas@kernel.org Date: Fri, 23 Mar 2018 16:50:46 -0500 From: Bjorn Helgaas To: Stephen Bates Cc: Logan Gunthorpe , Sinan Kaya , "linux-kernel@vger.kernel.org" , "linux-pci@vger.kernel.org" , "linux-nvme@lists.infradead.org" , "linux-rdma@vger.kernel.org" , "linux-nvdimm@lists.01.org" , "linux-block@vger.kernel.org" , Christoph Hellwig , Jens Axboe , Keith Busch , Sagi Grimberg , Bjorn Helgaas , Jason Gunthorpe , Max Gurtovoy , Dan Williams , =?iso-8859-1?B?Suly9G1l?= Glisse , Benjamin Herrenschmidt , Alex Williamson Subject: Re: [PATCH v3 01/11] PCI/P2PDMA: Support peer-to-peer memory Message-ID: <20180323215046.GC210003@bhelgaas-glaptop.roam.corp.google.com> References: <3e738f95-d73c-4182-2fa1-8664aafb1ab7@deltatee.com> <703aa92c-0c1c-4852-5887-6f6e6ccde0fb@codeaurora.org> <3ea80992-a0fc-08f2-d93d-ae0ec4e3f4ce@codeaurora.org> <4eb6850c-df1b-fd44-3ee0-d43a50270b53@deltatee.com> <757fca36-dee4-e070-669e-f2788bd78e41@codeaurora.org> <4f761f55-4e9a-dccb-d12f-c59d2cd689db@deltatee.com> <20180313230850.GA45763@bhelgaas-glaptop.roam.corp.google.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.9.2 (2017-12-15) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Thu, Mar 22, 2018 at 10:57:32PM +0000, Stephen Bates wrote: > > I've seen the response that peers directly below a Root Port could not > > DMA to each other through the Root Port because of the "route to self" > > issue, and I'm not disputing that. > > Bjorn > > You asked me for a reference to RTS in the PCIe specification. As > luck would have it I ended up in an Irish bar with Peter Onufryk > this week at OCP Summit. We discussed the topic. It is not > explicitly referred to as "Route to Self" and it's certainly not > explicit (or obvious) but r6.2.8.1 of the PCIe 4.0 specification > discusses error conditions for virtual PCI bridges. One of these > conditions (given in the very first bullet in that section) applies > to a request that is destined for the same port it came in on. When > this occurs the request must be terminated as a UR. Thanks for that reference! I suspect figure 10-3 in sec 10.1.1 might also be relevant, although it's buried in the ATS section. It shows internal routing between functions of a multifunction device. That suggests that the functions should be able to DMA to each other without those transactions ever appearing on the link. Popping way up the stack, my original point was that I'm trying to remove restrictions on what devices can participate in peer-to-peer DMA. I think it's fairly clear that in conventional PCI, any devices in the same PCI hierarchy, i.e., below the same host-to-PCI bridge, should be able to DMA to each other. The routing behavior of PCIe is supposed to be compatible with conventional PCI, and I would argue that this effectively requires multi-function PCIe devices to have the internal routing required to avoid the route-to-self issue. Bjorn