From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1755562AbeEHUNg (ORCPT ); Tue, 8 May 2018 16:13:36 -0400 Received: from mx1.redhat.com ([209.132.183.28]:53818 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751579AbeEHUNe (ORCPT ); Tue, 8 May 2018 16:13:34 -0400 Date: Tue, 8 May 2018 14:13:31 -0600 From: Alex Williamson To: Logan Gunthorpe Cc: Christian =?UTF-8?B?S8O2bmln?= , Bjorn Helgaas , linux-kernel@vger.kernel.org, linux-pci@vger.kernel.org, linux-nvme@lists.infradead.org, linux-rdma@vger.kernel.org, linux-nvdimm@lists.01.org, linux-block@vger.kernel.org, Stephen Bates , Christoph Hellwig , Jens Axboe , Keith Busch , Sagi Grimberg , Bjorn Helgaas , Jason Gunthorpe , Max Gurtovoy , Dan Williams , =?UTF-8?B?SsOpcsO0bWU=?= Glisse , Benjamin Herrenschmidt Subject: Re: [PATCH v4 04/14] PCI/P2PDMA: Clear ACS P2P flags for all devices behind switches Message-ID: <20180508141331.7cd737cb@w520.home> In-Reply-To: <5fc9b1c1-9208-06cc-0ec5-1f54c2520494@deltatee.com> References: <20180423233046.21476-1-logang@deltatee.com> <20180423233046.21476-5-logang@deltatee.com> <20180507231306.GG161390@bhelgaas-glaptop.roam.corp.google.com> <0b4183ef-e720-204b-9e85-b9eaf7a4136a@deltatee.com> <3584a6ac-95c7-5d23-1859-aee30605776e@deltatee.com> <20180508133407.57a46902@w520.home> <5fc9b1c1-9208-06cc-0ec5-1f54c2520494@deltatee.com> MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Tue, 8 May 2018 13:45:50 -0600 Logan Gunthorpe wrote: > On 08/05/18 01:34 PM, Alex Williamson wrote: > > They are not so unrelated, see the ACS Direct Translated P2P > > capability, which in fact must be implemented by switch downstream > > ports implementing ACS and works specifically with ATS. This appears to > > be the way the PCI SIG would intend for P2P to occur within an IOMMU > > managed topology, routing pre-translated DMA directly between peer > > devices while requiring non-translated requests to bounce through the > > IOMMU. Really, what's the value of having an I/O virtual address space > > provided by an IOMMU if we're going to allow physical DMA between > > downstream devices, couldn't we just turn off the IOMMU altogether? Of > > course ATS is not without holes itself, basically that we trust the > > endpoint's implementation of ATS implicitly. Thanks, > > I agree that this is what the SIG intends, but I don't think hardware > fully supports this methodology yet. The Direct Translated capability > just requires switches to forward packets that have the AT request type > set. It does not require them to do the translation or to support ATS > such that P2P requests can be translated by the IOMMU. I expect this is > so that an downstream device can implement ATS and not get messed up by > an upstream switch that doesn't support it. Well, I'm a bit confused, this patch series is specifically disabling ACS on switches, but per the spec downstream switch ports implementing ACS MUST implement direct translated P2P. So it seems the only potential gap here is the endpoint, which must support ATS or else there's nothing for direct translated P2P to do. The switch port plays no part in the actual translation of the request, ATS on the endpoint has already cached the translation and is now attempting to use it. For the switch port, this only becomes a routing decision, the request is already translated, therefore ACS RR and EC can be ignored to perform "normal" (direct) routing, as if ACS were not present. It would be a shame to go to all the trouble of creating this no-ACS mode to find out the target hardware supports ATS and should have simply used it, or we should have disabled the IOMMU altogether, which leaves ACS disabled. Thanks, Alex