From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ml01.01.org (Postfix) with ESMTPS id 41D7E20961089 for ; Tue, 8 May 2018 13:43:43 -0700 (PDT) Date: Tue, 8 May 2018 14:43:41 -0600 From: Alex Williamson Subject: Re: [PATCH v4 04/14] PCI/P2PDMA: Clear ACS P2P flags for all devices behind switches Message-ID: <20180508144341.0441b676@w520.home> In-Reply-To: References: <20180423233046.21476-1-logang@deltatee.com> <20180423233046.21476-5-logang@deltatee.com> <20180507231306.GG161390@bhelgaas-glaptop.roam.corp.google.com> <0b4183ef-e720-204b-9e85-b9eaf7a4136a@deltatee.com> <3584a6ac-95c7-5d23-1859-aee30605776e@deltatee.com> <20180508133407.57a46902@w520.home> <5fc9b1c1-9208-06cc-0ec5-1f54c2520494@deltatee.com> <20180508141331.7cd737cb@w520.home> MIME-Version: 1.0 List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: linux-nvdimm-bounces@lists.01.org Sender: "Linux-nvdimm" To: Logan Gunthorpe Cc: Jens Axboe , Keith Busch , linux-nvdimm@lists.01.org, linux-rdma@vger.kernel.org, linux-pci@vger.kernel.org, linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org, linux-block@vger.kernel.org, =?UTF-8?B?SsOpcsO0bWU=?= Glisse , Jason Gunthorpe , Bjorn Helgaas , Benjamin Herrenschmidt , Bjorn Helgaas , Max Gurtovoy , Christoph Hellwig , Christian =?UTF-8?B?S8O2bmln?= List-ID: On Tue, 8 May 2018 14:19:05 -0600 Logan Gunthorpe wrote: > On 08/05/18 02:13 PM, Alex Williamson wrote: > > Well, I'm a bit confused, this patch series is specifically disabling > > ACS on switches, but per the spec downstream switch ports implementing > > ACS MUST implement direct translated P2P. So it seems the only > > potential gap here is the endpoint, which must support ATS or else > > there's nothing for direct translated P2P to do. The switch port plays > > no part in the actual translation of the request, ATS on the endpoint > > has already cached the translation and is now attempting to use it. > > For the switch port, this only becomes a routing decision, the request > > is already translated, therefore ACS RR and EC can be ignored to > > perform "normal" (direct) routing, as if ACS were not present. It would > > be a shame to go to all the trouble of creating this no-ACS mode to find > > out the target hardware supports ATS and should have simply used it, or > > we should have disabled the IOMMU altogether, which leaves ACS disabled. > > Ah, ok, I didn't think it was the endpoint that had to implement ATS. > But in that case, for our application, we need NVMe cards and RDMA NICs > to all have ATS support and I expect that is just as unlikely. At least > none of the endpoints on my system support it. Maybe only certain GPUs > have this support. Yes, GPUs seem to be leading the pack in implementing ATS. So now the dumb question, why not simply turn off the IOMMU and thus ACS? The argument of using the IOMMU for security is rather diminished if we're specifically enabling devices to poke one another directly and clearly this isn't favorable for device assignment either. Are there target systems where this is not a simple kernel commandline option? Thanks, Alex _______________________________________________ Linux-nvdimm mailing list Linux-nvdimm@lists.01.org https://lists.01.org/mailman/listinfo/linux-nvdimm