From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.0 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 36D53C282CE for ; Wed, 22 May 2019 20:41:25 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id D06F120821 for ; Wed, 22 May 2019 20:41:24 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729762AbfEVUlY (ORCPT ); Wed, 22 May 2019 16:41:24 -0400 Received: from ale.deltatee.com ([207.54.116.67]:52802 "EHLO ale.deltatee.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1729528AbfEVUlY (ORCPT ); Wed, 22 May 2019 16:41:24 -0400 Received: from s01061831bf6ec98c.cg.shawcable.net ([68.147.80.180] helo=[192.168.6.141]) by ale.deltatee.com with esmtpsa (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.89) (envelope-from ) id 1hTY33-0003ne-TI; Wed, 22 May 2019 14:41:18 -0600 To: "Koenig, Christian" Cc: "linux-pci@vger.kernel.org" , Bjorn Helgaas References: From: Logan Gunthorpe Message-ID: <97aa52fc-f062-acf1-0e0c-5a4d1d505777@deltatee.com> Date: Wed, 22 May 2019 14:41:16 -0600 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.6.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-SA-Exim-Connect-IP: 68.147.80.180 X-SA-Exim-Rcpt-To: bhelgaas@google.com, linux-pci@vger.kernel.org, Christian.Koenig@amd.com X-SA-Exim-Mail-From: logang@deltatee.com Subject: Re: [PATCH] PCI/P2PDMA: Root complex whitelist should not apply when an IOMMU is present X-SA-Exim-Version: 4.2.1 (built Tue, 02 Aug 2016 21:08:31 +0000) X-SA-Exim-Scanned: Yes (on ale.deltatee.com) Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org On 2019-05-22 2:30 p.m., Koenig, Christian wrote: > Hi Logan, > > Am 22.05.2019 22:12 schrieb Logan Gunthorpe : > > [CAUTION: External Email] > > Presently, there is no path to DMA map P2PDMA memory, so if a TLP > targeting this memory hits the root complex and an IOMMU is present, > the IOMMU will reject the transaction, even if the RC would support > P2PDMA. > > So until the kernel knows to map these DMA addresses in the IOMMU, > we should not enable the whitelist when an IOMMU is present. > > > Well NAK, cause that is exactly what we are doing. Are you DMA-mapping the addresses outside the P2PDMA code? If so there's a huge mismatch with the existing users of P2PDMA (nvme-fabrics). If you're not dma-mapping then I can't see how it could work because the IOMMU should reject any requests to access those addresses. By adding the whitelist in this way you will break any user that attempts to use P2P in nvme-fabrics on whitelisted RCs with an IOMMU enabled. Currently, the users of P2PDMA use pci_p2pdma_map_sg() which only returns the PCI bus address. If P2PDMA transactions can now go through an IOMMU, then this is wrong and broken. We need to ensure that all users of P2PDMA map this memory in the same way. Which means, if the TLPs will go through an IOMMU they get dma-map'd and, if they don't, they use the PCI Bus address (as the current code does). Without the change proposed in this patch, I have to retract my review and NAK your patch until we can sort out the mapping issues. Logan