From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Return-Path: Subject: Re: IOAT DMA w/IOMMU To: Logan Gunthorpe , Eric Pilmore , Bjorn Helgaas Cc: linux-pci@vger.kernel.org, David Woodhouse , Alex Williamson , iommu@lists.linux-foundation.org References: <20180809184331.GB113140@bhelgaas-glaptop.roam.corp.google.com> <95e53520-b05f-4be4-e694-463d717453a8@gigaio.com> From: Kit Chow Message-ID: <3f503940-db6a-ec75-ed15-1db99d5d48ef@gigaio.com> Date: Thu, 9 Aug 2018 14:47:36 -0700 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed List-ID: On 08/09/2018 02:11 PM, Logan Gunthorpe wrote: > > On 09/08/18 02:57 PM, Kit Chow wrote: >> >> On 08/09/2018 01:11 PM, Logan Gunthorpe wrote: >>> On 09/08/18 01:47 PM, Kit Chow wrote: >>>>> I haven't tested this scenario but my guess would be that IOAT would >>>>> indeed go through the IOMMU and the PCI BAR address would need to be >>>>> properly mapped into the IOAT's IOVA. The fact that you see DMAR errors >>>>> is probably a good indication that this is the case. I really don't know >>>>> why you'd want to DMA something without mapping it. >>>> I have experimented with changing ntb_async_tx_submit to dma_map the PCI >>>> BAR >>>> address. With this, I get a different DMAR error: >>> What code did you use to do this? >> If you mean version of linux, it is 4.15.7.  Or specific dma_map call, I >> believe it was dma_map_single. > I mean the complete code you use to do the mapping so we can see if it's > correct. dma_map_single() seems like an odd choice, I expected to see > dma_map_resource(). Thanks for the suggestion!  Will try out dma_map_resource and report back. Kit >>>> DMAR: [DMA Write] Request device [07:00.4] fault addr ffd00000 >>>> [fault reason 12] non-zero reserved fields in PTE >>> Also, what device corresponds to 07:00.4 on your system? >> I believe 07.00.4 was the PLX dma device. I get the same error with ioat. > Using the mapping with the PLX dma device likely converts it from a pure > P2P request to one where the TLPs pass through the IOMMU. So the fact > that you get the same error with both means IOAT almost certainly goes > through the IOMMU and there's something wrong with the mapping setup. > > Logan