Re: IOAT DMA w/IOMMU

[Kit Chow <kchow@gigaio.com>]

On 08/09/2018 02:11 PM, Logan Gunthorpe wrote:
>
> On 09/08/18 02:57 PM, Kit Chow wrote:
>>
>> On 08/09/2018 01:11 PM, Logan Gunthorpe wrote:
>>> On 09/08/18 01:47 PM, Kit Chow wrote:
>>>>> I haven't tested this scenario but my guess would be that IOAT would
>>>>> indeed go through the IOMMU and the PCI BAR address would need to be
>>>>> properly mapped into the IOAT's IOVA. The fact that you see DMAR errors
>>>>> is probably a good indication that this is the case. I really don't know
>>>>> why you'd want to DMA something without mapping it.
>>>> I have experimented with changing ntb_async_tx_submit to dma_map the PCI
>>>> BAR
>>>> address. With this, I get a different DMAR error:
>>> What code did you use to do this?
>> If you mean version of linux, it is 4.15.7.  Or specific dma_map call, I
>> believe it was dma_map_single.
> I mean the complete code you use to do the mapping so we can see if it's
> correct. dma_map_single() seems like an odd choice, I expected to see
> dma_map_resource().
Thanks for the suggestion!  Will try out dma_map_resource and report back.

Kit

>>>> DMAR: [DMA Write] Request device [07:00.4] fault addr ffd00000
>>>> [fault reason 12] non-zero reserved fields in PTE
>>> Also, what device corresponds to 07:00.4 on your system?
>> I believe 07.00.4 was the PLX dma device. I get the same error with ioat.
> Using the mapping with the PLX dma device likely converts it from a pure
> P2P request to one where the TLPs pass through the IOMMU. So the fact
> that you get the same error with both means IOAT almost certainly goes
> through the IOMMU and there's something wrong with the mapping setup.
>
> Logan