Re: IOAT DMA w/IOMMU

[Kit Chow <kchow@gigaio.com>]

On 08/09/2018 02:36 PM, Logan Gunthorpe wrote:
>
> On 09/08/18 03:31 PM, Eric Pilmore wrote:
>> On Thu, Aug 9, 2018 at 12:35 PM, Logan Gunthorpe <logang@deltatee.com> wrote:
>>> Hey,
>>>
>>> On 09/08/18 12:51 PM, Eric Pilmore wrote:
>>>>>> Was wondering if anybody here has used IOAT DMA engines with an
>>>>>> IOMMU turned on (Xeon based system)? My specific question is really
>>>>>> whether it is possible to DMA (w/IOAT) to a PCI BAR address as the
>>>>>> destination without having to map that address to the IOVA space of
>>>>>> the DMA engine first (assuming the IOMMU is on)?
>>> I haven't tested this scenario but my guess would be that IOAT would
>>> indeed go through the IOMMU and the PCI BAR address would need to be
>>> properly mapped into the IOAT's IOVA. The fact that you see DMAR errors
>>> is probably a good indication that this is the case. I really don't know
>>> why you'd want to DMA something without mapping it.
>> The thought was to avoid paying the price of having to go through yet another
>> translation and also because it was not believed to be necessary anyway since
>> the DMA device could go straight to a PCI BAR address without the need for a
>> mapping.  We have been playing with two DMA engines, IOAT and PLX. The
>> PLX does not have any issues going straight to the PCI BAR address, but unlike
>> IOAT, PLX is sitting "in the PCI tree".
> Yes, you can do true P2P transactions with the PLX DMA engine. (Though
> you do have to be careful as technically you need to translate to a PCI
> bus address not the CPU physical address which are the same on x86; and
> you need to watch that ACS doesn't mess it up).
>
> It doesn't sound like it's possible to avoid the IOMMU when using the
> IOAT so you need the mapping. I would not expect the extra translation
> in the IOMMU to be noticeable, performance wise.
>
> Logan
Hi Logan,

Based on Dave Jiang's suggestion, I am looking into upstreaming the
ntb change to dma map the pci bar address in ntb_async_tx_submit() 
(along with
the intel iommu change to support dma_map_resource).

I only have access to intel hosts for testing (and possibly an AMD
host currently collecting dust) and am not sure how to go about getting
the proper test coverage for other architectures.

Any suggestions? Do you anticipate any issues with dma mapping the pci bar
address in ntb_async_tx_submit on non-x86 archs?  Do you or know of folks
who have ntb test setups with non-x86 hosts who might be able to help?

Thanks
Kit