From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Return-Path: Subject: Re: IOAT DMA w/IOMMU To: Logan Gunthorpe , Dave Jiang , Eric Pilmore , Bjorn Helgaas Cc: "linux-pci@vger.kernel.org" , David Woodhouse , Alex Williamson , "iommu@lists.linux-foundation.org" References: <3f503940-db6a-ec75-ed15-1db99d5d48ef@gigaio.com> <112A412BB11A1242B37129D931BCE5348382CFB2@fmsmsx117.amr.corp.intel.com> <1170c096-a749-6ee3-32d4-ebba1d12adff@gigaio.com> <929f3dfa-fbff-c582-3be5-c368f11feb7e@deltatee.com> <460cc47f-6a8e-67b0-01a0-02f0c51df258@intel.com> <0f653f8f-c7ae-831d-e238-2742590d9ea1@deltatee.com> <6525fa4d-7d9b-0136-7206-f8351230af46@intel.com> <499934e7-3734-1aee-37dd-b42a5d2a2608@intel.com> <9d135f61-20c2-e966-4d7d-3e154384ea08@gigaio.com> <0a9fa9c3-dbe4-6451-9c1a-06b662702f61@deltatee.com> From: Kit Chow Message-ID: Date: Mon, 13 Aug 2018 07:23:22 -0700 MIME-Version: 1.0 In-Reply-To: <0a9fa9c3-dbe4-6451-9c1a-06b662702f61@deltatee.com> Content-Type: text/plain; charset=utf-8; format=flowed List-ID: On 08/10/2018 07:10 PM, Logan Gunthorpe wrote: > > On 10/08/18 06:53 PM, Kit Chow wrote: >> I was able to finally succeed in doing the dma transfers over ioat only >> when prot has DMA_PTE_WRITE set by setting the direction to either >> DMA_FROM_DEVICE or DMA_BIDIRECTIONAL. Any ideas if the prot settings >> need to be changed? Are there any bad side effects if I used >> DMA_BIDIRECTIONAL? > Good to hear it. Without digging into the direction much all I can say > is that it can sometimes be very confusing what the direction is. Adding > another PCI device just adds to the confusion. Yep, confusing :). ======================= ============================================= DMA_NONE no direction (used for debugging) DMA_TO_DEVICE data is going from the memory to the device DMA_FROM_DEVICE data is coming from the device to the memory DMA_BIDIRECTIONAL direction isn't known ======================= ============================================= > I believe, the direction should be from the IOAT's point of view. So if > the IOAT is writing to the BAR you'd set DMA_FROM_DEVICE (ie. data is > coming from the IOAT) and if it's reading you'd set DMA_TO_DEVICE (ie. > data is going to the IOAT). It would certainly seem like DMA_TO_DEVICE would be the proper choice; IOAT is the plumbing to move host data (memory) to the bar address (device). Will go with what works and set DMA_FROM_DEVICE. In ntb_async_tx_submit, does the direction used for the dma_map routines for the src and dest addresses need to be consistent? And does the direction setting for the dmaengine_unmap_data have to be consistent with the direction used in dma_map_*? BTW, dmaengine_unmap routine only calls dma_unmap_page. Should it keep track of the dma_map routine used and call the corresponding dma_unmap routine?  In the case of the intel iommu, it doesn't matter. Thanks Kit > > Using DMA_BIDIRECTIONAL just forgoes any hardware security / protection > that the buffer would have in terms of direction. Generally it's good > practice to use the strictest direction you can. > >> Given that using the pci bar address as is without getting an iommu >> address results in the same "PTE Write access" error, I wonder if there >> is some internal 'prot' associated with the non-translated pci bar >> address that just needs to be tweaked to include DMA_PTE_WRITE??? > No, I don't think so. The 'prot' will be a property of the IOMMU. Not > having an entry is probably just the same (from the perspective of the > error you see) as only having an entry for reading. > > Logan