From: Kit Chow <kchow@gigaio.com>
To: Robin Murphy <robin.murphy@arm.com>,
Logan Gunthorpe <logang@deltatee.com>,
Dave Jiang <dave.jiang@intel.com>,
Eric Pilmore <epilmore@gigaio.com>,
Bjorn Helgaas <helgaas@kernel.org>
Cc: "linux-pci@vger.kernel.org" <linux-pci@vger.kernel.org>,
Alex Williamson <alex.williamson@redhat.com>,
David Woodhouse <dwmw2@infradead.org>,
"iommu@lists.linux-foundation.org"
<iommu@lists.linux-foundation.org>
Subject: Re: IOAT DMA w/IOMMU
Date: Mon, 13 Aug 2018 16:36:11 -0700 [thread overview]
Message-ID: <b0658bb5-4cbe-3b8d-e360-02c5640d2615@gigaio.com> (raw)
In-Reply-To: <3772d88a-e9ad-465a-c91f-df8f9c6219be@gigaio.com>
Taking a step back, I was a little surprised that dma_map_single
successfully returned an
iommu address for the pci bar address passed into it during my initial
experiment...
Internally, dma_map_single calls virt_to_page() to translate the
"virtual address" into a
page and intel_map page then calls page_to_phys() to convert the page to
a dma_addr_t.
The virt_to_page and page_to_phys routines don't appear to do any
validation and just
uses arithmetic to do the conversions.
the pci bar address (0x383c70e51578) does fall into a valid VA range in
x86_64 so
it could conceivably be a valid VA. So I tried an virtual address
inside the VA hole
and and it too returned without any errors.
virt_to_page(0x800000000000) -> 0xffffede000000000
page_to_phys(0xffffede000000000) -> 0xf80000000000
In arch/x86/include/asm/page.h, there is the following comment in regards to
validating the virtual address.
/*
* virt_to_page(kaddr) returns a valid pointer if and only if
* virt_addr_valid(kaddr) returns true.
*/
#define virt_to_page(kaddr) pfn_to_page(__pa(kaddr) >> PAGE_SHIFT)
So it looks like the validation by virt_addr_valid was somehow dropped
from the
virt_to_page code path. Does anyone have any ideas what happended to it?
Kit
(Resending, earlier message tagged as containing html subpart)
On 08/13/2018 08:21 AM, Kit Chow wrote:
>
>
> On 08/13/2018 07:59 AM, Robin Murphy wrote:
>> On 13/08/18 15:23, Kit Chow wrote:
>>> On 08/10/2018 07:10 PM, Logan Gunthorpe wrote:
>>>>
>>>> On 10/08/18 06:53 PM, Kit Chow wrote:
>>>>> I was able to finally succeed in doing the dma transfers over ioat
>>>>> only
>>>>> when prot has DMA_PTE_WRITE set by setting the direction to either
>>>>> DMA_FROM_DEVICE or DMA_BIDIRECTIONAL. Any ideas if the prot settings
>>>>> need to be changed? Are there any bad side effects if I used
>>>>> DMA_BIDIRECTIONAL?
>>>> Good to hear it. Without digging into the direction much all I can say
>>>> is that it can sometimes be very confusing what the direction is.
>>>> Adding
>>>> another PCI device just adds to the confusion.
>>> Yep, confusing :).
>>>
>>> ======================= =============================================
>>> DMA_NONE no direction (used for debugging)
>>> DMA_TO_DEVICE data is going from the memory to the device
>>> DMA_FROM_DEVICE data is coming from the device to the memory
>>> DMA_BIDIRECTIONAL direction isn't known
>>> ======================= =============================================
>>>
>>>> I believe, the direction should be from the IOAT's point of view.
>>>> So if
>>>> the IOAT is writing to the BAR you'd set DMA_FROM_DEVICE (ie. data is
>>>> coming from the IOAT) and if it's reading you'd set DMA_TO_DEVICE (ie.
>>>> data is going to the IOAT).
>>> It would certainly seem like DMA_TO_DEVICE would be the proper
>>> choice; IOAT is the plumbing to move host data (memory) to the bar
>>> address (device).
>>
>> Except that the "device" in question is the IOAT itself (more
>> generally, it means the device represented by the first argument to
>> dma_map_*() - the one actually emitting the reads and writes). The
>> context of a DMA API call is the individual mapping in question, not
>> whatever overall operation it may be part of - your example already
>> involves two separate mappings: one "from" system memory "to" the DMA
>> engine, and one "from" the DMA engine "to" PCI BAR memory.
>
> OK, that makes sense. The middleman (aka DMA engine device) is the
> key in the to/from puzzle. Thanks!
>
>
>>
>> Note that the DMA API's dma_direction is also distinct from the
>> dmaengine API's dma_transfer_direction, and there's plenty of fun to
>> be had mapping between the two - see pl330.c or rcar-dmac.c for other
>> examples of dma_map_resource() for slave devices - no guarantees that
>> those implementations are entirely correct (especially the one I
>> did!), but in practice they do make the "DMA engine behind an IOMMU"
>> case work for UARTs and similar straightforward slaves.
>>
>>> Will go with what works and set DMA_FROM_DEVICE.
>>>
>>> In ntb_async_tx_submit, does the direction used for the dma_map
>>> routines for the src and dest addresses need to be consistent?
>>
>> In general, the mappings of source and destination addresses would
>> typically have opposite directions as above, unless they're both
>> bidirectional.
>>
>>> And does the direction setting for the dmaengine_unmap_data have to
>>> be consistent with the direction used in dma_map_*?
>>
>> Yes, the arguments to an unmap are expected to match whatever was
>> passed to the corresponding map call. CONFIG_DMA_API_DEBUG should
>> help catch any mishaps.
>>
>> Robin.
>>
>>> BTW, dmaengine_unmap routine only calls dma_unmap_page. Should it
>>> keep track of the dma_map routine used and call the corresponding
>>> dma_unmap routine? In the case of the intel iommu, it doesn't matter.
>>>
>>> Thanks
>>> Kit
>>>
>>>>
>>>> Using DMA_BIDIRECTIONAL just forgoes any hardware security /
>>>> protection
>>>> that the buffer would have in terms of direction. Generally it's good
>>>> practice to use the strictest direction you can.
>>>>
>>>>> Given that using the pci bar address as is without getting an iommu
>>>>> address results in the same "PTE Write access" error, I wonder if
>>>>> there
>>>>> is some internal 'prot' associated with the non-translated pci bar
>>>>> address that just needs to be tweaked to include DMA_PTE_WRITE???
>>>> No, I don't think so. The 'prot' will be a property of the IOMMU. Not
>>>> having an entry is probably just the same (from the perspective of the
>>>> error you see) as only having an entry for reading.
>>>>
>>>> Logan
>>>
>>> _______________________________________________
>>> iommu mailing list
>>> iommu@lists.linux-foundation.org
>>> https://lists.linuxfoundation.org/mailman/listinfo/iommu
>
next prev parent reply other threads:[~2018-08-13 23:36 UTC|newest]
Thread overview: 48+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-08-09 18:14 IOAT DMA w/IOMMU Eric Pilmore
2018-08-09 18:43 ` Bjorn Helgaas
2018-08-09 18:51 ` Eric Pilmore
2018-08-09 19:35 ` Logan Gunthorpe
2018-08-09 19:47 ` Kit Chow
2018-08-09 20:11 ` Logan Gunthorpe
2018-08-09 20:57 ` Kit Chow
2018-08-09 21:11 ` Logan Gunthorpe
2018-08-09 21:47 ` Kit Chow
2018-08-09 22:40 ` Jiang, Dave
2018-08-09 22:48 ` Kit Chow
2018-08-09 22:50 ` Logan Gunthorpe
2018-08-09 23:00 ` Kit Chow
2018-08-10 16:02 ` Kit Chow
2018-08-10 16:23 ` Kit Chow
2018-08-10 16:24 ` Logan Gunthorpe
2018-08-10 16:24 ` Logan Gunthorpe
2018-08-10 16:31 ` Dave Jiang
2018-08-10 16:33 ` Logan Gunthorpe
2018-08-10 17:01 ` Dave Jiang
2018-08-10 17:15 ` Logan Gunthorpe
2018-08-10 17:46 ` Dave Jiang
2018-08-11 0:53 ` Kit Chow
2018-08-11 2:10 ` Logan Gunthorpe
2018-08-13 14:23 ` Kit Chow
2018-08-13 14:59 ` Robin Murphy
2018-08-13 15:21 ` Kit Chow
2018-08-13 23:30 ` Kit Chow
2018-08-13 23:39 ` Logan Gunthorpe
2018-08-13 23:48 ` Kit Chow
2018-08-13 23:50 ` Logan Gunthorpe
2018-08-14 13:47 ` Kit Chow
2018-08-14 14:03 ` Robin Murphy
2018-08-13 23:36 ` Kit Chow [this message]
2018-08-09 21:31 ` Eric Pilmore
2018-08-09 21:36 ` Logan Gunthorpe
2018-08-16 17:16 ` Kit Chow
2018-08-16 17:21 ` Logan Gunthorpe
2018-08-16 18:53 ` Kit Chow
2018-08-16 18:56 ` Logan Gunthorpe
2018-08-21 23:18 ` Eric Pilmore
2018-08-21 23:20 ` Logan Gunthorpe
2018-08-21 23:28 ` Eric Pilmore
2018-08-21 23:35 ` Logan Gunthorpe
2018-08-21 23:45 ` Eric Pilmore
2018-08-21 23:53 ` Logan Gunthorpe
2018-08-21 23:59 ` Eric Pilmore
2018-08-21 23:30 ` Eric Pilmore
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=b0658bb5-4cbe-3b8d-e360-02c5640d2615@gigaio.com \
--to=kchow@gigaio.com \
--cc=alex.williamson@redhat.com \
--cc=dave.jiang@intel.com \
--cc=dwmw2@infradead.org \
--cc=epilmore@gigaio.com \
--cc=helgaas@kernel.org \
--cc=iommu@lists.linux-foundation.org \
--cc=linux-pci@vger.kernel.org \
--cc=logang@deltatee.com \
--cc=robin.murphy@arm.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).