From: Kit Chow <kchow@gigaio.com>
To: "Jiang, Dave" <dave.jiang@intel.com>,
Logan Gunthorpe <logang@deltatee.com>,
Eric Pilmore <epilmore@gigaio.com>,
Bjorn Helgaas <helgaas@kernel.org>
Cc: "linux-pci@vger.kernel.org" <linux-pci@vger.kernel.org>,
David Woodhouse <dwmw2@infradead.org>,
Alex Williamson <alex.williamson@redhat.com>,
"iommu@lists.linux-foundation.org"
<iommu@lists.linux-foundation.org>
Subject: Re: IOAT DMA w/IOMMU
Date: Thu, 9 Aug 2018 15:48:27 -0700 [thread overview]
Message-ID: <b8af340c-d40b-cb8c-987f-b394f1d6ec3e@gigaio.com> (raw)
In-Reply-To: <112A412BB11A1242B37129D931BCE5348382CFB2@fmsmsx117.amr.corp.intel.com>
On 08/09/2018 03:40 PM, Jiang, Dave wrote:
>> -----Original Message-----
>> From: linux-pci-owner@vger.kernel.org [mailto:linux-pci-owner@vger.kernel.org] On Behalf Of Kit Chow
>> Sent: Thursday, August 9, 2018 2:48 PM
>> To: Logan Gunthorpe <logang@deltatee.com>; Eric Pilmore <epilmore@gigaio.com>; Bjorn Helgaas <helgaas@kernel.org>
>> Cc: linux-pci@vger.kernel.org; David Woodhouse <dwmw2@infradead.org>; Alex Williamson <alex.williamson@redhat.com>;
>> iommu@lists.linux-foundation.org
>> Subject: Re: IOAT DMA w/IOMMU
>>
>>
>>
>> On 08/09/2018 02:11 PM, Logan Gunthorpe wrote:
>>> On 09/08/18 02:57 PM, Kit Chow wrote:
>>>> On 08/09/2018 01:11 PM, Logan Gunthorpe wrote:
>>>>> On 09/08/18 01:47 PM, Kit Chow wrote:
>>>>>>> I haven't tested this scenario but my guess would be that IOAT would
>>>>>>> indeed go through the IOMMU and the PCI BAR address would need to be
>>>>>>> properly mapped into the IOAT's IOVA. The fact that you see DMAR errors
>>>>>>> is probably a good indication that this is the case. I really don't know
>>>>>>> why you'd want to DMA something without mapping it.
>>>>>> I have experimented with changing ntb_async_tx_submit to dma_map the PCI
>>>>>> BAR
>>>>>> address. With this, I get a different DMAR error:
>>>>> What code did you use to do this?
>>>> If you mean version of linux, it is 4.15.7. Or specific dma_map call, I
>>>> believe it was dma_map_single.
>>> I mean the complete code you use to do the mapping so we can see if it's
>>> correct. dma_map_single() seems like an odd choice, I expected to see
>>> dma_map_resource().
>> Thanks for the suggestion! Will try out dma_map_resource and report back.
>>
>> Kit
> Kit, I was able to try this on the Skylake Xeon platform with Intel NTB and ioatdma and I did not encounter any issues. Unfortunately I do not have my pre-Skylake platform to test anymore to verify.
We are expecting to get skylakes within a couple of weeks and I'll have
a chance to give it a go on there.
Based on Logan's comments, I am very hopeful that the dma_map_resource
will make things work on the older platforms...
Thanks
Kit
>>>>>> DMAR: [DMA Write] Request device [07:00.4] fault addr ffd00000
>>>>>> [fault reason 12] non-zero reserved fields in PTE
>>>>> Also, what device corresponds to 07:00.4 on your system?
>>>> I believe 07.00.4 was the PLX dma device. I get the same error with ioat.
>>> Using the mapping with the PLX dma device likely converts it from a pure
>>> P2P request to one where the TLPs pass through the IOMMU. So the fact
>>> that you get the same error with both means IOAT almost certainly goes
>>> through the IOMMU and there's something wrong with the mapping setup.
>>>
>>> Logan
next prev parent reply other threads:[~2018-08-09 22:48 UTC|newest]
Thread overview: 48+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-08-09 18:14 IOAT DMA w/IOMMU Eric Pilmore
2018-08-09 18:43 ` Bjorn Helgaas
2018-08-09 18:51 ` Eric Pilmore
2018-08-09 19:35 ` Logan Gunthorpe
2018-08-09 19:47 ` Kit Chow
2018-08-09 20:11 ` Logan Gunthorpe
2018-08-09 20:57 ` Kit Chow
2018-08-09 21:11 ` Logan Gunthorpe
2018-08-09 21:47 ` Kit Chow
2018-08-09 22:40 ` Jiang, Dave
2018-08-09 22:48 ` Kit Chow [this message]
2018-08-09 22:50 ` Logan Gunthorpe
2018-08-09 23:00 ` Kit Chow
2018-08-10 16:02 ` Kit Chow
2018-08-10 16:23 ` Kit Chow
2018-08-10 16:24 ` Logan Gunthorpe
2018-08-10 16:24 ` Logan Gunthorpe
2018-08-10 16:31 ` Dave Jiang
2018-08-10 16:33 ` Logan Gunthorpe
2018-08-10 17:01 ` Dave Jiang
2018-08-10 17:15 ` Logan Gunthorpe
2018-08-10 17:46 ` Dave Jiang
2018-08-11 0:53 ` Kit Chow
2018-08-11 2:10 ` Logan Gunthorpe
2018-08-13 14:23 ` Kit Chow
2018-08-13 14:59 ` Robin Murphy
2018-08-13 15:21 ` Kit Chow
2018-08-13 23:30 ` Kit Chow
2018-08-13 23:39 ` Logan Gunthorpe
2018-08-13 23:48 ` Kit Chow
2018-08-13 23:50 ` Logan Gunthorpe
2018-08-14 13:47 ` Kit Chow
2018-08-14 14:03 ` Robin Murphy
2018-08-13 23:36 ` Kit Chow
2018-08-09 21:31 ` Eric Pilmore
2018-08-09 21:36 ` Logan Gunthorpe
2018-08-16 17:16 ` Kit Chow
2018-08-16 17:21 ` Logan Gunthorpe
2018-08-16 18:53 ` Kit Chow
2018-08-16 18:56 ` Logan Gunthorpe
2018-08-21 23:18 ` Eric Pilmore
2018-08-21 23:20 ` Logan Gunthorpe
2018-08-21 23:28 ` Eric Pilmore
2018-08-21 23:35 ` Logan Gunthorpe
2018-08-21 23:45 ` Eric Pilmore
2018-08-21 23:53 ` Logan Gunthorpe
2018-08-21 23:59 ` Eric Pilmore
2018-08-21 23:30 ` Eric Pilmore
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=b8af340c-d40b-cb8c-987f-b394f1d6ec3e@gigaio.com \
--to=kchow@gigaio.com \
--cc=alex.williamson@redhat.com \
--cc=dave.jiang@intel.com \
--cc=dwmw2@infradead.org \
--cc=epilmore@gigaio.com \
--cc=helgaas@kernel.org \
--cc=iommu@lists.linux-foundation.org \
--cc=linux-pci@vger.kernel.org \
--cc=logang@deltatee.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).