From: Auger Eric <eric.auger@redhat.com>
To: Nate Watterson <nwatters@codeaurora.org>,
Robin Murphy <robin.murphy@arm.com>,
Joerg Roedel <joro@8bytes.org>,
iommu@lists.linux-foundation.org, linux-kernel@vger.kernel.org
Cc: shankerd@codeaurora.org
Subject: Re: [PATCH] iommu/dma: Setup iova_domain granule for IOMMU_DMA_MSI cookies
Date: Tue, 16 May 2017 22:20:56 +0200 [thread overview]
Message-ID: <131c7b30-ab20-6ba0-743b-f61095f878c0@redhat.com> (raw)
In-Reply-To: <2fa1c68c-2c9a-850c-d5b8-0e30cfd077b0@codeaurora.org>
Hi Nate,
On 16/05/2017 22:07, Nate Watterson wrote:
>
>
> On 5/16/2017 3:55 PM, Auger Eric wrote:
>> Hi,
>>
>> On 13/04/2017 21:38, Nate Watterson wrote:
>>> Hi Robin,
>>>
>>> On 4/13/2017 7:21 AM, Robin Murphy wrote:
>>>> Hi Nate,
>>>>
>>>> On 13/04/17 09:55, Nate Watterson wrote:
>>>>> Currently, the __iommu_dma_{map/free} functions call
>>>>> iova_{offset/align}
>>>>> making them unsuitable for use with iommu_domains having an
>>>>> IOMMU_DMA_MSI
>>>>> cookie since the cookie's iova_domain member, iovad, is uninitialized.
>>>>>
>>>>> Now that iommu_dma_get_msi_page() calls __iommu_dma_map() regardless
>>>>> of cookie type, failures are being seen when mapping MSI target
>>>>> addresses for devices attached to UNMANAGED domains. To work around
>>>>> this issue, the iova_domain granule for IOMMU_DMA_MSI cookies is
>>>>> initialized to the value returned by cookie_msi_granule().
>>>>
>>>> Oh bum. Thanks for the report.
>>>>
>>>> However, I really don't like bodging around it with deliberate
>>>> undefined
>>>> behaviour. Fixing things properly doesn't seem too hard:
>>>
>>> I was not especially please with my solution, but I wanted to avoid
>>> potentially missing any other spots in the code where granule was
>>> used uninitialized. The compile time check made me feel a little
>>> less dirty about innappropriately using the iova_domain with MSI
>>> cookies.
>>>
>>>>
>>>> ----->8-----
>>>> diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c
>>>> index 8348f366ddd1..62618e77bedc 100644
>>>> --- a/drivers/iommu/dma-iommu.c
>>>> +++ b/drivers/iommu/dma-iommu.c
>>>> @@ -396,13 +396,13 @@ static void iommu_dma_free_iova(struct
>>>> iommu_dma_cookie *cookie,
>>>> dma_addr_t iova, size_t size)
>>>> {
>>>> struct iova_domain *iovad = &cookie->iovad;
>>>> - unsigned long shift = iova_shift(iovad);
>>>>
>>>> /* The MSI case is only ever cleaning up its most recent
>>>> allocation */
>>>> if (cookie->type == IOMMU_DMA_MSI_COOKIE)
>>>> cookie->msi_iova -= size;
>>>> else
>>>> - free_iova_fast(iovad, iova >> shift, size >> shift);
>>>> + free_iova_fast(iovad, iova_pfn(iovad, iova),
>>>> + size >> iova_shift(iovad));
>>>> }
>>>>
>>>> static void __iommu_dma_unmap(struct iommu_domain *domain,
>>>> dma_addr_t
>>>> dma_addr,
>>>> @@ -617,11 +617,14 @@ static dma_addr_t __iommu_dma_map(struct device
>>>> *dev, phys_addr_t phys,
>>>> {
>>>> struct iommu_domain *domain = iommu_get_domain_for_dev(dev);
>>>> struct iommu_dma_cookie *cookie = domain->iova_cookie;
>>>> - struct iova_domain *iovad = &cookie->iovad;
>>>> - size_t iova_off = iova_offset(iovad, phys);
>>>> + size_t iova_off = 0;
>>>> dma_addr_t iova;
>>>>
>>>> - size = iova_align(iovad, size + iova_off);
>>>> + if (cookie->type == IOMMU_DMA_IOVA_COOKIE) {
>>>> + iova_off = iova_offset(&cookie->iovad, phys);
>>>> + size = iova_align(&cookie->iovad, size + iova_off);
>>>> + }
>>>> +
>>>> iova = iommu_dma_alloc_iova(domain, size, dma_get_mask(dev),
>>>> dev);
>>>> if (!iova)
>>>> return DMA_ERROR_CODE;
>>>> -----8<-----
>>>>
>>>> Untested, and you'll probably want to double-check it anyway given that
>>>> the original oversight was mine in the first place ;)
>>>
>>> This looks good to me. As Shanker has already mentioned, it does fix the
>>> faults we were previously seeing with direct device assignment. I also
>>> verified that there aren't any other obvious cases of a granule == 0
>>> being used in the dma_iommu code by adding BUG_ON(!iovad->granule) to
>>> iova_{mask/align/offset/...} and running a variety of tests without
>>> issue.
>>>
>>> Are you going to post the patch?
>>
>> I also noticed PCIe passthrough/ARM is broken for me with 4.12-r1. I
>> tested Robin's patch as well, on Cavium ThunderX, and this fixes the
>> faults I have seen.
>>
>> Has anyone sent a formal patch?
>
> iommu/dma: Don't touch invalid iova_domain members
thanks!
Best Regards
Eric
>
>>
>> Thanks
>>
>> Eric
>>
>>>
>>>>
>>>> Robin.
>>>>
>>>>> Fixes: a44e6657585b ("iommu/dma: Clean up MSI IOVA allocation")
>>>>> Signed-off-by: Nate Watterson <nwatters@codeaurora.org>
>>>>> ---
>>>>> drivers/iommu/dma-iommu.c | 10 ++++++++++
>>>>> 1 file changed, 10 insertions(+)
>>>>>
>>>>> diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c
>>>>> index 8348f366..d7b0816 100644
>>>>> --- a/drivers/iommu/dma-iommu.c
>>>>> +++ b/drivers/iommu/dma-iommu.c
>>>>> @@ -127,6 +127,16 @@ int iommu_get_msi_cookie(struct iommu_domain
>>>>> *domain, dma_addr_t base)
>>>>> cookie->msi_iova = base;
>>>>> domain->iova_cookie = cookie;
>>>>> +
>>>>> + /*
>>>>> + * Setup granule for compatibility with __iommu_dma_{alloc/free}
>>>>> and
>>>>> + * add a compile time check to ensure that writing granule won't
>>>>> + * clobber msi_iova.
>>>>> + */
>>>>> + cookie->iovad.granule = cookie_msi_granule(cookie);
>>>>> + BUILD_BUG_ON(offsetof(struct iova_domain, granule) <
>>>>> + sizeof(cookie->msi_iova));
>>>>> +
>>>>> return 0;
>>>>> }
>>>>> EXPORT_SYMBOL(iommu_get_msi_cookie);
>>>>>
>>>>
>
prev parent reply other threads:[~2017-05-16 20:21 UTC|newest]
Thread overview: 8+ messages / expand[flat|nested] mbox.gz Atom feed top
2017-04-13 8:55 [PATCH] iommu/dma: Setup iova_domain granule for IOMMU_DMA_MSI cookies Nate Watterson
2017-04-13 11:21 ` Robin Murphy
2017-04-13 14:29 ` Shanker Donthineni
2017-05-04 13:32 ` Bharat Bhushan
2017-04-13 19:38 ` Nate Watterson
2017-05-16 19:55 ` Auger Eric
2017-05-16 20:07 ` Nate Watterson
2017-05-16 20:20 ` Auger Eric [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=131c7b30-ab20-6ba0-743b-f61095f878c0@redhat.com \
--to=eric.auger@redhat.com \
--cc=iommu@lists.linux-foundation.org \
--cc=joro@8bytes.org \
--cc=linux-kernel@vger.kernel.org \
--cc=nwatters@codeaurora.org \
--cc=robin.murphy@arm.com \
--cc=shankerd@codeaurora.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).