From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752909AbdEPUHj (ORCPT ); Tue, 16 May 2017 16:07:39 -0400 Received: from smtp.codeaurora.org ([198.145.29.96]:55620 "EHLO smtp.codeaurora.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750930AbdEPUHf (ORCPT ); Tue, 16 May 2017 16:07:35 -0400 DMARC-Filter: OpenDMARC Filter v1.3.2 smtp.codeaurora.org 53A816081B Authentication-Results: pdx-caf-mail.web.codeaurora.org; dmarc=none (p=none dis=none) header.from=codeaurora.org Authentication-Results: pdx-caf-mail.web.codeaurora.org; spf=none smtp.mailfrom=nwatters@codeaurora.org Subject: Re: [PATCH] iommu/dma: Setup iova_domain granule for IOMMU_DMA_MSI cookies To: Auger Eric , Robin Murphy , Joerg Roedel , iommu@lists.linux-foundation.org, linux-kernel@vger.kernel.org Cc: shankerd@codeaurora.org References: <1492073715-7811-1-git-send-email-nwatters@codeaurora.org> <90fe2111-42e5-ecb1-d0bd-10e06883ae5d@arm.com> <95dfaf27-9a74-7a6c-fa77-0fcfe1edbe6e@codeaurora.org> <694f109c-78ec-ec1e-037f-0a3807c4ef6d@redhat.com> From: Nate Watterson Message-ID: <2fa1c68c-2c9a-850c-d5b8-0e30cfd077b0@codeaurora.org> Date: Tue, 16 May 2017 16:07:32 -0400 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.1.0 MIME-Version: 1.0 In-Reply-To: <694f109c-78ec-ec1e-037f-0a3807c4ef6d@redhat.com> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 5/16/2017 3:55 PM, Auger Eric wrote: > Hi, > > On 13/04/2017 21:38, Nate Watterson wrote: >> Hi Robin, >> >> On 4/13/2017 7:21 AM, Robin Murphy wrote: >>> Hi Nate, >>> >>> On 13/04/17 09:55, Nate Watterson wrote: >>>> Currently, the __iommu_dma_{map/free} functions call iova_{offset/align} >>>> making them unsuitable for use with iommu_domains having an >>>> IOMMU_DMA_MSI >>>> cookie since the cookie's iova_domain member, iovad, is uninitialized. >>>> >>>> Now that iommu_dma_get_msi_page() calls __iommu_dma_map() regardless >>>> of cookie type, failures are being seen when mapping MSI target >>>> addresses for devices attached to UNMANAGED domains. To work around >>>> this issue, the iova_domain granule for IOMMU_DMA_MSI cookies is >>>> initialized to the value returned by cookie_msi_granule(). >>> >>> Oh bum. Thanks for the report. >>> >>> However, I really don't like bodging around it with deliberate undefined >>> behaviour. Fixing things properly doesn't seem too hard: >> >> I was not especially please with my solution, but I wanted to avoid >> potentially missing any other spots in the code where granule was >> used uninitialized. The compile time check made me feel a little >> less dirty about innappropriately using the iova_domain with MSI >> cookies. >> >>> >>> ----->8----- >>> diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c >>> index 8348f366ddd1..62618e77bedc 100644 >>> --- a/drivers/iommu/dma-iommu.c >>> +++ b/drivers/iommu/dma-iommu.c >>> @@ -396,13 +396,13 @@ static void iommu_dma_free_iova(struct >>> iommu_dma_cookie *cookie, >>> dma_addr_t iova, size_t size) >>> { >>> struct iova_domain *iovad = &cookie->iovad; >>> - unsigned long shift = iova_shift(iovad); >>> >>> /* The MSI case is only ever cleaning up its most recent >>> allocation */ >>> if (cookie->type == IOMMU_DMA_MSI_COOKIE) >>> cookie->msi_iova -= size; >>> else >>> - free_iova_fast(iovad, iova >> shift, size >> shift); >>> + free_iova_fast(iovad, iova_pfn(iovad, iova), >>> + size >> iova_shift(iovad)); >>> } >>> >>> static void __iommu_dma_unmap(struct iommu_domain *domain, dma_addr_t >>> dma_addr, >>> @@ -617,11 +617,14 @@ static dma_addr_t __iommu_dma_map(struct device >>> *dev, phys_addr_t phys, >>> { >>> struct iommu_domain *domain = iommu_get_domain_for_dev(dev); >>> struct iommu_dma_cookie *cookie = domain->iova_cookie; >>> - struct iova_domain *iovad = &cookie->iovad; >>> - size_t iova_off = iova_offset(iovad, phys); >>> + size_t iova_off = 0; >>> dma_addr_t iova; >>> >>> - size = iova_align(iovad, size + iova_off); >>> + if (cookie->type == IOMMU_DMA_IOVA_COOKIE) { >>> + iova_off = iova_offset(&cookie->iovad, phys); >>> + size = iova_align(&cookie->iovad, size + iova_off); >>> + } >>> + >>> iova = iommu_dma_alloc_iova(domain, size, dma_get_mask(dev), >>> dev); >>> if (!iova) >>> return DMA_ERROR_CODE; >>> -----8<----- >>> >>> Untested, and you'll probably want to double-check it anyway given that >>> the original oversight was mine in the first place ;) >> >> This looks good to me. As Shanker has already mentioned, it does fix the >> faults we were previously seeing with direct device assignment. I also >> verified that there aren't any other obvious cases of a granule == 0 >> being used in the dma_iommu code by adding BUG_ON(!iovad->granule) to >> iova_{mask/align/offset/...} and running a variety of tests without >> issue. >> >> Are you going to post the patch? > > I also noticed PCIe passthrough/ARM is broken for me with 4.12-r1. I > tested Robin's patch as well, on Cavium ThunderX, and this fixes the > faults I have seen. > > Has anyone sent a formal patch? iommu/dma: Don't touch invalid iova_domain members > > Thanks > > Eric > >> >>> >>> Robin. >>> >>>> Fixes: a44e6657585b ("iommu/dma: Clean up MSI IOVA allocation") >>>> Signed-off-by: Nate Watterson >>>> --- >>>> drivers/iommu/dma-iommu.c | 10 ++++++++++ >>>> 1 file changed, 10 insertions(+) >>>> >>>> diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c >>>> index 8348f366..d7b0816 100644 >>>> --- a/drivers/iommu/dma-iommu.c >>>> +++ b/drivers/iommu/dma-iommu.c >>>> @@ -127,6 +127,16 @@ int iommu_get_msi_cookie(struct iommu_domain >>>> *domain, dma_addr_t base) >>>> cookie->msi_iova = base; >>>> domain->iova_cookie = cookie; >>>> + >>>> + /* >>>> + * Setup granule for compatibility with __iommu_dma_{alloc/free} >>>> and >>>> + * add a compile time check to ensure that writing granule won't >>>> + * clobber msi_iova. >>>> + */ >>>> + cookie->iovad.granule = cookie_msi_granule(cookie); >>>> + BUILD_BUG_ON(offsetof(struct iova_domain, granule) < >>>> + sizeof(cookie->msi_iova)); >>>> + >>>> return 0; >>>> } >>>> EXPORT_SYMBOL(iommu_get_msi_cookie); >>>> >>> -- Qualcomm Datacenter Technologies as an affiliate of Qualcomm Technologies, Inc. Qualcomm Technologies, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project. From mboxrd@z Thu Jan 1 00:00:00 1970 From: Nate Watterson Subject: Re: [PATCH] iommu/dma: Setup iova_domain granule for IOMMU_DMA_MSI cookies Date: Tue, 16 May 2017 16:07:32 -0400 Message-ID: <2fa1c68c-2c9a-850c-d5b8-0e30cfd077b0@codeaurora.org> References: <1492073715-7811-1-git-send-email-nwatters@codeaurora.org> <90fe2111-42e5-ecb1-d0bd-10e06883ae5d@arm.com> <95dfaf27-9a74-7a6c-fa77-0fcfe1edbe6e@codeaurora.org> <694f109c-78ec-ec1e-037f-0a3807c4ef6d@redhat.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; Format="flowed" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <694f109c-78ec-ec1e-037f-0a3807c4ef6d-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> Content-Language: en-US List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: iommu-bounces-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org Errors-To: iommu-bounces-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org To: Auger Eric , Robin Murphy , Joerg Roedel , iommu-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org, linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org Cc: shankerd-sgV2jX0FEOL9JmXXK+q4OQ@public.gmane.org List-Id: iommu@lists.linux-foundation.org On 5/16/2017 3:55 PM, Auger Eric wrote: > Hi, > > On 13/04/2017 21:38, Nate Watterson wrote: >> Hi Robin, >> >> On 4/13/2017 7:21 AM, Robin Murphy wrote: >>> Hi Nate, >>> >>> On 13/04/17 09:55, Nate Watterson wrote: >>>> Currently, the __iommu_dma_{map/free} functions call iova_{offset/align} >>>> making them unsuitable for use with iommu_domains having an >>>> IOMMU_DMA_MSI >>>> cookie since the cookie's iova_domain member, iovad, is uninitialized. >>>> >>>> Now that iommu_dma_get_msi_page() calls __iommu_dma_map() regardless >>>> of cookie type, failures are being seen when mapping MSI target >>>> addresses for devices attached to UNMANAGED domains. To work around >>>> this issue, the iova_domain granule for IOMMU_DMA_MSI cookies is >>>> initialized to the value returned by cookie_msi_granule(). >>> >>> Oh bum. Thanks for the report. >>> >>> However, I really don't like bodging around it with deliberate undefined >>> behaviour. Fixing things properly doesn't seem too hard: >> >> I was not especially please with my solution, but I wanted to avoid >> potentially missing any other spots in the code where granule was >> used uninitialized. The compile time check made me feel a little >> less dirty about innappropriately using the iova_domain with MSI >> cookies. >> >>> >>> ----->8----- >>> diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c >>> index 8348f366ddd1..62618e77bedc 100644 >>> --- a/drivers/iommu/dma-iommu.c >>> +++ b/drivers/iommu/dma-iommu.c >>> @@ -396,13 +396,13 @@ static void iommu_dma_free_iova(struct >>> iommu_dma_cookie *cookie, >>> dma_addr_t iova, size_t size) >>> { >>> struct iova_domain *iovad = &cookie->iovad; >>> - unsigned long shift = iova_shift(iovad); >>> >>> /* The MSI case is only ever cleaning up its most recent >>> allocation */ >>> if (cookie->type == IOMMU_DMA_MSI_COOKIE) >>> cookie->msi_iova -= size; >>> else >>> - free_iova_fast(iovad, iova >> shift, size >> shift); >>> + free_iova_fast(iovad, iova_pfn(iovad, iova), >>> + size >> iova_shift(iovad)); >>> } >>> >>> static void __iommu_dma_unmap(struct iommu_domain *domain, dma_addr_t >>> dma_addr, >>> @@ -617,11 +617,14 @@ static dma_addr_t __iommu_dma_map(struct device >>> *dev, phys_addr_t phys, >>> { >>> struct iommu_domain *domain = iommu_get_domain_for_dev(dev); >>> struct iommu_dma_cookie *cookie = domain->iova_cookie; >>> - struct iova_domain *iovad = &cookie->iovad; >>> - size_t iova_off = iova_offset(iovad, phys); >>> + size_t iova_off = 0; >>> dma_addr_t iova; >>> >>> - size = iova_align(iovad, size + iova_off); >>> + if (cookie->type == IOMMU_DMA_IOVA_COOKIE) { >>> + iova_off = iova_offset(&cookie->iovad, phys); >>> + size = iova_align(&cookie->iovad, size + iova_off); >>> + } >>> + >>> iova = iommu_dma_alloc_iova(domain, size, dma_get_mask(dev), >>> dev); >>> if (!iova) >>> return DMA_ERROR_CODE; >>> -----8<----- >>> >>> Untested, and you'll probably want to double-check it anyway given that >>> the original oversight was mine in the first place ;) >> >> This looks good to me. As Shanker has already mentioned, it does fix the >> faults we were previously seeing with direct device assignment. I also >> verified that there aren't any other obvious cases of a granule == 0 >> being used in the dma_iommu code by adding BUG_ON(!iovad->granule) to >> iova_{mask/align/offset/...} and running a variety of tests without >> issue. >> >> Are you going to post the patch? > > I also noticed PCIe passthrough/ARM is broken for me with 4.12-r1. I > tested Robin's patch as well, on Cavium ThunderX, and this fixes the > faults I have seen. > > Has anyone sent a formal patch? iommu/dma: Don't touch invalid iova_domain members > > Thanks > > Eric > >> >>> >>> Robin. >>> >>>> Fixes: a44e6657585b ("iommu/dma: Clean up MSI IOVA allocation") >>>> Signed-off-by: Nate Watterson >>>> --- >>>> drivers/iommu/dma-iommu.c | 10 ++++++++++ >>>> 1 file changed, 10 insertions(+) >>>> >>>> diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c >>>> index 8348f366..d7b0816 100644 >>>> --- a/drivers/iommu/dma-iommu.c >>>> +++ b/drivers/iommu/dma-iommu.c >>>> @@ -127,6 +127,16 @@ int iommu_get_msi_cookie(struct iommu_domain >>>> *domain, dma_addr_t base) >>>> cookie->msi_iova = base; >>>> domain->iova_cookie = cookie; >>>> + >>>> + /* >>>> + * Setup granule for compatibility with __iommu_dma_{alloc/free} >>>> and >>>> + * add a compile time check to ensure that writing granule won't >>>> + * clobber msi_iova. >>>> + */ >>>> + cookie->iovad.granule = cookie_msi_granule(cookie); >>>> + BUILD_BUG_ON(offsetof(struct iova_domain, granule) < >>>> + sizeof(cookie->msi_iova)); >>>> + >>>> return 0; >>>> } >>>> EXPORT_SYMBOL(iommu_get_msi_cookie); >>>> >>> -- Qualcomm Datacenter Technologies as an affiliate of Qualcomm Technologies, Inc. Qualcomm Technologies, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project.