From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754051AbdEDNcS (ORCPT ); Thu, 4 May 2017 09:32:18 -0400 Received: from mail-he1eur01on0067.outbound.protection.outlook.com ([104.47.0.67]:30718 "EHLO EUR01-HE1-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1751565AbdEDNcK (ORCPT ); Thu, 4 May 2017 09:32:10 -0400 From: Bharat Bhushan To: "shankerd@codeaurora.org" , Robin Murphy , Nate Watterson , "Joerg Roedel" , "iommu@lists.linux-foundation.org" , "linux-kernel@vger.kernel.org" Subject: RE: [PATCH] iommu/dma: Setup iova_domain granule for IOMMU_DMA_MSI cookies Thread-Topic: [PATCH] iommu/dma: Setup iova_domain granule for IOMMU_DMA_MSI cookies Thread-Index: AQHStEgebX2reUgIZUeeAbNtuGiEcaHDXDGAgCDrdTA= Date: Thu, 4 May 2017 13:32:07 +0000 Message-ID: References: <1492073715-7811-1-git-send-email-nwatters@codeaurora.org> <90fe2111-42e5-ecb1-d0bd-10e06883ae5d@arm.com> <383509cb-ae33-e0c4-1c06-8b162acdb555@codeaurora.org> In-Reply-To: <383509cb-ae33-e0c4-1c06-8b162acdb555@codeaurora.org> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: codeaurora.org; dkim=none (message not signed) header.d=none;codeaurora.org; dmarc=none action=none header.from=nxp.com; x-originating-ip: [192.88.169.1] x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1;AM5PR0401MB2547;7:qljkW6OgftGkaDtM8GK7RV/7GIBQlEtLSQqFKBFoyeVdVB5YdvVomuK7P7RFq90SZJH+oBc6ZcDArqufrDg6ZozGjHNid2nU1ESuLOwe85BL+ACHc1jK1YKSd+kJ++2vg2mWO9pukB06Zd3cnwJ4j0AOcYDOvUsNEYkgIUcprQRit4OhRHqpHysbOMx8TQwgcqRC1PHQ2wUrQIecFRe/ntIkJO9QoNrlrB11/HpLOaBVzVaG8ySQiMDrUA5EtQcDREk88RwuldBwjCNwvgIzkIfYKPYgbGs9PebgFrnX11st2wSpm9z6pXLotn1ZCZxLiGqt3X/M8sZz1u1rVqGyDQ== x-ms-office365-filtering-correlation-id: ec1fae8a-2b9e-4e46-6f70-08d492f1f60f x-ms-office365-filtering-ht: Tenant x-microsoft-antispam: UriScan:;BCL:0;PCL:0;RULEID:(22001)(2017030254075)(48565401081)(201703131423075)(201703031133081);SRVR:AM5PR0401MB2547; x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:(158342451672863)(180628864354917)(9452136761055); x-exchange-antispam-report-cfa-test: BCL:0;PCL:0;RULEID:(6040450)(601004)(2401047)(5005006)(8121501046)(10201501046)(3002001)(93006095)(93001095)(6055026)(6041248)(20161123555025)(20161123558100)(20161123564025)(20161123560025)(201703131423075)(201702281528075)(201703061421075)(201703061406153)(20161123562025)(6072148);SRVR:AM5PR0401MB2547;BCL:0;PCL:0;RULEID:;SRVR:AM5PR0401MB2547; x-forefront-prvs: 02973C87BC x-forefront-antispam-report: SFV:NSPM;SFS:(10009020)(6009001)(39450400003)(39840400002)(39860400002)(39850400002)(39400400002)(39410400002)(24454002)(13464003)(377454003)(2900100001)(6506006)(53936002)(66066001)(6306002)(86362001)(77096006)(6436002)(55016002)(99286003)(478600001)(74316002)(9686003)(2201001)(6246003)(38730400002)(2950100002)(3846002)(3660700001)(189998001)(6116002)(102836003)(33656002)(122556002)(2906002)(3280700002)(229853002)(7696004)(7736002)(50986999)(76176999)(305945005)(54356999)(5660300001)(53546009)(8936002)(25786009)(5890100001)(2501003)(81166006);DIR:OUT;SFP:1101;SCL:1;SRVR:AM5PR0401MB2547;H:AM5PR0401MB2545.eurprd04.prod.outlook.com;FPR:;SPF:None;MLV:sfv;LANG:en; spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 X-OriginatorOrg: nxp.com X-MS-Exchange-CrossTenant-originalarrivaltime: 04 May 2017 13:32:07.8062 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 686ea1d3-bc2b-4c6f-a92c-d99c5c301635 X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM5PR0401MB2547 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from base64 to 8bit by mail.home.local id v44DWVkI019056 Hi Robin, I faced same issue on our platform and debugged to get the root cause of the issue. Also fixed in somewhat similar way, cleared iova_off and not change size for if cookie->type is IOMMU_DMA_MSI_COOKIE. Anyways that was not as good as below changes. While just now I saw this was already discussed and solved. Should have looked earlier to save time, but the debugging was some learning experience 😊 I have tested your below change and it works fine on NXP platform. Thanks -Bharat > -----Original Message----- > From: iommu-bounces@lists.linux-foundation.org [mailto:iommu- > bounces@lists.linux-foundation.org] On Behalf Of Shanker Donthineni > Sent: Thursday, April 13, 2017 8:00 PM > To: Robin Murphy ; Nate Watterson > ; Joerg Roedel ; > iommu@lists.linux-foundation.org; linux-kernel@vger.kernel.org > Subject: Re: [PATCH] iommu/dma: Setup iova_domain granule for > IOMMU_DMA_MSI cookies > > Hi Robin, > > I tested your changes and the device pass-through feature works fine on > QDF2400 server platform. Maybe Nate comments on the patch contents but > it fixes the problem. > > > @@ -317,13 +317,13 @@ static void iommu_dma_free_iova(struct > iommu_dma_cookie *cookie, > dma_addr_t iova, size_t size) { > struct iova_domain *iovad = &cookie->iovad; > - unsigned long shift = iova_shift(iovad); > > /* The MSI case is only ever cleaning up its most recent allocation */ > if (cookie->type == IOMMU_DMA_MSI_COOKIE) > cookie->msi_iova -= size; > else > - free_iova_fast(iovad, iova >> shift, size >> shift); > + free_iova_fast(iovad, iova_pfn(iovad, iova), > + size >> iova_shift(iovad)); > } > > static void __iommu_dma_unmap(struct iommu_domain *domain, > dma_addr_t dma_addr, @@ -538,11 +538,14 @@ static dma_addr_t > __iommu_dma_map(struct device *dev, phys_addr_t phys, { > struct iommu_domain *domain = iommu_get_domain_for_dev(dev); > struct iommu_dma_cookie *cookie = domain->iova_cookie; > - struct iova_domain *iovad = &cookie->iovad; > - size_t iova_off = iova_offset(iovad, phys); > + size_t iova_off = 0; > dma_addr_t iova; > > - size = iova_align(iovad, size + iova_off); > + if (cookie->type == IOMMU_DMA_IOVA_COOKIE) { > + iova_off = iova_offset(&cookie->iovad, phys); > + size = iova_align(&cookie->iovad, size + iova_off); > + } > > > On 04/13/2017 06:21 AM, Robin Murphy wrote: > > Hi Nate, > > > > On 13/04/17 09:55, Nate Watterson wrote: > >> Currently, the __iommu_dma_{map/free} functions call > >> iova_{offset/align} making them unsuitable for use with iommu_domains > >> having an IOMMU_DMA_MSI cookie since the cookie's iova_domain > member, iovad, is uninitialized. > >> > >> Now that iommu_dma_get_msi_page() calls __iommu_dma_map() > regardless > >> of cookie type, failures are being seen when mapping MSI target > >> addresses for devices attached to UNMANAGED domains. To work > around > >> this issue, the iova_domain granule for IOMMU_DMA_MSI cookies is > >> initialized to the value returned by cookie_msi_granule(). > > Oh bum. Thanks for the report. > > > > However, I really don't like bodging around it with deliberate > > undefined behaviour. Fixing things properly doesn't seem too hard: > > > > ----->8----- > > diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c > > index 8348f366ddd1..62618e77bedc 100644 > > --- a/drivers/iommu/dma-iommu.c > > +++ b/drivers/iommu/dma-iommu.c > > @@ -396,13 +396,13 @@ static void iommu_dma_free_iova(struct > > iommu_dma_cookie *cookie, > > dma_addr_t iova, size_t size) { > > struct iova_domain *iovad = &cookie->iovad; > > - unsigned long shift = iova_shift(iovad); > > > > /* The MSI case is only ever cleaning up its most recent > > allocation */ > > if (cookie->type == IOMMU_DMA_MSI_COOKIE) > > cookie->msi_iova -= size; > > else > > - free_iova_fast(iovad, iova >> shift, size >> shift); > > + free_iova_fast(iovad, iova_pfn(iovad, iova), > > + size >> iova_shift(iovad)); > > } > > > > static void __iommu_dma_unmap(struct iommu_domain *domain, > dma_addr_t > > dma_addr, @@ -617,11 +617,14 @@ static dma_addr_t > > __iommu_dma_map(struct device *dev, phys_addr_t phys, { > > struct iommu_domain *domain = iommu_get_domain_for_dev(dev); > > struct iommu_dma_cookie *cookie = domain->iova_cookie; > > - struct iova_domain *iovad = &cookie->iovad; > > - size_t iova_off = iova_offset(iovad, phys); > > + size_t iova_off = 0; > > dma_addr_t iova; > > > > - size = iova_align(iovad, size + iova_off); > > + if (cookie->type == IOMMU_DMA_IOVA_COOKIE) { > > + iova_off = iova_offset(&cookie->iovad, phys); > > + size = iova_align(&cookie->iovad, size + iova_off); > > + } > > + > > iova = iommu_dma_alloc_iova(domain, size, dma_get_mask(dev), > dev); > > if (!iova) > > return DMA_ERROR_CODE; > > -----8<----- > > > > Untested, and you'll probably want to double-check it anyway given > > that the original oversight was mine in the first place ;) > > > > Robin. > > > >> Fixes: a44e6657585b ("iommu/dma: Clean up MSI IOVA allocation") > >> Signed-off-by: Nate Watterson > >> --- > >> drivers/iommu/dma-iommu.c | 10 ++++++++++ > >> 1 file changed, 10 insertions(+) > >> > >> diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c > >> index 8348f366..d7b0816 100644 > >> --- a/drivers/iommu/dma-iommu.c > >> +++ b/drivers/iommu/dma-iommu.c > >> @@ -127,6 +127,16 @@ int iommu_get_msi_cookie(struct > iommu_domain > >> *domain, dma_addr_t base) > >> > >> cookie->msi_iova = base; > >> domain->iova_cookie = cookie; > >> + > >> + /* > >> + * Setup granule for compatibility with __iommu_dma_{alloc/free} > and > >> + * add a compile time check to ensure that writing granule won't > >> + * clobber msi_iova. > >> + */ > >> + cookie->iovad.granule = cookie_msi_granule(cookie); > >> + BUILD_BUG_ON(offsetof(struct iova_domain, granule) < > >> + sizeof(cookie->msi_iova)); > >> + > >> return 0; > >> } > >> EXPORT_SYMBOL(iommu_get_msi_cookie); > >> > > -- > Shanker Donthineni > Qualcomm Datacenter Technologies, Inc. as an affiliate of Qualcomm > Technologies, Inc. > Qualcomm Technologies, Inc. is a member of the Code Aurora Forum, a Linux > Foundation Collaborative Project. > > _______________________________________________ > iommu mailing list > iommu@lists.linux-foundation.org > https://lists.linuxfoundation.org/mailman/listinfo/iommu