From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.3 required=3.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE, SPF_PASS,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2A67AC3A5A6 for ; Thu, 19 Sep 2019 14:35:47 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id E17C62067B for ; Thu, 19 Sep 2019 14:35:46 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="pdpL7xJY" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org E17C62067B Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=huawei.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+infradead-linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender:Content-Type: Content-Transfer-Encoding:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:Date:Message-ID:References: To:Subject:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=6DrVk2l85CwOrib/nSwujoQgtBSWaLKwdEXAe+N8/go=; b=pdpL7xJYFrQsUZtfRn43fQ6gP KUoPDDPocWS5i8LaYhsy/sez1OffXSc/8ipxDrZWGqENA8HA/yPanawlo6x+UfeCGd5YOFzo5MA5X FSUTXxcj6Bcj70xxRXvsDo9ty4EfYXNOnjyDbFi/dBLNZd3jVNq911J37D/a7egQ18RcPLrmzPnEr bLI+FvTkzB2KhTuTltsmqzrK/6qxeOaURe4dZ7O+nKuCKtuxtG3t+K2TptgSNUn+i97uRv1YbUykJ bT95EhBe56dDIgPXUM/64Xycrl2T6pjik5JlKP3BH4sPkG69bK0Cu6hd7H8fxCdX3hHlSYtM+HmjA nQCwGAU+g==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.92.2 #3 (Red Hat Linux)) id 1iAxWz-0007WZ-Qg; Thu, 19 Sep 2019 14:35:37 +0000 Received: from szxga06-in.huawei.com ([45.249.212.32] helo=huawei.com) by bombadil.infradead.org with esmtps (Exim 4.92.2 #3 (Red Hat Linux)) id 1iAxWw-0007Vu-Cf for linux-arm-kernel@lists.infradead.org; Thu, 19 Sep 2019 14:35:36 +0000 Received: from DGGEMS412-HUB.china.huawei.com (unknown [172.30.72.59]) by Forcepoint Email with ESMTP id D07CF8849E71097B62BE; Thu, 19 Sep 2019 22:35:25 +0800 (CST) Received: from [127.0.0.1] (10.202.227.179) by DGGEMS412-HUB.china.huawei.com (10.3.19.212) with Microsoft SMTP Server id 14.3.439.0; Thu, 19 Sep 2019 22:35:16 +0800 From: John Garry Subject: Re: arm64 iommu groups issue To: Robin Murphy , Marc Zyngier , "Will Deacon" , Lorenzo Pieralisi , Sudeep Holla , "Guohanjun (Hanjun Guo)" References: <9625faf4-48ef-2dd3-d82f-931d9cf26976@huawei.com> <4768c541-ebf4-61d5-0c5e-77dee83f8f94@arm.com> Message-ID: Date: Thu, 19 Sep 2019 15:35:08 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.3.0 MIME-Version: 1.0 In-Reply-To: <4768c541-ebf4-61d5-0c5e-77dee83f8f94@arm.com> X-Originating-IP: [10.202.227.179] X-CFilter-Loop: Reflected X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20190919_073535_011785_4CA8EE73 X-CRM114-Status: GOOD ( 15.09 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: "linux-kernel@vger.kernel.org" , Alex Williamson , Shameer Kolothum , Linuxarm , iommu , Bjorn Helgaas , "linux-arm-kernel@lists.infradead.org" Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset="us-ascii"; Format="flowed" Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+infradead-linux-arm-kernel=archiver.kernel.org@lists.infradead.org On 19/09/2019 14:25, Robin Murphy wrote: >> When the port eventually probes it gets a new, separate group. >> >> This all seems to be as the built-in module init ordering is as >> follows: pcieport drv, smmu drv, mlx5 drv >> >> I notice that if I build the mlx5 drv as a ko and insert after boot, >> all functions + pcieport are in the same group: >> >> [ 11.530046] hisi_sas_v2_hw HISI0162:01: Adding to iommu group 0 >> [ 17.301093] hns_dsaf HISI00B2:00: Adding to iommu group 1 >> [ 18.743600] ehci-platform PNP0D20:00: Adding to iommu group 2 >> [ 20.212284] pcieport 0002:f8:00.0: Adding to iommu group 3 >> [ 20.356303] pcieport 0004:88:00.0: Adding to iommu group 4 >> [ 20.493337] pcieport 0005:78:00.0: Adding to iommu group 5 >> [ 20.702999] pcieport 000a:10:00.0: Adding to iommu group 6 >> [ 20.859183] pcieport 000c:20:00.0: Adding to iommu group 7 >> [ 20.996140] pcieport 000d:30:00.0: Adding to iommu group 8 >> [ 21.152637] serial 0002:f9:00.0: Adding to iommu group 3 >> [ 21.346991] serial 0002:f9:00.1: Adding to iommu group 3 >> [ 100.754306] mlx5_core 000a:11:00.0: Adding to iommu group 6 >> [ 101.420156] mlx5_core 000a:11:00.1: Adding to iommu group 6 >> [ 292.481714] mlx5_core 000a:11:00.2: Adding to iommu group 6 >> [ 293.281061] mlx5_core 000a:11:00.3: Adding to iommu group 6 >> >> This does seem like a problem for arm64 platforms which don't support >> ACS, yet enable an SMMU. Maybe also a problem even if they do support >> ACS. >> >> Opinion? > Hi Robin, > Yeah, this is less than ideal. For sure. Our production D05 boards don't ship with the SMMU enabled in BIOS, but it would be slightly concerning in this regard if they did. > One way to bodge it might be to make > pci_device_group() also walk downwards to see if any non-ACS-isolated > children already have a group, rather than assuming that groups get > allocated in hierarchical order, but that's far from ideal. Agree. My own workaround was to hack the mentioned iort code to defer the PF probe if the parent port had also yet to probe. > > The underlying issue is that, for historical reasons, OF/IORT-based > IOMMU drivers have ended up with group allocation being tied to endpoint > driver probing via the dma_configure() mechanism (long story short, > driver probe is the only thing which can be delayed in order to wait for > a specific IOMMU instance to be ready).However, in the meantime, the > IOMMU API internals have evolved sufficiently that I think there's a way > to really put things right - I have the spark of an idea which I'll try > to sketch out ASAP... > OK, great. Thanks, John > Robin. _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel