From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.5 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 82D71C636C9 for ; Wed, 21 Jul 2021 15:27:30 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 4EEE061244 for ; Wed, 21 Jul 2021 15:27:30 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 4EEE061244 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Subject:Cc:To:From:Message-ID:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=gE4GOe3awxptFLw1+o33jEEFbe2100N1tp0czehMkr8=; b=ICtSl/wZQ+biWO 8lzjJzf5NPbIBfpA4ct7RxGZgADpvwor4dvzooK9DRq3Z4bfbiUoxzyosRtBWgC37PbReKi2Cyubw t9cSJ2GJWbwwWQoCFXr32doEt/RXn4XLCJLhVrkwmEsAHjb9J1hcQCntF4p+Jkd6XsLARR9oFdIGn fttOC2nQUI6uyfOtxNCfyFBc2o3UsALW/8dXJzsY1JSKItolf5/sEz3A4MxaLFvvUhbtJrTp6B3nB gkZi15K8rxqeSUzaweIkjzwTNm6CtXrbz3pLn5/Vq0sbHQCGferFEjPbRNWLPyIvu3ZBL56faNhm2 8hACVnQU93SavJUkoWiA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1m6E6K-00GICs-82; Wed, 21 Jul 2021 15:25:36 +0000 Received: from mail.kernel.org ([198.145.29.99]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1m6Dix-00G9VZ-HW for linux-arm-kernel@lists.infradead.org; Wed, 21 Jul 2021 15:01:29 +0000 Received: from disco-boy.misterjones.org (disco-boy.misterjones.org [51.254.78.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 2D6A16120E; Wed, 21 Jul 2021 15:01:27 +0000 (UTC) Received: from sofa.misterjones.org ([185.219.108.64] helo=why.misterjones.org) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1m6Div-0005Ox-3p; Wed, 21 Jul 2021 16:01:25 +0100 Date: Wed, 21 Jul 2021 16:01:24 +0100 Message-ID: <877dhj3e4b.wl-maz@kernel.org> From: Marc Zyngier To: Robin Murphy Cc: Bixuan Cui , iommu@lists.linux-foundation.org, linux-kernel@vger.kernel.org, will@kernel.org, weiyongjun1@huawei.com, john.wanghui@huawei.com, dingtianhong@huawei.com, thunder.leizhen@huawei.com, guohanjun@huawei.com, joro@8bytes.org, jean-philippe@linaro.org, Jonathan.Cameron@huawei.com, song.bao.hua@hisilicon.com, linux-arm-kernel@lists.infradead.org Subject: Re: [PATCH -next] iommu/arm-smmu-v3: Add suspend and resume support In-Reply-To: <848befb0-7a9a-0b2b-8be9-3dfa02919488@arm.com> References: <20210721013350.17664-1-cuibixuan@huawei.com> <4e506481-5f6c-9c5e-eda3-300861581080@arm.com> <878s1z3j68.wl-maz@kernel.org> <848befb0-7a9a-0b2b-8be9-3dfa02919488@arm.com> User-Agent: Wanderlust/2.15.9 (Almost Unreal) SEMI-EPG/1.14.7 (Harue) FLIM-LB/1.14.9 (=?UTF-8?B?R29qxY0=?=) APEL-LB/10.8 EasyPG/1.0.0 Emacs/27.1 (x86_64-pc-linux-gnu) MULE/6.0 (HANACHIRUSATO) MIME-Version: 1.0 (generated by SEMI-EPG 1.14.7 - "Harue") X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: robin.murphy@arm.com, cuibixuan@huawei.com, iommu@lists.linux-foundation.org, linux-kernel@vger.kernel.org, will@kernel.org, weiyongjun1@huawei.com, john.wanghui@huawei.com, dingtianhong@huawei.com, thunder.leizhen@huawei.com, guohanjun@huawei.com, joro@8bytes.org, jean-philippe@linaro.org, Jonathan.Cameron@huawei.com, song.bao.hua@hisilicon.com, linux-arm-kernel@lists.infradead.org X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210721_080127_723169_31AFD18D X-CRM114-Status: GOOD ( 36.13 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Wed, 21 Jul 2021 14:59:47 +0100, Robin Murphy wrote: > > On 2021-07-21 14:12, Marc Zyngier wrote: > > On Wed, 21 Jul 2021 12:42:14 +0100, > > Robin Murphy wrote: > >> > >> [ +Marc for MSI bits ] > >> > >> On 2021-07-21 02:33, Bixuan Cui wrote: > >>> Add suspend and resume support for arm-smmu-v3 by low-power mode. > >>> > >>> When the smmu is suspended, it is powered off and the registers are > >>> cleared. So saves the msi_msg context during msi interrupt initialization > >>> of smmu. When resume happens it calls arm_smmu_device_reset() to restore > >>> the registers. > >>> > >>> Signed-off-by: Bixuan Cui > >>> Reviewed-by: Wei Yongjun > >>> Reviewed-by: Zhen Lei > >>> Reviewed-by: Ding Tianhong > >>> Reviewed-by: Hanjun Guo > >>> --- > >>> > >>> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 72 ++++++++++++++++++--- > >>> 1 file changed, 64 insertions(+), 8 deletions(-) > >>> > >>> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c > >>> index 235f9bdaeaf2..bf1163acbcb1 100644 > >>> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c > >>> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c > >>> @@ -40,6 +40,7 @@ MODULE_PARM_DESC(disable_bypass, > >>> static bool disable_msipolling; > >>> module_param(disable_msipolling, bool, 0444); > >>> +static bool bypass; > >>> MODULE_PARM_DESC(disable_msipolling, > >>> "Disable MSI-based polling for CMD_SYNC completion."); > >>> @@ -3129,11 +3130,37 @@ static void arm_smmu_write_msi_msg(struct > >>> msi_desc *desc, struct msi_msg *msg) > >>> doorbell = (((u64)msg->address_hi) << 32) | msg->address_lo; > >>> doorbell &= MSI_CFG0_ADDR_MASK; > >>> + /* Saves the msg context for resume if desc->msg is empty */ > >>> + if (desc->msg.address_lo == 0 && desc->msg.address_hi == 0) { > >>> + desc->msg.address_lo = msg->address_lo; > >>> + desc->msg.address_hi = msg->address_hi; > >>> + desc->msg.data = msg->data; > >>> + } > >> > >> My gut feeling is that this is something a device driver maybe > >> shouldn't be poking into, but I'm not entirely familiar with the area > >> :/ > > > > Certainly not. If you rely on the message being stored into the > > descriptors, then implement this in the core code, like we do for PCI. > > Ah, so it would be an acceptable compromise to *read* desc->msg (and > thus avoid having to store our own copy of the message) if the core > was guaranteed to cache it? That's good to know, thanks. Yeah, vfio, a couple of other weird drivers and (*surprise!*) ia64 are using this kind of trick. I don't see a reason not to implement that for platform-MSI (although level signalling may be interesting...), or even to move it into the core MSI code. > > >>> + > >>> writeq_relaxed(doorbell, smmu->base + cfg[0]); > >>> writel_relaxed(msg->data, smmu->base + cfg[1]); > >>> writel_relaxed(ARM_SMMU_MEMATTR_DEVICE_nGnRE, smmu->base + cfg[2]); > >>> } > >>> +static void arm_smmu_resume_msis(struct arm_smmu_device *smmu) > >>> +{ > >>> + struct msi_desc *desc; > >>> + struct device *dev = smmu->dev; > >>> + > >>> + for_each_msi_entry(desc, dev) { > >>> + switch (desc->platform.msi_index) { > >>> + case EVTQ_MSI_INDEX: > >>> + case GERROR_MSI_INDEX: > >>> + case PRIQ_MSI_INDEX: > >>> + arm_smmu_write_msi_msg(desc, &(desc->msg)); > > > > Consider using get_cached_msi_msg() instead of using the internals of > > the descriptor. > > Oh, there's even a proper API for it, marvellous! I hadn't managed to > dig that far myself :) It is a bit odd in the sense that it takes a copy of the message instead of returning a pointer, but at least this solves lifetime issues. Thanks, M. -- Without deviation from the norm, progress is not possible. _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel