From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-14.1 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id F3EF4C47098 for ; Thu, 3 Jun 2021 08:55:21 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id C472D613DE for ; Thu, 3 Jun 2021 08:55:21 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org C472D613DE Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=solid-run.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:Subject:Message-ID:Date:From: In-Reply-To:References:MIME-Version:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=ORl0DBKo0oLCaBojrlhhqeNSvMq60lBZ9P9RnTkr8ms=; b=F2Vxm4PaZa2AdW KM8JXLpDGoIWfbHjhU4w6Rl5bR4ZegPlop6u52L4EfQJIqR+Zq53LzcrQUOUBZqQmjN2HHvMSueWE qFglzEfsb7bgwjZ/o10Ps/4/alIuwF1z2p4/JBc1Ao4W9+FuNGJFj9jayGaNWtXn+9oBelBImVZPq h5umc5bQ7py6wjyk9qBCeODm3KWioI/TuyIVbFCAzNqKkgK3nlRJqPpyaZ4bv4esHbuz45FVDJQRJ tozZNPWw0CgQhtmFHsu67eBlTWdXHOik1IPPYvk0aNx6YC25GGMWBKZ1ovqMI41wcufzDh8SFuwYZ wjJGKCxD3GX9ikH/Cl1A==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1loj6i-007nCU-9o; Thu, 03 Jun 2021 08:53:40 +0000 Received: from mail-vs1-xe31.google.com ([2607:f8b0:4864:20::e31]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1loj6a-007n9f-WE for linux-arm-kernel@lists.infradead.org; Thu, 03 Jun 2021 08:53:35 +0000 Received: by mail-vs1-xe31.google.com with SMTP id s22so2587558vsl.10 for ; Thu, 03 Jun 2021 01:53:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=solid-run-com.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=Myemz5k16Nb16eM+VNAAZnl9WcdJHNpws7yrBRTSzQ8=; b=FIiLMwkjVLQwvIDeLZi02NT81EiYl8Vfp2TJ4+4aniF2IcsesHL+BydP/wXTPVWmgI SBAcJ2BAoTaOeIrGslJ5XmYP49OoR3Tu4uRa3pvDMs5SBYgCy+yG4VRr4v6lMaBYnDLc SKOeIG230/Eizv18tylHUlXlsoTt3u48K+zr+c9e31PVBp+K0OMj/IAUWRemEvdT+Or1 p1zgDyza8IIruANBM7NvSo8ggNoATA7iKA4qQO+XsS0UTBvAR6KHhUpDQo/0CcRjMH3n mOIamavU9Bjb/bEFlzcfgqPwYMjHD5VeKXFFJyQKqgskoI/jocKzYi8bTUUQCfuW9xer xung== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=Myemz5k16Nb16eM+VNAAZnl9WcdJHNpws7yrBRTSzQ8=; b=crCv3m50iW+FFz8rAYHGeSrQThcscStn3E0cb/yNV8siraE4wz713i5hanpP+Rq/AT oKXe0dZxEODfDhVjV1stui4cALKZsmAsoSndZohNusnK7mPoBZuSSWkB5fWVpt60dc8R nhVNRNmgU6pBKNJQXSqczGMCZ0ahFjg7nx4nj1NrkPn0nr9YGoy42sabOVh8W4orwWIl 2686kcOjnsFpWMLrgy6pyIVr8X/Pk+idRLXpt2kICQAzhHHzxpI3sQreKj5ydWr3RuPg AgkCuBa4uq1c4P/xkTTLDdW8GIabWdVpkNV0czarFMb1sYHMiRSZZYMZT/fKX2NWB08z 9h2g== X-Gm-Message-State: AOAM533EmCPvYMgRLmRGDxKwUXthDSm2lxLwShf+8sweHRQC54GzQnBI mlxAGQ9/94NZlBFEQoFLJwiX+tv9b97nvNYTNHwhxQ== X-Google-Smtp-Source: ABdhPJxj+4IT8+6qr07qq87ubJ2nmPl8BZS/9F8+PlebID2YL/LATooVU+fmNI4LSTztODs2WUyyuopyqMoKE7EXvwM= X-Received: by 2002:a67:b406:: with SMTP id x6mr7014288vsl.10.1622710410065; Thu, 03 Jun 2021 01:53:30 -0700 (PDT) MIME-Version: 1.0 References: <20210524110222.2212-1-shameerali.kolothum.thodi@huawei.com> <20210524110222.2212-8-shameerali.kolothum.thodi@huawei.com> In-Reply-To: <20210524110222.2212-8-shameerali.kolothum.thodi@huawei.com> From: Jon Nettleton Date: Thu, 3 Jun 2021 10:52:53 +0200 Message-ID: Subject: Re: [PATCH v5 7/8] iommu/arm-smmu: Get associated RMR info and install bypass SMR To: Shameer Kolothum Cc: linux-arm-kernel , ACPI Devel Maling List , iommu@lists.linux-foundation.org, Linuxarm , Lorenzo Pieralisi , joro@8bytes.org, Robin Murphy , wanghuiqiang , Hanjun Guo , Steven Price , Sami.Mujawar@arm.com, eric.auger@redhat.com, yangyicong X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210603_015333_959421_741470C4 X-CRM114-Status: GOOD ( 32.93 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Mon, May 24, 2021 at 1:04 PM Shameer Kolothum wrote: > > From: Jon Nettleton > > Check if there is any RMR info associated with the devices behind > the SMMU and if any, install bypass SMRs for them. This is to > keep any ongoing traffic associated with these devices alive > when we enable/reset SMMU during probe(). > > Signed-off-by: Jon Nettleton > Signed-off-by: Steven Price > Signed-off-by: Shameer Kolothum > --- > drivers/iommu/arm/arm-smmu/arm-smmu.c | 65 +++++++++++++++++++++++++++ > 1 file changed, 65 insertions(+) > > diff --git a/drivers/iommu/arm/arm-smmu/arm-smmu.c b/drivers/iommu/arm/arm-smmu/arm-smmu.c > index 6f72c4d208ca..56db3d3238fc 100644 > --- a/drivers/iommu/arm/arm-smmu/arm-smmu.c > +++ b/drivers/iommu/arm/arm-smmu/arm-smmu.c > @@ -2042,6 +2042,67 @@ err_reset_platform_ops: __maybe_unused; > return err; > } > > +static void arm_smmu_rmr_install_bypass_smr(struct arm_smmu_device *smmu) > +{ > + struct list_head rmr_list; > + struct iommu_resv_region *e; > + int i, cnt = 0; > + u32 smr; > + u32 reg; > + > + INIT_LIST_HEAD(&rmr_list); > + if (iommu_dma_get_rmrs(dev_fwnode(smmu->dev), &rmr_list)) > + return; > + > + reg = arm_smmu_gr0_read(smmu, ARM_SMMU_GR0_sCR0); > + > + if ((reg & ARM_SMMU_sCR0_USFCFG) && !(reg & ARM_SMMU_sCR0_CLIENTPD)) { > + /* > + * SMMU is already enabled and disallowing bypass, so preserve > + * the existing SMRs > + */ > + for (i = 0; i < smmu->num_mapping_groups; i++) { > + smr = arm_smmu_gr0_read(smmu, ARM_SMMU_GR0_SMR(i)); > + if (!FIELD_GET(ARM_SMMU_SMR_VALID, smr)) > + continue; > + smmu->smrs[i].id = FIELD_GET(ARM_SMMU_SMR_ID, smr); > + smmu->smrs[i].mask = FIELD_GET(ARM_SMMU_SMR_MASK, smr); > + smmu->smrs[i].valid = true; > + } > + } > + > + list_for_each_entry(e, &rmr_list, list) { > + u32 sid = e->fw_data.rmr.sid; > + > + i = arm_smmu_find_sme(smmu, sid, ~0); > + if (i < 0) > + continue; > + if (smmu->s2crs[i].count == 0) { > + smmu->smrs[i].id = sid; > + smmu->smrs[i].mask = ~0; > + smmu->smrs[i].valid = true; > + } > + smmu->s2crs[i].count++; > + smmu->s2crs[i].type = S2CR_TYPE_BYPASS; > + smmu->s2crs[i].privcfg = S2CR_PRIVCFG_DEFAULT; > + smmu->s2crs[i].cbndx = 0xff; > + > + cnt++; > + } > + > + if ((reg & ARM_SMMU_sCR0_USFCFG) && !(reg & ARM_SMMU_sCR0_CLIENTPD)) { > + /* Remove the valid bit for unused SMRs */ > + for (i = 0; i < smmu->num_mapping_groups; i++) { > + if (smmu->s2crs[i].count == 0) > + smmu->smrs[i].valid = false; > + } > + } > + > + dev_notice(smmu->dev, "\tpreserved %d boot mapping%s\n", cnt, > + cnt == 1 ? "" : "s"); > + iommu_dma_put_rmrs(dev_fwnode(smmu->dev), &rmr_list); > +} > + > static int arm_smmu_device_probe(struct platform_device *pdev) > { > struct resource *res; > @@ -2168,6 +2229,10 @@ static int arm_smmu_device_probe(struct platform_device *pdev) > } > > platform_set_drvdata(pdev, smmu); > + > + /* Check for RMRs and install bypass SMRs if any */ > + arm_smmu_rmr_install_bypass_smr(smmu); > + > arm_smmu_device_reset(smmu); > arm_smmu_test_smr_masks(smmu); > > -- > 2.17.1 > Shameer and Robin I finally got around to updating edk2 and the HoneyComb IORT tables to reflect the new standards. Out of the box the new patchset was generating errors immediatly after the smmu bringup. arm-smmu arm-smmu.0.auto: Unhandled context fault: fsr=0x402, iova=0x2080000140, fsynr=0x1d0040, cbfrsynra=0x4000, cb=0 These errors were generated even with disable_bypass=0 I tracked down the issue to This code is skipped as Robin said would be correct > + if ((reg & ARM_SMMU_sCR0_USFCFG) && !(reg & ARM_SMMU_sCR0_CLIENTPD)) { > + /* > + * SMMU is already enabled and disallowing bypass, so preserve > + * the existing SMRs > + */ > + for (i = 0; i < smmu->num_mapping_groups; i++) { > + smr = arm_smmu_gr0_read(smmu, ARM_SMMU_GR0_SMR(i)); > + if (!FIELD_GET(ARM_SMMU_SMR_VALID, smr)) > + continue; > + smmu->smrs[i].id = FIELD_GET(ARM_SMMU_SMR_ID, smr); > + smmu->smrs[i].mask = FIELD_GET(ARM_SMMU_SMR_MASK, smr); > + smmu->smrs[i].valid = true; > + }[ 2.707729] arm-smmu: setting up rmr list on 0x4000 [ 2.712598] arm-smmu: s2crs count is 0 smrs index 0x0 [ 2.717638] arm-smmu: s2crs count is 0 smrs id is 0x4000 [ 2.722939] arm-smmu: s2crs count is 0 smrs mask is 0x8000 [ 2.728417] arm-smmu arm-smmu.0.auto: preserved 1 boot mapping > + } Then this code block was hit which is correct > + if (smmu->s2crs[i].count == 0) { > + smmu->smrs[i].id = sid; > + smmu->smrs[i].mask = ~0; > + smmu->smrs[i].valid = true; > + } The mask was causing the issue. If I think ammended that segment to read the mask as setup by the hardware everything was back to functioning both with and without disable_bypass set. Some debug from that section when it is working [ 2.712598] arm-smmu: s2crs count is 0 smrs index 0x0 [ 2.717638] arm-smmu: s2crs count is 0 smrs id is 0x4000 [ 2.722939] arm-smmu: s2crs count is 0 smrs mask is 0x8000 [ 2.728417] arm-smmu arm-smmu.0.auto: preserved 1 boot mapping Robin if anything jumps out at you let me know, otherwise I will debug further. -Jon _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel