From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4BDD6C47082 for ; Thu, 3 Jun 2021 08:53:43 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 3179060FF1 for ; Thu, 3 Jun 2021 08:53:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229640AbhFCIz0 (ORCPT ); Thu, 3 Jun 2021 04:55:26 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54668 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229623AbhFCIz0 (ORCPT ); Thu, 3 Jun 2021 04:55:26 -0400 Received: from mail-vs1-xe2f.google.com (mail-vs1-xe2f.google.com [IPv6:2607:f8b0:4864:20::e2f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 65479C06174A for ; Thu, 3 Jun 2021 01:53:32 -0700 (PDT) Received: by mail-vs1-xe2f.google.com with SMTP id f15so2585075vsq.12 for ; Thu, 03 Jun 2021 01:53:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=solid-run-com.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=Myemz5k16Nb16eM+VNAAZnl9WcdJHNpws7yrBRTSzQ8=; b=FIiLMwkjVLQwvIDeLZi02NT81EiYl8Vfp2TJ4+4aniF2IcsesHL+BydP/wXTPVWmgI SBAcJ2BAoTaOeIrGslJ5XmYP49OoR3Tu4uRa3pvDMs5SBYgCy+yG4VRr4v6lMaBYnDLc SKOeIG230/Eizv18tylHUlXlsoTt3u48K+zr+c9e31PVBp+K0OMj/IAUWRemEvdT+Or1 p1zgDyza8IIruANBM7NvSo8ggNoATA7iKA4qQO+XsS0UTBvAR6KHhUpDQo/0CcRjMH3n mOIamavU9Bjb/bEFlzcfgqPwYMjHD5VeKXFFJyQKqgskoI/jocKzYi8bTUUQCfuW9xer xung== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=Myemz5k16Nb16eM+VNAAZnl9WcdJHNpws7yrBRTSzQ8=; b=muwuzx0zhgKvMKUopVOOSEmNJ7sHIBWDAWhApy8b3s7En8F5KRdHdSQLp7w5hGoO38 +KpYVCs6D5Y9+3BfWD1Qk1Djs/0MBWQel69StidcNkKEruYcKqRLhprYXF5lTSJUI6gn NUEzVxJ5OEydd1F0V6vn/udgKLv9VuRKn9jcD93QpnLUmtGZTz+AtU5Z+la81KAyacmv u8bhqD1LS4B14QZ0amT3JrB1BmMQLd7Gb1dOT5nEbh2p7kXTZsuhFAYx53xUY0vzpBIy 7BGBaMg00qBxu7yFTETbVttv4Ni9XSn2Rtm3lQHqu50k7kGgLmP/wcuw8ftaKZD8CBNV cgyA== X-Gm-Message-State: AOAM5336RhaKDcBNTIW3BaQcIwuAOyFhbk6gBRxhfJhni36bdXNUu614 r0pFe5INzifFxZwTWrvoN6CROyssAjPxyJMS/xlKXg== X-Google-Smtp-Source: ABdhPJxj+4IT8+6qr07qq87ubJ2nmPl8BZS/9F8+PlebID2YL/LATooVU+fmNI4LSTztODs2WUyyuopyqMoKE7EXvwM= X-Received: by 2002:a67:b406:: with SMTP id x6mr7014288vsl.10.1622710410065; Thu, 03 Jun 2021 01:53:30 -0700 (PDT) MIME-Version: 1.0 References: <20210524110222.2212-1-shameerali.kolothum.thodi@huawei.com> <20210524110222.2212-8-shameerali.kolothum.thodi@huawei.com> In-Reply-To: <20210524110222.2212-8-shameerali.kolothum.thodi@huawei.com> From: Jon Nettleton Date: Thu, 3 Jun 2021 10:52:53 +0200 Message-ID: Subject: Re: [PATCH v5 7/8] iommu/arm-smmu: Get associated RMR info and install bypass SMR To: Shameer Kolothum Cc: linux-arm-kernel , ACPI Devel Maling List , iommu@lists.linux-foundation.org, Linuxarm , Lorenzo Pieralisi , joro@8bytes.org, Robin Murphy , wanghuiqiang , Hanjun Guo , Steven Price , Sami.Mujawar@arm.com, eric.auger@redhat.com, yangyicong Content-Type: text/plain; charset="UTF-8" Precedence: bulk List-ID: X-Mailing-List: linux-acpi@vger.kernel.org On Mon, May 24, 2021 at 1:04 PM Shameer Kolothum wrote: > > From: Jon Nettleton > > Check if there is any RMR info associated with the devices behind > the SMMU and if any, install bypass SMRs for them. This is to > keep any ongoing traffic associated with these devices alive > when we enable/reset SMMU during probe(). > > Signed-off-by: Jon Nettleton > Signed-off-by: Steven Price > Signed-off-by: Shameer Kolothum > --- > drivers/iommu/arm/arm-smmu/arm-smmu.c | 65 +++++++++++++++++++++++++++ > 1 file changed, 65 insertions(+) > > diff --git a/drivers/iommu/arm/arm-smmu/arm-smmu.c b/drivers/iommu/arm/arm-smmu/arm-smmu.c > index 6f72c4d208ca..56db3d3238fc 100644 > --- a/drivers/iommu/arm/arm-smmu/arm-smmu.c > +++ b/drivers/iommu/arm/arm-smmu/arm-smmu.c > @@ -2042,6 +2042,67 @@ err_reset_platform_ops: __maybe_unused; > return err; > } > > +static void arm_smmu_rmr_install_bypass_smr(struct arm_smmu_device *smmu) > +{ > + struct list_head rmr_list; > + struct iommu_resv_region *e; > + int i, cnt = 0; > + u32 smr; > + u32 reg; > + > + INIT_LIST_HEAD(&rmr_list); > + if (iommu_dma_get_rmrs(dev_fwnode(smmu->dev), &rmr_list)) > + return; > + > + reg = arm_smmu_gr0_read(smmu, ARM_SMMU_GR0_sCR0); > + > + if ((reg & ARM_SMMU_sCR0_USFCFG) && !(reg & ARM_SMMU_sCR0_CLIENTPD)) { > + /* > + * SMMU is already enabled and disallowing bypass, so preserve > + * the existing SMRs > + */ > + for (i = 0; i < smmu->num_mapping_groups; i++) { > + smr = arm_smmu_gr0_read(smmu, ARM_SMMU_GR0_SMR(i)); > + if (!FIELD_GET(ARM_SMMU_SMR_VALID, smr)) > + continue; > + smmu->smrs[i].id = FIELD_GET(ARM_SMMU_SMR_ID, smr); > + smmu->smrs[i].mask = FIELD_GET(ARM_SMMU_SMR_MASK, smr); > + smmu->smrs[i].valid = true; > + } > + } > + > + list_for_each_entry(e, &rmr_list, list) { > + u32 sid = e->fw_data.rmr.sid; > + > + i = arm_smmu_find_sme(smmu, sid, ~0); > + if (i < 0) > + continue; > + if (smmu->s2crs[i].count == 0) { > + smmu->smrs[i].id = sid; > + smmu->smrs[i].mask = ~0; > + smmu->smrs[i].valid = true; > + } > + smmu->s2crs[i].count++; > + smmu->s2crs[i].type = S2CR_TYPE_BYPASS; > + smmu->s2crs[i].privcfg = S2CR_PRIVCFG_DEFAULT; > + smmu->s2crs[i].cbndx = 0xff; > + > + cnt++; > + } > + > + if ((reg & ARM_SMMU_sCR0_USFCFG) && !(reg & ARM_SMMU_sCR0_CLIENTPD)) { > + /* Remove the valid bit for unused SMRs */ > + for (i = 0; i < smmu->num_mapping_groups; i++) { > + if (smmu->s2crs[i].count == 0) > + smmu->smrs[i].valid = false; > + } > + } > + > + dev_notice(smmu->dev, "\tpreserved %d boot mapping%s\n", cnt, > + cnt == 1 ? "" : "s"); > + iommu_dma_put_rmrs(dev_fwnode(smmu->dev), &rmr_list); > +} > + > static int arm_smmu_device_probe(struct platform_device *pdev) > { > struct resource *res; > @@ -2168,6 +2229,10 @@ static int arm_smmu_device_probe(struct platform_device *pdev) > } > > platform_set_drvdata(pdev, smmu); > + > + /* Check for RMRs and install bypass SMRs if any */ > + arm_smmu_rmr_install_bypass_smr(smmu); > + > arm_smmu_device_reset(smmu); > arm_smmu_test_smr_masks(smmu); > > -- > 2.17.1 > Shameer and Robin I finally got around to updating edk2 and the HoneyComb IORT tables to reflect the new standards. Out of the box the new patchset was generating errors immediatly after the smmu bringup. arm-smmu arm-smmu.0.auto: Unhandled context fault: fsr=0x402, iova=0x2080000140, fsynr=0x1d0040, cbfrsynra=0x4000, cb=0 These errors were generated even with disable_bypass=0 I tracked down the issue to This code is skipped as Robin said would be correct > + if ((reg & ARM_SMMU_sCR0_USFCFG) && !(reg & ARM_SMMU_sCR0_CLIENTPD)) { > + /* > + * SMMU is already enabled and disallowing bypass, so preserve > + * the existing SMRs > + */ > + for (i = 0; i < smmu->num_mapping_groups; i++) { > + smr = arm_smmu_gr0_read(smmu, ARM_SMMU_GR0_SMR(i)); > + if (!FIELD_GET(ARM_SMMU_SMR_VALID, smr)) > + continue; > + smmu->smrs[i].id = FIELD_GET(ARM_SMMU_SMR_ID, smr); > + smmu->smrs[i].mask = FIELD_GET(ARM_SMMU_SMR_MASK, smr); > + smmu->smrs[i].valid = true; > + }[ 2.707729] arm-smmu: setting up rmr list on 0x4000 [ 2.712598] arm-smmu: s2crs count is 0 smrs index 0x0 [ 2.717638] arm-smmu: s2crs count is 0 smrs id is 0x4000 [ 2.722939] arm-smmu: s2crs count is 0 smrs mask is 0x8000 [ 2.728417] arm-smmu arm-smmu.0.auto: preserved 1 boot mapping > + } Then this code block was hit which is correct > + if (smmu->s2crs[i].count == 0) { > + smmu->smrs[i].id = sid; > + smmu->smrs[i].mask = ~0; > + smmu->smrs[i].valid = true; > + } The mask was causing the issue. If I think ammended that segment to read the mask as setup by the hardware everything was back to functioning both with and without disable_bypass set. Some debug from that section when it is working [ 2.712598] arm-smmu: s2crs count is 0 smrs index 0x0 [ 2.717638] arm-smmu: s2crs count is 0 smrs id is 0x4000 [ 2.722939] arm-smmu: s2crs count is 0 smrs mask is 0x8000 [ 2.728417] arm-smmu arm-smmu.0.auto: preserved 1 boot mapping Robin if anything jumps out at you let me know, otherwise I will debug further. -Jon From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5E15FC47082 for ; Thu, 3 Jun 2021 08:53:35 +0000 (UTC) Received: from smtp2.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 10056613E7 for ; Thu, 3 Jun 2021 08:53:35 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 10056613E7 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=solid-run.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=iommu-bounces@lists.linux-foundation.org Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id D721E40510; Thu, 3 Jun 2021 08:53:34 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id BkGGbh9cgRWG; Thu, 3 Jun 2021 08:53:33 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp2.osuosl.org (Postfix) with ESMTP id 9A606400DB; Thu, 3 Jun 2021 08:53:33 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 7604DC000D; Thu, 3 Jun 2021 08:53:33 +0000 (UTC) Received: from smtp1.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by lists.linuxfoundation.org (Postfix) with ESMTP id 5D951C0001 for ; Thu, 3 Jun 2021 08:53:32 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id 515E583E45 for ; Thu, 3 Jun 2021 08:53:32 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp1.osuosl.org (amavisd-new); dkim=pass (2048-bit key) header.d=solid-run-com.20150623.gappssmtp.com Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id C1pv4GNNZufO for ; Thu, 3 Jun 2021 08:53:31 +0000 (UTC) X-Greylist: whitelisted by SQLgrey-1.8.0 Received: from mail-vs1-xe32.google.com (mail-vs1-xe32.google.com [IPv6:2607:f8b0:4864:20::e32]) by smtp1.osuosl.org (Postfix) with ESMTPS id 52D0A83E1F for ; Thu, 3 Jun 2021 08:53:31 +0000 (UTC) Received: by mail-vs1-xe32.google.com with SMTP id i29so2591394vsr.11 for ; Thu, 03 Jun 2021 01:53:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=solid-run-com.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=Myemz5k16Nb16eM+VNAAZnl9WcdJHNpws7yrBRTSzQ8=; b=FIiLMwkjVLQwvIDeLZi02NT81EiYl8Vfp2TJ4+4aniF2IcsesHL+BydP/wXTPVWmgI SBAcJ2BAoTaOeIrGslJ5XmYP49OoR3Tu4uRa3pvDMs5SBYgCy+yG4VRr4v6lMaBYnDLc SKOeIG230/Eizv18tylHUlXlsoTt3u48K+zr+c9e31PVBp+K0OMj/IAUWRemEvdT+Or1 p1zgDyza8IIruANBM7NvSo8ggNoATA7iKA4qQO+XsS0UTBvAR6KHhUpDQo/0CcRjMH3n mOIamavU9Bjb/bEFlzcfgqPwYMjHD5VeKXFFJyQKqgskoI/jocKzYi8bTUUQCfuW9xer xung== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=Myemz5k16Nb16eM+VNAAZnl9WcdJHNpws7yrBRTSzQ8=; b=hQWXTaxCl9WRDDogT6DIzQ5gQfmpQJ/g5JIOTTyrh2npcMvaeqJbazUM+yqZm6sEgK QUSNp6jZza9Xlb1XRkag3eCxuNlmB7BP0pRgTgK2+uPQZUu4NxbjqsYlB3+rHurPGwmE /LY8Fwti4d/6CJhWO5Xwz+BGJ54k/xL8EAsQEvDzPwxKbccipcVwXyg9tsFUKpqUtWFg kjwx3dsT/ciT6XYBOLH7Cz2BDcFdP2CuMZC3rXaKxHXhg+ThQKbVkFTPSOEghe34Hrm5 jLoLUNtxpQgQXtNHAz9eUS73bwCeZd0LUckh2Gehze+la1i2msQMZyAUIN5FiylJUZGU Rz8g== X-Gm-Message-State: AOAM531M1Yq8UN4HxnbsD0R+rt66L5kVjQp2vMmy3Ju0e2PLQTpC04x0 giqadbPqHaQ/s0ent2QWGWnpnZdlFbYrLgCBufo7tQ== X-Google-Smtp-Source: ABdhPJxj+4IT8+6qr07qq87ubJ2nmPl8BZS/9F8+PlebID2YL/LATooVU+fmNI4LSTztODs2WUyyuopyqMoKE7EXvwM= X-Received: by 2002:a67:b406:: with SMTP id x6mr7014288vsl.10.1622710410065; Thu, 03 Jun 2021 01:53:30 -0700 (PDT) MIME-Version: 1.0 References: <20210524110222.2212-1-shameerali.kolothum.thodi@huawei.com> <20210524110222.2212-8-shameerali.kolothum.thodi@huawei.com> In-Reply-To: <20210524110222.2212-8-shameerali.kolothum.thodi@huawei.com> From: Jon Nettleton Date: Thu, 3 Jun 2021 10:52:53 +0200 Message-ID: Subject: Re: [PATCH v5 7/8] iommu/arm-smmu: Get associated RMR info and install bypass SMR To: Shameer Kolothum Cc: Linuxarm , Steven Price , ACPI Devel Maling List , iommu@lists.linux-foundation.org, wanghuiqiang , Hanjun Guo , yangyicong , Sami.Mujawar@arm.com, Robin Murphy , linux-arm-kernel X-BeenThere: iommu@lists.linux-foundation.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Development issues for Linux IOMMU support List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: iommu-bounces@lists.linux-foundation.org Sender: "iommu" On Mon, May 24, 2021 at 1:04 PM Shameer Kolothum wrote: > > From: Jon Nettleton > > Check if there is any RMR info associated with the devices behind > the SMMU and if any, install bypass SMRs for them. This is to > keep any ongoing traffic associated with these devices alive > when we enable/reset SMMU during probe(). > > Signed-off-by: Jon Nettleton > Signed-off-by: Steven Price > Signed-off-by: Shameer Kolothum > --- > drivers/iommu/arm/arm-smmu/arm-smmu.c | 65 +++++++++++++++++++++++++++ > 1 file changed, 65 insertions(+) > > diff --git a/drivers/iommu/arm/arm-smmu/arm-smmu.c b/drivers/iommu/arm/arm-smmu/arm-smmu.c > index 6f72c4d208ca..56db3d3238fc 100644 > --- a/drivers/iommu/arm/arm-smmu/arm-smmu.c > +++ b/drivers/iommu/arm/arm-smmu/arm-smmu.c > @@ -2042,6 +2042,67 @@ err_reset_platform_ops: __maybe_unused; > return err; > } > > +static void arm_smmu_rmr_install_bypass_smr(struct arm_smmu_device *smmu) > +{ > + struct list_head rmr_list; > + struct iommu_resv_region *e; > + int i, cnt = 0; > + u32 smr; > + u32 reg; > + > + INIT_LIST_HEAD(&rmr_list); > + if (iommu_dma_get_rmrs(dev_fwnode(smmu->dev), &rmr_list)) > + return; > + > + reg = arm_smmu_gr0_read(smmu, ARM_SMMU_GR0_sCR0); > + > + if ((reg & ARM_SMMU_sCR0_USFCFG) && !(reg & ARM_SMMU_sCR0_CLIENTPD)) { > + /* > + * SMMU is already enabled and disallowing bypass, so preserve > + * the existing SMRs > + */ > + for (i = 0; i < smmu->num_mapping_groups; i++) { > + smr = arm_smmu_gr0_read(smmu, ARM_SMMU_GR0_SMR(i)); > + if (!FIELD_GET(ARM_SMMU_SMR_VALID, smr)) > + continue; > + smmu->smrs[i].id = FIELD_GET(ARM_SMMU_SMR_ID, smr); > + smmu->smrs[i].mask = FIELD_GET(ARM_SMMU_SMR_MASK, smr); > + smmu->smrs[i].valid = true; > + } > + } > + > + list_for_each_entry(e, &rmr_list, list) { > + u32 sid = e->fw_data.rmr.sid; > + > + i = arm_smmu_find_sme(smmu, sid, ~0); > + if (i < 0) > + continue; > + if (smmu->s2crs[i].count == 0) { > + smmu->smrs[i].id = sid; > + smmu->smrs[i].mask = ~0; > + smmu->smrs[i].valid = true; > + } > + smmu->s2crs[i].count++; > + smmu->s2crs[i].type = S2CR_TYPE_BYPASS; > + smmu->s2crs[i].privcfg = S2CR_PRIVCFG_DEFAULT; > + smmu->s2crs[i].cbndx = 0xff; > + > + cnt++; > + } > + > + if ((reg & ARM_SMMU_sCR0_USFCFG) && !(reg & ARM_SMMU_sCR0_CLIENTPD)) { > + /* Remove the valid bit for unused SMRs */ > + for (i = 0; i < smmu->num_mapping_groups; i++) { > + if (smmu->s2crs[i].count == 0) > + smmu->smrs[i].valid = false; > + } > + } > + > + dev_notice(smmu->dev, "\tpreserved %d boot mapping%s\n", cnt, > + cnt == 1 ? "" : "s"); > + iommu_dma_put_rmrs(dev_fwnode(smmu->dev), &rmr_list); > +} > + > static int arm_smmu_device_probe(struct platform_device *pdev) > { > struct resource *res; > @@ -2168,6 +2229,10 @@ static int arm_smmu_device_probe(struct platform_device *pdev) > } > > platform_set_drvdata(pdev, smmu); > + > + /* Check for RMRs and install bypass SMRs if any */ > + arm_smmu_rmr_install_bypass_smr(smmu); > + > arm_smmu_device_reset(smmu); > arm_smmu_test_smr_masks(smmu); > > -- > 2.17.1 > Shameer and Robin I finally got around to updating edk2 and the HoneyComb IORT tables to reflect the new standards. Out of the box the new patchset was generating errors immediatly after the smmu bringup. arm-smmu arm-smmu.0.auto: Unhandled context fault: fsr=0x402, iova=0x2080000140, fsynr=0x1d0040, cbfrsynra=0x4000, cb=0 These errors were generated even with disable_bypass=0 I tracked down the issue to This code is skipped as Robin said would be correct > + if ((reg & ARM_SMMU_sCR0_USFCFG) && !(reg & ARM_SMMU_sCR0_CLIENTPD)) { > + /* > + * SMMU is already enabled and disallowing bypass, so preserve > + * the existing SMRs > + */ > + for (i = 0; i < smmu->num_mapping_groups; i++) { > + smr = arm_smmu_gr0_read(smmu, ARM_SMMU_GR0_SMR(i)); > + if (!FIELD_GET(ARM_SMMU_SMR_VALID, smr)) > + continue; > + smmu->smrs[i].id = FIELD_GET(ARM_SMMU_SMR_ID, smr); > + smmu->smrs[i].mask = FIELD_GET(ARM_SMMU_SMR_MASK, smr); > + smmu->smrs[i].valid = true; > + }[ 2.707729] arm-smmu: setting up rmr list on 0x4000 [ 2.712598] arm-smmu: s2crs count is 0 smrs index 0x0 [ 2.717638] arm-smmu: s2crs count is 0 smrs id is 0x4000 [ 2.722939] arm-smmu: s2crs count is 0 smrs mask is 0x8000 [ 2.728417] arm-smmu arm-smmu.0.auto: preserved 1 boot mapping > + } Then this code block was hit which is correct > + if (smmu->s2crs[i].count == 0) { > + smmu->smrs[i].id = sid; > + smmu->smrs[i].mask = ~0; > + smmu->smrs[i].valid = true; > + } The mask was causing the issue. If I think ammended that segment to read the mask as setup by the hardware everything was back to functioning both with and without disable_bypass set. Some debug from that section when it is working [ 2.712598] arm-smmu: s2crs count is 0 smrs index 0x0 [ 2.717638] arm-smmu: s2crs count is 0 smrs id is 0x4000 [ 2.722939] arm-smmu: s2crs count is 0 smrs mask is 0x8000 [ 2.728417] arm-smmu arm-smmu.0.auto: preserved 1 boot mapping Robin if anything jumps out at you let me know, otherwise I will debug further. -Jon _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-14.1 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id F3EF4C47098 for ; Thu, 3 Jun 2021 08:55:21 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id C472D613DE for ; Thu, 3 Jun 2021 08:55:21 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org C472D613DE Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=solid-run.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:Subject:Message-ID:Date:From: In-Reply-To:References:MIME-Version:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=ORl0DBKo0oLCaBojrlhhqeNSvMq60lBZ9P9RnTkr8ms=; b=F2Vxm4PaZa2AdW KM8JXLpDGoIWfbHjhU4w6Rl5bR4ZegPlop6u52L4EfQJIqR+Zq53LzcrQUOUBZqQmjN2HHvMSueWE qFglzEfsb7bgwjZ/o10Ps/4/alIuwF1z2p4/JBc1Ao4W9+FuNGJFj9jayGaNWtXn+9oBelBImVZPq h5umc5bQ7py6wjyk9qBCeODm3KWioI/TuyIVbFCAzNqKkgK3nlRJqPpyaZ4bv4esHbuz45FVDJQRJ tozZNPWw0CgQhtmFHsu67eBlTWdXHOik1IPPYvk0aNx6YC25GGMWBKZ1ovqMI41wcufzDh8SFuwYZ wjJGKCxD3GX9ikH/Cl1A==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1loj6i-007nCU-9o; Thu, 03 Jun 2021 08:53:40 +0000 Received: from mail-vs1-xe31.google.com ([2607:f8b0:4864:20::e31]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1loj6a-007n9f-WE for linux-arm-kernel@lists.infradead.org; Thu, 03 Jun 2021 08:53:35 +0000 Received: by mail-vs1-xe31.google.com with SMTP id s22so2587558vsl.10 for ; Thu, 03 Jun 2021 01:53:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=solid-run-com.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=Myemz5k16Nb16eM+VNAAZnl9WcdJHNpws7yrBRTSzQ8=; b=FIiLMwkjVLQwvIDeLZi02NT81EiYl8Vfp2TJ4+4aniF2IcsesHL+BydP/wXTPVWmgI SBAcJ2BAoTaOeIrGslJ5XmYP49OoR3Tu4uRa3pvDMs5SBYgCy+yG4VRr4v6lMaBYnDLc SKOeIG230/Eizv18tylHUlXlsoTt3u48K+zr+c9e31PVBp+K0OMj/IAUWRemEvdT+Or1 p1zgDyza8IIruANBM7NvSo8ggNoATA7iKA4qQO+XsS0UTBvAR6KHhUpDQo/0CcRjMH3n mOIamavU9Bjb/bEFlzcfgqPwYMjHD5VeKXFFJyQKqgskoI/jocKzYi8bTUUQCfuW9xer xung== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=Myemz5k16Nb16eM+VNAAZnl9WcdJHNpws7yrBRTSzQ8=; b=crCv3m50iW+FFz8rAYHGeSrQThcscStn3E0cb/yNV8siraE4wz713i5hanpP+Rq/AT oKXe0dZxEODfDhVjV1stui4cALKZsmAsoSndZohNusnK7mPoBZuSSWkB5fWVpt60dc8R nhVNRNmgU6pBKNJQXSqczGMCZ0ahFjg7nx4nj1NrkPn0nr9YGoy42sabOVh8W4orwWIl 2686kcOjnsFpWMLrgy6pyIVr8X/Pk+idRLXpt2kICQAzhHHzxpI3sQreKj5ydWr3RuPg AgkCuBa4uq1c4P/xkTTLDdW8GIabWdVpkNV0czarFMb1sYHMiRSZZYMZT/fKX2NWB08z 9h2g== X-Gm-Message-State: AOAM533EmCPvYMgRLmRGDxKwUXthDSm2lxLwShf+8sweHRQC54GzQnBI mlxAGQ9/94NZlBFEQoFLJwiX+tv9b97nvNYTNHwhxQ== X-Google-Smtp-Source: ABdhPJxj+4IT8+6qr07qq87ubJ2nmPl8BZS/9F8+PlebID2YL/LATooVU+fmNI4LSTztODs2WUyyuopyqMoKE7EXvwM= X-Received: by 2002:a67:b406:: with SMTP id x6mr7014288vsl.10.1622710410065; Thu, 03 Jun 2021 01:53:30 -0700 (PDT) MIME-Version: 1.0 References: <20210524110222.2212-1-shameerali.kolothum.thodi@huawei.com> <20210524110222.2212-8-shameerali.kolothum.thodi@huawei.com> In-Reply-To: <20210524110222.2212-8-shameerali.kolothum.thodi@huawei.com> From: Jon Nettleton Date: Thu, 3 Jun 2021 10:52:53 +0200 Message-ID: Subject: Re: [PATCH v5 7/8] iommu/arm-smmu: Get associated RMR info and install bypass SMR To: Shameer Kolothum Cc: linux-arm-kernel , ACPI Devel Maling List , iommu@lists.linux-foundation.org, Linuxarm , Lorenzo Pieralisi , joro@8bytes.org, Robin Murphy , wanghuiqiang , Hanjun Guo , Steven Price , Sami.Mujawar@arm.com, eric.auger@redhat.com, yangyicong X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210603_015333_959421_741470C4 X-CRM114-Status: GOOD ( 32.93 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Mon, May 24, 2021 at 1:04 PM Shameer Kolothum wrote: > > From: Jon Nettleton > > Check if there is any RMR info associated with the devices behind > the SMMU and if any, install bypass SMRs for them. This is to > keep any ongoing traffic associated with these devices alive > when we enable/reset SMMU during probe(). > > Signed-off-by: Jon Nettleton > Signed-off-by: Steven Price > Signed-off-by: Shameer Kolothum > --- > drivers/iommu/arm/arm-smmu/arm-smmu.c | 65 +++++++++++++++++++++++++++ > 1 file changed, 65 insertions(+) > > diff --git a/drivers/iommu/arm/arm-smmu/arm-smmu.c b/drivers/iommu/arm/arm-smmu/arm-smmu.c > index 6f72c4d208ca..56db3d3238fc 100644 > --- a/drivers/iommu/arm/arm-smmu/arm-smmu.c > +++ b/drivers/iommu/arm/arm-smmu/arm-smmu.c > @@ -2042,6 +2042,67 @@ err_reset_platform_ops: __maybe_unused; > return err; > } > > +static void arm_smmu_rmr_install_bypass_smr(struct arm_smmu_device *smmu) > +{ > + struct list_head rmr_list; > + struct iommu_resv_region *e; > + int i, cnt = 0; > + u32 smr; > + u32 reg; > + > + INIT_LIST_HEAD(&rmr_list); > + if (iommu_dma_get_rmrs(dev_fwnode(smmu->dev), &rmr_list)) > + return; > + > + reg = arm_smmu_gr0_read(smmu, ARM_SMMU_GR0_sCR0); > + > + if ((reg & ARM_SMMU_sCR0_USFCFG) && !(reg & ARM_SMMU_sCR0_CLIENTPD)) { > + /* > + * SMMU is already enabled and disallowing bypass, so preserve > + * the existing SMRs > + */ > + for (i = 0; i < smmu->num_mapping_groups; i++) { > + smr = arm_smmu_gr0_read(smmu, ARM_SMMU_GR0_SMR(i)); > + if (!FIELD_GET(ARM_SMMU_SMR_VALID, smr)) > + continue; > + smmu->smrs[i].id = FIELD_GET(ARM_SMMU_SMR_ID, smr); > + smmu->smrs[i].mask = FIELD_GET(ARM_SMMU_SMR_MASK, smr); > + smmu->smrs[i].valid = true; > + } > + } > + > + list_for_each_entry(e, &rmr_list, list) { > + u32 sid = e->fw_data.rmr.sid; > + > + i = arm_smmu_find_sme(smmu, sid, ~0); > + if (i < 0) > + continue; > + if (smmu->s2crs[i].count == 0) { > + smmu->smrs[i].id = sid; > + smmu->smrs[i].mask = ~0; > + smmu->smrs[i].valid = true; > + } > + smmu->s2crs[i].count++; > + smmu->s2crs[i].type = S2CR_TYPE_BYPASS; > + smmu->s2crs[i].privcfg = S2CR_PRIVCFG_DEFAULT; > + smmu->s2crs[i].cbndx = 0xff; > + > + cnt++; > + } > + > + if ((reg & ARM_SMMU_sCR0_USFCFG) && !(reg & ARM_SMMU_sCR0_CLIENTPD)) { > + /* Remove the valid bit for unused SMRs */ > + for (i = 0; i < smmu->num_mapping_groups; i++) { > + if (smmu->s2crs[i].count == 0) > + smmu->smrs[i].valid = false; > + } > + } > + > + dev_notice(smmu->dev, "\tpreserved %d boot mapping%s\n", cnt, > + cnt == 1 ? "" : "s"); > + iommu_dma_put_rmrs(dev_fwnode(smmu->dev), &rmr_list); > +} > + > static int arm_smmu_device_probe(struct platform_device *pdev) > { > struct resource *res; > @@ -2168,6 +2229,10 @@ static int arm_smmu_device_probe(struct platform_device *pdev) > } > > platform_set_drvdata(pdev, smmu); > + > + /* Check for RMRs and install bypass SMRs if any */ > + arm_smmu_rmr_install_bypass_smr(smmu); > + > arm_smmu_device_reset(smmu); > arm_smmu_test_smr_masks(smmu); > > -- > 2.17.1 > Shameer and Robin I finally got around to updating edk2 and the HoneyComb IORT tables to reflect the new standards. Out of the box the new patchset was generating errors immediatly after the smmu bringup. arm-smmu arm-smmu.0.auto: Unhandled context fault: fsr=0x402, iova=0x2080000140, fsynr=0x1d0040, cbfrsynra=0x4000, cb=0 These errors were generated even with disable_bypass=0 I tracked down the issue to This code is skipped as Robin said would be correct > + if ((reg & ARM_SMMU_sCR0_USFCFG) && !(reg & ARM_SMMU_sCR0_CLIENTPD)) { > + /* > + * SMMU is already enabled and disallowing bypass, so preserve > + * the existing SMRs > + */ > + for (i = 0; i < smmu->num_mapping_groups; i++) { > + smr = arm_smmu_gr0_read(smmu, ARM_SMMU_GR0_SMR(i)); > + if (!FIELD_GET(ARM_SMMU_SMR_VALID, smr)) > + continue; > + smmu->smrs[i].id = FIELD_GET(ARM_SMMU_SMR_ID, smr); > + smmu->smrs[i].mask = FIELD_GET(ARM_SMMU_SMR_MASK, smr); > + smmu->smrs[i].valid = true; > + }[ 2.707729] arm-smmu: setting up rmr list on 0x4000 [ 2.712598] arm-smmu: s2crs count is 0 smrs index 0x0 [ 2.717638] arm-smmu: s2crs count is 0 smrs id is 0x4000 [ 2.722939] arm-smmu: s2crs count is 0 smrs mask is 0x8000 [ 2.728417] arm-smmu arm-smmu.0.auto: preserved 1 boot mapping > + } Then this code block was hit which is correct > + if (smmu->s2crs[i].count == 0) { > + smmu->smrs[i].id = sid; > + smmu->smrs[i].mask = ~0; > + smmu->smrs[i].valid = true; > + } The mask was causing the issue. If I think ammended that segment to read the mask as setup by the hardware everything was back to functioning both with and without disable_bypass set. Some debug from that section when it is working [ 2.712598] arm-smmu: s2crs count is 0 smrs index 0x0 [ 2.717638] arm-smmu: s2crs count is 0 smrs id is 0x4000 [ 2.722939] arm-smmu: s2crs count is 0 smrs mask is 0x8000 [ 2.728417] arm-smmu arm-smmu.0.auto: preserved 1 boot mapping Robin if anything jumps out at you let me know, otherwise I will debug further. -Jon _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel