From: Robin Murphy <robin.murphy@arm.com>
To: Jon Nettleton <jon@solid-run.com>, Steven Price <steven.price@arm.com>
Cc: Shameer Kolothum <shameerali.kolothum.thodi@huawei.com>,
linux-arm-kernel <linux-arm-kernel@lists.infradead.org>,
ACPI Devel Maling List <linux-acpi@vger.kernel.org>,
iommu@lists.linux-foundation.org, Linuxarm <linuxarm@huawei.com>,
Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>,
joro@8bytes.org, wanghuiqiang <wanghuiqiang@huawei.com>,
Hanjun Guo <guohanjun@huawei.com>,
Sami.Mujawar@arm.com, eric.auger@redhat.com,
yangyicong <yangyicong@huawei.com>
Subject: Re: [PATCH v5 7/8] iommu/arm-smmu: Get associated RMR info and install bypass SMR
Date: Mon, 14 Jun 2021 10:23:19 +0100 [thread overview]
Message-ID: <729b1d4a-a63c-15ef-ef05-8be59f1da118@arm.com> (raw)
In-Reply-To: <CABdtJHtj0UVQMCEJKeN5gb=kyRgZg7qiD9vqg_9qytVMVRFbGQ@mail.gmail.com>
On 2021-06-13 08:40, Jon Nettleton wrote:
> On Thu, Jun 3, 2021 at 1:51 PM Jon Nettleton <jon@solid-run.com> wrote:
>>
>> On Thu, Jun 3, 2021 at 1:27 PM Steven Price <steven.price@arm.com> wrote:
>>>
>>> On 03/06/2021 09:52, Jon Nettleton wrote:
>>>> On Mon, May 24, 2021 at 1:04 PM Shameer Kolothum
>>>> <shameerali.kolothum.thodi@huawei.com> wrote:
>>>>>
>>>>> From: Jon Nettleton <jon@solid-run.com>
>>>>>
>>>>> Check if there is any RMR info associated with the devices behind
>>>>> the SMMU and if any, install bypass SMRs for them. This is to
>>>>> keep any ongoing traffic associated with these devices alive
>>>>> when we enable/reset SMMU during probe().
>>>>>
>>>>> Signed-off-by: Jon Nettleton <jon@solid-run.com>
>>>>> Signed-off-by: Steven Price <steven.price@arm.com>
>>>>> Signed-off-by: Shameer Kolothum <shameerali.kolothum.thodi@huawei.com>
>>>>> ---
>>>>> drivers/iommu/arm/arm-smmu/arm-smmu.c | 65 +++++++++++++++++++++++++++
>>>>> 1 file changed, 65 insertions(+)
>>>>>
>>>>> diff --git a/drivers/iommu/arm/arm-smmu/arm-smmu.c b/drivers/iommu/arm/arm-smmu/arm-smmu.c
>>>>> index 6f72c4d208ca..56db3d3238fc 100644
>>>>> --- a/drivers/iommu/arm/arm-smmu/arm-smmu.c
>>>>> +++ b/drivers/iommu/arm/arm-smmu/arm-smmu.c
>>>>> @@ -2042,6 +2042,67 @@ err_reset_platform_ops: __maybe_unused;
>>>>> return err;
>>>>> }
>>>>>
>>>>> +static void arm_smmu_rmr_install_bypass_smr(struct arm_smmu_device *smmu)
>>>>> +{
>>>>> + struct list_head rmr_list;
>>>>> + struct iommu_resv_region *e;
>>>>> + int i, cnt = 0;
>>>>> + u32 smr;
>>>>> + u32 reg;
>>>>> +
>>>>> + INIT_LIST_HEAD(&rmr_list);
>>>>> + if (iommu_dma_get_rmrs(dev_fwnode(smmu->dev), &rmr_list))
>>>>> + return;
>>>>> +
>>>>> + reg = arm_smmu_gr0_read(smmu, ARM_SMMU_GR0_sCR0);
>>>>> +
>>>>> + if ((reg & ARM_SMMU_sCR0_USFCFG) && !(reg & ARM_SMMU_sCR0_CLIENTPD)) {
>>>>> + /*
>>>>> + * SMMU is already enabled and disallowing bypass, so preserve
>>>>> + * the existing SMRs
>>>>> + */
>>>>> + for (i = 0; i < smmu->num_mapping_groups; i++) {
>>>>> + smr = arm_smmu_gr0_read(smmu, ARM_SMMU_GR0_SMR(i));
>>>>> + if (!FIELD_GET(ARM_SMMU_SMR_VALID, smr))
>>>>> + continue;
>>>>> + smmu->smrs[i].id = FIELD_GET(ARM_SMMU_SMR_ID, smr);
>>>>> + smmu->smrs[i].mask = FIELD_GET(ARM_SMMU_SMR_MASK, smr);
>>>>> + smmu->smrs[i].valid = true;
>>>>> + }
>>>>> + }
>>>>> +
>>>>> + list_for_each_entry(e, &rmr_list, list) {
>>>>> + u32 sid = e->fw_data.rmr.sid;
>>>>> +
>>>>> + i = arm_smmu_find_sme(smmu, sid, ~0);
>>>>> + if (i < 0)
>>>>> + continue;
>>>>> + if (smmu->s2crs[i].count == 0) {
>>>>> + smmu->smrs[i].id = sid;
>>>>> + smmu->smrs[i].mask = ~0;
>>>
>>> Looking at this code again, that mask looks wrong! According to the SMMU
>>> spec MASK[i]==1 means ID[i] is ignored. I.e. this means completely
>>> ignore the ID...
>>>
>>> I'm not at all sure why they designed the hardware that way round.
>>>
>>>>> + smmu->smrs[i].valid = true;
>>>>> + }
>>>>> + smmu->s2crs[i].count++;
>>>>> + smmu->s2crs[i].type = S2CR_TYPE_BYPASS;
>>>>> + smmu->s2crs[i].privcfg = S2CR_PRIVCFG_DEFAULT;
>>>>> + smmu->s2crs[i].cbndx = 0xff;
>>>>> +
>>>>> + cnt++;
>>>>> + }
>>>>> +
>>>>> + if ((reg & ARM_SMMU_sCR0_USFCFG) && !(reg & ARM_SMMU_sCR0_CLIENTPD)) {
>>>>> + /* Remove the valid bit for unused SMRs */
>>>>> + for (i = 0; i < smmu->num_mapping_groups; i++) {
>>>>> + if (smmu->s2crs[i].count == 0)
>>>>> + smmu->smrs[i].valid = false;
>>>>> + }
>>>>> + }
>>>>> +
>>>>> + dev_notice(smmu->dev, "\tpreserved %d boot mapping%s\n", cnt,
>>>>> + cnt == 1 ? "" : "s");
>>>>> + iommu_dma_put_rmrs(dev_fwnode(smmu->dev), &rmr_list);
>>>>> +}
>>>>> +
>>>>> static int arm_smmu_device_probe(struct platform_device *pdev)
>>>>> {
>>>>> struct resource *res;
>>>>> @@ -2168,6 +2229,10 @@ static int arm_smmu_device_probe(struct platform_device *pdev)
>>>>> }
>>>>>
>>>>> platform_set_drvdata(pdev, smmu);
>>>>> +
>>>>> + /* Check for RMRs and install bypass SMRs if any */
>>>>> + arm_smmu_rmr_install_bypass_smr(smmu);
>>>>> +
>>>>> arm_smmu_device_reset(smmu);
>>>>> arm_smmu_test_smr_masks(smmu);
>>>>>
>>>>> --
>>>>> 2.17.1
>>>>>
>>>>
>>>> Shameer and Robin
>>>>
>>>> I finally got around to updating edk2 and the HoneyComb IORT tables to
>>>> reflect the new standards.
>>>> Out of the box the new patchset was generating errors immediatly after
>>>> the smmu bringup.
>>>>
>>>> arm-smmu arm-smmu.0.auto: Unhandled context fault: fsr=0x402, iova=0x2080000140,
>>>> fsynr=0x1d0040, cbfrsynra=0x4000, cb=0
>>>>
>>>> These errors were generated even with disable_bypass=0
>>>>
>>>> I tracked down the issue to
>>>>
>>>> This code is skipped as Robin said would be correct
>>>
>>> If you're skipping the first bit of code, then that (hopefully) means
>>> that the SMMU is disabled...
>>>
>>>>> + if ((reg & ARM_SMMU_sCR0_USFCFG) && !(reg & ARM_SMMU_sCR0_CLIENTPD)) {
>>>>> + /*
>>>>> + * SMMU is already enabled and disallowing bypass, so preserve
>>>>> + * the existing SMRs
>>>>> + */
>>>>> + for (i = 0; i < smmu->num_mapping_groups; i++) {
>>>>> + smr = arm_smmu_gr0_read(smmu, ARM_SMMU_GR0_SMR(i));
>>>>> + if (!FIELD_GET(ARM_SMMU_SMR_VALID, smr))
>>>>> + continue;
>>>>> + smmu->smrs[i].id = FIELD_GET(ARM_SMMU_SMR_ID, smr);
>>>>> + smmu->smrs[i].mask = FIELD_GET(ARM_SMMU_SMR_MASK, smr);
>>>>> + smmu->smrs[i].valid = true;
>>>>> + }[ 2.707729] arm-smmu: setting up rmr list on 0x4000
>>>> [ 2.712598] arm-smmu: s2crs count is 0 smrs index 0x0
>>>> [ 2.717638] arm-smmu: s2crs count is 0 smrs id is 0x4000
>>>> [ 2.722939] arm-smmu: s2crs count is 0 smrs mask is 0x8000
>>>> [ 2.728417] arm-smmu arm-smmu.0.auto: preserved 1 boot mapping
>>>>
>>>>> + }
>>>>
>>>> Then this code block was hit which is correct
>>>>
>>>>> + if (smmu->s2crs[i].count == 0) {
>>>>> + smmu->smrs[i].id = sid;
>>>>> + smmu->smrs[i].mask = ~0;
>>>>> + smmu->smrs[i].valid = true;
>>>>> + }
>>>>
>>>> The mask was causing the issue. If I think ammended that segment to read
>>>> the mask as setup by the hardware everything was back to functioning both
>>>> with and without disable_bypass set.
>>>
>>> ...so reading a mask from it doesn't sounds quite right.
>>>
>>> Can you have a go with a corrected mask of '0' rather than all-1s and
>>> see if that helps? My guess is that the mask of ~0 was causing multiple
>>> matches in the SMRs which is 'UNPREDICTABLE'.
>>>
>>> Sadly in my test setup there's only the one device behind the SMMU so
>>> I didn't spot this (below patch works for me, but that's not saying
>>> much).
>>>
>>> Steve
>>>
>>> --->8---
>>> diff --git a/drivers/iommu/arm/arm-smmu/arm-smmu.c b/drivers/iommu/arm/arm-smmu/arm-smmu.c
>>> index 56db3d3238fc..44831d0c78e4 100644
>>> --- a/drivers/iommu/arm/arm-smmu/arm-smmu.c
>>> +++ b/drivers/iommu/arm/arm-smmu/arm-smmu.c
>>> @@ -2079,7 +2079,7 @@ static void arm_smmu_rmr_install_bypass_smr(struct arm_smmu_device *smmu)
>>> continue;
>>> if (smmu->s2crs[i].count == 0) {
>>> smmu->smrs[i].id = sid;
>>> - smmu->smrs[i].mask = ~0;
>>> + smmu->smrs[i].mask = 0;
>>> smmu->smrs[i].valid = true;
>>> }
>>> smmu->s2crs[i].count++;
>>
>> Yes this works fine. Thanks
>
> Shameer,
>
> Can you pick up this change into your next patch set? Also are there
> any objections to adding this to the SMMUv2 code from the maintainers?
Urgh, I was rather confused here since I knew I'd already written a
review of an earlier version pointing this out along with a couple of
other issues... then I found it still sat in my drafts folder :(
Let me just "rebase" those comments to v5...
Robin.
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
next prev parent reply other threads:[~2021-06-14 9:25 UTC|newest]
Thread overview: 40+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-05-24 11:02 [PATCH v5 0/8] ACPI/IORT: Support for IORT RMR node Shameer Kolothum
2021-05-24 11:02 ` [PATCH v5 1/8] ACPI/IORT: Add support for RMR node parsing Shameer Kolothum
2021-06-14 11:14 ` Robin Murphy
2021-06-14 12:37 ` Shameerali Kolothum Thodi
2021-05-24 11:02 ` [PATCH v5 2/8] iommu/dma: Introduce generic helper to retrieve RMR info Shameer Kolothum
2021-05-24 15:35 ` kernel test robot
2021-05-24 11:02 ` [PATCH v5 3/8] ACPI/IORT: Add a helper to retrieve RMR memory regions Shameer Kolothum
2021-05-26 7:53 ` Laurentiu Tudor
2021-05-26 16:36 ` Shameerali Kolothum Thodi
2021-05-26 17:11 ` Laurentiu Tudor
2021-06-03 12:27 ` Jon Nettleton
2021-06-03 12:32 ` Laurentiu Tudor
2021-05-27 4:25 ` Jon Nettleton
2021-06-14 10:35 ` Robin Murphy
2021-06-14 11:23 ` Robin Murphy
2021-06-14 12:49 ` Shameerali Kolothum Thodi
2021-06-29 17:34 ` Jon Nettleton
2021-07-04 7:38 ` Jon Nettleton
2021-07-05 9:10 ` Shameerali Kolothum Thodi
2021-05-24 11:02 ` [PATCH v5 4/8] iommu/arm-smmu-v3: Introduce strtab init helper Shameer Kolothum
2021-05-24 11:02 ` [PATCH v5 5/8] iommu/arm-smmu-v3: Add bypass flag to arm_smmu_write_strtab_ent() Shameer Kolothum
2021-06-14 10:23 ` Robin Murphy
2021-06-14 12:51 ` Shameerali Kolothum Thodi
2021-05-24 11:02 ` [PATCH v5 6/8] iommu/arm-smmu-v3: Get associated RMR info and install Shameer Kolothum
2021-06-14 10:15 ` Robin Murphy
2021-05-24 11:02 ` [PATCH v5 7/8] iommu/arm-smmu: Get associated RMR info and install bypass SMR Shameer Kolothum
2021-06-03 8:52 ` Jon Nettleton
2021-06-03 11:27 ` Steven Price
2021-06-03 11:51 ` Jon Nettleton
2021-06-13 7:40 ` Jon Nettleton
2021-06-14 9:23 ` Robin Murphy [this message]
2021-06-14 10:06 ` Robin Murphy
2021-06-14 16:51 ` Shameerali Kolothum Thodi
2021-06-15 8:02 ` Jon Nettleton
2021-06-29 7:03 ` Jon Nettleton
2021-06-29 13:22 ` Robin Murphy
2021-06-29 16:25 ` Jon Nettleton
2021-06-30 8:50 ` Shameerali Kolothum Thodi
2021-05-24 11:02 ` [PATCH v5 8/8] iommu/dma: Reserve any RMR regions associated with a dev Shameer Kolothum
2021-05-24 15:18 ` [PATCH v5 0/8] ACPI/IORT: Support for IORT RMR node Steven Price
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=729b1d4a-a63c-15ef-ef05-8be59f1da118@arm.com \
--to=robin.murphy@arm.com \
--cc=Sami.Mujawar@arm.com \
--cc=eric.auger@redhat.com \
--cc=guohanjun@huawei.com \
--cc=iommu@lists.linux-foundation.org \
--cc=jon@solid-run.com \
--cc=joro@8bytes.org \
--cc=linux-acpi@vger.kernel.org \
--cc=linux-arm-kernel@lists.infradead.org \
--cc=linuxarm@huawei.com \
--cc=lorenzo.pieralisi@arm.com \
--cc=shameerali.kolothum.thodi@huawei.com \
--cc=steven.price@arm.com \
--cc=wanghuiqiang@huawei.com \
--cc=yangyicong@huawei.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).