Need help to debug Arm SMMU issue seen on guests with passthrough devices

Need help to debug Arm SMMU issue seen on guests with passthrough devices

[Ayan Kumar Halder]

Hi Arm/SMMU experts,

Scenario :- I am trying to assign a device (eg mmc) to a guest which 
uses smmu. I start the guest using "xl create -c ...". It works fine for 
the first time. I am able to access the device.

Now, when I destroy the guest and create again, I see this issue

(XEN) smmu: /axi/smmu@fd800000:         GFSR 0x00000002, GFSYNR0 
0x00000000, GFSYNR1 0x00000871, GFSYNR2 0x00000000

0x871 is the stream id used by mmc.

We are using SMMU-500. So from the Arm SMMU v2 specs, it is 
"Unidentified stream fault" for stream id "0x871".

This issue is seen regardless we use 'mmu-masters' or 'iommus' property 
in xen device tree.

I can confirm that this issue was not seen in xen-4.14.3. It is seen in 
our (Xilinx's) latest release ie xen-4.16.1.

I am using our downstream fork of Xen. I can't readily use the upstream 
as we use lot of EEMI codebase.


Can someone try to see if this issue exists in the latest upstream Xen ?

Any pointers on how to debug this further ?

Kind regards,

Ayan

Re: Need help to debug Arm SMMU issue seen on guests with passthrough devices

[Jan Beulich]

On 21.04.2022 13:12, Ayan Kumar Halder wrote:
> Hi Arm/SMMU experts,
> 
> Scenario :- I am trying to assign a device (eg mmc) to a guest which 
> uses smmu. I start the guest using "xl create -c ...". It works fine for 
> the first time. I am able to access the device.
> 
> Now, when I destroy the guest and create again, I see this issue
> 
> (XEN) smmu: /axi/smmu@fd800000:         GFSR 0x00000002, GFSYNR0 
> 0x00000000, GFSYNR1 0x00000871, GFSYNR2 0x00000000
> 
> 0x871 is the stream id used by mmc.
> 
> We are using SMMU-500. So from the Arm SMMU v2 specs, it is 
> "Unidentified stream fault" for stream id "0x871".

I have no clue what any of this means. I wonder how I ended up on your
To: list. I'd like to ask everyone to drop me from the recipients list
in their replies.

Jan

> This issue is seen regardless we use 'mmu-masters' or 'iommus' property 
> in xen device tree.
> 
> I can confirm that this issue was not seen in xen-4.14.3. It is seen in 
> our (Xilinx's) latest release ie xen-4.16.1.
> 
> I am using our downstream fork of Xen. I can't readily use the upstream 
> as we use lot of EEMI codebase.
> 
> 
> Can someone try to see if this issue exists in the latest upstream Xen ?
> 
> Any pointers on how to debug this further ?
> 
> Kind regards,
> 
> Ayan
> 
> 
> 
> 
> 

Re: Need help to debug Arm SMMU issue seen on guests with passthrough devices

[Rahul Singh]

[-- Attachment #1.1: Type: text/plain, Size: 1141 bytes --]

Hi Ayan,

> On 21 Apr 2022, at 12:12 pm, Ayan Kumar Halder <ayan.kumar.halder@xilinx.com> wrote:
>
> Hi Arm/SMMU experts,
>
> Scenario :- I am trying to assign a device (eg mmc) to a guest which uses smmu. I start the guest using "xl create -c ...". It works fine for the first time. I am able to access the device.
>
> Now, when I destroy the guest and create again, I see this issue

Did you observe any other Xen error while deassigning and assigning the device ?
>
> (XEN) smmu: /axi/smmu@fd800000:         GFSR 0x00000002, GFSYNR0 0x00000000, GFSYNR1 0x00000871, GFSYNR2 0x00000000
>
> 0x871 is the stream id used by mmc.
>
> We are using SMMU-500. So from the Arm SMMU v2 specs, it is "Unidentified stream fault" for stream id "0x871".

Yes, you are right above error means that Unidentified stream fault. That means SMMU hardware is not set up for streamId 0x0000087
I just try to go through the SMMU code and observe that this issue may be related to iommu group but I am not maybe I am wrong.
I just created the patch that is not tested. Please try to apply the patch and test it once.



Regards,
Rahul


[-- Attachment #1.2: Type: text/html, Size: 1745 bytes --]

[-- Attachment #2: 0001-xen-arm-smmuv1-remove-iommu-group-when-deassign.patch --]
[-- Type: application/octet-stream, Size: 943 bytes --]

From b0f2c38d5856a8c19cd04f70d3bd8b8cd2256bbd Mon Sep 17 00:00:00 2001
Message-Id: <b0f2c38d5856a8c19cd04f70d3bd8b8cd2256bbd.1650546228.git.rahul.singh@arm.com>
From: Rahul Singh <rahul.singh@arm.com>
Date: Thu, 21 Apr 2022 13:42:20 +0100
Subject: [PATCH] xen/arm: smmuv1: remove iommu group when deassign

Signed-off-by: Rahul Singh <rahul.singh@arm.com>
Change-Id: I941c5541c24df496bd3143f1c75f1366af5d8ca4
---
 xen/drivers/passthrough/arm/smmu.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/xen/drivers/passthrough/arm/smmu.c b/xen/drivers/passthrough/arm/smmu.c
index c9dfc4caa0..3831480ed7 100644
--- a/xen/drivers/passthrough/arm/smmu.c
+++ b/xen/drivers/passthrough/arm/smmu.c
@@ -1690,6 +1690,8 @@ static void arm_smmu_detach_dev(struct iommu_domain *domain, struct device *dev)
 	if (cfg)
 		arm_smmu_master_free_smes(cfg);
 
+	iommu_group_put(dev_iommu_group(dev));
+	dev_iommu_group(dev) = NULL;
 }
 
 #if 0 /*
-- 
2.25.1

Re: Need help to debug Arm SMMU issue seen on guests with passthrough devices

[Ayan Kumar Halder]

Hi Rahul,

On 21/04/2022 14:21, Rahul Singh wrote:
> Hi Ayan,
>
> > On 21 Apr 2022, at 12:12 pm, Ayan Kumar Halder 
> <ayan.kumar.halder@xilinx.com> wrote:
> >
> > Hi Arm/SMMU experts,
> >
> > Scenario :- I am trying to assign a device (eg mmc) to a guest which 
> uses smmu. I start the guest using "xl create -c ...". It works fine 
> for the first time. I am able to access the device.
> >
> > Now, when I destroy the guest and create again, I see this issue
>
> Did you observe any other Xen error while deassigning and assigning 
> the device ?
> >
> > (XEN) smmu: /axi/smmu@fd800000:         GFSR 0x00000002, GFSYNR0 
> 0x00000000, GFSYNR1 0x00000871, GFSYNR2 0x00000000
> >
> > 0x871 is the stream id used by mmc.
> >
> > We are using SMMU-500. So from the Arm SMMU v2 specs, it is 
> "Unidentified stream fault" for stream id "0x871".
>
> Yes, you are right above error means that Unidentified stream fault. 
> That means SMMU hardware is not set up for streamId 0x0000087
> I just try to go through the SMMU code and observe that this issue may 
> be related to iommu group but I am not maybe I am wrong.
> I just created the patch that is not tested. Please try to apply the 
> patch and test it once.

Many thanks for looking into this. With your patch, the issue is resolved.

Are you going to send out this patch for review ? You can add my 
"Tested-by" tag.

- Ayan

>
>
>
> Regards,
> Rahul
>

Re: Need help to debug Arm SMMU issue seen on guests with passthrough devices

[Rahul Singh]

Hi Ayan,

> On 21 Apr 2022, at 5:44 pm, Ayan Kumar Halder <ayan.kumar.halder@xilinx.com> wrote:
> 
> Hi Rahul,
> 
> On 21/04/2022 14:21, Rahul Singh wrote:
>> Hi Ayan,
>> 
>> > On 21 Apr 2022, at 12:12 pm, Ayan Kumar Halder <ayan.kumar.halder@xilinx.com> wrote:
>> >
>> > Hi Arm/SMMU experts,
>> >
>> > Scenario :- I am trying to assign a device (eg mmc) to a guest which uses smmu. I start the guest using "xl create -c ...". It works fine for the first time. I am able to access the device.
>> >
>> > Now, when I destroy the guest and create again, I see this issue
>> 
>> Did you observe any other Xen error while deassigning and assigning the device ?
>> >
>> > (XEN) smmu: /axi/smmu@fd800000:  GFSR 0x00000002, GFSYNR0 0x00000000, GFSYNR1 0x00000871, GFSYNR2 0x00000000
>> >
>> > 0x871 is the stream id used by mmc.
>> >
>> > We are using SMMU-500. So from the Arm SMMU v2 specs, it is "Unidentified stream fault" for stream id "0x871".
>> 
>> Yes, you are right above error means that Unidentified stream fault. That means SMMU hardware is not set up for streamId 0x0000087
>> I just try to go through the SMMU code and observe that this issue may be related to iommu group but I am not maybe I am wrong.
>> I just created the patch that is not tested. Please try to apply the patch and test it once.
> 
> Many thanks for looking into this. With your patch, the issue is resolved.
> 
> Are you going to send out this patch for review ? You can add my "Tested-by" tag.
> 

Good to know that the issue has been resolved. Yes, I will send the patch for review.

Regards,
Rahul