Re: [XEN PATCH v1] xen/arm : Add support for SMMUv3 driver

From: Rahul Singh <Rahul.Singh@arm.com>
To: Oleksandr Andrushchenko <Oleksandr_Andrushchenko@epam.com>
Cc: Bertrand Marquis <Bertrand.Marquis@arm.com>,
	Oleksandr Andrushchenko <andr2000@gmail.com>,
	Julien Grall <julien@xen.org>,
	"xen-devel@lists.xenproject.org" <xen-devel@lists.xenproject.org>,
	Jan Beulich <jbeulich@suse.com>, Paul Durrant <paul@xen.org>,
	Stefano Stabellini <sstabellini@kernel.org>,
	Volodymyr Babchuk <Volodymyr_Babchuk@epam.com>
Subject: Re: [XEN PATCH v1] xen/arm : Add support for SMMUv3 driver
Date: Fri, 6 Nov 2020 14:41:08 +0000	[thread overview]
Message-ID: <1390C05F-445F-4349-A672-4D7373C301B8@arm.com> (raw)
In-Reply-To: <d2eb2db3-7038-3850-310b-4676102e0a55@epam.com>

Hello Oleksandr,

> On 6 Nov 2020, at 2:22 pm, Oleksandr Andrushchenko <Oleksandr_Andrushchenko@epam.com> wrote:
> 
> Hi, Rahul!
> 
> On 11/6/20 3:58 PM, Rahul Singh wrote:
>> Hello Oleksandr,
>> 
>>> On 6 Nov 2020, at 1:00 pm, Oleksandr Andrushchenko <Oleksandr_Andrushchenko@epam.com> wrote:
>>> 
>>> Hello, Rahul!
>>> 
>>> On 11/6/20 2:48 PM, Rahul Singh wrote:
>>>> Hello Oleksandr,
>>>> 
>>>>> On 2 Nov 2020, at 10:12 am, Oleksandr Andrushchenko <Oleksandr_Andrushchenko@epam.com> wrote:
>>>>> 
>>>>> Hi,
>>>>> 
>>>>> On 11/2/20 11:55 AM, Bertrand Marquis wrote:
>>>>>> Hi,
>>>>>> 
>>>>>>> On 2 Nov 2020, at 05:55, Oleksandr Andrushchenko <andr2000@gmail.com> wrote:
>>>>>>> 
>>>>>>> Hi, Julien!
>>>>>>> 
>>>>>>> On 10/30/20 7:18 PM, Julien Grall wrote:
>>>>>>>> Hi Oleksandr,
>>>>>>>> 
>>>>>>>> On 30/10/2020 10:44, Oleksandr Andrushchenko wrote:
>>>>>>>>> On 10/20/20 6:25 PM, Rahul Singh wrote:
>>>>>>>>>> Add support for ARM architected SMMUv3 implementations. It is based on
>>>>>>>>>> the Linux SMMUv3 driver.
>>>>>>>>>> 
>>>>>>>>>> Major differences between the Linux driver are as follows:
>>>>>>>>>> 1. Only Stage-2 translation is supported as compared to the Linux driver
>>>>>>>>>>      that supports both Stage-1 and Stage-2 translations.
>>>>>>>>> First of all thank you for the efforts!
>>>>>>>>> 
>>>>>>>>> I tried the patch with QEMU and would like to know if my understanding correct
>>>>>>>>> 
>>>>>>>>> that this combination will not work as of now:
>>>>>>>>> 
>>>>>>>>> (XEN) SMMUv3: /smmuv3@9050000: SMMUv3: DT value = eventq
>>>>>>>>> (XEN) Data Abort Trap. Syndrome=0x1940010
>>>>>>>>> (XEN) Walking Hypervisor VA 0x40031000 on CPU0 via TTBR 0x00000000b8469000
>>>>>>>>> (XEN) 0TH[0x0] = 0x00000000b8468f7f
>>>>>>>>> 
>>>>>>>>> [snip]
>>>>>>>>> 
>>>>>>>>> If this is expected then is there any plan to make QEMU work as well?
>>>>>>>>> 
>>>>>>>>> I see [1] says that "Only stage 1 and AArch64 PTW are supported." on QEMU side.
>>>>>>>> Just for clarication, you are trying to boot Xen on QEMU, right?
>>>>>>> Exactly
>>>>>>>> You might be able to use the stage-1 page-tables to isolate each device in Xen. However, I don't think you will be able to share the P2M because the page-tables layout between stage-1 and stage-2 is different.
>>>>>>> So, it is even more work then
>>>>>> Overall it would make more sense to spend some time adding proper support in Qemu then trying to modify the driver to support Qemu right now.
>>>>>> 
>>>>>>>>> We are interested in QEMU/SMMUv3 as a flexible platform for PCI passthrough
>>>>>>>>> 
>>>>>>>>> implementation, so it could allow testing different setups and configurations with QEMU.
>>>>>>>> I would recommend to get the SMMU supporting supporting stage-2 page-tables.
>>>>>>> You mean in QEMU?
>>>>>> See before.
>>>>>> 
>>>>>>>> Regardless that, I think Xen should be able to say the SMMU is not supported rather than crashing.
>>>>>>> Yes, that would be nice
>>>>>> Fully agree and we will look into that.
>>>>>> 
>>>>>> Anything you could share so that we could quickly reproduce your setup would be more then great.
>>>>> Nothing special,
>>>>> 
>>>>> qemu/aarch64-softmmu/qemu-system-aarch64 -machine type=virt -machine virt,gic-version=2 \
>>>>> 
>>>>> -machine virtualization=true -cpu cortex-a57 -smp 4 -m 2048 -nic user,hostfwd=tcp:127.0.0.1:2222-:22 \
>>>>> 
>>>>> -nographic -serial mon:stdio [..snip..]
>>>>> 
>>>>> I also set iommu to smmuv3 in my tests, QEMU emulator version 4.2.1
>>>> I just checked and confirmed that QEMU is booting with XEN SMMUv3 patch and XEN is able to say SMMU translation is not supported. As XEN supports Stage-2 translation and QEMU supports Stage-1 only.
>>>> 
>>>> 
>>>> (XEN) SMMUv3: /smmuv3@9050000: SMMUv3: DT value = eventq
>>>> (XEN) SMMUv3: /smmuv3@9050000: IDR0.COHACC overridden by FW configuration (false)
>>>> (XEN) SMMUv3: /smmuv3@9050000: no translation support!
>>>> (XEN) I/O virtualisation disabled
>>>> 
>>>> Only difference I observed is that you have to add option "-machine virt,iommu=smmuv3 “ when launching the QEMU.
>>> I do use the option
>> I used "-machine virt,iommu=smmuv3 “  option while creating the virt-dtb and while launching the QEMU.
>> I also observed the same error what you observed if I am not using the "-machine virt,iommu=smmuv3 “ options when launching the QEMU so I thought this might be case for you also but anyways you have use the options it might be other issue.
> 
> Hm, probably that was on my side as now I can see:
> 
> (XEN) SMMUv3: /smmuv3@9050000: SMMUv3: DT value = eventq
> (XEN) SMMUv3: /smmuv3@9050000: IDR0.COHACC overridden by FW configuration (false)
> (XEN) SMMUv3: /smmuv3@9050000: no translation support!
> (XEN) I/O virtualisation disabled
> (XEN)
> (XEN) ****************************************
> (XEN) Panic on CPU 0:
> (XEN) Couldn't configure correctly all the IOMMUs.
> (XEN) ****************************************
> (XEN)
> (XEN) Manual reset required ('noreboot' specified)
> 
> So, sorry for the noise, I might have misconfigured something it seems
> 
> When you say "Xen is booting" do you mean you see the same panic?

Yes I observe the same. 

We have to decide now if for SMMUv3 there is no translation support do we have to print the logs and move forward  or as above return error to iommu_setup that will cal panic().

Regards,
Rahul

> 
> Thank you,
> 
> Oleksandr
> 
>> 
>>>> Please let me know if it also works for you.
>>> Well, I should have reported that earlier that I do not use the staging Xen at the moment,
>>> 
>>> it is 4.14.0. So, can this be a problem with that Xen version?
>> I don’t think so this is the problem with the XEN version.
>>> Anyways, if it works with the staging then everything looks ok
>>> 
>>> Thank you,
>>> 
>>> Oleksandr
>>> 
>>>>>> Regards
>>>>>> Bertrand
>>>>>> 
>>>>>>>> Cheers,
>>>>>>>> 
>>>>>>> Thank you,
>>>>>>> 
>>>>>>> Oleksandr
>> Regards,
>> Rahul