Re: [PATCH 10/11] xen/arm: Do not map PCI ECAM space to Domain-0's p2m

From: Rahul Singh <Rahul.Singh@arm.com>
To: Oleksandr Andrushchenko <Oleksandr_Andrushchenko@epam.com>
Cc: "Stefano Stabellini" <sstabellini@kernel.org>,
	"Julien Grall" <julien.grall.oss@gmail.com>,
	"Oleksandr Andrushchenko" <andr2000@gmail.com>,
	xen-devel <xen-devel@lists.xenproject.org>,
	"Oleksandr Tyshchenko" <Oleksandr_Tyshchenko@epam.com>,
	"Volodymyr Babchuk" <Volodymyr_Babchuk@epam.com>,
	"Artem Mygaiev" <Artem_Mygaiev@epam.com>,
	"Roger Pau Monné" <roger.pau@citrix.com>,
	"Bertrand Marquis" <Bertrand.Marquis@arm.com>
Subject: Re: [PATCH 10/11] xen/arm: Do not map PCI ECAM space to Domain-0's p2m
Date: Wed, 15 Sep 2021 16:42:02 +0000	[thread overview]
Message-ID: <5AFD57E4-1C4B-4781-ADAC-BA3C4AC2E2D1@arm.com> (raw)
In-Reply-To: <ee38bc53-c590-24d2-9061-3d4fd33f58bd@epam.com>

Hi Oleksandr,

> On 15 Sep 2021, at 6:35 am, Oleksandr Andrushchenko <Oleksandr_Andrushchenko@epam.com> wrote:
> 
> Hi, Stefano, Rahul!
> 
> On 15.09.21 03:36, Stefano Stabellini wrote:
>> On Tue, 14 Sep 2021, Oleksandr Andrushchenko wrote:
>>> With the patch above I have the following log in Domain-0:
>>> 
>>> generic-armv8-xt-dom0 login: root
>>> root@generic-armv8-xt-dom0:~# (XEN) *** Serial input to Xen (type 'CTRL-a' three times to switch input)
>>> (XEN) ==== PCI devices ====
>>> (XEN) ==== segment 0000 ====
>>> (XEN) 0000:03:00.0 - d0 - node -1
>>> (XEN) 0000:02:02.0 - d0 - node -1
>>> (XEN) 0000:02:01.0 - d0 - node -1
>>> (XEN) 0000:02:00.0 - d0 - node -1
>>> (XEN) 0000:01:00.0 - d0 - node -1
>>> (XEN) 0000:00:00.0 - d0 - node -1
>>> (XEN) *** Serial input to DOM0 (type 'CTRL-a' three times to switch input)
>>> 
>>> root@generic-armv8-xt-dom0:~# modprobe e1000e
>>> [   46.104729] e1000e: Intel(R) PRO/1000 Network Driver
>>> [   46.105479] e1000e: Copyright(c) 1999 - 2015 Intel Corporation.
>>> [   46.107297] e1000e 0000:03:00.0: enabling device (0000 -> 0002)
>>> (XEN) map [e0000, e001f] -> 0xe0000 for d0
>>> (XEN) map [e0020, e003f] -> 0xe0020 for d0
>>> [   46.178513] e1000e 0000:03:00.0: Interrupt Throttling Rate (ints/sec) set to dynamic conservative mode
>>> [   46.189668] pci_msi_setup_msi_irqs
>>> [   46.191016] nwl_compose_msi_msg msg at fe440000
>>> (XEN) traps.c:2014:d0v0 HSR=0x00000093810047 pc=0xffff8000104b4b00 gva=0xffff800010fa5000 gpa=0x000000e0040000
>>> [   46.200455] Unhandled fault at 0xffff800010fa5000
>>> 
>>> [snip]
>>> 
>>> [   46.233079] Call trace:
>>> [   46.233559]  __pci_write_msi_msg+0x70/0x180
>>> [   46.234149]  pci_msi_domain_write_msg+0x28/0x30
>>> [   46.234869]  msi_domain_activate+0x5c/0x88
>>> 
>>>  From the above you can see that BARs are mapped for Domain-0 now
>>> 
>>> only when an assigned PCI device gets enabled in Domain-0.
>>> 
>>> Another thing to note is that we crash on MSI-X access as BARs are mapped
>>> 
>>> to the domain while enabling memory decoding in the COMMAND register,
>>> 
>>> but MSI-X are handled differently, e.g. we have MSI-X holes in the mappings.
>>> 
>>> This is because before this change the whole PCI aperture was mapped into
>>> 
>>> Domain-0 and it is not. Thus, MSI-X holes are left unmapped now and there
>>> 
>>> was no need to do so, e.g. they were always mapped into Domain-0 and
>>> 
>>> emulated for guests.
>>> 
>>> Please note that one cannot use xl pci-attach in this case to attach the PCI device
>>> 
>>> in question to Domain-0 as (please see the log) that device is already attached.
>>> 
>>> Neither it can be detached and re-attached. So, without mapping MSI-X holes for
>>> 
>>> Domain-0 the device becomes unusable in Domain-0. At the same time the device
>>> 
>>> can be successfully passed to DomU.
>>> 
>>> 
>>> Julien, Stefano! Please let me know how can we proceed with this.
>> What was the plan for MSI-X in Dom0?
> It just worked because we mapped everything
>> 
>> Given that Dom0 interacts with a virtual-ITS and virtual-GIC rather than
>> a physical-ITS and physical-GIC, I imagine that it wasn't correct for
>> Dom0 to write to the real MSI-X table directly?
>> 
>> Shouldn't Dom0 get emulated MSI-X tables like any DomU?
>> 
>> Otherwise, if Dom0 is expected to have the real MSI-X tables mapped, then
>> I would suggest to map them at the same time as the BARs. But I am
>> thinking that Dom0 should get emulated MSI-X tables, not physical MSI-X
>> tables.
> 
> Yes, it seems more than reasonable to enable emulation for Domain-0
> 
> as well. Other than that, Stefano, do you think we are good to go with
> 
> the changes I did in order to unmap everything for Domain-0?
> 
> Rahul, it seems we will need a change to vPCI/MSI-X so we can also
> 
> trap Domain-0 for MSI-X tables.

I agree that we need emulated MSI-X tables for Dom0 also. Let me check on this and come back to you.

Regards,
Rahul
> 
> Thank you,
> 
> Oleksandr