Re: Xen data from meta-virtualization layer

[Rahul Singh <Rahul.Singh@arm.com>]

Hello ,

> On 22 Nov 2020, at 10:55 pm, Leo Krueger <leo.krueger@zal.aero> wrote:
> 
> Hi Julien,
> 
> finally I could try out what you suggested, please find my answers inline.
> 
>> -----Ursprüngliche Nachricht-----
>> Von: Julien Grall <julien@xen.org>
>> Gesendet: Mittwoch, 18. November 2020 13:24
>> An: Stefano Stabellini <stefano.stabellini@xilinx.com>; Leo Krueger
>> <leo.krueger@zal.aero>
>> Cc: Peng Fan <peng.fan@nxp.com>; brucea@xilinx.com; Cornelia Bruelhart
>> <cornelia.bruelhart@zal.aero>; oleksandr_andrushchenko@epam.com; xen-
>> devel@lists.xenproject.org; Bertrand.Marquis@arm.com
>> Betreff: Re: AW: AW: AW: AW: Xen data from meta-virtualization layer
>> 
>> Hi,
>> 
>> On 17/11/2020 23:53, Stefano Stabellini wrote:
>>> Adding Bertrand, Oleksandr, Julien, and others -- they have a more
>>> recent experience with GICv3 ITS than me and might be able to help.
>>> I am attaching the device tree Leo sent a few days ago for reference.
>>> 
>>> 
>>> Typically when you can set the ethernet link up and no packets are
>>> exchanged it is because of a missing interrupt. In this case a missing
>>> MSI.
>>> 
>>> Bertrand, I believe you tried the GIC ITS driver with PCI devices
>>> recently. It is expected to work correctly with MSIs in Dom0, right?
>> 
>> OSSTest has some hardware (e.g. Thunder-X) where ITS is required to boot
>> Dom0. I haven't seen any failure on recent Xen. We are testing 4.11 and
>> onwards on Thunder-X.
>> 
>> However, it may be possible that some more work is necessary for other
>> hardware (e.g. workaround, missing code...). See more below.
>> 
>>> 
>>> 
>>> 
>>> On Tue, 17 Nov 2020, Leo Krueger wrote:
>>>> Hi,
>>>> 
>>>> I enabled CONFIG_HAS_ITS (what a stupid mistake by me to not set it
>>>> before...) but then had to add the following node to my device tree
>>>> 
>>>> 	gic_lpi_base: syscon@0x80000000 {
>>>> 		compatible = "gic-lpi-base";
>> 
>> I couldn't find this compatible defined/used in Linux 5.10-rc4. @Leo, could
>> you clarify which flavor/version of Linux you are using?
> 
> It is Linux 4.19 from Yocto (Warror release). XEN 4.13.2.
> While searching around the Internet for any solution, I came across [0] which contained the gic-lpi-base node.
> So I just tried adding it (quite desperate I know) and voila, it at least brought me one step further (XEN exposing the ITS)...
> 
>> 
>>>> 		reg = <0x0 0x80000000 0x0 0x100000>;
>>>> 		max-gic-redistributors = <2>;
>>>> 	};
>>>> 
>>>> to somehow change something in regard to the ITS and MSI/MSI-X
>>>> 
>>>> (XEN) GICv3 initialization:
>>>> (XEN)       gic_dist_addr=0x00000006000000
>>>> (XEN)       gic_maintenance_irq=25
>>>> (XEN)       gic_rdist_stride=0
>>>> (XEN)       gic_rdist_regions=1
>>>> (XEN)       redistributor regions:
>>>> (XEN)         - region 0: 0x00000006040000 - 0x00000006080000
>>>> (XEN) GICv3: using at most 57344 LPIs on the host.
>>>> (XEN) GICv3: 288 lines, (IID 0001143b).
>>>> (XEN) GICv3: Found ITS @0x6020000
>>>> (XEN) using non-cacheable ITS command queue
>>>> (XEN) GICv3: CPU0: Found redistributor in region 0 @000000004001c000
>>>> 
>>>> [    0.000000] GICv3: Distributor has no Range Selector support
>>>> [    0.000000] GICv3: no VLPI support, no direct LPI support
>>>> [    0.000000] ITS [mem 0x06020000-0x0603ffff]
>>>> [    0.000000] ITS@0x0000000006020000: allocated 65536 Devices
>> @dc880000 (flat, esz 8, psz 64K, shr 1)
>>>> [    0.000000] ITS@0x0000000006020000: allocated 32768 Interrupt
>> Collections @dc820000 (flat, esz 2, psz 64K, shr 1)
>>>> [    0.000000] GIC: using LPI property table @0x00000000dc830000
>>>> [    0.000000] GICv3: CPU0: found redistributor 0 region
>> 0:0x0000000006040000
>>>> [    0.000000] CPU0: using LPI pending table @0x00000000dc840000
>>>> ...
>>>> [    0.040080] Platform MSI: gic-its domain created
>>>> [    0.040136] PCI/MSI: /interrupt-controller/gic-its domain created
>>>> [    0.040181] fsl-mc MSI: /interrupt-controller/gic-its domain created
>>>> 
>>>> 
>>>> Still I am ending up with the " Failed to add - passthrough or MSI/MSI-X
>> might fail!" log messages for some PCI devices, but at least the on-board
>> ethernet ports (fsl_enetc ) are initialized.
>>>> I can set the link up and a link is successfully established.
>> 
>> This message is normal. Xen on Arm is not yet aware of PCI devices and
>> therefore the hypercalls to add PCI devices will return -EOPNOTSUPP.
>> 
>> However, this is not an issue because the virtual ITS implementation will
>> allow dom0 to configure any devices.
>> 
>>>> 
>>>> But (!) I cannot receive or transmit anything (no error message...) and
>> there seem to be no interrupts:
>>>> 
>>>> 29:          0   ITS-MSI   1 Edge      gbe0-rxtx0
>>>>  32:          0   ITS-MSI 8192 Edge      ptp_qoriq
>>>> 
>>>> (from /proc/interrupts).
>>>> 
>>>> Any idea on this one? I keep digging and checking whether my device tree
>> needs some additional fixes.
>> 
>> Can you apply patch [1] and provide the logs? This will dump more
>> information about the command received by the vITS and the one send to
>> the host ITS.
> 
> For XEN 4.13.2 I had to adapt your patch slightly [1], see below (yes I know, quite ugly in parts).
> Find attached the boot log and an output of "xl dmesg" which is truncated due to the large amount of messages.
> 
> When enabling the network interface (gbe0), the following output is visible:
> 
> root@kontron-sal28:~# ip link set up dev gbe0
> (XEN) vgic-v3-its.c:902:d0v0 vITS  cmd 0x0c: 000000170000000c 0000000000000001 0000000000000000 0000000000000000
> (XEN) vgic-v3-its.c:902:d0v0 vITS  cmd 0x05: 0000000000000005 0000000000000000 0000000000000000 0000000000000000
> [   34.034598] Atheros 8031 ethernet 0000:00:00.3:05: attached PHY driver [Atheros 8031 ethernet] (mii_bus:phy_addr=0000:00:00.3:05, irq=POLL)
> [   34.041111] 8021q: adding VLAN 0 to HW filter on device gbe0
> [   34.041209] IPv6: ADDRCONF(NETDEV_UP): gbe0: link is not ready
> root@kontron-sal28:~# [   35.041951] fsl_enetc 0000:00:00.0 gbe0: Link is Down
> [   38.114426] fsl_enetc 0000:00:00.0 gbe0: Link is Up - 1Gbps/Full - flow control off
> [   38.114508] IPv6: ADDRCONF(NETDEV_CHANGE): gbe0: link becomes ready
> 
> Does that tell you anything?
> 

I just checked the logs shared, what I found out that there’s is an error while booting to configure the MSI for the PCI device because of that there will be case that Device Id generate out-of-band is not mapped correctly to ITS device table created while initialising the MSI for the device. 
I might be wrong let someone else also comments on this. 

 
[    0.173964] OF: /soc/pcie@1f0000000: Invalid msi-map translation - no match for rid 0xf8 on           (null)
 
Regards,
Rahul

>> 
>> Note that Xen will need to be build with CONFIG_DEBUG=y in order to get
>> some of the messages.
>> 
>> [...]
>> 
>>>>>> [    0.000000] GICv3: Distributor has no Range Selector support
>>>>>> 
>>>>>> [    0.000000] GICv3: no VLPI support, no direct LPI support
>>>>>> 
>>>>>> [    0.000000] GICv3: CPU0: found redistributor 0 region
>>>>>> 0:0x0000000006040000
>>>>> 
>>>>> "no VLPI support" is very suspicious, it looks like Dom0 doesn't
>>>>> find any ITS support.
>> VLPI is a feature that was introduced in GICv4 to directly inject LPI in the
>> guest. So this is normal to see this message when running on Xen because
>> we are going to only expose a GICv3 to a domain until at least we support
>> nested virt.
>> 
>> However, you were right about that Xen didn't expose the ITS because the
>> following lines were missing:
>> 
>> [    0.000000] ITS@0x0000000006020000: allocated 65536 Devices @dc880000
>> (flat, esz 8, psz 64K, shr 1)
>> 
>> Cheers,
> 
> Best regards,
> Leo
> 
>> 
>> [1]
>> diff --git a/xen/arch/arm/gic-v3-its.c b/xen/arch/arm/gic-v3-its.c index
>> 9558bad96ac3..8a0a02308e74 100644
>> --- a/xen/arch/arm/gic-v3-its.c
>> +++ b/xen/arch/arm/gic-v3-its.c
>> @@ -87,6 +87,10 @@ static int its_send_command(struct host_its *hw_its,
>> const void *its_cmd)
>>      /* No ITS commands from an interrupt handler (at the moment). */
>>      ASSERT(!in_irq());
>> 
>> +    printk(XENLOG_WARNING, "pITS  cmd 0x%02lx: %016lx %016lx %016lx
>> %016lx\n",
>> +           its_cmd_get_command(command),
>> +           command[0], command[1], command[2], command[3]);
>> +
>>      spin_lock(&hw_its->cmd_lock);
>> 
>>      do {
>> diff --git a/xen/arch/arm/gic-v3-lpi.c b/xen/arch/arm/gic-v3-lpi.c index
>> 869bc97fa1aa..e7c5bcd8d423 100644
>> --- a/xen/arch/arm/gic-v3-lpi.c
>> +++ b/xen/arch/arm/gic-v3-lpi.c
>> @@ -183,7 +183,10 @@ void gicv3_do_LPI(unsigned int lpi)
>>      /* Find out if a guest mapped something to this physical LPI. */
>>      hlpip = gic_get_host_lpi(lpi);
>>      if ( !hlpip )
>> +    {
>> +        printk("%s: Received LPI %u but it is not mapped?\n", __func__,
>> lpi);
>>          goto out;
>> +    }
>> 
>>      hlpi.data = read_u64_atomic(&hlpip->data);
>> 
>> @@ -222,6 +225,9 @@ void gicv3_lpi_update_host_entry(uint32_t host_lpi,
>> int domain_id,
>>  {
>>      union host_lpi *hlpip, hlpi;
>> 
>> +    printk("%s: host_lpi %u domain %d virq_lpi %u\n",
>> +           __func__, host_lpi, domain_id, virq_lpi);
>> +
>>      ASSERT(host_lpi >= LPI_OFFSET);
>> 
>>      host_lpi -= LPI_OFFSET;
>> diff --git a/xen/arch/arm/vgic-v3-its.c b/xen/arch/arm/vgic-v3-its.c index
>> 58d939b85f92..89ef137b3e6b 100644
>> --- a/xen/arch/arm/vgic-v3-its.c
>> +++ b/xen/arch/arm/vgic-v3-its.c
>> @@ -897,7 +897,7 @@ out_unlock:
>> 
>>  static void dump_its_command(uint64_t *command)
>>  {
>> -    gdprintk(XENLOG_WARNING, "  cmd 0x%02lx: %016lx %016lx %016lx
>> %016lx\n",
>> +    gdprintk(XENLOG_WARNING, "vITS  cmd 0x%02lx: %016lx %016lx %016lx
>> %016lx\n",
>>               its_cmd_get_command(command),
>>               command[0], command[1], command[2], command[3]);
>>  }
>> @@ -926,6 +926,8 @@ static int vgic_its_handle_cmds(struct domain *d,
>> struct virt_its *its)
>>          if ( ret )
>>              return ret;
>> 
>> +        dump_its_command(command);
>> +
>>          switch ( its_cmd_get_command(command) )
>>          {
>>          case GITS_CMD_CLEAR:
>> 
>> 
>> --
>> Julien Grall
> 
> [0] https://www.mail-archive.com/u-boot@lists.denx.de/msg379708.html
> [1] 
> diff --git a/xen/arch/arm/gic-v3-its.c b/xen/arch/arm/gic-v3-its.c
> index 9558bad96a..d175ba52b0 100644
> --- a/xen/arch/arm/gic-v3-its.c
> +++ b/xen/arch/arm/gic-v3-its.c
> @@ -87,6 +87,10 @@ static int its_send_command(struct host_its *hw_its, const void *its_cmd)
>     /* No ITS commands from an interrupt handler (at the moment). */
>     ASSERT(!in_irq());
> 
> +    printk(XENLOG_WARNING "pITS  cmd 0x%02lx: %016lx %016lx %016lx %016lx\n",
> +        (((uint64_t *) its_cmd)[0] >> 0) & GENMASK(8 - 1, 0),
> +        ((uint64_t *) its_cmd)[0], ((uint64_t *) its_cmd)[1], ((uint64_t *) its_cmd)[2], ((uint64_t *) its_cmd)[3]);
> +
>     spin_lock(&hw_its->cmd_lock);
> 
>     do {
> diff --git a/xen/arch/arm/gic-v3-lpi.c b/xen/arch/arm/gic-v3-lpi.c
> index 78b9521b21..2c3b0fc9e5 100644
> --- a/xen/arch/arm/gic-v3-lpi.c
> +++ b/xen/arch/arm/gic-v3-lpi.c
> @@ -181,8 +181,10 @@ void gicv3_do_LPI(unsigned int lpi)
> 
>     /* Find out if a guest mapped something to this physical LPI. */
>     hlpip = gic_get_host_lpi(lpi);
> -    if ( !hlpip )
> +    if ( !hlpip ) {
> +        printk("%s: Received LPI %u but it is not mapped?\n", __func__, lpi);
>         goto out;
> +    }
> 
>     hlpi.data = read_u64_atomic(&hlpip->data);
> 
> @@ -221,6 +223,9 @@ void gicv3_lpi_update_host_entry(uint32_t host_lpi, int domain_id,
> {
>     union host_lpi *hlpip, hlpi;
> 
> +    printk("%s: host_lpi %u domain %d virt_lpi %u\n",
> +        __func__, host_lpi, domain_id, virt_lpi);
> +
>     ASSERT(host_lpi >= LPI_OFFSET);
> 
>     host_lpi -= LPI_OFFSET;
> diff --git a/xen/arch/arm/vgic-v3-its.c b/xen/arch/arm/vgic-v3-its.c
> index 6e153c698d..dd5081ef80 100644
> --- a/xen/arch/arm/vgic-v3-its.c
> +++ b/xen/arch/arm/vgic-v3-its.c
> @@ -897,7 +897,7 @@ out_unlock:
> 
> static void dump_its_command(uint64_t *command)
> {
> -    gdprintk(XENLOG_WARNING, "  cmd 0x%02lx: %016lx %016lx %016lx %016lx\n",
> +    gdprintk(XENLOG_WARNING, "vITS  cmd 0x%02lx: %016lx %016lx %016lx %016lx\n",
>              its_cmd_get_command(command),
>              command[0], command[1], command[2], command[3]);
> }
> @@ -926,6 +926,8 @@ static int vgic_its_handle_cmds(struct domain *d, struct virt_its *its)
>         if ( ret )
>             return ret;
> 
> +        dump_its_command(command);
> +
>         switch ( its_cmd_get_command(command) )
>         {
>         case GITS_CMD_CLEAR:
> <boot-xendebug.log><xen-debug-output.txt>