Re: Xen data from meta-virtualization layer

[Rahul Singh <Rahul.Singh@arm.com>]

Hello Stefano,


> On 17 Nov 2020, at 11:53 pm, Stefano Stabellini <stefano.stabellini@xilinx.com> wrote:
> 
> Adding Bertrand, Oleksandr, Julien, and others -- they have a more
> recent experience with GICv3 ITS than me and might be able to help.
> I am attaching the device tree Leo sent a few days ago for reference.
> 
> 
> Typically when you can set the ethernet link up and no packets are
> exchanged it is because of a missing interrupt. In this case a missing
> MSI.
> 
> Bertrand, I believe you tried the GIC ITS driver with PCI devices
> recently. It is expected to work correctly with MSIs in Dom0, right?
> 

Yes we are using the XEN GIC ITS driver and MSI interrupts is working fine in DOM0.

20:        112          0          0          0   ITS-MSI 1572864 Edge      eth0
 21:        441          0          0          0   ITS-MSI 3670016 Edge      eth1
 22:       4286          0          0          0   ITS-MSI 4194304 Edge      xhci_hcd


Regards,
Rahul

> 
> On Tue, 17 Nov 2020, Leo Krueger wrote:
>> Hi,
>> 
>> I enabled CONFIG_HAS_ITS (what a stupid mistake by me to not set it before...) but then had to add the following node to my device tree
>> 
>> 	gic_lpi_base: syscon@0x80000000 {
>> 		compatible = "gic-lpi-base";
>> 		reg = <0x0 0x80000000 0x0 0x100000>;
>> 		max-gic-redistributors = <2>;
>> 	};
>> 
>> to somehow change something in regard to the ITS and MSI/MSI-X
>> 
>> (XEN) GICv3 initialization:
>> (XEN)       gic_dist_addr=0x00000006000000
>> (XEN)       gic_maintenance_irq=25
>> (XEN)       gic_rdist_stride=0
>> (XEN)       gic_rdist_regions=1
>> (XEN)       redistributor regions:
>> (XEN)         - region 0: 0x00000006040000 - 0x00000006080000
>> (XEN) GICv3: using at most 57344 LPIs on the host.
>> (XEN) GICv3: 288 lines, (IID 0001143b).
>> (XEN) GICv3: Found ITS @0x6020000
>> (XEN) using non-cacheable ITS command queue
>> (XEN) GICv3: CPU0: Found redistributor in region 0 @000000004001c000
>> 
>> [    0.000000] GICv3: Distributor has no Range Selector support
>> [    0.000000] GICv3: no VLPI support, no direct LPI support
>> [    0.000000] ITS [mem 0x06020000-0x0603ffff]
>> [    0.000000] ITS@0x0000000006020000: allocated 65536 Devices @dc880000 (flat, esz 8, psz 64K, shr 1)
>> [    0.000000] ITS@0x0000000006020000: allocated 32768 Interrupt Collections @dc820000 (flat, esz 2, psz 64K, shr 1)
>> [    0.000000] GIC: using LPI property table @0x00000000dc830000
>> [    0.000000] GICv3: CPU0: found redistributor 0 region 0:0x0000000006040000
>> [    0.000000] CPU0: using LPI pending table @0x00000000dc840000
>> ...
>> [    0.040080] Platform MSI: gic-its domain created
>> [    0.040136] PCI/MSI: /interrupt-controller/gic-its domain created
>> [    0.040181] fsl-mc MSI: /interrupt-controller/gic-its domain created
>> 
>> 
>> Still I am ending up with the " Failed to add - passthrough or MSI/MSI-X might fail!" log messages for some PCI devices, but at least the on-board ethernet ports (fsl_enetc ) are initialized.
>> I can set the link up and a link is successfully established.
>> 
>> But (!) I cannot receive or transmit anything (no error message...) and there seem to be no interrupts:
>> 
>> 29:          0   ITS-MSI   1 Edge      gbe0-rxtx0
>> 32:          0   ITS-MSI 8192 Edge      ptp_qoriq
>> 
>> (from /proc/interrupts).
>> 
>> Any idea on this one? I keep digging and checking whether my device tree needs some additional fixes.
>> 
>> Kind regards,
>> Leo
>> 
>> --
>> Leo Krüger, M.Sc.
>> Senior Systems Engineer Distributed Systems
>> Intelligent Digital Cabin
>> 
>> ZAL Zentrum für Angewandte Luftfahrtforschung GmbH
>> Hein-Saß-Weg 22
>> 21129 Hamburg
>>  
>> +49 (0) 40 248 595-154
>> 
>> zal.aero | twitter.com/ZALTechCenter | facebook.com/ZALTechCenter 
>> 
>> ZAL Zentrum für Angewandte Luftfahrtforschung GmbH 
>> Sitz der Gesellschaft / Legal Domicile: Hamburg 
>> Registergericht / Registration Court: Amtsgericht Hamburg HRB 110232
>> Vorsitzender des Aufsichtsrates / Chairman of the Supervisory Board: StR Andreas Rieckhof
>> Geschäftsführung / Board of Management: Roland Gerhards
>> 
>> Disclaimer:
>> This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have
>> received this mail in error), please notify the sender immediately and destroy this e-mail. Any unauthorised copying, 
>> disclosure or distribution of the material in this e-mail is strictly forbidden.
>> 
>>> -----Ursprüngliche Nachricht-----
>>> Von: Stefano Stabellini <stefano.stabellini@xilinx.com>
>>> Gesendet: Dienstag, 17. November 2020 01:59
>>> An: Leo Krueger <leo.krueger@zal.aero>
>>> Cc: Peng Fan <peng.fan@nxp.com>; Stefano Stabellini
>>> <stefano.stabellini@xilinx.com>; brucea@xilinx.com; Cornelia Bruelhart
>>> <cornelia.bruelhart@zal.aero>
>>> Betreff: Re: AW: AW: AW: Xen data from meta-virtualization layer
>>> 
>>> Replies inline below
>>> 
>>> 
>>> On Sun, 15 Nov 2020, Leo Krueger wrote:
>>>> Hi Peng, hi Stefano,
>>>> 
>>>> 
>>>> 
>>>> sorry for the long silence…
>>>> 
>>>> 
>>>> 
>>>> I tried the change suggested (and hope I didn’t do anything wrong
>>>> while doing so…) on top of XEN 4.13.2 (before, I always tried with
>>>> 4.12 but wanted to give 4.13.2 a try as well) but I do not see any difference,
>>> still the same “unhandled context fault” log entries pop up and I cannot
>>> access my sdcard.
>>>> 
>>>> 
>>>> 
>>>> As it seems to work without respectively disabled iommu, that would be
>>>> fine for me for now. What I am worried about a bit more is PCIe or
>>> MSI/MSIX to be exact.
>>>> 
>>>> 
>>>> 
>>>> Here is the gic-v3 and its node from my device tree:
>>>> 
>>>> 
>>>> 
>>>> interrupt-controller@6000000 {
>>>> 
>>>>         compatible = "arm,gic-v3";
>>>> 
>>>>         #address-cells = <0x2>;
>>>> 
>>>>         #size-cells = <0x2>;
>>>> 
>>>>         ranges;
>>>> 
>>>>         reg = <0x0 0x6000000 0x0 0x10000 0x0 0x6040000 0x0 0x40000>;
>>>> 
>>>>         #interrupt-cells = <0x3>;
>>>> 
>>>>         interrupt-controller;
>>>> 
>>>>         interrupts = <0x1 0x9 0xf08>;
>>>> 
>>>>         phandle = <0x1>;
>>>> 
>>>> 
>>>> 
>>>>         gic-its@6020000 {
>>>> 
>>>>                 compatible = "arm,gic-v3-its";
>>>> 
>>>>                 msi-controller;
>>>> 
>>>>                 reg = <0x0 0x6020000 0x0 0x20000>;
>>>> 
>>>>                 phandle = <0xd>;
>>>> 
>>>>         };
>>>> 
>>>> };
>>>> 
>>>> 
>>>> 
>>>> And here are some kernel log excerpts related to GIC when booting
>>> without (!) XEN:
>>>> 
>>>> 
>>>> 
>>>> [    0.000000] GICv3: GIC: Using split EOI/Deactivate mode
>>>> 
>>>> [    0.000000] GICv3: Distributor has no Range Selector support
>>>> 
>>>> [    0.000000] GICv3: no VLPI support, no direct LPI support
>>>> 
>>>> [    0.000000] ITS [mem 0x06020000-0x0603ffff]
>>>> 
>>>> [    0.000000] ITS@0x0000000006020000: allocated 65536 Devices
>>>> @20fb880000 (flat, esz 8, psz 64K, shr 0)
>>>> 
>>>> [    0.000000] ITS: using cache flushing for cmd queue
>>>> 
>>>> [    0.000000] GIC: using LPI property table @0x00000020fb830000
>>>> 
>>>> [    0.000000] GICv3: CPU0: found redistributor 0 region
>>>> 0:0x0000000006040000
>>>> 
>>>> [    0.000000] CPU0: using LPI pending table @0x00000020fb840000
>>>> 
>>>> [    0.000000] GIC: using cache flushing for LPI property table
>>>> 
>>>> 
>>>> 
>>>> However, when booting with XEN, only the following three lines are
>>> contained in the kernel log:
>>>> 
>>>> 
>>>> 
>>>> [    0.000000] GICv3: Distributor has no Range Selector support
>>>> 
>>>> [    0.000000] GICv3: no VLPI support, no direct LPI support
>>>> 
>>>> [    0.000000] GICv3: CPU0: found redistributor 0 region
>>>> 0:0x0000000006040000
>>> 
>>> "no VLPI support" is very suspicious, it looks like Dom0 doesn't find any ITS
>>> support.
>>> 
>>> Can you double check that you have the ITS driver in Xen built-in? That would
>>> be CONFIG_HAS_ITS. If you do "make menuconfig" and enable "Configure
>>> standard Xen features (expert users)" you should get a new option "GICv3
>>> ITS MSI controller support" under "Architecture Features".
>>> Make sure to enable it.
>>> 
>>> Let me know if that works!
> <devicetree.dts>

Regards,
Rahul