Adding Bertrand, Oleksandr, Julien, and others -- they have a more recent experience with GICv3 ITS than me and might be able to help. I am attaching the device tree Leo sent a few days ago for reference. Typically when you can set the ethernet link up and no packets are exchanged it is because of a missing interrupt. In this case a missing MSI. Bertrand, I believe you tried the GIC ITS driver with PCI devices recently. It is expected to work correctly with MSIs in Dom0, right? On Tue, 17 Nov 2020, Leo Krueger wrote: > Hi, > > I enabled CONFIG_HAS_ITS (what a stupid mistake by me to not set it before...) but then had to add the following node to my device tree > > gic_lpi_base: syscon@0x80000000 { > compatible = "gic-lpi-base"; > reg = <0x0 0x80000000 0x0 0x100000>; > max-gic-redistributors = <2>; > }; > > to somehow change something in regard to the ITS and MSI/MSI-X > > (XEN) GICv3 initialization: > (XEN) gic_dist_addr=0x00000006000000 > (XEN) gic_maintenance_irq=25 > (XEN) gic_rdist_stride=0 > (XEN) gic_rdist_regions=1 > (XEN) redistributor regions: > (XEN) - region 0: 0x00000006040000 - 0x00000006080000 > (XEN) GICv3: using at most 57344 LPIs on the host. > (XEN) GICv3: 288 lines, (IID 0001143b). > (XEN) GICv3: Found ITS @0x6020000 > (XEN) using non-cacheable ITS command queue > (XEN) GICv3: CPU0: Found redistributor in region 0 @000000004001c000 > > [ 0.000000] GICv3: Distributor has no Range Selector support > [ 0.000000] GICv3: no VLPI support, no direct LPI support > [ 0.000000] ITS [mem 0x06020000-0x0603ffff] > [ 0.000000] ITS@0x0000000006020000: allocated 65536 Devices @dc880000 (flat, esz 8, psz 64K, shr 1) > [ 0.000000] ITS@0x0000000006020000: allocated 32768 Interrupt Collections @dc820000 (flat, esz 2, psz 64K, shr 1) > [ 0.000000] GIC: using LPI property table @0x00000000dc830000 > [ 0.000000] GICv3: CPU0: found redistributor 0 region 0:0x0000000006040000 > [ 0.000000] CPU0: using LPI pending table @0x00000000dc840000 > ... > [ 0.040080] Platform MSI: gic-its domain created > [ 0.040136] PCI/MSI: /interrupt-controller/gic-its domain created > [ 0.040181] fsl-mc MSI: /interrupt-controller/gic-its domain created > > > Still I am ending up with the " Failed to add - passthrough or MSI/MSI-X might fail!" log messages for some PCI devices, but at least the on-board ethernet ports (fsl_enetc ) are initialized. > I can set the link up and a link is successfully established. > > But (!) I cannot receive or transmit anything (no error message...) and there seem to be no interrupts: > > 29: 0 ITS-MSI 1 Edge gbe0-rxtx0 > 32: 0 ITS-MSI 8192 Edge ptp_qoriq > > (from /proc/interrupts). > > Any idea on this one? I keep digging and checking whether my device tree needs some additional fixes. > > Kind regards, > Leo > > -- > Leo Krüger, M.Sc. > Senior Systems Engineer Distributed Systems > Intelligent Digital Cabin > > ZAL Zentrum für Angewandte Luftfahrtforschung GmbH > Hein-Saß-Weg 22 > 21129 Hamburg >   > +49 (0) 40 248 595-154 > > zal.aero | twitter.com/ZALTechCenter | facebook.com/ZALTechCenter > > ZAL Zentrum für Angewandte Luftfahrtforschung GmbH > Sitz der Gesellschaft / Legal Domicile: Hamburg > Registergericht / Registration Court: Amtsgericht Hamburg HRB 110232 > Vorsitzender des Aufsichtsrates / Chairman of the Supervisory Board: StR Andreas Rieckhof > Geschäftsführung / Board of Management: Roland Gerhards > > Disclaimer: > This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have > received this mail in error), please notify the sender immediately and destroy this e-mail. Any unauthorised copying, > disclosure or distribution of the material in this e-mail is strictly forbidden. > > > -----Ursprüngliche Nachricht----- > > Von: Stefano Stabellini > > Gesendet: Dienstag, 17. November 2020 01:59 > > An: Leo Krueger > > Cc: Peng Fan ; Stefano Stabellini > > ; brucea@xilinx.com; Cornelia Bruelhart > > > > Betreff: Re: AW: AW: AW: Xen data from meta-virtualization layer > > > > Replies inline below > > > > > > On Sun, 15 Nov 2020, Leo Krueger wrote: > > > Hi Peng, hi Stefano, > > > > > > > > > > > > sorry for the long silence… > > > > > > > > > > > > I tried the change suggested (and hope I didn’t do anything wrong > > > while doing so…) on top of XEN 4.13.2 (before, I always tried with > > > 4.12 but wanted to give 4.13.2 a try as well) but I do not see any difference, > > still the same “unhandled context fault” log entries pop up and I cannot > > access my sdcard. > > > > > > > > > > > > As it seems to work without respectively disabled iommu, that would be > > > fine for me for now. What I am worried about a bit more is PCIe or > > MSI/MSIX to be exact. > > > > > > > > > > > > Here is the gic-v3 and its node from my device tree: > > > > > > > > > > > > interrupt-controller@6000000 { > > > > > >         compatible = "arm,gic-v3"; > > > > > >         #address-cells = <0x2>; > > > > > >         #size-cells = <0x2>; > > > > > >         ranges; > > > > > >         reg = <0x0 0x6000000 0x0 0x10000 0x0 0x6040000 0x0 0x40000>; > > > > > >         #interrupt-cells = <0x3>; > > > > > >         interrupt-controller; > > > > > >         interrupts = <0x1 0x9 0xf08>; > > > > > >         phandle = <0x1>; > > > > > > > > > > > >         gic-its@6020000 { > > > > > >                 compatible = "arm,gic-v3-its"; > > > > > >                 msi-controller; > > > > > >                 reg = <0x0 0x6020000 0x0 0x20000>; > > > > > >                 phandle = <0xd>; > > > > > >         }; > > > > > > }; > > > > > > > > > > > > And here are some kernel log excerpts related to GIC when booting > > without (!) XEN: > > > > > > > > > > > > [    0.000000] GICv3: GIC: Using split EOI/Deactivate mode > > > > > > [    0.000000] GICv3: Distributor has no Range Selector support > > > > > > [    0.000000] GICv3: no VLPI support, no direct LPI support > > > > > > [    0.000000] ITS [mem 0x06020000-0x0603ffff] > > > > > > [    0.000000] ITS@0x0000000006020000: allocated 65536 Devices > > > @20fb880000 (flat, esz 8, psz 64K, shr 0) > > > > > > [    0.000000] ITS: using cache flushing for cmd queue > > > > > > [    0.000000] GIC: using LPI property table @0x00000020fb830000 > > > > > > [    0.000000] GICv3: CPU0: found redistributor 0 region > > > 0:0x0000000006040000 > > > > > > [    0.000000] CPU0: using LPI pending table @0x00000020fb840000 > > > > > > [    0.000000] GIC: using cache flushing for LPI property table > > > > > > > > > > > > However, when booting with XEN, only the following three lines are > > contained in the kernel log: > > > > > > > > > > > > [    0.000000] GICv3: Distributor has no Range Selector support > > > > > > [    0.000000] GICv3: no VLPI support, no direct LPI support > > > > > > [    0.000000] GICv3: CPU0: found redistributor 0 region > > > 0:0x0000000006040000 > > > > "no VLPI support" is very suspicious, it looks like Dom0 doesn't find any ITS > > support. > > > > Can you double check that you have the ITS driver in Xen built-in? That would > > be CONFIG_HAS_ITS. If you do "make menuconfig" and enable "Configure > > standard Xen features (expert users)" you should get a new option "GICv3 > > ITS MSI controller support" under "Architecture Features". > > Make sure to enable it. > > > > Let me know if that works! >