Re: Keystone Issue

[CodeWiz2280 <codewiz2280@gmail.com>]

There does appear to be a secondary (CIC) controller that can forward
events to the GIC-400 and EDMA controllers for the keystone 2 family.
Admittedly, i'm not sure how it is being used with regards to the
peripherals.  I only see mention of the GIC-400 parent for the devices
in the device tree.  Maybe Bertrand has a better idea on whether any
peripherals go through the CIC first?  I see that gic_interrupt ()
fires once in Xen, which calls doIRQ to push out the virtual interrupt
to the dom0 kernel.  The dom0 kernel then handles the interrupt and
returns, but gic_interrupt() never fires again in Xen.

On Mon, Jun 8, 2020 at 12:13 PM Stefano Stabellini
<sstabellini@kernel.org> wrote:
>
>
>
> On Mon, 8 Jun 2020, CodeWiz2280 wrote:
> > It actually shows only 1 interrupt for any of the devices in that list
> > (e.g. spi, ttyS0, ethernet) so you're probably right on the money with
> > it being an interrupt acknowledge issue.  Any help you can provide is
> > greatly appreciated.
> >
> > On Mon, Jun 8, 2020 at 4:40 AM Bertrand Marquis
> > <Bertrand.Marquis@arm.com> wrote:
> > >
> > >
> > >
> > > > On 5 Jun 2020, at 20:12, CodeWiz2280 <codewiz2280@gmail.com> wrote:
> > > >
> > > > On Fri, Jun 5, 2020 at 11:05 AM CodeWiz2280 <codewiz2280@gmail.com> wrote:
> > > >>
> > > >> On Fri, Jun 5, 2020 at 8:47 AM Bertrand Marquis
> > > >> <Bertrand.Marquis@arm.com> wrote:
> > > >>>
> > > >>>
> > > >>>
> > > >>>> On 5 Jun 2020, at 13:42, CodeWiz2280 <codewiz2280@gmail.com> wrote:
> > > >>>>
> > > >>>> On Fri, Jun 5, 2020 at 8:30 AM Julien Grall <julien@xen.org> wrote:
> > > >>>>>
> > > >>>>> Hi,
> > > >>>>>
> > > >>>>> On 05/06/2020 13:25, CodeWiz2280 wrote:
> > > >>>>>> The Keystone uses the netcp driver, which has interrupts from 40-79
> > > >>>>>> listed in the device tree (arch/arm/boot/keystone-k2e-netcp.dtsi).
> > > >>>>>> I'm using the same device tree between my non-xen standalone kernel
> > > >>>>>> and my dom0 kernel booted by xen.  In the standalone (non-xen) kernel
> > > >>>>>> the ethernet works fine, but I don't see any of its interrupts in the
> > > >>>>>> output of /proc/iomem.  I'm not seeing them in /proc/iomem when
> > > >>>>>> running dom0 under Xen either.  When booting with Xen I get this
> > > >>>>>> behavior where the ifconfig output shows 1 RX message and 1 TX
> > > >>>>>> message, and then nothing else.
> > > >>>>>
> > > >>>>> I am not sure whether this is a typo in the e-mail. /proc/iomem is
> > > >>>>> listing the list of the MMIO regions. You want to use /proc/interrupts.
> > > >>>>>
> > > >>>>> Can you confirm which path you are dumping?
> > > >>>> Yes, that was a typo.  Sorry about that.  I meant that I am dumping
> > > >>>> /proc/interrupts and do not
> > > >>>> see them under the non-xen kernel or xen booted dom0.
> > > >>>
> > > >>> Could you post both /proc/interrupts content ?
> > > >>
> > > >> Standalone non-xen kernel (Ethernet works)
> > > >> # cat /proc/interrupts
> > > >>           CPU0       CPU1       CPU2       CPU3
> > > >> 17:          0          0          0          0     GICv2  29 Level
> > > >>  arch_timer
> > > >> 18:       9856       1202        457        650     GICv2  30 Level
> > > >>  arch_timer
> > > >> 21:          0          0          0          0     GICv2 142 Edge
> > > >>  timer-keystone
> > > >> 22:          0          0          0          0     GICv2  52 Edge      arm-pmu
> > > >> 23:          0          0          0          0     GICv2  53 Edge      arm-pmu
> > > >> 24:          0          0          0          0     GICv2  54 Edge      arm-pmu
> > > >> 25:          0          0          0          0     GICv2  55 Edge      arm-pmu
> > > >> 26:          0          0          0          0     GICv2  36 Edge
> > > >>  26202a0.keystone_irq
> > > >> 27:       1435          0          0          0     GICv2 309 Edge      ttyS0
> > > >> 29:          0          0          0          0     GICv2 315 Edge
> > > >>  2530000.i2c
> > > >> 30:          1          0          0          0     GICv2 318 Edge
> > > >>  2530400.i2c
> > > >> 31:          0          0          0          0     GICv2 321 Edge
> > > >>  2530800.i2c
> > > >> 32:         69          0          0          0     GICv2 324 Edge
> > > >>  21000400.spi
> > > >> 33:          0          0          0          0     GICv2 328 Edge
> > > >>  21000600.spi
> > > >> 34:          0          0          0          0     GICv2 332 Edge
> > > >>  21000800.spi
> > > >> 70:          0          0          0          0     GICv2 417 Edge
> > > >>  ks-pcie-error-irq
> > > >> 79:          0          0          0          0   PCI-MSI   0 Edge
> > > >>  PCIe PME, aerdrv
> > > >> 88:         57          0          0          0     GICv2  80 Level
> > > >>  hwqueue-528
> > > >> 89:         57          0          0          0     GICv2  81 Level
> > > >>  hwqueue-529
> > > >> 90:         47          0          0          0     GICv2  82 Level
> > > >>  hwqueue-530
> > > >> 91:         41          0          0          0     GICv2  83 Level
> > > >>  hwqueue-531
> > > >> IPI0:          0          0          0          0  CPU wakeup interrupts
> > > >> IPI1:          0          0          0          0  Timer broadcast interrupts
> > > >> IPI2:        730        988       1058        937  Rescheduling interrupts
> > > >> IPI3:          2          3          4          6  Function call interrupts
> > > >> IPI4:          0          0          0          0  CPU stop interrupts
> > > >> IPI5:          0          0          0          0  IRQ work interrupts
> > > >> IPI6:          0          0          0          0  completion interrupts
> > > >>
> > > >> Xen dom0 (Ethernet stops)
> > > >> # cat /proc/interrupts
> > > >>           CPU0
> > > >> 18:      10380     GIC-0  27 Level     arch_timer
> > > >> 19:          0     GIC-0 142 Edge      timer-keystone
> > > >> 20:         88     GIC-0  16 Level     events
> > > >> 21:          0   xen-dyn     Edge    -event     xenbus
> > > >> 22:          0     GIC-0  36 Edge      26202a0.keystone_irq
> > > >> 23:          1     GIC-0 312 Edge      ttyS0
> > > >> 25:          1     GIC-0 318 Edge
> > > >> 27:          1     GIC-0 324 Edge      21000400.spi
> > > >> 28:          0     GIC-0 328 Edge      21000600.spi
> > > >> 29:          0     GIC-0 332 Edge      21000800.spi
> > > >> 65:          0     GIC-0 417 Edge      ks-pcie-error-irq
> > > >> 74:          0   PCI-MSI   0 Edge      PCIe PME, aerdrv
> > > >> 83:          1     GIC-0  80 Level     hwqueue-528
> > > >> 84:          1     GIC-0  81 Level     hwqueue-529
> > > >> 85:          1     GIC-0  82 Level     hwqueue-530
> > > >> 86:          1     GIC-0  83 Level     hwqueue-531
> > > >> 115:         87   xen-dyn     Edge    -virq      hvc_console
> > > >> IPI0:          0  CPU wakeup interrupts
> > > >> IPI1:          0  Timer broadcast interrupts
> > > >> IPI2:          0  Rescheduling interrupts
> > > >> IPI3:          0  Function call interrupts
> > > >> IPI4:          0  CPU stop interrupts
> > > >> IPI5:          0  IRQ work interrupts
> > > >> IPI6:          0  completion interrupts
> > > >> Err:          0
> > > > After getting a chance to look at this a little more, I believe the
> > > > TX/RX interrupts for the ethernets map like this:
> > > >
> > > > eth0 Rx  - hwqueue-528
> > > > eth1 Rx - hwqueue-529
> > > > eth0 Tx  - hwqueue-530
> > > > eth1 Tx - hwqueue-531
> > > >>
> > > > The interrupt counts in the standlone working kernel seem to roughly
> > > > correspond to the counts of Tx/Rx messages in ifconfig.  Going on
> > > > that, its clear that only 1 interrupt has been received for Tx and 1
> > > > for Rx in the Xen Dom0 equivalent.  Any thoughts on this?
> > >
> > > This definitely look like an interrupt acknowledgement issue.
> > > This could be caused by 2 things I remember of:
> > > - front vs level interrupts
> > > - a problem with forwarded interrupt acknowledgement.
> > > I think there was something related to that where the vcpu ack was not properly
> > > handled on a keystone and I had to change the way the interrupt was acked for
> > > forwarded hardware interrupts.
>
> Is there maybe some sort of secondary interrupt controller (secondary in
> addition to the GIC) or interrupt "concentrator" on KeyStone?
>
> Or is it just a small deviation from normal GIC behavior?