Re: Keystone Issue

[Bertrand Marquis <Bertrand.Marquis@arm.com>]

Hi,

> On 9 Jun 2020, at 15:33, CodeWiz2280 <codewiz2280@gmail.com> wrote:
> 
> There does appear to be a secondary (CIC) controller that can forward
> events to the GIC-400 and EDMA controllers for the keystone 2 family.
> Admittedly, i'm not sure how it is being used with regards to the
> peripherals.  I only see mention of the GIC-400 parent for the devices
> in the device tree.  Maybe Bertrand has a better idea on whether any
> peripherals go through the CIC first?  I see that gic_interrupt ()
> fires once in Xen, which calls doIRQ to push out the virtual interrupt
> to the dom0 kernel.  The dom0 kernel then handles the interrupt and
> returns, but gic_interrupt() never fires again in Xen.

I do not remember of any CIC but the behaviour definitely look like an interrupt acknowledge problem.

Could you try the following:
--- a/xen/arch/arm/gic-v2.c
+++ b/xen/arch/arm/gic-v2.c
@@ -667,6 +667,9 @@ static void gicv2_guest_irq_end(struct irq_desc *desc)
     /* Lower the priority of the IRQ */
     gicv2_eoi_irq(desc);
     /* Deactivation happens in maintenance interrupt / via GICV */
+
+    /* Test for Keystone2 */
+    gicv2_dir_irq(desc);
 }

I think the problem I had was related to the vgic not deactivating properly the interrupt.
This might make the interrupt fire indefinitely !!

Regards
Bertrand

> 
> On Mon, Jun 8, 2020 at 12:13 PM Stefano Stabellini
> <sstabellini@kernel.org> wrote:
>> 
>> 
>> 
>> On Mon, 8 Jun 2020, CodeWiz2280 wrote:
>>> It actually shows only 1 interrupt for any of the devices in that list
>>> (e.g. spi, ttyS0, ethernet) so you're probably right on the money with
>>> it being an interrupt acknowledge issue.  Any help you can provide is
>>> greatly appreciated.
>>> 
>>> On Mon, Jun 8, 2020 at 4:40 AM Bertrand Marquis
>>> <Bertrand.Marquis@arm.com> wrote:
>>>> 
>>>> 
>>>> 
>>>>> On 5 Jun 2020, at 20:12, CodeWiz2280 <codewiz2280@gmail.com> wrote:
>>>>> 
>>>>> On Fri, Jun 5, 2020 at 11:05 AM CodeWiz2280 <codewiz2280@gmail.com> wrote:
>>>>>> 
>>>>>> On Fri, Jun 5, 2020 at 8:47 AM Bertrand Marquis
>>>>>> <Bertrand.Marquis@arm.com> wrote:
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>>> On 5 Jun 2020, at 13:42, CodeWiz2280 <codewiz2280@gmail.com> wrote:
>>>>>>>> 
>>>>>>>> On Fri, Jun 5, 2020 at 8:30 AM Julien Grall <julien@xen.org> wrote:
>>>>>>>>> 
>>>>>>>>> Hi,
>>>>>>>>> 
>>>>>>>>> On 05/06/2020 13:25, CodeWiz2280 wrote:
>>>>>>>>>> The Keystone uses the netcp driver, which has interrupts from 40-79
>>>>>>>>>> listed in the device tree (arch/arm/boot/keystone-k2e-netcp.dtsi).
>>>>>>>>>> I'm using the same device tree between my non-xen standalone kernel
>>>>>>>>>> and my dom0 kernel booted by xen.  In the standalone (non-xen) kernel
>>>>>>>>>> the ethernet works fine, but I don't see any of its interrupts in the
>>>>>>>>>> output of /proc/iomem.  I'm not seeing them in /proc/iomem when
>>>>>>>>>> running dom0 under Xen either.  When booting with Xen I get this
>>>>>>>>>> behavior where the ifconfig output shows 1 RX message and 1 TX
>>>>>>>>>> message, and then nothing else.
>>>>>>>>> 
>>>>>>>>> I am not sure whether this is a typo in the e-mail. /proc/iomem is
>>>>>>>>> listing the list of the MMIO regions. You want to use /proc/interrupts.
>>>>>>>>> 
>>>>>>>>> Can you confirm which path you are dumping?
>>>>>>>> Yes, that was a typo.  Sorry about that.  I meant that I am dumping
>>>>>>>> /proc/interrupts and do not
>>>>>>>> see them under the non-xen kernel or xen booted dom0.
>>>>>>> 
>>>>>>> Could you post both /proc/interrupts content ?
>>>>>> 
>>>>>> Standalone non-xen kernel (Ethernet works)
>>>>>> # cat /proc/interrupts
>>>>>>          CPU0       CPU1       CPU2       CPU3
>>>>>> 17:          0          0          0          0     GICv2  29 Level
>>>>>> arch_timer
>>>>>> 18:       9856       1202        457        650     GICv2  30 Level
>>>>>> arch_timer
>>>>>> 21:          0          0          0          0     GICv2 142 Edge
>>>>>> timer-keystone
>>>>>> 22:          0          0          0          0     GICv2  52 Edge      arm-pmu
>>>>>> 23:          0          0          0          0     GICv2  53 Edge      arm-pmu
>>>>>> 24:          0          0          0          0     GICv2  54 Edge      arm-pmu
>>>>>> 25:          0          0          0          0     GICv2  55 Edge      arm-pmu
>>>>>> 26:          0          0          0          0     GICv2  36 Edge
>>>>>> 26202a0.keystone_irq
>>>>>> 27:       1435          0          0          0     GICv2 309 Edge      ttyS0
>>>>>> 29:          0          0          0          0     GICv2 315 Edge
>>>>>> 2530000.i2c
>>>>>> 30:          1          0          0          0     GICv2 318 Edge
>>>>>> 2530400.i2c
>>>>>> 31:          0          0          0          0     GICv2 321 Edge
>>>>>> 2530800.i2c
>>>>>> 32:         69          0          0          0     GICv2 324 Edge
>>>>>> 21000400.spi
>>>>>> 33:          0          0          0          0     GICv2 328 Edge
>>>>>> 21000600.spi
>>>>>> 34:          0          0          0          0     GICv2 332 Edge
>>>>>> 21000800.spi
>>>>>> 70:          0          0          0          0     GICv2 417 Edge
>>>>>> ks-pcie-error-irq
>>>>>> 79:          0          0          0          0   PCI-MSI   0 Edge
>>>>>> PCIe PME, aerdrv
>>>>>> 88:         57          0          0          0     GICv2  80 Level
>>>>>> hwqueue-528
>>>>>> 89:         57          0          0          0     GICv2  81 Level
>>>>>> hwqueue-529
>>>>>> 90:         47          0          0          0     GICv2  82 Level
>>>>>> hwqueue-530
>>>>>> 91:         41          0          0          0     GICv2  83 Level
>>>>>> hwqueue-531
>>>>>> IPI0:          0          0          0          0  CPU wakeup interrupts
>>>>>> IPI1:          0          0          0          0  Timer broadcast interrupts
>>>>>> IPI2:        730        988       1058        937  Rescheduling interrupts
>>>>>> IPI3:          2          3          4          6  Function call interrupts
>>>>>> IPI4:          0          0          0          0  CPU stop interrupts
>>>>>> IPI5:          0          0          0          0  IRQ work interrupts
>>>>>> IPI6:          0          0          0          0  completion interrupts
>>>>>> 
>>>>>> Xen dom0 (Ethernet stops)
>>>>>> # cat /proc/interrupts
>>>>>>          CPU0
>>>>>> 18:      10380     GIC-0  27 Level     arch_timer
>>>>>> 19:          0     GIC-0 142 Edge      timer-keystone
>>>>>> 20:         88     GIC-0  16 Level     events
>>>>>> 21:          0   xen-dyn     Edge    -event     xenbus
>>>>>> 22:          0     GIC-0  36 Edge      26202a0.keystone_irq
>>>>>> 23:          1     GIC-0 312 Edge      ttyS0
>>>>>> 25:          1     GIC-0 318 Edge
>>>>>> 27:          1     GIC-0 324 Edge      21000400.spi
>>>>>> 28:          0     GIC-0 328 Edge      21000600.spi
>>>>>> 29:          0     GIC-0 332 Edge      21000800.spi
>>>>>> 65:          0     GIC-0 417 Edge      ks-pcie-error-irq
>>>>>> 74:          0   PCI-MSI   0 Edge      PCIe PME, aerdrv
>>>>>> 83:          1     GIC-0  80 Level     hwqueue-528
>>>>>> 84:          1     GIC-0  81 Level     hwqueue-529
>>>>>> 85:          1     GIC-0  82 Level     hwqueue-530
>>>>>> 86:          1     GIC-0  83 Level     hwqueue-531
>>>>>> 115:         87   xen-dyn     Edge    -virq      hvc_console
>>>>>> IPI0:          0  CPU wakeup interrupts
>>>>>> IPI1:          0  Timer broadcast interrupts
>>>>>> IPI2:          0  Rescheduling interrupts
>>>>>> IPI3:          0  Function call interrupts
>>>>>> IPI4:          0  CPU stop interrupts
>>>>>> IPI5:          0  IRQ work interrupts
>>>>>> IPI6:          0  completion interrupts
>>>>>> Err:          0
>>>>> After getting a chance to look at this a little more, I believe the
>>>>> TX/RX interrupts for the ethernets map like this:
>>>>> 
>>>>> eth0 Rx  - hwqueue-528
>>>>> eth1 Rx - hwqueue-529
>>>>> eth0 Tx  - hwqueue-530
>>>>> eth1 Tx - hwqueue-531
>>>>>> 
>>>>> The interrupt counts in the standlone working kernel seem to roughly
>>>>> correspond to the counts of Tx/Rx messages in ifconfig.  Going on
>>>>> that, its clear that only 1 interrupt has been received for Tx and 1
>>>>> for Rx in the Xen Dom0 equivalent.  Any thoughts on this?
>>>> 
>>>> This definitely look like an interrupt acknowledgement issue.
>>>> This could be caused by 2 things I remember of:
>>>> - front vs level interrupts
>>>> - a problem with forwarded interrupt acknowledgement.
>>>> I think there was something related to that where the vcpu ack was not properly
>>>> handled on a keystone and I had to change the way the interrupt was acked for
>>>> forwarded hardware interrupts.
>> 
>> Is there maybe some sort of secondary interrupt controller (secondary in
>> addition to the GIC) or interrupt "concentrator" on KeyStone?
>> 
>> Or is it just a small deviation from normal GIC behavior?