Re: XIVE VFIO kernel resample failure in INTx mode under heavy load

["Cédric Le Goater" <clg@kaod.org>]

Timothy,

On 3/16/22 17:29, Cédric Le Goater wrote:
> Hello,
> 
> 
>> I've been struggling for some time with what is looking like a
>> potential bug in QEMU/KVM on the POWER9 platform.  It appears that
>> in XIVE mode, when the in-kernel IRQ chip is enabled, an external
>> device that rapidly asserts IRQs via the legacy INTx level mechanism
>> will only receive one interrupt in the KVM guest.
> 
> Indeed. I could reproduce with a pass-through PCI adapter using
> 'pci=nomsi'. The virtio devices operate correctly but the network
> adapter only receives one event (*):
> 
> 
> $ cat /proc/interrupts
>             CPU0       CPU1       CPU2       CPU3       CPU4       CPU5       CPU6       CPU7
>   16:       2198       1378       1519       1216          0          0          0          0  XIVE-IPI   0 Edge      IPI-0
>   17:          0          0          0          0       2003       1936       1335       1507  XIVE-IPI   1 Edge      IPI-1
>   18:          0       6401          0          0          0          0          0          0  XIVE-IRQ 4609 Level     virtio3, virtio0, virtio2
>   19:          0          0          0          0          0        204          0          0  XIVE-IRQ 4610 Level     virtio1
>   20:          0          0          0          0          0          0          0          0  XIVE-IRQ 4608 Level     xhci-hcd:usb1
>   21:          0          1          0          0          0          0          0          0  XIVE-IRQ 4612 Level     eth1 (*)
>   23:          0          0          0          0          0          0          0          0  XIVE-IRQ 4096 Edge      RAS_EPOW
>   24:          0          0          0          0          0          0          0          0  XIVE-IRQ 4592 Edge      hvc_console
>   26:          0          0          0          0          0          0          0          0  XIVE-IRQ 4097 Edge      RAS_HOTPLUG
> 
>> Changing any one of those items appears to avoid the glitch, e.g. XICS
> 
> XICS is very different from XIVE. The driver implements the previous
> interrupt controller architecture (P5-P8) and the hypervisor mediates
> the delivery to the guest. With XIVE, vCPUs are directly signaled by
> the IC. When under KVM, we use different KVM devices for each mode :
> 
> * KVM XIVE is a XICS-on-XIVE implementation (P9/P10 hosts) for guests
>    not using the XIVE native interface. RHEL7 for instance.
> * KVM XIVE native is a XIVE implementation (P9/P10 hosts) for guests
>    using the XIVE native interface. Linux > 4.14.
> * KVM XICS is for P8 hosts (no XIVE HW)
> 
> VFIO adds some complexity with the source events. I think the problem
> comes from the assertion state. I will talk about it later.
> 
>> mode with the in-kernel IRQ chip works (all interrupts are passed
>> through),
> 
> All interrupts are passed through using XIVE also. Run 'info pic' in
> the monitor. On the host, check the IRQ mapping in the debugfs file :
> 
>    /sys/kernel/debug/powerpc/kvm-xive-*
> 
>> and XIVE mode with the in-kernel IRQ chip disabled also works. 
> 
> In that case, no KVM device backs the QEMU device and all state
> is in one place.
> 
>> We
>> are also not seeing any problems in XIVE mode with the in-kernel
>> chip from MSI/MSI-X devices.
> 
> Yes. pass-through devices are expected to operate correctly :)
> 
>> The device in question is a real time card that needs to raise an
>> interrupt every 1ms.  It works perfectly on the host, but fails in
>> the guest -- with the in-kernel IRQ chip and XIVE enabled, it
>> receives exactly one interrupt, at which point the host continues to
>> see INTx+ but the guest sees INTX-, and the IRQ handler in the guest
>> kernel is never reentered.
> 
> ok. Same symptom as the scenario above.
> 
>> We have also seen some very rare glitches where, over a long period
>> of time, we can enter a similar deadlock in XICS mode.
> 
> with the in-kernel XICS IRQ chip ?
> 
>> Disabling
>> the in-kernel IRQ chip in XIVE mode will also lead to the lockup
>> with this device, since the userspace IRQ emulation cannot keep up
>> with the rapid interrupt firing (measurements show around 100ms
>> required for processing each interrupt in the user mode).
> 
> MSI emulation in QEMU is slower indeed (35%). LSI is very slow because
> it is handled as a special case in the device/driver. To maintain the
> assertion state, all LSI handling is done with a special HCALL :
> H_INT_ESB which is implemented in QEMU. This generates a lot of back
> and forth in the KVM stack.
> 
>> My understanding is the resample mechanism does some clever tricks
>> with level IRQs, but that QEMU needs to check if the IRQ is still
>> asserted by the device on guest EOI.
> 
> Yes. the problem is in that area.
> 
>> Since a failure here would
>> explain these symptoms I'm wondering if there is a bug in either
>> QEMU or KVM for POWER / pSeries (SPAPr) where the IRQ is not
>> resampled and therefore not re-fired in the guest?
> 
> KVM I would say. The assertion state is maintained in KVM for the KVM
> XICS-on-XIVE implementation and in QEMU for the KVM XIVE native
> device. These are good candidates. I will take a look.

All works fine with KVM_CAP_IRQFD_RESAMPLE=false in QEMU. Can you please
try this workaround for now ? I could reach 934 Mbits/sec on the passthru
device.

I clearly overlooked that part and it has been 3 years.

Thanks,

C.