Re: XIVE VFIO kernel resample failure in INTx mode under heavy load

[Alexey Kardashevskiy <aik@ozlabs.ru>]

On 3/17/22 06:16, Cédric Le Goater wrote:
> Timothy,
> 
> On 3/16/22 17:29, Cédric Le Goater wrote:
>> Hello,
>>
>>
>>> I've been struggling for some time with what is looking like a
>>> potential bug in QEMU/KVM on the POWER9 platform.  It appears that
>>> in XIVE mode, when the in-kernel IRQ chip is enabled, an external
>>> device that rapidly asserts IRQs via the legacy INTx level mechanism
>>> will only receive one interrupt in the KVM guest.
>>
>> Indeed. I could reproduce with a pass-through PCI adapter using
>> 'pci=nomsi'. The virtio devices operate correctly but the network
>> adapter only receives one event (*):
>>
>>
>> $ cat /proc/interrupts
>>             CPU0       CPU1       CPU2       CPU3       CPU4       
>> CPU5       CPU6       CPU7
>>   16:       2198       1378       1519       1216          0          
>> 0          0          0  XIVE-IPI   0 Edge      IPI-0
>>   17:          0          0          0          0       2003       
>> 1936       1335       1507  XIVE-IPI   1 Edge      IPI-1
>>   18:          0       6401          0          0          0          
>> 0          0          0  XIVE-IRQ 4609 Level     virtio3, virtio0, 
>> virtio2
>>   19:          0          0          0          0          0        
>> 204          0          0  XIVE-IRQ 4610 Level     virtio1
>>   20:          0          0          0          0          0          
>> 0          0          0  XIVE-IRQ 4608 Level     xhci-hcd:usb1
>>   21:          0          1          0          0          0          
>> 0          0          0  XIVE-IRQ 4612 Level     eth1 (*)
>>   23:          0          0          0          0          0          
>> 0          0          0  XIVE-IRQ 4096 Edge      RAS_EPOW
>>   24:          0          0          0          0          0          
>> 0          0          0  XIVE-IRQ 4592 Edge      hvc_console
>>   26:          0          0          0          0          0          
>> 0          0          0  XIVE-IRQ 4097 Edge      RAS_HOTPLUG
>>
>>> Changing any one of those items appears to avoid the glitch, e.g. XICS
>>
>> XICS is very different from XIVE. The driver implements the previous
>> interrupt controller architecture (P5-P8) and the hypervisor mediates
>> the delivery to the guest. With XIVE, vCPUs are directly signaled by
>> the IC. When under KVM, we use different KVM devices for each mode :
>>
>> * KVM XIVE is a XICS-on-XIVE implementation (P9/P10 hosts) for guests
>>    not using the XIVE native interface. RHEL7 for instance.
>> * KVM XIVE native is a XIVE implementation (P9/P10 hosts) for guests
>>    using the XIVE native interface. Linux > 4.14.
>> * KVM XICS is for P8 hosts (no XIVE HW)
>>
>> VFIO adds some complexity with the source events. I think the problem
>> comes from the assertion state. I will talk about it later.
>>
>>> mode with the in-kernel IRQ chip works (all interrupts are passed
>>> through),
>>
>> All interrupts are passed through using XIVE also. Run 'info pic' in
>> the monitor. On the host, check the IRQ mapping in the debugfs file :
>>
>>    /sys/kernel/debug/powerpc/kvm-xive-*
>>
>>> and XIVE mode with the in-kernel IRQ chip disabled also works. 
>>
>> In that case, no KVM device backs the QEMU device and all state
>> is in one place.
>>
>>> We
>>> are also not seeing any problems in XIVE mode with the in-kernel
>>> chip from MSI/MSI-X devices.
>>
>> Yes. pass-through devices are expected to operate correctly :)
>>
>>> The device in question is a real time card that needs to raise an
>>> interrupt every 1ms.  It works perfectly on the host, but fails in
>>> the guest -- with the in-kernel IRQ chip and XIVE enabled, it
>>> receives exactly one interrupt, at which point the host continues to
>>> see INTx+ but the guest sees INTX-, and the IRQ handler in the guest
>>> kernel is never reentered.
>>
>> ok. Same symptom as the scenario above.
>>
>>> We have also seen some very rare glitches where, over a long period
>>> of time, we can enter a similar deadlock in XICS mode.
>>
>> with the in-kernel XICS IRQ chip ?
>>
>>> Disabling
>>> the in-kernel IRQ chip in XIVE mode will also lead to the lockup
>>> with this device, since the userspace IRQ emulation cannot keep up
>>> with the rapid interrupt firing (measurements show around 100ms
>>> required for processing each interrupt in the user mode).
>>
>> MSI emulation in QEMU is slower indeed (35%). LSI is very slow because
>> it is handled as a special case in the device/driver. To maintain the
>> assertion state, all LSI handling is done with a special HCALL :
>> H_INT_ESB which is implemented in QEMU. This generates a lot of back
>> and forth in the KVM stack.
>>
>>> My understanding is the resample mechanism does some clever tricks
>>> with level IRQs, but that QEMU needs to check if the IRQ is still
>>> asserted by the device on guest EOI.
>>
>> Yes. the problem is in that area.
>>
>>> Since a failure here would
>>> explain these symptoms I'm wondering if there is a bug in either
>>> QEMU or KVM for POWER / pSeries (SPAPr) where the IRQ is not
>>> resampled and therefore not re-fired in the guest?
>>
>> KVM I would say. The assertion state is maintained in KVM for the KVM
>> XICS-on-XIVE implementation and in QEMU for the KVM XIVE native
>> device. These are good candidates. I will take a look.
> 
> All works fine with KVM_CAP_IRQFD_RESAMPLE=false in QEMU. Can you please
> try this workaround for now ? I could reach 934 Mbits/sec on the passthru
> device.
> 
> I clearly overlooked that part and it has been 3 years.


Disabling KVM_CAP_IRQFD_RESAMPLE on XIVE-capable machines seems to be 
the right fix actually.

XIVE == baremetal/vm POWER9 and newer.
XICS == baremetal/vm POWER8 and older, or VMs on any POWER (backward 
compat.).

Tested on POWER9 with a passed through XHCI host and "-append pci=nomsi" 
and "-machine pseries,ic-mode=xics,kernel_irqchip=on" (and s/xics/xive/).

When it is XIVE-on-XIVE (host and guest are XIVE), INTx is emulated in 
the QEMU's H_INT_ESB handler and IRQFD_RESAMPLE is just useless in such 
case (as it is designed to eliminate going to the userspace for the 
EOI->INTx unmasking) and there is no pathway to call the eventfd's 
irqfd_resampler_ack() from QEMU. So the VM's XHCI device receives 
exactly 1 interrupt and that is it. "kernel_irqchip=off" fixes it 
(obviously).

When it is XICS-on-XIVE (host is XIVE and guest is XICS), then the VM 
receives 100000 interrupts and then it gets frozen (__report_bad_irq() 
is called). Which happens because (unlike XICS-on-XICS), the host XIVE's 
xive_(rm|vm)_h_eoi() does not call irqfd_resampler_ack(). This fixes it:

=============
diff --git a/arch/powerpc/kvm/book3s_xive_template.c 
b/arch/powerpc/kvm/book3s_xive_template.c
index b0015e05d99a..9f0d8e5c7f4b 100644 

--- a/arch/powerpc/kvm/book3s_xive_template.c 

+++ b/arch/powerpc/kvm/book3s_xive_template.c 

@@ -595,6 +595,8 @@ X_STATIC int GLUE(X_PFX,h_eoi)(struct kvm_vcpu 
*vcpu, unsigned long xirr)
         xc->hw_cppr = xc->cppr; 

         __x_writeb(xc->cppr, __x_tima + TM_QW1_OS + TM_CPPR); 

 

+       kvm_notify_acked_irq(vcpu->kvm, 0, irq); 

+ 

         return rc; 

  } 

=============

The host's XICS does call kvm_notify_acked_irq() (I did not test that 
but the code seems to be doing so).

After re-reading what I just wrote, I am leaning towards disabling use 
of KVM_CAP_IRQFD_RESAMPLE as it seems last worked on POWER8 and never 
since :)

Did I miss something in the picture (hey Cedric)?


--
Alexey