Re: [PATCH 0/5] Threaded MSI interrupt for VFIO PCI device

[Yunhong Jiang <yunhong.jiang@linux.intel.com>]

On Thu, Dec 03, 2015 at 11:55:53AM -0700, Alex Williamson wrote:
> On Thu, 2015-12-03 at 10:22 -0800, Yunhong Jiang wrote:
> > When assigning a VFIO device to a KVM guest with low latency requirement, it  
> > is better to handle the interrupt in the hard interrupt context, to reduce 
> > the context switch to/from the IRQ thread.
> > 
> > Based on discussion on https://lkml.org/lkml/2015/10/26/764, the VFIO msi 
> > interrupt is changed to use request_threaded_irq(). The primary interrupt 
> > handler tries to set the guest interrupt atomically. If it fails to achieve 
> > it, a threaded interrupt handler will be invoked.
> > 
> > The irq_bypass manager is extended for this purpose. The KVM eventfd will 
> > provide a irqbypass consumer to handle the interrupt at hard interrupt 
> > context. The producer will invoke the consumer's handler then.
> 
> Do you have any performance data?  Thanks,

Sorry, I should include the data on the commit messages.

The test:
Launch a VM with a FPGA device, which triggers an interrpt every 1ms. The VM 
is launched on a isolated CPU with NONHZ_FULL enabled.
Two data are collected.  On the guest, a special program will check the 
latency from the time the interrupt been triggered to the time the interrupt 
received by guest IRS. On the host, I use the perf to collect the perf data.

The performance Data with the patch:

Host perf data:
[root@otcnfv02 test-tools]# perf kvm stat record -C 34 sleep 10
[root@otcnfv02 test-tools]# perf kvm stat report
Analyze events for all VMs, all VCPUs:
	     VM-EXIT    Samples  Samples%     Time%    Min Time    Max Time
Avg time
  EXTERNAL_INTERRUPT       9997    98.55%    99.31%      1.73us     17.07us
2.09us ( +-   0.21% )
   PAUSE_INSTRUCTION        127     1.25%     0.51%      0.69us      1.20us
0.84us ( +-   1.34% )
	   MSR_WRITE         20     0.20%     0.18%      1.62us      3.21us
1.93us ( +-   3.95% )

Guest data:
[nfv@rt-test-1 ~]$ ./run-int-test.sh
Latency is Min: 3.74us Max: 20.08us Avg: 4.49us
No of interrupts = 74995

The performance data without the patch:
Host perf data:
[root@otcnfv02 test-tools]# perf kvm stat record -C 34 sleep 10
[root@otcnfv02 test-tools]# perf kvm stat report
Analyze events for all VMs, all VCPUs:
	     VM-EXIT    Samples  Samples%     Time%    Min Time    Max Time
Avg time
   PAUSE_INSTRUCTION     141136    87.74%    50.39%      0.69us      8.51us      
0.77us ( +-   0.07% )
  EXTERNAL_INTERRUPT      19701    12.25%    49.59%      2.31us     15.93us      
5.46us ( +-   0.12% )
	   MSR_WRITE         24     0.01%     0.02%      1.51us      2.22us      
1.91us ( +-   1.91% )

Notice:
The EXTERNAL_INTERRUPT VMExit is different w/ the patch (9997) and w/o the 
patch (19701). It is because with threaded IRQ, the NOHZ_FULL it not working 
because two threads on the pCPU, thus we have both the FPGA device interrupt 
and the tick timer interrupt. After calculation, the average time for the 
FPGA device interrupt is 4.72 us.

Guest data:
[nfv@rt-test-1 ~]$ ./run-int-test.sh
Latency is Min: 6.70us Max: 50.38us Avg: 7.44us
No of interrupts = 42639

Thanks
--jyh

> 
> Alex
>