From mboxrd@z Thu Jan  1 00:00:00 1970
From: Radim =?utf-8?B?S3LEjW3DocWZ?= <rkrcmar@redhat.com>
Subject: Re: Long interrupt latency in guest
Date: Mon, 30 Jan 2017 19:03:31 +0100
Message-ID: <20170130180330.GA4867@potion>
References: <3875C02542CA2945BF761013C1F5B8E57F3DDD50@ORSMSX109.amr.corp.intel.com>
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Cc: "kvm@vger.kernel.org" <kvm@vger.kernel.org>
To: "Su, David W" <david.w.su@intel.com>
Return-path: <kvm-owner@vger.kernel.org>
Received: from mx1.redhat.com ([209.132.183.28]:40148 "EHLO mx1.redhat.com"
        rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP
        id S1752174AbdA3SDo (ORCPT <rfc822;kvm@vger.kernel.org>);
        Mon, 30 Jan 2017 13:03:44 -0500
Content-Disposition: inline
In-Reply-To: <3875C02542CA2945BF761013C1F5B8E57F3DDD50@ORSMSX109.amr.corp.intel.com>
Sender: kvm-owner@vger.kernel.org
List-ID: <kvm.vger.kernel.org>

2017-01-25 23:57+0000, Su, David W:
> We use vfio-pci to expose a NIC device to a guest and a packet generator generating time stamped packets to measure latency.  The NIC is programmed to generate an interrupt when it receives a packet.  qemu cpu thread is bound to a fixed CPU core and the NIC interrupts are bound to the same core.  The host CPU supports APIC virtualization.  We have observed sometimes an interrupt is delayed for a relatively long time (mini seconds) before being delivered to the guest.
> 
> It seems there is a small window in the function vcpu_enter_guest in arch/x86/kvm/x86.c, where an interrupt from a device managed by vfio-pci is queued in PIR after PIR has synchronized to VIRR.  This will cause the interrupt not delivered to the guest until the next VM exit-entry cycle.  Refer to the code snippet below, if a device interrupt arrives after the KVM_REQ_EVENT check block and before local_irq_disable(), the interrupt request will be in PIR but not in VIRR.  In a worst case scenario, the interrupt would get lost if another interrupt from the device arrived before a VM exit occurred.

Good analysis.

>   /*
>    * KVM_REQ_EVENT is not set when posted interrupts are set by
>    * VT-d hardware, so we have to update RVI unconditionally.
>    */
>   if (kvm_lapic_enabled(vcpu)) {
>     /*
>      * Update architecture specific hints for APIC
>      * virtual interrupt delivery.
>      */
>     if (kvm_x86_ops->hwapic_irr_update)
>       kvm_x86_ops->hwapic_irr_update(vcpu,
>         kvm_lapic_find_highest_irr(vcpu));
>   }
> 
>   if (kvm_check_request(KVM_REQ_EVENT, vcpu) || req_int_win) {
> 
>   ...
>   ...
>   ...
> 
>   local_irq_disable();
> 
>   if (vcpu->mode == EXITING_GUEST_MODE || vcpu->requests
>       || need_resched() || signal_pending(current)) {
> 
> 
> Moving the "if (kvm_lapic_enabled(vcpu))" block to after the "if (vcpu->mode == EXITING_GUEST_MODE ..." block resolved the long interrupt latency issue in my limited testing, but I'm not sure if this is going to break something else.

hwapic_irr_update() or kvm_lapic_find_highest_irr() might set
vcpu->requests, so they should be just after local_irq_disable().
A drawback is that kvm_lapic_find_highest_irr() sets KVM_REQ_EVENT on
every PIR sync and thus forces useless re-runs of vcpu_enter_guest().

Paolo's pending patches eliminated the useless KVM_REQ_EVENT and also
moved the update just after local_irq_disable().
(Without noticing that it fixes a bug.)

Please verify that the delay goes away with patches from
https://lkml.org/lkml/2016/12/19/352

Thanks.