From mboxrd@z Thu Jan  1 00:00:00 1970
From: "Su, David W" <david.w.su@intel.com>
Subject: Long interrupt latency in guest
Date: Wed, 25 Jan 2017 23:57:33 +0000
Message-ID: <3875C02542CA2945BF761013C1F5B8E57F3DDD50@ORSMSX109.amr.corp.intel.com>
Mime-Version: 1.0
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: 8BIT
To: "kvm@vger.kernel.org" <kvm@vger.kernel.org>
Return-path: <kvm-owner@vger.kernel.org>
Received: from mga07.intel.com ([134.134.136.100]:24019 "EHLO mga07.intel.com"
        rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP
        id S1750998AbdAYX5g (ORCPT <rfc822;kvm@vger.kernel.org>);
        Wed, 25 Jan 2017 18:57:36 -0500
Content-Language: en-US
Sender: kvm-owner@vger.kernel.org
List-ID: <kvm.vger.kernel.org>

We use vfio-pci to expose a NIC device to a guest and a packet generator generating time stamped packets to measure latency.  The NIC is programmed to generate an interrupt when it receives a packet.  qemu cpu thread is bound to a fixed CPU core and the NIC interrupts are bound to the same core.  The host CPU supports APIC virtualization.  We have observed sometimes an interrupt is delayed for a relatively long time (mini seconds) before being delivered to the guest.

It seems there is a small window in the function vcpu_enter_guest in arch/x86/kvm/x86.c, where an interrupt from a device managed by vfio-pci is queued in PIR after PIR has synchronized to VIRR.  This will cause the interrupt not delivered to the guest until the next VM exit-entry cycle.  Refer to the code snippet below, if a device interrupt arrives after the KVM_REQ_EVENT check block and before local_irq_disable(), the interrupt request will be in PIR but not in VIRR.  In a worst case scenario, the interrupt would get lost if another interrupt from the device arrived before a VM exit occurred.

  /*
   * KVM_REQ_EVENT is not set when posted interrupts are set by
   * VT-d hardware, so we have to update RVI unconditionally.
   */
  if (kvm_lapic_enabled(vcpu)) {
    /*
     * Update architecture specific hints for APIC
     * virtual interrupt delivery.
     */
    if (kvm_x86_ops->hwapic_irr_update)
      kvm_x86_ops->hwapic_irr_update(vcpu,
        kvm_lapic_find_highest_irr(vcpu));
  }

  if (kvm_check_request(KVM_REQ_EVENT, vcpu) || req_int_win) {

  ...
  ...
  ...

  local_irq_disable();

  if (vcpu->mode == EXITING_GUEST_MODE || vcpu->requests
      || need_resched() || signal_pending(current)) {


Moving the "if (kvm_lapic_enabled(vcpu))" block to after the "if (vcpu->mode == EXITING_GUEST_MODE ..." block resolved the long interrupt latency issue in my limited testing, but I'm not sure if this is going to break something else.