From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Tian, Kevin" Subject: RE: [PATCH 23/31] nVMX: Correct handling of interrupt injection Date: Wed, 25 May 2011 17:18:36 +0800 Message-ID: <625BA99ED14B2D499DC4E29D8138F1505C9BFA3AD4@shsmsx502.ccr.corp.intel.com> References: <1305575004-nyh@il.ibm.com> <201105161955.p4GJtgKc001996@rice.haifa.ibm.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 8BIT Cc: "gleb@redhat.com" , "avi@redhat.com" To: Nadav Har'El , "kvm@vger.kernel.org" Return-path: Received: from mga09.intel.com ([134.134.136.24]:41666 "EHLO mga09.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753315Ab1EYJTE convert rfc822-to-8bit (ORCPT ); Wed, 25 May 2011 05:19:04 -0400 In-Reply-To: <201105161955.p4GJtgKc001996@rice.haifa.ibm.com> Content-Language: en-US Sender: kvm-owner@vger.kernel.org List-ID: > From: Nadav Har'El > Sent: Tuesday, May 17, 2011 3:56 AM > > The code in this patch correctly emulates external-interrupt injection > while a nested guest L2 is running. > > Because of this code's relative un-obviousness, I include here a longer-than- > usual justification for what it does - much longer than the code itself ;-) > > To understand how to correctly emulate interrupt injection while L2 is > running, let's look first at what we need to emulate: How would things look > like if the extra L0 hypervisor layer is removed, and instead of L0 injecting > an interrupt, we had hardware delivering an interrupt? > > Now we have L1 running on bare metal with a guest L2, and the hardware > generates an interrupt. Assuming that L1 set PIN_BASED_EXT_INTR_MASK to > 1, and > VM_EXIT_ACK_INTR_ON_EXIT to 0 (we'll revisit these assumptions below), > what > happens now is this: The processor exits from L2 to L1, with an external- > interrupt exit reason but without an interrupt vector. L1 runs, with > interrupts disabled, and it doesn't yet know what the interrupt was. Soon > after, it enables interrupts and only at that moment, it gets the interrupt > from the processor. when L1 is KVM, Linux handles this interrupt. > > Now we need exactly the same thing to happen when that L1->L2 system runs > on top of L0, instead of real hardware. This is how we do this: > > When L0 wants to inject an interrupt, it needs to exit from L2 to L1, with > external-interrupt exit reason (with an invalid interrupt vector), and run L1. > Just like in the bare metal case, it likely can't deliver the interrupt to > L1 now because L1 is running with interrupts disabled, in which case it turns > on the interrupt window when running L1 after the exit. L1 will soon enable > interrupts, and at that point L0 will gain control again and inject the > interrupt to L1. > > Finally, there is an extra complication in the code: when nested_run_pending, > we cannot return to L1 now, and must launch L2. We need to remember the > interrupt we wanted to inject (and not clear it now), and do it on the > next exit. > > The above explanation shows that the relative strangeness of the nested > interrupt injection code in this patch, and the extra interrupt-window > exit incurred, are in fact necessary for accurate emulation, and are not > just an unoptimized implementation. > > Let's revisit now the two assumptions made above: > > If L1 turns off PIN_BASED_EXT_INTR_MASK (no hypervisor that I know > does, by the way), things are simple: L0 may inject the interrupt directly > to the L2 guest - using the normal code path that injects to any guest. > We support this case in the code below. > > If L1 turns on VM_EXIT_ACK_INTR_ON_EXIT (again, no hypervisor that I know > does), things look very different from the description above: L1 expects > to see an exit from L2 with the interrupt vector already filled in the exit > information, and does not expect to be interrupted again with this interrupt. > The current code does not (yet) support this case, so we do not allow the > VM_EXIT_ACK_INTR_ON_EXIT exit-control to be turned on by L1. > > Signed-off-by: Nadav Har'El > --- > arch/x86/kvm/vmx.c | 36 ++++++++++++++++++++++++++++++++++++ > 1 file changed, 36 insertions(+) > > --- .before/arch/x86/kvm/vmx.c 2011-05-16 22:36:49.000000000 +0300 > +++ .after/arch/x86/kvm/vmx.c 2011-05-16 22:36:49.000000000 +0300 > @@ -1788,6 +1788,7 @@ static __init void nested_vmx_setup_ctls > > /* exit controls */ > nested_vmx_exit_ctls_low = 0; > + /* Note that guest use of VM_EXIT_ACK_INTR_ON_EXIT is not supported. > */ > #ifdef CONFIG_X86_64 > nested_vmx_exit_ctls_high = VM_EXIT_HOST_ADDR_SPACE_SIZE; > #else > @@ -3733,9 +3734,25 @@ out: > return ret; > } > > +/* > + * In nested virtualization, check if L1 asked to exit on external interrupts. > + * For most existing hypervisors, this will always return true. > + */ > +static bool nested_exit_on_intr(struct kvm_vcpu *vcpu) > +{ > + return get_vmcs12(vcpu)->pin_based_vm_exec_control & > + PIN_BASED_EXT_INTR_MASK; > +} > + > static void enable_irq_window(struct kvm_vcpu *vcpu) > { > u32 cpu_based_vm_exec_control; > + if (is_guest_mode(vcpu) && nested_exit_on_intr(vcpu)) > + /* We can get here when nested_run_pending caused > + * vmx_interrupt_allowed() to return false. In this case, do > + * nothing - the interrupt will be injected later. > + */ I think this is not a rare path? when vcpu is in guest mode with L2 as current vmx context, this function could be invoked multiple times since kvm thread can be scheduled out/in randomly. > + return; > > cpu_based_vm_exec_control = > vmcs_read32(CPU_BASED_VM_EXEC_CONTROL); > cpu_based_vm_exec_control |= CPU_BASED_VIRTUAL_INTR_PENDING; > @@ -3858,6 +3875,17 @@ static void vmx_set_nmi_mask(struct kvm_ > > static int vmx_interrupt_allowed(struct kvm_vcpu *vcpu) > { > + if (is_guest_mode(vcpu) && nested_exit_on_intr(vcpu)) { > + struct vmcs12 *vmcs12; > + if (to_vmx(vcpu)->nested.nested_run_pending) > + return 0; Well, now I can see why you require this special 'nested_run_pending' flag because there're places where L0 injects virtual interrupts right after VMLAUNCH/VMRESUME emulation and before entering L2. :-) > + nested_vmx_vmexit(vcpu); > + vmcs12 = get_vmcs12(vcpu); > + vmcs12->vm_exit_reason = EXIT_REASON_EXTERNAL_INTERRUPT; > + vmcs12->vm_exit_intr_info = 0; > + /* fall through to normal code, but now in L1, not L2 */ > + } > + This is a bad place to add this logic. vmx_interrupt_allowed is simply a query function but you make it an absolute trigger point for switching from L2 to L1. This is fine as now only point calling vmx_interrupt_allowed is when there's vNMI pending. But it's dangerous to have such assumption for pending events inside vmx_interrupt_allowed. On the other hand, I think there's one area which is not handled timely. I think you need to kick a L2->L1 transition when L0 wants to inject virtual interrupt. Consider your current logic: a) L2 is running on cpu1 b) L0 on cpu 0 decides to post a virtual interrupt to L1. An IPI is issued to cpu1 after updating virqchip c) L2 on cpu0 vmexit to L0, and checks whether L0 or L1 should handle the event. As it's an external interrupt, L0 will handle it. As it's a notification IPI, nothing is required. d) L0 on cpu0 then decides to resume, and find KVM_REQ_EVENT At this point you only add logic to enable_irq_window, but there's no action to trigger L2->L1 transition. So what will happen? Will the event be injected into L2 instead or pend until next switch happens due to other cause? Thanks Kevin