From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753312AbdKIKsl (ORCPT ); Thu, 9 Nov 2017 05:48:41 -0500 Received: from mx1.redhat.com ([209.132.183.28]:40180 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752930AbdKIKsk (ORCPT ); Thu, 9 Nov 2017 05:48:40 -0500 Subject: Re: [PATCH RESEND 2/3] KVM: Add paravirt remote TLB flush To: Wanpeng Li , linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: =?UTF-8?B?UmFkaW0gS3LEjW3DocWZ?= , Wanpeng Li , Eduardo Valentin References: <1510192934-5369-1-git-send-email-wanpeng.li@hotmail.com> <1510192934-5369-3-git-send-email-wanpeng.li@hotmail.com> From: Paolo Bonzini Message-ID: Date: Thu, 9 Nov 2017 11:48:35 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.4.0 MIME-Version: 1.0 In-Reply-To: <1510192934-5369-3-git-send-email-wanpeng.li@hotmail.com> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 8bit X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.32]); Thu, 09 Nov 2017 10:48:39 +0000 (UTC) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 09/11/2017 03:02, Wanpeng Li wrote: > From: Wanpeng Li > > Remote flushing api's does a busy wait which is fine in bare-metal > scenario. But with-in the guest, the vcpus might have been pre-empted > or blocked. In this scenario, the initator vcpu would end up > busy-waiting for a long amount of time. > > This patch set implements para-virt flush tlbs making sure that it > does not wait for vcpus that are sleeping. And all the sleeping vcpus > flush the tlb on guest enter. > > The best result is achieved when we're overcommiting the host by running > multiple vCPUs on each pCPU. In this case PV tlb flush avoids touching > vCPUs which are not scheduled and avoid the wait on the main CPU. > > Test on a Haswell i7 desktop 4 cores (2HT), so 8 pCPUs, running ebizzy in > one linux guest. > > ebizzy -M > vanilla optimized boost > 8 vCPUs 10152 10083 -0.68% > 16 vCPUs 1224 4866 297.5% > 24 vCPUs 1109 3871 249% > 32 vCPUs 1025 3375 229.3% > > Cc: Paolo Bonzini > Cc: Radim Krčmář > Signed-off-by: Wanpeng Li > --- > arch/x86/include/uapi/asm/kvm_para.h | 1 + > arch/x86/kernel/kvm.c | 29 +++++++++++++++++++++++++++++ > 2 files changed, 30 insertions(+) > > diff --git a/arch/x86/include/uapi/asm/kvm_para.h b/arch/x86/include/uapi/asm/kvm_para.h > index ff23ce9..189e354 100644 > --- a/arch/x86/include/uapi/asm/kvm_para.h > +++ b/arch/x86/include/uapi/asm/kvm_para.h > @@ -52,6 +52,7 @@ struct kvm_steal_time { > > #define KVM_VCPU_NOT_PREEMPTED (0 << 0) > #define KVM_VCPU_PREEMPTED (1 << 0) > +#define KVM_VCPU_SHOULD_FLUSH (1 << 1) > > #define KVM_CLOCK_PAIRING_WALLCLOCK 0 > struct kvm_clock_pairing { > diff --git a/arch/x86/kernel/kvm.c b/arch/x86/kernel/kvm.c > index 1b1b641..2e2f3ae 100644 > --- a/arch/x86/kernel/kvm.c > +++ b/arch/x86/kernel/kvm.c > @@ -465,6 +465,33 @@ static void __init kvm_apf_trap_init(void) > update_intr_gate(X86_TRAP_PF, async_page_fault); > } > > +static void kvm_flush_tlb_others(const struct cpumask *cpumask, > + const struct flush_tlb_info *info) > +{ > + u8 state; > + int cpu; > + struct kvm_steal_time *src; > + cpumask_t flushmask; > + > + > + cpumask_copy(&flushmask, cpumask); > + /* > + * We have to call flush only on online vCPUs. And > + * queue flush_on_enter for pre-empted vCPUs > + */ > + for_each_cpu(cpu, cpumask) { > + src = &per_cpu(steal_time, cpu); > + state = src->preempted; > + if ((state & KVM_VCPU_PREEMPTED)) { > + if (cmpxchg(&src->preempted, state, state | 1 << > + KVM_VCPU_SHOULD_FLUSH)) > + cpumask_clear_cpu(cpu, &flushmask); > + } > + } > + > + native_flush_tlb_others(&flushmask, info); > +} > + > void __init kvm_guest_init(void) > { > int i; > @@ -484,6 +511,8 @@ void __init kvm_guest_init(void) > pv_time_ops.steal_clock = kvm_steal_clock; > } > > + pv_mmu_ops.flush_tlb_others = kvm_flush_tlb_others; This needs to be keyed on a new CPUID feature bit. Eduardo is also adding a new "PV_DEDICATED" hint and you might disable PV TLB flush when PV_DEDICATED is set. Paolo > if (kvm_para_has_feature(KVM_FEATURE_PV_EOI)) > apic_set_eoi_write(kvm_guest_apic_eoi_write); > >