From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.0 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI,SIGNED_OFF_BY, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A8729C5B57D for ; Wed, 3 Jul 2019 00:49:00 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 77F0C218EA for ; Wed, 3 Jul 2019 00:49:00 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="RlR5HeXA" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727349AbfGCAs4 (ORCPT ); Tue, 2 Jul 2019 20:48:56 -0400 Received: from mail-ot1-f65.google.com ([209.85.210.65]:32893 "EHLO mail-ot1-f65.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727127AbfGCAs4 (ORCPT ); Tue, 2 Jul 2019 20:48:56 -0400 Received: by mail-ot1-f65.google.com with SMTP id q20so536403otl.0; Tue, 02 Jul 2019 17:48:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=45BsG3LbCDzj8igWscjEhFYwH/bXI9tYws0EYmAe5o0=; b=RlR5HeXAIYRo74Y047tIsFyMhy8MF7pMRB+B3vI6PB3OjxaPM01VHlefj6Q18IxGGH tkHARN2W1bhlG++JE0W4+JmMuZw9SFQayKTjVeYkCxhsZ/FhIL6HLCxb4lmkHw6oj1eH VHQp4IQXTG63WzPE7mNhZyIoFbkWmjcHIT8V6YUVlZxa8I8Hs4nZBQmt9th4/YEszsEj qs5qLwZx6+/prjQKBbwrN4UKxuqE5v9F953CYVzpumZ4JoSpRgmVwGo9EcgQBSob2JBG 0ZdC5ml+/anJ6itiDTwlTJMFW/gRhwLebMsZmIrgLqVo71hR3NMG2i/kemeS/ER7Tpec FFJQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=45BsG3LbCDzj8igWscjEhFYwH/bXI9tYws0EYmAe5o0=; b=CYkrbXDuPrvMFYTr1S1FAq2eoBcemNBcrOP64LcsYiNK0BMTsXZV5pQAVb1ZFbaQqx H8ZnDfx78/jHe+FOhEGzcVGhFX9MMFVd7BQYYxl0A9MCU3gUylhJJI6mwoBNIefWPUQw O8aMoxoPNvw/WxUjJH9v0Hph+ngWrVLc6hcUVxcxqeZ6BruxUblgcAptHslu71iJ9ZD1 9D2+5OPzmvjRtlC/wPv2IhbFohGzncuVYHL50bLP4SmBvtLHrcaYta388nfZXhoKup34 pMY+rorFGjYncAqOzCX2XOe2MblNZY2YL31HMMajHDmmES0HXS+QWiGv9lMFVcA9ACk8 WvFQ== X-Gm-Message-State: APjAAAXBuTVhKPPsbHoSGVfB5j1dX6jrgEszRmk3L9C1kQXDXz5h4iy/ Do2CraTPvVwrJhQEXv0iD7szMc2y9pxzO6srzNKudA== X-Google-Smtp-Source: APXvYqyYH9ZkrslDEpoYp8hkvS7yhfva8jU9IqK0d8MGe6sZMgWnpUiyyuJo3saUDZI6ZXXv0stu7i7XIs+SoUse5FI= X-Received: by 2002:a9d:62c4:: with SMTP id z4mr25845573otk.56.1562114935125; Tue, 02 Jul 2019 17:48:55 -0700 (PDT) MIME-Version: 1.0 References: <1560474949-20497-1-git-send-email-wanpengli@tencent.com> <1560474949-20497-2-git-send-email-wanpengli@tencent.com> In-Reply-To: From: Wanpeng Li Date: Wed, 3 Jul 2019 08:48:47 +0800 Message-ID: Subject: Re: [PATCH v4 1/2] KVM: LAPIC: Optimize timer latency consider world switch time To: LKML , kvm Cc: Paolo Bonzini , =?UTF-8?B?UmFkaW0gS3LEjW3DocWZ?= , Sean Christopherson Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Hi Paolo, how about this patchset? Patch 2/2 is easy to take, do you have more concern about patch 1/2? On Fri, 28 Jun 2019 at 16:29, Wanpeng Li wrote: > > ping again, > On Fri, 21 Jun 2019 at 17:44, Wanpeng Li wrote: > > > > ping, > > On Fri, 14 Jun 2019 at 09:15, Wanpeng Li wrote: > > > > > > From: Wanpeng Li > > > > > > Advance lapic timer tries to hidden the hypervisor overhead between t= he > > > host emulated timer fires and the guest awares the timer is fired. Ho= wever, > > > even though after more sustaining optimizations, kvm-unit-tests/tscde= adline_latency > > > still awares ~1000 cycles latency since we lost the time between the = end of > > > wait_lapic_expire and the guest awares the timer is fired. There are > > > codes between the end of wait_lapic_expire and the world switch, furt= hermore, > > > the world switch itself also has overhead. Actually the guest_tsc is = equal > > > to the target deadline time in wait_lapic_expire is too late, guest w= ill > > > aware the latency between the end of wait_lapic_expire() and after vm= entry > > > to the guest. This patch takes this time into consideration. > > > > > > The vmentry_advance_ns module parameter is conservative 25ns by defau= lt(thanks > > > to Radim's kvm-unit-tests/vmentry_latency.flat), it can be tuned/rewo= rked in > > > the future. > > > > > > Cc: Paolo Bonzini > > > Cc: Radim Kr=C4=8Dm=C3=A1=C5=99 > > > Cc: Sean Christopherson > > > Signed-off-by: Wanpeng Li > > > --- > > > v3 -> v4: > > > * default value is 25ns > > > * compute vmentry_advance_cycles in kvm_set_tsc_khz() path > > > v2 -> v3: > > > * read-only module parameter > > > * get_vmentry_advance_cycles() not inline > > > v1 -> v2: > > > * rename get_vmentry_advance_delta to get_vmentry_advance_cycles > > > * cache vmentry_advance_cycles by setting param bit 0 > > > * add param max limit > > > > > > arch/x86/kvm/lapic.c | 21 ++++++++++++++++++--- > > > arch/x86/kvm/lapic.h | 2 ++ > > > arch/x86/kvm/vmx/vmx.c | 3 ++- > > > arch/x86/kvm/x86.c | 12 ++++++++++-- > > > arch/x86/kvm/x86.h | 2 ++ > > > 5 files changed, 34 insertions(+), 6 deletions(-) > > > > > > diff --git a/arch/x86/kvm/lapic.c b/arch/x86/kvm/lapic.c > > > index e82a18c..e92e4e5 100644 > > > --- a/arch/x86/kvm/lapic.c > > > +++ b/arch/x86/kvm/lapic.c > > > @@ -1528,6 +1528,19 @@ static inline void adjust_lapic_timer_advance(= struct kvm_vcpu *vcpu, > > > apic->lapic_timer.timer_advance_ns =3D timer_advance_ns; > > > } > > > > > > +u64 compute_vmentry_advance_cycles(struct kvm_vcpu *vcpu) > > > +{ > > > + u64 cycles; > > > + struct kvm_lapic *apic =3D vcpu->arch.apic; > > > + > > > + cycles =3D vmentry_advance_ns * vcpu->arch.virtual_tsc_khz; > > > + do_div(cycles, 1000000); > > > + > > > + apic->lapic_timer.vmentry_advance_cycles =3D cycles; > > > + > > > + return cycles; > > > +} > > > + > > > void kvm_wait_lapic_expire(struct kvm_vcpu *vcpu) > > > { > > > struct kvm_lapic *apic =3D vcpu->arch.apic; > > > @@ -1541,7 +1554,8 @@ void kvm_wait_lapic_expire(struct kvm_vcpu *vcp= u) > > > > > > tsc_deadline =3D apic->lapic_timer.expired_tscdeadline; > > > apic->lapic_timer.expired_tscdeadline =3D 0; > > > - guest_tsc =3D kvm_read_l1_tsc(vcpu, rdtsc()); > > > + guest_tsc =3D kvm_read_l1_tsc(vcpu, rdtsc()) + > > > + apic->lapic_timer.vmentry_advance_cycles; > > > apic->lapic_timer.advance_expire_delta =3D guest_tsc - tsc_de= adline; > > > > > > if (guest_tsc < tsc_deadline) > > > @@ -1569,7 +1583,8 @@ static void start_sw_tscdeadline(struct kvm_lap= ic *apic) > > > local_irq_save(flags); > > > > > > now =3D ktime_get(); > > > - guest_tsc =3D kvm_read_l1_tsc(vcpu, rdtsc()); > > > + guest_tsc =3D kvm_read_l1_tsc(vcpu, rdtsc()) + > > > + apic->lapic_timer.vmentry_advance_cycles; > > > > > > ns =3D (tscdeadline - guest_tsc) * 1000000ULL; > > > do_div(ns, this_tsc_khz); > > > @@ -2326,7 +2341,7 @@ int kvm_create_lapic(struct kvm_vcpu *vcpu, int= timer_advance_ns) > > > apic->lapic_timer.timer_advance_ns =3D timer_advance_= ns; > > > apic->lapic_timer.timer_advance_adjust_done =3D true; > > > } > > > - > > > + apic->lapic_timer.vmentry_advance_cycles =3D 0; > > > > > > /* > > > * APIC is created enabled. This will prevent kvm_lapic_set_b= ase from > > > diff --git a/arch/x86/kvm/lapic.h b/arch/x86/kvm/lapic.h > > > index 3674717..7c38950 100644 > > > --- a/arch/x86/kvm/lapic.h > > > +++ b/arch/x86/kvm/lapic.h > > > @@ -33,6 +33,7 @@ struct kvm_timer { > > > u64 expired_tscdeadline; > > > u32 timer_advance_ns; > > > s64 advance_expire_delta; > > > + u64 vmentry_advance_cycles; > > > atomic_t pending; /* accumulated trigge= red timers */ > > > bool hv_timer_in_use; > > > bool timer_advance_adjust_done; > > > @@ -226,6 +227,7 @@ static inline int kvm_lapic_latched_init(struct k= vm_vcpu *vcpu) > > > bool kvm_apic_pending_eoi(struct kvm_vcpu *vcpu, int vector); > > > > > > void kvm_wait_lapic_expire(struct kvm_vcpu *vcpu); > > > +u64 compute_vmentry_advance_cycles(struct kvm_vcpu *vcpu); > > > > > > bool kvm_intr_is_single_vcpu_fast(struct kvm *kvm, struct kvm_lapic_= irq *irq, > > > struct kvm_vcpu **dest_vcpu); > > > diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c > > > index 8fbea03..dc81c78 100644 > > > --- a/arch/x86/kvm/vmx/vmx.c > > > +++ b/arch/x86/kvm/vmx/vmx.c > > > @@ -7064,7 +7064,8 @@ static int vmx_set_hv_timer(struct kvm_vcpu *vc= pu, u64 guest_deadline_tsc, > > > > > > vmx =3D to_vmx(vcpu); > > > tscl =3D rdtsc(); > > > - guest_tscl =3D kvm_read_l1_tsc(vcpu, tscl); > > > + guest_tscl =3D kvm_read_l1_tsc(vcpu, tscl) + > > > + vcpu->arch.apic->lapic_timer.vmentry_advance_cycles; > > > delta_tsc =3D max(guest_deadline_tsc, guest_tscl) - guest_tsc= l; > > > lapic_timer_advance_cycles =3D nsec_to_cycles(vcpu, > > > ktimer->timer_adv= ance_ns); > > > diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c > > > index 0a05a4e..5e79b6c 100644 > > > --- a/arch/x86/kvm/x86.c > > > +++ b/arch/x86/kvm/x86.c > > > @@ -145,6 +145,12 @@ module_param(tsc_tolerance_ppm, uint, S_IRUGO | = S_IWUSR); > > > static int __read_mostly lapic_timer_advance_ns =3D -1; > > > module_param(lapic_timer_advance_ns, int, S_IRUGO | S_IWUSR); > > > > > > +/* > > > + * lapic timer vmentry advance (tscdeadline mode only) in nanosecond= s. > > > + */ > > > +u32 __read_mostly vmentry_advance_ns =3D 25; > > > +module_param(vmentry_advance_ns, uint, S_IRUGO); > > > + > > > static bool __read_mostly vector_hashing =3D true; > > > module_param(vector_hashing, bool, S_IRUGO); > > > > > > @@ -1592,6 +1598,8 @@ static int kvm_set_tsc_khz(struct kvm_vcpu *vcp= u, u32 user_tsc_khz) > > > kvm_get_time_scale(user_tsc_khz * 1000LL, NSEC_PER_SEC, > > > &vcpu->arch.virtual_tsc_shift, > > > &vcpu->arch.virtual_tsc_mult); > > > + if (user_tsc_khz !=3D vcpu->arch.virtual_tsc_khz) > > > + compute_vmentry_advance_cycles(vcpu); > > > vcpu->arch.virtual_tsc_khz =3D user_tsc_khz; > > > > > > /* > > > @@ -9134,8 +9142,6 @@ int kvm_arch_vcpu_init(struct kvm_vcpu *vcpu) > > > } > > > vcpu->arch.pio_data =3D page_address(page); > > > > > > - kvm_set_tsc_khz(vcpu, max_tsc_khz); > > > - > > > r =3D kvm_mmu_create(vcpu); > > > if (r < 0) > > > goto fail_free_pio_data; > > > @@ -9148,6 +9154,8 @@ int kvm_arch_vcpu_init(struct kvm_vcpu *vcpu) > > > } else > > > static_key_slow_inc(&kvm_no_apic_vcpu); > > > > > > + kvm_set_tsc_khz(vcpu, max_tsc_khz); > > > + > > > vcpu->arch.mce_banks =3D kzalloc(KVM_MAX_MCE_BANKS * sizeof(u= 64) * 4, > > > GFP_KERNEL_ACCOUNT); > > > if (!vcpu->arch.mce_banks) { > > > diff --git a/arch/x86/kvm/x86.h b/arch/x86/kvm/x86.h > > > index e08a128..9998989 100644 > > > --- a/arch/x86/kvm/x86.h > > > +++ b/arch/x86/kvm/x86.h > > > @@ -299,6 +299,8 @@ extern u64 kvm_supported_xcr0(void); > > > > > > extern unsigned int min_timer_period_us; > > > > > > +extern unsigned int vmentry_advance_ns; > > > + > > > extern bool enable_vmware_backdoor; > > > > > > extern struct static_key kvm_no_apic_vcpu; > > > -- > > > 2.7.4 > > >