From: Vitaly Kuznetsov <vkuznets@redhat.com> To: kvm@vger.kernel.org, x86@kernel.org Cc: "Paolo Bonzini" <pbonzini@redhat.com>, "Radim Krčmář" <rkrcmar@redhat.com>, "Thomas Gleixner" <tglx@linutronix.de>, "Ingo Molnar" <mingo@redhat.com>, "H. Peter Anvin" <hpa@zytor.com>, "K. Y. Srinivasan" <kys@microsoft.com>, "Haiyang Zhang" <haiyangz@microsoft.com>, "Stephen Hemminger" <sthemmin@microsoft.com>, "Michael Kelley (EOSG)" <Michael.H.Kelley@microsoft.com>, "Andy Lutomirski" <luto@kernel.org>, "Mohammed Gamal" <mmorsy@redhat.com>, "Cathy Avery" <cavery@redhat.com>, "Roman Kagan" <rkagan@virtuozzo.com>, linux-kernel@vger.kernel.org, devel@linuxdriverproject.org Subject: [PATCH v2 7/7] x86/kvm: support Hyper-V reenlightenment Date: Wed, 13 Dec 2017 16:09:45 +0100 [thread overview] Message-ID: <20171213150945.24054-8-vkuznets@redhat.com> (raw) In-Reply-To: <20171213150945.24054-1-vkuznets@redhat.com> When we run nested KVM on Hyper-V guests we need to update masterclocks for all guests when L1 migrates to a host with different TSC frequency. Implement the procedure in the following way: - Pause all guests. - Tell our host (Hyper-V) to stop emulating TSC accesses. - Update our gtod copy, recompute clocks. - Unpause all guests. This is somewhat similar to cpufreq but we have two important differences: we can only disable TSC emulation globally (on all CPUs) and we don't know the new TSC frequency until we turn the emulation off so we can't 'prepare' ourselves to the event. Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com> --- arch/x86/kvm/x86.c | 45 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 45 insertions(+) diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index 843a8596f57e..d98e5165c1ad 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -68,6 +68,7 @@ #include <asm/div64.h> #include <asm/irq_remapping.h> #include <asm/mshyperv.h> +#include <asm/hypervisor.h> #define CREATE_TRACE_POINTS #include "trace.h" @@ -5932,6 +5933,43 @@ static void tsc_khz_changed(void *data) __this_cpu_write(cpu_tsc_khz, khz); } +void kvm_hyperv_tsc_notifier(void) +{ +#ifdef CONFIG_X86_64 + struct kvm *kvm; + struct kvm_vcpu *vcpu; + int cpu; + + spin_lock(&kvm_lock); + list_for_each_entry(kvm, &vm_list, vm_list) + kvm_make_mclock_inprogress_request(kvm); + + hyperv_stop_tsc_emulation(); + + /* TSC frequency always matches when on Hyper-V */ + for_each_present_cpu(cpu) + per_cpu(cpu_tsc_khz, cpu) = tsc_khz; + kvm_max_guest_tsc_khz = tsc_khz; + + list_for_each_entry(kvm, &vm_list, vm_list) { + struct kvm_arch *ka = &kvm->arch; + + spin_lock(&ka->pvclock_gtod_sync_lock); + + pvclock_update_vm_gtod_copy(kvm); + + kvm_for_each_vcpu(cpu, vcpu, kvm) + kvm_make_request(KVM_REQ_CLOCK_UPDATE, vcpu); + + kvm_for_each_vcpu(cpu, vcpu, kvm) + kvm_clear_request(KVM_REQ_MCLOCK_INPROGRESS, vcpu); + + spin_unlock(&ka->pvclock_gtod_sync_lock); + } + spin_unlock(&kvm_lock); +#endif +} + static int kvmclock_cpufreq_notifier(struct notifier_block *nb, unsigned long val, void *data) { @@ -6217,6 +6255,9 @@ int kvm_arch_init(void *opaque) kvm_lapic_init(); #ifdef CONFIG_X86_64 pvclock_gtod_register_notifier(&pvclock_gtod_notifier); + + if (x86_hyper_type == X86_HYPER_MS_HYPERV) + set_hv_tscchange_cb(kvm_hyperv_tsc_notifier); #endif return 0; @@ -6229,6 +6270,10 @@ int kvm_arch_init(void *opaque) void kvm_arch_exit(void) { +#ifdef CONFIG_X86_64 + if (x86_hyper_type == X86_HYPER_MS_HYPERV) + clear_hv_tscchange_cb(); +#endif kvm_lapic_exit(); perf_unregister_guest_info_callbacks(&kvm_guest_cbs); -- 2.14.3
WARNING: multiple messages have this Message-ID (diff)
From: Vitaly Kuznetsov <vkuznets@redhat.com> To: kvm@vger.kernel.org, x86@kernel.org Cc: "Stephen Hemminger" <sthemmin@microsoft.com>, "Radim Krčmář" <rkrcmar@redhat.com>, "Haiyang Zhang" <haiyangz@microsoft.com>, linux-kernel@vger.kernel.org, devel@linuxdriverproject.org, "Michael Kelley (EOSG)" <Michael.H.Kelley@microsoft.com>, "Ingo Molnar" <mingo@redhat.com>, "Roman Kagan" <rkagan@virtuozzo.com>, "Andy Lutomirski" <luto@kernel.org>, "H. Peter Anvin" <hpa@zytor.com>, "Paolo Bonzini" <pbonzini@redhat.com>, "Thomas Gleixner" <tglx@linutronix.de>, "Mohammed Gamal" <mmorsy@redhat.com> Subject: [PATCH v2 7/7] x86/kvm: support Hyper-V reenlightenment Date: Wed, 13 Dec 2017 16:09:45 +0100 [thread overview] Message-ID: <20171213150945.24054-8-vkuznets@redhat.com> (raw) In-Reply-To: <20171213150945.24054-1-vkuznets@redhat.com> When we run nested KVM on Hyper-V guests we need to update masterclocks for all guests when L1 migrates to a host with different TSC frequency. Implement the procedure in the following way: - Pause all guests. - Tell our host (Hyper-V) to stop emulating TSC accesses. - Update our gtod copy, recompute clocks. - Unpause all guests. This is somewhat similar to cpufreq but we have two important differences: we can only disable TSC emulation globally (on all CPUs) and we don't know the new TSC frequency until we turn the emulation off so we can't 'prepare' ourselves to the event. Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com> --- arch/x86/kvm/x86.c | 45 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 45 insertions(+) diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index 843a8596f57e..d98e5165c1ad 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -68,6 +68,7 @@ #include <asm/div64.h> #include <asm/irq_remapping.h> #include <asm/mshyperv.h> +#include <asm/hypervisor.h> #define CREATE_TRACE_POINTS #include "trace.h" @@ -5932,6 +5933,43 @@ static void tsc_khz_changed(void *data) __this_cpu_write(cpu_tsc_khz, khz); } +void kvm_hyperv_tsc_notifier(void) +{ +#ifdef CONFIG_X86_64 + struct kvm *kvm; + struct kvm_vcpu *vcpu; + int cpu; + + spin_lock(&kvm_lock); + list_for_each_entry(kvm, &vm_list, vm_list) + kvm_make_mclock_inprogress_request(kvm); + + hyperv_stop_tsc_emulation(); + + /* TSC frequency always matches when on Hyper-V */ + for_each_present_cpu(cpu) + per_cpu(cpu_tsc_khz, cpu) = tsc_khz; + kvm_max_guest_tsc_khz = tsc_khz; + + list_for_each_entry(kvm, &vm_list, vm_list) { + struct kvm_arch *ka = &kvm->arch; + + spin_lock(&ka->pvclock_gtod_sync_lock); + + pvclock_update_vm_gtod_copy(kvm); + + kvm_for_each_vcpu(cpu, vcpu, kvm) + kvm_make_request(KVM_REQ_CLOCK_UPDATE, vcpu); + + kvm_for_each_vcpu(cpu, vcpu, kvm) + kvm_clear_request(KVM_REQ_MCLOCK_INPROGRESS, vcpu); + + spin_unlock(&ka->pvclock_gtod_sync_lock); + } + spin_unlock(&kvm_lock); +#endif +} + static int kvmclock_cpufreq_notifier(struct notifier_block *nb, unsigned long val, void *data) { @@ -6217,6 +6255,9 @@ int kvm_arch_init(void *opaque) kvm_lapic_init(); #ifdef CONFIG_X86_64 pvclock_gtod_register_notifier(&pvclock_gtod_notifier); + + if (x86_hyper_type == X86_HYPER_MS_HYPERV) + set_hv_tscchange_cb(kvm_hyperv_tsc_notifier); #endif return 0; @@ -6229,6 +6270,10 @@ int kvm_arch_init(void *opaque) void kvm_arch_exit(void) { +#ifdef CONFIG_X86_64 + if (x86_hyper_type == X86_HYPER_MS_HYPERV) + clear_hv_tscchange_cb(); +#endif kvm_lapic_exit(); perf_unregister_guest_info_callbacks(&kvm_guest_cbs); -- 2.14.3
next prev parent reply other threads:[~2017-12-13 15:11 UTC|newest] Thread overview: 28+ messages / expand[flat|nested] mbox.gz Atom feed top 2017-12-13 15:09 [PATCH v2 0/7] x86/kvm/hyperv: stable clocksorce for L2 guests when running nested KVM on Hyper-V Vitaly Kuznetsov 2017-12-13 15:09 ` Vitaly Kuznetsov 2017-12-13 15:09 ` [PATCH v2 1/7] x86/hyper-v: check for required priviliges in hyperv_init() Vitaly Kuznetsov 2017-12-13 15:09 ` Vitaly Kuznetsov 2018-01-14 20:32 ` Thomas Gleixner 2018-01-14 20:32 ` Thomas Gleixner 2017-12-13 15:09 ` [PATCH v2 2/7] x86/hyper-v: add a function to read both TSC and TSC page value simulateneously Vitaly Kuznetsov 2018-01-14 20:35 ` Thomas Gleixner 2018-01-14 20:35 ` Thomas Gleixner 2018-01-15 10:45 ` Vitaly Kuznetsov 2018-01-15 10:45 ` Vitaly Kuznetsov 2017-12-13 15:09 ` [PATCH v2 3/7] x86/hyper-v: reenlightenment notifications support Vitaly Kuznetsov 2017-12-13 15:09 ` Vitaly Kuznetsov 2018-01-14 20:42 ` Thomas Gleixner 2017-12-13 15:09 ` [PATCH v2 4/7] x86/hyper-v: redirect reenlightment notifications on CPU offlining Vitaly Kuznetsov 2017-12-13 15:09 ` Vitaly Kuznetsov 2018-01-14 20:44 ` Thomas Gleixner 2018-01-15 10:44 ` Vitaly Kuznetsov 2018-01-15 10:44 ` Vitaly Kuznetsov 2017-12-13 15:09 ` [PATCH v2 5/7] x86/irq: Count Hyper-V reenlightenment interrupts Vitaly Kuznetsov 2017-12-13 15:09 ` Vitaly Kuznetsov 2017-12-13 15:09 ` [PATCH v2 6/7] x86/kvm: pass stable clocksource to guests when running nested on Hyper-V Vitaly Kuznetsov 2017-12-13 15:09 ` Vitaly Kuznetsov 2017-12-13 15:09 ` Vitaly Kuznetsov [this message] 2017-12-13 15:09 ` [PATCH v2 7/7] x86/kvm: support Hyper-V reenlightenment Vitaly Kuznetsov 2017-12-15 9:01 ` [PATCH v2 0/7] x86/kvm/hyperv: stable clocksorce for L2 guests when running nested KVM on Hyper-V Paolo Bonzini 2017-12-15 9:30 ` Vitaly Kuznetsov 2018-01-03 14:57 ` Vitaly Kuznetsov
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=20171213150945.24054-8-vkuznets@redhat.com \ --to=vkuznets@redhat.com \ --cc=Michael.H.Kelley@microsoft.com \ --cc=cavery@redhat.com \ --cc=devel@linuxdriverproject.org \ --cc=haiyangz@microsoft.com \ --cc=hpa@zytor.com \ --cc=kvm@vger.kernel.org \ --cc=kys@microsoft.com \ --cc=linux-kernel@vger.kernel.org \ --cc=luto@kernel.org \ --cc=mingo@redhat.com \ --cc=mmorsy@redhat.com \ --cc=pbonzini@redhat.com \ --cc=rkagan@virtuozzo.com \ --cc=rkrcmar@redhat.com \ --cc=sthemmin@microsoft.com \ --cc=tglx@linutronix.de \ --cc=x86@kernel.org \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.