* [PATCH RFC] Revert "KVM: VMX: Micro-optimize vmexit time when not exposing PMU"
@ 2020-06-19 9:40 Vitaly Kuznetsov
2020-06-19 10:03 ` Xu, Like
2020-06-19 12:13 ` Paolo Bonzini
0 siblings, 2 replies; 4+ messages in thread
From: Vitaly Kuznetsov @ 2020-06-19 9:40 UTC (permalink / raw)
To: kvm, Paolo Bonzini, Sean Christopherson, Wanpeng Li
Cc: Jim Mattson, Maxime Coquelin, linux-kernel
Guest crashes are observed on a Cascade Lake system when 'perf top' is
launched on the host, e.g.
BUG: unable to handle kernel paging request at fffffe0000073038
PGD 7ffa7067 P4D 7ffa7067 PUD 7ffa6067 PMD 7ffa5067 PTE ffffffffff120
Oops: 0000 [#1] SMP PTI
CPU: 1 PID: 1 Comm: systemd Not tainted 4.18.0+ #380
...
Call Trace:
serial8250_console_write+0xfe/0x1f0
call_console_drivers.constprop.0+0x9d/0x120
console_unlock+0x1ea/0x460
Call traces are different but the crash is imminent. The problem was
blindly bisected to the commit 041bc42ce2d0 ("KVM: VMX: Micro-optimize
vmexit time when not exposing PMU"). It was also confirmed that the
issue goes away if PMU is exposed to the guest.
With some instrumentation of the guest we can see what is being switched
(when we do atomic_switch_perf_msrs()):
vmx_vcpu_run: switching 2 msrs
vmx_vcpu_run: switching MSR38f guest: 70000000d host: 70000000f
vmx_vcpu_run: switching MSR3f1 guest: 0 host: 2
The current guess is that PEBS (MSR_IA32_PEBS_ENABLE, 0x3f1) is to blame.
Regardless of whether PMU is exposed to the guest or not, PEBS needs to
be disabled upon switch.
This reverts commit 041bc42ce2d0efac3b85bbb81dea8c74b81f4ef9.
Reported-by: Maxime Coquelin <maxime.coquelin@redhat.com>
Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
---
- Perf/KVM interractions are a mystery to me, thus RFC.
---
arch/x86/kvm/vmx/vmx.c | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)
diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c
index 36c771728c8c..b1a23ad986ff 100644
--- a/arch/x86/kvm/vmx/vmx.c
+++ b/arch/x86/kvm/vmx/vmx.c
@@ -6728,8 +6728,7 @@ static fastpath_t vmx_vcpu_run(struct kvm_vcpu *vcpu)
pt_guest_enter(vmx);
- if (vcpu_to_pmu(vcpu)->version)
- atomic_switch_perf_msrs(vmx);
+ atomic_switch_perf_msrs(vmx);
atomic_switch_umwait_control_msr(vmx);
if (enable_preemption_timer)
--
2.25.4
^ permalink raw reply related [flat|nested] 4+ messages in thread
* Re: [PATCH RFC] Revert "KVM: VMX: Micro-optimize vmexit time when not exposing PMU"
2020-06-19 9:40 [PATCH RFC] Revert "KVM: VMX: Micro-optimize vmexit time when not exposing PMU" Vitaly Kuznetsov
@ 2020-06-19 10:03 ` Xu, Like
2020-06-19 11:29 ` Vitaly Kuznetsov
2020-06-19 12:13 ` Paolo Bonzini
1 sibling, 1 reply; 4+ messages in thread
From: Xu, Like @ 2020-06-19 10:03 UTC (permalink / raw)
To: Vitaly Kuznetsov, kvm, Paolo Bonzini, Sean Christopherson, Wanpeng Li
Cc: Jim Mattson, Maxime Coquelin, linux-kernel
On 2020/6/19 17:40, Vitaly Kuznetsov wrote:
> Guest crashes are observed on a Cascade Lake system when 'perf top' is
> launched on the host, e.g.
Interesting, is it specific to Cascade Lake?
Would you mind sharing the output of
"cpuid -r -l 1 -1" and "cat /proc/cpuinfo| grep microcode | uniq" with us ?
Thanks,
Like Xu
>
> BUG: unable to handle kernel paging request at fffffe0000073038
> PGD 7ffa7067 P4D 7ffa7067 PUD 7ffa6067 PMD 7ffa5067 PTE ffffffffff120
> Oops: 0000 [#1] SMP PTI
> CPU: 1 PID: 1 Comm: systemd Not tainted 4.18.0+ #380
> ...
> Call Trace:
> serial8250_console_write+0xfe/0x1f0
> call_console_drivers.constprop.0+0x9d/0x120
> console_unlock+0x1ea/0x460
>
> Call traces are different but the crash is imminent. The problem was
> blindly bisected to the commit 041bc42ce2d0 ("KVM: VMX: Micro-optimize
> vmexit time when not exposing PMU"). It was also confirmed that the
> issue goes away if PMU is exposed to the guest.
>
> With some instrumentation of the guest we can see what is being switched
> (when we do atomic_switch_perf_msrs()):
>
> vmx_vcpu_run: switching 2 msrs
> vmx_vcpu_run: switching MSR38f guest: 70000000d host: 70000000f
> vmx_vcpu_run: switching MSR3f1 guest: 0 host: 2
>
> The current guess is that PEBS (MSR_IA32_PEBS_ENABLE, 0x3f1) is to blame.
> Regardless of whether PMU is exposed to the guest or not, PEBS needs to
> be disabled upon switch.
>
> This reverts commit 041bc42ce2d0efac3b85bbb81dea8c74b81f4ef9.
>
> Reported-by: Maxime Coquelin <maxime.coquelin@redhat.com>
> Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
> ---
> - Perf/KVM interractions are a mystery to me, thus RFC.
> ---
> arch/x86/kvm/vmx/vmx.c | 3 +--
> 1 file changed, 1 insertion(+), 2 deletions(-)
>
> diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c
> index 36c771728c8c..b1a23ad986ff 100644
> --- a/arch/x86/kvm/vmx/vmx.c
> +++ b/arch/x86/kvm/vmx/vmx.c
> @@ -6728,8 +6728,7 @@ static fastpath_t vmx_vcpu_run(struct kvm_vcpu *vcpu)
>
> pt_guest_enter(vmx);
>
> - if (vcpu_to_pmu(vcpu)->version)
> - atomic_switch_perf_msrs(vmx);
> + atomic_switch_perf_msrs(vmx);
> atomic_switch_umwait_control_msr(vmx);
>
> if (enable_preemption_timer)
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [PATCH RFC] Revert "KVM: VMX: Micro-optimize vmexit time when not exposing PMU"
2020-06-19 10:03 ` Xu, Like
@ 2020-06-19 11:29 ` Vitaly Kuznetsov
0 siblings, 0 replies; 4+ messages in thread
From: Vitaly Kuznetsov @ 2020-06-19 11:29 UTC (permalink / raw)
To: like.xu, kvm
Cc: Jim Mattson, Maxime Coquelin, linux-kernel, Paolo Bonzini,
Sean Christopherson, Wanpeng Li
"Xu, Like" <like.xu@intel.com> writes:
> On 2020/6/19 17:40, Vitaly Kuznetsov wrote:
>> Guest crashes are observed on a Cascade Lake system when 'perf top' is
>> launched on the host, e.g.
> Interesting, is it specific to Cascade Lake?
>
Actually no, just reproduced this on a Haswell system. If you run the
guest with "-cpu host,-pmu" and do 'perf top' (on the host) the guest
crashes immediately.
> Would you mind sharing the output of
> "cpuid -r -l 1 -1" and "cat /proc/cpuinfo| grep microcode | uniq" with us ?
Sure (this is probably unrelated because the issue also reproduces on
Haswell but still):
# cpuid -r -l 1 -1
CPU:
0x00000001 0x00: eax=0x00050657 ebx=0x03200800 ecx=0x7ffefbff edx=0xbfebfbff
# cat /proc/cpuinfo| grep microcode | uniq
microcode : 0x500002c
This is "Intel(R) Xeon(R) Gold 5218 CPU @ 2.30GHz"
--
Vitaly
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [PATCH RFC] Revert "KVM: VMX: Micro-optimize vmexit time when not exposing PMU"
2020-06-19 9:40 [PATCH RFC] Revert "KVM: VMX: Micro-optimize vmexit time when not exposing PMU" Vitaly Kuznetsov
2020-06-19 10:03 ` Xu, Like
@ 2020-06-19 12:13 ` Paolo Bonzini
1 sibling, 0 replies; 4+ messages in thread
From: Paolo Bonzini @ 2020-06-19 12:13 UTC (permalink / raw)
To: Vitaly Kuznetsov, kvm, Sean Christopherson, Wanpeng Li
Cc: Jim Mattson, Maxime Coquelin, linux-kernel
On 19/06/20 11:40, Vitaly Kuznetsov wrote:
> Guest crashes are observed on a Cascade Lake system when 'perf top' is
> launched on the host, e.g.
>
> BUG: unable to handle kernel paging request at fffffe0000073038
> PGD 7ffa7067 P4D 7ffa7067 PUD 7ffa6067 PMD 7ffa5067 PTE ffffffffff120
> Oops: 0000 [#1] SMP PTI
> CPU: 1 PID: 1 Comm: systemd Not tainted 4.18.0+ #380
> ...
> Call Trace:
> serial8250_console_write+0xfe/0x1f0
> call_console_drivers.constprop.0+0x9d/0x120
> console_unlock+0x1ea/0x460
>
> Call traces are different but the crash is imminent. The problem was
> blindly bisected to the commit 041bc42ce2d0 ("KVM: VMX: Micro-optimize
> vmexit time when not exposing PMU"). It was also confirmed that the
> issue goes away if PMU is exposed to the guest.
>
> With some instrumentation of the guest we can see what is being switched
> (when we do atomic_switch_perf_msrs()):
>
> vmx_vcpu_run: switching 2 msrs
> vmx_vcpu_run: switching MSR38f guest: 70000000d host: 70000000f
> vmx_vcpu_run: switching MSR3f1 guest: 0 host: 2
>
> The current guess is that PEBS (MSR_IA32_PEBS_ENABLE, 0x3f1) is to blame.
> Regardless of whether PMU is exposed to the guest or not, PEBS needs to
> be disabled upon switch.
>
> This reverts commit 041bc42ce2d0efac3b85bbb81dea8c74b81f4ef9.
>
> Reported-by: Maxime Coquelin <maxime.coquelin@redhat.com>
> Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
> ---
> - Perf/KVM interractions are a mystery to me, thus RFC.
> ---
> arch/x86/kvm/vmx/vmx.c | 3 +--
> 1 file changed, 1 insertion(+), 2 deletions(-)
>
> diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c
> index 36c771728c8c..b1a23ad986ff 100644
> --- a/arch/x86/kvm/vmx/vmx.c
> +++ b/arch/x86/kvm/vmx/vmx.c
> @@ -6728,8 +6728,7 @@ static fastpath_t vmx_vcpu_run(struct kvm_vcpu *vcpu)
>
> pt_guest_enter(vmx);
>
> - if (vcpu_to_pmu(vcpu)->version)
> - atomic_switch_perf_msrs(vmx);
> + atomic_switch_perf_msrs(vmx);
> atomic_switch_umwait_control_msr(vmx);
>
> if (enable_preemption_timer)
>
Queued, thanks.
Paolo
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2020-06-19 12:15 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-06-19 9:40 [PATCH RFC] Revert "KVM: VMX: Micro-optimize vmexit time when not exposing PMU" Vitaly Kuznetsov
2020-06-19 10:03 ` Xu, Like
2020-06-19 11:29 ` Vitaly Kuznetsov
2020-06-19 12:13 ` Paolo Bonzini
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).