[PATCH] smp/hotplug, x86/vmware: Put offline vCPUs in halt instead of mwait

[PATCH] smp/hotplug, x86/vmware: Put offline vCPUs in halt instead of mwait

[Srivatsa S. Bhat]

From: Srivatsa S. Bhat (VMware) <srivatsa@csail.mit.edu>

VMware ESXi allows enabling a passthru mwait CPU-idle state in the
guest using the following VMX option:

monitor_control.mwait_in_guest = "TRUE"

This lets a vCPU in mwait to remain in guest context (instead of
yielding to the hypervisor via a VMEXIT), which helps speed up
wakeups from idle.

However, this runs into problems with CPU hotplug, because the Linux
CPU offline path prefers to put the vCPU-to-be-offlined in mwait
state, whenever mwait is available. As a result, since a vCPU in mwait
remains in guest context and does not yield to the hypervisor, an
offline vCPU *appears* to be 100% busy as viewed from ESXi, which
prevents the hypervisor from running other vCPUs or workloads on the
corresponding pCPU (particularly when vCPU - pCPU mappings are
statically defined by the user). [ Note that such a vCPU is not
actually busy spinning though; it remains in mwait idle state in the
guest ].

Fix this by overriding the CPU offline play_dead() callback for VMware
hypervisor, by putting the CPU in halt state (which actually yields to
the hypervisor), even if mwait support is available.

Signed-off-by: Srivatsa S. Bhat (VMware) <srivatsa@csail.mit.edu>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Borislav Petkov <bp@alien8.de>
Cc: Dave Hansen <dave.hansen@linux.intel.com>
Cc: "H. Peter Anvin" <hpa@zytor.com>
Cc: Alexey Makhalov <amakhalov@vmware.com>
Cc: Juergen Gross <jgross@suse.com>
Cc: x86@kernel.org
Cc: VMware PV-Drivers Reviewers <pv-drivers@vmware.com>
---

 arch/x86/kernel/cpu/vmware.c |   17 +++++++++++++++++
 1 file changed, 17 insertions(+)

diff --git a/arch/x86/kernel/cpu/vmware.c b/arch/x86/kernel/cpu/vmware.c
index c04b933f48d3..420e359ed9bb 100644
--- a/arch/x86/kernel/cpu/vmware.c
+++ b/arch/x86/kernel/cpu/vmware.c
@@ -27,6 +27,7 @@
 #include <linux/clocksource.h>
 #include <linux/cpu.h>
 #include <linux/reboot.h>
+#include <linux/tboot.h>
 #include <linux/static_call.h>
 #include <asm/div64.h>
 #include <asm/x86_init.h>
@@ -312,6 +313,21 @@ static int vmware_cpu_down_prepare(unsigned int cpu)
 	local_irq_enable();
 	return 0;
 }
+
+static void vmware_play_dead(void)
+{
+	play_dead_common();
+	tboot_shutdown(TB_SHUTDOWN_WFS);
+
+	/*
+	 * Put the vCPU going offline in halt instead of mwait (even
+	 * if mwait support is available), to make sure that the
+	 * offline vCPU yields to the hypervisor (which may not happen
+	 * with mwait, for example, if the guest's VMX is configured
+	 * to retain the vCPU in guest context upon mwait).
+	 */
+	hlt_play_dead();
+}
 #endif
 
 static __init int activate_jump_labels(void)
@@ -349,6 +365,7 @@ static void __init vmware_paravirt_ops_setup(void)
 #ifdef CONFIG_SMP
 		smp_ops.smp_prepare_boot_cpu =
 			vmware_smp_prepare_boot_cpu;
+		smp_ops.play_dead = vmware_play_dead;
 		if (cpuhp_setup_state_nocalls(CPUHP_AP_ONLINE_DYN,
 					      "x86/vmware:online",
 					      vmware_cpu_online,

[PATCH] smp/hotplug, x86/vmware: Put offline vCPUs in halt instead of mwait

[Srivatsa S. Bhat]

From: Srivatsa S. Bhat (VMware) <srivatsa@csail.mit.edu>

VMware ESXi allows enabling a passthru mwait CPU-idle state in the
guest using the following VMX option:

monitor_control.mwait_in_guest = "TRUE"

This lets a vCPU in mwait to remain in guest context (instead of
yielding to the hypervisor via a VMEXIT), which helps speed up
wakeups from idle.

However, this runs into problems with CPU hotplug, because the Linux
CPU offline path prefers to put the vCPU-to-be-offlined in mwait
state, whenever mwait is available. As a result, since a vCPU in mwait
remains in guest context and does not yield to the hypervisor, an
offline vCPU *appears* to be 100% busy as viewed from ESXi, which
prevents the hypervisor from running other vCPUs or workloads on the
corresponding pCPU (particularly when vCPU - pCPU mappings are
statically defined by the user). [ Note that such a vCPU is not
actually busy spinning though; it remains in mwait idle state in the
guest ].

Fix this by overriding the CPU offline play_dead() callback for VMware
hypervisor, by putting the CPU in halt state (which actually yields to
the hypervisor), even if mwait support is available.

Signed-off-by: Srivatsa S. Bhat (VMware) <srivatsa@csail.mit.edu>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Borislav Petkov <bp@alien8.de>
Cc: Dave Hansen <dave.hansen@linux.intel.com>
Cc: "H. Peter Anvin" <hpa@zytor.com>
Cc: Alexey Makhalov <amakhalov@vmware.com>
Cc: Juergen Gross <jgross@suse.com>
Cc: x86@kernel.org
Cc: VMware PV-Drivers Reviewers <pv-drivers@vmware.com>
---

 arch/x86/kernel/cpu/vmware.c |   17 +++++++++++++++++
 1 file changed, 17 insertions(+)

diff --git a/arch/x86/kernel/cpu/vmware.c b/arch/x86/kernel/cpu/vmware.c
index c04b933f48d3..420e359ed9bb 100644
--- a/arch/x86/kernel/cpu/vmware.c
+++ b/arch/x86/kernel/cpu/vmware.c
@@ -27,6 +27,7 @@
 #include <linux/clocksource.h>
 #include <linux/cpu.h>
 #include <linux/reboot.h>
+#include <linux/tboot.h>
 #include <linux/static_call.h>
 #include <asm/div64.h>
 #include <asm/x86_init.h>
@@ -312,6 +313,21 @@ static int vmware_cpu_down_prepare(unsigned int cpu)
 	local_irq_enable();
 	return 0;
 }
+
+static void vmware_play_dead(void)
+{
+	play_dead_common();
+	tboot_shutdown(TB_SHUTDOWN_WFS);
+
+	/*
+	 * Put the vCPU going offline in halt instead of mwait (even
+	 * if mwait support is available), to make sure that the
+	 * offline vCPU yields to the hypervisor (which may not happen
+	 * with mwait, for example, if the guest's VMX is configured
+	 * to retain the vCPU in guest context upon mwait).
+	 */
+	hlt_play_dead();
+}
 #endif
 
 static __init int activate_jump_labels(void)
@@ -349,6 +365,7 @@ static void __init vmware_paravirt_ops_setup(void)
 #ifdef CONFIG_SMP
 		smp_ops.smp_prepare_boot_cpu =
 			vmware_smp_prepare_boot_cpu;
+		smp_ops.play_dead = vmware_play_dead;
 		if (cpuhp_setup_state_nocalls(CPUHP_AP_ONLINE_DYN,
 					      "x86/vmware:online",
 					      vmware_cpu_online,


_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [PATCH] smp/hotplug, x86/vmware: Put offline vCPUs in halt instead of mwait

[Srivatsa S. Bhat]

Hi Boris, Thomas, Peter,

On 7/21/22 1:44 PM, Srivatsa S. Bhat wrote:
> From: Srivatsa S. Bhat (VMware) <srivatsa@csail.mit.edu>
> 
> VMware ESXi allows enabling a passthru mwait CPU-idle state in the
> guest using the following VMX option:
> 
> monitor_control.mwait_in_guest = "TRUE"
> 
> This lets a vCPU in mwait to remain in guest context (instead of
> yielding to the hypervisor via a VMEXIT), which helps speed up
> wakeups from idle.
> 
> However, this runs into problems with CPU hotplug, because the Linux
> CPU offline path prefers to put the vCPU-to-be-offlined in mwait
> state, whenever mwait is available. As a result, since a vCPU in mwait
> remains in guest context and does not yield to the hypervisor, an
> offline vCPU *appears* to be 100% busy as viewed from ESXi, which
> prevents the hypervisor from running other vCPUs or workloads on the
> corresponding pCPU (particularly when vCPU - pCPU mappings are
> statically defined by the user). [ Note that such a vCPU is not
> actually busy spinning though; it remains in mwait idle state in the
> guest ].
> 
> Fix this by overriding the CPU offline play_dead() callback for VMware
> hypervisor, by putting the CPU in halt state (which actually yields to
> the hypervisor), even if mwait support is available.
> 
> Signed-off-by: Srivatsa S. Bhat (VMware) <srivatsa@csail.mit.edu>
> Cc: Thomas Gleixner <tglx@linutronix.de>
> Cc: Peter Zijlstra <peterz@infradead.org>
> Cc: Ingo Molnar <mingo@redhat.com>
> Cc: Borislav Petkov <bp@alien8.de>
> Cc: Dave Hansen <dave.hansen@linux.intel.com>
> Cc: "H. Peter Anvin" <hpa@zytor.com>
> Cc: Alexey Makhalov <amakhalov@vmware.com>
> Cc: Juergen Gross <jgross@suse.com>
> Cc: x86@kernel.org
> Cc: VMware PV-Drivers Reviewers <pv-drivers@vmware.com>
> ---

Could you share your thoughts on this patch when you get a chance,
please? I verified that this patch still applies cleanly on current
mainline (6.0-rc6). I'm happy to resend the patch though, if it helps.

Thank you!

Regards,
Srivatsa

> 
>  arch/x86/kernel/cpu/vmware.c |   17 +++++++++++++++++
>  1 file changed, 17 insertions(+)
> 
> diff --git a/arch/x86/kernel/cpu/vmware.c b/arch/x86/kernel/cpu/vmware.c
> index c04b933f48d3..420e359ed9bb 100644
> --- a/arch/x86/kernel/cpu/vmware.c
> +++ b/arch/x86/kernel/cpu/vmware.c
> @@ -27,6 +27,7 @@
>  #include <linux/clocksource.h>
>  #include <linux/cpu.h>
>  #include <linux/reboot.h>
> +#include <linux/tboot.h>
>  #include <linux/static_call.h>
>  #include <asm/div64.h>
>  #include <asm/x86_init.h>
> @@ -312,6 +313,21 @@ static int vmware_cpu_down_prepare(unsigned int cpu)
>  	local_irq_enable();
>  	return 0;
>  }
> +
> +static void vmware_play_dead(void)
> +{
> +	play_dead_common();
> +	tboot_shutdown(TB_SHUTDOWN_WFS);
> +
> +	/*
> +	 * Put the vCPU going offline in halt instead of mwait (even
> +	 * if mwait support is available), to make sure that the
> +	 * offline vCPU yields to the hypervisor (which may not happen
> +	 * with mwait, for example, if the guest's VMX is configured
> +	 * to retain the vCPU in guest context upon mwait).
> +	 */
> +	hlt_play_dead();
> +}
>  #endif
>  
>  static __init int activate_jump_labels(void)
> @@ -349,6 +365,7 @@ static void __init vmware_paravirt_ops_setup(void)
>  #ifdef CONFIG_SMP
>  		smp_ops.smp_prepare_boot_cpu =
>  			vmware_smp_prepare_boot_cpu;
> +		smp_ops.play_dead = vmware_play_dead;
>  		if (cpuhp_setup_state_nocalls(CPUHP_AP_ONLINE_DYN,
>  					      "x86/vmware:online",
>  					      vmware_cpu_online,
> 
> 
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [PATCH] smp/hotplug, x86/vmware: Put offline vCPUs in halt instead of mwait

[Srivatsa S. Bhat]

Hi Boris, Thomas, Peter,

On 7/21/22 1:44 PM, Srivatsa S. Bhat wrote:
> From: Srivatsa S. Bhat (VMware) <srivatsa@csail.mit.edu>
> 
> VMware ESXi allows enabling a passthru mwait CPU-idle state in the
> guest using the following VMX option:
> 
> monitor_control.mwait_in_guest = "TRUE"
> 
> This lets a vCPU in mwait to remain in guest context (instead of
> yielding to the hypervisor via a VMEXIT), which helps speed up
> wakeups from idle.
> 
> However, this runs into problems with CPU hotplug, because the Linux
> CPU offline path prefers to put the vCPU-to-be-offlined in mwait
> state, whenever mwait is available. As a result, since a vCPU in mwait
> remains in guest context and does not yield to the hypervisor, an
> offline vCPU *appears* to be 100% busy as viewed from ESXi, which
> prevents the hypervisor from running other vCPUs or workloads on the
> corresponding pCPU (particularly when vCPU - pCPU mappings are
> statically defined by the user). [ Note that such a vCPU is not
> actually busy spinning though; it remains in mwait idle state in the
> guest ].
> 
> Fix this by overriding the CPU offline play_dead() callback for VMware
> hypervisor, by putting the CPU in halt state (which actually yields to
> the hypervisor), even if mwait support is available.
> 
> Signed-off-by: Srivatsa S. Bhat (VMware) <srivatsa@csail.mit.edu>
> Cc: Thomas Gleixner <tglx@linutronix.de>
> Cc: Peter Zijlstra <peterz@infradead.org>
> Cc: Ingo Molnar <mingo@redhat.com>
> Cc: Borislav Petkov <bp@alien8.de>
> Cc: Dave Hansen <dave.hansen@linux.intel.com>
> Cc: "H. Peter Anvin" <hpa@zytor.com>
> Cc: Alexey Makhalov <amakhalov@vmware.com>
> Cc: Juergen Gross <jgross@suse.com>
> Cc: x86@kernel.org
> Cc: VMware PV-Drivers Reviewers <pv-drivers@vmware.com>
> ---

Could you share your thoughts on this patch when you get a chance,
please? I verified that this patch still applies cleanly on current
mainline (6.0-rc6). I'm happy to resend the patch though, if it helps.

Thank you!

Regards,
Srivatsa

> 
>  arch/x86/kernel/cpu/vmware.c |   17 +++++++++++++++++
>  1 file changed, 17 insertions(+)
> 
> diff --git a/arch/x86/kernel/cpu/vmware.c b/arch/x86/kernel/cpu/vmware.c
> index c04b933f48d3..420e359ed9bb 100644
> --- a/arch/x86/kernel/cpu/vmware.c
> +++ b/arch/x86/kernel/cpu/vmware.c
> @@ -27,6 +27,7 @@
>  #include <linux/clocksource.h>
>  #include <linux/cpu.h>
>  #include <linux/reboot.h>
> +#include <linux/tboot.h>
>  #include <linux/static_call.h>
>  #include <asm/div64.h>
>  #include <asm/x86_init.h>
> @@ -312,6 +313,21 @@ static int vmware_cpu_down_prepare(unsigned int cpu)
>  	local_irq_enable();
>  	return 0;
>  }
> +
> +static void vmware_play_dead(void)
> +{
> +	play_dead_common();
> +	tboot_shutdown(TB_SHUTDOWN_WFS);
> +
> +	/*
> +	 * Put the vCPU going offline in halt instead of mwait (even
> +	 * if mwait support is available), to make sure that the
> +	 * offline vCPU yields to the hypervisor (which may not happen
> +	 * with mwait, for example, if the guest's VMX is configured
> +	 * to retain the vCPU in guest context upon mwait).
> +	 */
> +	hlt_play_dead();
> +}
>  #endif
>  
>  static __init int activate_jump_labels(void)
> @@ -349,6 +365,7 @@ static void __init vmware_paravirt_ops_setup(void)
>  #ifdef CONFIG_SMP
>  		smp_ops.smp_prepare_boot_cpu =
>  			vmware_smp_prepare_boot_cpu;
> +		smp_ops.play_dead = vmware_play_dead;
>  		if (cpuhp_setup_state_nocalls(CPUHP_AP_ONLINE_DYN,
>  					      "x86/vmware:online",
>  					      vmware_cpu_online,
> 
> 

Re: [PATCH] smp/hotplug, x86/vmware: Put offline vCPUs in halt instead of mwait

[Peter Zijlstra]

On Thu, Jul 21, 2022 at 01:44:33PM -0700, Srivatsa S. Bhat wrote:
> From: Srivatsa S. Bhat (VMware) <srivatsa@csail.mit.edu>
> 
> VMware ESXi allows enabling a passthru mwait CPU-idle state in the
> guest using the following VMX option:
> 
> monitor_control.mwait_in_guest = "TRUE"
> 
> This lets a vCPU in mwait to remain in guest context (instead of
> yielding to the hypervisor via a VMEXIT), which helps speed up
> wakeups from idle.
> 
> However, this runs into problems with CPU hotplug, because the Linux
> CPU offline path prefers to put the vCPU-to-be-offlined in mwait
> state, whenever mwait is available. As a result, since a vCPU in mwait
> remains in guest context and does not yield to the hypervisor, an
> offline vCPU *appears* to be 100% busy as viewed from ESXi, which
> prevents the hypervisor from running other vCPUs or workloads on the
> corresponding pCPU (particularly when vCPU - pCPU mappings are
> statically defined by the user).

I would hope vCPU pinning is a mandatory thing when MWAIT passthrough it
set?

> [ Note that such a vCPU is not
> actually busy spinning though; it remains in mwait idle state in the
> guest ].
> 
> Fix this by overriding the CPU offline play_dead() callback for VMware
> hypervisor, by putting the CPU in halt state (which actually yields to
> the hypervisor), even if mwait support is available.
> 
> Signed-off-by: Srivatsa S. Bhat (VMware) <srivatsa@csail.mit.edu>
> ---

> +static void vmware_play_dead(void)
> +{
> +	play_dead_common();
> +	tboot_shutdown(TB_SHUTDOWN_WFS);
> +
> +	/*
> +	 * Put the vCPU going offline in halt instead of mwait (even
> +	 * if mwait support is available), to make sure that the
> +	 * offline vCPU yields to the hypervisor (which may not happen
> +	 * with mwait, for example, if the guest's VMX is configured
> +	 * to retain the vCPU in guest context upon mwait).
> +	 */
> +	hlt_play_dead();
> +}
>  #endif
>  
>  static __init int activate_jump_labels(void)
> @@ -349,6 +365,7 @@ static void __init vmware_paravirt_ops_setup(void)
>  #ifdef CONFIG_SMP
>  		smp_ops.smp_prepare_boot_cpu =
>  			vmware_smp_prepare_boot_cpu;
> +		smp_ops.play_dead = vmware_play_dead;
>  		if (cpuhp_setup_state_nocalls(CPUHP_AP_ONLINE_DYN,
>  					      "x86/vmware:online",
>  					      vmware_cpu_online,

No real objection here; but would not something like the below fix the
problem more generally? I'm thinking MWAIT passthrough for *any*
hypervisor doesn't want play_dead to use it.

diff --git a/arch/x86/kernel/smpboot.c b/arch/x86/kernel/smpboot.c
index f24227bc3220..166cb3aaca8a 100644
--- a/arch/x86/kernel/smpboot.c
+++ b/arch/x86/kernel/smpboot.c
@@ -1759,6 +1759,8 @@ static inline void mwait_play_dead(void)
 		return;
 	if (!this_cpu_has(X86_FEATURE_CLFLUSH))
 		return;
+	if (this_cpu_has(X86_FEATURE_HYPERVISOR))
+		return;
 	if (__this_cpu_read(cpu_info.cpuid_level) < CPUID_MWAIT_LEAF)
 		return;
 

Re: [PATCH] smp/hotplug, x86/vmware: Put offline vCPUs in halt instead of mwait

[Peter Zijlstra]

On Thu, Jul 21, 2022 at 01:44:33PM -0700, Srivatsa S. Bhat wrote:
> From: Srivatsa S. Bhat (VMware) <srivatsa@csail.mit.edu>
> 
> VMware ESXi allows enabling a passthru mwait CPU-idle state in the
> guest using the following VMX option:
> 
> monitor_control.mwait_in_guest = "TRUE"
> 
> This lets a vCPU in mwait to remain in guest context (instead of
> yielding to the hypervisor via a VMEXIT), which helps speed up
> wakeups from idle.
> 
> However, this runs into problems with CPU hotplug, because the Linux
> CPU offline path prefers to put the vCPU-to-be-offlined in mwait
> state, whenever mwait is available. As a result, since a vCPU in mwait
> remains in guest context and does not yield to the hypervisor, an
> offline vCPU *appears* to be 100% busy as viewed from ESXi, which
> prevents the hypervisor from running other vCPUs or workloads on the
> corresponding pCPU (particularly when vCPU - pCPU mappings are
> statically defined by the user).

I would hope vCPU pinning is a mandatory thing when MWAIT passthrough it
set?

> [ Note that such a vCPU is not
> actually busy spinning though; it remains in mwait idle state in the
> guest ].
> 
> Fix this by overriding the CPU offline play_dead() callback for VMware
> hypervisor, by putting the CPU in halt state (which actually yields to
> the hypervisor), even if mwait support is available.
> 
> Signed-off-by: Srivatsa S. Bhat (VMware) <srivatsa@csail.mit.edu>
> ---

> +static void vmware_play_dead(void)
> +{
> +	play_dead_common();
> +	tboot_shutdown(TB_SHUTDOWN_WFS);
> +
> +	/*
> +	 * Put the vCPU going offline in halt instead of mwait (even
> +	 * if mwait support is available), to make sure that the
> +	 * offline vCPU yields to the hypervisor (which may not happen
> +	 * with mwait, for example, if the guest's VMX is configured
> +	 * to retain the vCPU in guest context upon mwait).
> +	 */
> +	hlt_play_dead();
> +}
>  #endif
>  
>  static __init int activate_jump_labels(void)
> @@ -349,6 +365,7 @@ static void __init vmware_paravirt_ops_setup(void)
>  #ifdef CONFIG_SMP
>  		smp_ops.smp_prepare_boot_cpu =
>  			vmware_smp_prepare_boot_cpu;
> +		smp_ops.play_dead = vmware_play_dead;
>  		if (cpuhp_setup_state_nocalls(CPUHP_AP_ONLINE_DYN,
>  					      "x86/vmware:online",
>  					      vmware_cpu_online,

No real objection here; but would not something like the below fix the
problem more generally? I'm thinking MWAIT passthrough for *any*
hypervisor doesn't want play_dead to use it.

diff --git a/arch/x86/kernel/smpboot.c b/arch/x86/kernel/smpboot.c
index f24227bc3220..166cb3aaca8a 100644
--- a/arch/x86/kernel/smpboot.c
+++ b/arch/x86/kernel/smpboot.c
@@ -1759,6 +1759,8 @@ static inline void mwait_play_dead(void)
 		return;
 	if (!this_cpu_has(X86_FEATURE_CLFLUSH))
 		return;
+	if (this_cpu_has(X86_FEATURE_HYPERVISOR))
+		return;
 	if (__this_cpu_read(cpu_info.cpuid_level) < CPUID_MWAIT_LEAF)
 		return;
 
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [PATCH] smp/hotplug, x86/vmware: Put offline vCPUs in halt instead of mwait

[Borislav Petkov]

+ kvm ML and leaving the whole mail quoted in for them.

On Fri, Sep 23, 2022 at 09:05:26AM +0200, Peter Zijlstra wrote:
> On Thu, Jul 21, 2022 at 01:44:33PM -0700, Srivatsa S. Bhat wrote:
> > From: Srivatsa S. Bhat (VMware) <srivatsa@csail.mit.edu>
> > 
> > VMware ESXi allows enabling a passthru mwait CPU-idle state in the
> > guest using the following VMX option:
> > 
> > monitor_control.mwait_in_guest = "TRUE"
> > 
> > This lets a vCPU in mwait to remain in guest context (instead of
> > yielding to the hypervisor via a VMEXIT), which helps speed up
> > wakeups from idle.
> > 
> > However, this runs into problems with CPU hotplug, because the Linux
> > CPU offline path prefers to put the vCPU-to-be-offlined in mwait
> > state, whenever mwait is available. As a result, since a vCPU in mwait
> > remains in guest context and does not yield to the hypervisor, an
> > offline vCPU *appears* to be 100% busy as viewed from ESXi, which
> > prevents the hypervisor from running other vCPUs or workloads on the
> > corresponding pCPU (particularly when vCPU - pCPU mappings are
> > statically defined by the user).
> 
> I would hope vCPU pinning is a mandatory thing when MWAIT passthrough it
> set?
> 
> > [ Note that such a vCPU is not
> > actually busy spinning though; it remains in mwait idle state in the
> > guest ].
> > 
> > Fix this by overriding the CPU offline play_dead() callback for VMware
> > hypervisor, by putting the CPU in halt state (which actually yields to
> > the hypervisor), even if mwait support is available.
> > 
> > Signed-off-by: Srivatsa S. Bhat (VMware) <srivatsa@csail.mit.edu>
> > ---
> 
> > +static void vmware_play_dead(void)
> > +{
> > +	play_dead_common();
> > +	tboot_shutdown(TB_SHUTDOWN_WFS);
> > +
> > +	/*
> > +	 * Put the vCPU going offline in halt instead of mwait (even
> > +	 * if mwait support is available), to make sure that the
> > +	 * offline vCPU yields to the hypervisor (which may not happen
> > +	 * with mwait, for example, if the guest's VMX is configured
> > +	 * to retain the vCPU in guest context upon mwait).
> > +	 */
> > +	hlt_play_dead();
> > +}
> >  #endif
> >  
> >  static __init int activate_jump_labels(void)
> > @@ -349,6 +365,7 @@ static void __init vmware_paravirt_ops_setup(void)
> >  #ifdef CONFIG_SMP
> >  		smp_ops.smp_prepare_boot_cpu =
> >  			vmware_smp_prepare_boot_cpu;
> > +		smp_ops.play_dead = vmware_play_dead;
> >  		if (cpuhp_setup_state_nocalls(CPUHP_AP_ONLINE_DYN,
> >  					      "x86/vmware:online",
> >  					      vmware_cpu_online,
> 
> No real objection here; but would not something like the below fix the
> problem more generally? I'm thinking MWAIT passthrough for *any*
> hypervisor doesn't want play_dead to use it.
> 
> diff --git a/arch/x86/kernel/smpboot.c b/arch/x86/kernel/smpboot.c
> index f24227bc3220..166cb3aaca8a 100644
> --- a/arch/x86/kernel/smpboot.c
> +++ b/arch/x86/kernel/smpboot.c
> @@ -1759,6 +1759,8 @@ static inline void mwait_play_dead(void)
>  		return;
>  	if (!this_cpu_has(X86_FEATURE_CLFLUSH))
>  		return;
> +	if (this_cpu_has(X86_FEATURE_HYPERVISOR))
> +		return;
>  	if (__this_cpu_read(cpu_info.cpuid_level) < CPUID_MWAIT_LEAF)
>  		return;

Yeah, it would be nice if we could get a consensus here from all
relevant HVs.

-- 
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette

Re: [PATCH] smp/hotplug, x86/vmware: Put offline vCPUs in halt instead of mwait

[Borislav Petkov]

+ kvm ML and leaving the whole mail quoted in for them.

On Fri, Sep 23, 2022 at 09:05:26AM +0200, Peter Zijlstra wrote:
> On Thu, Jul 21, 2022 at 01:44:33PM -0700, Srivatsa S. Bhat wrote:
> > From: Srivatsa S. Bhat (VMware) <srivatsa@csail.mit.edu>
> > 
> > VMware ESXi allows enabling a passthru mwait CPU-idle state in the
> > guest using the following VMX option:
> > 
> > monitor_control.mwait_in_guest = "TRUE"
> > 
> > This lets a vCPU in mwait to remain in guest context (instead of
> > yielding to the hypervisor via a VMEXIT), which helps speed up
> > wakeups from idle.
> > 
> > However, this runs into problems with CPU hotplug, because the Linux
> > CPU offline path prefers to put the vCPU-to-be-offlined in mwait
> > state, whenever mwait is available. As a result, since a vCPU in mwait
> > remains in guest context and does not yield to the hypervisor, an
> > offline vCPU *appears* to be 100% busy as viewed from ESXi, which
> > prevents the hypervisor from running other vCPUs or workloads on the
> > corresponding pCPU (particularly when vCPU - pCPU mappings are
> > statically defined by the user).
> 
> I would hope vCPU pinning is a mandatory thing when MWAIT passthrough it
> set?
> 
> > [ Note that such a vCPU is not
> > actually busy spinning though; it remains in mwait idle state in the
> > guest ].
> > 
> > Fix this by overriding the CPU offline play_dead() callback for VMware
> > hypervisor, by putting the CPU in halt state (which actually yields to
> > the hypervisor), even if mwait support is available.
> > 
> > Signed-off-by: Srivatsa S. Bhat (VMware) <srivatsa@csail.mit.edu>
> > ---
> 
> > +static void vmware_play_dead(void)
> > +{
> > +	play_dead_common();
> > +	tboot_shutdown(TB_SHUTDOWN_WFS);
> > +
> > +	/*
> > +	 * Put the vCPU going offline in halt instead of mwait (even
> > +	 * if mwait support is available), to make sure that the
> > +	 * offline vCPU yields to the hypervisor (which may not happen
> > +	 * with mwait, for example, if the guest's VMX is configured
> > +	 * to retain the vCPU in guest context upon mwait).
> > +	 */
> > +	hlt_play_dead();
> > +}
> >  #endif
> >  
> >  static __init int activate_jump_labels(void)
> > @@ -349,6 +365,7 @@ static void __init vmware_paravirt_ops_setup(void)
> >  #ifdef CONFIG_SMP
> >  		smp_ops.smp_prepare_boot_cpu =
> >  			vmware_smp_prepare_boot_cpu;
> > +		smp_ops.play_dead = vmware_play_dead;
> >  		if (cpuhp_setup_state_nocalls(CPUHP_AP_ONLINE_DYN,
> >  					      "x86/vmware:online",
> >  					      vmware_cpu_online,
> 
> No real objection here; but would not something like the below fix the
> problem more generally? I'm thinking MWAIT passthrough for *any*
> hypervisor doesn't want play_dead to use it.
> 
> diff --git a/arch/x86/kernel/smpboot.c b/arch/x86/kernel/smpboot.c
> index f24227bc3220..166cb3aaca8a 100644
> --- a/arch/x86/kernel/smpboot.c
> +++ b/arch/x86/kernel/smpboot.c
> @@ -1759,6 +1759,8 @@ static inline void mwait_play_dead(void)
>  		return;
>  	if (!this_cpu_has(X86_FEATURE_CLFLUSH))
>  		return;
> +	if (this_cpu_has(X86_FEATURE_HYPERVISOR))
> +		return;
>  	if (__this_cpu_read(cpu_info.cpuid_level) < CPUID_MWAIT_LEAF)
>  		return;

Yeah, it would be nice if we could get a consensus here from all
relevant HVs.

-- 
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [PATCH] smp/hotplug, x86/vmware: Put offline vCPUs in halt instead of mwait

[Juergen Gross]

[-- Attachment #1.1.1: Type: text/plain, Size: 3200 bytes --]

On 23.09.22 09:05, Peter Zijlstra wrote:
> On Thu, Jul 21, 2022 at 01:44:33PM -0700, Srivatsa S. Bhat wrote:
>> From: Srivatsa S. Bhat (VMware) <srivatsa@csail.mit.edu>
>>
>> VMware ESXi allows enabling a passthru mwait CPU-idle state in the
>> guest using the following VMX option:
>>
>> monitor_control.mwait_in_guest = "TRUE"
>>
>> This lets a vCPU in mwait to remain in guest context (instead of
>> yielding to the hypervisor via a VMEXIT), which helps speed up
>> wakeups from idle.
>>
>> However, this runs into problems with CPU hotplug, because the Linux
>> CPU offline path prefers to put the vCPU-to-be-offlined in mwait
>> state, whenever mwait is available. As a result, since a vCPU in mwait
>> remains in guest context and does not yield to the hypervisor, an
>> offline vCPU *appears* to be 100% busy as viewed from ESXi, which
>> prevents the hypervisor from running other vCPUs or workloads on the
>> corresponding pCPU (particularly when vCPU - pCPU mappings are
>> statically defined by the user).
> 
> I would hope vCPU pinning is a mandatory thing when MWAIT passthrough it
> set?
> 
>> [ Note that such a vCPU is not
>> actually busy spinning though; it remains in mwait idle state in the
>> guest ].
>>
>> Fix this by overriding the CPU offline play_dead() callback for VMware
>> hypervisor, by putting the CPU in halt state (which actually yields to
>> the hypervisor), even if mwait support is available.
>>
>> Signed-off-by: Srivatsa S. Bhat (VMware) <srivatsa@csail.mit.edu>
>> ---
> 
>> +static void vmware_play_dead(void)
>> +{
>> +	play_dead_common();
>> +	tboot_shutdown(TB_SHUTDOWN_WFS);
>> +
>> +	/*
>> +	 * Put the vCPU going offline in halt instead of mwait (even
>> +	 * if mwait support is available), to make sure that the
>> +	 * offline vCPU yields to the hypervisor (which may not happen
>> +	 * with mwait, for example, if the guest's VMX is configured
>> +	 * to retain the vCPU in guest context upon mwait).
>> +	 */
>> +	hlt_play_dead();
>> +}
>>   #endif
>>   
>>   static __init int activate_jump_labels(void)
>> @@ -349,6 +365,7 @@ static void __init vmware_paravirt_ops_setup(void)
>>   #ifdef CONFIG_SMP
>>   		smp_ops.smp_prepare_boot_cpu =
>>   			vmware_smp_prepare_boot_cpu;
>> +		smp_ops.play_dead = vmware_play_dead;
>>   		if (cpuhp_setup_state_nocalls(CPUHP_AP_ONLINE_DYN,
>>   					      "x86/vmware:online",
>>   					      vmware_cpu_online,
> 
> No real objection here; but would not something like the below fix the
> problem more generally? I'm thinking MWAIT passthrough for *any*
> hypervisor doesn't want play_dead to use it.
> 
> diff --git a/arch/x86/kernel/smpboot.c b/arch/x86/kernel/smpboot.c
> index f24227bc3220..166cb3aaca8a 100644
> --- a/arch/x86/kernel/smpboot.c
> +++ b/arch/x86/kernel/smpboot.c
> @@ -1759,6 +1759,8 @@ static inline void mwait_play_dead(void)
>   		return;
>   	if (!this_cpu_has(X86_FEATURE_CLFLUSH))
>   		return;
> +	if (this_cpu_has(X86_FEATURE_HYPERVISOR))
> +		return;
>   	if (__this_cpu_read(cpu_info.cpuid_level) < CPUID_MWAIT_LEAF)
>   		return;
>   

With my Xen hat on I agree with this approach.


Juergen

[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 3149 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 495 bytes --]

Re: [PATCH] smp/hotplug, x86/vmware: Put offline vCPUs in halt instead of mwait

[Juergen Gross via Virtualization]

[-- Attachment #1.1.1.1: Type: text/plain, Size: 3200 bytes --]

On 23.09.22 09:05, Peter Zijlstra wrote:
> On Thu, Jul 21, 2022 at 01:44:33PM -0700, Srivatsa S. Bhat wrote:
>> From: Srivatsa S. Bhat (VMware) <srivatsa@csail.mit.edu>
>>
>> VMware ESXi allows enabling a passthru mwait CPU-idle state in the
>> guest using the following VMX option:
>>
>> monitor_control.mwait_in_guest = "TRUE"
>>
>> This lets a vCPU in mwait to remain in guest context (instead of
>> yielding to the hypervisor via a VMEXIT), which helps speed up
>> wakeups from idle.
>>
>> However, this runs into problems with CPU hotplug, because the Linux
>> CPU offline path prefers to put the vCPU-to-be-offlined in mwait
>> state, whenever mwait is available. As a result, since a vCPU in mwait
>> remains in guest context and does not yield to the hypervisor, an
>> offline vCPU *appears* to be 100% busy as viewed from ESXi, which
>> prevents the hypervisor from running other vCPUs or workloads on the
>> corresponding pCPU (particularly when vCPU - pCPU mappings are
>> statically defined by the user).
> 
> I would hope vCPU pinning is a mandatory thing when MWAIT passthrough it
> set?
> 
>> [ Note that such a vCPU is not
>> actually busy spinning though; it remains in mwait idle state in the
>> guest ].
>>
>> Fix this by overriding the CPU offline play_dead() callback for VMware
>> hypervisor, by putting the CPU in halt state (which actually yields to
>> the hypervisor), even if mwait support is available.
>>
>> Signed-off-by: Srivatsa S. Bhat (VMware) <srivatsa@csail.mit.edu>
>> ---
> 
>> +static void vmware_play_dead(void)
>> +{
>> +	play_dead_common();
>> +	tboot_shutdown(TB_SHUTDOWN_WFS);
>> +
>> +	/*
>> +	 * Put the vCPU going offline in halt instead of mwait (even
>> +	 * if mwait support is available), to make sure that the
>> +	 * offline vCPU yields to the hypervisor (which may not happen
>> +	 * with mwait, for example, if the guest's VMX is configured
>> +	 * to retain the vCPU in guest context upon mwait).
>> +	 */
>> +	hlt_play_dead();
>> +}
>>   #endif
>>   
>>   static __init int activate_jump_labels(void)
>> @@ -349,6 +365,7 @@ static void __init vmware_paravirt_ops_setup(void)
>>   #ifdef CONFIG_SMP
>>   		smp_ops.smp_prepare_boot_cpu =
>>   			vmware_smp_prepare_boot_cpu;
>> +		smp_ops.play_dead = vmware_play_dead;
>>   		if (cpuhp_setup_state_nocalls(CPUHP_AP_ONLINE_DYN,
>>   					      "x86/vmware:online",
>>   					      vmware_cpu_online,
> 
> No real objection here; but would not something like the below fix the
> problem more generally? I'm thinking MWAIT passthrough for *any*
> hypervisor doesn't want play_dead to use it.
> 
> diff --git a/arch/x86/kernel/smpboot.c b/arch/x86/kernel/smpboot.c
> index f24227bc3220..166cb3aaca8a 100644
> --- a/arch/x86/kernel/smpboot.c
> +++ b/arch/x86/kernel/smpboot.c
> @@ -1759,6 +1759,8 @@ static inline void mwait_play_dead(void)
>   		return;
>   	if (!this_cpu_has(X86_FEATURE_CLFLUSH))
>   		return;
> +	if (this_cpu_has(X86_FEATURE_HYPERVISOR))
> +		return;
>   	if (__this_cpu_read(cpu_info.cpuid_level) < CPUID_MWAIT_LEAF)
>   		return;
>   

With my Xen hat on I agree with this approach.


Juergen

[-- Attachment #1.1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 3149 bytes --]

[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 495 bytes --]

[-- Attachment #2: Type: text/plain, Size: 183 bytes --]

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [PATCH] smp/hotplug, x86/vmware: Put offline vCPUs in halt instead of mwait

[Srivatsa S. Bhat]

On 9/23/22 3:45 AM, Borislav Petkov wrote:
> + kvm ML and leaving the whole mail quoted in for them.
> 
> On Fri, Sep 23, 2022 at 09:05:26AM +0200, Peter Zijlstra wrote:
>> On Thu, Jul 21, 2022 at 01:44:33PM -0700, Srivatsa S. Bhat wrote:
>>> From: Srivatsa S. Bhat (VMware) <srivatsa@csail.mit.edu>
>>>
>>> VMware ESXi allows enabling a passthru mwait CPU-idle state in the
>>> guest using the following VMX option:
>>>
>>> monitor_control.mwait_in_guest = "TRUE"
>>>
>>> This lets a vCPU in mwait to remain in guest context (instead of
>>> yielding to the hypervisor via a VMEXIT), which helps speed up
>>> wakeups from idle.
>>>
>>> However, this runs into problems with CPU hotplug, because the Linux
>>> CPU offline path prefers to put the vCPU-to-be-offlined in mwait
>>> state, whenever mwait is available. As a result, since a vCPU in mwait
>>> remains in guest context and does not yield to the hypervisor, an
>>> offline vCPU *appears* to be 100% busy as viewed from ESXi, which
>>> prevents the hypervisor from running other vCPUs or workloads on the
>>> corresponding pCPU (particularly when vCPU - pCPU mappings are
>>> statically defined by the user).
>>
>> I would hope vCPU pinning is a mandatory thing when MWAIT passthrough it
>> set?
>>
>>> [ Note that such a vCPU is not
>>> actually busy spinning though; it remains in mwait idle state in the
>>> guest ].
>>>
>>> Fix this by overriding the CPU offline play_dead() callback for VMware
>>> hypervisor, by putting the CPU in halt state (which actually yields to
>>> the hypervisor), even if mwait support is available.
>>>
>>> Signed-off-by: Srivatsa S. Bhat (VMware) <srivatsa@csail.mit.edu>
>>> ---
>>
>>> +static void vmware_play_dead(void)
>>> +{
>>> +	play_dead_common();
>>> +	tboot_shutdown(TB_SHUTDOWN_WFS);
>>> +
>>> +	/*
>>> +	 * Put the vCPU going offline in halt instead of mwait (even
>>> +	 * if mwait support is available), to make sure that the
>>> +	 * offline vCPU yields to the hypervisor (which may not happen
>>> +	 * with mwait, for example, if the guest's VMX is configured
>>> +	 * to retain the vCPU in guest context upon mwait).
>>> +	 */
>>> +	hlt_play_dead();
>>> +}
>>>  #endif
>>>  
>>>  static __init int activate_jump_labels(void)
>>> @@ -349,6 +365,7 @@ static void __init vmware_paravirt_ops_setup(void)
>>>  #ifdef CONFIG_SMP
>>>  		smp_ops.smp_prepare_boot_cpu =
>>>  			vmware_smp_prepare_boot_cpu;
>>> +		smp_ops.play_dead = vmware_play_dead;
>>>  		if (cpuhp_setup_state_nocalls(CPUHP_AP_ONLINE_DYN,
>>>  					      "x86/vmware:online",
>>>  					      vmware_cpu_online,
>>
>> No real objection here; but would not something like the below fix the
>> problem more generally? I'm thinking MWAIT passthrough for *any*
>> hypervisor doesn't want play_dead to use it.
>>

That would be better indeed, thank you!

>> diff --git a/arch/x86/kernel/smpboot.c b/arch/x86/kernel/smpboot.c
>> index f24227bc3220..166cb3aaca8a 100644
>> --- a/arch/x86/kernel/smpboot.c
>> +++ b/arch/x86/kernel/smpboot.c
>> @@ -1759,6 +1759,8 @@ static inline void mwait_play_dead(void)
>>  		return;
>>  	if (!this_cpu_has(X86_FEATURE_CLFLUSH))
>>  		return;
>> +	if (this_cpu_has(X86_FEATURE_HYPERVISOR))
>> +		return;
>>  	if (__this_cpu_read(cpu_info.cpuid_level) < CPUID_MWAIT_LEAF)
>>  		return;
> 
> Yeah, it would be nice if we could get a consensus here from all
> relevant HVs.
> 

I'll send out a v2 after trying out these changes.

Thank you!

Regards,
Srivatsa
VMware Photon OS

Re: [PATCH] smp/hotplug, x86/vmware: Put offline vCPUs in halt instead of mwait

[Srivatsa S. Bhat]

On 9/23/22 3:45 AM, Borislav Petkov wrote:
> + kvm ML and leaving the whole mail quoted in for them.
> 
> On Fri, Sep 23, 2022 at 09:05:26AM +0200, Peter Zijlstra wrote:
>> On Thu, Jul 21, 2022 at 01:44:33PM -0700, Srivatsa S. Bhat wrote:
>>> From: Srivatsa S. Bhat (VMware) <srivatsa@csail.mit.edu>
>>>
>>> VMware ESXi allows enabling a passthru mwait CPU-idle state in the
>>> guest using the following VMX option:
>>>
>>> monitor_control.mwait_in_guest = "TRUE"
>>>
>>> This lets a vCPU in mwait to remain in guest context (instead of
>>> yielding to the hypervisor via a VMEXIT), which helps speed up
>>> wakeups from idle.
>>>
>>> However, this runs into problems with CPU hotplug, because the Linux
>>> CPU offline path prefers to put the vCPU-to-be-offlined in mwait
>>> state, whenever mwait is available. As a result, since a vCPU in mwait
>>> remains in guest context and does not yield to the hypervisor, an
>>> offline vCPU *appears* to be 100% busy as viewed from ESXi, which
>>> prevents the hypervisor from running other vCPUs or workloads on the
>>> corresponding pCPU (particularly when vCPU - pCPU mappings are
>>> statically defined by the user).
>>
>> I would hope vCPU pinning is a mandatory thing when MWAIT passthrough it
>> set?
>>
>>> [ Note that such a vCPU is not
>>> actually busy spinning though; it remains in mwait idle state in the
>>> guest ].
>>>
>>> Fix this by overriding the CPU offline play_dead() callback for VMware
>>> hypervisor, by putting the CPU in halt state (which actually yields to
>>> the hypervisor), even if mwait support is available.
>>>
>>> Signed-off-by: Srivatsa S. Bhat (VMware) <srivatsa@csail.mit.edu>
>>> ---
>>
>>> +static void vmware_play_dead(void)
>>> +{
>>> +	play_dead_common();
>>> +	tboot_shutdown(TB_SHUTDOWN_WFS);
>>> +
>>> +	/*
>>> +	 * Put the vCPU going offline in halt instead of mwait (even
>>> +	 * if mwait support is available), to make sure that the
>>> +	 * offline vCPU yields to the hypervisor (which may not happen
>>> +	 * with mwait, for example, if the guest's VMX is configured
>>> +	 * to retain the vCPU in guest context upon mwait).
>>> +	 */
>>> +	hlt_play_dead();
>>> +}
>>>  #endif
>>>  
>>>  static __init int activate_jump_labels(void)
>>> @@ -349,6 +365,7 @@ static void __init vmware_paravirt_ops_setup(void)
>>>  #ifdef CONFIG_SMP
>>>  		smp_ops.smp_prepare_boot_cpu =
>>>  			vmware_smp_prepare_boot_cpu;
>>> +		smp_ops.play_dead = vmware_play_dead;
>>>  		if (cpuhp_setup_state_nocalls(CPUHP_AP_ONLINE_DYN,
>>>  					      "x86/vmware:online",
>>>  					      vmware_cpu_online,
>>
>> No real objection here; but would not something like the below fix the
>> problem more generally? I'm thinking MWAIT passthrough for *any*
>> hypervisor doesn't want play_dead to use it.
>>

That would be better indeed, thank you!

>> diff --git a/arch/x86/kernel/smpboot.c b/arch/x86/kernel/smpboot.c
>> index f24227bc3220..166cb3aaca8a 100644
>> --- a/arch/x86/kernel/smpboot.c
>> +++ b/arch/x86/kernel/smpboot.c
>> @@ -1759,6 +1759,8 @@ static inline void mwait_play_dead(void)
>>  		return;
>>  	if (!this_cpu_has(X86_FEATURE_CLFLUSH))
>>  		return;
>> +	if (this_cpu_has(X86_FEATURE_HYPERVISOR))
>> +		return;
>>  	if (__this_cpu_read(cpu_info.cpuid_level) < CPUID_MWAIT_LEAF)
>>  		return;
> 
> Yeah, it would be nice if we could get a consensus here from all
> relevant HVs.
> 

I'll send out a v2 after trying out these changes.

Thank you!

Regards,
Srivatsa
VMware Photon OS
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization