[PATCH v2 0/2] Enable PV qspinlock for Hyper-V

[PATCH v2 0/2] Enable PV qspinlock for Hyper-V

[Yi Sun]

v1->v2:
    - compile hv_spinlock.c only when CONFIG_PARAVIRT_SPINLOCKS enabled
    - merge v1 patch 2/3 to single patch
    - remove part of the boilerplate in hv_spinlock.c
    - declare hv_pvspin as __initdata
    - remove spin_wait_info and hv_notify_long_spin_wait because
      SpinWaitInfo is a standalone feature.
    - add comments for reading HV_X64_MSR_GUEST_IDLE
    - replace pr_warn to pr_info
    - use pr_fmt instead of the 'hv:' prefix
    - register callback function for smp_ops.smp_prepare_boot_cpu
      to initialize hyper-v spinlock

This patch adds the necessary Hyper-V specific code to allow
PV qspinlock work on Hyper-V.

In wait callback function, read HV_X64_MSR_GUEST_IDLE MSR
to trigger the guest's transition to the idle power state
which can be exited by an IPI even if IF flag is disabled.

In kick callback function, just send platform IPI to make
waiting vcpu exit idle state.

In vcpu_is_preempted callback function, return false directly
because Hyper-V does not provide such interface so far.


Cc: "K. Y. Srinivasan" <kys@microsoft.com>
Cc: Haiyang Zhang <haiyangz@microsoft.com>
Cc: Stephen Hemminger <sthemmin@microsoft.com>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Thomas Gleixner <tglx@linutronix.de>

Yi Sun (2):
  X86/Hyper-V: Add Guest IDLE MSR support
  locking/pvqspinlock, hv: Enable PV qspinlock for Hyper-V

 Documentation/admin-guide/kernel-parameters.txt |  5 ++
 arch/x86/hyperv/Makefile                        |  4 ++
 arch/x86/hyperv/hv_spinlock.c                   | 81 +++++++++++++++++++++++++
 arch/x86/include/asm/hyperv-tlfs.h              |  5 ++
 arch/x86/include/asm/mshyperv.h                 |  3 +
 arch/x86/kernel/cpu/mshyperv.c                  | 12 ++++
 6 files changed, 110 insertions(+)
 create mode 100644 arch/x86/hyperv/hv_spinlock.c

-- 
1.9.1

[PATCH v2 1/2] X86/Hyper-V: Add Guest IDLE MSR support

[Yi Sun]

Hyper-V may expose a HV_X64_MSR_GUEST_IDLE MSR. We can read
HYPERV_CPUID_FEATURES eax to check it. Read this MSR can
trigger the guest's transition to the idle power state which
can be exited by an IPI even if IF flag is disabled.

Cc: "K. Y. Srinivasan" <kys@microsoft.com>
Cc: Haiyang Zhang <haiyangz@microsoft.com>
Cc: Stephen Hemminger <sthemmin@microsoft.com>
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
---
 arch/x86/include/asm/hyperv-tlfs.h | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/arch/x86/include/asm/hyperv-tlfs.h b/arch/x86/include/asm/hyperv-tlfs.h
index e977b6b..2a2fa17 100644
--- a/arch/x86/include/asm/hyperv-tlfs.h
+++ b/arch/x86/include/asm/hyperv-tlfs.h
@@ -38,6 +38,8 @@
 #define HV_MSR_TIME_REF_COUNT_AVAILABLE		(1 << 1)
 /* Partition reference TSC MSR is available */
 #define HV_MSR_REFERENCE_TSC_AVAILABLE		(1 << 9)
+/* Partition Guest IDLE MSR is available */
+#define HV_X64_MSR_GUEST_IDLE_AVAILABLE		(1 << 10)
 
 /* A partition's reference time stamp counter (TSC) page */
 #define HV_X64_MSR_REFERENCE_TSC		0x40000021
@@ -246,6 +248,9 @@
 #define HV_X64_MSR_STIMER3_CONFIG		0x400000B6
 #define HV_X64_MSR_STIMER3_COUNT		0x400000B7
 
+/* Hyper-V guest idle MSR */
+#define HV_X64_MSR_GUEST_IDLE			0x400000F0
+
 /* Hyper-V guest crash notification MSR's */
 #define HV_X64_MSR_CRASH_P0			0x40000100
 #define HV_X64_MSR_CRASH_P1			0x40000101
-- 
1.9.1

[PATCH v2 2/2] locking/pvqspinlock, hv: Enable PV qspinlock for Hyper-V

[Yi Sun]

Follow PV spinlock mechanism to implement the callback functions
to allow the CPU idling and kicking operations on Hyper-V.

Cc: "K. Y. Srinivasan" <kys@microsoft.com>
Cc: Haiyang Zhang <haiyangz@microsoft.com>
Cc: Stephen Hemminger <sthemmin@microsoft.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
---
v1->v2:
    - compile hv_spinlock.c only when CONFIG_PARAVIRT_SPINLOCKS enabled
    - merge v1 patch 2/3 to single patch
      (suggested by Thomas Gleixner)
    - remove part of the boilerplate in hv_spinlock.c
      (suggested by Thomas Gleixner)
    - declare hv_pvspin as __initdata
      (suggested by Thomas Gleixner)
    - remove spin_wait_info and hv_notify_long_spin_wait because
      SpinWaitInfo is a standalone feature.
    - add comments for reading HV_X64_MSR_GUEST_IDLE
      (suggested by Thomas Gleixner)
    - replace pr_warn to pr_info
      (suggested by Thomas Gleixner)
    - use pr_fmt instead of the 'hv:' prefix
      (suggested by Thomas Gleixner)
    - register callback function for smp_ops.smp_prepare_boot_cpu
      to initialize hyper-v spinlock
      (suggested by Thomas Gleixner)
---
 Documentation/admin-guide/kernel-parameters.txt |  5 ++
 arch/x86/hyperv/Makefile                        |  4 ++
 arch/x86/hyperv/hv_spinlock.c                   | 81 +++++++++++++++++++++++++
 arch/x86/include/asm/mshyperv.h                 |  3 +
 arch/x86/kernel/cpu/mshyperv.c                  | 12 ++++
 5 files changed, 105 insertions(+)
 create mode 100644 arch/x86/hyperv/hv_spinlock.c

diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt
index 92eb1f4..0fc8448 100644
--- a/Documentation/admin-guide/kernel-parameters.txt
+++ b/Documentation/admin-guide/kernel-parameters.txt
@@ -1385,6 +1385,11 @@
 	hvc_iucv_allow=	[S390]	Comma-separated list of z/VM user IDs.
 				If specified, z/VM IUCV HVC accepts connections
 				from listed z/VM user IDs only.
+
+	hv_nopvspin	[X86,HYPER_V]
+			Disables the ticketlock slowpath using HYPER-V PV
+			optimizations.
+
 	keep_bootcon	[KNL]
 			Do not unregister boot console at start. This is only
 			useful for debugging when something happens in the window
diff --git a/arch/x86/hyperv/Makefile b/arch/x86/hyperv/Makefile
index b21ee65..1c11f94 100644
--- a/arch/x86/hyperv/Makefile
+++ b/arch/x86/hyperv/Makefile
@@ -1,2 +1,6 @@
 obj-y			:= hv_init.o mmu.o nested.o
 obj-$(CONFIG_X86_64)	+= hv_apic.o
+
+ifdef CONFIG_X86_64
+obj-$(CONFIG_PARAVIRT_SPINLOCKS)	+= hv_spinlock.o
+endif
diff --git a/arch/x86/hyperv/hv_spinlock.c b/arch/x86/hyperv/hv_spinlock.c
new file mode 100644
index 0000000..0dafcc6
--- /dev/null
+++ b/arch/x86/hyperv/hv_spinlock.c
@@ -0,0 +1,81 @@
+// SPDX-License-Identifier: GPL-2.0
+
+/*
+ * Hyper-V specific spinlock code.
+ *
+ * Copyright (C) 2018, Intel, Inc.
+ *
+ * Author : Yi Sun <yi.y.sun@intel.com>
+ */
+
+#define pr_fmt(fmt) "hv: " fmt
+
+#include <linux/kernel_stat.h>
+#include <linux/spinlock.h>
+#include <linux/debugfs.h>
+#include <linux/log2.h>
+#include <linux/gfp.h>
+
+#include <asm/mshyperv.h>
+#include <asm/hyperv-tlfs.h>
+#include <asm/paravirt.h>
+#include <asm/qspinlock.h>
+#include <asm/apic.h>
+
+static bool __initdata hv_pvspin = true;
+
+static void hv_qlock_kick(int cpu)
+{
+	apic->send_IPI(cpu, X86_PLATFORM_IPI_VECTOR);
+}
+
+static void hv_qlock_wait(u8 *byte, u8 val)
+{
+	unsigned long msr_val;
+
+	if (READ_ONCE(*byte) != val)
+		return;
+
+	/*
+	 * Read HV_X64_MSR_GUEST_IDLE MSR can trigger the guest's
+	 * transition to the idle power state which can be exited
+	 * by an IPI even if IF flag is disabled.
+	 */
+	if (ms_hyperv.features & HV_X64_MSR_GUEST_IDLE_AVAILABLE)
+		rdmsrl(HV_X64_MSR_GUEST_IDLE, msr_val);
+}
+
+/*
+ * Hyper-V does not support this so far.
+ */
+bool hv_vcpu_is_preempted(int vcpu)
+{
+	return false;
+}
+PV_CALLEE_SAVE_REGS_THUNK(hv_vcpu_is_preempted);
+
+void __init hv_init_spinlocks(void)
+{
+	if (!hv_pvspin ||
+	    !apic ||
+	    !(ms_hyperv.hints & HV_X64_CLUSTER_IPI_RECOMMENDED) ||
+	    !(ms_hyperv.features & HV_X64_MSR_GUEST_IDLE_AVAILABLE)) {
+		pr_info("PV spinlocks disabled\n");
+		return;
+	}
+	pr_info("PV spinlocks enabled\n");
+
+	__pv_init_lock_hash();
+	pv_lock_ops.queued_spin_lock_slowpath = __pv_queued_spin_lock_slowpath;
+	pv_lock_ops.queued_spin_unlock = PV_CALLEE_SAVE(__pv_queued_spin_unlock);
+	pv_lock_ops.wait = hv_qlock_wait;
+	pv_lock_ops.kick = hv_qlock_kick;
+	pv_lock_ops.vcpu_is_preempted = PV_CALLEE_SAVE(hv_vcpu_is_preempted);
+}
+
+static __init int hv_parse_nopvspin(char *arg)
+{
+	hv_pvspin = false;
+	return 0;
+}
+early_param("hv_nopvspin", hv_parse_nopvspin);
diff --git a/arch/x86/include/asm/mshyperv.h b/arch/x86/include/asm/mshyperv.h
index f377044..ac36ea9 100644
--- a/arch/x86/include/asm/mshyperv.h
+++ b/arch/x86/include/asm/mshyperv.h
@@ -355,6 +355,8 @@ static inline int cpumask_to_vpset(struct hv_vpset *vpset,
 static inline void hv_apic_init(void) {}
 #endif
 
+void __init hv_init_spinlocks(void);
+
 #else /* CONFIG_HYPERV */
 static inline void hyperv_init(void) {}
 static inline bool hv_is_hyperv_initialized(void) { return false; }
@@ -368,6 +370,7 @@ static inline struct hv_vp_assist_page *hv_get_vp_assist_page(unsigned int cpu)
 	return NULL;
 }
 static inline int hyperv_flush_guest_mapping(u64 as) { return -1; }
+void __init hv_init_spinlocks(void) {}
 #endif /* CONFIG_HYPERV */
 
 #ifdef CONFIG_HYPERV_TSCPAGE
diff --git a/arch/x86/kernel/cpu/mshyperv.c b/arch/x86/kernel/cpu/mshyperv.c
index ad12733..4d12d3c 100644
--- a/arch/x86/kernel/cpu/mshyperv.c
+++ b/arch/x86/kernel/cpu/mshyperv.c
@@ -199,6 +199,14 @@ static unsigned long hv_get_tsc_khz(void)
 	return freq / 1000;
 }
 
+#if defined(CONFIG_SMP) && defined(CONFIG_PARAVIRT_SPINLOCKS)
+static void __init hv_smp_prepare_boot_cpu(void)
+{
+	native_smp_prepare_boot_cpu();
+	hv_init_spinlocks();
+}
+#endif
+
 static void __init ms_hyperv_init_platform(void)
 {
 	int hv_host_info_eax;
@@ -304,6 +312,10 @@ static void __init ms_hyperv_init_platform(void)
 		alloc_intr_gate(HYPERV_STIMER0_VECTOR,
 				hv_stimer0_callback_vector);
 #endif
+
+#if defined(CONFIG_SMP) && defined(CONFIG_PARAVIRT_SPINLOCKS)
+	smp_ops.smp_prepare_boot_cpu = hv_smp_prepare_boot_cpu;
+#endif
 }
 
 const __initconst struct hypervisor_x86 x86_hyper_ms_hyperv = {
-- 
1.9.1

RE: [PATCH v2 1/2] X86/Hyper-V: Add Guest IDLE MSR support

[Michael Kelley (EOSG)]

From: Yi Sun <yi.y.sun@linux.intel.com>  Sent: Friday, September 21, 2018 12:25 AM
> 
> Hyper-V may expose a HV_X64_MSR_GUEST_IDLE MSR. We can read
> HYPERV_CPUID_FEATURES eax to check it. Read this MSR can
> trigger the guest's transition to the idle power state which
> can be exited by an IPI even if IF flag is disabled.
> 
> Cc: "K. Y. Srinivasan" <kys@microsoft.com>
> Cc: Haiyang Zhang <haiyangz@microsoft.com>
> Cc: Stephen Hemminger <sthemmin@microsoft.com>
> Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
> ---
>  arch/x86/include/asm/hyperv-tlfs.h | 5 +++++
>  1 file changed, 5 insertions(+)
> 

Reviewed-by: Michael Kelley <mikelley@microsoft.com>

RE: [PATCH v2 2/2] locking/pvqspinlock, hv: Enable PV qspinlock for Hyper-V

[Michael Kelley (EOSG)]

From: Yi Sun <yi.y.sun@linux.intel.com> Sent: Friday, September 21, 2018 12:25 AM
> +
> +#define pr_fmt(fmt) "hv: " fmt

Other Hyper-V messages use "Hyper-V: " as the prefix, not "hv: ".  Take
a quick look at 'dmesg' output for reference.

> +
> +#include <linux/kernel_stat.h>
> +#include <linux/spinlock.h>
> +#include <linux/debugfs.h>
> +#include <linux/log2.h>
> +#include <linux/gfp.h>

Some of these #includes look like they might be leftovers from
some other code.  Please check and see whether kernel_stat.h,
debugsfs.h, log2.h, and gfp.h are actually needed.

> +static void hv_qlock_wait(u8 *byte, u8 val)
> +{
> +	unsigned long msr_val;
> +
> +	if (READ_ONCE(*byte) != val)
> +		return;
> +
> +	/*
> +	 * Read HV_X64_MSR_GUEST_IDLE MSR can trigger the guest's
> +	 * transition to the idle power state which can be exited
> +	 * by an IPI even if IF flag is disabled.
> +	 */
> +	if (ms_hyperv.features & HV_X64_MSR_GUEST_IDLE_AVAILABLE)

I can't see a case where this test is actually needed.  hv_qlock_wait()
can only get called if the flag is set when hv_init_spinlocks() is run, and
the flag value doesn't change after it is set.

> +		rdmsrl(HV_X64_MSR_GUEST_IDLE, msr_val);
> +}

Michael

Re: [PATCH v2 2/2] locking/pvqspinlock, hv: Enable PV qspinlock for Hyper-V

[kbuild test robot]

[-- Attachment #1: Type: text/plain, Size: 1780 bytes --]

Hi Yi,

Thank you for the patch! Yet something to improve:

[auto build test ERROR on tip/x86/core]
[also build test ERROR on v4.19-rc4 next-20180921]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system]

url:    https://github.com/0day-ci/linux/commits/Yi-Sun/Enable-PV-qspinlock-for-Hyper-V/20180921-234519
config: x86_64-randconfig-s1-09212334 (attached as .config)
compiler: gcc-6 (Debian 6.4.0-9) 6.4.0 20171026
reproduce:
        # save the attached .config to linux build tree
        make ARCH=x86_64 

All errors (new ones prefixed by >>):

   arch/x86/kernel/cpu/mshyperv.o: In function `hv_init_spinlocks':
>> arch/x86/include/asm/mshyperv.h:373: multiple definition of `hv_init_spinlocks'
   arch/x86/entry/vdso/vma.o:arch/x86/include/asm/mshyperv.h:373: first defined here

vim +373 arch/x86/include/asm/mshyperv.h

   359	
   360	#else /* CONFIG_HYPERV */
   361	static inline void hyperv_init(void) {}
   362	static inline bool hv_is_hyperv_initialized(void) { return false; }
   363	static inline void hyperv_cleanup(void) {}
   364	static inline void hyperv_setup_mmu_ops(void) {}
   365	static inline void set_hv_tscchange_cb(void (*cb)(void)) {}
   366	static inline void clear_hv_tscchange_cb(void) {}
   367	static inline void hyperv_stop_tsc_emulation(void) {};
   368	static inline struct hv_vp_assist_page *hv_get_vp_assist_page(unsigned int cpu)
   369	{
   370		return NULL;
   371	}
   372	static inline int hyperv_flush_guest_mapping(u64 as) { return -1; }
 > 373	void __init hv_init_spinlocks(void) {}
   374	#endif /* CONFIG_HYPERV */
   375	

---
0-DAY kernel test infrastructure                Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all                   Intel Corporation

[-- Attachment #2: .config.gz --]
[-- Type: application/gzip, Size: 31145 bytes --]

Re: [PATCH v2 2/2] locking/pvqspinlock, hv: Enable PV qspinlock for Hyper-V

[Yi Sun]

On 18-09-21 17:02:54, Michael Kelley (EOSG) wrote:
> From: Yi Sun <yi.y.sun@linux.intel.com> Sent: Friday, September 21, 2018 12:25 AM
> > +
> > +#define pr_fmt(fmt) "hv: " fmt
> 
> Other Hyper-V messages use "Hyper-V: " as the prefix, not "hv: ".  Take
> a quick look at 'dmesg' output for reference.
> 
Will modify this. Thanks!

> > +
> > +#include <linux/kernel_stat.h>
> > +#include <linux/spinlock.h>
> > +#include <linux/debugfs.h>
> > +#include <linux/log2.h>
> > +#include <linux/gfp.h>
> 
> Some of these #includes look like they might be leftovers from
> some other code.  Please check and see whether kernel_stat.h,
> debugsfs.h, log2.h, and gfp.h are actually needed.
> 
Sure, I will check them.

> > +static void hv_qlock_wait(u8 *byte, u8 val)
> > +{
> > +	unsigned long msr_val;
> > +
> > +	if (READ_ONCE(*byte) != val)
> > +		return;
> > +
> > +	/*
> > +	 * Read HV_X64_MSR_GUEST_IDLE MSR can trigger the guest's
> > +	 * transition to the idle power state which can be exited
> > +	 * by an IPI even if IF flag is disabled.
> > +	 */
> > +	if (ms_hyperv.features & HV_X64_MSR_GUEST_IDLE_AVAILABLE)
> 
> I can't see a case where this test is actually needed.  hv_qlock_wait()
> can only get called if the flag is set when hv_init_spinlocks() is run, and
> the flag value doesn't change after it is set.
> 
Yes, it is redundant. Will remove it.

> > +		rdmsrl(HV_X64_MSR_GUEST_IDLE, msr_val);
> > +}
> 
> Michael

Re: [PATCH v2 2/2] locking/pvqspinlock, hv: Enable PV qspinlock for Hyper-V

[kbuild test robot]

[-- Attachment #1: Type: text/plain, Size: 884 bytes --]

Hi Yi,

Thank you for the patch! Yet something to improve:

[auto build test ERROR on tip/x86/core]
[also build test ERROR on v4.19-rc5 next-20180925]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system]

url:    https://github.com/0day-ci/linux/commits/Yi-Sun/Enable-PV-qspinlock-for-Hyper-V/20180921-234519
config: i386-allyesconfig (attached as .config)
compiler: gcc-7 (Debian 7.3.0-1) 7.3.0
reproduce:
        # save the attached .config to linux build tree
        make ARCH=i386 

All errors (new ones prefixed by >>):

   arch/x86/kernel/cpu/mshyperv.o: In function `hv_smp_prepare_boot_cpu':
>> mshyperv.c:(.init.text+0x15): undefined reference to `hv_init_spinlocks'

---
0-DAY kernel test infrastructure                Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all                   Intel Corporation

[-- Attachment #2: .config.gz --]
[-- Type: application/gzip, Size: 64465 bytes --]

Re: [PATCH v2 2/2] locking/pvqspinlock, hv: Enable PV qspinlock for Hyper-V

[Thomas Gleixner]

On Fri, 21 Sep 2018, Yi Sun wrote:
>  static inline int hyperv_flush_guest_mapping(u64 as) { return -1; }
> +void __init hv_init_spinlocks(void) {}

That stub want's to be static inline void hv_....

But as the code which uses it is ifdef guarded anyway, you don't need it at
all. We add stub functions to avoid te ifdeffery in the code so the
compiler can optimize them out, but if the ifdef guards are required for
other reasons, then providing the stub for nothing is a pointless exercise.

Thanks,

	tglx