[PATCH v10 0/7] Enable Linux guests on Hyper-V on ARM64

[PATCH v10 0/7] Enable Linux guests on Hyper-V on ARM64

[Michael Kelley]

This series enables Linux guests running on Hyper-V on ARM64
hardware. New ARM64-specific code in arch/arm64/hyperv initializes
Hyper-V, including its interrupts and hypercall mechanism.
Existing architecture independent drivers for Hyper-V's VMbus and
synthetic devices just work when built for ARM64. Hyper-V code is
built and included in the image and modules only if CONFIG_HYPERV
is enabled.

The seven patches are organized as follows:

1) Fix some minor places where x86-dependencies crept into
   architecture independent Hyper-V code

2) Add definitions and functions for making Hyper-V hypercalls
   and getting/setting virtual processor registers provided by
   Hyper-V

3) Add architecture specific definitions needed by the
   architecture independent Hyper-V clocksource driver in
   drivers/clocksource/hyperv_timer.c. Update the clocksource
   driver to be initialized on ARM64.

4) Add functions needed by the arch independent VMbus driver
   for reporting a panic to Hyper-V and as stubs for the kexec
   and crash handlers.

5) Add Hyper-V initialization code and utility functions that
   report Hyper-v status.

6) Export screen_info so it may be used by the Hyper-V frame buffer
   driver built as a module. It is already exported for x86,
   powerpc, and alpha architectures.

7) Make CONFIG_HYPERV selectable on ARM64 in addition to x86/x64.

Hyper-V on ARM64 runs with a 4 Kbyte page size, but allows guests
with 4K/16K/64K page size. Linux guests with this patch series
work with all three supported ARM64 page sizes.

The Hyper-V vPCI driver at drivers/pci/host/pci-hyperv.c has
x86/x64-specific code and is not being built for ARM64. Fixing
this driver to enable vPCI devices on ARM64 will be done later.
The Hyper-V balloon driver also recently picked up some x86
dependencies that will be fixed separately, so it currently
doesn't build for ARM64 and CONFIG_HYPERV_BALLOON must be "N".

In a few cases, terminology from the x86/x64 world has been carried
over into the ARM64 code ("MSR", "TSC").  Hyper-V still uses the
x86/x64 terminology and has not replaced it with something more
generic, so the code uses the Hyper-V terminology.  This will be
fixed when Hyper-V updates the usage in the TLFS.

This patch set is based on the 5.13-rc1 code tree, plus a patch
from Sudeep Holla that implements SMCCC v1.2 HVC calls:
https://lore.kernel.org/linux-arm-kernel/20210505093843.3308691-2-sudeep.holla@arm.com/

Changes in v10:
* Drop previous Patch 1 for SMCCC v1.2 HVC calls in favor of
  Sudeep Holla's patch [Mark Rutland]
* Use new helper functions in 5.13 for checking Hyper-V hypercall
  return status
* Set up per-CPU hypercall argument page that is now used by the
  Hyper-V balloon driver
* Rebase on 5.13-RC1

Changes in v9:
* Added Patch 1 to enable making an SMCCC compliant hypercall
  that returns results in other than registers X0 thru X3, per
  version 1.2 and later of the SMCCC spec.
* Using the ability to return results in registers X6 and X7,
  converted hv_get_vpreg_128() to use a "fast" hypercall that
  passes inputs and outputs in registers, and in doing so eliminated
  a lot of memory allocation complexity.
* Cleaned up some extra blank lines and use of spaces in aligning
  local variables. [Sunil Muthuswamy]
* Based on discussion about future directions, reverted the
  population of hv_vp_index array to use a cpuhp state instead
  of a hypercall, which is like it was in v7 and earlier.

Changes in v8:
* Removed a lot of code based on refactoring the boundary between
  arch independent and arch dependent code for Hyper-V, per comments
  from Arnd Bergmann. The removed code was either duplicated on
  the x86 side, or has been folded into architecture independent
  code as not really being architecture dependent.
* Added config dependency on !CONFIG_CPU_BIG_ENDIAN [Arnd Bergmann]
* Reworked the approach to Hyper-V initialization. The functionality
  is the same, but is now structured like the Xen code with an early
  init function called in setup_arch() and an early initcall to
  finish the initialization. [Arnd Bergmann]

Changes in v7:
* Separately upstreamed split of hyperv-tlfs.h into arch dependent
  and independent versions.  In this patch set, update the ARM64
  hyperv-tlfs.h to include architecture independent definitions.
  This approach eliminates a lot of lines of otherwise duplicated
  code on the ARM64 side.
* Break ARM64 mshyperv.h into smaller pieces. Have an initial
  baseline, and add code along with patches for a particular
  functional area. [Marc Zyngier]
* In mshyperv.h, use static inline functions instead of #defines
  where possible. [Arnd Bergmann]
* Use VMbus INTID obtained from ACPI DSDT instead of hardcoding.
  The STIMER INTID is still hardcoded because it is needed
  before Linux has initialized the ACPI subsystem, so it can't
  be obtained from the DSDT.  Wedging it into the GTDT seems
  dubious, so was not done. [Marc Zyngier]
* Update Hyper-V page size allocation functions to use
  alloc_page() if PAGE_SIZE == HV_HYP_PAGE_SIZE [Arnd
  Bergmann]
* Various other minor changes based on feedback and to rebase
  to latest linux-next [Marc Zyngier and Arnd Bergmann]

Changes in v6:
* Use SMCCC hypercall interface instead of direct invocation
  of HVC instruction and the Hyper-V hypercall interface
  [Marc Zyngier]
* Reimplemented functions to alloc/free Hyper-V size pages
  using kmalloc/kfree since kmalloc now guarantees alignment of
  power of 2 size allocations [Marc Zyngier]
* Export screen_info in arm64 architecture so it can be used
  by the Hyper-V buffer driver built as a module
* Renamed source file arch/arm64/hyperv/hv_init.c to hv_core.c
  to better reflect its content
* Fixed the bit position of certain feature flags presented by
  Hyper-V to the guest.  The bit positions on ARM64 don't match
  the position on x86 like originally thought.
* Minor fixups to rebase to 5.6-rc5 linux-next

Changes in v5:
* Minor fixups to rebase to 5.4-rc1 linux-next

Changes in v4:
* Moved clock-related code into an architecture independent
  Hyper-V clocksource driver that is already upstream. Clock
  related code is removed from this patch set except for the
  ARM64 specific interrupt handler. [Marc Zyngier]
* Separately upstreamed the split of mshyperv.h into arch independent
  and arch dependent portions. The arch independent portion has been
  removed from this patch set.
* Divided patch #2 of the series into multiple smaller patches
  [Marc Zyngier]
* Changed a dozen or so smaller things based on feedback
  [Marc Zyngier, Will Deacon]
* Added functions to alloc/free Hyper-V size pages for use by
  drivers for Hyper-V synthetic devices when updated to not assume
  guest page size and Hyper-v page size are the same

Changes in v3:
* Added initialization of hv_vp_index array like was recently
  added on x86 branch [KY Srinivasan]
* Changed Hyper-V ARM64 register symbols to be all uppercase 
  instead of mixed case [KY Srinivasan]
* Separated mshyperv.h into two files, one architecture
  independent and one architecture dependent. After this code
  is upstream, will make changes to the x86 code to use the
  architecture independent file and remove duplication. And
  once we have a multi-architecture Hyper-V TLFS, will do a
  separate patch to split hyperv-tlfs.h in the same way.
  [KY Srinivasan]
* Minor tweaks to rebase to latest linux-next code

Changes in v2:
* Removed patch to implement slow_virt_to_phys() on ARM64.
  Use of slow_virt_to_phys() in arch independent Hyper-V
  drivers has been eliminated by commit 6ba34171bcbd
  ("Drivers: hv: vmbus: Remove use of slow_virt_to_phys()")
* Minor tweaks to rebase to latest linux-next code

Michael Kelley (7):
  asm-generic: hyperv: Fix incorrect architecture dependencies
  arm64: hyperv: Add Hyper-V hypercall and register access utilities
  arm64: hyperv: Add Hyper-V clocksource/clockevent support
  arm64: hyperv: Add kexec and panic handlers
  arm64: hyperv: Initialize hypervisor on boot
  arm64: efi: Export screen_info
  Drivers: hv: Enable Hyper-V code to be built on ARM64

 MAINTAINERS                          |   3 +
 arch/arm64/Kbuild                    |   1 +
 arch/arm64/hyperv/Makefile           |   2 +
 arch/arm64/hyperv/hv_core.c          | 182 +++++++++++++++++++++++++++++++
 arch/arm64/hyperv/mshyperv.c         | 206 +++++++++++++++++++++++++++++++++++
 arch/arm64/include/asm/hyperv-tlfs.h |  69 ++++++++++++
 arch/arm64/include/asm/mshyperv.h    |  72 ++++++++++++
 arch/arm64/kernel/efi.c              |   1 +
 arch/arm64/kernel/setup.c            |   4 +
 arch/x86/include/asm/mshyperv.h      |   3 -
 drivers/clocksource/hyperv_timer.c   |  14 +++
 drivers/hv/Kconfig                   |   3 +-
 include/asm-generic/mshyperv.h       |   5 +
 13 files changed, 561 insertions(+), 4 deletions(-)
 create mode 100644 arch/arm64/hyperv/Makefile
 create mode 100644 arch/arm64/hyperv/hv_core.c
 create mode 100644 arch/arm64/hyperv/mshyperv.c
 create mode 100644 arch/arm64/include/asm/hyperv-tlfs.h
 create mode 100644 arch/arm64/include/asm/mshyperv.h

-- 
1.8.3.1

[PATCH v10 1/7] asm-generic: hyperv: Fix incorrect architecture dependencies

[Michael Kelley]

Move the declaration of hv_root_partition and hyperv_pcpu_input_arg
from the x86-specific mshyperv.h to the arch independent mshyperv.h
since they are used by arch independent code.  While here, add a
missing #include needed to compile correctly on ARM64.

Signed-off-by: Michael Kelley <mikelley@microsoft.com>
---
 arch/x86/include/asm/mshyperv.h | 3 ---
 include/asm-generic/mshyperv.h  | 5 +++++
 2 files changed, 5 insertions(+), 3 deletions(-)

diff --git a/arch/x86/include/asm/mshyperv.h b/arch/x86/include/asm/mshyperv.h
index 67ff0d6..45c48b0 100644
--- a/arch/x86/include/asm/mshyperv.h
+++ b/arch/x86/include/asm/mshyperv.h
@@ -36,7 +36,6 @@ static inline u64 hv_get_register(unsigned int reg)
 extern int hyperv_init_cpuhp;
 
 extern void *hv_hypercall_pg;
-extern void  __percpu  **hyperv_pcpu_input_arg;
 extern void  __percpu  **hyperv_pcpu_output_arg;
 
 extern u64 hv_current_partition_id;
@@ -170,8 +169,6 @@ int hyperv_fill_flush_guest_mapping_list(
 		struct hv_guest_mapping_flush_list *flush,
 		u64 start_gfn, u64 end_gfn);
 
-extern bool hv_root_partition;
-
 #ifdef CONFIG_X86_64
 void hv_apic_init(void);
 void __init hv_init_spinlocks(void);
diff --git a/include/asm-generic/mshyperv.h b/include/asm-generic/mshyperv.h
index 9a000ba..22c92b8 100644
--- a/include/asm-generic/mshyperv.h
+++ b/include/asm-generic/mshyperv.h
@@ -22,6 +22,7 @@
 #include <linux/atomic.h>
 #include <linux/bitops.h>
 #include <linux/cpumask.h>
+#include <linux/nmi.h>
 #include <asm/ptrace.h>
 #include <asm/hyperv-tlfs.h>
 
@@ -151,6 +152,8 @@ static inline void vmbus_signal_eom(struct hv_message *msg, u32 old_msg_type)
 extern int vmbus_interrupt;
 extern int vmbus_irq;
 
+extern bool hv_root_partition;
+
 #if IS_ENABLED(CONFIG_HYPERV)
 /*
  * Hypervisor's notion of virtual processor ID is different from
@@ -161,6 +164,8 @@ static inline void vmbus_signal_eom(struct hv_message *msg, u32 old_msg_type)
 extern u32 *hv_vp_index;
 extern u32 hv_max_vp_index;
 
+extern void  __percpu  **hyperv_pcpu_input_arg;
+
 /* Sentinel value for an uninitialized entry in hv_vp_index array */
 #define VP_INVAL	U32_MAX
 
-- 
1.8.3.1

[PATCH v10 2/7] arm64: hyperv: Add Hyper-V hypercall and register access utilities

[Michael Kelley]

hyperv-tlfs.h defines Hyper-V interfaces from the Hyper-V Top Level
Functional Spec (TLFS), and #includes the architecture-independent
part of hyperv-tlfs.h in include/asm-generic.  The published TLFS
is distinctly oriented to x86/x64, so the ARM64-specific
hyperv-tlfs.h includes information for ARM64 that is not yet formally
published. The TLFS is available here:

  docs.microsoft.com/en-us/virtualization/hyper-v-on-windows/reference/tlfs

mshyperv.h defines Linux-specific structures and routines for
interacting with Hyper-V on ARM64, and #includes the architecture-
independent part of mshyperv.h in include/asm-generic.

Use these definitions to provide utility functions to make
Hyper-V hypercalls and to get and set Hyper-V provided
registers associated with a virtual processor.

Signed-off-by: Michael Kelley <mikelley@microsoft.com>
Reviewed-by: Sunil Muthuswamy <sunilmut@microsoft.com>
---
 MAINTAINERS                          |   3 +
 arch/arm64/Kbuild                    |   1 +
 arch/arm64/hyperv/Makefile           |   2 +
 arch/arm64/hyperv/hv_core.c          | 130 +++++++++++++++++++++++++++++++++++
 arch/arm64/include/asm/hyperv-tlfs.h |  69 +++++++++++++++++++
 arch/arm64/include/asm/mshyperv.h    |  54 +++++++++++++++
 6 files changed, 259 insertions(+)
 create mode 100644 arch/arm64/hyperv/Makefile
 create mode 100644 arch/arm64/hyperv/hv_core.c
 create mode 100644 arch/arm64/include/asm/hyperv-tlfs.h
 create mode 100644 arch/arm64/include/asm/mshyperv.h

diff --git a/MAINTAINERS b/MAINTAINERS
index bd7aff0c..6c650cb 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -8437,6 +8437,9 @@ T:	git git://git.kernel.org/pub/scm/linux/kernel/git/hyperv/linux.git
 F:	Documentation/ABI/stable/sysfs-bus-vmbus
 F:	Documentation/ABI/testing/debugfs-hyperv
 F:	Documentation/networking/device_drivers/ethernet/microsoft/netvsc.rst
+F:	arch/arm64/hyperv
+F:	arch/arm64/include/asm/hyperv-tlfs.h
+F:	arch/arm64/include/asm/mshyperv.h
 F:	arch/x86/hyperv
 F:	arch/x86/include/asm/hyperv-tlfs.h
 F:	arch/x86/include/asm/mshyperv.h
diff --git a/arch/arm64/Kbuild b/arch/arm64/Kbuild
index d646582..7a37608 100644
--- a/arch/arm64/Kbuild
+++ b/arch/arm64/Kbuild
@@ -3,4 +3,5 @@ obj-y			+= kernel/ mm/
 obj-$(CONFIG_NET)	+= net/
 obj-$(CONFIG_KVM)	+= kvm/
 obj-$(CONFIG_XEN)	+= xen/
+obj-$(subst m,y,$(CONFIG_HYPERV))	+= hyperv/
 obj-$(CONFIG_CRYPTO)	+= crypto/
diff --git a/arch/arm64/hyperv/Makefile b/arch/arm64/hyperv/Makefile
new file mode 100644
index 0000000..1697d30
--- /dev/null
+++ b/arch/arm64/hyperv/Makefile
@@ -0,0 +1,2 @@
+# SPDX-License-Identifier: GPL-2.0
+obj-y		:= hv_core.o
diff --git a/arch/arm64/hyperv/hv_core.c b/arch/arm64/hyperv/hv_core.c
new file mode 100644
index 0000000..34004a5
--- /dev/null
+++ b/arch/arm64/hyperv/hv_core.c
@@ -0,0 +1,130 @@
+// SPDX-License-Identifier: GPL-2.0
+
+/*
+ * Low level utility routines for interacting with Hyper-V.
+ *
+ * Copyright (C) 2021, Microsoft, Inc.
+ *
+ * Author : Michael Kelley <mikelley@microsoft.com>
+ */
+
+#include <linux/types.h>
+#include <linux/export.h>
+#include <linux/mm.h>
+#include <linux/hyperv.h>
+#include <linux/arm-smccc.h>
+#include <linux/module.h>
+#include <asm-generic/bug.h>
+#include <asm/hyperv-tlfs.h>
+#include <asm/mshyperv.h>
+
+/*
+ * hv_do_hypercall- Invoke the specified hypercall
+ */
+u64 hv_do_hypercall(u64 control, void *input, void *output)
+{
+	struct arm_smccc_res	res;
+	u64			input_address;
+	u64			output_address;
+
+	input_address = input ? virt_to_phys(input) : 0;
+	output_address = output ? virt_to_phys(output) : 0;
+
+	arm_smccc_1_1_hvc(HV_FUNC_ID, control,
+			  input_address, output_address, &res);
+	return res.a0;
+}
+EXPORT_SYMBOL_GPL(hv_do_hypercall);
+
+/*
+ * hv_do_fast_hypercall8 -- Invoke the specified hypercall
+ * with arguments in registers instead of physical memory.
+ * Avoids the overhead of virt_to_phys for simple hypercalls.
+ */
+
+u64 hv_do_fast_hypercall8(u16 code, u64 input)
+{
+	struct arm_smccc_res	res;
+	u64			control;
+
+	control = (u64)code | HV_HYPERCALL_FAST_BIT;
+
+	arm_smccc_1_1_hvc(HV_FUNC_ID, control, input, &res);
+	return res.a0;
+}
+EXPORT_SYMBOL_GPL(hv_do_fast_hypercall8);
+
+/*
+ * Set a single VP register to a 64-bit value.
+ */
+void hv_set_vpreg(u32 msr, u64 value)
+{
+	struct arm_smccc_res res;
+
+	arm_smccc_1_1_hvc(
+		HV_FUNC_ID,
+		HVCALL_SET_VP_REGISTERS | HV_HYPERCALL_FAST_BIT |
+			HV_HYPERCALL_REP_COMP_1,
+		HV_PARTITION_ID_SELF,
+		HV_VP_INDEX_SELF,
+		msr,
+		0,
+		value,
+		0,
+		&res);
+
+	/*
+	 * Something is fundamentally broken in the hypervisor if
+	 * setting a VP register fails. There's really no way to
+	 * continue as a guest VM, so panic.
+	 */
+	BUG_ON(!hv_result_success(res.a0));
+}
+EXPORT_SYMBOL_GPL(hv_set_vpreg);
+
+/*
+ * Get the value of a single VP register.  One version
+ * returns just 64 bits and another returns the full 128 bits.
+ * The two versions are separate to avoid complicating the
+ * calling sequence for the more frequently used 64 bit version.
+ */
+
+void hv_get_vpreg_128(u32 msr, struct hv_get_vp_registers_output *result)
+{
+	struct arm_smccc_1_2_regs args;
+	struct arm_smccc_1_2_regs res;
+
+	args.a0 = HV_FUNC_ID;
+	args.a1 = HVCALL_GET_VP_REGISTERS | HV_HYPERCALL_FAST_BIT |
+			HV_HYPERCALL_REP_COMP_1;
+	args.a2 = HV_PARTITION_ID_SELF;
+	args.a3 = HV_VP_INDEX_SELF;
+	args.a4 = msr;
+
+	/*
+	 * Use the SMCCC 1.2 interface because the results are in registers
+	 * beyond X0-X3.
+	 */
+	arm_smccc_1_2_hvc(&args, &res);
+
+	/*
+	 * Something is fundamentally broken in the hypervisor if
+	 * getting a VP register fails. There's really no way to
+	 * continue as a guest VM, so panic.
+	 */
+	BUG_ON(!hv_result_success(res.a0));
+
+	result->as64.low = res.a6;
+	result->as64.high = res.a7;
+}
+EXPORT_SYMBOL_GPL(hv_get_vpreg_128);
+
+u64 hv_get_vpreg(u32 msr)
+{
+	struct hv_get_vp_registers_output output;
+
+	hv_get_vpreg_128(msr, &output);
+
+	return output.as64.low;
+}
+EXPORT_SYMBOL_GPL(hv_get_vpreg);
diff --git a/arch/arm64/include/asm/hyperv-tlfs.h b/arch/arm64/include/asm/hyperv-tlfs.h
new file mode 100644
index 0000000..4d964a7
--- /dev/null
+++ b/arch/arm64/include/asm/hyperv-tlfs.h
@@ -0,0 +1,69 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+
+/*
+ * This file contains definitions from the Hyper-V Hypervisor Top-Level
+ * Functional Specification (TLFS):
+ * https://docs.microsoft.com/en-us/virtualization/hyper-v-on-windows/reference/tlfs
+ *
+ * Copyright (C) 2021, Microsoft, Inc.
+ *
+ * Author : Michael Kelley <mikelley@microsoft.com>
+ */
+
+#ifndef _ASM_HYPERV_TLFS_H
+#define _ASM_HYPERV_TLFS_H
+
+#include <linux/types.h>
+
+/*
+ * All data structures defined in the TLFS that are shared between Hyper-V
+ * and a guest VM use Little Endian byte ordering.  This matches the default
+ * byte ordering of Linux running on ARM64, so no special handling is required.
+ */
+
+/*
+ * These Hyper-V registers provide information equivalent to the CPUID
+ * instruction on x86/x64.
+ */
+#define HV_REGISTER_HYPERVISOR_VERSION		0x00000100 /*CPUID 0x40000002 */
+#define HV_REGISTER_FEATURES			0x00000200 /*CPUID 0x40000003 */
+#define HV_REGISTER_ENLIGHTENMENTS		0x00000201 /*CPUID 0x40000004 */
+
+/*
+ * Group C Features. See the asm-generic version of hyperv-tlfs.h
+ * for a description of Feature Groups.
+ */
+
+/* Crash MSRs available */
+#define HV_FEATURE_GUEST_CRASH_MSR_AVAILABLE	BIT(8)
+
+/* STIMER direct mode is available */
+#define HV_STIMER_DIRECT_MODE_AVAILABLE		BIT(13)
+
+/*
+ * Synthetic register definitions equivalent to MSRs on x86/x64
+ */
+#define HV_REGISTER_CRASH_P0		0x00000210
+#define HV_REGISTER_CRASH_P1		0x00000211
+#define HV_REGISTER_CRASH_P2		0x00000212
+#define HV_REGISTER_CRASH_P3		0x00000213
+#define HV_REGISTER_CRASH_P4		0x00000214
+#define HV_REGISTER_CRASH_CTL		0x00000215
+
+#define HV_REGISTER_GUEST_OSID		0x00090002
+#define HV_REGISTER_VP_INDEX		0x00090003
+#define HV_REGISTER_TIME_REF_COUNT	0x00090004
+#define HV_REGISTER_REFERENCE_TSC	0x00090017
+
+#define HV_REGISTER_SINT0		0x000A0000
+#define HV_REGISTER_SCONTROL		0x000A0010
+#define HV_REGISTER_SIEFP		0x000A0012
+#define HV_REGISTER_SIMP		0x000A0013
+#define HV_REGISTER_EOM			0x000A0014
+
+#define HV_REGISTER_STIMER0_CONFIG	0x000B0000
+#define HV_REGISTER_STIMER0_COUNT	0x000B0001
+
+#include <asm-generic/hyperv-tlfs.h>
+
+#endif
diff --git a/arch/arm64/include/asm/mshyperv.h b/arch/arm64/include/asm/mshyperv.h
new file mode 100644
index 0000000..c448704
--- /dev/null
+++ b/arch/arm64/include/asm/mshyperv.h
@@ -0,0 +1,54 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+
+/*
+ * Linux-specific definitions for managing interactions with Microsoft's
+ * Hyper-V hypervisor. The definitions in this file are specific to
+ * the ARM64 architecture.  See include/asm-generic/mshyperv.h for
+ * definitions are that architecture independent.
+ *
+ * Definitions that are specified in the Hyper-V Top Level Functional
+ * Spec (TLFS) should not go in this file, but should instead go in
+ * hyperv-tlfs.h.
+ *
+ * Copyright (C) 2021, Microsoft, Inc.
+ *
+ * Author : Michael Kelley <mikelley@microsoft.com>
+ */
+
+#ifndef _ASM_MSHYPERV_H
+#define _ASM_MSHYPERV_H
+
+#include <linux/types.h>
+#include <linux/arm-smccc.h>
+#include <asm/hyperv-tlfs.h>
+
+/*
+ * Declare calls to get and set Hyper-V VP register values on ARM64, which
+ * requires a hypercall.
+ */
+
+extern void hv_set_vpreg(u32 reg, u64 value);
+extern u64 hv_get_vpreg(u32 reg);
+extern void hv_get_vpreg_128(u32 reg, struct hv_get_vp_registers_output *result);
+
+static inline void hv_set_register(unsigned int reg, u64 value)
+{
+	hv_set_vpreg(reg, value);
+}
+
+static inline u64 hv_get_register(unsigned int reg)
+{
+	return hv_get_vpreg(reg);
+}
+
+/* SMCCC hypercall parameters */
+#define HV_SMCCC_FUNC_NUMBER	1
+#define HV_FUNC_ID	ARM_SMCCC_CALL_VAL(			\
+				ARM_SMCCC_STD_CALL,		\
+				ARM_SMCCC_SMC_64,		\
+				ARM_SMCCC_OWNER_VENDOR_HYP,	\
+				HV_SMCCC_FUNC_NUMBER)
+
+#include <asm-generic/mshyperv.h>
+
+#endif
-- 
1.8.3.1

[PATCH v10 3/7] arm64: hyperv: Add Hyper-V clocksource/clockevent support

[Michael Kelley]

Add architecture specific definitions and functions needed
by the architecture independent Hyper-V clocksource driver.
Update the Hyper-V clocksource driver to be initialized
on ARM64.

Signed-off-by: Michael Kelley <mikelley@microsoft.com>
Reviewed-by: Sunil Muthuswamy <sunilmut@microsoft.com>
---
 arch/arm64/include/asm/mshyperv.h  | 12 ++++++++++++
 drivers/clocksource/hyperv_timer.c | 14 ++++++++++++++
 2 files changed, 26 insertions(+)

diff --git a/arch/arm64/include/asm/mshyperv.h b/arch/arm64/include/asm/mshyperv.h
index c448704..b17299c 100644
--- a/arch/arm64/include/asm/mshyperv.h
+++ b/arch/arm64/include/asm/mshyperv.h
@@ -21,6 +21,7 @@
 #include <linux/types.h>
 #include <linux/arm-smccc.h>
 #include <asm/hyperv-tlfs.h>
+#include <clocksource/arm_arch_timer.h>
 
 /*
  * Declare calls to get and set Hyper-V VP register values on ARM64, which
@@ -41,6 +42,17 @@ static inline u64 hv_get_register(unsigned int reg)
 	return hv_get_vpreg(reg);
 }
 
+/* Define the interrupt ID used by STIMER0 Direct Mode interrupts. This
+ * value can't come from ACPI tables because it is needed before the
+ * Linux ACPI subsystem is initialized.
+ */
+#define HYPERV_STIMER0_VECTOR	31
+
+static inline u64 hv_get_raw_timer(void)
+{
+	return arch_timer_read_counter();
+}
+
 /* SMCCC hypercall parameters */
 #define HV_SMCCC_FUNC_NUMBER	1
 #define HV_FUNC_ID	ARM_SMCCC_CALL_VAL(			\
diff --git a/drivers/clocksource/hyperv_timer.c b/drivers/clocksource/hyperv_timer.c
index 977fd05..270ad9c 100644
--- a/drivers/clocksource/hyperv_timer.c
+++ b/drivers/clocksource/hyperv_timer.c
@@ -569,3 +569,17 @@ void __init hv_init_clocksource(void)
 	hv_setup_sched_clock(read_hv_sched_clock_msr);
 }
 EXPORT_SYMBOL_GPL(hv_init_clocksource);
+
+/* Initialize everything on ARM64 */
+static int __init hyperv_timer_init(struct acpi_table_header *table)
+{
+	if (!hv_is_hyperv_initialized())
+		return -EINVAL;
+
+	hv_init_clocksource();
+	if (hv_stimer_alloc(true))
+		return -EINVAL;
+
+	return 0;
+}
+TIMER_ACPI_DECLARE(hyperv, ACPI_SIG_GTDT, hyperv_timer_init);
-- 
1.8.3.1

[PATCH v10 4/7] arm64: hyperv: Add kexec and panic handlers

[Michael Kelley]

Add function to inform Hyper-V about a guest panic.

Also add functions to set up and remove kexec and panic
handlers, which are currently unused on ARM64 but are
called from architecture independent code in the VMbus
driver.

This code is built only when CONFIG_HYPERV is enabled.

Signed-off-by: Michael Kelley <mikelley@microsoft.com>
Reviewed-by: Sunil Muthuswamy <sunilmut@microsoft.com>
---
 arch/arm64/hyperv/Makefile   |  2 +-
 arch/arm64/hyperv/hv_core.c  | 52 ++++++++++++++++++++++++++++++++++++++++++
 arch/arm64/hyperv/mshyperv.c | 54 ++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 107 insertions(+), 1 deletion(-)
 create mode 100644 arch/arm64/hyperv/mshyperv.c

diff --git a/arch/arm64/hyperv/Makefile b/arch/arm64/hyperv/Makefile
index 1697d30..87c31c0 100644
--- a/arch/arm64/hyperv/Makefile
+++ b/arch/arm64/hyperv/Makefile
@@ -1,2 +1,2 @@
 # SPDX-License-Identifier: GPL-2.0
-obj-y		:= hv_core.o
+obj-y		:= hv_core.o mshyperv.o
diff --git a/arch/arm64/hyperv/hv_core.c b/arch/arm64/hyperv/hv_core.c
index 34004a5..56391f3 100644
--- a/arch/arm64/hyperv/hv_core.c
+++ b/arch/arm64/hyperv/hv_core.c
@@ -128,3 +128,55 @@ u64 hv_get_vpreg(u32 msr)
 	return output.as64.low;
 }
 EXPORT_SYMBOL_GPL(hv_get_vpreg);
+
+/*
+ * hyperv_report_panic - report a panic to Hyper-V.  This function uses
+ * the older version of the Hyper-V interface that admittedly doesn't
+ * pass enough information to be useful beyond just recording the
+ * occurrence of a panic. The parallel hv_kmsg_dump() uses the
+ * new interface that allows reporting 4 Kbytes of data, which is much
+ * more useful. Hyper-V on ARM64 always supports the newer interface, but
+ * we retain support for the older version because the sysadmin is allowed
+ * to disable the newer version via sysctl in case of information security
+ * concerns about the more verbose version.
+ */
+void hyperv_report_panic(struct pt_regs *regs, long err, bool in_die)
+{
+	static bool	panic_reported;
+	u64		guest_id;
+
+	/* Don't report a panic to Hyper-V if we're not going to panic */
+	if (in_die && !panic_on_oops)
+		return;
+
+	/*
+	 * We prefer to report panic on 'die' chain as we have proper
+	 * registers to report, but if we miss it (e.g. on BUG()) we need
+	 * to report it on 'panic'.
+	 *
+	 * Calling code in the 'die' and 'panic' paths ensures that only
+	 * one CPU is running this code, so no atomicity is needed.
+	 */
+	if (panic_reported)
+		return;
+	panic_reported = true;
+
+	guest_id = hv_get_vpreg(HV_REGISTER_GUEST_OSID);
+
+	/*
+	 * Hyper-V provides the ability to store only 5 values.
+	 * Pick the passed in error value, the guest_id, and the PC.
+	 * The first two general registers are added arbitrarily.
+	 */
+	hv_set_vpreg(HV_REGISTER_CRASH_P0, err);
+	hv_set_vpreg(HV_REGISTER_CRASH_P1, guest_id);
+	hv_set_vpreg(HV_REGISTER_CRASH_P2, regs->pc);
+	hv_set_vpreg(HV_REGISTER_CRASH_P3, regs->regs[0]);
+	hv_set_vpreg(HV_REGISTER_CRASH_P4, regs->regs[1]);
+
+	/*
+	 * Let Hyper-V know there is crash data available
+	 */
+	hv_set_vpreg(HV_REGISTER_CRASH_CTL, HV_CRASH_CTL_CRASH_NOTIFY);
+}
+EXPORT_SYMBOL_GPL(hyperv_report_panic);
diff --git a/arch/arm64/hyperv/mshyperv.c b/arch/arm64/hyperv/mshyperv.c
new file mode 100644
index 0000000..d202b4c
--- /dev/null
+++ b/arch/arm64/hyperv/mshyperv.c
@@ -0,0 +1,54 @@
+// SPDX-License-Identifier: GPL-2.0
+
+/*
+ * Core routines for interacting with Microsoft's Hyper-V hypervisor.
+ * Includes hypervisor initialization, and handling of crashes and
+ * kexecs through a set of static "handler" variables set by the
+ * architecture independent VMbus driver.
+ *
+ * Copyright (C) 2021, Microsoft, Inc.
+ *
+ * Author : Michael Kelley <mikelley@microsoft.com>
+ */
+
+#include <linux/types.h>
+#include <linux/export.h>
+#include <linux/ptrace.h>
+
+/*
+ * The VMbus handler functions are no-ops on ARM64 because
+ * VMbus interrupts are handled as percpu IRQs.
+ */
+void hv_setup_vmbus_handler(void (*handler)(void))
+{
+}
+EXPORT_SYMBOL_GPL(hv_setup_vmbus_handler);
+
+void hv_remove_vmbus_handler(void)
+{
+}
+EXPORT_SYMBOL_GPL(hv_remove_vmbus_handler);
+
+/*
+ * The kexec and crash handler functions are
+ * currently no-ops on ARM64.
+ */
+void hv_setup_kexec_handler(void (*handler)(void))
+{
+}
+EXPORT_SYMBOL_GPL(hv_setup_kexec_handler);
+
+void hv_remove_kexec_handler(void)
+{
+}
+EXPORT_SYMBOL_GPL(hv_remove_kexec_handler);
+
+void hv_setup_crash_handler(void (*handler)(struct pt_regs *regs))
+{
+}
+EXPORT_SYMBOL_GPL(hv_setup_crash_handler);
+
+void hv_remove_crash_handler(void)
+{
+}
+EXPORT_SYMBOL_GPL(hv_remove_crash_handler);
-- 
1.8.3.1

[PATCH v10 5/7] arm64: hyperv: Initialize hypervisor on boot

[Michael Kelley]

Add ARM64-specific code to initialize the Hyper-V
hypervisor when booting as a guest VM. Provide functions
and data structures indicating hypervisor status that
are needed by VMbus driver.

This code is built only when CONFIG_HYPERV is enabled.

Signed-off-by: Michael Kelley <mikelley@microsoft.com>
---
 arch/arm64/hyperv/mshyperv.c      | 152 ++++++++++++++++++++++++++++++++++++++
 arch/arm64/include/asm/mshyperv.h |   6 ++
 arch/arm64/kernel/setup.c         |   4 +
 3 files changed, 162 insertions(+)

diff --git a/arch/arm64/hyperv/mshyperv.c b/arch/arm64/hyperv/mshyperv.c
index d202b4c..95f7e4e 100644
--- a/arch/arm64/hyperv/mshyperv.c
+++ b/arch/arm64/hyperv/mshyperv.c
@@ -14,6 +14,158 @@
 #include <linux/types.h>
 #include <linux/export.h>
 #include <linux/ptrace.h>
+#include <linux/errno.h>
+#include <linux/acpi.h>
+#include <linux/version.h>
+#include <linux/cpuhotplug.h>
+#include <linux/slab.h>
+#include <linux/cpumask.h>
+#include <asm/mshyperv.h>
+
+static bool		hyperv_initialized;
+struct ms_hyperv_info	ms_hyperv __ro_after_init;
+EXPORT_SYMBOL_GPL(ms_hyperv);
+
+bool	hv_root_partition;
+EXPORT_SYMBOL_GPL(hv_root_partition);
+
+u32	*hv_vp_index;
+EXPORT_SYMBOL_GPL(hv_vp_index);
+
+u32	hv_max_vp_index;
+EXPORT_SYMBOL_GPL(hv_max_vp_index);
+
+void __percpu **hyperv_pcpu_input_arg;
+EXPORT_SYMBOL_GPL(hyperv_pcpu_input_arg);
+
+static int hv_cpu_init(unsigned int cpu)
+{
+	void **input_arg = (void **)this_cpu_ptr(hyperv_pcpu_input_arg);
+
+	hv_vp_index[cpu] = hv_get_vpreg(HV_REGISTER_VP_INDEX);
+
+	*input_arg = kmalloc(HV_HYP_PAGE_SIZE, GFP_KERNEL);
+	if (!(*input_arg))
+		return -ENOMEM;
+
+	return 0;
+}
+
+void __init hyperv_early_init(void)
+{
+	struct hv_get_vp_registers_output	result;
+	u32	a, b, c, d;
+	u64	guest_id;
+
+	/*
+	 * If we're in a VM on Hyper-V, the ACPI hypervisor_id field will
+	 * have the string "MsHyperV".
+	 */
+	if (strncmp((char *)&acpi_gbl_FADT.hypervisor_id, "MsHyperV", 8))
+		return;
+
+	/* Setup the guest ID */
+	guest_id = generate_guest_id(0, LINUX_VERSION_CODE, 0);
+	hv_set_vpreg(HV_REGISTER_GUEST_OSID, guest_id);
+
+	/* Get the features and hints from Hyper-V */
+	hv_get_vpreg_128(HV_REGISTER_FEATURES, &result);
+	ms_hyperv.features = result.as32.a;
+	ms_hyperv.priv_high = result.as32.b;
+	ms_hyperv.misc_features = result.as32.c;
+
+	hv_get_vpreg_128(HV_REGISTER_ENLIGHTENMENTS, &result);
+	ms_hyperv.hints = result.as32.a;
+
+	pr_info("Hyper-V: privilege flags low 0x%x, high 0x%x, hints 0x%x, misc 0x%x\n",
+		ms_hyperv.features, ms_hyperv.priv_high, ms_hyperv.hints,
+		ms_hyperv.misc_features);
+
+	/*
+	 * If Hyper-V has crash notifications, set crash_kexec_post_notifiers
+	 * so that we will report the panic to Hyper-V before running kdump.
+	 */
+	if (ms_hyperv.misc_features & HV_FEATURE_GUEST_CRASH_MSR_AVAILABLE)
+		crash_kexec_post_notifiers = true;
+
+	/* Get information about the Hyper-V host version */
+	hv_get_vpreg_128(HV_REGISTER_HYPERVISOR_VERSION, &result);
+	a = result.as32.a;
+	b = result.as32.b;
+	c = result.as32.c;
+	d = result.as32.d;
+	pr_info("Hyper-V: Host Build %d.%d.%d.%d-%d-%d\n",
+		b >> 16, b & 0xFFFF, a,	d & 0xFFFFFF, c, d >> 24);
+
+	hyperv_initialized = true;
+}
+
+static int __init hyperv_init(void)
+{
+	int	i, ret;
+
+	hyperv_pcpu_input_arg = alloc_percpu(void  *);
+	if (!hyperv_pcpu_input_arg)
+		return -ENOMEM;
+
+	/* Allocate and initialize percpu VP index array */
+	hv_max_vp_index = num_possible_cpus();
+	hv_vp_index = kmalloc_array(hv_max_vp_index, sizeof(*hv_vp_index),
+				    GFP_KERNEL);
+	if (!hv_vp_index) {
+		ret = -ENOMEM;
+		goto free_input_arg;
+	}
+
+	for (i = 0; i < hv_max_vp_index; i++)
+		hv_vp_index[i] = VP_INVAL;
+
+	if (cpuhp_setup_state(CPUHP_AP_ONLINE_DYN, "arm64/hyperv_init:online",
+					hv_cpu_init, NULL) < 0) {
+		ret = -EINVAL;
+		goto free_vp_index;
+	}
+
+	return 0;
+
+free_vp_index:
+	kfree(hv_vp_index);
+	hv_vp_index = NULL;
+
+free_input_arg:
+	hv_max_vp_index = 0;
+	free_percpu(hyperv_pcpu_input_arg);
+	hyperv_pcpu_input_arg = NULL;
+	return ret;
+}
+
+early_initcall(hyperv_init);
+
+/* This routine is called before kexec/kdump. It does required cleanup. */
+void hyperv_cleanup(void)
+{
+	hv_set_vpreg(HV_REGISTER_GUEST_OSID, 0);
+
+}
+EXPORT_SYMBOL_GPL(hyperv_cleanup);
+
+bool hv_is_hyperv_initialized(void)
+{
+	return hyperv_initialized;
+}
+EXPORT_SYMBOL_GPL(hv_is_hyperv_initialized);
+
+bool hv_is_hibernation_supported(void)
+{
+	return false;
+}
+EXPORT_SYMBOL_GPL(hv_is_hibernation_supported);
+
+bool hv_is_isolation_supported(void)
+{
+	return false;
+}
+EXPORT_SYMBOL_GPL(hv_is_isolation_supported);
 
 /*
  * The VMbus handler functions are no-ops on ARM64 because
diff --git a/arch/arm64/include/asm/mshyperv.h b/arch/arm64/include/asm/mshyperv.h
index b17299c..86ca5c5 100644
--- a/arch/arm64/include/asm/mshyperv.h
+++ b/arch/arm64/include/asm/mshyperv.h
@@ -23,6 +23,12 @@
 #include <asm/hyperv-tlfs.h>
 #include <clocksource/arm_arch_timer.h>
 
+#if IS_ENABLED(CONFIG_HYPERV)
+void __init hyperv_early_init(void);
+#else
+static inline void hyperv_early_init(void) {};
+#endif
+
 /*
  * Declare calls to get and set Hyper-V VP register values on ARM64, which
  * requires a hypercall.
diff --git a/arch/arm64/kernel/setup.c b/arch/arm64/kernel/setup.c
index 61845c0..7b17d6a 100644
--- a/arch/arm64/kernel/setup.c
+++ b/arch/arm64/kernel/setup.c
@@ -49,6 +49,7 @@
 #include <asm/traps.h>
 #include <asm/efi.h>
 #include <asm/xen/hypervisor.h>
+#include <asm/mshyperv.h>
 #include <asm/mmu_context.h>
 
 static int num_standard_resources;
@@ -355,6 +356,9 @@ void __init __no_sanitize_address setup_arch(char **cmdline_p)
 	if (acpi_disabled)
 		unflatten_device_tree();
 
+	/* Do after acpi_boot_table_init() so local FADT is available */
+	hyperv_early_init();
+
 	bootmem_init();
 
 	kasan_init();
-- 
1.8.3.1

[PATCH v10 6/7] arm64: efi: Export screen_info

[Michael Kelley]

The Hyper-V frame buffer driver may be built as a module, and
it needs access to screen_info. So export screen_info.

Signed-off-by: Michael Kelley <mikelley@microsoft.com>
Acked-by: Ard Biesheuvel <ardb@kernel.org>
---
 arch/arm64/kernel/efi.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/arch/arm64/kernel/efi.c b/arch/arm64/kernel/efi.c
index fa02efb..e1be6c4 100644
--- a/arch/arm64/kernel/efi.c
+++ b/arch/arm64/kernel/efi.c
@@ -55,6 +55,7 @@ static __init pteval_t create_mapping_protection(efi_memory_desc_t *md)
 
 /* we will fill this structure from the stub, so don't put it in .bss */
 struct screen_info screen_info __section(".data");
+EXPORT_SYMBOL(screen_info);
 
 int __init efi_create_mapping(struct mm_struct *mm, efi_memory_desc_t *md)
 {
-- 
1.8.3.1

[PATCH v10 7/7] Drivers: hv: Enable Hyper-V code to be built on ARM64

[Michael Kelley]

Update drivers/hv/Kconfig so CONFIG_HYPERV can be selected on
ARM64, causing the Hyper-V specific code to be built.

Signed-off-by: Michael Kelley <mikelley@microsoft.com>
Reviewed-by: Sunil Muthuswamy <sunilmut@microsoft.com>
---
 drivers/hv/Kconfig | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/drivers/hv/Kconfig b/drivers/hv/Kconfig
index 66c794d..efb7585 100644
--- a/drivers/hv/Kconfig
+++ b/drivers/hv/Kconfig
@@ -4,7 +4,8 @@ menu "Microsoft Hyper-V guest support"
 
 config HYPERV
 	tristate "Microsoft Hyper-V client drivers"
-	depends on X86 && ACPI && X86_LOCAL_APIC && HYPERVISOR_GUEST
+	depends on ACPI && ((X86 && X86_LOCAL_APIC && HYPERVISOR_GUEST) \
+		|| (ARM64 && !CPU_BIG_ENDIAN))
 	select PARAVIRT
 	select X86_HV_CALLBACK_VECTOR
 	help
-- 
1.8.3.1

Re: [PATCH v10 0/7] Enable Linux guests on Hyper-V on ARM64

[Sudeep Holla]

Hi Will,

On Wed, May 12, 2021 at 10:37:40AM -0700, Michael Kelley wrote:
>
> This patch set is based on the 5.13-rc1 code tree, plus a patch
> from Sudeep Holla that implements SMCCC v1.2 HVC calls:
> https://lore.kernel.org/linux-arm-kernel/20210505093843.3308691-2-sudeep.holla@arm.com/
>

Assuming that you will be handling v5.14, I am asking you. I plan to post
the above mention patch that implements SMCCC v1.2 SMC/HVC independent
of my FFA series with small change as suggested by Mark R.

Irrespective of readiness of this series or FFA, is it possible to pull
the patch and share a branch based on v5.13-rc1(Arm SoC requirement), so
that if FFA is ready I can pull the branch along with FFA patches and send
it to Arm SoC. You can do the same for this series if it gets ready.
I am just trying to avoid last minute confusion. Let me know if you have
any alternative suggestions.

--
Regards,
Sudeep

Re: [PATCH v10 5/7] arm64: hyperv: Initialize hypervisor on boot

[Wei Liu]

On Wed, May 12, 2021 at 10:37:45AM -0700, Michael Kelley wrote:
> Add ARM64-specific code to initialize the Hyper-V
> hypervisor when booting as a guest VM. Provide functions
> and data structures indicating hypervisor status that
> are needed by VMbus driver.
> 
> This code is built only when CONFIG_HYPERV is enabled.
> 
> Signed-off-by: Michael Kelley <mikelley@microsoft.com>
[...]
>  /*
>   * Declare calls to get and set Hyper-V VP register values on ARM64, which
>   * requires a hypercall.
> diff --git a/arch/arm64/kernel/setup.c b/arch/arm64/kernel/setup.c
> index 61845c0..7b17d6a 100644
> --- a/arch/arm64/kernel/setup.c
> +++ b/arch/arm64/kernel/setup.c
> @@ -49,6 +49,7 @@
>  #include <asm/traps.h>
>  #include <asm/efi.h>
>  #include <asm/xen/hypervisor.h>
> +#include <asm/mshyperv.h>
>  #include <asm/mmu_context.h>
>  
>  static int num_standard_resources;
> @@ -355,6 +356,9 @@ void __init __no_sanitize_address setup_arch(char **cmdline_p)
>  	if (acpi_disabled)
>  		unflatten_device_tree();
>  
> +	/* Do after acpi_boot_table_init() so local FADT is available */
> +	hyperv_early_init();
> +

Arm maintainers, this requires your attention. Thanks.

The rest is Hyper-V specific, feel free to skip that portion.

Wei.

>  	bootmem_init();
>  
>  	kasan_init();
> -- 
> 1.8.3.1
> 

Re: [PATCH v10 3/7] arm64: hyperv: Add Hyper-V clocksource/clockevent support

[Mark Rutland]

Hi Michael,

On Wed, May 12, 2021 at 10:37:43AM -0700, Michael Kelley wrote:
> Add architecture specific definitions and functions needed
> by the architecture independent Hyper-V clocksource driver.
> Update the Hyper-V clocksource driver to be initialized
> on ARM64.

Previously we've said that for a clocksource we must use the architected
counter, since that's necessary for things like the VDSO to work
correctly and efficiently.

Given that, I'm a bit confused that we're registering a per-cpu
clocksource that is in part based on the architected counter. Likewise,
I don't entirely follow why it's necessary to PV the clock_event_device.

Are the architected counter and timer reliable without this PV
infrastructure? Why do we need to PV either of those?

Thanks,
Mark.

> 
> Signed-off-by: Michael Kelley <mikelley@microsoft.com>
> Reviewed-by: Sunil Muthuswamy <sunilmut@microsoft.com>
> ---
>  arch/arm64/include/asm/mshyperv.h  | 12 ++++++++++++
>  drivers/clocksource/hyperv_timer.c | 14 ++++++++++++++
>  2 files changed, 26 insertions(+)
> 
> diff --git a/arch/arm64/include/asm/mshyperv.h b/arch/arm64/include/asm/mshyperv.h
> index c448704..b17299c 100644
> --- a/arch/arm64/include/asm/mshyperv.h
> +++ b/arch/arm64/include/asm/mshyperv.h
> @@ -21,6 +21,7 @@
>  #include <linux/types.h>
>  #include <linux/arm-smccc.h>
>  #include <asm/hyperv-tlfs.h>
> +#include <clocksource/arm_arch_timer.h>
>  
>  /*
>   * Declare calls to get and set Hyper-V VP register values on ARM64, which
> @@ -41,6 +42,17 @@ static inline u64 hv_get_register(unsigned int reg)
>  	return hv_get_vpreg(reg);
>  }
>  
> +/* Define the interrupt ID used by STIMER0 Direct Mode interrupts. This
> + * value can't come from ACPI tables because it is needed before the
> + * Linux ACPI subsystem is initialized.
> + */
> +#define HYPERV_STIMER0_VECTOR	31
> +
> +static inline u64 hv_get_raw_timer(void)
> +{
> +	return arch_timer_read_counter();
> +}
> +
>  /* SMCCC hypercall parameters */
>  #define HV_SMCCC_FUNC_NUMBER	1
>  #define HV_FUNC_ID	ARM_SMCCC_CALL_VAL(			\
> diff --git a/drivers/clocksource/hyperv_timer.c b/drivers/clocksource/hyperv_timer.c
> index 977fd05..270ad9c 100644
> --- a/drivers/clocksource/hyperv_timer.c
> +++ b/drivers/clocksource/hyperv_timer.c
> @@ -569,3 +569,17 @@ void __init hv_init_clocksource(void)
>  	hv_setup_sched_clock(read_hv_sched_clock_msr);
>  }
>  EXPORT_SYMBOL_GPL(hv_init_clocksource);
> +
> +/* Initialize everything on ARM64 */
> +static int __init hyperv_timer_init(struct acpi_table_header *table)
> +{
> +	if (!hv_is_hyperv_initialized())
> +		return -EINVAL;
> +
> +	hv_init_clocksource();
> +	if (hv_stimer_alloc(true))
> +		return -EINVAL;
> +
> +	return 0;
> +}
> +TIMER_ACPI_DECLARE(hyperv, ACPI_SIG_GTDT, hyperv_timer_init);
> -- 
> 1.8.3.1
> 

Re: [PATCH v10 2/7] arm64: hyperv: Add Hyper-V hypercall and register access utilities

[Mark Rutland]

On Wed, May 12, 2021 at 10:37:42AM -0700, Michael Kelley wrote:
> hyperv-tlfs.h defines Hyper-V interfaces from the Hyper-V Top Level
> Functional Spec (TLFS), and #includes the architecture-independent
> part of hyperv-tlfs.h in include/asm-generic.  The published TLFS
> is distinctly oriented to x86/x64, so the ARM64-specific
> hyperv-tlfs.h includes information for ARM64 that is not yet formally
> published. The TLFS is available here:
> 
>   docs.microsoft.com/en-us/virtualization/hyper-v-on-windows/reference/tlfs
> 
> mshyperv.h defines Linux-specific structures and routines for
> interacting with Hyper-V on ARM64, and #includes the architecture-
> independent part of mshyperv.h in include/asm-generic.
> 
> Use these definitions to provide utility functions to make
> Hyper-V hypercalls and to get and set Hyper-V provided
> registers associated with a virtual processor.
> 
> Signed-off-by: Michael Kelley <mikelley@microsoft.com>
> Reviewed-by: Sunil Muthuswamy <sunilmut@microsoft.com>
> ---
>  MAINTAINERS                          |   3 +
>  arch/arm64/Kbuild                    |   1 +
>  arch/arm64/hyperv/Makefile           |   2 +
>  arch/arm64/hyperv/hv_core.c          | 130 +++++++++++++++++++++++++++++++++++
>  arch/arm64/include/asm/hyperv-tlfs.h |  69 +++++++++++++++++++
>  arch/arm64/include/asm/mshyperv.h    |  54 +++++++++++++++
>  6 files changed, 259 insertions(+)
>  create mode 100644 arch/arm64/hyperv/Makefile
>  create mode 100644 arch/arm64/hyperv/hv_core.c
>  create mode 100644 arch/arm64/include/asm/hyperv-tlfs.h
>  create mode 100644 arch/arm64/include/asm/mshyperv.h

> +/*
> + * hv_do_hypercall- Invoke the specified hypercall
> + */
> +u64 hv_do_hypercall(u64 control, void *input, void *output)
> +{
> +	struct arm_smccc_res	res;
> +	u64			input_address;
> +	u64			output_address;
> +
> +	input_address = input ? virt_to_phys(input) : 0;
> +	output_address = output ? virt_to_phys(output) : 0;

I may have asked this before, but are `input` and `output` always linear
map pointers, or can they ever be vmalloc pointers?

Otherwise, this looks fine to me.

Mark.

RE: [PATCH v10 2/7] arm64: hyperv: Add Hyper-V hypercall and register access utilities

[Michael Kelley]

From: Mark Rutland <mark.rutland@arm.com> Sent: Friday, May 14, 2021 5:53 AM
> 
> On Wed, May 12, 2021 at 10:37:42AM -0700, Michael Kelley wrote:
> > hyperv-tlfs.h defines Hyper-V interfaces from the Hyper-V Top Level
> > Functional Spec (TLFS), and #includes the architecture-independent
> > part of hyperv-tlfs.h in include/asm-generic.  The published TLFS
> > is distinctly oriented to x86/x64, so the ARM64-specific
> > hyperv-tlfs.h includes information for ARM64 that is not yet formally
> > published. The TLFS is available here:
> >
> >   docs.microsoft.com/en-us/virtualization/hyper-v-on-windows/reference/tlfs
> >
> > mshyperv.h defines Linux-specific structures and routines for
> > interacting with Hyper-V on ARM64, and #includes the architecture-
> > independent part of mshyperv.h in include/asm-generic.
> >
> > Use these definitions to provide utility functions to make
> > Hyper-V hypercalls and to get and set Hyper-V provided
> > registers associated with a virtual processor.
> >
> > Signed-off-by: Michael Kelley <mikelley@microsoft.com>
> > Reviewed-by: Sunil Muthuswamy <sunilmut@microsoft.com>
> > ---
> >  MAINTAINERS                          |   3 +
> >  arch/arm64/Kbuild                    |   1 +
> >  arch/arm64/hyperv/Makefile           |   2 +
> >  arch/arm64/hyperv/hv_core.c          | 130 +++++++++++++++++++++++++++++++++++
> >  arch/arm64/include/asm/hyperv-tlfs.h |  69 +++++++++++++++++++
> >  arch/arm64/include/asm/mshyperv.h    |  54 +++++++++++++++
> >  6 files changed, 259 insertions(+)
> >  create mode 100644 arch/arm64/hyperv/Makefile
> >  create mode 100644 arch/arm64/hyperv/hv_core.c
> >  create mode 100644 arch/arm64/include/asm/hyperv-tlfs.h
> >  create mode 100644 arch/arm64/include/asm/mshyperv.h
> 
> > +/*
> > + * hv_do_hypercall- Invoke the specified hypercall
> > + */
> > +u64 hv_do_hypercall(u64 control, void *input, void *output)
> > +{
> > +	struct arm_smccc_res	res;
> > +	u64			input_address;
> > +	u64			output_address;
> > +
> > +	input_address = input ? virt_to_phys(input) : 0;
> > +	output_address = output ? virt_to_phys(output) : 0;
> 
> I may have asked this before, but are `input` and `output` always linear
> map pointers, or can they ever be vmalloc pointers?
> 
> Otherwise, this looks fine to me.
> 

The caller must ensure that hypercall arguments are aligned to
4 Kbytes, and no larger than 4 Kbytes, since that's the page size
used by Hyper-V regardless of the guest page size.  A per-CPU
4 Kbyte memory area (hyperv_pcpu_input_arg) meeting these
requirements is pre-allocated that callers can use for this purpose.
On the x86 side there's also a similar hyperv_pcpu_output_arg,
but there are no use cases for it on ARM64.

Michael

RE: [PATCH v10 3/7] arm64: hyperv: Add Hyper-V clocksource/clockevent support

[Michael Kelley]

From: Mark Rutland <mark.rutland@arm.com> Sent: Friday, May 14, 2021 5:37 AM
> 
> Hi Michael,
> 
> On Wed, May 12, 2021 at 10:37:43AM -0700, Michael Kelley wrote:
> > Add architecture specific definitions and functions needed
> > by the architecture independent Hyper-V clocksource driver.
> > Update the Hyper-V clocksource driver to be initialized
> > on ARM64.
> 
> Previously we've said that for a clocksource we must use the architected
> counter, since that's necessary for things like the VDSO to work
> correctly and efficiently.
> 
> Given that, I'm a bit confused that we're registering a per-cpu
> clocksource that is in part based on the architected counter. Likewise,
> I don't entirely follow why it's necessary to PV the clock_event_device.
> 
> Are the architected counter and timer reliable without this PV
> infrastructure? Why do we need to PV either of those?
> 
> Thanks,
> Mark.
> 

For the clocksource, we have a requirement to live migrate VMs
between Hyper-V hosts running on hardware that may have different
arch counter frequencies (it's not conformant to the ARM v8.6 1 GHz
requirement).  The Hyper-V virtualization does scaling to handle the
frequency difference.  And yes, there's a tradeoff with vDSO not
working, though we have an out-of-tree vDSO implementation that
we can use when necessary.

For clockevents, the only timer interrupt that Hyper-V provides
in a guest VM is its virtualized "STIMER" interrupt.  There's no
virtualization of the ARM arch timer in the guest.

Michael

> >
> > Signed-off-by: Michael Kelley <mikelley@microsoft.com>
> > Reviewed-by: Sunil Muthuswamy <sunilmut@microsoft.com>
> > ---
> >  arch/arm64/include/asm/mshyperv.h  | 12 ++++++++++++
> >  drivers/clocksource/hyperv_timer.c | 14 ++++++++++++++
> >  2 files changed, 26 insertions(+)
> >
> > diff --git a/arch/arm64/include/asm/mshyperv.h b/arch/arm64/include/asm/mshyperv.h
> > index c448704..b17299c 100644
> > --- a/arch/arm64/include/asm/mshyperv.h
> > +++ b/arch/arm64/include/asm/mshyperv.h
> > @@ -21,6 +21,7 @@
> >  #include <linux/types.h>
> >  #include <linux/arm-smccc.h>
> >  #include <asm/hyperv-tlfs.h>
> > +#include <clocksource/arm_arch_timer.h>
> >
> >  /*
> >   * Declare calls to get and set Hyper-V VP register values on ARM64, which
> > @@ -41,6 +42,17 @@ static inline u64 hv_get_register(unsigned int reg)
> >  	return hv_get_vpreg(reg);
> >  }
> >
> > +/* Define the interrupt ID used by STIMER0 Direct Mode interrupts. This
> > + * value can't come from ACPI tables because it is needed before the
> > + * Linux ACPI subsystem is initialized.
> > + */
> > +#define HYPERV_STIMER0_VECTOR	31
> > +
> > +static inline u64 hv_get_raw_timer(void)
> > +{
> > +	return arch_timer_read_counter();
> > +}
> > +
> >  /* SMCCC hypercall parameters */
> >  #define HV_SMCCC_FUNC_NUMBER	1
> >  #define HV_FUNC_ID	ARM_SMCCC_CALL_VAL(			\
> > diff --git a/drivers/clocksource/hyperv_timer.c b/drivers/clocksource/hyperv_timer.c
> > index 977fd05..270ad9c 100644
> > --- a/drivers/clocksource/hyperv_timer.c
> > +++ b/drivers/clocksource/hyperv_timer.c
> > @@ -569,3 +569,17 @@ void __init hv_init_clocksource(void)
> >  	hv_setup_sched_clock(read_hv_sched_clock_msr);
> >  }
> >  EXPORT_SYMBOL_GPL(hv_init_clocksource);
> > +
> > +/* Initialize everything on ARM64 */
> > +static int __init hyperv_timer_init(struct acpi_table_header *table)
> > +{
> > +	if (!hv_is_hyperv_initialized())
> > +		return -EINVAL;
> > +
> > +	hv_init_clocksource();
> > +	if (hv_stimer_alloc(true))
> > +		return -EINVAL;
> > +
> > +	return 0;
> > +}
> > +TIMER_ACPI_DECLARE(hyperv, ACPI_SIG_GTDT, hyperv_timer_init);
> > --
> > 1.8.3.1
> >

Re: [PATCH v10 2/7] arm64: hyperv: Add Hyper-V hypercall and register access utilities

[Mark Rutland]

On Fri, May 14, 2021 at 03:14:41PM +0000, Michael Kelley wrote:
> From: Mark Rutland <mark.rutland@arm.com> Sent: Friday, May 14, 2021 5:53 AM
> > 
> > On Wed, May 12, 2021 at 10:37:42AM -0700, Michael Kelley wrote:
> > > hyperv-tlfs.h defines Hyper-V interfaces from the Hyper-V Top Level
> > > Functional Spec (TLFS), and #includes the architecture-independent
> > > part of hyperv-tlfs.h in include/asm-generic.  The published TLFS
> > > is distinctly oriented to x86/x64, so the ARM64-specific
> > > hyperv-tlfs.h includes information for ARM64 that is not yet formally
> > > published. The TLFS is available here:
> > >
> > >   docs.microsoft.com/en-us/virtualization/hyper-v-on-windows/reference/tlfs
> > >
> > > mshyperv.h defines Linux-specific structures and routines for
> > > interacting with Hyper-V on ARM64, and #includes the architecture-
> > > independent part of mshyperv.h in include/asm-generic.
> > >
> > > Use these definitions to provide utility functions to make
> > > Hyper-V hypercalls and to get and set Hyper-V provided
> > > registers associated with a virtual processor.
> > >
> > > Signed-off-by: Michael Kelley <mikelley@microsoft.com>
> > > Reviewed-by: Sunil Muthuswamy <sunilmut@microsoft.com>
> > > ---
> > >  MAINTAINERS                          |   3 +
> > >  arch/arm64/Kbuild                    |   1 +
> > >  arch/arm64/hyperv/Makefile           |   2 +
> > >  arch/arm64/hyperv/hv_core.c          | 130 +++++++++++++++++++++++++++++++++++
> > >  arch/arm64/include/asm/hyperv-tlfs.h |  69 +++++++++++++++++++
> > >  arch/arm64/include/asm/mshyperv.h    |  54 +++++++++++++++
> > >  6 files changed, 259 insertions(+)
> > >  create mode 100644 arch/arm64/hyperv/Makefile
> > >  create mode 100644 arch/arm64/hyperv/hv_core.c
> > >  create mode 100644 arch/arm64/include/asm/hyperv-tlfs.h
> > >  create mode 100644 arch/arm64/include/asm/mshyperv.h
> > 
> > > +/*
> > > + * hv_do_hypercall- Invoke the specified hypercall
> > > + */
> > > +u64 hv_do_hypercall(u64 control, void *input, void *output)
> > > +{
> > > +	struct arm_smccc_res	res;
> > > +	u64			input_address;
> > > +	u64			output_address;
> > > +
> > > +	input_address = input ? virt_to_phys(input) : 0;
> > > +	output_address = output ? virt_to_phys(output) : 0;
> > 
> > I may have asked this before, but are `input` and `output` always linear
> > map pointers, or can they ever be vmalloc pointers?
> > 
> > Otherwise, this looks fine to me.
> 
> The caller must ensure that hypercall arguments are aligned to
> 4 Kbytes, and no larger than 4 Kbytes, since that's the page size
> used by Hyper-V regardless of the guest page size.  A per-CPU
> 4 Kbyte memory area (hyperv_pcpu_input_arg) meeting these
> requirements is pre-allocated that callers can use for this purpose.

What I was trying to find out was how that was allocated, as vmalloc()'d
pointers aren't legitimate to pass to virt_to_phys().

From scanning ahead to patch 5, I see that memory comes from kmalloc(),
and so it is legitimate to use virt_to_phys().


I see; and from patch 5 I see that memory come from kmalloc(), and will
therefore be part of the linear map, and so virt_to_phys() is
legitimate.

What I was asking here was how that memory was allocated. So long as
those are the only buffers used, this looks fine to me.

Thanks,
Mark.

Re: [PATCH v10 3/7] arm64: hyperv: Add Hyper-V clocksource/clockevent support

[Mark Rutland]

On Fri, May 14, 2021 at 03:35:15PM +0000, Michael Kelley wrote:
> From: Mark Rutland <mark.rutland@arm.com> Sent: Friday, May 14, 2021 5:37 AM
> > On Wed, May 12, 2021 at 10:37:43AM -0700, Michael Kelley wrote:
> > > Add architecture specific definitions and functions needed
> > > by the architecture independent Hyper-V clocksource driver.
> > > Update the Hyper-V clocksource driver to be initialized
> > > on ARM64.
> > 
> > Previously we've said that for a clocksource we must use the architected
> > counter, since that's necessary for things like the VDSO to work
> > correctly and efficiently.
> > 
> > Given that, I'm a bit confused that we're registering a per-cpu
> > clocksource that is in part based on the architected counter. Likewise,
> > I don't entirely follow why it's necessary to PV the clock_event_device.
> > 
> > Are the architected counter and timer reliable without this PV
> > infrastructure? Why do we need to PV either of those?
> > 
> > Thanks,
> > Mark.
> 
> For the clocksource, we have a requirement to live migrate VMs
> between Hyper-V hosts running on hardware that may have different
> arch counter frequencies (it's not conformant to the ARM v8.6 1 GHz
> requirement).  The Hyper-V virtualization does scaling to handle the
> frequency difference.  And yes, there's a tradeoff with vDSO not
> working, though we have an out-of-tree vDSO implementation that
> we can use when necessary.

Just to be clear, the vDSO is *one example* of something that won't
function correctly. More generally, because this undermines core
architectural guarantees, it requires more invasive changes (e.g. we'd
have to weaken the sanity checks, and not use the counter in things like
kexec paths), impacts any architectural features tied to the generic
timer/counter (e.g. the event stream, SPE and tracing, future features),
and means that other SW (e.g. bootloaders and other EFI applications)
are unlikley to function correctly in this environment.

I am very much not keen on trying to PV this.

What does the guest see when it reads CNTFRQ_EL0? Does this match the
real HW value (and can this change over time)? Or is this entirely
synthetic?

What do the ACPI tables look like in the guest? Is there a GTDT table at
all?

How does the counter event stream behave?

Are there other architectural features which Hyper-V does not implement
for a guest?

Is there anything else that may change across a migration? e.g. MIDR?
MPIDR? Any of the ID registers?

> For clockevents, the only timer interrupt that Hyper-V provides
> in a guest VM is its virtualized "STIMER" interrupt.  There's no
> virtualization of the ARM arch timer in the guest.

I think that is rather unfortunate, given it's a core architectural
feature. Is it just the interrupt that's missing? i.e. does all the
PE-local functionality behave as the architecture requires?

Thanks,
Mark.

> 
> > >
> > > Signed-off-by: Michael Kelley <mikelley@microsoft.com>
> > > Reviewed-by: Sunil Muthuswamy <sunilmut@microsoft.com>
> > > ---
> > >  arch/arm64/include/asm/mshyperv.h  | 12 ++++++++++++
> > >  drivers/clocksource/hyperv_timer.c | 14 ++++++++++++++
> > >  2 files changed, 26 insertions(+)
> > >
> > > diff --git a/arch/arm64/include/asm/mshyperv.h b/arch/arm64/include/asm/mshyperv.h
> > > index c448704..b17299c 100644
> > > --- a/arch/arm64/include/asm/mshyperv.h
> > > +++ b/arch/arm64/include/asm/mshyperv.h
> > > @@ -21,6 +21,7 @@
> > >  #include <linux/types.h>
> > >  #include <linux/arm-smccc.h>
> > >  #include <asm/hyperv-tlfs.h>
> > > +#include <clocksource/arm_arch_timer.h>
> > >
> > >  /*
> > >   * Declare calls to get and set Hyper-V VP register values on ARM64, which
> > > @@ -41,6 +42,17 @@ static inline u64 hv_get_register(unsigned int reg)
> > >  	return hv_get_vpreg(reg);
> > >  }
> > >
> > > +/* Define the interrupt ID used by STIMER0 Direct Mode interrupts. This
> > > + * value can't come from ACPI tables because it is needed before the
> > > + * Linux ACPI subsystem is initialized.
> > > + */
> > > +#define HYPERV_STIMER0_VECTOR	31
> > > +
> > > +static inline u64 hv_get_raw_timer(void)
> > > +{
> > > +	return arch_timer_read_counter();
> > > +}
> > > +
> > >  /* SMCCC hypercall parameters */
> > >  #define HV_SMCCC_FUNC_NUMBER	1
> > >  #define HV_FUNC_ID	ARM_SMCCC_CALL_VAL(			\
> > > diff --git a/drivers/clocksource/hyperv_timer.c b/drivers/clocksource/hyperv_timer.c
> > > index 977fd05..270ad9c 100644
> > > --- a/drivers/clocksource/hyperv_timer.c
> > > +++ b/drivers/clocksource/hyperv_timer.c
> > > @@ -569,3 +569,17 @@ void __init hv_init_clocksource(void)
> > >  	hv_setup_sched_clock(read_hv_sched_clock_msr);
> > >  }
> > >  EXPORT_SYMBOL_GPL(hv_init_clocksource);
> > > +
> > > +/* Initialize everything on ARM64 */
> > > +static int __init hyperv_timer_init(struct acpi_table_header *table)
> > > +{
> > > +	if (!hv_is_hyperv_initialized())
> > > +		return -EINVAL;
> > > +
> > > +	hv_init_clocksource();
> > > +	if (hv_stimer_alloc(true))
> > > +		return -EINVAL;
> > > +
> > > +	return 0;
> > > +}
> > > +TIMER_ACPI_DECLARE(hyperv, ACPI_SIG_GTDT, hyperv_timer_init);
> > > --
> > > 1.8.3.1
> > >

RE: [PATCH v10 2/7] arm64: hyperv: Add Hyper-V hypercall and register access utilities

[Michael Kelley]

From: Mark Rutland <mark.rutland@arm.com> Sent: Monday, May 17, 2021 4:45 AM
> 
> On Fri, May 14, 2021 at 03:14:41PM +0000, Michael Kelley wrote:
> > From: Mark Rutland <mark.rutland@arm.com> Sent: Friday, May 14, 2021 5:53 AM
> > >
> > > On Wed, May 12, 2021 at 10:37:42AM -0700, Michael Kelley wrote:
> > > > hyperv-tlfs.h defines Hyper-V interfaces from the Hyper-V Top Level
> > > > Functional Spec (TLFS), and #includes the architecture-independent
> > > > part of hyperv-tlfs.h in include/asm-generic.  The published TLFS
> > > > is distinctly oriented to x86/x64, so the ARM64-specific
> > > > hyperv-tlfs.h includes information for ARM64 that is not yet formally
> > > > published. The TLFS is available here:
> > > >
> > > >   docs.microsoft.com/en-us/virtualization/hyper-v-on-windows/reference/tlfs
> > > >
> > > > mshyperv.h defines Linux-specific structures and routines for
> > > > interacting with Hyper-V on ARM64, and #includes the architecture-
> > > > independent part of mshyperv.h in include/asm-generic.
> > > >
> > > > Use these definitions to provide utility functions to make
> > > > Hyper-V hypercalls and to get and set Hyper-V provided
> > > > registers associated with a virtual processor.
> > > >
> > > > Signed-off-by: Michael Kelley <mikelley@microsoft.com>
> > > > Reviewed-by: Sunil Muthuswamy <sunilmut@microsoft.com>
> > > > ---
> > > >  MAINTAINERS                          |   3 +
> > > >  arch/arm64/Kbuild                    |   1 +
> > > >  arch/arm64/hyperv/Makefile           |   2 +
> > > >  arch/arm64/hyperv/hv_core.c          | 130
> +++++++++++++++++++++++++++++++++++
> > > >  arch/arm64/include/asm/hyperv-tlfs.h |  69 +++++++++++++++++++
> > > >  arch/arm64/include/asm/mshyperv.h    |  54 +++++++++++++++
> > > >  6 files changed, 259 insertions(+)
> > > >  create mode 100644 arch/arm64/hyperv/Makefile
> > > >  create mode 100644 arch/arm64/hyperv/hv_core.c
> > > >  create mode 100644 arch/arm64/include/asm/hyperv-tlfs.h
> > > >  create mode 100644 arch/arm64/include/asm/mshyperv.h
> > >
> > > > +/*
> > > > + * hv_do_hypercall- Invoke the specified hypercall
> > > > + */
> > > > +u64 hv_do_hypercall(u64 control, void *input, void *output)
> > > > +{
> > > > +	struct arm_smccc_res	res;
> > > > +	u64			input_address;
> > > > +	u64			output_address;
> > > > +
> > > > +	input_address = input ? virt_to_phys(input) : 0;
> > > > +	output_address = output ? virt_to_phys(output) : 0;
> > >
> > > I may have asked this before, but are `input` and `output` always linear
> > > map pointers, or can they ever be vmalloc pointers?
> > >
> > > Otherwise, this looks fine to me.
> >
> > The caller must ensure that hypercall arguments are aligned to
> > 4 Kbytes, and no larger than 4 Kbytes, since that's the page size
> > used by Hyper-V regardless of the guest page size.  A per-CPU
> > 4 Kbyte memory area (hyperv_pcpu_input_arg) meeting these
> > requirements is pre-allocated that callers can use for this purpose.
> 
> What I was trying to find out was how that was allocated, as vmalloc()'d
> pointers aren't legitimate to pass to virt_to_phys().
> 
> From scanning ahead to patch 5, I see that memory comes from kmalloc(),
> and so it is legitimate to use virt_to_phys().
> 
> 
> I see; and from patch 5 I see that memory come from kmalloc(), and will
> therefore be part of the linear map, and so virt_to_phys() is
> legitimate.
> 
> What I was asking here was how that memory was allocated. So long as
> those are the only buffers used, this looks fine to me.
> 

There is no vmalloc() or stack local memory used as arguments to a
hypercall.   Call sites know about this requirement along with the
requirement of being aligned to 4 Kbytes and no larger than 4 Kbytes.

Michael

RE: [PATCH v10 3/7] arm64: hyperv: Add Hyper-V clocksource/clockevent support

[Michael Kelley]

From: Mark Rutland <mark.rutland@arm.com> Sent: Monday, May 17, 2021 6:08 AM
> 
> On Fri, May 14, 2021 at 03:35:15PM +0000, Michael Kelley wrote:
> > From: Mark Rutland <mark.rutland@arm.com> Sent: Friday, May 14, 2021 5:37 AM
> > > On Wed, May 12, 2021 at 10:37:43AM -0700, Michael Kelley wrote:
> > > > Add architecture specific definitions and functions needed
> > > > by the architecture independent Hyper-V clocksource driver.
> > > > Update the Hyper-V clocksource driver to be initialized
> > > > on ARM64.
> > >
> > > Previously we've said that for a clocksource we must use the architected
> > > counter, since that's necessary for things like the VDSO to work
> > > correctly and efficiently.
> > >
> > > Given that, I'm a bit confused that we're registering a per-cpu
> > > clocksource that is in part based on the architected counter. Likewise,
> > > I don't entirely follow why it's necessary to PV the clock_event_device.
> > >
> > > Are the architected counter and timer reliable without this PV
> > > infrastructure? Why do we need to PV either of those?
> > >
> > > Thanks,
> > > Mark.
> >
> > For the clocksource, we have a requirement to live migrate VMs
> > between Hyper-V hosts running on hardware that may have different
> > arch counter frequencies (it's not conformant to the ARM v8.6 1 GHz
> > requirement).  The Hyper-V virtualization does scaling to handle the
> > frequency difference.  And yes, there's a tradeoff with vDSO not
> > working, though we have an out-of-tree vDSO implementation that
> > we can use when necessary.
> 
> Just to be clear, the vDSO is *one example* of something that won't
> function correctly. More generally, because this undermines core
> architectural guarantees, it requires more invasive changes (e.g. we'd
> have to weaken the sanity checks, and not use the counter in things like
> kexec paths), impacts any architectural features tied to the generic
> timer/counter (e.g. the event stream, SPE and tracing, future features),
> and means that other SW (e.g. bootloaders and other EFI applications)
> are unlikley to function correctly in this environment.
> 
> I am very much not keen on trying to PV this.
> 
> What does the guest see when it reads CNTFRQ_EL0? Does this match the
> real HW value (and can this change over time)? Or is this entirely
> synthetic?
> 
> What do the ACPI tables look like in the guest? Is there a GTDT table at
> all?
> 
> How does the counter event stream behave?
> 
> Are there other architectural features which Hyper-V does not implement
> for a guest?
> 
> Is there anything else that may change across a migration? e.g. MIDR?
> MPIDR? Any of the ID registers?

The ARMv8 architectural system counter and associated registers are visible
and functional in a VM on Hyper-V.   The "arch_sys_counter" clocksource is
instantiated by the arm_arch_timer.c driver based on the GTDT in the guest,
and a Linux guest on Hyper-V runs fine with this clocksource.  Low level code
like bootloaders and EFI applications work normally.

The Hyper-V virtualization provides another Linux clocksource that is an
overlay on the arch counter and that provides time consistency across a live
migration. Live migration of ARM64 VMs on Hyper-V is not functional today,
but the Hyper-V team believes they can make it functional.  I have not
explored with them the live migration implications of things beyond time
consistency, like event streams, CNTFRQ_EL0, MIDR/MPIDR, etc.

Would a summary of your point be that live migration across hardware
with different arch counter frequencies is likely to not be possible with
100% fidelity because of these other dependencies on the arch counter
frequency?  (hence the fixed 1 GHz frequency in ARM v8.6)

> 
> > For clockevents, the only timer interrupt that Hyper-V provides
> > in a guest VM is its virtualized "STIMER" interrupt.  There's no
> > virtualization of the ARM arch timer in the guest.
> 
> I think that is rather unfortunate, given it's a core architectural
> feature. Is it just the interrupt that's missing? i.e. does all the
> PE-local functionality behave as the architecture requires?

Right off the bat, I don't know about timer-related PE-local
functionality as it's not exercised in a Linux VM on Hyper-V that is
using STIMER-based clockevents.  I'll explore with the Hyper-V
team.  My impression is that enabling the ARM arch timer in a
guest VM is more work for Hyper-V than just wiring up an
interrupt.

Michael

> 
> Thanks,
> Mark.
> 
> >
> > > >
> > > > Signed-off-by: Michael Kelley <mikelley@microsoft.com>
> > > > Reviewed-by: Sunil Muthuswamy <sunilmut@microsoft.com>
> > > > ---
> > > >  arch/arm64/include/asm/mshyperv.h  | 12 ++++++++++++
> > > >  drivers/clocksource/hyperv_timer.c | 14 ++++++++++++++
> > > >  2 files changed, 26 insertions(+)
> > > >
> > > > diff --git a/arch/arm64/include/asm/mshyperv.h
> b/arch/arm64/include/asm/mshyperv.h
> > > > index c448704..b17299c 100644
> > > > --- a/arch/arm64/include/asm/mshyperv.h
> > > > +++ b/arch/arm64/include/asm/mshyperv.h
> > > > @@ -21,6 +21,7 @@
> > > >  #include <linux/types.h>
> > > >  #include <linux/arm-smccc.h>
> > > >  #include <asm/hyperv-tlfs.h>
> > > > +#include <clocksource/arm_arch_timer.h>
> > > >
> > > >  /*
> > > >   * Declare calls to get and set Hyper-V VP register values on ARM64, which
> > > > @@ -41,6 +42,17 @@ static inline u64 hv_get_register(unsigned int reg)
> > > >  	return hv_get_vpreg(reg);
> > > >  }
> > > >
> > > > +/* Define the interrupt ID used by STIMER0 Direct Mode interrupts. This
> > > > + * value can't come from ACPI tables because it is needed before the
> > > > + * Linux ACPI subsystem is initialized.
> > > > + */
> > > > +#define HYPERV_STIMER0_VECTOR	31
> > > > +
> > > > +static inline u64 hv_get_raw_timer(void)
> > > > +{
> > > > +	return arch_timer_read_counter();
> > > > +}
> > > > +
> > > >  /* SMCCC hypercall parameters */
> > > >  #define HV_SMCCC_FUNC_NUMBER	1
> > > >  #define HV_FUNC_ID	ARM_SMCCC_CALL_VAL(			\
> > > > diff --git a/drivers/clocksource/hyperv_timer.c b/drivers/clocksource/hyperv_timer.c
> > > > index 977fd05..270ad9c 100644
> > > > --- a/drivers/clocksource/hyperv_timer.c
> > > > +++ b/drivers/clocksource/hyperv_timer.c
> > > > @@ -569,3 +569,17 @@ void __init hv_init_clocksource(void)
> > > >  	hv_setup_sched_clock(read_hv_sched_clock_msr);
> > > >  }
> > > >  EXPORT_SYMBOL_GPL(hv_init_clocksource);
> > > > +
> > > > +/* Initialize everything on ARM64 */
> > > > +static int __init hyperv_timer_init(struct acpi_table_header *table)
> > > > +{
> > > > +	if (!hv_is_hyperv_initialized())
> > > > +		return -EINVAL;
> > > > +
> > > > +	hv_init_clocksource();
> > > > +	if (hv_stimer_alloc(true))
> > > > +		return -EINVAL;
> > > > +
> > > > +	return 0;
> > > > +}
> > > > +TIMER_ACPI_DECLARE(hyperv, ACPI_SIG_GTDT, hyperv_timer_init);
> > > > --
> > > > 1.8.3.1
> > > >

Re: [PATCH v10 3/7] arm64: hyperv: Add Hyper-V clocksource/clockevent support

[Mark Rutland]

On Mon, May 17, 2021 at 05:27:49PM +0000, Michael Kelley wrote:
> From: Mark Rutland <mark.rutland@arm.com> Sent: Monday, May 17, 2021 6:08 AM
> > 
> > On Fri, May 14, 2021 at 03:35:15PM +0000, Michael Kelley wrote:
> > > From: Mark Rutland <mark.rutland@arm.com> Sent: Friday, May 14, 2021 5:37 AM
> > > > On Wed, May 12, 2021 at 10:37:43AM -0700, Michael Kelley wrote:
> > > > > Add architecture specific definitions and functions needed
> > > > > by the architecture independent Hyper-V clocksource driver.
> > > > > Update the Hyper-V clocksource driver to be initialized
> > > > > on ARM64.
> > > >
> > > > Previously we've said that for a clocksource we must use the architected
> > > > counter, since that's necessary for things like the VDSO to work
> > > > correctly and efficiently.
> > > >
> > > > Given that, I'm a bit confused that we're registering a per-cpu
> > > > clocksource that is in part based on the architected counter. Likewise,
> > > > I don't entirely follow why it's necessary to PV the clock_event_device.
> > > >
> > > > Are the architected counter and timer reliable without this PV
> > > > infrastructure? Why do we need to PV either of those?
> > > >
> > > > Thanks,
> > > > Mark.
> > >
> > > For the clocksource, we have a requirement to live migrate VMs
> > > between Hyper-V hosts running on hardware that may have different
> > > arch counter frequencies (it's not conformant to the ARM v8.6 1 GHz
> > > requirement).  The Hyper-V virtualization does scaling to handle the
> > > frequency difference.  And yes, there's a tradeoff with vDSO not
> > > working, though we have an out-of-tree vDSO implementation that
> > > we can use when necessary.
> > 
> > Just to be clear, the vDSO is *one example* of something that won't
> > function correctly. More generally, because this undermines core
> > architectural guarantees, it requires more invasive changes (e.g. we'd
> > have to weaken the sanity checks, and not use the counter in things like
> > kexec paths), impacts any architectural features tied to the generic
> > timer/counter (e.g. the event stream, SPE and tracing, future features),
> > and means that other SW (e.g. bootloaders and other EFI applications)
> > are unlikley to function correctly in this environment.
> > 
> > I am very much not keen on trying to PV this.
> > 
> > What does the guest see when it reads CNTFRQ_EL0? Does this match the
> > real HW value (and can this change over time)? Or is this entirely
> > synthetic?
> > 
> > What do the ACPI tables look like in the guest? Is there a GTDT table at
> > all?
> > 
> > How does the counter event stream behave?
> > 
> > Are there other architectural features which Hyper-V does not implement
> > for a guest?
> > 
> > Is there anything else that may change across a migration? e.g. MIDR?
> > MPIDR? Any of the ID registers?
> 
> The ARMv8 architectural system counter and associated registers are visible
> and functional in a VM on Hyper-V.   The "arch_sys_counter" clocksource is
> instantiated by the arm_arch_timer.c driver based on the GTDT in the guest,
> and a Linux guest on Hyper-V runs fine with this clocksource.  Low level code
> like bootloaders and EFI applications work normally.

That's good to hear!

One potential issue worth noting is that as those pieces of software are
unlikely to handle counter frequency changes reliably, and so may not
behave correctly if live-migrated.

> The Hyper-V virtualization provides another Linux clocksource that is an
> overlay on the arch counter and that provides time consistency across a live
> migration. Live migration of ARM64 VMs on Hyper-V is not functional today,
> but the Hyper-V team believes they can make it functional.  I have not
> explored with them the live migration implications of things beyond time
> consistency, like event streams, CNTFRQ_EL0, MIDR/MPIDR, etc.
> 
> Would a summary of your point be that live migration across hardware
> with different arch counter frequencies is likely to not be possible with
> 100% fidelity because of these other dependencies on the arch counter
> frequency?  (hence the fixed 1 GHz frequency in ARM v8.6)

Yes.

In addition, there are a larger set of things necessarily exposed to VMs
that mean that live migration isn't all that practical except betweenm
identical machines (where the counter frequency should be identical),
and the timer frequency might just be the canary in the coalmine. For
example, the cache properties enumerated in CTR_EL0 cannot necessarily
be emulated on another machine.

> > > For clockevents, the only timer interrupt that Hyper-V provides
> > > in a guest VM is its virtualized "STIMER" interrupt.  There's no
> > > virtualization of the ARM arch timer in the guest.
> > 
> > I think that is rather unfortunate, given it's a core architectural
> > feature. Is it just the interrupt that's missing? i.e. does all the
> > PE-local functionality behave as the architecture requires?
> 
> Right off the bat, I don't know about timer-related PE-local
> functionality as it's not exercised in a Linux VM on Hyper-V that is
> using STIMER-based clockevents.  I'll explore with the Hyper-V
> team.  My impression is that enabling the ARM arch timer in a
> guest VM is more work for Hyper-V than just wiring up an
> interrupt.

Thanks for chasing this up!

Mark.

RE: [PATCH v10 3/7] arm64: hyperv: Add Hyper-V clocksource/clockevent support

[Michael Kelley]

From: Mark Rutland <mark.rutland@arm.com> Sent: Tuesday, May 18, 2021 10:00 AM
> 
> On Mon, May 17, 2021 at 05:27:49PM +0000, Michael Kelley wrote:
> > From: Mark Rutland <mark.rutland@arm.com> Sent: Monday, May 17, 2021 6:08 AM
> > >
> > > On Fri, May 14, 2021 at 03:35:15PM +0000, Michael Kelley wrote:
> > > > From: Mark Rutland <mark.rutland@arm.com> Sent: Friday, May 14, 2021 5:37 AM
> > > > > On Wed, May 12, 2021 at 10:37:43AM -0700, Michael Kelley wrote:
> > > > > > Add architecture specific definitions and functions needed
> > > > > > by the architecture independent Hyper-V clocksource driver.
> > > > > > Update the Hyper-V clocksource driver to be initialized
> > > > > > on ARM64.
> > > > >
> > > > > Previously we've said that for a clocksource we must use the architected
> > > > > counter, since that's necessary for things like the VDSO to work
> > > > > correctly and efficiently.
> > > > >
> > > > > Given that, I'm a bit confused that we're registering a per-cpu
> > > > > clocksource that is in part based on the architected counter. Likewise,
> > > > > I don't entirely follow why it's necessary to PV the clock_event_device.
> > > > >
> > > > > Are the architected counter and timer reliable without this PV
> > > > > infrastructure? Why do we need to PV either of those?
> > > > >
> > > > > Thanks,
> > > > > Mark.
> > > >
> > > > For the clocksource, we have a requirement to live migrate VMs
> > > > between Hyper-V hosts running on hardware that may have different
> > > > arch counter frequencies (it's not conformant to the ARM v8.6 1 GHz
> > > > requirement).  The Hyper-V virtualization does scaling to handle the
> > > > frequency difference.  And yes, there's a tradeoff with vDSO not
> > > > working, though we have an out-of-tree vDSO implementation that
> > > > we can use when necessary.
> > >
> > > Just to be clear, the vDSO is *one example* of something that won't
> > > function correctly. More generally, because this undermines core
> > > architectural guarantees, it requires more invasive changes (e.g. we'd
> > > have to weaken the sanity checks, and not use the counter in things like
> > > kexec paths), impacts any architectural features tied to the generic
> > > timer/counter (e.g. the event stream, SPE and tracing, future features),
> > > and means that other SW (e.g. bootloaders and other EFI applications)
> > > are unlikley to function correctly in this environment.
> > >
> > > I am very much not keen on trying to PV this.
> > >
> > > What does the guest see when it reads CNTFRQ_EL0? Does this match the
> > > real HW value (and can this change over time)? Or is this entirely
> > > synthetic?
> > >
> > > What do the ACPI tables look like in the guest? Is there a GTDT table at
> > > all?
> > >
> > > How does the counter event stream behave?
> > >
> > > Are there other architectural features which Hyper-V does not implement
> > > for a guest?
> > >
> > > Is there anything else that may change across a migration? e.g. MIDR?
> > > MPIDR? Any of the ID registers?
> >
> > The ARMv8 architectural system counter and associated registers are visible
> > and functional in a VM on Hyper-V.   The "arch_sys_counter" clocksource is
> > instantiated by the arm_arch_timer.c driver based on the GTDT in the guest,
> > and a Linux guest on Hyper-V runs fine with this clocksource.  Low level code
> > like bootloaders and EFI applications work normally.
> 
> That's good to hear!
> 
> One potential issue worth noting is that as those pieces of software are
> unlikely to handle counter frequency changes reliably, and so may not
> behave correctly if live-migrated.
> 
> > The Hyper-V virtualization provides another Linux clocksource that is an
> > overlay on the arch counter and that provides time consistency across a live
> > migration. Live migration of ARM64 VMs on Hyper-V is not functional today,
> > but the Hyper-V team believes they can make it functional.  I have not
> > explored with them the live migration implications of things beyond time
> > consistency, like event streams, CNTFRQ_EL0, MIDR/MPIDR, etc.
> >
> > Would a summary of your point be that live migration across hardware
> > with different arch counter frequencies is likely to not be possible with
> > 100% fidelity because of these other dependencies on the arch counter
> > frequency?  (hence the fixed 1 GHz frequency in ARM v8.6)
> 
> Yes.
> 
> In addition, there are a larger set of things necessarily exposed to VMs
> that mean that live migration isn't all that practical except betweenm
> identical machines (where the counter frequency should be identical),
> and the timer frequency might just be the canary in the coalmine. For
> example, the cache properties enumerated in CTR_EL0 cannot necessarily
> be emulated on another machine.
> 
> > > > For clockevents, the only timer interrupt that Hyper-V provides
> > > > in a guest VM is its virtualized "STIMER" interrupt.  There's no
> > > > virtualization of the ARM arch timer in the guest.
> > >
> > > I think that is rather unfortunate, given it's a core architectural
> > > feature. Is it just the interrupt that's missing? i.e. does all the
> > > PE-local functionality behave as the architecture requires?
> >
> > Right off the bat, I don't know about timer-related PE-local
> > functionality as it's not exercised in a Linux VM on Hyper-V that is
> > using STIMER-based clockevents.  I'll explore with the Hyper-V
> > team.  My impression is that enabling the ARM arch timer in a
> > guest VM is more work for Hyper-V than just wiring up an
> > interrupt.
> 
> Thanks for chasing this up!
> 

I've had a couple rounds of discussions with the Hyper-V team.   For
the clocksource we've agreed to table the live migration discussion, and
I'll resubmit the code so that arm_arch_timer.c provides the
standard arch_sys_counter clocksource.  As noted previously, this just
works for a Hyper-V guest.  The live migration discussion may come
back later after a deeper investigation by Hyper-V.

For clockevents, there's not a near term fix.  It's more than just plumbing
an interrupt for Hyper-V to virtualize the ARM64 arch timer in a guest VM.
From their perspective there's also benefit in having a timer abstraction
that's independent of the architecture, and in the Linux guest, the STIMER
code is common across x86/x64 and ARM64.  It follows the standard Linux
clockevents model, as it should. The code is already in use in out-of-tree
builds in the Linux VMs included in Windows 10 on ARM64 as part of the
so-called "Windows Subsystem for Linux".

So I'm hoping we can get this core support for ARM64 guests on Hyper-V
into upstream using the existing STIMER support.  At some point, Hyper-V
will do the virtualization of the ARM64 arch timer, but we don't want to
have to stay out-of-tree until after that happens.

Thoughts?

Michael

Re: [PATCH v10 3/7] arm64: hyperv: Add Hyper-V clocksource/clockevent support

[Mark Rutland]

Hi Michael,

[trimming the bulk of the thrread]

On Tue, Jun 08, 2021 at 03:36:06PM +0000, Michael Kelley wrote:
> I've had a couple rounds of discussions with the Hyper-V team.   For
> the clocksource we've agreed to table the live migration discussion, and
> I'll resubmit the code so that arm_arch_timer.c provides the
> standard arch_sys_counter clocksource.  As noted previously, this just
> works for a Hyper-V guest.  The live migration discussion may come
> back later after a deeper investigation by Hyper-V.

Great; thanks for this!

> For clockevents, there's not a near term fix.  It's more than just plumbing
> an interrupt for Hyper-V to virtualize the ARM64 arch timer in a guest VM.
> From their perspective there's also benefit in having a timer abstraction
> that's independent of the architecture, and in the Linux guest, the STIMER
> code is common across x86/x64 and ARM64.  It follows the standard Linux
> clockevents model, as it should. The code is already in use in out-of-tree
> builds in the Linux VMs included in Windows 10 on ARM64 as part of the
> so-called "Windows Subsystem for Linux".
> 
> So I'm hoping we can get this core support for ARM64 guests on Hyper-V
> into upstream using the existing STIMER support.  At some point, Hyper-V
> will do the virtualization of the ARM64 arch timer, but we don't want to
> have to stay out-of-tree until after that happens.

My main concern here is making sure that we can rely on architected
properties, and don't have to special-case architected bits for hyperv
(or any other hypervisor), since that inevitably causes longer-term
pain.

While in abstract I'm not as worried about using the timer
clock_event_device specifically, that same driver provides the
clocksource and the event stream, and I want those to work as usual,
without being tied into the hyperv code. IIUC that will require some
work, since the driver won't register if the GTDT is missing timer
interrupts (or if there is no GTDT).

I think it really depends on what that looks like.

Thanks,
Mark.

RE: [PATCH v10 3/7] arm64: hyperv: Add Hyper-V clocksource/clockevent support

[Michael Kelley]

From: Mark Rutland <mark.rutland@arm.com> Sent: Thursday, June 10, 2021 9:45 AM
> 
> Hi Michael,
> 
> [trimming the bulk of the thrread]
> 
> On Tue, Jun 08, 2021 at 03:36:06PM +0000, Michael Kelley wrote:
> > I've had a couple rounds of discussions with the Hyper-V team.   For
> > the clocksource we've agreed to table the live migration discussion, and
> > I'll resubmit the code so that arm_arch_timer.c provides the
> > standard arch_sys_counter clocksource.  As noted previously, this just
> > works for a Hyper-V guest.  The live migration discussion may come
> > back later after a deeper investigation by Hyper-V.
> 
> Great; thanks for this!
> 
> > For clockevents, there's not a near term fix.  It's more than just plumbing
> > an interrupt for Hyper-V to virtualize the ARM64 arch timer in a guest VM.
> > From their perspective there's also benefit in having a timer abstraction
> > that's independent of the architecture, and in the Linux guest, the STIMER
> > code is common across x86/x64 and ARM64.  It follows the standard Linux
> > clockevents model, as it should. The code is already in use in out-of-tree
> > builds in the Linux VMs included in Windows 10 on ARM64 as part of the
> > so-called "Windows Subsystem for Linux".
> >
> > So I'm hoping we can get this core support for ARM64 guests on Hyper-V
> > into upstream using the existing STIMER support.  At some point, Hyper-V
> > will do the virtualization of the ARM64 arch timer, but we don't want to
> > have to stay out-of-tree until after that happens.
> 
> My main concern here is making sure that we can rely on architected
> properties, and don't have to special-case architected bits for hyperv
> (or any other hypervisor), since that inevitably causes longer-term
> pain.
> 
> While in abstract I'm not as worried about using the timer
> clock_event_device specifically, that same driver provides the
> clocksource and the event stream, and I want those to work as usual,
> without being tied into the hyperv code. IIUC that will require some
> work, since the driver won't register if the GTDT is missing timer
> interrupts (or if there is no GTDT).
> 
> I think it really depends on what that looks like.

Mark,

Here are the details:

The existing initialization and registration code in arm_arch_timer.c
works in a Hyper-V guest with no changes.  As previously mentioned,
the GTDT exists and is correctly populated.  Even though it isn't used,
there's a PPI INTID specified for the virtual timer, just so
the "arm_sys_timer" clockevent can be initialized and registered.
The IRQ shows up in the output of "cat /proc/interrupts" with zero counts
for all CPUs since no interrupts are ever generated.  The EL1 virtual
timer registers (CNTV_CVAL_EL0, CNTV_TVAL_EL0, and CNTV_CTL_EL0)
are accessible in the VM.  The "arm_sys_timer" clockevent is left in
a shutdown state with CNTV_CTL_EL0.ENABLE set to zero when the
Hyper-V STIMER clockevent is registered with a higher rating.

Event streams are initialized and the __delay() implementation
for ARM64 inside the kernel works.  However, on the Ampere
eMAG hardware I'm using for testing, the WFE instruction returns
more quickly than it should even though the event stream fields in
CNTKCTL_EL1 are correct.  I have a query in to the Hyper-V team 
to see if they are trapping WFE and just returning, vs. perhaps the
eMAG processor takes the easy way out and has WFE just return
immediately.  I'm not knowledgeable about other uses of timer
event streams, so let me know if there are other usage scenarios
I should check.

Finally, the "arch_sys_counter" clocksource gets initialized and
setup correctly.  If the Hyper-V clocksource is also initialized,
you can flip between the two clocksources at runtime as expected.
If the Hyper-V clocksource is not setup, then Linux in the VM runs
fine with the "arch_sys_counter" clocksource.

Michael

RE: [PATCH v10 3/7] arm64: hyperv: Add Hyper-V clocksource/clockevent support

[Michael Kelley]

From: Michael Kelley <mikelley@microsoft.com> Sent: Sunday, June 13, 2021 7:42 PM
> 
> From: Mark Rutland <mark.rutland@arm.com> Sent: Thursday, June 10, 2021 9:45 AM
> >
> > Hi Michael,
> >
> > [trimming the bulk of the thrread]
> >
> > On Tue, Jun 08, 2021 at 03:36:06PM +0000, Michael Kelley wrote:
> > > I've had a couple rounds of discussions with the Hyper-V team.   For
> > > the clocksource we've agreed to table the live migration discussion, and
> > > I'll resubmit the code so that arm_arch_timer.c provides the
> > > standard arch_sys_counter clocksource.  As noted previously, this just
> > > works for a Hyper-V guest.  The live migration discussion may come
> > > back later after a deeper investigation by Hyper-V.
> >
> > Great; thanks for this!
> >
> > > For clockevents, there's not a near term fix.  It's more than just plumbing
> > > an interrupt for Hyper-V to virtualize the ARM64 arch timer in a guest VM.
> > > From their perspective there's also benefit in having a timer abstraction
> > > that's independent of the architecture, and in the Linux guest, the STIMER
> > > code is common across x86/x64 and ARM64.  It follows the standard Linux
> > > clockevents model, as it should. The code is already in use in out-of-tree
> > > builds in the Linux VMs included in Windows 10 on ARM64 as part of the
> > > so-called "Windows Subsystem for Linux".
> > >
> > > So I'm hoping we can get this core support for ARM64 guests on Hyper-V
> > > into upstream using the existing STIMER support.  At some point, Hyper-V
> > > will do the virtualization of the ARM64 arch timer, but we don't want to
> > > have to stay out-of-tree until after that happens.
> >
> > My main concern here is making sure that we can rely on architected
> > properties, and don't have to special-case architected bits for hyperv
> > (or any other hypervisor), since that inevitably causes longer-term
> > pain.
> >
> > While in abstract I'm not as worried about using the timer
> > clock_event_device specifically, that same driver provides the
> > clocksource and the event stream, and I want those to work as usual,
> > without being tied into the hyperv code. IIUC that will require some
> > work, since the driver won't register if the GTDT is missing timer
> > interrupts (or if there is no GTDT).
> >
> > I think it really depends on what that looks like.
> 
> Mark,
> 
> Here are the details:
> 
> The existing initialization and registration code in arm_arch_timer.c
> works in a Hyper-V guest with no changes.  As previously mentioned,
> the GTDT exists and is correctly populated.  Even though it isn't used,
> there's a PPI INTID specified for the virtual timer, just so
> the "arm_sys_timer" clockevent can be initialized and registered.
> The IRQ shows up in the output of "cat /proc/interrupts" with zero counts
> for all CPUs since no interrupts are ever generated.  The EL1 virtual
> timer registers (CNTV_CVAL_EL0, CNTV_TVAL_EL0, and CNTV_CTL_EL0)
> are accessible in the VM.  The "arm_sys_timer" clockevent is left in
> a shutdown state with CNTV_CTL_EL0.ENABLE set to zero when the
> Hyper-V STIMER clockevent is registered with a higher rating.
> 
> Event streams are initialized and the __delay() implementation
> for ARM64 inside the kernel works.  However, on the Ampere
> eMAG hardware I'm using for testing, the WFE instruction returns
> more quickly than it should even though the event stream fields in
> CNTKCTL_EL1 are correct.  I have a query in to the Hyper-V team
> to see if they are trapping WFE and just returning, vs. perhaps the
> eMAG processor takes the easy way out and has WFE just return
> immediately.  I'm not knowledgeable about other uses of timer
> event streams, so let me know if there are other usage scenarios
> I should check.

I confirmed that Hyper-V is not trapping the WFE instruction.  And
on a Marvell TX2 and on an Ampere Altra, the counter event stream
and WFE provide the expected delay.  Evidently WFE on the eMAG
doesn't actually delay.  Bottom line:  event streams work as expected
in a Hyper-V VM.  No changes needed to arm_arch_timer.[ch].

Michael

> 
> Finally, the "arch_sys_counter" clocksource gets initialized and
> setup correctly.  If the Hyper-V clocksource is also initialized,
> you can flip between the two clocksources at runtime as expected.
> If the Hyper-V clocksource is not setup, then Linux in the VM runs
> fine with the "arch_sys_counter" clocksource.
> 
> Michael

Re: [PATCH v10 3/7] arm64: hyperv: Add Hyper-V clocksource/clockevent support

[Mark Rutland]

Hi Michael,

Thanks for all this; comments inline below. I've added Marc Zyngier, who
co-maintains the architected timer code.

On Mon, Jun 14, 2021 at 02:42:23AM +0000, Michael Kelley wrote:
> From: Mark Rutland <mark.rutland@arm.com> Sent: Thursday, June 10, 2021 9:45 AM
> > On Tue, Jun 08, 2021 at 03:36:06PM +0000, Michael Kelley wrote:
> > > I've had a couple rounds of discussions with the Hyper-V team.   For
> > > the clocksource we've agreed to table the live migration discussion, and
> > > I'll resubmit the code so that arm_arch_timer.c provides the
> > > standard arch_sys_counter clocksource.  As noted previously, this just
> > > works for a Hyper-V guest.  The live migration discussion may come
> > > back later after a deeper investigation by Hyper-V.
> > 
> > Great; thanks for this!
> > 
> > > For clockevents, there's not a near term fix.  It's more than just plumbing
> > > an interrupt for Hyper-V to virtualize the ARM64 arch timer in a guest VM.
> > > From their perspective there's also benefit in having a timer abstraction
> > > that's independent of the architecture, and in the Linux guest, the STIMER
> > > code is common across x86/x64 and ARM64.  It follows the standard Linux
> > > clockevents model, as it should. The code is already in use in out-of-tree
> > > builds in the Linux VMs included in Windows 10 on ARM64 as part of the
> > > so-called "Windows Subsystem for Linux".
> > >
> > > So I'm hoping we can get this core support for ARM64 guests on Hyper-V
> > > into upstream using the existing STIMER support.  At some point, Hyper-V
> > > will do the virtualization of the ARM64 arch timer, but we don't want to
> > > have to stay out-of-tree until after that happens.
> > 
> > My main concern here is making sure that we can rely on architected
> > properties, and don't have to special-case architected bits for hyperv
> > (or any other hypervisor), since that inevitably causes longer-term
> > pain.
> > 
> > While in abstract I'm not as worried about using the timer
> > clock_event_device specifically, that same driver provides the
> > clocksource and the event stream, and I want those to work as usual,
> > without being tied into the hyperv code. IIUC that will require some
> > work, since the driver won't register if the GTDT is missing timer
> > interrupts (or if there is no GTDT).
> > 
> > I think it really depends on what that looks like.
> 
> Mark,
> 
> Here are the details:
> 
> The existing initialization and registration code in arm_arch_timer.c
> works in a Hyper-V guest with no changes.  As previously mentioned,
> the GTDT exists and is correctly populated.  Even though it isn't used,
> there's a PPI INTID specified for the virtual timer, just so
> the "arm_sys_timer" clockevent can be initialized and registered.
> The IRQ shows up in the output of "cat /proc/interrupts" with zero counts
> for all CPUs since no interrupts are ever generated. The EL1 virtual
> timer registers (CNTV_CVAL_EL0, CNTV_TVAL_EL0, and CNTV_CTL_EL0)
> are accessible in the VM.  The "arm_sys_timer" clockevent is left in
> a shutdown state with CNTV_CTL_EL0.ENABLE set to zero when the
> Hyper-V STIMER clockevent is registered with a higher rating.

This concerns me, since we're lying to the kernel, and assuming that it
will never try to use this timer. I appreciate that evidently we don't
happen to rely on that today if you register a higher priority timer,
but that does open us up to future fragility (e.g. if we added sanity
checks when registering timers), and IIRC there are ways for userspace
to change the clockevent device today.

> Event streams are initialized and the __delay() implementation
> for ARM64 inside the kernel works.  However, on the Ampere
> eMAG hardware I'm using for testing, the WFE instruction returns
> more quickly than it should even though the event stream fields in
> CNTKCTL_EL1 are correct.  I have a query in to the Hyper-V team 
> to see if they are trapping WFE and just returning, vs. perhaps the
> eMAG processor takes the easy way out and has WFE just return
> immediately.  I'm not knowledgeable about other uses of timer
> event streams, so let me know if there are other usage scenarios
> I should check.

I saw your reply confirming that this is gnerally working as expected
(and that Hyper-V is not trapping WFE) so this sounds fine to me.

> Finally, the "arch_sys_counter" clocksource gets initialized and
> setup correctly.  If the Hyper-V clocksource is also initialized,
> you can flip between the two clocksources at runtime as expected.
> If the Hyper-V clocksource is not setup, then Linux in the VM runs
> fine with the "arch_sys_counter" clocksource.

Great!

As above, my remaining concern here is fragility around the
clockevent_device; I'm not keen that we're lying (in the GTDT) that
interrupts are wired up when they not functional, and while you can get
away with that today, that relies on kernel implementation details that
could change.

Ideally, Hyper-V would provide the architectural timer (as it's already
claiming to in the GTDT), things would "just work", and the Hyper-V
timer would be an optimization rather than a functional necessity.

You mentioned above that Hyper-V will virtualize the timer "at some
point" -- is that already planned, and when is that likely to be?

Marc, do you have any thoughts on this?

Thanks,
Mark.

Re: [PATCH v10 3/7] arm64: hyperv: Add Hyper-V clocksource/clockevent support

[Marc Zyngier]

On Tue, 22 Jun 2021 10:54:12 +0100,
Mark Rutland <mark.rutland@arm.com> wrote:
> 
> Hi Michael,
> 
> Thanks for all this; comments inline below. I've added Marc Zyngier, who
> co-maintains the architected timer code.
> 
> On Mon, Jun 14, 2021 at 02:42:23AM +0000, Michael Kelley wrote:
> > From: Mark Rutland <mark.rutland@arm.com> Sent: Thursday, June 10, 2021 9:45 AM
> > > On Tue, Jun 08, 2021 at 03:36:06PM +0000, Michael Kelley wrote:
> > > > I've had a couple rounds of discussions with the Hyper-V team.   For
> > > > the clocksource we've agreed to table the live migration discussion, and
> > > > I'll resubmit the code so that arm_arch_timer.c provides the
> > > > standard arch_sys_counter clocksource.  As noted previously, this just
> > > > works for a Hyper-V guest.  The live migration discussion may come
> > > > back later after a deeper investigation by Hyper-V.
> > > 
> > > Great; thanks for this!
> > > 
> > > > For clockevents, there's not a near term fix.  It's more than just plumbing
> > > > an interrupt for Hyper-V to virtualize the ARM64 arch timer in a guest VM.
> > > > From their perspective there's also benefit in having a timer abstraction
> > > > that's independent of the architecture, and in the Linux guest, the STIMER
> > > > code is common across x86/x64 and ARM64.  It follows the standard Linux
> > > > clockevents model, as it should. The code is already in use in out-of-tree
> > > > builds in the Linux VMs included in Windows 10 on ARM64 as part of the
> > > > so-called "Windows Subsystem for Linux".
> > > >
> > > > So I'm hoping we can get this core support for ARM64 guests on Hyper-V
> > > > into upstream using the existing STIMER support.  At some point, Hyper-V
> > > > will do the virtualization of the ARM64 arch timer, but we don't want to
> > > > have to stay out-of-tree until after that happens.
> > > 
> > > My main concern here is making sure that we can rely on architected
> > > properties, and don't have to special-case architected bits for hyperv
> > > (or any other hypervisor), since that inevitably causes longer-term
> > > pain.
> > > 
> > > While in abstract I'm not as worried about using the timer
> > > clock_event_device specifically, that same driver provides the
> > > clocksource and the event stream, and I want those to work as usual,
> > > without being tied into the hyperv code. IIUC that will require some
> > > work, since the driver won't register if the GTDT is missing timer
> > > interrupts (or if there is no GTDT).
> > > 
> > > I think it really depends on what that looks like.
> > 
> > Mark,
> > 
> > Here are the details:
> > 
> > The existing initialization and registration code in arm_arch_timer.c
> > works in a Hyper-V guest with no changes.  As previously mentioned,
> > the GTDT exists and is correctly populated.  Even though it isn't used,
> > there's a PPI INTID specified for the virtual timer, just so
> > the "arm_sys_timer" clockevent can be initialized and registered.
> > The IRQ shows up in the output of "cat /proc/interrupts" with zero counts
> > for all CPUs since no interrupts are ever generated. The EL1 virtual
> > timer registers (CNTV_CVAL_EL0, CNTV_TVAL_EL0, and CNTV_CTL_EL0)
> > are accessible in the VM.  The "arm_sys_timer" clockevent is left in
> > a shutdown state with CNTV_CTL_EL0.ENABLE set to zero when the
> > Hyper-V STIMER clockevent is registered with a higher rating.
> 
> This concerns me, since we're lying to the kernel, and assuming that it
> will never try to use this timer. I appreciate that evidently we don't
> happen to rely on that today if you register a higher priority timer,
> but that does open us up to future fragility (e.g. if we added sanity
> checks when registering timers), and IIRC there are ways for userspace
> to change the clockevent device today.

Indeed. Userspace can perfectly unbind the clockevent using
/sys/devices/system/clockevents/clockevent*/unbind_device, and the
kernel will be happy to switch to the next per-cpu timer, which
happens to be the arch timer. Oh wait...

>
> > Event streams are initialized and the __delay() implementation
> > for ARM64 inside the kernel works.  However, on the Ampere
> > eMAG hardware I'm using for testing, the WFE instruction returns
> > more quickly than it should even though the event stream fields in
> > CNTKCTL_EL1 are correct.  I have a query in to the Hyper-V team 
> > to see if they are trapping WFE and just returning, vs. perhaps the
> > eMAG processor takes the easy way out and has WFE just return
> > immediately.  I'm not knowledgeable about other uses of timer
> > event streams, so let me know if there are other usage scenarios
> > I should check.
> 
> I saw your reply confirming that this is gnerally working as expected
> (and that Hyper-V is not trapping WFE) so this sounds fine to me.
> 
> > Finally, the "arch_sys_counter" clocksource gets initialized and
> > setup correctly.  If the Hyper-V clocksource is also initialized,
> > you can flip between the two clocksources at runtime as expected.
> > If the Hyper-V clocksource is not setup, then Linux in the VM runs
> > fine with the "arch_sys_counter" clocksource.
> 
> Great!
> 
> As above, my remaining concern here is fragility around the
> clockevent_device; I'm not keen that we're lying (in the GTDT) that
> interrupts are wired up when they not functional, and while you can get
> away with that today, that relies on kernel implementation details that
> could change.
> 
> Ideally, Hyper-V would provide the architectural timer (as it's already
> claiming to in the GTDT), things would "just work", and the Hyper-V
> timer would be an optimization rather than a functional necessity.
> 
> You mentioned above that Hyper-V will virtualize the timer "at some
> point" -- is that already planned, and when is that likely to be?
> 
> Marc, do you have any thoughts on this?

Overall, lying to the kernel is a bad idea. Only implementing half of
the architecture is another bad idea. I doubt the combination of two
bad ideas produces a good one.

If Hyper-V guests need to use another timer (for migration purposes?),
that's fine. But we rely on both the base architecture to be
completely implemented *and* on advertised features to be functional.
I think this has been our position since the first Hyper-V patches
were posted... 3 years ago?

What is the hold up for reliably virtualising the arch timer,
including interrupt delivery?

Thanks,

	M.

-- 
Without deviation from the norm, progress is not possible.

RE: [PATCH v10 3/7] arm64: hyperv: Add Hyper-V clocksource/clockevent support

[Michael Kelley]

From: Marc Zyngier <maz@kernel.org> Sent: Wednesday, June 23, 2021 1:56 AM
> 
> On Tue, 22 Jun 2021 10:54:12 +0100,
> Mark Rutland <mark.rutland@arm.com> wrote:
> >
> > Hi Michael,
> >
> > Thanks for all this; comments inline below. I've added Marc Zyngier, who
> > co-maintains the architected timer code.
> >
> > On Mon, Jun 14, 2021 at 02:42:23AM +0000, Michael Kelley wrote:
> > > From: Mark Rutland <mark.rutland@arm.com> Sent: Thursday, June 10, 2021 9:45 AM
> > > > On Tue, Jun 08, 2021 at 03:36:06PM +0000, Michael Kelley wrote:
> > > > > I've had a couple rounds of discussions with the Hyper-V team.   For
> > > > > the clocksource we've agreed to table the live migration discussion, and
> > > > > I'll resubmit the code so that arm_arch_timer.c provides the
> > > > > standard arch_sys_counter clocksource.  As noted previously, this just
> > > > > works for a Hyper-V guest.  The live migration discussion may come
> > > > > back later after a deeper investigation by Hyper-V.
> > > >
> > > > Great; thanks for this!
> > > >
> > > > > For clockevents, there's not a near term fix.  It's more than just plumbing
> > > > > an interrupt for Hyper-V to virtualize the ARM64 arch timer in a guest VM.
> > > > > From their perspective there's also benefit in having a timer abstraction
> > > > > that's independent of the architecture, and in the Linux guest, the STIMER
> > > > > code is common across x86/x64 and ARM64.  It follows the standard Linux
> > > > > clockevents model, as it should. The code is already in use in out-of-tree
> > > > > builds in the Linux VMs included in Windows 10 on ARM64 as part of the
> > > > > so-called "Windows Subsystem for Linux".
> > > > >
> > > > > So I'm hoping we can get this core support for ARM64 guests on Hyper-V
> > > > > into upstream using the existing STIMER support.  At some point, Hyper-V
> > > > > will do the virtualization of the ARM64 arch timer, but we don't want to
> > > > > have to stay out-of-tree until after that happens.
> > > >
> > > > My main concern here is making sure that we can rely on architected
> > > > properties, and don't have to special-case architected bits for hyperv
> > > > (or any other hypervisor), since that inevitably causes longer-term
> > > > pain.
> > > >
> > > > While in abstract I'm not as worried about using the timer
> > > > clock_event_device specifically, that same driver provides the
> > > > clocksource and the event stream, and I want those to work as usual,
> > > > without being tied into the hyperv code. IIUC that will require some
> > > > work, since the driver won't register if the GTDT is missing timer
> > > > interrupts (or if there is no GTDT).
> > > >
> > > > I think it really depends on what that looks like.
> > >
> > > Mark,
> > >
> > > Here are the details:
> > >
> > > The existing initialization and registration code in arm_arch_timer.c
> > > works in a Hyper-V guest with no changes.  As previously mentioned,
> > > the GTDT exists and is correctly populated.  Even though it isn't used,
> > > there's a PPI INTID specified for the virtual timer, just so
> > > the "arm_sys_timer" clockevent can be initialized and registered.
> > > The IRQ shows up in the output of "cat /proc/interrupts" with zero counts
> > > for all CPUs since no interrupts are ever generated. The EL1 virtual
> > > timer registers (CNTV_CVAL_EL0, CNTV_TVAL_EL0, and CNTV_CTL_EL0)
> > > are accessible in the VM.  The "arm_sys_timer" clockevent is left in
> > > a shutdown state with CNTV_CTL_EL0.ENABLE set to zero when the
> > > Hyper-V STIMER clockevent is registered with a higher rating.
> >
> > This concerns me, since we're lying to the kernel, and assuming that it
> > will never try to use this timer. I appreciate that evidently we don't
> > happen to rely on that today if you register a higher priority timer,
> > but that does open us up to future fragility (e.g. if we added sanity
> > checks when registering timers), and IIRC there are ways for userspace
> > to change the clockevent device today.
> 
> Indeed. Userspace can perfectly unbind the clockevent using
> /sys/devices/system/clockevents/clockevent*/unbind_device, and the
> kernel will be happy to switch to the next per-cpu timer, which
> happens to be the arch timer. Oh wait...
> 
> >
> > > Event streams are initialized and the __delay() implementation
> > > for ARM64 inside the kernel works.  However, on the Ampere
> > > eMAG hardware I'm using for testing, the WFE instruction returns
> > > more quickly than it should even though the event stream fields in
> > > CNTKCTL_EL1 are correct.  I have a query in to the Hyper-V team
> > > to see if they are trapping WFE and just returning, vs. perhaps the
> > > eMAG processor takes the easy way out and has WFE just return
> > > immediately.  I'm not knowledgeable about other uses of timer
> > > event streams, so let me know if there are other usage scenarios
> > > I should check.
> >
> > I saw your reply confirming that this is gnerally working as expected
> > (and that Hyper-V is not trapping WFE) so this sounds fine to me.
> >
> > > Finally, the "arch_sys_counter" clocksource gets initialized and
> > > setup correctly.  If the Hyper-V clocksource is also initialized,
> > > you can flip between the two clocksources at runtime as expected.
> > > If the Hyper-V clocksource is not setup, then Linux in the VM runs
> > > fine with the "arch_sys_counter" clocksource.
> >
> > Great!
> >
> > As above, my remaining concern here is fragility around the
> > clockevent_device; I'm not keen that we're lying (in the GTDT) that
> > interrupts are wired up when they not functional, and while you can get
> > away with that today, that relies on kernel implementation details that
> > could change.
> >
> > Ideally, Hyper-V would provide the architectural timer (as it's already
> > claiming to in the GTDT), things would "just work", and the Hyper-V
> > timer would be an optimization rather than a functional necessity.
> >
> > You mentioned above that Hyper-V will virtualize the timer "at some
> > point" -- is that already planned, and when is that likely to be?
> >
> > Marc, do you have any thoughts on this?
> 
> Overall, lying to the kernel is a bad idea. Only implementing half of
> the architecture is another bad idea. I doubt the combination of two
> bad ideas produces a good one.
> 
> If Hyper-V guests need to use another timer (for migration purposes?),
> that's fine. But we rely on both the base architecture to be
> completely implemented *and* on advertised features to be functional.
> I think this has been our position since the first Hyper-V patches
> were posted... 3 years ago?
> 
> What is the hold up for reliably virtualising the arch timer,
> including interrupt delivery?

Marc (and Mark) --

In our early interactions about the Hyper-V clocks and timers, the code
was a bit spread out, and you suggested moving all the clocksource
and clockevent stuff to a driver under drivers/clocksource.  See
https://lore.kernel.org/lkml/e0374a07-809c-cabd-2eb6-e6b5ad84742e@arm.com/.
That was a good change independent of any ARM64 considerations,
but I read (or perhaps overread) your comments to say that it was OK
to use these Hyper-V para-virtualized clocks/timers instead of the ARM64
architectural ones in a Hyper-V VM.  They work and it's what the Hyper-V
guys wanted to do anyway, so having Hyper-V offer the ARM64 arch
counter and timer in a VM hasn't been a priority.  They had other stuff that
didn't work at all on ARM64, so that's where their attention went.

I agree that it would be better to have the ARM64 arch counter/timer
fully implemented in a Hyper-V VM.  But we're wanting to find a practical
way to move forward, and in doing so confine any rough edges to Hyper-V
VMs and the Hyper-V specific code in the kernel tree. We're maintaining
and shipping the code out-of-tree based on Hyper-V ARM64 current behavior
and would like to get this core enablement code upstream. Sure, unbinding
the Hyper-V clockevent doesn't work, but that's not a problem in any use
cases we see from customers.

All that said, our discussions with the Hyper-V team are continuing.  We're
still in the process of seeing what's practical to get and when.

Michael

> 
> Thanks,
> 
> 	M.
> 
> --
> Without deviation from the norm, progress is not possible.