* [PATCH v10 1/7] asm-generic: hyperv: Fix incorrect architecture dependencies
2021-05-12 17:37 [PATCH v10 0/7] Enable Linux guests on Hyper-V on ARM64 Michael Kelley
@ 2021-05-12 17:37 ` Michael Kelley
2021-05-12 17:37 ` [PATCH v10 2/7] arm64: hyperv: Add Hyper-V hypercall and register access utilities Michael Kelley
` (6 subsequent siblings)
7 siblings, 0 replies; 26+ messages in thread
From: Michael Kelley @ 2021-05-12 17:37 UTC (permalink / raw)
To: will, catalin.marinas, mark.rutland, lorenzo.pieralisi,
sudeep.holla, linux-arm-kernel, linux-kernel, linux-hyperv,
linux-efi, arnd, wei.liu, ardb, daniel.lezcano, kys
Cc: mikelley
Move the declaration of hv_root_partition and hyperv_pcpu_input_arg
from the x86-specific mshyperv.h to the arch independent mshyperv.h
since they are used by arch independent code. While here, add a
missing #include needed to compile correctly on ARM64.
Signed-off-by: Michael Kelley <mikelley@microsoft.com>
---
arch/x86/include/asm/mshyperv.h | 3 ---
include/asm-generic/mshyperv.h | 5 +++++
2 files changed, 5 insertions(+), 3 deletions(-)
diff --git a/arch/x86/include/asm/mshyperv.h b/arch/x86/include/asm/mshyperv.h
index 67ff0d6..45c48b0 100644
--- a/arch/x86/include/asm/mshyperv.h
+++ b/arch/x86/include/asm/mshyperv.h
@@ -36,7 +36,6 @@ static inline u64 hv_get_register(unsigned int reg)
extern int hyperv_init_cpuhp;
extern void *hv_hypercall_pg;
-extern void __percpu **hyperv_pcpu_input_arg;
extern void __percpu **hyperv_pcpu_output_arg;
extern u64 hv_current_partition_id;
@@ -170,8 +169,6 @@ int hyperv_fill_flush_guest_mapping_list(
struct hv_guest_mapping_flush_list *flush,
u64 start_gfn, u64 end_gfn);
-extern bool hv_root_partition;
-
#ifdef CONFIG_X86_64
void hv_apic_init(void);
void __init hv_init_spinlocks(void);
diff --git a/include/asm-generic/mshyperv.h b/include/asm-generic/mshyperv.h
index 9a000ba..22c92b8 100644
--- a/include/asm-generic/mshyperv.h
+++ b/include/asm-generic/mshyperv.h
@@ -22,6 +22,7 @@
#include <linux/atomic.h>
#include <linux/bitops.h>
#include <linux/cpumask.h>
+#include <linux/nmi.h>
#include <asm/ptrace.h>
#include <asm/hyperv-tlfs.h>
@@ -151,6 +152,8 @@ static inline void vmbus_signal_eom(struct hv_message *msg, u32 old_msg_type)
extern int vmbus_interrupt;
extern int vmbus_irq;
+extern bool hv_root_partition;
+
#if IS_ENABLED(CONFIG_HYPERV)
/*
* Hypervisor's notion of virtual processor ID is different from
@@ -161,6 +164,8 @@ static inline void vmbus_signal_eom(struct hv_message *msg, u32 old_msg_type)
extern u32 *hv_vp_index;
extern u32 hv_max_vp_index;
+extern void __percpu **hyperv_pcpu_input_arg;
+
/* Sentinel value for an uninitialized entry in hv_vp_index array */
#define VP_INVAL U32_MAX
--
1.8.3.1
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 26+ messages in thread* [PATCH v10 2/7] arm64: hyperv: Add Hyper-V hypercall and register access utilities
2021-05-12 17:37 [PATCH v10 0/7] Enable Linux guests on Hyper-V on ARM64 Michael Kelley
2021-05-12 17:37 ` [PATCH v10 1/7] asm-generic: hyperv: Fix incorrect architecture dependencies Michael Kelley
@ 2021-05-12 17:37 ` Michael Kelley
2021-05-14 12:52 ` Mark Rutland
2021-05-12 17:37 ` [PATCH v10 3/7] arm64: hyperv: Add Hyper-V clocksource/clockevent support Michael Kelley
` (5 subsequent siblings)
7 siblings, 1 reply; 26+ messages in thread
From: Michael Kelley @ 2021-05-12 17:37 UTC (permalink / raw)
To: will, catalin.marinas, mark.rutland, lorenzo.pieralisi,
sudeep.holla, linux-arm-kernel, linux-kernel, linux-hyperv,
linux-efi, arnd, wei.liu, ardb, daniel.lezcano, kys
Cc: mikelley
hyperv-tlfs.h defines Hyper-V interfaces from the Hyper-V Top Level
Functional Spec (TLFS), and #includes the architecture-independent
part of hyperv-tlfs.h in include/asm-generic. The published TLFS
is distinctly oriented to x86/x64, so the ARM64-specific
hyperv-tlfs.h includes information for ARM64 that is not yet formally
published. The TLFS is available here:
docs.microsoft.com/en-us/virtualization/hyper-v-on-windows/reference/tlfs
mshyperv.h defines Linux-specific structures and routines for
interacting with Hyper-V on ARM64, and #includes the architecture-
independent part of mshyperv.h in include/asm-generic.
Use these definitions to provide utility functions to make
Hyper-V hypercalls and to get and set Hyper-V provided
registers associated with a virtual processor.
Signed-off-by: Michael Kelley <mikelley@microsoft.com>
Reviewed-by: Sunil Muthuswamy <sunilmut@microsoft.com>
---
MAINTAINERS | 3 +
arch/arm64/Kbuild | 1 +
arch/arm64/hyperv/Makefile | 2 +
arch/arm64/hyperv/hv_core.c | 130 +++++++++++++++++++++++++++++++++++
arch/arm64/include/asm/hyperv-tlfs.h | 69 +++++++++++++++++++
arch/arm64/include/asm/mshyperv.h | 54 +++++++++++++++
6 files changed, 259 insertions(+)
create mode 100644 arch/arm64/hyperv/Makefile
create mode 100644 arch/arm64/hyperv/hv_core.c
create mode 100644 arch/arm64/include/asm/hyperv-tlfs.h
create mode 100644 arch/arm64/include/asm/mshyperv.h
diff --git a/MAINTAINERS b/MAINTAINERS
index bd7aff0c..6c650cb 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -8437,6 +8437,9 @@ T: git git://git.kernel.org/pub/scm/linux/kernel/git/hyperv/linux.git
F: Documentation/ABI/stable/sysfs-bus-vmbus
F: Documentation/ABI/testing/debugfs-hyperv
F: Documentation/networking/device_drivers/ethernet/microsoft/netvsc.rst
+F: arch/arm64/hyperv
+F: arch/arm64/include/asm/hyperv-tlfs.h
+F: arch/arm64/include/asm/mshyperv.h
F: arch/x86/hyperv
F: arch/x86/include/asm/hyperv-tlfs.h
F: arch/x86/include/asm/mshyperv.h
diff --git a/arch/arm64/Kbuild b/arch/arm64/Kbuild
index d646582..7a37608 100644
--- a/arch/arm64/Kbuild
+++ b/arch/arm64/Kbuild
@@ -3,4 +3,5 @@ obj-y += kernel/ mm/
obj-$(CONFIG_NET) += net/
obj-$(CONFIG_KVM) += kvm/
obj-$(CONFIG_XEN) += xen/
+obj-$(subst m,y,$(CONFIG_HYPERV)) += hyperv/
obj-$(CONFIG_CRYPTO) += crypto/
diff --git a/arch/arm64/hyperv/Makefile b/arch/arm64/hyperv/Makefile
new file mode 100644
index 0000000..1697d30
--- /dev/null
+++ b/arch/arm64/hyperv/Makefile
@@ -0,0 +1,2 @@
+# SPDX-License-Identifier: GPL-2.0
+obj-y := hv_core.o
diff --git a/arch/arm64/hyperv/hv_core.c b/arch/arm64/hyperv/hv_core.c
new file mode 100644
index 0000000..34004a5
--- /dev/null
+++ b/arch/arm64/hyperv/hv_core.c
@@ -0,0 +1,130 @@
+// SPDX-License-Identifier: GPL-2.0
+
+/*
+ * Low level utility routines for interacting with Hyper-V.
+ *
+ * Copyright (C) 2021, Microsoft, Inc.
+ *
+ * Author : Michael Kelley <mikelley@microsoft.com>
+ */
+
+#include <linux/types.h>
+#include <linux/export.h>
+#include <linux/mm.h>
+#include <linux/hyperv.h>
+#include <linux/arm-smccc.h>
+#include <linux/module.h>
+#include <asm-generic/bug.h>
+#include <asm/hyperv-tlfs.h>
+#include <asm/mshyperv.h>
+
+/*
+ * hv_do_hypercall- Invoke the specified hypercall
+ */
+u64 hv_do_hypercall(u64 control, void *input, void *output)
+{
+ struct arm_smccc_res res;
+ u64 input_address;
+ u64 output_address;
+
+ input_address = input ? virt_to_phys(input) : 0;
+ output_address = output ? virt_to_phys(output) : 0;
+
+ arm_smccc_1_1_hvc(HV_FUNC_ID, control,
+ input_address, output_address, &res);
+ return res.a0;
+}
+EXPORT_SYMBOL_GPL(hv_do_hypercall);
+
+/*
+ * hv_do_fast_hypercall8 -- Invoke the specified hypercall
+ * with arguments in registers instead of physical memory.
+ * Avoids the overhead of virt_to_phys for simple hypercalls.
+ */
+
+u64 hv_do_fast_hypercall8(u16 code, u64 input)
+{
+ struct arm_smccc_res res;
+ u64 control;
+
+ control = (u64)code | HV_HYPERCALL_FAST_BIT;
+
+ arm_smccc_1_1_hvc(HV_FUNC_ID, control, input, &res);
+ return res.a0;
+}
+EXPORT_SYMBOL_GPL(hv_do_fast_hypercall8);
+
+/*
+ * Set a single VP register to a 64-bit value.
+ */
+void hv_set_vpreg(u32 msr, u64 value)
+{
+ struct arm_smccc_res res;
+
+ arm_smccc_1_1_hvc(
+ HV_FUNC_ID,
+ HVCALL_SET_VP_REGISTERS | HV_HYPERCALL_FAST_BIT |
+ HV_HYPERCALL_REP_COMP_1,
+ HV_PARTITION_ID_SELF,
+ HV_VP_INDEX_SELF,
+ msr,
+ 0,
+ value,
+ 0,
+ &res);
+
+ /*
+ * Something is fundamentally broken in the hypervisor if
+ * setting a VP register fails. There's really no way to
+ * continue as a guest VM, so panic.
+ */
+ BUG_ON(!hv_result_success(res.a0));
+}
+EXPORT_SYMBOL_GPL(hv_set_vpreg);
+
+/*
+ * Get the value of a single VP register. One version
+ * returns just 64 bits and another returns the full 128 bits.
+ * The two versions are separate to avoid complicating the
+ * calling sequence for the more frequently used 64 bit version.
+ */
+
+void hv_get_vpreg_128(u32 msr, struct hv_get_vp_registers_output *result)
+{
+ struct arm_smccc_1_2_regs args;
+ struct arm_smccc_1_2_regs res;
+
+ args.a0 = HV_FUNC_ID;
+ args.a1 = HVCALL_GET_VP_REGISTERS | HV_HYPERCALL_FAST_BIT |
+ HV_HYPERCALL_REP_COMP_1;
+ args.a2 = HV_PARTITION_ID_SELF;
+ args.a3 = HV_VP_INDEX_SELF;
+ args.a4 = msr;
+
+ /*
+ * Use the SMCCC 1.2 interface because the results are in registers
+ * beyond X0-X3.
+ */
+ arm_smccc_1_2_hvc(&args, &res);
+
+ /*
+ * Something is fundamentally broken in the hypervisor if
+ * getting a VP register fails. There's really no way to
+ * continue as a guest VM, so panic.
+ */
+ BUG_ON(!hv_result_success(res.a0));
+
+ result->as64.low = res.a6;
+ result->as64.high = res.a7;
+}
+EXPORT_SYMBOL_GPL(hv_get_vpreg_128);
+
+u64 hv_get_vpreg(u32 msr)
+{
+ struct hv_get_vp_registers_output output;
+
+ hv_get_vpreg_128(msr, &output);
+
+ return output.as64.low;
+}
+EXPORT_SYMBOL_GPL(hv_get_vpreg);
diff --git a/arch/arm64/include/asm/hyperv-tlfs.h b/arch/arm64/include/asm/hyperv-tlfs.h
new file mode 100644
index 0000000..4d964a7
--- /dev/null
+++ b/arch/arm64/include/asm/hyperv-tlfs.h
@@ -0,0 +1,69 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+
+/*
+ * This file contains definitions from the Hyper-V Hypervisor Top-Level
+ * Functional Specification (TLFS):
+ * https://docs.microsoft.com/en-us/virtualization/hyper-v-on-windows/reference/tlfs
+ *
+ * Copyright (C) 2021, Microsoft, Inc.
+ *
+ * Author : Michael Kelley <mikelley@microsoft.com>
+ */
+
+#ifndef _ASM_HYPERV_TLFS_H
+#define _ASM_HYPERV_TLFS_H
+
+#include <linux/types.h>
+
+/*
+ * All data structures defined in the TLFS that are shared between Hyper-V
+ * and a guest VM use Little Endian byte ordering. This matches the default
+ * byte ordering of Linux running on ARM64, so no special handling is required.
+ */
+
+/*
+ * These Hyper-V registers provide information equivalent to the CPUID
+ * instruction on x86/x64.
+ */
+#define HV_REGISTER_HYPERVISOR_VERSION 0x00000100 /*CPUID 0x40000002 */
+#define HV_REGISTER_FEATURES 0x00000200 /*CPUID 0x40000003 */
+#define HV_REGISTER_ENLIGHTENMENTS 0x00000201 /*CPUID 0x40000004 */
+
+/*
+ * Group C Features. See the asm-generic version of hyperv-tlfs.h
+ * for a description of Feature Groups.
+ */
+
+/* Crash MSRs available */
+#define HV_FEATURE_GUEST_CRASH_MSR_AVAILABLE BIT(8)
+
+/* STIMER direct mode is available */
+#define HV_STIMER_DIRECT_MODE_AVAILABLE BIT(13)
+
+/*
+ * Synthetic register definitions equivalent to MSRs on x86/x64
+ */
+#define HV_REGISTER_CRASH_P0 0x00000210
+#define HV_REGISTER_CRASH_P1 0x00000211
+#define HV_REGISTER_CRASH_P2 0x00000212
+#define HV_REGISTER_CRASH_P3 0x00000213
+#define HV_REGISTER_CRASH_P4 0x00000214
+#define HV_REGISTER_CRASH_CTL 0x00000215
+
+#define HV_REGISTER_GUEST_OSID 0x00090002
+#define HV_REGISTER_VP_INDEX 0x00090003
+#define HV_REGISTER_TIME_REF_COUNT 0x00090004
+#define HV_REGISTER_REFERENCE_TSC 0x00090017
+
+#define HV_REGISTER_SINT0 0x000A0000
+#define HV_REGISTER_SCONTROL 0x000A0010
+#define HV_REGISTER_SIEFP 0x000A0012
+#define HV_REGISTER_SIMP 0x000A0013
+#define HV_REGISTER_EOM 0x000A0014
+
+#define HV_REGISTER_STIMER0_CONFIG 0x000B0000
+#define HV_REGISTER_STIMER0_COUNT 0x000B0001
+
+#include <asm-generic/hyperv-tlfs.h>
+
+#endif
diff --git a/arch/arm64/include/asm/mshyperv.h b/arch/arm64/include/asm/mshyperv.h
new file mode 100644
index 0000000..c448704
--- /dev/null
+++ b/arch/arm64/include/asm/mshyperv.h
@@ -0,0 +1,54 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+
+/*
+ * Linux-specific definitions for managing interactions with Microsoft's
+ * Hyper-V hypervisor. The definitions in this file are specific to
+ * the ARM64 architecture. See include/asm-generic/mshyperv.h for
+ * definitions are that architecture independent.
+ *
+ * Definitions that are specified in the Hyper-V Top Level Functional
+ * Spec (TLFS) should not go in this file, but should instead go in
+ * hyperv-tlfs.h.
+ *
+ * Copyright (C) 2021, Microsoft, Inc.
+ *
+ * Author : Michael Kelley <mikelley@microsoft.com>
+ */
+
+#ifndef _ASM_MSHYPERV_H
+#define _ASM_MSHYPERV_H
+
+#include <linux/types.h>
+#include <linux/arm-smccc.h>
+#include <asm/hyperv-tlfs.h>
+
+/*
+ * Declare calls to get and set Hyper-V VP register values on ARM64, which
+ * requires a hypercall.
+ */
+
+extern void hv_set_vpreg(u32 reg, u64 value);
+extern u64 hv_get_vpreg(u32 reg);
+extern void hv_get_vpreg_128(u32 reg, struct hv_get_vp_registers_output *result);
+
+static inline void hv_set_register(unsigned int reg, u64 value)
+{
+ hv_set_vpreg(reg, value);
+}
+
+static inline u64 hv_get_register(unsigned int reg)
+{
+ return hv_get_vpreg(reg);
+}
+
+/* SMCCC hypercall parameters */
+#define HV_SMCCC_FUNC_NUMBER 1
+#define HV_FUNC_ID ARM_SMCCC_CALL_VAL( \
+ ARM_SMCCC_STD_CALL, \
+ ARM_SMCCC_SMC_64, \
+ ARM_SMCCC_OWNER_VENDOR_HYP, \
+ HV_SMCCC_FUNC_NUMBER)
+
+#include <asm-generic/mshyperv.h>
+
+#endif
--
1.8.3.1
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 26+ messages in thread* Re: [PATCH v10 2/7] arm64: hyperv: Add Hyper-V hypercall and register access utilities
2021-05-12 17:37 ` [PATCH v10 2/7] arm64: hyperv: Add Hyper-V hypercall and register access utilities Michael Kelley
@ 2021-05-14 12:52 ` Mark Rutland
2021-05-14 15:14 ` Michael Kelley
0 siblings, 1 reply; 26+ messages in thread
From: Mark Rutland @ 2021-05-14 12:52 UTC (permalink / raw)
To: Michael Kelley
Cc: will, catalin.marinas, lorenzo.pieralisi, sudeep.holla,
linux-arm-kernel, linux-kernel, linux-hyperv, linux-efi, arnd,
wei.liu, ardb, daniel.lezcano, kys
On Wed, May 12, 2021 at 10:37:42AM -0700, Michael Kelley wrote:
> hyperv-tlfs.h defines Hyper-V interfaces from the Hyper-V Top Level
> Functional Spec (TLFS), and #includes the architecture-independent
> part of hyperv-tlfs.h in include/asm-generic. The published TLFS
> is distinctly oriented to x86/x64, so the ARM64-specific
> hyperv-tlfs.h includes information for ARM64 that is not yet formally
> published. The TLFS is available here:
>
> docs.microsoft.com/en-us/virtualization/hyper-v-on-windows/reference/tlfs
>
> mshyperv.h defines Linux-specific structures and routines for
> interacting with Hyper-V on ARM64, and #includes the architecture-
> independent part of mshyperv.h in include/asm-generic.
>
> Use these definitions to provide utility functions to make
> Hyper-V hypercalls and to get and set Hyper-V provided
> registers associated with a virtual processor.
>
> Signed-off-by: Michael Kelley <mikelley@microsoft.com>
> Reviewed-by: Sunil Muthuswamy <sunilmut@microsoft.com>
> ---
> MAINTAINERS | 3 +
> arch/arm64/Kbuild | 1 +
> arch/arm64/hyperv/Makefile | 2 +
> arch/arm64/hyperv/hv_core.c | 130 +++++++++++++++++++++++++++++++++++
> arch/arm64/include/asm/hyperv-tlfs.h | 69 +++++++++++++++++++
> arch/arm64/include/asm/mshyperv.h | 54 +++++++++++++++
> 6 files changed, 259 insertions(+)
> create mode 100644 arch/arm64/hyperv/Makefile
> create mode 100644 arch/arm64/hyperv/hv_core.c
> create mode 100644 arch/arm64/include/asm/hyperv-tlfs.h
> create mode 100644 arch/arm64/include/asm/mshyperv.h
> +/*
> + * hv_do_hypercall- Invoke the specified hypercall
> + */
> +u64 hv_do_hypercall(u64 control, void *input, void *output)
> +{
> + struct arm_smccc_res res;
> + u64 input_address;
> + u64 output_address;
> +
> + input_address = input ? virt_to_phys(input) : 0;
> + output_address = output ? virt_to_phys(output) : 0;
I may have asked this before, but are `input` and `output` always linear
map pointers, or can they ever be vmalloc pointers?
Otherwise, this looks fine to me.
Mark.
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 26+ messages in thread* RE: [PATCH v10 2/7] arm64: hyperv: Add Hyper-V hypercall and register access utilities
2021-05-14 12:52 ` Mark Rutland
@ 2021-05-14 15:14 ` Michael Kelley
2021-05-17 11:44 ` Mark Rutland
0 siblings, 1 reply; 26+ messages in thread
From: Michael Kelley @ 2021-05-14 15:14 UTC (permalink / raw)
To: Mark Rutland
Cc: will, catalin.marinas, lorenzo.pieralisi, sudeep.holla,
linux-arm-kernel, linux-kernel, linux-hyperv, linux-efi, arnd,
wei.liu, ardb, daniel.lezcano, KY Srinivasan
From: Mark Rutland <mark.rutland@arm.com> Sent: Friday, May 14, 2021 5:53 AM
>
> On Wed, May 12, 2021 at 10:37:42AM -0700, Michael Kelley wrote:
> > hyperv-tlfs.h defines Hyper-V interfaces from the Hyper-V Top Level
> > Functional Spec (TLFS), and #includes the architecture-independent
> > part of hyperv-tlfs.h in include/asm-generic. The published TLFS
> > is distinctly oriented to x86/x64, so the ARM64-specific
> > hyperv-tlfs.h includes information for ARM64 that is not yet formally
> > published. The TLFS is available here:
> >
> > docs.microsoft.com/en-us/virtualization/hyper-v-on-windows/reference/tlfs
> >
> > mshyperv.h defines Linux-specific structures and routines for
> > interacting with Hyper-V on ARM64, and #includes the architecture-
> > independent part of mshyperv.h in include/asm-generic.
> >
> > Use these definitions to provide utility functions to make
> > Hyper-V hypercalls and to get and set Hyper-V provided
> > registers associated with a virtual processor.
> >
> > Signed-off-by: Michael Kelley <mikelley@microsoft.com>
> > Reviewed-by: Sunil Muthuswamy <sunilmut@microsoft.com>
> > ---
> > MAINTAINERS | 3 +
> > arch/arm64/Kbuild | 1 +
> > arch/arm64/hyperv/Makefile | 2 +
> > arch/arm64/hyperv/hv_core.c | 130 +++++++++++++++++++++++++++++++++++
> > arch/arm64/include/asm/hyperv-tlfs.h | 69 +++++++++++++++++++
> > arch/arm64/include/asm/mshyperv.h | 54 +++++++++++++++
> > 6 files changed, 259 insertions(+)
> > create mode 100644 arch/arm64/hyperv/Makefile
> > create mode 100644 arch/arm64/hyperv/hv_core.c
> > create mode 100644 arch/arm64/include/asm/hyperv-tlfs.h
> > create mode 100644 arch/arm64/include/asm/mshyperv.h
>
> > +/*
> > + * hv_do_hypercall- Invoke the specified hypercall
> > + */
> > +u64 hv_do_hypercall(u64 control, void *input, void *output)
> > +{
> > + struct arm_smccc_res res;
> > + u64 input_address;
> > + u64 output_address;
> > +
> > + input_address = input ? virt_to_phys(input) : 0;
> > + output_address = output ? virt_to_phys(output) : 0;
>
> I may have asked this before, but are `input` and `output` always linear
> map pointers, or can they ever be vmalloc pointers?
>
> Otherwise, this looks fine to me.
>
The caller must ensure that hypercall arguments are aligned to
4 Kbytes, and no larger than 4 Kbytes, since that's the page size
used by Hyper-V regardless of the guest page size. A per-CPU
4 Kbyte memory area (hyperv_pcpu_input_arg) meeting these
requirements is pre-allocated that callers can use for this purpose.
On the x86 side there's also a similar hyperv_pcpu_output_arg,
but there are no use cases for it on ARM64.
Michael
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 26+ messages in thread* Re: [PATCH v10 2/7] arm64: hyperv: Add Hyper-V hypercall and register access utilities
2021-05-14 15:14 ` Michael Kelley
@ 2021-05-17 11:44 ` Mark Rutland
2021-05-17 16:41 ` Michael Kelley
0 siblings, 1 reply; 26+ messages in thread
From: Mark Rutland @ 2021-05-17 11:44 UTC (permalink / raw)
To: Michael Kelley
Cc: will, catalin.marinas, lorenzo.pieralisi, sudeep.holla,
linux-arm-kernel, linux-kernel, linux-hyperv, linux-efi, arnd,
wei.liu, ardb, daniel.lezcano, KY Srinivasan
On Fri, May 14, 2021 at 03:14:41PM +0000, Michael Kelley wrote:
> From: Mark Rutland <mark.rutland@arm.com> Sent: Friday, May 14, 2021 5:53 AM
> >
> > On Wed, May 12, 2021 at 10:37:42AM -0700, Michael Kelley wrote:
> > > hyperv-tlfs.h defines Hyper-V interfaces from the Hyper-V Top Level
> > > Functional Spec (TLFS), and #includes the architecture-independent
> > > part of hyperv-tlfs.h in include/asm-generic. The published TLFS
> > > is distinctly oriented to x86/x64, so the ARM64-specific
> > > hyperv-tlfs.h includes information for ARM64 that is not yet formally
> > > published. The TLFS is available here:
> > >
> > > docs.microsoft.com/en-us/virtualization/hyper-v-on-windows/reference/tlfs
> > >
> > > mshyperv.h defines Linux-specific structures and routines for
> > > interacting with Hyper-V on ARM64, and #includes the architecture-
> > > independent part of mshyperv.h in include/asm-generic.
> > >
> > > Use these definitions to provide utility functions to make
> > > Hyper-V hypercalls and to get and set Hyper-V provided
> > > registers associated with a virtual processor.
> > >
> > > Signed-off-by: Michael Kelley <mikelley@microsoft.com>
> > > Reviewed-by: Sunil Muthuswamy <sunilmut@microsoft.com>
> > > ---
> > > MAINTAINERS | 3 +
> > > arch/arm64/Kbuild | 1 +
> > > arch/arm64/hyperv/Makefile | 2 +
> > > arch/arm64/hyperv/hv_core.c | 130 +++++++++++++++++++++++++++++++++++
> > > arch/arm64/include/asm/hyperv-tlfs.h | 69 +++++++++++++++++++
> > > arch/arm64/include/asm/mshyperv.h | 54 +++++++++++++++
> > > 6 files changed, 259 insertions(+)
> > > create mode 100644 arch/arm64/hyperv/Makefile
> > > create mode 100644 arch/arm64/hyperv/hv_core.c
> > > create mode 100644 arch/arm64/include/asm/hyperv-tlfs.h
> > > create mode 100644 arch/arm64/include/asm/mshyperv.h
> >
> > > +/*
> > > + * hv_do_hypercall- Invoke the specified hypercall
> > > + */
> > > +u64 hv_do_hypercall(u64 control, void *input, void *output)
> > > +{
> > > + struct arm_smccc_res res;
> > > + u64 input_address;
> > > + u64 output_address;
> > > +
> > > + input_address = input ? virt_to_phys(input) : 0;
> > > + output_address = output ? virt_to_phys(output) : 0;
> >
> > I may have asked this before, but are `input` and `output` always linear
> > map pointers, or can they ever be vmalloc pointers?
> >
> > Otherwise, this looks fine to me.
>
> The caller must ensure that hypercall arguments are aligned to
> 4 Kbytes, and no larger than 4 Kbytes, since that's the page size
> used by Hyper-V regardless of the guest page size. A per-CPU
> 4 Kbyte memory area (hyperv_pcpu_input_arg) meeting these
> requirements is pre-allocated that callers can use for this purpose.
What I was trying to find out was how that was allocated, as vmalloc()'d
pointers aren't legitimate to pass to virt_to_phys().
From scanning ahead to patch 5, I see that memory comes from kmalloc(),
and so it is legitimate to use virt_to_phys().
I see; and from patch 5 I see that memory come from kmalloc(), and will
therefore be part of the linear map, and so virt_to_phys() is
legitimate.
What I was asking here was how that memory was allocated. So long as
those are the only buffers used, this looks fine to me.
Thanks,
Mark.
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 26+ messages in thread* RE: [PATCH v10 2/7] arm64: hyperv: Add Hyper-V hypercall and register access utilities
2021-05-17 11:44 ` Mark Rutland
@ 2021-05-17 16:41 ` Michael Kelley
0 siblings, 0 replies; 26+ messages in thread
From: Michael Kelley @ 2021-05-17 16:41 UTC (permalink / raw)
To: Mark Rutland
Cc: will, catalin.marinas, lorenzo.pieralisi, sudeep.holla,
linux-arm-kernel, linux-kernel, linux-hyperv, linux-efi, arnd,
wei.liu, ardb, daniel.lezcano, KY Srinivasan
From: Mark Rutland <mark.rutland@arm.com> Sent: Monday, May 17, 2021 4:45 AM
>
> On Fri, May 14, 2021 at 03:14:41PM +0000, Michael Kelley wrote:
> > From: Mark Rutland <mark.rutland@arm.com> Sent: Friday, May 14, 2021 5:53 AM
> > >
> > > On Wed, May 12, 2021 at 10:37:42AM -0700, Michael Kelley wrote:
> > > > hyperv-tlfs.h defines Hyper-V interfaces from the Hyper-V Top Level
> > > > Functional Spec (TLFS), and #includes the architecture-independent
> > > > part of hyperv-tlfs.h in include/asm-generic. The published TLFS
> > > > is distinctly oriented to x86/x64, so the ARM64-specific
> > > > hyperv-tlfs.h includes information for ARM64 that is not yet formally
> > > > published. The TLFS is available here:
> > > >
> > > > docs.microsoft.com/en-us/virtualization/hyper-v-on-windows/reference/tlfs
> > > >
> > > > mshyperv.h defines Linux-specific structures and routines for
> > > > interacting with Hyper-V on ARM64, and #includes the architecture-
> > > > independent part of mshyperv.h in include/asm-generic.
> > > >
> > > > Use these definitions to provide utility functions to make
> > > > Hyper-V hypercalls and to get and set Hyper-V provided
> > > > registers associated with a virtual processor.
> > > >
> > > > Signed-off-by: Michael Kelley <mikelley@microsoft.com>
> > > > Reviewed-by: Sunil Muthuswamy <sunilmut@microsoft.com>
> > > > ---
> > > > MAINTAINERS | 3 +
> > > > arch/arm64/Kbuild | 1 +
> > > > arch/arm64/hyperv/Makefile | 2 +
> > > > arch/arm64/hyperv/hv_core.c | 130
> +++++++++++++++++++++++++++++++++++
> > > > arch/arm64/include/asm/hyperv-tlfs.h | 69 +++++++++++++++++++
> > > > arch/arm64/include/asm/mshyperv.h | 54 +++++++++++++++
> > > > 6 files changed, 259 insertions(+)
> > > > create mode 100644 arch/arm64/hyperv/Makefile
> > > > create mode 100644 arch/arm64/hyperv/hv_core.c
> > > > create mode 100644 arch/arm64/include/asm/hyperv-tlfs.h
> > > > create mode 100644 arch/arm64/include/asm/mshyperv.h
> > >
> > > > +/*
> > > > + * hv_do_hypercall- Invoke the specified hypercall
> > > > + */
> > > > +u64 hv_do_hypercall(u64 control, void *input, void *output)
> > > > +{
> > > > + struct arm_smccc_res res;
> > > > + u64 input_address;
> > > > + u64 output_address;
> > > > +
> > > > + input_address = input ? virt_to_phys(input) : 0;
> > > > + output_address = output ? virt_to_phys(output) : 0;
> > >
> > > I may have asked this before, but are `input` and `output` always linear
> > > map pointers, or can they ever be vmalloc pointers?
> > >
> > > Otherwise, this looks fine to me.
> >
> > The caller must ensure that hypercall arguments are aligned to
> > 4 Kbytes, and no larger than 4 Kbytes, since that's the page size
> > used by Hyper-V regardless of the guest page size. A per-CPU
> > 4 Kbyte memory area (hyperv_pcpu_input_arg) meeting these
> > requirements is pre-allocated that callers can use for this purpose.
>
> What I was trying to find out was how that was allocated, as vmalloc()'d
> pointers aren't legitimate to pass to virt_to_phys().
>
> From scanning ahead to patch 5, I see that memory comes from kmalloc(),
> and so it is legitimate to use virt_to_phys().
>
>
> I see; and from patch 5 I see that memory come from kmalloc(), and will
> therefore be part of the linear map, and so virt_to_phys() is
> legitimate.
>
> What I was asking here was how that memory was allocated. So long as
> those are the only buffers used, this looks fine to me.
>
There is no vmalloc() or stack local memory used as arguments to a
hypercall. Call sites know about this requirement along with the
requirement of being aligned to 4 Kbytes and no larger than 4 Kbytes.
Michael
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 26+ messages in thread
* [PATCH v10 3/7] arm64: hyperv: Add Hyper-V clocksource/clockevent support
2021-05-12 17:37 [PATCH v10 0/7] Enable Linux guests on Hyper-V on ARM64 Michael Kelley
2021-05-12 17:37 ` [PATCH v10 1/7] asm-generic: hyperv: Fix incorrect architecture dependencies Michael Kelley
2021-05-12 17:37 ` [PATCH v10 2/7] arm64: hyperv: Add Hyper-V hypercall and register access utilities Michael Kelley
@ 2021-05-12 17:37 ` Michael Kelley
2021-05-14 12:37 ` Mark Rutland
2021-05-12 17:37 ` [PATCH v10 4/7] arm64: hyperv: Add kexec and panic handlers Michael Kelley
` (4 subsequent siblings)
7 siblings, 1 reply; 26+ messages in thread
From: Michael Kelley @ 2021-05-12 17:37 UTC (permalink / raw)
To: will, catalin.marinas, mark.rutland, lorenzo.pieralisi,
sudeep.holla, linux-arm-kernel, linux-kernel, linux-hyperv,
linux-efi, arnd, wei.liu, ardb, daniel.lezcano, kys
Cc: mikelley
Add architecture specific definitions and functions needed
by the architecture independent Hyper-V clocksource driver.
Update the Hyper-V clocksource driver to be initialized
on ARM64.
Signed-off-by: Michael Kelley <mikelley@microsoft.com>
Reviewed-by: Sunil Muthuswamy <sunilmut@microsoft.com>
---
arch/arm64/include/asm/mshyperv.h | 12 ++++++++++++
drivers/clocksource/hyperv_timer.c | 14 ++++++++++++++
2 files changed, 26 insertions(+)
diff --git a/arch/arm64/include/asm/mshyperv.h b/arch/arm64/include/asm/mshyperv.h
index c448704..b17299c 100644
--- a/arch/arm64/include/asm/mshyperv.h
+++ b/arch/arm64/include/asm/mshyperv.h
@@ -21,6 +21,7 @@
#include <linux/types.h>
#include <linux/arm-smccc.h>
#include <asm/hyperv-tlfs.h>
+#include <clocksource/arm_arch_timer.h>
/*
* Declare calls to get and set Hyper-V VP register values on ARM64, which
@@ -41,6 +42,17 @@ static inline u64 hv_get_register(unsigned int reg)
return hv_get_vpreg(reg);
}
+/* Define the interrupt ID used by STIMER0 Direct Mode interrupts. This
+ * value can't come from ACPI tables because it is needed before the
+ * Linux ACPI subsystem is initialized.
+ */
+#define HYPERV_STIMER0_VECTOR 31
+
+static inline u64 hv_get_raw_timer(void)
+{
+ return arch_timer_read_counter();
+}
+
/* SMCCC hypercall parameters */
#define HV_SMCCC_FUNC_NUMBER 1
#define HV_FUNC_ID ARM_SMCCC_CALL_VAL( \
diff --git a/drivers/clocksource/hyperv_timer.c b/drivers/clocksource/hyperv_timer.c
index 977fd05..270ad9c 100644
--- a/drivers/clocksource/hyperv_timer.c
+++ b/drivers/clocksource/hyperv_timer.c
@@ -569,3 +569,17 @@ void __init hv_init_clocksource(void)
hv_setup_sched_clock(read_hv_sched_clock_msr);
}
EXPORT_SYMBOL_GPL(hv_init_clocksource);
+
+/* Initialize everything on ARM64 */
+static int __init hyperv_timer_init(struct acpi_table_header *table)
+{
+ if (!hv_is_hyperv_initialized())
+ return -EINVAL;
+
+ hv_init_clocksource();
+ if (hv_stimer_alloc(true))
+ return -EINVAL;
+
+ return 0;
+}
+TIMER_ACPI_DECLARE(hyperv, ACPI_SIG_GTDT, hyperv_timer_init);
--
1.8.3.1
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 26+ messages in thread* Re: [PATCH v10 3/7] arm64: hyperv: Add Hyper-V clocksource/clockevent support
2021-05-12 17:37 ` [PATCH v10 3/7] arm64: hyperv: Add Hyper-V clocksource/clockevent support Michael Kelley
@ 2021-05-14 12:37 ` Mark Rutland
2021-05-14 15:35 ` Michael Kelley
0 siblings, 1 reply; 26+ messages in thread
From: Mark Rutland @ 2021-05-14 12:37 UTC (permalink / raw)
To: Michael Kelley
Cc: will, catalin.marinas, lorenzo.pieralisi, sudeep.holla,
linux-arm-kernel, linux-kernel, linux-hyperv, linux-efi, arnd,
wei.liu, ardb, daniel.lezcano, kys
Hi Michael,
On Wed, May 12, 2021 at 10:37:43AM -0700, Michael Kelley wrote:
> Add architecture specific definitions and functions needed
> by the architecture independent Hyper-V clocksource driver.
> Update the Hyper-V clocksource driver to be initialized
> on ARM64.
Previously we've said that for a clocksource we must use the architected
counter, since that's necessary for things like the VDSO to work
correctly and efficiently.
Given that, I'm a bit confused that we're registering a per-cpu
clocksource that is in part based on the architected counter. Likewise,
I don't entirely follow why it's necessary to PV the clock_event_device.
Are the architected counter and timer reliable without this PV
infrastructure? Why do we need to PV either of those?
Thanks,
Mark.
>
> Signed-off-by: Michael Kelley <mikelley@microsoft.com>
> Reviewed-by: Sunil Muthuswamy <sunilmut@microsoft.com>
> ---
> arch/arm64/include/asm/mshyperv.h | 12 ++++++++++++
> drivers/clocksource/hyperv_timer.c | 14 ++++++++++++++
> 2 files changed, 26 insertions(+)
>
> diff --git a/arch/arm64/include/asm/mshyperv.h b/arch/arm64/include/asm/mshyperv.h
> index c448704..b17299c 100644
> --- a/arch/arm64/include/asm/mshyperv.h
> +++ b/arch/arm64/include/asm/mshyperv.h
> @@ -21,6 +21,7 @@
> #include <linux/types.h>
> #include <linux/arm-smccc.h>
> #include <asm/hyperv-tlfs.h>
> +#include <clocksource/arm_arch_timer.h>
>
> /*
> * Declare calls to get and set Hyper-V VP register values on ARM64, which
> @@ -41,6 +42,17 @@ static inline u64 hv_get_register(unsigned int reg)
> return hv_get_vpreg(reg);
> }
>
> +/* Define the interrupt ID used by STIMER0 Direct Mode interrupts. This
> + * value can't come from ACPI tables because it is needed before the
> + * Linux ACPI subsystem is initialized.
> + */
> +#define HYPERV_STIMER0_VECTOR 31
> +
> +static inline u64 hv_get_raw_timer(void)
> +{
> + return arch_timer_read_counter();
> +}
> +
> /* SMCCC hypercall parameters */
> #define HV_SMCCC_FUNC_NUMBER 1
> #define HV_FUNC_ID ARM_SMCCC_CALL_VAL( \
> diff --git a/drivers/clocksource/hyperv_timer.c b/drivers/clocksource/hyperv_timer.c
> index 977fd05..270ad9c 100644
> --- a/drivers/clocksource/hyperv_timer.c
> +++ b/drivers/clocksource/hyperv_timer.c
> @@ -569,3 +569,17 @@ void __init hv_init_clocksource(void)
> hv_setup_sched_clock(read_hv_sched_clock_msr);
> }
> EXPORT_SYMBOL_GPL(hv_init_clocksource);
> +
> +/* Initialize everything on ARM64 */
> +static int __init hyperv_timer_init(struct acpi_table_header *table)
> +{
> + if (!hv_is_hyperv_initialized())
> + return -EINVAL;
> +
> + hv_init_clocksource();
> + if (hv_stimer_alloc(true))
> + return -EINVAL;
> +
> + return 0;
> +}
> +TIMER_ACPI_DECLARE(hyperv, ACPI_SIG_GTDT, hyperv_timer_init);
> --
> 1.8.3.1
>
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 26+ messages in thread* RE: [PATCH v10 3/7] arm64: hyperv: Add Hyper-V clocksource/clockevent support
2021-05-14 12:37 ` Mark Rutland
@ 2021-05-14 15:35 ` Michael Kelley
2021-05-17 13:08 ` Mark Rutland
0 siblings, 1 reply; 26+ messages in thread
From: Michael Kelley @ 2021-05-14 15:35 UTC (permalink / raw)
To: Mark Rutland
Cc: will, catalin.marinas, lorenzo.pieralisi, sudeep.holla,
linux-arm-kernel, linux-kernel, linux-hyperv, linux-efi, arnd,
wei.liu, ardb, daniel.lezcano, KY Srinivasan
From: Mark Rutland <mark.rutland@arm.com> Sent: Friday, May 14, 2021 5:37 AM
>
> Hi Michael,
>
> On Wed, May 12, 2021 at 10:37:43AM -0700, Michael Kelley wrote:
> > Add architecture specific definitions and functions needed
> > by the architecture independent Hyper-V clocksource driver.
> > Update the Hyper-V clocksource driver to be initialized
> > on ARM64.
>
> Previously we've said that for a clocksource we must use the architected
> counter, since that's necessary for things like the VDSO to work
> correctly and efficiently.
>
> Given that, I'm a bit confused that we're registering a per-cpu
> clocksource that is in part based on the architected counter. Likewise,
> I don't entirely follow why it's necessary to PV the clock_event_device.
>
> Are the architected counter and timer reliable without this PV
> infrastructure? Why do we need to PV either of those?
>
> Thanks,
> Mark.
>
For the clocksource, we have a requirement to live migrate VMs
between Hyper-V hosts running on hardware that may have different
arch counter frequencies (it's not conformant to the ARM v8.6 1 GHz
requirement). The Hyper-V virtualization does scaling to handle the
frequency difference. And yes, there's a tradeoff with vDSO not
working, though we have an out-of-tree vDSO implementation that
we can use when necessary.
For clockevents, the only timer interrupt that Hyper-V provides
in a guest VM is its virtualized "STIMER" interrupt. There's no
virtualization of the ARM arch timer in the guest.
Michael
> >
> > Signed-off-by: Michael Kelley <mikelley@microsoft.com>
> > Reviewed-by: Sunil Muthuswamy <sunilmut@microsoft.com>
> > ---
> > arch/arm64/include/asm/mshyperv.h | 12 ++++++++++++
> > drivers/clocksource/hyperv_timer.c | 14 ++++++++++++++
> > 2 files changed, 26 insertions(+)
> >
> > diff --git a/arch/arm64/include/asm/mshyperv.h b/arch/arm64/include/asm/mshyperv.h
> > index c448704..b17299c 100644
> > --- a/arch/arm64/include/asm/mshyperv.h
> > +++ b/arch/arm64/include/asm/mshyperv.h
> > @@ -21,6 +21,7 @@
> > #include <linux/types.h>
> > #include <linux/arm-smccc.h>
> > #include <asm/hyperv-tlfs.h>
> > +#include <clocksource/arm_arch_timer.h>
> >
> > /*
> > * Declare calls to get and set Hyper-V VP register values on ARM64, which
> > @@ -41,6 +42,17 @@ static inline u64 hv_get_register(unsigned int reg)
> > return hv_get_vpreg(reg);
> > }
> >
> > +/* Define the interrupt ID used by STIMER0 Direct Mode interrupts. This
> > + * value can't come from ACPI tables because it is needed before the
> > + * Linux ACPI subsystem is initialized.
> > + */
> > +#define HYPERV_STIMER0_VECTOR 31
> > +
> > +static inline u64 hv_get_raw_timer(void)
> > +{
> > + return arch_timer_read_counter();
> > +}
> > +
> > /* SMCCC hypercall parameters */
> > #define HV_SMCCC_FUNC_NUMBER 1
> > #define HV_FUNC_ID ARM_SMCCC_CALL_VAL( \
> > diff --git a/drivers/clocksource/hyperv_timer.c b/drivers/clocksource/hyperv_timer.c
> > index 977fd05..270ad9c 100644
> > --- a/drivers/clocksource/hyperv_timer.c
> > +++ b/drivers/clocksource/hyperv_timer.c
> > @@ -569,3 +569,17 @@ void __init hv_init_clocksource(void)
> > hv_setup_sched_clock(read_hv_sched_clock_msr);
> > }
> > EXPORT_SYMBOL_GPL(hv_init_clocksource);
> > +
> > +/* Initialize everything on ARM64 */
> > +static int __init hyperv_timer_init(struct acpi_table_header *table)
> > +{
> > + if (!hv_is_hyperv_initialized())
> > + return -EINVAL;
> > +
> > + hv_init_clocksource();
> > + if (hv_stimer_alloc(true))
> > + return -EINVAL;
> > +
> > + return 0;
> > +}
> > +TIMER_ACPI_DECLARE(hyperv, ACPI_SIG_GTDT, hyperv_timer_init);
> > --
> > 1.8.3.1
> >
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 26+ messages in thread* Re: [PATCH v10 3/7] arm64: hyperv: Add Hyper-V clocksource/clockevent support
2021-05-14 15:35 ` Michael Kelley
@ 2021-05-17 13:08 ` Mark Rutland
2021-05-17 17:27 ` Michael Kelley
0 siblings, 1 reply; 26+ messages in thread
From: Mark Rutland @ 2021-05-17 13:08 UTC (permalink / raw)
To: Michael Kelley
Cc: will, catalin.marinas, lorenzo.pieralisi, sudeep.holla,
linux-arm-kernel, linux-kernel, linux-hyperv, linux-efi, arnd,
wei.liu, ardb, daniel.lezcano, KY Srinivasan
On Fri, May 14, 2021 at 03:35:15PM +0000, Michael Kelley wrote:
> From: Mark Rutland <mark.rutland@arm.com> Sent: Friday, May 14, 2021 5:37 AM
> > On Wed, May 12, 2021 at 10:37:43AM -0700, Michael Kelley wrote:
> > > Add architecture specific definitions and functions needed
> > > by the architecture independent Hyper-V clocksource driver.
> > > Update the Hyper-V clocksource driver to be initialized
> > > on ARM64.
> >
> > Previously we've said that for a clocksource we must use the architected
> > counter, since that's necessary for things like the VDSO to work
> > correctly and efficiently.
> >
> > Given that, I'm a bit confused that we're registering a per-cpu
> > clocksource that is in part based on the architected counter. Likewise,
> > I don't entirely follow why it's necessary to PV the clock_event_device.
> >
> > Are the architected counter and timer reliable without this PV
> > infrastructure? Why do we need to PV either of those?
> >
> > Thanks,
> > Mark.
>
> For the clocksource, we have a requirement to live migrate VMs
> between Hyper-V hosts running on hardware that may have different
> arch counter frequencies (it's not conformant to the ARM v8.6 1 GHz
> requirement). The Hyper-V virtualization does scaling to handle the
> frequency difference. And yes, there's a tradeoff with vDSO not
> working, though we have an out-of-tree vDSO implementation that
> we can use when necessary.
Just to be clear, the vDSO is *one example* of something that won't
function correctly. More generally, because this undermines core
architectural guarantees, it requires more invasive changes (e.g. we'd
have to weaken the sanity checks, and not use the counter in things like
kexec paths), impacts any architectural features tied to the generic
timer/counter (e.g. the event stream, SPE and tracing, future features),
and means that other SW (e.g. bootloaders and other EFI applications)
are unlikley to function correctly in this environment.
I am very much not keen on trying to PV this.
What does the guest see when it reads CNTFRQ_EL0? Does this match the
real HW value (and can this change over time)? Or is this entirely
synthetic?
What do the ACPI tables look like in the guest? Is there a GTDT table at
all?
How does the counter event stream behave?
Are there other architectural features which Hyper-V does not implement
for a guest?
Is there anything else that may change across a migration? e.g. MIDR?
MPIDR? Any of the ID registers?
> For clockevents, the only timer interrupt that Hyper-V provides
> in a guest VM is its virtualized "STIMER" interrupt. There's no
> virtualization of the ARM arch timer in the guest.
I think that is rather unfortunate, given it's a core architectural
feature. Is it just the interrupt that's missing? i.e. does all the
PE-local functionality behave as the architecture requires?
Thanks,
Mark.
>
> > >
> > > Signed-off-by: Michael Kelley <mikelley@microsoft.com>
> > > Reviewed-by: Sunil Muthuswamy <sunilmut@microsoft.com>
> > > ---
> > > arch/arm64/include/asm/mshyperv.h | 12 ++++++++++++
> > > drivers/clocksource/hyperv_timer.c | 14 ++++++++++++++
> > > 2 files changed, 26 insertions(+)
> > >
> > > diff --git a/arch/arm64/include/asm/mshyperv.h b/arch/arm64/include/asm/mshyperv.h
> > > index c448704..b17299c 100644
> > > --- a/arch/arm64/include/asm/mshyperv.h
> > > +++ b/arch/arm64/include/asm/mshyperv.h
> > > @@ -21,6 +21,7 @@
> > > #include <linux/types.h>
> > > #include <linux/arm-smccc.h>
> > > #include <asm/hyperv-tlfs.h>
> > > +#include <clocksource/arm_arch_timer.h>
> > >
> > > /*
> > > * Declare calls to get and set Hyper-V VP register values on ARM64, which
> > > @@ -41,6 +42,17 @@ static inline u64 hv_get_register(unsigned int reg)
> > > return hv_get_vpreg(reg);
> > > }
> > >
> > > +/* Define the interrupt ID used by STIMER0 Direct Mode interrupts. This
> > > + * value can't come from ACPI tables because it is needed before the
> > > + * Linux ACPI subsystem is initialized.
> > > + */
> > > +#define HYPERV_STIMER0_VECTOR 31
> > > +
> > > +static inline u64 hv_get_raw_timer(void)
> > > +{
> > > + return arch_timer_read_counter();
> > > +}
> > > +
> > > /* SMCCC hypercall parameters */
> > > #define HV_SMCCC_FUNC_NUMBER 1
> > > #define HV_FUNC_ID ARM_SMCCC_CALL_VAL( \
> > > diff --git a/drivers/clocksource/hyperv_timer.c b/drivers/clocksource/hyperv_timer.c
> > > index 977fd05..270ad9c 100644
> > > --- a/drivers/clocksource/hyperv_timer.c
> > > +++ b/drivers/clocksource/hyperv_timer.c
> > > @@ -569,3 +569,17 @@ void __init hv_init_clocksource(void)
> > > hv_setup_sched_clock(read_hv_sched_clock_msr);
> > > }
> > > EXPORT_SYMBOL_GPL(hv_init_clocksource);
> > > +
> > > +/* Initialize everything on ARM64 */
> > > +static int __init hyperv_timer_init(struct acpi_table_header *table)
> > > +{
> > > + if (!hv_is_hyperv_initialized())
> > > + return -EINVAL;
> > > +
> > > + hv_init_clocksource();
> > > + if (hv_stimer_alloc(true))
> > > + return -EINVAL;
> > > +
> > > + return 0;
> > > +}
> > > +TIMER_ACPI_DECLARE(hyperv, ACPI_SIG_GTDT, hyperv_timer_init);
> > > --
> > > 1.8.3.1
> > >
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 26+ messages in thread* RE: [PATCH v10 3/7] arm64: hyperv: Add Hyper-V clocksource/clockevent support
2021-05-17 13:08 ` Mark Rutland
@ 2021-05-17 17:27 ` Michael Kelley
2021-05-18 17:00 ` Mark Rutland
0 siblings, 1 reply; 26+ messages in thread
From: Michael Kelley @ 2021-05-17 17:27 UTC (permalink / raw)
To: Mark Rutland
Cc: will, catalin.marinas, lorenzo.pieralisi, sudeep.holla,
linux-arm-kernel, linux-kernel, linux-hyperv, linux-efi, arnd,
wei.liu, ardb, daniel.lezcano, KY Srinivasan
From: Mark Rutland <mark.rutland@arm.com> Sent: Monday, May 17, 2021 6:08 AM
>
> On Fri, May 14, 2021 at 03:35:15PM +0000, Michael Kelley wrote:
> > From: Mark Rutland <mark.rutland@arm.com> Sent: Friday, May 14, 2021 5:37 AM
> > > On Wed, May 12, 2021 at 10:37:43AM -0700, Michael Kelley wrote:
> > > > Add architecture specific definitions and functions needed
> > > > by the architecture independent Hyper-V clocksource driver.
> > > > Update the Hyper-V clocksource driver to be initialized
> > > > on ARM64.
> > >
> > > Previously we've said that for a clocksource we must use the architected
> > > counter, since that's necessary for things like the VDSO to work
> > > correctly and efficiently.
> > >
> > > Given that, I'm a bit confused that we're registering a per-cpu
> > > clocksource that is in part based on the architected counter. Likewise,
> > > I don't entirely follow why it's necessary to PV the clock_event_device.
> > >
> > > Are the architected counter and timer reliable without this PV
> > > infrastructure? Why do we need to PV either of those?
> > >
> > > Thanks,
> > > Mark.
> >
> > For the clocksource, we have a requirement to live migrate VMs
> > between Hyper-V hosts running on hardware that may have different
> > arch counter frequencies (it's not conformant to the ARM v8.6 1 GHz
> > requirement). The Hyper-V virtualization does scaling to handle the
> > frequency difference. And yes, there's a tradeoff with vDSO not
> > working, though we have an out-of-tree vDSO implementation that
> > we can use when necessary.
>
> Just to be clear, the vDSO is *one example* of something that won't
> function correctly. More generally, because this undermines core
> architectural guarantees, it requires more invasive changes (e.g. we'd
> have to weaken the sanity checks, and not use the counter in things like
> kexec paths), impacts any architectural features tied to the generic
> timer/counter (e.g. the event stream, SPE and tracing, future features),
> and means that other SW (e.g. bootloaders and other EFI applications)
> are unlikley to function correctly in this environment.
>
> I am very much not keen on trying to PV this.
>
> What does the guest see when it reads CNTFRQ_EL0? Does this match the
> real HW value (and can this change over time)? Or is this entirely
> synthetic?
>
> What do the ACPI tables look like in the guest? Is there a GTDT table at
> all?
>
> How does the counter event stream behave?
>
> Are there other architectural features which Hyper-V does not implement
> for a guest?
>
> Is there anything else that may change across a migration? e.g. MIDR?
> MPIDR? Any of the ID registers?
The ARMv8 architectural system counter and associated registers are visible
and functional in a VM on Hyper-V. The "arch_sys_counter" clocksource is
instantiated by the arm_arch_timer.c driver based on the GTDT in the guest,
and a Linux guest on Hyper-V runs fine with this clocksource. Low level code
like bootloaders and EFI applications work normally.
The Hyper-V virtualization provides another Linux clocksource that is an
overlay on the arch counter and that provides time consistency across a live
migration. Live migration of ARM64 VMs on Hyper-V is not functional today,
but the Hyper-V team believes they can make it functional. I have not
explored with them the live migration implications of things beyond time
consistency, like event streams, CNTFRQ_EL0, MIDR/MPIDR, etc.
Would a summary of your point be that live migration across hardware
with different arch counter frequencies is likely to not be possible with
100% fidelity because of these other dependencies on the arch counter
frequency? (hence the fixed 1 GHz frequency in ARM v8.6)
>
> > For clockevents, the only timer interrupt that Hyper-V provides
> > in a guest VM is its virtualized "STIMER" interrupt. There's no
> > virtualization of the ARM arch timer in the guest.
>
> I think that is rather unfortunate, given it's a core architectural
> feature. Is it just the interrupt that's missing? i.e. does all the
> PE-local functionality behave as the architecture requires?
Right off the bat, I don't know about timer-related PE-local
functionality as it's not exercised in a Linux VM on Hyper-V that is
using STIMER-based clockevents. I'll explore with the Hyper-V
team. My impression is that enabling the ARM arch timer in a
guest VM is more work for Hyper-V than just wiring up an
interrupt.
Michael
>
> Thanks,
> Mark.
>
> >
> > > >
> > > > Signed-off-by: Michael Kelley <mikelley@microsoft.com>
> > > > Reviewed-by: Sunil Muthuswamy <sunilmut@microsoft.com>
> > > > ---
> > > > arch/arm64/include/asm/mshyperv.h | 12 ++++++++++++
> > > > drivers/clocksource/hyperv_timer.c | 14 ++++++++++++++
> > > > 2 files changed, 26 insertions(+)
> > > >
> > > > diff --git a/arch/arm64/include/asm/mshyperv.h
> b/arch/arm64/include/asm/mshyperv.h
> > > > index c448704..b17299c 100644
> > > > --- a/arch/arm64/include/asm/mshyperv.h
> > > > +++ b/arch/arm64/include/asm/mshyperv.h
> > > > @@ -21,6 +21,7 @@
> > > > #include <linux/types.h>
> > > > #include <linux/arm-smccc.h>
> > > > #include <asm/hyperv-tlfs.h>
> > > > +#include <clocksource/arm_arch_timer.h>
> > > >
> > > > /*
> > > > * Declare calls to get and set Hyper-V VP register values on ARM64, which
> > > > @@ -41,6 +42,17 @@ static inline u64 hv_get_register(unsigned int reg)
> > > > return hv_get_vpreg(reg);
> > > > }
> > > >
> > > > +/* Define the interrupt ID used by STIMER0 Direct Mode interrupts. This
> > > > + * value can't come from ACPI tables because it is needed before the
> > > > + * Linux ACPI subsystem is initialized.
> > > > + */
> > > > +#define HYPERV_STIMER0_VECTOR 31
> > > > +
> > > > +static inline u64 hv_get_raw_timer(void)
> > > > +{
> > > > + return arch_timer_read_counter();
> > > > +}
> > > > +
> > > > /* SMCCC hypercall parameters */
> > > > #define HV_SMCCC_FUNC_NUMBER 1
> > > > #define HV_FUNC_ID ARM_SMCCC_CALL_VAL( \
> > > > diff --git a/drivers/clocksource/hyperv_timer.c b/drivers/clocksource/hyperv_timer.c
> > > > index 977fd05..270ad9c 100644
> > > > --- a/drivers/clocksource/hyperv_timer.c
> > > > +++ b/drivers/clocksource/hyperv_timer.c
> > > > @@ -569,3 +569,17 @@ void __init hv_init_clocksource(void)
> > > > hv_setup_sched_clock(read_hv_sched_clock_msr);
> > > > }
> > > > EXPORT_SYMBOL_GPL(hv_init_clocksource);
> > > > +
> > > > +/* Initialize everything on ARM64 */
> > > > +static int __init hyperv_timer_init(struct acpi_table_header *table)
> > > > +{
> > > > + if (!hv_is_hyperv_initialized())
> > > > + return -EINVAL;
> > > > +
> > > > + hv_init_clocksource();
> > > > + if (hv_stimer_alloc(true))
> > > > + return -EINVAL;
> > > > +
> > > > + return 0;
> > > > +}
> > > > +TIMER_ACPI_DECLARE(hyperv, ACPI_SIG_GTDT, hyperv_timer_init);
> > > > --
> > > > 1.8.3.1
> > > >
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 26+ messages in thread* Re: [PATCH v10 3/7] arm64: hyperv: Add Hyper-V clocksource/clockevent support
2021-05-17 17:27 ` Michael Kelley
@ 2021-05-18 17:00 ` Mark Rutland
2021-06-08 15:36 ` Michael Kelley
0 siblings, 1 reply; 26+ messages in thread
From: Mark Rutland @ 2021-05-18 17:00 UTC (permalink / raw)
To: Michael Kelley
Cc: will, catalin.marinas, lorenzo.pieralisi, sudeep.holla,
linux-arm-kernel, linux-kernel, linux-hyperv, linux-efi, arnd,
wei.liu, ardb, daniel.lezcano, KY Srinivasan
On Mon, May 17, 2021 at 05:27:49PM +0000, Michael Kelley wrote:
> From: Mark Rutland <mark.rutland@arm.com> Sent: Monday, May 17, 2021 6:08 AM
> >
> > On Fri, May 14, 2021 at 03:35:15PM +0000, Michael Kelley wrote:
> > > From: Mark Rutland <mark.rutland@arm.com> Sent: Friday, May 14, 2021 5:37 AM
> > > > On Wed, May 12, 2021 at 10:37:43AM -0700, Michael Kelley wrote:
> > > > > Add architecture specific definitions and functions needed
> > > > > by the architecture independent Hyper-V clocksource driver.
> > > > > Update the Hyper-V clocksource driver to be initialized
> > > > > on ARM64.
> > > >
> > > > Previously we've said that for a clocksource we must use the architected
> > > > counter, since that's necessary for things like the VDSO to work
> > > > correctly and efficiently.
> > > >
> > > > Given that, I'm a bit confused that we're registering a per-cpu
> > > > clocksource that is in part based on the architected counter. Likewise,
> > > > I don't entirely follow why it's necessary to PV the clock_event_device.
> > > >
> > > > Are the architected counter and timer reliable without this PV
> > > > infrastructure? Why do we need to PV either of those?
> > > >
> > > > Thanks,
> > > > Mark.
> > >
> > > For the clocksource, we have a requirement to live migrate VMs
> > > between Hyper-V hosts running on hardware that may have different
> > > arch counter frequencies (it's not conformant to the ARM v8.6 1 GHz
> > > requirement). The Hyper-V virtualization does scaling to handle the
> > > frequency difference. And yes, there's a tradeoff with vDSO not
> > > working, though we have an out-of-tree vDSO implementation that
> > > we can use when necessary.
> >
> > Just to be clear, the vDSO is *one example* of something that won't
> > function correctly. More generally, because this undermines core
> > architectural guarantees, it requires more invasive changes (e.g. we'd
> > have to weaken the sanity checks, and not use the counter in things like
> > kexec paths), impacts any architectural features tied to the generic
> > timer/counter (e.g. the event stream, SPE and tracing, future features),
> > and means that other SW (e.g. bootloaders and other EFI applications)
> > are unlikley to function correctly in this environment.
> >
> > I am very much not keen on trying to PV this.
> >
> > What does the guest see when it reads CNTFRQ_EL0? Does this match the
> > real HW value (and can this change over time)? Or is this entirely
> > synthetic?
> >
> > What do the ACPI tables look like in the guest? Is there a GTDT table at
> > all?
> >
> > How does the counter event stream behave?
> >
> > Are there other architectural features which Hyper-V does not implement
> > for a guest?
> >
> > Is there anything else that may change across a migration? e.g. MIDR?
> > MPIDR? Any of the ID registers?
>
> The ARMv8 architectural system counter and associated registers are visible
> and functional in a VM on Hyper-V. The "arch_sys_counter" clocksource is
> instantiated by the arm_arch_timer.c driver based on the GTDT in the guest,
> and a Linux guest on Hyper-V runs fine with this clocksource. Low level code
> like bootloaders and EFI applications work normally.
That's good to hear!
One potential issue worth noting is that as those pieces of software are
unlikely to handle counter frequency changes reliably, and so may not
behave correctly if live-migrated.
> The Hyper-V virtualization provides another Linux clocksource that is an
> overlay on the arch counter and that provides time consistency across a live
> migration. Live migration of ARM64 VMs on Hyper-V is not functional today,
> but the Hyper-V team believes they can make it functional. I have not
> explored with them the live migration implications of things beyond time
> consistency, like event streams, CNTFRQ_EL0, MIDR/MPIDR, etc.
>
> Would a summary of your point be that live migration across hardware
> with different arch counter frequencies is likely to not be possible with
> 100% fidelity because of these other dependencies on the arch counter
> frequency? (hence the fixed 1 GHz frequency in ARM v8.6)
Yes.
In addition, there are a larger set of things necessarily exposed to VMs
that mean that live migration isn't all that practical except betweenm
identical machines (where the counter frequency should be identical),
and the timer frequency might just be the canary in the coalmine. For
example, the cache properties enumerated in CTR_EL0 cannot necessarily
be emulated on another machine.
> > > For clockevents, the only timer interrupt that Hyper-V provides
> > > in a guest VM is its virtualized "STIMER" interrupt. There's no
> > > virtualization of the ARM arch timer in the guest.
> >
> > I think that is rather unfortunate, given it's a core architectural
> > feature. Is it just the interrupt that's missing? i.e. does all the
> > PE-local functionality behave as the architecture requires?
>
> Right off the bat, I don't know about timer-related PE-local
> functionality as it's not exercised in a Linux VM on Hyper-V that is
> using STIMER-based clockevents. I'll explore with the Hyper-V
> team. My impression is that enabling the ARM arch timer in a
> guest VM is more work for Hyper-V than just wiring up an
> interrupt.
Thanks for chasing this up!
Mark.
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 26+ messages in thread
* RE: [PATCH v10 3/7] arm64: hyperv: Add Hyper-V clocksource/clockevent support
2021-05-18 17:00 ` Mark Rutland
@ 2021-06-08 15:36 ` Michael Kelley
2021-06-10 16:45 ` Mark Rutland
0 siblings, 1 reply; 26+ messages in thread
From: Michael Kelley @ 2021-06-08 15:36 UTC (permalink / raw)
To: Mark Rutland
Cc: will, catalin.marinas, lorenzo.pieralisi, sudeep.holla,
linux-arm-kernel, linux-kernel, linux-hyperv, linux-efi, arnd,
wei.liu, ardb, daniel.lezcano, KY Srinivasan
From: Mark Rutland <mark.rutland@arm.com> Sent: Tuesday, May 18, 2021 10:00 AM
>
> On Mon, May 17, 2021 at 05:27:49PM +0000, Michael Kelley wrote:
> > From: Mark Rutland <mark.rutland@arm.com> Sent: Monday, May 17, 2021 6:08 AM
> > >
> > > On Fri, May 14, 2021 at 03:35:15PM +0000, Michael Kelley wrote:
> > > > From: Mark Rutland <mark.rutland@arm.com> Sent: Friday, May 14, 2021 5:37 AM
> > > > > On Wed, May 12, 2021 at 10:37:43AM -0700, Michael Kelley wrote:
> > > > > > Add architecture specific definitions and functions needed
> > > > > > by the architecture independent Hyper-V clocksource driver.
> > > > > > Update the Hyper-V clocksource driver to be initialized
> > > > > > on ARM64.
> > > > >
> > > > > Previously we've said that for a clocksource we must use the architected
> > > > > counter, since that's necessary for things like the VDSO to work
> > > > > correctly and efficiently.
> > > > >
> > > > > Given that, I'm a bit confused that we're registering a per-cpu
> > > > > clocksource that is in part based on the architected counter. Likewise,
> > > > > I don't entirely follow why it's necessary to PV the clock_event_device.
> > > > >
> > > > > Are the architected counter and timer reliable without this PV
> > > > > infrastructure? Why do we need to PV either of those?
> > > > >
> > > > > Thanks,
> > > > > Mark.
> > > >
> > > > For the clocksource, we have a requirement to live migrate VMs
> > > > between Hyper-V hosts running on hardware that may have different
> > > > arch counter frequencies (it's not conformant to the ARM v8.6 1 GHz
> > > > requirement). The Hyper-V virtualization does scaling to handle the
> > > > frequency difference. And yes, there's a tradeoff with vDSO not
> > > > working, though we have an out-of-tree vDSO implementation that
> > > > we can use when necessary.
> > >
> > > Just to be clear, the vDSO is *one example* of something that won't
> > > function correctly. More generally, because this undermines core
> > > architectural guarantees, it requires more invasive changes (e.g. we'd
> > > have to weaken the sanity checks, and not use the counter in things like
> > > kexec paths), impacts any architectural features tied to the generic
> > > timer/counter (e.g. the event stream, SPE and tracing, future features),
> > > and means that other SW (e.g. bootloaders and other EFI applications)
> > > are unlikley to function correctly in this environment.
> > >
> > > I am very much not keen on trying to PV this.
> > >
> > > What does the guest see when it reads CNTFRQ_EL0? Does this match the
> > > real HW value (and can this change over time)? Or is this entirely
> > > synthetic?
> > >
> > > What do the ACPI tables look like in the guest? Is there a GTDT table at
> > > all?
> > >
> > > How does the counter event stream behave?
> > >
> > > Are there other architectural features which Hyper-V does not implement
> > > for a guest?
> > >
> > > Is there anything else that may change across a migration? e.g. MIDR?
> > > MPIDR? Any of the ID registers?
> >
> > The ARMv8 architectural system counter and associated registers are visible
> > and functional in a VM on Hyper-V. The "arch_sys_counter" clocksource is
> > instantiated by the arm_arch_timer.c driver based on the GTDT in the guest,
> > and a Linux guest on Hyper-V runs fine with this clocksource. Low level code
> > like bootloaders and EFI applications work normally.
>
> That's good to hear!
>
> One potential issue worth noting is that as those pieces of software are
> unlikely to handle counter frequency changes reliably, and so may not
> behave correctly if live-migrated.
>
> > The Hyper-V virtualization provides another Linux clocksource that is an
> > overlay on the arch counter and that provides time consistency across a live
> > migration. Live migration of ARM64 VMs on Hyper-V is not functional today,
> > but the Hyper-V team believes they can make it functional. I have not
> > explored with them the live migration implications of things beyond time
> > consistency, like event streams, CNTFRQ_EL0, MIDR/MPIDR, etc.
> >
> > Would a summary of your point be that live migration across hardware
> > with different arch counter frequencies is likely to not be possible with
> > 100% fidelity because of these other dependencies on the arch counter
> > frequency? (hence the fixed 1 GHz frequency in ARM v8.6)
>
> Yes.
>
> In addition, there are a larger set of things necessarily exposed to VMs
> that mean that live migration isn't all that practical except betweenm
> identical machines (where the counter frequency should be identical),
> and the timer frequency might just be the canary in the coalmine. For
> example, the cache properties enumerated in CTR_EL0 cannot necessarily
> be emulated on another machine.
>
> > > > For clockevents, the only timer interrupt that Hyper-V provides
> > > > in a guest VM is its virtualized "STIMER" interrupt. There's no
> > > > virtualization of the ARM arch timer in the guest.
> > >
> > > I think that is rather unfortunate, given it's a core architectural
> > > feature. Is it just the interrupt that's missing? i.e. does all the
> > > PE-local functionality behave as the architecture requires?
> >
> > Right off the bat, I don't know about timer-related PE-local
> > functionality as it's not exercised in a Linux VM on Hyper-V that is
> > using STIMER-based clockevents. I'll explore with the Hyper-V
> > team. My impression is that enabling the ARM arch timer in a
> > guest VM is more work for Hyper-V than just wiring up an
> > interrupt.
>
> Thanks for chasing this up!
>
I've had a couple rounds of discussions with the Hyper-V team. For
the clocksource we've agreed to table the live migration discussion, and
I'll resubmit the code so that arm_arch_timer.c provides the
standard arch_sys_counter clocksource. As noted previously, this just
works for a Hyper-V guest. The live migration discussion may come
back later after a deeper investigation by Hyper-V.
For clockevents, there's not a near term fix. It's more than just plumbing
an interrupt for Hyper-V to virtualize the ARM64 arch timer in a guest VM.
From their perspective there's also benefit in having a timer abstraction
that's independent of the architecture, and in the Linux guest, the STIMER
code is common across x86/x64 and ARM64. It follows the standard Linux
clockevents model, as it should. The code is already in use in out-of-tree
builds in the Linux VMs included in Windows 10 on ARM64 as part of the
so-called "Windows Subsystem for Linux".
So I'm hoping we can get this core support for ARM64 guests on Hyper-V
into upstream using the existing STIMER support. At some point, Hyper-V
will do the virtualization of the ARM64 arch timer, but we don't want to
have to stay out-of-tree until after that happens.
Thoughts?
Michael
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [PATCH v10 3/7] arm64: hyperv: Add Hyper-V clocksource/clockevent support
2021-06-08 15:36 ` Michael Kelley
@ 2021-06-10 16:45 ` Mark Rutland
2021-06-14 2:42 ` Michael Kelley
0 siblings, 1 reply; 26+ messages in thread
From: Mark Rutland @ 2021-06-10 16:45 UTC (permalink / raw)
To: Michael Kelley
Cc: will, catalin.marinas, lorenzo.pieralisi, sudeep.holla,
linux-arm-kernel, linux-kernel, linux-hyperv, linux-efi, arnd,
wei.liu, ardb, daniel.lezcano, KY Srinivasan
Hi Michael,
[trimming the bulk of the thrread]
On Tue, Jun 08, 2021 at 03:36:06PM +0000, Michael Kelley wrote:
> I've had a couple rounds of discussions with the Hyper-V team. For
> the clocksource we've agreed to table the live migration discussion, and
> I'll resubmit the code so that arm_arch_timer.c provides the
> standard arch_sys_counter clocksource. As noted previously, this just
> works for a Hyper-V guest. The live migration discussion may come
> back later after a deeper investigation by Hyper-V.
Great; thanks for this!
> For clockevents, there's not a near term fix. It's more than just plumbing
> an interrupt for Hyper-V to virtualize the ARM64 arch timer in a guest VM.
> From their perspective there's also benefit in having a timer abstraction
> that's independent of the architecture, and in the Linux guest, the STIMER
> code is common across x86/x64 and ARM64. It follows the standard Linux
> clockevents model, as it should. The code is already in use in out-of-tree
> builds in the Linux VMs included in Windows 10 on ARM64 as part of the
> so-called "Windows Subsystem for Linux".
>
> So I'm hoping we can get this core support for ARM64 guests on Hyper-V
> into upstream using the existing STIMER support. At some point, Hyper-V
> will do the virtualization of the ARM64 arch timer, but we don't want to
> have to stay out-of-tree until after that happens.
My main concern here is making sure that we can rely on architected
properties, and don't have to special-case architected bits for hyperv
(or any other hypervisor), since that inevitably causes longer-term
pain.
While in abstract I'm not as worried about using the timer
clock_event_device specifically, that same driver provides the
clocksource and the event stream, and I want those to work as usual,
without being tied into the hyperv code. IIUC that will require some
work, since the driver won't register if the GTDT is missing timer
interrupts (or if there is no GTDT).
I think it really depends on what that looks like.
Thanks,
Mark.
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 26+ messages in thread
* RE: [PATCH v10 3/7] arm64: hyperv: Add Hyper-V clocksource/clockevent support
2021-06-10 16:45 ` Mark Rutland
@ 2021-06-14 2:42 ` Michael Kelley
2021-06-16 20:17 ` Michael Kelley
2021-06-22 9:54 ` Mark Rutland
0 siblings, 2 replies; 26+ messages in thread
From: Michael Kelley @ 2021-06-14 2:42 UTC (permalink / raw)
To: Mark Rutland
Cc: will, catalin.marinas, lorenzo.pieralisi, sudeep.holla,
linux-arm-kernel, linux-kernel, linux-hyperv, linux-efi, arnd,
wei.liu, ardb, daniel.lezcano, KY Srinivasan
From: Mark Rutland <mark.rutland@arm.com> Sent: Thursday, June 10, 2021 9:45 AM
>
> Hi Michael,
>
> [trimming the bulk of the thrread]
>
> On Tue, Jun 08, 2021 at 03:36:06PM +0000, Michael Kelley wrote:
> > I've had a couple rounds of discussions with the Hyper-V team. For
> > the clocksource we've agreed to table the live migration discussion, and
> > I'll resubmit the code so that arm_arch_timer.c provides the
> > standard arch_sys_counter clocksource. As noted previously, this just
> > works for a Hyper-V guest. The live migration discussion may come
> > back later after a deeper investigation by Hyper-V.
>
> Great; thanks for this!
>
> > For clockevents, there's not a near term fix. It's more than just plumbing
> > an interrupt for Hyper-V to virtualize the ARM64 arch timer in a guest VM.
> > From their perspective there's also benefit in having a timer abstraction
> > that's independent of the architecture, and in the Linux guest, the STIMER
> > code is common across x86/x64 and ARM64. It follows the standard Linux
> > clockevents model, as it should. The code is already in use in out-of-tree
> > builds in the Linux VMs included in Windows 10 on ARM64 as part of the
> > so-called "Windows Subsystem for Linux".
> >
> > So I'm hoping we can get this core support for ARM64 guests on Hyper-V
> > into upstream using the existing STIMER support. At some point, Hyper-V
> > will do the virtualization of the ARM64 arch timer, but we don't want to
> > have to stay out-of-tree until after that happens.
>
> My main concern here is making sure that we can rely on architected
> properties, and don't have to special-case architected bits for hyperv
> (or any other hypervisor), since that inevitably causes longer-term
> pain.
>
> While in abstract I'm not as worried about using the timer
> clock_event_device specifically, that same driver provides the
> clocksource and the event stream, and I want those to work as usual,
> without being tied into the hyperv code. IIUC that will require some
> work, since the driver won't register if the GTDT is missing timer
> interrupts (or if there is no GTDT).
>
> I think it really depends on what that looks like.
Mark,
Here are the details:
The existing initialization and registration code in arm_arch_timer.c
works in a Hyper-V guest with no changes. As previously mentioned,
the GTDT exists and is correctly populated. Even though it isn't used,
there's a PPI INTID specified for the virtual timer, just so
the "arm_sys_timer" clockevent can be initialized and registered.
The IRQ shows up in the output of "cat /proc/interrupts" with zero counts
for all CPUs since no interrupts are ever generated. The EL1 virtual
timer registers (CNTV_CVAL_EL0, CNTV_TVAL_EL0, and CNTV_CTL_EL0)
are accessible in the VM. The "arm_sys_timer" clockevent is left in
a shutdown state with CNTV_CTL_EL0.ENABLE set to zero when the
Hyper-V STIMER clockevent is registered with a higher rating.
Event streams are initialized and the __delay() implementation
for ARM64 inside the kernel works. However, on the Ampere
eMAG hardware I'm using for testing, the WFE instruction returns
more quickly than it should even though the event stream fields in
CNTKCTL_EL1 are correct. I have a query in to the Hyper-V team
to see if they are trapping WFE and just returning, vs. perhaps the
eMAG processor takes the easy way out and has WFE just return
immediately. I'm not knowledgeable about other uses of timer
event streams, so let me know if there are other usage scenarios
I should check.
Finally, the "arch_sys_counter" clocksource gets initialized and
setup correctly. If the Hyper-V clocksource is also initialized,
you can flip between the two clocksources at runtime as expected.
If the Hyper-V clocksource is not setup, then Linux in the VM runs
fine with the "arch_sys_counter" clocksource.
Michael
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 26+ messages in thread
* RE: [PATCH v10 3/7] arm64: hyperv: Add Hyper-V clocksource/clockevent support
2021-06-14 2:42 ` Michael Kelley
@ 2021-06-16 20:17 ` Michael Kelley
2021-06-22 9:54 ` Mark Rutland
1 sibling, 0 replies; 26+ messages in thread
From: Michael Kelley @ 2021-06-16 20:17 UTC (permalink / raw)
To: Mark Rutland
Cc: will, catalin.marinas, lorenzo.pieralisi, sudeep.holla,
linux-arm-kernel, linux-kernel, linux-hyperv, linux-efi, arnd,
wei.liu, ardb, daniel.lezcano, KY Srinivasan
From: Michael Kelley <mikelley@microsoft.com> Sent: Sunday, June 13, 2021 7:42 PM
>
> From: Mark Rutland <mark.rutland@arm.com> Sent: Thursday, June 10, 2021 9:45 AM
> >
> > Hi Michael,
> >
> > [trimming the bulk of the thrread]
> >
> > On Tue, Jun 08, 2021 at 03:36:06PM +0000, Michael Kelley wrote:
> > > I've had a couple rounds of discussions with the Hyper-V team. For
> > > the clocksource we've agreed to table the live migration discussion, and
> > > I'll resubmit the code so that arm_arch_timer.c provides the
> > > standard arch_sys_counter clocksource. As noted previously, this just
> > > works for a Hyper-V guest. The live migration discussion may come
> > > back later after a deeper investigation by Hyper-V.
> >
> > Great; thanks for this!
> >
> > > For clockevents, there's not a near term fix. It's more than just plumbing
> > > an interrupt for Hyper-V to virtualize the ARM64 arch timer in a guest VM.
> > > From their perspective there's also benefit in having a timer abstraction
> > > that's independent of the architecture, and in the Linux guest, the STIMER
> > > code is common across x86/x64 and ARM64. It follows the standard Linux
> > > clockevents model, as it should. The code is already in use in out-of-tree
> > > builds in the Linux VMs included in Windows 10 on ARM64 as part of the
> > > so-called "Windows Subsystem for Linux".
> > >
> > > So I'm hoping we can get this core support for ARM64 guests on Hyper-V
> > > into upstream using the existing STIMER support. At some point, Hyper-V
> > > will do the virtualization of the ARM64 arch timer, but we don't want to
> > > have to stay out-of-tree until after that happens.
> >
> > My main concern here is making sure that we can rely on architected
> > properties, and don't have to special-case architected bits for hyperv
> > (or any other hypervisor), since that inevitably causes longer-term
> > pain.
> >
> > While in abstract I'm not as worried about using the timer
> > clock_event_device specifically, that same driver provides the
> > clocksource and the event stream, and I want those to work as usual,
> > without being tied into the hyperv code. IIUC that will require some
> > work, since the driver won't register if the GTDT is missing timer
> > interrupts (or if there is no GTDT).
> >
> > I think it really depends on what that looks like.
>
> Mark,
>
> Here are the details:
>
> The existing initialization and registration code in arm_arch_timer.c
> works in a Hyper-V guest with no changes. As previously mentioned,
> the GTDT exists and is correctly populated. Even though it isn't used,
> there's a PPI INTID specified for the virtual timer, just so
> the "arm_sys_timer" clockevent can be initialized and registered.
> The IRQ shows up in the output of "cat /proc/interrupts" with zero counts
> for all CPUs since no interrupts are ever generated. The EL1 virtual
> timer registers (CNTV_CVAL_EL0, CNTV_TVAL_EL0, and CNTV_CTL_EL0)
> are accessible in the VM. The "arm_sys_timer" clockevent is left in
> a shutdown state with CNTV_CTL_EL0.ENABLE set to zero when the
> Hyper-V STIMER clockevent is registered with a higher rating.
>
> Event streams are initialized and the __delay() implementation
> for ARM64 inside the kernel works. However, on the Ampere
> eMAG hardware I'm using for testing, the WFE instruction returns
> more quickly than it should even though the event stream fields in
> CNTKCTL_EL1 are correct. I have a query in to the Hyper-V team
> to see if they are trapping WFE and just returning, vs. perhaps the
> eMAG processor takes the easy way out and has WFE just return
> immediately. I'm not knowledgeable about other uses of timer
> event streams, so let me know if there are other usage scenarios
> I should check.
I confirmed that Hyper-V is not trapping the WFE instruction. And
on a Marvell TX2 and on an Ampere Altra, the counter event stream
and WFE provide the expected delay. Evidently WFE on the eMAG
doesn't actually delay. Bottom line: event streams work as expected
in a Hyper-V VM. No changes needed to arm_arch_timer.[ch].
Michael
>
> Finally, the "arch_sys_counter" clocksource gets initialized and
> setup correctly. If the Hyper-V clocksource is also initialized,
> you can flip between the two clocksources at runtime as expected.
> If the Hyper-V clocksource is not setup, then Linux in the VM runs
> fine with the "arch_sys_counter" clocksource.
>
> Michael
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [PATCH v10 3/7] arm64: hyperv: Add Hyper-V clocksource/clockevent support
2021-06-14 2:42 ` Michael Kelley
2021-06-16 20:17 ` Michael Kelley
@ 2021-06-22 9:54 ` Mark Rutland
2021-06-23 8:56 ` Marc Zyngier
1 sibling, 1 reply; 26+ messages in thread
From: Mark Rutland @ 2021-06-22 9:54 UTC (permalink / raw)
To: Michael Kelley, Marc Zyngier
Cc: will, catalin.marinas, lorenzo.pieralisi, sudeep.holla,
linux-arm-kernel, linux-kernel, linux-hyperv, linux-efi, arnd,
wei.liu, ardb, daniel.lezcano, KY Srinivasan
Hi Michael,
Thanks for all this; comments inline below. I've added Marc Zyngier, who
co-maintains the architected timer code.
On Mon, Jun 14, 2021 at 02:42:23AM +0000, Michael Kelley wrote:
> From: Mark Rutland <mark.rutland@arm.com> Sent: Thursday, June 10, 2021 9:45 AM
> > On Tue, Jun 08, 2021 at 03:36:06PM +0000, Michael Kelley wrote:
> > > I've had a couple rounds of discussions with the Hyper-V team. For
> > > the clocksource we've agreed to table the live migration discussion, and
> > > I'll resubmit the code so that arm_arch_timer.c provides the
> > > standard arch_sys_counter clocksource. As noted previously, this just
> > > works for a Hyper-V guest. The live migration discussion may come
> > > back later after a deeper investigation by Hyper-V.
> >
> > Great; thanks for this!
> >
> > > For clockevents, there's not a near term fix. It's more than just plumbing
> > > an interrupt for Hyper-V to virtualize the ARM64 arch timer in a guest VM.
> > > From their perspective there's also benefit in having a timer abstraction
> > > that's independent of the architecture, and in the Linux guest, the STIMER
> > > code is common across x86/x64 and ARM64. It follows the standard Linux
> > > clockevents model, as it should. The code is already in use in out-of-tree
> > > builds in the Linux VMs included in Windows 10 on ARM64 as part of the
> > > so-called "Windows Subsystem for Linux".
> > >
> > > So I'm hoping we can get this core support for ARM64 guests on Hyper-V
> > > into upstream using the existing STIMER support. At some point, Hyper-V
> > > will do the virtualization of the ARM64 arch timer, but we don't want to
> > > have to stay out-of-tree until after that happens.
> >
> > My main concern here is making sure that we can rely on architected
> > properties, and don't have to special-case architected bits for hyperv
> > (or any other hypervisor), since that inevitably causes longer-term
> > pain.
> >
> > While in abstract I'm not as worried about using the timer
> > clock_event_device specifically, that same driver provides the
> > clocksource and the event stream, and I want those to work as usual,
> > without being tied into the hyperv code. IIUC that will require some
> > work, since the driver won't register if the GTDT is missing timer
> > interrupts (or if there is no GTDT).
> >
> > I think it really depends on what that looks like.
>
> Mark,
>
> Here are the details:
>
> The existing initialization and registration code in arm_arch_timer.c
> works in a Hyper-V guest with no changes. As previously mentioned,
> the GTDT exists and is correctly populated. Even though it isn't used,
> there's a PPI INTID specified for the virtual timer, just so
> the "arm_sys_timer" clockevent can be initialized and registered.
> The IRQ shows up in the output of "cat /proc/interrupts" with zero counts
> for all CPUs since no interrupts are ever generated. The EL1 virtual
> timer registers (CNTV_CVAL_EL0, CNTV_TVAL_EL0, and CNTV_CTL_EL0)
> are accessible in the VM. The "arm_sys_timer" clockevent is left in
> a shutdown state with CNTV_CTL_EL0.ENABLE set to zero when the
> Hyper-V STIMER clockevent is registered with a higher rating.
This concerns me, since we're lying to the kernel, and assuming that it
will never try to use this timer. I appreciate that evidently we don't
happen to rely on that today if you register a higher priority timer,
but that does open us up to future fragility (e.g. if we added sanity
checks when registering timers), and IIRC there are ways for userspace
to change the clockevent device today.
> Event streams are initialized and the __delay() implementation
> for ARM64 inside the kernel works. However, on the Ampere
> eMAG hardware I'm using for testing, the WFE instruction returns
> more quickly than it should even though the event stream fields in
> CNTKCTL_EL1 are correct. I have a query in to the Hyper-V team
> to see if they are trapping WFE and just returning, vs. perhaps the
> eMAG processor takes the easy way out and has WFE just return
> immediately. I'm not knowledgeable about other uses of timer
> event streams, so let me know if there are other usage scenarios
> I should check.
I saw your reply confirming that this is gnerally working as expected
(and that Hyper-V is not trapping WFE) so this sounds fine to me.
> Finally, the "arch_sys_counter" clocksource gets initialized and
> setup correctly. If the Hyper-V clocksource is also initialized,
> you can flip between the two clocksources at runtime as expected.
> If the Hyper-V clocksource is not setup, then Linux in the VM runs
> fine with the "arch_sys_counter" clocksource.
Great!
As above, my remaining concern here is fragility around the
clockevent_device; I'm not keen that we're lying (in the GTDT) that
interrupts are wired up when they not functional, and while you can get
away with that today, that relies on kernel implementation details that
could change.
Ideally, Hyper-V would provide the architectural timer (as it's already
claiming to in the GTDT), things would "just work", and the Hyper-V
timer would be an optimization rather than a functional necessity.
You mentioned above that Hyper-V will virtualize the timer "at some
point" -- is that already planned, and when is that likely to be?
Marc, do you have any thoughts on this?
Thanks,
Mark.
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [PATCH v10 3/7] arm64: hyperv: Add Hyper-V clocksource/clockevent support
2021-06-22 9:54 ` Mark Rutland
@ 2021-06-23 8:56 ` Marc Zyngier
2021-06-28 2:21 ` Michael Kelley
0 siblings, 1 reply; 26+ messages in thread
From: Marc Zyngier @ 2021-06-23 8:56 UTC (permalink / raw)
To: Mark Rutland
Cc: Michael Kelley, will, catalin.marinas, lorenzo.pieralisi,
sudeep.holla, linux-arm-kernel, linux-kernel, linux-hyperv,
linux-efi, arnd, wei.liu, ardb, daniel.lezcano, KY Srinivasan
On Tue, 22 Jun 2021 10:54:12 +0100,
Mark Rutland <mark.rutland@arm.com> wrote:
>
> Hi Michael,
>
> Thanks for all this; comments inline below. I've added Marc Zyngier, who
> co-maintains the architected timer code.
>
> On Mon, Jun 14, 2021 at 02:42:23AM +0000, Michael Kelley wrote:
> > From: Mark Rutland <mark.rutland@arm.com> Sent: Thursday, June 10, 2021 9:45 AM
> > > On Tue, Jun 08, 2021 at 03:36:06PM +0000, Michael Kelley wrote:
> > > > I've had a couple rounds of discussions with the Hyper-V team. For
> > > > the clocksource we've agreed to table the live migration discussion, and
> > > > I'll resubmit the code so that arm_arch_timer.c provides the
> > > > standard arch_sys_counter clocksource. As noted previously, this just
> > > > works for a Hyper-V guest. The live migration discussion may come
> > > > back later after a deeper investigation by Hyper-V.
> > >
> > > Great; thanks for this!
> > >
> > > > For clockevents, there's not a near term fix. It's more than just plumbing
> > > > an interrupt for Hyper-V to virtualize the ARM64 arch timer in a guest VM.
> > > > From their perspective there's also benefit in having a timer abstraction
> > > > that's independent of the architecture, and in the Linux guest, the STIMER
> > > > code is common across x86/x64 and ARM64. It follows the standard Linux
> > > > clockevents model, as it should. The code is already in use in out-of-tree
> > > > builds in the Linux VMs included in Windows 10 on ARM64 as part of the
> > > > so-called "Windows Subsystem for Linux".
> > > >
> > > > So I'm hoping we can get this core support for ARM64 guests on Hyper-V
> > > > into upstream using the existing STIMER support. At some point, Hyper-V
> > > > will do the virtualization of the ARM64 arch timer, but we don't want to
> > > > have to stay out-of-tree until after that happens.
> > >
> > > My main concern here is making sure that we can rely on architected
> > > properties, and don't have to special-case architected bits for hyperv
> > > (or any other hypervisor), since that inevitably causes longer-term
> > > pain.
> > >
> > > While in abstract I'm not as worried about using the timer
> > > clock_event_device specifically, that same driver provides the
> > > clocksource and the event stream, and I want those to work as usual,
> > > without being tied into the hyperv code. IIUC that will require some
> > > work, since the driver won't register if the GTDT is missing timer
> > > interrupts (or if there is no GTDT).
> > >
> > > I think it really depends on what that looks like.
> >
> > Mark,
> >
> > Here are the details:
> >
> > The existing initialization and registration code in arm_arch_timer.c
> > works in a Hyper-V guest with no changes. As previously mentioned,
> > the GTDT exists and is correctly populated. Even though it isn't used,
> > there's a PPI INTID specified for the virtual timer, just so
> > the "arm_sys_timer" clockevent can be initialized and registered.
> > The IRQ shows up in the output of "cat /proc/interrupts" with zero counts
> > for all CPUs since no interrupts are ever generated. The EL1 virtual
> > timer registers (CNTV_CVAL_EL0, CNTV_TVAL_EL0, and CNTV_CTL_EL0)
> > are accessible in the VM. The "arm_sys_timer" clockevent is left in
> > a shutdown state with CNTV_CTL_EL0.ENABLE set to zero when the
> > Hyper-V STIMER clockevent is registered with a higher rating.
>
> This concerns me, since we're lying to the kernel, and assuming that it
> will never try to use this timer. I appreciate that evidently we don't
> happen to rely on that today if you register a higher priority timer,
> but that does open us up to future fragility (e.g. if we added sanity
> checks when registering timers), and IIRC there are ways for userspace
> to change the clockevent device today.
Indeed. Userspace can perfectly unbind the clockevent using
/sys/devices/system/clockevents/clockevent*/unbind_device, and the
kernel will be happy to switch to the next per-cpu timer, which
happens to be the arch timer. Oh wait...
>
> > Event streams are initialized and the __delay() implementation
> > for ARM64 inside the kernel works. However, on the Ampere
> > eMAG hardware I'm using for testing, the WFE instruction returns
> > more quickly than it should even though the event stream fields in
> > CNTKCTL_EL1 are correct. I have a query in to the Hyper-V team
> > to see if they are trapping WFE and just returning, vs. perhaps the
> > eMAG processor takes the easy way out and has WFE just return
> > immediately. I'm not knowledgeable about other uses of timer
> > event streams, so let me know if there are other usage scenarios
> > I should check.
>
> I saw your reply confirming that this is gnerally working as expected
> (and that Hyper-V is not trapping WFE) so this sounds fine to me.
>
> > Finally, the "arch_sys_counter" clocksource gets initialized and
> > setup correctly. If the Hyper-V clocksource is also initialized,
> > you can flip between the two clocksources at runtime as expected.
> > If the Hyper-V clocksource is not setup, then Linux in the VM runs
> > fine with the "arch_sys_counter" clocksource.
>
> Great!
>
> As above, my remaining concern here is fragility around the
> clockevent_device; I'm not keen that we're lying (in the GTDT) that
> interrupts are wired up when they not functional, and while you can get
> away with that today, that relies on kernel implementation details that
> could change.
>
> Ideally, Hyper-V would provide the architectural timer (as it's already
> claiming to in the GTDT), things would "just work", and the Hyper-V
> timer would be an optimization rather than a functional necessity.
>
> You mentioned above that Hyper-V will virtualize the timer "at some
> point" -- is that already planned, and when is that likely to be?
>
> Marc, do you have any thoughts on this?
Overall, lying to the kernel is a bad idea. Only implementing half of
the architecture is another bad idea. I doubt the combination of two
bad ideas produces a good one.
If Hyper-V guests need to use another timer (for migration purposes?),
that's fine. But we rely on both the base architecture to be
completely implemented *and* on advertised features to be functional.
I think this has been our position since the first Hyper-V patches
were posted... 3 years ago?
What is the hold up for reliably virtualising the arch timer,
including interrupt delivery?
Thanks,
M.
--
Without deviation from the norm, progress is not possible.
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 26+ messages in thread
* RE: [PATCH v10 3/7] arm64: hyperv: Add Hyper-V clocksource/clockevent support
2021-06-23 8:56 ` Marc Zyngier
@ 2021-06-28 2:21 ` Michael Kelley
0 siblings, 0 replies; 26+ messages in thread
From: Michael Kelley @ 2021-06-28 2:21 UTC (permalink / raw)
To: Marc Zyngier, Mark Rutland
Cc: will, catalin.marinas, lorenzo.pieralisi, sudeep.holla,
linux-arm-kernel, linux-kernel, linux-hyperv, linux-efi, arnd,
wei.liu, ardb, daniel.lezcano, KY Srinivasan
From: Marc Zyngier <maz@kernel.org> Sent: Wednesday, June 23, 2021 1:56 AM
>
> On Tue, 22 Jun 2021 10:54:12 +0100,
> Mark Rutland <mark.rutland@arm.com> wrote:
> >
> > Hi Michael,
> >
> > Thanks for all this; comments inline below. I've added Marc Zyngier, who
> > co-maintains the architected timer code.
> >
> > On Mon, Jun 14, 2021 at 02:42:23AM +0000, Michael Kelley wrote:
> > > From: Mark Rutland <mark.rutland@arm.com> Sent: Thursday, June 10, 2021 9:45 AM
> > > > On Tue, Jun 08, 2021 at 03:36:06PM +0000, Michael Kelley wrote:
> > > > > I've had a couple rounds of discussions with the Hyper-V team. For
> > > > > the clocksource we've agreed to table the live migration discussion, and
> > > > > I'll resubmit the code so that arm_arch_timer.c provides the
> > > > > standard arch_sys_counter clocksource. As noted previously, this just
> > > > > works for a Hyper-V guest. The live migration discussion may come
> > > > > back later after a deeper investigation by Hyper-V.
> > > >
> > > > Great; thanks for this!
> > > >
> > > > > For clockevents, there's not a near term fix. It's more than just plumbing
> > > > > an interrupt for Hyper-V to virtualize the ARM64 arch timer in a guest VM.
> > > > > From their perspective there's also benefit in having a timer abstraction
> > > > > that's independent of the architecture, and in the Linux guest, the STIMER
> > > > > code is common across x86/x64 and ARM64. It follows the standard Linux
> > > > > clockevents model, as it should. The code is already in use in out-of-tree
> > > > > builds in the Linux VMs included in Windows 10 on ARM64 as part of the
> > > > > so-called "Windows Subsystem for Linux".
> > > > >
> > > > > So I'm hoping we can get this core support for ARM64 guests on Hyper-V
> > > > > into upstream using the existing STIMER support. At some point, Hyper-V
> > > > > will do the virtualization of the ARM64 arch timer, but we don't want to
> > > > > have to stay out-of-tree until after that happens.
> > > >
> > > > My main concern here is making sure that we can rely on architected
> > > > properties, and don't have to special-case architected bits for hyperv
> > > > (or any other hypervisor), since that inevitably causes longer-term
> > > > pain.
> > > >
> > > > While in abstract I'm not as worried about using the timer
> > > > clock_event_device specifically, that same driver provides the
> > > > clocksource and the event stream, and I want those to work as usual,
> > > > without being tied into the hyperv code. IIUC that will require some
> > > > work, since the driver won't register if the GTDT is missing timer
> > > > interrupts (or if there is no GTDT).
> > > >
> > > > I think it really depends on what that looks like.
> > >
> > > Mark,
> > >
> > > Here are the details:
> > >
> > > The existing initialization and registration code in arm_arch_timer.c
> > > works in a Hyper-V guest with no changes. As previously mentioned,
> > > the GTDT exists and is correctly populated. Even though it isn't used,
> > > there's a PPI INTID specified for the virtual timer, just so
> > > the "arm_sys_timer" clockevent can be initialized and registered.
> > > The IRQ shows up in the output of "cat /proc/interrupts" with zero counts
> > > for all CPUs since no interrupts are ever generated. The EL1 virtual
> > > timer registers (CNTV_CVAL_EL0, CNTV_TVAL_EL0, and CNTV_CTL_EL0)
> > > are accessible in the VM. The "arm_sys_timer" clockevent is left in
> > > a shutdown state with CNTV_CTL_EL0.ENABLE set to zero when the
> > > Hyper-V STIMER clockevent is registered with a higher rating.
> >
> > This concerns me, since we're lying to the kernel, and assuming that it
> > will never try to use this timer. I appreciate that evidently we don't
> > happen to rely on that today if you register a higher priority timer,
> > but that does open us up to future fragility (e.g. if we added sanity
> > checks when registering timers), and IIRC there are ways for userspace
> > to change the clockevent device today.
>
> Indeed. Userspace can perfectly unbind the clockevent using
> /sys/devices/system/clockevents/clockevent*/unbind_device, and the
> kernel will be happy to switch to the next per-cpu timer, which
> happens to be the arch timer. Oh wait...
>
> >
> > > Event streams are initialized and the __delay() implementation
> > > for ARM64 inside the kernel works. However, on the Ampere
> > > eMAG hardware I'm using for testing, the WFE instruction returns
> > > more quickly than it should even though the event stream fields in
> > > CNTKCTL_EL1 are correct. I have a query in to the Hyper-V team
> > > to see if they are trapping WFE and just returning, vs. perhaps the
> > > eMAG processor takes the easy way out and has WFE just return
> > > immediately. I'm not knowledgeable about other uses of timer
> > > event streams, so let me know if there are other usage scenarios
> > > I should check.
> >
> > I saw your reply confirming that this is gnerally working as expected
> > (and that Hyper-V is not trapping WFE) so this sounds fine to me.
> >
> > > Finally, the "arch_sys_counter" clocksource gets initialized and
> > > setup correctly. If the Hyper-V clocksource is also initialized,
> > > you can flip between the two clocksources at runtime as expected.
> > > If the Hyper-V clocksource is not setup, then Linux in the VM runs
> > > fine with the "arch_sys_counter" clocksource.
> >
> > Great!
> >
> > As above, my remaining concern here is fragility around the
> > clockevent_device; I'm not keen that we're lying (in the GTDT) that
> > interrupts are wired up when they not functional, and while you can get
> > away with that today, that relies on kernel implementation details that
> > could change.
> >
> > Ideally, Hyper-V would provide the architectural timer (as it's already
> > claiming to in the GTDT), things would "just work", and the Hyper-V
> > timer would be an optimization rather than a functional necessity.
> >
> > You mentioned above that Hyper-V will virtualize the timer "at some
> > point" -- is that already planned, and when is that likely to be?
> >
> > Marc, do you have any thoughts on this?
>
> Overall, lying to the kernel is a bad idea. Only implementing half of
> the architecture is another bad idea. I doubt the combination of two
> bad ideas produces a good one.
>
> If Hyper-V guests need to use another timer (for migration purposes?),
> that's fine. But we rely on both the base architecture to be
> completely implemented *and* on advertised features to be functional.
> I think this has been our position since the first Hyper-V patches
> were posted... 3 years ago?
>
> What is the hold up for reliably virtualising the arch timer,
> including interrupt delivery?
Marc (and Mark) --
In our early interactions about the Hyper-V clocks and timers, the code
was a bit spread out, and you suggested moving all the clocksource
and clockevent stuff to a driver under drivers/clocksource. See
https://lore.kernel.org/lkml/e0374a07-809c-cabd-2eb6-e6b5ad84742e@arm.com/.
That was a good change independent of any ARM64 considerations,
but I read (or perhaps overread) your comments to say that it was OK
to use these Hyper-V para-virtualized clocks/timers instead of the ARM64
architectural ones in a Hyper-V VM. They work and it's what the Hyper-V
guys wanted to do anyway, so having Hyper-V offer the ARM64 arch
counter and timer in a VM hasn't been a priority. They had other stuff that
didn't work at all on ARM64, so that's where their attention went.
I agree that it would be better to have the ARM64 arch counter/timer
fully implemented in a Hyper-V VM. But we're wanting to find a practical
way to move forward, and in doing so confine any rough edges to Hyper-V
VMs and the Hyper-V specific code in the kernel tree. We're maintaining
and shipping the code out-of-tree based on Hyper-V ARM64 current behavior
and would like to get this core enablement code upstream. Sure, unbinding
the Hyper-V clockevent doesn't work, but that's not a problem in any use
cases we see from customers.
All that said, our discussions with the Hyper-V team are continuing. We're
still in the process of seeing what's practical to get and when.
Michael
>
> Thanks,
>
> M.
>
> --
> Without deviation from the norm, progress is not possible.
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 26+ messages in thread
* [PATCH v10 4/7] arm64: hyperv: Add kexec and panic handlers
2021-05-12 17:37 [PATCH v10 0/7] Enable Linux guests on Hyper-V on ARM64 Michael Kelley
` (2 preceding siblings ...)
2021-05-12 17:37 ` [PATCH v10 3/7] arm64: hyperv: Add Hyper-V clocksource/clockevent support Michael Kelley
@ 2021-05-12 17:37 ` Michael Kelley
2021-05-12 17:37 ` [PATCH v10 5/7] arm64: hyperv: Initialize hypervisor on boot Michael Kelley
` (3 subsequent siblings)
7 siblings, 0 replies; 26+ messages in thread
From: Michael Kelley @ 2021-05-12 17:37 UTC (permalink / raw)
To: will, catalin.marinas, mark.rutland, lorenzo.pieralisi,
sudeep.holla, linux-arm-kernel, linux-kernel, linux-hyperv,
linux-efi, arnd, wei.liu, ardb, daniel.lezcano, kys
Cc: mikelley
Add function to inform Hyper-V about a guest panic.
Also add functions to set up and remove kexec and panic
handlers, which are currently unused on ARM64 but are
called from architecture independent code in the VMbus
driver.
This code is built only when CONFIG_HYPERV is enabled.
Signed-off-by: Michael Kelley <mikelley@microsoft.com>
Reviewed-by: Sunil Muthuswamy <sunilmut@microsoft.com>
---
arch/arm64/hyperv/Makefile | 2 +-
arch/arm64/hyperv/hv_core.c | 52 ++++++++++++++++++++++++++++++++++++++++++
arch/arm64/hyperv/mshyperv.c | 54 ++++++++++++++++++++++++++++++++++++++++++++
3 files changed, 107 insertions(+), 1 deletion(-)
create mode 100644 arch/arm64/hyperv/mshyperv.c
diff --git a/arch/arm64/hyperv/Makefile b/arch/arm64/hyperv/Makefile
index 1697d30..87c31c0 100644
--- a/arch/arm64/hyperv/Makefile
+++ b/arch/arm64/hyperv/Makefile
@@ -1,2 +1,2 @@
# SPDX-License-Identifier: GPL-2.0
-obj-y := hv_core.o
+obj-y := hv_core.o mshyperv.o
diff --git a/arch/arm64/hyperv/hv_core.c b/arch/arm64/hyperv/hv_core.c
index 34004a5..56391f3 100644
--- a/arch/arm64/hyperv/hv_core.c
+++ b/arch/arm64/hyperv/hv_core.c
@@ -128,3 +128,55 @@ u64 hv_get_vpreg(u32 msr)
return output.as64.low;
}
EXPORT_SYMBOL_GPL(hv_get_vpreg);
+
+/*
+ * hyperv_report_panic - report a panic to Hyper-V. This function uses
+ * the older version of the Hyper-V interface that admittedly doesn't
+ * pass enough information to be useful beyond just recording the
+ * occurrence of a panic. The parallel hv_kmsg_dump() uses the
+ * new interface that allows reporting 4 Kbytes of data, which is much
+ * more useful. Hyper-V on ARM64 always supports the newer interface, but
+ * we retain support for the older version because the sysadmin is allowed
+ * to disable the newer version via sysctl in case of information security
+ * concerns about the more verbose version.
+ */
+void hyperv_report_panic(struct pt_regs *regs, long err, bool in_die)
+{
+ static bool panic_reported;
+ u64 guest_id;
+
+ /* Don't report a panic to Hyper-V if we're not going to panic */
+ if (in_die && !panic_on_oops)
+ return;
+
+ /*
+ * We prefer to report panic on 'die' chain as we have proper
+ * registers to report, but if we miss it (e.g. on BUG()) we need
+ * to report it on 'panic'.
+ *
+ * Calling code in the 'die' and 'panic' paths ensures that only
+ * one CPU is running this code, so no atomicity is needed.
+ */
+ if (panic_reported)
+ return;
+ panic_reported = true;
+
+ guest_id = hv_get_vpreg(HV_REGISTER_GUEST_OSID);
+
+ /*
+ * Hyper-V provides the ability to store only 5 values.
+ * Pick the passed in error value, the guest_id, and the PC.
+ * The first two general registers are added arbitrarily.
+ */
+ hv_set_vpreg(HV_REGISTER_CRASH_P0, err);
+ hv_set_vpreg(HV_REGISTER_CRASH_P1, guest_id);
+ hv_set_vpreg(HV_REGISTER_CRASH_P2, regs->pc);
+ hv_set_vpreg(HV_REGISTER_CRASH_P3, regs->regs[0]);
+ hv_set_vpreg(HV_REGISTER_CRASH_P4, regs->regs[1]);
+
+ /*
+ * Let Hyper-V know there is crash data available
+ */
+ hv_set_vpreg(HV_REGISTER_CRASH_CTL, HV_CRASH_CTL_CRASH_NOTIFY);
+}
+EXPORT_SYMBOL_GPL(hyperv_report_panic);
diff --git a/arch/arm64/hyperv/mshyperv.c b/arch/arm64/hyperv/mshyperv.c
new file mode 100644
index 0000000..d202b4c
--- /dev/null
+++ b/arch/arm64/hyperv/mshyperv.c
@@ -0,0 +1,54 @@
+// SPDX-License-Identifier: GPL-2.0
+
+/*
+ * Core routines for interacting with Microsoft's Hyper-V hypervisor.
+ * Includes hypervisor initialization, and handling of crashes and
+ * kexecs through a set of static "handler" variables set by the
+ * architecture independent VMbus driver.
+ *
+ * Copyright (C) 2021, Microsoft, Inc.
+ *
+ * Author : Michael Kelley <mikelley@microsoft.com>
+ */
+
+#include <linux/types.h>
+#include <linux/export.h>
+#include <linux/ptrace.h>
+
+/*
+ * The VMbus handler functions are no-ops on ARM64 because
+ * VMbus interrupts are handled as percpu IRQs.
+ */
+void hv_setup_vmbus_handler(void (*handler)(void))
+{
+}
+EXPORT_SYMBOL_GPL(hv_setup_vmbus_handler);
+
+void hv_remove_vmbus_handler(void)
+{
+}
+EXPORT_SYMBOL_GPL(hv_remove_vmbus_handler);
+
+/*
+ * The kexec and crash handler functions are
+ * currently no-ops on ARM64.
+ */
+void hv_setup_kexec_handler(void (*handler)(void))
+{
+}
+EXPORT_SYMBOL_GPL(hv_setup_kexec_handler);
+
+void hv_remove_kexec_handler(void)
+{
+}
+EXPORT_SYMBOL_GPL(hv_remove_kexec_handler);
+
+void hv_setup_crash_handler(void (*handler)(struct pt_regs *regs))
+{
+}
+EXPORT_SYMBOL_GPL(hv_setup_crash_handler);
+
+void hv_remove_crash_handler(void)
+{
+}
+EXPORT_SYMBOL_GPL(hv_remove_crash_handler);
--
1.8.3.1
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 26+ messages in thread* [PATCH v10 5/7] arm64: hyperv: Initialize hypervisor on boot
2021-05-12 17:37 [PATCH v10 0/7] Enable Linux guests on Hyper-V on ARM64 Michael Kelley
` (3 preceding siblings ...)
2021-05-12 17:37 ` [PATCH v10 4/7] arm64: hyperv: Add kexec and panic handlers Michael Kelley
@ 2021-05-12 17:37 ` Michael Kelley
2021-05-13 15:16 ` Wei Liu
2021-05-12 17:37 ` [PATCH v10 6/7] arm64: efi: Export screen_info Michael Kelley
` (2 subsequent siblings)
7 siblings, 1 reply; 26+ messages in thread
From: Michael Kelley @ 2021-05-12 17:37 UTC (permalink / raw)
To: will, catalin.marinas, mark.rutland, lorenzo.pieralisi,
sudeep.holla, linux-arm-kernel, linux-kernel, linux-hyperv,
linux-efi, arnd, wei.liu, ardb, daniel.lezcano, kys
Cc: mikelley
Add ARM64-specific code to initialize the Hyper-V
hypervisor when booting as a guest VM. Provide functions
and data structures indicating hypervisor status that
are needed by VMbus driver.
This code is built only when CONFIG_HYPERV is enabled.
Signed-off-by: Michael Kelley <mikelley@microsoft.com>
---
arch/arm64/hyperv/mshyperv.c | 152 ++++++++++++++++++++++++++++++++++++++
arch/arm64/include/asm/mshyperv.h | 6 ++
arch/arm64/kernel/setup.c | 4 +
3 files changed, 162 insertions(+)
diff --git a/arch/arm64/hyperv/mshyperv.c b/arch/arm64/hyperv/mshyperv.c
index d202b4c..95f7e4e 100644
--- a/arch/arm64/hyperv/mshyperv.c
+++ b/arch/arm64/hyperv/mshyperv.c
@@ -14,6 +14,158 @@
#include <linux/types.h>
#include <linux/export.h>
#include <linux/ptrace.h>
+#include <linux/errno.h>
+#include <linux/acpi.h>
+#include <linux/version.h>
+#include <linux/cpuhotplug.h>
+#include <linux/slab.h>
+#include <linux/cpumask.h>
+#include <asm/mshyperv.h>
+
+static bool hyperv_initialized;
+struct ms_hyperv_info ms_hyperv __ro_after_init;
+EXPORT_SYMBOL_GPL(ms_hyperv);
+
+bool hv_root_partition;
+EXPORT_SYMBOL_GPL(hv_root_partition);
+
+u32 *hv_vp_index;
+EXPORT_SYMBOL_GPL(hv_vp_index);
+
+u32 hv_max_vp_index;
+EXPORT_SYMBOL_GPL(hv_max_vp_index);
+
+void __percpu **hyperv_pcpu_input_arg;
+EXPORT_SYMBOL_GPL(hyperv_pcpu_input_arg);
+
+static int hv_cpu_init(unsigned int cpu)
+{
+ void **input_arg = (void **)this_cpu_ptr(hyperv_pcpu_input_arg);
+
+ hv_vp_index[cpu] = hv_get_vpreg(HV_REGISTER_VP_INDEX);
+
+ *input_arg = kmalloc(HV_HYP_PAGE_SIZE, GFP_KERNEL);
+ if (!(*input_arg))
+ return -ENOMEM;
+
+ return 0;
+}
+
+void __init hyperv_early_init(void)
+{
+ struct hv_get_vp_registers_output result;
+ u32 a, b, c, d;
+ u64 guest_id;
+
+ /*
+ * If we're in a VM on Hyper-V, the ACPI hypervisor_id field will
+ * have the string "MsHyperV".
+ */
+ if (strncmp((char *)&acpi_gbl_FADT.hypervisor_id, "MsHyperV", 8))
+ return;
+
+ /* Setup the guest ID */
+ guest_id = generate_guest_id(0, LINUX_VERSION_CODE, 0);
+ hv_set_vpreg(HV_REGISTER_GUEST_OSID, guest_id);
+
+ /* Get the features and hints from Hyper-V */
+ hv_get_vpreg_128(HV_REGISTER_FEATURES, &result);
+ ms_hyperv.features = result.as32.a;
+ ms_hyperv.priv_high = result.as32.b;
+ ms_hyperv.misc_features = result.as32.c;
+
+ hv_get_vpreg_128(HV_REGISTER_ENLIGHTENMENTS, &result);
+ ms_hyperv.hints = result.as32.a;
+
+ pr_info("Hyper-V: privilege flags low 0x%x, high 0x%x, hints 0x%x, misc 0x%x\n",
+ ms_hyperv.features, ms_hyperv.priv_high, ms_hyperv.hints,
+ ms_hyperv.misc_features);
+
+ /*
+ * If Hyper-V has crash notifications, set crash_kexec_post_notifiers
+ * so that we will report the panic to Hyper-V before running kdump.
+ */
+ if (ms_hyperv.misc_features & HV_FEATURE_GUEST_CRASH_MSR_AVAILABLE)
+ crash_kexec_post_notifiers = true;
+
+ /* Get information about the Hyper-V host version */
+ hv_get_vpreg_128(HV_REGISTER_HYPERVISOR_VERSION, &result);
+ a = result.as32.a;
+ b = result.as32.b;
+ c = result.as32.c;
+ d = result.as32.d;
+ pr_info("Hyper-V: Host Build %d.%d.%d.%d-%d-%d\n",
+ b >> 16, b & 0xFFFF, a, d & 0xFFFFFF, c, d >> 24);
+
+ hyperv_initialized = true;
+}
+
+static int __init hyperv_init(void)
+{
+ int i, ret;
+
+ hyperv_pcpu_input_arg = alloc_percpu(void *);
+ if (!hyperv_pcpu_input_arg)
+ return -ENOMEM;
+
+ /* Allocate and initialize percpu VP index array */
+ hv_max_vp_index = num_possible_cpus();
+ hv_vp_index = kmalloc_array(hv_max_vp_index, sizeof(*hv_vp_index),
+ GFP_KERNEL);
+ if (!hv_vp_index) {
+ ret = -ENOMEM;
+ goto free_input_arg;
+ }
+
+ for (i = 0; i < hv_max_vp_index; i++)
+ hv_vp_index[i] = VP_INVAL;
+
+ if (cpuhp_setup_state(CPUHP_AP_ONLINE_DYN, "arm64/hyperv_init:online",
+ hv_cpu_init, NULL) < 0) {
+ ret = -EINVAL;
+ goto free_vp_index;
+ }
+
+ return 0;
+
+free_vp_index:
+ kfree(hv_vp_index);
+ hv_vp_index = NULL;
+
+free_input_arg:
+ hv_max_vp_index = 0;
+ free_percpu(hyperv_pcpu_input_arg);
+ hyperv_pcpu_input_arg = NULL;
+ return ret;
+}
+
+early_initcall(hyperv_init);
+
+/* This routine is called before kexec/kdump. It does required cleanup. */
+void hyperv_cleanup(void)
+{
+ hv_set_vpreg(HV_REGISTER_GUEST_OSID, 0);
+
+}
+EXPORT_SYMBOL_GPL(hyperv_cleanup);
+
+bool hv_is_hyperv_initialized(void)
+{
+ return hyperv_initialized;
+}
+EXPORT_SYMBOL_GPL(hv_is_hyperv_initialized);
+
+bool hv_is_hibernation_supported(void)
+{
+ return false;
+}
+EXPORT_SYMBOL_GPL(hv_is_hibernation_supported);
+
+bool hv_is_isolation_supported(void)
+{
+ return false;
+}
+EXPORT_SYMBOL_GPL(hv_is_isolation_supported);
/*
* The VMbus handler functions are no-ops on ARM64 because
diff --git a/arch/arm64/include/asm/mshyperv.h b/arch/arm64/include/asm/mshyperv.h
index b17299c..86ca5c5 100644
--- a/arch/arm64/include/asm/mshyperv.h
+++ b/arch/arm64/include/asm/mshyperv.h
@@ -23,6 +23,12 @@
#include <asm/hyperv-tlfs.h>
#include <clocksource/arm_arch_timer.h>
+#if IS_ENABLED(CONFIG_HYPERV)
+void __init hyperv_early_init(void);
+#else
+static inline void hyperv_early_init(void) {};
+#endif
+
/*
* Declare calls to get and set Hyper-V VP register values on ARM64, which
* requires a hypercall.
diff --git a/arch/arm64/kernel/setup.c b/arch/arm64/kernel/setup.c
index 61845c0..7b17d6a 100644
--- a/arch/arm64/kernel/setup.c
+++ b/arch/arm64/kernel/setup.c
@@ -49,6 +49,7 @@
#include <asm/traps.h>
#include <asm/efi.h>
#include <asm/xen/hypervisor.h>
+#include <asm/mshyperv.h>
#include <asm/mmu_context.h>
static int num_standard_resources;
@@ -355,6 +356,9 @@ void __init __no_sanitize_address setup_arch(char **cmdline_p)
if (acpi_disabled)
unflatten_device_tree();
+ /* Do after acpi_boot_table_init() so local FADT is available */
+ hyperv_early_init();
+
bootmem_init();
kasan_init();
--
1.8.3.1
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 26+ messages in thread* Re: [PATCH v10 5/7] arm64: hyperv: Initialize hypervisor on boot
2021-05-12 17:37 ` [PATCH v10 5/7] arm64: hyperv: Initialize hypervisor on boot Michael Kelley
@ 2021-05-13 15:16 ` Wei Liu
0 siblings, 0 replies; 26+ messages in thread
From: Wei Liu @ 2021-05-13 15:16 UTC (permalink / raw)
To: Michael Kelley
Cc: will, catalin.marinas, mark.rutland, lorenzo.pieralisi,
sudeep.holla, linux-arm-kernel, linux-kernel, linux-hyperv,
linux-efi, arnd, wei.liu, ardb, daniel.lezcano, kys
On Wed, May 12, 2021 at 10:37:45AM -0700, Michael Kelley wrote:
> Add ARM64-specific code to initialize the Hyper-V
> hypervisor when booting as a guest VM. Provide functions
> and data structures indicating hypervisor status that
> are needed by VMbus driver.
>
> This code is built only when CONFIG_HYPERV is enabled.
>
> Signed-off-by: Michael Kelley <mikelley@microsoft.com>
[...]
> /*
> * Declare calls to get and set Hyper-V VP register values on ARM64, which
> * requires a hypercall.
> diff --git a/arch/arm64/kernel/setup.c b/arch/arm64/kernel/setup.c
> index 61845c0..7b17d6a 100644
> --- a/arch/arm64/kernel/setup.c
> +++ b/arch/arm64/kernel/setup.c
> @@ -49,6 +49,7 @@
> #include <asm/traps.h>
> #include <asm/efi.h>
> #include <asm/xen/hypervisor.h>
> +#include <asm/mshyperv.h>
> #include <asm/mmu_context.h>
>
> static int num_standard_resources;
> @@ -355,6 +356,9 @@ void __init __no_sanitize_address setup_arch(char **cmdline_p)
> if (acpi_disabled)
> unflatten_device_tree();
>
> + /* Do after acpi_boot_table_init() so local FADT is available */
> + hyperv_early_init();
> +
Arm maintainers, this requires your attention. Thanks.
The rest is Hyper-V specific, feel free to skip that portion.
Wei.
> bootmem_init();
>
> kasan_init();
> --
> 1.8.3.1
>
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 26+ messages in thread
* [PATCH v10 6/7] arm64: efi: Export screen_info
2021-05-12 17:37 [PATCH v10 0/7] Enable Linux guests on Hyper-V on ARM64 Michael Kelley
` (4 preceding siblings ...)
2021-05-12 17:37 ` [PATCH v10 5/7] arm64: hyperv: Initialize hypervisor on boot Michael Kelley
@ 2021-05-12 17:37 ` Michael Kelley
2021-05-12 17:37 ` [PATCH v10 7/7] Drivers: hv: Enable Hyper-V code to be built on ARM64 Michael Kelley
2021-05-13 13:17 ` [PATCH v10 0/7] Enable Linux guests on Hyper-V " Sudeep Holla
7 siblings, 0 replies; 26+ messages in thread
From: Michael Kelley @ 2021-05-12 17:37 UTC (permalink / raw)
To: will, catalin.marinas, mark.rutland, lorenzo.pieralisi,
sudeep.holla, linux-arm-kernel, linux-kernel, linux-hyperv,
linux-efi, arnd, wei.liu, ardb, daniel.lezcano, kys
Cc: mikelley
The Hyper-V frame buffer driver may be built as a module, and
it needs access to screen_info. So export screen_info.
Signed-off-by: Michael Kelley <mikelley@microsoft.com>
Acked-by: Ard Biesheuvel <ardb@kernel.org>
---
arch/arm64/kernel/efi.c | 1 +
1 file changed, 1 insertion(+)
diff --git a/arch/arm64/kernel/efi.c b/arch/arm64/kernel/efi.c
index fa02efb..e1be6c4 100644
--- a/arch/arm64/kernel/efi.c
+++ b/arch/arm64/kernel/efi.c
@@ -55,6 +55,7 @@ static __init pteval_t create_mapping_protection(efi_memory_desc_t *md)
/* we will fill this structure from the stub, so don't put it in .bss */
struct screen_info screen_info __section(".data");
+EXPORT_SYMBOL(screen_info);
int __init efi_create_mapping(struct mm_struct *mm, efi_memory_desc_t *md)
{
--
1.8.3.1
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 26+ messages in thread* [PATCH v10 7/7] Drivers: hv: Enable Hyper-V code to be built on ARM64
2021-05-12 17:37 [PATCH v10 0/7] Enable Linux guests on Hyper-V on ARM64 Michael Kelley
` (5 preceding siblings ...)
2021-05-12 17:37 ` [PATCH v10 6/7] arm64: efi: Export screen_info Michael Kelley
@ 2021-05-12 17:37 ` Michael Kelley
2021-05-13 13:17 ` [PATCH v10 0/7] Enable Linux guests on Hyper-V " Sudeep Holla
7 siblings, 0 replies; 26+ messages in thread
From: Michael Kelley @ 2021-05-12 17:37 UTC (permalink / raw)
To: will, catalin.marinas, mark.rutland, lorenzo.pieralisi,
sudeep.holla, linux-arm-kernel, linux-kernel, linux-hyperv,
linux-efi, arnd, wei.liu, ardb, daniel.lezcano, kys
Cc: mikelley
Update drivers/hv/Kconfig so CONFIG_HYPERV can be selected on
ARM64, causing the Hyper-V specific code to be built.
Signed-off-by: Michael Kelley <mikelley@microsoft.com>
Reviewed-by: Sunil Muthuswamy <sunilmut@microsoft.com>
---
drivers/hv/Kconfig | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/drivers/hv/Kconfig b/drivers/hv/Kconfig
index 66c794d..efb7585 100644
--- a/drivers/hv/Kconfig
+++ b/drivers/hv/Kconfig
@@ -4,7 +4,8 @@ menu "Microsoft Hyper-V guest support"
config HYPERV
tristate "Microsoft Hyper-V client drivers"
- depends on X86 && ACPI && X86_LOCAL_APIC && HYPERVISOR_GUEST
+ depends on ACPI && ((X86 && X86_LOCAL_APIC && HYPERVISOR_GUEST) \
+ || (ARM64 && !CPU_BIG_ENDIAN))
select PARAVIRT
select X86_HV_CALLBACK_VECTOR
help
--
1.8.3.1
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 26+ messages in thread* Re: [PATCH v10 0/7] Enable Linux guests on Hyper-V on ARM64
2021-05-12 17:37 [PATCH v10 0/7] Enable Linux guests on Hyper-V on ARM64 Michael Kelley
` (6 preceding siblings ...)
2021-05-12 17:37 ` [PATCH v10 7/7] Drivers: hv: Enable Hyper-V code to be built on ARM64 Michael Kelley
@ 2021-05-13 13:17 ` Sudeep Holla
7 siblings, 0 replies; 26+ messages in thread
From: Sudeep Holla @ 2021-05-13 13:17 UTC (permalink / raw)
To: will, catalin.marinas
Cc: Michael Kelley, mark.rutland, lorenzo.pieralisi,
linux-arm-kernel, linux-kernel, linux-hyperv, linux-efi, arnd,
wei.liu, ardb, daniel.lezcano, kys
Hi Will,
On Wed, May 12, 2021 at 10:37:40AM -0700, Michael Kelley wrote:
>
> This patch set is based on the 5.13-rc1 code tree, plus a patch
> from Sudeep Holla that implements SMCCC v1.2 HVC calls:
> https://lore.kernel.org/linux-arm-kernel/20210505093843.3308691-2-sudeep.holla@arm.com/
>
Assuming that you will be handling v5.14, I am asking you. I plan to post
the above mention patch that implements SMCCC v1.2 SMC/HVC independent
of my FFA series with small change as suggested by Mark R.
Irrespective of readiness of this series or FFA, is it possible to pull
the patch and share a branch based on v5.13-rc1(Arm SoC requirement), so
that if FFA is ready I can pull the branch along with FFA patches and send
it to Arm SoC. You can do the same for this series if it gets ready.
I am just trying to avoid last minute confusion. Let me know if you have
any alternative suggestions.
--
Regards,
Sudeep
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 26+ messages in thread