[PATCH 0/4] Hyper-V: Enable Linux guests on Hyper-V on ARM64

[PATCH 0/4] Hyper-V: Enable Linux guests on Hyper-V on ARM64

[kys]

From: "K. Y. Srinivasan" <kys@microsoft.com>

This series enables Linux guests running on Hyper-V on ARM64
hardware. New ARM64-specific code in arch/arm64/hyperv initializes
Hyper-V, including its synthetic clocks and hypercall mechanism.
Existing architecture independent drivers for Hyper-V's VMbus and
synthetic devices just work when built for ARM64. Hyper-V code is
built and included in the image and modules only if CONFIG_HYPERV
is enabled.

The four patches are organized as follows:
1) Add include files that define the Hyper-V interface as
   described in the Hyper-V Top Level Functional Spec (TLFS), plus
   additional definitions specific to Linux running on Hyper-V.

2) Add core Hyper-V support on ARM64, including hypercalls,
   synthetic clock initialization, and interrupt handlers.

3) Update the existing VMbus driver to generalize interrupt
   management across x86/x64 and ARM64.

4) Make CONFIG_HYPERV selectable on ARM64 in addition to x86/x64.

Some areas of Linux guests on Hyper-V on ARM64 are a work-
in-progress, primarily due to work still being done in Hyper-V:

* Hyper-V on ARM64 currently runs with a 4 Kbyte page size, and only
  supports guests with a 4 Kbyte page size. Because Hyper-V uses
  shared pages to communicate between the guest and the hypervisor,
  there are open design decisions on the page size to use when
  the guest is using 16K/64K pages.  Once those issues are
  resolved and Hyper-V fully supports 16K/64K guest pages, changes
  may be needed in the Linux drivers for Hyper-V synthetic devices.

* Hyper-V on ARM64 does not currently support mapping PCI devices
  into the guest address space. The Hyper-V PCI driver at
  drivers/pci/host/pci-hyperv.c has x86/x64-specific code and is
  not being built for ARM64.

In a few cases, terminology from the x86/x64 world has been carried
over into the ARM64 code ("MSR", "TSC").  Hyper-V still uses the
x86/x64 terminology and has not replaced it with something more
generic, so the code uses the Hyper-V terminology.  This will be
fixed when Hyper-V updates the usage in the TLFS.


Michael Kelley (4):
  arm64: hyperv: Add core Hyper-V include files
  arm64: hyperv: Add support for Hyper-V as a hypervisor
  Drivers: hv: vmbus: Add hooks for per-CPU IRQ
  Drivers: hv: Enable CONFIG_HYPERV on ARM64

 MAINTAINERS                          |   4 +
 arch/arm64/Makefile                  |   1 +
 arch/arm64/hyperv/Makefile           |   2 +
 arch/arm64/hyperv/hv_hvc.S           |  54 ++++
 arch/arm64/hyperv/hv_init.c          | 441 +++++++++++++++++++++++++++
 arch/arm64/hyperv/mshyperv.c         | 178 +++++++++++
 arch/arm64/include/asm/hyperv-tlfs.h | 338 ++++++++++++++++++++
 arch/arm64/include/asm/mshyperv.h    | 116 +++++++
 arch/x86/include/asm/mshyperv.h      |   4 +
 drivers/hv/Kconfig                   |   3 +-
 drivers/hv/hv.c                      |   2 +
 include/asm-generic/mshyperv.h       | 240 +++++++++++++++
 12 files changed, 1382 insertions(+), 1 deletion(-)
 create mode 100644 arch/arm64/hyperv/Makefile
 create mode 100644 arch/arm64/hyperv/hv_hvc.S
 create mode 100644 arch/arm64/hyperv/hv_init.c
 create mode 100644 arch/arm64/hyperv/mshyperv.c
 create mode 100644 arch/arm64/include/asm/hyperv-tlfs.h
 create mode 100644 arch/arm64/include/asm/mshyperv.h
 create mode 100644 include/asm-generic/mshyperv.h

-- 
2.19.1

[PATCH 0/4] Hyper-V: Enable Linux guests on Hyper-V on ARM64

[kys at linuxonhyperv.com]

From: "K. Y. Srinivasan" <kys@microsoft.com>

This series enables Linux guests running on Hyper-V on ARM64
hardware. New ARM64-specific code in arch/arm64/hyperv initializes
Hyper-V, including its synthetic clocks and hypercall mechanism.
Existing architecture independent drivers for Hyper-V's VMbus and
synthetic devices just work when built for ARM64. Hyper-V code is
built and included in the image and modules only if CONFIG_HYPERV
is enabled.

The four patches are organized as follows:
1) Add include files that define the Hyper-V interface as
   described in the Hyper-V Top Level Functional Spec (TLFS), plus
   additional definitions specific to Linux running on Hyper-V.

2) Add core Hyper-V support on ARM64, including hypercalls,
   synthetic clock initialization, and interrupt handlers.

3) Update the existing VMbus driver to generalize interrupt
   management across x86/x64 and ARM64.

4) Make CONFIG_HYPERV selectable on ARM64 in addition to x86/x64.

Some areas of Linux guests on Hyper-V on ARM64 are a work-
in-progress, primarily due to work still being done in Hyper-V:

* Hyper-V on ARM64 currently runs with a 4 Kbyte page size, and only
  supports guests with a 4 Kbyte page size. Because Hyper-V uses
  shared pages to communicate between the guest and the hypervisor,
  there are open design decisions on the page size to use when
  the guest is using 16K/64K pages.  Once those issues are
  resolved and Hyper-V fully supports 16K/64K guest pages, changes
  may be needed in the Linux drivers for Hyper-V synthetic devices.

* Hyper-V on ARM64 does not currently support mapping PCI devices
  into the guest address space. The Hyper-V PCI driver at
  drivers/pci/host/pci-hyperv.c has x86/x64-specific code and is
  not being built for ARM64.

In a few cases, terminology from the x86/x64 world has been carried
over into the ARM64 code ("MSR", "TSC").  Hyper-V still uses the
x86/x64 terminology and has not replaced it with something more
generic, so the code uses the Hyper-V terminology.  This will be
fixed when Hyper-V updates the usage in the TLFS.


Michael Kelley (4):
  arm64: hyperv: Add core Hyper-V include files
  arm64: hyperv: Add support for Hyper-V as a hypervisor
  Drivers: hv: vmbus: Add hooks for per-CPU IRQ
  Drivers: hv: Enable CONFIG_HYPERV on ARM64

 MAINTAINERS                          |   4 +
 arch/arm64/Makefile                  |   1 +
 arch/arm64/hyperv/Makefile           |   2 +
 arch/arm64/hyperv/hv_hvc.S           |  54 ++++
 arch/arm64/hyperv/hv_init.c          | 441 +++++++++++++++++++++++++++
 arch/arm64/hyperv/mshyperv.c         | 178 +++++++++++
 arch/arm64/include/asm/hyperv-tlfs.h | 338 ++++++++++++++++++++
 arch/arm64/include/asm/mshyperv.h    | 116 +++++++
 arch/x86/include/asm/mshyperv.h      |   4 +
 drivers/hv/Kconfig                   |   3 +-
 drivers/hv/hv.c                      |   2 +
 include/asm-generic/mshyperv.h       | 240 +++++++++++++++
 12 files changed, 1382 insertions(+), 1 deletion(-)
 create mode 100644 arch/arm64/hyperv/Makefile
 create mode 100644 arch/arm64/hyperv/hv_hvc.S
 create mode 100644 arch/arm64/hyperv/hv_init.c
 create mode 100644 arch/arm64/hyperv/mshyperv.c
 create mode 100644 arch/arm64/include/asm/hyperv-tlfs.h
 create mode 100644 arch/arm64/include/asm/mshyperv.h
 create mode 100644 include/asm-generic/mshyperv.h

-- 
2.19.1

[PATCH 1/4] arm64: hyperv: Add core Hyper-V include files

[kys]

From: Michael Kelley <mikelley@microsoft.com>

hyperv-tlfs.h defines Hyper-V interfaces from the Hyper-V Top Level
Functional Spec (TLFS). The TLFS is distinctly oriented to x86/x64,
and Hyper-V has not separated out the architecture-dependent parts into
x86/x64 vs. ARM64. So hyperv-tlfs.h includes information for ARM64
that is not yet formally published. The TLFS is available here:

  docs.microsoft.com/en-us/virtualization/hyper-v-on-windows/reference/tlfs

mshyperv.h defines Linux-specific structures and routines for
interacting with Hyper-V. It is split into an ARM64 specific file
and an architecture independent file in include/asm-generic.

Signed-off-by: Michael Kelley <mikelley@microsoft.com>
Signed-off-by: K. Y. Srinivasan <kys@microsoft.com>
---
 MAINTAINERS                          |   3 +
 arch/arm64/include/asm/hyperv-tlfs.h | 338 +++++++++++++++++++++++++++
 arch/arm64/include/asm/mshyperv.h    | 116 +++++++++
 include/asm-generic/mshyperv.h       | 240 +++++++++++++++++++
 4 files changed, 697 insertions(+)
 create mode 100644 arch/arm64/include/asm/hyperv-tlfs.h
 create mode 100644 arch/arm64/include/asm/mshyperv.h
 create mode 100644 include/asm-generic/mshyperv.h

diff --git a/MAINTAINERS b/MAINTAINERS
index f4855974f325..72f19cef4c48 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -6835,6 +6835,8 @@ F:	arch/x86/include/asm/trace/hyperv.h
 F:	arch/x86/include/asm/hyperv-tlfs.h
 F:	arch/x86/kernel/cpu/mshyperv.c
 F:	arch/x86/hyperv
+F:	arch/arm64/include/asm/hyperv-tlfs.h
+F:	arch/arm64/include/asm/mshyperv.h
 F:	drivers/hid/hid-hyperv.c
 F:	drivers/hv/
 F:	drivers/input/serio/hyperv-keyboard.c
@@ -6846,6 +6848,7 @@ F:	drivers/video/fbdev/hyperv_fb.c
 F:	net/vmw_vsock/hyperv_transport.c
 F:	include/linux/hyperv.h
 F:	include/uapi/linux/hyperv.h
+F:	include/asm-generic/mshyperv.h
 F:	tools/hv/
 F:	Documentation/ABI/stable/sysfs-bus-vmbus
 
diff --git a/arch/arm64/include/asm/hyperv-tlfs.h b/arch/arm64/include/asm/hyperv-tlfs.h
new file mode 100644
index 000000000000..924e37600e92
--- /dev/null
+++ b/arch/arm64/include/asm/hyperv-tlfs.h
@@ -0,0 +1,338 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+
+/*
+ * This file contains definitions from the Hyper-V Hypervisor Top-Level
+ * Functional Specification (TLFS):
+ * https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fvirtualization%2Fhyper-v-on-windows%2Freference%2Ftlfs&amp;data=02%7C01%7Ckys%40microsoft.com%7Cc831a45fd63e4a4b083908d641216aa8%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636768009113747528&amp;sdata=jRSrs9ZWXdmeS7LQUEpoSyUfBS7a5KLYy%2FolFdE2tI0%3D&amp;reserved=0
+ *
+ * Copyright (C) 2018, Microsoft, Inc.
+ *
+ * Author : Michael Kelley <mikelley@microsoft.com>
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License version 2 as published
+ * by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE, GOOD TITLE or
+ * NON INFRINGEMENT.  See the GNU General Public License for more
+ * details.
+ */
+
+#ifndef _ASM_ARM64_HYPERV_H
+#define _ASM_ARM64_HYPERV_H
+
+#include <linux/types.h>
+
+/*
+ * These Hyper-V registers provide information equivalent to the CPUID
+ * instruction on x86/x64.
+ */
+#define HV_REGISTER_HYPERVISOR_VERSION		0x00000100 /*CPUID 0x40000002 */
+#define	HV_REGISTER_PRIVILEGES_AND_FEATURES	0x00000200 /*CPUID 0x40000003 */
+#define	HV_REGISTER_FEATURES			0x00000201 /*CPUID 0x40000004 */
+#define	HV_REGISTER_IMPLEMENTATION_LIMITS	0x00000202 /*CPUID 0x40000005 */
+#define HV_ARM64_REGISTER_INTERFACE_VERSION	0x00090006 /*CPUID 0x40000001 */
+
+/*
+ * Feature identification. HvRegisterPrivilegesAndFeaturesInfo returns a
+ * 128-bit value with flags indicating which features are available to the
+ * partition based upon the current partition privileges. The 128-bit
+ * value is broken up with different portions stored in different 32-bit
+ * fields in the ms_hyperv structure.
+ */
+
+/* Partition Reference Counter available*/
+#define HV_MSR_TIME_REF_COUNT_AVAILABLE		(1 << 1)
+
+/*
+ * Synthetic Timers available
+ */
+#define HV_MSR_SYNTIMER_AVAILABLE		(1 << 3)
+
+/* Frequency MSRs available */
+#define HV_FEATURE_FREQUENCY_MSRS_AVAILABLE	(1 << 8)
+
+/* Reference TSC available */
+#define HV_MSR_REFERENCE_TSC_AVAILABLE		(1 << 9)
+
+/* Crash MSR available */
+#define HV_FEATURE_GUEST_CRASH_MSR_AVAILABLE	(1 << 10)
+
+
+/*
+ * This group of flags is in the high order 64-bits of the returned
+ * 128-bit value.
+ */
+
+/* STIMER direct mode is available */
+#define HV_STIMER_DIRECT_MODE_AVAILABLE		(1 << 19)
+
+/*
+ * Implementation recommendations in register
+ * HvRegisterFeaturesInfo. Indicates which behaviors the hypervisor
+ * recommends the OS implement for optimal performance.
+ */
+
+/*
+ * Recommend not using Auto EOI
+ */
+#define HV_DEPRECATING_AEOI_RECOMMENDED		(1 << 9)
+
+/*
+ * Synthetic register definitions equivalent to MSRs on x86/x64
+ */
+#define HV_REGISTER_CRASH_P0		0x00000210
+#define HV_REGISTER_CRASH_P1		0x00000211
+#define HV_REGISTER_CRASH_P2		0x00000212
+#define HV_REGISTER_CRASH_P3		0x00000213
+#define HV_REGISTER_CRASH_P4		0x00000214
+#define HV_REGISTER_CRASH_CTL		0x00000215
+
+#define HV_REGISTER_GUEST_OSID		0x00090002
+#define HV_REGISTER_VPINDEX		0x00090003
+#define HV_REGISTER_TIME_REFCOUNT	0x00090004
+#define HV_REGISTER_REFERENCE_TSC	0x00090017
+
+#define HV_REGISTER_SINT0		0x000A0000
+#define HV_REGISTER_SINT1		0x000A0001
+#define HV_REGISTER_SINT2		0x000A0002
+#define HV_REGISTER_SINT3		0x000A0003
+#define HV_REGISTER_SINT4		0x000A0004
+#define HV_REGISTER_SINT5		0x000A0005
+#define HV_REGISTER_SINT6		0x000A0006
+#define HV_REGISTER_SINT7		0x000A0007
+#define HV_REGISTER_SINT8		0x000A0008
+#define HV_REGISTER_SINT9		0x000A0009
+#define HV_REGISTER_SINT10		0x000A000A
+#define HV_REGISTER_SINT11		0x000A000B
+#define HV_REGISTER_SINT12		0x000A000C
+#define HV_REGISTER_SINT13		0x000A000D
+#define HV_REGISTER_SINT14		0x000A000E
+#define HV_REGISTER_SINT15		0x000A000F
+#define HV_REGISTER_SCONTROL		0x000A0010
+#define HV_REGISTER_SVERSION		0x000A0011
+#define HV_REGISTER_SIFP		0x000A0012
+#define HV_REGISTER_SIPP		0x000A0013
+#define HV_REGISTER_EOM			0x000A0014
+#define HV_REGISTER_SIRBP		0x000A0015
+
+#define HV_REGISTER_STIMER0_CONFIG	0x000B0000
+#define HV_REGISTER_STIMER0_COUNT	0x000B0001
+#define HV_REGISTER_STIMER1_CONFIG	0x000B0002
+#define HV_REGISTER_STIMER1_COUNT	0x000B0003
+#define HV_REGISTER_STIMER2_CONFIG	0x000B0004
+#define HV_REGISTER_STIMER2_COUNT	0x000B0005
+#define HV_REGISTER_STIMER3_CONFIG	0x000B0006
+#define HV_REGISTER_STIMER3_COUNT	0x000B0007
+
+/*
+ * Crash notification flags.
+ */
+#define HV_CRASH_CTL_CRASH_NOTIFY_MSG	BIT_ULL(62)
+#define HV_CRASH_CTL_CRASH_NOTIFY	BIT_ULL(63)
+
+/*
+ * The guest OS needs to register the guest ID with the hypervisor.
+ * The guest ID is a 64 bit entity and the structure of this ID is
+ * specified in the Hyper-V specification:
+ *
+ * msdn.microsoft.com/en-us/library/windows/hardware/ff542653%28v=vs.85%29.aspx
+ *
+ * While the current guideline does not specify how Linux guest ID(s)
+ * need to be generated, our plan is to publish the guidelines for
+ * Linux and other guest operating systems that currently are hosted
+ * on Hyper-V. The implementation here conforms to this yet
+ * unpublished guidelines.
+ *
+ *
+ * Bit(s)
+ * 63 - Indicates if the OS is Open Source or not; 1 is Open Source
+ * 62:56 - Os Type; Linux is 0x100
+ * 55:48 - Distro specific identification
+ * 47:16 - Linux kernel version number
+ * 15:0  - Distro specific identification
+ *
+ *
+ */
+#define HV_LINUX_VENDOR_ID              0x8100
+
+/* Declare the various hypercall operations. */
+#define HVCALL_FLUSH_VIRTUAL_ADDRESS_SPACE	0x0002
+#define HVCALL_FLUSH_VIRTUAL_ADDRESS_LIST	0x0003
+#define HVCALL_NOTIFY_LONG_SPIN_WAIT		0x0008
+#define HVCALL_SEND_IPI				0x000b
+#define HVCALL_FLUSH_VIRTUAL_ADDRESS_SPACE_EX	0x0013
+#define HVCALL_FLUSH_VIRTUAL_ADDRESS_LIST_EX	0x0014
+#define HVCALL_SEND_IPI_EX			0x0015
+#define HVCALL_GET_VP_REGISTERS			0x0050
+#define HVCALL_SET_VP_REGISTERS			0x0051
+#define HVCALL_POST_MESSAGE			0x005c
+#define HVCALL_SIGNAL_EVENT			0x005d
+#define HVCALL_RETARGET_INTERRUPT		0x007e
+#define HVCALL_START_VIRTUAL_PROCESSOR		0x0099
+#define HVCALL_GET_VP_INDEX_FROM_APICID		0x009a
+
+/* Declare standard hypercall field values. */
+#define HV_PARTITION_ID_SELF                    ((u64)-1)
+#define HV_VP_INDEX_SELF                        ((u32)-2)
+
+#define HV_HYPERCALL_FAST_BIT                   BIT(16)
+#define HV_HYPERCALL_REP_COUNT_1                BIT_ULL(32)
+#define HV_HYPERCALL_RESULT_MASK                GENMASK_ULL(15, 0)
+
+/* Define the hypercall status result */
+
+union hv_hypercall_status {
+	u64 as_uint64;
+	struct {
+		u16 status;
+		u16 reserved;
+		u16 reps_completed;  /* Low 12 bits */
+		u16 reserved2;
+	};
+};
+
+/* hypercall status code */
+#define HV_STATUS_SUCCESS			0
+#define HV_STATUS_INVALID_HYPERCALL_CODE	2
+#define HV_STATUS_INVALID_HYPERCALL_INPUT	3
+#define HV_STATUS_INVALID_ALIGNMENT		4
+#define HV_STATUS_INSUFFICIENT_MEMORY		11
+#define HV_STATUS_INVALID_CONNECTION_ID		18
+#define HV_STATUS_INSUFFICIENT_BUFFERS		19
+
+/* Define output layout for Get VP Register hypercall */
+struct hv_get_vp_register_output {
+	u64 registervaluelow;
+	u64 registervaluehigh;
+};
+
+#define HV_FLUSH_ALL_PROCESSORS			BIT(0)
+#define HV_FLUSH_ALL_VIRTUAL_ADDRESS_SPACES	BIT(1)
+#define HV_FLUSH_NON_GLOBAL_MAPPINGS_ONLY	BIT(2)
+#define HV_FLUSH_USE_EXTENDED_RANGE_FORMAT	BIT(3)
+
+enum HV_GENERIC_SET_FORMAT {
+	HV_GENERIC_SET_SPARSE_4K,
+	HV_GENERIC_SET_ALL,
+};
+
+/*
+ * The Hyper-V TimeRefCount register and the TSC
+ * page provide a guest VM clock with 100ns tick rate
+ */
+#define HV_CLOCK_HZ (NSEC_PER_SEC/100)
+
+/*
+ * The fields in this structure are set by Hyper-V and read
+ * by the Linux guest.  They should be accessed with READ_ONCE()
+ * so the compiler doesn't optimize in a way that will cause
+ * problems.
+ */
+struct ms_hyperv_tsc_page {
+	u32 tsc_sequence;
+	u32 reserved1;
+	u64 tsc_scale;
+	s64 tsc_offset;
+	u64 reserved2[509];
+};
+
+/* Define the number of synthetic interrupt sources. */
+#define HV_SYNIC_SINT_COUNT		(16)
+/* Define the expected SynIC version. */
+#define HV_SYNIC_VERSION_1		(0x1)
+
+#define HV_SYNIC_CONTROL_ENABLE		(1ULL << 0)
+#define HV_SYNIC_SIMP_ENABLE		(1ULL << 0)
+#define HV_SYNIC_SIEFP_ENABLE		(1ULL << 0)
+#define HV_SYNIC_SINT_MASKED		(1ULL << 16)
+#define HV_SYNIC_SINT_AUTO_EOI		(1ULL << 17)
+#define HV_SYNIC_SINT_VECTOR_MASK	(0xFF)
+
+#define HV_SYNIC_STIMER_COUNT		(4)
+
+/* Define synthetic interrupt controller message constants. */
+#define HV_MESSAGE_SIZE			(256)
+#define HV_MESSAGE_PAYLOAD_BYTE_COUNT	(240)
+#define HV_MESSAGE_PAYLOAD_QWORD_COUNT	(30)
+
+/* Define hypervisor message types. */
+enum hv_message_type {
+	HVMSG_NONE			= 0x00000000,
+
+	/* Memory access messages. */
+	HVMSG_UNMAPPED_GPA		= 0x80000000,
+	HVMSG_GPA_INTERCEPT		= 0x80000001,
+
+	/* Timer notification messages. */
+	HVMSG_TIMER_EXPIRED		= 0x80000010,
+
+	/* Error messages. */
+	HVMSG_INVALID_VP_REGISTER_VALUE	= 0x80000020,
+	HVMSG_UNRECOVERABLE_EXCEPTION	= 0x80000021,
+	HVMSG_UNSUPPORTED_FEATURE	= 0x80000022,
+
+	/* Trace buffer complete messages. */
+	HVMSG_EVENTLOG_BUFFERCOMPLETE	= 0x80000040,
+};
+
+/* Define synthetic interrupt controller message flags. */
+union hv_message_flags {
+	__u8 asu8;
+	struct {
+		__u8 msg_pending:1;
+		__u8 reserved:7;
+	};
+};
+
+/* Define port identifier type. */
+union hv_port_id {
+	__u32 asu32;
+	struct {
+		__u32 id:24;
+		__u32 reserved:8;
+	} u;
+};
+
+/* Define synthetic interrupt controller message header. */
+struct hv_message_header {
+	__u32 message_type;
+	__u8 payload_size;
+	union hv_message_flags message_flags;
+	__u8 reserved[2];
+	union {
+		__u64 sender;
+		union hv_port_id port;
+	};
+};
+
+/* Define synthetic interrupt controller message format. */
+struct hv_message {
+	struct hv_message_header header;
+	union {
+		__u64 payload[HV_MESSAGE_PAYLOAD_QWORD_COUNT];
+	} u;
+};
+
+/* Define the synthetic interrupt message page layout. */
+struct hv_message_page {
+	struct hv_message sint_message[HV_SYNIC_SINT_COUNT];
+};
+
+/* Define timer message payload structure. */
+struct hv_timer_message_payload {
+	__u32 timer_index;
+	__u32 reserved;
+	__u64 expiration_time;	/* When the timer expired */
+	__u64 delivery_time;	/* When the message was delivered */
+};
+
+#define HV_STIMER_ENABLE		(1ULL << 0)
+#define HV_STIMER_PERIODIC		(1ULL << 1)
+#define HV_STIMER_LAZY			(1ULL << 2)
+#define HV_STIMER_AUTOENABLE		(1ULL << 3)
+#define HV_STIMER_SINT(config)		(__u8)(((config) >> 16) & 0x0F)
+
+#endif
diff --git a/arch/arm64/include/asm/mshyperv.h b/arch/arm64/include/asm/mshyperv.h
new file mode 100644
index 000000000000..a87c431d58b3
--- /dev/null
+++ b/arch/arm64/include/asm/mshyperv.h
@@ -0,0 +1,116 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+
+/*
+ * Linux-specific definitions for managing interactions with Microsoft's
+ * Hyper-V hypervisor. The definitions in this file are specific to
+ * the ARM64 architecture.  See include/asm-generic/mshyperv.h for
+ * definitions are that architecture independent.
+ *
+ * Definitions that are specified in the Hyper-V Top Level Functional
+ * Spec (TLFS) should not go in this file, but should instead go in
+ * hyperv-tlfs.h.
+ *
+ * Copyright (C) 2018, Microsoft, Inc.
+ *
+ * Author : Michael Kelley <mikelley@microsoft.com>
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License version 2 as published
+ * by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE, GOOD TITLE or
+ * NON INFRINGEMENT.  See the GNU General Public License for more
+ * details.
+ */
+
+#ifndef _ASM_ARM64_MSHYPERV_H
+#define _ASM_ARM64_MSHYPERV_H
+
+#include <linux/types.h>
+#include <linux/interrupt.h>
+#include <linux/clocksource.h>
+#include <linux/irq.h>
+#include <linux/irqdesc.h>
+#include <asm/hyperv-tlfs.h>
+
+/*
+ * Define the IRQ numbers/vectors used by Hyper-V VMbus interrupts
+ * and by STIMER0 Direct Mode interrupts. Hyper-V should be supplying
+ * these values through ACPI, but there are no other interrupting
+ * devices in a Hyper-V VM on ARM64, so it's OK to hard code for now.
+ * The "CALLBACK_VECTOR" terminology is a left-over from the x86/x64
+ * world that is used in architecture independent Hyper-V code.
+ */
+#define HYPERVISOR_CALLBACK_VECTOR 16
+#define	HV_STIMER0_IRQNR	   17
+
+extern u64 hv_do_hvc(u64 control, ...);
+extern u64 hv_do_hvc_fast_get(u64 control, u64 input1, u64 input2, u64 input3,
+		struct hv_get_vp_register_output *output);
+
+/*
+ * Declare calls to get and set Hyper-V VP register values on ARM64, which
+ * requires a hypercall.
+ */
+extern void hv_set_vpreg(u32 reg, u64 value);
+extern u64 hv_get_vpreg(u32 reg);
+extern void hv_get_vpreg_128(u32 reg, struct hv_get_vp_register_output *result);
+
+/*
+ * Use the Hyper-V provided stimer0 as the timer that is made
+ * available to the architecture independent Hyper-V drivers.
+ */
+#define hv_init_timer(timer, tick) \
+		hv_set_vpreg(HV_REGISTER_STIMER0_COUNT + (2*timer), tick)
+#define hv_init_timer_config(timer, val) \
+		hv_set_vpreg(HV_REGISTER_STIMER0_CONFIG + (2*timer), val)
+#define hv_get_current_tick(tick) \
+		(tick = hv_get_vpreg(HV_REGISTER_TIME_REFCOUNT))
+
+#define hv_get_simp(val) (val = hv_get_vpreg(HV_REGISTER_SIPP))
+#define hv_set_simp(val) hv_set_vpreg(HV_REGISTER_SIPP, val)
+
+#define hv_get_siefp(val) (val = hv_get_vpreg(HV_REGISTER_SIFP))
+#define hv_set_siefp(val) hv_set_vpreg(HV_REGISTER_SIFP, val)
+
+#define hv_get_synic_state(val) (val = hv_get_vpreg(HV_REGISTER_SCONTROL))
+#define hv_set_synic_state(val) hv_set_vpreg(HV_REGISTER_SCONTROL, val)
+
+#define hv_get_vp_index(index) (index = hv_get_vpreg(HV_REGISTER_VPINDEX))
+
+#define hv_signal_eom()	hv_set_vpreg(HV_REGISTER_EOM, 0)
+
+/*
+ * Hyper-V SINT registers are numbered sequentially, so we can just
+ * add the SINT number to the register number of SINT0
+ */
+#define hv_get_synint_state(sint_num, val) \
+		(val = hv_get_vpreg(HV_REGISTER_SINT0 + sint_num))
+#define hv_set_synint_state(sint_num, val) \
+		hv_set_vpreg(HV_REGISTER_SINT0 + sint_num, val)
+
+#define hv_get_crash_ctl(val) \
+		(val = hv_get_vpreg(HV_REGISTER_CRASH_CTL))
+
+#if IS_ENABLED(CONFIG_HYPERV)
+#define hv_enable_stimer0_percpu_irq(irq)	enable_percpu_irq(irq, 0)
+#define hv_disable_stimer0_percpu_irq(irq)	disable_percpu_irq(irq)
+#endif
+
+/* ARM64 specific code to read the hardware clock */
+static inline u64 hv_read_hwclock(void)
+{
+	u64 result;
+
+	isb();
+	result = read_sysreg(cntvct_el0);
+	isb();
+
+	return result;
+}
+
+#include <asm-generic/mshyperv.h>
+
+#endif
diff --git a/include/asm-generic/mshyperv.h b/include/asm-generic/mshyperv.h
new file mode 100644
index 000000000000..fbe1ec89c85a
--- /dev/null
+++ b/include/asm-generic/mshyperv.h
@@ -0,0 +1,240 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+
+/*
+ * Linux-specific definitions for managing interactions with Microsoft's
+ * Hyper-V hypervisor. The definitions in this file are architecture
+ * independent. See arch/<arch>/include/asm/mshyperv.h for definitions
+ * that are specific to architecture <arch>.
+ *
+ * Definitions that are specified in the Hyper-V Top Level Functional
+ * Spec (TLFS) should not go in this file, but should instead go in
+ * hyperv-tlfs.h.
+ *
+ * Copyright (C) 2018, Microsoft, Inc.
+ *
+ * Author : Michael Kelley <mikelley@microsoft.com>
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License version 2 as published
+ * by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE, GOOD TITLE or
+ * NON INFRINGEMENT.  See the GNU General Public License for more
+ * details.
+ */
+
+#ifndef _ASM_GENERIC_MSHYPERV_H
+#define _ASM_GENERIC_MSHYPERV_H
+
+#include <linux/types.h>
+#include <linux/interrupt.h>
+#include <linux/clocksource.h>
+#include <linux/irq.h>
+#include <linux/irqdesc.h>
+#include <asm/hyperv-tlfs.h>
+
+/*
+ * Hyper-V always runs with a page size of 4096. These definitions
+ * are used when communicating with Hyper-V using guest physical
+ * pages and guest physical page addresses, since the guest page
+ * size may not be 4096 on ARM64.
+ */
+#define HV_HYP_PAGE_SIZE	4096
+#define HV_HYP_PAGE_SHIFT	12
+#define HV_HYP_PAGE_MASK	(~(HV_HYP_PAGE_SIZE - 1))
+
+
+struct ms_hyperv_info {
+	u32 features;
+	u32 misc_features;
+	u32 hints;
+	u32 max_vp_index;
+	u32 max_lp_index;
+};
+extern struct ms_hyperv_info ms_hyperv;
+
+extern u64 hv_do_hypercall(u64 control, void *inputaddr, void *outputaddr);
+extern u64 hv_do_fast_hypercall8(u16 control, u64 input8);
+
+/*
+ * The guest OS needs to register the guest ID with the hypervisor.
+ * The guest ID is a 64 bit entity and the structure of this ID is
+ * specified in the Hyper-V specification:
+ *
+ * msdn.microsoft.com/en-us/library/windows/hardware/ff542653%28v=vs.85%29.aspx
+ *
+ * While the current guideline does not specify how Linux guest ID(s)
+ * need to be generated, our plan is to publish the guidelines for
+ * Linux and other guest operating systems that currently are hosted
+ * on Hyper-V. The implementation here conforms to this yet
+ * unpublished guidelines.
+ *
+ *
+ * Bit(s)
+ * 63 - Indicates if the OS is Open Source or not; 1 is Open Source
+ * 62:56 - Os Type; Linux is 0x100
+ * 55:48 - Distro specific identification
+ * 47:16 - Linux kernel version number
+ * 15:0  - Distro specific identification
+ *
+ * Generate the guest ID based on the guideline described above.
+ */
+
+static inline  __u64 generate_guest_id(__u64 d_info1, __u64 kernel_version,
+				       __u64 d_info2)
+{
+	__u64 guest_id = 0;
+
+	guest_id = (((__u64)HV_LINUX_VENDOR_ID) << 48);
+	guest_id |= (d_info1 << 48);
+	guest_id |= (kernel_version << 16);
+	guest_id |= d_info2;
+
+	return guest_id;
+}
+
+
+/* Free the message slot and signal end-of-message if required */
+static inline void vmbus_signal_eom(struct hv_message *msg, u32 old_msg_type)
+{
+	/*
+	 * On crash we're reading some other CPU's message page and we need
+	 * to be careful: this other CPU may already had cleared the header
+	 * and the host may already had delivered some other message there.
+	 * In case we blindly write msg->header.message_type we're going
+	 * to lose it. We can still lose a message of the same type but
+	 * we count on the fact that there can only be one
+	 * CHANNELMSG_UNLOAD_RESPONSE and we don't care about other messages
+	 * on crash.
+	 */
+	if (cmpxchg(&msg->header.message_type, old_msg_type,
+		    HVMSG_NONE) != old_msg_type)
+		return;
+
+	/*
+	 * Make sure the write to MessageType (ie set to
+	 * HVMSG_NONE) happens before we read the
+	 * MessagePending and EOMing. Otherwise, the EOMing
+	 * will not deliver any more messages since there is
+	 * no empty slot
+	 */
+	mb();
+
+	if (msg->header.message_flags.msg_pending) {
+		/*
+		 * This will cause message queue rescan to
+		 * possibly deliver another msg from the
+		 * hypervisor
+		 */
+		hv_signal_eom();
+	}
+}
+
+void hv_setup_vmbus_irq(void (*handler)(void));
+void hv_remove_vmbus_irq(void);
+void hv_enable_vmbus_irq(void);
+void hv_disable_vmbus_irq(void);
+
+void hv_setup_kexec_handler(void (*handler)(void));
+void hv_remove_kexec_handler(void);
+void hv_setup_crash_handler(void (*handler)(struct pt_regs *regs));
+void hv_remove_crash_handler(void);
+
+#if IS_ENABLED(CONFIG_HYPERV)
+extern struct clocksource *hyperv_cs;
+
+/*
+ * Hypervisor's notion of virtual processor ID is different from
+ * Linux' notion of CPU ID. This information can only be retrieved
+ * in the context of the calling CPU. Setup a map for easy access
+ * to this information.
+ */
+extern u32 *hv_vp_index;
+extern u32 hv_max_vp_index;
+
+/* Sentinel value for an uninitialized entry in hv_vp_index array */
+#define VP_INVAL	U32_MAX
+
+/**
+ * hv_cpu_number_to_vp_number() - Map CPU to VP.
+ * @cpu_number: CPU number in Linux terms
+ *
+ * This function returns the mapping between the Linux processor
+ * number and the hypervisor's virtual processor number, useful
+ * in making hypercalls and such that talk about specific
+ * processors.
+ *
+ * Return: Virtual processor number in Hyper-V terms
+ */
+static inline int hv_cpu_number_to_vp_number(int cpu_number)
+{
+	return hv_vp_index[cpu_number];
+}
+
+void hyperv_report_panic(struct pt_regs *regs, long err);
+void hyperv_report_panic_msg(phys_addr_t pa, size_t size);
+bool hv_is_hyperv_initialized(void);
+void hyperv_cleanup(void);
+#else /* CONFIG_HYPERV */
+static inline bool hv_is_hyperv_initialized(void) { return false; }
+static inline void hyperv_cleanup(void) {}
+#endif /* CONFIG_HYPERV */
+
+#if IS_ENABLED(CONFIG_HYPERV)
+extern int hv_setup_stimer0_irq(int *irq, int *vector, void (*handler)(void));
+extern void hv_remove_stimer0_irq(int irq);
+#endif
+
+static inline u64 hv_read_tsc_page_tsc(const struct ms_hyperv_tsc_page *tsc_pg,
+				       u64 *cur_tsc)
+{
+	u64	scale, offset;
+	u32	sequence;
+
+	/*
+	 * The protocol for reading Hyper-V TSC page is specified in Hypervisor
+	 * Top-Level Functional Specification.  To get the reference time we
+	 * must do the following:
+	 * - READ ReferenceTscSequence
+	 *   A special '0' value indicates the time source is unreliable and we
+	 *   need to use something else.
+	 * - ReferenceTime =
+	 *     ((HWclock val) * ReferenceTscScale) >> 64) + ReferenceTscOffset
+	 * - READ ReferenceTscSequence again. In case its value has changed
+	 *   since our first reading we need to discard ReferenceTime and repeat
+	 *   the whole sequence as the hypervisor was updating the page in
+	 *   between.
+	 */
+	do {
+		sequence = READ_ONCE(tsc_pg->tsc_sequence);
+		/*
+		 * Make sure we read sequence before we read other values from
+		 * TSC page.
+		 */
+		smp_rmb();
+
+		scale = READ_ONCE(tsc_pg->tsc_scale);
+		offset = READ_ONCE(tsc_pg->tsc_offset);
+		*cur_tsc = hv_read_hwclock();
+
+		/*
+		 * Make sure we read sequence after we read all other values
+		 * from TSC page.
+		 */
+		smp_rmb();
+
+	} while (READ_ONCE(tsc_pg->tsc_sequence) != sequence);
+
+	return mul_u64_u64_shr(*cur_tsc, scale, 64) + offset;
+}
+
+static inline u64 hv_read_tsc_page(const struct ms_hyperv_tsc_page *tsc_pg)
+{
+	u64 cur_tsc;
+
+	return hv_read_tsc_page_tsc(tsc_pg, &cur_tsc);
+}
+
+#endif
-- 
2.19.1

[PATCH 1/4] arm64: hyperv: Add core Hyper-V include files

[kys at linuxonhyperv.com]

From: Michael Kelley <mikelley@microsoft.com>

hyperv-tlfs.h defines Hyper-V interfaces from the Hyper-V Top Level
Functional Spec (TLFS). The TLFS is distinctly oriented to x86/x64,
and Hyper-V has not separated out the architecture-dependent parts into
x86/x64 vs. ARM64. So hyperv-tlfs.h includes information for ARM64
that is not yet formally published. The TLFS is available here:

  docs.microsoft.com/en-us/virtualization/hyper-v-on-windows/reference/tlfs

mshyperv.h defines Linux-specific structures and routines for
interacting with Hyper-V. It is split into an ARM64 specific file
and an architecture independent file in include/asm-generic.

Signed-off-by: Michael Kelley <mikelley@microsoft.com>
Signed-off-by: K. Y. Srinivasan <kys@microsoft.com>
---
 MAINTAINERS                          |   3 +
 arch/arm64/include/asm/hyperv-tlfs.h | 338 +++++++++++++++++++++++++++
 arch/arm64/include/asm/mshyperv.h    | 116 +++++++++
 include/asm-generic/mshyperv.h       | 240 +++++++++++++++++++
 4 files changed, 697 insertions(+)
 create mode 100644 arch/arm64/include/asm/hyperv-tlfs.h
 create mode 100644 arch/arm64/include/asm/mshyperv.h
 create mode 100644 include/asm-generic/mshyperv.h

diff --git a/MAINTAINERS b/MAINTAINERS
index f4855974f325..72f19cef4c48 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -6835,6 +6835,8 @@ F:	arch/x86/include/asm/trace/hyperv.h
 F:	arch/x86/include/asm/hyperv-tlfs.h
 F:	arch/x86/kernel/cpu/mshyperv.c
 F:	arch/x86/hyperv
+F:	arch/arm64/include/asm/hyperv-tlfs.h
+F:	arch/arm64/include/asm/mshyperv.h
 F:	drivers/hid/hid-hyperv.c
 F:	drivers/hv/
 F:	drivers/input/serio/hyperv-keyboard.c
@@ -6846,6 +6848,7 @@ F:	drivers/video/fbdev/hyperv_fb.c
 F:	net/vmw_vsock/hyperv_transport.c
 F:	include/linux/hyperv.h
 F:	include/uapi/linux/hyperv.h
+F:	include/asm-generic/mshyperv.h
 F:	tools/hv/
 F:	Documentation/ABI/stable/sysfs-bus-vmbus
 
diff --git a/arch/arm64/include/asm/hyperv-tlfs.h b/arch/arm64/include/asm/hyperv-tlfs.h
new file mode 100644
index 000000000000..924e37600e92
--- /dev/null
+++ b/arch/arm64/include/asm/hyperv-tlfs.h
@@ -0,0 +1,338 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+
+/*
+ * This file contains definitions from the Hyper-V Hypervisor Top-Level
+ * Functional Specification (TLFS):
+ * https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fvirtualization%2Fhyper-v-on-windows%2Freference%2Ftlfs&amp;data=02%7C01%7Ckys%40microsoft.com%7Cc831a45fd63e4a4b083908d641216aa8%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636768009113747528&amp;sdata=jRSrs9ZWXdmeS7LQUEpoSyUfBS7a5KLYy%2FolFdE2tI0%3D&amp;reserved=0
+ *
+ * Copyright (C) 2018, Microsoft, Inc.
+ *
+ * Author : Michael Kelley <mikelley@microsoft.com>
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License version 2 as published
+ * by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE, GOOD TITLE or
+ * NON INFRINGEMENT.  See the GNU General Public License for more
+ * details.
+ */
+
+#ifndef _ASM_ARM64_HYPERV_H
+#define _ASM_ARM64_HYPERV_H
+
+#include <linux/types.h>
+
+/*
+ * These Hyper-V registers provide information equivalent to the CPUID
+ * instruction on x86/x64.
+ */
+#define HV_REGISTER_HYPERVISOR_VERSION		0x00000100 /*CPUID 0x40000002 */
+#define	HV_REGISTER_PRIVILEGES_AND_FEATURES	0x00000200 /*CPUID 0x40000003 */
+#define	HV_REGISTER_FEATURES			0x00000201 /*CPUID 0x40000004 */
+#define	HV_REGISTER_IMPLEMENTATION_LIMITS	0x00000202 /*CPUID 0x40000005 */
+#define HV_ARM64_REGISTER_INTERFACE_VERSION	0x00090006 /*CPUID 0x40000001 */
+
+/*
+ * Feature identification. HvRegisterPrivilegesAndFeaturesInfo returns a
+ * 128-bit value with flags indicating which features are available to the
+ * partition based upon the current partition privileges. The 128-bit
+ * value is broken up with different portions stored in different 32-bit
+ * fields in the ms_hyperv structure.
+ */
+
+/* Partition Reference Counter available*/
+#define HV_MSR_TIME_REF_COUNT_AVAILABLE		(1 << 1)
+
+/*
+ * Synthetic Timers available
+ */
+#define HV_MSR_SYNTIMER_AVAILABLE		(1 << 3)
+
+/* Frequency MSRs available */
+#define HV_FEATURE_FREQUENCY_MSRS_AVAILABLE	(1 << 8)
+
+/* Reference TSC available */
+#define HV_MSR_REFERENCE_TSC_AVAILABLE		(1 << 9)
+
+/* Crash MSR available */
+#define HV_FEATURE_GUEST_CRASH_MSR_AVAILABLE	(1 << 10)
+
+
+/*
+ * This group of flags is in the high order 64-bits of the returned
+ * 128-bit value.
+ */
+
+/* STIMER direct mode is available */
+#define HV_STIMER_DIRECT_MODE_AVAILABLE		(1 << 19)
+
+/*
+ * Implementation recommendations in register
+ * HvRegisterFeaturesInfo. Indicates which behaviors the hypervisor
+ * recommends the OS implement for optimal performance.
+ */
+
+/*
+ * Recommend not using Auto EOI
+ */
+#define HV_DEPRECATING_AEOI_RECOMMENDED		(1 << 9)
+
+/*
+ * Synthetic register definitions equivalent to MSRs on x86/x64
+ */
+#define HV_REGISTER_CRASH_P0		0x00000210
+#define HV_REGISTER_CRASH_P1		0x00000211
+#define HV_REGISTER_CRASH_P2		0x00000212
+#define HV_REGISTER_CRASH_P3		0x00000213
+#define HV_REGISTER_CRASH_P4		0x00000214
+#define HV_REGISTER_CRASH_CTL		0x00000215
+
+#define HV_REGISTER_GUEST_OSID		0x00090002
+#define HV_REGISTER_VPINDEX		0x00090003
+#define HV_REGISTER_TIME_REFCOUNT	0x00090004
+#define HV_REGISTER_REFERENCE_TSC	0x00090017
+
+#define HV_REGISTER_SINT0		0x000A0000
+#define HV_REGISTER_SINT1		0x000A0001
+#define HV_REGISTER_SINT2		0x000A0002
+#define HV_REGISTER_SINT3		0x000A0003
+#define HV_REGISTER_SINT4		0x000A0004
+#define HV_REGISTER_SINT5		0x000A0005
+#define HV_REGISTER_SINT6		0x000A0006
+#define HV_REGISTER_SINT7		0x000A0007
+#define HV_REGISTER_SINT8		0x000A0008
+#define HV_REGISTER_SINT9		0x000A0009
+#define HV_REGISTER_SINT10		0x000A000A
+#define HV_REGISTER_SINT11		0x000A000B
+#define HV_REGISTER_SINT12		0x000A000C
+#define HV_REGISTER_SINT13		0x000A000D
+#define HV_REGISTER_SINT14		0x000A000E
+#define HV_REGISTER_SINT15		0x000A000F
+#define HV_REGISTER_SCONTROL		0x000A0010
+#define HV_REGISTER_SVERSION		0x000A0011
+#define HV_REGISTER_SIFP		0x000A0012
+#define HV_REGISTER_SIPP		0x000A0013
+#define HV_REGISTER_EOM			0x000A0014
+#define HV_REGISTER_SIRBP		0x000A0015
+
+#define HV_REGISTER_STIMER0_CONFIG	0x000B0000
+#define HV_REGISTER_STIMER0_COUNT	0x000B0001
+#define HV_REGISTER_STIMER1_CONFIG	0x000B0002
+#define HV_REGISTER_STIMER1_COUNT	0x000B0003
+#define HV_REGISTER_STIMER2_CONFIG	0x000B0004
+#define HV_REGISTER_STIMER2_COUNT	0x000B0005
+#define HV_REGISTER_STIMER3_CONFIG	0x000B0006
+#define HV_REGISTER_STIMER3_COUNT	0x000B0007
+
+/*
+ * Crash notification flags.
+ */
+#define HV_CRASH_CTL_CRASH_NOTIFY_MSG	BIT_ULL(62)
+#define HV_CRASH_CTL_CRASH_NOTIFY	BIT_ULL(63)
+
+/*
+ * The guest OS needs to register the guest ID with the hypervisor.
+ * The guest ID is a 64 bit entity and the structure of this ID is
+ * specified in the Hyper-V specification:
+ *
+ * msdn.microsoft.com/en-us/library/windows/hardware/ff542653%28v=vs.85%29.aspx
+ *
+ * While the current guideline does not specify how Linux guest ID(s)
+ * need to be generated, our plan is to publish the guidelines for
+ * Linux and other guest operating systems that currently are hosted
+ * on Hyper-V. The implementation here conforms to this yet
+ * unpublished guidelines.
+ *
+ *
+ * Bit(s)
+ * 63 - Indicates if the OS is Open Source or not; 1 is Open Source
+ * 62:56 - Os Type; Linux is 0x100
+ * 55:48 - Distro specific identification
+ * 47:16 - Linux kernel version number
+ * 15:0  - Distro specific identification
+ *
+ *
+ */
+#define HV_LINUX_VENDOR_ID              0x8100
+
+/* Declare the various hypercall operations. */
+#define HVCALL_FLUSH_VIRTUAL_ADDRESS_SPACE	0x0002
+#define HVCALL_FLUSH_VIRTUAL_ADDRESS_LIST	0x0003
+#define HVCALL_NOTIFY_LONG_SPIN_WAIT		0x0008
+#define HVCALL_SEND_IPI				0x000b
+#define HVCALL_FLUSH_VIRTUAL_ADDRESS_SPACE_EX	0x0013
+#define HVCALL_FLUSH_VIRTUAL_ADDRESS_LIST_EX	0x0014
+#define HVCALL_SEND_IPI_EX			0x0015
+#define HVCALL_GET_VP_REGISTERS			0x0050
+#define HVCALL_SET_VP_REGISTERS			0x0051
+#define HVCALL_POST_MESSAGE			0x005c
+#define HVCALL_SIGNAL_EVENT			0x005d
+#define HVCALL_RETARGET_INTERRUPT		0x007e
+#define HVCALL_START_VIRTUAL_PROCESSOR		0x0099
+#define HVCALL_GET_VP_INDEX_FROM_APICID		0x009a
+
+/* Declare standard hypercall field values. */
+#define HV_PARTITION_ID_SELF                    ((u64)-1)
+#define HV_VP_INDEX_SELF                        ((u32)-2)
+
+#define HV_HYPERCALL_FAST_BIT                   BIT(16)
+#define HV_HYPERCALL_REP_COUNT_1                BIT_ULL(32)
+#define HV_HYPERCALL_RESULT_MASK                GENMASK_ULL(15, 0)
+
+/* Define the hypercall status result */
+
+union hv_hypercall_status {
+	u64 as_uint64;
+	struct {
+		u16 status;
+		u16 reserved;
+		u16 reps_completed;  /* Low 12 bits */
+		u16 reserved2;
+	};
+};
+
+/* hypercall status code */
+#define HV_STATUS_SUCCESS			0
+#define HV_STATUS_INVALID_HYPERCALL_CODE	2
+#define HV_STATUS_INVALID_HYPERCALL_INPUT	3
+#define HV_STATUS_INVALID_ALIGNMENT		4
+#define HV_STATUS_INSUFFICIENT_MEMORY		11
+#define HV_STATUS_INVALID_CONNECTION_ID		18
+#define HV_STATUS_INSUFFICIENT_BUFFERS		19
+
+/* Define output layout for Get VP Register hypercall */
+struct hv_get_vp_register_output {
+	u64 registervaluelow;
+	u64 registervaluehigh;
+};
+
+#define HV_FLUSH_ALL_PROCESSORS			BIT(0)
+#define HV_FLUSH_ALL_VIRTUAL_ADDRESS_SPACES	BIT(1)
+#define HV_FLUSH_NON_GLOBAL_MAPPINGS_ONLY	BIT(2)
+#define HV_FLUSH_USE_EXTENDED_RANGE_FORMAT	BIT(3)
+
+enum HV_GENERIC_SET_FORMAT {
+	HV_GENERIC_SET_SPARSE_4K,
+	HV_GENERIC_SET_ALL,
+};
+
+/*
+ * The Hyper-V TimeRefCount register and the TSC
+ * page provide a guest VM clock with 100ns tick rate
+ */
+#define HV_CLOCK_HZ (NSEC_PER_SEC/100)
+
+/*
+ * The fields in this structure are set by Hyper-V and read
+ * by the Linux guest.  They should be accessed with READ_ONCE()
+ * so the compiler doesn't optimize in a way that will cause
+ * problems.
+ */
+struct ms_hyperv_tsc_page {
+	u32 tsc_sequence;
+	u32 reserved1;
+	u64 tsc_scale;
+	s64 tsc_offset;
+	u64 reserved2[509];
+};
+
+/* Define the number of synthetic interrupt sources. */
+#define HV_SYNIC_SINT_COUNT		(16)
+/* Define the expected SynIC version. */
+#define HV_SYNIC_VERSION_1		(0x1)
+
+#define HV_SYNIC_CONTROL_ENABLE		(1ULL << 0)
+#define HV_SYNIC_SIMP_ENABLE		(1ULL << 0)
+#define HV_SYNIC_SIEFP_ENABLE		(1ULL << 0)
+#define HV_SYNIC_SINT_MASKED		(1ULL << 16)
+#define HV_SYNIC_SINT_AUTO_EOI		(1ULL << 17)
+#define HV_SYNIC_SINT_VECTOR_MASK	(0xFF)
+
+#define HV_SYNIC_STIMER_COUNT		(4)
+
+/* Define synthetic interrupt controller message constants. */
+#define HV_MESSAGE_SIZE			(256)
+#define HV_MESSAGE_PAYLOAD_BYTE_COUNT	(240)
+#define HV_MESSAGE_PAYLOAD_QWORD_COUNT	(30)
+
+/* Define hypervisor message types. */
+enum hv_message_type {
+	HVMSG_NONE			= 0x00000000,
+
+	/* Memory access messages. */
+	HVMSG_UNMAPPED_GPA		= 0x80000000,
+	HVMSG_GPA_INTERCEPT		= 0x80000001,
+
+	/* Timer notification messages. */
+	HVMSG_TIMER_EXPIRED		= 0x80000010,
+
+	/* Error messages. */
+	HVMSG_INVALID_VP_REGISTER_VALUE	= 0x80000020,
+	HVMSG_UNRECOVERABLE_EXCEPTION	= 0x80000021,
+	HVMSG_UNSUPPORTED_FEATURE	= 0x80000022,
+
+	/* Trace buffer complete messages. */
+	HVMSG_EVENTLOG_BUFFERCOMPLETE	= 0x80000040,
+};
+
+/* Define synthetic interrupt controller message flags. */
+union hv_message_flags {
+	__u8 asu8;
+	struct {
+		__u8 msg_pending:1;
+		__u8 reserved:7;
+	};
+};
+
+/* Define port identifier type. */
+union hv_port_id {
+	__u32 asu32;
+	struct {
+		__u32 id:24;
+		__u32 reserved:8;
+	} u;
+};
+
+/* Define synthetic interrupt controller message header. */
+struct hv_message_header {
+	__u32 message_type;
+	__u8 payload_size;
+	union hv_message_flags message_flags;
+	__u8 reserved[2];
+	union {
+		__u64 sender;
+		union hv_port_id port;
+	};
+};
+
+/* Define synthetic interrupt controller message format. */
+struct hv_message {
+	struct hv_message_header header;
+	union {
+		__u64 payload[HV_MESSAGE_PAYLOAD_QWORD_COUNT];
+	} u;
+};
+
+/* Define the synthetic interrupt message page layout. */
+struct hv_message_page {
+	struct hv_message sint_message[HV_SYNIC_SINT_COUNT];
+};
+
+/* Define timer message payload structure. */
+struct hv_timer_message_payload {
+	__u32 timer_index;
+	__u32 reserved;
+	__u64 expiration_time;	/* When the timer expired */
+	__u64 delivery_time;	/* When the message was delivered */
+};
+
+#define HV_STIMER_ENABLE		(1ULL << 0)
+#define HV_STIMER_PERIODIC		(1ULL << 1)
+#define HV_STIMER_LAZY			(1ULL << 2)
+#define HV_STIMER_AUTOENABLE		(1ULL << 3)
+#define HV_STIMER_SINT(config)		(__u8)(((config) >> 16) & 0x0F)
+
+#endif
diff --git a/arch/arm64/include/asm/mshyperv.h b/arch/arm64/include/asm/mshyperv.h
new file mode 100644
index 000000000000..a87c431d58b3
--- /dev/null
+++ b/arch/arm64/include/asm/mshyperv.h
@@ -0,0 +1,116 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+
+/*
+ * Linux-specific definitions for managing interactions with Microsoft's
+ * Hyper-V hypervisor. The definitions in this file are specific to
+ * the ARM64 architecture.  See include/asm-generic/mshyperv.h for
+ * definitions are that architecture independent.
+ *
+ * Definitions that are specified in the Hyper-V Top Level Functional
+ * Spec (TLFS) should not go in this file, but should instead go in
+ * hyperv-tlfs.h.
+ *
+ * Copyright (C) 2018, Microsoft, Inc.
+ *
+ * Author : Michael Kelley <mikelley@microsoft.com>
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License version 2 as published
+ * by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE, GOOD TITLE or
+ * NON INFRINGEMENT.  See the GNU General Public License for more
+ * details.
+ */
+
+#ifndef _ASM_ARM64_MSHYPERV_H
+#define _ASM_ARM64_MSHYPERV_H
+
+#include <linux/types.h>
+#include <linux/interrupt.h>
+#include <linux/clocksource.h>
+#include <linux/irq.h>
+#include <linux/irqdesc.h>
+#include <asm/hyperv-tlfs.h>
+
+/*
+ * Define the IRQ numbers/vectors used by Hyper-V VMbus interrupts
+ * and by STIMER0 Direct Mode interrupts. Hyper-V should be supplying
+ * these values through ACPI, but there are no other interrupting
+ * devices in a Hyper-V VM on ARM64, so it's OK to hard code for now.
+ * The "CALLBACK_VECTOR" terminology is a left-over from the x86/x64
+ * world that is used in architecture independent Hyper-V code.
+ */
+#define HYPERVISOR_CALLBACK_VECTOR 16
+#define	HV_STIMER0_IRQNR	   17
+
+extern u64 hv_do_hvc(u64 control, ...);
+extern u64 hv_do_hvc_fast_get(u64 control, u64 input1, u64 input2, u64 input3,
+		struct hv_get_vp_register_output *output);
+
+/*
+ * Declare calls to get and set Hyper-V VP register values on ARM64, which
+ * requires a hypercall.
+ */
+extern void hv_set_vpreg(u32 reg, u64 value);
+extern u64 hv_get_vpreg(u32 reg);
+extern void hv_get_vpreg_128(u32 reg, struct hv_get_vp_register_output *result);
+
+/*
+ * Use the Hyper-V provided stimer0 as the timer that is made
+ * available to the architecture independent Hyper-V drivers.
+ */
+#define hv_init_timer(timer, tick) \
+		hv_set_vpreg(HV_REGISTER_STIMER0_COUNT + (2*timer), tick)
+#define hv_init_timer_config(timer, val) \
+		hv_set_vpreg(HV_REGISTER_STIMER0_CONFIG + (2*timer), val)
+#define hv_get_current_tick(tick) \
+		(tick = hv_get_vpreg(HV_REGISTER_TIME_REFCOUNT))
+
+#define hv_get_simp(val) (val = hv_get_vpreg(HV_REGISTER_SIPP))
+#define hv_set_simp(val) hv_set_vpreg(HV_REGISTER_SIPP, val)
+
+#define hv_get_siefp(val) (val = hv_get_vpreg(HV_REGISTER_SIFP))
+#define hv_set_siefp(val) hv_set_vpreg(HV_REGISTER_SIFP, val)
+
+#define hv_get_synic_state(val) (val = hv_get_vpreg(HV_REGISTER_SCONTROL))
+#define hv_set_synic_state(val) hv_set_vpreg(HV_REGISTER_SCONTROL, val)
+
+#define hv_get_vp_index(index) (index = hv_get_vpreg(HV_REGISTER_VPINDEX))
+
+#define hv_signal_eom()	hv_set_vpreg(HV_REGISTER_EOM, 0)
+
+/*
+ * Hyper-V SINT registers are numbered sequentially, so we can just
+ * add the SINT number to the register number of SINT0
+ */
+#define hv_get_synint_state(sint_num, val) \
+		(val = hv_get_vpreg(HV_REGISTER_SINT0 + sint_num))
+#define hv_set_synint_state(sint_num, val) \
+		hv_set_vpreg(HV_REGISTER_SINT0 + sint_num, val)
+
+#define hv_get_crash_ctl(val) \
+		(val = hv_get_vpreg(HV_REGISTER_CRASH_CTL))
+
+#if IS_ENABLED(CONFIG_HYPERV)
+#define hv_enable_stimer0_percpu_irq(irq)	enable_percpu_irq(irq, 0)
+#define hv_disable_stimer0_percpu_irq(irq)	disable_percpu_irq(irq)
+#endif
+
+/* ARM64 specific code to read the hardware clock */
+static inline u64 hv_read_hwclock(void)
+{
+	u64 result;
+
+	isb();
+	result = read_sysreg(cntvct_el0);
+	isb();
+
+	return result;
+}
+
+#include <asm-generic/mshyperv.h>
+
+#endif
diff --git a/include/asm-generic/mshyperv.h b/include/asm-generic/mshyperv.h
new file mode 100644
index 000000000000..fbe1ec89c85a
--- /dev/null
+++ b/include/asm-generic/mshyperv.h
@@ -0,0 +1,240 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+
+/*
+ * Linux-specific definitions for managing interactions with Microsoft's
+ * Hyper-V hypervisor. The definitions in this file are architecture
+ * independent. See arch/<arch>/include/asm/mshyperv.h for definitions
+ * that are specific to architecture <arch>.
+ *
+ * Definitions that are specified in the Hyper-V Top Level Functional
+ * Spec (TLFS) should not go in this file, but should instead go in
+ * hyperv-tlfs.h.
+ *
+ * Copyright (C) 2018, Microsoft, Inc.
+ *
+ * Author : Michael Kelley <mikelley@microsoft.com>
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License version 2 as published
+ * by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE, GOOD TITLE or
+ * NON INFRINGEMENT.  See the GNU General Public License for more
+ * details.
+ */
+
+#ifndef _ASM_GENERIC_MSHYPERV_H
+#define _ASM_GENERIC_MSHYPERV_H
+
+#include <linux/types.h>
+#include <linux/interrupt.h>
+#include <linux/clocksource.h>
+#include <linux/irq.h>
+#include <linux/irqdesc.h>
+#include <asm/hyperv-tlfs.h>
+
+/*
+ * Hyper-V always runs with a page size of 4096. These definitions
+ * are used when communicating with Hyper-V using guest physical
+ * pages and guest physical page addresses, since the guest page
+ * size may not be 4096 on ARM64.
+ */
+#define HV_HYP_PAGE_SIZE	4096
+#define HV_HYP_PAGE_SHIFT	12
+#define HV_HYP_PAGE_MASK	(~(HV_HYP_PAGE_SIZE - 1))
+
+
+struct ms_hyperv_info {
+	u32 features;
+	u32 misc_features;
+	u32 hints;
+	u32 max_vp_index;
+	u32 max_lp_index;
+};
+extern struct ms_hyperv_info ms_hyperv;
+
+extern u64 hv_do_hypercall(u64 control, void *inputaddr, void *outputaddr);
+extern u64 hv_do_fast_hypercall8(u16 control, u64 input8);
+
+/*
+ * The guest OS needs to register the guest ID with the hypervisor.
+ * The guest ID is a 64 bit entity and the structure of this ID is
+ * specified in the Hyper-V specification:
+ *
+ * msdn.microsoft.com/en-us/library/windows/hardware/ff542653%28v=vs.85%29.aspx
+ *
+ * While the current guideline does not specify how Linux guest ID(s)
+ * need to be generated, our plan is to publish the guidelines for
+ * Linux and other guest operating systems that currently are hosted
+ * on Hyper-V. The implementation here conforms to this yet
+ * unpublished guidelines.
+ *
+ *
+ * Bit(s)
+ * 63 - Indicates if the OS is Open Source or not; 1 is Open Source
+ * 62:56 - Os Type; Linux is 0x100
+ * 55:48 - Distro specific identification
+ * 47:16 - Linux kernel version number
+ * 15:0  - Distro specific identification
+ *
+ * Generate the guest ID based on the guideline described above.
+ */
+
+static inline  __u64 generate_guest_id(__u64 d_info1, __u64 kernel_version,
+				       __u64 d_info2)
+{
+	__u64 guest_id = 0;
+
+	guest_id = (((__u64)HV_LINUX_VENDOR_ID) << 48);
+	guest_id |= (d_info1 << 48);
+	guest_id |= (kernel_version << 16);
+	guest_id |= d_info2;
+
+	return guest_id;
+}
+
+
+/* Free the message slot and signal end-of-message if required */
+static inline void vmbus_signal_eom(struct hv_message *msg, u32 old_msg_type)
+{
+	/*
+	 * On crash we're reading some other CPU's message page and we need
+	 * to be careful: this other CPU may already had cleared the header
+	 * and the host may already had delivered some other message there.
+	 * In case we blindly write msg->header.message_type we're going
+	 * to lose it. We can still lose a message of the same type but
+	 * we count on the fact that there can only be one
+	 * CHANNELMSG_UNLOAD_RESPONSE and we don't care about other messages
+	 * on crash.
+	 */
+	if (cmpxchg(&msg->header.message_type, old_msg_type,
+		    HVMSG_NONE) != old_msg_type)
+		return;
+
+	/*
+	 * Make sure the write to MessageType (ie set to
+	 * HVMSG_NONE) happens before we read the
+	 * MessagePending and EOMing. Otherwise, the EOMing
+	 * will not deliver any more messages since there is
+	 * no empty slot
+	 */
+	mb();
+
+	if (msg->header.message_flags.msg_pending) {
+		/*
+		 * This will cause message queue rescan to
+		 * possibly deliver another msg from the
+		 * hypervisor
+		 */
+		hv_signal_eom();
+	}
+}
+
+void hv_setup_vmbus_irq(void (*handler)(void));
+void hv_remove_vmbus_irq(void);
+void hv_enable_vmbus_irq(void);
+void hv_disable_vmbus_irq(void);
+
+void hv_setup_kexec_handler(void (*handler)(void));
+void hv_remove_kexec_handler(void);
+void hv_setup_crash_handler(void (*handler)(struct pt_regs *regs));
+void hv_remove_crash_handler(void);
+
+#if IS_ENABLED(CONFIG_HYPERV)
+extern struct clocksource *hyperv_cs;
+
+/*
+ * Hypervisor's notion of virtual processor ID is different from
+ * Linux' notion of CPU ID. This information can only be retrieved
+ * in the context of the calling CPU. Setup a map for easy access
+ * to this information.
+ */
+extern u32 *hv_vp_index;
+extern u32 hv_max_vp_index;
+
+/* Sentinel value for an uninitialized entry in hv_vp_index array */
+#define VP_INVAL	U32_MAX
+
+/**
+ * hv_cpu_number_to_vp_number() - Map CPU to VP.
+ * @cpu_number: CPU number in Linux terms
+ *
+ * This function returns the mapping between the Linux processor
+ * number and the hypervisor's virtual processor number, useful
+ * in making hypercalls and such that talk about specific
+ * processors.
+ *
+ * Return: Virtual processor number in Hyper-V terms
+ */
+static inline int hv_cpu_number_to_vp_number(int cpu_number)
+{
+	return hv_vp_index[cpu_number];
+}
+
+void hyperv_report_panic(struct pt_regs *regs, long err);
+void hyperv_report_panic_msg(phys_addr_t pa, size_t size);
+bool hv_is_hyperv_initialized(void);
+void hyperv_cleanup(void);
+#else /* CONFIG_HYPERV */
+static inline bool hv_is_hyperv_initialized(void) { return false; }
+static inline void hyperv_cleanup(void) {}
+#endif /* CONFIG_HYPERV */
+
+#if IS_ENABLED(CONFIG_HYPERV)
+extern int hv_setup_stimer0_irq(int *irq, int *vector, void (*handler)(void));
+extern void hv_remove_stimer0_irq(int irq);
+#endif
+
+static inline u64 hv_read_tsc_page_tsc(const struct ms_hyperv_tsc_page *tsc_pg,
+				       u64 *cur_tsc)
+{
+	u64	scale, offset;
+	u32	sequence;
+
+	/*
+	 * The protocol for reading Hyper-V TSC page is specified in Hypervisor
+	 * Top-Level Functional Specification.  To get the reference time we
+	 * must do the following:
+	 * - READ ReferenceTscSequence
+	 *   A special '0' value indicates the time source is unreliable and we
+	 *   need to use something else.
+	 * - ReferenceTime =
+	 *     ((HWclock val) * ReferenceTscScale) >> 64) + ReferenceTscOffset
+	 * - READ ReferenceTscSequence again. In case its value has changed
+	 *   since our first reading we need to discard ReferenceTime and repeat
+	 *   the whole sequence as the hypervisor was updating the page in
+	 *   between.
+	 */
+	do {
+		sequence = READ_ONCE(tsc_pg->tsc_sequence);
+		/*
+		 * Make sure we read sequence before we read other values from
+		 * TSC page.
+		 */
+		smp_rmb();
+
+		scale = READ_ONCE(tsc_pg->tsc_scale);
+		offset = READ_ONCE(tsc_pg->tsc_offset);
+		*cur_tsc = hv_read_hwclock();
+
+		/*
+		 * Make sure we read sequence after we read all other values
+		 * from TSC page.
+		 */
+		smp_rmb();
+
+	} while (READ_ONCE(tsc_pg->tsc_sequence) != sequence);
+
+	return mul_u64_u64_shr(*cur_tsc, scale, 64) + offset;
+}
+
+static inline u64 hv_read_tsc_page(const struct ms_hyperv_tsc_page *tsc_pg)
+{
+	u64 cur_tsc;
+
+	return hv_read_tsc_page_tsc(tsc_pg, &cur_tsc);
+}
+
+#endif
-- 
2.19.1

[PATCH 2/4] arm64: hyperv: Add support for Hyper-V as a hypervisor

[kys]

From: Michael Kelley <mikelley@microsoft.com>

Add ARM64-specific code to enable Hyper-V. This code includes:
* Detecting Hyper-V and initializing the guest/Hyper-V interface
* Setting up Hyper-V's synthetic clocks
* Making hypercalls using the HVC instruction
* Setting up VMbus and stimer0 interrupts
* Setting up kexec and crash handlers
This code is architecture dependent code and is mostly driven by
architecture independent code in the VMbus driver in drivers/hv/hv.c
and drivers/hv/vmbus_drv.c.

This code is built only when CONFIG_HYPERV is enabled.

Signed-off-by: Michael Kelley <mikelley@microsoft.com>
Signed-off-by: K. Y. Srinivasan <kys@microsoft.com>
---
 MAINTAINERS                  |   1 +
 arch/arm64/Makefile          |   1 +
 arch/arm64/hyperv/Makefile   |   2 +
 arch/arm64/hyperv/hv_hvc.S   |  54 +++++
 arch/arm64/hyperv/hv_init.c  | 441 +++++++++++++++++++++++++++++++++++
 arch/arm64/hyperv/mshyperv.c | 178 ++++++++++++++
 6 files changed, 677 insertions(+)
 create mode 100644 arch/arm64/hyperv/Makefile
 create mode 100644 arch/arm64/hyperv/hv_hvc.S
 create mode 100644 arch/arm64/hyperv/hv_init.c
 create mode 100644 arch/arm64/hyperv/mshyperv.c

diff --git a/MAINTAINERS b/MAINTAINERS
index 72f19cef4c48..326eeb32a0cd 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -6837,6 +6837,7 @@ F:	arch/x86/kernel/cpu/mshyperv.c
 F:	arch/x86/hyperv
 F:	arch/arm64/include/asm/hyperv-tlfs.h
 F:	arch/arm64/include/asm/mshyperv.h
+F:	arch/arm64/hyperv
 F:	drivers/hid/hid-hyperv.c
 F:	drivers/hv/
 F:	drivers/input/serio/hyperv-keyboard.c
diff --git a/arch/arm64/Makefile b/arch/arm64/Makefile
index 6cb9fc7e9382..ad9ec0579553 100644
--- a/arch/arm64/Makefile
+++ b/arch/arm64/Makefile
@@ -106,6 +106,7 @@ core-y		+= arch/arm64/kernel/ arch/arm64/mm/
 core-$(CONFIG_NET) += arch/arm64/net/
 core-$(CONFIG_KVM) += arch/arm64/kvm/
 core-$(CONFIG_XEN) += arch/arm64/xen/
+core-$(CONFIG_HYPERV) += arch/arm64/hyperv/
 core-$(CONFIG_CRYPTO) += arch/arm64/crypto/
 libs-y		:= arch/arm64/lib/ $(libs-y)
 core-$(CONFIG_EFI_STUB) += $(objtree)/drivers/firmware/efi/libstub/lib.a
diff --git a/arch/arm64/hyperv/Makefile b/arch/arm64/hyperv/Makefile
new file mode 100644
index 000000000000..988eda55330c
--- /dev/null
+++ b/arch/arm64/hyperv/Makefile
@@ -0,0 +1,2 @@
+# SPDX-License-Identifier: GPL-2.0
+obj-y		:= hv_init.o hv_hvc.o mshyperv.o
diff --git a/arch/arm64/hyperv/hv_hvc.S b/arch/arm64/hyperv/hv_hvc.S
new file mode 100644
index 000000000000..82636969b4f2
--- /dev/null
+++ b/arch/arm64/hyperv/hv_hvc.S
@@ -0,0 +1,54 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+
+/*
+ * Microsoft Hyper-V hypervisor invocation routines
+ *
+ * Copyright (C) 2018, Microsoft, Inc.
+ *
+ * Author : Michael Kelley <mikelley@microsoft.com>
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License version 2 as published
+ * by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE, GOOD TITLE or
+ * NON INFRINGEMENT.  See the GNU General Public License for more
+ * details.
+ */
+
+#include <linux/linkage.h>
+
+	.text
+/*
+ * Do the HVC instruction.  For Hyper-V the argument is always 1.
+ * x0 contains the hypercall control value, while additional registers
+ * vary depending on the hypercall, and whether the hypercall arguments
+ * are in memory or in registers (a "fast" hypercall per the Hyper-V
+ * TLFS).  When the arguments are in memory x1 is the guest physical
+ * address of the input arguments, and x2 is the guest physical
+ * address of the output arguments.  When the arguments are in
+ * registers, the register values depends on the hypercall.  Note
+ * that this version cannot return any values in registers.
+ */
+ENTRY(hv_do_hvc)
+	hvc #1
+	ret
+ENDPROC(hv_do_hvc)
+
+/*
+ * This variant of HVC invocation is for hv_get_vpreg and
+ * hv_get_vpreg_128. The input parameters are passed in registers
+ * along with a pointer in x4 to where the output result should
+ * be stored. The output is returned in x15 and x16.  x18 is used as
+ * scratch space to avoid buildng a stack frame, as Hyper-V does
+ * not preserve registers x0-x17.
+ */
+ENTRY(hv_do_hvc_fast_get)
+	mov x18, x4
+	hvc #1
+	str x15,[x18]
+	str x16,[x18,#8]
+	ret
+ENDPROC(hv_do_hvc_fast_get)
diff --git a/arch/arm64/hyperv/hv_init.c b/arch/arm64/hyperv/hv_init.c
new file mode 100644
index 000000000000..aa1a8c09d989
--- /dev/null
+++ b/arch/arm64/hyperv/hv_init.c
@@ -0,0 +1,441 @@
+// SPDX-License-Identifier: GPL-2.0
+
+/*
+ * Initialization of the interface with Microsoft's Hyper-V hypervisor,
+ * and various low level utility routines for interacting with Hyper-V.
+ *
+ * Copyright (C) 2018, Microsoft, Inc.
+ *
+ * Author : Michael Kelley <mikelley@microsoft.com>
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License version 2 as published
+ * by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE, GOOD TITLE or
+ * NON INFRINGEMENT.  See the GNU General Public License for more
+ * details.
+ */
+
+
+#include <linux/types.h>
+#include <linux/version.h>
+#include <linux/export.h>
+#include <linux/vmalloc.h>
+#include <linux/mm.h>
+#include <linux/clocksource.h>
+#include <linux/sched_clock.h>
+#include <linux/acpi.h>
+#include <linux/module.h>
+#include <linux/hyperv.h>
+#include <linux/slab.h>
+#include <linux/cpuhotplug.h>
+#include <linux/psci.h>
+#include <asm-generic/bug.h>
+#include <asm/hypervisor.h>
+#include <asm/hyperv-tlfs.h>
+#include <asm/mshyperv.h>
+#include <asm/sysreg.h>
+#include <clocksource/arm_arch_timer.h>
+
+static bool	hyperv_initialized;
+struct		ms_hyperv_info ms_hyperv;
+EXPORT_SYMBOL_GPL(ms_hyperv);
+
+static struct ms_hyperv_tsc_page *tsc_pg;
+
+struct ms_hyperv_tsc_page *hv_get_tsc_page(void)
+{
+	return tsc_pg;
+}
+EXPORT_SYMBOL_GPL(hv_get_tsc_page);
+
+static u64 read_hv_sched_clock_tsc(void)
+{
+	u64 current_tick = hv_read_tsc_page(tsc_pg);
+
+	if (current_tick == U64_MAX)
+		current_tick = hv_get_vpreg(HV_REGISTER_TIME_REFCOUNT);
+
+	return current_tick;
+}
+
+static u64 read_hv_clock_tsc(struct clocksource *arg)
+{
+	u64 current_tick = hv_read_tsc_page(tsc_pg);
+
+	if (current_tick == U64_MAX)
+		current_tick = hv_get_vpreg(HV_REGISTER_TIME_REFCOUNT);
+
+	return current_tick;
+}
+
+static struct clocksource hyperv_cs_tsc = {
+		.name		= "hyperv_clocksource_tsc_page",
+		.rating		= 400,
+		.read		= read_hv_clock_tsc,
+		.mask		= CLOCKSOURCE_MASK(64),
+		.flags		= CLOCK_SOURCE_IS_CONTINUOUS,
+};
+
+static u64 read_hv_sched_clock_msr(void)
+{
+	return hv_get_vpreg(HV_REGISTER_TIME_REFCOUNT);
+}
+
+static u64 read_hv_clock_msr(struct clocksource *arg)
+{
+	return hv_get_vpreg(HV_REGISTER_TIME_REFCOUNT);
+}
+
+static struct clocksource hyperv_cs_msr = {
+	.name		= "hyperv_clocksource_msr",
+	.rating		= 400,
+	.read		= read_hv_clock_msr,
+	.mask		= CLOCKSOURCE_MASK(64),
+	.flags		= CLOCK_SOURCE_IS_CONTINUOUS,
+};
+
+struct clocksource *hyperv_cs;
+EXPORT_SYMBOL_GPL(hyperv_cs);
+
+u32 *hv_vp_index;
+EXPORT_SYMBOL_GPL(hv_vp_index);
+
+u32 hv_max_vp_index;
+
+static int hv_cpu_init(unsigned int cpu)
+{
+	u64 msr_vp_index;
+
+	hv_get_vp_index(msr_vp_index);
+
+	hv_vp_index[smp_processor_id()] = msr_vp_index;
+
+	if (msr_vp_index > hv_max_vp_index)
+		hv_max_vp_index = msr_vp_index;
+
+	return 0;
+}
+
+/*
+ * This function is invoked via the ACPI clocksource probe mechanism. We
+ * don't actually use any values from the ACPI GTDT table, but we set up
+ * the Hyper-V synthetic clocksource and do other initialization for
+ * interacting with Hyper-V the first time.  Using early_initcall to invoke
+ * this function is too late because interrupts are already enabled at that
+ * point, and sched_clock_register must run before interrupts are enabled.
+ *
+ * 1. Setup the guest ID.
+ * 2. Get features and hints info from Hyper-V
+ * 3. Setup per-cpu VP indices.
+ * 4. Register Hyper-V specific clocksource.
+ * 5. Register the scheduler clock.
+ */
+
+static int __init hyperv_init(struct acpi_table_header *table)
+{
+	struct hv_get_vp_register_output result;
+	u32	a, b, c, d;
+	u64	guest_id;
+	int	i;
+
+	/*
+	 * If we're in a VM on Hyper-V, the ACPI hypervisor_id field will
+	 * have the string "MsHyperV".
+	 */
+	if (strncmp((char *)&acpi_gbl_FADT.hypervisor_id, "MsHyperV", 8))
+		return 1;
+
+	/* Setup the guest ID */
+	guest_id = generate_guest_id(0, LINUX_VERSION_CODE, 0);
+	hv_set_vpreg(HV_REGISTER_GUEST_OSID, guest_id);
+
+	/* Get the features and hints from Hyper-V */
+	hv_get_vpreg_128(HV_REGISTER_PRIVILEGES_AND_FEATURES, &result);
+	ms_hyperv.features = lower_32_bits(result.registervaluelow);
+	ms_hyperv.misc_features = upper_32_bits(result.registervaluehigh);
+
+	hv_get_vpreg_128(HV_REGISTER_FEATURES, &result);
+	ms_hyperv.hints = lower_32_bits(result.registervaluelow);
+
+	pr_info("Hyper-V: Features 0x%x, hints 0x%x\n",
+		ms_hyperv.features, ms_hyperv.hints);
+
+	/*
+	 * Direct mode is the only option for STIMERs provided Hyper-V
+	 * on ARM64, so Hyper-V doesn't actually set the flag.  But add the
+	 * flag so the architecture independent code in drivers/hv/hv.c
+	 * will correctly use that mode.
+	 */
+	ms_hyperv.misc_features |= HV_STIMER_DIRECT_MODE_AVAILABLE;
+
+	/*
+	 * Hyper-V on ARM64 doesn't support AutoEOI.  Add the hint
+	 * that tells architecture independent code not to use this
+	 * feature.
+	 */
+	ms_hyperv.hints |= HV_DEPRECATING_AEOI_RECOMMENDED;
+
+	/* Get information about the Hyper-V host version */
+	hv_get_vpreg_128(HV_REGISTER_HYPERVISOR_VERSION, &result);
+	a = lower_32_bits(result.registervaluelow);
+	b = upper_32_bits(result.registervaluelow);
+	c = lower_32_bits(result.registervaluehigh);
+	d = upper_32_bits(result.registervaluehigh);
+	pr_info("Hyper-V: Host Build %d.%d.%d.%d-%d-%d\n",
+		b >> 16, b & 0xFFFF, a, d & 0xFFFFFF, c, d >> 24);
+
+	/* Allocate percpu VP index */
+	hv_vp_index = kmalloc_array(num_possible_cpus(), sizeof(*hv_vp_index),
+				    GFP_KERNEL);
+	if (!hv_vp_index)
+		return 1;
+
+	for (i = 0; i < num_possible_cpus(); i++)
+		hv_vp_index[i] = VP_INVAL;
+
+	if (cpuhp_setup_state(CPUHP_AP_ONLINE_DYN, "arm64/hyperv_init:online",
+			      hv_cpu_init, NULL) < 0)
+		goto free_vp_index;
+
+	/*
+	 * Try to set up what Hyper-V calls the "TSC reference page", which
+	 * uses the ARM Generic Timer virtual counter with some scaling
+	 * information to provide a fast and stable guest VM clocksource.
+	 * If the TSC reference page can't be set up, fall back to reading
+	 * the guest clock provided by Hyper-V's synthetic reference time
+	 * register.
+	 */
+	if (ms_hyperv.features & HV_MSR_REFERENCE_TSC_AVAILABLE) {
+
+		u64		tsc_msr;
+		phys_addr_t	phys_addr;
+
+		tsc_pg = __vmalloc(HV_HYP_PAGE_SIZE, GFP_KERNEL, PAGE_KERNEL);
+		if (tsc_pg) {
+			phys_addr = page_to_phys(vmalloc_to_page(tsc_pg));
+			tsc_msr = hv_get_vpreg(HV_REGISTER_REFERENCE_TSC);
+			tsc_msr &= GENMASK_ULL(11, 0);
+			tsc_msr = tsc_msr | 0x1 | (u64)phys_addr;
+			hv_set_vpreg(HV_REGISTER_REFERENCE_TSC, tsc_msr);
+			hyperv_cs = &hyperv_cs_tsc;
+			sched_clock_register(read_hv_sched_clock_tsc,
+						64, HV_CLOCK_HZ);
+		}
+	}
+
+	if (!hyperv_cs &&
+	    (ms_hyperv.features & HV_MSR_TIME_REF_COUNT_AVAILABLE)) {
+		hyperv_cs = &hyperv_cs_msr;
+		sched_clock_register(read_hv_sched_clock_msr,
+						64, HV_CLOCK_HZ);
+	}
+
+	if (hyperv_cs) {
+		hyperv_cs->archdata.vdso_direct = false;
+		clocksource_register_hz(hyperv_cs, HV_CLOCK_HZ);
+	}
+
+	hyperv_initialized = true;
+	return 0;
+
+free_vp_index:
+	kfree(hv_vp_index);
+	hv_vp_index = NULL;
+	return 1;
+}
+TIMER_ACPI_DECLARE(hyperv, ACPI_SIG_GTDT, hyperv_init);
+
+/*
+ * This routine is called before kexec/kdump, it does the required cleanup.
+ */
+void hyperv_cleanup(void)
+{
+	/* Reset our OS id */
+	hv_set_vpreg(HV_REGISTER_GUEST_OSID, 0);
+
+}
+EXPORT_SYMBOL_GPL(hyperv_cleanup);
+
+/*
+ * hv_do_hypercall- Invoke the specified hypercall
+ */
+u64 hv_do_hypercall(u64 control, void *input, void *output)
+{
+	u64 input_address;
+	u64 output_address;
+
+	input_address = input ? virt_to_phys(input) : 0;
+	output_address = output ? virt_to_phys(output) : 0;
+	return hv_do_hvc(control, input_address, output_address);
+}
+EXPORT_SYMBOL_GPL(hv_do_hypercall);
+
+/*
+ * hv_do_fast_hypercall8 -- Invoke the specified hypercall
+ * with arguments in registers instead of physical memory.
+ * Avoids the overhead of virt_to_phys for simple hypercalls.
+ */
+
+u64 hv_do_fast_hypercall8(u16 code, u64 input)
+{
+	u64 control;
+
+	control = (u64)code | HV_HYPERCALL_FAST_BIT;
+	return hv_do_hvc(control, input);
+}
+EXPORT_SYMBOL_GPL(hv_do_fast_hypercall8);
+
+
+/*
+ * Set a single VP register to a 64-bit value.
+ */
+void hv_set_vpreg(u32 msr, u64 value)
+{
+	union hv_hypercall_status status;
+
+	status.as_uint64 = hv_do_hvc(
+		HVCALL_SET_VP_REGISTERS | HV_HYPERCALL_FAST_BIT |
+			HV_HYPERCALL_REP_COUNT_1,
+		HV_PARTITION_ID_SELF,
+		HV_VP_INDEX_SELF,
+		msr,
+		0,
+		value,
+		0);
+
+	/*
+	 * Something is fundamentally broken in the hypervisor if
+	 * setting a VP register fails. There's really no way to
+	 * continue as a guest VM, so panic.
+	 */
+	BUG_ON(status.status != HV_STATUS_SUCCESS);
+}
+EXPORT_SYMBOL_GPL(hv_set_vpreg);
+
+
+/*
+ * Get the value of a single VP register, and only the low order 64 bits.
+ */
+u64 hv_get_vpreg(u32 msr)
+{
+	union hv_hypercall_status status;
+	struct hv_get_vp_register_output output;
+
+	status.as_uint64 = hv_do_hvc_fast_get(
+		HVCALL_GET_VP_REGISTERS | HV_HYPERCALL_FAST_BIT |
+			HV_HYPERCALL_REP_COUNT_1,
+		HV_PARTITION_ID_SELF,
+		HV_VP_INDEX_SELF,
+		msr,
+		&output);
+
+	/*
+	 * Something is fundamentally broken in the hypervisor if
+	 * getting a VP register fails. There's really no way to
+	 * continue as a guest VM, so panic.
+	 */
+	BUG_ON(status.status != HV_STATUS_SUCCESS);
+
+	return output.registervaluelow;
+}
+EXPORT_SYMBOL_GPL(hv_get_vpreg);
+
+/*
+ * Get the value of a single VP register that is 128 bits in size.  This is a
+ * separate call in order to avoid complicating the calling sequence for
+ * the much more frequently used 64-bit version.
+ */
+void hv_get_vpreg_128(u32 msr, struct hv_get_vp_register_output *result)
+{
+	union hv_hypercall_status status;
+
+	status.as_uint64 = hv_do_hvc_fast_get(
+		HVCALL_GET_VP_REGISTERS | HV_HYPERCALL_FAST_BIT |
+			HV_HYPERCALL_REP_COUNT_1,
+		HV_PARTITION_ID_SELF,
+		HV_VP_INDEX_SELF,
+		msr,
+		result);
+
+	/*
+	 * Something is fundamentally broken in the hypervisor if
+	 * getting a VP register fails. There's really no way to
+	 * continue as a guest VM, so panic.
+	 */
+	BUG_ON(status.status != HV_STATUS_SUCCESS);
+
+	return;
+
+}
+EXPORT_SYMBOL_GPL(hv_get_vpreg_128);
+
+void hyperv_report_panic(struct pt_regs *regs, long err)
+{
+	static bool panic_reported;
+	u64 guest_id;
+
+	/*
+	 * We prefer to report panic on 'die' chain as we have proper
+	 * registers to report, but if we miss it (e.g. on BUG()) we need
+	 * to report it on 'panic'.
+	 */
+	if (panic_reported)
+		return;
+	panic_reported = true;
+
+	guest_id = hv_get_vpreg(HV_REGISTER_GUEST_OSID);
+
+	/*
+	 * Hyper-V provides the ability to store only 5 values.
+	 * Pick the passed in error value, the guest_id, and the PC.
+	 * The first two general registers are added arbitrarily.
+	 */
+	hv_set_vpreg(HV_REGISTER_CRASH_P0, err);
+	hv_set_vpreg(HV_REGISTER_CRASH_P1, guest_id);
+	hv_set_vpreg(HV_REGISTER_CRASH_P2, regs->pc);
+	hv_set_vpreg(HV_REGISTER_CRASH_P3, regs->regs[0]);
+	hv_set_vpreg(HV_REGISTER_CRASH_P4, regs->regs[1]);
+
+	/*
+	 * Let Hyper-V know there is crash data available
+	 */
+	hv_set_vpreg(HV_REGISTER_CRASH_CTL, HV_CRASH_CTL_CRASH_NOTIFY);
+}
+EXPORT_SYMBOL_GPL(hyperv_report_panic);
+
+/*
+ * hyperv_report_panic_msg - report panic message to Hyper-V
+ * @pa: physical address of the panic page containing the message
+ * @size: size of the message in the page
+ */
+void hyperv_report_panic_msg(phys_addr_t pa, size_t size)
+{
+	/*
+	 * P3 to contain the physical address of the panic page & P4 to
+	 * contain the size of the panic data in that page. Rest of the
+	 * registers are no-op when the NOTIFY_MSG flag is set.
+	 */
+	hv_set_vpreg(HV_REGISTER_CRASH_P0, 0);
+	hv_set_vpreg(HV_REGISTER_CRASH_P1, 0);
+	hv_set_vpreg(HV_REGISTER_CRASH_P2, 0);
+	hv_set_vpreg(HV_REGISTER_CRASH_P3, pa);
+	hv_set_vpreg(HV_REGISTER_CRASH_P4, size);
+
+	/*
+	 * Let Hyper-V know there is crash data available along with
+	 * the panic message.
+	 */
+	hv_set_vpreg(HV_REGISTER_CRASH_CTL,
+	       (HV_CRASH_CTL_CRASH_NOTIFY | HV_CRASH_CTL_CRASH_NOTIFY_MSG));
+}
+EXPORT_SYMBOL_GPL(hyperv_report_panic_msg);
+
+bool hv_is_hyperv_initialized(void)
+{
+	return hyperv_initialized;
+}
+EXPORT_SYMBOL_GPL(hv_is_hyperv_initialized);
diff --git a/arch/arm64/hyperv/mshyperv.c b/arch/arm64/hyperv/mshyperv.c
new file mode 100644
index 000000000000..3ef055599412
--- /dev/null
+++ b/arch/arm64/hyperv/mshyperv.c
@@ -0,0 +1,178 @@
+// SPDX-License-Identifier: GPL-2.0
+
+/*
+ * Core routines for interacting with Microsoft's Hyper-V hypervisor,
+ * including setting up VMbus and STIMER interrupts, and handling
+ * crashes and kexecs. These interactions are through a set of
+ * static "handler" variables set by the architecture independent
+ * VMbus and STIMER drivers.  This design is used to meet x86/x64
+ * requirements for avoiding direct linkages and allowing the VMbus
+ * and STIMER drivers to be unloaded and reloaded.
+ *
+ * Copyright (C) 2018, Microsoft, Inc.
+ *
+ * Author : Michael Kelley <mikelley@microsoft.com>
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License version 2 as published
+ * by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE, GOOD TITLE or
+ * NON INFRINGEMENT.  See the GNU General Public License for more
+ * details.
+ */
+
+#include <linux/types.h>
+#include <linux/export.h>
+#include <linux/interrupt.h>
+#include <linux/kexec.h>
+#include <linux/acpi.h>
+#include <linux/ptrace.h>
+#include <asm/hyperv-tlfs.h>
+#include <asm/mshyperv.h>
+
+static void (*vmbus_handler)(void);
+static void (*hv_stimer0_handler)(void);
+static void (*hv_kexec_handler)(void);
+static void (*hv_crash_handler)(struct pt_regs *regs);
+
+static int vmbus_irq;
+static long __percpu *vmbus_evt;
+static long __percpu *stimer0_evt;
+
+irqreturn_t hyperv_vector_handler(int irq, void *dev_id)
+{
+	if (vmbus_handler)
+		vmbus_handler();
+	return IRQ_HANDLED;
+}
+
+/* Must be done just once */
+void hv_setup_vmbus_irq(void (*handler)(void))
+{
+	int result;
+
+	vmbus_handler = handler;
+	vmbus_irq = acpi_register_gsi(NULL, HYPERVISOR_CALLBACK_VECTOR,
+				 ACPI_LEVEL_SENSITIVE, ACPI_ACTIVE_HIGH);
+	if (vmbus_irq <= 0) {
+		pr_err("Can't register Hyper-V VMBus GSI. Error %d",
+			vmbus_irq);
+		vmbus_irq = 0;
+		return;
+	}
+	vmbus_evt = alloc_percpu(long);
+	result = request_percpu_irq(vmbus_irq, hyperv_vector_handler,
+			"Hyper-V VMbus", vmbus_evt);
+	if (result) {
+		pr_err("Can't request Hyper-V VMBus IRQ %d. Error %d",
+			vmbus_irq, result);
+		free_percpu(vmbus_evt);
+		acpi_unregister_gsi(vmbus_irq);
+		vmbus_irq = 0;
+	}
+}
+EXPORT_SYMBOL_GPL(hv_setup_vmbus_irq);
+
+/* Must be done just once */
+void hv_remove_vmbus_irq(void)
+{
+	if (vmbus_irq) {
+		free_percpu_irq(vmbus_irq, vmbus_evt);
+		free_percpu(vmbus_evt);
+		acpi_unregister_gsi(vmbus_irq);
+	}
+}
+EXPORT_SYMBOL_GPL(hv_remove_vmbus_irq);
+
+/* Must be done by each CPU */
+void hv_enable_vmbus_irq(void)
+{
+	enable_percpu_irq(vmbus_irq, 0);
+}
+EXPORT_SYMBOL_GPL(hv_enable_vmbus_irq);
+
+/* Must be done by each CPU */
+void hv_disable_vmbus_irq(void)
+{
+	disable_percpu_irq(vmbus_irq);
+}
+EXPORT_SYMBOL_GPL(hv_disable_vmbus_irq);
+
+/* Routines to do per-architecture handling of STIMER0 when in Direct Mode */
+
+static irqreturn_t hv_stimer0_vector_handler(int irq, void *dev_id)
+{
+	if (hv_stimer0_handler)
+		hv_stimer0_handler();
+	return IRQ_HANDLED;
+}
+
+int hv_setup_stimer0_irq(int *irq, int *vector, void (*handler)(void))
+{
+	int localirq;
+	int result;
+
+	localirq = acpi_register_gsi(NULL, HV_STIMER0_IRQNR,
+			ACPI_LEVEL_SENSITIVE, ACPI_ACTIVE_HIGH);
+	if (localirq <= 0) {
+		pr_err("Can't register Hyper-V stimer0 GSI. Error %d",
+			localirq);
+		*irq = 0;
+		return -1;
+	}
+	stimer0_evt = alloc_percpu(long);
+	result = request_percpu_irq(localirq, hv_stimer0_vector_handler,
+					 "Hyper-V stimer0", stimer0_evt);
+	if (result) {
+		pr_err("Can't request Hyper-V stimer0 IRQ %d. Error %d",
+			localirq, result);
+		free_percpu(stimer0_evt);
+		acpi_unregister_gsi(localirq);
+		*irq = 0;
+		return -1;
+	}
+
+	hv_stimer0_handler = handler;
+	*vector = HV_STIMER0_IRQNR;
+	*irq = localirq;
+	return 0;
+}
+EXPORT_SYMBOL_GPL(hv_setup_stimer0_irq);
+
+void hv_remove_stimer0_irq(int irq)
+{
+	hv_stimer0_handler = NULL;
+	if (irq) {
+		free_percpu_irq(irq, stimer0_evt);
+		free_percpu(stimer0_evt);
+		acpi_unregister_gsi(irq);
+	}
+}
+EXPORT_SYMBOL_GPL(hv_remove_stimer0_irq);
+
+void hv_setup_kexec_handler(void (*handler)(void))
+{
+	hv_kexec_handler = handler;
+}
+EXPORT_SYMBOL_GPL(hv_setup_kexec_handler);
+
+void hv_remove_kexec_handler(void)
+{
+	hv_kexec_handler = NULL;
+}
+EXPORT_SYMBOL_GPL(hv_remove_kexec_handler);
+
+void hv_setup_crash_handler(void (*handler)(struct pt_regs *regs))
+{
+	hv_crash_handler = handler;
+}
+EXPORT_SYMBOL_GPL(hv_setup_crash_handler);
+
+void hv_remove_crash_handler(void)
+{
+	hv_crash_handler = NULL;
+}
+EXPORT_SYMBOL_GPL(hv_remove_crash_handler);
-- 
2.19.1

[PATCH 2/4] arm64: hyperv: Add support for Hyper-V as a hypervisor

[kys at linuxonhyperv.com]

From: Michael Kelley <mikelley@microsoft.com>

Add ARM64-specific code to enable Hyper-V. This code includes:
* Detecting Hyper-V and initializing the guest/Hyper-V interface
* Setting up Hyper-V's synthetic clocks
* Making hypercalls using the HVC instruction
* Setting up VMbus and stimer0 interrupts
* Setting up kexec and crash handlers
This code is architecture dependent code and is mostly driven by
architecture independent code in the VMbus driver in drivers/hv/hv.c
and drivers/hv/vmbus_drv.c.

This code is built only when CONFIG_HYPERV is enabled.

Signed-off-by: Michael Kelley <mikelley@microsoft.com>
Signed-off-by: K. Y. Srinivasan <kys@microsoft.com>
---
 MAINTAINERS                  |   1 +
 arch/arm64/Makefile          |   1 +
 arch/arm64/hyperv/Makefile   |   2 +
 arch/arm64/hyperv/hv_hvc.S   |  54 +++++
 arch/arm64/hyperv/hv_init.c  | 441 +++++++++++++++++++++++++++++++++++
 arch/arm64/hyperv/mshyperv.c | 178 ++++++++++++++
 6 files changed, 677 insertions(+)
 create mode 100644 arch/arm64/hyperv/Makefile
 create mode 100644 arch/arm64/hyperv/hv_hvc.S
 create mode 100644 arch/arm64/hyperv/hv_init.c
 create mode 100644 arch/arm64/hyperv/mshyperv.c

diff --git a/MAINTAINERS b/MAINTAINERS
index 72f19cef4c48..326eeb32a0cd 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -6837,6 +6837,7 @@ F:	arch/x86/kernel/cpu/mshyperv.c
 F:	arch/x86/hyperv
 F:	arch/arm64/include/asm/hyperv-tlfs.h
 F:	arch/arm64/include/asm/mshyperv.h
+F:	arch/arm64/hyperv
 F:	drivers/hid/hid-hyperv.c
 F:	drivers/hv/
 F:	drivers/input/serio/hyperv-keyboard.c
diff --git a/arch/arm64/Makefile b/arch/arm64/Makefile
index 6cb9fc7e9382..ad9ec0579553 100644
--- a/arch/arm64/Makefile
+++ b/arch/arm64/Makefile
@@ -106,6 +106,7 @@ core-y		+= arch/arm64/kernel/ arch/arm64/mm/
 core-$(CONFIG_NET) += arch/arm64/net/
 core-$(CONFIG_KVM) += arch/arm64/kvm/
 core-$(CONFIG_XEN) += arch/arm64/xen/
+core-$(CONFIG_HYPERV) += arch/arm64/hyperv/
 core-$(CONFIG_CRYPTO) += arch/arm64/crypto/
 libs-y		:= arch/arm64/lib/ $(libs-y)
 core-$(CONFIG_EFI_STUB) += $(objtree)/drivers/firmware/efi/libstub/lib.a
diff --git a/arch/arm64/hyperv/Makefile b/arch/arm64/hyperv/Makefile
new file mode 100644
index 000000000000..988eda55330c
--- /dev/null
+++ b/arch/arm64/hyperv/Makefile
@@ -0,0 +1,2 @@
+# SPDX-License-Identifier: GPL-2.0
+obj-y		:= hv_init.o hv_hvc.o mshyperv.o
diff --git a/arch/arm64/hyperv/hv_hvc.S b/arch/arm64/hyperv/hv_hvc.S
new file mode 100644
index 000000000000..82636969b4f2
--- /dev/null
+++ b/arch/arm64/hyperv/hv_hvc.S
@@ -0,0 +1,54 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+
+/*
+ * Microsoft Hyper-V hypervisor invocation routines
+ *
+ * Copyright (C) 2018, Microsoft, Inc.
+ *
+ * Author : Michael Kelley <mikelley@microsoft.com>
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License version 2 as published
+ * by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE, GOOD TITLE or
+ * NON INFRINGEMENT.  See the GNU General Public License for more
+ * details.
+ */
+
+#include <linux/linkage.h>
+
+	.text
+/*
+ * Do the HVC instruction.  For Hyper-V the argument is always 1.
+ * x0 contains the hypercall control value, while additional registers
+ * vary depending on the hypercall, and whether the hypercall arguments
+ * are in memory or in registers (a "fast" hypercall per the Hyper-V
+ * TLFS).  When the arguments are in memory x1 is the guest physical
+ * address of the input arguments, and x2 is the guest physical
+ * address of the output arguments.  When the arguments are in
+ * registers, the register values depends on the hypercall.  Note
+ * that this version cannot return any values in registers.
+ */
+ENTRY(hv_do_hvc)
+	hvc #1
+	ret
+ENDPROC(hv_do_hvc)
+
+/*
+ * This variant of HVC invocation is for hv_get_vpreg and
+ * hv_get_vpreg_128. The input parameters are passed in registers
+ * along with a pointer in x4 to where the output result should
+ * be stored. The output is returned in x15 and x16.  x18 is used as
+ * scratch space to avoid buildng a stack frame, as Hyper-V does
+ * not preserve registers x0-x17.
+ */
+ENTRY(hv_do_hvc_fast_get)
+	mov x18, x4
+	hvc #1
+	str x15,[x18]
+	str x16,[x18,#8]
+	ret
+ENDPROC(hv_do_hvc_fast_get)
diff --git a/arch/arm64/hyperv/hv_init.c b/arch/arm64/hyperv/hv_init.c
new file mode 100644
index 000000000000..aa1a8c09d989
--- /dev/null
+++ b/arch/arm64/hyperv/hv_init.c
@@ -0,0 +1,441 @@
+// SPDX-License-Identifier: GPL-2.0
+
+/*
+ * Initialization of the interface with Microsoft's Hyper-V hypervisor,
+ * and various low level utility routines for interacting with Hyper-V.
+ *
+ * Copyright (C) 2018, Microsoft, Inc.
+ *
+ * Author : Michael Kelley <mikelley@microsoft.com>
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License version 2 as published
+ * by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE, GOOD TITLE or
+ * NON INFRINGEMENT.  See the GNU General Public License for more
+ * details.
+ */
+
+
+#include <linux/types.h>
+#include <linux/version.h>
+#include <linux/export.h>
+#include <linux/vmalloc.h>
+#include <linux/mm.h>
+#include <linux/clocksource.h>
+#include <linux/sched_clock.h>
+#include <linux/acpi.h>
+#include <linux/module.h>
+#include <linux/hyperv.h>
+#include <linux/slab.h>
+#include <linux/cpuhotplug.h>
+#include <linux/psci.h>
+#include <asm-generic/bug.h>
+#include <asm/hypervisor.h>
+#include <asm/hyperv-tlfs.h>
+#include <asm/mshyperv.h>
+#include <asm/sysreg.h>
+#include <clocksource/arm_arch_timer.h>
+
+static bool	hyperv_initialized;
+struct		ms_hyperv_info ms_hyperv;
+EXPORT_SYMBOL_GPL(ms_hyperv);
+
+static struct ms_hyperv_tsc_page *tsc_pg;
+
+struct ms_hyperv_tsc_page *hv_get_tsc_page(void)
+{
+	return tsc_pg;
+}
+EXPORT_SYMBOL_GPL(hv_get_tsc_page);
+
+static u64 read_hv_sched_clock_tsc(void)
+{
+	u64 current_tick = hv_read_tsc_page(tsc_pg);
+
+	if (current_tick == U64_MAX)
+		current_tick = hv_get_vpreg(HV_REGISTER_TIME_REFCOUNT);
+
+	return current_tick;
+}
+
+static u64 read_hv_clock_tsc(struct clocksource *arg)
+{
+	u64 current_tick = hv_read_tsc_page(tsc_pg);
+
+	if (current_tick == U64_MAX)
+		current_tick = hv_get_vpreg(HV_REGISTER_TIME_REFCOUNT);
+
+	return current_tick;
+}
+
+static struct clocksource hyperv_cs_tsc = {
+		.name		= "hyperv_clocksource_tsc_page",
+		.rating		= 400,
+		.read		= read_hv_clock_tsc,
+		.mask		= CLOCKSOURCE_MASK(64),
+		.flags		= CLOCK_SOURCE_IS_CONTINUOUS,
+};
+
+static u64 read_hv_sched_clock_msr(void)
+{
+	return hv_get_vpreg(HV_REGISTER_TIME_REFCOUNT);
+}
+
+static u64 read_hv_clock_msr(struct clocksource *arg)
+{
+	return hv_get_vpreg(HV_REGISTER_TIME_REFCOUNT);
+}
+
+static struct clocksource hyperv_cs_msr = {
+	.name		= "hyperv_clocksource_msr",
+	.rating		= 400,
+	.read		= read_hv_clock_msr,
+	.mask		= CLOCKSOURCE_MASK(64),
+	.flags		= CLOCK_SOURCE_IS_CONTINUOUS,
+};
+
+struct clocksource *hyperv_cs;
+EXPORT_SYMBOL_GPL(hyperv_cs);
+
+u32 *hv_vp_index;
+EXPORT_SYMBOL_GPL(hv_vp_index);
+
+u32 hv_max_vp_index;
+
+static int hv_cpu_init(unsigned int cpu)
+{
+	u64 msr_vp_index;
+
+	hv_get_vp_index(msr_vp_index);
+
+	hv_vp_index[smp_processor_id()] = msr_vp_index;
+
+	if (msr_vp_index > hv_max_vp_index)
+		hv_max_vp_index = msr_vp_index;
+
+	return 0;
+}
+
+/*
+ * This function is invoked via the ACPI clocksource probe mechanism. We
+ * don't actually use any values from the ACPI GTDT table, but we set up
+ * the Hyper-V synthetic clocksource and do other initialization for
+ * interacting with Hyper-V the first time.  Using early_initcall to invoke
+ * this function is too late because interrupts are already enabled at that
+ * point, and sched_clock_register must run before interrupts are enabled.
+ *
+ * 1. Setup the guest ID.
+ * 2. Get features and hints info from Hyper-V
+ * 3. Setup per-cpu VP indices.
+ * 4. Register Hyper-V specific clocksource.
+ * 5. Register the scheduler clock.
+ */
+
+static int __init hyperv_init(struct acpi_table_header *table)
+{
+	struct hv_get_vp_register_output result;
+	u32	a, b, c, d;
+	u64	guest_id;
+	int	i;
+
+	/*
+	 * If we're in a VM on Hyper-V, the ACPI hypervisor_id field will
+	 * have the string "MsHyperV".
+	 */
+	if (strncmp((char *)&acpi_gbl_FADT.hypervisor_id, "MsHyperV", 8))
+		return 1;
+
+	/* Setup the guest ID */
+	guest_id = generate_guest_id(0, LINUX_VERSION_CODE, 0);
+	hv_set_vpreg(HV_REGISTER_GUEST_OSID, guest_id);
+
+	/* Get the features and hints from Hyper-V */
+	hv_get_vpreg_128(HV_REGISTER_PRIVILEGES_AND_FEATURES, &result);
+	ms_hyperv.features = lower_32_bits(result.registervaluelow);
+	ms_hyperv.misc_features = upper_32_bits(result.registervaluehigh);
+
+	hv_get_vpreg_128(HV_REGISTER_FEATURES, &result);
+	ms_hyperv.hints = lower_32_bits(result.registervaluelow);
+
+	pr_info("Hyper-V: Features 0x%x, hints 0x%x\n",
+		ms_hyperv.features, ms_hyperv.hints);
+
+	/*
+	 * Direct mode is the only option for STIMERs provided Hyper-V
+	 * on ARM64, so Hyper-V doesn't actually set the flag.  But add the
+	 * flag so the architecture independent code in drivers/hv/hv.c
+	 * will correctly use that mode.
+	 */
+	ms_hyperv.misc_features |= HV_STIMER_DIRECT_MODE_AVAILABLE;
+
+	/*
+	 * Hyper-V on ARM64 doesn't support AutoEOI.  Add the hint
+	 * that tells architecture independent code not to use this
+	 * feature.
+	 */
+	ms_hyperv.hints |= HV_DEPRECATING_AEOI_RECOMMENDED;
+
+	/* Get information about the Hyper-V host version */
+	hv_get_vpreg_128(HV_REGISTER_HYPERVISOR_VERSION, &result);
+	a = lower_32_bits(result.registervaluelow);
+	b = upper_32_bits(result.registervaluelow);
+	c = lower_32_bits(result.registervaluehigh);
+	d = upper_32_bits(result.registervaluehigh);
+	pr_info("Hyper-V: Host Build %d.%d.%d.%d-%d-%d\n",
+		b >> 16, b & 0xFFFF, a, d & 0xFFFFFF, c, d >> 24);
+
+	/* Allocate percpu VP index */
+	hv_vp_index = kmalloc_array(num_possible_cpus(), sizeof(*hv_vp_index),
+				    GFP_KERNEL);
+	if (!hv_vp_index)
+		return 1;
+
+	for (i = 0; i < num_possible_cpus(); i++)
+		hv_vp_index[i] = VP_INVAL;
+
+	if (cpuhp_setup_state(CPUHP_AP_ONLINE_DYN, "arm64/hyperv_init:online",
+			      hv_cpu_init, NULL) < 0)
+		goto free_vp_index;
+
+	/*
+	 * Try to set up what Hyper-V calls the "TSC reference page", which
+	 * uses the ARM Generic Timer virtual counter with some scaling
+	 * information to provide a fast and stable guest VM clocksource.
+	 * If the TSC reference page can't be set up, fall back to reading
+	 * the guest clock provided by Hyper-V's synthetic reference time
+	 * register.
+	 */
+	if (ms_hyperv.features & HV_MSR_REFERENCE_TSC_AVAILABLE) {
+
+		u64		tsc_msr;
+		phys_addr_t	phys_addr;
+
+		tsc_pg = __vmalloc(HV_HYP_PAGE_SIZE, GFP_KERNEL, PAGE_KERNEL);
+		if (tsc_pg) {
+			phys_addr = page_to_phys(vmalloc_to_page(tsc_pg));
+			tsc_msr = hv_get_vpreg(HV_REGISTER_REFERENCE_TSC);
+			tsc_msr &= GENMASK_ULL(11, 0);
+			tsc_msr = tsc_msr | 0x1 | (u64)phys_addr;
+			hv_set_vpreg(HV_REGISTER_REFERENCE_TSC, tsc_msr);
+			hyperv_cs = &hyperv_cs_tsc;
+			sched_clock_register(read_hv_sched_clock_tsc,
+						64, HV_CLOCK_HZ);
+		}
+	}
+
+	if (!hyperv_cs &&
+	    (ms_hyperv.features & HV_MSR_TIME_REF_COUNT_AVAILABLE)) {
+		hyperv_cs = &hyperv_cs_msr;
+		sched_clock_register(read_hv_sched_clock_msr,
+						64, HV_CLOCK_HZ);
+	}
+
+	if (hyperv_cs) {
+		hyperv_cs->archdata.vdso_direct = false;
+		clocksource_register_hz(hyperv_cs, HV_CLOCK_HZ);
+	}
+
+	hyperv_initialized = true;
+	return 0;
+
+free_vp_index:
+	kfree(hv_vp_index);
+	hv_vp_index = NULL;
+	return 1;
+}
+TIMER_ACPI_DECLARE(hyperv, ACPI_SIG_GTDT, hyperv_init);
+
+/*
+ * This routine is called before kexec/kdump, it does the required cleanup.
+ */
+void hyperv_cleanup(void)
+{
+	/* Reset our OS id */
+	hv_set_vpreg(HV_REGISTER_GUEST_OSID, 0);
+
+}
+EXPORT_SYMBOL_GPL(hyperv_cleanup);
+
+/*
+ * hv_do_hypercall- Invoke the specified hypercall
+ */
+u64 hv_do_hypercall(u64 control, void *input, void *output)
+{
+	u64 input_address;
+	u64 output_address;
+
+	input_address = input ? virt_to_phys(input) : 0;
+	output_address = output ? virt_to_phys(output) : 0;
+	return hv_do_hvc(control, input_address, output_address);
+}
+EXPORT_SYMBOL_GPL(hv_do_hypercall);
+
+/*
+ * hv_do_fast_hypercall8 -- Invoke the specified hypercall
+ * with arguments in registers instead of physical memory.
+ * Avoids the overhead of virt_to_phys for simple hypercalls.
+ */
+
+u64 hv_do_fast_hypercall8(u16 code, u64 input)
+{
+	u64 control;
+
+	control = (u64)code | HV_HYPERCALL_FAST_BIT;
+	return hv_do_hvc(control, input);
+}
+EXPORT_SYMBOL_GPL(hv_do_fast_hypercall8);
+
+
+/*
+ * Set a single VP register to a 64-bit value.
+ */
+void hv_set_vpreg(u32 msr, u64 value)
+{
+	union hv_hypercall_status status;
+
+	status.as_uint64 = hv_do_hvc(
+		HVCALL_SET_VP_REGISTERS | HV_HYPERCALL_FAST_BIT |
+			HV_HYPERCALL_REP_COUNT_1,
+		HV_PARTITION_ID_SELF,
+		HV_VP_INDEX_SELF,
+		msr,
+		0,
+		value,
+		0);
+
+	/*
+	 * Something is fundamentally broken in the hypervisor if
+	 * setting a VP register fails. There's really no way to
+	 * continue as a guest VM, so panic.
+	 */
+	BUG_ON(status.status != HV_STATUS_SUCCESS);
+}
+EXPORT_SYMBOL_GPL(hv_set_vpreg);
+
+
+/*
+ * Get the value of a single VP register, and only the low order 64 bits.
+ */
+u64 hv_get_vpreg(u32 msr)
+{
+	union hv_hypercall_status status;
+	struct hv_get_vp_register_output output;
+
+	status.as_uint64 = hv_do_hvc_fast_get(
+		HVCALL_GET_VP_REGISTERS | HV_HYPERCALL_FAST_BIT |
+			HV_HYPERCALL_REP_COUNT_1,
+		HV_PARTITION_ID_SELF,
+		HV_VP_INDEX_SELF,
+		msr,
+		&output);
+
+	/*
+	 * Something is fundamentally broken in the hypervisor if
+	 * getting a VP register fails. There's really no way to
+	 * continue as a guest VM, so panic.
+	 */
+	BUG_ON(status.status != HV_STATUS_SUCCESS);
+
+	return output.registervaluelow;
+}
+EXPORT_SYMBOL_GPL(hv_get_vpreg);
+
+/*
+ * Get the value of a single VP register that is 128 bits in size.  This is a
+ * separate call in order to avoid complicating the calling sequence for
+ * the much more frequently used 64-bit version.
+ */
+void hv_get_vpreg_128(u32 msr, struct hv_get_vp_register_output *result)
+{
+	union hv_hypercall_status status;
+
+	status.as_uint64 = hv_do_hvc_fast_get(
+		HVCALL_GET_VP_REGISTERS | HV_HYPERCALL_FAST_BIT |
+			HV_HYPERCALL_REP_COUNT_1,
+		HV_PARTITION_ID_SELF,
+		HV_VP_INDEX_SELF,
+		msr,
+		result);
+
+	/*
+	 * Something is fundamentally broken in the hypervisor if
+	 * getting a VP register fails. There's really no way to
+	 * continue as a guest VM, so panic.
+	 */
+	BUG_ON(status.status != HV_STATUS_SUCCESS);
+
+	return;
+
+}
+EXPORT_SYMBOL_GPL(hv_get_vpreg_128);
+
+void hyperv_report_panic(struct pt_regs *regs, long err)
+{
+	static bool panic_reported;
+	u64 guest_id;
+
+	/*
+	 * We prefer to report panic on 'die' chain as we have proper
+	 * registers to report, but if we miss it (e.g. on BUG()) we need
+	 * to report it on 'panic'.
+	 */
+	if (panic_reported)
+		return;
+	panic_reported = true;
+
+	guest_id = hv_get_vpreg(HV_REGISTER_GUEST_OSID);
+
+	/*
+	 * Hyper-V provides the ability to store only 5 values.
+	 * Pick the passed in error value, the guest_id, and the PC.
+	 * The first two general registers are added arbitrarily.
+	 */
+	hv_set_vpreg(HV_REGISTER_CRASH_P0, err);
+	hv_set_vpreg(HV_REGISTER_CRASH_P1, guest_id);
+	hv_set_vpreg(HV_REGISTER_CRASH_P2, regs->pc);
+	hv_set_vpreg(HV_REGISTER_CRASH_P3, regs->regs[0]);
+	hv_set_vpreg(HV_REGISTER_CRASH_P4, regs->regs[1]);
+
+	/*
+	 * Let Hyper-V know there is crash data available
+	 */
+	hv_set_vpreg(HV_REGISTER_CRASH_CTL, HV_CRASH_CTL_CRASH_NOTIFY);
+}
+EXPORT_SYMBOL_GPL(hyperv_report_panic);
+
+/*
+ * hyperv_report_panic_msg - report panic message to Hyper-V
+ * @pa: physical address of the panic page containing the message
+ * @size: size of the message in the page
+ */
+void hyperv_report_panic_msg(phys_addr_t pa, size_t size)
+{
+	/*
+	 * P3 to contain the physical address of the panic page & P4 to
+	 * contain the size of the panic data in that page. Rest of the
+	 * registers are no-op when the NOTIFY_MSG flag is set.
+	 */
+	hv_set_vpreg(HV_REGISTER_CRASH_P0, 0);
+	hv_set_vpreg(HV_REGISTER_CRASH_P1, 0);
+	hv_set_vpreg(HV_REGISTER_CRASH_P2, 0);
+	hv_set_vpreg(HV_REGISTER_CRASH_P3, pa);
+	hv_set_vpreg(HV_REGISTER_CRASH_P4, size);
+
+	/*
+	 * Let Hyper-V know there is crash data available along with
+	 * the panic message.
+	 */
+	hv_set_vpreg(HV_REGISTER_CRASH_CTL,
+	       (HV_CRASH_CTL_CRASH_NOTIFY | HV_CRASH_CTL_CRASH_NOTIFY_MSG));
+}
+EXPORT_SYMBOL_GPL(hyperv_report_panic_msg);
+
+bool hv_is_hyperv_initialized(void)
+{
+	return hyperv_initialized;
+}
+EXPORT_SYMBOL_GPL(hv_is_hyperv_initialized);
diff --git a/arch/arm64/hyperv/mshyperv.c b/arch/arm64/hyperv/mshyperv.c
new file mode 100644
index 000000000000..3ef055599412
--- /dev/null
+++ b/arch/arm64/hyperv/mshyperv.c
@@ -0,0 +1,178 @@
+// SPDX-License-Identifier: GPL-2.0
+
+/*
+ * Core routines for interacting with Microsoft's Hyper-V hypervisor,
+ * including setting up VMbus and STIMER interrupts, and handling
+ * crashes and kexecs. These interactions are through a set of
+ * static "handler" variables set by the architecture independent
+ * VMbus and STIMER drivers.  This design is used to meet x86/x64
+ * requirements for avoiding direct linkages and allowing the VMbus
+ * and STIMER drivers to be unloaded and reloaded.
+ *
+ * Copyright (C) 2018, Microsoft, Inc.
+ *
+ * Author : Michael Kelley <mikelley@microsoft.com>
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License version 2 as published
+ * by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE, GOOD TITLE or
+ * NON INFRINGEMENT.  See the GNU General Public License for more
+ * details.
+ */
+
+#include <linux/types.h>
+#include <linux/export.h>
+#include <linux/interrupt.h>
+#include <linux/kexec.h>
+#include <linux/acpi.h>
+#include <linux/ptrace.h>
+#include <asm/hyperv-tlfs.h>
+#include <asm/mshyperv.h>
+
+static void (*vmbus_handler)(void);
+static void (*hv_stimer0_handler)(void);
+static void (*hv_kexec_handler)(void);
+static void (*hv_crash_handler)(struct pt_regs *regs);
+
+static int vmbus_irq;
+static long __percpu *vmbus_evt;
+static long __percpu *stimer0_evt;
+
+irqreturn_t hyperv_vector_handler(int irq, void *dev_id)
+{
+	if (vmbus_handler)
+		vmbus_handler();
+	return IRQ_HANDLED;
+}
+
+/* Must be done just once */
+void hv_setup_vmbus_irq(void (*handler)(void))
+{
+	int result;
+
+	vmbus_handler = handler;
+	vmbus_irq = acpi_register_gsi(NULL, HYPERVISOR_CALLBACK_VECTOR,
+				 ACPI_LEVEL_SENSITIVE, ACPI_ACTIVE_HIGH);
+	if (vmbus_irq <= 0) {
+		pr_err("Can't register Hyper-V VMBus GSI. Error %d",
+			vmbus_irq);
+		vmbus_irq = 0;
+		return;
+	}
+	vmbus_evt = alloc_percpu(long);
+	result = request_percpu_irq(vmbus_irq, hyperv_vector_handler,
+			"Hyper-V VMbus", vmbus_evt);
+	if (result) {
+		pr_err("Can't request Hyper-V VMBus IRQ %d. Error %d",
+			vmbus_irq, result);
+		free_percpu(vmbus_evt);
+		acpi_unregister_gsi(vmbus_irq);
+		vmbus_irq = 0;
+	}
+}
+EXPORT_SYMBOL_GPL(hv_setup_vmbus_irq);
+
+/* Must be done just once */
+void hv_remove_vmbus_irq(void)
+{
+	if (vmbus_irq) {
+		free_percpu_irq(vmbus_irq, vmbus_evt);
+		free_percpu(vmbus_evt);
+		acpi_unregister_gsi(vmbus_irq);
+	}
+}
+EXPORT_SYMBOL_GPL(hv_remove_vmbus_irq);
+
+/* Must be done by each CPU */
+void hv_enable_vmbus_irq(void)
+{
+	enable_percpu_irq(vmbus_irq, 0);
+}
+EXPORT_SYMBOL_GPL(hv_enable_vmbus_irq);
+
+/* Must be done by each CPU */
+void hv_disable_vmbus_irq(void)
+{
+	disable_percpu_irq(vmbus_irq);
+}
+EXPORT_SYMBOL_GPL(hv_disable_vmbus_irq);
+
+/* Routines to do per-architecture handling of STIMER0 when in Direct Mode */
+
+static irqreturn_t hv_stimer0_vector_handler(int irq, void *dev_id)
+{
+	if (hv_stimer0_handler)
+		hv_stimer0_handler();
+	return IRQ_HANDLED;
+}
+
+int hv_setup_stimer0_irq(int *irq, int *vector, void (*handler)(void))
+{
+	int localirq;
+	int result;
+
+	localirq = acpi_register_gsi(NULL, HV_STIMER0_IRQNR,
+			ACPI_LEVEL_SENSITIVE, ACPI_ACTIVE_HIGH);
+	if (localirq <= 0) {
+		pr_err("Can't register Hyper-V stimer0 GSI. Error %d",
+			localirq);
+		*irq = 0;
+		return -1;
+	}
+	stimer0_evt = alloc_percpu(long);
+	result = request_percpu_irq(localirq, hv_stimer0_vector_handler,
+					 "Hyper-V stimer0", stimer0_evt);
+	if (result) {
+		pr_err("Can't request Hyper-V stimer0 IRQ %d. Error %d",
+			localirq, result);
+		free_percpu(stimer0_evt);
+		acpi_unregister_gsi(localirq);
+		*irq = 0;
+		return -1;
+	}
+
+	hv_stimer0_handler = handler;
+	*vector = HV_STIMER0_IRQNR;
+	*irq = localirq;
+	return 0;
+}
+EXPORT_SYMBOL_GPL(hv_setup_stimer0_irq);
+
+void hv_remove_stimer0_irq(int irq)
+{
+	hv_stimer0_handler = NULL;
+	if (irq) {
+		free_percpu_irq(irq, stimer0_evt);
+		free_percpu(stimer0_evt);
+		acpi_unregister_gsi(irq);
+	}
+}
+EXPORT_SYMBOL_GPL(hv_remove_stimer0_irq);
+
+void hv_setup_kexec_handler(void (*handler)(void))
+{
+	hv_kexec_handler = handler;
+}
+EXPORT_SYMBOL_GPL(hv_setup_kexec_handler);
+
+void hv_remove_kexec_handler(void)
+{
+	hv_kexec_handler = NULL;
+}
+EXPORT_SYMBOL_GPL(hv_remove_kexec_handler);
+
+void hv_setup_crash_handler(void (*handler)(struct pt_regs *regs))
+{
+	hv_crash_handler = handler;
+}
+EXPORT_SYMBOL_GPL(hv_setup_crash_handler);
+
+void hv_remove_crash_handler(void)
+{
+	hv_crash_handler = NULL;
+}
+EXPORT_SYMBOL_GPL(hv_remove_crash_handler);
-- 
2.19.1

[PATCH 3/4] Drivers: hv: vmbus: Add hooks for per-CPU IRQ

[kys]

From: Michael Kelley <mikelley@microsoft.com>

Add hooks to enable/disable a per-CPU IRQ for VMbus. These hooks
are in the architecture independent setup and shutdown paths for
Hyper-V, and are needed by Linux guests on Hyper-V on ARM64.  The
x86/x64 implementation is null because VMbus interrupts on x86/x64
don't use an IRQ.

Signed-off-by: Michael Kelley <mikelley@microsoft.com>
Signed-off-by: K. Y. Srinivasan <kys@microsoft.com>
---
 arch/x86/include/asm/mshyperv.h | 4 ++++
 drivers/hv/hv.c                 | 2 ++
 2 files changed, 6 insertions(+)

diff --git a/arch/x86/include/asm/mshyperv.h b/arch/x86/include/asm/mshyperv.h
index 0d6271cce198..8d97bd3a13a6 100644
--- a/arch/x86/include/asm/mshyperv.h
+++ b/arch/x86/include/asm/mshyperv.h
@@ -109,6 +109,10 @@ void hyperv_vector_handler(struct pt_regs *regs);
 void hv_setup_vmbus_irq(void (*handler)(void));
 void hv_remove_vmbus_irq(void);
 
+/* On x86/x64, there isn't a real IRQ to be enabled/disable */
+static inline void hv_enable_vmbus_irq(void) {}
+static inline void hv_disable_vmbus_irq(void) {}
+
 void hv_setup_kexec_handler(void (*handler)(void));
 void hv_remove_kexec_handler(void);
 void hv_setup_crash_handler(void (*handler)(struct pt_regs *regs));
diff --git a/drivers/hv/hv.c b/drivers/hv/hv.c
index 166c2501de17..d0bb09a4bd73 100644
--- a/drivers/hv/hv.c
+++ b/drivers/hv/hv.c
@@ -307,6 +307,7 @@ int hv_synic_init(unsigned int cpu)
 	hv_set_siefp(siefp.as_uint64);
 
 	/* Setup the shared SINT. */
+	hv_enable_vmbus_irq();
 	hv_get_synint_state(VMBUS_MESSAGE_SINT, shared_sint.as_uint64);
 
 	shared_sint.vector = HYPERVISOR_CALLBACK_VECTOR;
@@ -434,6 +435,7 @@ int hv_synic_cleanup(unsigned int cpu)
 	/* Disable the global synic bit */
 	sctrl.enable = 0;
 	hv_set_synic_state(sctrl.as_uint64);
+	hv_disable_vmbus_irq();
 
 	return 0;
 }
-- 
2.19.1

[PATCH 3/4] Drivers: hv: vmbus: Add hooks for per-CPU IRQ

[kys at linuxonhyperv.com]

From: Michael Kelley <mikelley@microsoft.com>

Add hooks to enable/disable a per-CPU IRQ for VMbus. These hooks
are in the architecture independent setup and shutdown paths for
Hyper-V, and are needed by Linux guests on Hyper-V on ARM64.  The
x86/x64 implementation is null because VMbus interrupts on x86/x64
don't use an IRQ.

Signed-off-by: Michael Kelley <mikelley@microsoft.com>
Signed-off-by: K. Y. Srinivasan <kys@microsoft.com>
---
 arch/x86/include/asm/mshyperv.h | 4 ++++
 drivers/hv/hv.c                 | 2 ++
 2 files changed, 6 insertions(+)

diff --git a/arch/x86/include/asm/mshyperv.h b/arch/x86/include/asm/mshyperv.h
index 0d6271cce198..8d97bd3a13a6 100644
--- a/arch/x86/include/asm/mshyperv.h
+++ b/arch/x86/include/asm/mshyperv.h
@@ -109,6 +109,10 @@ void hyperv_vector_handler(struct pt_regs *regs);
 void hv_setup_vmbus_irq(void (*handler)(void));
 void hv_remove_vmbus_irq(void);
 
+/* On x86/x64, there isn't a real IRQ to be enabled/disable */
+static inline void hv_enable_vmbus_irq(void) {}
+static inline void hv_disable_vmbus_irq(void) {}
+
 void hv_setup_kexec_handler(void (*handler)(void));
 void hv_remove_kexec_handler(void);
 void hv_setup_crash_handler(void (*handler)(struct pt_regs *regs));
diff --git a/drivers/hv/hv.c b/drivers/hv/hv.c
index 166c2501de17..d0bb09a4bd73 100644
--- a/drivers/hv/hv.c
+++ b/drivers/hv/hv.c
@@ -307,6 +307,7 @@ int hv_synic_init(unsigned int cpu)
 	hv_set_siefp(siefp.as_uint64);
 
 	/* Setup the shared SINT. */
+	hv_enable_vmbus_irq();
 	hv_get_synint_state(VMBUS_MESSAGE_SINT, shared_sint.as_uint64);
 
 	shared_sint.vector = HYPERVISOR_CALLBACK_VECTOR;
@@ -434,6 +435,7 @@ int hv_synic_cleanup(unsigned int cpu)
 	/* Disable the global synic bit */
 	sctrl.enable = 0;
 	hv_set_synic_state(sctrl.as_uint64);
+	hv_disable_vmbus_irq();
 
 	return 0;
 }
-- 
2.19.1

[PATCH 4/4] Drivers: hv: Enable CONFIG_HYPERV on ARM64

[kys]

From: Michael Kelley <mikelley@microsoft.com>

Update drivers/hv/Kconfig so CONFIG_HYPERV can be selected on ARM64,
causing the Hyper-V specific code to be built.

Signed-off-by: Michael Kelley <mikelley@microsoft.com>
Signed-off-by: K. Y. Srinivasan <kys@microsoft.com>
---
 drivers/hv/Kconfig | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/drivers/hv/Kconfig b/drivers/hv/Kconfig
index 97954f575c3f..c3e11a2f9c70 100644
--- a/drivers/hv/Kconfig
+++ b/drivers/hv/Kconfig
@@ -4,7 +4,8 @@ menu "Microsoft Hyper-V guest support"
 
 config HYPERV
 	tristate "Microsoft Hyper-V client drivers"
-	depends on X86 && ACPI && PCI && X86_LOCAL_APIC && HYPERVISOR_GUEST
+	depends on ACPI && PCI && \
+			((X86 && X86_LOCAL_APIC && HYPERVISOR_GUEST) || ARM64)
 	select PARAVIRT
 	help
 	  Select this option to run Linux as a Hyper-V client operating
-- 
2.19.1

[PATCH 4/4] Drivers: hv: Enable CONFIG_HYPERV on ARM64

[kys at linuxonhyperv.com]

From: Michael Kelley <mikelley@microsoft.com>

Update drivers/hv/Kconfig so CONFIG_HYPERV can be selected on ARM64,
causing the Hyper-V specific code to be built.

Signed-off-by: Michael Kelley <mikelley@microsoft.com>
Signed-off-by: K. Y. Srinivasan <kys@microsoft.com>
---
 drivers/hv/Kconfig | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/drivers/hv/Kconfig b/drivers/hv/Kconfig
index 97954f575c3f..c3e11a2f9c70 100644
--- a/drivers/hv/Kconfig
+++ b/drivers/hv/Kconfig
@@ -4,7 +4,8 @@ menu "Microsoft Hyper-V guest support"
 
 config HYPERV
 	tristate "Microsoft Hyper-V client drivers"
-	depends on X86 && ACPI && PCI && X86_LOCAL_APIC && HYPERVISOR_GUEST
+	depends on ACPI && PCI && \
+			((X86 && X86_LOCAL_APIC && HYPERVISOR_GUEST) || ARM64)
 	select PARAVIRT
 	help
 	  Select this option to run Linux as a Hyper-V client operating
-- 
2.19.1

Re: [PATCH 1/4] arm64: hyperv: Add core Hyper-V include files

[Greg KH]

On Thu, Nov 22, 2018 at 03:10:56AM +0000, kys@linuxonhyperv.com wrote:
> --- /dev/null
> +++ b/arch/arm64/include/asm/hyperv-tlfs.h
> @@ -0,0 +1,338 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +
> +/*
> + * This file contains definitions from the Hyper-V Hypervisor Top-Level
> + * Functional Specification (TLFS):
> + * https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fvirtualization%2Fhyper-v-on-windows%2Freference%2Ftlfs&data=02%7C01%7Ckys%40microsoft.com%7Cc831a45fd63e4a4b083908d641216aa8%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636768009113747528&sdata=jRSrs9ZWXdmeS7LQUEpoSyUfBS7a5KLYy%2FolFdE2tI0%3D&reserved=0

"interesting" url :(

Care to use a real one?

thanks,

greg k-h

[PATCH 1/4] arm64: hyperv: Add core Hyper-V include files

[Greg KH]

On Thu, Nov 22, 2018 at 03:10:56AM +0000, kys at linuxonhyperv.com wrote:
> --- /dev/null
> +++ b/arch/arm64/include/asm/hyperv-tlfs.h
> @@ -0,0 +1,338 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +
> +/*
> + * This file contains definitions from the Hyper-V Hypervisor Top-Level
> + * Functional Specification (TLFS):
> + * https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fvirtualization%2Fhyper-v-on-windows%2Freference%2Ftlfs&data=02%7C01%7Ckys%40microsoft.com%7Cc831a45fd63e4a4b083908d641216aa8%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636768009113747528&sdata=jRSrs9ZWXdmeS7LQUEpoSyUfBS7a5KLYy%2FolFdE2tI0%3D&reserved=0

"interesting" url :(

Care to use a real one?

thanks,

greg k-h

Re: [PATCH 2/4] arm64: hyperv: Add support for Hyper-V as a hypervisor

[Greg KH]

On Thu, Nov 22, 2018 at 03:10:57AM +0000, kys@linuxonhyperv.com wrote:
> --- /dev/null
> +++ b/arch/arm64/hyperv/hv_hvc.S
> @@ -0,0 +1,54 @@
> +/* SPDX-License-Identifier: GPL-2.0 */

Using SPDX is good, but:

> +
> +/*
> + * Microsoft Hyper-V hypervisor invocation routines
> + *
> + * Copyright (C) 2018, Microsoft, Inc.
> + *
> + * Author : Michael Kelley <mikelley@microsoft.com>
> + *
> + * This program is free software; you can redistribute it and/or modify it
> + * under the terms of the GNU General Public License version 2 as published
> + * by the Free Software Foundation.
> + *
> + * This program is distributed in the hope that it will be useful, but
> + * WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE, GOOD TITLE or
> + * NON INFRINGEMENT.  See the GNU General Public License for more
> + * details.

No need for these two paragraphs at all.  If you add them now, someone
will just come along later and rip them out.  Please don't include them
in the first place.

Not to mention the first paragraph doesn't make sense, think about
"which" version 2 the FSF published (hint, there was 5 or 7 or so...)
That's why we have one in the kernel tree for everyone to rely on.

thanks,

greg k-h

[PATCH 2/4] arm64: hyperv: Add support for Hyper-V as a hypervisor

[Greg KH]

On Thu, Nov 22, 2018 at 03:10:57AM +0000, kys at linuxonhyperv.com wrote:
> --- /dev/null
> +++ b/arch/arm64/hyperv/hv_hvc.S
> @@ -0,0 +1,54 @@
> +/* SPDX-License-Identifier: GPL-2.0 */

Using SPDX is good, but:

> +
> +/*
> + * Microsoft Hyper-V hypervisor invocation routines
> + *
> + * Copyright (C) 2018, Microsoft, Inc.
> + *
> + * Author : Michael Kelley <mikelley@microsoft.com>
> + *
> + * This program is free software; you can redistribute it and/or modify it
> + * under the terms of the GNU General Public License version 2 as published
> + * by the Free Software Foundation.
> + *
> + * This program is distributed in the hope that it will be useful, but
> + * WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE, GOOD TITLE or
> + * NON INFRINGEMENT.  See the GNU General Public License for more
> + * details.

No need for these two paragraphs at all.  If you add them now, someone
will just come along later and rip them out.  Please don't include them
in the first place.

Not to mention the first paragraph doesn't make sense, think about
"which" version 2 the FSF published (hint, there was 5 or 7 or so...)
That's why we have one in the kernel tree for everyone to rely on.

thanks,

greg k-h

Re: [PATCH 3/4] Drivers: hv: vmbus: Add hooks for per-CPU IRQ

[Greg KH]

On Thu, Nov 22, 2018 at 03:10:58AM +0000, kys@linuxonhyperv.com wrote:
> From: Michael Kelley <mikelley@microsoft.com>
> 
> Add hooks to enable/disable a per-CPU IRQ for VMbus. These hooks
> are in the architecture independent setup and shutdown paths for
> Hyper-V, and are needed by Linux guests on Hyper-V on ARM64.  The
> x86/x64 implementation is null because VMbus interrupts on x86/x64
> don't use an IRQ.
> 
> Signed-off-by: Michael Kelley <mikelley@microsoft.com>
> Signed-off-by: K. Y. Srinivasan <kys@microsoft.com>
> ---
>  arch/x86/include/asm/mshyperv.h | 4 ++++
>  drivers/hv/hv.c                 | 2 ++
>  2 files changed, 6 insertions(+)
> 
> diff --git a/arch/x86/include/asm/mshyperv.h b/arch/x86/include/asm/mshyperv.h
> index 0d6271cce198..8d97bd3a13a6 100644
> --- a/arch/x86/include/asm/mshyperv.h
> +++ b/arch/x86/include/asm/mshyperv.h
> @@ -109,6 +109,10 @@ void hyperv_vector_handler(struct pt_regs *regs);
>  void hv_setup_vmbus_irq(void (*handler)(void));
>  void hv_remove_vmbus_irq(void);
>  
> +/* On x86/x64, there isn't a real IRQ to be enabled/disable */
> +static inline void hv_enable_vmbus_irq(void) {}
> +static inline void hv_disable_vmbus_irq(void) {}
> +
>  void hv_setup_kexec_handler(void (*handler)(void));
>  void hv_remove_kexec_handler(void);
>  void hv_setup_crash_handler(void (*handler)(struct pt_regs *regs));
> diff --git a/drivers/hv/hv.c b/drivers/hv/hv.c
> index 166c2501de17..d0bb09a4bd73 100644
> --- a/drivers/hv/hv.c
> +++ b/drivers/hv/hv.c
> @@ -307,6 +307,7 @@ int hv_synic_init(unsigned int cpu)
>  	hv_set_siefp(siefp.as_uint64);
>  
>  	/* Setup the shared SINT. */
> +	hv_enable_vmbus_irq();
>  	hv_get_synint_state(VMBUS_MESSAGE_SINT, shared_sint.as_uint64);
>  
>  	shared_sint.vector = HYPERVISOR_CALLBACK_VECTOR;
> @@ -434,6 +435,7 @@ int hv_synic_cleanup(unsigned int cpu)
>  	/* Disable the global synic bit */
>  	sctrl.enable = 0;
>  	hv_set_synic_state(sctrl.as_uint64);
> +	hv_disable_vmbus_irq();
>  
>  	return 0;
>  }
> -- 
> 2.19.1

You created "null" hooks that do nothing, for no one in this patch
series, why?

Add them when they are needed not now.  If I saw this code in the tree,
I would just go delete it as it is because it is not used at all.

thanks,

greg k-h

[PATCH 3/4] Drivers: hv: vmbus: Add hooks for per-CPU IRQ

[Greg KH]

On Thu, Nov 22, 2018 at 03:10:58AM +0000, kys at linuxonhyperv.com wrote:
> From: Michael Kelley <mikelley@microsoft.com>
> 
> Add hooks to enable/disable a per-CPU IRQ for VMbus. These hooks
> are in the architecture independent setup and shutdown paths for
> Hyper-V, and are needed by Linux guests on Hyper-V on ARM64.  The
> x86/x64 implementation is null because VMbus interrupts on x86/x64
> don't use an IRQ.
> 
> Signed-off-by: Michael Kelley <mikelley@microsoft.com>
> Signed-off-by: K. Y. Srinivasan <kys@microsoft.com>
> ---
>  arch/x86/include/asm/mshyperv.h | 4 ++++
>  drivers/hv/hv.c                 | 2 ++
>  2 files changed, 6 insertions(+)
> 
> diff --git a/arch/x86/include/asm/mshyperv.h b/arch/x86/include/asm/mshyperv.h
> index 0d6271cce198..8d97bd3a13a6 100644
> --- a/arch/x86/include/asm/mshyperv.h
> +++ b/arch/x86/include/asm/mshyperv.h
> @@ -109,6 +109,10 @@ void hyperv_vector_handler(struct pt_regs *regs);
>  void hv_setup_vmbus_irq(void (*handler)(void));
>  void hv_remove_vmbus_irq(void);
>  
> +/* On x86/x64, there isn't a real IRQ to be enabled/disable */
> +static inline void hv_enable_vmbus_irq(void) {}
> +static inline void hv_disable_vmbus_irq(void) {}
> +
>  void hv_setup_kexec_handler(void (*handler)(void));
>  void hv_remove_kexec_handler(void);
>  void hv_setup_crash_handler(void (*handler)(struct pt_regs *regs));
> diff --git a/drivers/hv/hv.c b/drivers/hv/hv.c
> index 166c2501de17..d0bb09a4bd73 100644
> --- a/drivers/hv/hv.c
> +++ b/drivers/hv/hv.c
> @@ -307,6 +307,7 @@ int hv_synic_init(unsigned int cpu)
>  	hv_set_siefp(siefp.as_uint64);
>  
>  	/* Setup the shared SINT. */
> +	hv_enable_vmbus_irq();
>  	hv_get_synint_state(VMBUS_MESSAGE_SINT, shared_sint.as_uint64);
>  
>  	shared_sint.vector = HYPERVISOR_CALLBACK_VECTOR;
> @@ -434,6 +435,7 @@ int hv_synic_cleanup(unsigned int cpu)
>  	/* Disable the global synic bit */
>  	sctrl.enable = 0;
>  	hv_set_synic_state(sctrl.as_uint64);
> +	hv_disable_vmbus_irq();
>  
>  	return 0;
>  }
> -- 
> 2.19.1

You created "null" hooks that do nothing, for no one in this patch
series, why?

Add them when they are needed not now.  If I saw this code in the tree,
I would just go delete it as it is because it is not used at all.

thanks,

greg k-h

RE: [PATCH 3/4] Drivers: hv: vmbus: Add hooks for per-CPU IRQ

[Michael Kelley]

From: Greg KH <gregkh@linuxfoundation.org>  Monday, November 26, 2018 11:21 AM

> > diff --git a/arch/x86/include/asm/mshyperv.h b/arch/x86/include/asm/mshyperv.h
> > index 0d6271cce198..8d97bd3a13a6 100644
> > --- a/arch/x86/include/asm/mshyperv.h
> > +++ b/arch/x86/include/asm/mshyperv.h
> > @@ -109,6 +109,10 @@ void hyperv_vector_handler(struct pt_regs *regs);
> >  void hv_setup_vmbus_irq(void (*handler)(void));
> >  void hv_remove_vmbus_irq(void);
> >
> > +/* On x86/x64, there isn't a real IRQ to be enabled/disable */
> > +static inline void hv_enable_vmbus_irq(void) {}
> > +static inline void hv_disable_vmbus_irq(void) {}
> > +
> >  void hv_setup_kexec_handler(void (*handler)(void));
> >  void hv_remove_kexec_handler(void);
> >  void hv_setup_crash_handler(void (*handler)(struct pt_regs *regs));
> > diff --git a/drivers/hv/hv.c b/drivers/hv/hv.c
> > index 166c2501de17..d0bb09a4bd73 100644
> > --- a/drivers/hv/hv.c
> > +++ b/drivers/hv/hv.c
> > @@ -307,6 +307,7 @@ int hv_synic_init(unsigned int cpu)
> >  	hv_set_siefp(siefp.as_uint64);
> >
> >  	/* Setup the shared SINT. */
> > +	hv_enable_vmbus_irq();
> >  	hv_get_synint_state(VMBUS_MESSAGE_SINT, shared_sint.as_uint64);
> >
> >  	shared_sint.vector = HYPERVISOR_CALLBACK_VECTOR;
> > @@ -434,6 +435,7 @@ int hv_synic_cleanup(unsigned int cpu)
> >  	/* Disable the global synic bit */
> >  	sctrl.enable = 0;
> >  	hv_set_synic_state(sctrl.as_uint64);
> > +	hv_disable_vmbus_irq();
> >
> >  	return 0;
> >  }
> > --
> > 2.19.1
> 
> You created "null" hooks that do nothing, for no one in this patch
> series, why?
> 

hv_enable_vmbus_irq() and hv_disable_vmbus_irq() have non-null
implementations in the ARM64 code in patch 2 of this series.  The
implementations are in the new file arch/arm64/hyperv/mshyperv.c.
Or am I misunderstanding your point?

Michael

[PATCH 3/4] Drivers: hv: vmbus: Add hooks for per-CPU IRQ

[Michael Kelley]

From: Greg KH <gregkh@linuxfoundation.org>  Monday, November 26, 2018 11:21 AM

> > diff --git a/arch/x86/include/asm/mshyperv.h b/arch/x86/include/asm/mshyperv.h
> > index 0d6271cce198..8d97bd3a13a6 100644
> > --- a/arch/x86/include/asm/mshyperv.h
> > +++ b/arch/x86/include/asm/mshyperv.h
> > @@ -109,6 +109,10 @@ void hyperv_vector_handler(struct pt_regs *regs);
> >  void hv_setup_vmbus_irq(void (*handler)(void));
> >  void hv_remove_vmbus_irq(void);
> >
> > +/* On x86/x64, there isn't a real IRQ to be enabled/disable */
> > +static inline void hv_enable_vmbus_irq(void) {}
> > +static inline void hv_disable_vmbus_irq(void) {}
> > +
> >  void hv_setup_kexec_handler(void (*handler)(void));
> >  void hv_remove_kexec_handler(void);
> >  void hv_setup_crash_handler(void (*handler)(struct pt_regs *regs));
> > diff --git a/drivers/hv/hv.c b/drivers/hv/hv.c
> > index 166c2501de17..d0bb09a4bd73 100644
> > --- a/drivers/hv/hv.c
> > +++ b/drivers/hv/hv.c
> > @@ -307,6 +307,7 @@ int hv_synic_init(unsigned int cpu)
> >  	hv_set_siefp(siefp.as_uint64);
> >
> >  	/* Setup the shared SINT. */
> > +	hv_enable_vmbus_irq();
> >  	hv_get_synint_state(VMBUS_MESSAGE_SINT, shared_sint.as_uint64);
> >
> >  	shared_sint.vector = HYPERVISOR_CALLBACK_VECTOR;
> > @@ -434,6 +435,7 @@ int hv_synic_cleanup(unsigned int cpu)
> >  	/* Disable the global synic bit */
> >  	sctrl.enable = 0;
> >  	hv_set_synic_state(sctrl.as_uint64);
> > +	hv_disable_vmbus_irq();
> >
> >  	return 0;
> >  }
> > --
> > 2.19.1
> 
> You created "null" hooks that do nothing, for no one in this patch
> series, why?
> 

hv_enable_vmbus_irq() and hv_disable_vmbus_irq() have non-null
implementations in the ARM64 code in patch 2 of this series.  The
implementations are in the new file arch/arm64/hyperv/mshyperv.c.
Or am I misunderstanding your point?

Michael

Re: [PATCH 3/4] Drivers: hv: vmbus: Add hooks for per-CPU IRQ

[Greg KH]

On Mon, Nov 26, 2018 at 07:47:41PM +0000, Michael Kelley wrote:
> From: Greg KH <gregkh@linuxfoundation.org>  Monday, November 26, 2018 11:21 AM
> 
> > > diff --git a/arch/x86/include/asm/mshyperv.h b/arch/x86/include/asm/mshyperv.h
> > > index 0d6271cce198..8d97bd3a13a6 100644
> > > --- a/arch/x86/include/asm/mshyperv.h
> > > +++ b/arch/x86/include/asm/mshyperv.h
> > > @@ -109,6 +109,10 @@ void hyperv_vector_handler(struct pt_regs *regs);
> > >  void hv_setup_vmbus_irq(void (*handler)(void));
> > >  void hv_remove_vmbus_irq(void);
> > >
> > > +/* On x86/x64, there isn't a real IRQ to be enabled/disable */
> > > +static inline void hv_enable_vmbus_irq(void) {}
> > > +static inline void hv_disable_vmbus_irq(void) {}
> > > +
> > >  void hv_setup_kexec_handler(void (*handler)(void));
> > >  void hv_remove_kexec_handler(void);
> > >  void hv_setup_crash_handler(void (*handler)(struct pt_regs *regs));
> > > diff --git a/drivers/hv/hv.c b/drivers/hv/hv.c
> > > index 166c2501de17..d0bb09a4bd73 100644
> > > --- a/drivers/hv/hv.c
> > > +++ b/drivers/hv/hv.c
> > > @@ -307,6 +307,7 @@ int hv_synic_init(unsigned int cpu)
> > >  	hv_set_siefp(siefp.as_uint64);
> > >
> > >  	/* Setup the shared SINT. */
> > > +	hv_enable_vmbus_irq();
> > >  	hv_get_synint_state(VMBUS_MESSAGE_SINT, shared_sint.as_uint64);
> > >
> > >  	shared_sint.vector = HYPERVISOR_CALLBACK_VECTOR;
> > > @@ -434,6 +435,7 @@ int hv_synic_cleanup(unsigned int cpu)
> > >  	/* Disable the global synic bit */
> > >  	sctrl.enable = 0;
> > >  	hv_set_synic_state(sctrl.as_uint64);
> > > +	hv_disable_vmbus_irq();
> > >
> > >  	return 0;
> > >  }
> > > --
> > > 2.19.1
> > 
> > You created "null" hooks that do nothing, for no one in this patch
> > series, why?
> > 
> 
> hv_enable_vmbus_irq() and hv_disable_vmbus_irq() have non-null
> implementations in the ARM64 code in patch 2 of this series.  The
> implementations are in the new file arch/arm64/hyperv/mshyperv.c.
> Or am I misunderstanding your point?

So you use a hook in an earlier patch and then add it in a later one?

Shouldn't you do it the other way around?  As it is, the earlier patch
should not work properly, right?

thanks,

greg k-h

[PATCH 3/4] Drivers: hv: vmbus: Add hooks for per-CPU IRQ

[Greg KH]

On Mon, Nov 26, 2018 at 07:47:41PM +0000, Michael Kelley wrote:
> From: Greg KH <gregkh@linuxfoundation.org>  Monday, November 26, 2018 11:21 AM
> 
> > > diff --git a/arch/x86/include/asm/mshyperv.h b/arch/x86/include/asm/mshyperv.h
> > > index 0d6271cce198..8d97bd3a13a6 100644
> > > --- a/arch/x86/include/asm/mshyperv.h
> > > +++ b/arch/x86/include/asm/mshyperv.h
> > > @@ -109,6 +109,10 @@ void hyperv_vector_handler(struct pt_regs *regs);
> > >  void hv_setup_vmbus_irq(void (*handler)(void));
> > >  void hv_remove_vmbus_irq(void);
> > >
> > > +/* On x86/x64, there isn't a real IRQ to be enabled/disable */
> > > +static inline void hv_enable_vmbus_irq(void) {}
> > > +static inline void hv_disable_vmbus_irq(void) {}
> > > +
> > >  void hv_setup_kexec_handler(void (*handler)(void));
> > >  void hv_remove_kexec_handler(void);
> > >  void hv_setup_crash_handler(void (*handler)(struct pt_regs *regs));
> > > diff --git a/drivers/hv/hv.c b/drivers/hv/hv.c
> > > index 166c2501de17..d0bb09a4bd73 100644
> > > --- a/drivers/hv/hv.c
> > > +++ b/drivers/hv/hv.c
> > > @@ -307,6 +307,7 @@ int hv_synic_init(unsigned int cpu)
> > >  	hv_set_siefp(siefp.as_uint64);
> > >
> > >  	/* Setup the shared SINT. */
> > > +	hv_enable_vmbus_irq();
> > >  	hv_get_synint_state(VMBUS_MESSAGE_SINT, shared_sint.as_uint64);
> > >
> > >  	shared_sint.vector = HYPERVISOR_CALLBACK_VECTOR;
> > > @@ -434,6 +435,7 @@ int hv_synic_cleanup(unsigned int cpu)
> > >  	/* Disable the global synic bit */
> > >  	sctrl.enable = 0;
> > >  	hv_set_synic_state(sctrl.as_uint64);
> > > +	hv_disable_vmbus_irq();
> > >
> > >  	return 0;
> > >  }
> > > --
> > > 2.19.1
> > 
> > You created "null" hooks that do nothing, for no one in this patch
> > series, why?
> > 
> 
> hv_enable_vmbus_irq() and hv_disable_vmbus_irq() have non-null
> implementations in the ARM64 code in patch 2 of this series.  The
> implementations are in the new file arch/arm64/hyperv/mshyperv.c.
> Or am I misunderstanding your point?

So you use a hook in an earlier patch and then add it in a later one?

Shouldn't you do it the other way around?  As it is, the earlier patch
should not work properly, right?

thanks,

greg k-h

Re: [PATCH 1/4] arm64: hyperv: Add core Hyper-V include files

[Joshua R. Poulson]

That's https://docs.microsoft.com/en-us/virtualization/hyper-v-on-windows/reference/tlfs

Thanks, --jrp
On Mon, Nov 26, 2018 at 11:18 AM Greg KH <gregkh@linuxfoundation.org> wrote:
>
> On Thu, Nov 22, 2018 at 03:10:56AM +0000, kys@linuxonhyperv.com wrote:
> > --- /dev/null
> > +++ b/arch/arm64/include/asm/hyperv-tlfs.h
> > @@ -0,0 +1,338 @@
> > +/* SPDX-License-Identifier: GPL-2.0 */
> > +
> > +/*
> > + * This file contains definitions from the Hyper-V Hypervisor Top-Level
> > + * Functional Specification (TLFS):
> > + * https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fvirtualization%2Fhyper-v-on-windows%2Freference%2Ftlfs&amp;data=02%7C01%7Ckys%40microsoft.com%7Cc831a45fd63e4a4b083908d641216aa8%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636768009113747528&amp;sdata=jRSrs9ZWXdmeS7LQUEpoSyUfBS7a5KLYy%2FolFdE2tI0%3D&amp;reserved=0
>
> "interesting" url :(
>
> Care to use a real one?
>
> thanks,
>
> greg k-h

[PATCH 1/4] arm64: hyperv: Add core Hyper-V include files

[Joshua R. Poulson]

That's https://docs.microsoft.com/en-us/virtualization/hyper-v-on-windows/reference/tlfs

Thanks, --jrp
On Mon, Nov 26, 2018 at 11:18 AM Greg KH <gregkh@linuxfoundation.org> wrote:
>
> On Thu, Nov 22, 2018 at 03:10:56AM +0000, kys at linuxonhyperv.com wrote:
> > --- /dev/null
> > +++ b/arch/arm64/include/asm/hyperv-tlfs.h
> > @@ -0,0 +1,338 @@
> > +/* SPDX-License-Identifier: GPL-2.0 */
> > +
> > +/*
> > + * This file contains definitions from the Hyper-V Hypervisor Top-Level
> > + * Functional Specification (TLFS):
> > + * https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fvirtualization%2Fhyper-v-on-windows%2Freference%2Ftlfs&amp;data=02%7C01%7Ckys%40microsoft.com%7Cc831a45fd63e4a4b083908d641216aa8%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636768009113747528&amp;sdata=jRSrs9ZWXdmeS7LQUEpoSyUfBS7a5KLYy%2FolFdE2tI0%3D&amp;reserved=0
>
> "interesting" url :(
>
> Care to use a real one?
>
> thanks,
>
> greg k-h

RE: [PATCH 3/4] Drivers: hv: vmbus: Add hooks for per-CPU IRQ

[Michael Kelley]

From: Greg KH <gregkh@linuxfoundation.org>  Monday, November 26, 2018 11:57 AM

> > > You created "null" hooks that do nothing, for no one in this patch
> > > series, why?
> > >
> >
> > hv_enable_vmbus_irq() and hv_disable_vmbus_irq() have non-null
> > implementations in the ARM64 code in patch 2 of this series.  The
> > implementations are in the new file arch/arm64/hyperv/mshyperv.c.
> > Or am I misunderstanding your point?
> 
> So you use a hook in an earlier patch and then add it in a later one?
> 
> Shouldn't you do it the other way around?  As it is, the earlier patch
> should not work properly, right?

The earlier patch implements the hook on the ARM64 side but it is
unused -- it's not called.  The later patch then calls it.  Wouldn't the
other way around be backwards?

The general approach is for patches 1 and 2 of the series to provide
all the new code under arch/arm64 to enable Hyper-V.  But the code
won't get called (or even built) with just these two patches because
CONFIG_HYPERV can't be selected.  Patch 3 is separate because it
applies to architecture independent code and arch/x86 code -- I thought
there might be value in keeping the ARM64 and x86 patches distinct. 
Patch 4 applies to architecture independent code, and enables the
ARM64 code in patches 1 and 2 to be compiled and run when
CONFIG_HYPERV is selected.

If combining some of the patches in the series is a better approach, I'm
good with that.

Michael

[PATCH 3/4] Drivers: hv: vmbus: Add hooks for per-CPU IRQ

[Michael Kelley]

From: Greg KH <gregkh@linuxfoundation.org>  Monday, November 26, 2018 11:57 AM

> > > You created "null" hooks that do nothing, for no one in this patch
> > > series, why?
> > >
> >
> > hv_enable_vmbus_irq() and hv_disable_vmbus_irq() have non-null
> > implementations in the ARM64 code in patch 2 of this series.  The
> > implementations are in the new file arch/arm64/hyperv/mshyperv.c.
> > Or am I misunderstanding your point?
> 
> So you use a hook in an earlier patch and then add it in a later one?
> 
> Shouldn't you do it the other way around?  As it is, the earlier patch
> should not work properly, right?

The earlier patch implements the hook on the ARM64 side but it is
unused -- it's not called.  The later patch then calls it.  Wouldn't the
other way around be backwards?

The general approach is for patches 1 and 2 of the series to provide
all the new code under arch/arm64 to enable Hyper-V.  But the code
won't get called (or even built) with just these two patches because
CONFIG_HYPERV can't be selected.  Patch 3 is separate because it
applies to architecture independent code and arch/x86 code -- I thought
there might be value in keeping the ARM64 and x86 patches distinct. 
Patch 4 applies to architecture independent code, and enables the
ARM64 code in patches 1 and 2 to be compiled and run when
CONFIG_HYPERV is selected.

If combining some of the patches in the series is a better approach, I'm
good with that.

Michael

RE: [PATCH 1/4] arm64: hyperv: Add core Hyper-V include files

[KY Srinivasan]

> -----Original Message-----
> From: Greg KH <gregkh@linuxfoundation.org>
> Sent: Monday, November 26, 2018 11:18 AM
> To: KY Srinivasan <kys@microsoft.com>
> Cc: will.deacon@arm.com; catalin.marinas@armm.com;
> mark.rutland@arm.com; marc.zyngier@arm.com; linux-arm-
> kernel@lists.infradead.org; linux-kernel@vger.kernel.org;
> devel@linuxdriverproject.org; olaf@aepfle.de; apw@canonical.com;
> jasowang@redhat.com; Stephen Hemminger <sthemmin@microsoft.com>;
> Michael Kelley <mikelley@microsoft.com>; vkuznets
> <vkuznets@redhat.com>; Michael Kelley <mikelley@microsoft.com>
> Subject: Re: [PATCH 1/4] arm64: hyperv: Add core Hyper-V include files
> 
> On Thu, Nov 22, 2018 at 03:10:56AM +0000, kys@linuxonhyperv.com wrote:
> > --- /dev/null
> > +++ b/arch/arm64/include/asm/hyperv-tlfs.h
> > @@ -0,0 +1,338 @@
> > +/* SPDX-License-Identifier: GPL-2.0 */
> > +
> > +/*
> > + * This file contains definitions from the Hyper-V Hypervisor Top-Level
> > + * Functional Specification (TLFS):
> > + *
> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.
> microsoft.com%2Fen-us%2Fvirtualization%2Fhyper-v-on-
> windows%2Freference%2Ftlfs&amp;data=02%7C01%7Ckys%40microsoft.co
> m%7Cce3a8662cbbc4afe157508d653d3eadf%7C72f988bf86f141af91ab2d7cd0
> 11db47%7C1%7C0%7C636788566988323167&amp;sdata=VJHHMNbpOdAANd
> DNUD3UMuR8SkbBGOew7uy%2B%2BUCS2sE%3D&amp;reserved=0
> 
> "interesting" url :(
> 
> Care to use a real one?

Will fix and resend.

K. Y
> 
> thanks,
> 
> greg k-h

[PATCH 1/4] arm64: hyperv: Add core Hyper-V include files

[KY Srinivasan]

> -----Original Message-----
> From: Greg KH <gregkh@linuxfoundation.org>
> Sent: Monday, November 26, 2018 11:18 AM
> To: KY Srinivasan <kys@microsoft.com>
> Cc: will.deacon at arm.com; catalin.marinas at armm.com;
> mark.rutland at arm.com; marc.zyngier at arm.com; linux-arm-
> kernel at lists.infradead.org; linux-kernel at vger.kernel.org;
> devel at linuxdriverproject.org; olaf at aepfle.de; apw at canonical.com;
> jasowang at redhat.com; Stephen Hemminger <sthemmin@microsoft.com>;
> Michael Kelley <mikelley@microsoft.com>; vkuznets
> <vkuznets@redhat.com>; Michael Kelley <mikelley@microsoft.com>
> Subject: Re: [PATCH 1/4] arm64: hyperv: Add core Hyper-V include files
> 
> On Thu, Nov 22, 2018 at 03:10:56AM +0000, kys at linuxonhyperv.com wrote:
> > --- /dev/null
> > +++ b/arch/arm64/include/asm/hyperv-tlfs.h
> > @@ -0,0 +1,338 @@
> > +/* SPDX-License-Identifier: GPL-2.0 */
> > +
> > +/*
> > + * This file contains definitions from the Hyper-V Hypervisor Top-Level
> > + * Functional Specification (TLFS):
> > + *
> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.
> microsoft.com%2Fen-us%2Fvirtualization%2Fhyper-v-on-
> windows%2Freference%2Ftlfs&amp;data=02%7C01%7Ckys%40microsoft.co
> m%7Cce3a8662cbbc4afe157508d653d3eadf%7C72f988bf86f141af91ab2d7cd0
> 11db47%7C1%7C0%7C636788566988323167&amp;sdata=VJHHMNbpOdAANd
> DNUD3UMuR8SkbBGOew7uy%2B%2BUCS2sE%3D&amp;reserved=0
> 
> "interesting" url :(
> 
> Care to use a real one?

Will fix and resend.

K. Y
> 
> thanks,
> 
> greg k-h

Re: [PATCH 3/4] Drivers: hv: vmbus: Add hooks for per-CPU IRQ

[Greg KH]

On Mon, Nov 26, 2018 at 08:56:50PM +0000, Michael Kelley wrote:
> From: Greg KH <gregkh@linuxfoundation.org>  Monday, November 26, 2018 11:57 AM
> 
> > > > You created "null" hooks that do nothing, for no one in this patch
> > > > series, why?
> > > >
> > >
> > > hv_enable_vmbus_irq() and hv_disable_vmbus_irq() have non-null
> > > implementations in the ARM64 code in patch 2 of this series.  The
> > > implementations are in the new file arch/arm64/hyperv/mshyperv.c.
> > > Or am I misunderstanding your point?
> > 
> > So you use a hook in an earlier patch and then add it in a later one?
> > 
> > Shouldn't you do it the other way around?  As it is, the earlier patch
> > should not work properly, right?
> 
> The earlier patch implements the hook on the ARM64 side but it is
> unused -- it's not called.  The later patch then calls it.  Wouldn't the
> other way around be backwards?

Ah, it wasn't obvious that the previous patch added it at all, why not
just make that addition part of this patch?

> The general approach is for patches 1 and 2 of the series to provide
> all the new code under arch/arm64 to enable Hyper-V.  But the code
> won't get called (or even built) with just these two patches because
> CONFIG_HYPERV can't be selected.  Patch 3 is separate because it
> applies to architecture independent code and arch/x86 code -- I thought
> there might be value in keeping the ARM64 and x86 patches distinct. 
> Patch 4 applies to architecture independent code, and enables the
> ARM64 code in patches 1 and 2 to be compiled and run when
> CONFIG_HYPERV is selected.
> 
> If combining some of the patches in the series is a better approach, I'm
> good with that.

Ok, that makes more sense, if it is easier to get the ARM people to
review this, that's fine.  Doesn't seem like anyone did that yet :(

sorry for the noise,

greg k-h

[PATCH 3/4] Drivers: hv: vmbus: Add hooks for per-CPU IRQ

[Greg KH]

On Mon, Nov 26, 2018 at 08:56:50PM +0000, Michael Kelley wrote:
> From: Greg KH <gregkh@linuxfoundation.org>  Monday, November 26, 2018 11:57 AM
> 
> > > > You created "null" hooks that do nothing, for no one in this patch
> > > > series, why?
> > > >
> > >
> > > hv_enable_vmbus_irq() and hv_disable_vmbus_irq() have non-null
> > > implementations in the ARM64 code in patch 2 of this series.  The
> > > implementations are in the new file arch/arm64/hyperv/mshyperv.c.
> > > Or am I misunderstanding your point?
> > 
> > So you use a hook in an earlier patch and then add it in a later one?
> > 
> > Shouldn't you do it the other way around?  As it is, the earlier patch
> > should not work properly, right?
> 
> The earlier patch implements the hook on the ARM64 side but it is
> unused -- it's not called.  The later patch then calls it.  Wouldn't the
> other way around be backwards?

Ah, it wasn't obvious that the previous patch added it at all, why not
just make that addition part of this patch?

> The general approach is for patches 1 and 2 of the series to provide
> all the new code under arch/arm64 to enable Hyper-V.  But the code
> won't get called (or even built) with just these two patches because
> CONFIG_HYPERV can't be selected.  Patch 3 is separate because it
> applies to architecture independent code and arch/x86 code -- I thought
> there might be value in keeping the ARM64 and x86 patches distinct. 
> Patch 4 applies to architecture independent code, and enables the
> ARM64 code in patches 1 and 2 to be compiled and run when
> CONFIG_HYPERV is selected.
> 
> If combining some of the patches in the series is a better approach, I'm
> good with that.

Ok, that makes more sense, if it is easier to get the ARM people to
review this, that's fine.  Doesn't seem like anyone did that yet :(

sorry for the noise,

greg k-h

Re: [PATCH 3/4] Drivers: hv: vmbus: Add hooks for per-CPU IRQ

[Will Deacon]

On Tue, Nov 27, 2018 at 07:20:56AM +0100, Greg KH wrote:
> On Mon, Nov 26, 2018 at 08:56:50PM +0000, Michael Kelley wrote:
> > From: Greg KH <gregkh@linuxfoundation.org>  Monday, November 26, 2018 11:57 AM
> > 
> > > > > You created "null" hooks that do nothing, for no one in this patch
> > > > > series, why?
> > > > >
> > > >
> > > > hv_enable_vmbus_irq() and hv_disable_vmbus_irq() have non-null
> > > > implementations in the ARM64 code in patch 2 of this series.  The
> > > > implementations are in the new file arch/arm64/hyperv/mshyperv.c.
> > > > Or am I misunderstanding your point?
> > > 
> > > So you use a hook in an earlier patch and then add it in a later one?
> > > 
> > > Shouldn't you do it the other way around?  As it is, the earlier patch
> > > should not work properly, right?
> > 
> > The earlier patch implements the hook on the ARM64 side but it is
> > unused -- it's not called.  The later patch then calls it.  Wouldn't the
> > other way around be backwards?
> 
> Ah, it wasn't obvious that the previous patch added it at all, why not
> just make that addition part of this patch?
> 
> > The general approach is for patches 1 and 2 of the series to provide
> > all the new code under arch/arm64 to enable Hyper-V.  But the code
> > won't get called (or even built) with just these two patches because
> > CONFIG_HYPERV can't be selected.  Patch 3 is separate because it
> > applies to architecture independent code and arch/x86 code -- I thought
> > there might be value in keeping the ARM64 and x86 patches distinct. 
> > Patch 4 applies to architecture independent code, and enables the
> > ARM64 code in patches 1 and 2 to be compiled and run when
> > CONFIG_HYPERV is selected.
> > 
> > If combining some of the patches in the series is a better approach, I'm
> > good with that.
> 
> Ok, that makes more sense, if it is easier to get the ARM people to
> review this, that's fine.  Doesn't seem like anyone did that yet :(

It's on the list, but thanks for having a look as well!

Will

[PATCH 3/4] Drivers: hv: vmbus: Add hooks for per-CPU IRQ

[Will Deacon]

On Tue, Nov 27, 2018 at 07:20:56AM +0100, Greg KH wrote:
> On Mon, Nov 26, 2018 at 08:56:50PM +0000, Michael Kelley wrote:
> > From: Greg KH <gregkh@linuxfoundation.org>  Monday, November 26, 2018 11:57 AM
> > 
> > > > > You created "null" hooks that do nothing, for no one in this patch
> > > > > series, why?
> > > > >
> > > >
> > > > hv_enable_vmbus_irq() and hv_disable_vmbus_irq() have non-null
> > > > implementations in the ARM64 code in patch 2 of this series.  The
> > > > implementations are in the new file arch/arm64/hyperv/mshyperv.c.
> > > > Or am I misunderstanding your point?
> > > 
> > > So you use a hook in an earlier patch and then add it in a later one?
> > > 
> > > Shouldn't you do it the other way around?  As it is, the earlier patch
> > > should not work properly, right?
> > 
> > The earlier patch implements the hook on the ARM64 side but it is
> > unused -- it's not called.  The later patch then calls it.  Wouldn't the
> > other way around be backwards?
> 
> Ah, it wasn't obvious that the previous patch added it at all, why not
> just make that addition part of this patch?
> 
> > The general approach is for patches 1 and 2 of the series to provide
> > all the new code under arch/arm64 to enable Hyper-V.  But the code
> > won't get called (or even built) with just these two patches because
> > CONFIG_HYPERV can't be selected.  Patch 3 is separate because it
> > applies to architecture independent code and arch/x86 code -- I thought
> > there might be value in keeping the ARM64 and x86 patches distinct. 
> > Patch 4 applies to architecture independent code, and enables the
> > ARM64 code in patches 1 and 2 to be compiled and run when
> > CONFIG_HYPERV is selected.
> > 
> > If combining some of the patches in the series is a better approach, I'm
> > good with that.
> 
> Ok, that makes more sense, if it is easier to get the ARM people to
> review this, that's fine.  Doesn't seem like anyone did that yet :(

It's on the list, but thanks for having a look as well!

Will

RE: [PATCH 3/4] Drivers: hv: vmbus: Add hooks for per-CPU IRQ

[Michael Kelley]

From: Will Deacon <will.deacon@arm.com>  Sent: Tuesday, November 27, 2018 2:19 AM

> > > The general approach is for patches 1 and 2 of the series to provide
> > > all the new code under arch/arm64 to enable Hyper-V.  But the code
> > > won't get called (or even built) with just these two patches because
> > > CONFIG_HYPERV can't be selected.  Patch 3 is separate because it
> > > applies to architecture independent code and arch/x86 code -- I thought
> > > there might be value in keeping the ARM64 and x86 patches distinct.
> > > Patch 4 applies to architecture independent code, and enables the
> > > ARM64 code in patches 1 and 2 to be compiled and run when
> > > CONFIG_HYPERV is selected.
> > >
> > > If combining some of the patches in the series is a better approach, I'm
> > > good with that.
> >
> > Ok, that makes more sense, if it is easier to get the ARM people to
> > review this, that's fine.  Doesn't seem like anyone did that yet :(
> 
> It's on the list, but thanks for having a look as well!
> 
> Will

Will -- I'll hold off on sending a new version, pending comments from
the ARM64 maintainers.  Let me know if you prefer that I do otherwise.

Michael

RE: [PATCH 3/4] Drivers: hv: vmbus: Add hooks for per-CPU IRQ

[Michael Kelley]

From: Will Deacon <will.deacon@arm.com>  Sent: Tuesday, November 27, 2018 2:19 AM

> > > The general approach is for patches 1 and 2 of the series to provide
> > > all the new code under arch/arm64 to enable Hyper-V.  But the code
> > > won't get called (or even built) with just these two patches because
> > > CONFIG_HYPERV can't be selected.  Patch 3 is separate because it
> > > applies to architecture independent code and arch/x86 code -- I thought
> > > there might be value in keeping the ARM64 and x86 patches distinct.
> > > Patch 4 applies to architecture independent code, and enables the
> > > ARM64 code in patches 1 and 2 to be compiled and run when
> > > CONFIG_HYPERV is selected.
> > >
> > > If combining some of the patches in the series is a better approach, I'm
> > > good with that.
> >
> > Ok, that makes more sense, if it is easier to get the ARM people to
> > review this, that's fine.  Doesn't seem like anyone did that yet :(
> 
> It's on the list, but thanks for having a look as well!
> 
> Will

Will -- I'll hold off on sending a new version, pending comments from
the ARM64 maintainers.  Let me know if you prefer that I do otherwise.

Michael

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 1/4] arm64: hyperv: Add core Hyper-V include files

[Will Deacon]

Hi all,

On Thu, Nov 22, 2018 at 03:10:56AM +0000, kys@linuxonhyperv.com wrote:
> From: Michael Kelley <mikelley@microsoft.com>
> 
> hyperv-tlfs.h defines Hyper-V interfaces from the Hyper-V Top Level
> Functional Spec (TLFS). The TLFS is distinctly oriented to x86/x64,
> and Hyper-V has not separated out the architecture-dependent parts into
> x86/x64 vs. ARM64. So hyperv-tlfs.h includes information for ARM64
> that is not yet formally published. The TLFS is available here:

When do you plan to publish the spec? It's pretty hard to review this stuff
without knowing what it's supposed to look like.

>   docs.microsoft.com/en-us/virtualization/hyper-v-on-windows/reference/tlfs
> 
> mshyperv.h defines Linux-specific structures and routines for
> interacting with Hyper-V. It is split into an ARM64 specific file
> and an architecture independent file in include/asm-generic.
> 
> Signed-off-by: Michael Kelley <mikelley@microsoft.com>
> Signed-off-by: K. Y. Srinivasan <kys@microsoft.com>
> ---
>  MAINTAINERS                          |   3 +
>  arch/arm64/include/asm/hyperv-tlfs.h | 338 +++++++++++++++++++++++++++
>  arch/arm64/include/asm/mshyperv.h    | 116 +++++++++
>  include/asm-generic/mshyperv.h       | 240 +++++++++++++++++++
>  4 files changed, 697 insertions(+)
>  create mode 100644 arch/arm64/include/asm/hyperv-tlfs.h
>  create mode 100644 arch/arm64/include/asm/mshyperv.h
>  create mode 100644 include/asm-generic/mshyperv.h
> 
> diff --git a/MAINTAINERS b/MAINTAINERS
> index f4855974f325..72f19cef4c48 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -6835,6 +6835,8 @@ F:	arch/x86/include/asm/trace/hyperv.h
>  F:	arch/x86/include/asm/hyperv-tlfs.h
>  F:	arch/x86/kernel/cpu/mshyperv.c
>  F:	arch/x86/hyperv
> +F:	arch/arm64/include/asm/hyperv-tlfs.h
> +F:	arch/arm64/include/asm/mshyperv.h
>  F:	drivers/hid/hid-hyperv.c
>  F:	drivers/hv/
>  F:	drivers/input/serio/hyperv-keyboard.c
> @@ -6846,6 +6848,7 @@ F:	drivers/video/fbdev/hyperv_fb.c
>  F:	net/vmw_vsock/hyperv_transport.c
>  F:	include/linux/hyperv.h
>  F:	include/uapi/linux/hyperv.h
> +F:	include/asm-generic/mshyperv.h
>  F:	tools/hv/
>  F:	Documentation/ABI/stable/sysfs-bus-vmbus
>  
> diff --git a/arch/arm64/include/asm/hyperv-tlfs.h b/arch/arm64/include/asm/hyperv-tlfs.h
> new file mode 100644
> index 000000000000..924e37600e92
> --- /dev/null
> +++ b/arch/arm64/include/asm/hyperv-tlfs.h
> @@ -0,0 +1,338 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +
> +/*
> + * This file contains definitions from the Hyper-V Hypervisor Top-Level
> + * Functional Specification (TLFS):
> + * https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fvirtualization%2Fhyper-v-on-windows%2Freference%2Ftlfs&amp;data=02%7C01%7Ckys%40microsoft.com%7Cc831a45fd63e4a4b083908d641216aa8%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636768009113747528&amp;sdata=jRSrs9ZWXdmeS7LQUEpoSyUfBS7a5KLYy%2FolFdE2tI0%3D&amp;reserved=0

As mentioned elsewhere, please use a better link here and drop the license
boilerplate below.

> + *
> + * Copyright (C) 2018, Microsoft, Inc.
> + *
> + * Author : Michael Kelley <mikelley@microsoft.com>
> + *
> + * This program is free software; you can redistribute it and/or modify it
> + * under the terms of the GNU General Public License version 2 as published
> + * by the Free Software Foundation.
> + *
> + * This program is distributed in the hope that it will be useful, but
> + * WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE, GOOD TITLE or
> + * NON INFRINGEMENT.  See the GNU General Public License for more
> + * details.
> + */
> +
> +#ifndef _ASM_ARM64_HYPERV_H
> +#define _ASM_ARM64_HYPERV_H

__ASM_HYPER_V_H please

> +
> +#include <linux/types.h>
> +
> +/*
> + * These Hyper-V registers provide information equivalent to the CPUID
> + * instruction on x86/x64.
> + */
> +#define HV_REGISTER_HYPERVISOR_VERSION		0x00000100 /*CPUID 0x40000002 */
> +#define	HV_REGISTER_PRIVILEGES_AND_FEATURES	0x00000200 /*CPUID 0x40000003 */
> +#define	HV_REGISTER_FEATURES			0x00000201 /*CPUID 0x40000004 */
> +#define	HV_REGISTER_IMPLEMENTATION_LIMITS	0x00000202 /*CPUID 0x40000005 */
> +#define HV_ARM64_REGISTER_INTERFACE_VERSION	0x00090006 /*CPUID 0x40000001 */
> +
> +/*
> + * Feature identification. HvRegisterPrivilegesAndFeaturesInfo returns a
> + * 128-bit value with flags indicating which features are available to the
> + * partition based upon the current partition privileges. The 128-bit
> + * value is broken up with different portions stored in different 32-bit
> + * fields in the ms_hyperv structure.
> + */
> +
> +/* Partition Reference Counter available*/
> +#define HV_MSR_TIME_REF_COUNT_AVAILABLE		(1 << 1)
> +
> +/*
> + * Synthetic Timers available
> + */
> +#define HV_MSR_SYNTIMER_AVAILABLE		(1 << 3)
> +
> +/* Frequency MSRs available */
> +#define HV_FEATURE_FREQUENCY_MSRS_AVAILABLE	(1 << 8)
> +
> +/* Reference TSC available */
> +#define HV_MSR_REFERENCE_TSC_AVAILABLE		(1 << 9)
> +
> +/* Crash MSR available */
> +#define HV_FEATURE_GUEST_CRASH_MSR_AVAILABLE	(1 << 10)
> +
> +
> +/*
> + * This group of flags is in the high order 64-bits of the returned
> + * 128-bit value.
> + */
> +
> +/* STIMER direct mode is available */
> +#define HV_STIMER_DIRECT_MODE_AVAILABLE		(1 << 19)
> +
> +/*
> + * Implementation recommendations in register
> + * HvRegisterFeaturesInfo. Indicates which behaviors the hypervisor
> + * recommends the OS implement for optimal performance.
> + */
> +
> +/*
> + * Recommend not using Auto EOI
> + */
> +#define HV_DEPRECATING_AEOI_RECOMMENDED		(1 << 9)
> +
> +/*
> + * Synthetic register definitions equivalent to MSRs on x86/x64
> + */
> +#define HV_REGISTER_CRASH_P0		0x00000210
> +#define HV_REGISTER_CRASH_P1		0x00000211
> +#define HV_REGISTER_CRASH_P2		0x00000212
> +#define HV_REGISTER_CRASH_P3		0x00000213
> +#define HV_REGISTER_CRASH_P4		0x00000214
> +#define HV_REGISTER_CRASH_CTL		0x00000215
> +
> +#define HV_REGISTER_GUEST_OSID		0x00090002
> +#define HV_REGISTER_VPINDEX		0x00090003
> +#define HV_REGISTER_TIME_REFCOUNT	0x00090004
> +#define HV_REGISTER_REFERENCE_TSC	0x00090017
> +
> +#define HV_REGISTER_SINT0		0x000A0000
> +#define HV_REGISTER_SINT1		0x000A0001
> +#define HV_REGISTER_SINT2		0x000A0002
> +#define HV_REGISTER_SINT3		0x000A0003
> +#define HV_REGISTER_SINT4		0x000A0004
> +#define HV_REGISTER_SINT5		0x000A0005
> +#define HV_REGISTER_SINT6		0x000A0006
> +#define HV_REGISTER_SINT7		0x000A0007
> +#define HV_REGISTER_SINT8		0x000A0008
> +#define HV_REGISTER_SINT9		0x000A0009
> +#define HV_REGISTER_SINT10		0x000A000A
> +#define HV_REGISTER_SINT11		0x000A000B
> +#define HV_REGISTER_SINT12		0x000A000C
> +#define HV_REGISTER_SINT13		0x000A000D
> +#define HV_REGISTER_SINT14		0x000A000E
> +#define HV_REGISTER_SINT15		0x000A000F
> +#define HV_REGISTER_SCONTROL		0x000A0010
> +#define HV_REGISTER_SVERSION		0x000A0011
> +#define HV_REGISTER_SIFP		0x000A0012
> +#define HV_REGISTER_SIPP		0x000A0013
> +#define HV_REGISTER_EOM			0x000A0014
> +#define HV_REGISTER_SIRBP		0x000A0015
> +
> +#define HV_REGISTER_STIMER0_CONFIG	0x000B0000
> +#define HV_REGISTER_STIMER0_COUNT	0x000B0001
> +#define HV_REGISTER_STIMER1_CONFIG	0x000B0002
> +#define HV_REGISTER_STIMER1_COUNT	0x000B0003
> +#define HV_REGISTER_STIMER2_CONFIG	0x000B0004
> +#define HV_REGISTER_STIMER2_COUNT	0x000B0005
> +#define HV_REGISTER_STIMER3_CONFIG	0x000B0006
> +#define HV_REGISTER_STIMER3_COUNT	0x000B0007
> +
> +/*
> + * Crash notification flags.
> + */
> +#define HV_CRASH_CTL_CRASH_NOTIFY_MSG	BIT_ULL(62)
> +#define HV_CRASH_CTL_CRASH_NOTIFY	BIT_ULL(63)
> +
> +/*
> + * The guest OS needs to register the guest ID with the hypervisor.
> + * The guest ID is a 64 bit entity and the structure of this ID is
> + * specified in the Hyper-V specification:
> + *
> + * msdn.microsoft.com/en-us/library/windows/hardware/ff542653%28v=vs.85%29.aspx

Dead link :(

> + *
> + * While the current guideline does not specify how Linux guest ID(s)
> + * need to be generated, our plan is to publish the guidelines for
> + * Linux and other guest operating systems that currently are hosted
> + * on Hyper-V. The implementation here conforms to this yet
> + * unpublished guidelines.

Again, any timeline on publishing this stuff? Right now, this is just a file
full of magic numbers and I'm not sure we're in a good position to maintain
it based on the idea that you have a cunning plan.

> + *
> + *
> + * Bit(s)
> + * 63 - Indicates if the OS is Open Source or not; 1 is Open Source
> + * 62:56 - Os Type; Linux is 0x100
> + * 55:48 - Distro specific identification
> + * 47:16 - Linux kernel version number
> + * 15:0  - Distro specific identification
> + *
> + *
> + */
> +#define HV_LINUX_VENDOR_ID              0x8100
> +
> +/* Declare the various hypercall operations. */
> +#define HVCALL_FLUSH_VIRTUAL_ADDRESS_SPACE	0x0002
> +#define HVCALL_FLUSH_VIRTUAL_ADDRESS_LIST	0x0003
> +#define HVCALL_NOTIFY_LONG_SPIN_WAIT		0x0008
> +#define HVCALL_SEND_IPI				0x000b
> +#define HVCALL_FLUSH_VIRTUAL_ADDRESS_SPACE_EX	0x0013
> +#define HVCALL_FLUSH_VIRTUAL_ADDRESS_LIST_EX	0x0014
> +#define HVCALL_SEND_IPI_EX			0x0015
> +#define HVCALL_GET_VP_REGISTERS			0x0050
> +#define HVCALL_SET_VP_REGISTERS			0x0051
> +#define HVCALL_POST_MESSAGE			0x005c
> +#define HVCALL_SIGNAL_EVENT			0x005d
> +#define HVCALL_RETARGET_INTERRUPT		0x007e
> +#define HVCALL_START_VIRTUAL_PROCESSOR		0x0099
> +#define HVCALL_GET_VP_INDEX_FROM_APICID		0x009a

This all sounds very x86y...

> +/* Declare standard hypercall field values. */
> +#define HV_PARTITION_ID_SELF                    ((u64)-1)
> +#define HV_VP_INDEX_SELF                        ((u32)-2)
> +
> +#define HV_HYPERCALL_FAST_BIT                   BIT(16)
> +#define HV_HYPERCALL_REP_COUNT_1                BIT_ULL(32)
> +#define HV_HYPERCALL_RESULT_MASK                GENMASK_ULL(15, 0)
> +
> +/* Define the hypercall status result */
> +
> +union hv_hypercall_status {
> +	u64 as_uint64;
> +	struct {
> +		u16 status;
> +		u16 reserved;
> +		u16 reps_completed;  /* Low 12 bits */
> +		u16 reserved2;
> +	};
> +};
> +
> +/* hypercall status code */
> +#define HV_STATUS_SUCCESS			0
> +#define HV_STATUS_INVALID_HYPERCALL_CODE	2
> +#define HV_STATUS_INVALID_HYPERCALL_INPUT	3
> +#define HV_STATUS_INVALID_ALIGNMENT		4
> +#define HV_STATUS_INSUFFICIENT_MEMORY		11
> +#define HV_STATUS_INVALID_CONNECTION_ID		18
> +#define HV_STATUS_INSUFFICIENT_BUFFERS		19
> +
> +/* Define output layout for Get VP Register hypercall */
> +struct hv_get_vp_register_output {
> +	u64 registervaluelow;
> +	u64 registervaluehigh;
> +};
> +
> +#define HV_FLUSH_ALL_PROCESSORS			BIT(0)
> +#define HV_FLUSH_ALL_VIRTUAL_ADDRESS_SPACES	BIT(1)
> +#define HV_FLUSH_NON_GLOBAL_MAPPINGS_ONLY	BIT(2)
> +#define HV_FLUSH_USE_EXTENDED_RANGE_FORMAT	BIT(3)
> +
> +enum HV_GENERIC_SET_FORMAT {
> +	HV_GENERIC_SET_SPARSE_4K,
> +	HV_GENERIC_SET_ALL,
> +};
> +
> +/*
> + * The Hyper-V TimeRefCount register and the TSC
> + * page provide a guest VM clock with 100ns tick rate
> + */
> +#define HV_CLOCK_HZ (NSEC_PER_SEC/100)
> +
> +/*
> + * The fields in this structure are set by Hyper-V and read
> + * by the Linux guest.  They should be accessed with READ_ONCE()
> + * so the compiler doesn't optimize in a way that will cause
> + * problems.
> + */
> +struct ms_hyperv_tsc_page {
> +	u32 tsc_sequence;
> +	u32 reserved1;
> +	u64 tsc_scale;
> +	s64 tsc_offset;
> +	u64 reserved2[509];
> +};

It might be clearer to express this as a union with the page size:

struct ms_hyperv_tsc_page {
	union {
		struct {
			__u32	tsc_sequence;
			__u32	reserved1;
		};
		__u8	reserved2[HV_HYP_PAGE_SIZE];
	};
};

What's the required alignment on this structure?
Also, do you intend for 32-bit guests to use this ABI as well?

> +
> +/* Define the number of synthetic interrupt sources. */
> +#define HV_SYNIC_SINT_COUNT		(16)
> +/* Define the expected SynIC version. */
> +#define HV_SYNIC_VERSION_1		(0x1)
> +
> +#define HV_SYNIC_CONTROL_ENABLE		(1ULL << 0)
> +#define HV_SYNIC_SIMP_ENABLE		(1ULL << 0)
> +#define HV_SYNIC_SIEFP_ENABLE		(1ULL << 0)
> +#define HV_SYNIC_SINT_MASKED		(1ULL << 16)
> +#define HV_SYNIC_SINT_AUTO_EOI		(1ULL << 17)
> +#define HV_SYNIC_SINT_VECTOR_MASK	(0xFF)
> +
> +#define HV_SYNIC_STIMER_COUNT		(4)
> +
> +/* Define synthetic interrupt controller message constants. */
> +#define HV_MESSAGE_SIZE			(256)
> +#define HV_MESSAGE_PAYLOAD_BYTE_COUNT	(240)
> +#define HV_MESSAGE_PAYLOAD_QWORD_COUNT	(30)
> +
> +/* Define hypervisor message types. */
> +enum hv_message_type {
> +	HVMSG_NONE			= 0x00000000,
> +
> +	/* Memory access messages. */
> +	HVMSG_UNMAPPED_GPA		= 0x80000000,
> +	HVMSG_GPA_INTERCEPT		= 0x80000001,
> +
> +	/* Timer notification messages. */
> +	HVMSG_TIMER_EXPIRED		= 0x80000010,
> +
> +	/* Error messages. */
> +	HVMSG_INVALID_VP_REGISTER_VALUE	= 0x80000020,
> +	HVMSG_UNRECOVERABLE_EXCEPTION	= 0x80000021,
> +	HVMSG_UNSUPPORTED_FEATURE	= 0x80000022,
> +
> +	/* Trace buffer complete messages. */
> +	HVMSG_EVENTLOG_BUFFERCOMPLETE	= 0x80000040,
> +};
> +
> +/* Define synthetic interrupt controller message flags. */
> +union hv_message_flags {
> +	__u8 asu8;
> +	struct {
> +		__u8 msg_pending:1;
> +		__u8 reserved:7;
> +	};
> +};
> +
> +/* Define port identifier type. */
> +union hv_port_id {
> +	__u32 asu32;
> +	struct {
> +		__u32 id:24;
> +		__u32 reserved:8;
> +	} u;
> +};

Hmm, I thought bitfields weren't a good idea for this sort of thing. Isn't
it up to the compiler how they are laid out in memory?

> +
> +/* Define synthetic interrupt controller message header. */
> +struct hv_message_header {
> +	__u32 message_type;
> +	__u8 payload_size;
> +	union hv_message_flags message_flags;
> +	__u8 reserved[2];
> +	union {
> +		__u64 sender;
> +		union hv_port_id port;
> +	};
> +};
> +
> +/* Define synthetic interrupt controller message format. */
> +struct hv_message {
> +	struct hv_message_header header;
> +	union {
> +		__u64 payload[HV_MESSAGE_PAYLOAD_QWORD_COUNT];
> +	} u;
> +};
> +
> +/* Define the synthetic interrupt message page layout. */
> +struct hv_message_page {
> +	struct hv_message sint_message[HV_SYNIC_SINT_COUNT];
> +};
> +
> +/* Define timer message payload structure. */
> +struct hv_timer_message_payload {
> +	__u32 timer_index;
> +	__u32 reserved;
> +	__u64 expiration_time;	/* When the timer expired */
> +	__u64 delivery_time;	/* When the message was delivered */
> +};
> +
> +#define HV_STIMER_ENABLE		(1ULL << 0)
> +#define HV_STIMER_PERIODIC		(1ULL << 1)
> +#define HV_STIMER_LAZY			(1ULL << 2)
> +#define HV_STIMER_AUTOENABLE		(1ULL << 3)
> +#define HV_STIMER_SINT(config)		(__u8)(((config) >> 16) & 0x0F)
> +
> +#endif
> diff --git a/arch/arm64/include/asm/mshyperv.h b/arch/arm64/include/asm/mshyperv.h
> new file mode 100644
> index 000000000000..a87c431d58b3
> --- /dev/null
> +++ b/arch/arm64/include/asm/mshyperv.h
> @@ -0,0 +1,116 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +
> +/*
> + * Linux-specific definitions for managing interactions with Microsoft's
> + * Hyper-V hypervisor. The definitions in this file are specific to
> + * the ARM64 architecture.  See include/asm-generic/mshyperv.h for
> + * definitions are that architecture independent.
> + *
> + * Definitions that are specified in the Hyper-V Top Level Functional
> + * Spec (TLFS) should not go in this file, but should instead go in
> + * hyperv-tlfs.h.
> + *
> + * Copyright (C) 2018, Microsoft, Inc.
> + *
> + * Author : Michael Kelley <mikelley@microsoft.com>
> + *
> + * This program is free software; you can redistribute it and/or modify it
> + * under the terms of the GNU General Public License version 2 as published
> + * by the Free Software Foundation.
> + *
> + * This program is distributed in the hope that it will be useful, but
> + * WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE, GOOD TITLE or
> + * NON INFRINGEMENT.  See the GNU General Public License for more
> + * details.
> + */
> +
> +#ifndef _ASM_ARM64_MSHYPERV_H
> +#define _ASM_ARM64_MSHYPERV_H
> +
> +#include <linux/types.h>
> +#include <linux/interrupt.h>
> +#include <linux/clocksource.h>
> +#include <linux/irq.h>
> +#include <linux/irqdesc.h>
> +#include <asm/hyperv-tlfs.h>
> +
> +/*
> + * Define the IRQ numbers/vectors used by Hyper-V VMbus interrupts
> + * and by STIMER0 Direct Mode interrupts. Hyper-V should be supplying
> + * these values through ACPI, but there are no other interrupting
> + * devices in a Hyper-V VM on ARM64, so it's OK to hard code for now.
> + * The "CALLBACK_VECTOR" terminology is a left-over from the x86/x64
> + * world that is used in architecture independent Hyper-V code.
> + */
> +#define HYPERVISOR_CALLBACK_VECTOR 16
> +#define	HV_STIMER0_IRQNR	   17
> +
> +extern u64 hv_do_hvc(u64 control, ...);
> +extern u64 hv_do_hvc_fast_get(u64 control, u64 input1, u64 input2, u64 input3,
> +		struct hv_get_vp_register_output *output);
> +
> +/*
> + * Declare calls to get and set Hyper-V VP register values on ARM64, which
> + * requires a hypercall.
> + */
> +extern void hv_set_vpreg(u32 reg, u64 value);
> +extern u64 hv_get_vpreg(u32 reg);
> +extern void hv_get_vpreg_128(u32 reg, struct hv_get_vp_register_output *result);
> +
> +/*
> + * Use the Hyper-V provided stimer0 as the timer that is made
> + * available to the architecture independent Hyper-V drivers.
> + */
> +#define hv_init_timer(timer, tick) \
> +		hv_set_vpreg(HV_REGISTER_STIMER0_COUNT + (2*timer), tick)
> +#define hv_init_timer_config(timer, val) \
> +		hv_set_vpreg(HV_REGISTER_STIMER0_CONFIG + (2*timer), val)
> +#define hv_get_current_tick(tick) \
> +		(tick = hv_get_vpreg(HV_REGISTER_TIME_REFCOUNT))
> +
> +#define hv_get_simp(val) (val = hv_get_vpreg(HV_REGISTER_SIPP))
> +#define hv_set_simp(val) hv_set_vpreg(HV_REGISTER_SIPP, val)
> +
> +#define hv_get_siefp(val) (val = hv_get_vpreg(HV_REGISTER_SIFP))
> +#define hv_set_siefp(val) hv_set_vpreg(HV_REGISTER_SIFP, val)
> +
> +#define hv_get_synic_state(val) (val = hv_get_vpreg(HV_REGISTER_SCONTROL))
> +#define hv_set_synic_state(val) hv_set_vpreg(HV_REGISTER_SCONTROL, val)
> +
> +#define hv_get_vp_index(index) (index = hv_get_vpreg(HV_REGISTER_VPINDEX))
> +
> +#define hv_signal_eom()	hv_set_vpreg(HV_REGISTER_EOM, 0)
> +
> +/*
> + * Hyper-V SINT registers are numbered sequentially, so we can just
> + * add the SINT number to the register number of SINT0
> + */
> +#define hv_get_synint_state(sint_num, val) \
> +		(val = hv_get_vpreg(HV_REGISTER_SINT0 + sint_num))
> +#define hv_set_synint_state(sint_num, val) \
> +		hv_set_vpreg(HV_REGISTER_SINT0 + sint_num, val)
> +
> +#define hv_get_crash_ctl(val) \
> +		(val = hv_get_vpreg(HV_REGISTER_CRASH_CTL))
> +
> +#if IS_ENABLED(CONFIG_HYPERV)
> +#define hv_enable_stimer0_percpu_irq(irq)	enable_percpu_irq(irq, 0)
> +#define hv_disable_stimer0_percpu_irq(irq)	disable_percpu_irq(irq)
> +#endif
> +
> +/* ARM64 specific code to read the hardware clock */
> +static inline u64 hv_read_hwclock(void)
> +{
> +	u64 result;
> +
> +	isb();
> +	result = read_sysreg(cntvct_el0);
> +	isb();
> +
> +	return result;
> +}

Shouldn't you use arch_timer_read_counter() or arch_counter_get_cntvct()
instead?

> +#include <asm-generic/mshyperv.h>
> +
> +#endif
> diff --git a/include/asm-generic/mshyperv.h b/include/asm-generic/mshyperv.h
> new file mode 100644
> index 000000000000..fbe1ec89c85a
> --- /dev/null
> +++ b/include/asm-generic/mshyperv.h
> @@ -0,0 +1,240 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +
> +/*
> + * Linux-specific definitions for managing interactions with Microsoft's
> + * Hyper-V hypervisor. The definitions in this file are architecture
> + * independent. See arch/<arch>/include/asm/mshyperv.h for definitions
> + * that are specific to architecture <arch>.
> + *
> + * Definitions that are specified in the Hyper-V Top Level Functional
> + * Spec (TLFS) should not go in this file, but should instead go in
> + * hyperv-tlfs.h.
> + *
> + * Copyright (C) 2018, Microsoft, Inc.
> + *
> + * Author : Michael Kelley <mikelley@microsoft.com>
> + *
> + * This program is free software; you can redistribute it and/or modify it
> + * under the terms of the GNU General Public License version 2 as published
> + * by the Free Software Foundation.
> + *
> + * This program is distributed in the hope that it will be useful, but
> + * WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE, GOOD TITLE or
> + * NON INFRINGEMENT.  See the GNU General Public License for more
> + * details.
> + */
> +
> +#ifndef _ASM_GENERIC_MSHYPERV_H
> +#define _ASM_GENERIC_MSHYPERV_H
> +
> +#include <linux/types.h>
> +#include <linux/interrupt.h>
> +#include <linux/clocksource.h>
> +#include <linux/irq.h>
> +#include <linux/irqdesc.h>
> +#include <asm/hyperv-tlfs.h>
> +
> +/*
> + * Hyper-V always runs with a page size of 4096. These definitions
> + * are used when communicating with Hyper-V using guest physical
> + * pages and guest physical page addresses, since the guest page
> + * size may not be 4096 on ARM64.
> + */
> +#define HV_HYP_PAGE_SIZE	4096
> +#define HV_HYP_PAGE_SHIFT	12

Maybe define the page size in terms of the shift?

> +#define HV_HYP_PAGE_MASK	(~(HV_HYP_PAGE_SIZE - 1))
> +
> +
> +struct ms_hyperv_info {
> +	u32 features;
> +	u32 misc_features;
> +	u32 hints;
> +	u32 max_vp_index;
> +	u32 max_lp_index;
> +};

What is the define endianness for in-memory structures shared with the
hypervisor?

> +extern struct ms_hyperv_info ms_hyperv;
> +
> +extern u64 hv_do_hypercall(u64 control, void *inputaddr, void *outputaddr);
> +extern u64 hv_do_fast_hypercall8(u16 control, u64 input8);
> +
> +/*
> + * The guest OS needs to register the guest ID with the hypervisor.
> + * The guest ID is a 64 bit entity and the structure of this ID is
> + * specified in the Hyper-V specification:
> + *
> + * msdn.microsoft.com/en-us/library/windows/hardware/ff542653%28v=vs.85%29.aspx
> + *
> + * While the current guideline does not specify how Linux guest ID(s)
> + * need to be generated, our plan is to publish the guidelines for
> + * Linux and other guest operating systems that currently are hosted
> + * on Hyper-V. The implementation here conforms to this yet
> + * unpublished guidelines.
> + *
> + *
> + * Bit(s)
> + * 63 - Indicates if the OS is Open Source or not; 1 is Open Source
> + * 62:56 - Os Type; Linux is 0x100
> + * 55:48 - Distro specific identification
> + * 47:16 - Linux kernel version number
> + * 15:0  - Distro specific identification

This looks familiar...

> + * Generate the guest ID based on the guideline described above.
> + */
> +
> +static inline  __u64 generate_guest_id(__u64 d_info1, __u64 kernel_version,
> +				       __u64 d_info2)
> +{
> +	__u64 guest_id = 0;
> +
> +	guest_id = (((__u64)HV_LINUX_VENDOR_ID) << 48);
> +	guest_id |= (d_info1 << 48);
> +	guest_id |= (kernel_version << 16);
> +	guest_id |= d_info2;
> +
> +	return guest_id;
> +}
> +
> +
> +/* Free the message slot and signal end-of-message if required */
> +static inline void vmbus_signal_eom(struct hv_message *msg, u32 old_msg_type)
> +{
> +	/*
> +	 * On crash we're reading some other CPU's message page and we need
> +	 * to be careful: this other CPU may already had cleared the header
> +	 * and the host may already had delivered some other message there.
> +	 * In case we blindly write msg->header.message_type we're going
> +	 * to lose it. We can still lose a message of the same type but
> +	 * we count on the fact that there can only be one
> +	 * CHANNELMSG_UNLOAD_RESPONSE and we don't care about other messages
> +	 * on crash.
> +	 */
> +	if (cmpxchg(&msg->header.message_type, old_msg_type,
> +		    HVMSG_NONE) != old_msg_type)
> +		return;
> +
> +	/*
> +	 * Make sure the write to MessageType (ie set to
> +	 * HVMSG_NONE) happens before we read the
> +	 * MessagePending and EOMing. Otherwise, the EOMing
> +	 * will not deliver any more messages since there is
> +	 * no empty slot
> +	 */
> +	mb();

A successful cmpxchg() already implies full barriers, so this isn't needed.
I also wonder whether cmpxchg_acquire() would be sufficient here.

> +
> +	if (msg->header.message_flags.msg_pending) {
> +		/*
> +		 * This will cause message queue rescan to
> +		 * possibly deliver another msg from the
> +		 * hypervisor
> +		 */
> +		hv_signal_eom();
> +	}
> +}
> +
> +void hv_setup_vmbus_irq(void (*handler)(void));
> +void hv_remove_vmbus_irq(void);
> +void hv_enable_vmbus_irq(void);
> +void hv_disable_vmbus_irq(void);
> +
> +void hv_setup_kexec_handler(void (*handler)(void));
> +void hv_remove_kexec_handler(void);
> +void hv_setup_crash_handler(void (*handler)(struct pt_regs *regs));
> +void hv_remove_crash_handler(void);
> +
> +#if IS_ENABLED(CONFIG_HYPERV)
> +extern struct clocksource *hyperv_cs;
> +
> +/*
> + * Hypervisor's notion of virtual processor ID is different from
> + * Linux' notion of CPU ID. This information can only be retrieved
> + * in the context of the calling CPU. Setup a map for easy access
> + * to this information.
> + */
> +extern u32 *hv_vp_index;
> +extern u32 hv_max_vp_index;
> +
> +/* Sentinel value for an uninitialized entry in hv_vp_index array */
> +#define VP_INVAL	U32_MAX
> +
> +/**
> + * hv_cpu_number_to_vp_number() - Map CPU to VP.
> + * @cpu_number: CPU number in Linux terms
> + *
> + * This function returns the mapping between the Linux processor
> + * number and the hypervisor's virtual processor number, useful
> + * in making hypercalls and such that talk about specific
> + * processors.
> + *
> + * Return: Virtual processor number in Hyper-V terms
> + */
> +static inline int hv_cpu_number_to_vp_number(int cpu_number)
> +{
> +	return hv_vp_index[cpu_number];
> +}
> +
> +void hyperv_report_panic(struct pt_regs *regs, long err);
> +void hyperv_report_panic_msg(phys_addr_t pa, size_t size);
> +bool hv_is_hyperv_initialized(void);
> +void hyperv_cleanup(void);
> +#else /* CONFIG_HYPERV */
> +static inline bool hv_is_hyperv_initialized(void) { return false; }
> +static inline void hyperv_cleanup(void) {}
> +#endif /* CONFIG_HYPERV */
> +
> +#if IS_ENABLED(CONFIG_HYPERV)
> +extern int hv_setup_stimer0_irq(int *irq, int *vector, void (*handler)(void));
> +extern void hv_remove_stimer0_irq(int irq);
> +#endif
> +
> +static inline u64 hv_read_tsc_page_tsc(const struct ms_hyperv_tsc_page *tsc_pg,
> +				       u64 *cur_tsc)
> +{
> +	u64	scale, offset;
> +	u32	sequence;
> +
> +	/*
> +	 * The protocol for reading Hyper-V TSC page is specified in Hypervisor
> +	 * Top-Level Functional Specification.  To get the reference time we
> +	 * must do the following:
> +	 * - READ ReferenceTscSequence
> +	 *   A special '0' value indicates the time source is unreliable and we
> +	 *   need to use something else.
> +	 * - ReferenceTime =
> +	 *     ((HWclock val) * ReferenceTscScale) >> 64) + ReferenceTscOffset
> +	 * - READ ReferenceTscSequence again. In case its value has changed
> +	 *   since our first reading we need to discard ReferenceTime and repeat
> +	 *   the whole sequence as the hypervisor was updating the page in
> +	 *   between.
> +	 */
> +	do {
> +		sequence = READ_ONCE(tsc_pg->tsc_sequence);
> +		/*
> +		 * Make sure we read sequence before we read other values from
> +		 * TSC page.
> +		 */
> +		smp_rmb();
> +
> +		scale = READ_ONCE(tsc_pg->tsc_scale);
> +		offset = READ_ONCE(tsc_pg->tsc_offset);
> +		*cur_tsc = hv_read_hwclock();
> +
> +		/*
> +		 * Make sure we read sequence after we read all other values
> +		 * from TSC page.
> +		 */
> +		smp_rmb();
> +
> +	} while (READ_ONCE(tsc_pg->tsc_sequence) != sequence);

Could you explain what the writer side looks like, please? I'm a bit
confused as to why we're not using the bottom bit of the sequence
number to detect a concurrent update.

Will

Re: [PATCH 1/4] arm64: hyperv: Add core Hyper-V include files

[Will Deacon]

Hi all,

On Thu, Nov 22, 2018 at 03:10:56AM +0000, kys@linuxonhyperv.com wrote:
> From: Michael Kelley <mikelley@microsoft.com>
> 
> hyperv-tlfs.h defines Hyper-V interfaces from the Hyper-V Top Level
> Functional Spec (TLFS). The TLFS is distinctly oriented to x86/x64,
> and Hyper-V has not separated out the architecture-dependent parts into
> x86/x64 vs. ARM64. So hyperv-tlfs.h includes information for ARM64
> that is not yet formally published. The TLFS is available here:

When do you plan to publish the spec? It's pretty hard to review this stuff
without knowing what it's supposed to look like.

>   docs.microsoft.com/en-us/virtualization/hyper-v-on-windows/reference/tlfs
> 
> mshyperv.h defines Linux-specific structures and routines for
> interacting with Hyper-V. It is split into an ARM64 specific file
> and an architecture independent file in include/asm-generic.
> 
> Signed-off-by: Michael Kelley <mikelley@microsoft.com>
> Signed-off-by: K. Y. Srinivasan <kys@microsoft.com>
> ---
>  MAINTAINERS                          |   3 +
>  arch/arm64/include/asm/hyperv-tlfs.h | 338 +++++++++++++++++++++++++++
>  arch/arm64/include/asm/mshyperv.h    | 116 +++++++++
>  include/asm-generic/mshyperv.h       | 240 +++++++++++++++++++
>  4 files changed, 697 insertions(+)
>  create mode 100644 arch/arm64/include/asm/hyperv-tlfs.h
>  create mode 100644 arch/arm64/include/asm/mshyperv.h
>  create mode 100644 include/asm-generic/mshyperv.h
> 
> diff --git a/MAINTAINERS b/MAINTAINERS
> index f4855974f325..72f19cef4c48 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -6835,6 +6835,8 @@ F:	arch/x86/include/asm/trace/hyperv.h
>  F:	arch/x86/include/asm/hyperv-tlfs.h
>  F:	arch/x86/kernel/cpu/mshyperv.c
>  F:	arch/x86/hyperv
> +F:	arch/arm64/include/asm/hyperv-tlfs.h
> +F:	arch/arm64/include/asm/mshyperv.h
>  F:	drivers/hid/hid-hyperv.c
>  F:	drivers/hv/
>  F:	drivers/input/serio/hyperv-keyboard.c
> @@ -6846,6 +6848,7 @@ F:	drivers/video/fbdev/hyperv_fb.c
>  F:	net/vmw_vsock/hyperv_transport.c
>  F:	include/linux/hyperv.h
>  F:	include/uapi/linux/hyperv.h
> +F:	include/asm-generic/mshyperv.h
>  F:	tools/hv/
>  F:	Documentation/ABI/stable/sysfs-bus-vmbus
>  
> diff --git a/arch/arm64/include/asm/hyperv-tlfs.h b/arch/arm64/include/asm/hyperv-tlfs.h
> new file mode 100644
> index 000000000000..924e37600e92
> --- /dev/null
> +++ b/arch/arm64/include/asm/hyperv-tlfs.h
> @@ -0,0 +1,338 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +
> +/*
> + * This file contains definitions from the Hyper-V Hypervisor Top-Level
> + * Functional Specification (TLFS):
> + * https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fvirtualization%2Fhyper-v-on-windows%2Freference%2Ftlfs&amp;data=02%7C01%7Ckys%40microsoft.com%7Cc831a45fd63e4a4b083908d641216aa8%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636768009113747528&amp;sdata=jRSrs9ZWXdmeS7LQUEpoSyUfBS7a5KLYy%2FolFdE2tI0%3D&amp;reserved=0

As mentioned elsewhere, please use a better link here and drop the license
boilerplate below.

> + *
> + * Copyright (C) 2018, Microsoft, Inc.
> + *
> + * Author : Michael Kelley <mikelley@microsoft.com>
> + *
> + * This program is free software; you can redistribute it and/or modify it
> + * under the terms of the GNU General Public License version 2 as published
> + * by the Free Software Foundation.
> + *
> + * This program is distributed in the hope that it will be useful, but
> + * WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE, GOOD TITLE or
> + * NON INFRINGEMENT.  See the GNU General Public License for more
> + * details.
> + */
> +
> +#ifndef _ASM_ARM64_HYPERV_H
> +#define _ASM_ARM64_HYPERV_H

__ASM_HYPER_V_H please

> +
> +#include <linux/types.h>
> +
> +/*
> + * These Hyper-V registers provide information equivalent to the CPUID
> + * instruction on x86/x64.
> + */
> +#define HV_REGISTER_HYPERVISOR_VERSION		0x00000100 /*CPUID 0x40000002 */
> +#define	HV_REGISTER_PRIVILEGES_AND_FEATURES	0x00000200 /*CPUID 0x40000003 */
> +#define	HV_REGISTER_FEATURES			0x00000201 /*CPUID 0x40000004 */
> +#define	HV_REGISTER_IMPLEMENTATION_LIMITS	0x00000202 /*CPUID 0x40000005 */
> +#define HV_ARM64_REGISTER_INTERFACE_VERSION	0x00090006 /*CPUID 0x40000001 */
> +
> +/*
> + * Feature identification. HvRegisterPrivilegesAndFeaturesInfo returns a
> + * 128-bit value with flags indicating which features are available to the
> + * partition based upon the current partition privileges. The 128-bit
> + * value is broken up with different portions stored in different 32-bit
> + * fields in the ms_hyperv structure.
> + */
> +
> +/* Partition Reference Counter available*/
> +#define HV_MSR_TIME_REF_COUNT_AVAILABLE		(1 << 1)
> +
> +/*
> + * Synthetic Timers available
> + */
> +#define HV_MSR_SYNTIMER_AVAILABLE		(1 << 3)
> +
> +/* Frequency MSRs available */
> +#define HV_FEATURE_FREQUENCY_MSRS_AVAILABLE	(1 << 8)
> +
> +/* Reference TSC available */
> +#define HV_MSR_REFERENCE_TSC_AVAILABLE		(1 << 9)
> +
> +/* Crash MSR available */
> +#define HV_FEATURE_GUEST_CRASH_MSR_AVAILABLE	(1 << 10)
> +
> +
> +/*
> + * This group of flags is in the high order 64-bits of the returned
> + * 128-bit value.
> + */
> +
> +/* STIMER direct mode is available */
> +#define HV_STIMER_DIRECT_MODE_AVAILABLE		(1 << 19)
> +
> +/*
> + * Implementation recommendations in register
> + * HvRegisterFeaturesInfo. Indicates which behaviors the hypervisor
> + * recommends the OS implement for optimal performance.
> + */
> +
> +/*
> + * Recommend not using Auto EOI
> + */
> +#define HV_DEPRECATING_AEOI_RECOMMENDED		(1 << 9)
> +
> +/*
> + * Synthetic register definitions equivalent to MSRs on x86/x64
> + */
> +#define HV_REGISTER_CRASH_P0		0x00000210
> +#define HV_REGISTER_CRASH_P1		0x00000211
> +#define HV_REGISTER_CRASH_P2		0x00000212
> +#define HV_REGISTER_CRASH_P3		0x00000213
> +#define HV_REGISTER_CRASH_P4		0x00000214
> +#define HV_REGISTER_CRASH_CTL		0x00000215
> +
> +#define HV_REGISTER_GUEST_OSID		0x00090002
> +#define HV_REGISTER_VPINDEX		0x00090003
> +#define HV_REGISTER_TIME_REFCOUNT	0x00090004
> +#define HV_REGISTER_REFERENCE_TSC	0x00090017
> +
> +#define HV_REGISTER_SINT0		0x000A0000
> +#define HV_REGISTER_SINT1		0x000A0001
> +#define HV_REGISTER_SINT2		0x000A0002
> +#define HV_REGISTER_SINT3		0x000A0003
> +#define HV_REGISTER_SINT4		0x000A0004
> +#define HV_REGISTER_SINT5		0x000A0005
> +#define HV_REGISTER_SINT6		0x000A0006
> +#define HV_REGISTER_SINT7		0x000A0007
> +#define HV_REGISTER_SINT8		0x000A0008
> +#define HV_REGISTER_SINT9		0x000A0009
> +#define HV_REGISTER_SINT10		0x000A000A
> +#define HV_REGISTER_SINT11		0x000A000B
> +#define HV_REGISTER_SINT12		0x000A000C
> +#define HV_REGISTER_SINT13		0x000A000D
> +#define HV_REGISTER_SINT14		0x000A000E
> +#define HV_REGISTER_SINT15		0x000A000F
> +#define HV_REGISTER_SCONTROL		0x000A0010
> +#define HV_REGISTER_SVERSION		0x000A0011
> +#define HV_REGISTER_SIFP		0x000A0012
> +#define HV_REGISTER_SIPP		0x000A0013
> +#define HV_REGISTER_EOM			0x000A0014
> +#define HV_REGISTER_SIRBP		0x000A0015
> +
> +#define HV_REGISTER_STIMER0_CONFIG	0x000B0000
> +#define HV_REGISTER_STIMER0_COUNT	0x000B0001
> +#define HV_REGISTER_STIMER1_CONFIG	0x000B0002
> +#define HV_REGISTER_STIMER1_COUNT	0x000B0003
> +#define HV_REGISTER_STIMER2_CONFIG	0x000B0004
> +#define HV_REGISTER_STIMER2_COUNT	0x000B0005
> +#define HV_REGISTER_STIMER3_CONFIG	0x000B0006
> +#define HV_REGISTER_STIMER3_COUNT	0x000B0007
> +
> +/*
> + * Crash notification flags.
> + */
> +#define HV_CRASH_CTL_CRASH_NOTIFY_MSG	BIT_ULL(62)
> +#define HV_CRASH_CTL_CRASH_NOTIFY	BIT_ULL(63)
> +
> +/*
> + * The guest OS needs to register the guest ID with the hypervisor.
> + * The guest ID is a 64 bit entity and the structure of this ID is
> + * specified in the Hyper-V specification:
> + *
> + * msdn.microsoft.com/en-us/library/windows/hardware/ff542653%28v=vs.85%29.aspx

Dead link :(

> + *
> + * While the current guideline does not specify how Linux guest ID(s)
> + * need to be generated, our plan is to publish the guidelines for
> + * Linux and other guest operating systems that currently are hosted
> + * on Hyper-V. The implementation here conforms to this yet
> + * unpublished guidelines.

Again, any timeline on publishing this stuff? Right now, this is just a file
full of magic numbers and I'm not sure we're in a good position to maintain
it based on the idea that you have a cunning plan.

> + *
> + *
> + * Bit(s)
> + * 63 - Indicates if the OS is Open Source or not; 1 is Open Source
> + * 62:56 - Os Type; Linux is 0x100
> + * 55:48 - Distro specific identification
> + * 47:16 - Linux kernel version number
> + * 15:0  - Distro specific identification
> + *
> + *
> + */
> +#define HV_LINUX_VENDOR_ID              0x8100
> +
> +/* Declare the various hypercall operations. */
> +#define HVCALL_FLUSH_VIRTUAL_ADDRESS_SPACE	0x0002
> +#define HVCALL_FLUSH_VIRTUAL_ADDRESS_LIST	0x0003
> +#define HVCALL_NOTIFY_LONG_SPIN_WAIT		0x0008
> +#define HVCALL_SEND_IPI				0x000b
> +#define HVCALL_FLUSH_VIRTUAL_ADDRESS_SPACE_EX	0x0013
> +#define HVCALL_FLUSH_VIRTUAL_ADDRESS_LIST_EX	0x0014
> +#define HVCALL_SEND_IPI_EX			0x0015
> +#define HVCALL_GET_VP_REGISTERS			0x0050
> +#define HVCALL_SET_VP_REGISTERS			0x0051
> +#define HVCALL_POST_MESSAGE			0x005c
> +#define HVCALL_SIGNAL_EVENT			0x005d
> +#define HVCALL_RETARGET_INTERRUPT		0x007e
> +#define HVCALL_START_VIRTUAL_PROCESSOR		0x0099
> +#define HVCALL_GET_VP_INDEX_FROM_APICID		0x009a

This all sounds very x86y...

> +/* Declare standard hypercall field values. */
> +#define HV_PARTITION_ID_SELF                    ((u64)-1)
> +#define HV_VP_INDEX_SELF                        ((u32)-2)
> +
> +#define HV_HYPERCALL_FAST_BIT                   BIT(16)
> +#define HV_HYPERCALL_REP_COUNT_1                BIT_ULL(32)
> +#define HV_HYPERCALL_RESULT_MASK                GENMASK_ULL(15, 0)
> +
> +/* Define the hypercall status result */
> +
> +union hv_hypercall_status {
> +	u64 as_uint64;
> +	struct {
> +		u16 status;
> +		u16 reserved;
> +		u16 reps_completed;  /* Low 12 bits */
> +		u16 reserved2;
> +	};
> +};
> +
> +/* hypercall status code */
> +#define HV_STATUS_SUCCESS			0
> +#define HV_STATUS_INVALID_HYPERCALL_CODE	2
> +#define HV_STATUS_INVALID_HYPERCALL_INPUT	3
> +#define HV_STATUS_INVALID_ALIGNMENT		4
> +#define HV_STATUS_INSUFFICIENT_MEMORY		11
> +#define HV_STATUS_INVALID_CONNECTION_ID		18
> +#define HV_STATUS_INSUFFICIENT_BUFFERS		19
> +
> +/* Define output layout for Get VP Register hypercall */
> +struct hv_get_vp_register_output {
> +	u64 registervaluelow;
> +	u64 registervaluehigh;
> +};
> +
> +#define HV_FLUSH_ALL_PROCESSORS			BIT(0)
> +#define HV_FLUSH_ALL_VIRTUAL_ADDRESS_SPACES	BIT(1)
> +#define HV_FLUSH_NON_GLOBAL_MAPPINGS_ONLY	BIT(2)
> +#define HV_FLUSH_USE_EXTENDED_RANGE_FORMAT	BIT(3)
> +
> +enum HV_GENERIC_SET_FORMAT {
> +	HV_GENERIC_SET_SPARSE_4K,
> +	HV_GENERIC_SET_ALL,
> +};
> +
> +/*
> + * The Hyper-V TimeRefCount register and the TSC
> + * page provide a guest VM clock with 100ns tick rate
> + */
> +#define HV_CLOCK_HZ (NSEC_PER_SEC/100)
> +
> +/*
> + * The fields in this structure are set by Hyper-V and read
> + * by the Linux guest.  They should be accessed with READ_ONCE()
> + * so the compiler doesn't optimize in a way that will cause
> + * problems.
> + */
> +struct ms_hyperv_tsc_page {
> +	u32 tsc_sequence;
> +	u32 reserved1;
> +	u64 tsc_scale;
> +	s64 tsc_offset;
> +	u64 reserved2[509];
> +};

It might be clearer to express this as a union with the page size:

struct ms_hyperv_tsc_page {
	union {
		struct {
			__u32	tsc_sequence;
			__u32	reserved1;
		};
		__u8	reserved2[HV_HYP_PAGE_SIZE];
	};
};

What's the required alignment on this structure?
Also, do you intend for 32-bit guests to use this ABI as well?

> +
> +/* Define the number of synthetic interrupt sources. */
> +#define HV_SYNIC_SINT_COUNT		(16)
> +/* Define the expected SynIC version. */
> +#define HV_SYNIC_VERSION_1		(0x1)
> +
> +#define HV_SYNIC_CONTROL_ENABLE		(1ULL << 0)
> +#define HV_SYNIC_SIMP_ENABLE		(1ULL << 0)
> +#define HV_SYNIC_SIEFP_ENABLE		(1ULL << 0)
> +#define HV_SYNIC_SINT_MASKED		(1ULL << 16)
> +#define HV_SYNIC_SINT_AUTO_EOI		(1ULL << 17)
> +#define HV_SYNIC_SINT_VECTOR_MASK	(0xFF)
> +
> +#define HV_SYNIC_STIMER_COUNT		(4)
> +
> +/* Define synthetic interrupt controller message constants. */
> +#define HV_MESSAGE_SIZE			(256)
> +#define HV_MESSAGE_PAYLOAD_BYTE_COUNT	(240)
> +#define HV_MESSAGE_PAYLOAD_QWORD_COUNT	(30)
> +
> +/* Define hypervisor message types. */
> +enum hv_message_type {
> +	HVMSG_NONE			= 0x00000000,
> +
> +	/* Memory access messages. */
> +	HVMSG_UNMAPPED_GPA		= 0x80000000,
> +	HVMSG_GPA_INTERCEPT		= 0x80000001,
> +
> +	/* Timer notification messages. */
> +	HVMSG_TIMER_EXPIRED		= 0x80000010,
> +
> +	/* Error messages. */
> +	HVMSG_INVALID_VP_REGISTER_VALUE	= 0x80000020,
> +	HVMSG_UNRECOVERABLE_EXCEPTION	= 0x80000021,
> +	HVMSG_UNSUPPORTED_FEATURE	= 0x80000022,
> +
> +	/* Trace buffer complete messages. */
> +	HVMSG_EVENTLOG_BUFFERCOMPLETE	= 0x80000040,
> +};
> +
> +/* Define synthetic interrupt controller message flags. */
> +union hv_message_flags {
> +	__u8 asu8;
> +	struct {
> +		__u8 msg_pending:1;
> +		__u8 reserved:7;
> +	};
> +};
> +
> +/* Define port identifier type. */
> +union hv_port_id {
> +	__u32 asu32;
> +	struct {
> +		__u32 id:24;
> +		__u32 reserved:8;
> +	} u;
> +};

Hmm, I thought bitfields weren't a good idea for this sort of thing. Isn't
it up to the compiler how they are laid out in memory?

> +
> +/* Define synthetic interrupt controller message header. */
> +struct hv_message_header {
> +	__u32 message_type;
> +	__u8 payload_size;
> +	union hv_message_flags message_flags;
> +	__u8 reserved[2];
> +	union {
> +		__u64 sender;
> +		union hv_port_id port;
> +	};
> +};
> +
> +/* Define synthetic interrupt controller message format. */
> +struct hv_message {
> +	struct hv_message_header header;
> +	union {
> +		__u64 payload[HV_MESSAGE_PAYLOAD_QWORD_COUNT];
> +	} u;
> +};
> +
> +/* Define the synthetic interrupt message page layout. */
> +struct hv_message_page {
> +	struct hv_message sint_message[HV_SYNIC_SINT_COUNT];
> +};
> +
> +/* Define timer message payload structure. */
> +struct hv_timer_message_payload {
> +	__u32 timer_index;
> +	__u32 reserved;
> +	__u64 expiration_time;	/* When the timer expired */
> +	__u64 delivery_time;	/* When the message was delivered */
> +};
> +
> +#define HV_STIMER_ENABLE		(1ULL << 0)
> +#define HV_STIMER_PERIODIC		(1ULL << 1)
> +#define HV_STIMER_LAZY			(1ULL << 2)
> +#define HV_STIMER_AUTOENABLE		(1ULL << 3)
> +#define HV_STIMER_SINT(config)		(__u8)(((config) >> 16) & 0x0F)
> +
> +#endif
> diff --git a/arch/arm64/include/asm/mshyperv.h b/arch/arm64/include/asm/mshyperv.h
> new file mode 100644
> index 000000000000..a87c431d58b3
> --- /dev/null
> +++ b/arch/arm64/include/asm/mshyperv.h
> @@ -0,0 +1,116 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +
> +/*
> + * Linux-specific definitions for managing interactions with Microsoft's
> + * Hyper-V hypervisor. The definitions in this file are specific to
> + * the ARM64 architecture.  See include/asm-generic/mshyperv.h for
> + * definitions are that architecture independent.
> + *
> + * Definitions that are specified in the Hyper-V Top Level Functional
> + * Spec (TLFS) should not go in this file, but should instead go in
> + * hyperv-tlfs.h.
> + *
> + * Copyright (C) 2018, Microsoft, Inc.
> + *
> + * Author : Michael Kelley <mikelley@microsoft.com>
> + *
> + * This program is free software; you can redistribute it and/or modify it
> + * under the terms of the GNU General Public License version 2 as published
> + * by the Free Software Foundation.
> + *
> + * This program is distributed in the hope that it will be useful, but
> + * WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE, GOOD TITLE or
> + * NON INFRINGEMENT.  See the GNU General Public License for more
> + * details.
> + */
> +
> +#ifndef _ASM_ARM64_MSHYPERV_H
> +#define _ASM_ARM64_MSHYPERV_H
> +
> +#include <linux/types.h>
> +#include <linux/interrupt.h>
> +#include <linux/clocksource.h>
> +#include <linux/irq.h>
> +#include <linux/irqdesc.h>
> +#include <asm/hyperv-tlfs.h>
> +
> +/*
> + * Define the IRQ numbers/vectors used by Hyper-V VMbus interrupts
> + * and by STIMER0 Direct Mode interrupts. Hyper-V should be supplying
> + * these values through ACPI, but there are no other interrupting
> + * devices in a Hyper-V VM on ARM64, so it's OK to hard code for now.
> + * The "CALLBACK_VECTOR" terminology is a left-over from the x86/x64
> + * world that is used in architecture independent Hyper-V code.
> + */
> +#define HYPERVISOR_CALLBACK_VECTOR 16
> +#define	HV_STIMER0_IRQNR	   17
> +
> +extern u64 hv_do_hvc(u64 control, ...);
> +extern u64 hv_do_hvc_fast_get(u64 control, u64 input1, u64 input2, u64 input3,
> +		struct hv_get_vp_register_output *output);
> +
> +/*
> + * Declare calls to get and set Hyper-V VP register values on ARM64, which
> + * requires a hypercall.
> + */
> +extern void hv_set_vpreg(u32 reg, u64 value);
> +extern u64 hv_get_vpreg(u32 reg);
> +extern void hv_get_vpreg_128(u32 reg, struct hv_get_vp_register_output *result);
> +
> +/*
> + * Use the Hyper-V provided stimer0 as the timer that is made
> + * available to the architecture independent Hyper-V drivers.
> + */
> +#define hv_init_timer(timer, tick) \
> +		hv_set_vpreg(HV_REGISTER_STIMER0_COUNT + (2*timer), tick)
> +#define hv_init_timer_config(timer, val) \
> +		hv_set_vpreg(HV_REGISTER_STIMER0_CONFIG + (2*timer), val)
> +#define hv_get_current_tick(tick) \
> +		(tick = hv_get_vpreg(HV_REGISTER_TIME_REFCOUNT))
> +
> +#define hv_get_simp(val) (val = hv_get_vpreg(HV_REGISTER_SIPP))
> +#define hv_set_simp(val) hv_set_vpreg(HV_REGISTER_SIPP, val)
> +
> +#define hv_get_siefp(val) (val = hv_get_vpreg(HV_REGISTER_SIFP))
> +#define hv_set_siefp(val) hv_set_vpreg(HV_REGISTER_SIFP, val)
> +
> +#define hv_get_synic_state(val) (val = hv_get_vpreg(HV_REGISTER_SCONTROL))
> +#define hv_set_synic_state(val) hv_set_vpreg(HV_REGISTER_SCONTROL, val)
> +
> +#define hv_get_vp_index(index) (index = hv_get_vpreg(HV_REGISTER_VPINDEX))
> +
> +#define hv_signal_eom()	hv_set_vpreg(HV_REGISTER_EOM, 0)
> +
> +/*
> + * Hyper-V SINT registers are numbered sequentially, so we can just
> + * add the SINT number to the register number of SINT0
> + */
> +#define hv_get_synint_state(sint_num, val) \
> +		(val = hv_get_vpreg(HV_REGISTER_SINT0 + sint_num))
> +#define hv_set_synint_state(sint_num, val) \
> +		hv_set_vpreg(HV_REGISTER_SINT0 + sint_num, val)
> +
> +#define hv_get_crash_ctl(val) \
> +		(val = hv_get_vpreg(HV_REGISTER_CRASH_CTL))
> +
> +#if IS_ENABLED(CONFIG_HYPERV)
> +#define hv_enable_stimer0_percpu_irq(irq)	enable_percpu_irq(irq, 0)
> +#define hv_disable_stimer0_percpu_irq(irq)	disable_percpu_irq(irq)
> +#endif
> +
> +/* ARM64 specific code to read the hardware clock */
> +static inline u64 hv_read_hwclock(void)
> +{
> +	u64 result;
> +
> +	isb();
> +	result = read_sysreg(cntvct_el0);
> +	isb();
> +
> +	return result;
> +}

Shouldn't you use arch_timer_read_counter() or arch_counter_get_cntvct()
instead?

> +#include <asm-generic/mshyperv.h>
> +
> +#endif
> diff --git a/include/asm-generic/mshyperv.h b/include/asm-generic/mshyperv.h
> new file mode 100644
> index 000000000000..fbe1ec89c85a
> --- /dev/null
> +++ b/include/asm-generic/mshyperv.h
> @@ -0,0 +1,240 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +
> +/*
> + * Linux-specific definitions for managing interactions with Microsoft's
> + * Hyper-V hypervisor. The definitions in this file are architecture
> + * independent. See arch/<arch>/include/asm/mshyperv.h for definitions
> + * that are specific to architecture <arch>.
> + *
> + * Definitions that are specified in the Hyper-V Top Level Functional
> + * Spec (TLFS) should not go in this file, but should instead go in
> + * hyperv-tlfs.h.
> + *
> + * Copyright (C) 2018, Microsoft, Inc.
> + *
> + * Author : Michael Kelley <mikelley@microsoft.com>
> + *
> + * This program is free software; you can redistribute it and/or modify it
> + * under the terms of the GNU General Public License version 2 as published
> + * by the Free Software Foundation.
> + *
> + * This program is distributed in the hope that it will be useful, but
> + * WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE, GOOD TITLE or
> + * NON INFRINGEMENT.  See the GNU General Public License for more
> + * details.
> + */
> +
> +#ifndef _ASM_GENERIC_MSHYPERV_H
> +#define _ASM_GENERIC_MSHYPERV_H
> +
> +#include <linux/types.h>
> +#include <linux/interrupt.h>
> +#include <linux/clocksource.h>
> +#include <linux/irq.h>
> +#include <linux/irqdesc.h>
> +#include <asm/hyperv-tlfs.h>
> +
> +/*
> + * Hyper-V always runs with a page size of 4096. These definitions
> + * are used when communicating with Hyper-V using guest physical
> + * pages and guest physical page addresses, since the guest page
> + * size may not be 4096 on ARM64.
> + */
> +#define HV_HYP_PAGE_SIZE	4096
> +#define HV_HYP_PAGE_SHIFT	12

Maybe define the page size in terms of the shift?

> +#define HV_HYP_PAGE_MASK	(~(HV_HYP_PAGE_SIZE - 1))
> +
> +
> +struct ms_hyperv_info {
> +	u32 features;
> +	u32 misc_features;
> +	u32 hints;
> +	u32 max_vp_index;
> +	u32 max_lp_index;
> +};

What is the define endianness for in-memory structures shared with the
hypervisor?

> +extern struct ms_hyperv_info ms_hyperv;
> +
> +extern u64 hv_do_hypercall(u64 control, void *inputaddr, void *outputaddr);
> +extern u64 hv_do_fast_hypercall8(u16 control, u64 input8);
> +
> +/*
> + * The guest OS needs to register the guest ID with the hypervisor.
> + * The guest ID is a 64 bit entity and the structure of this ID is
> + * specified in the Hyper-V specification:
> + *
> + * msdn.microsoft.com/en-us/library/windows/hardware/ff542653%28v=vs.85%29.aspx
> + *
> + * While the current guideline does not specify how Linux guest ID(s)
> + * need to be generated, our plan is to publish the guidelines for
> + * Linux and other guest operating systems that currently are hosted
> + * on Hyper-V. The implementation here conforms to this yet
> + * unpublished guidelines.
> + *
> + *
> + * Bit(s)
> + * 63 - Indicates if the OS is Open Source or not; 1 is Open Source
> + * 62:56 - Os Type; Linux is 0x100
> + * 55:48 - Distro specific identification
> + * 47:16 - Linux kernel version number
> + * 15:0  - Distro specific identification

This looks familiar...

> + * Generate the guest ID based on the guideline described above.
> + */
> +
> +static inline  __u64 generate_guest_id(__u64 d_info1, __u64 kernel_version,
> +				       __u64 d_info2)
> +{
> +	__u64 guest_id = 0;
> +
> +	guest_id = (((__u64)HV_LINUX_VENDOR_ID) << 48);
> +	guest_id |= (d_info1 << 48);
> +	guest_id |= (kernel_version << 16);
> +	guest_id |= d_info2;
> +
> +	return guest_id;
> +}
> +
> +
> +/* Free the message slot and signal end-of-message if required */
> +static inline void vmbus_signal_eom(struct hv_message *msg, u32 old_msg_type)
> +{
> +	/*
> +	 * On crash we're reading some other CPU's message page and we need
> +	 * to be careful: this other CPU may already had cleared the header
> +	 * and the host may already had delivered some other message there.
> +	 * In case we blindly write msg->header.message_type we're going
> +	 * to lose it. We can still lose a message of the same type but
> +	 * we count on the fact that there can only be one
> +	 * CHANNELMSG_UNLOAD_RESPONSE and we don't care about other messages
> +	 * on crash.
> +	 */
> +	if (cmpxchg(&msg->header.message_type, old_msg_type,
> +		    HVMSG_NONE) != old_msg_type)
> +		return;
> +
> +	/*
> +	 * Make sure the write to MessageType (ie set to
> +	 * HVMSG_NONE) happens before we read the
> +	 * MessagePending and EOMing. Otherwise, the EOMing
> +	 * will not deliver any more messages since there is
> +	 * no empty slot
> +	 */
> +	mb();

A successful cmpxchg() already implies full barriers, so this isn't needed.
I also wonder whether cmpxchg_acquire() would be sufficient here.

> +
> +	if (msg->header.message_flags.msg_pending) {
> +		/*
> +		 * This will cause message queue rescan to
> +		 * possibly deliver another msg from the
> +		 * hypervisor
> +		 */
> +		hv_signal_eom();
> +	}
> +}
> +
> +void hv_setup_vmbus_irq(void (*handler)(void));
> +void hv_remove_vmbus_irq(void);
> +void hv_enable_vmbus_irq(void);
> +void hv_disable_vmbus_irq(void);
> +
> +void hv_setup_kexec_handler(void (*handler)(void));
> +void hv_remove_kexec_handler(void);
> +void hv_setup_crash_handler(void (*handler)(struct pt_regs *regs));
> +void hv_remove_crash_handler(void);
> +
> +#if IS_ENABLED(CONFIG_HYPERV)
> +extern struct clocksource *hyperv_cs;
> +
> +/*
> + * Hypervisor's notion of virtual processor ID is different from
> + * Linux' notion of CPU ID. This information can only be retrieved
> + * in the context of the calling CPU. Setup a map for easy access
> + * to this information.
> + */
> +extern u32 *hv_vp_index;
> +extern u32 hv_max_vp_index;
> +
> +/* Sentinel value for an uninitialized entry in hv_vp_index array */
> +#define VP_INVAL	U32_MAX
> +
> +/**
> + * hv_cpu_number_to_vp_number() - Map CPU to VP.
> + * @cpu_number: CPU number in Linux terms
> + *
> + * This function returns the mapping between the Linux processor
> + * number and the hypervisor's virtual processor number, useful
> + * in making hypercalls and such that talk about specific
> + * processors.
> + *
> + * Return: Virtual processor number in Hyper-V terms
> + */
> +static inline int hv_cpu_number_to_vp_number(int cpu_number)
> +{
> +	return hv_vp_index[cpu_number];
> +}
> +
> +void hyperv_report_panic(struct pt_regs *regs, long err);
> +void hyperv_report_panic_msg(phys_addr_t pa, size_t size);
> +bool hv_is_hyperv_initialized(void);
> +void hyperv_cleanup(void);
> +#else /* CONFIG_HYPERV */
> +static inline bool hv_is_hyperv_initialized(void) { return false; }
> +static inline void hyperv_cleanup(void) {}
> +#endif /* CONFIG_HYPERV */
> +
> +#if IS_ENABLED(CONFIG_HYPERV)
> +extern int hv_setup_stimer0_irq(int *irq, int *vector, void (*handler)(void));
> +extern void hv_remove_stimer0_irq(int irq);
> +#endif
> +
> +static inline u64 hv_read_tsc_page_tsc(const struct ms_hyperv_tsc_page *tsc_pg,
> +				       u64 *cur_tsc)
> +{
> +	u64	scale, offset;
> +	u32	sequence;
> +
> +	/*
> +	 * The protocol for reading Hyper-V TSC page is specified in Hypervisor
> +	 * Top-Level Functional Specification.  To get the reference time we
> +	 * must do the following:
> +	 * - READ ReferenceTscSequence
> +	 *   A special '0' value indicates the time source is unreliable and we
> +	 *   need to use something else.
> +	 * - ReferenceTime =
> +	 *     ((HWclock val) * ReferenceTscScale) >> 64) + ReferenceTscOffset
> +	 * - READ ReferenceTscSequence again. In case its value has changed
> +	 *   since our first reading we need to discard ReferenceTime and repeat
> +	 *   the whole sequence as the hypervisor was updating the page in
> +	 *   between.
> +	 */
> +	do {
> +		sequence = READ_ONCE(tsc_pg->tsc_sequence);
> +		/*
> +		 * Make sure we read sequence before we read other values from
> +		 * TSC page.
> +		 */
> +		smp_rmb();
> +
> +		scale = READ_ONCE(tsc_pg->tsc_scale);
> +		offset = READ_ONCE(tsc_pg->tsc_offset);
> +		*cur_tsc = hv_read_hwclock();
> +
> +		/*
> +		 * Make sure we read sequence after we read all other values
> +		 * from TSC page.
> +		 */
> +		smp_rmb();
> +
> +	} while (READ_ONCE(tsc_pg->tsc_sequence) != sequence);

Could you explain what the writer side looks like, please? I'm a bit
confused as to why we're not using the bottom bit of the sequence
number to detect a concurrent update.

Will

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 2/4] arm64: hyperv: Add support for Hyper-V as a hypervisor

[Will Deacon]

On Thu, Nov 22, 2018 at 03:10:57AM +0000, kys@linuxonhyperv.com wrote:
> From: Michael Kelley <mikelley@microsoft.com>
> 
> Add ARM64-specific code to enable Hyper-V. This code includes:
> * Detecting Hyper-V and initializing the guest/Hyper-V interface
> * Setting up Hyper-V's synthetic clocks
> * Making hypercalls using the HVC instruction
> * Setting up VMbus and stimer0 interrupts
> * Setting up kexec and crash handlers
> This code is architecture dependent code and is mostly driven by
> architecture independent code in the VMbus driver in drivers/hv/hv.c
> and drivers/hv/vmbus_drv.c.
> 
> This code is built only when CONFIG_HYPERV is enabled.
> 
> Signed-off-by: Michael Kelley <mikelley@microsoft.com>
> Signed-off-by: K. Y. Srinivasan <kys@microsoft.com>
> ---
>  MAINTAINERS                  |   1 +
>  arch/arm64/Makefile          |   1 +
>  arch/arm64/hyperv/Makefile   |   2 +
>  arch/arm64/hyperv/hv_hvc.S   |  54 +++++
>  arch/arm64/hyperv/hv_init.c  | 441 +++++++++++++++++++++++++++++++++++
>  arch/arm64/hyperv/mshyperv.c | 178 ++++++++++++++
>  6 files changed, 677 insertions(+)
>  create mode 100644 arch/arm64/hyperv/Makefile
>  create mode 100644 arch/arm64/hyperv/hv_hvc.S
>  create mode 100644 arch/arm64/hyperv/hv_init.c
>  create mode 100644 arch/arm64/hyperv/mshyperv.c
> 
> diff --git a/MAINTAINERS b/MAINTAINERS
> index 72f19cef4c48..326eeb32a0cd 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -6837,6 +6837,7 @@ F:	arch/x86/kernel/cpu/mshyperv.c
>  F:	arch/x86/hyperv
>  F:	arch/arm64/include/asm/hyperv-tlfs.h
>  F:	arch/arm64/include/asm/mshyperv.h
> +F:	arch/arm64/hyperv
>  F:	drivers/hid/hid-hyperv.c
>  F:	drivers/hv/
>  F:	drivers/input/serio/hyperv-keyboard.c
> diff --git a/arch/arm64/Makefile b/arch/arm64/Makefile
> index 6cb9fc7e9382..ad9ec0579553 100644
> --- a/arch/arm64/Makefile
> +++ b/arch/arm64/Makefile
> @@ -106,6 +106,7 @@ core-y		+= arch/arm64/kernel/ arch/arm64/mm/
>  core-$(CONFIG_NET) += arch/arm64/net/
>  core-$(CONFIG_KVM) += arch/arm64/kvm/
>  core-$(CONFIG_XEN) += arch/arm64/xen/
> +core-$(CONFIG_HYPERV) += arch/arm64/hyperv/
>  core-$(CONFIG_CRYPTO) += arch/arm64/crypto/
>  libs-y		:= arch/arm64/lib/ $(libs-y)
>  core-$(CONFIG_EFI_STUB) += $(objtree)/drivers/firmware/efi/libstub/lib.a
> diff --git a/arch/arm64/hyperv/Makefile b/arch/arm64/hyperv/Makefile
> new file mode 100644
> index 000000000000..988eda55330c
> --- /dev/null
> +++ b/arch/arm64/hyperv/Makefile
> @@ -0,0 +1,2 @@
> +# SPDX-License-Identifier: GPL-2.0
> +obj-y		:= hv_init.o hv_hvc.o mshyperv.o
> diff --git a/arch/arm64/hyperv/hv_hvc.S b/arch/arm64/hyperv/hv_hvc.S
> new file mode 100644
> index 000000000000..82636969b4f2
> --- /dev/null
> +++ b/arch/arm64/hyperv/hv_hvc.S
> @@ -0,0 +1,54 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +
> +/*
> + * Microsoft Hyper-V hypervisor invocation routines
> + *
> + * Copyright (C) 2018, Microsoft, Inc.
> + *
> + * Author : Michael Kelley <mikelley@microsoft.com>
> + *
> + * This program is free software; you can redistribute it and/or modify it
> + * under the terms of the GNU General Public License version 2 as published
> + * by the Free Software Foundation.
> + *
> + * This program is distributed in the hope that it will be useful, but
> + * WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE, GOOD TITLE or
> + * NON INFRINGEMENT.  See the GNU General Public License for more
> + * details.
> + */
> +
> +#include <linux/linkage.h>
> +
> +	.text
> +/*
> + * Do the HVC instruction.  For Hyper-V the argument is always 1.
> + * x0 contains the hypercall control value, while additional registers
> + * vary depending on the hypercall, and whether the hypercall arguments
> + * are in memory or in registers (a "fast" hypercall per the Hyper-V
> + * TLFS).  When the arguments are in memory x1 is the guest physical
> + * address of the input arguments, and x2 is the guest physical
> + * address of the output arguments.  When the arguments are in
> + * registers, the register values depends on the hypercall.  Note
> + * that this version cannot return any values in registers.
> + */
> +ENTRY(hv_do_hvc)
> +	hvc #1
> +	ret
> +ENDPROC(hv_do_hvc)
> +
> +/*
> + * This variant of HVC invocation is for hv_get_vpreg and
> + * hv_get_vpreg_128. The input parameters are passed in registers
> + * along with a pointer in x4 to where the output result should
> + * be stored. The output is returned in x15 and x16.  x18 is used as
> + * scratch space to avoid buildng a stack frame, as Hyper-V does
> + * not preserve registers x0-x17.
> + */
> +ENTRY(hv_do_hvc_fast_get)
> +	mov x18, x4
> +	hvc #1
> +	str x15,[x18]
> +	str x16,[x18,#8]
> +	ret
> +ENDPROC(hv_do_hvc_fast_get)

Why are you not following the ARM SMCCC? It would be /much/ better if you
could just follow the standard, which is already implemented by
include/linux/smccc.h.

Will

Re: [PATCH 2/4] arm64: hyperv: Add support for Hyper-V as a hypervisor

[Will Deacon]

On Thu, Nov 22, 2018 at 03:10:57AM +0000, kys@linuxonhyperv.com wrote:
> From: Michael Kelley <mikelley@microsoft.com>
> 
> Add ARM64-specific code to enable Hyper-V. This code includes:
> * Detecting Hyper-V and initializing the guest/Hyper-V interface
> * Setting up Hyper-V's synthetic clocks
> * Making hypercalls using the HVC instruction
> * Setting up VMbus and stimer0 interrupts
> * Setting up kexec and crash handlers
> This code is architecture dependent code and is mostly driven by
> architecture independent code in the VMbus driver in drivers/hv/hv.c
> and drivers/hv/vmbus_drv.c.
> 
> This code is built only when CONFIG_HYPERV is enabled.
> 
> Signed-off-by: Michael Kelley <mikelley@microsoft.com>
> Signed-off-by: K. Y. Srinivasan <kys@microsoft.com>
> ---
>  MAINTAINERS                  |   1 +
>  arch/arm64/Makefile          |   1 +
>  arch/arm64/hyperv/Makefile   |   2 +
>  arch/arm64/hyperv/hv_hvc.S   |  54 +++++
>  arch/arm64/hyperv/hv_init.c  | 441 +++++++++++++++++++++++++++++++++++
>  arch/arm64/hyperv/mshyperv.c | 178 ++++++++++++++
>  6 files changed, 677 insertions(+)
>  create mode 100644 arch/arm64/hyperv/Makefile
>  create mode 100644 arch/arm64/hyperv/hv_hvc.S
>  create mode 100644 arch/arm64/hyperv/hv_init.c
>  create mode 100644 arch/arm64/hyperv/mshyperv.c
> 
> diff --git a/MAINTAINERS b/MAINTAINERS
> index 72f19cef4c48..326eeb32a0cd 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -6837,6 +6837,7 @@ F:	arch/x86/kernel/cpu/mshyperv.c
>  F:	arch/x86/hyperv
>  F:	arch/arm64/include/asm/hyperv-tlfs.h
>  F:	arch/arm64/include/asm/mshyperv.h
> +F:	arch/arm64/hyperv
>  F:	drivers/hid/hid-hyperv.c
>  F:	drivers/hv/
>  F:	drivers/input/serio/hyperv-keyboard.c
> diff --git a/arch/arm64/Makefile b/arch/arm64/Makefile
> index 6cb9fc7e9382..ad9ec0579553 100644
> --- a/arch/arm64/Makefile
> +++ b/arch/arm64/Makefile
> @@ -106,6 +106,7 @@ core-y		+= arch/arm64/kernel/ arch/arm64/mm/
>  core-$(CONFIG_NET) += arch/arm64/net/
>  core-$(CONFIG_KVM) += arch/arm64/kvm/
>  core-$(CONFIG_XEN) += arch/arm64/xen/
> +core-$(CONFIG_HYPERV) += arch/arm64/hyperv/
>  core-$(CONFIG_CRYPTO) += arch/arm64/crypto/
>  libs-y		:= arch/arm64/lib/ $(libs-y)
>  core-$(CONFIG_EFI_STUB) += $(objtree)/drivers/firmware/efi/libstub/lib.a
> diff --git a/arch/arm64/hyperv/Makefile b/arch/arm64/hyperv/Makefile
> new file mode 100644
> index 000000000000..988eda55330c
> --- /dev/null
> +++ b/arch/arm64/hyperv/Makefile
> @@ -0,0 +1,2 @@
> +# SPDX-License-Identifier: GPL-2.0
> +obj-y		:= hv_init.o hv_hvc.o mshyperv.o
> diff --git a/arch/arm64/hyperv/hv_hvc.S b/arch/arm64/hyperv/hv_hvc.S
> new file mode 100644
> index 000000000000..82636969b4f2
> --- /dev/null
> +++ b/arch/arm64/hyperv/hv_hvc.S
> @@ -0,0 +1,54 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +
> +/*
> + * Microsoft Hyper-V hypervisor invocation routines
> + *
> + * Copyright (C) 2018, Microsoft, Inc.
> + *
> + * Author : Michael Kelley <mikelley@microsoft.com>
> + *
> + * This program is free software; you can redistribute it and/or modify it
> + * under the terms of the GNU General Public License version 2 as published
> + * by the Free Software Foundation.
> + *
> + * This program is distributed in the hope that it will be useful, but
> + * WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE, GOOD TITLE or
> + * NON INFRINGEMENT.  See the GNU General Public License for more
> + * details.
> + */
> +
> +#include <linux/linkage.h>
> +
> +	.text
> +/*
> + * Do the HVC instruction.  For Hyper-V the argument is always 1.
> + * x0 contains the hypercall control value, while additional registers
> + * vary depending on the hypercall, and whether the hypercall arguments
> + * are in memory or in registers (a "fast" hypercall per the Hyper-V
> + * TLFS).  When the arguments are in memory x1 is the guest physical
> + * address of the input arguments, and x2 is the guest physical
> + * address of the output arguments.  When the arguments are in
> + * registers, the register values depends on the hypercall.  Note
> + * that this version cannot return any values in registers.
> + */
> +ENTRY(hv_do_hvc)
> +	hvc #1
> +	ret
> +ENDPROC(hv_do_hvc)
> +
> +/*
> + * This variant of HVC invocation is for hv_get_vpreg and
> + * hv_get_vpreg_128. The input parameters are passed in registers
> + * along with a pointer in x4 to where the output result should
> + * be stored. The output is returned in x15 and x16.  x18 is used as
> + * scratch space to avoid buildng a stack frame, as Hyper-V does
> + * not preserve registers x0-x17.
> + */
> +ENTRY(hv_do_hvc_fast_get)
> +	mov x18, x4
> +	hvc #1
> +	str x15,[x18]
> +	str x16,[x18,#8]
> +	ret
> +ENDPROC(hv_do_hvc_fast_get)

Why are you not following the ARM SMCCC? It would be /much/ better if you
could just follow the standard, which is already implemented by
include/linux/smccc.h.

Will

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 2/4] arm64: hyperv: Add support for Hyper-V as a hypervisor

[Marc Zyngier]

On 22/11/2018 03:10, kys@linuxonhyperv.com wrote:
> From: Michael Kelley <mikelley@microsoft.com>
> 
> Add ARM64-specific code to enable Hyper-V. This code includes:
> * Detecting Hyper-V and initializing the guest/Hyper-V interface
> * Setting up Hyper-V's synthetic clocks
> * Making hypercalls using the HVC instruction
> * Setting up VMbus and stimer0 interrupts
> * Setting up kexec and crash handlers

This commit message is a clear indication that this should be split in
at least 5 different patches.

> This code is architecture dependent code and is mostly driven by
> architecture independent code in the VMbus driver in drivers/hv/hv.c
> and drivers/hv/vmbus_drv.c.
> 
> This code is built only when CONFIG_HYPERV is enabled.
> 
> Signed-off-by: Michael Kelley <mikelley@microsoft.com>
> Signed-off-by: K. Y. Srinivasan <kys@microsoft.com>
> ---
>  MAINTAINERS                  |   1 +
>  arch/arm64/Makefile          |   1 +
>  arch/arm64/hyperv/Makefile   |   2 +
>  arch/arm64/hyperv/hv_hvc.S   |  54 +++++
>  arch/arm64/hyperv/hv_init.c  | 441 +++++++++++++++++++++++++++++++++++
>  arch/arm64/hyperv/mshyperv.c | 178 ++++++++++++++
>  6 files changed, 677 insertions(+)
>  create mode 100644 arch/arm64/hyperv/Makefile
>  create mode 100644 arch/arm64/hyperv/hv_hvc.S
>  create mode 100644 arch/arm64/hyperv/hv_init.c
>  create mode 100644 arch/arm64/hyperv/mshyperv.c
> 
> diff --git a/MAINTAINERS b/MAINTAINERS
> index 72f19cef4c48..326eeb32a0cd 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -6837,6 +6837,7 @@ F:	arch/x86/kernel/cpu/mshyperv.c
>  F:	arch/x86/hyperv
>  F:	arch/arm64/include/asm/hyperv-tlfs.h
>  F:	arch/arm64/include/asm/mshyperv.h
> +F:	arch/arm64/hyperv
>  F:	drivers/hid/hid-hyperv.c
>  F:	drivers/hv/
>  F:	drivers/input/serio/hyperv-keyboard.c
> diff --git a/arch/arm64/Makefile b/arch/arm64/Makefile
> index 6cb9fc7e9382..ad9ec0579553 100644
> --- a/arch/arm64/Makefile
> +++ b/arch/arm64/Makefile
> @@ -106,6 +106,7 @@ core-y		+= arch/arm64/kernel/ arch/arm64/mm/
>  core-$(CONFIG_NET) += arch/arm64/net/
>  core-$(CONFIG_KVM) += arch/arm64/kvm/
>  core-$(CONFIG_XEN) += arch/arm64/xen/
> +core-$(CONFIG_HYPERV) += arch/arm64/hyperv/
>  core-$(CONFIG_CRYPTO) += arch/arm64/crypto/
>  libs-y		:= arch/arm64/lib/ $(libs-y)
>  core-$(CONFIG_EFI_STUB) += $(objtree)/drivers/firmware/efi/libstub/lib.a
> diff --git a/arch/arm64/hyperv/Makefile b/arch/arm64/hyperv/Makefile
> new file mode 100644
> index 000000000000..988eda55330c
> --- /dev/null
> +++ b/arch/arm64/hyperv/Makefile
> @@ -0,0 +1,2 @@
> +# SPDX-License-Identifier: GPL-2.0
> +obj-y		:= hv_init.o hv_hvc.o mshyperv.o
> diff --git a/arch/arm64/hyperv/hv_hvc.S b/arch/arm64/hyperv/hv_hvc.S
> new file mode 100644
> index 000000000000..82636969b4f2
> --- /dev/null
> +++ b/arch/arm64/hyperv/hv_hvc.S
> @@ -0,0 +1,54 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +
> +/*
> + * Microsoft Hyper-V hypervisor invocation routines
> + *
> + * Copyright (C) 2018, Microsoft, Inc.
> + *
> + * Author : Michael Kelley <mikelley@microsoft.com>
> + *
> + * This program is free software; you can redistribute it and/or modify it
> + * under the terms of the GNU General Public License version 2 as published
> + * by the Free Software Foundation.
> + *
> + * This program is distributed in the hope that it will be useful, but
> + * WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE, GOOD TITLE or
> + * NON INFRINGEMENT.  See the GNU General Public License for more
> + * details.
> + */
> +
> +#include <linux/linkage.h>
> +
> +	.text
> +/*
> + * Do the HVC instruction.  For Hyper-V the argument is always 1.
> + * x0 contains the hypercall control value, while additional registers
> + * vary depending on the hypercall, and whether the hypercall arguments
> + * are in memory or in registers (a "fast" hypercall per the Hyper-V
> + * TLFS).  When the arguments are in memory x1 is the guest physical
> + * address of the input arguments, and x2 is the guest physical
> + * address of the output arguments.  When the arguments are in
> + * registers, the register values depends on the hypercall.  Note
> + * that this version cannot return any values in registers.
> + */
> +ENTRY(hv_do_hvc)
> +	hvc #1
> +	ret
> +ENDPROC(hv_do_hvc)
> +
> +/*
> + * This variant of HVC invocation is for hv_get_vpreg and
> + * hv_get_vpreg_128. The input parameters are passed in registers
> + * along with a pointer in x4 to where the output result should
> + * be stored. The output is returned in x15 and x16.  x18 is used as
> + * scratch space to avoid buildng a stack frame, as Hyper-V does
> + * not preserve registers x0-x17.
> + */
> +ENTRY(hv_do_hvc_fast_get)
> +	mov x18, x4
> +	hvc #1
> +	str x15,[x18]
> +	str x16,[x18,#8]
> +	ret
> +ENDPROC(hv_do_hvc_fast_get)

As Will said, this isn't a viable option. Please follow SMCCC 1.1.

> diff --git a/arch/arm64/hyperv/hv_init.c b/arch/arm64/hyperv/hv_init.c
> new file mode 100644
> index 000000000000..aa1a8c09d989
> --- /dev/null
> +++ b/arch/arm64/hyperv/hv_init.c
> @@ -0,0 +1,441 @@
> +// SPDX-License-Identifier: GPL-2.0
> +
> +/*
> + * Initialization of the interface with Microsoft's Hyper-V hypervisor,
> + * and various low level utility routines for interacting with Hyper-V.
> + *
> + * Copyright (C) 2018, Microsoft, Inc.
> + *
> + * Author : Michael Kelley <mikelley@microsoft.com>
> + *
> + * This program is free software; you can redistribute it and/or modify it
> + * under the terms of the GNU General Public License version 2 as published
> + * by the Free Software Foundation.
> + *
> + * This program is distributed in the hope that it will be useful, but
> + * WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE, GOOD TITLE or
> + * NON INFRINGEMENT.  See the GNU General Public License for more
> + * details.
> + */
> +
> +
> +#include <linux/types.h>
> +#include <linux/version.h>
> +#include <linux/export.h>
> +#include <linux/vmalloc.h>
> +#include <linux/mm.h>
> +#include <linux/clocksource.h>
> +#include <linux/sched_clock.h>
> +#include <linux/acpi.h>
> +#include <linux/module.h>
> +#include <linux/hyperv.h>
> +#include <linux/slab.h>
> +#include <linux/cpuhotplug.h>
> +#include <linux/psci.h>
> +#include <asm-generic/bug.h>
> +#include <asm/hypervisor.h>
> +#include <asm/hyperv-tlfs.h>
> +#include <asm/mshyperv.h>
> +#include <asm/sysreg.h>
> +#include <clocksource/arm_arch_timer.h>
> +
> +static bool	hyperv_initialized;
> +struct		ms_hyperv_info ms_hyperv;
> +EXPORT_SYMBOL_GPL(ms_hyperv);

Who are the users of this structure? Should they go via accessors instead?

> +
> +static struct ms_hyperv_tsc_page *tsc_pg;
> +
> +struct ms_hyperv_tsc_page *hv_get_tsc_page(void)
> +{
> +	return tsc_pg;
> +}
> +EXPORT_SYMBOL_GPL(hv_get_tsc_page);
> +
> +static u64 read_hv_sched_clock_tsc(void)
> +{
> +	u64 current_tick = hv_read_tsc_page(tsc_pg);
> +
> +	if (current_tick == U64_MAX)
> +		current_tick = hv_get_vpreg(HV_REGISTER_TIME_REFCOUNT);
> +
> +	return current_tick;
> +}
> +
> +static u64 read_hv_clock_tsc(struct clocksource *arg)
> +{
> +	u64 current_tick = hv_read_tsc_page(tsc_pg);
> +
> +	if (current_tick == U64_MAX)
> +		current_tick = hv_get_vpreg(HV_REGISTER_TIME_REFCOUNT);
> +
> +	return current_tick;
> +}

Can't you define one function in terms of the other?

> +
> +static struct clocksource hyperv_cs_tsc = {
> +		.name		= "hyperv_clocksource_tsc_page",
> +		.rating		= 400,
> +		.read		= read_hv_clock_tsc,
> +		.mask		= CLOCKSOURCE_MASK(64),
> +		.flags		= CLOCK_SOURCE_IS_CONTINUOUS,
> +};
> +
> +static u64 read_hv_sched_clock_msr(void)
> +{
> +	return hv_get_vpreg(HV_REGISTER_TIME_REFCOUNT);
> +}
> +
> +static u64 read_hv_clock_msr(struct clocksource *arg)
> +{
> +	return hv_get_vpreg(HV_REGISTER_TIME_REFCOUNT);
> +}
> +
> +static struct clocksource hyperv_cs_msr = {
> +	.name		= "hyperv_clocksource_msr",
> +	.rating		= 400,
> +	.read		= read_hv_clock_msr,
> +	.mask		= CLOCKSOURCE_MASK(64),
> +	.flags		= CLOCK_SOURCE_IS_CONTINUOUS,
> +};
> +
> +struct clocksource *hyperv_cs;
> +EXPORT_SYMBOL_GPL(hyperv_cs);

Why? Who needs to poke this?

> +
> +u32 *hv_vp_index;
> +EXPORT_SYMBOL_GPL(hv_vp_index);
> +
> +u32 hv_max_vp_index;

Same thing.

> +
> +static int hv_cpu_init(unsigned int cpu)
> +{
> +	u64 msr_vp_index;
> +
> +	hv_get_vp_index(msr_vp_index);
> +
> +	hv_vp_index[smp_processor_id()] = msr_vp_index;
> +
> +	if (msr_vp_index > hv_max_vp_index)
> +		hv_max_vp_index = msr_vp_index;
> +
> +	return 0;
> +}

Is that some new way to describe a CPU topology? If so, why isn't that
exposed via the ACPI tables that the kernel already parses?

> +
> +/*
> + * This function is invoked via the ACPI clocksource probe mechanism. We
> + * don't actually use any values from the ACPI GTDT table, but we set up

This doesn't feel like a good idea at all. Piggy-backing on an existing
mechanism and use it for something completely different is not exactly
future-proof.

Also, if this is supposed to be a clocksource, why isn't that a
clocksource driver on its own right?

> + * the Hyper-V synthetic clocksource and do other initialization for
> + * interacting with Hyper-V the first time.  Using early_initcall to invoke
> + * this function is too late because interrupts are already enabled at that
> + * point, and sched_clock_register must run before interrupts are enabled.
> + *
> + * 1. Setup the guest ID.
> + * 2. Get features and hints info from Hyper-V
> + * 3. Setup per-cpu VP indices.
> + * 4. Register Hyper-V specific clocksource.
> + * 5. Register the scheduler clock.
> + */
> +
> +static int __init hyperv_init(struct acpi_table_header *table)
> +{
> +	struct hv_get_vp_register_output result;
> +	u32	a, b, c, d;
> +	u64	guest_id;
> +	int	i;
> +
> +	/*
> +	 * If we're in a VM on Hyper-V, the ACPI hypervisor_id field will
> +	 * have the string "MsHyperV".
> +	 */
> +	if (strncmp((char *)&acpi_gbl_FADT.hypervisor_id, "MsHyperV", 8))
> +		return 1;
> +
> +	/* Setup the guest ID */
> +	guest_id = generate_guest_id(0, LINUX_VERSION_CODE, 0);
> +	hv_set_vpreg(HV_REGISTER_GUEST_OSID, guest_id);
> +
> +	/* Get the features and hints from Hyper-V */
> +	hv_get_vpreg_128(HV_REGISTER_PRIVILEGES_AND_FEATURES, &result);
> +	ms_hyperv.features = lower_32_bits(result.registervaluelow);
> +	ms_hyperv.misc_features = upper_32_bits(result.registervaluehigh);
> +
> +	hv_get_vpreg_128(HV_REGISTER_FEATURES, &result);
> +	ms_hyperv.hints = lower_32_bits(result.registervaluelow);
> +
> +	pr_info("Hyper-V: Features 0x%x, hints 0x%x\n",
> +		ms_hyperv.features, ms_hyperv.hints);
> +
> +	/*
> +	 * Direct mode is the only option for STIMERs provided Hyper-V
> +	 * on ARM64, so Hyper-V doesn't actually set the flag.  But add the
> +	 * flag so the architecture independent code in drivers/hv/hv.c
> +	 * will correctly use that mode.
> +	 */
> +	ms_hyperv.misc_features |= HV_STIMER_DIRECT_MODE_AVAILABLE;
> +
> +	/*
> +	 * Hyper-V on ARM64 doesn't support AutoEOI.  Add the hint
> +	 * that tells architecture independent code not to use this
> +	 * feature.
> +	 */
> +	ms_hyperv.hints |= HV_DEPRECATING_AEOI_RECOMMENDED;
> +
> +	/* Get information about the Hyper-V host version */
> +	hv_get_vpreg_128(HV_REGISTER_HYPERVISOR_VERSION, &result);
> +	a = lower_32_bits(result.registervaluelow);
> +	b = upper_32_bits(result.registervaluelow);
> +	c = lower_32_bits(result.registervaluehigh);
> +	d = upper_32_bits(result.registervaluehigh);
> +	pr_info("Hyper-V: Host Build %d.%d.%d.%d-%d-%d\n",
> +		b >> 16, b & 0xFFFF, a, d & 0xFFFFFF, c, d >> 24);
> +
> +	/* Allocate percpu VP index */
> +	hv_vp_index = kmalloc_array(num_possible_cpus(), sizeof(*hv_vp_index),
> +				    GFP_KERNEL);

Why isn't this a percpu variable?

> +	if (!hv_vp_index)
> +		return 1;
> +
> +	for (i = 0; i < num_possible_cpus(); i++)
> +		hv_vp_index[i] = VP_INVAL;
> +
> +	if (cpuhp_setup_state(CPUHP_AP_ONLINE_DYN, "arm64/hyperv_init:online",
> +			      hv_cpu_init, NULL) < 0)
> +		goto free_vp_index;
> +
> +	/*
> +	 * Try to set up what Hyper-V calls the "TSC reference page", which
> +	 * uses the ARM Generic Timer virtual counter with some scaling
> +	 * information to provide a fast and stable guest VM clocksource.
> +	 * If the TSC reference page can't be set up, fall back to reading
> +	 * the guest clock provided by Hyper-V's synthetic reference time
> +	 * register.
> +	 */
> +	if (ms_hyperv.features & HV_MSR_REFERENCE_TSC_AVAILABLE) {
> +
> +		u64		tsc_msr;
> +		phys_addr_t	phys_addr;
> +
> +		tsc_pg = __vmalloc(HV_HYP_PAGE_SIZE, GFP_KERNEL, PAGE_KERNEL);

And why not vmalloc?

> +		if (tsc_pg) {
> +			phys_addr = page_to_phys(vmalloc_to_page(tsc_pg));
> +			tsc_msr = hv_get_vpreg(HV_REGISTER_REFERENCE_TSC);
> +			tsc_msr &= GENMASK_ULL(11, 0);

Magic.

> +			tsc_msr = tsc_msr | 0x1 | (u64)phys_addr;
> +			hv_set_vpreg(HV_REGISTER_REFERENCE_TSC, tsc_msr);
> +			hyperv_cs = &hyperv_cs_tsc;
> +			sched_clock_register(read_hv_sched_clock_tsc,
> +						64, HV_CLOCK_HZ);
> +		}
> +	}
> +
> +	if (!hyperv_cs &&
> +	    (ms_hyperv.features & HV_MSR_TIME_REF_COUNT_AVAILABLE)) {
> +		hyperv_cs = &hyperv_cs_msr;
> +		sched_clock_register(read_hv_sched_clock_msr,
> +						64, HV_CLOCK_HZ);
> +	}
> +
> +	if (hyperv_cs) {
> +		hyperv_cs->archdata.vdso_direct = false;
> +		clocksource_register_hz(hyperv_cs, HV_CLOCK_HZ);
> +	}
> +
> +	hyperv_initialized = true;
> +	return 0;
> +
> +free_vp_index:
> +	kfree(hv_vp_index);
> +	hv_vp_index = NULL;
> +	return 1;

????

> +}
> +TIMER_ACPI_DECLARE(hyperv, ACPI_SIG_GTDT, hyperv_init);
> +
> +/*
> + * This routine is called before kexec/kdump, it does the required cleanup.
> + */
> +void hyperv_cleanup(void)
> +{
> +	/* Reset our OS id */
> +	hv_set_vpreg(HV_REGISTER_GUEST_OSID, 0);
> +
> +}
> +EXPORT_SYMBOL_GPL(hyperv_cleanup);
> +
> +/*
> + * hv_do_hypercall- Invoke the specified hypercall
> + */
> +u64 hv_do_hypercall(u64 control, void *input, void *output)
> +{
> +	u64 input_address;
> +	u64 output_address;
> +
> +	input_address = input ? virt_to_phys(input) : 0;
> +	output_address = output ? virt_to_phys(output) : 0;
> +	return hv_do_hvc(control, input_address, output_address);
> +}
> +EXPORT_SYMBOL_GPL(hv_do_hypercall);
> +
> +/*
> + * hv_do_fast_hypercall8 -- Invoke the specified hypercall
> + * with arguments in registers instead of physical memory.
> + * Avoids the overhead of virt_to_phys for simple hypercalls.
> + */
> +
> +u64 hv_do_fast_hypercall8(u16 code, u64 input)
> +{
> +	u64 control;
> +
> +	control = (u64)code | HV_HYPERCALL_FAST_BIT;
> +	return hv_do_hvc(control, input);
> +}
> +EXPORT_SYMBOL_GPL(hv_do_fast_hypercall8);
> +
> +
> +/*
> + * Set a single VP register to a 64-bit value.
> + */
> +void hv_set_vpreg(u32 msr, u64 value)
> +{
> +	union hv_hypercall_status status;
> +
> +	status.as_uint64 = hv_do_hvc(
> +		HVCALL_SET_VP_REGISTERS | HV_HYPERCALL_FAST_BIT |
> +			HV_HYPERCALL_REP_COUNT_1,
> +		HV_PARTITION_ID_SELF,
> +		HV_VP_INDEX_SELF,
> +		msr,
> +		0,
> +		value,
> +		0);
> +
> +	/*
> +	 * Something is fundamentally broken in the hypervisor if
> +	 * setting a VP register fails. There's really no way to
> +	 * continue as a guest VM, so panic.
> +	 */
> +	BUG_ON(status.status != HV_STATUS_SUCCESS);
> +}
> +EXPORT_SYMBOL_GPL(hv_set_vpreg);
> +
> +
> +/*
> + * Get the value of a single VP register, and only the low order 64 bits.
> + */
> +u64 hv_get_vpreg(u32 msr)
> +{
> +	union hv_hypercall_status status;
> +	struct hv_get_vp_register_output output;
> +
> +	status.as_uint64 = hv_do_hvc_fast_get(
> +		HVCALL_GET_VP_REGISTERS | HV_HYPERCALL_FAST_BIT |
> +			HV_HYPERCALL_REP_COUNT_1,
> +		HV_PARTITION_ID_SELF,
> +		HV_VP_INDEX_SELF,
> +		msr,
> +		&output);
> +
> +	/*
> +	 * Something is fundamentally broken in the hypervisor if
> +	 * getting a VP register fails. There's really no way to
> +	 * continue as a guest VM, so panic.
> +	 */
> +	BUG_ON(status.status != HV_STATUS_SUCCESS);
> +
> +	return output.registervaluelow;
> +}
> +EXPORT_SYMBOL_GPL(hv_get_vpreg);
> +
> +/*
> + * Get the value of a single VP register that is 128 bits in size.  This is a
> + * separate call in order to avoid complicating the calling sequence for
> + * the much more frequently used 64-bit version.
> + */
> +void hv_get_vpreg_128(u32 msr, struct hv_get_vp_register_output *result)
> +{
> +	union hv_hypercall_status status;
> +
> +	status.as_uint64 = hv_do_hvc_fast_get(
> +		HVCALL_GET_VP_REGISTERS | HV_HYPERCALL_FAST_BIT |
> +			HV_HYPERCALL_REP_COUNT_1,
> +		HV_PARTITION_ID_SELF,
> +		HV_VP_INDEX_SELF,
> +		msr,
> +		result);
> +
> +	/*
> +	 * Something is fundamentally broken in the hypervisor if
> +	 * getting a VP register fails. There's really no way to
> +	 * continue as a guest VM, so panic.
> +	 */
> +	BUG_ON(status.status != HV_STATUS_SUCCESS);
> +
> +	return;
> +
> +}
> +EXPORT_SYMBOL_GPL(hv_get_vpreg_128);
> +
> +void hyperv_report_panic(struct pt_regs *regs, long err)
> +{
> +	static bool panic_reported;
> +	u64 guest_id;
> +
> +	/*
> +	 * We prefer to report panic on 'die' chain as we have proper
> +	 * registers to report, but if we miss it (e.g. on BUG()) we need
> +	 * to report it on 'panic'.
> +	 */
> +	if (panic_reported)
> +		return;
> +	panic_reported = true;
> +
> +	guest_id = hv_get_vpreg(HV_REGISTER_GUEST_OSID);
> +
> +	/*
> +	 * Hyper-V provides the ability to store only 5 values.
> +	 * Pick the passed in error value, the guest_id, and the PC.
> +	 * The first two general registers are added arbitrarily.
> +	 */
> +	hv_set_vpreg(HV_REGISTER_CRASH_P0, err);
> +	hv_set_vpreg(HV_REGISTER_CRASH_P1, guest_id);
> +	hv_set_vpreg(HV_REGISTER_CRASH_P2, regs->pc);
> +	hv_set_vpreg(HV_REGISTER_CRASH_P3, regs->regs[0]);
> +	hv_set_vpreg(HV_REGISTER_CRASH_P4, regs->regs[1]);
> +
> +	/*
> +	 * Let Hyper-V know there is crash data available
> +	 */
> +	hv_set_vpreg(HV_REGISTER_CRASH_CTL, HV_CRASH_CTL_CRASH_NOTIFY);
> +}
> +EXPORT_SYMBOL_GPL(hyperv_report_panic);
> +
> +/*
> + * hyperv_report_panic_msg - report panic message to Hyper-V
> + * @pa: physical address of the panic page containing the message
> + * @size: size of the message in the page
> + */
> +void hyperv_report_panic_msg(phys_addr_t pa, size_t size)
> +{
> +	/*
> +	 * P3 to contain the physical address of the panic page & P4 to
> +	 * contain the size of the panic data in that page. Rest of the
> +	 * registers are no-op when the NOTIFY_MSG flag is set.
> +	 */
> +	hv_set_vpreg(HV_REGISTER_CRASH_P0, 0);
> +	hv_set_vpreg(HV_REGISTER_CRASH_P1, 0);
> +	hv_set_vpreg(HV_REGISTER_CRASH_P2, 0);
> +	hv_set_vpreg(HV_REGISTER_CRASH_P3, pa);
> +	hv_set_vpreg(HV_REGISTER_CRASH_P4, size);
> +
> +	/*
> +	 * Let Hyper-V know there is crash data available along with
> +	 * the panic message.
> +	 */
> +	hv_set_vpreg(HV_REGISTER_CRASH_CTL,
> +	       (HV_CRASH_CTL_CRASH_NOTIFY | HV_CRASH_CTL_CRASH_NOTIFY_MSG));
> +}
> +EXPORT_SYMBOL_GPL(hyperv_report_panic_msg);
> +
> +bool hv_is_hyperv_initialized(void)
> +{
> +	return hyperv_initialized;
> +}
> +EXPORT_SYMBOL_GPL(hv_is_hyperv_initialized);
> diff --git a/arch/arm64/hyperv/mshyperv.c b/arch/arm64/hyperv/mshyperv.c
> new file mode 100644
> index 000000000000..3ef055599412
> --- /dev/null
> +++ b/arch/arm64/hyperv/mshyperv.c
> @@ -0,0 +1,178 @@
> +// SPDX-License-Identifier: GPL-2.0
> +
> +/*
> + * Core routines for interacting with Microsoft's Hyper-V hypervisor,
> + * including setting up VMbus and STIMER interrupts, and handling
> + * crashes and kexecs. These interactions are through a set of
> + * static "handler" variables set by the architecture independent
> + * VMbus and STIMER drivers.  This design is used to meet x86/x64
> + * requirements for avoiding direct linkages and allowing the VMbus
> + * and STIMER drivers to be unloaded and reloaded.
> + *
> + * Copyright (C) 2018, Microsoft, Inc.
> + *
> + * Author : Michael Kelley <mikelley@microsoft.com>
> + *
> + * This program is free software; you can redistribute it and/or modify it
> + * under the terms of the GNU General Public License version 2 as published
> + * by the Free Software Foundation.
> + *
> + * This program is distributed in the hope that it will be useful, but
> + * WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE, GOOD TITLE or
> + * NON INFRINGEMENT.  See the GNU General Public License for more
> + * details.
> + */
> +
> +#include <linux/types.h>
> +#include <linux/export.h>
> +#include <linux/interrupt.h>
> +#include <linux/kexec.h>
> +#include <linux/acpi.h>
> +#include <linux/ptrace.h>
> +#include <asm/hyperv-tlfs.h>
> +#include <asm/mshyperv.h>
> +
> +static void (*vmbus_handler)(void);
> +static void (*hv_stimer0_handler)(void);
> +static void (*hv_kexec_handler)(void);
> +static void (*hv_crash_handler)(struct pt_regs *regs);
> +
> +static int vmbus_irq;
> +static long __percpu *vmbus_evt;
> +static long __percpu *stimer0_evt;
> +
> +irqreturn_t hyperv_vector_handler(int irq, void *dev_id)
> +{
> +	if (vmbus_handler)
> +		vmbus_handler();

In which circumstances can this be NULL?

> +	return IRQ_HANDLED;
> +}
> +
> +/* Must be done just once */
> +void hv_setup_vmbus_irq(void (*handler)(void))
> +{
> +	int result;
> +
> +	vmbus_handler = handler;

If thismust only be done once, maybe you could check that it hasn't been
done already?

> +	vmbus_irq = acpi_register_gsi(NULL, HYPERVISOR_CALLBACK_VECTOR,
> +				 ACPI_LEVEL_SENSITIVE, ACPI_ACTIVE_HIGH);
> +	if (vmbus_irq <= 0) {
> +		pr_err("Can't register Hyper-V VMBus GSI. Error %d",
> +			vmbus_irq);
> +		vmbus_irq = 0;
> +		return;
> +	}
> +	vmbus_evt = alloc_percpu(long);
> +	result = request_percpu_irq(vmbus_irq, hyperv_vector_handler,
> +			"Hyper-V VMbus", vmbus_evt);

If this is a per-cpu interrupt, why isn't it signalled as a PPI, in an
architecture compliant way?

> +	if (result) {
> +		pr_err("Can't request Hyper-V VMBus IRQ %d. Error %d",
> +			vmbus_irq, result);
> +		free_percpu(vmbus_evt);
> +		acpi_unregister_gsi(vmbus_irq);
> +		vmbus_irq = 0;
> +	}
> +}
> +EXPORT_SYMBOL_GPL(hv_setup_vmbus_irq);
> +
> +/* Must be done just once */
> +void hv_remove_vmbus_irq(void)
> +{
> +	if (vmbus_irq) {
> +		free_percpu_irq(vmbus_irq, vmbus_evt);
> +		free_percpu(vmbus_evt);
> +		acpi_unregister_gsi(vmbus_irq);
> +	}
> +}
> +EXPORT_SYMBOL_GPL(hv_remove_vmbus_irq);
> +
> +/* Must be done by each CPU */
> +void hv_enable_vmbus_irq(void)
> +{
> +	enable_percpu_irq(vmbus_irq, 0);
> +}
> +EXPORT_SYMBOL_GPL(hv_enable_vmbus_irq);
> +
> +/* Must be done by each CPU */
> +void hv_disable_vmbus_irq(void)
> +{
> +	disable_percpu_irq(vmbus_irq);
> +}
> +EXPORT_SYMBOL_GPL(hv_disable_vmbus_irq);
> +
> +/* Routines to do per-architecture handling of STIMER0 when in Direct Mode */
> +
> +static irqreturn_t hv_stimer0_vector_handler(int irq, void *dev_id)
> +{
> +	if (hv_stimer0_handler)
> +		hv_stimer0_handler();
> +	return IRQ_HANDLED;
> +}
> +
> +int hv_setup_stimer0_irq(int *irq, int *vector, void (*handler)(void))
> +{
> +	int localirq;
> +	int result;
> +
> +	localirq = acpi_register_gsi(NULL, HV_STIMER0_IRQNR,
> +			ACPI_LEVEL_SENSITIVE, ACPI_ACTIVE_HIGH);
> +	if (localirq <= 0) {
> +		pr_err("Can't register Hyper-V stimer0 GSI. Error %d",
> +			localirq);
> +		*irq = 0;
> +		return -1;
> +	}
> +	stimer0_evt = alloc_percpu(long);
> +	result = request_percpu_irq(localirq, hv_stimer0_vector_handler,
> +					 "Hyper-V stimer0", stimer0_evt);
> +	if (result) {
> +		pr_err("Can't request Hyper-V stimer0 IRQ %d. Error %d",
> +			localirq, result);
> +		free_percpu(stimer0_evt);
> +		acpi_unregister_gsi(localirq);
> +		*irq = 0;
> +		return -1;
> +	}
> +
> +	hv_stimer0_handler = handler;
> +	*vector = HV_STIMER0_IRQNR;
> +	*irq = localirq;
> +	return 0;
> +}
> +EXPORT_SYMBOL_GPL(hv_setup_stimer0_irq);
> +
> +void hv_remove_stimer0_irq(int irq)
> +{
> +	hv_stimer0_handler = NULL;
> +	if (irq) {
> +		free_percpu_irq(irq, stimer0_evt);
> +		free_percpu(stimer0_evt);
> +		acpi_unregister_gsi(irq);
> +	}
> +}
> +EXPORT_SYMBOL_GPL(hv_remove_stimer0_irq);
> +
> +void hv_setup_kexec_handler(void (*handler)(void))
> +{
> +	hv_kexec_handler = handler;
> +}
> +EXPORT_SYMBOL_GPL(hv_setup_kexec_handler);
> +
> +void hv_remove_kexec_handler(void)
> +{
> +	hv_kexec_handler = NULL;
> +}
> +EXPORT_SYMBOL_GPL(hv_remove_kexec_handler);
> +
> +void hv_setup_crash_handler(void (*handler)(struct pt_regs *regs))
> +{
> +	hv_crash_handler = handler;
> +}
> +EXPORT_SYMBOL_GPL(hv_setup_crash_handler);
> +
> +void hv_remove_crash_handler(void)
> +{
> +	hv_crash_handler = NULL;
> +}
> +EXPORT_SYMBOL_GPL(hv_remove_crash_handler);
> 

Overall, this patch needs a massive split up, clean up, and a good dose
of documentation so that we can understand what you are trying to achieve.

Thanks,

	M.
-- 
Jazz is not dead. It just smells funny...

Re: [PATCH 2/4] arm64: hyperv: Add support for Hyper-V as a hypervisor

[Marc Zyngier]

On 22/11/2018 03:10, kys@linuxonhyperv.com wrote:
> From: Michael Kelley <mikelley@microsoft.com>
> 
> Add ARM64-specific code to enable Hyper-V. This code includes:
> * Detecting Hyper-V and initializing the guest/Hyper-V interface
> * Setting up Hyper-V's synthetic clocks
> * Making hypercalls using the HVC instruction
> * Setting up VMbus and stimer0 interrupts
> * Setting up kexec and crash handlers

This commit message is a clear indication that this should be split in
at least 5 different patches.

> This code is architecture dependent code and is mostly driven by
> architecture independent code in the VMbus driver in drivers/hv/hv.c
> and drivers/hv/vmbus_drv.c.
> 
> This code is built only when CONFIG_HYPERV is enabled.
> 
> Signed-off-by: Michael Kelley <mikelley@microsoft.com>
> Signed-off-by: K. Y. Srinivasan <kys@microsoft.com>
> ---
>  MAINTAINERS                  |   1 +
>  arch/arm64/Makefile          |   1 +
>  arch/arm64/hyperv/Makefile   |   2 +
>  arch/arm64/hyperv/hv_hvc.S   |  54 +++++
>  arch/arm64/hyperv/hv_init.c  | 441 +++++++++++++++++++++++++++++++++++
>  arch/arm64/hyperv/mshyperv.c | 178 ++++++++++++++
>  6 files changed, 677 insertions(+)
>  create mode 100644 arch/arm64/hyperv/Makefile
>  create mode 100644 arch/arm64/hyperv/hv_hvc.S
>  create mode 100644 arch/arm64/hyperv/hv_init.c
>  create mode 100644 arch/arm64/hyperv/mshyperv.c
> 
> diff --git a/MAINTAINERS b/MAINTAINERS
> index 72f19cef4c48..326eeb32a0cd 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -6837,6 +6837,7 @@ F:	arch/x86/kernel/cpu/mshyperv.c
>  F:	arch/x86/hyperv
>  F:	arch/arm64/include/asm/hyperv-tlfs.h
>  F:	arch/arm64/include/asm/mshyperv.h
> +F:	arch/arm64/hyperv
>  F:	drivers/hid/hid-hyperv.c
>  F:	drivers/hv/
>  F:	drivers/input/serio/hyperv-keyboard.c
> diff --git a/arch/arm64/Makefile b/arch/arm64/Makefile
> index 6cb9fc7e9382..ad9ec0579553 100644
> --- a/arch/arm64/Makefile
> +++ b/arch/arm64/Makefile
> @@ -106,6 +106,7 @@ core-y		+= arch/arm64/kernel/ arch/arm64/mm/
>  core-$(CONFIG_NET) += arch/arm64/net/
>  core-$(CONFIG_KVM) += arch/arm64/kvm/
>  core-$(CONFIG_XEN) += arch/arm64/xen/
> +core-$(CONFIG_HYPERV) += arch/arm64/hyperv/
>  core-$(CONFIG_CRYPTO) += arch/arm64/crypto/
>  libs-y		:= arch/arm64/lib/ $(libs-y)
>  core-$(CONFIG_EFI_STUB) += $(objtree)/drivers/firmware/efi/libstub/lib.a
> diff --git a/arch/arm64/hyperv/Makefile b/arch/arm64/hyperv/Makefile
> new file mode 100644
> index 000000000000..988eda55330c
> --- /dev/null
> +++ b/arch/arm64/hyperv/Makefile
> @@ -0,0 +1,2 @@
> +# SPDX-License-Identifier: GPL-2.0
> +obj-y		:= hv_init.o hv_hvc.o mshyperv.o
> diff --git a/arch/arm64/hyperv/hv_hvc.S b/arch/arm64/hyperv/hv_hvc.S
> new file mode 100644
> index 000000000000..82636969b4f2
> --- /dev/null
> +++ b/arch/arm64/hyperv/hv_hvc.S
> @@ -0,0 +1,54 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +
> +/*
> + * Microsoft Hyper-V hypervisor invocation routines
> + *
> + * Copyright (C) 2018, Microsoft, Inc.
> + *
> + * Author : Michael Kelley <mikelley@microsoft.com>
> + *
> + * This program is free software; you can redistribute it and/or modify it
> + * under the terms of the GNU General Public License version 2 as published
> + * by the Free Software Foundation.
> + *
> + * This program is distributed in the hope that it will be useful, but
> + * WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE, GOOD TITLE or
> + * NON INFRINGEMENT.  See the GNU General Public License for more
> + * details.
> + */
> +
> +#include <linux/linkage.h>
> +
> +	.text
> +/*
> + * Do the HVC instruction.  For Hyper-V the argument is always 1.
> + * x0 contains the hypercall control value, while additional registers
> + * vary depending on the hypercall, and whether the hypercall arguments
> + * are in memory or in registers (a "fast" hypercall per the Hyper-V
> + * TLFS).  When the arguments are in memory x1 is the guest physical
> + * address of the input arguments, and x2 is the guest physical
> + * address of the output arguments.  When the arguments are in
> + * registers, the register values depends on the hypercall.  Note
> + * that this version cannot return any values in registers.
> + */
> +ENTRY(hv_do_hvc)
> +	hvc #1
> +	ret
> +ENDPROC(hv_do_hvc)
> +
> +/*
> + * This variant of HVC invocation is for hv_get_vpreg and
> + * hv_get_vpreg_128. The input parameters are passed in registers
> + * along with a pointer in x4 to where the output result should
> + * be stored. The output is returned in x15 and x16.  x18 is used as
> + * scratch space to avoid buildng a stack frame, as Hyper-V does
> + * not preserve registers x0-x17.
> + */
> +ENTRY(hv_do_hvc_fast_get)
> +	mov x18, x4
> +	hvc #1
> +	str x15,[x18]
> +	str x16,[x18,#8]
> +	ret
> +ENDPROC(hv_do_hvc_fast_get)

As Will said, this isn't a viable option. Please follow SMCCC 1.1.

> diff --git a/arch/arm64/hyperv/hv_init.c b/arch/arm64/hyperv/hv_init.c
> new file mode 100644
> index 000000000000..aa1a8c09d989
> --- /dev/null
> +++ b/arch/arm64/hyperv/hv_init.c
> @@ -0,0 +1,441 @@
> +// SPDX-License-Identifier: GPL-2.0
> +
> +/*
> + * Initialization of the interface with Microsoft's Hyper-V hypervisor,
> + * and various low level utility routines for interacting with Hyper-V.
> + *
> + * Copyright (C) 2018, Microsoft, Inc.
> + *
> + * Author : Michael Kelley <mikelley@microsoft.com>
> + *
> + * This program is free software; you can redistribute it and/or modify it
> + * under the terms of the GNU General Public License version 2 as published
> + * by the Free Software Foundation.
> + *
> + * This program is distributed in the hope that it will be useful, but
> + * WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE, GOOD TITLE or
> + * NON INFRINGEMENT.  See the GNU General Public License for more
> + * details.
> + */
> +
> +
> +#include <linux/types.h>
> +#include <linux/version.h>
> +#include <linux/export.h>
> +#include <linux/vmalloc.h>
> +#include <linux/mm.h>
> +#include <linux/clocksource.h>
> +#include <linux/sched_clock.h>
> +#include <linux/acpi.h>
> +#include <linux/module.h>
> +#include <linux/hyperv.h>
> +#include <linux/slab.h>
> +#include <linux/cpuhotplug.h>
> +#include <linux/psci.h>
> +#include <asm-generic/bug.h>
> +#include <asm/hypervisor.h>
> +#include <asm/hyperv-tlfs.h>
> +#include <asm/mshyperv.h>
> +#include <asm/sysreg.h>
> +#include <clocksource/arm_arch_timer.h>
> +
> +static bool	hyperv_initialized;
> +struct		ms_hyperv_info ms_hyperv;
> +EXPORT_SYMBOL_GPL(ms_hyperv);

Who are the users of this structure? Should they go via accessors instead?

> +
> +static struct ms_hyperv_tsc_page *tsc_pg;
> +
> +struct ms_hyperv_tsc_page *hv_get_tsc_page(void)
> +{
> +	return tsc_pg;
> +}
> +EXPORT_SYMBOL_GPL(hv_get_tsc_page);
> +
> +static u64 read_hv_sched_clock_tsc(void)
> +{
> +	u64 current_tick = hv_read_tsc_page(tsc_pg);
> +
> +	if (current_tick == U64_MAX)
> +		current_tick = hv_get_vpreg(HV_REGISTER_TIME_REFCOUNT);
> +
> +	return current_tick;
> +}
> +
> +static u64 read_hv_clock_tsc(struct clocksource *arg)
> +{
> +	u64 current_tick = hv_read_tsc_page(tsc_pg);
> +
> +	if (current_tick == U64_MAX)
> +		current_tick = hv_get_vpreg(HV_REGISTER_TIME_REFCOUNT);
> +
> +	return current_tick;
> +}

Can't you define one function in terms of the other?

> +
> +static struct clocksource hyperv_cs_tsc = {
> +		.name		= "hyperv_clocksource_tsc_page",
> +		.rating		= 400,
> +		.read		= read_hv_clock_tsc,
> +		.mask		= CLOCKSOURCE_MASK(64),
> +		.flags		= CLOCK_SOURCE_IS_CONTINUOUS,
> +};
> +
> +static u64 read_hv_sched_clock_msr(void)
> +{
> +	return hv_get_vpreg(HV_REGISTER_TIME_REFCOUNT);
> +}
> +
> +static u64 read_hv_clock_msr(struct clocksource *arg)
> +{
> +	return hv_get_vpreg(HV_REGISTER_TIME_REFCOUNT);
> +}
> +
> +static struct clocksource hyperv_cs_msr = {
> +	.name		= "hyperv_clocksource_msr",
> +	.rating		= 400,
> +	.read		= read_hv_clock_msr,
> +	.mask		= CLOCKSOURCE_MASK(64),
> +	.flags		= CLOCK_SOURCE_IS_CONTINUOUS,
> +};
> +
> +struct clocksource *hyperv_cs;
> +EXPORT_SYMBOL_GPL(hyperv_cs);

Why? Who needs to poke this?

> +
> +u32 *hv_vp_index;
> +EXPORT_SYMBOL_GPL(hv_vp_index);
> +
> +u32 hv_max_vp_index;

Same thing.

> +
> +static int hv_cpu_init(unsigned int cpu)
> +{
> +	u64 msr_vp_index;
> +
> +	hv_get_vp_index(msr_vp_index);
> +
> +	hv_vp_index[smp_processor_id()] = msr_vp_index;
> +
> +	if (msr_vp_index > hv_max_vp_index)
> +		hv_max_vp_index = msr_vp_index;
> +
> +	return 0;
> +}

Is that some new way to describe a CPU topology? If so, why isn't that
exposed via the ACPI tables that the kernel already parses?

> +
> +/*
> + * This function is invoked via the ACPI clocksource probe mechanism. We
> + * don't actually use any values from the ACPI GTDT table, but we set up

This doesn't feel like a good idea at all. Piggy-backing on an existing
mechanism and use it for something completely different is not exactly
future-proof.

Also, if this is supposed to be a clocksource, why isn't that a
clocksource driver on its own right?

> + * the Hyper-V synthetic clocksource and do other initialization for
> + * interacting with Hyper-V the first time.  Using early_initcall to invoke
> + * this function is too late because interrupts are already enabled at that
> + * point, and sched_clock_register must run before interrupts are enabled.
> + *
> + * 1. Setup the guest ID.
> + * 2. Get features and hints info from Hyper-V
> + * 3. Setup per-cpu VP indices.
> + * 4. Register Hyper-V specific clocksource.
> + * 5. Register the scheduler clock.
> + */
> +
> +static int __init hyperv_init(struct acpi_table_header *table)
> +{
> +	struct hv_get_vp_register_output result;
> +	u32	a, b, c, d;
> +	u64	guest_id;
> +	int	i;
> +
> +	/*
> +	 * If we're in a VM on Hyper-V, the ACPI hypervisor_id field will
> +	 * have the string "MsHyperV".
> +	 */
> +	if (strncmp((char *)&acpi_gbl_FADT.hypervisor_id, "MsHyperV", 8))
> +		return 1;
> +
> +	/* Setup the guest ID */
> +	guest_id = generate_guest_id(0, LINUX_VERSION_CODE, 0);
> +	hv_set_vpreg(HV_REGISTER_GUEST_OSID, guest_id);
> +
> +	/* Get the features and hints from Hyper-V */
> +	hv_get_vpreg_128(HV_REGISTER_PRIVILEGES_AND_FEATURES, &result);
> +	ms_hyperv.features = lower_32_bits(result.registervaluelow);
> +	ms_hyperv.misc_features = upper_32_bits(result.registervaluehigh);
> +
> +	hv_get_vpreg_128(HV_REGISTER_FEATURES, &result);
> +	ms_hyperv.hints = lower_32_bits(result.registervaluelow);
> +
> +	pr_info("Hyper-V: Features 0x%x, hints 0x%x\n",
> +		ms_hyperv.features, ms_hyperv.hints);
> +
> +	/*
> +	 * Direct mode is the only option for STIMERs provided Hyper-V
> +	 * on ARM64, so Hyper-V doesn't actually set the flag.  But add the
> +	 * flag so the architecture independent code in drivers/hv/hv.c
> +	 * will correctly use that mode.
> +	 */
> +	ms_hyperv.misc_features |= HV_STIMER_DIRECT_MODE_AVAILABLE;
> +
> +	/*
> +	 * Hyper-V on ARM64 doesn't support AutoEOI.  Add the hint
> +	 * that tells architecture independent code not to use this
> +	 * feature.
> +	 */
> +	ms_hyperv.hints |= HV_DEPRECATING_AEOI_RECOMMENDED;
> +
> +	/* Get information about the Hyper-V host version */
> +	hv_get_vpreg_128(HV_REGISTER_HYPERVISOR_VERSION, &result);
> +	a = lower_32_bits(result.registervaluelow);
> +	b = upper_32_bits(result.registervaluelow);
> +	c = lower_32_bits(result.registervaluehigh);
> +	d = upper_32_bits(result.registervaluehigh);
> +	pr_info("Hyper-V: Host Build %d.%d.%d.%d-%d-%d\n",
> +		b >> 16, b & 0xFFFF, a, d & 0xFFFFFF, c, d >> 24);
> +
> +	/* Allocate percpu VP index */
> +	hv_vp_index = kmalloc_array(num_possible_cpus(), sizeof(*hv_vp_index),
> +				    GFP_KERNEL);

Why isn't this a percpu variable?

> +	if (!hv_vp_index)
> +		return 1;
> +
> +	for (i = 0; i < num_possible_cpus(); i++)
> +		hv_vp_index[i] = VP_INVAL;
> +
> +	if (cpuhp_setup_state(CPUHP_AP_ONLINE_DYN, "arm64/hyperv_init:online",
> +			      hv_cpu_init, NULL) < 0)
> +		goto free_vp_index;
> +
> +	/*
> +	 * Try to set up what Hyper-V calls the "TSC reference page", which
> +	 * uses the ARM Generic Timer virtual counter with some scaling
> +	 * information to provide a fast and stable guest VM clocksource.
> +	 * If the TSC reference page can't be set up, fall back to reading
> +	 * the guest clock provided by Hyper-V's synthetic reference time
> +	 * register.
> +	 */
> +	if (ms_hyperv.features & HV_MSR_REFERENCE_TSC_AVAILABLE) {
> +
> +		u64		tsc_msr;
> +		phys_addr_t	phys_addr;
> +
> +		tsc_pg = __vmalloc(HV_HYP_PAGE_SIZE, GFP_KERNEL, PAGE_KERNEL);

And why not vmalloc?

> +		if (tsc_pg) {
> +			phys_addr = page_to_phys(vmalloc_to_page(tsc_pg));
> +			tsc_msr = hv_get_vpreg(HV_REGISTER_REFERENCE_TSC);
> +			tsc_msr &= GENMASK_ULL(11, 0);

Magic.

> +			tsc_msr = tsc_msr | 0x1 | (u64)phys_addr;
> +			hv_set_vpreg(HV_REGISTER_REFERENCE_TSC, tsc_msr);
> +			hyperv_cs = &hyperv_cs_tsc;
> +			sched_clock_register(read_hv_sched_clock_tsc,
> +						64, HV_CLOCK_HZ);
> +		}
> +	}
> +
> +	if (!hyperv_cs &&
> +	    (ms_hyperv.features & HV_MSR_TIME_REF_COUNT_AVAILABLE)) {
> +		hyperv_cs = &hyperv_cs_msr;
> +		sched_clock_register(read_hv_sched_clock_msr,
> +						64, HV_CLOCK_HZ);
> +	}
> +
> +	if (hyperv_cs) {
> +		hyperv_cs->archdata.vdso_direct = false;
> +		clocksource_register_hz(hyperv_cs, HV_CLOCK_HZ);
> +	}
> +
> +	hyperv_initialized = true;
> +	return 0;
> +
> +free_vp_index:
> +	kfree(hv_vp_index);
> +	hv_vp_index = NULL;
> +	return 1;

????

> +}
> +TIMER_ACPI_DECLARE(hyperv, ACPI_SIG_GTDT, hyperv_init);
> +
> +/*
> + * This routine is called before kexec/kdump, it does the required cleanup.
> + */
> +void hyperv_cleanup(void)
> +{
> +	/* Reset our OS id */
> +	hv_set_vpreg(HV_REGISTER_GUEST_OSID, 0);
> +
> +}
> +EXPORT_SYMBOL_GPL(hyperv_cleanup);
> +
> +/*
> + * hv_do_hypercall- Invoke the specified hypercall
> + */
> +u64 hv_do_hypercall(u64 control, void *input, void *output)
> +{
> +	u64 input_address;
> +	u64 output_address;
> +
> +	input_address = input ? virt_to_phys(input) : 0;
> +	output_address = output ? virt_to_phys(output) : 0;
> +	return hv_do_hvc(control, input_address, output_address);
> +}
> +EXPORT_SYMBOL_GPL(hv_do_hypercall);
> +
> +/*
> + * hv_do_fast_hypercall8 -- Invoke the specified hypercall
> + * with arguments in registers instead of physical memory.
> + * Avoids the overhead of virt_to_phys for simple hypercalls.
> + */
> +
> +u64 hv_do_fast_hypercall8(u16 code, u64 input)
> +{
> +	u64 control;
> +
> +	control = (u64)code | HV_HYPERCALL_FAST_BIT;
> +	return hv_do_hvc(control, input);
> +}
> +EXPORT_SYMBOL_GPL(hv_do_fast_hypercall8);
> +
> +
> +/*
> + * Set a single VP register to a 64-bit value.
> + */
> +void hv_set_vpreg(u32 msr, u64 value)
> +{
> +	union hv_hypercall_status status;
> +
> +	status.as_uint64 = hv_do_hvc(
> +		HVCALL_SET_VP_REGISTERS | HV_HYPERCALL_FAST_BIT |
> +			HV_HYPERCALL_REP_COUNT_1,
> +		HV_PARTITION_ID_SELF,
> +		HV_VP_INDEX_SELF,
> +		msr,
> +		0,
> +		value,
> +		0);
> +
> +	/*
> +	 * Something is fundamentally broken in the hypervisor if
> +	 * setting a VP register fails. There's really no way to
> +	 * continue as a guest VM, so panic.
> +	 */
> +	BUG_ON(status.status != HV_STATUS_SUCCESS);
> +}
> +EXPORT_SYMBOL_GPL(hv_set_vpreg);
> +
> +
> +/*
> + * Get the value of a single VP register, and only the low order 64 bits.
> + */
> +u64 hv_get_vpreg(u32 msr)
> +{
> +	union hv_hypercall_status status;
> +	struct hv_get_vp_register_output output;
> +
> +	status.as_uint64 = hv_do_hvc_fast_get(
> +		HVCALL_GET_VP_REGISTERS | HV_HYPERCALL_FAST_BIT |
> +			HV_HYPERCALL_REP_COUNT_1,
> +		HV_PARTITION_ID_SELF,
> +		HV_VP_INDEX_SELF,
> +		msr,
> +		&output);
> +
> +	/*
> +	 * Something is fundamentally broken in the hypervisor if
> +	 * getting a VP register fails. There's really no way to
> +	 * continue as a guest VM, so panic.
> +	 */
> +	BUG_ON(status.status != HV_STATUS_SUCCESS);
> +
> +	return output.registervaluelow;
> +}
> +EXPORT_SYMBOL_GPL(hv_get_vpreg);
> +
> +/*
> + * Get the value of a single VP register that is 128 bits in size.  This is a
> + * separate call in order to avoid complicating the calling sequence for
> + * the much more frequently used 64-bit version.
> + */
> +void hv_get_vpreg_128(u32 msr, struct hv_get_vp_register_output *result)
> +{
> +	union hv_hypercall_status status;
> +
> +	status.as_uint64 = hv_do_hvc_fast_get(
> +		HVCALL_GET_VP_REGISTERS | HV_HYPERCALL_FAST_BIT |
> +			HV_HYPERCALL_REP_COUNT_1,
> +		HV_PARTITION_ID_SELF,
> +		HV_VP_INDEX_SELF,
> +		msr,
> +		result);
> +
> +	/*
> +	 * Something is fundamentally broken in the hypervisor if
> +	 * getting a VP register fails. There's really no way to
> +	 * continue as a guest VM, so panic.
> +	 */
> +	BUG_ON(status.status != HV_STATUS_SUCCESS);
> +
> +	return;
> +
> +}
> +EXPORT_SYMBOL_GPL(hv_get_vpreg_128);
> +
> +void hyperv_report_panic(struct pt_regs *regs, long err)
> +{
> +	static bool panic_reported;
> +	u64 guest_id;
> +
> +	/*
> +	 * We prefer to report panic on 'die' chain as we have proper
> +	 * registers to report, but if we miss it (e.g. on BUG()) we need
> +	 * to report it on 'panic'.
> +	 */
> +	if (panic_reported)
> +		return;
> +	panic_reported = true;
> +
> +	guest_id = hv_get_vpreg(HV_REGISTER_GUEST_OSID);
> +
> +	/*
> +	 * Hyper-V provides the ability to store only 5 values.
> +	 * Pick the passed in error value, the guest_id, and the PC.
> +	 * The first two general registers are added arbitrarily.
> +	 */
> +	hv_set_vpreg(HV_REGISTER_CRASH_P0, err);
> +	hv_set_vpreg(HV_REGISTER_CRASH_P1, guest_id);
> +	hv_set_vpreg(HV_REGISTER_CRASH_P2, regs->pc);
> +	hv_set_vpreg(HV_REGISTER_CRASH_P3, regs->regs[0]);
> +	hv_set_vpreg(HV_REGISTER_CRASH_P4, regs->regs[1]);
> +
> +	/*
> +	 * Let Hyper-V know there is crash data available
> +	 */
> +	hv_set_vpreg(HV_REGISTER_CRASH_CTL, HV_CRASH_CTL_CRASH_NOTIFY);
> +}
> +EXPORT_SYMBOL_GPL(hyperv_report_panic);
> +
> +/*
> + * hyperv_report_panic_msg - report panic message to Hyper-V
> + * @pa: physical address of the panic page containing the message
> + * @size: size of the message in the page
> + */
> +void hyperv_report_panic_msg(phys_addr_t pa, size_t size)
> +{
> +	/*
> +	 * P3 to contain the physical address of the panic page & P4 to
> +	 * contain the size of the panic data in that page. Rest of the
> +	 * registers are no-op when the NOTIFY_MSG flag is set.
> +	 */
> +	hv_set_vpreg(HV_REGISTER_CRASH_P0, 0);
> +	hv_set_vpreg(HV_REGISTER_CRASH_P1, 0);
> +	hv_set_vpreg(HV_REGISTER_CRASH_P2, 0);
> +	hv_set_vpreg(HV_REGISTER_CRASH_P3, pa);
> +	hv_set_vpreg(HV_REGISTER_CRASH_P4, size);
> +
> +	/*
> +	 * Let Hyper-V know there is crash data available along with
> +	 * the panic message.
> +	 */
> +	hv_set_vpreg(HV_REGISTER_CRASH_CTL,
> +	       (HV_CRASH_CTL_CRASH_NOTIFY | HV_CRASH_CTL_CRASH_NOTIFY_MSG));
> +}
> +EXPORT_SYMBOL_GPL(hyperv_report_panic_msg);
> +
> +bool hv_is_hyperv_initialized(void)
> +{
> +	return hyperv_initialized;
> +}
> +EXPORT_SYMBOL_GPL(hv_is_hyperv_initialized);
> diff --git a/arch/arm64/hyperv/mshyperv.c b/arch/arm64/hyperv/mshyperv.c
> new file mode 100644
> index 000000000000..3ef055599412
> --- /dev/null
> +++ b/arch/arm64/hyperv/mshyperv.c
> @@ -0,0 +1,178 @@
> +// SPDX-License-Identifier: GPL-2.0
> +
> +/*
> + * Core routines for interacting with Microsoft's Hyper-V hypervisor,
> + * including setting up VMbus and STIMER interrupts, and handling
> + * crashes and kexecs. These interactions are through a set of
> + * static "handler" variables set by the architecture independent
> + * VMbus and STIMER drivers.  This design is used to meet x86/x64
> + * requirements for avoiding direct linkages and allowing the VMbus
> + * and STIMER drivers to be unloaded and reloaded.
> + *
> + * Copyright (C) 2018, Microsoft, Inc.
> + *
> + * Author : Michael Kelley <mikelley@microsoft.com>
> + *
> + * This program is free software; you can redistribute it and/or modify it
> + * under the terms of the GNU General Public License version 2 as published
> + * by the Free Software Foundation.
> + *
> + * This program is distributed in the hope that it will be useful, but
> + * WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE, GOOD TITLE or
> + * NON INFRINGEMENT.  See the GNU General Public License for more
> + * details.
> + */
> +
> +#include <linux/types.h>
> +#include <linux/export.h>
> +#include <linux/interrupt.h>
> +#include <linux/kexec.h>
> +#include <linux/acpi.h>
> +#include <linux/ptrace.h>
> +#include <asm/hyperv-tlfs.h>
> +#include <asm/mshyperv.h>
> +
> +static void (*vmbus_handler)(void);
> +static void (*hv_stimer0_handler)(void);
> +static void (*hv_kexec_handler)(void);
> +static void (*hv_crash_handler)(struct pt_regs *regs);
> +
> +static int vmbus_irq;
> +static long __percpu *vmbus_evt;
> +static long __percpu *stimer0_evt;
> +
> +irqreturn_t hyperv_vector_handler(int irq, void *dev_id)
> +{
> +	if (vmbus_handler)
> +		vmbus_handler();

In which circumstances can this be NULL?

> +	return IRQ_HANDLED;
> +}
> +
> +/* Must be done just once */
> +void hv_setup_vmbus_irq(void (*handler)(void))
> +{
> +	int result;
> +
> +	vmbus_handler = handler;

If thismust only be done once, maybe you could check that it hasn't been
done already?

> +	vmbus_irq = acpi_register_gsi(NULL, HYPERVISOR_CALLBACK_VECTOR,
> +				 ACPI_LEVEL_SENSITIVE, ACPI_ACTIVE_HIGH);
> +	if (vmbus_irq <= 0) {
> +		pr_err("Can't register Hyper-V VMBus GSI. Error %d",
> +			vmbus_irq);
> +		vmbus_irq = 0;
> +		return;
> +	}
> +	vmbus_evt = alloc_percpu(long);
> +	result = request_percpu_irq(vmbus_irq, hyperv_vector_handler,
> +			"Hyper-V VMbus", vmbus_evt);

If this is a per-cpu interrupt, why isn't it signalled as a PPI, in an
architecture compliant way?

> +	if (result) {
> +		pr_err("Can't request Hyper-V VMBus IRQ %d. Error %d",
> +			vmbus_irq, result);
> +		free_percpu(vmbus_evt);
> +		acpi_unregister_gsi(vmbus_irq);
> +		vmbus_irq = 0;
> +	}
> +}
> +EXPORT_SYMBOL_GPL(hv_setup_vmbus_irq);
> +
> +/* Must be done just once */
> +void hv_remove_vmbus_irq(void)
> +{
> +	if (vmbus_irq) {
> +		free_percpu_irq(vmbus_irq, vmbus_evt);
> +		free_percpu(vmbus_evt);
> +		acpi_unregister_gsi(vmbus_irq);
> +	}
> +}
> +EXPORT_SYMBOL_GPL(hv_remove_vmbus_irq);
> +
> +/* Must be done by each CPU */
> +void hv_enable_vmbus_irq(void)
> +{
> +	enable_percpu_irq(vmbus_irq, 0);
> +}
> +EXPORT_SYMBOL_GPL(hv_enable_vmbus_irq);
> +
> +/* Must be done by each CPU */
> +void hv_disable_vmbus_irq(void)
> +{
> +	disable_percpu_irq(vmbus_irq);
> +}
> +EXPORT_SYMBOL_GPL(hv_disable_vmbus_irq);
> +
> +/* Routines to do per-architecture handling of STIMER0 when in Direct Mode */
> +
> +static irqreturn_t hv_stimer0_vector_handler(int irq, void *dev_id)
> +{
> +	if (hv_stimer0_handler)
> +		hv_stimer0_handler();
> +	return IRQ_HANDLED;
> +}
> +
> +int hv_setup_stimer0_irq(int *irq, int *vector, void (*handler)(void))
> +{
> +	int localirq;
> +	int result;
> +
> +	localirq = acpi_register_gsi(NULL, HV_STIMER0_IRQNR,
> +			ACPI_LEVEL_SENSITIVE, ACPI_ACTIVE_HIGH);
> +	if (localirq <= 0) {
> +		pr_err("Can't register Hyper-V stimer0 GSI. Error %d",
> +			localirq);
> +		*irq = 0;
> +		return -1;
> +	}
> +	stimer0_evt = alloc_percpu(long);
> +	result = request_percpu_irq(localirq, hv_stimer0_vector_handler,
> +					 "Hyper-V stimer0", stimer0_evt);
> +	if (result) {
> +		pr_err("Can't request Hyper-V stimer0 IRQ %d. Error %d",
> +			localirq, result);
> +		free_percpu(stimer0_evt);
> +		acpi_unregister_gsi(localirq);
> +		*irq = 0;
> +		return -1;
> +	}
> +
> +	hv_stimer0_handler = handler;
> +	*vector = HV_STIMER0_IRQNR;
> +	*irq = localirq;
> +	return 0;
> +}
> +EXPORT_SYMBOL_GPL(hv_setup_stimer0_irq);
> +
> +void hv_remove_stimer0_irq(int irq)
> +{
> +	hv_stimer0_handler = NULL;
> +	if (irq) {
> +		free_percpu_irq(irq, stimer0_evt);
> +		free_percpu(stimer0_evt);
> +		acpi_unregister_gsi(irq);
> +	}
> +}
> +EXPORT_SYMBOL_GPL(hv_remove_stimer0_irq);
> +
> +void hv_setup_kexec_handler(void (*handler)(void))
> +{
> +	hv_kexec_handler = handler;
> +}
> +EXPORT_SYMBOL_GPL(hv_setup_kexec_handler);
> +
> +void hv_remove_kexec_handler(void)
> +{
> +	hv_kexec_handler = NULL;
> +}
> +EXPORT_SYMBOL_GPL(hv_remove_kexec_handler);
> +
> +void hv_setup_crash_handler(void (*handler)(struct pt_regs *regs))
> +{
> +	hv_crash_handler = handler;
> +}
> +EXPORT_SYMBOL_GPL(hv_setup_crash_handler);
> +
> +void hv_remove_crash_handler(void)
> +{
> +	hv_crash_handler = NULL;
> +}
> +EXPORT_SYMBOL_GPL(hv_remove_crash_handler);
> 

Overall, this patch needs a massive split up, clean up, and a good dose
of documentation so that we can understand what you are trying to achieve.

Thanks,

	M.
-- 
Jazz is not dead. It just smells funny...

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 1/4] arm64: hyperv: Add core Hyper-V include files

[Vitaly Kuznetsov]

kys@linuxonhyperv.com writes:

> +
> +static inline u64 hv_read_tsc_page_tsc(const struct ms_hyperv_tsc_page *tsc_pg,
> +				       u64 *cur_tsc)
> +{
> +	u64	scale, offset;
> +	u32	sequence;
> +
> +	/*
> +	 * The protocol for reading Hyper-V TSC page is specified in Hypervisor
> +	 * Top-Level Functional Specification.  To get the reference time we
> +	 * must do the following:
> +	 * - READ ReferenceTscSequence
> +	 *   A special '0' value indicates the time source is unreliable and we
> +	 *   need to use something else.
> +	 * - ReferenceTime =
> +	 *     ((HWclock val) * ReferenceTscScale) >> 64) + ReferenceTscOffset
> +	 * - READ ReferenceTscSequence again. In case its value has changed
> +	 *   since our first reading we need to discard ReferenceTime and repeat
> +	 *   the whole sequence as the hypervisor was updating the page in
> +	 *   between.
> +	 */
> +	do {
> +		sequence = READ_ONCE(tsc_pg->tsc_sequence);

(sorry if this was already discussed before)

Current x86 code doing this actually checks for '0' here (note the
comment about this special value above):

		sequence = READ_ONCE(tsc_pg->tsc_sequence);
		if (!sequence)
			return U64_MAX;

Was it removed intentionally (and we need to fix the comment then)? 

-- 
Vitaly

Re: [PATCH 1/4] arm64: hyperv: Add core Hyper-V include files

[Vitaly Kuznetsov]

kys@linuxonhyperv.com writes:

> +
> +static inline u64 hv_read_tsc_page_tsc(const struct ms_hyperv_tsc_page *tsc_pg,
> +				       u64 *cur_tsc)
> +{
> +	u64	scale, offset;
> +	u32	sequence;
> +
> +	/*
> +	 * The protocol for reading Hyper-V TSC page is specified in Hypervisor
> +	 * Top-Level Functional Specification.  To get the reference time we
> +	 * must do the following:
> +	 * - READ ReferenceTscSequence
> +	 *   A special '0' value indicates the time source is unreliable and we
> +	 *   need to use something else.
> +	 * - ReferenceTime =
> +	 *     ((HWclock val) * ReferenceTscScale) >> 64) + ReferenceTscOffset
> +	 * - READ ReferenceTscSequence again. In case its value has changed
> +	 *   since our first reading we need to discard ReferenceTime and repeat
> +	 *   the whole sequence as the hypervisor was updating the page in
> +	 *   between.
> +	 */
> +	do {
> +		sequence = READ_ONCE(tsc_pg->tsc_sequence);

(sorry if this was already discussed before)

Current x86 code doing this actually checks for '0' here (note the
comment about this special value above):

		sequence = READ_ONCE(tsc_pg->tsc_sequence);
		if (!sequence)
			return U64_MAX;

Was it removed intentionally (and we need to fix the comment then)? 

-- 
Vitaly

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

RE: [PATCH 1/4] arm64: hyperv: Add core Hyper-V include files

[Michael Kelley]

From: Will Deacon <will.deacon@arm.com> Sent: Friday, December 7, 2018 5:43 AM

> > hyperv-tlfs.h defines Hyper-V interfaces from the Hyper-V Top Level
> > Functional Spec (TLFS). The TLFS is distinctly oriented to x86/x64,
> > and Hyper-V has not separated out the architecture-dependent parts into
> > x86/x64 vs. ARM64. So hyperv-tlfs.h includes information for ARM64
> > that is not yet formally published. The TLFS is available here:
> 
> When do you plan to publish the spec? It's pretty hard to review this stuff
> without knowing what it's supposed to look like.

I don't have a commitment from the Hyper-V team on when an updated TLFS
that covers ARM64 will be published.  I'm on the Linux side, and Hyper-V is a
separate group, but I'll raise the topic again with them.

> 
> >   docs.microsoft.com/en-us/virtualization/hyper-v-on-windows/reference/tlfs
> >
> > mshyperv.h defines Linux-specific structures and routines for
> > interacting with Hyper-V. It is split into an ARM64 specific file
> > and an architecture independent file in include/asm-generic.
> >
> > Signed-off-by: Michael Kelley <mikelley@microsoft.com>
> > Signed-off-by: K. Y. Srinivasan <kys@microsoft.com>
> > ---
> >  MAINTAINERS                          |   3 +
> >  arch/arm64/include/asm/hyperv-tlfs.h | 338 +++++++++++++++++++++++++++
> >  arch/arm64/include/asm/mshyperv.h    | 116 +++++++++
> >  include/asm-generic/mshyperv.h       | 240 +++++++++++++++++++
> >  4 files changed, 697 insertions(+)
> >  create mode 100644 arch/arm64/include/asm/hyperv-tlfs.h
> >  create mode 100644 arch/arm64/include/asm/mshyperv.h
> >  create mode 100644 include/asm-generic/mshyperv.h
> >
> > diff --git a/MAINTAINERS b/MAINTAINERS
> > index f4855974f325..72f19cef4c48 100644
> > --- a/MAINTAINERS
> > +++ b/MAINTAINERS
> > @@ -6835,6 +6835,8 @@ F:	arch/x86/include/asm/trace/hyperv.h
> >  F:	arch/x86/include/asm/hyperv-tlfs.h
> >  F:	arch/x86/kernel/cpu/mshyperv.c
> >  F:	arch/x86/hyperv
> > +F:	arch/arm64/include/asm/hyperv-tlfs.h
> > +F:	arch/arm64/include/asm/mshyperv.h
> >  F:	drivers/hid/hid-hyperv.c
> >  F:	drivers/hv/
> >  F:	drivers/input/serio/hyperv-keyboard.c
> > @@ -6846,6 +6848,7 @@ F:	drivers/video/fbdev/hyperv_fb.c
> >  F:	net/vmw_vsock/hyperv_transport.c
> >  F:	include/linux/hyperv.h
> >  F:	include/uapi/linux/hyperv.h
> > +F:	include/asm-generic/mshyperv.h
> >  F:	tools/hv/
> >  F:	Documentation/ABI/stable/sysfs-bus-vmbus
> >
> > diff --git a/arch/arm64/include/asm/hyperv-tlfs.h b/arch/arm64/include/asm/hyperv-
> tlfs.h
> > new file mode 100644
> > index 000000000000..924e37600e92
> > --- /dev/null
> > +++ b/arch/arm64/include/asm/hyperv-tlfs.h
> > @@ -0,0 +1,338 @@
> > +/* SPDX-License-Identifier: GPL-2.0 */
> > +
> > +/*
> > + * This file contains definitions from the Hyper-V Hypervisor Top-Level
> > + * Functional Specification (TLFS):
> > + *
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.microsoft.co
> m%2Fen-us%2Fvirtualization%2Fhyper-v-on-
> windows%2Freference%2Ftlfs&amp;data=02%7C01%7Cmikelley%40microsoft.com%7Ce1b
> dbb31db064623174f08d65c49cc3b%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C63
> 6797869386921804&amp;sdata=RiD05cDWC%2FPnXnis6U7EcfEfCjvb54uuKHRfifQhMEM%3D
> &amp;reserved=0
> 
> As mentioned elsewhere, please use a better link here and drop the license
> boilerplate below.

Agreed.  Will do so here and in other files in the next version of the patch.

> 
> > + *
> > + * Copyright (C) 2018, Microsoft, Inc.
> > + *
> > + * Author : Michael Kelley <mikelley@microsoft.com>
> > + *
> > + * This program is free software; you can redistribute it and/or modify it
> > + * under the terms of the GNU General Public License version 2 as published
> > + * by the Free Software Foundation.
> > + *
> > + * This program is distributed in the hope that it will be useful, but
> > + * WITHOUT ANY WARRANTY; without even the implied warranty of
> > + * MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE, GOOD TITLE or
> > + * NON INFRINGEMENT.  See the GNU General Public License for more
> > + * details.
> > + */
> > +
> > +#ifndef _ASM_ARM64_HYPERV_H
> > +#define _ASM_ARM64_HYPERV_H
> 
> __ASM_HYPER_V_H please

OK.

> 
> > +
> > +#include <linux/types.h>
> > +
> > +/*
> > + * These Hyper-V registers provide information equivalent to the CPUID
> > + * instruction on x86/x64.
> > + */
> > +#define HV_REGISTER_HYPERVISOR_VERSION		0x00000100 /*CPUID
> 0x40000002 */
> > +#define	HV_REGISTER_PRIVILEGES_AND_FEATURES	0x00000200 /*CPUID
> 0x40000003 */
> > +#define	HV_REGISTER_FEATURES			0x00000201 /*CPUID
> 0x40000004 */
> > +#define	HV_REGISTER_IMPLEMENTATION_LIMITS	0x00000202 /*CPUID
> 0x40000005 */
> > +#define HV_ARM64_REGISTER_INTERFACE_VERSION	0x00090006 /*CPUID
> 0x40000001 */
> > +
> > +/*
> > + * Feature identification. HvRegisterPrivilegesAndFeaturesInfo returns a
> > + * 128-bit value with flags indicating which features are available to the
> > + * partition based upon the current partition privileges. The 128-bit
> > + * value is broken up with different portions stored in different 32-bit
> > + * fields in the ms_hyperv structure.
> > + */
> > +
> > +/* Partition Reference Counter available*/
> > +#define HV_MSR_TIME_REF_COUNT_AVAILABLE		(1 << 1)
> > +
> > +/*
> > + * Synthetic Timers available
> > + */
> > +#define HV_MSR_SYNTIMER_AVAILABLE		(1 << 3)
> > +
> > +/* Frequency MSRs available */
> > +#define HV_FEATURE_FREQUENCY_MSRS_AVAILABLE	(1 << 8)
> > +
> > +/* Reference TSC available */
> > +#define HV_MSR_REFERENCE_TSC_AVAILABLE		(1 << 9)
> > +
> > +/* Crash MSR available */
> > +#define HV_FEATURE_GUEST_CRASH_MSR_AVAILABLE	(1 << 10)
> > +
> > +
> > +/*
> > + * This group of flags is in the high order 64-bits of the returned
> > + * 128-bit value.
> > + */
> > +
> > +/* STIMER direct mode is available */
> > +#define HV_STIMER_DIRECT_MODE_AVAILABLE		(1 << 19)
> > +
> > +/*
> > + * Implementation recommendations in register
> > + * HvRegisterFeaturesInfo. Indicates which behaviors the hypervisor
> > + * recommends the OS implement for optimal performance.
> > + */
> > +
> > +/*
> > + * Recommend not using Auto EOI
> > + */
> > +#define HV_DEPRECATING_AEOI_RECOMMENDED		(1 << 9)
> > +
> > +/*
> > + * Synthetic register definitions equivalent to MSRs on x86/x64
> > + */
> > +#define HV_REGISTER_CRASH_P0		0x00000210
> > +#define HV_REGISTER_CRASH_P1		0x00000211
> > +#define HV_REGISTER_CRASH_P2		0x00000212
> > +#define HV_REGISTER_CRASH_P3		0x00000213
> > +#define HV_REGISTER_CRASH_P4		0x00000214
> > +#define HV_REGISTER_CRASH_CTL		0x00000215
> > +
> > +#define HV_REGISTER_GUEST_OSID		0x00090002
> > +#define HV_REGISTER_VPINDEX		0x00090003
> > +#define HV_REGISTER_TIME_REFCOUNT	0x00090004
> > +#define HV_REGISTER_REFERENCE_TSC	0x00090017
> > +
> > +#define HV_REGISTER_SINT0		0x000A0000
> > +#define HV_REGISTER_SINT1		0x000A0001
> > +#define HV_REGISTER_SINT2		0x000A0002
> > +#define HV_REGISTER_SINT3		0x000A0003
> > +#define HV_REGISTER_SINT4		0x000A0004
> > +#define HV_REGISTER_SINT5		0x000A0005
> > +#define HV_REGISTER_SINT6		0x000A0006
> > +#define HV_REGISTER_SINT7		0x000A0007
> > +#define HV_REGISTER_SINT8		0x000A0008
> > +#define HV_REGISTER_SINT9		0x000A0009
> > +#define HV_REGISTER_SINT10		0x000A000A
> > +#define HV_REGISTER_SINT11		0x000A000B
> > +#define HV_REGISTER_SINT12		0x000A000C
> > +#define HV_REGISTER_SINT13		0x000A000D
> > +#define HV_REGISTER_SINT14		0x000A000E
> > +#define HV_REGISTER_SINT15		0x000A000F
> > +#define HV_REGISTER_SCONTROL		0x000A0010
> > +#define HV_REGISTER_SVERSION		0x000A0011
> > +#define HV_REGISTER_SIFP		0x000A0012
> > +#define HV_REGISTER_SIPP		0x000A0013
> > +#define HV_REGISTER_EOM			0x000A0014
> > +#define HV_REGISTER_SIRBP		0x000A0015
> > +
> > +#define HV_REGISTER_STIMER0_CONFIG	0x000B0000
> > +#define HV_REGISTER_STIMER0_COUNT	0x000B0001
> > +#define HV_REGISTER_STIMER1_CONFIG	0x000B0002
> > +#define HV_REGISTER_STIMER1_COUNT	0x000B0003
> > +#define HV_REGISTER_STIMER2_CONFIG	0x000B0004
> > +#define HV_REGISTER_STIMER2_COUNT	0x000B0005
> > +#define HV_REGISTER_STIMER3_CONFIG	0x000B0006
> > +#define HV_REGISTER_STIMER3_COUNT	0x000B0007
> > +
> > +/*
> > + * Crash notification flags.
> > + */
> > +#define HV_CRASH_CTL_CRASH_NOTIFY_MSG	BIT_ULL(62)
> > +#define HV_CRASH_CTL_CRASH_NOTIFY	BIT_ULL(63)
> > +
> > +/*
> > + * The guest OS needs to register the guest ID with the hypervisor.
> > + * The guest ID is a 64 bit entity and the structure of this ID is
> > + * specified in the Hyper-V specification:
> > + *
> > + * msdn.microsoft.com/en-us/library/windows/hardware/ff542653%28v=vs.85%29.aspx
> 
> Dead link :(

Hmmm.  It looks like that doc is gone and the info is now in the TLFS.  This
comment block shouldn't be here anyway.  As you noted later, there's another
copy in include/asm-generic/mshyperv.h, which is the right place.  I'll
get this cleaned up in the next version of the patch.

> 
> > + *
> > + * While the current guideline does not specify how Linux guest ID(s)
> > + * need to be generated, our plan is to publish the guidelines for
> > + * Linux and other guest operating systems that currently are hosted
> > + * on Hyper-V. The implementation here conforms to this yet
> > + * unpublished guidelines.
> 
> Again, any timeline on publishing this stuff? Right now, this is just a file
> full of magic numbers and I'm not sure we're in a good position to maintain
> it based on the idea that you have a cunning plan.

Fair point.

> 
> > + *
> > + *
> > + * Bit(s)
> > + * 63 - Indicates if the OS is Open Source or not; 1 is Open Source
> > + * 62:56 - Os Type; Linux is 0x100
> > + * 55:48 - Distro specific identification
> > + * 47:16 - Linux kernel version number
> > + * 15:0  - Distro specific identification
> > + *
> > + *
> > + */
> > +#define HV_LINUX_VENDOR_ID              0x8100
> > +
> > +/* Declare the various hypercall operations. */
> > +#define HVCALL_FLUSH_VIRTUAL_ADDRESS_SPACE	0x0002
> > +#define HVCALL_FLUSH_VIRTUAL_ADDRESS_LIST	0x0003
> > +#define HVCALL_NOTIFY_LONG_SPIN_WAIT		0x0008
> > +#define HVCALL_SEND_IPI				0x000b
> > +#define HVCALL_FLUSH_VIRTUAL_ADDRESS_SPACE_EX	0x0013
> > +#define HVCALL_FLUSH_VIRTUAL_ADDRESS_LIST_EX	0x0014
> > +#define HVCALL_SEND_IPI_EX			0x0015
> > +#define HVCALL_GET_VP_REGISTERS			0x0050
> > +#define HVCALL_SET_VP_REGISTERS			0x0051
> > +#define HVCALL_POST_MESSAGE			0x005c
> > +#define HVCALL_SIGNAL_EVENT			0x005d
> > +#define HVCALL_RETARGET_INTERRUPT		0x007e
> > +#define HVCALL_START_VIRTUAL_PROCESSOR		0x0099
> > +#define HVCALL_GET_VP_INDEX_FROM_APICID		0x009a
> 
> This all sounds very x86y...

Indeed it is.  Actually, I can delete this definition as the
GET_VP_INDEX_FROM_APICID hypercall is not used on ARM64.

> 
> > +/* Declare standard hypercall field values. */
> > +#define HV_PARTITION_ID_SELF                    ((u64)-1)
> > +#define HV_VP_INDEX_SELF                        ((u32)-2)
> > +
> > +#define HV_HYPERCALL_FAST_BIT                   BIT(16)
> > +#define HV_HYPERCALL_REP_COUNT_1                BIT_ULL(32)
> > +#define HV_HYPERCALL_RESULT_MASK                GENMASK_ULL(15, 0)
> > +
> > +/* Define the hypercall status result */
> > +
> > +union hv_hypercall_status {
> > +	u64 as_uint64;
> > +	struct {
> > +		u16 status;
> > +		u16 reserved;
> > +		u16 reps_completed;  /* Low 12 bits */
> > +		u16 reserved2;
> > +	};
> > +};
> > +
> > +/* hypercall status code */
> > +#define HV_STATUS_SUCCESS			0
> > +#define HV_STATUS_INVALID_HYPERCALL_CODE	2
> > +#define HV_STATUS_INVALID_HYPERCALL_INPUT	3
> > +#define HV_STATUS_INVALID_ALIGNMENT		4
> > +#define HV_STATUS_INSUFFICIENT_MEMORY		11
> > +#define HV_STATUS_INVALID_CONNECTION_ID		18
> > +#define HV_STATUS_INSUFFICIENT_BUFFERS		19
> > +
> > +/* Define output layout for Get VP Register hypercall */
> > +struct hv_get_vp_register_output {
> > +	u64 registervaluelow;
> > +	u64 registervaluehigh;
> > +};
> > +
> > +#define HV_FLUSH_ALL_PROCESSORS			BIT(0)
> > +#define HV_FLUSH_ALL_VIRTUAL_ADDRESS_SPACES	BIT(1)
> > +#define HV_FLUSH_NON_GLOBAL_MAPPINGS_ONLY	BIT(2)
> > +#define HV_FLUSH_USE_EXTENDED_RANGE_FORMAT	BIT(3)
> > +
> > +enum HV_GENERIC_SET_FORMAT {
> > +	HV_GENERIC_SET_SPARSE_4K,
> > +	HV_GENERIC_SET_ALL,
> > +};
> > +
> > +/*
> > + * The Hyper-V TimeRefCount register and the TSC
> > + * page provide a guest VM clock with 100ns tick rate
> > + */
> > +#define HV_CLOCK_HZ (NSEC_PER_SEC/100)
> > +
> > +/*
> > + * The fields in this structure are set by Hyper-V and read
> > + * by the Linux guest.  They should be accessed with READ_ONCE()
> > + * so the compiler doesn't optimize in a way that will cause
> > + * problems.
> > + */
> > +struct ms_hyperv_tsc_page {
> > +	u32 tsc_sequence;
> > +	u32 reserved1;
> > +	u64 tsc_scale;
> > +	s64 tsc_offset;
> > +	u64 reserved2[509];
> > +};
> 
> It might be clearer to express this as a union with the page size:
> 
> struct ms_hyperv_tsc_page {
> 	union {
> 		struct {
> 			__u32	tsc_sequence;
> 			__u32	reserved1;
> 		};
> 		__u8	reserved2[HV_HYP_PAGE_SIZE];
> 	};
> };
> 
> What's the required alignment on this structure?
> Also, do you intend for 32-bit guests to use this ABI as well?

The definition for struct ms_hyperv_tsc_page matches what
is written in section 12.6.2 of the Hyper-V TLFS, so I'm a little
reluctant to change it.  Alignment must be to Hyper-V's
4 Kbyte page size and the memory allocation for the structure
ensures that alignment when the guest page size is 4K.  In the
future, in order to work with Hyper-V, there's going to
be a general requirement for ARM64 guests to be able to allocate
4 Kbytes of memory that is 4 Kbyte aligned, even when we add
support for 16K/64K page sizes in the guest.  At the moment these
larger page sizes don't work due to incorrect assumptions about
page size in Hyper-V drivers that are architecture independent.
Fixing those mis-assumptions will be a separate set of patches.

There's no intent to support 32-bit guests running on Hyper-V for ARM64.

> 
> > +
> > +/* Define the number of synthetic interrupt sources. */
> > +#define HV_SYNIC_SINT_COUNT		(16)
> > +/* Define the expected SynIC version. */
> > +#define HV_SYNIC_VERSION_1		(0x1)
> > +
> > +#define HV_SYNIC_CONTROL_ENABLE		(1ULL << 0)
> > +#define HV_SYNIC_SIMP_ENABLE		(1ULL << 0)
> > +#define HV_SYNIC_SIEFP_ENABLE		(1ULL << 0)
> > +#define HV_SYNIC_SINT_MASKED		(1ULL << 16)
> > +#define HV_SYNIC_SINT_AUTO_EOI		(1ULL << 17)
> > +#define HV_SYNIC_SINT_VECTOR_MASK	(0xFF)
> > +
> > +#define HV_SYNIC_STIMER_COUNT		(4)
> > +
> > +/* Define synthetic interrupt controller message constants. */
> > +#define HV_MESSAGE_SIZE			(256)
> > +#define HV_MESSAGE_PAYLOAD_BYTE_COUNT	(240)
> > +#define HV_MESSAGE_PAYLOAD_QWORD_COUNT	(30)
> > +
> > +/* Define hypervisor message types. */
> > +enum hv_message_type {
> > +	HVMSG_NONE			= 0x00000000,
> > +
> > +	/* Memory access messages. */
> > +	HVMSG_UNMAPPED_GPA		= 0x80000000,
> > +	HVMSG_GPA_INTERCEPT		= 0x80000001,
> > +
> > +	/* Timer notification messages. */
> > +	HVMSG_TIMER_EXPIRED		= 0x80000010,
> > +
> > +	/* Error messages. */
> > +	HVMSG_INVALID_VP_REGISTER_VALUE	= 0x80000020,
> > +	HVMSG_UNRECOVERABLE_EXCEPTION	= 0x80000021,
> > +	HVMSG_UNSUPPORTED_FEATURE	= 0x80000022,
> > +
> > +	/* Trace buffer complete messages. */
> > +	HVMSG_EVENTLOG_BUFFERCOMPLETE	= 0x80000040,
> > +};
> > +
> > +/* Define synthetic interrupt controller message flags. */
> > +union hv_message_flags {
> > +	__u8 asu8;
> > +	struct {
> > +		__u8 msg_pending:1;
> > +		__u8 reserved:7;
> > +	};
> > +};
> > +
> > +/* Define port identifier type. */
> > +union hv_port_id {
> > +	__u32 asu32;
> > +	struct {
> > +		__u32 id:24;
> > +		__u32 reserved:8;
> > +	} u;
> > +};
> 
> Hmm, I thought bitfields weren't a good idea for this sort of thing. Isn't
> it up to the compiler how they are laid out in memory?

There's just been a discussion about this on the x86 side.  See
https://lkml.org/lkml/2018/11/27/116 and 
https://lkml.org/lkml/2018/11/30/848.   The data structures are
also typically defined as bitfields in the Hyper-V TLFS.  The decision was
to keep the bitfields but add __packed.  I'll do the same in the
next version of this patch.

> 
> > +
> > +/* Define synthetic interrupt controller message header. */
> > +struct hv_message_header {
> > +	__u32 message_type;
> > +	__u8 payload_size;
> > +	union hv_message_flags message_flags;
> > +	__u8 reserved[2];
> > +	union {
> > +		__u64 sender;
> > +		union hv_port_id port;
> > +	};
> > +};
> > +
> > +/* Define synthetic interrupt controller message format. */
> > +struct hv_message {
> > +	struct hv_message_header header;
> > +	union {
> > +		__u64 payload[HV_MESSAGE_PAYLOAD_QWORD_COUNT];
> > +	} u;
> > +};
> > +
> > +/* Define the synthetic interrupt message page layout. */
> > +struct hv_message_page {
> > +	struct hv_message sint_message[HV_SYNIC_SINT_COUNT];
> > +};
> > +
> > +/* Define timer message payload structure. */
> > +struct hv_timer_message_payload {
> > +	__u32 timer_index;
> > +	__u32 reserved;
> > +	__u64 expiration_time;	/* When the timer expired */
> > +	__u64 delivery_time;	/* When the message was delivered */
> > +};
> > +
> > +#define HV_STIMER_ENABLE		(1ULL << 0)
> > +#define HV_STIMER_PERIODIC		(1ULL << 1)
> > +#define HV_STIMER_LAZY			(1ULL << 2)
> > +#define HV_STIMER_AUTOENABLE		(1ULL << 3)
> > +#define HV_STIMER_SINT(config)		(__u8)(((config) >> 16) & 0x0F)
> > +
> > +#endif
> > diff --git a/arch/arm64/include/asm/mshyperv.h b/arch/arm64/include/asm/mshyperv.h
> > new file mode 100644
> > index 000000000000..a87c431d58b3
> > --- /dev/null
> > +++ b/arch/arm64/include/asm/mshyperv.h
> > @@ -0,0 +1,116 @@
> > +/* SPDX-License-Identifier: GPL-2.0 */
> > +
> > +/*
> > + * Linux-specific definitions for managing interactions with Microsoft's
> > + * Hyper-V hypervisor. The definitions in this file are specific to
> > + * the ARM64 architecture.  See include/asm-generic/mshyperv.h for
> > + * definitions are that architecture independent.
> > + *
> > + * Definitions that are specified in the Hyper-V Top Level Functional
> > + * Spec (TLFS) should not go in this file, but should instead go in
> > + * hyperv-tlfs.h.
> > + *
> > + * Copyright (C) 2018, Microsoft, Inc.
> > + *
> > + * Author : Michael Kelley <mikelley@microsoft.com>
> > + *
> > + * This program is free software; you can redistribute it and/or modify it
> > + * under the terms of the GNU General Public License version 2 as published
> > + * by the Free Software Foundation.
> > + *
> > + * This program is distributed in the hope that it will be useful, but
> > + * WITHOUT ANY WARRANTY; without even the implied warranty of
> > + * MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE, GOOD TITLE or
> > + * NON INFRINGEMENT.  See the GNU General Public License for more
> > + * details.
> > + */
> > +
> > +#ifndef _ASM_ARM64_MSHYPERV_H
> > +#define _ASM_ARM64_MSHYPERV_H
> > +
> > +#include <linux/types.h>
> > +#include <linux/interrupt.h>
> > +#include <linux/clocksource.h>
> > +#include <linux/irq.h>
> > +#include <linux/irqdesc.h>
> > +#include <asm/hyperv-tlfs.h>
> > +
> > +/*
> > + * Define the IRQ numbers/vectors used by Hyper-V VMbus interrupts
> > + * and by STIMER0 Direct Mode interrupts. Hyper-V should be supplying
> > + * these values through ACPI, but there are no other interrupting
> > + * devices in a Hyper-V VM on ARM64, so it's OK to hard code for now.
> > + * The "CALLBACK_VECTOR" terminology is a left-over from the x86/x64
> > + * world that is used in architecture independent Hyper-V code.
> > + */
> > +#define HYPERVISOR_CALLBACK_VECTOR 16
> > +#define	HV_STIMER0_IRQNR	   17
> > +
> > +extern u64 hv_do_hvc(u64 control, ...);
> > +extern u64 hv_do_hvc_fast_get(u64 control, u64 input1, u64 input2, u64 input3,
> > +		struct hv_get_vp_register_output *output);
> > +
> > +/*
> > + * Declare calls to get and set Hyper-V VP register values on ARM64, which
> > + * requires a hypercall.
> > + */
> > +extern void hv_set_vpreg(u32 reg, u64 value);
> > +extern u64 hv_get_vpreg(u32 reg);
> > +extern void hv_get_vpreg_128(u32 reg, struct hv_get_vp_register_output *result);
> > +
> > +/*
> > + * Use the Hyper-V provided stimer0 as the timer that is made
> > + * available to the architecture independent Hyper-V drivers.
> > + */
> > +#define hv_init_timer(timer, tick) \
> > +		hv_set_vpreg(HV_REGISTER_STIMER0_COUNT + (2*timer), tick)
> > +#define hv_init_timer_config(timer, val) \
> > +		hv_set_vpreg(HV_REGISTER_STIMER0_CONFIG + (2*timer), val)
> > +#define hv_get_current_tick(tick) \
> > +		(tick = hv_get_vpreg(HV_REGISTER_TIME_REFCOUNT))
> > +
> > +#define hv_get_simp(val) (val = hv_get_vpreg(HV_REGISTER_SIPP))
> > +#define hv_set_simp(val) hv_set_vpreg(HV_REGISTER_SIPP, val)
> > +
> > +#define hv_get_siefp(val) (val = hv_get_vpreg(HV_REGISTER_SIFP))
> > +#define hv_set_siefp(val) hv_set_vpreg(HV_REGISTER_SIFP, val)
> > +
> > +#define hv_get_synic_state(val) (val = hv_get_vpreg(HV_REGISTER_SCONTROL))
> > +#define hv_set_synic_state(val) hv_set_vpreg(HV_REGISTER_SCONTROL, val)
> > +
> > +#define hv_get_vp_index(index) (index = hv_get_vpreg(HV_REGISTER_VPINDEX))
> > +
> > +#define hv_signal_eom()	hv_set_vpreg(HV_REGISTER_EOM, 0)
> > +
> > +/*
> > + * Hyper-V SINT registers are numbered sequentially, so we can just
> > + * add the SINT number to the register number of SINT0
> > + */
> > +#define hv_get_synint_state(sint_num, val) \
> > +		(val = hv_get_vpreg(HV_REGISTER_SINT0 + sint_num))
> > +#define hv_set_synint_state(sint_num, val) \
> > +		hv_set_vpreg(HV_REGISTER_SINT0 + sint_num, val)
> > +
> > +#define hv_get_crash_ctl(val) \
> > +		(val = hv_get_vpreg(HV_REGISTER_CRASH_CTL))
> > +
> > +#if IS_ENABLED(CONFIG_HYPERV)
> > +#define hv_enable_stimer0_percpu_irq(irq)	enable_percpu_irq(irq, 0)
> > +#define hv_disable_stimer0_percpu_irq(irq)	disable_percpu_irq(irq)
> > +#endif
> > +
> > +/* ARM64 specific code to read the hardware clock */
> > +static inline u64 hv_read_hwclock(void)
> > +{
> > +	u64 result;
> > +
> > +	isb();
> > +	result = read_sysreg(cntvct_el0);
> > +	isb();
> > +
> > +	return result;
> > +}
> 
> Shouldn't you use arch_timer_read_counter() or arch_counter_get_cntvct()
> instead?

I don't think so.  Hyper-V provides its own clocksource on ARM64 like it does
x86.  It is not the standard ARM64 arch timer, and the ARM64 arch timer driver
that implements arch_timer_read_counter() and arch_counter_get_cntvct() is
not used.

> 
> > +#include <asm-generic/mshyperv.h>
> > +
> > +#endif
> > diff --git a/include/asm-generic/mshyperv.h b/include/asm-generic/mshyperv.h
> > new file mode 100644
> > index 000000000000..fbe1ec89c85a
> > --- /dev/null
> > +++ b/include/asm-generic/mshyperv.h
> > @@ -0,0 +1,240 @@
> > +/* SPDX-License-Identifier: GPL-2.0 */
> > +
> > +/*
> > + * Linux-specific definitions for managing interactions with Microsoft's
> > + * Hyper-V hypervisor. The definitions in this file are architecture
> > + * independent. See arch/<arch>/include/asm/mshyperv.h for definitions
> > + * that are specific to architecture <arch>.
> > + *
> > + * Definitions that are specified in the Hyper-V Top Level Functional
> > + * Spec (TLFS) should not go in this file, but should instead go in
> > + * hyperv-tlfs.h.
> > + *
> > + * Copyright (C) 2018, Microsoft, Inc.
> > + *
> > + * Author : Michael Kelley <mikelley@microsoft.com>
> > + *
> > + * This program is free software; you can redistribute it and/or modify it
> > + * under the terms of the GNU General Public License version 2 as published
> > + * by the Free Software Foundation.
> > + *
> > + * This program is distributed in the hope that it will be useful, but
> > + * WITHOUT ANY WARRANTY; without even the implied warranty of
> > + * MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE, GOOD TITLE or
> > + * NON INFRINGEMENT.  See the GNU General Public License for more
> > + * details.
> > + */
> > +
> > +#ifndef _ASM_GENERIC_MSHYPERV_H
> > +#define _ASM_GENERIC_MSHYPERV_H
> > +
> > +#include <linux/types.h>
> > +#include <linux/interrupt.h>
> > +#include <linux/clocksource.h>
> > +#include <linux/irq.h>
> > +#include <linux/irqdesc.h>
> > +#include <asm/hyperv-tlfs.h>
> > +
> > +/*
> > + * Hyper-V always runs with a page size of 4096. These definitions
> > + * are used when communicating with Hyper-V using guest physical
> > + * pages and guest physical page addresses, since the guest page
> > + * size may not be 4096 on ARM64.
> > + */
> > +#define HV_HYP_PAGE_SIZE	4096
> > +#define HV_HYP_PAGE_SHIFT	12
> 
> Maybe define the page size in terms of the shift?

Good idea.  I'll make that change.

> 
> > +#define HV_HYP_PAGE_MASK	(~(HV_HYP_PAGE_SIZE - 1))
> > +
> > +
> > +struct ms_hyperv_info {
> > +	u32 features;
> > +	u32 misc_features;
> > +	u32 hints;
> > +	u32 max_vp_index;
> > +	u32 max_lp_index;
> > +};
> 
> What is the define endianness for in-memory structures shared with the
> hypervisor?

It is little endian.  I'll add a comment to that effect as it is effectively
part of the ABI.

> 
> > +extern struct ms_hyperv_info ms_hyperv;
> > +
> > +extern u64 hv_do_hypercall(u64 control, void *inputaddr, void *outputaddr);
> > +extern u64 hv_do_fast_hypercall8(u16 control, u64 input8);
> > +
> > +/*
> > + * The guest OS needs to register the guest ID with the hypervisor.
> > + * The guest ID is a 64 bit entity and the structure of this ID is
> > + * specified in the Hyper-V specification:
> > + *
> > + * msdn.microsoft.com/en-us/library/windows/hardware/ff542653%28v=vs.85%29.aspx
> > + *
> > + * While the current guideline does not specify how Linux guest ID(s)
> > + * need to be generated, our plan is to publish the guidelines for
> > + * Linux and other guest operating systems that currently are hosted
> > + * on Hyper-V. The implementation here conforms to this yet
> > + * unpublished guidelines.
> > + *
> > + *
> > + * Bit(s)
> > + * 63 - Indicates if the OS is Open Source or not; 1 is Open Source
> > + * 62:56 - Os Type; Linux is 0x100
> > + * 55:48 - Distro specific identification
> > + * 47:16 - Linux kernel version number
> > + * 15:0  - Distro specific identification
> 
> This looks familiar...

Yes.  I'll remove the duplicate copy in the other file.

> 
> > + * Generate the guest ID based on the guideline described above.
> > + */
> > +
> > +static inline  __u64 generate_guest_id(__u64 d_info1, __u64 kernel_version,
> > +				       __u64 d_info2)
> > +{
> > +	__u64 guest_id = 0;
> > +
> > +	guest_id = (((__u64)HV_LINUX_VENDOR_ID) << 48);
> > +	guest_id |= (d_info1 << 48);
> > +	guest_id |= (kernel_version << 16);
> > +	guest_id |= d_info2;
> > +
> > +	return guest_id;
> > +}
> > +
> > +
> > +/* Free the message slot and signal end-of-message if required */
> > +static inline void vmbus_signal_eom(struct hv_message *msg, u32 old_msg_type)
> > +{
> > +	/*
> > +	 * On crash we're reading some other CPU's message page and we need
> > +	 * to be careful: this other CPU may already had cleared the header
> > +	 * and the host may already had delivered some other message there.
> > +	 * In case we blindly write msg->header.message_type we're going
> > +	 * to lose it. We can still lose a message of the same type but
> > +	 * we count on the fact that there can only be one
> > +	 * CHANNELMSG_UNLOAD_RESPONSE and we don't care about other messages
> > +	 * on crash.
> > +	 */
> > +	if (cmpxchg(&msg->header.message_type, old_msg_type,
> > +		    HVMSG_NONE) != old_msg_type)
> > +		return;
> > +
> > +	/*
> > +	 * Make sure the write to MessageType (ie set to
> > +	 * HVMSG_NONE) happens before we read the
> > +	 * MessagePending and EOMing. Otherwise, the EOMing
> > +	 * will not deliver any more messages since there is
> > +	 * no empty slot
> > +	 */
> > +	mb();
> 
> A successful cmpxchg() already implies full barriers, so this isn't needed.
> I also wonder whether cmpxchg_acquire() would be sufficient here.

OK.  I'll investigate the use of cmpxchg_acquire().

> 
> > +
> > +	if (msg->header.message_flags.msg_pending) {
> > +		/*
> > +		 * This will cause message queue rescan to
> > +		 * possibly deliver another msg from the
> > +		 * hypervisor
> > +		 */
> > +		hv_signal_eom();
> > +	}
> > +}
> > +
> > +void hv_setup_vmbus_irq(void (*handler)(void));
> > +void hv_remove_vmbus_irq(void);
> > +void hv_enable_vmbus_irq(void);
> > +void hv_disable_vmbus_irq(void);
> > +
> > +void hv_setup_kexec_handler(void (*handler)(void));
> > +void hv_remove_kexec_handler(void);
> > +void hv_setup_crash_handler(void (*handler)(struct pt_regs *regs));
> > +void hv_remove_crash_handler(void);
> > +
> > +#if IS_ENABLED(CONFIG_HYPERV)
> > +extern struct clocksource *hyperv_cs;
> > +
> > +/*
> > + * Hypervisor's notion of virtual processor ID is different from
> > + * Linux' notion of CPU ID. This information can only be retrieved
> > + * in the context of the calling CPU. Setup a map for easy access
> > + * to this information.
> > + */
> > +extern u32 *hv_vp_index;
> > +extern u32 hv_max_vp_index;
> > +
> > +/* Sentinel value for an uninitialized entry in hv_vp_index array */
> > +#define VP_INVAL	U32_MAX
> > +
> > +/**
> > + * hv_cpu_number_to_vp_number() - Map CPU to VP.
> > + * @cpu_number: CPU number in Linux terms
> > + *
> > + * This function returns the mapping between the Linux processor
> > + * number and the hypervisor's virtual processor number, useful
> > + * in making hypercalls and such that talk about specific
> > + * processors.
> > + *
> > + * Return: Virtual processor number in Hyper-V terms
> > + */
> > +static inline int hv_cpu_number_to_vp_number(int cpu_number)
> > +{
> > +	return hv_vp_index[cpu_number];
> > +}
> > +
> > +void hyperv_report_panic(struct pt_regs *regs, long err);
> > +void hyperv_report_panic_msg(phys_addr_t pa, size_t size);
> > +bool hv_is_hyperv_initialized(void);
> > +void hyperv_cleanup(void);
> > +#else /* CONFIG_HYPERV */
> > +static inline bool hv_is_hyperv_initialized(void) { return false; }
> > +static inline void hyperv_cleanup(void) {}
> > +#endif /* CONFIG_HYPERV */
> > +
> > +#if IS_ENABLED(CONFIG_HYPERV)
> > +extern int hv_setup_stimer0_irq(int *irq, int *vector, void (*handler)(void));
> > +extern void hv_remove_stimer0_irq(int irq);
> > +#endif
> > +
> > +static inline u64 hv_read_tsc_page_tsc(const struct ms_hyperv_tsc_page *tsc_pg,
> > +				       u64 *cur_tsc)
> > +{
> > +	u64	scale, offset;
> > +	u32	sequence;
> > +
> > +	/*
> > +	 * The protocol for reading Hyper-V TSC page is specified in Hypervisor
> > +	 * Top-Level Functional Specification.  To get the reference time we
> > +	 * must do the following:
> > +	 * - READ ReferenceTscSequence
> > +	 *   A special '0' value indicates the time source is unreliable and we
> > +	 *   need to use something else.
> > +	 * - ReferenceTime =
> > +	 *     ((HWclock val) * ReferenceTscScale) >> 64) + ReferenceTscOffset
> > +	 * - READ ReferenceTscSequence again. In case its value has changed
> > +	 *   since our first reading we need to discard ReferenceTime and repeat
> > +	 *   the whole sequence as the hypervisor was updating the page in
> > +	 *   between.
> > +	 */
> > +	do {
> > +		sequence = READ_ONCE(tsc_pg->tsc_sequence);
> > +		/*
> > +		 * Make sure we read sequence before we read other values from
> > +		 * TSC page.
> > +		 */
> > +		smp_rmb();
> > +
> > +		scale = READ_ONCE(tsc_pg->tsc_scale);
> > +		offset = READ_ONCE(tsc_pg->tsc_offset);
> > +		*cur_tsc = hv_read_hwclock();
> > +
> > +		/*
> > +		 * Make sure we read sequence after we read all other values
> > +		 * from TSC page.
> > +		 */
> > +		smp_rmb();
> > +
> > +	} while (READ_ONCE(tsc_pg->tsc_sequence) != sequence);
> 
> Could you explain what the writer side looks like, please? I'm a bit
> confused as to why we're not using the bottom bit of the sequence
> number to detect a concurrent update.

The writer side, of course, is Hyper-V.  The algorithm used follows the
code in Section 12.6.3 of the Hyper-V TLFS, though as pointed out in a
separate email by Vitaly Kuznetsov, I seemed to have dropped the test for
sequence value 0 when moving the code over from the x86 side.  I'll add
that back in.

I'm not clear on your thought about using the bottom bit of the sequence
number.   Is that better than just comparing the full 32-bit value?

Thanks for the review and all the great comments ...

Michael

> 
> Will

RE: [PATCH 1/4] arm64: hyperv: Add core Hyper-V include files

[Michael Kelley]

From: Will Deacon <will.deacon@arm.com> Sent: Friday, December 7, 2018 5:43 AM

> > hyperv-tlfs.h defines Hyper-V interfaces from the Hyper-V Top Level
> > Functional Spec (TLFS). The TLFS is distinctly oriented to x86/x64,
> > and Hyper-V has not separated out the architecture-dependent parts into
> > x86/x64 vs. ARM64. So hyperv-tlfs.h includes information for ARM64
> > that is not yet formally published. The TLFS is available here:
> 
> When do you plan to publish the spec? It's pretty hard to review this stuff
> without knowing what it's supposed to look like.

I don't have a commitment from the Hyper-V team on when an updated TLFS
that covers ARM64 will be published.  I'm on the Linux side, and Hyper-V is a
separate group, but I'll raise the topic again with them.

> 
> >   docs.microsoft.com/en-us/virtualization/hyper-v-on-windows/reference/tlfs
> >
> > mshyperv.h defines Linux-specific structures and routines for
> > interacting with Hyper-V. It is split into an ARM64 specific file
> > and an architecture independent file in include/asm-generic.
> >
> > Signed-off-by: Michael Kelley <mikelley@microsoft.com>
> > Signed-off-by: K. Y. Srinivasan <kys@microsoft.com>
> > ---
> >  MAINTAINERS                          |   3 +
> >  arch/arm64/include/asm/hyperv-tlfs.h | 338 +++++++++++++++++++++++++++
> >  arch/arm64/include/asm/mshyperv.h    | 116 +++++++++
> >  include/asm-generic/mshyperv.h       | 240 +++++++++++++++++++
> >  4 files changed, 697 insertions(+)
> >  create mode 100644 arch/arm64/include/asm/hyperv-tlfs.h
> >  create mode 100644 arch/arm64/include/asm/mshyperv.h
> >  create mode 100644 include/asm-generic/mshyperv.h
> >
> > diff --git a/MAINTAINERS b/MAINTAINERS
> > index f4855974f325..72f19cef4c48 100644
> > --- a/MAINTAINERS
> > +++ b/MAINTAINERS
> > @@ -6835,6 +6835,8 @@ F:	arch/x86/include/asm/trace/hyperv.h
> >  F:	arch/x86/include/asm/hyperv-tlfs.h
> >  F:	arch/x86/kernel/cpu/mshyperv.c
> >  F:	arch/x86/hyperv
> > +F:	arch/arm64/include/asm/hyperv-tlfs.h
> > +F:	arch/arm64/include/asm/mshyperv.h
> >  F:	drivers/hid/hid-hyperv.c
> >  F:	drivers/hv/
> >  F:	drivers/input/serio/hyperv-keyboard.c
> > @@ -6846,6 +6848,7 @@ F:	drivers/video/fbdev/hyperv_fb.c
> >  F:	net/vmw_vsock/hyperv_transport.c
> >  F:	include/linux/hyperv.h
> >  F:	include/uapi/linux/hyperv.h
> > +F:	include/asm-generic/mshyperv.h
> >  F:	tools/hv/
> >  F:	Documentation/ABI/stable/sysfs-bus-vmbus
> >
> > diff --git a/arch/arm64/include/asm/hyperv-tlfs.h b/arch/arm64/include/asm/hyperv-
> tlfs.h
> > new file mode 100644
> > index 000000000000..924e37600e92
> > --- /dev/null
> > +++ b/arch/arm64/include/asm/hyperv-tlfs.h
> > @@ -0,0 +1,338 @@
> > +/* SPDX-License-Identifier: GPL-2.0 */
> > +
> > +/*
> > + * This file contains definitions from the Hyper-V Hypervisor Top-Level
> > + * Functional Specification (TLFS):
> > + *
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.microsoft.co
> m%2Fen-us%2Fvirtualization%2Fhyper-v-on-
> windows%2Freference%2Ftlfs&amp;data=02%7C01%7Cmikelley%40microsoft.com%7Ce1b
> dbb31db064623174f08d65c49cc3b%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C63
> 6797869386921804&amp;sdata=RiD05cDWC%2FPnXnis6U7EcfEfCjvb54uuKHRfifQhMEM%3D
> &amp;reserved=0
> 
> As mentioned elsewhere, please use a better link here and drop the license
> boilerplate below.

Agreed.  Will do so here and in other files in the next version of the patch.

> 
> > + *
> > + * Copyright (C) 2018, Microsoft, Inc.
> > + *
> > + * Author : Michael Kelley <mikelley@microsoft.com>
> > + *
> > + * This program is free software; you can redistribute it and/or modify it
> > + * under the terms of the GNU General Public License version 2 as published
> > + * by the Free Software Foundation.
> > + *
> > + * This program is distributed in the hope that it will be useful, but
> > + * WITHOUT ANY WARRANTY; without even the implied warranty of
> > + * MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE, GOOD TITLE or
> > + * NON INFRINGEMENT.  See the GNU General Public License for more
> > + * details.
> > + */
> > +
> > +#ifndef _ASM_ARM64_HYPERV_H
> > +#define _ASM_ARM64_HYPERV_H
> 
> __ASM_HYPER_V_H please

OK.

> 
> > +
> > +#include <linux/types.h>
> > +
> > +/*
> > + * These Hyper-V registers provide information equivalent to the CPUID
> > + * instruction on x86/x64.
> > + */
> > +#define HV_REGISTER_HYPERVISOR_VERSION		0x00000100 /*CPUID
> 0x40000002 */
> > +#define	HV_REGISTER_PRIVILEGES_AND_FEATURES	0x00000200 /*CPUID
> 0x40000003 */
> > +#define	HV_REGISTER_FEATURES			0x00000201 /*CPUID
> 0x40000004 */
> > +#define	HV_REGISTER_IMPLEMENTATION_LIMITS	0x00000202 /*CPUID
> 0x40000005 */
> > +#define HV_ARM64_REGISTER_INTERFACE_VERSION	0x00090006 /*CPUID
> 0x40000001 */
> > +
> > +/*
> > + * Feature identification. HvRegisterPrivilegesAndFeaturesInfo returns a
> > + * 128-bit value with flags indicating which features are available to the
> > + * partition based upon the current partition privileges. The 128-bit
> > + * value is broken up with different portions stored in different 32-bit
> > + * fields in the ms_hyperv structure.
> > + */
> > +
> > +/* Partition Reference Counter available*/
> > +#define HV_MSR_TIME_REF_COUNT_AVAILABLE		(1 << 1)
> > +
> > +/*
> > + * Synthetic Timers available
> > + */
> > +#define HV_MSR_SYNTIMER_AVAILABLE		(1 << 3)
> > +
> > +/* Frequency MSRs available */
> > +#define HV_FEATURE_FREQUENCY_MSRS_AVAILABLE	(1 << 8)
> > +
> > +/* Reference TSC available */
> > +#define HV_MSR_REFERENCE_TSC_AVAILABLE		(1 << 9)
> > +
> > +/* Crash MSR available */
> > +#define HV_FEATURE_GUEST_CRASH_MSR_AVAILABLE	(1 << 10)
> > +
> > +
> > +/*
> > + * This group of flags is in the high order 64-bits of the returned
> > + * 128-bit value.
> > + */
> > +
> > +/* STIMER direct mode is available */
> > +#define HV_STIMER_DIRECT_MODE_AVAILABLE		(1 << 19)
> > +
> > +/*
> > + * Implementation recommendations in register
> > + * HvRegisterFeaturesInfo. Indicates which behaviors the hypervisor
> > + * recommends the OS implement for optimal performance.
> > + */
> > +
> > +/*
> > + * Recommend not using Auto EOI
> > + */
> > +#define HV_DEPRECATING_AEOI_RECOMMENDED		(1 << 9)
> > +
> > +/*
> > + * Synthetic register definitions equivalent to MSRs on x86/x64
> > + */
> > +#define HV_REGISTER_CRASH_P0		0x00000210
> > +#define HV_REGISTER_CRASH_P1		0x00000211
> > +#define HV_REGISTER_CRASH_P2		0x00000212
> > +#define HV_REGISTER_CRASH_P3		0x00000213
> > +#define HV_REGISTER_CRASH_P4		0x00000214
> > +#define HV_REGISTER_CRASH_CTL		0x00000215
> > +
> > +#define HV_REGISTER_GUEST_OSID		0x00090002
> > +#define HV_REGISTER_VPINDEX		0x00090003
> > +#define HV_REGISTER_TIME_REFCOUNT	0x00090004
> > +#define HV_REGISTER_REFERENCE_TSC	0x00090017
> > +
> > +#define HV_REGISTER_SINT0		0x000A0000
> > +#define HV_REGISTER_SINT1		0x000A0001
> > +#define HV_REGISTER_SINT2		0x000A0002
> > +#define HV_REGISTER_SINT3		0x000A0003
> > +#define HV_REGISTER_SINT4		0x000A0004
> > +#define HV_REGISTER_SINT5		0x000A0005
> > +#define HV_REGISTER_SINT6		0x000A0006
> > +#define HV_REGISTER_SINT7		0x000A0007
> > +#define HV_REGISTER_SINT8		0x000A0008
> > +#define HV_REGISTER_SINT9		0x000A0009
> > +#define HV_REGISTER_SINT10		0x000A000A
> > +#define HV_REGISTER_SINT11		0x000A000B
> > +#define HV_REGISTER_SINT12		0x000A000C
> > +#define HV_REGISTER_SINT13		0x000A000D
> > +#define HV_REGISTER_SINT14		0x000A000E
> > +#define HV_REGISTER_SINT15		0x000A000F
> > +#define HV_REGISTER_SCONTROL		0x000A0010
> > +#define HV_REGISTER_SVERSION		0x000A0011
> > +#define HV_REGISTER_SIFP		0x000A0012
> > +#define HV_REGISTER_SIPP		0x000A0013
> > +#define HV_REGISTER_EOM			0x000A0014
> > +#define HV_REGISTER_SIRBP		0x000A0015
> > +
> > +#define HV_REGISTER_STIMER0_CONFIG	0x000B0000
> > +#define HV_REGISTER_STIMER0_COUNT	0x000B0001
> > +#define HV_REGISTER_STIMER1_CONFIG	0x000B0002
> > +#define HV_REGISTER_STIMER1_COUNT	0x000B0003
> > +#define HV_REGISTER_STIMER2_CONFIG	0x000B0004
> > +#define HV_REGISTER_STIMER2_COUNT	0x000B0005
> > +#define HV_REGISTER_STIMER3_CONFIG	0x000B0006
> > +#define HV_REGISTER_STIMER3_COUNT	0x000B0007
> > +
> > +/*
> > + * Crash notification flags.
> > + */
> > +#define HV_CRASH_CTL_CRASH_NOTIFY_MSG	BIT_ULL(62)
> > +#define HV_CRASH_CTL_CRASH_NOTIFY	BIT_ULL(63)
> > +
> > +/*
> > + * The guest OS needs to register the guest ID with the hypervisor.
> > + * The guest ID is a 64 bit entity and the structure of this ID is
> > + * specified in the Hyper-V specification:
> > + *
> > + * msdn.microsoft.com/en-us/library/windows/hardware/ff542653%28v=vs.85%29.aspx
> 
> Dead link :(

Hmmm.  It looks like that doc is gone and the info is now in the TLFS.  This
comment block shouldn't be here anyway.  As you noted later, there's another
copy in include/asm-generic/mshyperv.h, which is the right place.  I'll
get this cleaned up in the next version of the patch.

> 
> > + *
> > + * While the current guideline does not specify how Linux guest ID(s)
> > + * need to be generated, our plan is to publish the guidelines for
> > + * Linux and other guest operating systems that currently are hosted
> > + * on Hyper-V. The implementation here conforms to this yet
> > + * unpublished guidelines.
> 
> Again, any timeline on publishing this stuff? Right now, this is just a file
> full of magic numbers and I'm not sure we're in a good position to maintain
> it based on the idea that you have a cunning plan.

Fair point.

> 
> > + *
> > + *
> > + * Bit(s)
> > + * 63 - Indicates if the OS is Open Source or not; 1 is Open Source
> > + * 62:56 - Os Type; Linux is 0x100
> > + * 55:48 - Distro specific identification
> > + * 47:16 - Linux kernel version number
> > + * 15:0  - Distro specific identification
> > + *
> > + *
> > + */
> > +#define HV_LINUX_VENDOR_ID              0x8100
> > +
> > +/* Declare the various hypercall operations. */
> > +#define HVCALL_FLUSH_VIRTUAL_ADDRESS_SPACE	0x0002
> > +#define HVCALL_FLUSH_VIRTUAL_ADDRESS_LIST	0x0003
> > +#define HVCALL_NOTIFY_LONG_SPIN_WAIT		0x0008
> > +#define HVCALL_SEND_IPI				0x000b
> > +#define HVCALL_FLUSH_VIRTUAL_ADDRESS_SPACE_EX	0x0013
> > +#define HVCALL_FLUSH_VIRTUAL_ADDRESS_LIST_EX	0x0014
> > +#define HVCALL_SEND_IPI_EX			0x0015
> > +#define HVCALL_GET_VP_REGISTERS			0x0050
> > +#define HVCALL_SET_VP_REGISTERS			0x0051
> > +#define HVCALL_POST_MESSAGE			0x005c
> > +#define HVCALL_SIGNAL_EVENT			0x005d
> > +#define HVCALL_RETARGET_INTERRUPT		0x007e
> > +#define HVCALL_START_VIRTUAL_PROCESSOR		0x0099
> > +#define HVCALL_GET_VP_INDEX_FROM_APICID		0x009a
> 
> This all sounds very x86y...

Indeed it is.  Actually, I can delete this definition as the
GET_VP_INDEX_FROM_APICID hypercall is not used on ARM64.

> 
> > +/* Declare standard hypercall field values. */
> > +#define HV_PARTITION_ID_SELF                    ((u64)-1)
> > +#define HV_VP_INDEX_SELF                        ((u32)-2)
> > +
> > +#define HV_HYPERCALL_FAST_BIT                   BIT(16)
> > +#define HV_HYPERCALL_REP_COUNT_1                BIT_ULL(32)
> > +#define HV_HYPERCALL_RESULT_MASK                GENMASK_ULL(15, 0)
> > +
> > +/* Define the hypercall status result */
> > +
> > +union hv_hypercall_status {
> > +	u64 as_uint64;
> > +	struct {
> > +		u16 status;
> > +		u16 reserved;
> > +		u16 reps_completed;  /* Low 12 bits */
> > +		u16 reserved2;
> > +	};
> > +};
> > +
> > +/* hypercall status code */
> > +#define HV_STATUS_SUCCESS			0
> > +#define HV_STATUS_INVALID_HYPERCALL_CODE	2
> > +#define HV_STATUS_INVALID_HYPERCALL_INPUT	3
> > +#define HV_STATUS_INVALID_ALIGNMENT		4
> > +#define HV_STATUS_INSUFFICIENT_MEMORY		11
> > +#define HV_STATUS_INVALID_CONNECTION_ID		18
> > +#define HV_STATUS_INSUFFICIENT_BUFFERS		19
> > +
> > +/* Define output layout for Get VP Register hypercall */
> > +struct hv_get_vp_register_output {
> > +	u64 registervaluelow;
> > +	u64 registervaluehigh;
> > +};
> > +
> > +#define HV_FLUSH_ALL_PROCESSORS			BIT(0)
> > +#define HV_FLUSH_ALL_VIRTUAL_ADDRESS_SPACES	BIT(1)
> > +#define HV_FLUSH_NON_GLOBAL_MAPPINGS_ONLY	BIT(2)
> > +#define HV_FLUSH_USE_EXTENDED_RANGE_FORMAT	BIT(3)
> > +
> > +enum HV_GENERIC_SET_FORMAT {
> > +	HV_GENERIC_SET_SPARSE_4K,
> > +	HV_GENERIC_SET_ALL,
> > +};
> > +
> > +/*
> > + * The Hyper-V TimeRefCount register and the TSC
> > + * page provide a guest VM clock with 100ns tick rate
> > + */
> > +#define HV_CLOCK_HZ (NSEC_PER_SEC/100)
> > +
> > +/*
> > + * The fields in this structure are set by Hyper-V and read
> > + * by the Linux guest.  They should be accessed with READ_ONCE()
> > + * so the compiler doesn't optimize in a way that will cause
> > + * problems.
> > + */
> > +struct ms_hyperv_tsc_page {
> > +	u32 tsc_sequence;
> > +	u32 reserved1;
> > +	u64 tsc_scale;
> > +	s64 tsc_offset;
> > +	u64 reserved2[509];
> > +};
> 
> It might be clearer to express this as a union with the page size:
> 
> struct ms_hyperv_tsc_page {
> 	union {
> 		struct {
> 			__u32	tsc_sequence;
> 			__u32	reserved1;
> 		};
> 		__u8	reserved2[HV_HYP_PAGE_SIZE];
> 	};
> };
> 
> What's the required alignment on this structure?
> Also, do you intend for 32-bit guests to use this ABI as well?

The definition for struct ms_hyperv_tsc_page matches what
is written in section 12.6.2 of the Hyper-V TLFS, so I'm a little
reluctant to change it.  Alignment must be to Hyper-V's
4 Kbyte page size and the memory allocation for the structure
ensures that alignment when the guest page size is 4K.  In the
future, in order to work with Hyper-V, there's going to
be a general requirement for ARM64 guests to be able to allocate
4 Kbytes of memory that is 4 Kbyte aligned, even when we add
support for 16K/64K page sizes in the guest.  At the moment these
larger page sizes don't work due to incorrect assumptions about
page size in Hyper-V drivers that are architecture independent.
Fixing those mis-assumptions will be a separate set of patches.

There's no intent to support 32-bit guests running on Hyper-V for ARM64.

> 
> > +
> > +/* Define the number of synthetic interrupt sources. */
> > +#define HV_SYNIC_SINT_COUNT		(16)
> > +/* Define the expected SynIC version. */
> > +#define HV_SYNIC_VERSION_1		(0x1)
> > +
> > +#define HV_SYNIC_CONTROL_ENABLE		(1ULL << 0)
> > +#define HV_SYNIC_SIMP_ENABLE		(1ULL << 0)
> > +#define HV_SYNIC_SIEFP_ENABLE		(1ULL << 0)
> > +#define HV_SYNIC_SINT_MASKED		(1ULL << 16)
> > +#define HV_SYNIC_SINT_AUTO_EOI		(1ULL << 17)
> > +#define HV_SYNIC_SINT_VECTOR_MASK	(0xFF)
> > +
> > +#define HV_SYNIC_STIMER_COUNT		(4)
> > +
> > +/* Define synthetic interrupt controller message constants. */
> > +#define HV_MESSAGE_SIZE			(256)
> > +#define HV_MESSAGE_PAYLOAD_BYTE_COUNT	(240)
> > +#define HV_MESSAGE_PAYLOAD_QWORD_COUNT	(30)
> > +
> > +/* Define hypervisor message types. */
> > +enum hv_message_type {
> > +	HVMSG_NONE			= 0x00000000,
> > +
> > +	/* Memory access messages. */
> > +	HVMSG_UNMAPPED_GPA		= 0x80000000,
> > +	HVMSG_GPA_INTERCEPT		= 0x80000001,
> > +
> > +	/* Timer notification messages. */
> > +	HVMSG_TIMER_EXPIRED		= 0x80000010,
> > +
> > +	/* Error messages. */
> > +	HVMSG_INVALID_VP_REGISTER_VALUE	= 0x80000020,
> > +	HVMSG_UNRECOVERABLE_EXCEPTION	= 0x80000021,
> > +	HVMSG_UNSUPPORTED_FEATURE	= 0x80000022,
> > +
> > +	/* Trace buffer complete messages. */
> > +	HVMSG_EVENTLOG_BUFFERCOMPLETE	= 0x80000040,
> > +};
> > +
> > +/* Define synthetic interrupt controller message flags. */
> > +union hv_message_flags {
> > +	__u8 asu8;
> > +	struct {
> > +		__u8 msg_pending:1;
> > +		__u8 reserved:7;
> > +	};
> > +};
> > +
> > +/* Define port identifier type. */
> > +union hv_port_id {
> > +	__u32 asu32;
> > +	struct {
> > +		__u32 id:24;
> > +		__u32 reserved:8;
> > +	} u;
> > +};
> 
> Hmm, I thought bitfields weren't a good idea for this sort of thing. Isn't
> it up to the compiler how they are laid out in memory?

There's just been a discussion about this on the x86 side.  See
https://lkml.org/lkml/2018/11/27/116 and 
https://lkml.org/lkml/2018/11/30/848.   The data structures are
also typically defined as bitfields in the Hyper-V TLFS.  The decision was
to keep the bitfields but add __packed.  I'll do the same in the
next version of this patch.

> 
> > +
> > +/* Define synthetic interrupt controller message header. */
> > +struct hv_message_header {
> > +	__u32 message_type;
> > +	__u8 payload_size;
> > +	union hv_message_flags message_flags;
> > +	__u8 reserved[2];
> > +	union {
> > +		__u64 sender;
> > +		union hv_port_id port;
> > +	};
> > +};
> > +
> > +/* Define synthetic interrupt controller message format. */
> > +struct hv_message {
> > +	struct hv_message_header header;
> > +	union {
> > +		__u64 payload[HV_MESSAGE_PAYLOAD_QWORD_COUNT];
> > +	} u;
> > +};
> > +
> > +/* Define the synthetic interrupt message page layout. */
> > +struct hv_message_page {
> > +	struct hv_message sint_message[HV_SYNIC_SINT_COUNT];
> > +};
> > +
> > +/* Define timer message payload structure. */
> > +struct hv_timer_message_payload {
> > +	__u32 timer_index;
> > +	__u32 reserved;
> > +	__u64 expiration_time;	/* When the timer expired */
> > +	__u64 delivery_time;	/* When the message was delivered */
> > +};
> > +
> > +#define HV_STIMER_ENABLE		(1ULL << 0)
> > +#define HV_STIMER_PERIODIC		(1ULL << 1)
> > +#define HV_STIMER_LAZY			(1ULL << 2)
> > +#define HV_STIMER_AUTOENABLE		(1ULL << 3)
> > +#define HV_STIMER_SINT(config)		(__u8)(((config) >> 16) & 0x0F)
> > +
> > +#endif
> > diff --git a/arch/arm64/include/asm/mshyperv.h b/arch/arm64/include/asm/mshyperv.h
> > new file mode 100644
> > index 000000000000..a87c431d58b3
> > --- /dev/null
> > +++ b/arch/arm64/include/asm/mshyperv.h
> > @@ -0,0 +1,116 @@
> > +/* SPDX-License-Identifier: GPL-2.0 */
> > +
> > +/*
> > + * Linux-specific definitions for managing interactions with Microsoft's
> > + * Hyper-V hypervisor. The definitions in this file are specific to
> > + * the ARM64 architecture.  See include/asm-generic/mshyperv.h for
> > + * definitions are that architecture independent.
> > + *
> > + * Definitions that are specified in the Hyper-V Top Level Functional
> > + * Spec (TLFS) should not go in this file, but should instead go in
> > + * hyperv-tlfs.h.
> > + *
> > + * Copyright (C) 2018, Microsoft, Inc.
> > + *
> > + * Author : Michael Kelley <mikelley@microsoft.com>
> > + *
> > + * This program is free software; you can redistribute it and/or modify it
> > + * under the terms of the GNU General Public License version 2 as published
> > + * by the Free Software Foundation.
> > + *
> > + * This program is distributed in the hope that it will be useful, but
> > + * WITHOUT ANY WARRANTY; without even the implied warranty of
> > + * MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE, GOOD TITLE or
> > + * NON INFRINGEMENT.  See the GNU General Public License for more
> > + * details.
> > + */
> > +
> > +#ifndef _ASM_ARM64_MSHYPERV_H
> > +#define _ASM_ARM64_MSHYPERV_H
> > +
> > +#include <linux/types.h>
> > +#include <linux/interrupt.h>
> > +#include <linux/clocksource.h>
> > +#include <linux/irq.h>
> > +#include <linux/irqdesc.h>
> > +#include <asm/hyperv-tlfs.h>
> > +
> > +/*
> > + * Define the IRQ numbers/vectors used by Hyper-V VMbus interrupts
> > + * and by STIMER0 Direct Mode interrupts. Hyper-V should be supplying
> > + * these values through ACPI, but there are no other interrupting
> > + * devices in a Hyper-V VM on ARM64, so it's OK to hard code for now.
> > + * The "CALLBACK_VECTOR" terminology is a left-over from the x86/x64
> > + * world that is used in architecture independent Hyper-V code.
> > + */
> > +#define HYPERVISOR_CALLBACK_VECTOR 16
> > +#define	HV_STIMER0_IRQNR	   17
> > +
> > +extern u64 hv_do_hvc(u64 control, ...);
> > +extern u64 hv_do_hvc_fast_get(u64 control, u64 input1, u64 input2, u64 input3,
> > +		struct hv_get_vp_register_output *output);
> > +
> > +/*
> > + * Declare calls to get and set Hyper-V VP register values on ARM64, which
> > + * requires a hypercall.
> > + */
> > +extern void hv_set_vpreg(u32 reg, u64 value);
> > +extern u64 hv_get_vpreg(u32 reg);
> > +extern void hv_get_vpreg_128(u32 reg, struct hv_get_vp_register_output *result);
> > +
> > +/*
> > + * Use the Hyper-V provided stimer0 as the timer that is made
> > + * available to the architecture independent Hyper-V drivers.
> > + */
> > +#define hv_init_timer(timer, tick) \
> > +		hv_set_vpreg(HV_REGISTER_STIMER0_COUNT + (2*timer), tick)
> > +#define hv_init_timer_config(timer, val) \
> > +		hv_set_vpreg(HV_REGISTER_STIMER0_CONFIG + (2*timer), val)
> > +#define hv_get_current_tick(tick) \
> > +		(tick = hv_get_vpreg(HV_REGISTER_TIME_REFCOUNT))
> > +
> > +#define hv_get_simp(val) (val = hv_get_vpreg(HV_REGISTER_SIPP))
> > +#define hv_set_simp(val) hv_set_vpreg(HV_REGISTER_SIPP, val)
> > +
> > +#define hv_get_siefp(val) (val = hv_get_vpreg(HV_REGISTER_SIFP))
> > +#define hv_set_siefp(val) hv_set_vpreg(HV_REGISTER_SIFP, val)
> > +
> > +#define hv_get_synic_state(val) (val = hv_get_vpreg(HV_REGISTER_SCONTROL))
> > +#define hv_set_synic_state(val) hv_set_vpreg(HV_REGISTER_SCONTROL, val)
> > +
> > +#define hv_get_vp_index(index) (index = hv_get_vpreg(HV_REGISTER_VPINDEX))
> > +
> > +#define hv_signal_eom()	hv_set_vpreg(HV_REGISTER_EOM, 0)
> > +
> > +/*
> > + * Hyper-V SINT registers are numbered sequentially, so we can just
> > + * add the SINT number to the register number of SINT0
> > + */
> > +#define hv_get_synint_state(sint_num, val) \
> > +		(val = hv_get_vpreg(HV_REGISTER_SINT0 + sint_num))
> > +#define hv_set_synint_state(sint_num, val) \
> > +		hv_set_vpreg(HV_REGISTER_SINT0 + sint_num, val)
> > +
> > +#define hv_get_crash_ctl(val) \
> > +		(val = hv_get_vpreg(HV_REGISTER_CRASH_CTL))
> > +
> > +#if IS_ENABLED(CONFIG_HYPERV)
> > +#define hv_enable_stimer0_percpu_irq(irq)	enable_percpu_irq(irq, 0)
> > +#define hv_disable_stimer0_percpu_irq(irq)	disable_percpu_irq(irq)
> > +#endif
> > +
> > +/* ARM64 specific code to read the hardware clock */
> > +static inline u64 hv_read_hwclock(void)
> > +{
> > +	u64 result;
> > +
> > +	isb();
> > +	result = read_sysreg(cntvct_el0);
> > +	isb();
> > +
> > +	return result;
> > +}
> 
> Shouldn't you use arch_timer_read_counter() or arch_counter_get_cntvct()
> instead?

I don't think so.  Hyper-V provides its own clocksource on ARM64 like it does
x86.  It is not the standard ARM64 arch timer, and the ARM64 arch timer driver
that implements arch_timer_read_counter() and arch_counter_get_cntvct() is
not used.

> 
> > +#include <asm-generic/mshyperv.h>
> > +
> > +#endif
> > diff --git a/include/asm-generic/mshyperv.h b/include/asm-generic/mshyperv.h
> > new file mode 100644
> > index 000000000000..fbe1ec89c85a
> > --- /dev/null
> > +++ b/include/asm-generic/mshyperv.h
> > @@ -0,0 +1,240 @@
> > +/* SPDX-License-Identifier: GPL-2.0 */
> > +
> > +/*
> > + * Linux-specific definitions for managing interactions with Microsoft's
> > + * Hyper-V hypervisor. The definitions in this file are architecture
> > + * independent. See arch/<arch>/include/asm/mshyperv.h for definitions
> > + * that are specific to architecture <arch>.
> > + *
> > + * Definitions that are specified in the Hyper-V Top Level Functional
> > + * Spec (TLFS) should not go in this file, but should instead go in
> > + * hyperv-tlfs.h.
> > + *
> > + * Copyright (C) 2018, Microsoft, Inc.
> > + *
> > + * Author : Michael Kelley <mikelley@microsoft.com>
> > + *
> > + * This program is free software; you can redistribute it and/or modify it
> > + * under the terms of the GNU General Public License version 2 as published
> > + * by the Free Software Foundation.
> > + *
> > + * This program is distributed in the hope that it will be useful, but
> > + * WITHOUT ANY WARRANTY; without even the implied warranty of
> > + * MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE, GOOD TITLE or
> > + * NON INFRINGEMENT.  See the GNU General Public License for more
> > + * details.
> > + */
> > +
> > +#ifndef _ASM_GENERIC_MSHYPERV_H
> > +#define _ASM_GENERIC_MSHYPERV_H
> > +
> > +#include <linux/types.h>
> > +#include <linux/interrupt.h>
> > +#include <linux/clocksource.h>
> > +#include <linux/irq.h>
> > +#include <linux/irqdesc.h>
> > +#include <asm/hyperv-tlfs.h>
> > +
> > +/*
> > + * Hyper-V always runs with a page size of 4096. These definitions
> > + * are used when communicating with Hyper-V using guest physical
> > + * pages and guest physical page addresses, since the guest page
> > + * size may not be 4096 on ARM64.
> > + */
> > +#define HV_HYP_PAGE_SIZE	4096
> > +#define HV_HYP_PAGE_SHIFT	12
> 
> Maybe define the page size in terms of the shift?

Good idea.  I'll make that change.

> 
> > +#define HV_HYP_PAGE_MASK	(~(HV_HYP_PAGE_SIZE - 1))
> > +
> > +
> > +struct ms_hyperv_info {
> > +	u32 features;
> > +	u32 misc_features;
> > +	u32 hints;
> > +	u32 max_vp_index;
> > +	u32 max_lp_index;
> > +};
> 
> What is the define endianness for in-memory structures shared with the
> hypervisor?

It is little endian.  I'll add a comment to that effect as it is effectively
part of the ABI.

> 
> > +extern struct ms_hyperv_info ms_hyperv;
> > +
> > +extern u64 hv_do_hypercall(u64 control, void *inputaddr, void *outputaddr);
> > +extern u64 hv_do_fast_hypercall8(u16 control, u64 input8);
> > +
> > +/*
> > + * The guest OS needs to register the guest ID with the hypervisor.
> > + * The guest ID is a 64 bit entity and the structure of this ID is
> > + * specified in the Hyper-V specification:
> > + *
> > + * msdn.microsoft.com/en-us/library/windows/hardware/ff542653%28v=vs.85%29.aspx
> > + *
> > + * While the current guideline does not specify how Linux guest ID(s)
> > + * need to be generated, our plan is to publish the guidelines for
> > + * Linux and other guest operating systems that currently are hosted
> > + * on Hyper-V. The implementation here conforms to this yet
> > + * unpublished guidelines.
> > + *
> > + *
> > + * Bit(s)
> > + * 63 - Indicates if the OS is Open Source or not; 1 is Open Source
> > + * 62:56 - Os Type; Linux is 0x100
> > + * 55:48 - Distro specific identification
> > + * 47:16 - Linux kernel version number
> > + * 15:0  - Distro specific identification
> 
> This looks familiar...

Yes.  I'll remove the duplicate copy in the other file.

> 
> > + * Generate the guest ID based on the guideline described above.
> > + */
> > +
> > +static inline  __u64 generate_guest_id(__u64 d_info1, __u64 kernel_version,
> > +				       __u64 d_info2)
> > +{
> > +	__u64 guest_id = 0;
> > +
> > +	guest_id = (((__u64)HV_LINUX_VENDOR_ID) << 48);
> > +	guest_id |= (d_info1 << 48);
> > +	guest_id |= (kernel_version << 16);
> > +	guest_id |= d_info2;
> > +
> > +	return guest_id;
> > +}
> > +
> > +
> > +/* Free the message slot and signal end-of-message if required */
> > +static inline void vmbus_signal_eom(struct hv_message *msg, u32 old_msg_type)
> > +{
> > +	/*
> > +	 * On crash we're reading some other CPU's message page and we need
> > +	 * to be careful: this other CPU may already had cleared the header
> > +	 * and the host may already had delivered some other message there.
> > +	 * In case we blindly write msg->header.message_type we're going
> > +	 * to lose it. We can still lose a message of the same type but
> > +	 * we count on the fact that there can only be one
> > +	 * CHANNELMSG_UNLOAD_RESPONSE and we don't care about other messages
> > +	 * on crash.
> > +	 */
> > +	if (cmpxchg(&msg->header.message_type, old_msg_type,
> > +		    HVMSG_NONE) != old_msg_type)
> > +		return;
> > +
> > +	/*
> > +	 * Make sure the write to MessageType (ie set to
> > +	 * HVMSG_NONE) happens before we read the
> > +	 * MessagePending and EOMing. Otherwise, the EOMing
> > +	 * will not deliver any more messages since there is
> > +	 * no empty slot
> > +	 */
> > +	mb();
> 
> A successful cmpxchg() already implies full barriers, so this isn't needed.
> I also wonder whether cmpxchg_acquire() would be sufficient here.

OK.  I'll investigate the use of cmpxchg_acquire().

> 
> > +
> > +	if (msg->header.message_flags.msg_pending) {
> > +		/*
> > +		 * This will cause message queue rescan to
> > +		 * possibly deliver another msg from the
> > +		 * hypervisor
> > +		 */
> > +		hv_signal_eom();
> > +	}
> > +}
> > +
> > +void hv_setup_vmbus_irq(void (*handler)(void));
> > +void hv_remove_vmbus_irq(void);
> > +void hv_enable_vmbus_irq(void);
> > +void hv_disable_vmbus_irq(void);
> > +
> > +void hv_setup_kexec_handler(void (*handler)(void));
> > +void hv_remove_kexec_handler(void);
> > +void hv_setup_crash_handler(void (*handler)(struct pt_regs *regs));
> > +void hv_remove_crash_handler(void);
> > +
> > +#if IS_ENABLED(CONFIG_HYPERV)
> > +extern struct clocksource *hyperv_cs;
> > +
> > +/*
> > + * Hypervisor's notion of virtual processor ID is different from
> > + * Linux' notion of CPU ID. This information can only be retrieved
> > + * in the context of the calling CPU. Setup a map for easy access
> > + * to this information.
> > + */
> > +extern u32 *hv_vp_index;
> > +extern u32 hv_max_vp_index;
> > +
> > +/* Sentinel value for an uninitialized entry in hv_vp_index array */
> > +#define VP_INVAL	U32_MAX
> > +
> > +/**
> > + * hv_cpu_number_to_vp_number() - Map CPU to VP.
> > + * @cpu_number: CPU number in Linux terms
> > + *
> > + * This function returns the mapping between the Linux processor
> > + * number and the hypervisor's virtual processor number, useful
> > + * in making hypercalls and such that talk about specific
> > + * processors.
> > + *
> > + * Return: Virtual processor number in Hyper-V terms
> > + */
> > +static inline int hv_cpu_number_to_vp_number(int cpu_number)
> > +{
> > +	return hv_vp_index[cpu_number];
> > +}
> > +
> > +void hyperv_report_panic(struct pt_regs *regs, long err);
> > +void hyperv_report_panic_msg(phys_addr_t pa, size_t size);
> > +bool hv_is_hyperv_initialized(void);
> > +void hyperv_cleanup(void);
> > +#else /* CONFIG_HYPERV */
> > +static inline bool hv_is_hyperv_initialized(void) { return false; }
> > +static inline void hyperv_cleanup(void) {}
> > +#endif /* CONFIG_HYPERV */
> > +
> > +#if IS_ENABLED(CONFIG_HYPERV)
> > +extern int hv_setup_stimer0_irq(int *irq, int *vector, void (*handler)(void));
> > +extern void hv_remove_stimer0_irq(int irq);
> > +#endif
> > +
> > +static inline u64 hv_read_tsc_page_tsc(const struct ms_hyperv_tsc_page *tsc_pg,
> > +				       u64 *cur_tsc)
> > +{
> > +	u64	scale, offset;
> > +	u32	sequence;
> > +
> > +	/*
> > +	 * The protocol for reading Hyper-V TSC page is specified in Hypervisor
> > +	 * Top-Level Functional Specification.  To get the reference time we
> > +	 * must do the following:
> > +	 * - READ ReferenceTscSequence
> > +	 *   A special '0' value indicates the time source is unreliable and we
> > +	 *   need to use something else.
> > +	 * - ReferenceTime =
> > +	 *     ((HWclock val) * ReferenceTscScale) >> 64) + ReferenceTscOffset
> > +	 * - READ ReferenceTscSequence again. In case its value has changed
> > +	 *   since our first reading we need to discard ReferenceTime and repeat
> > +	 *   the whole sequence as the hypervisor was updating the page in
> > +	 *   between.
> > +	 */
> > +	do {
> > +		sequence = READ_ONCE(tsc_pg->tsc_sequence);
> > +		/*
> > +		 * Make sure we read sequence before we read other values from
> > +		 * TSC page.
> > +		 */
> > +		smp_rmb();
> > +
> > +		scale = READ_ONCE(tsc_pg->tsc_scale);
> > +		offset = READ_ONCE(tsc_pg->tsc_offset);
> > +		*cur_tsc = hv_read_hwclock();
> > +
> > +		/*
> > +		 * Make sure we read sequence after we read all other values
> > +		 * from TSC page.
> > +		 */
> > +		smp_rmb();
> > +
> > +	} while (READ_ONCE(tsc_pg->tsc_sequence) != sequence);
> 
> Could you explain what the writer side looks like, please? I'm a bit
> confused as to why we're not using the bottom bit of the sequence
> number to detect a concurrent update.

The writer side, of course, is Hyper-V.  The algorithm used follows the
code in Section 12.6.3 of the Hyper-V TLFS, though as pointed out in a
separate email by Vitaly Kuznetsov, I seemed to have dropped the test for
sequence value 0 when moving the code over from the x86 side.  I'll add
that back in.

I'm not clear on your thought about using the bottom bit of the sequence
number.   Is that better than just comparing the full 32-bit value?

Thanks for the review and all the great comments ...

Michael

> 
> Will

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

RE: [PATCH 2/4] arm64: hyperv: Add support for Hyper-V as a hypervisor

[Michael Kelley]

From: Marc Zyngier <marc.zyngier@arm.com>  Sent: Friday, December 7, 2018 6:43 AM

> > Add ARM64-specific code to enable Hyper-V. This code includes:
> > * Detecting Hyper-V and initializing the guest/Hyper-V interface
> > * Setting up Hyper-V's synthetic clocks
> > * Making hypercalls using the HVC instruction
> > * Setting up VMbus and stimer0 interrupts
> > * Setting up kexec and crash handlers
> 
> This commit message is a clear indication that this should be split in
> at least 5 different patches.

OK, I'll work on separating into multiple layered patches in the next
version.

> 
> > This code is architecture dependent code and is mostly driven by
> > architecture independent code in the VMbus driver in drivers/hv/hv.c
> > and drivers/hv/vmbus_drv.c.
> >
> > This code is built only when CONFIG_HYPERV is enabled.
> >
> > Signed-off-by: Michael Kelley <mikelley@microsoft.com>
> > Signed-off-by: K. Y. Srinivasan <kys@microsoft.com>
> > ---
> >  MAINTAINERS                  |   1 +
> >  arch/arm64/Makefile          |   1 +
> >  arch/arm64/hyperv/Makefile   |   2 +
> >  arch/arm64/hyperv/hv_hvc.S   |  54 +++++
> >  arch/arm64/hyperv/hv_init.c  | 441 +++++++++++++++++++++++++++++++++++
> >  arch/arm64/hyperv/mshyperv.c | 178 ++++++++++++++
> >  6 files changed, 677 insertions(+)
> >  create mode 100644 arch/arm64/hyperv/Makefile
> >  create mode 100644 arch/arm64/hyperv/hv_hvc.S
> >  create mode 100644 arch/arm64/hyperv/hv_init.c
> >  create mode 100644 arch/arm64/hyperv/mshyperv.c
> >
> > diff --git a/MAINTAINERS b/MAINTAINERS
> > index 72f19cef4c48..326eeb32a0cd 100644
> > --- a/MAINTAINERS
> > +++ b/MAINTAINERS
> > @@ -6837,6 +6837,7 @@ F:	arch/x86/kernel/cpu/mshyperv.c
> >  F:	arch/x86/hyperv
> >  F:	arch/arm64/include/asm/hyperv-tlfs.h
> >  F:	arch/arm64/include/asm/mshyperv.h
> > +F:	arch/arm64/hyperv
> >  F:	drivers/hid/hid-hyperv.c
> >  F:	drivers/hv/
> >  F:	drivers/input/serio/hyperv-keyboard.c
> > diff --git a/arch/arm64/Makefile b/arch/arm64/Makefile
> > index 6cb9fc7e9382..ad9ec0579553 100644
> > --- a/arch/arm64/Makefile
> > +++ b/arch/arm64/Makefile
> > @@ -106,6 +106,7 @@ core-y		+= arch/arm64/kernel/ arch/arm64/mm/
> >  core-$(CONFIG_NET) += arch/arm64/net/
> >  core-$(CONFIG_KVM) += arch/arm64/kvm/
> >  core-$(CONFIG_XEN) += arch/arm64/xen/
> > +core-$(CONFIG_HYPERV) += arch/arm64/hyperv/
> >  core-$(CONFIG_CRYPTO) += arch/arm64/crypto/
> >  libs-y		:= arch/arm64/lib/ $(libs-y)
> >  core-$(CONFIG_EFI_STUB) += $(objtree)/drivers/firmware/efi/libstub/lib.a
> > diff --git a/arch/arm64/hyperv/Makefile b/arch/arm64/hyperv/Makefile
> > new file mode 100644
> > index 000000000000..988eda55330c
> > --- /dev/null
> > +++ b/arch/arm64/hyperv/Makefile
> > @@ -0,0 +1,2 @@
> > +# SPDX-License-Identifier: GPL-2.0
> > +obj-y		:= hv_init.o hv_hvc.o mshyperv.o
> > diff --git a/arch/arm64/hyperv/hv_hvc.S b/arch/arm64/hyperv/hv_hvc.S
> > new file mode 100644
> > index 000000000000..82636969b4f2
> > --- /dev/null
> > +++ b/arch/arm64/hyperv/hv_hvc.S
> > @@ -0,0 +1,54 @@
> > +/* SPDX-License-Identifier: GPL-2.0 */
> > +
> > +/*
> > + * Microsoft Hyper-V hypervisor invocation routines
> > + *
> > + * Copyright (C) 2018, Microsoft, Inc.
> > + *
> > + * Author : Michael Kelley <mikelley@microsoft.com>
> > + *
> > + * This program is free software; you can redistribute it and/or modify it
> > + * under the terms of the GNU General Public License version 2 as published
> > + * by the Free Software Foundation.
> > + *
> > + * This program is distributed in the hope that it will be useful, but
> > + * WITHOUT ANY WARRANTY; without even the implied warranty of
> > + * MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE, GOOD TITLE or
> > + * NON INFRINGEMENT.  See the GNU General Public License for more
> > + * details.
> > + */
> > +
> > +#include <linux/linkage.h>
> > +
> > +	.text
> > +/*
> > + * Do the HVC instruction.  For Hyper-V the argument is always 1.
> > + * x0 contains the hypercall control value, while additional registers
> > + * vary depending on the hypercall, and whether the hypercall arguments
> > + * are in memory or in registers (a "fast" hypercall per the Hyper-V
> > + * TLFS).  When the arguments are in memory x1 is the guest physical
> > + * address of the input arguments, and x2 is the guest physical
> > + * address of the output arguments.  When the arguments are in
> > + * registers, the register values depends on the hypercall.  Note
> > + * that this version cannot return any values in registers.
> > + */
> > +ENTRY(hv_do_hvc)
> > +	hvc #1
> > +	ret
> > +ENDPROC(hv_do_hvc)
> > +
> > +/*
> > + * This variant of HVC invocation is for hv_get_vpreg and
> > + * hv_get_vpreg_128. The input parameters are passed in registers
> > + * along with a pointer in x4 to where the output result should
> > + * be stored. The output is returned in x15 and x16.  x18 is used as
> > + * scratch space to avoid buildng a stack frame, as Hyper-V does
> > + * not preserve registers x0-x17.
> > + */
> > +ENTRY(hv_do_hvc_fast_get)
> > +	mov x18, x4
> > +	hvc #1
> > +	str x15,[x18]
> > +	str x16,[x18,#8]
> > +	ret
> > +ENDPROC(hv_do_hvc_fast_get)
> 
> As Will said, this isn't a viable option. Please follow SMCCC 1.1.

I'll have to start a conversation with the Hyper-V team about this.
I don't know why they chose to use HVC #1 or this register scheme
for output values.  It may be tough to change at this point because
there are Windows guests on Hyper-V for ARM64 that are already
using this approach.

> 
> > diff --git a/arch/arm64/hyperv/hv_init.c b/arch/arm64/hyperv/hv_init.c
> > new file mode 100644
> > index 000000000000..aa1a8c09d989
> > --- /dev/null
> > +++ b/arch/arm64/hyperv/hv_init.c
> > @@ -0,0 +1,441 @@
> > +// SPDX-License-Identifier: GPL-2.0
> > +
> > +/*
> > + * Initialization of the interface with Microsoft's Hyper-V hypervisor,
> > + * and various low level utility routines for interacting with Hyper-V.
> > + *
> > + * Copyright (C) 2018, Microsoft, Inc.
> > + *
> > + * Author : Michael Kelley <mikelley@microsoft.com>
> > + *
> > + * This program is free software; you can redistribute it and/or modify it
> > + * under the terms of the GNU General Public License version 2 as published
> > + * by the Free Software Foundation.
> > + *
> > + * This program is distributed in the hope that it will be useful, but
> > + * WITHOUT ANY WARRANTY; without even the implied warranty of
> > + * MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE, GOOD TITLE or
> > + * NON INFRINGEMENT.  See the GNU General Public License for more
> > + * details.
> > + */
> > +
> > +
> > +#include <linux/types.h>
> > +#include <linux/version.h>
> > +#include <linux/export.h>
> > +#include <linux/vmalloc.h>
> > +#include <linux/mm.h>
> > +#include <linux/clocksource.h>
> > +#include <linux/sched_clock.h>
> > +#include <linux/acpi.h>
> > +#include <linux/module.h>
> > +#include <linux/hyperv.h>
> > +#include <linux/slab.h>
> > +#include <linux/cpuhotplug.h>
> > +#include <linux/psci.h>
> > +#include <asm-generic/bug.h>
> > +#include <asm/hypervisor.h>
> > +#include <asm/hyperv-tlfs.h>
> > +#include <asm/mshyperv.h>
> > +#include <asm/sysreg.h>
> > +#include <clocksource/arm_arch_timer.h>
> > +
> > +static bool	hyperv_initialized;
> > +struct		ms_hyperv_info ms_hyperv;
> > +EXPORT_SYMBOL_GPL(ms_hyperv);
> 
> Who are the users of this structure? Should they go via accessors instead?

The structure is an aggregation of several flags fields that describe a myriad
of features and hints that may or may not be present on any particular version
of Hyper-V, plus the max virtual processor ID values.  Everything is read-only
after initialization.   Most of the references are to test one of the flags.  It's
a judgment call, but there are a lot of different flags with long names, and
writing accessors for each one doesn't seem to me to add any clarity.

> 
> > +
> > +static struct ms_hyperv_tsc_page *tsc_pg;
> > +
> > +struct ms_hyperv_tsc_page *hv_get_tsc_page(void)
> > +{
> > +	return tsc_pg;
> > +}
> > +EXPORT_SYMBOL_GPL(hv_get_tsc_page);
> > +
> > +static u64 read_hv_sched_clock_tsc(void)
> > +{
> > +	u64 current_tick = hv_read_tsc_page(tsc_pg);
> > +
> > +	if (current_tick == U64_MAX)
> > +		current_tick = hv_get_vpreg(HV_REGISTER_TIME_REFCOUNT);
> > +
> > +	return current_tick;
> > +}
> > +
> > +static u64 read_hv_clock_tsc(struct clocksource *arg)
> > +{
> > +	u64 current_tick = hv_read_tsc_page(tsc_pg);
> > +
> > +	if (current_tick == U64_MAX)
> > +		current_tick = hv_get_vpreg(HV_REGISTER_TIME_REFCOUNT);
> > +
> > +	return current_tick;
> > +}
> 
> Can't you define one function in terms of the other?

Yes, it looks like that works.  I'll change it in the next version of the
patch.

> 
> > +
> > +static struct clocksource hyperv_cs_tsc = {
> > +		.name		= "hyperv_clocksource_tsc_page",
> > +		.rating		= 400,
> > +		.read		= read_hv_clock_tsc,
> > +		.mask		= CLOCKSOURCE_MASK(64),
> > +		.flags		= CLOCK_SOURCE_IS_CONTINUOUS,
> > +};
> > +
> > +static u64 read_hv_sched_clock_msr(void)
> > +{
> > +	return hv_get_vpreg(HV_REGISTER_TIME_REFCOUNT);
> > +}
> > +
> > +static u64 read_hv_clock_msr(struct clocksource *arg)
> > +{
> > +	return hv_get_vpreg(HV_REGISTER_TIME_REFCOUNT);
> > +}
> > +
> > +static struct clocksource hyperv_cs_msr = {
> > +	.name		= "hyperv_clocksource_msr",
> > +	.rating		= 400,
> > +	.read		= read_hv_clock_msr,
> > +	.mask		= CLOCKSOURCE_MASK(64),
> > +	.flags		= CLOCK_SOURCE_IS_CONTINUOUS,
> > +};
> > +
> > +struct clocksource *hyperv_cs;
> > +EXPORT_SYMBOL_GPL(hyperv_cs);
> 
> Why? Who needs to poke this?

It's referenced in the architecture independent driver code
for the Hyper-V clocksource in drivers/hv/hv.c, and in the
code to sync the time with the Hyper-V host in
drivers/hv/hv_util.c.

> 
> > +
> > +u32 *hv_vp_index;
> > +EXPORT_SYMBOL_GPL(hv_vp_index);
> > +
> > +u32 hv_max_vp_index;
> 
> Same thing.

Agreed -- this one doesn't seem to be needed.  The variable
is not used outside of the architecture dependent Hyper-V code.

> 
> > +
> > +static int hv_cpu_init(unsigned int cpu)
> > +{
> > +	u64 msr_vp_index;
> > +
> > +	hv_get_vp_index(msr_vp_index);
> > +
> > +	hv_vp_index[smp_processor_id()] = msr_vp_index;
> > +
> > +	if (msr_vp_index > hv_max_vp_index)
> > +		hv_max_vp_index = msr_vp_index;
> > +
> > +	return 0;
> > +}
> 
> Is that some new way to describe a CPU topology? If so, why isn't that
> exposed via the ACPI tables that the kernel already parses?

Hyper-V's hypercall interface uses vCPU identifiers that are not
guaranteed to be consecutive integers or to match what ACPI shows.
No topology information is implied -- it's just unique identifiers.  The 
hv_vp_index array provides easy mapping from Linux's consecutive
integer IDs for CPUs when needed to construct hypercall arguments.

> 
> > +
> > +/*
> > + * This function is invoked via the ACPI clocksource probe mechanism. We
> > + * don't actually use any values from the ACPI GTDT table, but we set up
> 
> This doesn't feel like a good idea at all. Piggy-backing on an existing
> mechanism and use it for something completely different is not exactly
> future-proof.
> 
> Also, if this is supposed to be a clocksource, why isn't that a
> clocksource driver on its own right?

I agree this is not the right long term solution.  Is there a better place to
hang the initialization code?  Or should I just make an explicit call to
initialize Hyper-V at the right place?  On the x86 side, there's an
explicit framework for hypervisor-specific initialization routines to plug
into.  Maybe it's time for a basic version of such a framework on the
ARM64 side.  Thoughts on the best approach, both in the short-term and
the longer-term? If we put a framework in place, does that need to
happen before adding Hyper-V code, or afterwards as a cleanup?

And yes, Hyper-V does effectively have its own clocksource.  The
main code is in drivers/hv/hv.c, but it's not broken out as a separate
driver in drivers/clocksource, probably due to some history on the
x86 side that pre-dates me.  I'll have to research.

> 
> > + * the Hyper-V synthetic clocksource and do other initialization for
> > + * interacting with Hyper-V the first time.  Using early_initcall to invoke
> > + * this function is too late because interrupts are already enabled at that
> > + * point, and sched_clock_register must run before interrupts are enabled.
> > + *
> > + * 1. Setup the guest ID.
> > + * 2. Get features and hints info from Hyper-V
> > + * 3. Setup per-cpu VP indices.
> > + * 4. Register Hyper-V specific clocksource.
> > + * 5. Register the scheduler clock.
> > + */
> > +
> > +static int __init hyperv_init(struct acpi_table_header *table)
> > +{
> > +	struct hv_get_vp_register_output result;
> > +	u32	a, b, c, d;
> > +	u64	guest_id;
> > +	int	i;
> > +
> > +	/*
> > +	 * If we're in a VM on Hyper-V, the ACPI hypervisor_id field will
> > +	 * have the string "MsHyperV".
> > +	 */
> > +	if (strncmp((char *)&acpi_gbl_FADT.hypervisor_id, "MsHyperV", 8))
> > +		return 1;
> > +
> > +	/* Setup the guest ID */
> > +	guest_id = generate_guest_id(0, LINUX_VERSION_CODE, 0);
> > +	hv_set_vpreg(HV_REGISTER_GUEST_OSID, guest_id);
> > +
> > +	/* Get the features and hints from Hyper-V */
> > +	hv_get_vpreg_128(HV_REGISTER_PRIVILEGES_AND_FEATURES, &result);
> > +	ms_hyperv.features = lower_32_bits(result.registervaluelow);
> > +	ms_hyperv.misc_features = upper_32_bits(result.registervaluehigh);
> > +
> > +	hv_get_vpreg_128(HV_REGISTER_FEATURES, &result);
> > +	ms_hyperv.hints = lower_32_bits(result.registervaluelow);
> > +
> > +	pr_info("Hyper-V: Features 0x%x, hints 0x%x\n",
> > +		ms_hyperv.features, ms_hyperv.hints);
> > +
> > +	/*
> > +	 * Direct mode is the only option for STIMERs provided Hyper-V
> > +	 * on ARM64, so Hyper-V doesn't actually set the flag.  But add the
> > +	 * flag so the architecture independent code in drivers/hv/hv.c
> > +	 * will correctly use that mode.
> > +	 */
> > +	ms_hyperv.misc_features |= HV_STIMER_DIRECT_MODE_AVAILABLE;
> > +
> > +	/*
> > +	 * Hyper-V on ARM64 doesn't support AutoEOI.  Add the hint
> > +	 * that tells architecture independent code not to use this
> > +	 * feature.
> > +	 */
> > +	ms_hyperv.hints |= HV_DEPRECATING_AEOI_RECOMMENDED;
> > +
> > +	/* Get information about the Hyper-V host version */
> > +	hv_get_vpreg_128(HV_REGISTER_HYPERVISOR_VERSION, &result);
> > +	a = lower_32_bits(result.registervaluelow);
> > +	b = upper_32_bits(result.registervaluelow);
> > +	c = lower_32_bits(result.registervaluehigh);
> > +	d = upper_32_bits(result.registervaluehigh);
> > +	pr_info("Hyper-V: Host Build %d.%d.%d.%d-%d-%d\n",
> > +		b >> 16, b & 0xFFFF, a, d & 0xFFFFFF, c, d >> 24);
> > +
> > +	/* Allocate percpu VP index */
> > +	hv_vp_index = kmalloc_array(num_possible_cpus(), sizeof(*hv_vp_index),
> > +				    GFP_KERNEL);
> 
> Why isn't this a percpu variable?

In current code in the architecture independent Hyper-V drivers (as well
as some future Hyper-V enlightenments that aren't yet implemented
for ARM64), the running CPU needs to get the VP index values for any CPUs
in the hv_vp_index array.  Some of the code is performance sensitive, and
accessing a global array is faster than accessing other CPUs' per-cpu data.

> 
> > +	if (!hv_vp_index)
> > +		return 1;
> > +
> > +	for (i = 0; i < num_possible_cpus(); i++)
> > +		hv_vp_index[i] = VP_INVAL;
> > +
> > +	if (cpuhp_setup_state(CPUHP_AP_ONLINE_DYN, "arm64/hyperv_init:online",
> > +			      hv_cpu_init, NULL) < 0)
> > +		goto free_vp_index;
> > +
> > +	/*
> > +	 * Try to set up what Hyper-V calls the "TSC reference page", which
> > +	 * uses the ARM Generic Timer virtual counter with some scaling
> > +	 * information to provide a fast and stable guest VM clocksource.
> > +	 * If the TSC reference page can't be set up, fall back to reading
> > +	 * the guest clock provided by Hyper-V's synthetic reference time
> > +	 * register.
> > +	 */
> > +	if (ms_hyperv.features & HV_MSR_REFERENCE_TSC_AVAILABLE) {
> > +
> > +		u64		tsc_msr;
> > +		phys_addr_t	phys_addr;
> > +
> > +		tsc_pg = __vmalloc(HV_HYP_PAGE_SIZE, GFP_KERNEL, PAGE_KERNEL);
> 
> And why not vmalloc?

Looks like it could be just vmalloc().  I'll change it in the next version.

> 
> > +		if (tsc_pg) {
> > +			phys_addr = page_to_phys(vmalloc_to_page(tsc_pg));
> > +			tsc_msr = hv_get_vpreg(HV_REGISTER_REFERENCE_TSC);
> > +			tsc_msr &= GENMASK_ULL(11, 0);
> 
> Magic.

Per Section 12.6.1 of the Hyper-V TLFS, bits 11 thru 1 must be preserved, and
the line below sets bit 0.  I'll add some comments. :-)

> 
> > +			tsc_msr = tsc_msr | 0x1 | (u64)phys_addr;
> > +			hv_set_vpreg(HV_REGISTER_REFERENCE_TSC, tsc_msr);
> > +			hyperv_cs = &hyperv_cs_tsc;
> > +			sched_clock_register(read_hv_sched_clock_tsc,
> > +						64, HV_CLOCK_HZ);
> > +		}
> > +	}
> > +
> > +	if (!hyperv_cs &&
> > +	    (ms_hyperv.features & HV_MSR_TIME_REF_COUNT_AVAILABLE)) {
> > +		hyperv_cs = &hyperv_cs_msr;
> > +		sched_clock_register(read_hv_sched_clock_msr,
> > +						64, HV_CLOCK_HZ);
> > +	}
> > +
> > +	if (hyperv_cs) {
> > +		hyperv_cs->archdata.vdso_direct = false;
> > +		clocksource_register_hz(hyperv_cs, HV_CLOCK_HZ);
> > +	}
> > +
> > +	hyperv_initialized = true;
> > +	return 0;
> > +
> > +free_vp_index:
> > +	kfree(hv_vp_index);
> > +	hv_vp_index = NULL;
> > +	return 1;
> 
> ????

The return value is there because this function is implemented as a timer
initialization function, and that's what the function signature requires.
But maybe I'm not understanding your question.

> 
> > +}
> > +TIMER_ACPI_DECLARE(hyperv, ACPI_SIG_GTDT, hyperv_init);
> > +
> > +/*
> > + * This routine is called before kexec/kdump, it does the required cleanup.
> > + */
> > +void hyperv_cleanup(void)
> > +{
> > +	/* Reset our OS id */
> > +	hv_set_vpreg(HV_REGISTER_GUEST_OSID, 0);
> > +
> > +}
> > +EXPORT_SYMBOL_GPL(hyperv_cleanup);
> > +
> > +/*
> > + * hv_do_hypercall- Invoke the specified hypercall
> > + */
> > +u64 hv_do_hypercall(u64 control, void *input, void *output)
> > +{
> > +	u64 input_address;
> > +	u64 output_address;
> > +
> > +	input_address = input ? virt_to_phys(input) : 0;
> > +	output_address = output ? virt_to_phys(output) : 0;
> > +	return hv_do_hvc(control, input_address, output_address);
> > +}
> > +EXPORT_SYMBOL_GPL(hv_do_hypercall);
> > +
> > +/*
> > + * hv_do_fast_hypercall8 -- Invoke the specified hypercall
> > + * with arguments in registers instead of physical memory.
> > + * Avoids the overhead of virt_to_phys for simple hypercalls.
> > + */
> > +
> > +u64 hv_do_fast_hypercall8(u16 code, u64 input)
> > +{
> > +	u64 control;
> > +
> > +	control = (u64)code | HV_HYPERCALL_FAST_BIT;
> > +	return hv_do_hvc(control, input);
> > +}
> > +EXPORT_SYMBOL_GPL(hv_do_fast_hypercall8);
> > +
> > +
> > +/*
> > + * Set a single VP register to a 64-bit value.
> > + */
> > +void hv_set_vpreg(u32 msr, u64 value)
> > +{
> > +	union hv_hypercall_status status;
> > +
> > +	status.as_uint64 = hv_do_hvc(
> > +		HVCALL_SET_VP_REGISTERS | HV_HYPERCALL_FAST_BIT |
> > +			HV_HYPERCALL_REP_COUNT_1,
> > +		HV_PARTITION_ID_SELF,
> > +		HV_VP_INDEX_SELF,
> > +		msr,
> > +		0,
> > +		value,
> > +		0);
> > +
> > +	/*
> > +	 * Something is fundamentally broken in the hypervisor if
> > +	 * setting a VP register fails. There's really no way to
> > +	 * continue as a guest VM, so panic.
> > +	 */
> > +	BUG_ON(status.status != HV_STATUS_SUCCESS);
> > +}
> > +EXPORT_SYMBOL_GPL(hv_set_vpreg);
> > +
> > +
> > +/*
> > + * Get the value of a single VP register, and only the low order 64 bits.
> > + */
> > +u64 hv_get_vpreg(u32 msr)
> > +{
> > +	union hv_hypercall_status status;
> > +	struct hv_get_vp_register_output output;
> > +
> > +	status.as_uint64 = hv_do_hvc_fast_get(
> > +		HVCALL_GET_VP_REGISTERS | HV_HYPERCALL_FAST_BIT |
> > +			HV_HYPERCALL_REP_COUNT_1,
> > +		HV_PARTITION_ID_SELF,
> > +		HV_VP_INDEX_SELF,
> > +		msr,
> > +		&output);
> > +
> > +	/*
> > +	 * Something is fundamentally broken in the hypervisor if
> > +	 * getting a VP register fails. There's really no way to
> > +	 * continue as a guest VM, so panic.
> > +	 */
> > +	BUG_ON(status.status != HV_STATUS_SUCCESS);
> > +
> > +	return output.registervaluelow;
> > +}
> > +EXPORT_SYMBOL_GPL(hv_get_vpreg);
> > +
> > +/*
> > + * Get the value of a single VP register that is 128 bits in size.  This is a
> > + * separate call in order to avoid complicating the calling sequence for
> > + * the much more frequently used 64-bit version.
> > + */
> > +void hv_get_vpreg_128(u32 msr, struct hv_get_vp_register_output *result)
> > +{
> > +	union hv_hypercall_status status;
> > +
> > +	status.as_uint64 = hv_do_hvc_fast_get(
> > +		HVCALL_GET_VP_REGISTERS | HV_HYPERCALL_FAST_BIT |
> > +			HV_HYPERCALL_REP_COUNT_1,
> > +		HV_PARTITION_ID_SELF,
> > +		HV_VP_INDEX_SELF,
> > +		msr,
> > +		result);
> > +
> > +	/*
> > +	 * Something is fundamentally broken in the hypervisor if
> > +	 * getting a VP register fails. There's really no way to
> > +	 * continue as a guest VM, so panic.
> > +	 */
> > +	BUG_ON(status.status != HV_STATUS_SUCCESS);
> > +
> > +	return;
> > +
> > +}
> > +EXPORT_SYMBOL_GPL(hv_get_vpreg_128);
> > +
> > +void hyperv_report_panic(struct pt_regs *regs, long err)
> > +{
> > +	static bool panic_reported;
> > +	u64 guest_id;
> > +
> > +	/*
> > +	 * We prefer to report panic on 'die' chain as we have proper
> > +	 * registers to report, but if we miss it (e.g. on BUG()) we need
> > +	 * to report it on 'panic'.
> > +	 */
> > +	if (panic_reported)
> > +		return;
> > +	panic_reported = true;
> > +
> > +	guest_id = hv_get_vpreg(HV_REGISTER_GUEST_OSID);
> > +
> > +	/*
> > +	 * Hyper-V provides the ability to store only 5 values.
> > +	 * Pick the passed in error value, the guest_id, and the PC.
> > +	 * The first two general registers are added arbitrarily.
> > +	 */
> > +	hv_set_vpreg(HV_REGISTER_CRASH_P0, err);
> > +	hv_set_vpreg(HV_REGISTER_CRASH_P1, guest_id);
> > +	hv_set_vpreg(HV_REGISTER_CRASH_P2, regs->pc);
> > +	hv_set_vpreg(HV_REGISTER_CRASH_P3, regs->regs[0]);
> > +	hv_set_vpreg(HV_REGISTER_CRASH_P4, regs->regs[1]);
> > +
> > +	/*
> > +	 * Let Hyper-V know there is crash data available
> > +	 */
> > +	hv_set_vpreg(HV_REGISTER_CRASH_CTL, HV_CRASH_CTL_CRASH_NOTIFY);
> > +}
> > +EXPORT_SYMBOL_GPL(hyperv_report_panic);
> > +
> > +/*
> > + * hyperv_report_panic_msg - report panic message to Hyper-V
> > + * @pa: physical address of the panic page containing the message
> > + * @size: size of the message in the page
> > + */
> > +void hyperv_report_panic_msg(phys_addr_t pa, size_t size)
> > +{
> > +	/*
> > +	 * P3 to contain the physical address of the panic page & P4 to
> > +	 * contain the size of the panic data in that page. Rest of the
> > +	 * registers are no-op when the NOTIFY_MSG flag is set.
> > +	 */
> > +	hv_set_vpreg(HV_REGISTER_CRASH_P0, 0);
> > +	hv_set_vpreg(HV_REGISTER_CRASH_P1, 0);
> > +	hv_set_vpreg(HV_REGISTER_CRASH_P2, 0);
> > +	hv_set_vpreg(HV_REGISTER_CRASH_P3, pa);
> > +	hv_set_vpreg(HV_REGISTER_CRASH_P4, size);
> > +
> > +	/*
> > +	 * Let Hyper-V know there is crash data available along with
> > +	 * the panic message.
> > +	 */
> > +	hv_set_vpreg(HV_REGISTER_CRASH_CTL,
> > +	       (HV_CRASH_CTL_CRASH_NOTIFY | HV_CRASH_CTL_CRASH_NOTIFY_MSG));
> > +}
> > +EXPORT_SYMBOL_GPL(hyperv_report_panic_msg);
> > +
> > +bool hv_is_hyperv_initialized(void)
> > +{
> > +	return hyperv_initialized;
> > +}
> > +EXPORT_SYMBOL_GPL(hv_is_hyperv_initialized);
> > diff --git a/arch/arm64/hyperv/mshyperv.c b/arch/arm64/hyperv/mshyperv.c
> > new file mode 100644
> > index 000000000000..3ef055599412
> > --- /dev/null
> > +++ b/arch/arm64/hyperv/mshyperv.c
> > @@ -0,0 +1,178 @@
> > +// SPDX-License-Identifier: GPL-2.0
> > +
> > +/*
> > + * Core routines for interacting with Microsoft's Hyper-V hypervisor,
> > + * including setting up VMbus and STIMER interrupts, and handling
> > + * crashes and kexecs. These interactions are through a set of
> > + * static "handler" variables set by the architecture independent
> > + * VMbus and STIMER drivers.  This design is used to meet x86/x64
> > + * requirements for avoiding direct linkages and allowing the VMbus
> > + * and STIMER drivers to be unloaded and reloaded.
> > + *
> > + * Copyright (C) 2018, Microsoft, Inc.
> > + *
> > + * Author : Michael Kelley <mikelley@microsoft.com>
> > + *
> > + * This program is free software; you can redistribute it and/or modify it
> > + * under the terms of the GNU General Public License version 2 as published
> > + * by the Free Software Foundation.
> > + *
> > + * This program is distributed in the hope that it will be useful, but
> > + * WITHOUT ANY WARRANTY; without even the implied warranty of
> > + * MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE, GOOD TITLE or
> > + * NON INFRINGEMENT.  See the GNU General Public License for more
> > + * details.
> > + */
> > +
> > +#include <linux/types.h>
> > +#include <linux/export.h>
> > +#include <linux/interrupt.h>
> > +#include <linux/kexec.h>
> > +#include <linux/acpi.h>
> > +#include <linux/ptrace.h>
> > +#include <asm/hyperv-tlfs.h>
> > +#include <asm/mshyperv.h>
> > +
> > +static void (*vmbus_handler)(void);
> > +static void (*hv_stimer0_handler)(void);
> > +static void (*hv_kexec_handler)(void);
> > +static void (*hv_crash_handler)(struct pt_regs *regs);
> > +
> > +static int vmbus_irq;
> > +static long __percpu *vmbus_evt;
> > +static long __percpu *stimer0_evt;
> > +
> > +irqreturn_t hyperv_vector_handler(int irq, void *dev_id)
> > +{
> > +	if (vmbus_handler)
> > +		vmbus_handler();
> 
> In which circumstances can this be NULL?

Perhaps none.  This may be a leftover from the equivalent
x86 code.  I'll check and probably remove it.

> 
> > +	return IRQ_HANDLED;
> > +}
> > +
> > +/* Must be done just once */
> > +void hv_setup_vmbus_irq(void (*handler)(void))
> > +{
> > +	int result;
> > +
> > +	vmbus_handler = handler;
> 
> If thismust only be done once, maybe you could check that it hasn't been
> done already?

OK.

> 
> > +	vmbus_irq = acpi_register_gsi(NULL, HYPERVISOR_CALLBACK_VECTOR,
> > +				 ACPI_LEVEL_SENSITIVE, ACPI_ACTIVE_HIGH);
> > +	if (vmbus_irq <= 0) {
> > +		pr_err("Can't register Hyper-V VMBus GSI. Error %d",
> > +			vmbus_irq);
> > +		vmbus_irq = 0;
> > +		return;
> > +	}
> > +	vmbus_evt = alloc_percpu(long);
> > +	result = request_percpu_irq(vmbus_irq, hyperv_vector_handler,
> > +			"Hyper-V VMbus", vmbus_evt);
> 
> If this is a per-cpu interrupt, why isn't it signalled as a PPI, in an
> architecture compliant way?

Except for the code in this module, the interrupt handler for VMbus
interrupts is architecture independent.  But there's no support for
per-process interrupts on the x86, so the hypervisor interrupt vectors
are hard coded in the same IDT entry across all processors, and the
normal IRQ allocation mechanism is bypassed.  The above approach
assigns an ARM64 PPI (HYPERVISOR_CALLBACK_VECTOR is 16) in a
way that works with the arch independent interrupt handler.

Or maybe I'm missing your point.  If so, please set me straight.

> 
> > +	if (result) {
> > +		pr_err("Can't request Hyper-V VMBus IRQ %d. Error %d",
> > +			vmbus_irq, result);
> > +		free_percpu(vmbus_evt);
> > +		acpi_unregister_gsi(vmbus_irq);
> > +		vmbus_irq = 0;
> > +	}
> > +}
> > +EXPORT_SYMBOL_GPL(hv_setup_vmbus_irq);
> > +
> > +/* Must be done just once */
> > +void hv_remove_vmbus_irq(void)
> > +{
> > +	if (vmbus_irq) {
> > +		free_percpu_irq(vmbus_irq, vmbus_evt);
> > +		free_percpu(vmbus_evt);
> > +		acpi_unregister_gsi(vmbus_irq);
> > +	}
> > +}
> > +EXPORT_SYMBOL_GPL(hv_remove_vmbus_irq);
> > +
> > +/* Must be done by each CPU */
> > +void hv_enable_vmbus_irq(void)
> > +{
> > +	enable_percpu_irq(vmbus_irq, 0);
> > +}
> > +EXPORT_SYMBOL_GPL(hv_enable_vmbus_irq);
> > +
> > +/* Must be done by each CPU */
> > +void hv_disable_vmbus_irq(void)
> > +{
> > +	disable_percpu_irq(vmbus_irq);
> > +}
> > +EXPORT_SYMBOL_GPL(hv_disable_vmbus_irq);
> > +
> > +/* Routines to do per-architecture handling of STIMER0 when in Direct Mode */
> > +
> > +static irqreturn_t hv_stimer0_vector_handler(int irq, void *dev_id)
> > +{
> > +	if (hv_stimer0_handler)
> > +		hv_stimer0_handler();
> > +	return IRQ_HANDLED;
> > +}
> > +
> > +int hv_setup_stimer0_irq(int *irq, int *vector, void (*handler)(void))
> > +{
> > +	int localirq;
> > +	int result;
> > +
> > +	localirq = acpi_register_gsi(NULL, HV_STIMER0_IRQNR,
> > +			ACPI_LEVEL_SENSITIVE, ACPI_ACTIVE_HIGH);
> > +	if (localirq <= 0) {
> > +		pr_err("Can't register Hyper-V stimer0 GSI. Error %d",
> > +			localirq);
> > +		*irq = 0;
> > +		return -1;
> > +	}
> > +	stimer0_evt = alloc_percpu(long);
> > +	result = request_percpu_irq(localirq, hv_stimer0_vector_handler,
> > +					 "Hyper-V stimer0", stimer0_evt);
> > +	if (result) {
> > +		pr_err("Can't request Hyper-V stimer0 IRQ %d. Error %d",
> > +			localirq, result);
> > +		free_percpu(stimer0_evt);
> > +		acpi_unregister_gsi(localirq);
> > +		*irq = 0;
> > +		return -1;
> > +	}
> > +
> > +	hv_stimer0_handler = handler;
> > +	*vector = HV_STIMER0_IRQNR;
> > +	*irq = localirq;
> > +	return 0;
> > +}
> > +EXPORT_SYMBOL_GPL(hv_setup_stimer0_irq);
> > +
> > +void hv_remove_stimer0_irq(int irq)
> > +{
> > +	hv_stimer0_handler = NULL;
> > +	if (irq) {
> > +		free_percpu_irq(irq, stimer0_evt);
> > +		free_percpu(stimer0_evt);
> > +		acpi_unregister_gsi(irq);
> > +	}
> > +}
> > +EXPORT_SYMBOL_GPL(hv_remove_stimer0_irq);
> > +
> > +void hv_setup_kexec_handler(void (*handler)(void))
> > +{
> > +	hv_kexec_handler = handler;
> > +}
> > +EXPORT_SYMBOL_GPL(hv_setup_kexec_handler);
> > +
> > +void hv_remove_kexec_handler(void)
> > +{
> > +	hv_kexec_handler = NULL;
> > +}
> > +EXPORT_SYMBOL_GPL(hv_remove_kexec_handler);
> > +
> > +void hv_setup_crash_handler(void (*handler)(struct pt_regs *regs))
> > +{
> > +	hv_crash_handler = handler;
> > +}
> > +EXPORT_SYMBOL_GPL(hv_setup_crash_handler);
> > +
> > +void hv_remove_crash_handler(void)
> > +{
> > +	hv_crash_handler = NULL;
> > +}
> > +EXPORT_SYMBOL_GPL(hv_remove_crash_handler);
> >
> 
> Overall, this patch needs a massive split up, clean up, and a good dose
> of documentation so that we can understand what you are trying to achieve.

Thanks for the feedback.  I'll start work on a new version for the
changes that are clear, and await additional discussion on
a few issues where I need some further input.

Michael

> 
> Thanks,
> 
> 	M.
> --
> Jazz is not dead. It just smells funny...

RE: [PATCH 2/4] arm64: hyperv: Add support for Hyper-V as a hypervisor

[Michael Kelley]

From: Marc Zyngier <marc.zyngier@arm.com>  Sent: Friday, December 7, 2018 6:43 AM

> > Add ARM64-specific code to enable Hyper-V. This code includes:
> > * Detecting Hyper-V and initializing the guest/Hyper-V interface
> > * Setting up Hyper-V's synthetic clocks
> > * Making hypercalls using the HVC instruction
> > * Setting up VMbus and stimer0 interrupts
> > * Setting up kexec and crash handlers
> 
> This commit message is a clear indication that this should be split in
> at least 5 different patches.

OK, I'll work on separating into multiple layered patches in the next
version.

> 
> > This code is architecture dependent code and is mostly driven by
> > architecture independent code in the VMbus driver in drivers/hv/hv.c
> > and drivers/hv/vmbus_drv.c.
> >
> > This code is built only when CONFIG_HYPERV is enabled.
> >
> > Signed-off-by: Michael Kelley <mikelley@microsoft.com>
> > Signed-off-by: K. Y. Srinivasan <kys@microsoft.com>
> > ---
> >  MAINTAINERS                  |   1 +
> >  arch/arm64/Makefile          |   1 +
> >  arch/arm64/hyperv/Makefile   |   2 +
> >  arch/arm64/hyperv/hv_hvc.S   |  54 +++++
> >  arch/arm64/hyperv/hv_init.c  | 441 +++++++++++++++++++++++++++++++++++
> >  arch/arm64/hyperv/mshyperv.c | 178 ++++++++++++++
> >  6 files changed, 677 insertions(+)
> >  create mode 100644 arch/arm64/hyperv/Makefile
> >  create mode 100644 arch/arm64/hyperv/hv_hvc.S
> >  create mode 100644 arch/arm64/hyperv/hv_init.c
> >  create mode 100644 arch/arm64/hyperv/mshyperv.c
> >
> > diff --git a/MAINTAINERS b/MAINTAINERS
> > index 72f19cef4c48..326eeb32a0cd 100644
> > --- a/MAINTAINERS
> > +++ b/MAINTAINERS
> > @@ -6837,6 +6837,7 @@ F:	arch/x86/kernel/cpu/mshyperv.c
> >  F:	arch/x86/hyperv
> >  F:	arch/arm64/include/asm/hyperv-tlfs.h
> >  F:	arch/arm64/include/asm/mshyperv.h
> > +F:	arch/arm64/hyperv
> >  F:	drivers/hid/hid-hyperv.c
> >  F:	drivers/hv/
> >  F:	drivers/input/serio/hyperv-keyboard.c
> > diff --git a/arch/arm64/Makefile b/arch/arm64/Makefile
> > index 6cb9fc7e9382..ad9ec0579553 100644
> > --- a/arch/arm64/Makefile
> > +++ b/arch/arm64/Makefile
> > @@ -106,6 +106,7 @@ core-y		+= arch/arm64/kernel/ arch/arm64/mm/
> >  core-$(CONFIG_NET) += arch/arm64/net/
> >  core-$(CONFIG_KVM) += arch/arm64/kvm/
> >  core-$(CONFIG_XEN) += arch/arm64/xen/
> > +core-$(CONFIG_HYPERV) += arch/arm64/hyperv/
> >  core-$(CONFIG_CRYPTO) += arch/arm64/crypto/
> >  libs-y		:= arch/arm64/lib/ $(libs-y)
> >  core-$(CONFIG_EFI_STUB) += $(objtree)/drivers/firmware/efi/libstub/lib.a
> > diff --git a/arch/arm64/hyperv/Makefile b/arch/arm64/hyperv/Makefile
> > new file mode 100644
> > index 000000000000..988eda55330c
> > --- /dev/null
> > +++ b/arch/arm64/hyperv/Makefile
> > @@ -0,0 +1,2 @@
> > +# SPDX-License-Identifier: GPL-2.0
> > +obj-y		:= hv_init.o hv_hvc.o mshyperv.o
> > diff --git a/arch/arm64/hyperv/hv_hvc.S b/arch/arm64/hyperv/hv_hvc.S
> > new file mode 100644
> > index 000000000000..82636969b4f2
> > --- /dev/null
> > +++ b/arch/arm64/hyperv/hv_hvc.S
> > @@ -0,0 +1,54 @@
> > +/* SPDX-License-Identifier: GPL-2.0 */
> > +
> > +/*
> > + * Microsoft Hyper-V hypervisor invocation routines
> > + *
> > + * Copyright (C) 2018, Microsoft, Inc.
> > + *
> > + * Author : Michael Kelley <mikelley@microsoft.com>
> > + *
> > + * This program is free software; you can redistribute it and/or modify it
> > + * under the terms of the GNU General Public License version 2 as published
> > + * by the Free Software Foundation.
> > + *
> > + * This program is distributed in the hope that it will be useful, but
> > + * WITHOUT ANY WARRANTY; without even the implied warranty of
> > + * MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE, GOOD TITLE or
> > + * NON INFRINGEMENT.  See the GNU General Public License for more
> > + * details.
> > + */
> > +
> > +#include <linux/linkage.h>
> > +
> > +	.text
> > +/*
> > + * Do the HVC instruction.  For Hyper-V the argument is always 1.
> > + * x0 contains the hypercall control value, while additional registers
> > + * vary depending on the hypercall, and whether the hypercall arguments
> > + * are in memory or in registers (a "fast" hypercall per the Hyper-V
> > + * TLFS).  When the arguments are in memory x1 is the guest physical
> > + * address of the input arguments, and x2 is the guest physical
> > + * address of the output arguments.  When the arguments are in
> > + * registers, the register values depends on the hypercall.  Note
> > + * that this version cannot return any values in registers.
> > + */
> > +ENTRY(hv_do_hvc)
> > +	hvc #1
> > +	ret
> > +ENDPROC(hv_do_hvc)
> > +
> > +/*
> > + * This variant of HVC invocation is for hv_get_vpreg and
> > + * hv_get_vpreg_128. The input parameters are passed in registers
> > + * along with a pointer in x4 to where the output result should
> > + * be stored. The output is returned in x15 and x16.  x18 is used as
> > + * scratch space to avoid buildng a stack frame, as Hyper-V does
> > + * not preserve registers x0-x17.
> > + */
> > +ENTRY(hv_do_hvc_fast_get)
> > +	mov x18, x4
> > +	hvc #1
> > +	str x15,[x18]
> > +	str x16,[x18,#8]
> > +	ret
> > +ENDPROC(hv_do_hvc_fast_get)
> 
> As Will said, this isn't a viable option. Please follow SMCCC 1.1.

I'll have to start a conversation with the Hyper-V team about this.
I don't know why they chose to use HVC #1 or this register scheme
for output values.  It may be tough to change at this point because
there are Windows guests on Hyper-V for ARM64 that are already
using this approach.

> 
> > diff --git a/arch/arm64/hyperv/hv_init.c b/arch/arm64/hyperv/hv_init.c
> > new file mode 100644
> > index 000000000000..aa1a8c09d989
> > --- /dev/null
> > +++ b/arch/arm64/hyperv/hv_init.c
> > @@ -0,0 +1,441 @@
> > +// SPDX-License-Identifier: GPL-2.0
> > +
> > +/*
> > + * Initialization of the interface with Microsoft's Hyper-V hypervisor,
> > + * and various low level utility routines for interacting with Hyper-V.
> > + *
> > + * Copyright (C) 2018, Microsoft, Inc.
> > + *
> > + * Author : Michael Kelley <mikelley@microsoft.com>
> > + *
> > + * This program is free software; you can redistribute it and/or modify it
> > + * under the terms of the GNU General Public License version 2 as published
> > + * by the Free Software Foundation.
> > + *
> > + * This program is distributed in the hope that it will be useful, but
> > + * WITHOUT ANY WARRANTY; without even the implied warranty of
> > + * MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE, GOOD TITLE or
> > + * NON INFRINGEMENT.  See the GNU General Public License for more
> > + * details.
> > + */
> > +
> > +
> > +#include <linux/types.h>
> > +#include <linux/version.h>
> > +#include <linux/export.h>
> > +#include <linux/vmalloc.h>
> > +#include <linux/mm.h>
> > +#include <linux/clocksource.h>
> > +#include <linux/sched_clock.h>
> > +#include <linux/acpi.h>
> > +#include <linux/module.h>
> > +#include <linux/hyperv.h>
> > +#include <linux/slab.h>
> > +#include <linux/cpuhotplug.h>
> > +#include <linux/psci.h>
> > +#include <asm-generic/bug.h>
> > +#include <asm/hypervisor.h>
> > +#include <asm/hyperv-tlfs.h>
> > +#include <asm/mshyperv.h>
> > +#include <asm/sysreg.h>
> > +#include <clocksource/arm_arch_timer.h>
> > +
> > +static bool	hyperv_initialized;
> > +struct		ms_hyperv_info ms_hyperv;
> > +EXPORT_SYMBOL_GPL(ms_hyperv);
> 
> Who are the users of this structure? Should they go via accessors instead?

The structure is an aggregation of several flags fields that describe a myriad
of features and hints that may or may not be present on any particular version
of Hyper-V, plus the max virtual processor ID values.  Everything is read-only
after initialization.   Most of the references are to test one of the flags.  It's
a judgment call, but there are a lot of different flags with long names, and
writing accessors for each one doesn't seem to me to add any clarity.

> 
> > +
> > +static struct ms_hyperv_tsc_page *tsc_pg;
> > +
> > +struct ms_hyperv_tsc_page *hv_get_tsc_page(void)
> > +{
> > +	return tsc_pg;
> > +}
> > +EXPORT_SYMBOL_GPL(hv_get_tsc_page);
> > +
> > +static u64 read_hv_sched_clock_tsc(void)
> > +{
> > +	u64 current_tick = hv_read_tsc_page(tsc_pg);
> > +
> > +	if (current_tick == U64_MAX)
> > +		current_tick = hv_get_vpreg(HV_REGISTER_TIME_REFCOUNT);
> > +
> > +	return current_tick;
> > +}
> > +
> > +static u64 read_hv_clock_tsc(struct clocksource *arg)
> > +{
> > +	u64 current_tick = hv_read_tsc_page(tsc_pg);
> > +
> > +	if (current_tick == U64_MAX)
> > +		current_tick = hv_get_vpreg(HV_REGISTER_TIME_REFCOUNT);
> > +
> > +	return current_tick;
> > +}
> 
> Can't you define one function in terms of the other?

Yes, it looks like that works.  I'll change it in the next version of the
patch.

> 
> > +
> > +static struct clocksource hyperv_cs_tsc = {
> > +		.name		= "hyperv_clocksource_tsc_page",
> > +		.rating		= 400,
> > +		.read		= read_hv_clock_tsc,
> > +		.mask		= CLOCKSOURCE_MASK(64),
> > +		.flags		= CLOCK_SOURCE_IS_CONTINUOUS,
> > +};
> > +
> > +static u64 read_hv_sched_clock_msr(void)
> > +{
> > +	return hv_get_vpreg(HV_REGISTER_TIME_REFCOUNT);
> > +}
> > +
> > +static u64 read_hv_clock_msr(struct clocksource *arg)
> > +{
> > +	return hv_get_vpreg(HV_REGISTER_TIME_REFCOUNT);
> > +}
> > +
> > +static struct clocksource hyperv_cs_msr = {
> > +	.name		= "hyperv_clocksource_msr",
> > +	.rating		= 400,
> > +	.read		= read_hv_clock_msr,
> > +	.mask		= CLOCKSOURCE_MASK(64),
> > +	.flags		= CLOCK_SOURCE_IS_CONTINUOUS,
> > +};
> > +
> > +struct clocksource *hyperv_cs;
> > +EXPORT_SYMBOL_GPL(hyperv_cs);
> 
> Why? Who needs to poke this?

It's referenced in the architecture independent driver code
for the Hyper-V clocksource in drivers/hv/hv.c, and in the
code to sync the time with the Hyper-V host in
drivers/hv/hv_util.c.

> 
> > +
> > +u32 *hv_vp_index;
> > +EXPORT_SYMBOL_GPL(hv_vp_index);
> > +
> > +u32 hv_max_vp_index;
> 
> Same thing.

Agreed -- this one doesn't seem to be needed.  The variable
is not used outside of the architecture dependent Hyper-V code.

> 
> > +
> > +static int hv_cpu_init(unsigned int cpu)
> > +{
> > +	u64 msr_vp_index;
> > +
> > +	hv_get_vp_index(msr_vp_index);
> > +
> > +	hv_vp_index[smp_processor_id()] = msr_vp_index;
> > +
> > +	if (msr_vp_index > hv_max_vp_index)
> > +		hv_max_vp_index = msr_vp_index;
> > +
> > +	return 0;
> > +}
> 
> Is that some new way to describe a CPU topology? If so, why isn't that
> exposed via the ACPI tables that the kernel already parses?

Hyper-V's hypercall interface uses vCPU identifiers that are not
guaranteed to be consecutive integers or to match what ACPI shows.
No topology information is implied -- it's just unique identifiers.  The 
hv_vp_index array provides easy mapping from Linux's consecutive
integer IDs for CPUs when needed to construct hypercall arguments.

> 
> > +
> > +/*
> > + * This function is invoked via the ACPI clocksource probe mechanism. We
> > + * don't actually use any values from the ACPI GTDT table, but we set up
> 
> This doesn't feel like a good idea at all. Piggy-backing on an existing
> mechanism and use it for something completely different is not exactly
> future-proof.
> 
> Also, if this is supposed to be a clocksource, why isn't that a
> clocksource driver on its own right?

I agree this is not the right long term solution.  Is there a better place to
hang the initialization code?  Or should I just make an explicit call to
initialize Hyper-V at the right place?  On the x86 side, there's an
explicit framework for hypervisor-specific initialization routines to plug
into.  Maybe it's time for a basic version of such a framework on the
ARM64 side.  Thoughts on the best approach, both in the short-term and
the longer-term? If we put a framework in place, does that need to
happen before adding Hyper-V code, or afterwards as a cleanup?

And yes, Hyper-V does effectively have its own clocksource.  The
main code is in drivers/hv/hv.c, but it's not broken out as a separate
driver in drivers/clocksource, probably due to some history on the
x86 side that pre-dates me.  I'll have to research.

> 
> > + * the Hyper-V synthetic clocksource and do other initialization for
> > + * interacting with Hyper-V the first time.  Using early_initcall to invoke
> > + * this function is too late because interrupts are already enabled at that
> > + * point, and sched_clock_register must run before interrupts are enabled.
> > + *
> > + * 1. Setup the guest ID.
> > + * 2. Get features and hints info from Hyper-V
> > + * 3. Setup per-cpu VP indices.
> > + * 4. Register Hyper-V specific clocksource.
> > + * 5. Register the scheduler clock.
> > + */
> > +
> > +static int __init hyperv_init(struct acpi_table_header *table)
> > +{
> > +	struct hv_get_vp_register_output result;
> > +	u32	a, b, c, d;
> > +	u64	guest_id;
> > +	int	i;
> > +
> > +	/*
> > +	 * If we're in a VM on Hyper-V, the ACPI hypervisor_id field will
> > +	 * have the string "MsHyperV".
> > +	 */
> > +	if (strncmp((char *)&acpi_gbl_FADT.hypervisor_id, "MsHyperV", 8))
> > +		return 1;
> > +
> > +	/* Setup the guest ID */
> > +	guest_id = generate_guest_id(0, LINUX_VERSION_CODE, 0);
> > +	hv_set_vpreg(HV_REGISTER_GUEST_OSID, guest_id);
> > +
> > +	/* Get the features and hints from Hyper-V */
> > +	hv_get_vpreg_128(HV_REGISTER_PRIVILEGES_AND_FEATURES, &result);
> > +	ms_hyperv.features = lower_32_bits(result.registervaluelow);
> > +	ms_hyperv.misc_features = upper_32_bits(result.registervaluehigh);
> > +
> > +	hv_get_vpreg_128(HV_REGISTER_FEATURES, &result);
> > +	ms_hyperv.hints = lower_32_bits(result.registervaluelow);
> > +
> > +	pr_info("Hyper-V: Features 0x%x, hints 0x%x\n",
> > +		ms_hyperv.features, ms_hyperv.hints);
> > +
> > +	/*
> > +	 * Direct mode is the only option for STIMERs provided Hyper-V
> > +	 * on ARM64, so Hyper-V doesn't actually set the flag.  But add the
> > +	 * flag so the architecture independent code in drivers/hv/hv.c
> > +	 * will correctly use that mode.
> > +	 */
> > +	ms_hyperv.misc_features |= HV_STIMER_DIRECT_MODE_AVAILABLE;
> > +
> > +	/*
> > +	 * Hyper-V on ARM64 doesn't support AutoEOI.  Add the hint
> > +	 * that tells architecture independent code not to use this
> > +	 * feature.
> > +	 */
> > +	ms_hyperv.hints |= HV_DEPRECATING_AEOI_RECOMMENDED;
> > +
> > +	/* Get information about the Hyper-V host version */
> > +	hv_get_vpreg_128(HV_REGISTER_HYPERVISOR_VERSION, &result);
> > +	a = lower_32_bits(result.registervaluelow);
> > +	b = upper_32_bits(result.registervaluelow);
> > +	c = lower_32_bits(result.registervaluehigh);
> > +	d = upper_32_bits(result.registervaluehigh);
> > +	pr_info("Hyper-V: Host Build %d.%d.%d.%d-%d-%d\n",
> > +		b >> 16, b & 0xFFFF, a, d & 0xFFFFFF, c, d >> 24);
> > +
> > +	/* Allocate percpu VP index */
> > +	hv_vp_index = kmalloc_array(num_possible_cpus(), sizeof(*hv_vp_index),
> > +				    GFP_KERNEL);
> 
> Why isn't this a percpu variable?

In current code in the architecture independent Hyper-V drivers (as well
as some future Hyper-V enlightenments that aren't yet implemented
for ARM64), the running CPU needs to get the VP index values for any CPUs
in the hv_vp_index array.  Some of the code is performance sensitive, and
accessing a global array is faster than accessing other CPUs' per-cpu data.

> 
> > +	if (!hv_vp_index)
> > +		return 1;
> > +
> > +	for (i = 0; i < num_possible_cpus(); i++)
> > +		hv_vp_index[i] = VP_INVAL;
> > +
> > +	if (cpuhp_setup_state(CPUHP_AP_ONLINE_DYN, "arm64/hyperv_init:online",
> > +			      hv_cpu_init, NULL) < 0)
> > +		goto free_vp_index;
> > +
> > +	/*
> > +	 * Try to set up what Hyper-V calls the "TSC reference page", which
> > +	 * uses the ARM Generic Timer virtual counter with some scaling
> > +	 * information to provide a fast and stable guest VM clocksource.
> > +	 * If the TSC reference page can't be set up, fall back to reading
> > +	 * the guest clock provided by Hyper-V's synthetic reference time
> > +	 * register.
> > +	 */
> > +	if (ms_hyperv.features & HV_MSR_REFERENCE_TSC_AVAILABLE) {
> > +
> > +		u64		tsc_msr;
> > +		phys_addr_t	phys_addr;
> > +
> > +		tsc_pg = __vmalloc(HV_HYP_PAGE_SIZE, GFP_KERNEL, PAGE_KERNEL);
> 
> And why not vmalloc?

Looks like it could be just vmalloc().  I'll change it in the next version.

> 
> > +		if (tsc_pg) {
> > +			phys_addr = page_to_phys(vmalloc_to_page(tsc_pg));
> > +			tsc_msr = hv_get_vpreg(HV_REGISTER_REFERENCE_TSC);
> > +			tsc_msr &= GENMASK_ULL(11, 0);
> 
> Magic.

Per Section 12.6.1 of the Hyper-V TLFS, bits 11 thru 1 must be preserved, and
the line below sets bit 0.  I'll add some comments. :-)

> 
> > +			tsc_msr = tsc_msr | 0x1 | (u64)phys_addr;
> > +			hv_set_vpreg(HV_REGISTER_REFERENCE_TSC, tsc_msr);
> > +			hyperv_cs = &hyperv_cs_tsc;
> > +			sched_clock_register(read_hv_sched_clock_tsc,
> > +						64, HV_CLOCK_HZ);
> > +		}
> > +	}
> > +
> > +	if (!hyperv_cs &&
> > +	    (ms_hyperv.features & HV_MSR_TIME_REF_COUNT_AVAILABLE)) {
> > +		hyperv_cs = &hyperv_cs_msr;
> > +		sched_clock_register(read_hv_sched_clock_msr,
> > +						64, HV_CLOCK_HZ);
> > +	}
> > +
> > +	if (hyperv_cs) {
> > +		hyperv_cs->archdata.vdso_direct = false;
> > +		clocksource_register_hz(hyperv_cs, HV_CLOCK_HZ);
> > +	}
> > +
> > +	hyperv_initialized = true;
> > +	return 0;
> > +
> > +free_vp_index:
> > +	kfree(hv_vp_index);
> > +	hv_vp_index = NULL;
> > +	return 1;
> 
> ????

The return value is there because this function is implemented as a timer
initialization function, and that's what the function signature requires.
But maybe I'm not understanding your question.

> 
> > +}
> > +TIMER_ACPI_DECLARE(hyperv, ACPI_SIG_GTDT, hyperv_init);
> > +
> > +/*
> > + * This routine is called before kexec/kdump, it does the required cleanup.
> > + */
> > +void hyperv_cleanup(void)
> > +{
> > +	/* Reset our OS id */
> > +	hv_set_vpreg(HV_REGISTER_GUEST_OSID, 0);
> > +
> > +}
> > +EXPORT_SYMBOL_GPL(hyperv_cleanup);
> > +
> > +/*
> > + * hv_do_hypercall- Invoke the specified hypercall
> > + */
> > +u64 hv_do_hypercall(u64 control, void *input, void *output)
> > +{
> > +	u64 input_address;
> > +	u64 output_address;
> > +
> > +	input_address = input ? virt_to_phys(input) : 0;
> > +	output_address = output ? virt_to_phys(output) : 0;
> > +	return hv_do_hvc(control, input_address, output_address);
> > +}
> > +EXPORT_SYMBOL_GPL(hv_do_hypercall);
> > +
> > +/*
> > + * hv_do_fast_hypercall8 -- Invoke the specified hypercall
> > + * with arguments in registers instead of physical memory.
> > + * Avoids the overhead of virt_to_phys for simple hypercalls.
> > + */
> > +
> > +u64 hv_do_fast_hypercall8(u16 code, u64 input)
> > +{
> > +	u64 control;
> > +
> > +	control = (u64)code | HV_HYPERCALL_FAST_BIT;
> > +	return hv_do_hvc(control, input);
> > +}
> > +EXPORT_SYMBOL_GPL(hv_do_fast_hypercall8);
> > +
> > +
> > +/*
> > + * Set a single VP register to a 64-bit value.
> > + */
> > +void hv_set_vpreg(u32 msr, u64 value)
> > +{
> > +	union hv_hypercall_status status;
> > +
> > +	status.as_uint64 = hv_do_hvc(
> > +		HVCALL_SET_VP_REGISTERS | HV_HYPERCALL_FAST_BIT |
> > +			HV_HYPERCALL_REP_COUNT_1,
> > +		HV_PARTITION_ID_SELF,
> > +		HV_VP_INDEX_SELF,
> > +		msr,
> > +		0,
> > +		value,
> > +		0);
> > +
> > +	/*
> > +	 * Something is fundamentally broken in the hypervisor if
> > +	 * setting a VP register fails. There's really no way to
> > +	 * continue as a guest VM, so panic.
> > +	 */
> > +	BUG_ON(status.status != HV_STATUS_SUCCESS);
> > +}
> > +EXPORT_SYMBOL_GPL(hv_set_vpreg);
> > +
> > +
> > +/*
> > + * Get the value of a single VP register, and only the low order 64 bits.
> > + */
> > +u64 hv_get_vpreg(u32 msr)
> > +{
> > +	union hv_hypercall_status status;
> > +	struct hv_get_vp_register_output output;
> > +
> > +	status.as_uint64 = hv_do_hvc_fast_get(
> > +		HVCALL_GET_VP_REGISTERS | HV_HYPERCALL_FAST_BIT |
> > +			HV_HYPERCALL_REP_COUNT_1,
> > +		HV_PARTITION_ID_SELF,
> > +		HV_VP_INDEX_SELF,
> > +		msr,
> > +		&output);
> > +
> > +	/*
> > +	 * Something is fundamentally broken in the hypervisor if
> > +	 * getting a VP register fails. There's really no way to
> > +	 * continue as a guest VM, so panic.
> > +	 */
> > +	BUG_ON(status.status != HV_STATUS_SUCCESS);
> > +
> > +	return output.registervaluelow;
> > +}
> > +EXPORT_SYMBOL_GPL(hv_get_vpreg);
> > +
> > +/*
> > + * Get the value of a single VP register that is 128 bits in size.  This is a
> > + * separate call in order to avoid complicating the calling sequence for
> > + * the much more frequently used 64-bit version.
> > + */
> > +void hv_get_vpreg_128(u32 msr, struct hv_get_vp_register_output *result)
> > +{
> > +	union hv_hypercall_status status;
> > +
> > +	status.as_uint64 = hv_do_hvc_fast_get(
> > +		HVCALL_GET_VP_REGISTERS | HV_HYPERCALL_FAST_BIT |
> > +			HV_HYPERCALL_REP_COUNT_1,
> > +		HV_PARTITION_ID_SELF,
> > +		HV_VP_INDEX_SELF,
> > +		msr,
> > +		result);
> > +
> > +	/*
> > +	 * Something is fundamentally broken in the hypervisor if
> > +	 * getting a VP register fails. There's really no way to
> > +	 * continue as a guest VM, so panic.
> > +	 */
> > +	BUG_ON(status.status != HV_STATUS_SUCCESS);
> > +
> > +	return;
> > +
> > +}
> > +EXPORT_SYMBOL_GPL(hv_get_vpreg_128);
> > +
> > +void hyperv_report_panic(struct pt_regs *regs, long err)
> > +{
> > +	static bool panic_reported;
> > +	u64 guest_id;
> > +
> > +	/*
> > +	 * We prefer to report panic on 'die' chain as we have proper
> > +	 * registers to report, but if we miss it (e.g. on BUG()) we need
> > +	 * to report it on 'panic'.
> > +	 */
> > +	if (panic_reported)
> > +		return;
> > +	panic_reported = true;
> > +
> > +	guest_id = hv_get_vpreg(HV_REGISTER_GUEST_OSID);
> > +
> > +	/*
> > +	 * Hyper-V provides the ability to store only 5 values.
> > +	 * Pick the passed in error value, the guest_id, and the PC.
> > +	 * The first two general registers are added arbitrarily.
> > +	 */
> > +	hv_set_vpreg(HV_REGISTER_CRASH_P0, err);
> > +	hv_set_vpreg(HV_REGISTER_CRASH_P1, guest_id);
> > +	hv_set_vpreg(HV_REGISTER_CRASH_P2, regs->pc);
> > +	hv_set_vpreg(HV_REGISTER_CRASH_P3, regs->regs[0]);
> > +	hv_set_vpreg(HV_REGISTER_CRASH_P4, regs->regs[1]);
> > +
> > +	/*
> > +	 * Let Hyper-V know there is crash data available
> > +	 */
> > +	hv_set_vpreg(HV_REGISTER_CRASH_CTL, HV_CRASH_CTL_CRASH_NOTIFY);
> > +}
> > +EXPORT_SYMBOL_GPL(hyperv_report_panic);
> > +
> > +/*
> > + * hyperv_report_panic_msg - report panic message to Hyper-V
> > + * @pa: physical address of the panic page containing the message
> > + * @size: size of the message in the page
> > + */
> > +void hyperv_report_panic_msg(phys_addr_t pa, size_t size)
> > +{
> > +	/*
> > +	 * P3 to contain the physical address of the panic page & P4 to
> > +	 * contain the size of the panic data in that page. Rest of the
> > +	 * registers are no-op when the NOTIFY_MSG flag is set.
> > +	 */
> > +	hv_set_vpreg(HV_REGISTER_CRASH_P0, 0);
> > +	hv_set_vpreg(HV_REGISTER_CRASH_P1, 0);
> > +	hv_set_vpreg(HV_REGISTER_CRASH_P2, 0);
> > +	hv_set_vpreg(HV_REGISTER_CRASH_P3, pa);
> > +	hv_set_vpreg(HV_REGISTER_CRASH_P4, size);
> > +
> > +	/*
> > +	 * Let Hyper-V know there is crash data available along with
> > +	 * the panic message.
> > +	 */
> > +	hv_set_vpreg(HV_REGISTER_CRASH_CTL,
> > +	       (HV_CRASH_CTL_CRASH_NOTIFY | HV_CRASH_CTL_CRASH_NOTIFY_MSG));
> > +}
> > +EXPORT_SYMBOL_GPL(hyperv_report_panic_msg);
> > +
> > +bool hv_is_hyperv_initialized(void)
> > +{
> > +	return hyperv_initialized;
> > +}
> > +EXPORT_SYMBOL_GPL(hv_is_hyperv_initialized);
> > diff --git a/arch/arm64/hyperv/mshyperv.c b/arch/arm64/hyperv/mshyperv.c
> > new file mode 100644
> > index 000000000000..3ef055599412
> > --- /dev/null
> > +++ b/arch/arm64/hyperv/mshyperv.c
> > @@ -0,0 +1,178 @@
> > +// SPDX-License-Identifier: GPL-2.0
> > +
> > +/*
> > + * Core routines for interacting with Microsoft's Hyper-V hypervisor,
> > + * including setting up VMbus and STIMER interrupts, and handling
> > + * crashes and kexecs. These interactions are through a set of
> > + * static "handler" variables set by the architecture independent
> > + * VMbus and STIMER drivers.  This design is used to meet x86/x64
> > + * requirements for avoiding direct linkages and allowing the VMbus
> > + * and STIMER drivers to be unloaded and reloaded.
> > + *
> > + * Copyright (C) 2018, Microsoft, Inc.
> > + *
> > + * Author : Michael Kelley <mikelley@microsoft.com>
> > + *
> > + * This program is free software; you can redistribute it and/or modify it
> > + * under the terms of the GNU General Public License version 2 as published
> > + * by the Free Software Foundation.
> > + *
> > + * This program is distributed in the hope that it will be useful, but
> > + * WITHOUT ANY WARRANTY; without even the implied warranty of
> > + * MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE, GOOD TITLE or
> > + * NON INFRINGEMENT.  See the GNU General Public License for more
> > + * details.
> > + */
> > +
> > +#include <linux/types.h>
> > +#include <linux/export.h>
> > +#include <linux/interrupt.h>
> > +#include <linux/kexec.h>
> > +#include <linux/acpi.h>
> > +#include <linux/ptrace.h>
> > +#include <asm/hyperv-tlfs.h>
> > +#include <asm/mshyperv.h>
> > +
> > +static void (*vmbus_handler)(void);
> > +static void (*hv_stimer0_handler)(void);
> > +static void (*hv_kexec_handler)(void);
> > +static void (*hv_crash_handler)(struct pt_regs *regs);
> > +
> > +static int vmbus_irq;
> > +static long __percpu *vmbus_evt;
> > +static long __percpu *stimer0_evt;
> > +
> > +irqreturn_t hyperv_vector_handler(int irq, void *dev_id)
> > +{
> > +	if (vmbus_handler)
> > +		vmbus_handler();
> 
> In which circumstances can this be NULL?

Perhaps none.  This may be a leftover from the equivalent
x86 code.  I'll check and probably remove it.

> 
> > +	return IRQ_HANDLED;
> > +}
> > +
> > +/* Must be done just once */
> > +void hv_setup_vmbus_irq(void (*handler)(void))
> > +{
> > +	int result;
> > +
> > +	vmbus_handler = handler;
> 
> If thismust only be done once, maybe you could check that it hasn't been
> done already?

OK.

> 
> > +	vmbus_irq = acpi_register_gsi(NULL, HYPERVISOR_CALLBACK_VECTOR,
> > +				 ACPI_LEVEL_SENSITIVE, ACPI_ACTIVE_HIGH);
> > +	if (vmbus_irq <= 0) {
> > +		pr_err("Can't register Hyper-V VMBus GSI. Error %d",
> > +			vmbus_irq);
> > +		vmbus_irq = 0;
> > +		return;
> > +	}
> > +	vmbus_evt = alloc_percpu(long);
> > +	result = request_percpu_irq(vmbus_irq, hyperv_vector_handler,
> > +			"Hyper-V VMbus", vmbus_evt);
> 
> If this is a per-cpu interrupt, why isn't it signalled as a PPI, in an
> architecture compliant way?

Except for the code in this module, the interrupt handler for VMbus
interrupts is architecture independent.  But there's no support for
per-process interrupts on the x86, so the hypervisor interrupt vectors
are hard coded in the same IDT entry across all processors, and the
normal IRQ allocation mechanism is bypassed.  The above approach
assigns an ARM64 PPI (HYPERVISOR_CALLBACK_VECTOR is 16) in a
way that works with the arch independent interrupt handler.

Or maybe I'm missing your point.  If so, please set me straight.

> 
> > +	if (result) {
> > +		pr_err("Can't request Hyper-V VMBus IRQ %d. Error %d",
> > +			vmbus_irq, result);
> > +		free_percpu(vmbus_evt);
> > +		acpi_unregister_gsi(vmbus_irq);
> > +		vmbus_irq = 0;
> > +	}
> > +}
> > +EXPORT_SYMBOL_GPL(hv_setup_vmbus_irq);
> > +
> > +/* Must be done just once */
> > +void hv_remove_vmbus_irq(void)
> > +{
> > +	if (vmbus_irq) {
> > +		free_percpu_irq(vmbus_irq, vmbus_evt);
> > +		free_percpu(vmbus_evt);
> > +		acpi_unregister_gsi(vmbus_irq);
> > +	}
> > +}
> > +EXPORT_SYMBOL_GPL(hv_remove_vmbus_irq);
> > +
> > +/* Must be done by each CPU */
> > +void hv_enable_vmbus_irq(void)
> > +{
> > +	enable_percpu_irq(vmbus_irq, 0);
> > +}
> > +EXPORT_SYMBOL_GPL(hv_enable_vmbus_irq);
> > +
> > +/* Must be done by each CPU */
> > +void hv_disable_vmbus_irq(void)
> > +{
> > +	disable_percpu_irq(vmbus_irq);
> > +}
> > +EXPORT_SYMBOL_GPL(hv_disable_vmbus_irq);
> > +
> > +/* Routines to do per-architecture handling of STIMER0 when in Direct Mode */
> > +
> > +static irqreturn_t hv_stimer0_vector_handler(int irq, void *dev_id)
> > +{
> > +	if (hv_stimer0_handler)
> > +		hv_stimer0_handler();
> > +	return IRQ_HANDLED;
> > +}
> > +
> > +int hv_setup_stimer0_irq(int *irq, int *vector, void (*handler)(void))
> > +{
> > +	int localirq;
> > +	int result;
> > +
> > +	localirq = acpi_register_gsi(NULL, HV_STIMER0_IRQNR,
> > +			ACPI_LEVEL_SENSITIVE, ACPI_ACTIVE_HIGH);
> > +	if (localirq <= 0) {
> > +		pr_err("Can't register Hyper-V stimer0 GSI. Error %d",
> > +			localirq);
> > +		*irq = 0;
> > +		return -1;
> > +	}
> > +	stimer0_evt = alloc_percpu(long);
> > +	result = request_percpu_irq(localirq, hv_stimer0_vector_handler,
> > +					 "Hyper-V stimer0", stimer0_evt);
> > +	if (result) {
> > +		pr_err("Can't request Hyper-V stimer0 IRQ %d. Error %d",
> > +			localirq, result);
> > +		free_percpu(stimer0_evt);
> > +		acpi_unregister_gsi(localirq);
> > +		*irq = 0;
> > +		return -1;
> > +	}
> > +
> > +	hv_stimer0_handler = handler;
> > +	*vector = HV_STIMER0_IRQNR;
> > +	*irq = localirq;
> > +	return 0;
> > +}
> > +EXPORT_SYMBOL_GPL(hv_setup_stimer0_irq);
> > +
> > +void hv_remove_stimer0_irq(int irq)
> > +{
> > +	hv_stimer0_handler = NULL;
> > +	if (irq) {
> > +		free_percpu_irq(irq, stimer0_evt);
> > +		free_percpu(stimer0_evt);
> > +		acpi_unregister_gsi(irq);
> > +	}
> > +}
> > +EXPORT_SYMBOL_GPL(hv_remove_stimer0_irq);
> > +
> > +void hv_setup_kexec_handler(void (*handler)(void))
> > +{
> > +	hv_kexec_handler = handler;
> > +}
> > +EXPORT_SYMBOL_GPL(hv_setup_kexec_handler);
> > +
> > +void hv_remove_kexec_handler(void)
> > +{
> > +	hv_kexec_handler = NULL;
> > +}
> > +EXPORT_SYMBOL_GPL(hv_remove_kexec_handler);
> > +
> > +void hv_setup_crash_handler(void (*handler)(struct pt_regs *regs))
> > +{
> > +	hv_crash_handler = handler;
> > +}
> > +EXPORT_SYMBOL_GPL(hv_setup_crash_handler);
> > +
> > +void hv_remove_crash_handler(void)
> > +{
> > +	hv_crash_handler = NULL;
> > +}
> > +EXPORT_SYMBOL_GPL(hv_remove_crash_handler);
> >
> 
> Overall, this patch needs a massive split up, clean up, and a good dose
> of documentation so that we can understand what you are trying to achieve.

Thanks for the feedback.  I'll start work on a new version for the
changes that are clear, and await additional discussion on
a few issues where I need some further input.

Michael

> 
> Thanks,
> 
> 	M.
> --
> Jazz is not dead. It just smells funny...
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 2/4] arm64: hyperv: Add support for Hyper-V as a hypervisor

[Marc Zyngier]

Hi Michael,

On 12/12/2018 05:00, Michael Kelley wrote:
> From: Marc Zyngier <marc.zyngier@arm.com>  Sent: Friday, December 7, 2018 6:43 AM
> 
>>> Add ARM64-specific code to enable Hyper-V. This code includes:
>>> * Detecting Hyper-V and initializing the guest/Hyper-V interface
>>> * Setting up Hyper-V's synthetic clocks
>>> * Making hypercalls using the HVC instruction
>>> * Setting up VMbus and stimer0 interrupts
>>> * Setting up kexec and crash handlers
>>
>> This commit message is a clear indication that this should be split in
>> at least 5 different patches.
> 
> OK, I'll work on separating into multiple layered patches in the next
> version.

Thanks. This will definitely help the review process.

>>> +/*
>>> + * This variant of HVC invocation is for hv_get_vpreg and
>>> + * hv_get_vpreg_128. The input parameters are passed in registers
>>> + * along with a pointer in x4 to where the output result should
>>> + * be stored. The output is returned in x15 and x16.  x18 is used as
>>> + * scratch space to avoid buildng a stack frame, as Hyper-V does
>>> + * not preserve registers x0-x17.
>>> + */
>>> +ENTRY(hv_do_hvc_fast_get)
>>> +	mov x18, x4
>>> +	hvc #1
>>> +	str x15,[x18]
>>> +	str x16,[x18,#8]
>>> +	ret
>>> +ENDPROC(hv_do_hvc_fast_get)
>>
>> As Will said, this isn't a viable option. Please follow SMCCC 1.1.
> 
> I'll have to start a conversation with the Hyper-V team about this.
> I don't know why they chose to use HVC #1 or this register scheme
> for output values.  It may be tough to change at this point because
> there are Windows guests on Hyper-V for ARM64 that are already
> using this approach.

I appreciate you already have stuff in the wild, but there is definitely
a case to be made for supporting architecturally specified mechanisms in
a hypervisor, and SMCCC is definitely part of it (I'm certainly curious
of how you support the Spectre mitigation otherwise).

> 
>>
>>> diff --git a/arch/arm64/hyperv/hv_init.c b/arch/arm64/hyperv/hv_init.c
>>> new file mode 100644
>>> index 000000000000..aa1a8c09d989
>>> --- /dev/null
>>> +++ b/arch/arm64/hyperv/hv_init.c
>>> @@ -0,0 +1,441 @@
>>> +// SPDX-License-Identifier: GPL-2.0
>>> +
>>> +/*
>>> + * Initialization of the interface with Microsoft's Hyper-V hypervisor,
>>> + * and various low level utility routines for interacting with Hyper-V.
>>> + *
>>> + * Copyright (C) 2018, Microsoft, Inc.
>>> + *
>>> + * Author : Michael Kelley <mikelley@microsoft.com>
>>> + *
>>> + * This program is free software; you can redistribute it and/or modify it
>>> + * under the terms of the GNU General Public License version 2 as published
>>> + * by the Free Software Foundation.
>>> + *
>>> + * This program is distributed in the hope that it will be useful, but
>>> + * WITHOUT ANY WARRANTY; without even the implied warranty of
>>> + * MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE, GOOD TITLE or
>>> + * NON INFRINGEMENT.  See the GNU General Public License for more
>>> + * details.
>>> + */
>>> +
>>> +
>>> +#include <linux/types.h>
>>> +#include <linux/version.h>
>>> +#include <linux/export.h>
>>> +#include <linux/vmalloc.h>
>>> +#include <linux/mm.h>
>>> +#include <linux/clocksource.h>
>>> +#include <linux/sched_clock.h>
>>> +#include <linux/acpi.h>
>>> +#include <linux/module.h>
>>> +#include <linux/hyperv.h>
>>> +#include <linux/slab.h>
>>> +#include <linux/cpuhotplug.h>
>>> +#include <linux/psci.h>
>>> +#include <asm-generic/bug.h>
>>> +#include <asm/hypervisor.h>
>>> +#include <asm/hyperv-tlfs.h>
>>> +#include <asm/mshyperv.h>
>>> +#include <asm/sysreg.h>
>>> +#include <clocksource/arm_arch_timer.h>
>>> +
>>> +static bool	hyperv_initialized;
>>> +struct		ms_hyperv_info ms_hyperv;
>>> +EXPORT_SYMBOL_GPL(ms_hyperv);
>>
>> Who are the users of this structure? Should they go via accessors instead?
> 
> The structure is an aggregation of several flags fields that describe a myriad
> of features and hints that may or may not be present on any particular version
> of Hyper-V, plus the max virtual processor ID values.  Everything is read-only
> after initialization.   

nit: please consider using a __ro_after_init annotation in that case.

> Most of the references are to test one of the flags.  It's
> a judgment call, but there are a lot of different flags with long names, and
> writing accessors for each one doesn't seem to me to add any clarity.

Looking at that structure, it doesn't seem so bad, and you could easily
have generic accessors that take a flag as a parameter. Your call.

[...]

>>> +static struct clocksource hyperv_cs_msr = {
>>> +	.name		= "hyperv_clocksource_msr",
>>> +	.rating		= 400,
>>> +	.read		= read_hv_clock_msr,
>>> +	.mask		= CLOCKSOURCE_MASK(64),
>>> +	.flags		= CLOCK_SOURCE_IS_CONTINUOUS,
>>> +};
>>> +
>>> +struct clocksource *hyperv_cs;
>>> +EXPORT_SYMBOL_GPL(hyperv_cs);
>>
>> Why? Who needs to poke this?
> 
> It's referenced in the architecture independent driver code
> for the Hyper-V clocksource in drivers/hv/hv.c, and in the
> code to sync the time with the Hyper-V host in
> drivers/hv/hv_util.c.

Fair enough.

>>> +static int hv_cpu_init(unsigned int cpu)
>>> +{
>>> +	u64 msr_vp_index;
>>> +
>>> +	hv_get_vp_index(msr_vp_index);
>>> +
>>> +	hv_vp_index[smp_processor_id()] = msr_vp_index;
>>> +
>>> +	if (msr_vp_index > hv_max_vp_index)
>>> +		hv_max_vp_index = msr_vp_index;
>>> +
>>> +	return 0;
>>> +}
>>
>> Is that some new way to describe a CPU topology? If so, why isn't that
>> exposed via the ACPI tables that the kernel already parses?
> 
> Hyper-V's hypercall interface uses vCPU identifiers that are not
> guaranteed to be consecutive integers or to match what ACPI shows.
> No topology information is implied -- it's just unique identifiers.  The 
> hv_vp_index array provides easy mapping from Linux's consecutive
> integer IDs for CPUs when needed to construct hypercall arguments.

That's extremely odd. The hypervisor obviously knows which vCPU is doing
a hypercall, and if referencing another vCPU, the virtualized MPIDR_EL1
value should be used. I don't think deviating from the architecture is a
good idea (but I appreciate this is none of your doing). Following the
architecture would allow this code to directly use the cpu_logical_map
infrastructure we alreadu have.

> 
>>
>>> +
>>> +/*
>>> + * This function is invoked via the ACPI clocksource probe mechanism. We
>>> + * don't actually use any values from the ACPI GTDT table, but we set up
>>
>> This doesn't feel like a good idea at all. Piggy-backing on an existing
>> mechanism and use it for something completely different is not exactly
>> future-proof.
>>
>> Also, if this is supposed to be a clocksource, why isn't that a
>> clocksource driver on its own right?
> 
> I agree this is not the right long term solution.  Is there a better place to
> hang the initialization code?  Or should I just make an explicit call to
> initialize Hyper-V at the right place?  On the x86 side, there's an
> explicit framework for hypervisor-specific initialization routines to plug
> into.  Maybe it's time for a basic version of such a framework on the
> ARM64 side.  Thoughts on the best approach, both in the short-term and
> the longer-term? If we put a framework in place, does that need to
> happen before adding Hyper-V code, or afterwards as a cleanup?

If we're introducing an infrastructure, doing it before introducing more
stuff seems to be the logical option. But I'm not sure we really need
such an infrastructure in this case, unless we need to enforce some
specific ordering.

You could start by moving all the clocksource stuff to
drivers/clocksource and keep the same init mechanism for the time being.

> 
> And yes, Hyper-V does effectively have its own clocksource.  The
> main code is in drivers/hv/hv.c, but it's not broken out as a separate
> driver in drivers/clocksource, probably due to some history on the
> x86 side that pre-dates me.  I'll have to research.

OK.

> 
>>
>>> + * the Hyper-V synthetic clocksource and do other initialization for
>>> + * interacting with Hyper-V the first time.  Using early_initcall to invoke
>>> + * this function is too late because interrupts are already enabled at that
>>> + * point, and sched_clock_register must run before interrupts are enabled.
>>> + *
>>> + * 1. Setup the guest ID.
>>> + * 2. Get features and hints info from Hyper-V
>>> + * 3. Setup per-cpu VP indices.
>>> + * 4. Register Hyper-V specific clocksource.
>>> + * 5. Register the scheduler clock.
>>> + */
>>> +
>>> +static int __init hyperv_init(struct acpi_table_header *table)
>>> +{
>>> +	struct hv_get_vp_register_output result;
>>> +	u32	a, b, c, d;
>>> +	u64	guest_id;
>>> +	int	i;
>>> +
>>> +	/*
>>> +	 * If we're in a VM on Hyper-V, the ACPI hypervisor_id field will
>>> +	 * have the string "MsHyperV".
>>> +	 */
>>> +	if (strncmp((char *)&acpi_gbl_FADT.hypervisor_id, "MsHyperV", 8))
>>> +		return 1;
>>> +
>>> +	/* Setup the guest ID */
>>> +	guest_id = generate_guest_id(0, LINUX_VERSION_CODE, 0);
>>> +	hv_set_vpreg(HV_REGISTER_GUEST_OSID, guest_id);
>>> +
>>> +	/* Get the features and hints from Hyper-V */
>>> +	hv_get_vpreg_128(HV_REGISTER_PRIVILEGES_AND_FEATURES, &result);
>>> +	ms_hyperv.features = lower_32_bits(result.registervaluelow);
>>> +	ms_hyperv.misc_features = upper_32_bits(result.registervaluehigh);
>>> +
>>> +	hv_get_vpreg_128(HV_REGISTER_FEATURES, &result);
>>> +	ms_hyperv.hints = lower_32_bits(result.registervaluelow);
>>> +
>>> +	pr_info("Hyper-V: Features 0x%x, hints 0x%x\n",
>>> +		ms_hyperv.features, ms_hyperv.hints);
>>> +
>>> +	/*
>>> +	 * Direct mode is the only option for STIMERs provided Hyper-V
>>> +	 * on ARM64, so Hyper-V doesn't actually set the flag.  But add the
>>> +	 * flag so the architecture independent code in drivers/hv/hv.c
>>> +	 * will correctly use that mode.
>>> +	 */
>>> +	ms_hyperv.misc_features |= HV_STIMER_DIRECT_MODE_AVAILABLE;
>>> +
>>> +	/*
>>> +	 * Hyper-V on ARM64 doesn't support AutoEOI.  Add the hint
>>> +	 * that tells architecture independent code not to use this
>>> +	 * feature.
>>> +	 */
>>> +	ms_hyperv.hints |= HV_DEPRECATING_AEOI_RECOMMENDED;
>>> +
>>> +	/* Get information about the Hyper-V host version */
>>> +	hv_get_vpreg_128(HV_REGISTER_HYPERVISOR_VERSION, &result);
>>> +	a = lower_32_bits(result.registervaluelow);
>>> +	b = upper_32_bits(result.registervaluelow);
>>> +	c = lower_32_bits(result.registervaluehigh);
>>> +	d = upper_32_bits(result.registervaluehigh);
>>> +	pr_info("Hyper-V: Host Build %d.%d.%d.%d-%d-%d\n",
>>> +		b >> 16, b & 0xFFFF, a, d & 0xFFFFFF, c, d >> 24);
>>> +
>>> +	/* Allocate percpu VP index */
>>> +	hv_vp_index = kmalloc_array(num_possible_cpus(), sizeof(*hv_vp_index),
>>> +				    GFP_KERNEL);
>>
>> Why isn't this a percpu variable?
> 
> In current code in the architecture independent Hyper-V drivers (as well
> as some future Hyper-V enlightenments that aren't yet implemented
> for ARM64), the running CPU needs to get the VP index values for any CPUs
> in the hv_vp_index array.  Some of the code is performance sensitive, and
> accessing a global array is faster than accessing other CPUs' per-cpu data.

Fair enough. But his seems to tie into the above discussion about the
use of MPIDR_EL1 and the logical CPU mapping.

[...]

>>> +free_vp_index:
>>> +	kfree(hv_vp_index);
>>> +	hv_vp_index = NULL;
>>> +	return 1;
>>
>> ????
> 
> The return value is there because this function is implemented as a timer
> initialization function, and that's what the function signature requires.
> But maybe I'm not understanding your question.

I'm slightly miffed by the "return 1". I'd expect explicit negative
return codes that describe the error.

[...]

>>> +	vmbus_irq = acpi_register_gsi(NULL, HYPERVISOR_CALLBACK_VECTOR,
>>> +				 ACPI_LEVEL_SENSITIVE, ACPI_ACTIVE_HIGH);
>>> +	if (vmbus_irq <= 0) {
>>> +		pr_err("Can't register Hyper-V VMBus GSI. Error %d",
>>> +			vmbus_irq);
>>> +		vmbus_irq = 0;
>>> +		return;
>>> +	}
>>> +	vmbus_evt = alloc_percpu(long);
>>> +	result = request_percpu_irq(vmbus_irq, hyperv_vector_handler,
>>> +			"Hyper-V VMbus", vmbus_evt);
>>
>> If this is a per-cpu interrupt, why isn't it signalled as a PPI, in an
>> architecture compliant way?
> 
> Except for the code in this module, the interrupt handler for VMbus
> interrupts is architecture independent.  But there's no support for
> per-process interrupts on the x86, so the hypervisor interrupt vectors
> are hard coded in the same IDT entry across all processors, and the
> normal IRQ allocation mechanism is bypassed.  The above approach
> assigns an ARM64 PPI (HYPERVISOR_CALLBACK_VECTOR is 16) in a
> way that works with the arch independent interrupt handler.
> 
> Or maybe I'm missing your point.  If so, please set me straight.

Sorry, I missed that HYPERVISOR_CALLBACK_VECTOR was a PPI. It looks OK now.

Thanks,

	M.
-- 
Jazz is not dead. It just smells funny...

Re: [PATCH 2/4] arm64: hyperv: Add support for Hyper-V as a hypervisor

[Marc Zyngier]

Hi Michael,

On 12/12/2018 05:00, Michael Kelley wrote:
> From: Marc Zyngier <marc.zyngier@arm.com>  Sent: Friday, December 7, 2018 6:43 AM
> 
>>> Add ARM64-specific code to enable Hyper-V. This code includes:
>>> * Detecting Hyper-V and initializing the guest/Hyper-V interface
>>> * Setting up Hyper-V's synthetic clocks
>>> * Making hypercalls using the HVC instruction
>>> * Setting up VMbus and stimer0 interrupts
>>> * Setting up kexec and crash handlers
>>
>> This commit message is a clear indication that this should be split in
>> at least 5 different patches.
> 
> OK, I'll work on separating into multiple layered patches in the next
> version.

Thanks. This will definitely help the review process.

>>> +/*
>>> + * This variant of HVC invocation is for hv_get_vpreg and
>>> + * hv_get_vpreg_128. The input parameters are passed in registers
>>> + * along with a pointer in x4 to where the output result should
>>> + * be stored. The output is returned in x15 and x16.  x18 is used as
>>> + * scratch space to avoid buildng a stack frame, as Hyper-V does
>>> + * not preserve registers x0-x17.
>>> + */
>>> +ENTRY(hv_do_hvc_fast_get)
>>> +	mov x18, x4
>>> +	hvc #1
>>> +	str x15,[x18]
>>> +	str x16,[x18,#8]
>>> +	ret
>>> +ENDPROC(hv_do_hvc_fast_get)
>>
>> As Will said, this isn't a viable option. Please follow SMCCC 1.1.
> 
> I'll have to start a conversation with the Hyper-V team about this.
> I don't know why they chose to use HVC #1 or this register scheme
> for output values.  It may be tough to change at this point because
> there are Windows guests on Hyper-V for ARM64 that are already
> using this approach.

I appreciate you already have stuff in the wild, but there is definitely
a case to be made for supporting architecturally specified mechanisms in
a hypervisor, and SMCCC is definitely part of it (I'm certainly curious
of how you support the Spectre mitigation otherwise).

> 
>>
>>> diff --git a/arch/arm64/hyperv/hv_init.c b/arch/arm64/hyperv/hv_init.c
>>> new file mode 100644
>>> index 000000000000..aa1a8c09d989
>>> --- /dev/null
>>> +++ b/arch/arm64/hyperv/hv_init.c
>>> @@ -0,0 +1,441 @@
>>> +// SPDX-License-Identifier: GPL-2.0
>>> +
>>> +/*
>>> + * Initialization of the interface with Microsoft's Hyper-V hypervisor,
>>> + * and various low level utility routines for interacting with Hyper-V.
>>> + *
>>> + * Copyright (C) 2018, Microsoft, Inc.
>>> + *
>>> + * Author : Michael Kelley <mikelley@microsoft.com>
>>> + *
>>> + * This program is free software; you can redistribute it and/or modify it
>>> + * under the terms of the GNU General Public License version 2 as published
>>> + * by the Free Software Foundation.
>>> + *
>>> + * This program is distributed in the hope that it will be useful, but
>>> + * WITHOUT ANY WARRANTY; without even the implied warranty of
>>> + * MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE, GOOD TITLE or
>>> + * NON INFRINGEMENT.  See the GNU General Public License for more
>>> + * details.
>>> + */
>>> +
>>> +
>>> +#include <linux/types.h>
>>> +#include <linux/version.h>
>>> +#include <linux/export.h>
>>> +#include <linux/vmalloc.h>
>>> +#include <linux/mm.h>
>>> +#include <linux/clocksource.h>
>>> +#include <linux/sched_clock.h>
>>> +#include <linux/acpi.h>
>>> +#include <linux/module.h>
>>> +#include <linux/hyperv.h>
>>> +#include <linux/slab.h>
>>> +#include <linux/cpuhotplug.h>
>>> +#include <linux/psci.h>
>>> +#include <asm-generic/bug.h>
>>> +#include <asm/hypervisor.h>
>>> +#include <asm/hyperv-tlfs.h>
>>> +#include <asm/mshyperv.h>
>>> +#include <asm/sysreg.h>
>>> +#include <clocksource/arm_arch_timer.h>
>>> +
>>> +static bool	hyperv_initialized;
>>> +struct		ms_hyperv_info ms_hyperv;
>>> +EXPORT_SYMBOL_GPL(ms_hyperv);
>>
>> Who are the users of this structure? Should they go via accessors instead?
> 
> The structure is an aggregation of several flags fields that describe a myriad
> of features and hints that may or may not be present on any particular version
> of Hyper-V, plus the max virtual processor ID values.  Everything is read-only
> after initialization.   

nit: please consider using a __ro_after_init annotation in that case.

> Most of the references are to test one of the flags.  It's
> a judgment call, but there are a lot of different flags with long names, and
> writing accessors for each one doesn't seem to me to add any clarity.

Looking at that structure, it doesn't seem so bad, and you could easily
have generic accessors that take a flag as a parameter. Your call.

[...]

>>> +static struct clocksource hyperv_cs_msr = {
>>> +	.name		= "hyperv_clocksource_msr",
>>> +	.rating		= 400,
>>> +	.read		= read_hv_clock_msr,
>>> +	.mask		= CLOCKSOURCE_MASK(64),
>>> +	.flags		= CLOCK_SOURCE_IS_CONTINUOUS,
>>> +};
>>> +
>>> +struct clocksource *hyperv_cs;
>>> +EXPORT_SYMBOL_GPL(hyperv_cs);
>>
>> Why? Who needs to poke this?
> 
> It's referenced in the architecture independent driver code
> for the Hyper-V clocksource in drivers/hv/hv.c, and in the
> code to sync the time with the Hyper-V host in
> drivers/hv/hv_util.c.

Fair enough.

>>> +static int hv_cpu_init(unsigned int cpu)
>>> +{
>>> +	u64 msr_vp_index;
>>> +
>>> +	hv_get_vp_index(msr_vp_index);
>>> +
>>> +	hv_vp_index[smp_processor_id()] = msr_vp_index;
>>> +
>>> +	if (msr_vp_index > hv_max_vp_index)
>>> +		hv_max_vp_index = msr_vp_index;
>>> +
>>> +	return 0;
>>> +}
>>
>> Is that some new way to describe a CPU topology? If so, why isn't that
>> exposed via the ACPI tables that the kernel already parses?
> 
> Hyper-V's hypercall interface uses vCPU identifiers that are not
> guaranteed to be consecutive integers or to match what ACPI shows.
> No topology information is implied -- it's just unique identifiers.  The 
> hv_vp_index array provides easy mapping from Linux's consecutive
> integer IDs for CPUs when needed to construct hypercall arguments.

That's extremely odd. The hypervisor obviously knows which vCPU is doing
a hypercall, and if referencing another vCPU, the virtualized MPIDR_EL1
value should be used. I don't think deviating from the architecture is a
good idea (but I appreciate this is none of your doing). Following the
architecture would allow this code to directly use the cpu_logical_map
infrastructure we alreadu have.

> 
>>
>>> +
>>> +/*
>>> + * This function is invoked via the ACPI clocksource probe mechanism. We
>>> + * don't actually use any values from the ACPI GTDT table, but we set up
>>
>> This doesn't feel like a good idea at all. Piggy-backing on an existing
>> mechanism and use it for something completely different is not exactly
>> future-proof.
>>
>> Also, if this is supposed to be a clocksource, why isn't that a
>> clocksource driver on its own right?
> 
> I agree this is not the right long term solution.  Is there a better place to
> hang the initialization code?  Or should I just make an explicit call to
> initialize Hyper-V at the right place?  On the x86 side, there's an
> explicit framework for hypervisor-specific initialization routines to plug
> into.  Maybe it's time for a basic version of such a framework on the
> ARM64 side.  Thoughts on the best approach, both in the short-term and
> the longer-term? If we put a framework in place, does that need to
> happen before adding Hyper-V code, or afterwards as a cleanup?

If we're introducing an infrastructure, doing it before introducing more
stuff seems to be the logical option. But I'm not sure we really need
such an infrastructure in this case, unless we need to enforce some
specific ordering.

You could start by moving all the clocksource stuff to
drivers/clocksource and keep the same init mechanism for the time being.

> 
> And yes, Hyper-V does effectively have its own clocksource.  The
> main code is in drivers/hv/hv.c, but it's not broken out as a separate
> driver in drivers/clocksource, probably due to some history on the
> x86 side that pre-dates me.  I'll have to research.

OK.

> 
>>
>>> + * the Hyper-V synthetic clocksource and do other initialization for
>>> + * interacting with Hyper-V the first time.  Using early_initcall to invoke
>>> + * this function is too late because interrupts are already enabled at that
>>> + * point, and sched_clock_register must run before interrupts are enabled.
>>> + *
>>> + * 1. Setup the guest ID.
>>> + * 2. Get features and hints info from Hyper-V
>>> + * 3. Setup per-cpu VP indices.
>>> + * 4. Register Hyper-V specific clocksource.
>>> + * 5. Register the scheduler clock.
>>> + */
>>> +
>>> +static int __init hyperv_init(struct acpi_table_header *table)
>>> +{
>>> +	struct hv_get_vp_register_output result;
>>> +	u32	a, b, c, d;
>>> +	u64	guest_id;
>>> +	int	i;
>>> +
>>> +	/*
>>> +	 * If we're in a VM on Hyper-V, the ACPI hypervisor_id field will
>>> +	 * have the string "MsHyperV".
>>> +	 */
>>> +	if (strncmp((char *)&acpi_gbl_FADT.hypervisor_id, "MsHyperV", 8))
>>> +		return 1;
>>> +
>>> +	/* Setup the guest ID */
>>> +	guest_id = generate_guest_id(0, LINUX_VERSION_CODE, 0);
>>> +	hv_set_vpreg(HV_REGISTER_GUEST_OSID, guest_id);
>>> +
>>> +	/* Get the features and hints from Hyper-V */
>>> +	hv_get_vpreg_128(HV_REGISTER_PRIVILEGES_AND_FEATURES, &result);
>>> +	ms_hyperv.features = lower_32_bits(result.registervaluelow);
>>> +	ms_hyperv.misc_features = upper_32_bits(result.registervaluehigh);
>>> +
>>> +	hv_get_vpreg_128(HV_REGISTER_FEATURES, &result);
>>> +	ms_hyperv.hints = lower_32_bits(result.registervaluelow);
>>> +
>>> +	pr_info("Hyper-V: Features 0x%x, hints 0x%x\n",
>>> +		ms_hyperv.features, ms_hyperv.hints);
>>> +
>>> +	/*
>>> +	 * Direct mode is the only option for STIMERs provided Hyper-V
>>> +	 * on ARM64, so Hyper-V doesn't actually set the flag.  But add the
>>> +	 * flag so the architecture independent code in drivers/hv/hv.c
>>> +	 * will correctly use that mode.
>>> +	 */
>>> +	ms_hyperv.misc_features |= HV_STIMER_DIRECT_MODE_AVAILABLE;
>>> +
>>> +	/*
>>> +	 * Hyper-V on ARM64 doesn't support AutoEOI.  Add the hint
>>> +	 * that tells architecture independent code not to use this
>>> +	 * feature.
>>> +	 */
>>> +	ms_hyperv.hints |= HV_DEPRECATING_AEOI_RECOMMENDED;
>>> +
>>> +	/* Get information about the Hyper-V host version */
>>> +	hv_get_vpreg_128(HV_REGISTER_HYPERVISOR_VERSION, &result);
>>> +	a = lower_32_bits(result.registervaluelow);
>>> +	b = upper_32_bits(result.registervaluelow);
>>> +	c = lower_32_bits(result.registervaluehigh);
>>> +	d = upper_32_bits(result.registervaluehigh);
>>> +	pr_info("Hyper-V: Host Build %d.%d.%d.%d-%d-%d\n",
>>> +		b >> 16, b & 0xFFFF, a, d & 0xFFFFFF, c, d >> 24);
>>> +
>>> +	/* Allocate percpu VP index */
>>> +	hv_vp_index = kmalloc_array(num_possible_cpus(), sizeof(*hv_vp_index),
>>> +				    GFP_KERNEL);
>>
>> Why isn't this a percpu variable?
> 
> In current code in the architecture independent Hyper-V drivers (as well
> as some future Hyper-V enlightenments that aren't yet implemented
> for ARM64), the running CPU needs to get the VP index values for any CPUs
> in the hv_vp_index array.  Some of the code is performance sensitive, and
> accessing a global array is faster than accessing other CPUs' per-cpu data.

Fair enough. But his seems to tie into the above discussion about the
use of MPIDR_EL1 and the logical CPU mapping.

[...]

>>> +free_vp_index:
>>> +	kfree(hv_vp_index);
>>> +	hv_vp_index = NULL;
>>> +	return 1;
>>
>> ????
> 
> The return value is there because this function is implemented as a timer
> initialization function, and that's what the function signature requires.
> But maybe I'm not understanding your question.

I'm slightly miffed by the "return 1". I'd expect explicit negative
return codes that describe the error.

[...]

>>> +	vmbus_irq = acpi_register_gsi(NULL, HYPERVISOR_CALLBACK_VECTOR,
>>> +				 ACPI_LEVEL_SENSITIVE, ACPI_ACTIVE_HIGH);
>>> +	if (vmbus_irq <= 0) {
>>> +		pr_err("Can't register Hyper-V VMBus GSI. Error %d",
>>> +			vmbus_irq);
>>> +		vmbus_irq = 0;
>>> +		return;
>>> +	}
>>> +	vmbus_evt = alloc_percpu(long);
>>> +	result = request_percpu_irq(vmbus_irq, hyperv_vector_handler,
>>> +			"Hyper-V VMbus", vmbus_evt);
>>
>> If this is a per-cpu interrupt, why isn't it signalled as a PPI, in an
>> architecture compliant way?
> 
> Except for the code in this module, the interrupt handler for VMbus
> interrupts is architecture independent.  But there's no support for
> per-process interrupts on the x86, so the hypervisor interrupt vectors
> are hard coded in the same IDT entry across all processors, and the
> normal IRQ allocation mechanism is bypassed.  The above approach
> assigns an ARM64 PPI (HYPERVISOR_CALLBACK_VECTOR is 16) in a
> way that works with the arch independent interrupt handler.
> 
> Or maybe I'm missing your point.  If so, please set me straight.

Sorry, I missed that HYPERVISOR_CALLBACK_VECTOR was a PPI. It looks OK now.

Thanks,

	M.
-- 
Jazz is not dead. It just smells funny...

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

RE: [PATCH 2/4] arm64: hyperv: Add support for Hyper-V as a hypervisor

[Michael Kelley]

From: Marc Zyngier <marc.zyngier@arm.com> Sent: Thursday, December 13, 2018 3:23 AM

> >> As Will said, this isn't a viable option. Please follow SMCCC 1.1.
> >
> > I'll have to start a conversation with the Hyper-V team about this.
> > I don't know why they chose to use HVC #1 or this register scheme
> > for output values.  It may be tough to change at this point because
> > there are Windows guests on Hyper-V for ARM64 that are already
> > using this approach.
> 
> I appreciate you already have stuff in the wild, but there is definitely
> a case to be made for supporting architecturally specified mechanisms in
> a hypervisor, and SMCCC is definitely part of it (I'm certainly curious
> of how you support the Spectre mitigation otherwise).
>

The Hyper-V guys I need to discuss this with are not back from the
holidays until January 7th.  I'll follow up on this thread once I've
had that conversation.

> >>> +static int hv_cpu_init(unsigned int cpu)
> >>> +{
> >>> +	u64 msr_vp_index;
> >>> +
> >>> +	hv_get_vp_index(msr_vp_index);
> >>> +
> >>> +	hv_vp_index[smp_processor_id()] = msr_vp_index;
> >>> +
> >>> +	if (msr_vp_index > hv_max_vp_index)
> >>> +		hv_max_vp_index = msr_vp_index;
> >>> +
> >>> +	return 0;
> >>> +}
> >>
> >> Is that some new way to describe a CPU topology? If so, why isn't that
> >> exposed via the ACPI tables that the kernel already parses?
> >
> > Hyper-V's hypercall interface uses vCPU identifiers that are not
> > guaranteed to be consecutive integers or to match what ACPI shows.
> > No topology information is implied -- it's just unique identifiers.  The
> > hv_vp_index array provides easy mapping from Linux's consecutive
> > integer IDs for CPUs when needed to construct hypercall arguments.
> 
> That's extremely odd. The hypervisor obviously knows which vCPU is doing
> a hypercall, and if referencing another vCPU, the virtualized MPIDR_EL1
> value should be used. I don't think deviating from the architecture is a
> good idea (but I appreciate this is none of your doing). Following the
> architecture would allow this code to directly use the cpu_logical_map
> infrastructure we already have.

I see what you are getting at.  However, some Hyper-V hypercalls allow
specifying arbitrary sets of vCPUs.  These hypercalls are used to define
target processors in the virtual PCI code (which I have not yet brought over
to ARM64) and in enlightenments for IPIs and TLB flushes (used by
Windows guests and Linux guests on x86, but not yet brought over to Linux
ARM64, if they ever will be).  These hypercalls take bitmaps as arguments,
similar to a Linux cpumask, as defined in Sections 7.8.7.3 thru 7.8.7.5 in the
Hyper-V TLFS.  So Hyper-V defines its own VP index that is akin to the index 
into the cpu_logical_map, though it may not be the same mapping.  My earlier
comments may have been misleading -- the Hyper-V VP index is an integer
ranging from 0 thru (# vCPUs - 1).

With these requirements, Hyper-V defining its own VP index seems like
a reasonable thing to do.  And since Hyper-V provides the same hypercall
interfaces for both x86 and ARM64 implementations, and for Windows
guests, there's not much choice but to use the Hyper-V VP index as specified.

Michael

RE: [PATCH 2/4] arm64: hyperv: Add support for Hyper-V as a hypervisor

[Michael Kelley]

From: Marc Zyngier <marc.zyngier@arm.com> Sent: Thursday, December 13, 2018 3:23 AM

> >> As Will said, this isn't a viable option. Please follow SMCCC 1.1.
> >
> > I'll have to start a conversation with the Hyper-V team about this.
> > I don't know why they chose to use HVC #1 or this register scheme
> > for output values.  It may be tough to change at this point because
> > there are Windows guests on Hyper-V for ARM64 that are already
> > using this approach.
> 
> I appreciate you already have stuff in the wild, but there is definitely
> a case to be made for supporting architecturally specified mechanisms in
> a hypervisor, and SMCCC is definitely part of it (I'm certainly curious
> of how you support the Spectre mitigation otherwise).
>

The Hyper-V guys I need to discuss this with are not back from the
holidays until January 7th.  I'll follow up on this thread once I've
had that conversation.

> >>> +static int hv_cpu_init(unsigned int cpu)
> >>> +{
> >>> +	u64 msr_vp_index;
> >>> +
> >>> +	hv_get_vp_index(msr_vp_index);
> >>> +
> >>> +	hv_vp_index[smp_processor_id()] = msr_vp_index;
> >>> +
> >>> +	if (msr_vp_index > hv_max_vp_index)
> >>> +		hv_max_vp_index = msr_vp_index;
> >>> +
> >>> +	return 0;
> >>> +}
> >>
> >> Is that some new way to describe a CPU topology? If so, why isn't that
> >> exposed via the ACPI tables that the kernel already parses?
> >
> > Hyper-V's hypercall interface uses vCPU identifiers that are not
> > guaranteed to be consecutive integers or to match what ACPI shows.
> > No topology information is implied -- it's just unique identifiers.  The
> > hv_vp_index array provides easy mapping from Linux's consecutive
> > integer IDs for CPUs when needed to construct hypercall arguments.
> 
> That's extremely odd. The hypervisor obviously knows which vCPU is doing
> a hypercall, and if referencing another vCPU, the virtualized MPIDR_EL1
> value should be used. I don't think deviating from the architecture is a
> good idea (but I appreciate this is none of your doing). Following the
> architecture would allow this code to directly use the cpu_logical_map
> infrastructure we already have.

I see what you are getting at.  However, some Hyper-V hypercalls allow
specifying arbitrary sets of vCPUs.  These hypercalls are used to define
target processors in the virtual PCI code (which I have not yet brought over
to ARM64) and in enlightenments for IPIs and TLB flushes (used by
Windows guests and Linux guests on x86, but not yet brought over to Linux
ARM64, if they ever will be).  These hypercalls take bitmaps as arguments,
similar to a Linux cpumask, as defined in Sections 7.8.7.3 thru 7.8.7.5 in the
Hyper-V TLFS.  So Hyper-V defines its own VP index that is akin to the index 
into the cpu_logical_map, though it may not be the same mapping.  My earlier
comments may have been misleading -- the Hyper-V VP index is an integer
ranging from 0 thru (# vCPUs - 1).

With these requirements, Hyper-V defining its own VP index seems like
a reasonable thing to do.  And since Hyper-V provides the same hypercall
interfaces for both x86 and ARM64 implementations, and for Windows
guests, there's not much choice but to use the Hyper-V VP index as specified.

Michael


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

RE: [PATCH 2/4] arm64: hyperv: Add support for Hyper-V as a hypervisor

[Michael Kelley]

From: Michael Kelley  Sent: Friday, January 4, 2019 12:05 PM
> 
> > >> As Will said, this isn't a viable option. Please follow SMCCC 1.1.
> > >
> > > I'll have to start a conversation with the Hyper-V team about this.
> > > I don't know why they chose to use HVC #1 or this register scheme
> > > for output values.  It may be tough to change at this point because
> > > there are Windows guests on Hyper-V for ARM64 that are already
> > > using this approach.
> >
> > I appreciate you already have stuff in the wild, but there is definitely
> > a case to be made for supporting architecturally specified mechanisms in
> > a hypervisor, and SMCCC is definitely part of it (I'm certainly curious
> > of how you support the Spectre mitigation otherwise).
> >
> 
> The Hyper-V guys I need to discuss this with are not back from the
> holidays until January 7th.  I'll follow up on this thread once I've
> had that conversation.
> 

Feedback from the Hyper-V guys is that they believe the Hyper-V
specific hypercall sequence *is* compliant with SMCCC 1.1, in the
sense of being outside the requirements per Section 2.9 since the
Hyper-V hypercall sequence uses HVC #1.  Hyper-V wanted to use a
simpler calling sequence that doesn't have the full register
save/restore requirements, for better performance.  The details
of the Hyper-V hypercall sequence are documented internally,
but we do still need to get it published externally as part of a
Hyper-V TLFS version that includes ARM64.

Hyper-V uses the full SMC Calling Conventions in other places,
such as for PSCI calls, for SMCs to EL3, and for the Spectre
mitigation related calls.

Michael

RE: [PATCH 2/4] arm64: hyperv: Add support for Hyper-V as a hypervisor

[Michael Kelley]

From: Michael Kelley  Sent: Friday, January 4, 2019 12:05 PM
> 
> > >> As Will said, this isn't a viable option. Please follow SMCCC 1.1.
> > >
> > > I'll have to start a conversation with the Hyper-V team about this.
> > > I don't know why they chose to use HVC #1 or this register scheme
> > > for output values.  It may be tough to change at this point because
> > > there are Windows guests on Hyper-V for ARM64 that are already
> > > using this approach.
> >
> > I appreciate you already have stuff in the wild, but there is definitely
> > a case to be made for supporting architecturally specified mechanisms in
> > a hypervisor, and SMCCC is definitely part of it (I'm certainly curious
> > of how you support the Spectre mitigation otherwise).
> >
> 
> The Hyper-V guys I need to discuss this with are not back from the
> holidays until January 7th.  I'll follow up on this thread once I've
> had that conversation.
> 

Feedback from the Hyper-V guys is that they believe the Hyper-V
specific hypercall sequence *is* compliant with SMCCC 1.1, in the
sense of being outside the requirements per Section 2.9 since the
Hyper-V hypercall sequence uses HVC #1.  Hyper-V wanted to use a
simpler calling sequence that doesn't have the full register
save/restore requirements, for better performance.  The details
of the Hyper-V hypercall sequence are documented internally,
but we do still need to get it published externally as part of a
Hyper-V TLFS version that includes ARM64.

Hyper-V uses the full SMC Calling Conventions in other places,
such as for PSCI calls, for SMCs to EL3, and for the Spectre
mitigation related calls.

Michael
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

RE: [PATCH 0/4] Hyper-V: Enable Linux guests on Hyper-V on ARM64

[KY Srinivasan]

> -----Original Message-----
> From: kys@linuxonhyperv.com <kys@linuxonhyperv.com>
> Sent: Wednesday, November 21, 2018 6:51 PM
> To: gregkh@linuxfoundation.org; linux-kernel@vger.kernel.org;
> devel@linuxdriverproject.org; olaf@aepfle.de; apw@canonical.com;
> jasowang@redhat.com; Stephen Hemminger <sthemmin@microsoft.com>;
> Michael Kelley <mikelley@microsoft.com>; vkuznets
> <vkuznets@redhat.com>
> Cc: KY Srinivasan <kys@microsoft.com>
> Subject: [PATCH 0/4] Hyper-V: Enable Linux guests on Hyper-V on ARM64
> 
> From: "K. Y. Srinivasan" <kys@microsoft.com>
> 
> This series enables Linux guests running on Hyper-V on ARM64
> hardware. New ARM64-specific code in arch/arm64/hyperv initializes
> Hyper-V, including its synthetic clocks and hypercall mechanism.
> Existing architecture independent drivers for Hyper-V's VMbus and
> synthetic devices just work when built for ARM64. Hyper-V code is
> built and included in the image and modules only if CONFIG_HYPERV
> is enabled.
> 
> The four patches are organized as follows:
> 1) Add include files that define the Hyper-V interface as
>    described in the Hyper-V Top Level Functional Spec (TLFS), plus
>    additional definitions specific to Linux running on Hyper-V.
> 
> 2) Add core Hyper-V support on ARM64, including hypercalls,
>    synthetic clock initialization, and interrupt handlers.
> 
> 3) Update the existing VMbus driver to generalize interrupt
>    management across x86/x64 and ARM64.
> 
> 4) Make CONFIG_HYPERV selectable on ARM64 in addition to x86/x64.
> 
> Some areas of Linux guests on Hyper-V on ARM64 are a work-
> in-progress, primarily due to work still being done in Hyper-V:
> 
> * Hyper-V on ARM64 currently runs with a 4 Kbyte page size, and only
>   supports guests with a 4 Kbyte page size. Because Hyper-V uses
>   shared pages to communicate between the guest and the hypervisor,
>   there are open design decisions on the page size to use when
>   the guest is using 16K/64K pages.  Once those issues are
>   resolved and Hyper-V fully supports 16K/64K guest pages, changes
>   may be needed in the Linux drivers for Hyper-V synthetic devices.
> 
> * Hyper-V on ARM64 does not currently support mapping PCI devices
>   into the guest address space. The Hyper-V PCI driver at
>   drivers/pci/host/pci-hyperv.c has x86/x64-specific code and is
>   not being built for ARM64.
> 
> In a few cases, terminology from the x86/x64 world has been carried
> over into the ARM64 code ("MSR", "TSC").  Hyper-V still uses the
> x86/x64 terminology and has not replaced it with something more
> generic, so the code uses the Hyper-V terminology.  This will be
> fixed when Hyper-V updates the usage in the TLFS.
> 
> 
> Michael Kelley (4):
>   arm64: hyperv: Add core Hyper-V include files
>   arm64: hyperv: Add support for Hyper-V as a hypervisor
>   Drivers: hv: vmbus: Add hooks for per-CPU IRQ
>   Drivers: hv: Enable CONFIG_HYPERV on ARM64
> 
>  MAINTAINERS                          |   4 +
>  arch/arm64/Makefile                  |   1 +
>  arch/arm64/hyperv/Makefile           |   2 +
>  arch/arm64/hyperv/hv_hvc.S           |  54 ++++
>  arch/arm64/hyperv/hv_init.c          | 441 +++++++++++++++++++++++++++
>  arch/arm64/hyperv/mshyperv.c         | 178 +++++++++++
>  arch/arm64/include/asm/hyperv-tlfs.h | 338 ++++++++++++++++++++
>  arch/arm64/include/asm/mshyperv.h    | 116 +++++++
>  arch/x86/include/asm/mshyperv.h      |   4 +
>  drivers/hv/Kconfig                   |   3 +-
>  drivers/hv/hv.c                      |   2 +
>  include/asm-generic/mshyperv.h       | 240 +++++++++++++++
>  12 files changed, 1382 insertions(+), 1 deletion(-)
>  create mode 100644 arch/arm64/hyperv/Makefile
>  create mode 100644 arch/arm64/hyperv/hv_hvc.S
>  create mode 100644 arch/arm64/hyperv/hv_init.c
>  create mode 100644 arch/arm64/hyperv/mshyperv.c
>  create mode 100644 arch/arm64/include/asm/hyperv-tlfs.h
>  create mode 100644 arch/arm64/include/asm/mshyperv.h
>  create mode 100644 include/asm-generic/mshyperv.h
> 

Greg,

Please drop this. I will be resending shortly.

K. Y
> --
> 2.19.1

[PATCH 0/4] Hyper-V: Enable Linux guests on Hyper-V on ARM64

[kys]

From: "K. Y. Srinivasan" <kys@microsoft.com>

This series enables Linux guests running on Hyper-V on ARM64
hardware. New ARM64-specific code in arch/arm64/hyperv initializes
Hyper-V, including its synthetic clocks and hypercall mechanism.
Existing architecture independent drivers for Hyper-V's VMbus and
synthetic devices just work when built for ARM64. Hyper-V code is
built and included in the image and modules only if CONFIG_HYPERV
is enabled.

The four patches are organized as follows:
1) Add include files that define the Hyper-V interface as
   described in the Hyper-V Top Level Functional Spec (TLFS), plus
   additional definitions specific to Linux running on Hyper-V.

2) Add core Hyper-V support on ARM64, including hypercalls,
   synthetic clock initialization, and interrupt handlers.

3) Update the existing VMbus driver to generalize interrupt
   management across x86/x64 and ARM64.

4) Make CONFIG_HYPERV selectable on ARM64 in addition to x86/x64.

Some areas of Linux guests on Hyper-V on ARM64 are a work-
in-progress, primarily due to work still being done in Hyper-V:

* Hyper-V on ARM64 currently runs with a 4 Kbyte page size, and only
  supports guests with a 4 Kbyte page size. Because Hyper-V uses
  shared pages to communicate between the guest and the hypervisor,
  there are open design decisions on the page size to use when
  the guest is using 16K/64K pages.  Once those issues are
  resolved and Hyper-V fully supports 16K/64K guest pages, changes
  may be needed in the Linux drivers for Hyper-V synthetic devices.

* Hyper-V on ARM64 does not currently support mapping PCI devices
  into the guest address space. The Hyper-V PCI driver at
  drivers/pci/host/pci-hyperv.c has x86/x64-specific code and is
  not being built for ARM64.

In a few cases, terminology from the x86/x64 world has been carried
over into the ARM64 code ("MSR", "TSC").  Hyper-V still uses the
x86/x64 terminology and has not replaced it with something more
generic, so the code uses the Hyper-V terminology.  This will be
fixed when Hyper-V updates the usage in the TLFS.


Michael Kelley (4):
  arm64: hyperv: Add core Hyper-V include files
  arm64: hyperv: Add support for Hyper-V as a hypervisor
  Drivers: hv: vmbus: Add hooks for per-CPU IRQ
  Drivers: hv: Enable CONFIG_HYPERV on ARM64

 MAINTAINERS                          |   4 +
 arch/arm64/Makefile                  |   1 +
 arch/arm64/hyperv/Makefile           |   2 +
 arch/arm64/hyperv/hv_hvc.S           |  54 ++++
 arch/arm64/hyperv/hv_init.c          | 441 +++++++++++++++++++++++++++
 arch/arm64/hyperv/mshyperv.c         | 178 +++++++++++
 arch/arm64/include/asm/hyperv-tlfs.h | 338 ++++++++++++++++++++
 arch/arm64/include/asm/mshyperv.h    | 116 +++++++
 arch/x86/include/asm/mshyperv.h      |   4 +
 drivers/hv/Kconfig                   |   3 +-
 drivers/hv/hv.c                      |   2 +
 include/asm-generic/mshyperv.h       | 240 +++++++++++++++
 12 files changed, 1382 insertions(+), 1 deletion(-)
 create mode 100644 arch/arm64/hyperv/Makefile
 create mode 100644 arch/arm64/hyperv/hv_hvc.S
 create mode 100644 arch/arm64/hyperv/hv_init.c
 create mode 100644 arch/arm64/hyperv/mshyperv.c
 create mode 100644 arch/arm64/include/asm/hyperv-tlfs.h
 create mode 100644 arch/arm64/include/asm/mshyperv.h
 create mode 100644 include/asm-generic/mshyperv.h

-- 
2.19.1