[Xen-devel] [PATCH 0/8] More Hyper-V infrastructure

[Xen-devel] [PATCH 0/8] More Hyper-V infrastructure

[Wei Liu]

This patch sereis implements several important functionalities to run
Xen on top of Hyper-V.

See individual patches for more details.

I can only test them lightly and look at disassembly output to check
correctness at this stage. At the very last Xen on Hyper-V boots up
okay, so it is not breaking anything.

Wei.


Cc: Jan Beulich <jbeulich@suse.com>
Cc: Andrew Cooper <andrew.cooper3@citrix.com>
Cc: Wei Liu <wl@xen.org>
Cc: Roger Pau Monné <roger.pau@citrix.com>
Cc: Michael Kelley <mikelley@microsoft.com>
Cc: Paul Durrant <paul@xen.org>

Wei Liu (8):
  x86: include xen/lib.h in guest/pvh-boot.h
  x86/hyperv: detect absolutely necessary MSRs
  x86: rename guest/hypercall.h to guest/xen-hypercall.h
  x86/hyperv: setup hypercall page
  x86/hyperv: provide Hyper-V hypercall functions
  x86/hyperv: provide percpu hypercall input page
  x86/hyperv: retrieve vp_index from Hyper-V
  x86/hyperv: setup VP assist page

 xen/arch/x86/guest/hyperv/hyperv.c            | 125 +++++++++++++++++-
 xen/include/asm-x86/guest.h                   |   2 +-
 xen/include/asm-x86/guest/hyperv-hypercall.h  | 105 +++++++++++++++
 xen/include/asm-x86/guest/hyperv.h            |  10 ++
 xen/include/asm-x86/guest/pvh-boot.h          |   2 +
 .../guest/{hypercall.h => xen-hypercall.h}    |   2 +-
 6 files changed, 241 insertions(+), 5 deletions(-)
 create mode 100644 xen/include/asm-x86/guest/hyperv-hypercall.h
 rename xen/include/asm-x86/guest/{hypercall.h => xen-hypercall.h} (99%)

-- 
2.20.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[Xen-devel] [PATCH 1/8] x86: include xen/lib.h in guest/pvh-boot.h

[Wei Liu]

It needs ASSERT_UNREACHABLE.

Signed-off-by: Wei Liu <liuwe@microsoft.com>
---
 xen/include/asm-x86/guest/pvh-boot.h | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/xen/include/asm-x86/guest/pvh-boot.h b/xen/include/asm-x86/guest/pvh-boot.h
index b8a76c4eed..48ffd1a0b1 100644
--- a/xen/include/asm-x86/guest/pvh-boot.h
+++ b/xen/include/asm-x86/guest/pvh-boot.h
@@ -30,6 +30,8 @@ void pvh_print_info(void);
 
 #else
 
+#include <xen/lib.h>
+
 #define pvh_boot 0
 
 static inline void pvh_init(multiboot_info_t **mbi, module_t **mod)
-- 
2.20.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[Xen-devel] [PATCH 2/8] x86/hyperv: detect absolutely necessary MSRs

[Wei Liu]

If they are not available, disable Hyper-V related features.

Signed-off-by: Wei Liu <liuwe@microsoft.com>
---
 xen/arch/x86/guest/hyperv/hyperv.c | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/xen/arch/x86/guest/hyperv/hyperv.c b/xen/arch/x86/guest/hyperv/hyperv.c
index 2e70b4aa82..c6a26c5453 100644
--- a/xen/arch/x86/guest/hyperv/hyperv.c
+++ b/xen/arch/x86/guest/hyperv/hyperv.c
@@ -32,6 +32,8 @@ static const struct hypervisor_ops ops = {
 const struct hypervisor_ops *__init hyperv_probe(void)
 {
     uint32_t eax, ebx, ecx, edx;
+    uint64_t required_msrs = HV_X64_MSR_HYPERCALL_AVAILABLE |
+        HV_X64_MSR_VP_INDEX_AVAILABLE;
 
     cpuid(0x40000000, &eax, &ebx, &ecx, &edx);
     if ( !((ebx == 0x7263694d) &&  /* "Micr" */
@@ -57,6 +59,15 @@ const struct hypervisor_ops *__init hyperv_probe(void)
     ms_hyperv.max_vp_index = eax;
     ms_hyperv.max_lp_index = ebx;
 
+    if ( (ms_hyperv.features & required_msrs) != required_msrs )
+    {
+        /* Oops, required MSRs are not available. Treat this as
+         * "Hyper-V is not available".
+         */
+        memset(&ms_hyperv, 0, sizeof(ms_hyperv));
+        return NULL;
+    }
+
     return &ops;
 }
 
-- 
2.20.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[Xen-devel] [PATCH 3/8] x86: rename guest/hypercall.h to guest/xen-hypercall.h

[Wei Liu]

We will provide a header file for Hyper-V hypercalls.

No functional change.

Signed-off-by: Wei Liu <liuwe@microsoft.com>
---
 xen/include/asm-x86/guest.h                                | 2 +-
 xen/include/asm-x86/guest/{hypercall.h => xen-hypercall.h} | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)
 rename xen/include/asm-x86/guest/{hypercall.h => xen-hypercall.h} (99%)

diff --git a/xen/include/asm-x86/guest.h b/xen/include/asm-x86/guest.h
index 94448606d4..7f09c643d4 100644
--- a/xen/include/asm-x86/guest.h
+++ b/xen/include/asm-x86/guest.h
@@ -19,11 +19,11 @@
 #ifndef __X86_GUEST_H__
 #define __X86_GUEST_H__
 
-#include <asm/guest/hypercall.h>
 #include <asm/guest/hyperv.h>
 #include <asm/guest/hypervisor.h>
 #include <asm/guest/pvh-boot.h>
 #include <asm/guest/xen.h>
+#include <asm/guest/xen-hypercall.h>
 #include <asm/pv/shim.h>
 
 #endif /* __X86_GUEST_H__ */
diff --git a/xen/include/asm-x86/guest/hypercall.h b/xen/include/asm-x86/guest/xen-hypercall.h
similarity index 99%
rename from xen/include/asm-x86/guest/hypercall.h
rename to xen/include/asm-x86/guest/xen-hypercall.h
index d0d2f5022d..681d3aecd7 100644
--- a/xen/include/asm-x86/guest/hypercall.h
+++ b/xen/include/asm-x86/guest/xen-hypercall.h
@@ -1,5 +1,5 @@
 /******************************************************************************
- * asm-x86/guest/hypercall.h
+ * asm-x86/guest/xen-hypercall.h
  *
  * This program is free software; you can redistribute it and/or
  * modify it under the terms and conditions of the GNU General Public
-- 
2.20.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[Xen-devel] [PATCH 4/8] x86/hyperv: setup hypercall page

[Wei Liu]

Signed-off-by: Wei Liu <liuwe@microsoft.com>
---
 xen/arch/x86/guest/hyperv/hyperv.c | 41 +++++++++++++++++++++++++++---
 1 file changed, 38 insertions(+), 3 deletions(-)

diff --git a/xen/arch/x86/guest/hyperv/hyperv.c b/xen/arch/x86/guest/hyperv/hyperv.c
index c6a26c5453..438910c8cb 100644
--- a/xen/arch/x86/guest/hyperv/hyperv.c
+++ b/xen/arch/x86/guest/hyperv/hyperv.c
@@ -19,16 +19,17 @@
  * Copyright (c) 2019 Microsoft.
  */
 #include <xen/init.h>
+#include <xen/domain_page.h>
 
 #include <asm/guest.h>
 #include <asm/guest/hyperv-tlfs.h>
 
 struct ms_hyperv_info __read_mostly ms_hyperv;
 
-static const struct hypervisor_ops ops = {
-    .name = "Hyper-V",
-};
+void *hv_hypercall;
+static struct page_info *hv_hypercall_page;
 
+static const struct hypervisor_ops ops;
 const struct hypervisor_ops *__init hyperv_probe(void)
 {
     uint32_t eax, ebx, ecx, edx;
@@ -71,6 +72,40 @@ const struct hypervisor_ops *__init hyperv_probe(void)
     return &ops;
 }
 
+static void __init setup_hypercall_page(void)
+{
+    union hv_x64_msr_hypercall_contents hypercall_msr;
+
+    /* Unfortunately there isn't a really good way to unwind Xen to
+     * not use Hyper-V hooks, so panic if anything goes wrong.
+     *
+     * In practice if page allocation fails this early on it is
+     * unlikely we can get a working system later.
+     */
+    hv_hypercall_page = alloc_domheap_page(NULL, 0);
+    if ( !hv_hypercall_page )
+        panic("Failed to allocate Hyper-V hypercall page\n");
+
+    hv_hypercall = __map_domain_page_global(hv_hypercall_page);
+    if ( !hv_hypercall )
+        panic("Failed to map Hyper-V hypercall page\n");
+
+    rdmsrl(HV_X64_MSR_HYPERCALL, hypercall_msr.as_uint64);
+    hypercall_msr.enable = 1;
+    hypercall_msr.guest_physical_address = page_to_maddr(hv_hypercall_page);
+    wrmsrl(HV_X64_MSR_HYPERCALL, hypercall_msr.as_uint64);
+}
+
+static void __init setup(void)
+{
+    setup_hypercall_page();
+}
+
+static const struct hypervisor_ops ops = {
+    .name = "Hyper-V",
+    .setup = setup,
+};
+
 /*
  * Local variables:
  * mode: C
-- 
2.20.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[Xen-devel] [PATCH 5/8] x86/hyperv: provide Hyper-V hypercall functions

[Wei Liu]

These functions will be used later to make hypercalls to Hyper-V.

I couldn't find reference in TLFS that Hyper-V clobbers flags and
r9-r11, but Linux's commit message says it does. Err on the safe side.

Signed-off-by: Wei Liu <liuwe@microsoft.com>
---
 xen/include/asm-x86/guest/hyperv-hypercall.h | 105 +++++++++++++++++++
 1 file changed, 105 insertions(+)
 create mode 100644 xen/include/asm-x86/guest/hyperv-hypercall.h

diff --git a/xen/include/asm-x86/guest/hyperv-hypercall.h b/xen/include/asm-x86/guest/hyperv-hypercall.h
new file mode 100644
index 0000000000..6017123be5
--- /dev/null
+++ b/xen/include/asm-x86/guest/hyperv-hypercall.h
@@ -0,0 +1,105 @@
+/******************************************************************************
+ * asm-x86/guest/hyperv-hypercall.h
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms and conditions of the GNU General Public
+ * License, version 2, as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public
+ * License along with this program; If not, see <http://www.gnu.org/licenses/>.
+ *
+ * Copyright (c) 2019 Microsoft.
+ */
+
+#ifndef __X86_HYPERV_HYPERCALL_H__
+#define __X86_HYPERV_HYPERCALL_H__
+
+#include <xen/types.h>
+
+#include <asm/asm_defns.h>
+#include <asm/guest/hyperv-tlfs.h>
+#include <asm/page.h>
+
+extern void *hv_hypercall;
+
+static inline uint64_t hv_do_hypercall(uint64_t control, paddr_t input, paddr_t output)
+{
+    uint64_t status;
+
+    if ( !hv_hypercall )
+        return ~0ULL;
+
+    asm volatile ("mov %[output], %%r8\n"
+                  "call *%[hypercall_page]"
+                  : "=a" (status), "+c" (control),
+                    "+d" (input) ASM_CALL_CONSTRAINT
+                  : [output] "rm" (output),
+                    [hypercall_page] "m" (hv_hypercall)
+                  : "cc", "memory", "r8", "r9", "r10", "r11");
+
+    return status;
+}
+
+static inline uint64_t hv_do_fast_hypercall(uint16_t code,
+                                            uint64_t input1, uint64_t input2)
+{
+    uint64_t status;
+    uint64_t control = (uint64_t)code | HV_HYPERCALL_FAST_BIT;
+
+    if ( !hv_hypercall )
+        return ~0ULL;
+
+    asm volatile ("mov %[input2], %%r8\n"
+                  "call *%[hypercall_page]"
+                  : "=a" (status), "+c" (control),
+                    "+d" (input1) ASM_CALL_CONSTRAINT
+                  : [input2] "rm" (input2),
+                    [hypercall_page] "m" (hv_hypercall)
+                  : "cc", "r8", "r9", "r10", "r11");
+
+    return status;
+}
+
+static inline uint64_t hv_do_rep_hypercall(uint16_t code, uint16_t rep_count,
+                                           uint16_t varhead_size,
+                                           paddr_t input, paddr_t output)
+{
+    uint64_t control = code;
+    uint64_t status;
+    uint16_t rep_comp;
+
+    control |= (uint64_t)varhead_size << HV_HYPERCALL_VARHEAD_OFFSET;
+    control |= (uint64_t)rep_count << HV_HYPERCALL_REP_COMP_OFFSET;
+
+    do {
+        status = hv_do_hypercall(control, input, output);
+        if ( (status & HV_HYPERCALL_RESULT_MASK) != HV_STATUS_SUCCESS )
+            break;
+
+        rep_comp = (status & HV_HYPERCALL_REP_COMP_MASK) >>
+            HV_HYPERCALL_REP_COMP_OFFSET;
+
+        control &= ~HV_HYPERCALL_REP_START_MASK;
+        control |= (uint64_t)rep_comp << HV_HYPERCALL_REP_COMP_OFFSET;
+
+    } while ( rep_comp < rep_count );
+
+    return status;
+}
+
+#endif /* __X86_HYPERV_HYPERCALL_H__ */
+
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * tab-width: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
-- 
2.20.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[Xen-devel] [PATCH 6/8] x86/hyperv: provide percpu hypercall input page

[Wei Liu]

Hyper-V's input / output argument must be 8 bytes aligned an not cross
page boundary. The easiest way to satisfy those requirements is to use
percpu page.

For the foreseeable future we only need to provide input for TLB
and APIC hypercalls, so skip setting up an output page.

The page tracking structure is not bound to hypercall because it is a
common pattern for Xen to write guest physical address to Hyper-V while
at the same time accessing the page via a pointer.

We will also need to provide an ap_setup hook for secondary cpus to
setup its own input page.

Signed-off-by: Wei Liu <liuwe@microsoft.com>
---
 xen/arch/x86/guest/hyperv/hyperv.c | 26 ++++++++++++++++++++++++++
 xen/include/asm-x86/guest/hyperv.h |  8 ++++++++
 2 files changed, 34 insertions(+)

diff --git a/xen/arch/x86/guest/hyperv/hyperv.c b/xen/arch/x86/guest/hyperv/hyperv.c
index 438910c8cb..67667936e9 100644
--- a/xen/arch/x86/guest/hyperv/hyperv.c
+++ b/xen/arch/x86/guest/hyperv/hyperv.c
@@ -28,6 +28,7 @@ struct ms_hyperv_info __read_mostly ms_hyperv;
 
 void *hv_hypercall;
 static struct page_info *hv_hypercall_page;
+DEFINE_PER_CPU_READ_MOSTLY(struct hyperv_pcpu_page, hv_pcpu_input_arg);
 
 static const struct hypervisor_ops ops;
 const struct hypervisor_ops *__init hyperv_probe(void)
@@ -96,14 +97,39 @@ static void __init setup_hypercall_page(void)
     wrmsrl(HV_X64_MSR_HYPERCALL, hypercall_msr.as_uint64);
 }
 
+static void setup_hypercall_pcpu_arg(void)
+{
+    struct page_info *pg;
+    void *mapping;
+    unsigned int cpu = smp_processor_id();
+
+    pg = alloc_domheap_page(NULL, 0);
+    if ( !pg )
+        panic("Failed to setup hypercall input page for %u\n", cpu);
+
+    mapping = __map_domain_page_global(pg);
+    if ( !mapping )
+        panic("Failed to map hypercall input page for %u\n", cpu);
+
+    this_cpu(hv_pcpu_input_arg).maddr = page_to_maddr(pg);
+    this_cpu(hv_pcpu_input_arg).mapping = mapping;
+}
+
 static void __init setup(void)
 {
     setup_hypercall_page();
+    setup_hypercall_pcpu_arg();
+}
+
+static void ap_setup(void)
+{
+    setup_hypercall_pcpu_arg();
 }
 
 static const struct hypervisor_ops ops = {
     .name = "Hyper-V",
     .setup = setup,
+    .ap_setup = ap_setup,
 };
 
 /*
diff --git a/xen/include/asm-x86/guest/hyperv.h b/xen/include/asm-x86/guest/hyperv.h
index c7a7f32bd5..83f297468f 100644
--- a/xen/include/asm-x86/guest/hyperv.h
+++ b/xen/include/asm-x86/guest/hyperv.h
@@ -51,6 +51,8 @@ static inline uint64_t hv_scale_tsc(uint64_t tsc, uint64_t scale,
 
 #ifdef CONFIG_HYPERV_GUEST
 
+#include <xen/percpu.h>
+
 #include <asm/guest/hypervisor.h>
 
 struct ms_hyperv_info {
@@ -63,6 +65,12 @@ struct ms_hyperv_info {
 };
 extern struct ms_hyperv_info ms_hyperv;
 
+struct hyperv_pcpu_page {
+    paddr_t maddr;
+    void *mapping;
+};
+DECLARE_PER_CPU(struct hyperv_pcpu_page, hv_pcpu_input_arg);
+
 const struct hypervisor_ops *hyperv_probe(void);
 
 #else
-- 
2.20.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[Xen-devel] [PATCH 7/8] x86/hyperv: retrieve vp_index from Hyper-V

[Wei Liu]

This will be useful when invoking hypercall that targets specific
vcpu(s).

Signed-off-by: Wei Liu <liuwe@microsoft.com>
---
 xen/arch/x86/guest/hyperv/hyperv.c | 12 ++++++++++++
 xen/include/asm-x86/guest/hyperv.h |  1 +
 2 files changed, 13 insertions(+)

diff --git a/xen/arch/x86/guest/hyperv/hyperv.c b/xen/arch/x86/guest/hyperv/hyperv.c
index 67667936e9..da3a8cd85d 100644
--- a/xen/arch/x86/guest/hyperv/hyperv.c
+++ b/xen/arch/x86/guest/hyperv/hyperv.c
@@ -29,6 +29,7 @@ struct ms_hyperv_info __read_mostly ms_hyperv;
 void *hv_hypercall;
 static struct page_info *hv_hypercall_page;
 DEFINE_PER_CPU_READ_MOSTLY(struct hyperv_pcpu_page, hv_pcpu_input_arg);
+DEFINE_PER_CPU_READ_MOSTLY(unsigned int, hv_vp_index);
 
 static const struct hypervisor_ops ops;
 const struct hypervisor_ops *__init hyperv_probe(void)
@@ -115,15 +116,26 @@ static void setup_hypercall_pcpu_arg(void)
     this_cpu(hv_pcpu_input_arg).mapping = mapping;
 }
 
+static void setup_vp_index(void)
+{
+    uint64_t vp_index_msr;
+
+    rdmsrl(HV_X64_MSR_VP_INDEX, vp_index_msr);
+
+    this_cpu(hv_vp_index) = vp_index_msr;
+}
+
 static void __init setup(void)
 {
     setup_hypercall_page();
     setup_hypercall_pcpu_arg();
+    setup_vp_index();
 }
 
 static void ap_setup(void)
 {
     setup_hypercall_pcpu_arg();
+    setup_vp_index();
 }
 
 static const struct hypervisor_ops ops = {
diff --git a/xen/include/asm-x86/guest/hyperv.h b/xen/include/asm-x86/guest/hyperv.h
index 83f297468f..4b635829f3 100644
--- a/xen/include/asm-x86/guest/hyperv.h
+++ b/xen/include/asm-x86/guest/hyperv.h
@@ -70,6 +70,7 @@ struct hyperv_pcpu_page {
     void *mapping;
 };
 DECLARE_PER_CPU(struct hyperv_pcpu_page, hv_pcpu_input_arg);
+DECLARE_PER_CPU(unsigned int, hv_vp_index);
 
 const struct hypervisor_ops *hyperv_probe(void);
 
-- 
2.20.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[Xen-devel] [PATCH 8/8] x86/hyperv: setup VP assist page

[Wei Liu]

VP assist page is rather important as we need to toggle some bits in
that page such that L1 guest can make hypercalls directly to L0 Hyper-V.

Preemptively split out set_vp_assist page which will be used in the resume
path.

Signed-off-by: Wei Liu <liuwe@microsoft.com>
---
 xen/arch/x86/guest/hyperv/hyperv.c | 35 ++++++++++++++++++++++++++++++
 xen/include/asm-x86/guest/hyperv.h |  1 +
 2 files changed, 36 insertions(+)

diff --git a/xen/arch/x86/guest/hyperv/hyperv.c b/xen/arch/x86/guest/hyperv/hyperv.c
index da3a8cd85d..a88b9ae6d9 100644
--- a/xen/arch/x86/guest/hyperv/hyperv.c
+++ b/xen/arch/x86/guest/hyperv/hyperv.c
@@ -30,6 +30,7 @@ void *hv_hypercall;
 static struct page_info *hv_hypercall_page;
 DEFINE_PER_CPU_READ_MOSTLY(struct hyperv_pcpu_page, hv_pcpu_input_arg);
 DEFINE_PER_CPU_READ_MOSTLY(unsigned int, hv_vp_index);
+DEFINE_PER_CPU_READ_MOSTLY(struct hyperv_pcpu_page, hv_vp_assist);
 
 static const struct hypervisor_ops ops;
 const struct hypervisor_ops *__init hyperv_probe(void)
@@ -125,17 +126,51 @@ static void setup_vp_index(void)
     this_cpu(hv_vp_index) = vp_index_msr;
 }
 
+static void set_vp_assist(void)
+{
+    uint64_t val = paddr_to_pfn(this_cpu(hv_vp_assist).maddr);
+
+    val = (val << HV_X64_MSR_VP_ASSIST_PAGE_ADDRESS_SHIFT) |
+        HV_X64_MSR_VP_ASSIST_PAGE_ENABLE;
+
+    wrmsrl(HV_X64_MSR_VP_ASSIST_PAGE, val);
+}
+
+static void setup_vp_assist(void)
+{
+    struct page_info *pg;
+    void *mapping;
+    unsigned int cpu = smp_processor_id();
+
+    pg = alloc_domheap_page(NULL, 0);
+    if ( !pg )
+        panic("Failed to allocate vp_assist page for %u\n", cpu);
+
+    mapping = __map_domain_page_global(pg);
+    if ( !mapping )
+        panic("Failed to map vp_assist page for %u\n", cpu);
+
+    clear_page(mapping);
+
+    this_cpu(hv_vp_assist).maddr = page_to_maddr(pg);
+    this_cpu(hv_vp_assist).mapping = mapping;
+
+    set_vp_assist();
+}
+
 static void __init setup(void)
 {
     setup_hypercall_page();
     setup_hypercall_pcpu_arg();
     setup_vp_index();
+    setup_vp_assist();
 }
 
 static void ap_setup(void)
 {
     setup_hypercall_pcpu_arg();
     setup_vp_index();
+    setup_vp_assist();
 }
 
 static const struct hypervisor_ops ops = {
diff --git a/xen/include/asm-x86/guest/hyperv.h b/xen/include/asm-x86/guest/hyperv.h
index 4b635829f3..917f4e02c2 100644
--- a/xen/include/asm-x86/guest/hyperv.h
+++ b/xen/include/asm-x86/guest/hyperv.h
@@ -71,6 +71,7 @@ struct hyperv_pcpu_page {
 };
 DECLARE_PER_CPU(struct hyperv_pcpu_page, hv_pcpu_input_arg);
 DECLARE_PER_CPU(unsigned int, hv_vp_index);
+DECLARE_PER_CPU(struct hyperv_pcpu_page, hv_vp_assist);
 
 const struct hypervisor_ops *hyperv_probe(void);
 
-- 
2.20.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] [PATCH 4/8] x86/hyperv: setup hypercall page

[Michael Kelley]

From: Wei Liu <wei.liu.xen@gmail.com> On Behalf Of Wei Liu  Sent: Sunday, December 29, 2019 10:34 AM
> 
> Signed-off-by: Wei Liu <liuwe@microsoft.com>
> ---
>  xen/arch/x86/guest/hyperv/hyperv.c | 41 +++++++++++++++++++++++++++---
>  1 file changed, 38 insertions(+), 3 deletions(-)
> 
> diff --git a/xen/arch/x86/guest/hyperv/hyperv.c b/xen/arch/x86/guest/hyperv/hyperv.c
> index c6a26c5453..438910c8cb 100644
> --- a/xen/arch/x86/guest/hyperv/hyperv.c
> +++ b/xen/arch/x86/guest/hyperv/hyperv.c
> @@ -19,16 +19,17 @@
>   * Copyright (c) 2019 Microsoft.
>   */
>  #include <xen/init.h>
> +#include <xen/domain_page.h>
> 
>  #include <asm/guest.h>
>  #include <asm/guest/hyperv-tlfs.h>
> 
>  struct ms_hyperv_info __read_mostly ms_hyperv;
> 
> -static const struct hypervisor_ops ops = {
> -    .name = "Hyper-V",
> -};
> +void *hv_hypercall;
> +static struct page_info *hv_hypercall_page;
> 
> +static const struct hypervisor_ops ops;
>  const struct hypervisor_ops *__init hyperv_probe(void)
>  {
>      uint32_t eax, ebx, ecx, edx;
> @@ -71,6 +72,40 @@ const struct hypervisor_ops *__init hyperv_probe(void)
>      return &ops;
>  }
> 
> +static void __init setup_hypercall_page(void)
> +{
> +    union hv_x64_msr_hypercall_contents hypercall_msr;
> +
> +    /* Unfortunately there isn't a really good way to unwind Xen to
> +     * not use Hyper-V hooks, so panic if anything goes wrong.
> +     *
> +     * In practice if page allocation fails this early on it is
> +     * unlikely we can get a working system later.
> +     */
> +    hv_hypercall_page = alloc_domheap_page(NULL, 0);
> +    if ( !hv_hypercall_page )
> +        panic("Failed to allocate Hyper-V hypercall page\n");
> +
> +    hv_hypercall = __map_domain_page_global(hv_hypercall_page);
> +    if ( !hv_hypercall )
> +        panic("Failed to map Hyper-V hypercall page\n");
> +
> +    rdmsrl(HV_X64_MSR_HYPERCALL, hypercall_msr.as_uint64);
> +    hypercall_msr.enable = 1;
> +    hypercall_msr.guest_physical_address = page_to_maddr(hv_hypercall_page);

The "guest_physical_address" field is actually the guest physical page number.
So the physical address needs to be right shifted 12 bits before being stored
here.  I'd recommend using HV_HYP_PAGE_SHIFT from hyperv-tlfs.h as
the shift value; it was introduced to deal with the possibility that the page
size used and expected by the Hyper-V interface is different from the page
size used by the guest VM (which can happen on ARM64, though not on x86).

Michael

> +    wrmsrl(HV_X64_MSR_HYPERCALL, hypercall_msr.as_uint64);
> +}
> +
> +static void __init setup(void)
> +{
> +    setup_hypercall_page();
> +}
> +
> +static const struct hypervisor_ops ops = {
> +    .name = "Hyper-V",
> +    .setup = setup,
> +};
> +
>  /*
>   * Local variables:
>   * mode: C
> --
> 2.20.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] [PATCH 8/8] x86/hyperv: setup VP assist page

[Michael Kelley]

From: Wei Liu <wei.liu.xen@gmail.com> On Behalf Of Wei Liu  Sent: Sunday, December 29, 2019 10:34 AM
> 
> VP assist page is rather important as we need to toggle some bits in
> that page such that L1 guest can make hypercalls directly to L0 Hyper-V.
> 
> Preemptively split out set_vp_assist page which will be used in the resume
> path.
> 
> Signed-off-by: Wei Liu <liuwe@microsoft.com>
> ---
>  xen/arch/x86/guest/hyperv/hyperv.c | 35 ++++++++++++++++++++++++++++++
>  xen/include/asm-x86/guest/hyperv.h |  1 +
>  2 files changed, 36 insertions(+)
> 
> diff --git a/xen/arch/x86/guest/hyperv/hyperv.c b/xen/arch/x86/guest/hyperv/hyperv.c
> index da3a8cd85d..a88b9ae6d9 100644
> --- a/xen/arch/x86/guest/hyperv/hyperv.c
> +++ b/xen/arch/x86/guest/hyperv/hyperv.c
> @@ -30,6 +30,7 @@ void *hv_hypercall;
>  static struct page_info *hv_hypercall_page;
>  DEFINE_PER_CPU_READ_MOSTLY(struct hyperv_pcpu_page, hv_pcpu_input_arg);
>  DEFINE_PER_CPU_READ_MOSTLY(unsigned int, hv_vp_index);
> +DEFINE_PER_CPU_READ_MOSTLY(struct hyperv_pcpu_page, hv_vp_assist);
> 
>  static const struct hypervisor_ops ops;
>  const struct hypervisor_ops *__init hyperv_probe(void)
> @@ -125,17 +126,51 @@ static void setup_vp_index(void)
>      this_cpu(hv_vp_index) = vp_index_msr;
>  }
> 
> +static void set_vp_assist(void)
> +{
> +    uint64_t val = paddr_to_pfn(this_cpu(hv_vp_assist).maddr);
> +
> +    val = (val << HV_X64_MSR_VP_ASSIST_PAGE_ADDRESS_SHIFT) | 

I'd recommend using HV_HYP_PAGE_SHIFT instead of
HV_X64_MSR_VP_ASSIST_PAGE_ADDRESS_SHIFT.  On the Linux side,
I'm planning to remove that #define and the similar
*_PAGE_ADDRESS_MASK in favor of the newer HV_HYP_PAGE_* values.
There's nothing special about the VP assist page, so using the generic
#defines based on the Hyper-V page size is reasonable.

Michael

> +        HV_X64_MSR_VP_ASSIST_PAGE_ENABLE;
> +
> +    wrmsrl(HV_X64_MSR_VP_ASSIST_PAGE, val);
> +}
> +
> +static void setup_vp_assist(void)
> +{
> +    struct page_info *pg;
> +    void *mapping;
> +    unsigned int cpu = smp_processor_id();
> +
> +    pg = alloc_domheap_page(NULL, 0);
> +    if ( !pg )
> +        panic("Failed to allocate vp_assist page for %u\n", cpu);
> +
> +    mapping = __map_domain_page_global(pg);
> +    if ( !mapping )
> +        panic("Failed to map vp_assist page for %u\n", cpu);
> +
> +    clear_page(mapping);
> +
> +    this_cpu(hv_vp_assist).maddr = page_to_maddr(pg);
> +    this_cpu(hv_vp_assist).mapping = mapping;
> +
> +    set_vp_assist();
> +}
> +
>  static void __init setup(void)
>  {
>      setup_hypercall_page();
>      setup_hypercall_pcpu_arg();
>      setup_vp_index();
> +    setup_vp_assist();
>  }
> 
>  static void ap_setup(void)
>  {
>      setup_hypercall_pcpu_arg();
>      setup_vp_index();
> +    setup_vp_assist();
>  }
> 
>  static const struct hypervisor_ops ops = {
> diff --git a/xen/include/asm-x86/guest/hyperv.h b/xen/include/asm-x86/guest/hyperv.h
> index 4b635829f3..917f4e02c2 100644
> --- a/xen/include/asm-x86/guest/hyperv.h
> +++ b/xen/include/asm-x86/guest/hyperv.h
> @@ -71,6 +71,7 @@ struct hyperv_pcpu_page {
>  };
>  DECLARE_PER_CPU(struct hyperv_pcpu_page, hv_pcpu_input_arg);
>  DECLARE_PER_CPU(unsigned int, hv_vp_index);
> +DECLARE_PER_CPU(struct hyperv_pcpu_page, hv_vp_assist);
> 
>  const struct hypervisor_ops *hyperv_probe(void);
> 
> --
> 2.20.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] [PATCH 4/8] x86/hyperv: setup hypercall page

[Wei Liu]

On Sun, Dec 29, 2019 at 07:54:30PM +0000, Michael Kelley wrote:
[...]
> > 
> > +static void __init setup_hypercall_page(void)
> > +{
> > +    union hv_x64_msr_hypercall_contents hypercall_msr;
> > +
> > +    /* Unfortunately there isn't a really good way to unwind Xen to
> > +     * not use Hyper-V hooks, so panic if anything goes wrong.
> > +     *
> > +     * In practice if page allocation fails this early on it is
> > +     * unlikely we can get a working system later.
> > +     */
> > +    hv_hypercall_page = alloc_domheap_page(NULL, 0);
> > +    if ( !hv_hypercall_page )
> > +        panic("Failed to allocate Hyper-V hypercall page\n");
> > +
> > +    hv_hypercall = __map_domain_page_global(hv_hypercall_page);
> > +    if ( !hv_hypercall )
> > +        panic("Failed to map Hyper-V hypercall page\n");
> > +
> > +    rdmsrl(HV_X64_MSR_HYPERCALL, hypercall_msr.as_uint64);
> > +    hypercall_msr.enable = 1;
> > +    hypercall_msr.guest_physical_address = page_to_maddr(hv_hypercall_page);
> 
> The "guest_physical_address" field is actually the guest physical page number.
> So the physical address needs to be right shifted 12 bits before being stored
> here.  I'd recommend using HV_HYP_PAGE_SHIFT from hyperv-tlfs.h as
> the shift value; it was introduced to deal with the possibility that the page
> size used and expected by the Hyper-V interface is different from the page
> size used by the guest VM (which can happen on ARM64, though not on x86).

Good catch, and thanks for the tip here.

I will fix this in the next version.

Wei.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] [PATCH 8/8] x86/hyperv: setup VP assist page

[Wei Liu]

On Sun, Dec 29, 2019 at 07:59:48PM +0000, Michael Kelley wrote:
> From: Wei Liu <wei.liu.xen@gmail.com> On Behalf Of Wei Liu  Sent: Sunday, December 29, 2019 10:34 AM
> > 
> > VP assist page is rather important as we need to toggle some bits in
> > that page such that L1 guest can make hypercalls directly to L0 Hyper-V.
> > 
> > Preemptively split out set_vp_assist page which will be used in the resume
> > path.
> > 
> > Signed-off-by: Wei Liu <liuwe@microsoft.com>
> > ---
> >  xen/arch/x86/guest/hyperv/hyperv.c | 35 ++++++++++++++++++++++++++++++
> >  xen/include/asm-x86/guest/hyperv.h |  1 +
> >  2 files changed, 36 insertions(+)
> > 
> > diff --git a/xen/arch/x86/guest/hyperv/hyperv.c b/xen/arch/x86/guest/hyperv/hyperv.c
> > index da3a8cd85d..a88b9ae6d9 100644
> > --- a/xen/arch/x86/guest/hyperv/hyperv.c
> > +++ b/xen/arch/x86/guest/hyperv/hyperv.c
> > @@ -30,6 +30,7 @@ void *hv_hypercall;
> >  static struct page_info *hv_hypercall_page;
> >  DEFINE_PER_CPU_READ_MOSTLY(struct hyperv_pcpu_page, hv_pcpu_input_arg);
> >  DEFINE_PER_CPU_READ_MOSTLY(unsigned int, hv_vp_index);
> > +DEFINE_PER_CPU_READ_MOSTLY(struct hyperv_pcpu_page, hv_vp_assist);
> > 
> >  static const struct hypervisor_ops ops;
> >  const struct hypervisor_ops *__init hyperv_probe(void)
> > @@ -125,17 +126,51 @@ static void setup_vp_index(void)
> >      this_cpu(hv_vp_index) = vp_index_msr;
> >  }
> > 
> > +static void set_vp_assist(void)
> > +{
> > +    uint64_t val = paddr_to_pfn(this_cpu(hv_vp_assist).maddr);
> > +
> > +    val = (val << HV_X64_MSR_VP_ASSIST_PAGE_ADDRESS_SHIFT) | 
> 
> I'd recommend using HV_HYP_PAGE_SHIFT instead of
> HV_X64_MSR_VP_ASSIST_PAGE_ADDRESS_SHIFT.  On the Linux side,
> I'm planning to remove that #define and the similar
> *_PAGE_ADDRESS_MASK in favor of the newer HV_HYP_PAGE_* values.
> There's nothing special about the VP assist page, so using the generic
> #defines based on the Hyper-V page size is reasonable.

Ack.

Wei.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] [PATCH 4/8] x86/hyperv: setup hypercall page

[Andrew Cooper]

On 29/12/2019 18:33, Wei Liu wrote:
> @@ -71,6 +72,40 @@ const struct hypervisor_ops *__init hyperv_probe(void)
>      return &ops;
>  }
>  
> +static void __init setup_hypercall_page(void)
> +{
> +    union hv_x64_msr_hypercall_contents hypercall_msr;
> +
> +    /* Unfortunately there isn't a really good way to unwind Xen to
> +     * not use Hyper-V hooks, so panic if anything goes wrong.
> +     *
> +     * In practice if page allocation fails this early on it is
> +     * unlikely we can get a working system later.
> +     */
> +    hv_hypercall_page = alloc_domheap_page(NULL, 0);
> +    if ( !hv_hypercall_page )
> +        panic("Failed to allocate Hyper-V hypercall page\n");
> +
> +    hv_hypercall = __map_domain_page_global(hv_hypercall_page);
> +    if ( !hv_hypercall )
> +        panic("Failed to map Hyper-V hypercall page\n");

I really hope this doesn't actually function correctly.  This should
result in an NX mapping.

See feedback on the next patch for an alternative suggestion.

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] [PATCH 5/8] x86/hyperv: provide Hyper-V hypercall functions

[Andrew Cooper]

On 29/12/2019 18:33, Wei Liu wrote:
> These functions will be used later to make hypercalls to Hyper-V.
>
> I couldn't find reference in TLFS that Hyper-V clobbers flags and
> r9-r11, but Linux's commit message says it does. Err on the safe side.
>
> Signed-off-by: Wei Liu <liuwe@microsoft.com>
> ---
>  xen/include/asm-x86/guest/hyperv-hypercall.h | 105 +++++++++++++++++++
>  1 file changed, 105 insertions(+)
>  create mode 100644 xen/include/asm-x86/guest/hyperv-hypercall.h
>
> diff --git a/xen/include/asm-x86/guest/hyperv-hypercall.h b/xen/include/asm-x86/guest/hyperv-hypercall.h
> new file mode 100644
> index 0000000000..6017123be5
> --- /dev/null
> +++ b/xen/include/asm-x86/guest/hyperv-hypercall.h
> @@ -0,0 +1,105 @@
> +/******************************************************************************
> + * asm-x86/guest/hyperv-hypercall.h
> + *
> + * This program is free software; you can redistribute it and/or
> + * modify it under the terms and conditions of the GNU General Public
> + * License, version 2, as published by the Free Software Foundation.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
> + * General Public License for more details.
> + *
> + * You should have received a copy of the GNU General Public
> + * License along with this program; If not, see <http://www.gnu.org/licenses/>.
> + *
> + * Copyright (c) 2019 Microsoft.
> + */
> +
> +#ifndef __X86_HYPERV_HYPERCALL_H__
> +#define __X86_HYPERV_HYPERCALL_H__
> +
> +#include <xen/types.h>
> +
> +#include <asm/asm_defns.h>
> +#include <asm/guest/hyperv-tlfs.h>
> +#include <asm/page.h>
> +
> +extern void *hv_hypercall;
> +
> +static inline uint64_t hv_do_hypercall(uint64_t control, paddr_t input, paddr_t output)
> +{
> +    uint64_t status;
> +
> +    if ( !hv_hypercall )
> +        return ~0ULL;
> +
> +    asm volatile ("mov %[output], %%r8\n"
> +                  "call *%[hypercall_page]"
> +                  : "=a" (status), "+c" (control),
> +                    "+d" (input) ASM_CALL_CONSTRAINT
> +                  : [output] "rm" (output),
> +                    [hypercall_page] "m" (hv_hypercall)
> +                  : "cc", "memory", "r8", "r9", "r10", "r11");
> +
> +    return status;
> +}

Indirect calls are expensive these days due to retpoline/IBRS, and in
this case, unnecessary.

You want something like:

asm ( ".pushsection \".text.page_aligned\", \"ax\", @progbits\n\t"
      ".align 4096\n\t"
      ".globl hyperv_hypercall\n\t"
      "hyperv_hypercall:\n\t"
      "mov -1, %rax\n\t"
      "ret\n\t"
      ".align 4096;\n\t" );

Which will put one page worth of space in .text.page_aligned (so it gets
mapped executable), at a location the linker can evaluate (so you can
use a direct call, and the disassembly will be easier to follow), which
is initialised to the "not ready yet" code so you don't need a runtime
check in every hypercall that you didn't get the order of initialisation
wrong at boot.

Alternatively, initialise to ud2 if some form of console can be reliably
be arranged to work from the very start.

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] [PATCH 4/8] x86/hyperv: setup hypercall page

[Wei Liu]

On Mon, Dec 30, 2019 at 12:55:22PM +0000, Andrew Cooper wrote:
> On 29/12/2019 18:33, Wei Liu wrote:
> > @@ -71,6 +72,40 @@ const struct hypervisor_ops *__init hyperv_probe(void)
> >      return &ops;
> >  }
> >  
> > +static void __init setup_hypercall_page(void)
> > +{
> > +    union hv_x64_msr_hypercall_contents hypercall_msr;
> > +
> > +    /* Unfortunately there isn't a really good way to unwind Xen to
> > +     * not use Hyper-V hooks, so panic if anything goes wrong.
> > +     *
> > +     * In practice if page allocation fails this early on it is
> > +     * unlikely we can get a working system later.
> > +     */
> > +    hv_hypercall_page = alloc_domheap_page(NULL, 0);
> > +    if ( !hv_hypercall_page )
> > +        panic("Failed to allocate Hyper-V hypercall page\n");
> > +
> > +    hv_hypercall = __map_domain_page_global(hv_hypercall_page);
> > +    if ( !hv_hypercall )
> > +        panic("Failed to map Hyper-V hypercall page\n");
> 
> I really hope this doesn't actually function correctly.  This should
> result in an NX mapping.
> 

Ah, stupid me. I had actually looked at Xen's implementation and thought
"wouldn't it be nice to save one page in the image". I clearly missed
that __map_domain_page_global makes the page NX.

Wei.

> See feedback on the next patch for an alternative suggestion.
> 
> ~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] [PATCH 5/8] x86/hyperv: provide Hyper-V hypercall functions

[Wei Liu]

On Mon, Dec 30, 2019 at 01:04:33PM +0000, Andrew Cooper wrote:
> On 29/12/2019 18:33, Wei Liu wrote:
> > These functions will be used later to make hypercalls to Hyper-V.
> >
> > I couldn't find reference in TLFS that Hyper-V clobbers flags and
> > r9-r11, but Linux's commit message says it does. Err on the safe side.
> >
> > Signed-off-by: Wei Liu <liuwe@microsoft.com>
> > ---
> >  xen/include/asm-x86/guest/hyperv-hypercall.h | 105 +++++++++++++++++++
> >  1 file changed, 105 insertions(+)
> >  create mode 100644 xen/include/asm-x86/guest/hyperv-hypercall.h
> >
> > diff --git a/xen/include/asm-x86/guest/hyperv-hypercall.h b/xen/include/asm-x86/guest/hyperv-hypercall.h
> > new file mode 100644
> > index 0000000000..6017123be5
> > --- /dev/null
> > +++ b/xen/include/asm-x86/guest/hyperv-hypercall.h
> > @@ -0,0 +1,105 @@
> > +/******************************************************************************
> > + * asm-x86/guest/hyperv-hypercall.h
> > + *
> > + * This program is free software; you can redistribute it and/or
> > + * modify it under the terms and conditions of the GNU General Public
> > + * License, version 2, as published by the Free Software Foundation.
> > + *
> > + * This program is distributed in the hope that it will be useful,
> > + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> > + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
> > + * General Public License for more details.
> > + *
> > + * You should have received a copy of the GNU General Public
> > + * License along with this program; If not, see <http://www.gnu.org/licenses/>.
> > + *
> > + * Copyright (c) 2019 Microsoft.
> > + */
> > +
> > +#ifndef __X86_HYPERV_HYPERCALL_H__
> > +#define __X86_HYPERV_HYPERCALL_H__
> > +
> > +#include <xen/types.h>
> > +
> > +#include <asm/asm_defns.h>
> > +#include <asm/guest/hyperv-tlfs.h>
> > +#include <asm/page.h>
> > +
> > +extern void *hv_hypercall;
> > +
> > +static inline uint64_t hv_do_hypercall(uint64_t control, paddr_t input, paddr_t output)
> > +{
> > +    uint64_t status;
> > +
> > +    if ( !hv_hypercall )
> > +        return ~0ULL;
> > +
> > +    asm volatile ("mov %[output], %%r8\n"
> > +                  "call *%[hypercall_page]"
> > +                  : "=a" (status), "+c" (control),
> > +                    "+d" (input) ASM_CALL_CONSTRAINT
> > +                  : [output] "rm" (output),
> > +                    [hypercall_page] "m" (hv_hypercall)
> > +                  : "cc", "memory", "r8", "r9", "r10", "r11");
> > +
> > +    return status;
> > +}
> 
> Indirect calls are expensive these days due to retpoline/IBRS, and in
> this case, unnecessary.
> 
> You want something like:
> 
> asm ( ".pushsection \".text.page_aligned\", \"ax\", @progbits\n\t"
>       ".align 4096\n\t"
>       ".globl hyperv_hypercall\n\t"
>       "hyperv_hypercall:\n\t"
>       "mov -1, %rax\n\t"
>       "ret\n\t"
>       ".align 4096;\n\t" );
> 
> Which will put one page worth of space in .text.page_aligned (so it gets
> mapped executable), at a location the linker can evaluate (so you can
> use a direct call, and the disassembly will be easier to follow), which
> is initialised to the "not ready yet" code so you don't need a runtime
> check in every hypercall that you didn't get the order of initialisation
> wrong at boot.

This works for me. Thanks.

Wei.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] [PATCH 4/8] x86/hyperv: setup hypercall page

[Andrew Cooper]

On 30/12/2019 13:33, Wei Liu wrote:
> On Mon, Dec 30, 2019 at 12:55:22PM +0000, Andrew Cooper wrote:
>> On 29/12/2019 18:33, Wei Liu wrote:
>>> @@ -71,6 +72,40 @@ const struct hypervisor_ops *__init hyperv_probe(void)
>>>      return &ops;
>>>  }
>>>  
>>> +static void __init setup_hypercall_page(void)
>>> +{
>>> +    union hv_x64_msr_hypercall_contents hypercall_msr;
>>> +
>>> +    /* Unfortunately there isn't a really good way to unwind Xen to
>>> +     * not use Hyper-V hooks, so panic if anything goes wrong.
>>> +     *
>>> +     * In practice if page allocation fails this early on it is
>>> +     * unlikely we can get a working system later.
>>> +     */
>>> +    hv_hypercall_page = alloc_domheap_page(NULL, 0);
>>> +    if ( !hv_hypercall_page )
>>> +        panic("Failed to allocate Hyper-V hypercall page\n");
>>> +
>>> +    hv_hypercall = __map_domain_page_global(hv_hypercall_page);
>>> +    if ( !hv_hypercall )
>>> +        panic("Failed to map Hyper-V hypercall page\n");
>> I really hope this doesn't actually function correctly.  This should
>> result in an NX mapping.
>>
> Ah, stupid me. I had actually looked at Xen's implementation and thought
> "wouldn't it be nice to save one page in the image".

Its 4k, and there is a lot to be said for not having random tiny
critical bits of infrastructure spread dynamically around GFN space.

> I clearly missed that __map_domain_page_global makes the page NX.

It is hidden in the depths of PAGE_HYPERVISOR.

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] [PATCH 2/8] x86/hyperv: detect absolutely necessary MSRs

[Paul Durrant]

On Sun, 29 Dec 2019 at 18:34, Wei Liu <wl@xen.org> wrote:
>
> If they are not available, disable Hyper-V related features.
>
> Signed-off-by: Wei Liu <liuwe@microsoft.com>
> ---
>  xen/arch/x86/guest/hyperv/hyperv.c | 11 +++++++++++
>  1 file changed, 11 insertions(+)
>
> diff --git a/xen/arch/x86/guest/hyperv/hyperv.c b/xen/arch/x86/guest/hyperv/hyperv.c
> index 2e70b4aa82..c6a26c5453 100644
> --- a/xen/arch/x86/guest/hyperv/hyperv.c
> +++ b/xen/arch/x86/guest/hyperv/hyperv.c
> @@ -32,6 +32,8 @@ static const struct hypervisor_ops ops = {
>  const struct hypervisor_ops *__init hyperv_probe(void)
>  {
>      uint32_t eax, ebx, ecx, edx;
> +    uint64_t required_msrs = HV_X64_MSR_HYPERCALL_AVAILABLE |
> +        HV_X64_MSR_VP_INDEX_AVAILABLE;
>
>      cpuid(0x40000000, &eax, &ebx, &ecx, &edx);
>      if ( !((ebx == 0x7263694d) &&  /* "Micr" */
> @@ -57,6 +59,15 @@ const struct hypervisor_ops *__init hyperv_probe(void)
>      ms_hyperv.max_vp_index = eax;
>      ms_hyperv.max_lp_index = ebx;
>
> +    if ( (ms_hyperv.features & required_msrs) != required_msrs )
> +    {
> +        /* Oops, required MSRs are not available. Treat this as
> +         * "Hyper-V is not available".
> +         */

That block comment violates default coding style.

  Paul

> +        memset(&ms_hyperv, 0, sizeof(ms_hyperv));
> +        return NULL;
> +    }
> +
>      return &ops;
>  }
>
> --
> 2.20.1
>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] [PATCH 3/8] x86: rename guest/hypercall.h to guest/xen-hypercall.h

[Paul Durrant]

On Sun, 29 Dec 2019 at 18:34, Wei Liu <wl@xen.org> wrote:
>
> We will provide a header file for Hyper-V hypercalls.
>
> No functional change.
>
> Signed-off-by: Wei Liu <liuwe@microsoft.com>

Reviewed-by: Paul Durrant <paul@xen.org>

> ---
>  xen/include/asm-x86/guest.h                                | 2 +-
>  xen/include/asm-x86/guest/{hypercall.h => xen-hypercall.h} | 2 +-
>  2 files changed, 2 insertions(+), 2 deletions(-)
>  rename xen/include/asm-x86/guest/{hypercall.h => xen-hypercall.h} (99%)
>
> diff --git a/xen/include/asm-x86/guest.h b/xen/include/asm-x86/guest.h
> index 94448606d4..7f09c643d4 100644
> --- a/xen/include/asm-x86/guest.h
> +++ b/xen/include/asm-x86/guest.h
> @@ -19,11 +19,11 @@
>  #ifndef __X86_GUEST_H__
>  #define __X86_GUEST_H__
>
> -#include <asm/guest/hypercall.h>
>  #include <asm/guest/hyperv.h>
>  #include <asm/guest/hypervisor.h>
>  #include <asm/guest/pvh-boot.h>
>  #include <asm/guest/xen.h>
> +#include <asm/guest/xen-hypercall.h>
>  #include <asm/pv/shim.h>
>
>  #endif /* __X86_GUEST_H__ */
> diff --git a/xen/include/asm-x86/guest/hypercall.h b/xen/include/asm-x86/guest/xen-hypercall.h
> similarity index 99%
> rename from xen/include/asm-x86/guest/hypercall.h
> rename to xen/include/asm-x86/guest/xen-hypercall.h
> index d0d2f5022d..681d3aecd7 100644
> --- a/xen/include/asm-x86/guest/hypercall.h
> +++ b/xen/include/asm-x86/guest/xen-hypercall.h
> @@ -1,5 +1,5 @@
>  /******************************************************************************
> - * asm-x86/guest/hypercall.h
> + * asm-x86/guest/xen-hypercall.h
>   *
>   * This program is free software; you can redistribute it and/or
>   * modify it under the terms and conditions of the GNU General Public
> --
> 2.20.1
>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] [PATCH 6/8] x86/hyperv: provide percpu hypercall input page

[Paul Durrant]

On Sun, 29 Dec 2019 at 18:35, Wei Liu <wl@xen.org> wrote:
>
> Hyper-V's input / output argument must be 8 bytes aligned an not cross
> page boundary. The easiest way to satisfy those requirements is to use
> percpu page.
>
> For the foreseeable future we only need to provide input for TLB
> and APIC hypercalls, so skip setting up an output page.
>
> The page tracking structure is not bound to hypercall because it is a
> common pattern for Xen to write guest physical address to Hyper-V while
> at the same time accessing the page via a pointer.
>
> We will also need to provide an ap_setup hook for secondary cpus to
> setup its own input page.
>
> Signed-off-by: Wei Liu <liuwe@microsoft.com>

Reviewed-by: Paul Durrant <paul@xen.org>

> ---
>  xen/arch/x86/guest/hyperv/hyperv.c | 26 ++++++++++++++++++++++++++
>  xen/include/asm-x86/guest/hyperv.h |  8 ++++++++
>  2 files changed, 34 insertions(+)
>
> diff --git a/xen/arch/x86/guest/hyperv/hyperv.c b/xen/arch/x86/guest/hyperv/hyperv.c
> index 438910c8cb..67667936e9 100644
> --- a/xen/arch/x86/guest/hyperv/hyperv.c
> +++ b/xen/arch/x86/guest/hyperv/hyperv.c
> @@ -28,6 +28,7 @@ struct ms_hyperv_info __read_mostly ms_hyperv;
>
>  void *hv_hypercall;
>  static struct page_info *hv_hypercall_page;
> +DEFINE_PER_CPU_READ_MOSTLY(struct hyperv_pcpu_page, hv_pcpu_input_arg);
>
>  static const struct hypervisor_ops ops;
>  const struct hypervisor_ops *__init hyperv_probe(void)
> @@ -96,14 +97,39 @@ static void __init setup_hypercall_page(void)
>      wrmsrl(HV_X64_MSR_HYPERCALL, hypercall_msr.as_uint64);
>  }
>
> +static void setup_hypercall_pcpu_arg(void)
> +{
> +    struct page_info *pg;
> +    void *mapping;
> +    unsigned int cpu = smp_processor_id();
> +
> +    pg = alloc_domheap_page(NULL, 0);
> +    if ( !pg )
> +        panic("Failed to setup hypercall input page for %u\n", cpu);
> +
> +    mapping = __map_domain_page_global(pg);
> +    if ( !mapping )
> +        panic("Failed to map hypercall input page for %u\n", cpu);
> +
> +    this_cpu(hv_pcpu_input_arg).maddr = page_to_maddr(pg);
> +    this_cpu(hv_pcpu_input_arg).mapping = mapping;
> +}
> +
>  static void __init setup(void)
>  {
>      setup_hypercall_page();
> +    setup_hypercall_pcpu_arg();
> +}
> +
> +static void ap_setup(void)
> +{
> +    setup_hypercall_pcpu_arg();
>  }
>
>  static const struct hypervisor_ops ops = {
>      .name = "Hyper-V",
>      .setup = setup,
> +    .ap_setup = ap_setup,
>  };
>
>  /*
> diff --git a/xen/include/asm-x86/guest/hyperv.h b/xen/include/asm-x86/guest/hyperv.h
> index c7a7f32bd5..83f297468f 100644
> --- a/xen/include/asm-x86/guest/hyperv.h
> +++ b/xen/include/asm-x86/guest/hyperv.h
> @@ -51,6 +51,8 @@ static inline uint64_t hv_scale_tsc(uint64_t tsc, uint64_t scale,
>
>  #ifdef CONFIG_HYPERV_GUEST
>
> +#include <xen/percpu.h>
> +
>  #include <asm/guest/hypervisor.h>
>
>  struct ms_hyperv_info {
> @@ -63,6 +65,12 @@ struct ms_hyperv_info {
>  };
>  extern struct ms_hyperv_info ms_hyperv;
>
> +struct hyperv_pcpu_page {
> +    paddr_t maddr;
> +    void *mapping;
> +};
> +DECLARE_PER_CPU(struct hyperv_pcpu_page, hv_pcpu_input_arg);
> +
>  const struct hypervisor_ops *hyperv_probe(void);
>
>  #else
> --
> 2.20.1
>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] [PATCH 7/8] x86/hyperv: retrieve vp_index from Hyper-V

[Paul Durrant]

On Sun, 29 Dec 2019 at 18:35, Wei Liu <wl@xen.org> wrote:
>
> This will be useful when invoking hypercall that targets specific
> vcpu(s).
>
> Signed-off-by: Wei Liu <liuwe@microsoft.com>
> ---
>  xen/arch/x86/guest/hyperv/hyperv.c | 12 ++++++++++++
>  xen/include/asm-x86/guest/hyperv.h |  1 +
>  2 files changed, 13 insertions(+)
>
> diff --git a/xen/arch/x86/guest/hyperv/hyperv.c b/xen/arch/x86/guest/hyperv/hyperv.c
> index 67667936e9..da3a8cd85d 100644
> --- a/xen/arch/x86/guest/hyperv/hyperv.c
> +++ b/xen/arch/x86/guest/hyperv/hyperv.c
> @@ -29,6 +29,7 @@ struct ms_hyperv_info __read_mostly ms_hyperv;
>  void *hv_hypercall;
>  static struct page_info *hv_hypercall_page;
>  DEFINE_PER_CPU_READ_MOSTLY(struct hyperv_pcpu_page, hv_pcpu_input_arg);
> +DEFINE_PER_CPU_READ_MOSTLY(unsigned int, hv_vp_index);
>
>  static const struct hypervisor_ops ops;
>  const struct hypervisor_ops *__init hyperv_probe(void)
> @@ -115,15 +116,26 @@ static void setup_hypercall_pcpu_arg(void)
>      this_cpu(hv_pcpu_input_arg).mapping = mapping;
>  }
>
> +static void setup_vp_index(void)
> +{
> +    uint64_t vp_index_msr;
> +
> +    rdmsrl(HV_X64_MSR_VP_INDEX, vp_index_msr);
> +
> +    this_cpu(hv_vp_index) = vp_index_msr;
> +}

Is it worth a separate function vs. bundling this into
setup_hypercall_pcpu_arg()?

  Paul

> +
>  static void __init setup(void)
>  {
>      setup_hypercall_page();
>      setup_hypercall_pcpu_arg();
> +    setup_vp_index();
>  }
>
>  static void ap_setup(void)
>  {
>      setup_hypercall_pcpu_arg();
> +    setup_vp_index();
>  }
>
>  static const struct hypervisor_ops ops = {
> diff --git a/xen/include/asm-x86/guest/hyperv.h b/xen/include/asm-x86/guest/hyperv.h
> index 83f297468f..4b635829f3 100644
> --- a/xen/include/asm-x86/guest/hyperv.h
> +++ b/xen/include/asm-x86/guest/hyperv.h
> @@ -70,6 +70,7 @@ struct hyperv_pcpu_page {
>      void *mapping;
>  };
>  DECLARE_PER_CPU(struct hyperv_pcpu_page, hv_pcpu_input_arg);
> +DECLARE_PER_CPU(unsigned int, hv_vp_index);
>
>  const struct hypervisor_ops *hyperv_probe(void);
>
> --
> 2.20.1
>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] [PATCH 7/8] x86/hyperv: retrieve vp_index from Hyper-V

[Wei Liu]

On Fri, Jan 03, 2020 at 11:11:39AM +0000, Paul Durrant wrote:
> On Sun, 29 Dec 2019 at 18:35, Wei Liu <wl@xen.org> wrote:
> >
> > This will be useful when invoking hypercall that targets specific
> > vcpu(s).
> >
> > Signed-off-by: Wei Liu <liuwe@microsoft.com>
> > ---
> >  xen/arch/x86/guest/hyperv/hyperv.c | 12 ++++++++++++
> >  xen/include/asm-x86/guest/hyperv.h |  1 +
> >  2 files changed, 13 insertions(+)
> >
> > diff --git a/xen/arch/x86/guest/hyperv/hyperv.c b/xen/arch/x86/guest/hyperv/hyperv.c
> > index 67667936e9..da3a8cd85d 100644
> > --- a/xen/arch/x86/guest/hyperv/hyperv.c
> > +++ b/xen/arch/x86/guest/hyperv/hyperv.c
> > @@ -29,6 +29,7 @@ struct ms_hyperv_info __read_mostly ms_hyperv;
> >  void *hv_hypercall;
> >  static struct page_info *hv_hypercall_page;
> >  DEFINE_PER_CPU_READ_MOSTLY(struct hyperv_pcpu_page, hv_pcpu_input_arg);
> > +DEFINE_PER_CPU_READ_MOSTLY(unsigned int, hv_vp_index);
> >
> >  static const struct hypervisor_ops ops;
> >  const struct hypervisor_ops *__init hyperv_probe(void)
> > @@ -115,15 +116,26 @@ static void setup_hypercall_pcpu_arg(void)
> >      this_cpu(hv_pcpu_input_arg).mapping = mapping;
> >  }
> >
> > +static void setup_vp_index(void)
> > +{
> > +    uint64_t vp_index_msr;
> > +
> > +    rdmsrl(HV_X64_MSR_VP_INDEX, vp_index_msr);
> > +
> > +    this_cpu(hv_vp_index) = vp_index_msr;
> > +}
> 
> Is it worth a separate function vs. bundling this into
> setup_hypercall_pcpu_arg()?

My thinking is this may be called in the resume path; while
setup_hypercall_pcpu_arg doesn't (because they just set aside a bunch of
guest pages).

That is just a precaution. I haven't found any text in TLFS that vp
index will / can change after resume. This function can certainly be
folded into the other if that's preferred.

Wei.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] [PATCH 1/8] x86: include xen/lib.h in guest/pvh-boot.h

[Jan Beulich]

On 29.12.2019 19:33, Wei Liu wrote:
> It needs ASSERT_UNREACHABLE.
> 
> Signed-off-by: Wei Liu <liuwe@microsoft.com>

Acked-by: Jan Beulich <jbeulich@suse.com>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] [PATCH 2/8] x86/hyperv: detect absolutely necessary MSRs

[Jan Beulich]

On 03.01.2020 12:01, Paul Durrant wrote:
> On Sun, 29 Dec 2019 at 18:34, Wei Liu <wl@xen.org> wrote:
>> --- a/xen/arch/x86/guest/hyperv/hyperv.c
>> +++ b/xen/arch/x86/guest/hyperv/hyperv.c
>> @@ -32,6 +32,8 @@ static const struct hypervisor_ops ops = {
>>  const struct hypervisor_ops *__init hyperv_probe(void)
>>  {
>>      uint32_t eax, ebx, ecx, edx;
>> +    uint64_t required_msrs = HV_X64_MSR_HYPERCALL_AVAILABLE |
>> +        HV_X64_MSR_VP_INDEX_AVAILABLE;
>>
>>      cpuid(0x40000000, &eax, &ebx, &ecx, &edx);
>>      if ( !((ebx == 0x7263694d) &&  /* "Micr" */
>> @@ -57,6 +59,15 @@ const struct hypervisor_ops *__init hyperv_probe(void)
>>      ms_hyperv.max_vp_index = eax;
>>      ms_hyperv.max_lp_index = ebx;
>>
>> +    if ( (ms_hyperv.features & required_msrs) != required_msrs )
>> +    {
>> +        /* Oops, required MSRs are not available. Treat this as
>> +         * "Hyper-V is not available".
>> +         */
> 
> That block comment violates default coding style.

And with it corrected
Acked-by: Jan Beulich <jbeulich@suse.com>

Jan

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] [PATCH 3/8] x86: rename guest/hypercall.h to guest/xen-hypercall.h

[Jan Beulich]

On 29.12.2019 19:33, Wei Liu wrote:
> We will provide a header file for Hyper-V hypercalls.
> 
> No functional change.
> 
> Signed-off-by: Wei Liu <liuwe@microsoft.com>

In principle
Acked-by: Jan Beulich <jbeulich@suse.com>
albeit ...

> ---
>  xen/include/asm-x86/guest.h                                | 2 +-
>  xen/include/asm-x86/guest/{hypercall.h => xen-hypercall.h} | 2 +-

... could we settle on guest/*-hcall.h to save on file name
length without (hopefully) becoming ambiguous?

Jan

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] [PATCH 3/8] x86: rename guest/hypercall.h to guest/xen-hypercall.h

[Wei Liu]

On Fri, Jan 03, 2020 at 05:16:53PM +0100, Jan Beulich wrote:
> On 29.12.2019 19:33, Wei Liu wrote:
> > We will provide a header file for Hyper-V hypercalls.
> > 
> > No functional change.
> > 
> > Signed-off-by: Wei Liu <liuwe@microsoft.com>
> 
> In principle
> Acked-by: Jan Beulich <jbeulich@suse.com>
> albeit ...
> 
> > ---
> >  xen/include/asm-x86/guest.h                                | 2 +-
> >  xen/include/asm-x86/guest/{hypercall.h => xen-hypercall.h} | 2 +-
> 
> ... could we settle on guest/*-hcall.h to save on file name
> length without (hopefully) becoming ambiguous?

That works for me.

Wei.

> 
> Jan

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel