[PATCH v2 0/6] tools/libs: add missing support of linear p2m_list, cleanup

[PATCH v2 0/6] tools/libs: add missing support of linear p2m_list, cleanup

[Juergen Gross]

There are some corners left which don't support the not so very new
linear p2m list of pv guests, which has been introduced in Linux kernel
3.19 and which is mandatory for non-legacy versions of Xen since kernel
4.14.

This series adds support for the linear p2m list where it is missing
(colo support and "xl dump-core").

In theory it should be possible to merge the p2m list mapping code
from migration handling and core dump handling, but this needs quite
some cleanup before this is possible.

The first three patches of this series are fixing real problems, so
I've put them at the start of this series, especially in order to make
backports easier.

The other three patches are only the first steps of cleanup. The main
work done here is to concentrate all p2m mapping in libxenguest instead
of having one implementation in each of libxenguest and libxenctrl.

Merging the two implementations should be rather easy, but this will
require to touch many lines of code, as the migration handling variant
seems to be more mature, but it is using the migration stream specific
structures heavily. So I'd like to have some confirmation that my way
to clean this up is the right one.

My idea would be to add the data needed for p2m mapping to struct
domain_info_context and replace the related fields in struct
xc_sr_context with a struct domain_info_context. Modifying the
interface of xc_core_arch_map_p2m() to take most current parameters
via struct domain_info_context would then enable migration coding to
use xc_core_arch_map_p2m() for mapping the p2m. xc_core_arch_map_p2m()
should look basically like the current migration p2m mapping code
afterwards.

Any comments to that plan?

Changes in V2:
- added missing #include in ocaml stub

Juergen Gross (6):
  tools/libs/guest: fix max_pfn setting in map_p2m()
  tools/libs/ctrl: fix xc_core_arch_map_p2m() to support linear p2m
    table
  tools/libs/ctrl: use common p2m mapping code in xc_domain_resume_any()
  tools/libs: move xc_resume.c to libxenguest
  tools/libs: move xc_core* from libxenctrl to libxenguest
  tools/libs/guest: make some definitions private to libxenguest

 tools/include/xenctrl.h                       |  63 ---
 tools/include/xenguest.h                      |  63 +++
 tools/libs/ctrl/Makefile                      |   4 -
 tools/libs/ctrl/xc_core_x86.c                 | 223 ----------
 tools/libs/ctrl/xc_domain.c                   |   2 -
 tools/libs/ctrl/xc_private.h                  |  43 +-
 tools/libs/guest/Makefile                     |   4 +
 .../libs/{ctrl/xc_core.c => guest/xg_core.c}  |   7 +-
 .../libs/{ctrl/xc_core.h => guest/xg_core.h}  |  15 +-
 .../xc_core_arm.c => guest/xg_core_arm.c}     |  31 +-
 .../xc_core_arm.h => guest/xg_core_arm.h}     |   0
 tools/libs/guest/xg_core_x86.c                | 399 ++++++++++++++++++
 .../xc_core_x86.h => guest/xg_core_x86.h}     |   0
 tools/libs/guest/xg_dom_boot.c                |   2 +-
 tools/libs/guest/xg_domain.c                  |  19 +-
 tools/libs/guest/xg_offline_page.c            |   2 +-
 tools/libs/guest/xg_private.h                 |  16 +-
 .../{ctrl/xc_resume.c => guest/xg_resume.c}   |  69 +--
 tools/libs/guest/xg_sr_save_x86_pv.c          |   2 +-
 tools/ocaml/libs/xc/xenctrl_stubs.c           |   1 +
 20 files changed, 545 insertions(+), 420 deletions(-)
 delete mode 100644 tools/libs/ctrl/xc_core_x86.c
 rename tools/libs/{ctrl/xc_core.c => guest/xg_core.c} (99%)
 rename tools/libs/{ctrl/xc_core.h => guest/xg_core.h} (92%)
 rename tools/libs/{ctrl/xc_core_arm.c => guest/xg_core_arm.c} (72%)
 rename tools/libs/{ctrl/xc_core_arm.h => guest/xg_core_arm.h} (100%)
 create mode 100644 tools/libs/guest/xg_core_x86.c
 rename tools/libs/{ctrl/xc_core_x86.h => guest/xg_core_x86.h} (100%)
 rename tools/libs/{ctrl/xc_resume.c => guest/xg_resume.c} (80%)

-- 
2.26.2

[PATCH v2 1/6] tools/libs/guest: fix max_pfn setting in map_p2m()

[Juergen Gross]

When setting the highest pfn used in the guest, don't subtract 1 from
the value read from the shared_info data. The value read already is
the correct pfn.

Fixes: 91e204d37f449 ("libxc: try to find last used pfn when migrating")
Signed-off-by: Juergen Gross <jgross@suse.com>
---
This is a backport candidate
---
 tools/libs/guest/xg_sr_save_x86_pv.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tools/libs/guest/xg_sr_save_x86_pv.c b/tools/libs/guest/xg_sr_save_x86_pv.c
index 4964f1f7b8..dae7f2817f 100644
--- a/tools/libs/guest/xg_sr_save_x86_pv.c
+++ b/tools/libs/guest/xg_sr_save_x86_pv.c
@@ -468,7 +468,7 @@ static int map_p2m(struct xc_sr_context *ctx)
 
     ctx->x86.pv.p2m_generation = ~0ULL;
     ctx->x86.pv.max_pfn = GET_FIELD(ctx->x86.pv.shinfo, arch.max_pfn,
-                                    ctx->x86.pv.width) - 1;
+                                    ctx->x86.pv.width);
     p2m_cr3 = GET_FIELD(ctx->x86.pv.shinfo, arch.p2m_cr3, ctx->x86.pv.width);
 
     return p2m_cr3 ? map_p2m_list(ctx, p2m_cr3) : map_p2m_tree(ctx);
-- 
2.26.2

Re: [PATCH v2 1/6] tools/libs/guest: fix max_pfn setting in map_p2m()

[Wei Liu]

On Mon, Apr 12, 2021 at 05:22:31PM +0200, Juergen Gross wrote:
> When setting the highest pfn used in the guest, don't subtract 1 from
> the value read from the shared_info data. The value read already is
> the correct pfn.
> 
> Fixes: 91e204d37f449 ("libxc: try to find last used pfn when migrating")
> Signed-off-by: Juergen Gross <jgross@suse.com>

Acked-by: Wei Liu <wl@xen.org>

[PATCH v2 2/6] tools/libs/ctrl: fix xc_core_arch_map_p2m() to support linear p2m table

[Juergen Gross]

The core of a pv linux guest produced via "xl dump-core" is not usable
as since kernel 4.14 only the linear p2m table is kept if Xen indicates
it is supporting that. Unfortunately xc_core_arch_map_p2m() is still
supporting the 3-level p2m tree only.

Fix that by copying the functionality of map_p2m() from libxenguest to
libxenctrl.

Additionally the mapped p2m isn't of a fixed length now, so the
interface to the mapping functions needs to be adapted. In order not to
add even more parameters, expand struct domain_info_context and use a
pointer to that as a parameter.

Fixes: dc6d60937121 ("libxc: set flag for support of linear p2m list in domain builder")
Signed-off-by: Juergen Gross <jgross@suse.com>
---
This is a backport candidate.
---
 tools/include/xenguest.h      |   1 +
 tools/libs/ctrl/xc_core.c     |   5 +-
 tools/libs/ctrl/xc_core.h     |   8 +-
 tools/libs/ctrl/xc_core_arm.c |  23 +--
 tools/libs/ctrl/xc_core_x86.c | 256 ++++++++++++++++++++++++++++------
 tools/libs/ctrl/xc_private.h  |   1 +
 tools/libs/guest/xg_domain.c  |  17 +--
 7 files changed, 233 insertions(+), 78 deletions(-)

diff --git a/tools/include/xenguest.h b/tools/include/xenguest.h
index 217022b6e7..781d40c119 100644
--- a/tools/include/xenguest.h
+++ b/tools/include/xenguest.h
@@ -700,6 +700,7 @@ struct xc_domain_meminfo {
     xen_pfn_t *pfn_type;
     xen_pfn_t *p2m_table;
     unsigned long p2m_size;
+    unsigned int p2m_frames;
 };
 
 int xc_map_domain_meminfo(xc_interface *xch, uint32_t domid,
diff --git a/tools/libs/ctrl/xc_core.c b/tools/libs/ctrl/xc_core.c
index b47ab2f6d8..9576bec5a3 100644
--- a/tools/libs/ctrl/xc_core.c
+++ b/tools/libs/ctrl/xc_core.c
@@ -574,8 +574,7 @@ xc_domain_dumpcore_via_callback(xc_interface *xch,
             goto out;
         }
 
-        sts = xc_core_arch_map_p2m(xch, dinfo->guest_width, &info, live_shinfo,
-                                   &p2m, &dinfo->p2m_size);
+        sts = xc_core_arch_map_p2m(xch, dinfo, &info, live_shinfo, &p2m);
         if ( sts != 0 )
             goto out;
 
@@ -945,7 +944,7 @@ out:
     if ( memory_map != NULL )
         free(memory_map);
     if ( p2m != NULL )
-        munmap(p2m, PAGE_SIZE * P2M_FL_ENTRIES);
+        munmap(p2m, PAGE_SIZE * dinfo->p2m_frames);
     if ( p2m_array != NULL )
         free(p2m_array);
     if ( pfn_array != NULL )
diff --git a/tools/libs/ctrl/xc_core.h b/tools/libs/ctrl/xc_core.h
index 36fb755da2..8ea1f93a10 100644
--- a/tools/libs/ctrl/xc_core.h
+++ b/tools/libs/ctrl/xc_core.h
@@ -138,14 +138,14 @@ int xc_core_arch_memory_map_get(xc_interface *xch,
                                 xc_dominfo_t *info, shared_info_any_t *live_shinfo,
                                 xc_core_memory_map_t **mapp,
                                 unsigned int *nr_entries);
-int xc_core_arch_map_p2m(xc_interface *xch, unsigned int guest_width,
+int xc_core_arch_map_p2m(xc_interface *xch, struct domain_info_context *dinfo,
                          xc_dominfo_t *info, shared_info_any_t *live_shinfo,
-                         xen_pfn_t **live_p2m, unsigned long *pfnp);
+                         xen_pfn_t **live_p2m);
 
-int xc_core_arch_map_p2m_writable(xc_interface *xch, unsigned int guest_width,
+int xc_core_arch_map_p2m_writable(xc_interface *xch, struct domain_info_context *dinfo,
                                   xc_dominfo_t *info,
                                   shared_info_any_t *live_shinfo,
-                                  xen_pfn_t **live_p2m, unsigned long *pfnp);
+                                  xen_pfn_t **live_p2m);
 
 int xc_core_arch_get_scratch_gpfn(xc_interface *xch, uint32_t domid,
                                   xen_pfn_t *gpfn);
diff --git a/tools/libs/ctrl/xc_core_arm.c b/tools/libs/ctrl/xc_core_arm.c
index 7b587b4cc5..93765a565f 100644
--- a/tools/libs/ctrl/xc_core_arm.c
+++ b/tools/libs/ctrl/xc_core_arm.c
@@ -66,33 +66,24 @@ xc_core_arch_memory_map_get(xc_interface *xch, struct xc_core_arch_context *unus
 
 static int
 xc_core_arch_map_p2m_rw(xc_interface *xch, struct domain_info_context *dinfo, xc_dominfo_t *info,
-                        shared_info_any_t *live_shinfo, xen_pfn_t **live_p2m,
-                        unsigned long *pfnp, int rw)
+                        shared_info_any_t *live_shinfo, xen_pfn_t **live_p2m, int rw)
 {
     errno = ENOSYS;
     return -1;
 }
 
 int
-xc_core_arch_map_p2m(xc_interface *xch, unsigned int guest_width, xc_dominfo_t *info,
-                        shared_info_any_t *live_shinfo, xen_pfn_t **live_p2m,
-                        unsigned long *pfnp)
+xc_core_arch_map_p2m(xc_interface *xch, struct domain_info_context *dinfo, xc_dominfo_t *info,
+                        shared_info_any_t *live_shinfo, xen_pfn_t **live_p2m)
 {
-    struct domain_info_context _dinfo = { .guest_width = guest_width };
-    struct domain_info_context *dinfo = &_dinfo;
-    return xc_core_arch_map_p2m_rw(xch, dinfo, info,
-                                   live_shinfo, live_p2m, pfnp, 0);
+    return xc_core_arch_map_p2m_rw(xch, dinfo, info, live_shinfo, live_p2m, 0);
 }
 
 int
-xc_core_arch_map_p2m_writable(xc_interface *xch, unsigned int guest_width, xc_dominfo_t *info,
-                              shared_info_any_t *live_shinfo, xen_pfn_t **live_p2m,
-                              unsigned long *pfnp)
+xc_core_arch_map_p2m_writable(xc_interface *xch, struct domain_info_context *dinfo, xc_dominfo_t *info,
+                              shared_info_any_t *live_shinfo, xen_pfn_t **live_p2m)
 {
-    struct domain_info_context _dinfo = { .guest_width = guest_width };
-    struct domain_info_context *dinfo = &_dinfo;
-    return xc_core_arch_map_p2m_rw(xch, dinfo, info,
-                                   live_shinfo, live_p2m, pfnp, 1);
+    return xc_core_arch_map_p2m_rw(xch, dinfo, info, live_shinfo, live_p2m, 1);
 }
 
 int
diff --git a/tools/libs/ctrl/xc_core_x86.c b/tools/libs/ctrl/xc_core_x86.c
index cb76e6207b..c8f71d4b75 100644
--- a/tools/libs/ctrl/xc_core_x86.c
+++ b/tools/libs/ctrl/xc_core_x86.c
@@ -17,6 +17,7 @@
  *
  */
 
+#include <inttypes.h>
 #include "xc_private.h"
 #include "xc_core.h"
 #include <xen/hvm/e820.h>
@@ -65,34 +66,169 @@ xc_core_arch_memory_map_get(xc_interface *xch, struct xc_core_arch_context *unus
     return 0;
 }
 
-static int
-xc_core_arch_map_p2m_rw(xc_interface *xch, struct domain_info_context *dinfo, xc_dominfo_t *info,
-                        shared_info_any_t *live_shinfo, xen_pfn_t **live_p2m,
-                        unsigned long *pfnp, int rw)
+static inline bool is_canonical_address(uint64_t vaddr)
 {
-    /* Double and single indirect references to the live P2M table */
-    xen_pfn_t *live_p2m_frame_list_list = NULL;
-    xen_pfn_t *live_p2m_frame_list = NULL;
-    /* Copies of the above. */
-    xen_pfn_t *p2m_frame_list_list = NULL;
-    xen_pfn_t *p2m_frame_list = NULL;
+    return ((int64_t)vaddr >> 47) == ((int64_t)vaddr >> 63);
+}
 
-    uint32_t dom = info->domid;
-    int ret = -1;
-    int err;
-    int i;
+/* Virtual address ranges reserved for hypervisor. */
+#define HYPERVISOR_VIRT_START_X86_64 0xFFFF800000000000ULL
+#define HYPERVISOR_VIRT_END_X86_64   0xFFFF87FFFFFFFFFFULL
 
-    if ( xc_domain_nr_gpfns(xch, info->domid, &dinfo->p2m_size) < 0 )
+#define HYPERVISOR_VIRT_START_X86_32 0x00000000F5800000ULL
+#define HYPERVISOR_VIRT_END_X86_32   0x00000000FFFFFFFFULL
+
+static xen_pfn_t *
+xc_core_arch_map_p2m_list_rw(xc_interface *xch, struct domain_info_context *dinfo,
+                             uint32_t dom, shared_info_any_t *live_shinfo,
+                             uint64_t p2m_cr3)
+{
+    uint64_t p2m_vaddr, p2m_end, mask, off;
+    xen_pfn_t p2m_mfn, mfn, saved_mfn, max_pfn;
+    uint64_t *ptes = NULL;
+    xen_pfn_t *mfns = NULL;
+    unsigned int fpp, n_pages, level, n_levels, shift,
+                 idx_start, idx_end, idx, saved_idx;
+
+    p2m_vaddr = GET_FIELD(live_shinfo, arch.p2m_vaddr, dinfo->guest_width);
+    fpp = PAGE_SIZE / dinfo->guest_width;
+    dinfo->p2m_frames = (dinfo->p2m_size - 1) / fpp + 1;
+    p2m_end = p2m_vaddr + dinfo->p2m_frames * PAGE_SIZE - 1;
+
+    if ( dinfo->guest_width == 8 )
     {
-        ERROR("Could not get maximum GPFN!");
-        goto out;
+        mask = 0x0000ffffffffffffULL;
+        n_levels = 4;
+        p2m_mfn = p2m_cr3 >> 12;
+        if ( !is_canonical_address(p2m_vaddr) ||
+             !is_canonical_address(p2m_end) ||
+             p2m_end < p2m_vaddr ||
+             (p2m_vaddr <= HYPERVISOR_VIRT_END_X86_64 &&
+              p2m_end > HYPERVISOR_VIRT_START_X86_64) )
+        {
+            ERROR("Bad virtual p2m address range %#" PRIx64 "-%#" PRIx64,
+                  p2m_vaddr, p2m_end);
+            errno = ERANGE;
+            goto out;
+        }
+    }
+    else
+    {
+        mask = 0x00000000ffffffffULL;
+        n_levels = 3;
+        if ( p2m_cr3 & ~mask )
+            p2m_mfn = ~0UL;
+        else
+            p2m_mfn = (uint32_t)((p2m_cr3 >> 12) | (p2m_cr3 << 20));
+        if ( p2m_vaddr > mask || p2m_end > mask || p2m_end < p2m_vaddr ||
+             (p2m_vaddr <= HYPERVISOR_VIRT_END_X86_32 &&
+              p2m_end > HYPERVISOR_VIRT_START_X86_32) )
+        {
+            ERROR("Bad virtual p2m address range %#" PRIx64 "-%#" PRIx64,
+                  p2m_vaddr, p2m_end);
+            errno = ERANGE;
+            goto out;
+        }
     }
 
-    if ( dinfo->p2m_size < info->nr_pages  )
+    mfns = malloc(sizeof(*mfns));
+    if ( !mfns )
     {
-        ERROR("p2m_size < nr_pages -1 (%lx < %lx", dinfo->p2m_size, info->nr_pages - 1);
+        ERROR("Cannot allocate memory for array of %u mfns", 1);
         goto out;
     }
+    mfns[0] = p2m_mfn;
+    off = 0;
+    saved_mfn = 0;
+    idx_start = idx_end = saved_idx = 0;
+
+    for ( level = n_levels; level > 0; level-- )
+    {
+        n_pages = idx_end - idx_start + 1;
+        ptes = xc_map_foreign_pages(xch, dom, PROT_READ, mfns, n_pages);
+        if ( !ptes )
+        {
+            PERROR("Failed to map %u page table pages for p2m list", n_pages);
+            goto out;
+        }
+        free(mfns);
+
+        shift = level * 9 + 3;
+        idx_start = ((p2m_vaddr - off) & mask) >> shift;
+        idx_end = ((p2m_end - off) & mask) >> shift;
+        idx = idx_end - idx_start + 1;
+        mfns = malloc(sizeof(*mfns) * idx);
+        if ( !mfns )
+        {
+            ERROR("Cannot allocate memory for array of %u mfns", idx);
+            goto out;
+        }
+
+        for ( idx = idx_start; idx <= idx_end; idx++ )
+        {
+            mfn = (ptes[idx] & 0x000ffffffffff000ULL) >> PAGE_SHIFT;
+            if ( mfn == 0 )
+            {
+                ERROR("Bad mfn %#lx during page table walk for vaddr %#" PRIx64 " at level %d of p2m list",
+                      mfn, off + ((uint64_t)idx << shift), level);
+                errno = ERANGE;
+                goto out;
+            }
+            mfns[idx - idx_start] = mfn;
+
+            /* Maximum pfn check at level 2. Same reasoning as for p2m tree. */
+            if ( level == 2 )
+            {
+                if ( mfn != saved_mfn )
+                {
+                    saved_mfn = mfn;
+                    saved_idx = idx - idx_start;
+                }
+            }
+        }
+
+        if ( level == 2 )
+        {
+            if ( saved_idx == idx_end )
+                saved_idx++;
+            max_pfn = ((xen_pfn_t)saved_idx << 9) * fpp;
+            if ( max_pfn < dinfo->p2m_size )
+            {
+                dinfo->p2m_size = max_pfn;
+                dinfo->p2m_frames = (dinfo->p2m_size + fpp - 1) / fpp;
+                p2m_end = p2m_vaddr + dinfo->p2m_frames * PAGE_SIZE - 1;
+                idx_end = idx_start + saved_idx;
+            }
+        }
+
+        munmap(ptes, n_pages * PAGE_SIZE);
+        ptes = NULL;
+        off = p2m_vaddr & ((mask >> shift) << shift);
+    }
+
+    return mfns;
+
+ out:
+    free(mfns);
+    if ( ptes )
+        munmap(ptes, n_pages * PAGE_SIZE);
+
+    return NULL;
+}
+
+static xen_pfn_t *
+xc_core_arch_map_p2m_tree_rw(xc_interface *xch, struct domain_info_context *dinfo,
+                             uint32_t dom, shared_info_any_t *live_shinfo)
+{
+    /* Double and single indirect references to the live P2M table */
+    xen_pfn_t *live_p2m_frame_list_list;
+    xen_pfn_t *live_p2m_frame_list = NULL;
+    /* Copies of the above. */
+    xen_pfn_t *p2m_frame_list_list = NULL;
+    xen_pfn_t *p2m_frame_list;
+
+    int err;
+    int i;
 
     live_p2m_frame_list_list =
         xc_map_foreign_range(xch, dom, PAGE_SIZE, PROT_READ,
@@ -151,10 +287,60 @@ xc_core_arch_map_p2m_rw(xc_interface *xch, struct domain_info_context *dinfo, xc
         for ( i = P2M_FL_ENTRIES - 1; i >= 0; i-- )
             p2m_frame_list[i] = ((uint32_t *)p2m_frame_list)[i];
 
+    dinfo->p2m_frames = P2M_FL_ENTRIES;
+
+    return p2m_frame_list;
+
+ out:
+    err = errno;
+
+    if ( live_p2m_frame_list_list )
+        munmap(live_p2m_frame_list_list, PAGE_SIZE);
+
+    if ( live_p2m_frame_list )
+        munmap(live_p2m_frame_list, P2M_FLL_ENTRIES * PAGE_SIZE);
+
+    free(p2m_frame_list_list);
+
+    errno = err;
+
+    return NULL;
+}
+
+static int
+xc_core_arch_map_p2m_rw(xc_interface *xch, struct domain_info_context *dinfo, xc_dominfo_t *info,
+                        shared_info_any_t *live_shinfo, xen_pfn_t **live_p2m, int rw)
+{
+    xen_pfn_t *p2m_frame_list = NULL;
+    uint64_t p2m_cr3;
+    uint32_t dom = info->domid;
+    int ret = -1;
+    int err;
+
+    if ( xc_domain_nr_gpfns(xch, info->domid, &dinfo->p2m_size) < 0 )
+    {
+        ERROR("Could not get maximum GPFN!");
+        goto out;
+    }
+
+    if ( dinfo->p2m_size < info->nr_pages  )
+    {
+        ERROR("p2m_size < nr_pages -1 (%lx < %lx", dinfo->p2m_size, info->nr_pages - 1);
+        goto out;
+    }
+
+    p2m_cr3 = GET_FIELD(live_shinfo, arch.p2m_cr3, dinfo->guest_width);
+
+    p2m_frame_list = p2m_cr3 ? xc_core_arch_map_p2m_list_rw(xch, dinfo, dom, live_shinfo, p2m_cr3)
+                             : xc_core_arch_map_p2m_tree_rw(xch, dinfo, dom, live_shinfo);
+
+    if ( !p2m_frame_list )
+        goto out;
+
     *live_p2m = xc_map_foreign_pages(xch, dom,
                                     rw ? (PROT_READ | PROT_WRITE) : PROT_READ,
                                     p2m_frame_list,
-                                    P2M_FL_ENTRIES);
+                                    dinfo->p2m_frames);
 
     if ( !*live_p2m )
     {
@@ -162,21 +348,11 @@ xc_core_arch_map_p2m_rw(xc_interface *xch, struct domain_info_context *dinfo, xc
         goto out;
     }
 
-    *pfnp = dinfo->p2m_size;
-
     ret = 0;
 
 out:
     err = errno;
 
-    if ( live_p2m_frame_list_list )
-        munmap(live_p2m_frame_list_list, PAGE_SIZE);
-
-    if ( live_p2m_frame_list )
-        munmap(live_p2m_frame_list, P2M_FLL_ENTRIES * PAGE_SIZE);
-
-    free(p2m_frame_list_list);
-
     free(p2m_frame_list);
 
     errno = err;
@@ -184,25 +360,17 @@ out:
 }
 
 int
-xc_core_arch_map_p2m(xc_interface *xch, unsigned int guest_width, xc_dominfo_t *info,
-                        shared_info_any_t *live_shinfo, xen_pfn_t **live_p2m,
-                        unsigned long *pfnp)
+xc_core_arch_map_p2m(xc_interface *xch, struct domain_info_context *dinfo, xc_dominfo_t *info,
+                        shared_info_any_t *live_shinfo, xen_pfn_t **live_p2m)
 {
-    struct domain_info_context _dinfo = { .guest_width = guest_width };
-    struct domain_info_context *dinfo = &_dinfo;
-    return xc_core_arch_map_p2m_rw(xch, dinfo, info,
-                                   live_shinfo, live_p2m, pfnp, 0);
+    return xc_core_arch_map_p2m_rw(xch, dinfo, info, live_shinfo, live_p2m, 0);
 }
 
 int
-xc_core_arch_map_p2m_writable(xc_interface *xch, unsigned int guest_width, xc_dominfo_t *info,
-                              shared_info_any_t *live_shinfo, xen_pfn_t **live_p2m,
-                              unsigned long *pfnp)
+xc_core_arch_map_p2m_writable(xc_interface *xch, struct domain_info_context *dinfo, xc_dominfo_t *info,
+                              shared_info_any_t *live_shinfo, xen_pfn_t **live_p2m)
 {
-    struct domain_info_context _dinfo = { .guest_width = guest_width };
-    struct domain_info_context *dinfo = &_dinfo;
-    return xc_core_arch_map_p2m_rw(xch, dinfo, info,
-                                   live_shinfo, live_p2m, pfnp, 1);
+    return xc_core_arch_map_p2m_rw(xch, dinfo, info, live_shinfo, live_p2m, 1);
 }
 
 int
diff --git a/tools/libs/ctrl/xc_private.h b/tools/libs/ctrl/xc_private.h
index f0b5f83ac8..8ebc0b59da 100644
--- a/tools/libs/ctrl/xc_private.h
+++ b/tools/libs/ctrl/xc_private.h
@@ -79,6 +79,7 @@ struct iovec {
 
 struct domain_info_context {
     unsigned int guest_width;
+    unsigned int p2m_frames;
     unsigned long p2m_size;
 };
 
diff --git a/tools/libs/guest/xg_domain.c b/tools/libs/guest/xg_domain.c
index 5019c84e0e..dd7db2cbd8 100644
--- a/tools/libs/guest/xg_domain.c
+++ b/tools/libs/guest/xg_domain.c
@@ -24,13 +24,9 @@
 
 int xc_unmap_domain_meminfo(xc_interface *xch, struct xc_domain_meminfo *minfo)
 {
-    struct domain_info_context _di = { .guest_width = minfo->guest_width,
-                                       .p2m_size = minfo->p2m_size};
-    struct domain_info_context *dinfo = &_di;
-
     free(minfo->pfn_type);
     if ( minfo->p2m_table )
-        munmap(minfo->p2m_table, P2M_FL_ENTRIES * PAGE_SIZE);
+        munmap(minfo->p2m_table, minfo->p2m_frames * PAGE_SIZE);
     minfo->p2m_table = NULL;
 
     return 0;
@@ -40,7 +36,6 @@ int xc_map_domain_meminfo(xc_interface *xch, uint32_t domid,
                           struct xc_domain_meminfo *minfo)
 {
     struct domain_info_context _di;
-    struct domain_info_context *dinfo = &_di;
 
     xc_dominfo_t info;
     shared_info_any_t *live_shinfo;
@@ -96,16 +91,16 @@ int xc_map_domain_meminfo(xc_interface *xch, uint32_t domid,
         return -1;
     }
 
-    if ( xc_core_arch_map_p2m_writable(xch, minfo->guest_width, &info,
-                                       live_shinfo, &minfo->p2m_table,
-                                       &minfo->p2m_size) )
+    if ( xc_core_arch_map_p2m_writable(xch, &_di, &info,
+                                       live_shinfo, &minfo->p2m_table) )
     {
         PERROR("Could not map the P2M table");
         munmap(live_shinfo, PAGE_SIZE);
         return -1;
     }
     munmap(live_shinfo, PAGE_SIZE);
-    _di.p2m_size = minfo->p2m_size;
+    minfo->p2m_size = _di.p2m_size;
+    minfo->p2m_frames = _di.p2m_frames;
 
     /* Make space and prepare for getting the PFN types */
     minfo->pfn_type = calloc(sizeof(*minfo->pfn_type), minfo->p2m_size);
@@ -141,7 +136,7 @@ failed:
     }
     if ( minfo->p2m_table )
     {
-        munmap(minfo->p2m_table, P2M_FL_ENTRIES * PAGE_SIZE);
+        munmap(minfo->p2m_table, minfo->p2m_frames * PAGE_SIZE);
         minfo->p2m_table = NULL;
     }
 
-- 
2.26.2

Re: [PATCH v2 2/6] tools/libs/ctrl: fix xc_core_arch_map_p2m() to support linear p2m table

[Wei Liu]

On Mon, Apr 12, 2021 at 05:22:32PM +0200, Juergen Gross wrote:
> The core of a pv linux guest produced via "xl dump-core" is not usable
> as since kernel 4.14 only the linear p2m table is kept if Xen indicates
> it is supporting that. Unfortunately xc_core_arch_map_p2m() is still
> supporting the 3-level p2m tree only.
> 
> Fix that by copying the functionality of map_p2m() from libxenguest to
> libxenctrl.
> 

So there are now two copies of the same logic? Is it possible to reduce
it to only one?

Wei.

Re: [PATCH v2 2/6] tools/libs/ctrl: fix xc_core_arch_map_p2m() to support linear p2m table

[Juergen Gross]

[-- Attachment #1.1.1: Type: text/plain, Size: 692 bytes --]

On 21.04.21 12:13, Wei Liu wrote:
> On Mon, Apr 12, 2021 at 05:22:32PM +0200, Juergen Gross wrote:
>> The core of a pv linux guest produced via "xl dump-core" is not usable
>> as since kernel 4.14 only the linear p2m table is kept if Xen indicates
>> it is supporting that. Unfortunately xc_core_arch_map_p2m() is still
>> supporting the 3-level p2m tree only.
>>
>> Fix that by copying the functionality of map_p2m() from libxenguest to
>> libxenctrl.
>>
> 
> So there are now two copies of the same logic? Is it possible to reduce
> it to only one?

Yes. See the intro mail of the series.

I wanted to fix the issue first, before doing the major cleanup work.


Juergen

[-- Attachment #1.1.2: OpenPGP_0xB0DE9DD628BF132F.asc --]
[-- Type: application/pgp-keys, Size: 3135 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 495 bytes --]

Re: [PATCH v2 2/6] tools/libs/ctrl: fix xc_core_arch_map_p2m() to support linear p2m table

[Wei Liu]

On Wed, Apr 21, 2021 at 12:17:49PM +0200, Juergen Gross wrote:
> On 21.04.21 12:13, Wei Liu wrote:
> > On Mon, Apr 12, 2021 at 05:22:32PM +0200, Juergen Gross wrote:
> > > The core of a pv linux guest produced via "xl dump-core" is not usable
> > > as since kernel 4.14 only the linear p2m table is kept if Xen indicates
> > > it is supporting that. Unfortunately xc_core_arch_map_p2m() is still
> > > supporting the 3-level p2m tree only.
> > > 
> > > Fix that by copying the functionality of map_p2m() from libxenguest to
> > > libxenctrl.
> > > 
> > 
> > So there are now two copies of the same logic? Is it possible to reduce
> > it to only one?
> 
> Yes. See the intro mail of the series.
> 
> I wanted to fix the issue first, before doing the major cleanup work.

Acked-by: Wei Liu <wl@xen.org>

[PATCH v2 3/6] tools/libs/ctrl: use common p2m mapping code in xc_domain_resume_any()

[Juergen Gross]

Instead of open coding the mapping of the p2m list use the already
existing xc_core_arch_map_p2m() call, especially as the current code
does not support guests with the linear p2m map. It should be noted
that this code is needed for colo/remus only.

Switching to xc_core_arch_map_p2m() drops the need to bail out for
bitness of tool stack and guest differing.

Signed-off-by: Juergen Gross <jgross@suse.com>
---
This might be a backport candidate
V2:
- add missing #include in ocaml stub (Andrew Cooper)
---
 tools/libs/ctrl/xc_resume.c         | 66 +++++++++--------------------
 tools/ocaml/libs/xc/xenctrl_stubs.c |  1 +
 2 files changed, 22 insertions(+), 45 deletions(-)

diff --git a/tools/libs/ctrl/xc_resume.c b/tools/libs/ctrl/xc_resume.c
index 94c6c9fb31..e3c8e83aa9 100644
--- a/tools/libs/ctrl/xc_resume.c
+++ b/tools/libs/ctrl/xc_resume.c
@@ -20,6 +20,7 @@
 #include <xen/foreign/x86_32.h>
 #include <xen/foreign/x86_64.h>
 #include <xen/hvm/params.h>
+#include "xc_core.h"
 
 static int modify_returncode(xc_interface *xch, uint32_t domid)
 {
@@ -137,12 +138,10 @@ static int xc_domain_resume_any(xc_interface *xch, uint32_t domid)
     struct domain_info_context _dinfo = { .guest_width = 0,
                                           .p2m_size = 0 };
     struct domain_info_context *dinfo = &_dinfo;
-    unsigned long mfn;
+    xen_pfn_t mfn, store_mfn, console_mfn;
     vcpu_guest_context_any_t ctxt;
-    start_info_t *start_info;
-    shared_info_t *shinfo = NULL;
-    xen_pfn_t *p2m_frame_list_list = NULL;
-    xen_pfn_t *p2m_frame_list = NULL;
+    start_info_any_t *start_info;
+    shared_info_any_t *shinfo = NULL;
     xen_pfn_t *p2m = NULL;
 #endif
 
@@ -164,11 +163,6 @@ static int xc_domain_resume_any(xc_interface *xch, uint32_t domid)
         PERROR("Could not get domain width");
         return rc;
     }
-    if ( dinfo->guest_width != sizeof(long) )
-    {
-        ERROR("Cannot resume uncooperative cross-address-size guests");
-        return rc;
-    }
 
     /* Map the shared info frame */
     shinfo = xc_map_foreign_range(xch, domid, PAGE_SIZE,
@@ -179,34 +173,8 @@ static int xc_domain_resume_any(xc_interface *xch, uint32_t domid)
         goto out;
     }
 
-    dinfo->p2m_size = shinfo->arch.max_pfn;
-
-    p2m_frame_list_list =
-        xc_map_foreign_range(xch, domid, PAGE_SIZE, PROT_READ,
-                             shinfo->arch.pfn_to_mfn_frame_list_list);
-    if ( p2m_frame_list_list == NULL )
-    {
-        ERROR("Couldn't map p2m_frame_list_list");
-        goto out;
-    }
-
-    p2m_frame_list = xc_map_foreign_pages(xch, domid, PROT_READ,
-                                          p2m_frame_list_list,
-                                          P2M_FLL_ENTRIES);
-    if ( p2m_frame_list == NULL )
-    {
-        ERROR("Couldn't map p2m_frame_list");
-        goto out;
-    }
-
-    /* Map all the frames of the pfn->mfn table. For migrate to succeed,
-       the guest must not change which frames are used for this purpose.
-       (its not clear why it would want to change them, and we'll be OK
-       from a safety POV anyhow. */
-    p2m = xc_map_foreign_pages(xch, domid, PROT_READ,
-                               p2m_frame_list,
-                               P2M_FL_ENTRIES);
-    if ( p2m == NULL )
+    /* Map the p2m list */
+    if ( xc_core_arch_map_p2m(xch, dinfo, &info, shinfo, &p2m) )
     {
         ERROR("Couldn't map p2m table");
         goto out;
@@ -228,8 +196,20 @@ static int xc_domain_resume_any(xc_interface *xch, uint32_t domid)
         goto out;
     }
 
-    start_info->store_mfn        = p2m[start_info->store_mfn];
-    start_info->console.domU.mfn = p2m[start_info->console.domU.mfn];
+    store_mfn = GET_FIELD(start_info, store_mfn, dinfo->guest_width);
+    console_mfn = GET_FIELD(start_info, console.domU.mfn, dinfo->guest_width);
+    if ( dinfo->guest_width == 4 )
+    {
+        store_mfn = ((uint32_t *)p2m)[store_mfn];
+        console_mfn = ((uint32_t *)p2m)[console_mfn];
+    }
+    else
+    {
+        store_mfn = ((uint64_t *)p2m)[store_mfn];
+        console_mfn = ((uint64_t *)p2m)[console_mfn];
+    }
+    SET_FIELD(start_info, store_mfn, store_mfn, dinfo->guest_width);
+    SET_FIELD(start_info, console.domU.mfn, console_mfn, dinfo->guest_width);
 
     munmap(start_info, PAGE_SIZE);
 #endif /* defined(__i386__) || defined(__x86_64__) */
@@ -250,11 +230,7 @@ static int xc_domain_resume_any(xc_interface *xch, uint32_t domid)
 out:
 #if defined(__i386__) || defined(__x86_64__)
     if (p2m)
-        munmap(p2m, P2M_FL_ENTRIES*PAGE_SIZE);
-    if (p2m_frame_list)
-        munmap(p2m_frame_list, P2M_FLL_ENTRIES*PAGE_SIZE);
-    if (p2m_frame_list_list)
-        munmap(p2m_frame_list_list, PAGE_SIZE);
+        munmap(p2m, dinfo->p2m_frames * PAGE_SIZE);
     if (shinfo)
         munmap(shinfo, PAGE_SIZE);
 #endif
diff --git a/tools/ocaml/libs/xc/xenctrl_stubs.c b/tools/ocaml/libs/xc/xenctrl_stubs.c
index d05d7bb30e..6e4bc567f5 100644
--- a/tools/ocaml/libs/xc/xenctrl_stubs.c
+++ b/tools/ocaml/libs/xc/xenctrl_stubs.c
@@ -32,6 +32,7 @@
 
 #define XC_WANT_COMPAT_MAP_FOREIGN_API
 #include <xenctrl.h>
+#include <xenguest.h>
 #include <xen-tools/libs.h>
 
 #include "mmap_stubs.h"
-- 
2.26.2

Re: [PATCH v2 3/6] tools/libs/ctrl: use common p2m mapping code in xc_domain_resume_any()

[Christian Lindig]

[-- Attachment #1: Type: text/plain, Size: 907 bytes --]



On 12 Apr 2021, at 16:22, Juergen Gross <jgross@suse.com<mailto:jgross@suse.com>> wrote:

Instead of open coding the mapping of the p2m list use the already
existing xc_core_arch_map_p2m() call, especially as the current code
does not support guests with the linear p2m map. It should be noted
that this code is needed for colo/remus only.
[..]
diff --git a/tools/ocaml/libs/xc/xenctrl_stubs.c b/tools/ocaml/libs/xc/xenctrl_stubs.c
index d05d7bb30e..6e4bc567f5 100644
--- a/tools/ocaml/libs/xc/xenctrl_stubs.c
+++ b/tools/ocaml/libs/xc/xenctrl_stubs.c
@@ -32,6 +32,7 @@

#define XC_WANT_COMPAT_MAP_FOREIGN_API
#include <xenctrl.h>
+#include <xenguest.h>
#include <xen-tools/libs.h>

#include "mmap_stubs.h"
—
2.26.2

Do we need this when there are no other changes in the file?

Acked-by: Christian Lindig <christian.lindig@citrix.com<mailto:christian.lindig@citrix.com>>



[-- Attachment #2: Type: text/html, Size: 1735 bytes --]

Re: [PATCH v2 3/6] tools/libs/ctrl: use common p2m mapping code in xc_domain_resume_any()

[Juergen Gross]

[-- Attachment #1.1.1: Type: text/plain, Size: 1124 bytes --]

On 13.04.21 17:00, Christian Lindig wrote:
> 
> 
>> On 12 Apr 2021, at 16:22, Juergen Gross <jgross@suse.com 
>> <mailto:jgross@suse.com>> wrote:
>>
>> Instead of open coding the mapping of the p2m list use the already
>> existing xc_core_arch_map_p2m() call, especially as the current code
>> does not support guests with the linear p2m map. It should be noted
>> that this code is needed for colo/remus only.
>> [..]
>> diff --git a/tools/ocaml/libs/xc/xenctrl_stubs.c 
>> b/tools/ocaml/libs/xc/xenctrl_stubs.c
>> index d05d7bb30e..6e4bc567f5 100644
>> --- a/tools/ocaml/libs/xc/xenctrl_stubs.c
>> +++ b/tools/ocaml/libs/xc/xenctrl_stubs.c
>> @@ -32,6 +32,7 @@
>>
>> #define XC_WANT_COMPAT_MAP_FOREIGN_API
>> #include <xenctrl.h>
>> +#include <xenguest.h>
>> #include <xen-tools/libs.h>
>>
>> #include "mmap_stubs.h"
>> —
>> 2.26.2
> 
> Do we need this when there are no other changes in the file?

Yes. Some definitions moved from xenctrl.h into xenguest.h.

> 
> Acked-by: Christian Lindig <christian.lindig@citrix.com 
> <mailto:christian.lindig@citrix.com>>

Thanks,

Juergen


[-- Attachment #1.1.2: OpenPGP_0xB0DE9DD628BF132F.asc --]
[-- Type: application/pgp-keys, Size: 3135 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 495 bytes --]

Re: [PATCH v2 3/6] tools/libs/ctrl: use common p2m mapping code in xc_domain_resume_any()

[Wei Liu]

On Mon, Apr 12, 2021 at 05:22:33PM +0200, Juergen Gross wrote:
> Instead of open coding the mapping of the p2m list use the already
> existing xc_core_arch_map_p2m() call, especially as the current code
> does not support guests with the linear p2m map. It should be noted
> that this code is needed for colo/remus only.
> 
> Switching to xc_core_arch_map_p2m() drops the need to bail out for
> bitness of tool stack and guest differing.
> 
> Signed-off-by: Juergen Gross <jgross@suse.com>

Acked-by: Wei Liu <wl@xen.org>

[PATCH v2 4/6] tools/libs: move xc_resume.c to libxenguest

[Juergen Gross]

The guest suspend functionality is already part of libxenguest. Move
the resume functionality from libxenctrl to libxenguest, too.

Signed-off-by: Juergen Gross <jgross@suse.com>
---
 tools/include/xenctrl.h                       | 63 -------------------
 tools/include/xenguest.h                      | 62 ++++++++++++++++++
 tools/libs/ctrl/Makefile                      |  1 -
 tools/libs/guest/Makefile                     |  1 +
 .../{ctrl/xc_resume.c => guest/xg_resume.c}   |  1 +
 5 files changed, 64 insertions(+), 64 deletions(-)
 rename tools/libs/{ctrl/xc_resume.c => guest/xg_resume.c} (99%)

diff --git a/tools/include/xenctrl.h b/tools/include/xenctrl.h
index 318920166c..8a6bede5b0 100644
--- a/tools/include/xenctrl.h
+++ b/tools/include/xenctrl.h
@@ -576,69 +576,6 @@ int xc_domain_destroy(xc_interface *xch,
                       uint32_t domid);
 
 
-/**
- * This function resumes a suspended domain. The domain should have
- * been previously suspended.
- *
- * Note that there are 'xc_domain_suspend' as suspending a domain
- * is quite the endeavour.
- *
- * For the purpose of this explanation there are three guests:
- * PV (using hypercalls for privilgied operations), HVM
- * (fully hardware virtualized guests using emulated devices for everything),
- * and PVHVM (PV aware with hardware virtualisation).
- *
- * HVM guest are the simplest - they suspend via S3 / S4 and resume from
- * S3 / S4. Upon resume they have to re-negotiate with the emulated devices.
- *
- * PV and PVHVM communicate via hypercalls for suspend (and resume).
- * For suspend the toolstack initiates the process by writing an value
- * in XenBus "control/shutdown" with the string "suspend".
- *
- * The PV guest stashes anything it deems neccessary in 'struct
- * start_info' in case of failure (PVHVM may ignore this) and calls
- * the SCHEDOP_shutdown::SHUTDOWN_suspend hypercall (for PV as
- * argument it passes the MFN to 'struct start_info').
- *
- * And then the guest is suspended.
- *
- * The checkpointing or notifying a guest that the suspend failed or
- * cancelled (in case of checkpoint) is by having the
- * SCHEDOP_shutdown::SHUTDOWN_suspend hypercall return a non-zero
- * value.
- *
- * The PV and PVHVM resume path are similar. For PV it would be
- * similar to bootup - figure out where the 'struct start_info' is (or
- * if the suspend was cancelled aka checkpointed - reuse the saved
- * values).
- *
- * From here on they differ depending whether the guest is PV or PVHVM
- * in specifics but follow overall the same path:
- *  - PV: Bringing up the vCPUS,
- *  - PVHVM: Setup vector callback,
- *  - Bring up vCPU runstates,
- *  - Remap the grant tables if checkpointing or setup from scratch,
- *
- *
- * If the resume was not checkpointing (or if suspend was succesful) we would
- * setup the PV timers and the different PV events. Lastly the PV drivers
- * re-negotiate with the backend.
- *
- * This function would return before the guest started resuming. That is
- * the guest would be in non-running state and its vCPU context would be
- * in the the SCHEDOP_shutdown::SHUTDOWN_suspend hypercall return path
- * (for PV and PVHVM). For HVM it would be in would be in QEMU emulated
- * BIOS handling S3 suspend.
- *
- * @parm xch a handle to an open hypervisor interface
- * @parm domid the domain id to resume
- * @parm fast use cooperative resume (guest must support this)
- * return 0 on success, -1 on failure
- */
-int xc_domain_resume(xc_interface *xch,
-		     uint32_t domid,
-		     int fast);
-
 /**
  * This function will shutdown a domain. This is intended for use in
  * fully-virtualized domains where this operation is analogous to the
diff --git a/tools/include/xenguest.h b/tools/include/xenguest.h
index 781d40c119..44300a8658 100644
--- a/tools/include/xenguest.h
+++ b/tools/include/xenguest.h
@@ -689,6 +689,68 @@ int xc_query_page_offline_status(xc_interface *xch, unsigned long start,
 
 int xc_exchange_page(xc_interface *xch, uint32_t domid, xen_pfn_t mfn);
 
+/**
+ * This function resumes a suspended domain. The domain should have
+ * been previously suspended.
+ *
+ * Note that there are 'xc_domain_suspend' as suspending a domain
+ * is quite the endeavour.
+ *
+ * For the purpose of this explanation there are three guests:
+ * PV (using hypercalls for privilgied operations), HVM
+ * (fully hardware virtualized guests using emulated devices for everything),
+ * and PVHVM (PV aware with hardware virtualisation).
+ *
+ * HVM guest are the simplest - they suspend via S3 / S4 and resume from
+ * S3 / S4. Upon resume they have to re-negotiate with the emulated devices.
+ *
+ * PV and PVHVM communicate via hypercalls for suspend (and resume).
+ * For suspend the toolstack initiates the process by writing an value
+ * in XenBus "control/shutdown" with the string "suspend".
+ *
+ * The PV guest stashes anything it deems neccessary in 'struct
+ * start_info' in case of failure (PVHVM may ignore this) and calls
+ * the SCHEDOP_shutdown::SHUTDOWN_suspend hypercall (for PV as
+ * argument it passes the MFN to 'struct start_info').
+ *
+ * And then the guest is suspended.
+ *
+ * The checkpointing or notifying a guest that the suspend failed or
+ * cancelled (in case of checkpoint) is by having the
+ * SCHEDOP_shutdown::SHUTDOWN_suspend hypercall return a non-zero
+ * value.
+ *
+ * The PV and PVHVM resume path are similar. For PV it would be
+ * similar to bootup - figure out where the 'struct start_info' is (or
+ * if the suspend was cancelled aka checkpointed - reuse the saved
+ * values).
+ *
+ * From here on they differ depending whether the guest is PV or PVHVM
+ * in specifics but follow overall the same path:
+ *  - PV: Bringing up the vCPUS,
+ *  - PVHVM: Setup vector callback,
+ *  - Bring up vCPU runstates,
+ *  - Remap the grant tables if checkpointing or setup from scratch,
+ *
+ *
+ * If the resume was not checkpointing (or if suspend was succesful) we would
+ * setup the PV timers and the different PV events. Lastly the PV drivers
+ * re-negotiate with the backend.
+ *
+ * This function would return before the guest started resuming. That is
+ * the guest would be in non-running state and its vCPU context would be
+ * in the the SCHEDOP_shutdown::SHUTDOWN_suspend hypercall return path
+ * (for PV and PVHVM). For HVM it would be in would be in QEMU emulated
+ * BIOS handling S3 suspend.
+ *
+ * @parm xch a handle to an open hypervisor interface
+ * @parm domid the domain id to resume
+ * @parm fast use cooperative resume (guest must support this)
+ * return 0 on success, -1 on failure
+ */
+int xc_domain_resume(xc_interface *xch,
+                     uint32_t domid,
+                     int fast);
 
 /**
  * Memory related information, such as PFN types, the P2M table,
diff --git a/tools/libs/ctrl/Makefile b/tools/libs/ctrl/Makefile
index ce9ecae710..fbeb3a3537 100644
--- a/tools/libs/ctrl/Makefile
+++ b/tools/libs/ctrl/Makefile
@@ -20,7 +20,6 @@ SRCS-y       += xc_rt.c
 SRCS-y       += xc_tbuf.c
 SRCS-y       += xc_pm.c
 SRCS-y       += xc_cpu_hotplug.c
-SRCS-y       += xc_resume.c
 SRCS-y       += xc_vm_event.c
 SRCS-y       += xc_vmtrace.c
 SRCS-y       += xc_monitor.c
diff --git a/tools/libs/guest/Makefile b/tools/libs/guest/Makefile
index 604e1695d6..8309bbcd20 100644
--- a/tools/libs/guest/Makefile
+++ b/tools/libs/guest/Makefile
@@ -9,6 +9,7 @@ endif
 SRCS-y += xg_private.c
 SRCS-y += xg_domain.c
 SRCS-y += xg_suspend.c
+SRCS-y += xg_resume.c
 ifeq ($(CONFIG_MIGRATE),y)
 SRCS-y += xg_sr_common.c
 SRCS-$(CONFIG_X86) += xg_sr_common_x86.c
diff --git a/tools/libs/ctrl/xc_resume.c b/tools/libs/guest/xg_resume.c
similarity index 99%
rename from tools/libs/ctrl/xc_resume.c
rename to tools/libs/guest/xg_resume.c
index e3c8e83aa9..3bdefb2eef 100644
--- a/tools/libs/ctrl/xc_resume.c
+++ b/tools/libs/guest/xg_resume.c
@@ -14,6 +14,7 @@
  */
 
 #include "xc_private.h"
+#include "xenguest.h"
 
 #if defined(__i386__) || defined(__x86_64__)
 
-- 
2.26.2

Re: [PATCH v2 4/6] tools/libs: move xc_resume.c to libxenguest

[Wei Liu]

On Mon, Apr 12, 2021 at 05:22:34PM +0200, Juergen Gross wrote:
> The guest suspend functionality is already part of libxenguest. Move
> the resume functionality from libxenctrl to libxenguest, too.
> 
> Signed-off-by: Juergen Gross <jgross@suse.com>

Acked-by: Wei Liu <wl@xen.org>

[PATCH v2 5/6] tools/libs: move xc_core* from libxenctrl to libxenguest

[Juergen Gross]

The functionality in xc_core* should be part of libxenguest instead
of libxenctrl. Users are already either in libxenguest, or in xl.
There is one single exception: xc_core_arch_auto_translated_physmap()
is being used by xc_domain_memory_mapping(), which is used by qemu.
So leave the xc_core_arch_auto_translated_physmap() functionality in
libxenctrl.

This will make it easier to merge common functionality of xc_core*
and xg_sr_save*.

Signed-off-by: Juergen Gross <jgross@suse.com>
---
 tools/libs/ctrl/Makefile                             |  3 ---
 tools/libs/ctrl/xc_domain.c                          |  2 --
 tools/libs/ctrl/xc_private.h                         | 12 ++++++++++++
 tools/libs/guest/Makefile                            |  3 +++
 tools/libs/{ctrl/xc_core.c => guest/xg_core.c}       |  2 +-
 tools/libs/{ctrl/xc_core.h => guest/xg_core.h}       |  5 ++---
 .../libs/{ctrl/xc_core_arm.c => guest/xg_core_arm.c} |  8 +-------
 .../libs/{ctrl/xc_core_arm.h => guest/xg_core_arm.h} |  0
 .../libs/{ctrl/xc_core_x86.c => guest/xg_core_x86.c} |  8 +-------
 .../libs/{ctrl/xc_core_x86.h => guest/xg_core_x86.h} |  0
 tools/libs/guest/xg_dom_boot.c                       |  2 +-
 tools/libs/guest/xg_domain.c                         |  2 +-
 tools/libs/guest/xg_offline_page.c                   |  2 +-
 tools/libs/guest/xg_resume.c                         |  2 +-
 14 files changed, 24 insertions(+), 27 deletions(-)
 rename tools/libs/{ctrl/xc_core.c => guest/xg_core.c} (99%)
 rename tools/libs/{ctrl/xc_core.h => guest/xg_core.h} (97%)
 rename tools/libs/{ctrl/xc_core_arm.c => guest/xg_core_arm.c} (96%)
 rename tools/libs/{ctrl/xc_core_arm.h => guest/xg_core_arm.h} (100%)
 rename tools/libs/{ctrl/xc_core_x86.c => guest/xg_core_x86.c} (99%)
 rename tools/libs/{ctrl/xc_core_x86.h => guest/xg_core_x86.h} (100%)

diff --git a/tools/libs/ctrl/Makefile b/tools/libs/ctrl/Makefile
index fbeb3a3537..519246b0d6 100644
--- a/tools/libs/ctrl/Makefile
+++ b/tools/libs/ctrl/Makefile
@@ -2,9 +2,6 @@ XEN_ROOT = $(CURDIR)/../../..
 include $(XEN_ROOT)/tools/Rules.mk
 
 SRCS-y       += xc_altp2m.c
-SRCS-y       += xc_core.c
-SRCS-$(CONFIG_X86) += xc_core_x86.c
-SRCS-$(CONFIG_ARM) += xc_core_arm.c
 SRCS-y       += xc_cpupool.c
 SRCS-y       += xc_domain.c
 SRCS-y       += xc_evtchn.c
diff --git a/tools/libs/ctrl/xc_domain.c b/tools/libs/ctrl/xc_domain.c
index e7cea4a17d..7d118848f1 100644
--- a/tools/libs/ctrl/xc_domain.c
+++ b/tools/libs/ctrl/xc_domain.c
@@ -19,8 +19,6 @@
  * Copyright (c) 2003, K A Fraser.
  */
 
-#include "xc_private.h"
-#include "xc_core.h"
 #include "xc_private.h"
 #include <xen/memory.h>
 #include <xen/hvm/hvm_op.h>
diff --git a/tools/libs/ctrl/xc_private.h b/tools/libs/ctrl/xc_private.h
index 8ebc0b59da..dff0f0289b 100644
--- a/tools/libs/ctrl/xc_private.h
+++ b/tools/libs/ctrl/xc_private.h
@@ -467,6 +467,18 @@ void *xc_vm_event_enable(xc_interface *xch, uint32_t domain_id, int param,
 
 int do_dm_op(xc_interface *xch, uint32_t domid, unsigned int nr_bufs, ...);
 
+#if defined (__i386__) || defined (__x86_64__)
+static inline int xc_core_arch_auto_translated_physmap(const xc_dominfo_t *info)
+{
+    return info->hvm;
+}
+#elif defined (__arm__) || defined(__aarch64__)
+static inline int xc_core_arch_auto_translated_physmap(const xc_dominfo_t *info)
+{
+    return 1;
+}
+#endif
+
 #endif /* __XC_PRIVATE_H__ */
 
 /*
diff --git a/tools/libs/guest/Makefile b/tools/libs/guest/Makefile
index 8309bbcd20..85a6f45592 100644
--- a/tools/libs/guest/Makefile
+++ b/tools/libs/guest/Makefile
@@ -24,6 +24,9 @@ SRCS-y += xg_offline_page.c
 else
 SRCS-y += xg_nomigrate.c
 endif
+SRCS-y       += xg_core.c
+SRCS-$(CONFIG_X86) += xg_core_x86.c
+SRCS-$(CONFIG_ARM) += xg_core_arm.c
 
 CFLAGS += -I$(XEN_libxenctrl)
 
diff --git a/tools/libs/ctrl/xc_core.c b/tools/libs/guest/xg_core.c
similarity index 99%
rename from tools/libs/ctrl/xc_core.c
rename to tools/libs/guest/xg_core.c
index 9576bec5a3..c52f1161c1 100644
--- a/tools/libs/ctrl/xc_core.c
+++ b/tools/libs/guest/xg_core.c
@@ -61,7 +61,7 @@
  */
 
 #include "xc_private.h"
-#include "xc_core.h"
+#include "xg_core.h"
 #include <stdlib.h>
 #include <unistd.h>
 
diff --git a/tools/libs/ctrl/xc_core.h b/tools/libs/guest/xg_core.h
similarity index 97%
rename from tools/libs/ctrl/xc_core.h
rename to tools/libs/guest/xg_core.h
index 8ea1f93a10..f07584aaa6 100644
--- a/tools/libs/ctrl/xc_core.h
+++ b/tools/libs/guest/xg_core.h
@@ -131,7 +131,6 @@ struct xc_core_memory_map {
     uint64_t    size;
 };
 typedef struct xc_core_memory_map xc_core_memory_map_t;
-int xc_core_arch_auto_translated_physmap(const xc_dominfo_t *info);
 struct xc_core_arch_context;
 int xc_core_arch_memory_map_get(xc_interface *xch,
                                 struct xc_core_arch_context *arch_ctxt,
@@ -152,9 +151,9 @@ int xc_core_arch_get_scratch_gpfn(xc_interface *xch, uint32_t domid,
 
 
 #if defined (__i386__) || defined (__x86_64__)
-# include "xc_core_x86.h"
+# include "xg_core_x86.h"
 #elif defined (__arm__) || defined(__aarch64__)
-# include "xc_core_arm.h"
+# include "xg_core_arm.h"
 #else
 # error "unsupported architecture"
 #endif
diff --git a/tools/libs/ctrl/xc_core_arm.c b/tools/libs/guest/xg_core_arm.c
similarity index 96%
rename from tools/libs/ctrl/xc_core_arm.c
rename to tools/libs/guest/xg_core_arm.c
index 93765a565f..aaabd07558 100644
--- a/tools/libs/ctrl/xc_core_arm.c
+++ b/tools/libs/guest/xg_core_arm.c
@@ -17,7 +17,7 @@
  */
 
 #include "xc_private.h"
-#include "xc_core.h"
+#include "xg_core.h"
 
 #include <xen-tools/libs.h>
 
@@ -31,12 +31,6 @@ xc_core_arch_gpfn_may_present(struct xc_core_arch_context *arch_ctxt,
     return 0;
 }
 
-int
-xc_core_arch_auto_translated_physmap(const xc_dominfo_t *info)
-{
-    return 1;
-}
-
 int
 xc_core_arch_memory_map_get(xc_interface *xch, struct xc_core_arch_context *unused,
                             xc_dominfo_t *info, shared_info_any_t *live_shinfo,
diff --git a/tools/libs/ctrl/xc_core_arm.h b/tools/libs/guest/xg_core_arm.h
similarity index 100%
rename from tools/libs/ctrl/xc_core_arm.h
rename to tools/libs/guest/xg_core_arm.h
diff --git a/tools/libs/ctrl/xc_core_x86.c b/tools/libs/guest/xg_core_x86.c
similarity index 99%
rename from tools/libs/ctrl/xc_core_x86.c
rename to tools/libs/guest/xg_core_x86.c
index c8f71d4b75..09f5d696ce 100644
--- a/tools/libs/ctrl/xc_core_x86.c
+++ b/tools/libs/guest/xg_core_x86.c
@@ -19,7 +19,7 @@
 
 #include <inttypes.h>
 #include "xc_private.h"
-#include "xc_core.h"
+#include "xg_core.h"
 #include <xen/hvm/e820.h>
 
 int
@@ -33,12 +33,6 @@ xc_core_arch_gpfn_may_present(struct xc_core_arch_context *arch_ctxt,
     return 1;
 }
 
-int
-xc_core_arch_auto_translated_physmap(const xc_dominfo_t *info)
-{
-    return info->hvm;
-}
-
 int
 xc_core_arch_memory_map_get(xc_interface *xch, struct xc_core_arch_context *unused,
                             xc_dominfo_t *info, shared_info_any_t *live_shinfo,
diff --git a/tools/libs/ctrl/xc_core_x86.h b/tools/libs/guest/xg_core_x86.h
similarity index 100%
rename from tools/libs/ctrl/xc_core_x86.h
rename to tools/libs/guest/xg_core_x86.h
diff --git a/tools/libs/guest/xg_dom_boot.c b/tools/libs/guest/xg_dom_boot.c
index 2a002e7349..dac96b17a5 100644
--- a/tools/libs/guest/xg_dom_boot.c
+++ b/tools/libs/guest/xg_dom_boot.c
@@ -31,7 +31,7 @@
 #include <zlib.h>
 
 #include "xg_private.h"
-#include "xc_core.h"
+#include "xg_core.h"
 #include <xen/hvm/params.h>
 #include <xen/grant_table.h>
 
diff --git a/tools/libs/guest/xg_domain.c b/tools/libs/guest/xg_domain.c
index dd7db2cbd8..155e337427 100644
--- a/tools/libs/guest/xg_domain.c
+++ b/tools/libs/guest/xg_domain.c
@@ -20,7 +20,7 @@
  */
 
 #include "xg_private.h"
-#include "xc_core.h"
+#include "xg_core.h"
 
 int xc_unmap_domain_meminfo(xc_interface *xch, struct xc_domain_meminfo *minfo)
 {
diff --git a/tools/libs/guest/xg_offline_page.c b/tools/libs/guest/xg_offline_page.c
index d4722f0e8c..cfe0e2d537 100644
--- a/tools/libs/guest/xg_offline_page.c
+++ b/tools/libs/guest/xg_offline_page.c
@@ -25,7 +25,7 @@
 #include <stdlib.h>
 #include <unistd.h>
 #include <sys/time.h>
-#include <xc_core.h>
+#include <xg_core.h>
 
 #include "xc_private.h"
 #include "xg_private.h"
diff --git a/tools/libs/guest/xg_resume.c b/tools/libs/guest/xg_resume.c
index 3bdefb2eef..d201c1488d 100644
--- a/tools/libs/guest/xg_resume.c
+++ b/tools/libs/guest/xg_resume.c
@@ -21,7 +21,7 @@
 #include <xen/foreign/x86_32.h>
 #include <xen/foreign/x86_64.h>
 #include <xen/hvm/params.h>
-#include "xc_core.h"
+#include "xg_core.h"
 
 static int modify_returncode(xc_interface *xch, uint32_t domid)
 {
-- 
2.26.2

Re: [PATCH v2 5/6] tools/libs: move xc_core* from libxenctrl to libxenguest

[Wei Liu]

On Mon, Apr 12, 2021 at 05:22:35PM +0200, Juergen Gross wrote:
> The functionality in xc_core* should be part of libxenguest instead
> of libxenctrl. Users are already either in libxenguest, or in xl.
> There is one single exception: xc_core_arch_auto_translated_physmap()
> is being used by xc_domain_memory_mapping(), which is used by qemu.
> So leave the xc_core_arch_auto_translated_physmap() functionality in
> libxenctrl.
> 
> This will make it easier to merge common functionality of xc_core*
> and xg_sr_save*.
> 
> Signed-off-by: Juergen Gross <jgross@suse.com>

Acked-by: Wei Liu <wl@xen.org>

[PATCH v2 6/6] tools/libs/guest: make some definitions private to libxenguest

[Juergen Gross]

There are some definitions which are used in libxenguest only now.
Move them from libxenctrl over to libxenguest.

Remove an unused macro.

Signed-off-by: Juergen Gross <jgross@suse.com>
---
 tools/libs/ctrl/xc_private.h   | 32 --------------------------------
 tools/libs/guest/xg_core.h     |  2 +-
 tools/libs/guest/xg_core_x86.c | 16 +++++++++++++++-
 tools/libs/guest/xg_private.h  | 16 +++++++++++++++-
 tools/libs/guest/xg_resume.c   |  2 +-
 5 files changed, 32 insertions(+), 36 deletions(-)

diff --git a/tools/libs/ctrl/xc_private.h b/tools/libs/ctrl/xc_private.h
index dff0f0289b..3e299b943f 100644
--- a/tools/libs/ctrl/xc_private.h
+++ b/tools/libs/ctrl/xc_private.h
@@ -65,38 +65,6 @@ struct iovec {
 
 #define ROUNDUP(_x,_w) (((unsigned long)(_x)+(1UL<<(_w))-1) & ~((1UL<<(_w))-1))
 
-#define GET_FIELD(_p, _f, _w) (((_w) == 8) ? ((_p)->x64._f) : ((_p)->x32._f))
-
-#define SET_FIELD(_p, _f, _v, _w) do {          \
-    if ((_w) == 8)                              \
-        (_p)->x64._f = (_v);                    \
-    else                                        \
-        (_p)->x32._f = (_v);                    \
-} while (0)
-
-/* XXX SMH: following skanky macros rely on variable p2m_size being set */
-/* XXX TJD: also, "guest_width" should be the guest's sizeof(unsigned long) */
-
-struct domain_info_context {
-    unsigned int guest_width;
-    unsigned int p2m_frames;
-    unsigned long p2m_size;
-};
-
-/* Number of xen_pfn_t in a page */
-#define FPP             (PAGE_SIZE/(dinfo->guest_width))
-
-/* Number of entries in the pfn_to_mfn_frame_list_list */
-#define P2M_FLL_ENTRIES (((dinfo->p2m_size)+(FPP*FPP)-1)/(FPP*FPP))
-
-/* Number of entries in the pfn_to_mfn_frame_list */
-#define P2M_FL_ENTRIES  (((dinfo->p2m_size)+FPP-1)/FPP)
-
-/* Size in bytes of the pfn_to_mfn_frame_list     */
-#define P2M_GUEST_FL_SIZE ((P2M_FL_ENTRIES) * (dinfo->guest_width))
-#define P2M_TOOLS_FL_SIZE ((P2M_FL_ENTRIES) *                           \
-                           max_t(size_t, sizeof(xen_pfn_t), dinfo->guest_width))
-
 #define DECLARE_DOMCTL struct xen_domctl domctl
 #define DECLARE_SYSCTL struct xen_sysctl sysctl
 #define DECLARE_PHYSDEV_OP struct physdev_op physdev_op
diff --git a/tools/libs/guest/xg_core.h b/tools/libs/guest/xg_core.h
index f07584aaa6..aaca9e0a8b 100644
--- a/tools/libs/guest/xg_core.h
+++ b/tools/libs/guest/xg_core.h
@@ -21,7 +21,7 @@
 #define XC_CORE_H
 
 #include "xen/version.h"
-#include "xc_private.h"
+#include "xg_private.h"
 #include "xen/libelf/elfstructs.h"
 
 /* section names */
diff --git a/tools/libs/guest/xg_core_x86.c b/tools/libs/guest/xg_core_x86.c
index 09f5d696ce..61106b98b8 100644
--- a/tools/libs/guest/xg_core_x86.c
+++ b/tools/libs/guest/xg_core_x86.c
@@ -18,10 +18,24 @@
  */
 
 #include <inttypes.h>
-#include "xc_private.h"
+#include "xg_private.h"
 #include "xg_core.h"
 #include <xen/hvm/e820.h>
 
+/* Number of xen_pfn_t in a page */
+#define FPP             (PAGE_SIZE/(dinfo->guest_width))
+
+/* Number of entries in the pfn_to_mfn_frame_list_list */
+#define P2M_FLL_ENTRIES (((dinfo->p2m_size)+(FPP*FPP)-1)/(FPP*FPP))
+
+/* Number of entries in the pfn_to_mfn_frame_list */
+#define P2M_FL_ENTRIES  (((dinfo->p2m_size)+FPP-1)/FPP)
+
+/* Size in bytes of the pfn_to_mfn_frame_list     */
+#define P2M_GUEST_FL_SIZE ((P2M_FL_ENTRIES) * (dinfo->guest_width))
+#define P2M_TOOLS_FL_SIZE ((P2M_FL_ENTRIES) * \
+                           max_t(size_t, sizeof(xen_pfn_t), dinfo->guest_width))
+
 int
 xc_core_arch_gpfn_may_present(struct xc_core_arch_context *arch_ctxt,
                               unsigned long pfn)
diff --git a/tools/libs/guest/xg_private.h b/tools/libs/guest/xg_private.h
index 8f9b257a2f..ff57f8b832 100644
--- a/tools/libs/guest/xg_private.h
+++ b/tools/libs/guest/xg_private.h
@@ -42,6 +42,21 @@
 #endif
 #endif
 
+#define GET_FIELD(_p, _f, _w) (((_w) == 8) ? ((_p)->x64._f) : ((_p)->x32._f))
+
+#define SET_FIELD(_p, _f, _v, _w) do {          \
+    if ((_w) == 8)                              \
+        (_p)->x64._f = (_v);                    \
+    else                                        \
+        (_p)->x32._f = (_v);                    \
+} while (0)
+
+struct domain_info_context {
+    unsigned int guest_width;
+    unsigned int p2m_frames;
+    unsigned long p2m_size;
+};
+
 struct xc_dom_loader {
     char *name;
     /* Sadly the error returns from these functions are not consistent: */
@@ -139,7 +154,6 @@ static inline xen_pfn_t xc_pfn_to_mfn(xen_pfn_t pfn, xen_pfn_t *p2m,
 /* Masks for PTE<->PFN conversions */
 #define MADDR_BITS_X86  ((dinfo->guest_width == 8) ? 52 : 44)
 #define MFN_MASK_X86    ((1ULL << (MADDR_BITS_X86 - PAGE_SHIFT_X86)) - 1)
-#define MADDR_MASK_X86  (MFN_MASK_X86 << PAGE_SHIFT_X86)
 
 int pin_table(xc_interface *xch, unsigned int type, unsigned long mfn,
               uint32_t dom);
diff --git a/tools/libs/guest/xg_resume.c b/tools/libs/guest/xg_resume.c
index d201c1488d..77e2451a3c 100644
--- a/tools/libs/guest/xg_resume.c
+++ b/tools/libs/guest/xg_resume.c
@@ -13,7 +13,7 @@
  * License along with this library; If not, see <http://www.gnu.org/licenses/>.
  */
 
-#include "xc_private.h"
+#include "xg_private.h"
 #include "xenguest.h"
 
 #if defined(__i386__) || defined(__x86_64__)
-- 
2.26.2

Re: [PATCH v2 6/6] tools/libs/guest: make some definitions private to libxenguest

[Wei Liu]

On Mon, Apr 12, 2021 at 05:22:36PM +0200, Juergen Gross wrote:
> There are some definitions which are used in libxenguest only now.
> Move them from libxenctrl over to libxenguest.
> 
> Remove an unused macro.
> 
> Signed-off-by: Juergen Gross <jgross@suse.com>

Acked-by: Wei Liu <wl@xen.org>

Re: [PATCH v2 0/6] tools/libs: add missing support of linear p2m_list, cleanup

[Juergen Gross]

[-- Attachment #1.1.1: Type: text/plain, Size: 4154 bytes --]

Ping?

On 12.04.21 17:22, Juergen Gross wrote:
> There are some corners left which don't support the not so very new
> linear p2m list of pv guests, which has been introduced in Linux kernel
> 3.19 and which is mandatory for non-legacy versions of Xen since kernel
> 4.14.
> 
> This series adds support for the linear p2m list where it is missing
> (colo support and "xl dump-core").
> 
> In theory it should be possible to merge the p2m list mapping code
> from migration handling and core dump handling, but this needs quite
> some cleanup before this is possible.
> 
> The first three patches of this series are fixing real problems, so
> I've put them at the start of this series, especially in order to make
> backports easier.
> 
> The other three patches are only the first steps of cleanup. The main
> work done here is to concentrate all p2m mapping in libxenguest instead
> of having one implementation in each of libxenguest and libxenctrl.
> 
> Merging the two implementations should be rather easy, but this will
> require to touch many lines of code, as the migration handling variant
> seems to be more mature, but it is using the migration stream specific
> structures heavily. So I'd like to have some confirmation that my way
> to clean this up is the right one.
> 
> My idea would be to add the data needed for p2m mapping to struct
> domain_info_context and replace the related fields in struct
> xc_sr_context with a struct domain_info_context. Modifying the
> interface of xc_core_arch_map_p2m() to take most current parameters
> via struct domain_info_context would then enable migration coding to
> use xc_core_arch_map_p2m() for mapping the p2m. xc_core_arch_map_p2m()
> should look basically like the current migration p2m mapping code
> afterwards.
> 
> Any comments to that plan?
> 
> Changes in V2:
> - added missing #include in ocaml stub
> 
> Juergen Gross (6):
>    tools/libs/guest: fix max_pfn setting in map_p2m()
>    tools/libs/ctrl: fix xc_core_arch_map_p2m() to support linear p2m
>      table
>    tools/libs/ctrl: use common p2m mapping code in xc_domain_resume_any()
>    tools/libs: move xc_resume.c to libxenguest
>    tools/libs: move xc_core* from libxenctrl to libxenguest
>    tools/libs/guest: make some definitions private to libxenguest
> 
>   tools/include/xenctrl.h                       |  63 ---
>   tools/include/xenguest.h                      |  63 +++
>   tools/libs/ctrl/Makefile                      |   4 -
>   tools/libs/ctrl/xc_core_x86.c                 | 223 ----------
>   tools/libs/ctrl/xc_domain.c                   |   2 -
>   tools/libs/ctrl/xc_private.h                  |  43 +-
>   tools/libs/guest/Makefile                     |   4 +
>   .../libs/{ctrl/xc_core.c => guest/xg_core.c}  |   7 +-
>   .../libs/{ctrl/xc_core.h => guest/xg_core.h}  |  15 +-
>   .../xc_core_arm.c => guest/xg_core_arm.c}     |  31 +-
>   .../xc_core_arm.h => guest/xg_core_arm.h}     |   0
>   tools/libs/guest/xg_core_x86.c                | 399 ++++++++++++++++++
>   .../xc_core_x86.h => guest/xg_core_x86.h}     |   0
>   tools/libs/guest/xg_dom_boot.c                |   2 +-
>   tools/libs/guest/xg_domain.c                  |  19 +-
>   tools/libs/guest/xg_offline_page.c            |   2 +-
>   tools/libs/guest/xg_private.h                 |  16 +-
>   .../{ctrl/xc_resume.c => guest/xg_resume.c}   |  69 +--
>   tools/libs/guest/xg_sr_save_x86_pv.c          |   2 +-
>   tools/ocaml/libs/xc/xenctrl_stubs.c           |   1 +
>   20 files changed, 545 insertions(+), 420 deletions(-)
>   delete mode 100644 tools/libs/ctrl/xc_core_x86.c
>   rename tools/libs/{ctrl/xc_core.c => guest/xg_core.c} (99%)
>   rename tools/libs/{ctrl/xc_core.h => guest/xg_core.h} (92%)
>   rename tools/libs/{ctrl/xc_core_arm.c => guest/xg_core_arm.c} (72%)
>   rename tools/libs/{ctrl/xc_core_arm.h => guest/xg_core_arm.h} (100%)
>   create mode 100644 tools/libs/guest/xg_core_x86.c
>   rename tools/libs/{ctrl/xc_core_x86.h => guest/xg_core_x86.h} (100%)
>   rename tools/libs/{ctrl/xc_resume.c => guest/xg_resume.c} (80%)
> 


[-- Attachment #1.1.2: OpenPGP_0xB0DE9DD628BF132F.asc --]
[-- Type: application/pgp-keys, Size: 3135 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 495 bytes --]

Re: [PATCH v2 0/6] tools/libs: add missing support of linear p2m_list, cleanup

[Juergen Gross]

[-- Attachment #1.1.1: Type: text/plain, Size: 4724 bytes --]

On 12.05.21 08:58, Juergen Gross wrote:
> Ping?

Now each patch has an Ack by Wei. Could the series be either applied or
get more comments, please?


Juergen

> 
> On 12.04.21 17:22, Juergen Gross wrote:
>> There are some corners left which don't support the not so very new
>> linear p2m list of pv guests, which has been introduced in Linux kernel
>> 3.19 and which is mandatory for non-legacy versions of Xen since kernel
>> 4.14.
>>
>> This series adds support for the linear p2m list where it is missing
>> (colo support and "xl dump-core").
>>
>> In theory it should be possible to merge the p2m list mapping code
>> from migration handling and core dump handling, but this needs quite
>> some cleanup before this is possible.
>>
>> The first three patches of this series are fixing real problems, so
>> I've put them at the start of this series, especially in order to make
>> backports easier.
>>
>> The other three patches are only the first steps of cleanup. The main
>> work done here is to concentrate all p2m mapping in libxenguest instead
>> of having one implementation in each of libxenguest and libxenctrl.
>>
>> Merging the two implementations should be rather easy, but this will
>> require to touch many lines of code, as the migration handling variant
>> seems to be more mature, but it is using the migration stream specific
>> structures heavily. So I'd like to have some confirmation that my way
>> to clean this up is the right one.
>>
>> My idea would be to add the data needed for p2m mapping to struct
>> domain_info_context and replace the related fields in struct
>> xc_sr_context with a struct domain_info_context. Modifying the
>> interface of xc_core_arch_map_p2m() to take most current parameters
>> via struct domain_info_context would then enable migration coding to
>> use xc_core_arch_map_p2m() for mapping the p2m. xc_core_arch_map_p2m()
>> should look basically like the current migration p2m mapping code
>> afterwards.
>>
>> Any comments to that plan?
>>
>> Changes in V2:
>> - added missing #include in ocaml stub
>>
>> Juergen Gross (6):
>>    tools/libs/guest: fix max_pfn setting in map_p2m()
>>    tools/libs/ctrl: fix xc_core_arch_map_p2m() to support linear p2m
>>      table
>>    tools/libs/ctrl: use common p2m mapping code in xc_domain_resume_any()
>>    tools/libs: move xc_resume.c to libxenguest
>>    tools/libs: move xc_core* from libxenctrl to libxenguest
>>    tools/libs/guest: make some definitions private to libxenguest
>>
>>   tools/include/xenctrl.h                       |  63 ---
>>   tools/include/xenguest.h                      |  63 +++
>>   tools/libs/ctrl/Makefile                      |   4 -
>>   tools/libs/ctrl/xc_core_x86.c                 | 223 ----------
>>   tools/libs/ctrl/xc_domain.c                   |   2 -
>>   tools/libs/ctrl/xc_private.h                  |  
43 +-
>>   tools/libs/guest/Makefile                     |   4 +
>>   .../libs/{ctrl/xc_core.c => guest/xg_core.c}  |   7 +-
>>   .../libs/{ctrl/xc_core.h => guest/xg_core.h}  |  15 
+-
>>   .../xc_core_arm.c => guest/xg_core_arm.c}     |  31 +-
>>   .../xc_core_arm.h => guest/xg_core_arm.h}     |   0
>>   tools/libs/guest/xg_core_x86.c                | 399 ++++++++++++++++++
>>   .../xc_core_x86.h => guest/xg_core_x86.h}     |   0
>>   tools/libs/guest/xg_dom_boot.c                |   2 
+-
>>   tools/libs/guest/xg_domain.c                  |  
19 +-
>>   tools/libs/guest/xg_offline_page.c            |   2 +-
>>   tools/libs/guest/xg_private.h                 |  16 +-
>>   .../{ctrl/xc_resume.c => guest/xg_resume.c}   |  
69 +--
>>   tools/libs/guest/xg_sr_save_x86_pv.c          |   2 +-
>>   tools/ocaml/libs/xc/xenctrl_stubs.c           |   1 +
>>   20 files changed, 545 insertions(+), 420 deletions(-)
>>   delete mode 100644 tools/libs/ctrl/xc_core_x86.c
>>   rename tools/libs/{ctrl/xc_core.c => guest/xg_core.c} (99%)
>>   rename tools/libs/{ctrl/xc_core.h => guest/xg_core.h} (92%)
>>   rename tools/libs/{ctrl/xc_core_arm.c => guest/xg_core_arm.c} 
(72%)
>>   rename tools/libs/{ctrl/xc_core_arm.h => guest/xg_core_arm.h} 
(100%)
>>   create mode 100644 tools/libs/guest/xg_core_x86.c
>>   rename tools/libs/{ctrl/xc_core_x86.h => guest/xg_core_x86.h} 
(100%)
>>   rename tools/libs/{ctrl/xc_resume.c => guest/xg_resume.c} (80%)
>>
> 


[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 3135 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 495 bytes --]

Re: [PATCH v2 0/6] tools/libs: add missing support of linear p2m_list, cleanup

[Juergen Gross]

[-- Attachment #1.1.1: Type: text/plain, Size: 4989 bytes --]

On 25.05.21 09:32, Juergen Gross wrote:
> On 12.05.21 08:58, Juergen Gross wrote:
>> Ping?
> 
> Now each patch has an Ack by Wei. Could the series be either applied or
> get more comments, please?

And another PING.

Do I need to setup a cron job pinging each day to get a reaction?


Juergen

> 
> 
> Juergen
> 
>>
>> On 12.04.21 17:22, Juergen Gross wrote:
>>> There are some corners left which don't support the not so very new
>>> linear p2m list of pv guests, which has been introduced in Linux kernel
>>> 3.19 and which is mandatory for non-legacy versions of Xen since kernel
>>> 4.14.
>>>
>>> This series adds support for the linear p2m list where it is missing
>>> (colo support and "xl dump-core").
>>>
>>> In theory it should be possible to merge the p2m list mapping code
>>> from migration handling and core dump handling, but this needs quite
>>> some cleanup before this is possible.
>>>
>>> The first three patches of this series are fixing real problems, so
>>> I've put them at the start of this series, especially in order to make
>>> backports easier.
>>>
>>> The other three patches are only the first steps of cleanup. The main
>>> work done here is to concentrate all p2m mapping in libxenguest instead
>>> of having one implementation in each of libxenguest and libxenctrl.
>>>
>>> Merging the two implementations should be rather easy, but this will
>>> require to touch many lines of code, as the migration handling variant
>>> seems to be more mature, but it is using the migration stream specific
>>> structures heavily. So I'd like to have some confirmation that my way
>>> to clean this up is the right one.
>>>
>>> My idea would be to add the data needed for p2m mapping to struct
>>> domain_info_context and replace the related fields in struct
>>> xc_sr_context with a struct domain_info_context. Modifying the
>>> interface of xc_core_arch_map_p2m() to take most current parameters
>>> via struct domain_info_context would then enable migration coding to
>>> use xc_core_arch_map_p2m() for mapping the p2m. xc_core_arch_map_p2m()
>>> should look basically like the current migration p2m mapping code
>>> afterwards.
>>>
>>> Any comments to that plan?
>>>
>>> Changes in V2:
>>> - added missing #include in ocaml stub
>>>
>>> Juergen Gross (6):
>>>    tools/libs/guest: fix max_pfn setting in map_p2m()
>>>    tools/libs/ctrl: fix xc_core_arch_map_p2m() to support linear p2m
>>>      table
>>>    tools/libs/ctrl: use common p2m mapping code in 
>>> xc_domain_resume_any()
>>>    tools/libs: move xc_resume.c to libxenguest
>>>    tools/libs: move xc_core* from libxenctrl to libxenguest
>>>    tools/libs/guest: make some definitions private to libxenguest
>>>
>>>   tools/include/xenctrl.h                       |  63 ---
>>>   tools/include/xenguest.h                      |  63 +++
>>>   tools/libs/ctrl/Makefile                      |   4 -
>>>   tools/libs/ctrl/xc_core_x86.c                 | 223 ----------
>>>   tools/libs/ctrl/xc_domain.c                   |   2 -
>>>   tools/libs/ctrl/xc_private.h                  | 
> 43 +-
>>>   tools/libs/guest/Makefile                     |   4 +
>>>   .../libs/{ctrl/xc_core.c => guest/xg_core.c}  |   7 +-
>>>   .../libs/{ctrl/xc_core.h => guest/xg_core.h}  |  15 
> +-
>>>   .../xc_core_arm.c => guest/xg_core_arm.c}     |  31 +-
>>>   .../xc_core_arm.h => guest/xg_core_arm.h}     |   0
>>>   tools/libs/guest/xg_core_x86.c                | 399 ++++++++++++++++++
>>>   .../xc_core_x86.h => guest/xg_core_x86.h}     |   0
>>>   tools/libs/guest/xg_dom_boot.c                |   2 

> +-
>>>   tools/libs/guest/xg_domain.c                  | 
> 19 +-
>>>   tools/libs/guest/xg_offline_page.c            |   2 +-
>>>   tools/libs/guest/xg_private.h                 |  16 +-
>>>   .../{ctrl/xc_resume.c => guest/xg_resume.c}   | 
> 69 +--
>>>   tools/libs/guest/xg_sr_save_x86_pv.c          |   2 +-
>>>   tools/ocaml/libs/xc/xenctrl_stubs.c           |   1 +
>>>   20 files changed, 545 insertions(+), 420 deletions(-)
>>>   delete mode 100644 tools/libs/ctrl/xc_core_x86.c
>>>   rename tools/libs/{ctrl/xc_core.c => guest/xg_core.c} (99%)
>>>   rename tools/libs/{ctrl/xc_core.h => guest/xg_core.h} (92%)
>>>   rename tools/libs/{ctrl/xc_core_arm.c => guest/xg_core_arm.c} 
> (72%)
>>>   rename tools/libs/{ctrl/xc_core_arm.h => guest/xg_core_arm.h} 
> (100%)
>>>   create mode 100644 tools/libs/guest/xg_core_x86.c
>>>   rename tools/libs/{ctrl/xc_core_x86.h => guest/xg_core_x86.h} 
> (100%)
>>>   rename tools/libs/{ctrl/xc_resume.c => guest/xg_resume.c} (80%)
>>>
>>
> 


[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 3135 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 495 bytes --]

[PATCH v2 0/6] tools/libs: add missing support of linear p2m_list, cleanup

[Juergen Gross]

This is a resend of V2 with the Acks folded in.

There are some corners left which don't support the not so very new
linear p2m list of pv guests, which has been introduced in Linux kernel
3.19 and which is mandatory for non-legacy versions of Xen since kernel
4.14.

This series adds support for the linear p2m list where it is missing
(colo support and "xl dump-core").

In theory it should be possible to merge the p2m list mapping code
from migration handling and core dump handling, but this needs quite
some cleanup before this is possible.

The first three patches of this series are fixing real problems, so
I've put them at the start of this series, especially in order to make
backports easier.

The other three patches are only the first steps of cleanup. The main
work done here is to concentrate all p2m mapping in libxenguest instead
of having one implementation in each of libxenguest and libxenctrl.

Merging the two implementations should be rather easy, but this will
require to touch many lines of code, as the migration handling variant
seems to be more mature, but it is using the migration stream specific
structures heavily. So I'd like to have some confirmation that my way
to clean this up is the right one.

My idea would be to add the data needed for p2m mapping to struct
domain_info_context and replace the related fields in struct
xc_sr_context with a struct domain_info_context. Modifying the
interface of xc_core_arch_map_p2m() to take most current parameters
via struct domain_info_context would then enable migration coding to
use xc_core_arch_map_p2m() for mapping the p2m. xc_core_arch_map_p2m()
should look basically like the current migration p2m mapping code
afterwards.

Changes in V2:
- added missing #include in ocaml stub

Juergen Gross (6):
  tools/libs/guest: fix max_pfn setting in map_p2m()
  tools/libs/ctrl: fix xc_core_arch_map_p2m() to support linear p2m
    table
  tools/libs/ctrl: use common p2m mapping code in xc_domain_resume_any()
  tools/libs: move xc_resume.c to libxenguest
  tools/libs: move xc_core* from libxenctrl to libxenguest
  tools/libs/guest: make some definitions private to libxenguest

 tools/include/xenctrl.h                       |  63 ---
 tools/include/xenguest.h                      |  63 +++
 tools/libs/ctrl/Makefile                      |   4 -
 tools/libs/ctrl/xc_core_x86.c                 | 223 ----------
 tools/libs/ctrl/xc_domain.c                   |   2 -
 tools/libs/ctrl/xc_private.h                  |  43 +-
 tools/libs/guest/Makefile                     |   4 +
 .../libs/{ctrl/xc_core.c => guest/xg_core.c}  |   7 +-
 .../libs/{ctrl/xc_core.h => guest/xg_core.h}  |  15 +-
 .../xc_core_arm.c => guest/xg_core_arm.c}     |  31 +-
 .../xc_core_arm.h => guest/xg_core_arm.h}     |   0
 tools/libs/guest/xg_core_x86.c                | 399 ++++++++++++++++++
 .../xc_core_x86.h => guest/xg_core_x86.h}     |   0
 tools/libs/guest/xg_dom_boot.c                |   2 +-
 tools/libs/guest/xg_domain.c                  |  19 +-
 tools/libs/guest/xg_offline_page.c            |   2 +-
 tools/libs/guest/xg_private.h                 |  16 +-
 .../{ctrl/xc_resume.c => guest/xg_resume.c}   |  69 +--
 tools/libs/guest/xg_sr_save_x86_pv.c          |   2 +-
 tools/ocaml/libs/xc/xenctrl_stubs.c           |   1 +
 20 files changed, 545 insertions(+), 420 deletions(-)
 delete mode 100644 tools/libs/ctrl/xc_core_x86.c
 rename tools/libs/{ctrl/xc_core.c => guest/xg_core.c} (99%)
 rename tools/libs/{ctrl/xc_core.h => guest/xg_core.h} (92%)
 rename tools/libs/{ctrl/xc_core_arm.c => guest/xg_core_arm.c} (72%)
 rename tools/libs/{ctrl/xc_core_arm.h => guest/xg_core_arm.h} (100%)
 create mode 100644 tools/libs/guest/xg_core_x86.c
 rename tools/libs/{ctrl/xc_core_x86.h => guest/xg_core_x86.h} (100%)
 rename tools/libs/{ctrl/xc_resume.c => guest/xg_resume.c} (80%)

-- 
2.26.2

[PATCH v2 6/6] tools/libs/guest: make some definitions private to libxenguest

[Juergen Gross]

There are some definitions which are used in libxenguest only now.
Move them from libxenctrl over to libxenguest.

Remove an unused macro.

Signed-off-by: Juergen Gross <jgross@suse.com>
Acked-by: Wei Liu <wl@xen.org>
---
 tools/libs/ctrl/xc_private.h   | 32 --------------------------------
 tools/libs/guest/xg_core.h     |  2 +-
 tools/libs/guest/xg_core_x86.c | 16 +++++++++++++++-
 tools/libs/guest/xg_private.h  | 16 +++++++++++++++-
 tools/libs/guest/xg_resume.c   |  2 +-
 5 files changed, 32 insertions(+), 36 deletions(-)

diff --git a/tools/libs/ctrl/xc_private.h b/tools/libs/ctrl/xc_private.h
index dff0f0289b..3e299b943f 100644
--- a/tools/libs/ctrl/xc_private.h
+++ b/tools/libs/ctrl/xc_private.h
@@ -65,38 +65,6 @@ struct iovec {
 
 #define ROUNDUP(_x,_w) (((unsigned long)(_x)+(1UL<<(_w))-1) & ~((1UL<<(_w))-1))
 
-#define GET_FIELD(_p, _f, _w) (((_w) == 8) ? ((_p)->x64._f) : ((_p)->x32._f))
-
-#define SET_FIELD(_p, _f, _v, _w) do {          \
-    if ((_w) == 8)                              \
-        (_p)->x64._f = (_v);                    \
-    else                                        \
-        (_p)->x32._f = (_v);                    \
-} while (0)
-
-/* XXX SMH: following skanky macros rely on variable p2m_size being set */
-/* XXX TJD: also, "guest_width" should be the guest's sizeof(unsigned long) */
-
-struct domain_info_context {
-    unsigned int guest_width;
-    unsigned int p2m_frames;
-    unsigned long p2m_size;
-};
-
-/* Number of xen_pfn_t in a page */
-#define FPP             (PAGE_SIZE/(dinfo->guest_width))
-
-/* Number of entries in the pfn_to_mfn_frame_list_list */
-#define P2M_FLL_ENTRIES (((dinfo->p2m_size)+(FPP*FPP)-1)/(FPP*FPP))
-
-/* Number of entries in the pfn_to_mfn_frame_list */
-#define P2M_FL_ENTRIES  (((dinfo->p2m_size)+FPP-1)/FPP)
-
-/* Size in bytes of the pfn_to_mfn_frame_list     */
-#define P2M_GUEST_FL_SIZE ((P2M_FL_ENTRIES) * (dinfo->guest_width))
-#define P2M_TOOLS_FL_SIZE ((P2M_FL_ENTRIES) *                           \
-                           max_t(size_t, sizeof(xen_pfn_t), dinfo->guest_width))
-
 #define DECLARE_DOMCTL struct xen_domctl domctl
 #define DECLARE_SYSCTL struct xen_sysctl sysctl
 #define DECLARE_PHYSDEV_OP struct physdev_op physdev_op
diff --git a/tools/libs/guest/xg_core.h b/tools/libs/guest/xg_core.h
index f07584aaa6..aaca9e0a8b 100644
--- a/tools/libs/guest/xg_core.h
+++ b/tools/libs/guest/xg_core.h
@@ -21,7 +21,7 @@
 #define XC_CORE_H
 
 #include "xen/version.h"
-#include "xc_private.h"
+#include "xg_private.h"
 #include "xen/libelf/elfstructs.h"
 
 /* section names */
diff --git a/tools/libs/guest/xg_core_x86.c b/tools/libs/guest/xg_core_x86.c
index 09f5d696ce..61106b98b8 100644
--- a/tools/libs/guest/xg_core_x86.c
+++ b/tools/libs/guest/xg_core_x86.c
@@ -18,10 +18,24 @@
  */
 
 #include <inttypes.h>
-#include "xc_private.h"
+#include "xg_private.h"
 #include "xg_core.h"
 #include <xen/hvm/e820.h>
 
+/* Number of xen_pfn_t in a page */
+#define FPP             (PAGE_SIZE/(dinfo->guest_width))
+
+/* Number of entries in the pfn_to_mfn_frame_list_list */
+#define P2M_FLL_ENTRIES (((dinfo->p2m_size)+(FPP*FPP)-1)/(FPP*FPP))
+
+/* Number of entries in the pfn_to_mfn_frame_list */
+#define P2M_FL_ENTRIES  (((dinfo->p2m_size)+FPP-1)/FPP)
+
+/* Size in bytes of the pfn_to_mfn_frame_list     */
+#define P2M_GUEST_FL_SIZE ((P2M_FL_ENTRIES) * (dinfo->guest_width))
+#define P2M_TOOLS_FL_SIZE ((P2M_FL_ENTRIES) * \
+                           max_t(size_t, sizeof(xen_pfn_t), dinfo->guest_width))
+
 int
 xc_core_arch_gpfn_may_present(struct xc_core_arch_context *arch_ctxt,
                               unsigned long pfn)
diff --git a/tools/libs/guest/xg_private.h b/tools/libs/guest/xg_private.h
index 25e46d7ce1..03d765da21 100644
--- a/tools/libs/guest/xg_private.h
+++ b/tools/libs/guest/xg_private.h
@@ -42,6 +42,21 @@
 #endif
 #endif
 
+#define GET_FIELD(_p, _f, _w) (((_w) == 8) ? ((_p)->x64._f) : ((_p)->x32._f))
+
+#define SET_FIELD(_p, _f, _v, _w) do {          \
+    if ((_w) == 8)                              \
+        (_p)->x64._f = (_v);                    \
+    else                                        \
+        (_p)->x32._f = (_v);                    \
+} while (0)
+
+struct domain_info_context {
+    unsigned int guest_width;
+    unsigned int p2m_frames;
+    unsigned long p2m_size;
+};
+
 struct xc_dom_loader {
     const char *name;
     /* Sadly the error returns from these functions are not consistent: */
@@ -139,7 +154,6 @@ static inline xen_pfn_t xc_pfn_to_mfn(xen_pfn_t pfn, xen_pfn_t *p2m,
 /* Masks for PTE<->PFN conversions */
 #define MADDR_BITS_X86  ((dinfo->guest_width == 8) ? 52 : 44)
 #define MFN_MASK_X86    ((1ULL << (MADDR_BITS_X86 - PAGE_SHIFT_X86)) - 1)
-#define MADDR_MASK_X86  (MFN_MASK_X86 << PAGE_SHIFT_X86)
 
 int pin_table(xc_interface *xch, unsigned int type, unsigned long mfn,
               uint32_t dom);
diff --git a/tools/libs/guest/xg_resume.c b/tools/libs/guest/xg_resume.c
index d201c1488d..77e2451a3c 100644
--- a/tools/libs/guest/xg_resume.c
+++ b/tools/libs/guest/xg_resume.c
@@ -13,7 +13,7 @@
  * License along with this library; If not, see <http://www.gnu.org/licenses/>.
  */
 
-#include "xc_private.h"
+#include "xg_private.h"
 #include "xenguest.h"
 
 #if defined(__i386__) || defined(__x86_64__)
-- 
2.26.2