[PATCH v19 for-4.14 00/13] VM forking

[PATCH v19 for-4.14 00/13] VM forking

[Tamas K Lengyel]

The following patches are part of the series that implement VM forking for
Intel HVM guests to allow for the fast creation of identical VMs without the
assosciated high startup costs of booting or restoring the VM from a savefile.

JIRA issue: https://xenproject.atlassian.net/browse/XEN-89

The fork operation is implemented as part of the "xl fork-vm" command:
    xl fork-vm -C <config> -Q <qemu-save-file> <parent_domid>

By default a fully functional fork is created. The user is in charge however to
create the appropriate config file for the fork and to generate the QEMU save
file before the fork-vm call is made. The config file needs to give the
fork a new name at minimum but other settings may also require changes. Certain
settings in the config file of both the parent and the fork have to be set to
default. Details are documented.

The interface also allows to split the forking into two steps:
    xl fork-vm --launch-dm no \
               -m <max-vcpus> \
			   -p <parent_domid>
	xl fork-vm --launch-dm late \
	           -C <config_file_for_fork> \
			   -Q <qemu_save_file> \
			   <fork_domid>

The split creation model is useful when the VM needs to be created as fast as
possible. The forked VM can be unpaused without the device model being launched
to be monitored and accessed via VMI. Note however that without its device
model running (depending on what is executing in the VM) it is bound to
misbehave or even crash when its trying to access devices that would be
emulated by QEMU. We anticipate that for certain use-cases this would be an
acceptable situation, in case for example when fuzzing is performed of code
segments that don't access such devices.

Launching the device model requires the QEMU Xen savefile to be generated
manually from the parent VM. This can be accomplished simply by connecting to
its QMP socket and issuing the "xen-save-devices-state" command. For example
using the standard tool socat these commands can be used to generate the file:
    socat - UNIX-CONNECT:/var/run/xen/qmp-libxl-<parent_domid>
	{ "execute": "qmp_capabilities" }
	{ "execute": "xen-save-devices-state", \
		"arguments": { "filename": "/path/to/save/qemu_state", \
						"live": false} }

The series has been tested with Windows VMs and functions as expected. Linux
VMs when forked from a running VM will have a frozen VNC screen. Linux VMs at
this time can only be forked with a working device model when the parent VM was
restored from a snapshot using "xl restore -p". This is a known limitation due
to Linux VMs having to be made aware of being saved/migrated.

New in v19:
	Including all the patches currently outstanding into the series
	Breaking up libxl/xl patch to many sub-patches to make it easier to review
	libxl/xl is now reduced to the bare essential to launch QEMU for a VM fork

Tamas K Lengyel (13):
  x86/mem_sharing: block interrupt injection for forks
  tools/libxc: xc_memshr_fork with interrupts blocked
  tools/libxl: Split libxl__domain_make
  tools/libxl: populate xenstore entries when launching dm for VM fork
  tools/libxl: Add checks for dm_restore_file
  tools/libxl: adjust domcreate_bootloader_done
  tools/libxl: Adjust libxl__build_pre
  tools/libxl: Adjust libxl__build_post
  tools/libxl: libxl__build_hvm_fork
  tools/libxl: set QEMU saved_state from dm_restore_file
  tools/libxl: Add VM forking public functions
  tools/xl: Add xl fork-vm command
  tools/xl: document fork-vm command

 docs/man/xl.1.pod.in             |  39 +++++++++
 tools/libxc/include/xenctrl.h    |   3 +-
 tools/libxc/xc_memshr.c          |   4 +-
 tools/libxl/libxl.h              |  10 +++
 tools/libxl/libxl_create.c       | 134 +++++++++++++++++++++++++------
 tools/libxl/libxl_dm.c           |   2 +-
 tools/libxl/libxl_dom.c          |  59 +++++++++++---
 tools/libxl/libxl_internal.h     |   5 +-
 tools/libxl/libxl_types.idl      |   1 +
 tools/xl/Makefile                |   2 +-
 tools/xl/xl.h                    |   4 +
 tools/xl/xl_cmdtable.c           |  13 +++
 tools/xl/xl_forkvm.c             | 122 ++++++++++++++++++++++++++++
 tools/xl/xl_vmcontrol.c          |  13 +++
 xen/arch/x86/hvm/vmx/intr.c      |   6 ++
 xen/arch/x86/mm/mem_sharing.c    |   6 +-
 xen/include/asm-x86/hvm/domain.h |   2 +-
 xen/include/public/memory.h      |   3 +
 18 files changed, 383 insertions(+), 45 deletions(-)
 create mode 100644 tools/xl/xl_forkvm.c

-- 
2.25.1

[PATCH v19 for-4.14 01/13] x86/mem_sharing: block interrupt injection for forks

[Tamas K Lengyel]

When running VM forks without device models (QEMU), it may
be undesirable for Xen to inject interrupts. When creating such forks from
Windows VMs we have observed the kernel trying to process interrupts
immediately after the fork is executed. However without QEMU running such
interrupt handling may not be possible because it may attempt to interact with
devices that are not emulated by a backend. In the best case scenario such
interrupt handling would only present a detour in the VM forks' execution
flow, but in the worst case as we actually observed can completely stall it.
By disabling interrupt injection a fuzzer can exercise the target code without
interference. For other use-cases this option probably doesn't make sense,
that's why this is not enabled by default.

Forks & memory sharing are only available on Intel CPUs so this only applies
to vmx. Note that this is part of the experimental VM forking feature that's
completely disabled by default and can only be enabled by using
XEN_CONFIG_EXPERT during compile time.

Signed-off-by: Tamas K Lengyel <tamas.lengyel@intel.com>
Reviewed-by: Roger Pau Monné <roger.pau@citrix.com>
---
 xen/arch/x86/hvm/vmx/intr.c      | 6 ++++++
 xen/arch/x86/mm/mem_sharing.c    | 6 +++++-
 xen/include/asm-x86/hvm/domain.h | 2 +-
 xen/include/public/memory.h      | 3 +++
 4 files changed, 15 insertions(+), 2 deletions(-)

diff --git a/xen/arch/x86/hvm/vmx/intr.c b/xen/arch/x86/hvm/vmx/intr.c
index 000e14af49..80bfbb4787 100644
--- a/xen/arch/x86/hvm/vmx/intr.c
+++ b/xen/arch/x86/hvm/vmx/intr.c
@@ -256,6 +256,12 @@ void vmx_intr_assist(void)
     if ( unlikely(v->arch.vm_event) && v->arch.vm_event->sync_event )
         return;
 
+#ifdef CONFIG_MEM_SHARING
+    /* Block event injection for VM fork if requested */
+    if ( unlikely(v->domain->arch.hvm.mem_sharing.block_interrupts) )
+        return;
+#endif
+
     /* Crank the handle on interrupt state. */
     pt_vector = pt_update_irq(v);
 
diff --git a/xen/arch/x86/mm/mem_sharing.c b/xen/arch/x86/mm/mem_sharing.c
index 19922ab5d1..c428fd16ce 100644
--- a/xen/arch/x86/mm/mem_sharing.c
+++ b/xen/arch/x86/mm/mem_sharing.c
@@ -2106,7 +2106,8 @@ int mem_sharing_memop(XEN_GUEST_HANDLE_PARAM(xen_mem_sharing_op_t) arg)
         rc = -EINVAL;
         if ( mso.u.fork.pad )
             goto out;
-        if ( mso.u.fork.flags & ~XENMEM_FORK_WITH_IOMMU_ALLOWED )
+        if ( mso.u.fork.flags &
+             ~(XENMEM_FORK_WITH_IOMMU_ALLOWED | XENMEM_FORK_BLOCK_INTERRUPTS) )
             goto out;
 
         rc = rcu_lock_live_remote_domain_by_id(mso.u.fork.parent_domain,
@@ -2134,6 +2135,9 @@ int mem_sharing_memop(XEN_GUEST_HANDLE_PARAM(xen_mem_sharing_op_t) arg)
             rc = hypercall_create_continuation(__HYPERVISOR_memory_op,
                                                "lh", XENMEM_sharing_op,
                                                arg);
+        else if ( !rc && (mso.u.fork.flags & XENMEM_FORK_BLOCK_INTERRUPTS) )
+            d->arch.hvm.mem_sharing.block_interrupts = true;
+
         rcu_unlock_domain(pd);
         break;
     }
diff --git a/xen/include/asm-x86/hvm/domain.h b/xen/include/asm-x86/hvm/domain.h
index 95fe18cddc..9d247baf4d 100644
--- a/xen/include/asm-x86/hvm/domain.h
+++ b/xen/include/asm-x86/hvm/domain.h
@@ -67,7 +67,7 @@ struct hvm_ioreq_server {
 #ifdef CONFIG_MEM_SHARING
 struct mem_sharing_domain
 {
-    bool enabled;
+    bool enabled, block_interrupts;
 
     /*
      * When releasing shared gfn's in a preemptible manner, recall where
diff --git a/xen/include/public/memory.h b/xen/include/public/memory.h
index dbd35305df..850bd72c52 100644
--- a/xen/include/public/memory.h
+++ b/xen/include/public/memory.h
@@ -536,7 +536,10 @@ struct xen_mem_sharing_op {
         } debug;
         struct mem_sharing_op_fork {      /* OP_FORK */
             domid_t parent_domain;        /* IN: parent's domain id */
+/* Only makes sense for short-lived forks */
 #define XENMEM_FORK_WITH_IOMMU_ALLOWED (1u << 0)
+/* Only makes sense for short-lived forks */
+#define XENMEM_FORK_BLOCK_INTERRUPTS   (1u << 1)
             uint16_t flags;               /* IN: optional settings */
             uint32_t pad;                 /* Must be set to 0 */
         } fork;
-- 
2.25.1

[PATCH v19 for-4.14 02/13] tools/libxc: xc_memshr_fork with interrupts blocked

[Tamas K Lengyel]

Toolstack side for creating forks with interrupt injection blocked.

Signed-off-by: Tamas K Lengyel <tamas.lengyel@intel.com>
Reviewed-by: Roger Pau Monné <roger.pau@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 tools/libxc/include/xenctrl.h | 3 ++-
 tools/libxc/xc_memshr.c       | 4 +++-
 2 files changed, 5 insertions(+), 2 deletions(-)

diff --git a/tools/libxc/include/xenctrl.h b/tools/libxc/include/xenctrl.h
index f9e17ae424..5eeee1de46 100644
--- a/tools/libxc/include/xenctrl.h
+++ b/tools/libxc/include/xenctrl.h
@@ -2241,7 +2241,8 @@ int xc_memshr_range_share(xc_interface *xch,
 int xc_memshr_fork(xc_interface *xch,
                    uint32_t source_domain,
                    uint32_t client_domain,
-                   bool allow_with_iommu);
+                   bool allow_with_iommu,
+                   bool block_interrupts);
 
 /*
  * Note: this function is only intended to be used on short-lived forks that
diff --git a/tools/libxc/xc_memshr.c b/tools/libxc/xc_memshr.c
index 2300cc7075..a6cfd7dccf 100644
--- a/tools/libxc/xc_memshr.c
+++ b/tools/libxc/xc_memshr.c
@@ -240,7 +240,7 @@ int xc_memshr_debug_gref(xc_interface *xch,
 }
 
 int xc_memshr_fork(xc_interface *xch, uint32_t pdomid, uint32_t domid,
-                   bool allow_with_iommu)
+                   bool allow_with_iommu, bool block_interrupts)
 {
     xen_mem_sharing_op_t mso;
 
@@ -251,6 +251,8 @@ int xc_memshr_fork(xc_interface *xch, uint32_t pdomid, uint32_t domid,
 
     if ( allow_with_iommu )
         mso.u.fork.flags |= XENMEM_FORK_WITH_IOMMU_ALLOWED;
+    if ( block_interrupts )
+        mso.u.fork.flags |= XENMEM_FORK_BLOCK_INTERRUPTS;
 
     return xc_memshr_memop(xch, domid, &mso);
 }
-- 
2.25.1

[PATCH v19 for-4.14 03/13] tools/libxl: Split libxl__domain_make

[Tamas K Lengyel]

Make part of libxl__domain_make into a separate function. No functional change.

Signed-off-by: Tamas K Lengyel <tamas.lengyel@intel.com>
---
 tools/libxl/libxl_create.c   | 62 +++++++++++++++++++++++-------------
 tools/libxl/libxl_internal.h |  4 ++-
 2 files changed, 42 insertions(+), 24 deletions(-)

diff --git a/tools/libxl/libxl_create.c b/tools/libxl/libxl_create.c
index 75862dc6ed..09cf99d304 100644
--- a/tools/libxl/libxl_create.c
+++ b/tools/libxl/libxl_create.c
@@ -579,15 +579,7 @@ int libxl__domain_make(libxl__gc *gc, libxl_domain_config *d_config,
                        uint32_t *domid, bool soft_reset)
 {
     libxl_ctx *ctx = libxl__gc_owner(gc);
-    int ret, rc, nb_vm;
-    const char *dom_type;
-    char *uuid_string;
-    char *dom_path, *vm_path, *libxl_path;
-    struct xs_permissions roperm[2];
-    struct xs_permissions rwperm[1];
-    struct xs_permissions noperm[1];
-    xs_transaction_t t = 0;
-    libxl_vminfo *vm_list;
+    int ret, rc;
 
     /* convenience aliases */
     libxl_domain_create_info *info = &d_config->c_info;
@@ -595,12 +587,6 @@ int libxl__domain_make(libxl__gc *gc, libxl_domain_config *d_config,
 
     assert(soft_reset || *domid == INVALID_DOMID);
 
-    uuid_string = libxl__uuid2string(gc, info->uuid);
-    if (!uuid_string) {
-        rc = ERROR_NOMEM;
-        goto out;
-    }
-
     if (!soft_reset) {
         struct xen_domctl_createdomain create = {
             .ssidref = info->ssidref,
@@ -731,7 +717,37 @@ int libxl__domain_make(libxl__gc *gc, libxl_domain_config *d_config,
         goto out;
     }
 
-    dom_path = libxl__xs_get_dompath(gc, *domid);
+    rc = libxl__domain_make_xs_entries(gc, d_config, state, *domid);
+
+ out:
+    return rc;
+}
+
+int libxl__domain_make_xs_entries(libxl__gc *gc, libxl_domain_config *d_config,
+                                  libxl__domain_build_state *state,
+                                  uint32_t domid)
+{
+    libxl_ctx *ctx = libxl__gc_owner(gc);
+    int rc, nb_vm;
+    const char *dom_type;
+    char *uuid_string;
+    char *dom_path, *vm_path, *libxl_path;
+    struct xs_permissions roperm[2];
+    struct xs_permissions rwperm[1];
+    struct xs_permissions noperm[1];
+    xs_transaction_t t = 0;
+    libxl_vminfo *vm_list;
+
+    /* convenience aliases */
+    libxl_domain_create_info *info = &d_config->c_info;
+
+    uuid_string = libxl__uuid2string(gc, info->uuid);
+    if (!uuid_string) {
+        rc = ERROR_NOMEM;
+        goto out;
+    }
+
+    dom_path = libxl__xs_get_dompath(gc, domid);
     if (!dom_path) {
         rc = ERROR_FAIL;
         goto out;
@@ -739,12 +755,12 @@ int libxl__domain_make(libxl__gc *gc, libxl_domain_config *d_config,
 
     vm_path = GCSPRINTF("/vm/%s", uuid_string);
     if (!vm_path) {
-        LOGD(ERROR, *domid, "cannot allocate create paths");
+        LOGD(ERROR, domid, "cannot allocate create paths");
         rc = ERROR_FAIL;
         goto out;
     }
 
-    libxl_path = libxl__xs_libxl_path(gc, *domid);
+    libxl_path = libxl__xs_libxl_path(gc, domid);
     if (!libxl_path) {
         rc = ERROR_FAIL;
         goto out;
@@ -755,10 +771,10 @@ int libxl__domain_make(libxl__gc *gc, libxl_domain_config *d_config,
 
     roperm[0].id = 0;
     roperm[0].perms = XS_PERM_NONE;
-    roperm[1].id = *domid;
+    roperm[1].id = domid;
     roperm[1].perms = XS_PERM_READ;
 
-    rwperm[0].id = *domid;
+    rwperm[0].id = domid;
     rwperm[0].perms = XS_PERM_NONE;
 
 retry_transaction:
@@ -776,7 +792,7 @@ retry_transaction:
                     noperm, ARRAY_SIZE(noperm));
 
     xs_write(ctx->xsh, t, GCSPRINTF("%s/vm", dom_path), vm_path, strlen(vm_path));
-    rc = libxl__domain_rename(gc, *domid, 0, info->name, t);
+    rc = libxl__domain_rename(gc, domid, 0, info->name, t);
     if (rc)
         goto out;
 
@@ -866,7 +882,7 @@ retry_transaction:
 
     vm_list = libxl_list_vm(ctx, &nb_vm);
     if (!vm_list) {
-        LOGD(ERROR, *domid, "cannot get number of running guests");
+        LOGD(ERROR, domid, "cannot get number of running guests");
         rc = ERROR_FAIL;
         goto out;
     }
@@ -890,7 +906,7 @@ retry_transaction:
             t = 0;
             goto retry_transaction;
         }
-        LOGED(ERROR, *domid, "domain creation ""xenstore transaction commit failed");
+        LOGED(ERROR, domid, "domain creation ""xenstore transaction commit failed");
         rc = ERROR_FAIL;
         goto out;
     }
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index c7ece066c4..19b367daca 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -1983,7 +1983,9 @@ _hidden int libxl__domain_make(libxl__gc *gc,
                                libxl_domain_config *d_config,
                                libxl__domain_build_state *state,
                                uint32_t *domid, bool soft_reset);
-
+_hidden int libxl__domain_make_xs_entries(libxl__gc *gc, libxl_domain_config *d_config,
+                                          libxl__domain_build_state *state,
+                                          uint32_t domid);
 _hidden int libxl__domain_build(libxl__gc *gc,
                                 libxl_domain_config *d_config,
                                 uint32_t domid,
-- 
2.25.1

[PATCH v19 for-4.14 04/13] tools/libxl: populate xenstore entries when launching dm for VM fork

[Tamas K Lengyel]

No need to call libxl__domain_make since the domain already exists, only need
to populate the xenstore entries via libxl__domain_make_xs_entries.

Signed-off-by: Tamas K Lengyel <tamas.lengyel@intel.com>
---
 tools/libxl/libxl_create.c  | 11 ++++++++++-
 tools/libxl/libxl_types.idl |  1 +
 2 files changed, 11 insertions(+), 1 deletion(-)

diff --git a/tools/libxl/libxl_create.c b/tools/libxl/libxl_create.c
index 09cf99d304..c3614e5a30 100644
--- a/tools/libxl/libxl_create.c
+++ b/tools/libxl/libxl_create.c
@@ -1244,7 +1244,13 @@ static void initiate_domain_create(libxl__egc *egc,
     ret = libxl__domain_config_setdefault(gc,d_config,domid);
     if (ret) goto error_out;
 
-    ret = libxl__domain_make(gc, d_config, dbs, &domid, dcs->soft_reset);
+    /* If no dm_restore_file is specified we are in the normal path */
+    if (!d_config->dm_restore_file)
+        ret = libxl__domain_make(gc, d_config, dbs, &domid, dcs->soft_reset);
+    else
+        ret = libxl__domain_make_xs_entries(gc, d_config, &dcs->build_state,
+                                            domid);
+
     if (ret) {
         LOGD(ERROR, domid, "cannot make domain: %d", ret);
         dcs->guest_domid = domid;
@@ -2052,6 +2058,9 @@ static int do_domain_create(libxl_ctx *ctx, libxl_domain_config *d_config,
     cdcs->dcs.domid = INVALID_DOMID;
     cdcs->dcs.soft_reset = false;
 
+    if (d_config->dm_restore_file)
+        cdcs->dcs.domid = *domid;
+
     if (cdcs->dcs.restore_params.checkpointed_stream ==
         LIBXL_CHECKPOINTED_STREAM_COLO) {
         cdcs->dcs.colo_proxy_script =
diff --git a/tools/libxl/libxl_types.idl b/tools/libxl/libxl_types.idl
index 9d3f05f399..b9cc139b0a 100644
--- a/tools/libxl/libxl_types.idl
+++ b/tools/libxl/libxl_types.idl
@@ -961,6 +961,7 @@ libxl_domain_config = Struct("domain_config", [
     ("on_watchdog", libxl_action_on_shutdown),
     ("on_crash", libxl_action_on_shutdown),
     ("on_soft_reset", libxl_action_on_shutdown),
+    ("dm_restore_file", string, {'const': True}),
     ], dir=DIR_IN)
 
 libxl_diskinfo = Struct("diskinfo", [
-- 
2.25.1

[PATCH v19 for-4.14 05/13] tools/libxl: Add checks for dm_restore_file

[Tamas K Lengyel]

We can skip a bunch of steps a normal domain creation would entail, similar
to how domain restore & soft_reset skips them.

Signed-off-by: Tamas K Lengyel <tamas.lengyel@intel.com>
---
 tools/libxl/libxl_create.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/tools/libxl/libxl_create.c b/tools/libxl/libxl_create.c
index c3614e5a30..3f0745acc6 100644
--- a/tools/libxl/libxl_create.c
+++ b/tools/libxl/libxl_create.c
@@ -1294,7 +1294,7 @@ static void initiate_domain_create(libxl__egc *egc,
     if (ret)
         goto error_out;
 
-    if (dbs->restore || dcs->soft_reset) {
+    if (dbs->restore || dcs->soft_reset || d_config->dm_restore_file) {
         LOGD(DEBUG, domid, "restoring, not running bootloader");
         domcreate_bootloader_done(egc, &dcs->bl, 0);
     } else  {
@@ -1370,7 +1370,7 @@ static void domcreate_bootloader_done(libxl__egc *egc,
     dcs->sdss.dm.callback = domcreate_devmodel_started;
     dcs->sdss.callback = domcreate_devmodel_started;
 
-    if (restore_fd < 0 && !dcs->soft_reset) {
+    if (restore_fd < 0 && !dcs->soft_reset && !d_config->dm_restore_file) {
         rc = libxl__domain_build(gc, d_config, domid, state);
         domcreate_rebuild_done(egc, dcs, rc);
         return;
-- 
2.25.1

[PATCH v19 for-4.14 06/13] tools/libxl: adjust domcreate_bootloader_done

[Tamas K Lengyel]

Add special handling when only the the device model needs launching for forks.

Signed-off-by: Tamas K Lengyel <tamas.lengyel@intel.com>
---
 tools/libxl/libxl_create.c   | 9 +++++++++
 tools/libxl/libxl_internal.h | 1 +
 2 files changed, 10 insertions(+)

diff --git a/tools/libxl/libxl_create.c b/tools/libxl/libxl_create.c
index 3f0745acc6..ab3ac096ee 100644
--- a/tools/libxl/libxl_create.c
+++ b/tools/libxl/libxl_create.c
@@ -1376,6 +1376,15 @@ static void domcreate_bootloader_done(libxl__egc *egc,
         return;
     }
 
+    if (d_config->dm_restore_file) {
+        dcs->srs.dcs = dcs;
+        dcs->srs.ao = ao;
+        state->forked_vm = true;
+        rc = libxl__domain_build(gc, d_config, domid, state);
+        domcreate_rebuild_done(egc, dcs, rc);
+        return;
+    }
+
     /* Prepare environment for domcreate_stream_done */
     dcs->srs.dcs = dcs;
 
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index 19b367daca..eaae955658 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -1376,6 +1376,7 @@ typedef struct {
 
     char *saved_state;
     int dm_monitor_fd;
+    bool forked_vm;
 
     libxl__file_reference pv_kernel;
     libxl__file_reference pv_ramdisk;
-- 
2.25.1

[PATCH v19 for-4.14 07/13] tools/libxl: Adjust libxl__build_pre

[Tamas K Lengyel]

Skips parts not relevant for VM forks. No functional change in existing code,
only relocating some bits that don't need to be done at the very end.

Signed-off-by: Tamas K Lengyel <tamas.lengyel@intel.com>
---
 tools/libxl/libxl_dom.c | 23 ++++++++++++++---------
 1 file changed, 14 insertions(+), 9 deletions(-)

diff --git a/tools/libxl/libxl_dom.c b/tools/libxl/libxl_dom.c
index dd1aff89a3..1b55097a1a 100644
--- a/tools/libxl/libxl_dom.c
+++ b/tools/libxl/libxl_dom.c
@@ -249,9 +249,12 @@ int libxl__build_pre(libxl__gc *gc, uint32_t domid,
     libxl_domain_build_info *const info = &d_config->b_info;
     libxl_ctx *ctx = libxl__gc_owner(gc);
     char *xs_domid, *con_domid;
-    int rc;
+    int rc = 0;
     uint64_t size;
 
+    if (state->forked_vm)
+        goto skip_fork;
+
     if (xc_domain_max_vcpus(ctx->xch, domid, info->max_vcpus) != 0) {
         LOG(ERROR, "Couldn't set max vcpu count");
         return ERROR_FAIL;
@@ -374,6 +377,16 @@ int libxl__build_pre(libxl__gc *gc, uint32_t domid,
         return ERROR_FAIL;
     }
 
+    if ( (rc = libxl__arch_domain_create(gc, d_config, domid)) )
+        return rc;
+
+    /* Construct a CPUID policy, but only for brand new domains.  Domains
+     * being migrated-in/restored have CPUID handled during the
+     * static_data_done() callback. */
+    if (!state->restore)
+        libxl__cpuid_legacy(ctx, domid, info);
+
+skip_fork:
     xs_domid = xs_read(ctx->xsh, XBT_NULL, "/tool/xenstored/domid", NULL);
     state->store_domid = xs_domid ? atoi(xs_domid) : 0;
     free(xs_domid);
@@ -385,14 +398,6 @@ int libxl__build_pre(libxl__gc *gc, uint32_t domid,
     state->store_port = xc_evtchn_alloc_unbound(ctx->xch, domid, state->store_domid);
     state->console_port = xc_evtchn_alloc_unbound(ctx->xch, domid, state->console_domid);
 
-    rc = libxl__arch_domain_create(gc, d_config, domid);
-
-    /* Construct a CPUID policy, but only for brand new domains.  Domains
-     * being migrated-in/restored have CPUID handled during the
-     * static_data_done() callback. */
-    if (!state->restore)
-        libxl__cpuid_legacy(ctx, domid, info);
-
     return rc;
 }
 
-- 
2.25.1

[PATCH v19 for-4.14 08/13] tools/libxl: Adjust libxl__build_post

[Tamas K Lengyel]

Skips parts not relevant to VM forks.

Signed-off-by: Tamas K Lengyel <tamas.lengyel@intel.com>
---
 tools/libxl/libxl_dom.c | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/tools/libxl/libxl_dom.c b/tools/libxl/libxl_dom.c
index 1b55097a1a..52d49437cc 100644
--- a/tools/libxl/libxl_dom.c
+++ b/tools/libxl/libxl_dom.c
@@ -455,6 +455,9 @@ int libxl__build_post(libxl__gc *gc, uint32_t domid,
     char **ents;
     int i, rc;
 
+    if (state->forked_vm)
+        goto skip_fork;
+
     if (info->num_vnuma_nodes && !info->num_vcpu_soft_affinity) {
         rc = set_vnuma_affinity(gc, domid, info);
         if (rc)
@@ -475,6 +478,7 @@ int libxl__build_post(libxl__gc *gc, uint32_t domid,
         }
     }
 
+skip_fork:
     ents = libxl__calloc(gc, 12 + (info->max_vcpus * 2) + 2, sizeof(char *));
     ents[0] = "memory/static-max";
     ents[1] = GCSPRINTF("%"PRId64, info->max_memkb);
-- 
2.25.1

[PATCH v19 for-4.14 09/13] tools/libxl: libxl__build_hvm_fork

[Tamas K Lengyel]

Add libxl__build_hvm_fork function that performs only the steps needed for VM
forks, skipping a large chunk of libxl__build_hvm.

Signed-off-by: Tamas K Lengyel <tamas.lengyel@intel.com>
---
 tools/libxl/libxl_dom.c | 32 +++++++++++++++++++++++++++++---
 1 file changed, 29 insertions(+), 3 deletions(-)

diff --git a/tools/libxl/libxl_dom.c b/tools/libxl/libxl_dom.c
index 52d49437cc..28117f0907 100644
--- a/tools/libxl/libxl_dom.c
+++ b/tools/libxl/libxl_dom.c
@@ -741,14 +741,15 @@ static int hvm_build_set_params(xc_interface *handle, uint32_t domid,
                                 libxl_domain_build_info *info,
                                 int store_evtchn, unsigned long *store_mfn,
                                 int console_evtchn, unsigned long *console_mfn,
-                                domid_t store_domid, domid_t console_domid)
+                                domid_t store_domid, domid_t console_domid,
+                                bool forked_vm)
 {
     struct hvm_info_table *va_hvm;
     uint8_t *va_map, sum;
     uint64_t str_mfn, cons_mfn;
     int i;
 
-    if (info->type == LIBXL_DOMAIN_TYPE_HVM) {
+    if (info->type == LIBXL_DOMAIN_TYPE_HVM && !forked_vm) {
         va_map = xc_map_foreign_range(handle, domid,
                                       XC_PAGE_SIZE, PROT_READ | PROT_WRITE,
                                       HVM_INFO_PFN);
@@ -1053,6 +1054,28 @@ out:
     return rc;
 }
 
+static int libxl__build_hvm_fork(libxl__gc *gc, uint32_t domid,
+                                 libxl_domain_config *d_config,
+                                 libxl__domain_build_state *state)
+{
+    libxl_ctx *ctx = libxl__gc_owner(gc);
+    libxl_domain_build_info *const info = &d_config->b_info;
+
+    int rc = hvm_build_set_params(ctx->xch, domid, info, state->store_port,
+                                  &state->store_mfn, state->console_port,
+                                  &state->console_mfn, state->store_domid,
+                                  state->console_domid, state->forked_vm);
+
+    if ( rc )
+        return rc;
+
+    return xc_dom_gnttab_seed(ctx->xch, domid, true,
+                              state->console_mfn,
+                              state->store_mfn,
+                              state->console_domid,
+                              state->store_domid);
+}
+
 int libxl__build_hvm(libxl__gc *gc, uint32_t domid,
               libxl_domain_config *d_config,
               libxl__domain_build_state *state)
@@ -1064,6 +1087,9 @@ int libxl__build_hvm(libxl__gc *gc, uint32_t domid,
     struct xc_dom_image *dom = NULL;
     bool device_model = info->type == LIBXL_DOMAIN_TYPE_HVM ? true : false;
 
+    if (state->forked_vm)
+        return libxl__build_hvm_fork(gc, domid, d_config, state);
+
     xc_dom_loginit(ctx->xch);
 
     /*
@@ -1188,7 +1214,7 @@ int libxl__build_hvm(libxl__gc *gc, uint32_t domid,
     rc = hvm_build_set_params(ctx->xch, domid, info, state->store_port,
                                &state->store_mfn, state->console_port,
                                &state->console_mfn, state->store_domid,
-                               state->console_domid);
+                               state->console_domid, false);
     if (rc != 0) {
         LOG(ERROR, "hvm build set params failed");
         goto out;
-- 
2.25.1

[PATCH v19 for-4.14 10/13] tools/libxl: set QEMU saved_state from dm_restore_file

[Tamas K Lengyel]

And make sure we don't remove the file once done.

Signed-off-by: Tamas K Lengyel <tamas.lengyel@intel.com>
---
 tools/libxl/libxl_create.c | 4 ++++
 tools/libxl/libxl_dm.c     | 2 +-
 2 files changed, 5 insertions(+), 1 deletion(-)

diff --git a/tools/libxl/libxl_create.c b/tools/libxl/libxl_create.c
index ab3ac096ee..27f790cae1 100644
--- a/tools/libxl/libxl_create.c
+++ b/tools/libxl/libxl_create.c
@@ -1602,6 +1602,7 @@ static void domcreate_rebuild_done(libxl__egc *egc,
     /* convenience aliases */
     const uint32_t domid = dcs->guest_domid;
     libxl_domain_config *const d_config = dcs->guest_config;
+    libxl__domain_build_state *const state = &dcs->build_state;
 
     if (ret) {
         LOGD(ERROR, domid, "cannot (re-)build domain: %d", ret);
@@ -1609,6 +1610,9 @@ static void domcreate_rebuild_done(libxl__egc *egc,
         goto error_out;
     }
 
+    if (d_config->dm_restore_file)
+        state->saved_state = GCSPRINTF("%s", d_config->dm_restore_file);
+
     store_libxl_entry(gc, domid, &d_config->b_info);
 
     libxl__multidev_begin(ao, &dcs->multidev);
diff --git a/tools/libxl/libxl_dm.c b/tools/libxl/libxl_dm.c
index f2dc5696b9..9b22836e12 100644
--- a/tools/libxl/libxl_dm.c
+++ b/tools/libxl/libxl_dm.c
@@ -3104,7 +3104,7 @@ static void device_model_spawn_outcome(libxl__egc *egc,
 
     libxl__domain_build_state *state = dmss->build_state;
 
-    if (state->saved_state) {
+    if (state->saved_state && !state->forked_vm) {
         ret2 = unlink(state->saved_state);
         if (ret2) {
             LOGED(ERROR, dmss->guest_domid, "%s: failed to remove device-model state %s",
-- 
2.25.1

[PATCH v19 for-4.14 11/13] tools/libxl: Add VM forking public functions

[Tamas K Lengyel]

Signed-off-by: Tamas K Lengyel <tamas.lengyel@intel.com>
---
 tools/libxl/libxl.h        | 10 +++++++++
 tools/libxl/libxl_create.c | 44 ++++++++++++++++++++++++++++++++++++++
 2 files changed, 54 insertions(+)

diff --git a/tools/libxl/libxl.h b/tools/libxl/libxl.h
index 71709dc585..79792d6e29 100644
--- a/tools/libxl/libxl.h
+++ b/tools/libxl/libxl.h
@@ -2704,6 +2704,16 @@ static inline int libxl_qemu_monitor_command_0x041200(libxl_ctx *ctx,
  */
 int libxl_clear_domid_history(libxl_ctx *ctx);
 
+/*
+ * Experimental VM forking functions
+ */
+int libxl_domain_fork_vm(libxl_ctx *ctx, uint32_t pdomid, uint32_t *domid)
+                         LIBXL_EXTERNAL_CALLERS_ONLY;
+
+int libxl_domain_fork_launch_dm(libxl_ctx *ctx, libxl_domain_config *d_config,
+                                uint32_t domid,
+                                const libxl_asyncprogress_how *aop_console_how)
+                                LIBXL_EXTERNAL_CALLERS_ONLY;
 #endif /* LIBXL_H */
 
 /*
diff --git a/tools/libxl/libxl_create.c b/tools/libxl/libxl_create.c
index 27f790cae1..9190e4e263 100644
--- a/tools/libxl/libxl_create.c
+++ b/tools/libxl/libxl_create.c
@@ -2339,6 +2339,50 @@ int libxl_domain_soft_reset(libxl_ctx *ctx,
                                 aop_console_how);
 }
 
+/*
+ * The parent domain is expected to be created with default settings for
+ * - max_evtch_port
+ * - max_grant_frames
+ * - max_maptrack_frames
+ */
+int libxl_domain_fork_vm(libxl_ctx *ctx, uint32_t pdomid, uint32_t *domid)
+{
+    int rc;
+    xc_dominfo_t info;
+    struct xen_domctl_createdomain create = {0};
+
+    if ( 1 != xc_domain_getinfo(ctx->xch, pdomid, 1, &info) )
+        return ERROR_INVAL;
+
+    if ( info.domid != pdomid || !info.hvm || !info.hap )
+        return ERROR_INVAL;
+
+    create.flags |= XEN_DOMCTL_CDF_hvm;
+    create.flags |= XEN_DOMCTL_CDF_hap;
+    create.flags |= XEN_DOMCTL_CDF_oos_off;
+    create.arch.emulation_flags = info.arch_config.emulation_flags;
+    create.ssidref = info.ssidref;
+    create.max_vcpus = info.max_vcpu_id + 1;
+    create.max_evtchn_port = 1023;
+    create.max_grant_frames = LIBXL_MAX_GRANT_FRAMES_DEFAULT;
+    create.max_maptrack_frames = LIBXL_MAX_MAPTRACK_FRAMES_DEFAULT;
+
+    if ( (rc = xc_domain_create(ctx->xch, domid, &create)) )
+        return rc;
+
+    if ( (rc = xc_memshr_fork(ctx->xch, pdomid, *domid, false, false)) )
+        xc_domain_destroy(ctx->xch, *domid);
+
+    return rc;
+}
+
+int libxl_domain_fork_launch_dm(libxl_ctx *ctx, libxl_domain_config *d_config,
+                                uint32_t domid,
+                                const libxl_asyncprogress_how *aop_console_how)
+{
+    return do_domain_create(ctx, d_config, &domid, -1, -1, 0, 0, aop_console_how);
+}
+
 /*
  * Local variables:
  * mode: C
-- 
2.25.1

[PATCH v19 for-4.14 12/13] tools/xl: Add xl fork-vm command

[Tamas K Lengyel]

Adding the xl fork-vm command, compiled only on x86. Only the essential bits
are available via this command to create a fork and launch QEMU for it. The
command still allows to perform the task in a split-model, first creating the
fork and launching QEMU only later.

Signed-off-by: Tamas K Lengyel <tamas.lengyel@intel.com>
---
 tools/xl/Makefile       |   2 +-
 tools/xl/xl.h           |   4 ++
 tools/xl/xl_cmdtable.c  |  13 +++++
 tools/xl/xl_forkvm.c    | 122 ++++++++++++++++++++++++++++++++++++++++
 tools/xl/xl_vmcontrol.c |  13 +++++
 5 files changed, 153 insertions(+), 1 deletion(-)
 create mode 100644 tools/xl/xl_forkvm.c

diff --git a/tools/xl/Makefile b/tools/xl/Makefile
index af4912e67a..073222233b 100644
--- a/tools/xl/Makefile
+++ b/tools/xl/Makefile
@@ -15,7 +15,7 @@ LDFLAGS += $(PTHREAD_LDFLAGS)
 CFLAGS_XL += $(CFLAGS_libxenlight)
 CFLAGS_XL += -Wshadow
 
-XL_OBJS-$(CONFIG_X86) = xl_psr.o
+XL_OBJS-$(CONFIG_X86) = xl_psr.o xl_forkvm.o
 XL_OBJS = xl.o xl_cmdtable.o xl_sxp.o xl_utils.o $(XL_OBJS-y)
 XL_OBJS += xl_parse.o xl_cpupool.o xl_flask.o
 XL_OBJS += xl_vtpm.o xl_block.o xl_nic.o xl_usb.o
diff --git a/tools/xl/xl.h b/tools/xl/xl.h
index 06569c6c4a..4b4442e875 100644
--- a/tools/xl/xl.h
+++ b/tools/xl/xl.h
@@ -50,6 +50,8 @@ struct domain_create {
     int migrate_fd; /* -1 means none */
     int send_back_fd; /* -1 means none */
     char **migration_domname_r; /* from malloc */
+    uint32_t dm_restore_domid; /* restore dm for this domid */
+    const char *dm_restore_file; /* path to dm restore file */
 };
 
 int create_domain(struct domain_create *dom_info);
@@ -131,6 +133,8 @@ int main_restore(int argc, char **argv);
 int main_migrate_receive(int argc, char **argv);
 int main_save(int argc, char **argv);
 int main_migrate(int argc, char **argv);
+int main_fork_vm(int argc, char **argv);
+int main_fork_launch_dm(int argc, char **argv);
 #endif
 int main_dump_core(int argc, char **argv);
 int main_pause(int argc, char **argv);
diff --git a/tools/xl/xl_cmdtable.c b/tools/xl/xl_cmdtable.c
index 08335394e5..523d955317 100644
--- a/tools/xl/xl_cmdtable.c
+++ b/tools/xl/xl_cmdtable.c
@@ -187,6 +187,19 @@ struct cmd_spec cmd_table[] = {
       "Restore a domain from a saved state",
       "- for internal use only",
     },
+#if defined(__i386__) || defined(__x86_64__)
+    { "fork-vm",
+      &main_fork_vm, 0, 1,
+      "Fork a domain from the running parent domid. Experimental. Most config settings must match parent.",
+      "[options] <Domid>",
+      "-h                           Print this help.\n"
+      "-C <config>                  Use config file for VM fork.\n"
+      "-Q <qemu-save-file>          Use qemu save file for VM fork.\n"
+      "--launch-dm <yes|no|late>    Launch device model (QEMU) for VM fork (default yes).\n"
+      "-p                           Do not unpause fork VM fork after operation.\n"
+      "-d                           Enable debug messages.\n"
+    },
+#endif
 #endif
     { "dump-core",
       &main_dump_core, 0, 1,
diff --git a/tools/xl/xl_forkvm.c b/tools/xl/xl_forkvm.c
new file mode 100644
index 0000000000..5ab57ae41b
--- /dev/null
+++ b/tools/xl/xl_forkvm.c
@@ -0,0 +1,122 @@
+/*
+ * Copyright 2020 Intel Corporation
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU Lesser General Public License as published
+ * by the Free Software Foundation; version 2.1 only. with the special
+ * exception on linking described in file LICENSE.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Lesser General Public License for more details.
+ */
+
+#include <fcntl.h>
+#include <inttypes.h>
+#include <stdlib.h>
+#include <sys/stat.h>
+#include <sys/types.h>
+#include <sys/utsname.h>
+#include <time.h>
+#include <unistd.h>
+
+#include <libxl.h>
+#include <libxl_utils.h>
+#include <libxlutil.h>
+
+#include "xl.h"
+#include "xl_utils.h"
+#include "xl_parse.h"
+
+int main_fork_vm(int argc, char **argv)
+{
+    int rc, debug = 0;
+    uint32_t domid_in = INVALID_DOMID, domid_out = INVALID_DOMID;
+    int launch_dm = 1;
+    bool pause = 0;
+    const char *config_file = NULL;
+    const char *dm_restore_file = NULL;
+
+    int opt;
+    static struct option opts[] = {
+        {"launch-dm", 1, 0, 'l'},
+        COMMON_LONG_OPTS
+    };
+
+    SWITCH_FOREACH_OPT(opt, "phdC:Q:l:", opts, "fork-vm", 1) {
+    case 'd':
+        debug = 1;
+        break;
+    case 'p':
+        pause = 1;
+        break;
+    case 'C':
+        config_file = optarg;
+        break;
+    case 'Q':
+        dm_restore_file = optarg;
+        break;
+    case 'l':
+        if ( !strcmp(optarg, "no") )
+            launch_dm = 0;
+        if ( !strcmp(optarg, "yes") )
+            launch_dm = 1;
+        if ( !strcmp(optarg, "late") )
+            launch_dm = 2;
+        break;
+    default:
+        fprintf(stderr, "Unimplemented option(s)\n");
+        return EXIT_FAILURE;
+    }
+
+    if (argc-optind == 1) {
+        domid_in = atoi(argv[optind]);
+    } else {
+        help("fork-vm");
+        return EXIT_FAILURE;
+    }
+
+    if (launch_dm && (!config_file || !dm_restore_file)) {
+        fprintf(stderr, "Currently you must provide both -C and -Q options\n");
+        return EXIT_FAILURE;
+    }
+
+    if (launch_dm == 2) {
+        domid_out = domid_in;
+        rc = EXIT_SUCCESS;
+    } else {
+        rc = libxl_domain_fork_vm(ctx, domid_in, &domid_out);
+    }
+
+    if (rc == EXIT_SUCCESS) {
+        if ( launch_dm ) {
+            struct domain_create dom_info;
+            memset(&dom_info, 0, sizeof(dom_info));
+            dom_info.dm_restore_domid = domid_out;
+            dom_info.dm_restore_file = dm_restore_file;
+            dom_info.debug = debug;
+            dom_info.paused = pause;
+            dom_info.config_file = config_file;
+            dom_info.migrate_fd = -1;
+            dom_info.send_back_fd = -1;
+            rc = create_domain(&dom_info) < 0 ? EXIT_FAILURE : EXIT_SUCCESS;
+        } else if ( !pause )
+            rc = libxl_domain_unpause(ctx, domid_out, NULL);
+    }
+
+    if (rc == EXIT_SUCCESS)
+        fprintf(stderr, "fork-vm command successfully returned domid: %u\n", domid_out);
+    else if ( domid_out != INVALID_DOMID )
+        libxl_domain_destroy(ctx, domid_out, 0);
+
+    return rc;
+}
+
+/*
+ * Local variables:
+ * mode: C
+ * c-basic-offset: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
diff --git a/tools/xl/xl_vmcontrol.c b/tools/xl/xl_vmcontrol.c
index 17b4514c94..508a7c70bb 100644
--- a/tools/xl/xl_vmcontrol.c
+++ b/tools/xl/xl_vmcontrol.c
@@ -676,6 +676,12 @@ int create_domain(struct domain_create *dom_info)
 
     int restoring = (restore_file || (migrate_fd >= 0));
 
+#if defined(__i386__) || defined(__x86_64__)
+    /* VM forking, restore dm for this domain */
+    uint32_t dm_restore_domid = dom_info->dm_restore_domid;
+    const char *dm_restore_file = dom_info->dm_restore_file;
+#endif
+
     libxl_domain_config_init(&d_config);
 
     if (restoring) {
@@ -934,6 +940,13 @@ start:
                                       0, autoconnect_console_how);
         domid = domid_soft_reset;
         domid_soft_reset = INVALID_DOMID;
+#if defined(__i386__) || defined(__x86_64__)
+    } else if (dm_restore_file) {
+        d_config.dm_restore_file = dm_restore_file;
+        ret = libxl_domain_fork_launch_dm(ctx, &d_config, dm_restore_domid,
+                                          autoconnect_console_how);
+        domid = dm_restore_domid;
+#endif
     } else {
         ret = libxl_domain_create_new(ctx, &d_config, &domid,
                                       0, autoconnect_console_how);
-- 
2.25.1

[PATCH v19 for-4.14 13/13] tools/xl: document fork-vm command

[Tamas K Lengyel]

Signed-off-by: Tamas K Lengyel <tamas.lengyel@intel.com>
---
 docs/man/xl.1.pod.in | 39 +++++++++++++++++++++++++++++++++++++++
 1 file changed, 39 insertions(+)

diff --git a/docs/man/xl.1.pod.in b/docs/man/xl.1.pod.in
index 09339282e6..9e87b0314f 100644
--- a/docs/man/xl.1.pod.in
+++ b/docs/man/xl.1.pod.in
@@ -708,6 +708,45 @@ above).
 
 =back
 
+=item B<fork-vm> [I<OPTIONS>] I<domain-id>
+
+Create a fork of a running VM.  The domain will be paused after the operation
+and remains paused while forks of it exist.  Experimental and x86 only.
+Forks can only be made of domains with HAP enabled and on Intel hardware.  The
+parent domain must be created with the xl toolstack and its configuration must
+not manually define max_grant_frames, max_maptrack_frames or max_event_channels.
+
+B<OPTIONS>
+
+=over 4
+
+=item B<-p>
+
+Leave the forked VM paused after creating it.  The parent always remains paused
+while there are forks active from it and that's enforced by the hypervisor.
+
+=item B<--launch-dm>
+
+Specify whether the device model (QEMU) should be launched for the fork.  Late
+launch allows to start the device model for an already running fork previously
+created with "--launch-dm no".
+
+=item B<-C>
+
+The config file to use when launching the device model.  Currently required when
+launching the device model.  Most config settings MUST match the parent domain
+exactly, only change VM name, disk path and network configurations.
+
+=item B<-Q>
+
+The path to the QEMU save file to use when launching the device model.  Currently
+required when launching the device model.  Generate it by connecting to the parent
+domain's QMP socket and issuing:
+ { "execute": "qmp_capabilities" }
+ { "execute": "xen-save-devices-state", "arguments": { "filename": "/path/to/qemu.save", "live": false} }
+
+=back
+
 =item B<sharing> [I<domain-id>]
 
 Display the number of shared pages for a specified domain. If no domain is
-- 
2.25.1

RE: [PATCH v19 for-4.14 00/13] VM forking

[Paul Durrant]

> -----Original Message-----
> From: Xen-devel <xen-devel-bounces@lists.xenproject.org> On Behalf Of Tamas K Lengyel
> Sent: 01 June 2020 14:22
> To: xen-devel@lists.xenproject.org
> Cc: Kevin Tian <kevin.tian@intel.com>; Stefano Stabellini <sstabellini@kernel.org>; Tamas K Lengyel
> <tamas.lengyel@intel.com>; Jun Nakajima <jun.nakajima@intel.com>; Wei Liu <wl@xen.org>; Andrew Cooper
> <andrew.cooper3@citrix.com>; Ian Jackson <ian.jackson@eu.citrix.com>; George Dunlap
> <george.dunlap@citrix.com>; Tamas K Lengyel <tamas@tklengyel.com>; Jan Beulich <jbeulich@suse.com>;
> Anthony PERARD <anthony.perard@citrix.com>; Julien Grall <julien@xen.org>; Roger Pau Monné
> <roger.pau@citrix.com>
> Subject: [PATCH v19 for-4.14 00/13] VM forking

Hi,

  This series looks to be largely un-acked so, since we are now past the freeze date, I don't really think it can go into 4.14. Is there a particular reason that you think it should be considered?

  Paul

> 
> The following patches are part of the series that implement VM forking for
> Intel HVM guests to allow for the fast creation of identical VMs without the
> assosciated high startup costs of booting or restoring the VM from a savefile.
> 
> JIRA issue: https://xenproject.atlassian.net/browse/XEN-89
> 
> The fork operation is implemented as part of the "xl fork-vm" command:
>     xl fork-vm -C <config> -Q <qemu-save-file> <parent_domid>
> 
> By default a fully functional fork is created. The user is in charge however to
> create the appropriate config file for the fork and to generate the QEMU save
> file before the fork-vm call is made. The config file needs to give the
> fork a new name at minimum but other settings may also require changes. Certain
> settings in the config file of both the parent and the fork have to be set to
> default. Details are documented.
> 
> The interface also allows to split the forking into two steps:
>     xl fork-vm --launch-dm no \
>                -m <max-vcpus> \
> 			   -p <parent_domid>
> 	xl fork-vm --launch-dm late \
> 	           -C <config_file_for_fork> \
> 			   -Q <qemu_save_file> \
> 			   <fork_domid>
> 
> The split creation model is useful when the VM needs to be created as fast as
> possible. The forked VM can be unpaused without the device model being launched
> to be monitored and accessed via VMI. Note however that without its device
> model running (depending on what is executing in the VM) it is bound to
> misbehave or even crash when its trying to access devices that would be
> emulated by QEMU. We anticipate that for certain use-cases this would be an
> acceptable situation, in case for example when fuzzing is performed of code
> segments that don't access such devices.
> 
> Launching the device model requires the QEMU Xen savefile to be generated
> manually from the parent VM. This can be accomplished simply by connecting to
> its QMP socket and issuing the "xen-save-devices-state" command. For example
> using the standard tool socat these commands can be used to generate the file:
>     socat - UNIX-CONNECT:/var/run/xen/qmp-libxl-<parent_domid>
> 	{ "execute": "qmp_capabilities" }
> 	{ "execute": "xen-save-devices-state", \
> 		"arguments": { "filename": "/path/to/save/qemu_state", \
> 						"live": false} }
> 
> The series has been tested with Windows VMs and functions as expected. Linux
> VMs when forked from a running VM will have a frozen VNC screen. Linux VMs at
> this time can only be forked with a working device model when the parent VM was
> restored from a snapshot using "xl restore -p". This is a known limitation due
> to Linux VMs having to be made aware of being saved/migrated.
> 
> New in v19:
> 	Including all the patches currently outstanding into the series
> 	Breaking up libxl/xl patch to many sub-patches to make it easier to review
> 	libxl/xl is now reduced to the bare essential to launch QEMU for a VM fork
> 
> Tamas K Lengyel (13):
>   x86/mem_sharing: block interrupt injection for forks
>   tools/libxc: xc_memshr_fork with interrupts blocked
>   tools/libxl: Split libxl__domain_make
>   tools/libxl: populate xenstore entries when launching dm for VM fork
>   tools/libxl: Add checks for dm_restore_file
>   tools/libxl: adjust domcreate_bootloader_done
>   tools/libxl: Adjust libxl__build_pre
>   tools/libxl: Adjust libxl__build_post
>   tools/libxl: libxl__build_hvm_fork
>   tools/libxl: set QEMU saved_state from dm_restore_file
>   tools/libxl: Add VM forking public functions
>   tools/xl: Add xl fork-vm command
>   tools/xl: document fork-vm command
> 
>  docs/man/xl.1.pod.in             |  39 +++++++++
>  tools/libxc/include/xenctrl.h    |   3 +-
>  tools/libxc/xc_memshr.c          |   4 +-
>  tools/libxl/libxl.h              |  10 +++
>  tools/libxl/libxl_create.c       | 134 +++++++++++++++++++++++++------
>  tools/libxl/libxl_dm.c           |   2 +-
>  tools/libxl/libxl_dom.c          |  59 +++++++++++---
>  tools/libxl/libxl_internal.h     |   5 +-
>  tools/libxl/libxl_types.idl      |   1 +
>  tools/xl/Makefile                |   2 +-
>  tools/xl/xl.h                    |   4 +
>  tools/xl/xl_cmdtable.c           |  13 +++
>  tools/xl/xl_forkvm.c             | 122 ++++++++++++++++++++++++++++
>  tools/xl/xl_vmcontrol.c          |  13 +++
>  xen/arch/x86/hvm/vmx/intr.c      |   6 ++
>  xen/arch/x86/mm/mem_sharing.c    |   6 +-
>  xen/include/asm-x86/hvm/domain.h |   2 +-
>  xen/include/public/memory.h      |   3 +
>  18 files changed, 383 insertions(+), 45 deletions(-)
>  create mode 100644 tools/xl/xl_forkvm.c
> 
> --
> 2.25.1
> 

Re: [PATCH v19 for-4.14 00/13] VM forking

[George Dunlap]

> On Jun 1, 2020, at 4:07 PM, Paul Durrant <xadimgnik@gmail.com> wrote:
> 
>> -----Original Message-----
>> From: Xen-devel <xen-devel-bounces@lists.xenproject.org> On Behalf Of Tamas K Lengyel
>> Sent: 01 June 2020 14:22
>> To: xen-devel@lists.xenproject.org
>> Cc: Kevin Tian <kevin.tian@intel.com>; Stefano Stabellini <sstabellini@kernel.org>; Tamas K Lengyel
>> <tamas.lengyel@intel.com>; Jun Nakajima <jun.nakajima@intel.com>; Wei Liu <wl@xen.org>; Andrew Cooper
>> <andrew.cooper3@citrix.com>; Ian Jackson <ian.jackson@eu.citrix.com>; George Dunlap
>> <george.dunlap@citrix.com>; Tamas K Lengyel <tamas@tklengyel.com>; Jan Beulich <jbeulich@suse.com>;
>> Anthony PERARD <anthony.perard@citrix.com>; Julien Grall <julien@xen.org>; Roger Pau Monné
>> <roger.pau@citrix.com>
>> Subject: [PATCH v19 for-4.14 00/13] VM forking
> 
> Hi,
> 
>  This series looks to be largely un-acked so, since we are now past the freeze date, I don't really think it can go into 4.14. Is there a particular reason that you think it should be considered?

Tamas’ project itself mainly uses libxc and below, as I understand; and so getting patches 1 and 2 in would be an important milestone; both have had R-b’s before the feature freeze.  Arguably patches 1 and 2 are a bug fix.  Patch 1 is missing VMX (or a general x86).

The libxl/xl side hasn’t, as I understand it, had significant review; I think that should probably wait until 4.15.

What do you think, Tamas?

 -George

Re: [PATCH v19 for-4.14 00/13] VM forking

[Tamas K Lengyel]

On Mon, Jun 1, 2020 at 11:11 AM George Dunlap <George.Dunlap@citrix.com> wrote:
>
>
>
> > On Jun 1, 2020, at 4:07 PM, Paul Durrant <xadimgnik@gmail.com> wrote:
> >
> >> -----Original Message-----
> >> From: Xen-devel <xen-devel-bounces@lists.xenproject.org> On Behalf Of Tamas K Lengyel
> >> Sent: 01 June 2020 14:22
> >> To: xen-devel@lists.xenproject.org
> >> Cc: Kevin Tian <kevin.tian@intel.com>; Stefano Stabellini <sstabellini@kernel.org>; Tamas K Lengyel
> >> <tamas.lengyel@intel.com>; Jun Nakajima <jun.nakajima@intel.com>; Wei Liu <wl@xen.org>; Andrew Cooper
> >> <andrew.cooper3@citrix.com>; Ian Jackson <ian.jackson@eu.citrix.com>; George Dunlap
> >> <george.dunlap@citrix.com>; Tamas K Lengyel <tamas@tklengyel.com>; Jan Beulich <jbeulich@suse.com>;
> >> Anthony PERARD <anthony.perard@citrix.com>; Julien Grall <julien@xen.org>; Roger Pau Monné
> >> <roger.pau@citrix.com>
> >> Subject: [PATCH v19 for-4.14 00/13] VM forking
> >
> > Hi,
> >
> >  This series looks to be largely un-acked so, since we are now past the freeze date, I don't really think it can go into 4.14. Is there a particular reason that you think it should be considered?
>
> Tamas’ project itself mainly uses libxc and below, as I understand; and so getting patches 1 and 2 in would be an important milestone; both have had R-b’s before the feature freeze.  Arguably patches 1 and 2 are a bug fix.  Patch 1 is missing VMX (or a general x86).

Correct. The first two patches going in would decide whether we will
be able to use the 4.14 release without having to carry out-of-tree
patches. Although as things stand at the moment regarding all the bugs
being discovered in 4.13 and 4.14 we will likely still have to
backport all of these patches to 4.12 by hand.

> The libxl/xl side hasn’t, as I understand it, had significant review; I think that should probably wait until 4.15.

Correct. It has been sent 19 times so far over a period of 9 months
with no feedback from any of the maintainers other then that it's hard
to review. We had some good discussion with other community members
but evidently non of the toolstack maintainers care too much about it.
I made the last-ditch effort to make it easier to review but at this
point we started implementing our own toolstack to interact with VM
forks.

> What do you think, Tamas?

If it's not going into 4.14 then it's going to be dropped. It has been
made solely for the benefit of the community to make the new VM
forking more accessible and useful for others. Without it the only way
to use the feature is to implement your own toolstack. Initially we
were hoping that integrating support to xl/libxl would eliminate the
need for us to implement our own parallel toolstack but since we have
to do that now anyway there is no benefit for us in carrying these
patches any further. It's disheartening we had to resort to that and I
certainly will try to avoid contributing to xl/libxl in the future
since I personally consider it a waste of time.

Thanks,
Tamas

RE: [PATCH v19 for-4.14 00/13] VM forking

[Paul Durrant]

> -----Original Message-----
> From: Tamas K Lengyel <tamas@tklengyel.com>
> Sent: 01 June 2020 19:38
> To: George Dunlap <George.Dunlap@citrix.com>
> Cc: paul@xen.org; Tamas K Lengyel <tamas.lengyel@intel.com>; xen-devel <xen-
> devel@lists.xenproject.org>; Kevin Tian <kevin.tian@intel.com>; Stefano Stabellini
> <sstabellini@kernel.org>; Jun Nakajima <jun.nakajima@intel.com>; Wei Liu <wl@xen.org>; Andrew Cooper
> <Andrew.Cooper3@citrix.com>; Ian Jackson <Ian.Jackson@citrix.com>; Jan Beulich <jbeulich@suse.com>;
> Anthony Perard <anthony.perard@citrix.com>; Julien Grall <julien@xen.org>; Roger Pau Monne
> <roger.pau@citrix.com>
> Subject: Re: [PATCH v19 for-4.14 00/13] VM forking
> 
> On Mon, Jun 1, 2020 at 11:11 AM George Dunlap <George.Dunlap@citrix.com> wrote:
> >
> >
> >
> > > On Jun 1, 2020, at 4:07 PM, Paul Durrant <xadimgnik@gmail.com> wrote:
> > >
> > >> -----Original Message-----
> > >> From: Xen-devel <xen-devel-bounces@lists.xenproject.org> On Behalf Of Tamas K Lengyel
> > >> Sent: 01 June 2020 14:22
> > >> To: xen-devel@lists.xenproject.org
> > >> Cc: Kevin Tian <kevin.tian@intel.com>; Stefano Stabellini <sstabellini@kernel.org>; Tamas K
> Lengyel
> > >> <tamas.lengyel@intel.com>; Jun Nakajima <jun.nakajima@intel.com>; Wei Liu <wl@xen.org>; Andrew
> Cooper
> > >> <andrew.cooper3@citrix.com>; Ian Jackson <ian.jackson@eu.citrix.com>; George Dunlap
> > >> <george.dunlap@citrix.com>; Tamas K Lengyel <tamas@tklengyel.com>; Jan Beulich
> <jbeulich@suse.com>;
> > >> Anthony PERARD <anthony.perard@citrix.com>; Julien Grall <julien@xen.org>; Roger Pau Monné
> > >> <roger.pau@citrix.com>
> > >> Subject: [PATCH v19 for-4.14 00/13] VM forking
> > >
> > > Hi,
> > >
> > >  This series looks to be largely un-acked so, since we are now past the freeze date, I don't
> really think it can go into 4.14. Is there a particular reason that you think it should be considered?
> >
> > Tamas’ project itself mainly uses libxc and below, as I understand; and so getting patches 1 and 2
> in would be an important milestone; both have had R-b’s before the feature freeze.  Arguably patches 1
> and 2 are a bug fix.  Patch 1 is missing VMX (or a general x86).
> 
> Correct. The first two patches going in would decide whether we will
> be able to use the 4.14 release without having to carry out-of-tree
> patches. Although as things stand at the moment regarding all the bugs
> being discovered in 4.13 and 4.14 we will likely still have to
> backport all of these patches to 4.12 by hand.

That sounds like a reasonable justification.

Maintainers/committers, can we please get those patches in a.s.a.p.?

  Paul

RE: [PATCH v19 for-4.14 01/13] x86/mem_sharing: block interrupt injection for forks

[Paul Durrant]

> -----Original Message-----
> From: Xen-devel <xen-devel-bounces@lists.xenproject.org> On Behalf Of Tamas K Lengyel
> Sent: 01 June 2020 14:22
> To: xen-devel@lists.xenproject.org
> Cc: Kevin Tian <kevin.tian@intel.com>; Stefano Stabellini <sstabellini@kernel.org>; Tamas K Lengyel
> <tamas.lengyel@intel.com>; Jun Nakajima <jun.nakajima@intel.com>; Wei Liu <wl@xen.org>; Andrew Cooper
> <andrew.cooper3@citrix.com>; Ian Jackson <ian.jackson@eu.citrix.com>; George Dunlap
> <george.dunlap@citrix.com>; Tamas K Lengyel <tamas@tklengyel.com>; Jan Beulich <jbeulich@suse.com>;
> Julien Grall <julien@xen.org>; Roger Pau Monné <roger.pau@citrix.com>
> Subject: [PATCH v19 for-4.14 01/13] x86/mem_sharing: block interrupt injection for forks
> 
> When running VM forks without device models (QEMU), it may
> be undesirable for Xen to inject interrupts. When creating such forks from
> Windows VMs we have observed the kernel trying to process interrupts
> immediately after the fork is executed. However without QEMU running such
> interrupt handling may not be possible because it may attempt to interact with
> devices that are not emulated by a backend. In the best case scenario such
> interrupt handling would only present a detour in the VM forks' execution
> flow, but in the worst case as we actually observed can completely stall it.
> By disabling interrupt injection a fuzzer can exercise the target code without
> interference. For other use-cases this option probably doesn't make sense,
> that's why this is not enabled by default.
> 
> Forks & memory sharing are only available on Intel CPUs so this only applies
> to vmx. Note that this is part of the experimental VM forking feature that's
> completely disabled by default and can only be enabled by using
> XEN_CONFIG_EXPERT during compile time.
> 
> Signed-off-by: Tamas K Lengyel <tamas.lengyel@intel.com>
> Reviewed-by: Roger Pau Monné <roger.pau@citrix.com>

Release-acked-by: Paul Durrant <paul@xen.org>

> ---
>  xen/arch/x86/hvm/vmx/intr.c      | 6 ++++++
>  xen/arch/x86/mm/mem_sharing.c    | 6 +++++-
>  xen/include/asm-x86/hvm/domain.h | 2 +-
>  xen/include/public/memory.h      | 3 +++
>  4 files changed, 15 insertions(+), 2 deletions(-)
> 
> diff --git a/xen/arch/x86/hvm/vmx/intr.c b/xen/arch/x86/hvm/vmx/intr.c
> index 000e14af49..80bfbb4787 100644
> --- a/xen/arch/x86/hvm/vmx/intr.c
> +++ b/xen/arch/x86/hvm/vmx/intr.c
> @@ -256,6 +256,12 @@ void vmx_intr_assist(void)
>      if ( unlikely(v->arch.vm_event) && v->arch.vm_event->sync_event )
>          return;
> 
> +#ifdef CONFIG_MEM_SHARING
> +    /* Block event injection for VM fork if requested */
> +    if ( unlikely(v->domain->arch.hvm.mem_sharing.block_interrupts) )
> +        return;
> +#endif
> +
>      /* Crank the handle on interrupt state. */
>      pt_vector = pt_update_irq(v);
> 
> diff --git a/xen/arch/x86/mm/mem_sharing.c b/xen/arch/x86/mm/mem_sharing.c
> index 19922ab5d1..c428fd16ce 100644
> --- a/xen/arch/x86/mm/mem_sharing.c
> +++ b/xen/arch/x86/mm/mem_sharing.c
> @@ -2106,7 +2106,8 @@ int mem_sharing_memop(XEN_GUEST_HANDLE_PARAM(xen_mem_sharing_op_t) arg)
>          rc = -EINVAL;
>          if ( mso.u.fork.pad )
>              goto out;
> -        if ( mso.u.fork.flags & ~XENMEM_FORK_WITH_IOMMU_ALLOWED )
> +        if ( mso.u.fork.flags &
> +             ~(XENMEM_FORK_WITH_IOMMU_ALLOWED | XENMEM_FORK_BLOCK_INTERRUPTS) )
>              goto out;
> 
>          rc = rcu_lock_live_remote_domain_by_id(mso.u.fork.parent_domain,
> @@ -2134,6 +2135,9 @@ int mem_sharing_memop(XEN_GUEST_HANDLE_PARAM(xen_mem_sharing_op_t) arg)
>              rc = hypercall_create_continuation(__HYPERVISOR_memory_op,
>                                                 "lh", XENMEM_sharing_op,
>                                                 arg);
> +        else if ( !rc && (mso.u.fork.flags & XENMEM_FORK_BLOCK_INTERRUPTS) )
> +            d->arch.hvm.mem_sharing.block_interrupts = true;
> +
>          rcu_unlock_domain(pd);
>          break;
>      }
> diff --git a/xen/include/asm-x86/hvm/domain.h b/xen/include/asm-x86/hvm/domain.h
> index 95fe18cddc..9d247baf4d 100644
> --- a/xen/include/asm-x86/hvm/domain.h
> +++ b/xen/include/asm-x86/hvm/domain.h
> @@ -67,7 +67,7 @@ struct hvm_ioreq_server {
>  #ifdef CONFIG_MEM_SHARING
>  struct mem_sharing_domain
>  {
> -    bool enabled;
> +    bool enabled, block_interrupts;
> 
>      /*
>       * When releasing shared gfn's in a preemptible manner, recall where
> diff --git a/xen/include/public/memory.h b/xen/include/public/memory.h
> index dbd35305df..850bd72c52 100644
> --- a/xen/include/public/memory.h
> +++ b/xen/include/public/memory.h
> @@ -536,7 +536,10 @@ struct xen_mem_sharing_op {
>          } debug;
>          struct mem_sharing_op_fork {      /* OP_FORK */
>              domid_t parent_domain;        /* IN: parent's domain id */
> +/* Only makes sense for short-lived forks */
>  #define XENMEM_FORK_WITH_IOMMU_ALLOWED (1u << 0)
> +/* Only makes sense for short-lived forks */
> +#define XENMEM_FORK_BLOCK_INTERRUPTS   (1u << 1)
>              uint16_t flags;               /* IN: optional settings */
>              uint32_t pad;                 /* Must be set to 0 */
>          } fork;
> --
> 2.25.1
> 

RE: [PATCH v19 for-4.14 02/13] tools/libxc: xc_memshr_fork with interrupts blocked

[Paul Durrant]

> -----Original Message-----
> From: Xen-devel <xen-devel-bounces@lists.xenproject.org> On Behalf Of Tamas K Lengyel
> Sent: 01 June 2020 14:22
> To: xen-devel@lists.xenproject.org
> Cc: Ian Jackson <ian.jackson@eu.citrix.com>; Tamas K Lengyel <tamas.lengyel@intel.com>; Wei Liu
> <wl@xen.org>; Roger Pau Monné <roger.pau@citrix.com>
> Subject: [PATCH v19 for-4.14 02/13] tools/libxc: xc_memshr_fork with interrupts blocked
> 
> Toolstack side for creating forks with interrupt injection blocked.
> 
> Signed-off-by: Tamas K Lengyel <tamas.lengyel@intel.com>
> Reviewed-by: Roger Pau Monné <roger.pau@citrix.com>
> Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>

Release-acked-by: Paul Durrant <paul@xen.org>

> ---
>  tools/libxc/include/xenctrl.h | 3 ++-
>  tools/libxc/xc_memshr.c       | 4 +++-
>  2 files changed, 5 insertions(+), 2 deletions(-)
> 
> diff --git a/tools/libxc/include/xenctrl.h b/tools/libxc/include/xenctrl.h
> index f9e17ae424..5eeee1de46 100644
> --- a/tools/libxc/include/xenctrl.h
> +++ b/tools/libxc/include/xenctrl.h
> @@ -2241,7 +2241,8 @@ int xc_memshr_range_share(xc_interface *xch,
>  int xc_memshr_fork(xc_interface *xch,
>                     uint32_t source_domain,
>                     uint32_t client_domain,
> -                   bool allow_with_iommu);
> +                   bool allow_with_iommu,
> +                   bool block_interrupts);
> 
>  /*
>   * Note: this function is only intended to be used on short-lived forks that
> diff --git a/tools/libxc/xc_memshr.c b/tools/libxc/xc_memshr.c
> index 2300cc7075..a6cfd7dccf 100644
> --- a/tools/libxc/xc_memshr.c
> +++ b/tools/libxc/xc_memshr.c
> @@ -240,7 +240,7 @@ int xc_memshr_debug_gref(xc_interface *xch,
>  }
> 
>  int xc_memshr_fork(xc_interface *xch, uint32_t pdomid, uint32_t domid,
> -                   bool allow_with_iommu)
> +                   bool allow_with_iommu, bool block_interrupts)
>  {
>      xen_mem_sharing_op_t mso;
> 
> @@ -251,6 +251,8 @@ int xc_memshr_fork(xc_interface *xch, uint32_t pdomid, uint32_t domid,
> 
>      if ( allow_with_iommu )
>          mso.u.fork.flags |= XENMEM_FORK_WITH_IOMMU_ALLOWED;
> +    if ( block_interrupts )
> +        mso.u.fork.flags |= XENMEM_FORK_BLOCK_INTERRUPTS;
> 
>      return xc_memshr_memop(xch, domid, &mso);
>  }
> --
> 2.25.1
> 

Re: [PATCH v19 for-4.14 00/13] VM forking

[Jan Beulich]

On 02.06.2020 09:37, Paul Durrant wrote:
> Maintainers/committers, can we please get those patches in a.s.a.p.?

The sole reason I didn't put in at least the 1st patch on Friday
(perhaps also the 2nd, as it is suitably ack-ed) was that it's
still missing a VMX maintainer ack, and Kevin had provided
comments on v2 of the smaller (2-patch) series.

Jan

Re: [PATCH v19 for-4.14 01/13] x86/mem_sharing: block interrupt injection for forks

[Wei Liu]

On Mon, Jun 01, 2020 at 06:21:35AM -0700, Tamas K Lengyel wrote:
> When running VM forks without device models (QEMU), it may
> be undesirable for Xen to inject interrupts. When creating such forks from
> Windows VMs we have observed the kernel trying to process interrupts
> immediately after the fork is executed. However without QEMU running such
> interrupt handling may not be possible because it may attempt to interact with
> devices that are not emulated by a backend. In the best case scenario such
> interrupt handling would only present a detour in the VM forks' execution
> flow, but in the worst case as we actually observed can completely stall it.
> By disabling interrupt injection a fuzzer can exercise the target code without
> interference. For other use-cases this option probably doesn't make sense,
> that's why this is not enabled by default.
> 
> Forks & memory sharing are only available on Intel CPUs so this only applies
> to vmx. Note that this is part of the experimental VM forking feature that's
> completely disabled by default and can only be enabled by using
> XEN_CONFIG_EXPERT during compile time.
> 
> Signed-off-by: Tamas K Lengyel <tamas.lengyel@intel.com>
> Reviewed-by: Roger Pau Monné <roger.pau@citrix.com>

Reviewed-by: Wei Liu <wl@xen.org>

Re: [PATCH v19 for-4.14 02/13] tools/libxc: xc_memshr_fork with interrupts blocked

[Wei Liu]

On Mon, Jun 01, 2020 at 06:21:36AM -0700, Tamas K Lengyel wrote:
> Toolstack side for creating forks with interrupt injection blocked.
> 
> Signed-off-by: Tamas K Lengyel <tamas.lengyel@intel.com>
> Reviewed-by: Roger Pau Monné <roger.pau@citrix.com>
> Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>

Acked-by: Wei Liu <wl@xen.org>

RE: [PATCH v19 for-4.14 01/13] x86/mem_sharing: block interrupt injection for forks

[Tian, Kevin]

> From: Lengyel, Tamas <tamas.lengyel@intel.com>
> Sent: Monday, June 1, 2020 9:22 PM
> 
> When running VM forks without device models (QEMU), it may
> be undesirable for Xen to inject interrupts. When creating such forks from
> Windows VMs we have observed the kernel trying to process interrupts
> immediately after the fork is executed. However without QEMU running such
> interrupt handling may not be possible because it may attempt to interact
> with
> devices that are not emulated by a backend. In the best case scenario such

I asked this question before. the interrupts could come from Xen itself, e.g.
due to timer virtualization. So I didn't get why it's desired to block all interrupts
just because no QEMU is running. Also it's weird why Windows VMs are
observed to process interrupts that are generated by QEMU when no such
backend emulation exists at all. It sounds like a workaround instead of a real
fix...


> interrupt handling would only present a detour in the VM forks' execution
> flow, but in the worst case as we actually observed can completely stall it.
> By disabling interrupt injection a fuzzer can exercise the target code without
> interference. For other use-cases this option probably doesn't make sense,
> that's why this is not enabled by default.
> 
> Forks & memory sharing are only available on Intel CPUs so this only applies
> to vmx. Note that this is part of the experimental VM forking feature that's
> completely disabled by default and can only be enabled by using
> XEN_CONFIG_EXPERT during compile time.
> 
> Signed-off-by: Tamas K Lengyel <tamas.lengyel@intel.com>
> Reviewed-by: Roger Pau Monné <roger.pau@citrix.com>
> ---
>  xen/arch/x86/hvm/vmx/intr.c      | 6 ++++++
>  xen/arch/x86/mm/mem_sharing.c    | 6 +++++-
>  xen/include/asm-x86/hvm/domain.h | 2 +-
>  xen/include/public/memory.h      | 3 +++
>  4 files changed, 15 insertions(+), 2 deletions(-)
> 
> diff --git a/xen/arch/x86/hvm/vmx/intr.c b/xen/arch/x86/hvm/vmx/intr.c
> index 000e14af49..80bfbb4787 100644
> --- a/xen/arch/x86/hvm/vmx/intr.c
> +++ b/xen/arch/x86/hvm/vmx/intr.c
> @@ -256,6 +256,12 @@ void vmx_intr_assist(void)
>      if ( unlikely(v->arch.vm_event) && v->arch.vm_event->sync_event )
>          return;
> 
> +#ifdef CONFIG_MEM_SHARING
> +    /* Block event injection for VM fork if requested */
> +    if ( unlikely(v->domain->arch.hvm.mem_sharing.block_interrupts) )
> +        return;
> +#endif
> +
>      /* Crank the handle on interrupt state. */
>      pt_vector = pt_update_irq(v);
> 
> diff --git a/xen/arch/x86/mm/mem_sharing.c
> b/xen/arch/x86/mm/mem_sharing.c
> index 19922ab5d1..c428fd16ce 100644
> --- a/xen/arch/x86/mm/mem_sharing.c
> +++ b/xen/arch/x86/mm/mem_sharing.c
> @@ -2106,7 +2106,8 @@ int
> mem_sharing_memop(XEN_GUEST_HANDLE_PARAM(xen_mem_sharing_op
> _t) arg)
>          rc = -EINVAL;
>          if ( mso.u.fork.pad )
>              goto out;
> -        if ( mso.u.fork.flags & ~XENMEM_FORK_WITH_IOMMU_ALLOWED )
> +        if ( mso.u.fork.flags &
> +             ~(XENMEM_FORK_WITH_IOMMU_ALLOWED |
> XENMEM_FORK_BLOCK_INTERRUPTS) )
>              goto out;
> 
>          rc = rcu_lock_live_remote_domain_by_id(mso.u.fork.parent_domain,
> @@ -2134,6 +2135,9 @@ int
> mem_sharing_memop(XEN_GUEST_HANDLE_PARAM(xen_mem_sharing_op
> _t) arg)
>              rc = hypercall_create_continuation(__HYPERVISOR_memory_op,
>                                                 "lh", XENMEM_sharing_op,
>                                                 arg);
> +        else if ( !rc && (mso.u.fork.flags &
> XENMEM_FORK_BLOCK_INTERRUPTS) )
> +            d->arch.hvm.mem_sharing.block_interrupts = true;
> +
>          rcu_unlock_domain(pd);
>          break;
>      }
> diff --git a/xen/include/asm-x86/hvm/domain.h b/xen/include/asm-
> x86/hvm/domain.h
> index 95fe18cddc..9d247baf4d 100644
> --- a/xen/include/asm-x86/hvm/domain.h
> +++ b/xen/include/asm-x86/hvm/domain.h
> @@ -67,7 +67,7 @@ struct hvm_ioreq_server {
>  #ifdef CONFIG_MEM_SHARING
>  struct mem_sharing_domain
>  {
> -    bool enabled;
> +    bool enabled, block_interrupts;
> 
>      /*
>       * When releasing shared gfn's in a preemptible manner, recall where
> diff --git a/xen/include/public/memory.h b/xen/include/public/memory.h
> index dbd35305df..850bd72c52 100644
> --- a/xen/include/public/memory.h
> +++ b/xen/include/public/memory.h
> @@ -536,7 +536,10 @@ struct xen_mem_sharing_op {
>          } debug;
>          struct mem_sharing_op_fork {      /* OP_FORK */
>              domid_t parent_domain;        /* IN: parent's domain id */
> +/* Only makes sense for short-lived forks */
>  #define XENMEM_FORK_WITH_IOMMU_ALLOWED (1u << 0)
> +/* Only makes sense for short-lived forks */
> +#define XENMEM_FORK_BLOCK_INTERRUPTS   (1u << 1)
>              uint16_t flags;               /* IN: optional settings */
>              uint32_t pad;                 /* Must be set to 0 */
>          } fork;
> --
> 2.25.1

RE: [PATCH v19 for-4.14 01/13] x86/mem_sharing: block interrupt injection for forks

[Tian, Kevin]

> From: Tian, Kevin
> Sent: Wednesday, June 10, 2020 7:44 AM
> 
> > From: Lengyel, Tamas <tamas.lengyel@intel.com>
> > Sent: Monday, June 1, 2020 9:22 PM
> >
> > When running VM forks without device models (QEMU), it may
> > be undesirable for Xen to inject interrupts. When creating such forks from
> > Windows VMs we have observed the kernel trying to process interrupts
> > immediately after the fork is executed. However without QEMU running
> such
> > interrupt handling may not be possible because it may attempt to interact
> > with
> > devices that are not emulated by a backend. In the best case scenario such
> 
> I asked this question before. the interrupts could come from Xen itself, e.g.
> due to timer virtualization. So I didn't get why it's desired to block all
> interrupts
> just because no QEMU is running. Also it's weird why Windows VMs are
> observed to process interrupts that are generated by QEMU when no such
> backend emulation exists at all. It sounds like a workaround instead of a real
> fix...

ok, I rechecked your reply. Looks it's about the case that parent VM has QEMU
and pending interrupts while you fork it into child VMs without QEMU so those
pending interrupts become problematic.

Reviewed-by: Kevin Tian <kevin.tian@intel.com>

> 
> 
> > interrupt handling would only present a detour in the VM forks' execution
> > flow, but in the worst case as we actually observed can completely stall it.
> > By disabling interrupt injection a fuzzer can exercise the target code
> without
> > interference. For other use-cases this option probably doesn't make sense,
> > that's why this is not enabled by default.
> >
> > Forks & memory sharing are only available on Intel CPUs so this only
> applies
> > to vmx. Note that this is part of the experimental VM forking feature that's
> > completely disabled by default and can only be enabled by using
> > XEN_CONFIG_EXPERT during compile time.
> >
> > Signed-off-by: Tamas K Lengyel <tamas.lengyel@intel.com>
> > Reviewed-by: Roger Pau Monné <roger.pau@citrix.com>
> > ---
> >  xen/arch/x86/hvm/vmx/intr.c      | 6 ++++++
> >  xen/arch/x86/mm/mem_sharing.c    | 6 +++++-
> >  xen/include/asm-x86/hvm/domain.h | 2 +-
> >  xen/include/public/memory.h      | 3 +++
> >  4 files changed, 15 insertions(+), 2 deletions(-)
> >
> > diff --git a/xen/arch/x86/hvm/vmx/intr.c b/xen/arch/x86/hvm/vmx/intr.c
> > index 000e14af49..80bfbb4787 100644
> > --- a/xen/arch/x86/hvm/vmx/intr.c
> > +++ b/xen/arch/x86/hvm/vmx/intr.c
> > @@ -256,6 +256,12 @@ void vmx_intr_assist(void)
> >      if ( unlikely(v->arch.vm_event) && v->arch.vm_event->sync_event )
> >          return;
> >
> > +#ifdef CONFIG_MEM_SHARING
> > +    /* Block event injection for VM fork if requested */
> > +    if ( unlikely(v->domain->arch.hvm.mem_sharing.block_interrupts) )
> > +        return;
> > +#endif
> > +
> >      /* Crank the handle on interrupt state. */
> >      pt_vector = pt_update_irq(v);
> >
> > diff --git a/xen/arch/x86/mm/mem_sharing.c
> > b/xen/arch/x86/mm/mem_sharing.c
> > index 19922ab5d1..c428fd16ce 100644
> > --- a/xen/arch/x86/mm/mem_sharing.c
> > +++ b/xen/arch/x86/mm/mem_sharing.c
> > @@ -2106,7 +2106,8 @@ int
> >
> mem_sharing_memop(XEN_GUEST_HANDLE_PARAM(xen_mem_sharing_op
> > _t) arg)
> >          rc = -EINVAL;
> >          if ( mso.u.fork.pad )
> >              goto out;
> > -        if ( mso.u.fork.flags & ~XENMEM_FORK_WITH_IOMMU_ALLOWED )
> > +        if ( mso.u.fork.flags &
> > +             ~(XENMEM_FORK_WITH_IOMMU_ALLOWED |
> > XENMEM_FORK_BLOCK_INTERRUPTS) )
> >              goto out;
> >
> >          rc = rcu_lock_live_remote_domain_by_id(mso.u.fork.parent_domain,
> > @@ -2134,6 +2135,9 @@ int
> >
> mem_sharing_memop(XEN_GUEST_HANDLE_PARAM(xen_mem_sharing_op
> > _t) arg)
> >              rc = hypercall_create_continuation(__HYPERVISOR_memory_op,
> >                                                 "lh", XENMEM_sharing_op,
> >                                                 arg);
> > +        else if ( !rc && (mso.u.fork.flags &
> > XENMEM_FORK_BLOCK_INTERRUPTS) )
> > +            d->arch.hvm.mem_sharing.block_interrupts = true;
> > +
> >          rcu_unlock_domain(pd);
> >          break;
> >      }
> > diff --git a/xen/include/asm-x86/hvm/domain.h b/xen/include/asm-
> > x86/hvm/domain.h
> > index 95fe18cddc..9d247baf4d 100644
> > --- a/xen/include/asm-x86/hvm/domain.h
> > +++ b/xen/include/asm-x86/hvm/domain.h
> > @@ -67,7 +67,7 @@ struct hvm_ioreq_server {
> >  #ifdef CONFIG_MEM_SHARING
> >  struct mem_sharing_domain
> >  {
> > -    bool enabled;
> > +    bool enabled, block_interrupts;
> >
> >      /*
> >       * When releasing shared gfn's in a preemptible manner, recall where
> > diff --git a/xen/include/public/memory.h b/xen/include/public/memory.h
> > index dbd35305df..850bd72c52 100644
> > --- a/xen/include/public/memory.h
> > +++ b/xen/include/public/memory.h
> > @@ -536,7 +536,10 @@ struct xen_mem_sharing_op {
> >          } debug;
> >          struct mem_sharing_op_fork {      /* OP_FORK */
> >              domid_t parent_domain;        /* IN: parent's domain id */
> > +/* Only makes sense for short-lived forks */
> >  #define XENMEM_FORK_WITH_IOMMU_ALLOWED (1u << 0)
> > +/* Only makes sense for short-lived forks */
> > +#define XENMEM_FORK_BLOCK_INTERRUPTS   (1u << 1)
> >              uint16_t flags;               /* IN: optional settings */
> >              uint32_t pad;                 /* Must be set to 0 */
> >          } fork;
> > --
> > 2.25.1

Re: [PATCH v19 for-4.14 01/13] x86/mem_sharing: block interrupt injection for forks

[Tamas K Lengyel]

On Tue, Jun 9, 2020 at 5:53 PM Tian, Kevin <kevin.tian@intel.com> wrote:
>
> > From: Tian, Kevin
> > Sent: Wednesday, June 10, 2020 7:44 AM
> >
> > > From: Lengyel, Tamas <tamas.lengyel@intel.com>
> > > Sent: Monday, June 1, 2020 9:22 PM
> > >
> > > When running VM forks without device models (QEMU), it may
> > > be undesirable for Xen to inject interrupts. When creating such forks from
> > > Windows VMs we have observed the kernel trying to process interrupts
> > > immediately after the fork is executed. However without QEMU running
> > such
> > > interrupt handling may not be possible because it may attempt to interact
> > > with
> > > devices that are not emulated by a backend. In the best case scenario such
> >
> > I asked this question before. the interrupts could come from Xen itself, e.g.
> > due to timer virtualization. So I didn't get why it's desired to block all
> > interrupts
> > just because no QEMU is running. Also it's weird why Windows VMs are
> > observed to process interrupts that are generated by QEMU when no such
> > backend emulation exists at all. It sounds like a workaround instead of a real
> > fix...
>
> ok, I rechecked your reply. Looks it's about the case that parent VM has QEMU
> and pending interrupts while you fork it into child VMs without QEMU so those
> pending interrupts become problematic.
>
> Reviewed-by: Kevin Tian <kevin.tian@intel.com>

HI Kevin,
thanks! That's the case but really we just want to block all
interrupts irrespective of where the are coming from. The fork VMs are
being reset hundreds or thousands of times per second during fuzzing,
so there is no point in injecting any interrupt at all in that
particular use-case.

Tamas

Re: [PATCH v19 for-4.14 00/13] VM forking

[Tamas K Lengyel]

On Tue, Jun 2, 2020 at 3:39 AM Jan Beulich <jbeulich@suse.com> wrote:
>
> On 02.06.2020 09:37, Paul Durrant wrote:
> > Maintainers/committers, can we please get those patches in a.s.a.p.?
>
> The sole reason I didn't put in at least the 1st patch on Friday
> (perhaps also the 2nd, as it is suitably ack-ed) was that it's
> still missing a VMX maintainer ack, and Kevin had provided
> comments on v2 of the smaller (2-patch) series.

Can we get the first patches from this series merged now with Kevin's
Reviewed-by he sent last week
(https://lists.xenproject.org/archives/html/xen-devel/2020-06/msg00732.html)?

Thanks,
Tamas

Re: [PATCH v19 for-4.14 00/13] VM forking

[Jan Beulich]

On 15.06.2020 21:27, Tamas K Lengyel wrote:
> On Tue, Jun 2, 2020 at 3:39 AM Jan Beulich <jbeulich@suse.com> wrote:
>>
>> On 02.06.2020 09:37, Paul Durrant wrote:
>>> Maintainers/committers, can we please get those patches in a.s.a.p.?
>>
>> The sole reason I didn't put in at least the 1st patch on Friday
>> (perhaps also the 2nd, as it is suitably ack-ed) was that it's
>> still missing a VMX maintainer ack, and Kevin had provided
>> comments on v2 of the smaller (2-patch) series.
> 
> Can we get the first patches from this series merged now with Kevin's
> Reviewed-by he sent last week
> (https://lists.xenproject.org/archives/html/xen-devel/2020-06/msg00732.html)?

Please can you be a little more patient? I've been out of the
office until yesterday afternoon, and I'm dealing with half
broken email now.

Jan

Re: [PATCH v19 for-4.14 00/13] VM forking

[Jan Beulich]

On 01.06.2020 15:21, Tamas K Lengyel wrote:
> Tamas K Lengyel (13):
>   x86/mem_sharing: block interrupt injection for forks
>   tools/libxc: xc_memshr_fork with interrupts blocked

I've committed these two, and I'll leave the rest to the tool stack
maintainers.

Jan

Re: [PATCH v19 for-4.14 00/13] VM forking

[Tamas K Lengyel]

On Tue, Jun 16, 2020 at 2:32 AM Jan Beulich <jbeulich@suse.com> wrote:
>
> On 01.06.2020 15:21, Tamas K Lengyel wrote:
> > Tamas K Lengyel (13):
> >   x86/mem_sharing: block interrupt injection for forks
> >   tools/libxc: xc_memshr_fork with interrupts blocked
>
> I've committed these two, and I'll leave the rest to the tool stack
> maintainers.

Thanks!

Tamas