[PATCH v11 00/11] toolstack-assisted approach to PVHVM guest kexec

[PATCH v11 00/11] toolstack-assisted approach to PVHVM guest kexec

[Vitaly Kuznetsov]

This patch series provides x86 PVHVM domains with an ability to perform
kexec/kdump-style operations.

Changes since v10 (also mentioned in individual patches) grouped by reviewer's
name:
[Wei Liu]
- PATCH 02: add Acked-by
- PATCH 11:
 - tested stubdom case (and fix minor glitches).
 - coding style fixes,  minor description change in xl.cfg.pod.5

[Konrad Rzeszutek Wilk]
- PATCH 02: add Reviewed-by:
- PATCH 06: return when evtchn_reset() fails
- PATCH 08: add Reviewed-by:

[Jan Beulich]
- PATCH 06: return when evtchn_reset() fails
- PATCH 08: add Acked-by:, minor code style change

[Ian Campbell]
- PATCH 08: add Acked-by: (ARM part)

Currently non-acked patch is only PATCH 11 (libxl).

Original description:

The list of currently known issues blocking kexec (and why the series is
required) is:
- Bound event channels.
- Registered vcpu_info.
- PIRQ/emuirq mappings.
- shared_info frame after XENMAPSPACE_shared_info operation.
- Active grant mappings.

Previously there were several attempts to solve these issues in different ways:
- Individual 'shutdown' hypercalls (e.g. VCPUOP_reset_vcpu_info) (we agreed
  that one 'reset everything' hypercall is better).
- Try building new domain reassigning old domain's memory (memory reassignment
  turned out being too cumbersome).
- Toolstack-less 'reset everything' (turned out being impossible because there
  is not enough knowledge in the hypervisor, e.g. interdomain channels are
  being set up by the toolstack).

This series is a mix of the previously sent 'toolstack-based' and
'reset everything' series. Here are some key points:
- No new domain is created.
- Domain is asking for soft reset with SCHEDOP_shutdown with
  SHUTDOWN_soft_reset shutdown reason.
- XEN_DOMCTL_soft_reset is being called by the toolstack.
- Device model is being restarted.

With regards to active grants. In this series we restart domain's device model,
remove it from xenstore and introduce it back. The only 'misbehaving' backend
keeping active mapping I currently observe is xenconsoled: currently it has no
interface to disconnect from a domain (it just periodically scans all domains
to determine if any of them are dead). This is not an issue for us because:
- This matches standard domain startup as this grant mapping is being set up by
  the toolstack.
- Guest domain is aware of this special page.
grant_table_warn_active_grants() is required to find possible misbehaving
backends in future.

v10 of the 'toolstack-assisted approach to pvhvm guest kexec' is available here:
http://lists.xen.org/archives/html/xen-devel/2015-07/msg05054.html

Vitaly Kuznetsov (11):
  xen: introduce SHUTDOWN_soft_reset shutdown reason
  libxl: support SHUTDOWN_soft_reset shutdown reason
  xl: introduce enum domain_restart_type
  xen: evtchn: make evtchn_reset() ready for soft reset
  xen: grant_table: implement grant_table_warn_active_grants()
  xen: Introduce XEN_DOMCTL_soft_reset
  flask: DOMCTL_soft_reset support
  xen: arch-specific hooks for domain_soft_reset()
  libxc: support XEN_DOMCTL_soft_reset operation
  libxl: add LIBXL_DEVICE_MODEL_SAVE_FILE
  (lib)xl: soft reset support

 docs/man/xl.cfg.pod.5                        |  11 ++
 tools/flask/policy/policy/modules/xen/xen.if |   2 +-
 tools/libxc/include/xenctrl.h                |   3 +
 tools/libxc/xc_domain.c                      |   9 ++
 tools/libxl/libxl.c                          |  22 ++-
 tools/libxl/libxl.h                          |  15 +++
 tools/libxl/libxl_create.c                   | 192 ++++++++++++++++++++++++---
 tools/libxl/libxl_dm.c                       |   2 +-
 tools/libxl/libxl_internal.h                 |   5 +
 tools/libxl/libxl_types.idl                  |   4 +
 tools/libxl/xl.h                             |   7 +
 tools/libxl/xl_cmdimpl.c                     |  58 +++++---
 tools/python/xen/lowlevel/xl/xl.c            |   1 +
 xen/arch/arm/domain.c                        |   5 +
 xen/arch/x86/domain.c                        |  81 +++++++++++
 xen/arch/x86/hvm/hvm.c                       |   5 +
 xen/common/domain.c                          |  53 ++++++--
 xen/common/domctl.c                          |   9 ++
 xen/common/event_channel.c                   |  43 +++---
 xen/common/grant_table.c                     |  35 +++++
 xen/common/shutdown.c                        |   6 +
 xen/include/asm-x86/hvm/hvm.h                |   1 +
 xen/include/public/domctl.h                  |   1 +
 xen/include/public/sched.h                   |  11 +-
 xen/include/xen/domain.h                     |   2 +
 xen/include/xen/event.h                      |   3 +
 xen/include/xen/grant_table.h                |   5 +
 xen/include/xen/sched.h                      |   2 +
 xen/xsm/flask/hooks.c                        |   3 +
 xen/xsm/flask/policy/access_vectors          |   2 +
 30 files changed, 524 insertions(+), 74 deletions(-)

-- 
2.4.3

[PATCH v11 01/11] xen: introduce SHUTDOWN_soft_reset shutdown reason

[Vitaly Kuznetsov]

This special type of shutdown is supposed to be used by PVHVM guests when
they want to perform some sort of kexec/kdump.

Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
Acked-by: Jan Beulich <jbeulich@suse.com>
Reviewed-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
---
Changes since v10:
- None
---
 xen/common/shutdown.c      |  6 ++++++
 xen/include/public/sched.h | 11 ++++++++++-
 2 files changed, 16 insertions(+), 1 deletion(-)

diff --git a/xen/common/shutdown.c b/xen/common/shutdown.c
index 9cfbf7a..03a8641 100644
--- a/xen/common/shutdown.c
+++ b/xen/common/shutdown.c
@@ -66,6 +66,12 @@ void hwdom_shutdown(u8 reason)
         machine_restart(0);
         break; /* not reached */
 
+    case SHUTDOWN_soft_reset:
+        printk("Hardware domain %d did unsupported soft reset, rebooting.\n",
+               hardware_domain->domain_id);
+        machine_restart(0);
+        break; /* not reached */
+
     default:
         printk("Hardware Dom%u shutdown (unknown reason %u): ",
                hardware_domain->domain_id, reason);
diff --git a/xen/include/public/sched.h b/xen/include/public/sched.h
index 4000ac9..2219696 100644
--- a/xen/include/public/sched.h
+++ b/xen/include/public/sched.h
@@ -159,7 +159,16 @@ DEFINE_XEN_GUEST_HANDLE(sched_watchdog_t);
 #define SHUTDOWN_suspend    2  /* Clean up, save suspend info, kill.         */
 #define SHUTDOWN_crash      3  /* Tell controller we've crashed.             */
 #define SHUTDOWN_watchdog   4  /* Restart because watchdog time expired.     */
-#define SHUTDOWN_MAX        4  /* Maximum valid shutdown reason.             */
+
+/*
+ * Domain asked to perform 'soft reset' for it. The expected behavior is to
+ * reset internal Xen state for the domain returning it to the point where it
+ * was created but leaving the domain's memory contents and vCPU contexts
+ * intact. This will allow the domain to start over and set up all Xen specific
+ * interfaces again.
+ */
+#define SHUTDOWN_soft_reset 5
+#define SHUTDOWN_MAX        5  /* Maximum valid shutdown reason.             */
 /* ` } */
 
 #endif /* __XEN_PUBLIC_SCHED_H__ */
-- 
2.4.3

[PATCH v11 02/11] libxl: support SHUTDOWN_soft_reset shutdown reason

[Vitaly Kuznetsov]

Use letter 'S' to indicate a domain in such state. Introduce new
'on_soft_reset' action and default it to 'restart' for now.

Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
Reviewed-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Acked-by: Wei Liu <wei.liu2@citrix.com>
---
Changes since v10:
- Add Acked-by: Wei Liu
- Add Reviewed-by: Konrad Rzeszutek Wilk
---
 docs/man/xl.cfg.pod.5             |  5 +++++
 tools/libxl/libxl_types.idl       |  2 ++
 tools/libxl/xl_cmdimpl.c          | 12 +++++++++++-
 tools/python/xen/lowlevel/xl/xl.c |  1 +
 4 files changed, 19 insertions(+), 1 deletion(-)

diff --git a/docs/man/xl.cfg.pod.5 b/docs/man/xl.cfg.pod.5
index 80e51bb..c6345b8 100644
--- a/docs/man/xl.cfg.pod.5
+++ b/docs/man/xl.cfg.pod.5
@@ -367,6 +367,11 @@ Default is C<destroy>.
 
 Action to take if the domain crashes.  Default is C<destroy>.
 
+=item B<on_soft_reset="ACTION">
+
+Action to take if the domain performs 'soft reset' (e.g. does kexec).
+Default is C<restart>.
+
 =back
 
 =head3 Direct Kernel Boot
diff --git a/tools/libxl/libxl_types.idl b/tools/libxl/libxl_types.idl
index ef346e7..9f6ec00 100644
--- a/tools/libxl/libxl_types.idl
+++ b/tools/libxl/libxl_types.idl
@@ -192,6 +192,7 @@ libxl_shutdown_reason = Enumeration("shutdown_reason", [
     (2, "suspend"),
     (3, "crash"),
     (4, "watchdog"),
+    (5, "soft_reset"),
     ], init_val = "LIBXL_SHUTDOWN_REASON_UNKNOWN")
 
 libxl_vga_interface_type = Enumeration("vga_interface_type", [
@@ -637,6 +638,7 @@ libxl_domain_config = Struct("domain_config", [
     ("on_reboot", libxl_action_on_shutdown),
     ("on_watchdog", libxl_action_on_shutdown),
     ("on_crash", libxl_action_on_shutdown),
+    ("on_soft_reset", libxl_action_on_shutdown),
     ], dir=DIR_IN)
 
 libxl_diskinfo = Struct("diskinfo", [
diff --git a/tools/libxl/xl_cmdimpl.c b/tools/libxl/xl_cmdimpl.c
index ebbb9a5..6fdc874 100644
--- a/tools/libxl/xl_cmdimpl.c
+++ b/tools/libxl/xl_cmdimpl.c
@@ -1419,6 +1419,13 @@ static void parse_config_data(const char *config_source,
         exit(1);
     }
 
+    if (xlu_cfg_get_string (config, "on_soft_reset", &buf, 0))
+        buf = "restart";
+    if (!parse_action_on_shutdown(buf, &d_config->on_soft_reset)) {
+        fprintf(stderr, "Unknown on_soft_reset action \"%s\" specified\n", buf);
+        exit(1);
+    }
+
     /* libxl_get_required_shadow_memory() must be called after final values
      * (default or specified) for vcpus and memory are set, because the
      * calculation depends on those values. */
@@ -2431,6 +2438,9 @@ static int handle_domain_death(uint32_t *r_domid,
     case LIBXL_SHUTDOWN_REASON_WATCHDOG:
         action = d_config->on_watchdog;
         break;
+    case LIBXL_SHUTDOWN_REASON_SOFT_RESET:
+        action = d_config->on_soft_reset;
+        break;
     default:
         LOG("Unknown shutdown reason code %d. Destroying domain.",
             event->u.domain_shutdown.shutdown_reason);
@@ -3860,7 +3870,7 @@ static void list_domains(bool verbose, bool context, bool claim, bool numa,
                          bool cpupool, const libxl_dominfo *info, int nb_domain)
 {
     int i;
-    static const char shutdown_reason_letters[]= "-rscw";
+    static const char shutdown_reason_letters[]= "-rscwS";
     libxl_bitmap nodemap;
     libxl_physinfo physinfo;
 
diff --git a/tools/python/xen/lowlevel/xl/xl.c b/tools/python/xen/lowlevel/xl/xl.c
index 20423a3..9b33731 100644
--- a/tools/python/xen/lowlevel/xl/xl.c
+++ b/tools/python/xen/lowlevel/xl/xl.c
@@ -783,6 +783,7 @@ PyMODINIT_FUNC initxl(void)
     _INT_CONST_LIBXL(m, SHUTDOWN_REASON_SUSPEND);
     _INT_CONST_LIBXL(m, SHUTDOWN_REASON_CRASH);
     _INT_CONST_LIBXL(m, SHUTDOWN_REASON_WATCHDOG);
+    _INT_CONST_LIBXL(m, SHUTDOWN_REASON_SOFT_RESET);
 
     genwrap__init(m);
 }
-- 
2.4.3

[PATCH v11 03/11] xl: introduce enum domain_restart_type

[Vitaly Kuznetsov]

As a preparation before adding new restart type (soft reset) put all
restart types into an enum.

Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
Reviewed-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
---
Changes since v10:
- None.
---
 tools/libxl/xl.h         |  6 ++++++
 tools/libxl/xl_cmdimpl.c | 23 ++++++++++-------------
 2 files changed, 16 insertions(+), 13 deletions(-)

diff --git a/tools/libxl/xl.h b/tools/libxl/xl.h
index 13bccba..6c19c0d 100644
--- a/tools/libxl/xl.h
+++ b/tools/libxl/xl.h
@@ -190,6 +190,12 @@ enum output_format {
 };
 extern enum output_format default_output_format;
 
+typedef enum {
+    DOMAIN_RESTART_NONE = 0,     /* No domain restart */
+    DOMAIN_RESTART_NORMAL,       /* Domain should be restarted */
+    DOMAIN_RESTART_RENAME,       /* Domain should be renamed and restarted */
+} domain_restart_type;
+
 extern void printf_info_sexp(int domid, libxl_domain_config *d_config, FILE *fh);
 
 #define XL_GLOBAL_CONFIG XEN_CONFIG_DIR "/xl.conf"
diff --git a/tools/libxl/xl_cmdimpl.c b/tools/libxl/xl_cmdimpl.c
index 6fdc874..c9bd839 100644
--- a/tools/libxl/xl_cmdimpl.c
+++ b/tools/libxl/xl_cmdimpl.c
@@ -2411,15 +2411,12 @@ static void reload_domain_config(uint32_t domid,
     }
 }
 
-/* Returns 1 if domain should be restarted,
- * 2 if domain should be renamed then restarted, or 0
- * Can update r_domid if domain is destroyed etc */
-static int handle_domain_death(uint32_t *r_domid,
-                               libxl_event *event,
-                               libxl_domain_config *d_config)
-
+/* Can update r_domid if domain is destroyed */
+static domain_restart_type handle_domain_death(uint32_t *r_domid,
+                                               libxl_event *event,
+                                               libxl_domain_config *d_config)
 {
-    int restart = 0;
+    domain_restart_type restart = DOMAIN_RESTART_NONE;
     libxl_action_on_shutdown action;
 
     switch (event->u.domain_shutdown.shutdown_reason) {
@@ -2474,12 +2471,12 @@ static int handle_domain_death(uint32_t *r_domid,
 
     case LIBXL_ACTION_ON_SHUTDOWN_RESTART_RENAME:
         reload_domain_config(*r_domid, d_config);
-        restart = 2;
+        restart = DOMAIN_RESTART_RENAME;
         break;
 
     case LIBXL_ACTION_ON_SHUTDOWN_RESTART:
         reload_domain_config(*r_domid, d_config);
-        restart = 1;
+        restart = DOMAIN_RESTART_NORMAL;
         /* fall-through */
     case LIBXL_ACTION_ON_SHUTDOWN_DESTROY:
         LOG("Domain %d needs to be cleaned up: destroying the domain",
@@ -2946,7 +2943,7 @@ start:
                 event->u.domain_shutdown.shutdown_reason,
                 event->u.domain_shutdown.shutdown_reason);
             switch (handle_domain_death(&domid, event, &d_config)) {
-            case 2:
+            case DOMAIN_RESTART_RENAME:
                 if (!preserve_domain(&domid, event, &d_config)) {
                     /* If we fail then exit leaving the old domain in place. */
                     ret = -1;
@@ -2954,7 +2951,7 @@ start:
                 }
 
                 /* Otherwise fall through and restart. */
-            case 1:
+            case DOMAIN_RESTART_NORMAL:
                 libxl_event_free(ctx, event);
                 libxl_evdisable_domain_death(ctx, deathw);
                 deathw = NULL;
@@ -2990,7 +2987,7 @@ start:
                 sleep(2);
                 goto start;
 
-            case 0:
+            case DOMAIN_RESTART_NONE:
                 LOG("Done. Exiting now");
                 libxl_event_free(ctx, event);
                 ret = 0;
-- 
2.4.3

[PATCH v11 04/11] xen: evtchn: make evtchn_reset() ready for soft reset

[Vitaly Kuznetsov]

We need to close all event channel so the domain performing soft reset
will be able to open them back.

Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
Reviewed-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Acked-by: Jan Beulich <jbeulich@suse.com>
---
Changes since v10:
- None.
---
 xen/common/event_channel.c | 43 +++++++++++++++++++------------------------
 xen/include/xen/event.h    |  3 +++
 2 files changed, 22 insertions(+), 24 deletions(-)

diff --git a/xen/common/event_channel.c b/xen/common/event_channel.c
index 46737cf..bd334f9 100644
--- a/xen/common/event_channel.c
+++ b/xen/common/event_channel.c
@@ -980,44 +980,28 @@ int evtchn_unmask(unsigned int port)
 }
 
 
-static long evtchn_reset(evtchn_reset_t *r)
+int evtchn_reset(struct domain *d)
 {
-    domid_t dom = r->dom;
-    struct domain *d;
-    int i, rc;
-
-    d = rcu_lock_domain_by_any_id(dom);
-    if ( d == NULL )
-        return -ESRCH;
+    unsigned int i;
 
-    rc = xsm_evtchn_reset(XSM_TARGET, current->domain, d);
-    if ( rc )
-        goto out;
+    if ( d != current->domain && !d->controller_pause_count )
+        return -EINVAL;
 
     for ( i = 0; port_is_valid(d, i); i++ )
         evtchn_close(d, i, 1);
 
     spin_lock(&d->event_lock);
 
-    if ( (dom == DOMID_SELF) && d->evtchn_fifo )
+    if ( d->evtchn_fifo )
     {
-        /*
-         * Guest domain called EVTCHNOP_reset with DOMID_SELF, destroying
-         * FIFO event array and control blocks, resetting evtchn_port_ops to
-         * evtchn_port_ops_2l.
-         */
+        /* Switching back to 2-level ABI. */
         evtchn_fifo_destroy(d);
         evtchn_2l_init(d);
     }
 
     spin_unlock(&d->event_lock);
 
-    rc = 0;
-
-out:
-    rcu_unlock_domain(d);
-
-    return rc;
+    return 0;
 }
 
 static long evtchn_set_priority(const struct evtchn_set_priority *set_priority)
@@ -1142,9 +1126,20 @@ long do_event_channel_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg)
 
     case EVTCHNOP_reset: {
         struct evtchn_reset reset;
+        struct domain *d;
+
         if ( copy_from_guest(&reset, arg, 1) != 0 )
             return -EFAULT;
-        rc = evtchn_reset(&reset);
+
+        d = rcu_lock_domain_by_any_id(reset.dom);
+        if ( d == NULL )
+            return -ESRCH;
+
+        rc = xsm_evtchn_reset(XSM_TARGET, current->domain, d);
+        if ( !rc )
+            rc = evtchn_reset(d);
+
+        rcu_unlock_domain(d);
         break;
     }
 
diff --git a/xen/include/xen/event.h b/xen/include/xen/event.h
index af923d1..b87924a 100644
--- a/xen/include/xen/event.h
+++ b/xen/include/xen/event.h
@@ -126,6 +126,9 @@ void evtchn_check_pollers(struct domain *d, unsigned int port);
 
 void evtchn_2l_init(struct domain *d);
 
+/* Close all event channels and reset to 2-level ABI. */
+int evtchn_reset(struct domain *d);
+
 /*
  * Low-level event channel port ops.
  */
-- 
2.4.3

[PATCH v11 05/11] xen: grant_table: implement grant_table_warn_active_grants()

[Vitaly Kuznetsov]

Log first 10 active grants for a domain. This function is going to be used
for soft reset, active grants on this path usually mean misbehaving backends
refusing to release their mappings on shutdown. We need that in addition to
the already existent 'g' keyhandler as such misbehaving backends can cause a
domain to crash right after the soft reset operation and 'g' option won't be
available in this case.

Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
Reviewed-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Acked-by: Jan Beulich <jbeulich@suse.com>
---
Changes since v10:
- None.
---
 xen/common/grant_table.c      | 35 +++++++++++++++++++++++++++++++++++
 xen/include/xen/grant_table.h |  5 +++++
 2 files changed, 40 insertions(+)

diff --git a/xen/common/grant_table.c b/xen/common/grant_table.c
index 2b449d5..23185e7 100644
--- a/xen/common/grant_table.c
+++ b/xen/common/grant_table.c
@@ -3352,6 +3352,41 @@ gnttab_release_mappings(
     }
 }
 
+void grant_table_warn_active_grants(struct domain *d)
+{
+    struct grant_table *gt = d->grant_table;
+    struct active_grant_entry *act;
+    grant_ref_t ref;
+    unsigned int nr_active = 0;
+
+#define WARN_GRANT_MAX 10
+
+    read_lock(&gt->lock);
+
+    for ( ref = 0; ref != nr_grant_entries(gt); ref++ )
+    {
+        act = active_entry_acquire(gt, ref);
+        if ( !act->pin )
+        {
+            active_entry_release(act);
+            continue;
+        }
+
+        nr_active++;
+        if ( nr_active <= WARN_GRANT_MAX )
+            printk(XENLOG_G_DEBUG "Dom%d has an active grant: GFN: %lx (MFN: %lx)\n",
+                   d->domain_id, act->gfn, act->frame);
+        active_entry_release(act);
+    }
+
+    if ( nr_active > WARN_GRANT_MAX )
+        printk(XENLOG_G_DEBUG "Dom%d has too many (%d) active grants to report\n",
+               d->domain_id, nr_active);
+
+    read_unlock(&gt->lock);
+
+#undef WARN_GRANT_MAX
+}
 
 void
 grant_table_destroy(
diff --git a/xen/include/xen/grant_table.h b/xen/include/xen/grant_table.h
index 5263fd6..1c29cee 100644
--- a/xen/include/xen/grant_table.h
+++ b/xen/include/xen/grant_table.h
@@ -89,6 +89,11 @@ void grant_table_destroy(
     struct domain *d);
 void grant_table_init_vcpu(struct vcpu *v);
 
+/*
+ * Check if domain has active grants and log first 10 of them.
+ */
+void grant_table_warn_active_grants(struct domain *d);
+
 /* Domain death release of granted mappings of other domains' memory. */
 void
 gnttab_release_mappings(
-- 
2.4.3

[PATCH v11 06/11] xen: Introduce XEN_DOMCTL_soft_reset

[Vitaly Kuznetsov]

New domctl resets state for a domain allowing it to 'start over': register
vcpu_info, switch to FIFO ABI for event channels. Still active grants are
being logged to help debugging misbehaving backends.

Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
Acked-by: Jan Beulich <jbeulich@suse.com>
---
Changes since v10:
- return when evtchn_reset() fails [Konrad Rzeszutek Wilk, Jan Beulich].
---
 xen/common/domain.c         | 49 +++++++++++++++++++++++++++++++++++++--------
 xen/common/domctl.c         |  9 +++++++++
 xen/include/public/domctl.h |  1 +
 xen/include/xen/sched.h     |  2 ++
 4 files changed, 53 insertions(+), 8 deletions(-)

diff --git a/xen/common/domain.c b/xen/common/domain.c
index 1b9fcfc..42c8caf 100644
--- a/xen/common/domain.c
+++ b/xen/common/domain.c
@@ -110,6 +110,16 @@ static void vcpu_check_shutdown(struct vcpu *v)
     spin_unlock(&d->shutdown_lock);
 }
 
+static void vcpu_info_reset(struct vcpu *v)
+{
+    struct domain *d = v->domain;
+
+    v->vcpu_info = ((v->vcpu_id < XEN_LEGACY_MAX_VCPUS)
+                    ? (vcpu_info_t *)&shared_info(d, vcpu_info[v->vcpu_id])
+                    : &dummy_vcpu_info);
+    v->vcpu_info_mfn = INVALID_MFN;
+}
+
 struct vcpu *alloc_vcpu(
     struct domain *d, unsigned int vcpu_id, unsigned int cpu_id)
 {
@@ -145,10 +155,7 @@ struct vcpu *alloc_vcpu(
         v->runstate.state = RUNSTATE_offline;        
         v->runstate.state_entry_time = NOW();
         set_bit(_VPF_down, &v->pause_flags);
-        v->vcpu_info = ((vcpu_id < XEN_LEGACY_MAX_VCPUS)
-                        ? (vcpu_info_t *)&shared_info(d, vcpu_info[vcpu_id])
-                        : &dummy_vcpu_info);
-        v->vcpu_info_mfn = INVALID_MFN;
+        vcpu_info_reset(v);
         init_waitqueue_vcpu(v);
     }
 
@@ -1038,6 +1045,34 @@ void domain_unpause_except_self(struct domain *d)
         domain_unpause(d);
 }
 
+int domain_soft_reset(struct domain *d)
+{
+    struct vcpu *v;
+    int rc;
+
+    spin_lock(&d->shutdown_lock);
+    for_each_vcpu ( d, v )
+        if ( !v->paused_for_shutdown )
+        {
+            spin_unlock(&d->shutdown_lock);
+            return -EINVAL;
+        }
+    spin_unlock(&d->shutdown_lock);
+
+    rc = evtchn_reset(d);
+    if ( rc )
+        return rc;
+
+    grant_table_warn_active_grants(d);
+
+    for_each_vcpu ( d, v )
+        unmap_vcpu_info(v);
+
+    domain_resume(d);
+
+    return 0;
+}
+
 int vcpu_reset(struct vcpu *v)
 {
     struct domain *d = v->domain;
@@ -1149,8 +1184,7 @@ int map_vcpu_info(struct vcpu *v, unsigned long gfn, unsigned offset)
 
 /*
  * Unmap the vcpu info page if the guest decided to place it somewhere
- * else.  This is only used from arch_domain_destroy, so there's no
- * need to do anything clever.
+ * else. This is used from arch_domain_destroy and domain_soft_reset.
  */
 void unmap_vcpu_info(struct vcpu *v)
 {
@@ -1163,8 +1197,7 @@ void unmap_vcpu_info(struct vcpu *v)
     unmap_domain_page_global((void *)
                              ((unsigned long)v->vcpu_info & PAGE_MASK));
 
-    v->vcpu_info = &dummy_vcpu_info;
-    v->vcpu_info_mfn = INVALID_MFN;
+    vcpu_info_reset(v);
 
     put_page_and_type(mfn_to_page(mfn));
 }
diff --git a/xen/common/domctl.c b/xen/common/domctl.c
index 7f959f3..41891b1 100644
--- a/xen/common/domctl.c
+++ b/xen/common/domctl.c
@@ -704,6 +704,15 @@ long do_domctl(XEN_GUEST_HANDLE_PARAM(xen_domctl_t) u_domctl)
         break;
     }
 
+    case XEN_DOMCTL_soft_reset:
+        if ( d == current->domain )
+        {
+            ret = -EINVAL;
+            break;
+        }
+        ret = domain_soft_reset(d);
+        break;
+
     case XEN_DOMCTL_destroydomain:
         ret = domain_kill(d);
         if ( ret == -ERESTART )
diff --git a/xen/include/public/domctl.h b/xen/include/public/domctl.h
index 675f021..794d4d5 100644
--- a/xen/include/public/domctl.h
+++ b/xen/include/public/domctl.h
@@ -1139,6 +1139,7 @@ struct xen_domctl {
 #define XEN_DOMCTL_psr_cmt_op                    75
 #define XEN_DOMCTL_monitor_op                    77
 #define XEN_DOMCTL_psr_cat_op                    78
+#define XEN_DOMCTL_soft_reset                    79
 #define XEN_DOMCTL_gdbsx_guestmemio            1000
 #define XEN_DOMCTL_gdbsx_pausevcpu             1001
 #define XEN_DOMCTL_gdbsx_unpausevcpu           1002
diff --git a/xen/include/xen/sched.h b/xen/include/xen/sched.h
index 73d3bc8..8053b5a 100644
--- a/xen/include/xen/sched.h
+++ b/xen/include/xen/sched.h
@@ -610,6 +610,8 @@ void domain_shutdown(struct domain *d, u8 reason);
 void domain_resume(struct domain *d);
 void domain_pause_for_debugger(void);
 
+int domain_soft_reset(struct domain *d);
+
 int vcpu_start_shutdown_deferral(struct vcpu *v);
 void vcpu_end_shutdown_deferral(struct vcpu *v);
 
-- 
2.4.3

[PATCH v11 07/11] flask: DOMCTL_soft_reset support

[Vitaly Kuznetsov]

Add new soft_reset vector to domain2 class, add it to create_domain
in the default policy.

Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
Acked-by: Daniel De Graaf <dgdegra@tycho.nsa.gov>
---
Changes since v10:
- None.
---
 tools/flask/policy/policy/modules/xen/xen.if | 2 +-
 xen/xsm/flask/hooks.c                        | 3 +++
 xen/xsm/flask/policy/access_vectors          | 2 ++
 3 files changed, 6 insertions(+), 1 deletion(-)

diff --git a/tools/flask/policy/policy/modules/xen/xen.if b/tools/flask/policy/policy/modules/xen/xen.if
index a2f25e1..32dd7b3 100644
--- a/tools/flask/policy/policy/modules/xen/xen.if
+++ b/tools/flask/policy/policy/modules/xen/xen.if
@@ -52,7 +52,7 @@ define(`create_domain_common', `
 			getaffinity setaffinity setvcpuextstate };
 	allow $1 $2:domain2 { set_cpuid settsc setscheduler setclaim
 			set_max_evtchn set_vnumainfo get_vnumainfo cacheflush
-			psr_cmt_op psr_cat_op };
+			psr_cmt_op psr_cat_op soft_reset };
 	allow $1 $2:security check_context;
 	allow $1 $2:shadow enable;
 	allow $1 $2:mmu { map_read map_write adjust memorymap physmap pinpage mmuext_op updatemp };
diff --git a/xen/xsm/flask/hooks.c b/xen/xsm/flask/hooks.c
index 7a4522e..aa37096 100644
--- a/xen/xsm/flask/hooks.c
+++ b/xen/xsm/flask/hooks.c
@@ -738,6 +738,9 @@ static int flask_domctl(struct domain *d, int cmd)
     case XEN_DOMCTL_psr_cat_op:
         return current_has_perm(d, SECCLASS_DOMAIN2, DOMAIN2__PSR_CAT_OP);
 
+    case XEN_DOMCTL_soft_reset:
+        return current_has_perm(d, SECCLASS_DOMAIN2, DOMAIN2__SOFT_RESET);
+
     default:
         printk("flask_domctl: Unknown op %d\n", cmd);
         return -EPERM;
diff --git a/xen/xsm/flask/policy/access_vectors b/xen/xsm/flask/policy/access_vectors
index 71495fd..a11c788 100644
--- a/xen/xsm/flask/policy/access_vectors
+++ b/xen/xsm/flask/policy/access_vectors
@@ -232,6 +232,8 @@ class domain2
 # XEN_DOMCTL_monitor_op
 # XEN_DOMCTL_vm_event_op
     vm_event
+# XEN_DOMCTL_soft_reset
+    soft_reset
 # XENMEM_access_op
     mem_access
 # XENMEM_paging_op
-- 
2.4.3

[PATCH v11 08/11] xen: arch-specific hooks for domain_soft_reset()

[Vitaly Kuznetsov]

x86-specific hook cleans up the pirq-emuirq mappings, destroys all ioreq
servers and and replaces the shared_info frame with an empty page to support
subsequent XENMAPSPACE_shared_info call.

ARM-specific hook is -ENOSYS for now.

Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
Reviewed-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Jan Beulich <jbeulich@suse.com>
---
Changes since v10:
- Add Reviewed-by: Konrad Rzeszutek Wilk.
- Add Acked-by: Ian Campbell [ARM part].
- Acked-by: Jan Beulich.
- Minor change according to Jan's comment.
---
 xen/arch/arm/domain.c         |  5 +++
 xen/arch/x86/domain.c         | 81 +++++++++++++++++++++++++++++++++++++++++++
 xen/arch/x86/hvm/hvm.c        |  5 +++
 xen/common/domain.c           |  8 +++--
 xen/include/asm-x86/hvm/hvm.h |  1 +
 xen/include/xen/domain.h      |  2 ++
 6 files changed, 100 insertions(+), 2 deletions(-)

diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
index b2bfc7d..5bdc2e9 100644
--- a/xen/arch/arm/domain.c
+++ b/xen/arch/arm/domain.c
@@ -655,6 +655,11 @@ void arch_domain_unpause(struct domain *d)
 {
 }
 
+int arch_domain_soft_reset(struct domain *d)
+{
+    return -ENOSYS;
+}
+
 static int is_guest_pv32_psr(uint32_t psr)
 {
     switch (psr & PSR_MODE_MASK)
diff --git a/xen/arch/x86/domain.c b/xen/arch/x86/domain.c
index 045f6ff..afc3e1e 100644
--- a/xen/arch/x86/domain.c
+++ b/xen/arch/x86/domain.c
@@ -710,6 +710,87 @@ void arch_domain_unpause(struct domain *d)
         viridian_time_ref_count_thaw(d);
 }
 
+int arch_domain_soft_reset(struct domain *d)
+{
+    struct page_info *page = virt_to_page(d->shared_info), *new_page;
+    int ret = 0;
+    struct domain *owner;
+    unsigned long mfn, gfn;
+    p2m_type_t p2mt;
+    unsigned int i;
+
+    /* Soft reset is supported for HVM/PVH domains only. */
+    if ( !has_hvm_container_domain(d) )
+        return -EINVAL;
+
+    hvm_domain_soft_reset(d);
+
+    spin_lock(&d->event_lock);
+    for ( i = 0; i < d->nr_pirqs ; i++ )
+    {
+        if ( domain_pirq_to_emuirq(d, i) != IRQ_UNBOUND )
+        {
+            ret = unmap_domain_pirq_emuirq(d, i);
+            if ( ret )
+                break;
+        }
+    }
+    spin_unlock(&d->event_lock);
+
+    if ( ret )
+        return ret;
+
+    /*
+     * The shared_info page needs to be replaced with a new page, otherwise we
+     * will get a hole if the domain does XENMAPSPACE_shared_info.
+     */
+
+    owner = page_get_owner_and_reference(page);
+    ASSERT( owner == d );
+
+    mfn = page_to_mfn(page);
+    gfn = mfn_to_gmfn(d, mfn);
+
+    /*
+     * gfn == INVALID_GFN indicates that the shared_info page was never mapped
+     * to the domain's address space and there is nothing to replace.
+     */
+    if ( gfn == INVALID_GFN )
+        goto exit_put_page;
+
+    if ( mfn_x(get_gfn_query(d, gfn, &p2mt)) != mfn )
+    {
+        printk(XENLOG_G_ERR "Failed to get Dom%d's shared_info GFN (%lx)\n",
+               d->domain_id, gfn);
+        ret = -EINVAL;
+        goto exit_put_page;
+    }
+
+    new_page = alloc_domheap_page(d, 0);
+    if ( !new_page )
+    {
+        printk(XENLOG_G_ERR "Failed to alloc a page to replace"
+               " Dom%d's shared_info frame %lx\n", d->domain_id, gfn);
+        ret = -ENOMEM;
+        goto exit_put_gfn;
+    }
+    guest_physmap_remove_page(d, gfn, mfn, PAGE_ORDER_4K);
+
+    ret = guest_physmap_add_page(d, gfn, page_to_mfn(new_page), PAGE_ORDER_4K);
+    if ( ret )
+    {
+        printk(XENLOG_G_ERR "Failed to add a page to replace"
+               " Dom%d's shared_info frame %lx\n", d->domain_id, gfn);
+        free_domheap_page(new_page);
+    }
+ exit_put_gfn:
+    put_gfn(d, gfn);
+ exit_put_page:
+    put_page(page);
+
+    return ret;
+}
+
 /*
  * These are the masks of CR4 bits (subject to hardware availability) which a
  * PV guest may not legitimiately attempt to modify.
diff --git a/xen/arch/x86/hvm/hvm.c b/xen/arch/x86/hvm/hvm.c
index 615fa89..b3c9359 100644
--- a/xen/arch/x86/hvm/hvm.c
+++ b/xen/arch/x86/hvm/hvm.c
@@ -6825,6 +6825,11 @@ bool_t altp2m_vcpu_emulate_ve(struct vcpu *v)
     return 0;
 }
 
+void hvm_domain_soft_reset(struct domain *d)
+{
+    hvm_destroy_all_ioreq_servers(d);
+}
+
 /*
  * Local variables:
  * mode: C
diff --git a/xen/common/domain.c b/xen/common/domain.c
index 42c8caf..8e7a282 100644
--- a/xen/common/domain.c
+++ b/xen/common/domain.c
@@ -1068,9 +1068,13 @@ int domain_soft_reset(struct domain *d)
     for_each_vcpu ( d, v )
         unmap_vcpu_info(v);
 
-    domain_resume(d);
+    rc = arch_domain_soft_reset(d);
+    if ( !rc )
+        domain_resume(d);
+    else
+        domain_crash(d);
 
-    return 0;
+    return rc;
 }
 
 int vcpu_reset(struct vcpu *v)
diff --git a/xen/include/asm-x86/hvm/hvm.h b/xen/include/asm-x86/hvm/hvm.h
index 68b216c..4d9d640 100644
--- a/xen/include/asm-x86/hvm/hvm.h
+++ b/xen/include/asm-x86/hvm/hvm.h
@@ -225,6 +225,7 @@ extern const struct hvm_function_table *start_vmx(void);
 int hvm_domain_initialise(struct domain *d);
 void hvm_domain_relinquish_resources(struct domain *d);
 void hvm_domain_destroy(struct domain *d);
+void hvm_domain_soft_reset(struct domain *d);
 
 int hvm_vcpu_initialise(struct vcpu *v);
 void hvm_vcpu_destroy(struct vcpu *v);
diff --git a/xen/include/xen/domain.h b/xen/include/xen/domain.h
index 848db8a..a469fe0 100644
--- a/xen/include/xen/domain.h
+++ b/xen/include/xen/domain.h
@@ -65,6 +65,8 @@ void arch_domain_shutdown(struct domain *d);
 void arch_domain_pause(struct domain *d);
 void arch_domain_unpause(struct domain *d);
 
+int arch_domain_soft_reset(struct domain *d);
+
 int arch_set_info_guest(struct vcpu *, vcpu_guest_context_u);
 void arch_get_info_guest(struct vcpu *, vcpu_guest_context_u);
 
-- 
2.4.3

[PATCH v11 09/11] libxc: support XEN_DOMCTL_soft_reset operation

[Vitaly Kuznetsov]

Introduce xc_domain_soft_reset() function supporting XEN_DOMCTL_soft_reset.

Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
Acked-by: Wei Liu <wei.liu2@citrix.com>
Reviewed-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
Changes since v10:
- None.
---
 tools/libxc/include/xenctrl.h | 3 +++
 tools/libxc/xc_domain.c       | 9 +++++++++
 2 files changed, 12 insertions(+)

diff --git a/tools/libxc/include/xenctrl.h b/tools/libxc/include/xenctrl.h
index de3c0ad..cbe98de 100644
--- a/tools/libxc/include/xenctrl.h
+++ b/tools/libxc/include/xenctrl.h
@@ -1288,6 +1288,9 @@ int xc_domain_setvnuma(xc_interface *xch,
                         unsigned int *vcpu_to_vnode,
                         unsigned int *vnode_to_pnode);
 
+int xc_domain_soft_reset(xc_interface *xch,
+                         uint32_t domid);
+
 #if defined(__i386__) || defined(__x86_64__)
 /*
  * PC BIOS standard E820 types and structure.
diff --git a/tools/libxc/xc_domain.c b/tools/libxc/xc_domain.c
index 780797f..62b2e45 100644
--- a/tools/libxc/xc_domain.c
+++ b/tools/libxc/xc_domain.c
@@ -2493,6 +2493,15 @@ int xc_domain_setvnuma(xc_interface *xch,
     return rc;
 }
 
+
+int xc_domain_soft_reset(xc_interface *xch,
+                         uint32_t domid)
+{
+    DECLARE_DOMCTL;
+    domctl.cmd = XEN_DOMCTL_soft_reset;
+    domctl.domain = (domid_t)domid;
+    return do_domctl(xch, &domctl);
+}
 /*
  * Local variables:
  * mode: C
-- 
2.4.3

[PATCH v11 10/11] libxl: add LIBXL_DEVICE_MODEL_SAVE_FILE

[Vitaly Kuznetsov]

Use this in libxl_dm instead of hard-coding.

Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
Changes since v10:
- s,XC,LIBXL,g to meet the XC_DEVICE_MODEL_RESTORE_FILE ->
  LIBXL_DEVICE_MODEL_RESTORE_FILE change.
---
 tools/libxl/libxl_dm.c       | 2 +-
 tools/libxl/libxl_internal.h | 1 +
 2 files changed, 2 insertions(+), 1 deletion(-)

diff --git a/tools/libxl/libxl_dm.c b/tools/libxl/libxl_dm.c
index c84085e..b7c6592 100644
--- a/tools/libxl/libxl_dm.c
+++ b/tools/libxl/libxl_dm.c
@@ -31,7 +31,7 @@ static const char *libxl_tapif_script(libxl__gc *gc)
 
 const char *libxl__device_model_savefile(libxl__gc *gc, uint32_t domid)
 {
-    return libxl__sprintf(gc, "/var/lib/xen/qemu-save.%d", domid);
+    return libxl__sprintf(gc, LIBXL_DEVICE_MODEL_SAVE_FILE".%d", domid);
 }
 
 static const char *qemu_xen_path(libxl__gc *gc)
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index 6ea6c83..c3c3b7b 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -90,6 +90,7 @@
 /* QEMU may be slow to load and start due to a bug in Linux where the I/O
  * subsystem sometime produce high latency under load. */
 #define LIBXL_DEVICE_MODEL_START_TIMEOUT 60
+#define LIBXL_DEVICE_MODEL_SAVE_FILE "/var/lib/xen/qemu-save" /* .$domid */
 #define LIBXL_DEVICE_MODEL_RESTORE_FILE "/var/lib/xen/qemu-resume" /* .$domid */
 #define LIBXL_STUBDOM_START_TIMEOUT 30
 #define LIBXL_QEMU_BODGE_TIMEOUT 2
-- 
2.4.3

[PATCH v11 11/11] (lib)xl: soft reset support

[Vitaly Kuznetsov]

Use existing create/restore path to perform 'soft reset' for HVM
domains. Tear everything down, e.g. destroy domain's device model,
remove the domain from xenstore, save toolstack record and start
over.

Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
---
Changes since v10:
- Adapt to 'migration v2' changes.
- Use LIBXL_DEVICE_MODEL_SAVE_FILE as Qemu save file (and rename it to
  LIBXL_DEVICE_MODEL_RESTORE_FILE later) to support stubdom case (as
  we connect consoles to both files on create.
- Fix coding style, minor description change in xl.cfg.pod.5 [Wei Liu]

Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
---
 docs/man/xl.cfg.pod.5        |   8 +-
 tools/libxl/libxl.c          |  22 ++++-
 tools/libxl/libxl.h          |  15 ++++
 tools/libxl/libxl_create.c   | 192 ++++++++++++++++++++++++++++++++++++++-----
 tools/libxl/libxl_internal.h |   4 +
 tools/libxl/libxl_types.idl  |   2 +
 tools/libxl/xl.h             |   1 +
 tools/libxl/xl_cmdimpl.c     |  25 +++++-
 8 files changed, 242 insertions(+), 27 deletions(-)

diff --git a/docs/man/xl.cfg.pod.5 b/docs/man/xl.cfg.pod.5
index c6345b8..d8c4186 100644
--- a/docs/man/xl.cfg.pod.5
+++ b/docs/man/xl.cfg.pod.5
@@ -349,6 +349,12 @@ destroy the domain.
 write a "coredump" of the domain to F</var/lib/xen/dump/NAME> and then
 restart the domain.
 
+=item B<soft-reset>
+
+Reset all Xen specific interfaces for the Xen-aware HVM domain allowing
+it to reestablish these interfaces and continue executing the domain. PV
+guest is not supported.
+
 =back
 
 The default for C<on_poweroff> is C<destroy>.
@@ -370,7 +376,7 @@ Action to take if the domain crashes.  Default is C<destroy>.
 =item B<on_soft_reset="ACTION">
 
 Action to take if the domain performs 'soft reset' (e.g. does kexec).
-Default is C<restart>.
+Default is C<soft-reset>.
 
 =back
 
diff --git a/tools/libxl/libxl.c b/tools/libxl/libxl.c
index 4f2eb24..a9b8b8b 100644
--- a/tools/libxl/libxl.c
+++ b/tools/libxl/libxl.c
@@ -1489,6 +1489,7 @@ void libxl__domain_destroy(libxl__egc *egc, libxl__domain_destroy_state *dds)
         dds->stubdom.ao = ao;
         dds->stubdom.domid = stubdomid;
         dds->stubdom.callback = stubdom_destroy_callback;
+        dds->stubdom.soft_reset = false;
         libxl__destroy_domid(egc, &dds->stubdom);
     } else {
         dds->stubdom_finished = 1;
@@ -1497,6 +1498,7 @@ void libxl__domain_destroy(libxl__egc *egc, libxl__domain_destroy_state *dds)
     dds->domain.ao = ao;
     dds->domain.domid = dds->domid;
     dds->domain.callback = domain_destroy_callback;
+    dds->domain.soft_reset = dds->soft_reset;
     libxl__destroy_domid(egc, &dds->domain);
 }
 
@@ -1677,10 +1679,14 @@ static void devices_destroy_cb(libxl__egc *egc,
 
     /* Clean up qemu-save and qemu-resume files. They are
      * intermediate files created by libxc. Unfortunately they
-     * don't fit in existing userdata scheme very well.
+     * don't fit in existing userdata scheme very well. In soft reset
+     * case we need to keep the file.
      */
-    rc = libxl__remove_file(gc, libxl__device_model_savefile(gc, domid));
-    if (rc < 0) goto out;
+    if (!dis->soft_reset) {
+        rc = libxl__remove_file(gc,
+                                libxl__device_model_savefile(gc, domid));
+        if (rc < 0) goto out;
+    }
     rc = libxl__remove_file(gc,
              GCSPRINTF(LIBXL_DEVICE_MODEL_RESTORE_FILE".%u", domid));
     if (rc < 0) goto out;
@@ -1691,7 +1697,15 @@ static void devices_destroy_cb(libxl__egc *egc,
         ctx->xch = xc_interface_open(ctx->lg,0,0);
         if (!ctx->xch) goto badchild;
 
-        rc = xc_domain_destroy(ctx->xch, domid);
+        if (!dis->soft_reset) {
+            rc = xc_domain_destroy(ctx->xch, domid);
+        } else {
+            rc = xc_domain_pause(ctx->xch, domid);
+            if (rc < 0) goto badchild;
+            rc = xc_domain_soft_reset(ctx->xch, domid);
+            if (rc < 0) goto badchild;
+            rc = xc_domain_unpause(ctx->xch, domid);
+        }
         if (rc < 0) goto badchild;
         _exit(0);
 
diff --git a/tools/libxl/libxl.h b/tools/libxl/libxl.h
index 5f9047c..4ee9fbf 100644
--- a/tools/libxl/libxl.h
+++ b/tools/libxl/libxl.h
@@ -205,6 +205,13 @@
 #define LIBXL_HAVE_BUILDINFO_ARM_GIC_VERSION 1
 
 /*
+ * LIBXL_HAVE_SOFT_RESET indicates that libxl supports performing
+ * 'soft reset' for domains and there is 'soft_reset' shutdown reason
+ * in enum libxl_shutdown_reason.
+ */
+#define LIBXL_HAVE_SOFT_RESET 1
+
+/*
  * libxl ABI compatibility
  *
  * The only guarantee which libxl makes regarding ABI compatibility
@@ -1132,6 +1139,14 @@ int static inline libxl_domain_create_restore_0x040200(
 
 #endif
 
+int libxl_domain_soft_reset(libxl_ctx *ctx,
+                            libxl_domain_config *d_config,
+                            uint32_t domid,
+                            const libxl_asyncop_how *ao_how,
+                            const libxl_asyncprogress_how
+                            *aop_console_how)
+                            LIBXL_EXTERNAL_CALLERS_ONLY;
+
   /* A progress report will be made via ao_console_how, of type
    * domain_create_console_available, when the domain's primary
    * console is available and can be connected to.
diff --git a/tools/libxl/libxl_create.c b/tools/libxl/libxl_create.c
index 5128160..86bf9dc 100644
--- a/tools/libxl/libxl_create.c
+++ b/tools/libxl/libxl_create.c
@@ -496,8 +496,6 @@ int libxl__domain_make(libxl__gc *gc, libxl_domain_config *d_config,
     /* convenience aliases */
     libxl_domain_create_info *info = &d_config->c_info;
 
-    assert(!libxl_domid_valid_guest(*domid));
-
     uuid_string = libxl__uuid2string(gc, info->uuid);
     if (!uuid_string) {
         rc = ERROR_NOMEM;
@@ -518,7 +516,6 @@ int libxl__domain_make(libxl__gc *gc, libxl_domain_config *d_config,
         }
         flags |= XEN_DOMCTL_CDF_hap;
     }
-    *domid = -1;
 
     /* Ultimately, handle is an array of 16 uint8_t, same as uuid */
     libxl_uuid_copy(ctx, (libxl_uuid *)handle, &info->uuid);
@@ -530,13 +527,17 @@ int libxl__domain_make(libxl__gc *gc, libxl_domain_config *d_config,
         goto out;
     }
 
-    ret = xc_domain_create_config(ctx->xch, info->ssidref,
-                                  handle, flags, domid,
-                                  xc_config);
-    if (ret < 0) {
-        LIBXL__LOG_ERRNO(ctx, LIBXL__LOG_ERROR, "domain creation fail");
-        rc = ERROR_FAIL;
-        goto out;
+    /* Valid domid here means we're soft resetting. */
+    if (!libxl_domid_valid_guest(*domid)) {
+        ret = xc_domain_create_config(ctx->xch, info->ssidref,
+                                      handle, flags, domid,
+                                      xc_config);
+        if (ret < 0) {
+            LIBXL__LOG_ERRNO(ctx, LIBXL__LOG_ERROR,
+                             "domain creation fail");
+            rc = ERROR_FAIL;
+            goto out;
+        }
     }
 
     rc = libxl__arch_domain_save_config(gc, d_config, xc_config);
@@ -771,9 +772,8 @@ static void initiate_domain_create(libxl__egc *egc,
     libxl_domain_config *const d_config = dcs->guest_config;
     libxl__domain_build_state *const state = &dcs->build_state;
     const int restore_fd = dcs->restore_fd;
-    memset(&dcs->build_state, 0, sizeof(dcs->build_state));
 
-    domid = 0;
+    domid = dcs->domid_soft_reset;
 
     if (d_config->c_info.ssid_label) {
         char *s = d_config->c_info.ssid_label;
@@ -937,7 +937,7 @@ static void initiate_domain_create(libxl__egc *egc,
             d_config->nics[i].devid = ++last_devid;
     }
 
-    if (restore_fd >= 0) {
+    if (restore_fd >= 0 || dcs->domid_soft_reset != INVALID_DOMID) {
         LOG(DEBUG, "restoring, not running bootloader");
         domcreate_bootloader_done(egc, &dcs->bl, 0);
     } else  {
@@ -1010,7 +1010,7 @@ static void domcreate_bootloader_done(libxl__egc *egc,
     dcs->dmss.dm.callback = domcreate_devmodel_started;
     dcs->dmss.callback = domcreate_devmodel_started;
 
-    if ( restore_fd < 0 ) {
+    if (restore_fd < 0 && dcs->domid_soft_reset == INVALID_DOMID) {
         rc = libxl__domain_build(gc, d_config, domid, state);
         domcreate_rebuild_done(egc, dcs, rc);
         return;
@@ -1030,8 +1030,10 @@ static void domcreate_bootloader_done(libxl__egc *egc,
     dcs->srs.completion_callback = domcreate_stream_done;
     dcs->srs.checkpoint_callback = remus_checkpoint_stream_done;
 
-    libxl__stream_read_start(egc, &dcs->srs);
-    return;
+    if (restore_fd >= 0) {
+        libxl__stream_read_start(egc, &dcs->srs);
+        return;
+    }
 
  out:
     domcreate_stream_done(egc, &dcs->srs, rc);
@@ -1120,9 +1122,12 @@ out:
         libxl__file_reference_unmap(&state->pv_ramdisk);
     }
 
-    esave = errno;
-    libxl_fd_set_nonblock(ctx, fd, 0);
-    errno = esave;
+    /* fd == -1 here means we're doing soft reset. */
+    if (fd != -1) {
+        esave = errno;
+        libxl_fd_set_nonblock(ctx, fd, 0);
+        errno = esave;
+    }
     domcreate_rebuild_done(egc, dcs, ret);
 }
 
@@ -1543,6 +1548,14 @@ typedef struct {
     uint32_t *domid_out;
 } libxl__app_domain_create_state;
 
+typedef struct {
+    libxl__app_domain_create_state cdcs;
+    libxl__domain_destroy_state dds;
+    libxl__domain_suspend_state dss;
+    char *toolstack_buf;
+    uint32_t toolstack_len;
+} libxl__domain_soft_reset_state;
+
 static void domain_create_cb(libxl__egc *egc,
                              libxl__domain_create_state *dcs,
                              int rc, uint32_t domid);
@@ -1565,6 +1578,7 @@ static int do_domain_create(libxl_ctx *ctx, libxl_domain_config *d_config,
     if (restore_fd > -1)
         cdcs->dcs.restore_params = *params;
     cdcs->dcs.callback = domain_create_cb;
+    cdcs->dcs.domid_soft_reset = INVALID_DOMID;
     libxl__ao_progress_gethow(&cdcs->dcs.aop_console_how, aop_console_how);
     cdcs->domid_out = domid;
 
@@ -1573,6 +1587,131 @@ static int do_domain_create(libxl_ctx *ctx, libxl_domain_config *d_config,
     return AO_INPROGRESS;
 }
 
+static void domain_soft_reset_cb(libxl__egc *egc,
+                                 libxl__domain_destroy_state *dds,
+                                 int rc)
+{
+    STATE_AO_GC(dds->ao);
+    libxl__domain_soft_reset_state *srs = CONTAINER_OF(dds, *srs, dds);
+    libxl__app_domain_create_state *cdcs = &srs->cdcs;
+    char *savefile, *restorefile;
+
+    if (rc) {
+        LOG(ERROR, "destruction of domain %u failed.", dds->domid);
+        goto error;
+    }
+
+    cdcs->dcs.guest_domid = dds->domid;
+    rc = libxl__restore_emulator_xenstore_data(&cdcs->dcs, srs->toolstack_buf,
+                                               srs->toolstack_len);
+    if (rc) {
+        LOG(ERROR, "failed to restore toolstack record.");
+        goto error;
+    }
+
+    savefile = GCSPRINTF(LIBXL_DEVICE_MODEL_SAVE_FILE".%d", dds->domid);
+    restorefile = GCSPRINTF(LIBXL_DEVICE_MODEL_RESTORE_FILE".%d", dds->domid);
+    rc = rename(savefile, restorefile);
+    if (rc) {
+        LOG(ERROR, "failed to rename dm save file.");
+        goto error;
+    }
+
+    initiate_domain_create(egc, &cdcs->dcs);
+    return;
+
+error:
+    domcreate_complete(egc, &cdcs->dcs, rc);
+}
+
+static int do_domain_soft_reset(libxl_ctx *ctx,
+                                libxl_domain_config *d_config,
+                                uint32_t domid_soft_reset,
+                                const libxl_asyncop_how *ao_how,
+                                const libxl_asyncprogress_how
+                                *aop_console_how)
+{
+    AO_CREATE(ctx, 0, ao_how);
+    libxl__domain_soft_reset_state *srs;
+    libxl__app_domain_create_state *cdcs;
+    libxl__domain_create_state *dcs;
+    libxl__domain_build_state *state;
+    libxl__domain_suspend_state *dss;
+    char *dom_path, *xs_store_mfn, *xs_console_mfn;
+    uint32_t domid_out;
+    int rc;
+
+    GCNEW(srs);
+    cdcs = &srs->cdcs;
+    dcs = &cdcs->dcs;
+    state = &dcs->build_state;
+    dss = &srs->dss;
+
+    srs->cdcs.dcs.ao = ao;
+    srs->cdcs.dcs.guest_config = d_config;
+    libxl_domain_config_init(&srs->cdcs.dcs.guest_config_saved);
+    libxl_domain_config_copy(ctx, &srs->cdcs.dcs.guest_config_saved,
+                             d_config);
+    cdcs->dcs.restore_fd = -1;
+    cdcs->dcs.domid_soft_reset = domid_soft_reset;
+    cdcs->dcs.callback = domain_create_cb;
+    libxl__ao_progress_gethow(&srs->cdcs.dcs.aop_console_how,
+                              aop_console_how);
+    cdcs->domid_out = &domid_out;
+
+    dom_path = libxl__xs_get_dompath(gc, domid_soft_reset);
+    if (!dom_path) {
+        LOG(ERROR, "failed to read domain path");
+        return AO_CREATE_FAIL(ERROR_FAIL);
+    }
+
+    xs_store_mfn = xs_read(ctx->xsh, XBT_NULL,
+                           GCSPRINTF("%s/store/ring-ref", dom_path),
+                           NULL);
+    state->store_mfn = xs_store_mfn ? atol(xs_store_mfn): 0;
+    free(xs_store_mfn);
+
+    xs_console_mfn = xs_read(ctx->xsh, XBT_NULL,
+                             GCSPRINTF("%s/console/ring-ref", dom_path),
+                             NULL);
+    state->console_mfn = xs_console_mfn ? atol(xs_console_mfn): 0;
+    free(xs_console_mfn);
+
+    dss->ao = ao;
+    dss->domid = domid_soft_reset;
+    dss->dm_savefile = GCSPRINTF(LIBXL_DEVICE_MODEL_SAVE_FILE".%d",
+                                 domid_soft_reset);
+
+    rc = libxl__save_emulator_xenstore_data(dss, &srs->toolstack_buf,
+                                            &srs->toolstack_len);
+    if (rc) {
+        LOG(ERROR, "failed to save toolstack record.");
+        return AO_CREATE_FAIL(ERROR_FAIL);
+    }
+
+    rc = libxl__domain_suspend_device_model(gc, dss);
+    if (rc) {
+        LOG(ERROR, "failed to suspend device model.");
+        return AO_CREATE_FAIL(ERROR_FAIL);
+    }
+
+    /*
+     * Ask all backends to disconnect by removing the domain from
+     * xenstore. On the creation path the domain will be introduced to
+     * xenstore again with probably different store/console/...
+     * channels.
+     */
+    xs_release_domain(ctx->xsh, cdcs->dcs.domid_soft_reset);
+
+    srs->dds.ao = ao;
+    srs->dds.domid = domid_soft_reset;
+    srs->dds.callback = domain_soft_reset_cb;
+    srs->dds.soft_reset = true;
+    libxl__domain_destroy(egc, &srs->dds);
+
+    return AO_INPROGRESS;
+}
+
 static void domain_create_cb(libxl__egc *egc,
                              libxl__domain_create_state *dcs,
                              int rc, uint32_t domid)
@@ -1604,6 +1743,21 @@ int libxl_domain_create_restore(libxl_ctx *ctx, libxl_domain_config *d_config,
                             ao_how, aop_console_how);
 }
 
+int libxl_domain_soft_reset(libxl_ctx *ctx,
+                            libxl_domain_config *d_config,
+                            uint32_t domid,
+                            const libxl_asyncop_how *ao_how,
+                            const libxl_asyncprogress_how
+                            *aop_console_how)
+{
+    libxl_domain_build_info *const info = &d_config->b_info;
+
+    if (info->type != LIBXL_DOMAIN_TYPE_HVM) return ERROR_INVAL;
+
+    return do_domain_soft_reset(ctx, d_config, domid, ao_how,
+                                aop_console_how);
+}
+
 /*
  * Local variables:
  * mode: C
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index c3c3b7b..0a81ca8 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -111,6 +111,7 @@
 #define STUBDOM_SPECIAL_CONSOLES 3
 #define TAP_DEVICE_SUFFIX "-emu"
 #define DOMID_XS_PATH "domid"
+#define INVALID_DOMID ~0
 
 #define ARRAY_SIZE(a) (sizeof(a) / sizeof(a[0]))
 
@@ -3214,6 +3215,7 @@ struct libxl__destroy_domid_state {
     /* private to implementation */
     libxl__devices_remove_state drs;
     libxl__ev_child destroyer;
+    bool soft_reset;
 };
 
 struct libxl__domain_destroy_state {
@@ -3228,6 +3230,7 @@ struct libxl__domain_destroy_state {
     int stubdom_finished;
     libxl__destroy_domid_state domain;
     int domain_finished;
+    bool soft_reset;
 };
 
 /*
@@ -3392,6 +3395,7 @@ struct libxl__domain_create_state {
     libxl_domain_config guest_config_saved; /* vanilla config */
     int restore_fd, libxc_fd;
     libxl_domain_restore_params restore_params;
+    uint32_t domid_soft_reset;
     libxl__domain_create_cb *callback;
     libxl_asyncprogress_how aop_console_how;
     /* private to domain_create */
diff --git a/tools/libxl/libxl_types.idl b/tools/libxl/libxl_types.idl
index 9f6ec00..87d8255 100644
--- a/tools/libxl/libxl_types.idl
+++ b/tools/libxl/libxl_types.idl
@@ -138,6 +138,8 @@ libxl_action_on_shutdown = Enumeration("action_on_shutdown", [
 
     (5, "COREDUMP_DESTROY"),
     (6, "COREDUMP_RESTART"),
+
+    (7, "SOFT_RESET"),
     ], init_val = "LIBXL_ACTION_ON_SHUTDOWN_DESTROY")
 
 libxl_trigger = Enumeration("trigger", [
diff --git a/tools/libxl/xl.h b/tools/libxl/xl.h
index 6c19c0d..0021112 100644
--- a/tools/libxl/xl.h
+++ b/tools/libxl/xl.h
@@ -194,6 +194,7 @@ typedef enum {
     DOMAIN_RESTART_NONE = 0,     /* No domain restart */
     DOMAIN_RESTART_NORMAL,       /* Domain should be restarted */
     DOMAIN_RESTART_RENAME,       /* Domain should be renamed and restarted */
+    DOMAIN_RESTART_SOFT_RESET,   /* Soft reset should be performed */
 } domain_restart_type;
 
 extern void printf_info_sexp(int domid, libxl_domain_config *d_config, FILE *fh);
diff --git a/tools/libxl/xl_cmdimpl.c b/tools/libxl/xl_cmdimpl.c
index c9bd839..2414d11 100644
--- a/tools/libxl/xl_cmdimpl.c
+++ b/tools/libxl/xl_cmdimpl.c
@@ -133,6 +133,8 @@ static const char *action_on_shutdown_names[] = {
 
     [LIBXL_ACTION_ON_SHUTDOWN_COREDUMP_DESTROY] = "coredump-destroy",
     [LIBXL_ACTION_ON_SHUTDOWN_COREDUMP_RESTART] = "coredump-restart",
+
+    [LIBXL_ACTION_ON_SHUTDOWN_SOFT_RESET] = "soft-reset",
 };
 
 /* Optional data, in order:
@@ -1420,7 +1422,7 @@ static void parse_config_data(const char *config_source,
     }
 
     if (xlu_cfg_get_string (config, "on_soft_reset", &buf, 0))
-        buf = "restart";
+        buf = "soft-reset";
     if (!parse_action_on_shutdown(buf, &d_config->on_soft_reset)) {
         fprintf(stderr, "Unknown on_soft_reset action \"%s\" specified\n", buf);
         exit(1);
@@ -2485,6 +2487,11 @@ static domain_restart_type handle_domain_death(uint32_t *r_domid,
         *r_domid = INVALID_DOMID;
         break;
 
+    case LIBXL_ACTION_ON_SHUTDOWN_SOFT_RESET:
+        reload_domain_config(*r_domid, d_config);
+        restart = DOMAIN_RESTART_SOFT_RESET;
+        break;
+
     case LIBXL_ACTION_ON_SHUTDOWN_COREDUMP_DESTROY:
     case LIBXL_ACTION_ON_SHUTDOWN_COREDUMP_RESTART:
         /* Already handled these above. */
@@ -2661,6 +2668,7 @@ static uint32_t create_domain(struct domain_create *dom_info)
     int restore_fd_to_close = -1;
     const libxl_asyncprogress_how *autoconnect_console_how;
     struct save_file_header hdr;
+    uint32_t domid_soft_reset = INVALID_DOMID;
 
     int restoring = (restore_file || (migrate_fd >= 0));
 
@@ -2875,7 +2883,13 @@ start:
          * restore/migrate-receive it again.
          */
         restoring = 0;
-    }else{
+    } else if (domid_soft_reset != INVALID_DOMID) {
+        /* Do soft reset. */
+        ret = libxl_domain_soft_reset(ctx, &d_config, domid_soft_reset,
+                                      0, autoconnect_console_how);
+        domid = domid_soft_reset;
+        domid_soft_reset = INVALID_DOMID;
+    } else {
         ret = libxl_domain_create_new(ctx, &d_config, &domid,
                                       0, autoconnect_console_how);
     }
@@ -2943,8 +2957,13 @@ start:
                 event->u.domain_shutdown.shutdown_reason,
                 event->u.domain_shutdown.shutdown_reason);
             switch (handle_domain_death(&domid, event, &d_config)) {
+            case DOMAIN_RESTART_SOFT_RESET:
+                domid_soft_reset = domid;
+                domid = INVALID_DOMID;
+                /* fall through */
             case DOMAIN_RESTART_RENAME:
-                if (!preserve_domain(&domid, event, &d_config)) {
+                if (domid_soft_reset == INVALID_DOMID &&
+                    !preserve_domain(&domid, event, &d_config)) {
                     /* If we fail then exit leaving the old domain in place. */
                     ret = -1;
                     goto out;
-- 
2.4.3

Re: [PATCH v11 11/11] (lib)xl: soft reset support

[Wei Liu]

Sorry for the delay.

FYI all other patches of this series were applied by Jan. You only need
to resend this one.

See below for a few comments.

On Fri, Sep 04, 2015 at 03:39:51PM +0200, Vitaly Kuznetsov wrote:
> Use existing create/restore path to perform 'soft reset' for HVM
> domains. Tear everything down, e.g. destroy domain's device model,
> remove the domain from xenstore, save toolstack record and start
> over.
> 
> Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
> ---
> Changes since v10:
> - Adapt to 'migration v2' changes.
> - Use LIBXL_DEVICE_MODEL_SAVE_FILE as Qemu save file (and rename it to
>   LIBXL_DEVICE_MODEL_RESTORE_FILE later) to support stubdom case (as
>   we connect consoles to both files on create.
> - Fix coding style, minor description change in xl.cfg.pod.5 [Wei Liu]
> 
> Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
> ---
>  docs/man/xl.cfg.pod.5        |   8 +-
>  tools/libxl/libxl.c          |  22 ++++-
>  tools/libxl/libxl.h          |  15 ++++
>  tools/libxl/libxl_create.c   | 192 ++++++++++++++++++++++++++++++++++++++-----
>  tools/libxl/libxl_internal.h |   4 +
>  tools/libxl/libxl_types.idl  |   2 +
>  tools/libxl/xl.h             |   1 +
>  tools/libxl/xl_cmdimpl.c     |  25 +++++-
>  8 files changed, 242 insertions(+), 27 deletions(-)
> 
> diff --git a/docs/man/xl.cfg.pod.5 b/docs/man/xl.cfg.pod.5
> index c6345b8..d8c4186 100644
> --- a/docs/man/xl.cfg.pod.5
> +++ b/docs/man/xl.cfg.pod.5
> @@ -349,6 +349,12 @@ destroy the domain.
>  write a "coredump" of the domain to F</var/lib/xen/dump/NAME> and then
>  restart the domain.
>  
> +=item B<soft-reset>
> +
> +Reset all Xen specific interfaces for the Xen-aware HVM domain allowing
> +it to reestablish these interfaces and continue executing the domain. PV
> +guest is not supported.
> +

And "non-Xen-aware HVM will crash" ?  If there is no definite answer to
guest state maybe just saying "PV guest and non-Xen-aware HVM guests are
not supported" ?

It's important to let user know about the consequence because libxl
doesn't actually stop you from soft-resetting a HVM guest that is not
Xen-aware.

[...]
> +static int do_domain_soft_reset(libxl_ctx *ctx,
> +                                libxl_domain_config *d_config,
> +                                uint32_t domid_soft_reset,
> +                                const libxl_asyncop_how *ao_how,
> +                                const libxl_asyncprogress_how
> +                                *aop_console_how)
> +{
> +    AO_CREATE(ctx, 0, ao_how);
> +    libxl__domain_soft_reset_state *srs;
> +    libxl__app_domain_create_state *cdcs;
> +    libxl__domain_create_state *dcs;
> +    libxl__domain_build_state *state;
> +    libxl__domain_suspend_state *dss;
> +    char *dom_path, *xs_store_mfn, *xs_console_mfn;
> +    uint32_t domid_out;
> +    int rc;
> +
> +    GCNEW(srs);
> +    cdcs = &srs->cdcs;
> +    dcs = &cdcs->dcs;
> +    state = &dcs->build_state;
> +    dss = &srs->dss;
> +
> +    srs->cdcs.dcs.ao = ao;
> +    srs->cdcs.dcs.guest_config = d_config;
> +    libxl_domain_config_init(&srs->cdcs.dcs.guest_config_saved);
> +    libxl_domain_config_copy(ctx, &srs->cdcs.dcs.guest_config_saved,
> +                             d_config);
> +    cdcs->dcs.restore_fd = -1;
> +    cdcs->dcs.domid_soft_reset = domid_soft_reset;
> +    cdcs->dcs.callback = domain_create_cb;
> +    libxl__ao_progress_gethow(&srs->cdcs.dcs.aop_console_how,
> +                              aop_console_how);
> +    cdcs->domid_out = &domid_out;
> +
> +    dom_path = libxl__xs_get_dompath(gc, domid_soft_reset);
> +    if (!dom_path) {
> +        LOG(ERROR, "failed to read domain path");
> +        return AO_CREATE_FAIL(ERROR_FAIL);

Sorry for not noticing this earlier. My bad and apologise.

This should be

       if (!dom_path) {
           LOG(...);
           rc = ERROR_FAIL;
	   goto out;
       }

I.e. please use goto style error handling.

> +    }
> +
> +    xs_store_mfn = xs_read(ctx->xsh, XBT_NULL,
> +                           GCSPRINTF("%s/store/ring-ref", dom_path),
> +                           NULL);
> +    state->store_mfn = xs_store_mfn ? atol(xs_store_mfn): 0;
> +    free(xs_store_mfn);
> +
> +    xs_console_mfn = xs_read(ctx->xsh, XBT_NULL,
> +                             GCSPRINTF("%s/console/ring-ref", dom_path),
> +                             NULL);
> +    state->console_mfn = xs_console_mfn ? atol(xs_console_mfn): 0;
> +    free(xs_console_mfn);
> +
> +    dss->ao = ao;
> +    dss->domid = domid_soft_reset;
> +    dss->dm_savefile = GCSPRINTF(LIBXL_DEVICE_MODEL_SAVE_FILE".%d",
> +                                 domid_soft_reset);
> +
> +    rc = libxl__save_emulator_xenstore_data(dss, &srs->toolstack_buf,
> +                                            &srs->toolstack_len);
> +    if (rc) {
> +        LOG(ERROR, "failed to save toolstack record.");
> +        return AO_CREATE_FAIL(ERROR_FAIL);

Ditto, please use goto style error handling.

Furthermore please preserve rc from libxl__save_emulator_xenstore_data
instead of rewriting it with ERROR_FAIL.


> +    }
> +
> +    rc = libxl__domain_suspend_device_model(gc, dss);
> +    if (rc) {
> +        LOG(ERROR, "failed to suspend device model.");
> +        return AO_CREATE_FAIL(ERROR_FAIL);

Same here.

> +    }
> +
> +    /*
> +     * Ask all backends to disconnect by removing the domain from
> +     * xenstore. On the creation path the domain will be introduced to
> +     * xenstore again with probably different store/console/...
> +     * channels.
> +     */
> +    xs_release_domain(ctx->xsh, cdcs->dcs.domid_soft_reset);
> +
> +    srs->dds.ao = ao;
> +    srs->dds.domid = domid_soft_reset;
> +    srs->dds.callback = domain_soft_reset_cb;
> +    srs->dds.soft_reset = true;
> +    libxl__domain_destroy(egc, &srs->dds);
> +
> +    return AO_INPROGRESS;

out:
      return AO_CREATE_FAIL(rc);

> +}
> +

Wei.

Re: [PATCH v11 11/11] (lib)xl: soft reset support

[Vitaly Kuznetsov]

Wei Liu <wei.liu2@citrix.com> writes:

> Sorry for the delay.
>
> FYI all other patches of this series were applied by Jan. You only need
> to resend this one.

Cool, I will.

>
> See below for a few comments.
>
> On Fri, Sep 04, 2015 at 03:39:51PM +0200, Vitaly Kuznetsov wrote:
>> Use existing create/restore path to perform 'soft reset' for HVM
>> domains. Tear everything down, e.g. destroy domain's device model,
>> remove the domain from xenstore, save toolstack record and start
>> over.
>> 
>> Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
>> ---
>> Changes since v10:
>> - Adapt to 'migration v2' changes.
>> - Use LIBXL_DEVICE_MODEL_SAVE_FILE as Qemu save file (and rename it to
>>   LIBXL_DEVICE_MODEL_RESTORE_FILE later) to support stubdom case (as
>>   we connect consoles to both files on create.
>> - Fix coding style, minor description change in xl.cfg.pod.5 [Wei Liu]
>> 
>> Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
>> ---
>>  docs/man/xl.cfg.pod.5        |   8 +-
>>  tools/libxl/libxl.c          |  22 ++++-
>>  tools/libxl/libxl.h          |  15 ++++
>>  tools/libxl/libxl_create.c   | 192 ++++++++++++++++++++++++++++++++++++++-----
>>  tools/libxl/libxl_internal.h |   4 +
>>  tools/libxl/libxl_types.idl  |   2 +
>>  tools/libxl/xl.h             |   1 +
>>  tools/libxl/xl_cmdimpl.c     |  25 +++++-
>>  8 files changed, 242 insertions(+), 27 deletions(-)
>> 
>> diff --git a/docs/man/xl.cfg.pod.5 b/docs/man/xl.cfg.pod.5
>> index c6345b8..d8c4186 100644
>> --- a/docs/man/xl.cfg.pod.5
>> +++ b/docs/man/xl.cfg.pod.5
>> @@ -349,6 +349,12 @@ destroy the domain.
>>  write a "coredump" of the domain to F</var/lib/xen/dump/NAME> and then
>>  restart the domain.
>>  
>> +=item B<soft-reset>
>> +
>> +Reset all Xen specific interfaces for the Xen-aware HVM domain allowing
>> +it to reestablish these interfaces and continue executing the domain. PV
>> +guest is not supported.
>> +
>
> And "non-Xen-aware HVM will crash" ?

Sorry, I should have replied to that suggestion earlier. Non-Xen-aware
HVM guest can't really trigger this action and (in theory) is capable of
doing kexec without any assistance.

> If there is no definite answer to
> guest state maybe just saying "PV guest and non-Xen-aware HVM guests are
> not supported" ?

This sounds correct.

>
> It's important to let user know about the consequence because libxl
> doesn't actually stop you from soft-resetting a HVM guest that is not
> Xen-aware.

The question is who (and when/how) is going to trigger this action? In
case someone does that while HVM domain (doesn't really matter if it is
Xen-aware or not) does not expect this action it will crash.

*In theory* nothing bad is going to happen to a non-Xen-aware HVM guest
if someone else will trigger this action for it (e.g. on 'reset'
signal), it will just get an assistance it doesn't need.

[...]

-- 
  Vitaly

Re: [PATCH v11 11/11] (lib)xl: soft reset support

[Wei Liu]

On Mon, Sep 14, 2015 at 06:54:51PM +0200, Vitaly Kuznetsov wrote:
> Wei Liu <wei.liu2@citrix.com> writes:
> 
> > Sorry for the delay.
> >
> > FYI all other patches of this series were applied by Jan. You only need
> > to resend this one.
> 
> Cool, I will.
> 
> >
> > See below for a few comments.
> >
> > On Fri, Sep 04, 2015 at 03:39:51PM +0200, Vitaly Kuznetsov wrote:
> >> Use existing create/restore path to perform 'soft reset' for HVM
> >> domains. Tear everything down, e.g. destroy domain's device model,
> >> remove the domain from xenstore, save toolstack record and start
> >> over.
> >> 
> >> Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
> >> ---
> >> Changes since v10:
> >> - Adapt to 'migration v2' changes.
> >> - Use LIBXL_DEVICE_MODEL_SAVE_FILE as Qemu save file (and rename it to
> >>   LIBXL_DEVICE_MODEL_RESTORE_FILE later) to support stubdom case (as
> >>   we connect consoles to both files on create.
> >> - Fix coding style, minor description change in xl.cfg.pod.5 [Wei Liu]
> >> 
> >> Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
> >> ---
> >>  docs/man/xl.cfg.pod.5        |   8 +-
> >>  tools/libxl/libxl.c          |  22 ++++-
> >>  tools/libxl/libxl.h          |  15 ++++
> >>  tools/libxl/libxl_create.c   | 192 ++++++++++++++++++++++++++++++++++++++-----
> >>  tools/libxl/libxl_internal.h |   4 +
> >>  tools/libxl/libxl_types.idl  |   2 +
> >>  tools/libxl/xl.h             |   1 +
> >>  tools/libxl/xl_cmdimpl.c     |  25 +++++-
> >>  8 files changed, 242 insertions(+), 27 deletions(-)
> >> 
> >> diff --git a/docs/man/xl.cfg.pod.5 b/docs/man/xl.cfg.pod.5
> >> index c6345b8..d8c4186 100644
> >> --- a/docs/man/xl.cfg.pod.5
> >> +++ b/docs/man/xl.cfg.pod.5
> >> @@ -349,6 +349,12 @@ destroy the domain.
> >>  write a "coredump" of the domain to F</var/lib/xen/dump/NAME> and then
> >>  restart the domain.
> >>  
> >> +=item B<soft-reset>
> >> +
> >> +Reset all Xen specific interfaces for the Xen-aware HVM domain allowing
> >> +it to reestablish these interfaces and continue executing the domain. PV
> >> +guest is not supported.
> >> +
> >
> > And "non-Xen-aware HVM will crash" ?
> 
> Sorry, I should have replied to that suggestion earlier. Non-Xen-aware
> HVM guest can't really trigger this action and (in theory) is capable of
> doing kexec without any assistance.
> 
> > If there is no definite answer to
> > guest state maybe just saying "PV guest and non-Xen-aware HVM guests are
> > not supported" ?
> 
> This sounds correct.
> 
> >
> > It's important to let user know about the consequence because libxl
> > doesn't actually stop you from soft-resetting a HVM guest that is not
> > Xen-aware.
> 
> The question is who (and when/how) is going to trigger this action? In
> case someone does that while HVM domain (doesn't really matter if it is
> Xen-aware or not) does not expect this action it will crash.
> 
> *In theory* nothing bad is going to happen to a non-Xen-aware HVM guest
> if someone else will trigger this action for it (e.g. on 'reset'
> signal), it will just get an assistance it doesn't need.
> 

OK then. We should still make it clear it is unsupported for
non-Xen-aware HVM guest even if it happens to work.

Wei.

> [...]
> 
> -- 
>   Vitaly

Re: [PATCH v11 11/11] (lib)xl: soft reset support

[Ian Campbell]

On Mon, 2015-09-14 at 17:15 +0100, Wei Liu wrote:
> FYI all other patches of this series were applied by Jan. You only need
> to resend this one.

Jan,

I appreciate you do not want to do so routinely for all commits you make
but when you are partially applying a series it would be very useful if you
could drop a line to the thread (i.e. the 00/NN mail) explaining what you
have applied vs not applied. Particularly when it is ten elevenths of the
series and so easy to miss that the eleventh patch has not been included.

I was under the incorrect impression that all 11 patches had gone in so had
removed this work from my queue.

Thanks,
Ian.

> 
> See below for a few comments.
> 
> On Fri, Sep 04, 2015 at 03:39:51PM +0200, Vitaly Kuznetsov wrote:
> > Use existing create/restore path to perform 'soft reset' for HVM
> > domains. Tear everything down, e.g. destroy domain's device model,
> > remove the domain from xenstore, save toolstack record and start
> > over.
> > 
> > Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
> > ---
> > Changes since v10:
> > - Adapt to 'migration v2' changes.
> > - Use LIBXL_DEVICE_MODEL_SAVE_FILE as Qemu save file (and rename it to
> >   LIBXL_DEVICE_MODEL_RESTORE_FILE later) to support stubdom case (as
> >   we connect consoles to both files on create.
> > - Fix coding style, minor description change in xl.cfg.pod.5 [Wei Liu]
> > 
> > Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
> > ---
> >  docs/man/xl.cfg.pod.5        |   8 +-
> >  tools/libxl/libxl.c          |  22 ++++-
> >  tools/libxl/libxl.h          |  15 ++++
> >  tools/libxl/libxl_create.c   | 192
> > ++++++++++++++++++++++++++++++++++++++-----
> >  tools/libxl/libxl_internal.h |   4 +
> >  tools/libxl/libxl_types.idl  |   2 +
> >  tools/libxl/xl.h             |   1 +
> >  tools/libxl/xl_cmdimpl.c     |  25 +++++-
> >  8 files changed, 242 insertions(+), 27 deletions(-)
> > 
> > diff --git a/docs/man/xl.cfg.pod.5 b/docs/man/xl.cfg.pod.5
> > index c6345b8..d8c4186 100644
> > --- a/docs/man/xl.cfg.pod.5
> > +++ b/docs/man/xl.cfg.pod.5
> > @@ -349,6 +349,12 @@ destroy the domain.
> >  write a "coredump" of the domain to F</var/lib/xen/dump/NAME> and then
> >  restart the domain.
> >  
> > +=item B<soft-reset>
> > +
> > +Reset all Xen specific interfaces for the Xen-aware HVM domain
> > allowing
> > +it to reestablish these interfaces and continue executing the domain.
> > PV
> > +guest is not supported.
> > +
> 
> And "non-Xen-aware HVM will crash" ?  If there is no definite answer to
> guest state maybe just saying "PV guest and non-Xen-aware HVM guests are
> not supported" ?
> 
> It's important to let user know about the consequence because libxl
> doesn't actually stop you from soft-resetting a HVM guest that is not
> Xen-aware.
> 
> [...]
> > +static int do_domain_soft_reset(libxl_ctx *ctx,
> > +                                libxl_domain_config *d_config,
> > +                                uint32_t domid_soft_reset,
> > +                                const libxl_asyncop_how *ao_how,
> > +                                const libxl_asyncprogress_how
> > +                                *aop_console_how)
> > +{
> > +    AO_CREATE(ctx, 0, ao_how);
> > +    libxl__domain_soft_reset_state *srs;
> > +    libxl__app_domain_create_state *cdcs;
> > +    libxl__domain_create_state *dcs;
> > +    libxl__domain_build_state *state;
> > +    libxl__domain_suspend_state *dss;
> > +    char *dom_path, *xs_store_mfn, *xs_console_mfn;
> > +    uint32_t domid_out;
> > +    int rc;
> > +
> > +    GCNEW(srs);
> > +    cdcs = &srs->cdcs;
> > +    dcs = &cdcs->dcs;
> > +    state = &dcs->build_state;
> > +    dss = &srs->dss;
> > +
> > +    srs->cdcs.dcs.ao = ao;
> > +    srs->cdcs.dcs.guest_config = d_config;
> > +    libxl_domain_config_init(&srs->cdcs.dcs.guest_config_saved);
> > +    libxl_domain_config_copy(ctx, &srs->cdcs.dcs.guest_config_saved,
> > +                             d_config);
> > +    cdcs->dcs.restore_fd = -1;
> > +    cdcs->dcs.domid_soft_reset = domid_soft_reset;
> > +    cdcs->dcs.callback = domain_create_cb;
> > +    libxl__ao_progress_gethow(&srs->cdcs.dcs.aop_console_how,
> > +                              aop_console_how);
> > +    cdcs->domid_out = &domid_out;
> > +
> > +    dom_path = libxl__xs_get_dompath(gc, domid_soft_reset);
> > +    if (!dom_path) {
> > +        LOG(ERROR, "failed to read domain path");
> > +        return AO_CREATE_FAIL(ERROR_FAIL);
> 
> Sorry for not noticing this earlier. My bad and apologise.
> 
> This should be
> 
>        if (!dom_path) {
>            LOG(...);
>            rc = ERROR_FAIL;
> 	   goto out;
>        }
> 
> I.e. please use goto style error handling.
> 
> > +    }
> > +
> > +    xs_store_mfn = xs_read(ctx->xsh, XBT_NULL,
> > +                           GCSPRINTF("%s/store/ring-ref", dom_path),
> > +                           NULL);
> > +    state->store_mfn = xs_store_mfn ? atol(xs_store_mfn): 0;
> > +    free(xs_store_mfn);
> > +
> > +    xs_console_mfn = xs_read(ctx->xsh, XBT_NULL,
> > +                             GCSPRINTF("%s/console/ring-ref",
> > dom_path),
> > +                             NULL);
> > +    state->console_mfn = xs_console_mfn ? atol(xs_console_mfn): 0;
> > +    free(xs_console_mfn);
> > +
> > +    dss->ao = ao;
> > +    dss->domid = domid_soft_reset;
> > +    dss->dm_savefile = GCSPRINTF(LIBXL_DEVICE_MODEL_SAVE_FILE".%d",
> > +                                 domid_soft_reset);
> > +
> > +    rc = libxl__save_emulator_xenstore_data(dss, &srs->toolstack_buf,
> > +                                            &srs->toolstack_len);
> > +    if (rc) {
> > +        LOG(ERROR, "failed to save toolstack record.");
> > +        return AO_CREATE_FAIL(ERROR_FAIL);
> 
> Ditto, please use goto style error handling.
> 
> Furthermore please preserve rc from libxl__save_emulator_xenstore_data
> instead of rewriting it with ERROR_FAIL.
> 
> 
> > +    }
> > +
> > +    rc = libxl__domain_suspend_device_model(gc, dss);
> > +    if (rc) {
> > +        LOG(ERROR, "failed to suspend device model.");
> > +        return AO_CREATE_FAIL(ERROR_FAIL);
> 
> Same here.
> 
> > +    }
> > +
> > +    /*
> > +     * Ask all backends to disconnect by removing the domain from
> > +     * xenstore. On the creation path the domain will be introduced to
> > +     * xenstore again with probably different store/console/...
> > +     * channels.
> > +     */
> > +    xs_release_domain(ctx->xsh, cdcs->dcs.domid_soft_reset);
> > +
> > +    srs->dds.ao = ao;
> > +    srs->dds.domid = domid_soft_reset;
> > +    srs->dds.callback = domain_soft_reset_cb;
> > +    srs->dds.soft_reset = true;
> > +    libxl__domain_destroy(egc, &srs->dds);
> > +
> > +    return AO_INPROGRESS;
> 
> out:
>       return AO_CREATE_FAIL(rc);
> 
> > +}
> > +
> 
> Wei.

Re: [PATCH v11 11/11] (lib)xl: soft reset support

[Jan Beulich]

>>> On 15.09.15 at 10:38, <ian.campbell@citrix.com> wrote:
> On Mon, 2015-09-14 at 17:15 +0100, Wei Liu wrote:
>> FYI all other patches of this series were applied by Jan. You only need
>> to resend this one.
> 
> I appreciate you do not want to do so routinely for all commits you make
> but when you are partially applying a series it would be very useful if you
> could drop a line to the thread (i.e. the 00/NN mail) explaining what you
> have applied vs not applied. Particularly when it is ten elevenths of the
> series and so easy to miss that the eleventh patch has not been included.
> 
> I was under the incorrect impression that all 11 patches had gone in so had
> removed this work from my queue.

I'm sorry; I'll try to remember to do better next time. In this case,
since I explicitly got feedback from Wei that he's still planning to
look at that patch, I assumed there would be another submission
(or an explicit subsequent ack) of that one patch anyway.

Jan