* [PATCH 1/2] libxc: support to resume uncooperative HVM guests
2016-03-31 15:59 [PATCH 0/2] Remaining prerequisite patches for COLO Wei Liu
@ 2016-03-31 15:59 ` Wei Liu
2016-04-01 15:59 ` Olaf Hering
2016-03-31 15:59 ` [PATCH 2/2] libxc: Document xc_domain_resume Wei Liu
2016-04-01 13:45 ` [PATCH 0/2] Remaining prerequisite patches for COLO Ian Jackson
2 siblings, 1 reply; 8+ messages in thread
From: Wei Liu @ 2016-03-31 15:59 UTC (permalink / raw)
To: Xen-devel
Cc: Changlong Xie, Wei Liu, Wen Congyang, Ian Jackson, Yang Hongyang
From: Wen Congyang <wency@cn.fujitsu.com>
Before this patch:
1. Suspend
a. PVHVM and PV: we use the same way to suspend the guest (send the
suspend request to the guest). If the guest doesn't support evtchn, the
xenstore variant will be used, suspending the guest via XenBus control
node.
b. Pure HVM: we call xc_domain_shutdown(..., SHUTDOWN_suspend) to suspend
the guest.
2. Resume:
a. Fast path (fast=1)
Do not change the guest state. We call libxl__domain_resume(.., 1) which
calls xc_domain_resume(..., 1 /* fast=1*/) to resume the guest.
PV: Modify the return code to 1, and than call the domctl:
XEN_DOMCTL_resumedomain
PVHVM: same with PV
Pure HVM: Do nothing in modify_returncode, and than call the domctl:
XEN_DOMCTL_resumedomain
b. Slow path (fast=0)
Used when the guest's state have been changed. Will call
libxl__domain_resume(..., 0) to resume the guest.
PV: Update start info, and reset all secondary CPU states. Than call
the domctl: XEN_DOMCTL_resumedomain
PVHVM: Can not be resumed. You will get the following error message:
"Cannot resume uncooperative HVM guests"
Pure HVM: Same with PVHVM
After this patch:
1. Suspend
Unchanged.
2. Resume
a. Fast path
Unchanged.
b. Slow path
PV: Unchanged.
PVHVM: Call XEN_DOMCTL_resumedomain to resume the guest. Because we
don't modify the return code, PV drivers will disconnect
and reconnect.
The guest ends up doing the XENMAPSPACE_shared_info
XENMEM_add_to_physmap hypercall and resetting all of its CPU
states to point to the shared_info (except the ones past 32).
That is the Linux kernel does that - regardless whether the
SCHEDOP_shutdown:SHUTDOWN_suspend returns 1 or not.
Pure HVM: Call XEN_DOMCTL_resumedomain to resume the guest.
Under COLO, we will update the guest's state (modify memory, CPU registers,
device state etc). In this case, we cannot use the fast path to resume it.
Keep the return code 0, and use the slow path to resume the guest.
Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
Signed-off-by: Yang Hongyang <hongyang.yang@easystack.cn>
Reviewed-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
[ wei: reformat commit message a bit ]
Signed-off-by: Wei Liu <wei.liu2@citrix.com>
---
tools/libxc/xc_resume.c | 25 +++++++++++++++++++++----
1 file changed, 21 insertions(+), 4 deletions(-)
diff --git a/tools/libxc/xc_resume.c b/tools/libxc/xc_resume.c
index 5bf4cf0..6bf7d9c 100644
--- a/tools/libxc/xc_resume.c
+++ b/tools/libxc/xc_resume.c
@@ -109,6 +109,26 @@ static int xc_domain_resume_cooperative(xc_interface *xch, uint32_t domid)
return do_domctl(xch, &domctl);
}
+static int xc_domain_resume_hvm(xc_interface *xch, uint32_t domid)
+{
+ DECLARE_DOMCTL;
+
+ /*
+ * The domctl XEN_DOMCTL_resumedomain unpause each vcpu. After
+ * the domctl, the guest will run.
+ *
+ * If it is PVHVM, the guest called the hypercall
+ * SCHEDOP_shutdown:SHUTDOWN_suspend
+ * to suspend itself. We don't modify the return code, so the PV driver
+ * will disconnect and reconnect.
+ *
+ * If it is a HVM, the guest will continue running.
+ */
+ domctl.cmd = XEN_DOMCTL_resumedomain;
+ domctl.domain = domid;
+ return do_domctl(xch, &domctl);
+}
+
static int xc_domain_resume_any(xc_interface *xch, uint32_t domid)
{
DECLARE_DOMCTL;
@@ -138,10 +158,7 @@ static int xc_domain_resume_any(xc_interface *xch, uint32_t domid)
*/
#if defined(__i386__) || defined(__x86_64__)
if ( info.hvm )
- {
- ERROR("Cannot resume uncooperative HVM guests");
- return rc;
- }
+ return xc_domain_resume_hvm(xch, domid);
if ( xc_domain_get_guest_width(xch, domid, &dinfo->guest_width) != 0 )
{
--
2.1.4
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 8+ messages in thread* [PATCH 2/2] libxc: Document xc_domain_resume
2016-03-31 15:59 [PATCH 0/2] Remaining prerequisite patches for COLO Wei Liu
2016-03-31 15:59 ` [PATCH 1/2] libxc: support to resume uncooperative HVM guests Wei Liu
@ 2016-03-31 15:59 ` Wei Liu
2016-04-01 13:45 ` [PATCH 0/2] Remaining prerequisite patches for COLO Ian Jackson
2 siblings, 0 replies; 8+ messages in thread
From: Wei Liu @ 2016-03-31 15:59 UTC (permalink / raw)
To: Xen-devel; +Cc: Wei Liu, Changlong Xie, Ian Jackson, Wen Congyang
From: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Document the save and suspend mechanism.
Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Signed-off-by: Wei Liu <wei.liu2@citrix.com>
---
tools/libxc/include/xenctrl.h | 50 +++++++++++++++++++++++++++++++++++++++++++
1 file changed, 50 insertions(+)
diff --git a/tools/libxc/include/xenctrl.h b/tools/libxc/include/xenctrl.h
index a9e4dc1..013a3a3 100644
--- a/tools/libxc/include/xenctrl.h
+++ b/tools/libxc/include/xenctrl.h
@@ -565,6 +565,56 @@ int xc_domain_destroy(xc_interface *xch,
* This function resumes a suspended domain. The domain should have
* been previously suspended.
*
+ * Note that there are 'xc_domain_suspend' as suspending a domain
+ * is quite the endeavour.
+ *
+ * For the purpose of this explanation there are three guests:
+ * PV (using hypercalls for privilgied operations), HVM
+ * (fully hardware virtualized guests using emulated devices for everything),
+ * and PVHVM (PV aware with hardware virtualisation).
+ *
+ * HVM guest are the simplest - they suspend via S3 / S4 and resume from
+ * S3 / S4. Upon resume they have to re-negotiate with the emulated devices.
+ *
+ * PV and PVHVM communicate via hypercalls for suspend (and resume).
+ * For suspend the toolstack initiates the process by writing an value
+ * in XenBus "control/shutdown" with the string "suspend".
+ *
+ * The PV guest stashes anything it deems neccessary in 'struct
+ * start_info' in case of failure (PVHVM may ignore this) and calls
+ * the SCHEDOP_shutdown::SHUTDOWN_suspend hypercall (for PV as
+ * argument it passes the MFN to 'struct start_info').
+ *
+ * And then the guest is suspended.
+ *
+ * The checkpointing or notifying a guest that the suspend failed or
+ * cancelled (in case of checkpoint) is by having the
+ * SCHEDOP_shutdown::SHUTDOWN_suspend hypercall return a non-zero
+ * value.
+ *
+ * The PV and PVHVM resume path are similar. For PV it would be
+ * similar to bootup - figure out where the 'struct start_info' is (or
+ * if the suspend was cancelled aka checkpointed - reuse the saved
+ * values).
+ *
+ * From here on they differ depending whether the guest is PV or PVHVM
+ * in specifics but follow overall the same path:
+ * - PV: Bringing up the vCPUS,
+ * - PVHVM: Setup vector callback,
+ * - Bring up vCPU runstates,
+ * - Remap the grant tables if checkpointing or setup from scratch,
+ *
+ *
+ * If the resume was not checkpointing (or if suspend was succesful) we would
+ * setup the PV timers and the different PV events. Lastly the PV drivers
+ * re-negotiate with the backend.
+ *
+ * This function would return before the guest started resuming. That is
+ * the guest would be in non-running state and its vCPU context would be
+ * in the the SCHEDOP_shutdown::SHUTDOWN_suspend hypercall return path
+ * (for PV and PVHVM). For HVM it would be in would be in QEMU emulated
+ * BIOS handling S3 suspend.
+ *
* @parm xch a handle to an open hypervisor interface
* @parm domid the domain id to resume
* @parm fast use cooperative resume (guest must support this)
--
2.1.4
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 8+ messages in thread