From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.8 required=3.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,HTML_MESSAGE,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 931FCC07E96 for ; Thu, 8 Jul 2021 12:29:19 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id D959B61396 for ; Thu, 8 Jul 2021 12:29:18 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org D959B61396 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:39270 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1m1T9Z-0001BJ-Tq for qemu-devel@archiver.kernel.org; Thu, 08 Jul 2021 08:29:17 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:60382) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1m1T6a-0005eX-7P for qemu-devel@nongnu.org; Thu, 08 Jul 2021 08:26:12 -0400 Received: from mail-ej1-x635.google.com ([2a00:1450:4864:20::635]:44749) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1m1T6W-0008D9-SS for qemu-devel@nongnu.org; Thu, 08 Jul 2021 08:26:11 -0400 Received: by mail-ej1-x635.google.com with SMTP id he13so9261155ejc.11 for ; Thu, 08 Jul 2021 05:26:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=nOPzsmni+igZnchMBfXRsnH3K30lk4O7eSzMw+/++VQ=; b=talgFxfh6rx9nKmEMRCvSX4uKIB9rVCfu922YGqgKgSSuozfXqCutXc7KBoyE30MrD zTGfhHPSXJhQxryFJ9Fr+Q6mJYn8wxPy/XxCn+Pf4VC+M/VMUGx+wgUGC3DsDsyW25Hz cHm++Vavu/zh+JLrBWEYhsLU34jHyA5Gqk4Rq8VWUHT9a1MNCW4OmkSTVpxTqwMaLDCK XlZyVWdXs2MkFswvMYlESwZo5NoxPsF2j4sMa+hzqsKD/NkJMLVeUNifYn+zxgQtvmjV 03wCsK/VI7m6CDgWgO4YbavqxbV9Kwn2uwjq8KZijRbGh9BuD0QLYNjB6K6TUaegToRe Bo4Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=nOPzsmni+igZnchMBfXRsnH3K30lk4O7eSzMw+/++VQ=; b=kytxT0o7ypLH/HBi3tJDH978OLrs7Do8jiOkspWgKzvn0Jd6VrycNrefo/eiooEDLw x2UdGIxtNT1N3BhJBVofLSkQ9SpL7GzEJWv8CcI1vNjCMWD6spQJnzogpOSjkrxYCoPs vjSFPYMEKt9Qbk7U5Z91IRIVEKQYitoOZwhy7T3JnwWSqHViJCyu+2W2iYdu1ok8ISiE nyDyI3ejuXftbzsTNTK4CfutZBQ0c+TMOzFCONn2aNV7jjFdOVxZFpk8xUgSf0H9+ID9 JJKtpyYZJ+FrFqwZ/Y5m12WqC/LRFmUOqWxe3qbFno32DOd9lPaDs20QF7tMopg68wDi 8DQw== X-Gm-Message-State: AOAM530IDChDd+1dpjI48uDazXFYZQ30lZ/u7bBzuu1YxbvosAWBhS2Q Y6tDg7u/bNPSh8wEVjCjaMPT4NftS/L5VrVWqmk= X-Google-Smtp-Source: ABdhPJy5ypxyzeboXhXlkbRbtNf8ndmiZDGpbHbMce/lJCHBltLsVHD0cXiS7JM9BLtn5lCtov1JZNlNyWGit5Cd0YA= X-Received: by 2002:a17:906:4fd6:: with SMTP id i22mr17217105ejw.92.1625747167435; Thu, 08 Jul 2021 05:26:07 -0700 (PDT) MIME-Version: 1.0 References: <1625678434-240960-1-git-send-email-steven.sistare@oracle.com> <1625678434-240960-3-git-send-email-steven.sistare@oracle.com> In-Reply-To: <1625678434-240960-3-git-send-email-steven.sistare@oracle.com> From: =?UTF-8?B?TWFyYy1BbmRyw6kgTHVyZWF1?= Date: Thu, 8 Jul 2021 16:25:55 +0400 Message-ID: Subject: Re: [PATCH V5 02/25] cpr: reboot mode To: Steve Sistare Content-Type: multipart/alternative; boundary="00000000000068e1b605c69bc29d" Received-SPF: pass client-ip=2a00:1450:4864:20::635; envelope-from=marcandre.lureau@gmail.com; helo=mail-ej1-x635.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Jason Zeng , Juan Quintela , Eric Blake , "Michael S. Tsirkin" , QEMU , "Dr. David Alan Gilbert" , Alex Williamson , Stefan Hajnoczi , Paolo Bonzini , "Daniel P. Berrange" , =?UTF-8?Q?Philippe_Mathieu=2DDaud=C3=A9?= , =?UTF-8?B?QWxleCBCZW5uw6ll?= , Markus Armbruster Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" --00000000000068e1b605c69bc29d Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hi On Wed, Jul 7, 2021 at 9:45 PM Steve Sistare wrote: > Provide the cprsave and cprload functions for live update. These save an= d > restore VM state, with minimal guest pause time, so that qemu may be > updated > to a new version in between. > > cprsave stops the VM and saves vmstate to an ordinary file. It supports > any > type of guest image and block device, but the caller must not modify gues= t > block devices between cprsave and cprload. > > cprsave supports several modes, the first of which is reboot. In this > mode, > the caller invokes cprsave and then terminates qemu. The caller may then > update the host kernel and system software and reboot. The caller resume= s > the guest by running qemu with the same arguments as the original process > and invoking cprload. To use this mode, guest ram must be mapped to a > persistent shared memory file such as /dev/dax0.0 or /dev/shm PKRAM. > > The reboot mode supports vfio devices if the caller first suspends the > guest, such as by issuing guest-suspend-ram to the qemu guest agent. The > guest drivers' suspend methods flush outstanding requests and re-initiali= ze > the devices, and thus there is no device state to save and restore. > > cprload loads state from the file. If the VM was running at cprsave time= , > then VM execution resumes. If the VM was suspended at cprsave time, then > the caller must issue a system_wakeup command to resume. > > Signed-off-by: Steve Sistare > --- > MAINTAINERS | 7 +++ > include/migration/cpr.h | 17 ++++++ > include/sysemu/runstate.h | 1 + > migration/cpr.c | 149 > ++++++++++++++++++++++++++++++++++++++++++++++ > migration/meson.build | 1 + > migration/savevm.h | 2 + > softmmu/runstate.c | 21 ++++++- > 7 files changed, 197 insertions(+), 1 deletion(-) > create mode 100644 include/migration/cpr.h > create mode 100644 migration/cpr.c > > diff --git a/MAINTAINERS b/MAINTAINERS > index 684142e..c3573aa 100644 > --- a/MAINTAINERS > +++ b/MAINTAINERS > @@ -2858,6 +2858,13 @@ F: net/colo* > F: net/filter-rewriter.c > F: net/filter-mirror.c > > +CPR > +M: Steve Sistare > +M: Mark Kanda > +S: Maintained > +F: include/migration/cpr.h > +F: migration/cpr.c > + > Record/replay > M: Pavel Dovgalyuk > R: Paolo Bonzini > diff --git a/include/migration/cpr.h b/include/migration/cpr.h > new file mode 100644 > index 0000000..bffee19 > --- /dev/null > +++ b/include/migration/cpr.h > @@ -0,0 +1,17 @@ > +/* > + * Copyright (c) 2021 Oracle and/or its affiliates. > + * > + * This work is licensed under the terms of the GNU GPL, version 2. > + * See the COPYING file in the top-level directory. > + */ > + > +#ifndef MIGRATION_CPR_H > +#define MIGRATION_CPR_H > + > +#include "qapi/qapi-types-cpr.h" > + > +void cprsave(const char *file, CprMode mode, Error **errp); > I'd rather use "path" or "filename". +void cprexec(strList *args, Error **errp); > +void cprload(const char *file, Error **errp); > same It's recommended to return a bool value TRUE for success. (see include/qapi/error.h) + > +#endif > diff --git a/include/sysemu/runstate.h b/include/sysemu/runstate.h > index a535691..ed4b735 100644 > --- a/include/sysemu/runstate.h > +++ b/include/sysemu/runstate.h > @@ -51,6 +51,7 @@ void qemu_system_reset_request(ShutdownCause reason); > void qemu_system_suspend_request(void); > void qemu_register_suspend_notifier(Notifier *notifier); > bool qemu_wakeup_suspend_enabled(void); > +void qemu_system_start_on_wake_request(void); > I suggest introducing the function in a preliminary commit. Also for consistency with the rest of symbols, use "wakeup". void qemu_system_wakeup_request(WakeupReason reason, Error **errp); > void qemu_system_wakeup_enable(WakeupReason reason, bool enabled); > void qemu_register_wakeup_notifier(Notifier *notifier); > diff --git a/migration/cpr.c b/migration/cpr.c > new file mode 100644 > index 0000000..c5bad8a > --- /dev/null > +++ b/migration/cpr.c > @@ -0,0 +1,149 @@ > +/* > + * Copyright (c) 2021 Oracle and/or its affiliates. > + * > + * This work is licensed under the terms of the GNU GPL, version 2. > + * See the COPYING file in the top-level directory. > + */ > + > +#include "qemu/osdep.h" > +#include "monitor/monitor.h" > +#include "migration.h" > +#include "migration/snapshot.h" > +#include "chardev/char.h" > +#include "migration/misc.h" > +#include "migration/cpr.h" > +#include "migration/global_state.h" > +#include "qemu-file-channel.h" > +#include "qemu-file.h" > +#include "savevm.h" > +#include "qapi/error.h" > +#include "qapi/qmp/qerror.h" > +#include "qemu/error-report.h" > +#include "io/channel-buffer.h" > +#include "io/channel-file.h" > +#include "sysemu/cpu-timers.h" > +#include "sysemu/runstate.h" > +#include "sysemu/runstate-action.h" > +#include "sysemu/sysemu.h" > +#include "sysemu/replay.h" > +#include "sysemu/xen.h" > +#include "hw/vfio/vfio-common.h" > +#include "hw/virtio/vhost.h" > + > +QEMUFile *qf_file_open(const char *path, int flags, int mode, > + const char *name, Error **errp) > None of our functions have qf_ prefix. We are not very consistent with QEMUFile functions, but I suggest to spell it out qemu_file_open(). Also, it should probably be in migration/qemu-file.c. +{ > I'd ERRP_GUARD on every function with an errp argument. + QIOChannelFile *fioc; > Let's not miss an opportunity to use g_auto g_autoptr(QIOChannelFile) fioc =3D NULL; + QIOChannel *ioc; > + QEMUFile *f; > + > + if (flags & O_RDWR) { > + error_setg(errp, "qf_file_open %s: O_RDWR not supported", path); > + return 0; > + } > + > + fioc =3D qio_channel_file_new_path(path, flags, mode, errp); > + if (!fioc) { > + return 0; > + } > + > + ioc =3D QIO_CHANNEL(fioc); > + qio_channel_set_name(ioc, name); > + f =3D (flags & O_WRONLY) ? qemu_fopen_channel_output(ioc) : > + qemu_fopen_channel_input(ioc); > + object_unref(OBJECT(fioc)); With g_auto, can be removed, and value returned directly. + return f; > +} > + > +void cprsave(const char *file, CprMode mode, Error **errp) > +{ > + int ret; > + QEMUFile *f; > + int saved_vm_running =3D runstate_is_running(); > + > + if (mode =3D=3D CPR_MODE_REBOOT && qemu_ram_volatile(errp)) { > + return; > + } > + > + if (migrate_colo_enabled()) { > + error_setg(errp, "error: cprsave does not support x-colo"); > Remove error: + return; > + } > + > + if (replay_mode !=3D REPLAY_MODE_NONE) { > + error_setg(errp, "error: cprsave does not support replay"); > same + return; > + } > + > + f =3D qf_file_open(file, O_CREAT | O_WRONLY | O_TRUNC, 0600, "cprsav= e", > errp); > + if (!f) { > + return; > + } > + > + if (global_state_store()) { > + error_setg(errp, "Error saving global state"); > + qemu_fclose(f); > + return; > + } > Could be called before opening cprsave file? + if (runstate_check(RUN_STATE_SUSPENDED)) { > + /* Update timers_state before saving. Suspend did not so do. */ > + cpu_disable_ticks(); > + } > + vm_stop(RUN_STATE_SAVE_VM); > + > + ret =3D qemu_save_device_state(f); > + qemu_fclose(f); > + if (ret < 0) { > + error_setg(errp, "Error %d while saving VM state", ret); > + goto err; > Needless goto / labels. > + } > + > + goto done; > + > +err: > + if (saved_vm_running) { > + vm_start(); > + } > +done: > + return; > +} > + > +void cprload(const char *file, Error **errp) > +{ > + QEMUFile *f; > + int ret; > + RunState state; > + > + if (runstate_is_running()) { > + error_setg(errp, "cprload called for a running VM"); > + return; > + } > + > + f =3D qf_file_open(file, O_RDONLY, 0, "cprload", errp); > + if (!f) { > + return; > + } > + > + if (qemu_get_be32(f) !=3D QEMU_VM_FILE_MAGIC || > + qemu_get_be32(f) !=3D QEMU_VM_FILE_VERSION) { > + error_setg(errp, "error: %s is not a vmstate file", file); > f is leaked + return; > + } > + > + ret =3D qemu_load_device_state(f); > + qemu_fclose(f); > + if (ret < 0) { > + error_setg(errp, "Error %d while loading VM state", ret); > + return; > + } > + > + state =3D global_state_get_runstate(); > + if (state =3D=3D RUN_STATE_RUNNING) { > + vm_start(); > + } else { > + runstate_set(state); > + if (runstate_check(RUN_STATE_SUSPENDED)) { > + qemu_system_start_on_wake_request(); > + } > + } > +} > diff --git a/migration/meson.build b/migration/meson.build > index f8714dc..fd59281 100644 > --- a/migration/meson.build > +++ b/migration/meson.build > @@ -15,6 +15,7 @@ softmmu_ss.add(files( > 'channel.c', > 'colo-failover.c', > 'colo.c', > + 'cpr.c', > 'exec.c', > 'fd.c', > 'global_state.c', > diff --git a/migration/savevm.h b/migration/savevm.h > index 6461342..ce5d710 100644 > --- a/migration/savevm.h > +++ b/migration/savevm.h > @@ -67,5 +67,7 @@ int qemu_loadvm_state_main(QEMUFile *f, > MigrationIncomingState *mis); > int qemu_load_device_state(QEMUFile *f); > int qemu_savevm_state_complete_precopy_non_iterable(QEMUFile *f, > bool in_postcopy, bool inactivate_disks); > +QEMUFile *qf_file_open(const char *path, int flags, int mode, > + const char *name, Error **errp); > > #endif > diff --git a/softmmu/runstate.c b/softmmu/runstate.c > index 10d9b73..7fe4967 100644 > --- a/softmmu/runstate.c > +++ b/softmmu/runstate.c > @@ -115,6 +115,8 @@ static const RunStateTransition > runstate_transitions_def[] =3D { > { RUN_STATE_PRELAUNCH, RUN_STATE_RUNNING }, > { RUN_STATE_PRELAUNCH, RUN_STATE_FINISH_MIGRATE }, > { RUN_STATE_PRELAUNCH, RUN_STATE_INMIGRATE }, > + { RUN_STATE_PRELAUNCH, RUN_STATE_SUSPENDED }, > + { RUN_STATE_PRELAUNCH, RUN_STATE_PAUSED }, > > { RUN_STATE_FINISH_MIGRATE, RUN_STATE_RUNNING }, > { RUN_STATE_FINISH_MIGRATE, RUN_STATE_PAUSED }, > @@ -335,6 +337,7 @@ void vm_state_notify(bool running, RunState state) > } > } > > +static bool start_on_wake_requested; > static ShutdownCause reset_requested; > static ShutdownCause shutdown_requested; > static int shutdown_signal; > @@ -562,6 +565,11 @@ void qemu_register_suspend_notifier(Notifier > *notifier) > notifier_list_add(&suspend_notifiers, notifier); > } > > +void qemu_system_start_on_wake_request(void) > +{ > + start_on_wake_requested =3D true; > +} > + > void qemu_system_wakeup_request(WakeupReason reason, Error **errp) > { > trace_system_wakeup_request(reason); > @@ -574,7 +582,18 @@ void qemu_system_wakeup_request(WakeupReason reason, > Error **errp) > if (!(wakeup_reason_mask & (1 << reason))) { > return; > } > - runstate_set(RUN_STATE_RUNNING); > + > + /* > + * Must call vm_start if it has never been called, to invoke the sta= te > + * change callbacks for the first time. > + */ > + if (start_on_wake_requested) { > + start_on_wake_requested =3D false; > + vm_start(); > + } else { > + runstate_set(RUN_STATE_RUNNING); > + } > + > wakeup_reason =3D reason; > qemu_notify_event(); > } > -- > 1.8.3.1 > > > --=20 Marc-Andr=C3=A9 Lureau --00000000000068e1b605c69bc29d Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi

On Wed, Jul 7, 2021 at 9:45 PM Stev= e Sistare <steven.sistare@o= racle.com> wrote:
Provide the cprsave and cprload functions for live update.=C2=A0= These save and
restore VM state, with minimal guest pause time, so that qemu may be update= d
to a new version in between.

cprsave stops the VM and saves vmstate to an ordinary file.=C2=A0 It suppor= ts any
type of guest image and block device, but the caller must not modify guest<= br> block devices between cprsave and cprload.

cprsave supports several modes, the first of which is reboot.=C2=A0 In this= mode,
the caller invokes cprsave and then terminates qemu.=C2=A0 The caller may t= hen
update the host kernel and system software and reboot.=C2=A0 The caller res= umes
the guest by running qemu with the same arguments as the original process and invoking cprload.=C2=A0 To use this mode, guest ram must be mapped to a=
persistent shared memory file such as /dev/dax0.0 or /dev/shm PKRAM.

The reboot mode supports vfio devices if the caller first suspends the
guest, such as by issuing guest-suspend-ram to the qemu guest agent.=C2=A0 = The
guest drivers' suspend methods flush outstanding requests and re-initia= lize
the devices, and thus there is no device state to save and restore.

cprload loads state from the file.=C2=A0 If the VM was running at cprsave t= ime,
then VM execution resumes.=C2=A0 If the VM was suspended at cprsave time, t= hen
the caller must issue a system_wakeup command to resume.

Signed-off-by: Steve Sistare <steven.sistare@oracle.com>
---
=C2=A0MAINTAINERS=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0|= =C2=A0 =C2=A07 +++
=C2=A0include/migration/cpr.h=C2=A0 =C2=A0|=C2=A0 17 ++++++
=C2=A0include/sysemu/runstate.h |=C2=A0 =C2=A01 +
=C2=A0migration/cpr.c=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0| 149 +++++++= +++++++++++++++++++++++++++++++++++++++
=C2=A0migration/meson.build=C2=A0 =C2=A0 =C2=A0|=C2=A0 =C2=A01 +
=C2=A0migration/savevm.h=C2=A0 =C2=A0 =C2=A0 =C2=A0 |=C2=A0 =C2=A02 +
=C2=A0softmmu/runstate.c=C2=A0 =C2=A0 =C2=A0 =C2=A0 |=C2=A0 21 ++++++-
=C2=A07 files changed, 197 insertions(+), 1 deletion(-)
=C2=A0create mode 100644 include/migration/cpr.h
=C2=A0create mode 100644 migration/cpr.c

diff --git a/MAINTAINERS b/MAINTAINERS
index 684142e..c3573aa 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -2858,6 +2858,13 @@ F: net/colo*
=C2=A0F: net/filter-rewriter.c
=C2=A0F: net/filter-mirror.c

+CPR
+M: Steve Sistare <steven.sistare@oracle.com>
+M: Mark Kanda <mark.kanda@oracle.com>
+S: Maintained
+F: include/migration/cpr.h
+F: migration/cpr.c
+
=C2=A0Record/replay
=C2=A0M: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
=C2=A0R: Paolo Bonzini <pbonzini@redhat.com>
diff --git a/include/migration/cpr.h b/include/migration/cpr.h
new file mode 100644
index 0000000..bffee19
--- /dev/null
+++ b/include/migration/cpr.h
@@ -0,0 +1,17 @@
+/*
+ * Copyright (c) 2021 Oracle and/or its affiliates.
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2.
+ * See the COPYING file in the top-level directory.
+ */
+
+#ifndef MIGRATION_CPR_H
+#define MIGRATION_CPR_H
+
+#include "qapi/qapi-types-cpr.h"
+
+void cprsave(const char *file, CprMode mode, Error **errp);

I'd rather use "path" or "filename= ".

+void cprexec(strList *args, Error **errp);
+void cprload(const char *file, Error **errp);

same

It's recommended to return a bool= value TRUE for success.
(see include/qapi/error.h)
+
+#endif
diff --git a/include/sysemu/runstate.h b/include/sysemu/runstate.h
index a535691..ed4b735 100644
--- a/include/sysemu/runstate.h
+++ b/include/sysemu/runstate.h
@@ -51,6 +51,7 @@ void qemu_system_reset_request(ShutdownCause reason);
=C2=A0void qemu_system_suspend_request(void);
=C2=A0void qemu_register_suspend_notifier(Notifier *notifier);
=C2=A0bool qemu_wakeup_suspend_enabled(void);
+void qemu_system_start_on_wake_request(void);

I suggest introducing the function in a preliminary commit.

Also for consistency with the rest of symbols, use "= ;wakeup".

=C2=A0void qemu_system_wakeup_request(WakeupReason reason, Error **errp); =C2=A0void qemu_system_wakeup_enable(WakeupReason reason, bool enabled); =C2=A0void qemu_register_wakeup_notifier(Notifier *notifier);
diff --git a/migration/cpr.c b/migration/cpr.c
new file mode 100644
index 0000000..c5bad8a
--- /dev/null
+++ b/migration/cpr.c
@@ -0,0 +1,149 @@
+/*
+ * Copyright (c) 2021 Oracle and/or its affiliates.
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2.
+ * See the COPYING file in the top-level directory.
+ */
+
+#include "qemu/osdep.h"
+#include "monitor/monitor.h"
+#include "migration.h"
+#include "migration/snapshot.h"
+#include "chardev/char.h"
+#include "migration/misc.h"
+#include "migration/cpr.h"
+#include "migration/global_state.h"
+#include "qemu-file-channel.h"
+#include "qemu-file.h"
+#include "savevm.h"
+#include "qapi/error.h"
+#include "qapi/qmp/qerror.h"
+#include "qemu/error-report.h"
+#include "io/channel-buffer.h"
+#include "io/channel-file.h"
+#include "sysemu/cpu-timers.h"
+#include "sysemu/runstate.h"
+#include "sysemu/runstate-action.h"
+#include "sysemu/sysemu.h"
+#include "sysemu/replay.h"
+#include "sysemu/xen.h"
+#include "hw/vfio/vfio-common.h"
+#include "hw/virtio/vhost.h"
+
+QEMUFile *qf_file_open(const char *path, int flags, int mode,
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 const char *name, Error **errp)

None of our functions have qf_ prefix. We are not = very consistent with QEMUFile functions, but I suggest to spell it out qemu= _file_open().

Also, it should probably be in migra= tion/qemu-file.c.

+{

I'd ERRP_GUARD on every function= with an errp argument.

+=C2=A0 =C2=A0 QIOChannelFile *fioc;

Le= t's not miss an opportunity to use g_auto
=C2=A0 =C2=A0 g_aut= optr(QIOChannelFile) fioc =3D NULL;

+=C2=A0 =C2=A0 QIOChannel *ioc;
+=C2=A0 =C2=A0 QEMUFile *f;
+
+=C2=A0 =C2=A0 if (flags & O_RDWR) {
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 error_setg(errp, "qf_file_open %s: O_RDWR= not supported", path);
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 return 0;
+=C2=A0 =C2=A0 }
+
+=C2=A0 =C2=A0 fioc =3D qio_channel_file_new_path(path, flags, mode, errp);=
+=C2=A0 =C2=A0 if (!fioc) {
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 return 0;
+=C2=A0 =C2=A0 }
+
+=C2=A0 =C2=A0 ioc =3D QIO_CHANNEL(fioc);
+=C2=A0 =C2=A0 qio_channel_set_name(ioc, name);
+=C2=A0 =C2=A0 f =3D (flags & O_WRONLY) ? qemu_fopen_channel_output(ioc= ) :
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0qemu_fopen_channel_input(ioc);
=C2=A0+=C2=A0 =C2=A0 object_unref(OBJECT(fioc));
=C2=A0
With g_auto, can be removed, and value returned= directly.

+=C2=A0 =C2=A0 return f;
+}
+
+void cprsave(const char *file, CprMode mode, Error **errp)
+{
+=C2=A0 =C2=A0 int ret;
+=C2=A0 =C2=A0 QEMUFile *f;
+=C2=A0 =C2=A0 int saved_vm_running =3D runstate_is_running();
+
+=C2=A0 =C2=A0 if (mode =3D=3D CPR_MODE_REBOOT && qemu_ram_volatile= (errp)) {
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 return;
+=C2=A0 =C2=A0 }
+
+=C2=A0 =C2=A0 if (migrate_colo_enabled()) {
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 error_setg(errp, "error: cprsave does not= support x-colo");

Remove error:

+=C2=A0 =C2=A0 =C2=A0 =C2=A0 return;
+=C2=A0 =C2=A0 }
+
+=C2=A0 =C2=A0 if (replay_mode !=3D REPLAY_MODE_NONE) {
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 error_setg(errp, "error: cprsave does not= support replay");

same
=
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 return;
+=C2=A0 =C2=A0 }
+
+=C2=A0 =C2=A0 f =3D qf_file_open(file, O_CREAT | O_WRONLY | O_TRUNC, 0600,= "cprsave", errp);
+=C2=A0 =C2=A0 if (!f) {
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 return;
+=C2=A0 =C2=A0 }
+
+=C2=A0 =C2=A0 if (global_state_store()) {
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 error_setg(errp, "Error saving global sta= te");
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 qemu_fclose(f);
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 return;
+=C2=A0 =C2=A0 }

Could be called before= opening cprsave file?

+=C2=A0 =C2=A0 if (runstate_check(RUN_STATE_SUSPENDED)) {
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 /* Update timers_state before saving.=C2=A0 Su= spend did not so do. */
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 cpu_disable_ticks();
+=C2=A0 =C2=A0 }
+=C2=A0 =C2=A0 vm_stop(RUN_STATE_SAVE_VM);
+
+=C2=A0 =C2=A0 ret =3D qemu_save_device_state(f);
+=C2=A0 =C2=A0 qemu_fclose(f);
+=C2=A0 =C2=A0 if (ret < 0) {
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 error_setg(errp, "Error %d while saving V= M state", ret);
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 goto err;

Needless goto / labels.
=C2= =A0
+=C2=A0 =C2=A0 }=
+
+=C2=A0 =C2=A0 goto done;
+
+err:
+=C2=A0 =C2=A0 if (saved_vm_running) {
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 vm_start();
+=C2=A0 =C2=A0 }
+done:
+=C2=A0 =C2=A0 return;
+}
+
+void cprload(const char *file, Error **errp)
+{
+=C2=A0 =C2=A0 QEMUFile *f;
+=C2=A0 =C2=A0 int ret;
+=C2=A0 =C2=A0 RunState state;
+
+=C2=A0 =C2=A0 if (runstate_is_running()) {
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 error_setg(errp, "cprload called for a ru= nning VM");
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 return;
+=C2=A0 =C2=A0 }
+
+=C2=A0 =C2=A0 f =3D qf_file_open(file, O_RDONLY, 0, "cprload", e= rrp);
+=C2=A0 =C2=A0 if (!f) {
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 return;
+=C2=A0 =C2=A0 }
+
+=C2=A0 =C2=A0 if (qemu_get_be32(f) !=3D QEMU_VM_FILE_MAGIC ||
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 qemu_get_be32(f) !=3D QEMU_VM_FILE_VERSION) {<= br> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 error_setg(errp, "error: %s is not a vmst= ate file", file);

f is leaked

+=C2=A0 =C2=A0 =C2=A0 =C2=A0 return;
+=C2=A0 =C2=A0 }
+
+=C2=A0 =C2=A0 ret =3D qemu_load_device_state(f);
+=C2=A0 =C2=A0 qemu_fclose(f);
+=C2=A0 =C2=A0 if (ret < 0) {
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 error_setg(errp, "Error %d while loading = VM state", ret);
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 return;
+=C2=A0 =C2=A0 }
+
+=C2=A0 =C2=A0 state =3D global_state_get_runstate();
+=C2=A0 =C2=A0 if (state =3D=3D RUN_STATE_RUNNING) {
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 vm_start();
+=C2=A0 =C2=A0 } else {
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 runstate_set(state);
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 if (runstate_check(RUN_STATE_SUSPENDED)) {
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 qemu_system_start_on_wake_reques= t();
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 }
+=C2=A0 =C2=A0 }
+}
diff --git a/migration/meson.build b/migration/meson.build
index f8714dc..fd59281 100644
--- a/migration/meson.build
+++ b/migration/meson.build
@@ -15,6 +15,7 @@ softmmu_ss.add(files(
=C2=A0 =C2=A0'channel.c',
=C2=A0 =C2=A0'colo-failover.c',
=C2=A0 =C2=A0'colo.c',
+=C2=A0 'cpr.c',
=C2=A0 =C2=A0'exec.c',
=C2=A0 =C2=A0'fd.c',
=C2=A0 =C2=A0'global_state.c',
diff --git a/migration/savevm.h b/migration/savevm.h
index 6461342..ce5d710 100644
--- a/migration/savevm.h
+++ b/migration/savevm.h
@@ -67,5 +67,7 @@ int qemu_loadvm_state_main(QEMUFile *f, MigrationIncoming= State *mis);
=C2=A0int qemu_load_device_state(QEMUFile *f);
=C2=A0int qemu_savevm_state_complete_precopy_non_iterable(QEMUFile *f,
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0bool in_postcopy, bool inactivate_disks);=
+QEMUFile *qf_file_open(const char *path, int flags, int mode,
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0const char *name, Error **errp);

=C2=A0#endif
diff --git a/softmmu/runstate.c b/softmmu/runstate.c
index 10d9b73..7fe4967 100644
--- a/softmmu/runstate.c
+++ b/softmmu/runstate.c
@@ -115,6 +115,8 @@ static const RunStateTransition runstate_transitions_de= f[] =3D {
=C2=A0 =C2=A0 =C2=A0{ RUN_STATE_PRELAUNCH, RUN_STATE_RUNNING },
=C2=A0 =C2=A0 =C2=A0{ RUN_STATE_PRELAUNCH, RUN_STATE_FINISH_MIGRATE },
=C2=A0 =C2=A0 =C2=A0{ RUN_STATE_PRELAUNCH, RUN_STATE_INMIGRATE },
+=C2=A0 =C2=A0 { RUN_STATE_PRELAUNCH, RUN_STATE_SUSPENDED },
+=C2=A0 =C2=A0 { RUN_STATE_PRELAUNCH, RUN_STATE_PAUSED },

=C2=A0 =C2=A0 =C2=A0{ RUN_STATE_FINISH_MIGRATE, RUN_STATE_RUNNING },
=C2=A0 =C2=A0 =C2=A0{ RUN_STATE_FINISH_MIGRATE, RUN_STATE_PAUSED },
@@ -335,6 +337,7 @@ void vm_state_notify(bool running, RunState state)
=C2=A0 =C2=A0 =C2=A0}
=C2=A0}

+static bool start_on_wake_requested;
=C2=A0static ShutdownCause reset_requested;
=C2=A0static ShutdownCause shutdown_requested;
=C2=A0static int shutdown_signal;
@@ -562,6 +565,11 @@ void qemu_register_suspend_notifier(Notifier *notifier= )
=C2=A0 =C2=A0 =C2=A0notifier_list_add(&suspend_notifiers, notifier); =C2=A0}

+void qemu_system_start_on_wake_request(void)
+{
+=C2=A0 =C2=A0 start_on_wake_requested =3D true;
+}
+
=C2=A0void qemu_system_wakeup_request(WakeupReason reason, Error **errp) =C2=A0{
=C2=A0 =C2=A0 =C2=A0trace_system_wakeup_request(reason);
@@ -574,7 +582,18 @@ void qemu_system_wakeup_request(WakeupReason reason, E= rror **errp)
=C2=A0 =C2=A0 =C2=A0if (!(wakeup_reason_mask & (1 << reason))) {<= br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0return;
=C2=A0 =C2=A0 =C2=A0}
-=C2=A0 =C2=A0 runstate_set(RUN_STATE_RUNNING);
+
+=C2=A0 =C2=A0 /*
+=C2=A0 =C2=A0 =C2=A0* Must call vm_start if it has never been called, to i= nvoke the state
+=C2=A0 =C2=A0 =C2=A0* change callbacks for the first time.
+=C2=A0 =C2=A0 =C2=A0*/
+=C2=A0 =C2=A0 if (start_on_wake_requested) {
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 start_on_wake_requested =3D false;
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 vm_start();
+=C2=A0 =C2=A0 } else {
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 runstate_set(RUN_STATE_RUNNING);
+=C2=A0 =C2=A0 }
+
=C2=A0 =C2=A0 =C2=A0wakeup_reason =3D reason;
=C2=A0 =C2=A0 =C2=A0qemu_notify_event();
=C2=A0}
--
1.8.3.1




--
Marc-Andr=C3=A9 Lureau
--00000000000068e1b605c69bc29d--