From: Steve Sistare <steven.sistare@oracle.com>
To: qemu-devel@nongnu.org
Cc: "Daniel P. Berrange" <berrange@redhat.com>,
"Michael S. Tsirkin" <mst@redhat.com>,
"Alex Bennée" <alex.bennee@linaro.org>,
"Juan Quintela" <quintela@redhat.com>,
"Dr. David Alan Gilbert" <dgilbert@redhat.com>,
"Markus Armbruster" <armbru@redhat.com>,
"Alex Williamson" <alex.williamson@redhat.com>,
"Steve Sistare" <steven.sistare@oracle.com>,
"Stefan Hajnoczi" <stefanha@redhat.com>,
"Marc-André Lureau" <marcandre.lureau@redhat.com>,
"Paolo Bonzini" <pbonzini@redhat.com>,
"Philippe Mathieu-Daudé" <philmd@redhat.com>
Subject: [PATCH V1 00/32] Live Update
Date: Thu, 30 Jul 2020 08:14:04 -0700 [thread overview]
Message-ID: <1596122076-341293-1-git-send-email-steven.sistare@oracle.com> (raw)
Improve and extend the qemu functions that save and restore VM state so a
guest may be suspended and resumed with minimal pause time. qemu may be
updated to a new version in between.
The first set of patches adds the cprsave and cprload commands to save and
restore VM state, and allow the host kernel to be updated and rebooted in
between. The VM must create guest RAM in a persistent shared memory file,
such as /dev/dax0.0 or persistant /dev/shm PKRAM as proposed in
https://lore.kernel.org/lkml/1588812129-8596-1-git-send-email-anthony.yznaga@oracle.com/
cprsave stops the VCPUs and saves VM device state in a simple file, and
thus supports any type of guest image and block device. The caller must
not modify the VM's block devices between cprsave and cprload.
cprsave and cprload support guests with vfio devices if the caller first
suspends the guest by issuing guest-suspend-ram to the qemu guest agent.
The guest drivers suspend methods flush outstanding requests and re-
initialize the devices, and thus there is no device state to save and
restore.
1 savevm: add vmstate handler iterators
2 savevm: VM handlers mode mask
3 savevm: QMP command for cprsave
4 savevm: HMP Command for cprsave
5 savevm: QMP command for cprload
6 savevm: HMP Command for cprload
7 savevm: QMP command for cprinfo
8 savevm: HMP command for cprinfo
9 savevm: prevent cprsave if memory is volatile
10 kvmclock: restore paused KVM clock
11 cpu: disable ticks when suspended
12 vl: pause option
13 gdbstub: gdb support for suspended state
The next patches add a restart method that eliminates the persistent memory
constraint, and allows qemu to be updated across the restart, but does not
allow host reboot. Anonymous memory segments used by the guest are
preserved across a re-exec of qemu, mapped at the same VA, via a proposed
madvise(MADV_DOEXEC) option in the Linux kernel. See
https://lore.kernel.org/lkml/1595869887-23307-1-git-send-email-anthony.yznaga@oracle.com/
14 savevm: VMS_RESTART and cprsave restart
15 vl: QEMU_START_FREEZE env var
16 oslib: add qemu_clr_cloexec
17 util: env var helpers
18 osdep: import MADV_DOEXEC
19 memory: ram_block_add cosmetic changes
20 vl: add helper to request re-exec
21 exec, memory: exec(3) to restart
22 char: qio_channel_socket_accept reuse fd
23 char: save/restore chardev socket fds
24 ui: save/restore vnc socket fds
25 char: save/restore chardev pty fds
26 monitor: save/restore QMP negotiation status
27 vhost: reset vhost devices upon cprsave
28 char: restore terminal on restart
The next patches extend the restart method to save and restore vfio-pci
state, eliminating the requirement for a guest agent. The vfio container,
group, and device descriptors are preserved across the qemu re-exec.
29 pci: export pci_update_mappings
30 vfio-pci: save and restore
31 vfio-pci: trace pci config
32 vfio-pci: improved tracing
Here is an example of updating qemu from v4.2.0 to v4.2.1 using
"cprload restart". The software update is performed while the guest is
running to minimize downtime.
window 1 | window 2
|
# qemu-system-x86_64 ... |
QEMU 4.2.0 monitor - type 'help' ... |
(qemu) info status |
VM status: running |
| # yum update qemu
(qemu) cprsave /tmp/qemu.sav restart |
QEMU 4.2.1 monitor - type 'help' ... |
(qemu) info status |
VM status: paused (prelaunch) |
(qemu) cprload /tmp/qemu.sav |
(qemu) info status |
VM status: running |
Here is an example of updating the host kernel using "cprload reboot"
window 1 | window 2
|
# qemu-system-x86_64 ...mem-path=/dev/dax0.0 ...|
QEMU 4.2.1 monitor - type 'help' ... |
(qemu) info status |
VM status: running |
| # yum update kernel-uek
(qemu) cprsave /tmp/qemu.sav restart |
|
# systemctl kexec |
kexec_core: Starting new kernel |
... |
|
# qemu-system-x86_64 ...mem-path=/dev/dax0.0 ...|
QEMU 4.2.1 monitor - type 'help' ... |
(qemu) info status |
VM status: paused (prelaunch) |
(qemu) cprload /tmp/qemu.sav |
(qemu) info status |
VM status: running |
Mark Kanda (5):
char: qio_channel_socket_accept reuse fd
char: save/restore chardev socket fds
ui: save/restore vnc socket fds
monitor: save/restore QMP negotiation status
vhost: reset vhost devices upon cprsave
Steve Sistare (27):
savevm: add vmstate handler iterators
savevm: VM handlers mode mask
savevm: QMP command for cprsave
savevm: HMP Command for cprsave
savevm: QMP command for cprload
savevm: HMP Command for cprload
savevm: QMP command for cprinfo
savevm: HMP command for cprinfo
savevm: prevent cprsave if memory is volatile
kvmclock: restore paused KVM clock
cpu: disable ticks when suspended
vl: pause option
gdbstub: gdb support for suspended state
savevm: VMS_RESTART and cprsave restart
vl: QEMU_START_FREEZE env var
oslib: add qemu_clr_cloexec
util: env var helpers
osdep: import MADV_DOEXEC
memory: ram_block_add cosmetic changes
vl: add helper to request re-exec
exec, memory: exec(3) to restart
char: save/restore chardev pty fds
char: restore terminal on restart
pci: export pci_update_mappings
vfio-pci: save and restore
vfio-pci: trace pci config
vfio-pci: improved tracing
MAINTAINERS | 7 ++
accel/kvm/kvm-all.c | 8 +-
accel/kvm/trace-events | 3 +-
chardev/char-pty.c | 38 +++++--
chardev/char-socket.c | 35 ++++++
chardev/char-stdio.c | 7 ++
chardev/char.c | 16 +++
exec.c | 88 +++++++++++++--
gdbstub.c | 11 +-
hmp-commands.hx | 46 ++++++++
hw/i386/kvm/clock.c | 6 +-
hw/pci/msix.c | 1 +
hw/pci/pci.c | 17 +--
hw/pci/trace-events | 5 +-
hw/vfio/common.c | 115 ++++++++++++++++----
hw/vfio/pci.c | 179 ++++++++++++++++++++++++++++++-
hw/vfio/platform.c | 2 +-
hw/vfio/trace-events | 11 +-
hw/virtio/vhost.c | 12 +++
include/chardev/char.h | 8 ++
include/exec/memory.h | 4 +
include/hw/pci/pci.h | 2 +
include/hw/vfio/vfio-common.h | 4 +-
include/io/channel-socket.h | 3 +-
include/migration/register.h | 3 +
include/migration/vmstate.h | 11 ++
include/monitor/hmp.h | 3 +
include/qemu/cutils.h | 1 +
include/qemu/env.h | 31 ++++++
include/qemu/osdep.h | 8 ++
include/sysemu/sysemu.h | 10 ++
io/channel-socket.c | 12 ++-
io/net-listener.c | 4 +-
migration/block.c | 1 +
migration/migration.c | 4 +-
migration/ram.c | 1 +
migration/savevm.c | 237 ++++++++++++++++++++++++++++++++++++-----
migration/savevm.h | 4 +-
monitor/hmp-cmds.c | 28 +++++
monitor/qmp-cmds.c | 16 +++
monitor/qmp.c | 42 ++++++++
qapi/migration.json | 35 ++++++
qapi/pragma.json | 1 +
qemu-options.hx | 9 ++
scsi/qemu-pr-helper.c | 2 +-
softmmu/vl.c | 65 ++++++++++-
tests/qtest/tpm-emu.c | 2 +-
tests/test-char.c | 2 +-
tests/test-io-channel-socket.c | 4 +-
trace-events | 2 +
ui/vnc.c | 153 +++++++++++++++++++++-----
util/Makefile.objs | 2 +-
util/env.c | 132 +++++++++++++++++++++++
util/oslib-posix.c | 9 ++
util/oslib-win32.c | 4 +
55 files changed, 1331 insertions(+), 135 deletions(-)
create mode 100644 include/qemu/env.h
create mode 100644 util/env.c
--
1.8.3.1
next reply other threads:[~2020-07-30 15:42 UTC|newest]
Thread overview: 118+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-07-30 15:14 Steve Sistare [this message]
2020-07-30 15:14 ` [PATCH V1 01/32] savevm: add vmstate handler iterators Steve Sistare
2020-09-11 16:24 ` Dr. David Alan Gilbert
2020-09-24 21:43 ` Steven Sistare
2020-09-25 9:07 ` Dr. David Alan Gilbert
2020-07-30 15:14 ` [PATCH V1 02/32] savevm: VM handlers mode mask Steve Sistare
2020-07-30 15:14 ` [PATCH V1 03/32] savevm: QMP command for cprsave Steve Sistare
2020-07-30 16:12 ` Eric Blake
2020-07-30 17:52 ` Steven Sistare
2020-09-11 16:43 ` Dr. David Alan Gilbert
2020-09-25 18:43 ` Steven Sistare
2020-09-25 22:22 ` Steven Sistare
2020-07-30 15:14 ` [PATCH V1 04/32] savevm: HMP Command " Steve Sistare
2020-09-11 16:57 ` Dr. David Alan Gilbert
2020-09-24 21:44 ` Steven Sistare
2020-09-25 9:26 ` Dr. David Alan Gilbert
2020-07-30 15:14 ` [PATCH V1 05/32] savevm: QMP command for cprload Steve Sistare
2020-07-30 16:14 ` Eric Blake
2020-07-30 18:00 ` Steven Sistare
2020-09-11 17:18 ` Dr. David Alan Gilbert
2020-09-24 21:49 ` Steven Sistare
2020-07-30 15:14 ` [PATCH V1 06/32] savevm: HMP Command " Steve Sistare
2020-07-30 15:14 ` [PATCH V1 07/32] savevm: QMP command for cprinfo Steve Sistare
2020-07-30 16:17 ` Eric Blake
2020-07-30 18:02 ` Steven Sistare
2020-07-30 15:14 ` [PATCH V1 08/32] savevm: HMP " Steve Sistare
2020-09-11 17:27 ` Dr. David Alan Gilbert
2020-09-24 21:50 ` Steven Sistare
2020-07-30 15:14 ` [PATCH V1 09/32] savevm: prevent cprsave if memory is volatile Steve Sistare
2020-09-11 17:35 ` Dr. David Alan Gilbert
2020-09-24 21:51 ` Steven Sistare
2020-07-30 15:14 ` [PATCH V1 10/32] kvmclock: restore paused KVM clock Steve Sistare
2020-09-11 17:50 ` Dr. David Alan Gilbert
2020-09-25 18:07 ` Steven Sistare
2020-07-30 15:14 ` [PATCH V1 11/32] cpu: disable ticks when suspended Steve Sistare
2020-09-11 17:53 ` Dr. David Alan Gilbert
2020-09-24 20:42 ` Steven Sistare
2020-09-25 9:03 ` Dr. David Alan Gilbert
2020-07-30 15:14 ` [PATCH V1 12/32] vl: pause option Steve Sistare
2020-07-30 16:20 ` Eric Blake
2020-07-30 18:11 ` Steven Sistare
2020-07-31 10:07 ` Daniel P. Berrangé
2020-07-31 15:18 ` Steven Sistare
2020-07-30 17:03 ` Alex Bennée
2020-07-30 18:14 ` Steven Sistare
2020-07-31 9:44 ` Alex Bennée
2020-09-11 17:59 ` Dr. David Alan Gilbert
2020-09-24 21:51 ` Steven Sistare
2020-07-30 15:14 ` [PATCH V1 13/32] gdbstub: gdb support for suspended state Steve Sistare
2020-09-11 18:41 ` Dr. David Alan Gilbert
2020-09-24 21:51 ` Steven Sistare
2020-07-30 15:14 ` [PATCH V1 14/32] savevm: VMS_RESTART and cprsave restart Steve Sistare
2020-07-30 16:22 ` Eric Blake
2020-07-30 18:14 ` Steven Sistare
2020-09-11 18:44 ` Dr. David Alan Gilbert
2020-09-24 21:44 ` Steven Sistare
2020-07-30 15:14 ` [PATCH V1 15/32] vl: QEMU_START_FREEZE env var Steve Sistare
2020-09-11 18:49 ` Dr. David Alan Gilbert
2020-09-24 21:47 ` Steven Sistare
2020-09-25 15:52 ` Dr. David Alan Gilbert
2020-07-30 15:14 ` [PATCH V1 16/32] oslib: add qemu_clr_cloexec Steve Sistare
2020-09-11 18:52 ` Dr. David Alan Gilbert
2020-07-30 15:14 ` [PATCH V1 17/32] util: env var helpers Steve Sistare
2020-09-11 19:00 ` Dr. David Alan Gilbert
2020-09-24 21:52 ` Steven Sistare
2020-07-30 15:14 ` [PATCH V1 18/32] osdep: import MADV_DOEXEC Steve Sistare
2020-08-17 18:30 ` Steven Sistare
2020-08-17 20:48 ` Alex Williamson
2020-08-17 21:20 ` Steven Sistare
2020-08-17 21:44 ` Alex Williamson
2020-08-18 2:42 ` Alex Williamson
2020-08-19 21:52 ` Steven Sistare
2020-08-24 22:30 ` Alex Williamson
2020-10-08 16:32 ` Steven Sistare
2020-10-15 20:36 ` Alex Williamson
2020-10-19 16:33 ` Steven Sistare
2020-10-26 18:28 ` Steven Sistare
2020-07-30 15:14 ` [PATCH V1 19/32] memory: ram_block_add cosmetic changes Steve Sistare
2020-07-30 15:14 ` [PATCH V1 20/32] vl: add helper to request re-exec Steve Sistare
2020-07-30 15:14 ` [PATCH V1 21/32] exec, memory: exec(3) to restart Steve Sistare
2020-07-30 15:14 ` [PATCH V1 22/32] char: qio_channel_socket_accept reuse fd Steve Sistare
2020-09-15 17:33 ` Dr. David Alan Gilbert
2020-09-15 17:53 ` Daniel P. Berrangé
2020-09-24 21:54 ` Steven Sistare
2020-07-30 15:14 ` [PATCH V1 23/32] char: save/restore chardev socket fds Steve Sistare
2020-07-30 15:14 ` [PATCH V1 24/32] ui: save/restore vnc " Steve Sistare
2020-07-31 9:06 ` Daniel P. Berrangé
2020-07-31 16:51 ` Steven Sistare
2020-07-30 15:14 ` [PATCH V1 25/32] char: save/restore chardev pty fds Steve Sistare
2020-07-30 15:14 ` [PATCH V1 26/32] monitor: save/restore QMP negotiation status Steve Sistare
2020-07-30 15:14 ` [PATCH V1 27/32] vhost: reset vhost devices upon cprsave Steve Sistare
2020-07-30 15:14 ` [PATCH V1 28/32] char: restore terminal on restart Steve Sistare
2020-07-30 15:14 ` [PATCH V1 29/32] pci: export pci_update_mappings Steve Sistare
2020-07-30 15:14 ` [PATCH V1 30/32] vfio-pci: save and restore Steve Sistare
2020-08-06 10:22 ` Jason Zeng
2020-08-07 20:38 ` Steven Sistare
2020-08-10 3:50 ` Jason Zeng
2020-08-19 21:15 ` Steven Sistare
2020-08-20 10:33 ` Jason Zeng
2020-10-07 21:25 ` Steven Sistare
2020-07-30 15:14 ` [PATCH V1 31/32] vfio-pci: trace pci config Steve Sistare
2020-07-30 15:14 ` [PATCH V1 32/32] vfio-pci: improved tracing Steve Sistare
2020-09-15 18:49 ` Dr. David Alan Gilbert
2020-09-24 21:52 ` Steven Sistare
2020-07-30 16:52 ` [PATCH V1 00/32] Live Update Daniel P. Berrangé
2020-07-30 18:48 ` Steven Sistare
2020-07-31 8:53 ` Daniel P. Berrangé
2020-07-31 15:27 ` Steven Sistare
2020-07-31 15:52 ` Daniel P. Berrangé
2020-07-31 17:20 ` Steven Sistare
2020-08-11 19:08 ` Dr. David Alan Gilbert
2020-07-30 17:15 ` Paolo Bonzini
2020-07-30 19:09 ` Steven Sistare
2020-07-30 21:39 ` Paolo Bonzini
2020-07-31 19:22 ` Steven Sistare
2020-07-30 17:49 ` Dr. David Alan Gilbert
2020-07-30 19:31 ` Steven Sistare
2020-08-04 18:18 ` Steven Sistare
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1596122076-341293-1-git-send-email-steven.sistare@oracle.com \
--to=steven.sistare@oracle.com \
--cc=alex.bennee@linaro.org \
--cc=alex.williamson@redhat.com \
--cc=armbru@redhat.com \
--cc=berrange@redhat.com \
--cc=dgilbert@redhat.com \
--cc=marcandre.lureau@redhat.com \
--cc=mst@redhat.com \
--cc=pbonzini@redhat.com \
--cc=philmd@redhat.com \
--cc=qemu-devel@nongnu.org \
--cc=quintela@redhat.com \
--cc=stefanha@redhat.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).