From: Steve Sistare <steven.sistare@oracle.com> To: qemu-devel@nongnu.org Cc: "Daniel P. Berrange" <berrange@redhat.com>, "Michael S. Tsirkin" <mst@redhat.com>, "Alex Bennée" <alex.bennee@linaro.org>, "Juan Quintela" <quintela@redhat.com>, "Dr. David Alan Gilbert" <dgilbert@redhat.com>, "Markus Armbruster" <armbru@redhat.com>, "Alex Williamson" <alex.williamson@redhat.com>, "Steve Sistare" <steven.sistare@oracle.com>, "Stefan Hajnoczi" <stefanha@redhat.com>, "Marc-André Lureau" <marcandre.lureau@redhat.com>, "Paolo Bonzini" <pbonzini@redhat.com>, "Philippe Mathieu-Daudé" <philmd@redhat.com> Subject: [PATCH V1 00/32] Live Update Date: Thu, 30 Jul 2020 08:14:04 -0700 [thread overview] Message-ID: <1596122076-341293-1-git-send-email-steven.sistare@oracle.com> (raw) Improve and extend the qemu functions that save and restore VM state so a guest may be suspended and resumed with minimal pause time. qemu may be updated to a new version in between. The first set of patches adds the cprsave and cprload commands to save and restore VM state, and allow the host kernel to be updated and rebooted in between. The VM must create guest RAM in a persistent shared memory file, such as /dev/dax0.0 or persistant /dev/shm PKRAM as proposed in https://lore.kernel.org/lkml/1588812129-8596-1-git-send-email-anthony.yznaga@oracle.com/ cprsave stops the VCPUs and saves VM device state in a simple file, and thus supports any type of guest image and block device. The caller must not modify the VM's block devices between cprsave and cprload. cprsave and cprload support guests with vfio devices if the caller first suspends the guest by issuing guest-suspend-ram to the qemu guest agent. The guest drivers suspend methods flush outstanding requests and re- initialize the devices, and thus there is no device state to save and restore. 1 savevm: add vmstate handler iterators 2 savevm: VM handlers mode mask 3 savevm: QMP command for cprsave 4 savevm: HMP Command for cprsave 5 savevm: QMP command for cprload 6 savevm: HMP Command for cprload 7 savevm: QMP command for cprinfo 8 savevm: HMP command for cprinfo 9 savevm: prevent cprsave if memory is volatile 10 kvmclock: restore paused KVM clock 11 cpu: disable ticks when suspended 12 vl: pause option 13 gdbstub: gdb support for suspended state The next patches add a restart method that eliminates the persistent memory constraint, and allows qemu to be updated across the restart, but does not allow host reboot. Anonymous memory segments used by the guest are preserved across a re-exec of qemu, mapped at the same VA, via a proposed madvise(MADV_DOEXEC) option in the Linux kernel. See https://lore.kernel.org/lkml/1595869887-23307-1-git-send-email-anthony.yznaga@oracle.com/ 14 savevm: VMS_RESTART and cprsave restart 15 vl: QEMU_START_FREEZE env var 16 oslib: add qemu_clr_cloexec 17 util: env var helpers 18 osdep: import MADV_DOEXEC 19 memory: ram_block_add cosmetic changes 20 vl: add helper to request re-exec 21 exec, memory: exec(3) to restart 22 char: qio_channel_socket_accept reuse fd 23 char: save/restore chardev socket fds 24 ui: save/restore vnc socket fds 25 char: save/restore chardev pty fds 26 monitor: save/restore QMP negotiation status 27 vhost: reset vhost devices upon cprsave 28 char: restore terminal on restart The next patches extend the restart method to save and restore vfio-pci state, eliminating the requirement for a guest agent. The vfio container, group, and device descriptors are preserved across the qemu re-exec. 29 pci: export pci_update_mappings 30 vfio-pci: save and restore 31 vfio-pci: trace pci config 32 vfio-pci: improved tracing Here is an example of updating qemu from v4.2.0 to v4.2.1 using "cprload restart". The software update is performed while the guest is running to minimize downtime. window 1 | window 2 | # qemu-system-x86_64 ... | QEMU 4.2.0 monitor - type 'help' ... | (qemu) info status | VM status: running | | # yum update qemu (qemu) cprsave /tmp/qemu.sav restart | QEMU 4.2.1 monitor - type 'help' ... | (qemu) info status | VM status: paused (prelaunch) | (qemu) cprload /tmp/qemu.sav | (qemu) info status | VM status: running | Here is an example of updating the host kernel using "cprload reboot" window 1 | window 2 | # qemu-system-x86_64 ...mem-path=/dev/dax0.0 ...| QEMU 4.2.1 monitor - type 'help' ... | (qemu) info status | VM status: running | | # yum update kernel-uek (qemu) cprsave /tmp/qemu.sav restart | | # systemctl kexec | kexec_core: Starting new kernel | ... | | # qemu-system-x86_64 ...mem-path=/dev/dax0.0 ...| QEMU 4.2.1 monitor - type 'help' ... | (qemu) info status | VM status: paused (prelaunch) | (qemu) cprload /tmp/qemu.sav | (qemu) info status | VM status: running | Mark Kanda (5): char: qio_channel_socket_accept reuse fd char: save/restore chardev socket fds ui: save/restore vnc socket fds monitor: save/restore QMP negotiation status vhost: reset vhost devices upon cprsave Steve Sistare (27): savevm: add vmstate handler iterators savevm: VM handlers mode mask savevm: QMP command for cprsave savevm: HMP Command for cprsave savevm: QMP command for cprload savevm: HMP Command for cprload savevm: QMP command for cprinfo savevm: HMP command for cprinfo savevm: prevent cprsave if memory is volatile kvmclock: restore paused KVM clock cpu: disable ticks when suspended vl: pause option gdbstub: gdb support for suspended state savevm: VMS_RESTART and cprsave restart vl: QEMU_START_FREEZE env var oslib: add qemu_clr_cloexec util: env var helpers osdep: import MADV_DOEXEC memory: ram_block_add cosmetic changes vl: add helper to request re-exec exec, memory: exec(3) to restart char: save/restore chardev pty fds char: restore terminal on restart pci: export pci_update_mappings vfio-pci: save and restore vfio-pci: trace pci config vfio-pci: improved tracing MAINTAINERS | 7 ++ accel/kvm/kvm-all.c | 8 +- accel/kvm/trace-events | 3 +- chardev/char-pty.c | 38 +++++-- chardev/char-socket.c | 35 ++++++ chardev/char-stdio.c | 7 ++ chardev/char.c | 16 +++ exec.c | 88 +++++++++++++-- gdbstub.c | 11 +- hmp-commands.hx | 46 ++++++++ hw/i386/kvm/clock.c | 6 +- hw/pci/msix.c | 1 + hw/pci/pci.c | 17 +-- hw/pci/trace-events | 5 +- hw/vfio/common.c | 115 ++++++++++++++++---- hw/vfio/pci.c | 179 ++++++++++++++++++++++++++++++- hw/vfio/platform.c | 2 +- hw/vfio/trace-events | 11 +- hw/virtio/vhost.c | 12 +++ include/chardev/char.h | 8 ++ include/exec/memory.h | 4 + include/hw/pci/pci.h | 2 + include/hw/vfio/vfio-common.h | 4 +- include/io/channel-socket.h | 3 +- include/migration/register.h | 3 + include/migration/vmstate.h | 11 ++ include/monitor/hmp.h | 3 + include/qemu/cutils.h | 1 + include/qemu/env.h | 31 ++++++ include/qemu/osdep.h | 8 ++ include/sysemu/sysemu.h | 10 ++ io/channel-socket.c | 12 ++- io/net-listener.c | 4 +- migration/block.c | 1 + migration/migration.c | 4 +- migration/ram.c | 1 + migration/savevm.c | 237 ++++++++++++++++++++++++++++++++++++----- migration/savevm.h | 4 +- monitor/hmp-cmds.c | 28 +++++ monitor/qmp-cmds.c | 16 +++ monitor/qmp.c | 42 ++++++++ qapi/migration.json | 35 ++++++ qapi/pragma.json | 1 + qemu-options.hx | 9 ++ scsi/qemu-pr-helper.c | 2 +- softmmu/vl.c | 65 ++++++++++- tests/qtest/tpm-emu.c | 2 +- tests/test-char.c | 2 +- tests/test-io-channel-socket.c | 4 +- trace-events | 2 + ui/vnc.c | 153 +++++++++++++++++++++----- util/Makefile.objs | 2 +- util/env.c | 132 +++++++++++++++++++++++ util/oslib-posix.c | 9 ++ util/oslib-win32.c | 4 + 55 files changed, 1331 insertions(+), 135 deletions(-) create mode 100644 include/qemu/env.h create mode 100644 util/env.c -- 1.8.3.1
next reply other threads:[~2020-07-30 15:42 UTC|newest] Thread overview: 118+ messages / expand[flat|nested] mbox.gz Atom feed top 2020-07-30 15:14 Steve Sistare [this message] 2020-07-30 15:14 ` [PATCH V1 01/32] savevm: add vmstate handler iterators Steve Sistare 2020-09-11 16:24 ` Dr. David Alan Gilbert 2020-09-24 21:43 ` Steven Sistare 2020-09-25 9:07 ` Dr. David Alan Gilbert 2020-07-30 15:14 ` [PATCH V1 02/32] savevm: VM handlers mode mask Steve Sistare 2020-07-30 15:14 ` [PATCH V1 03/32] savevm: QMP command for cprsave Steve Sistare 2020-07-30 16:12 ` Eric Blake 2020-07-30 17:52 ` Steven Sistare 2020-09-11 16:43 ` Dr. David Alan Gilbert 2020-09-25 18:43 ` Steven Sistare 2020-09-25 22:22 ` Steven Sistare 2020-07-30 15:14 ` [PATCH V1 04/32] savevm: HMP Command " Steve Sistare 2020-09-11 16:57 ` Dr. David Alan Gilbert 2020-09-24 21:44 ` Steven Sistare 2020-09-25 9:26 ` Dr. David Alan Gilbert 2020-07-30 15:14 ` [PATCH V1 05/32] savevm: QMP command for cprload Steve Sistare 2020-07-30 16:14 ` Eric Blake 2020-07-30 18:00 ` Steven Sistare 2020-09-11 17:18 ` Dr. David Alan Gilbert 2020-09-24 21:49 ` Steven Sistare 2020-07-30 15:14 ` [PATCH V1 06/32] savevm: HMP Command " Steve Sistare 2020-07-30 15:14 ` [PATCH V1 07/32] savevm: QMP command for cprinfo Steve Sistare 2020-07-30 16:17 ` Eric Blake 2020-07-30 18:02 ` Steven Sistare 2020-07-30 15:14 ` [PATCH V1 08/32] savevm: HMP " Steve Sistare 2020-09-11 17:27 ` Dr. David Alan Gilbert 2020-09-24 21:50 ` Steven Sistare 2020-07-30 15:14 ` [PATCH V1 09/32] savevm: prevent cprsave if memory is volatile Steve Sistare 2020-09-11 17:35 ` Dr. David Alan Gilbert 2020-09-24 21:51 ` Steven Sistare 2020-07-30 15:14 ` [PATCH V1 10/32] kvmclock: restore paused KVM clock Steve Sistare 2020-09-11 17:50 ` Dr. David Alan Gilbert 2020-09-25 18:07 ` Steven Sistare 2020-07-30 15:14 ` [PATCH V1 11/32] cpu: disable ticks when suspended Steve Sistare 2020-09-11 17:53 ` Dr. David Alan Gilbert 2020-09-24 20:42 ` Steven Sistare 2020-09-25 9:03 ` Dr. David Alan Gilbert 2020-07-30 15:14 ` [PATCH V1 12/32] vl: pause option Steve Sistare 2020-07-30 16:20 ` Eric Blake 2020-07-30 18:11 ` Steven Sistare 2020-07-31 10:07 ` Daniel P. Berrangé 2020-07-31 15:18 ` Steven Sistare 2020-07-30 17:03 ` Alex Bennée 2020-07-30 18:14 ` Steven Sistare 2020-07-31 9:44 ` Alex Bennée 2020-09-11 17:59 ` Dr. David Alan Gilbert 2020-09-24 21:51 ` Steven Sistare 2020-07-30 15:14 ` [PATCH V1 13/32] gdbstub: gdb support for suspended state Steve Sistare 2020-09-11 18:41 ` Dr. David Alan Gilbert 2020-09-24 21:51 ` Steven Sistare 2020-07-30 15:14 ` [PATCH V1 14/32] savevm: VMS_RESTART and cprsave restart Steve Sistare 2020-07-30 16:22 ` Eric Blake 2020-07-30 18:14 ` Steven Sistare 2020-09-11 18:44 ` Dr. David Alan Gilbert 2020-09-24 21:44 ` Steven Sistare 2020-07-30 15:14 ` [PATCH V1 15/32] vl: QEMU_START_FREEZE env var Steve Sistare 2020-09-11 18:49 ` Dr. David Alan Gilbert 2020-09-24 21:47 ` Steven Sistare 2020-09-25 15:52 ` Dr. David Alan Gilbert 2020-07-30 15:14 ` [PATCH V1 16/32] oslib: add qemu_clr_cloexec Steve Sistare 2020-09-11 18:52 ` Dr. David Alan Gilbert 2020-07-30 15:14 ` [PATCH V1 17/32] util: env var helpers Steve Sistare 2020-09-11 19:00 ` Dr. David Alan Gilbert 2020-09-24 21:52 ` Steven Sistare 2020-07-30 15:14 ` [PATCH V1 18/32] osdep: import MADV_DOEXEC Steve Sistare 2020-08-17 18:30 ` Steven Sistare 2020-08-17 20:48 ` Alex Williamson 2020-08-17 21:20 ` Steven Sistare 2020-08-17 21:44 ` Alex Williamson 2020-08-18 2:42 ` Alex Williamson 2020-08-19 21:52 ` Steven Sistare 2020-08-24 22:30 ` Alex Williamson 2020-10-08 16:32 ` Steven Sistare 2020-10-15 20:36 ` Alex Williamson 2020-10-19 16:33 ` Steven Sistare 2020-10-26 18:28 ` Steven Sistare 2020-07-30 15:14 ` [PATCH V1 19/32] memory: ram_block_add cosmetic changes Steve Sistare 2020-07-30 15:14 ` [PATCH V1 20/32] vl: add helper to request re-exec Steve Sistare 2020-07-30 15:14 ` [PATCH V1 21/32] exec, memory: exec(3) to restart Steve Sistare 2020-07-30 15:14 ` [PATCH V1 22/32] char: qio_channel_socket_accept reuse fd Steve Sistare 2020-09-15 17:33 ` Dr. David Alan Gilbert 2020-09-15 17:53 ` Daniel P. Berrangé 2020-09-24 21:54 ` Steven Sistare 2020-07-30 15:14 ` [PATCH V1 23/32] char: save/restore chardev socket fds Steve Sistare 2020-07-30 15:14 ` [PATCH V1 24/32] ui: save/restore vnc " Steve Sistare 2020-07-31 9:06 ` Daniel P. Berrangé 2020-07-31 16:51 ` Steven Sistare 2020-07-30 15:14 ` [PATCH V1 25/32] char: save/restore chardev pty fds Steve Sistare 2020-07-30 15:14 ` [PATCH V1 26/32] monitor: save/restore QMP negotiation status Steve Sistare 2020-07-30 15:14 ` [PATCH V1 27/32] vhost: reset vhost devices upon cprsave Steve Sistare 2020-07-30 15:14 ` [PATCH V1 28/32] char: restore terminal on restart Steve Sistare 2020-07-30 15:14 ` [PATCH V1 29/32] pci: export pci_update_mappings Steve Sistare 2020-07-30 15:14 ` [PATCH V1 30/32] vfio-pci: save and restore Steve Sistare 2020-08-06 10:22 ` Jason Zeng 2020-08-07 20:38 ` Steven Sistare 2020-08-10 3:50 ` Jason Zeng 2020-08-19 21:15 ` Steven Sistare 2020-08-20 10:33 ` Jason Zeng 2020-10-07 21:25 ` Steven Sistare 2020-07-30 15:14 ` [PATCH V1 31/32] vfio-pci: trace pci config Steve Sistare 2020-07-30 15:14 ` [PATCH V1 32/32] vfio-pci: improved tracing Steve Sistare 2020-09-15 18:49 ` Dr. David Alan Gilbert 2020-09-24 21:52 ` Steven Sistare 2020-07-30 16:52 ` [PATCH V1 00/32] Live Update Daniel P. Berrangé 2020-07-30 18:48 ` Steven Sistare 2020-07-31 8:53 ` Daniel P. Berrangé 2020-07-31 15:27 ` Steven Sistare 2020-07-31 15:52 ` Daniel P. Berrangé 2020-07-31 17:20 ` Steven Sistare 2020-08-11 19:08 ` Dr. David Alan Gilbert 2020-07-30 17:15 ` Paolo Bonzini 2020-07-30 19:09 ` Steven Sistare 2020-07-30 21:39 ` Paolo Bonzini 2020-07-31 19:22 ` Steven Sistare 2020-07-30 17:49 ` Dr. David Alan Gilbert 2020-07-30 19:31 ` Steven Sistare 2020-08-04 18:18 ` Steven Sistare
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=1596122076-341293-1-git-send-email-steven.sistare@oracle.com \ --to=steven.sistare@oracle.com \ --cc=alex.bennee@linaro.org \ --cc=alex.williamson@redhat.com \ --cc=armbru@redhat.com \ --cc=berrange@redhat.com \ --cc=dgilbert@redhat.com \ --cc=marcandre.lureau@redhat.com \ --cc=mst@redhat.com \ --cc=pbonzini@redhat.com \ --cc=philmd@redhat.com \ --cc=qemu-devel@nongnu.org \ --cc=quintela@redhat.com \ --cc=stefanha@redhat.com \ --subject='Re: [PATCH V1 00/32] Live Update' \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: link
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).