QEMU-Devel Archive on lore.kernel.org
 help / color / Atom feed
From: Steve Sistare <steven.sistare@oracle.com>
To: qemu-devel@nongnu.org
Cc: "Daniel P. Berrange" <berrange@redhat.com>,
	"Michael S. Tsirkin" <mst@redhat.com>,
	"Alex Bennée" <alex.bennee@linaro.org>,
	"Juan Quintela" <quintela@redhat.com>,
	"Dr. David Alan Gilbert" <dgilbert@redhat.com>,
	"Markus Armbruster" <armbru@redhat.com>,
	"Alex Williamson" <alex.williamson@redhat.com>,
	"Steve Sistare" <steven.sistare@oracle.com>,
	"Stefan Hajnoczi" <stefanha@redhat.com>,
	"Marc-André Lureau" <marcandre.lureau@redhat.com>,
	"Paolo Bonzini" <pbonzini@redhat.com>,
	"Philippe Mathieu-Daudé" <philmd@redhat.com>
Subject: [PATCH V1 00/32] Live Update
Date: Thu, 30 Jul 2020 08:14:04 -0700
Message-ID: <1596122076-341293-1-git-send-email-steven.sistare@oracle.com> (raw)

Improve and extend the qemu functions that save and restore VM state so a
guest may be suspended and resumed with minimal pause time.  qemu may be
updated to a new version in between.

The first set of patches adds the cprsave and cprload commands to save and
restore VM state, and allow the host kernel to be updated and rebooted in
between.  The VM must create guest RAM in a persistent shared memory file,
such as /dev/dax0.0 or persistant /dev/shm PKRAM as proposed in 
https://lore.kernel.org/lkml/1588812129-8596-1-git-send-email-anthony.yznaga@oracle.com/

cprsave stops the VCPUs and saves VM device state in a simple file, and
thus supports any type of guest image and block device.  The caller must
not modify the VM's block devices between cprsave and cprload.

cprsave and cprload support guests with vfio devices if the caller first
suspends the guest by issuing guest-suspend-ram to the qemu guest agent.
The guest drivers suspend methods flush outstanding requests and re-
initialize the devices, and thus there is no device state to save and
restore.

   1 savevm: add vmstate handler iterators
   2 savevm: VM handlers mode mask
   3 savevm: QMP command for cprsave
   4 savevm: HMP Command for cprsave
   5 savevm: QMP command for cprload
   6 savevm: HMP Command for cprload
   7 savevm: QMP command for cprinfo
   8 savevm: HMP command for cprinfo
   9 savevm: prevent cprsave if memory is volatile
  10 kvmclock: restore paused KVM clock
  11 cpu: disable ticks when suspended
  12 vl: pause option
  13 gdbstub: gdb support for suspended state

The next patches add a restart method that eliminates the persistent memory
constraint, and allows qemu to be updated across the restart, but does not
allow host reboot.  Anonymous memory segments used by the guest are
preserved across a re-exec of qemu, mapped at the same VA, via a proposed
madvise(MADV_DOEXEC) option in the Linux kernel.  See
https://lore.kernel.org/lkml/1595869887-23307-1-git-send-email-anthony.yznaga@oracle.com/

  14 savevm: VMS_RESTART and cprsave restart
  15 vl: QEMU_START_FREEZE env var
  16 oslib: add qemu_clr_cloexec
  17 util: env var helpers
  18 osdep: import MADV_DOEXEC
  19 memory: ram_block_add cosmetic changes
  20 vl: add helper to request re-exec
  21 exec, memory: exec(3) to restart
  22 char: qio_channel_socket_accept reuse fd
  23 char: save/restore chardev socket fds
  24 ui: save/restore vnc socket fds
  25 char: save/restore chardev pty fds
  26 monitor: save/restore QMP negotiation status
  27 vhost: reset vhost devices upon cprsave
  28 char: restore terminal on restart

The next patches extend the restart method to save and restore vfio-pci
state, eliminating the requirement for a guest agent.  The vfio container,
group, and device descriptors are preserved across the qemu re-exec.

  29 pci: export pci_update_mappings
  30 vfio-pci: save and restore
  31 vfio-pci: trace pci config
  32 vfio-pci: improved tracing

Here is an example of updating qemu from v4.2.0 to v4.2.1 using 
"cprload restart".  The software update is performed while the guest is
running to minimize downtime.

window 1				| window 2
					|
# qemu-system-x86_64 ... 		|
QEMU 4.2.0 monitor - type 'help' ...	|
(qemu) info status			|
VM status: running			|
					| # yum update qemu
(qemu) cprsave /tmp/qemu.sav restart	|
QEMU 4.2.1 monitor - type 'help' ...	|
(qemu) info status			|
VM status: paused (prelaunch)		|
(qemu) cprload /tmp/qemu.sav		|
(qemu) info status			|
VM status: running			|


Here is an example of updating the host kernel using "cprload reboot"

window 1					| window 2
						|
# qemu-system-x86_64 ...mem-path=/dev/dax0.0 ...|
QEMU 4.2.1 monitor - type 'help' ...		|
(qemu) info status				|
VM status: running				|
						| # yum update kernel-uek
(qemu) cprsave /tmp/qemu.sav restart		|
						|
# systemctl kexec				|
kexec_core: Starting new kernel			|
...						|
						|
# qemu-system-x86_64 ...mem-path=/dev/dax0.0 ...|
QEMU 4.2.1 monitor - type 'help' ...		|
(qemu) info status				|
VM status: paused (prelaunch)			|
(qemu) cprload /tmp/qemu.sav			|
(qemu) info status				|
VM status: running				|


Mark Kanda (5):
  char: qio_channel_socket_accept reuse fd
  char: save/restore chardev socket fds
  ui: save/restore vnc socket fds
  monitor: save/restore QMP negotiation status
  vhost: reset vhost devices upon cprsave

Steve Sistare (27):
  savevm: add vmstate handler iterators
  savevm: VM handlers mode mask
  savevm: QMP command for cprsave
  savevm: HMP Command for cprsave
  savevm: QMP command for cprload
  savevm: HMP Command for cprload
  savevm: QMP command for cprinfo
  savevm: HMP command for cprinfo
  savevm: prevent cprsave if memory is volatile
  kvmclock: restore paused KVM clock
  cpu: disable ticks when suspended
  vl: pause option
  gdbstub: gdb support for suspended state
  savevm: VMS_RESTART and cprsave restart
  vl: QEMU_START_FREEZE env var
  oslib: add qemu_clr_cloexec
  util: env var helpers
  osdep: import MADV_DOEXEC
  memory: ram_block_add cosmetic changes
  vl: add helper to request re-exec
  exec, memory: exec(3) to restart
  char: save/restore chardev pty fds
  char: restore terminal on restart
  pci: export pci_update_mappings
  vfio-pci: save and restore
  vfio-pci: trace pci config
  vfio-pci: improved tracing

 MAINTAINERS                    |   7 ++
 accel/kvm/kvm-all.c            |   8 +-
 accel/kvm/trace-events         |   3 +-
 chardev/char-pty.c             |  38 +++++--
 chardev/char-socket.c          |  35 ++++++
 chardev/char-stdio.c           |   7 ++
 chardev/char.c                 |  16 +++
 exec.c                         |  88 +++++++++++++--
 gdbstub.c                      |  11 +-
 hmp-commands.hx                |  46 ++++++++
 hw/i386/kvm/clock.c            |   6 +-
 hw/pci/msix.c                  |   1 +
 hw/pci/pci.c                   |  17 +--
 hw/pci/trace-events            |   5 +-
 hw/vfio/common.c               | 115 ++++++++++++++++----
 hw/vfio/pci.c                  | 179 ++++++++++++++++++++++++++++++-
 hw/vfio/platform.c             |   2 +-
 hw/vfio/trace-events           |  11 +-
 hw/virtio/vhost.c              |  12 +++
 include/chardev/char.h         |   8 ++
 include/exec/memory.h          |   4 +
 include/hw/pci/pci.h           |   2 +
 include/hw/vfio/vfio-common.h  |   4 +-
 include/io/channel-socket.h    |   3 +-
 include/migration/register.h   |   3 +
 include/migration/vmstate.h    |  11 ++
 include/monitor/hmp.h          |   3 +
 include/qemu/cutils.h          |   1 +
 include/qemu/env.h             |  31 ++++++
 include/qemu/osdep.h           |   8 ++
 include/sysemu/sysemu.h        |  10 ++
 io/channel-socket.c            |  12 ++-
 io/net-listener.c              |   4 +-
 migration/block.c              |   1 +
 migration/migration.c          |   4 +-
 migration/ram.c                |   1 +
 migration/savevm.c             | 237 ++++++++++++++++++++++++++++++++++++-----
 migration/savevm.h             |   4 +-
 monitor/hmp-cmds.c             |  28 +++++
 monitor/qmp-cmds.c             |  16 +++
 monitor/qmp.c                  |  42 ++++++++
 qapi/migration.json            |  35 ++++++
 qapi/pragma.json               |   1 +
 qemu-options.hx                |   9 ++
 scsi/qemu-pr-helper.c          |   2 +-
 softmmu/vl.c                   |  65 ++++++++++-
 tests/qtest/tpm-emu.c          |   2 +-
 tests/test-char.c              |   2 +-
 tests/test-io-channel-socket.c |   4 +-
 trace-events                   |   2 +
 ui/vnc.c                       | 153 +++++++++++++++++++++-----
 util/Makefile.objs             |   2 +-
 util/env.c                     | 132 +++++++++++++++++++++++
 util/oslib-posix.c             |   9 ++
 util/oslib-win32.c             |   4 +
 55 files changed, 1331 insertions(+), 135 deletions(-)
 create mode 100644 include/qemu/env.h
 create mode 100644 util/env.c

-- 
1.8.3.1



             reply index

Thread overview: 93+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-07-30 15:14 Steve Sistare [this message]
2020-07-30 15:14 ` [PATCH V1 01/32] savevm: add vmstate handler iterators Steve Sistare
2020-09-11 16:24   ` Dr. David Alan Gilbert
2020-07-30 15:14 ` [PATCH V1 02/32] savevm: VM handlers mode mask Steve Sistare
2020-07-30 15:14 ` [PATCH V1 03/32] savevm: QMP command for cprsave Steve Sistare
2020-07-30 16:12   ` Eric Blake
2020-07-30 17:52     ` Steven Sistare
2020-09-11 16:43   ` Dr. David Alan Gilbert
2020-07-30 15:14 ` [PATCH V1 04/32] savevm: HMP Command " Steve Sistare
2020-09-11 16:57   ` Dr. David Alan Gilbert
2020-07-30 15:14 ` [PATCH V1 05/32] savevm: QMP command for cprload Steve Sistare
2020-07-30 16:14   ` Eric Blake
2020-07-30 18:00     ` Steven Sistare
2020-09-11 17:18       ` Dr. David Alan Gilbert
2020-07-30 15:14 ` [PATCH V1 06/32] savevm: HMP Command " Steve Sistare
2020-07-30 15:14 ` [PATCH V1 07/32] savevm: QMP command for cprinfo Steve Sistare
2020-07-30 16:17   ` Eric Blake
2020-07-30 18:02     ` Steven Sistare
2020-07-30 15:14 ` [PATCH V1 08/32] savevm: HMP " Steve Sistare
2020-09-11 17:27   ` Dr. David Alan Gilbert
2020-07-30 15:14 ` [PATCH V1 09/32] savevm: prevent cprsave if memory is volatile Steve Sistare
2020-09-11 17:35   ` Dr. David Alan Gilbert
2020-07-30 15:14 ` [PATCH V1 10/32] kvmclock: restore paused KVM clock Steve Sistare
2020-09-11 17:50   ` Dr. David Alan Gilbert
2020-07-30 15:14 ` [PATCH V1 11/32] cpu: disable ticks when suspended Steve Sistare
2020-09-11 17:53   ` Dr. David Alan Gilbert
2020-07-30 15:14 ` [PATCH V1 12/32] vl: pause option Steve Sistare
2020-07-30 16:20   ` Eric Blake
2020-07-30 18:11     ` Steven Sistare
2020-07-31 10:07       ` Daniel P. Berrangé
2020-07-31 15:18         ` Steven Sistare
2020-07-30 17:03   ` Alex Bennée
2020-07-30 18:14     ` Steven Sistare
2020-07-31  9:44       ` Alex Bennée
2020-09-11 17:59       ` Dr. David Alan Gilbert
2020-07-30 15:14 ` [PATCH V1 13/32] gdbstub: gdb support for suspended state Steve Sistare
2020-09-11 18:41   ` Dr. David Alan Gilbert
2020-07-30 15:14 ` [PATCH V1 14/32] savevm: VMS_RESTART and cprsave restart Steve Sistare
2020-07-30 16:22   ` Eric Blake
2020-07-30 18:14     ` Steven Sistare
2020-09-11 18:44   ` Dr. David Alan Gilbert
2020-07-30 15:14 ` [PATCH V1 15/32] vl: QEMU_START_FREEZE env var Steve Sistare
2020-09-11 18:49   ` Dr. David Alan Gilbert
2020-07-30 15:14 ` [PATCH V1 16/32] oslib: add qemu_clr_cloexec Steve Sistare
2020-09-11 18:52   ` Dr. David Alan Gilbert
2020-07-30 15:14 ` [PATCH V1 17/32] util: env var helpers Steve Sistare
2020-09-11 19:00   ` Dr. David Alan Gilbert
2020-07-30 15:14 ` [PATCH V1 18/32] osdep: import MADV_DOEXEC Steve Sistare
2020-08-17 18:30   ` Steven Sistare
2020-08-17 20:48     ` Alex Williamson
2020-08-17 21:20       ` Steven Sistare
2020-08-17 21:44         ` Alex Williamson
2020-08-18  2:42           ` Alex Williamson
2020-08-19 21:52             ` Steven Sistare
2020-08-24 22:30               ` Alex Williamson
2020-07-30 15:14 ` [PATCH V1 19/32] memory: ram_block_add cosmetic changes Steve Sistare
2020-07-30 15:14 ` [PATCH V1 20/32] vl: add helper to request re-exec Steve Sistare
2020-07-30 15:14 ` [PATCH V1 21/32] exec, memory: exec(3) to restart Steve Sistare
2020-07-30 15:14 ` [PATCH V1 22/32] char: qio_channel_socket_accept reuse fd Steve Sistare
2020-09-15 17:33   ` Dr. David Alan Gilbert
2020-09-15 17:53     ` Daniel P. Berrangé
2020-07-30 15:14 ` [PATCH V1 23/32] char: save/restore chardev socket fds Steve Sistare
2020-07-30 15:14 ` [PATCH V1 24/32] ui: save/restore vnc " Steve Sistare
2020-07-31  9:06   ` Daniel P. Berrangé
2020-07-31 16:51     ` Steven Sistare
2020-07-30 15:14 ` [PATCH V1 25/32] char: save/restore chardev pty fds Steve Sistare
2020-07-30 15:14 ` [PATCH V1 26/32] monitor: save/restore QMP negotiation status Steve Sistare
2020-07-30 15:14 ` [PATCH V1 27/32] vhost: reset vhost devices upon cprsave Steve Sistare
2020-07-30 15:14 ` [PATCH V1 28/32] char: restore terminal on restart Steve Sistare
2020-07-30 15:14 ` [PATCH V1 29/32] pci: export pci_update_mappings Steve Sistare
2020-07-30 15:14 ` [PATCH V1 30/32] vfio-pci: save and restore Steve Sistare
2020-08-06 10:22   ` Jason Zeng
2020-08-07 20:38     ` Steven Sistare
2020-08-10  3:50       ` Jason Zeng
2020-08-19 21:15         ` Steven Sistare
2020-08-20 10:33           ` Jason Zeng
2020-07-30 15:14 ` [PATCH V1 31/32] vfio-pci: trace pci config Steve Sistare
2020-07-30 15:14 ` [PATCH V1 32/32] vfio-pci: improved tracing Steve Sistare
2020-09-15 18:49   ` Dr. David Alan Gilbert
2020-07-30 16:52 ` [PATCH V1 00/32] Live Update Daniel P. Berrangé
2020-07-30 18:48   ` Steven Sistare
2020-07-31  8:53     ` Daniel P. Berrangé
2020-07-31 15:27       ` Steven Sistare
2020-07-31 15:52         ` Daniel P. Berrangé
2020-07-31 17:20           ` Steven Sistare
2020-08-11 19:08           ` Dr. David Alan Gilbert
2020-07-30 17:15 ` Paolo Bonzini
2020-07-30 19:09   ` Steven Sistare
2020-07-30 21:39     ` Paolo Bonzini
2020-07-31 19:22       ` Steven Sistare
2020-07-30 17:49 ` Dr. David Alan Gilbert
2020-07-30 19:31   ` Steven Sistare
2020-08-04 18:18 ` Steven Sistare

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=1596122076-341293-1-git-send-email-steven.sistare@oracle.com \
    --to=steven.sistare@oracle.com \
    --cc=alex.bennee@linaro.org \
    --cc=alex.williamson@redhat.com \
    --cc=armbru@redhat.com \
    --cc=berrange@redhat.com \
    --cc=dgilbert@redhat.com \
    --cc=marcandre.lureau@redhat.com \
    --cc=mst@redhat.com \
    --cc=pbonzini@redhat.com \
    --cc=philmd@redhat.com \
    --cc=qemu-devel@nongnu.org \
    --cc=quintela@redhat.com \
    --cc=stefanha@redhat.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

QEMU-Devel Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/qemu-devel/0 qemu-devel/git/0.git
	git clone --mirror https://lore.kernel.org/qemu-devel/1 qemu-devel/git/1.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 qemu-devel qemu-devel/ https://lore.kernel.org/qemu-devel \
		qemu-devel@nongnu.org
	public-inbox-index qemu-devel

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.nongnu.qemu-devel


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git