Re: [PATCH V9 00/46] Live Update

[Steven Sistare <steven.sistare@oracle.com>]

This series desperately needs review in its intersection with live migration.
The code in other areas has been reviewed and revised multiple times -- thank you!

David, Juan, can you spare some time to review this?  I have done my best to order 
the patches logically (see the labelled groups in this email), and to provide 
complete and clear cover letter and commit messages. Can I do anything to facilitate,
like doing a code walk through via zoom?

And of course, I welcome anyone's feedback.

Here is the original posting.

https://lore.kernel.org/qemu-devel/1658851843-236870-1-git-send-email-steven.sistare@oracle.com/

- Steve

On 7/26/2022 12:09 PM, Steve Sistare wrote:
> This version of the live update patch series integrates live update into the
> live migration framework.  The new interfaces are:
>   * mode (migration parameter)
>   * cpr-exec-args (migration parameter)
>   * file (migration URI)
>   * migrate-mode-enable (command-line argument)
>   * only-cpr-capable (command-line argument)
> 
> Provide the cpr-exec and cpr-reboot migration modes for live update.  These
> save and restore VM state, with minimal guest pause time, so that qemu may be
> updated to a new version in between.  The caller sets the mode parameter
> before invoking the migrate or migrate-incoming commands.
> 
> In cpr-reboot mode, the migrate command saves state to a file, allowing
> one to quit qemu, reboot to an updated kernel, start an updated version of
> qemu, and resume via the migrate-incoming command.  The caller must specify
> a migration URI that writes to and reads from a file.  Unlike normal mode,
> the use of certain local storage options does not block the migration, but
> the caller must not modify guest block devices between the quit and restart.
> The guest RAM memory-backend must be shared, and the @x-ignore-shared
> migration capability must be set, to avoid saving it to the file.  Guest RAM
> must be non-volatile across reboot, which can be achieved by backing it with
> a dax device, or /dev/shm PKRAM as proposed in
> https://lore.kernel.org/lkml/1617140178-8773-1-git-send-email-anthony.yznaga@oracle.com
> but this is not enforced.  The restarted qemu arguments must match those used
> to initially start qemu, plus the -incoming option.
> 
> The reboot mode supports vfio devices if the caller first suspends the guest,
> such as by issuing guest-suspend-ram to the qemu guest agent.  The guest
> drivers' suspend methods flush outstanding requests and re-initialize the
> devices, and thus there is no device state to save and restore.  After
> issuing migrate-incoming, the caller must issue a system_wakeup command to
> resume.
> 
> In cpr-exec mode, the migrate command saves state to a file and directly
> exec's a new version of qemu on the same host, replacing the original process
> while retaining its PID.  The caller must specify a migration URI that writes
> to and reads from a file, and resumes execution via the migrate-incoming
> command.  Arguments for the new qemu process are taken from the cpr-exec-args
> migration parameter, and must include the -incoming option.
> 
> Guest RAM must be backed by a memory backend with share=on, but cannot be
> memory-backend-ram.  The memory is re-mmap'd in the updated process, so guest
> ram is efficiently preserved in place, albeit with new virtual addresses.
> In addition, the '-migrate-mode-enable cpr-exec' option is required.  This
> causes secondary guest ram blocks (those not specified on the command line)
> to be allocated by mmap'ing a memfd.  The memfds are kept open across exec,
> their values are saved in special cpr state which is retrieved after exec,
> and they are re-mmap'd.  Since guest RAM is not copied, and storage blocks
> are not migrated, the caller must disable all capabilities related to page
> and block copy.  The implementation ignores all related parameters.
> 
> The exec mode supports vfio devices by preserving the vfio container, group,
> device, and event descriptors across the qemu re-exec, and by updating DMA
> mapping virtual addresses using VFIO_DMA_UNMAP_FLAG_VADDR and
> VFIO_DMA_MAP_FLAG_VADDR as defined in
>   https://lore.kernel.org/kvm/1611939252-7240-1-git-send-email-steven.sistare@oracle.com
> and integrated in Linux kernel 5.12.
> 
> Here is an example of updating qemu from v7.0.50 to v7.0.51 using exec mode.
> The software update is performed while the guest is running to minimize
> downtime.
> 
> window 1                                        | window 2
>                                                 |
> # qemu-system-$arch ...                         |
>   -migrate-mode-enable cpr-exec                 |
> QEMU 7.0.50 monitor - type 'help' ...           |
> (qemu) info status                              |
> VM status: running                              |
>                                                 | # yum update qemu
> (qemu) migrate_set_parameter mode cpr-exec      |
> (qemu) migrate_set_parameter cpr-exec-args      |
>   qemu-system-$arch ... -incoming defer         |
> (qemu) migrate -d file:/tmp/qemu.sav            |
> QEMU 7.0.51 monitor - type 'help' ...           |
> (qemu) info status                              |
> VM status: paused (inmigrate)                   |
> (qemu) migrate_incoming file:/tmp/qemu.sav      |
> (qemu) info status                              |
> VM status: running                              |
> 
> 
> Here is an example of updating the host kernel using reboot mode.
> 
> window 1                                        | window 2
>                                                 |
> # qemu-system-$arch ... mem-path=/dev/dax0.0    |
>   -migrate-mode-enable cpr-reboot               |
> QEMU 7.0.50 monitor - type 'help' ...           |
> (qemu) info status                              |
> VM status: running                              |
>                                                 | # yum update kernel-uek
> (qemu) migrate_set_parameter mode cpr-reboot    |
> (qemu) migrate -d file:/tmp/qemu.sav            |
> (qemu) quit                                     |
>                                                 |
> # systemctl kexec                               |
> kexec_core: Starting new kernel                 |
> ...                                             |
>                                                 |
> # qemu-system-$arch mem-path=/dev/dax0.0 ...    |
>   -incoming defer                               |
> QEMU 7.0.51 monitor - type 'help' ...           |
> (qemu) info status                              |
> VM status: paused (inmigrate)                   |
> (qemu) migrate_incoming file:/tmp/qemu.sav      |
> (qemu) info status                              |
> VM status: running                              |
> 
> Changes from V8 to V9:
>   vfio:
>     - free all cpr state during unwind in vfio_connect_container
>     - change cpr_resave_fd to return void, and avoid new unwind cases
>     - delete incorrect .unmigratable=1 in vmstate handlers
>     - add route batching in vfio_claim_vectors
>     - simplified vfio intx cpr code
>     - fix commit message for 'recover from unmap-all-vaddr failure'
>     - verify suspended runstate for cpr-reboot mode
>   Other:
>     - delete cpr-save, cpr-exec, cpr-load
>     - delete ram block vmstate handlers that were added in V8
>     - rename cpr-enable option to migrate-mode-enable
>     - add file URI for migration
>     - add mode and cpr-exec-args migration parameters
>     - add per-mode migration blockers
>     - add mode checks in migration notifiers
>     - fix suspended runstate during migration
>     - replace RAM_ANON flag with RAM_NAMED_FILE
>     - support memory-backend-epc
> 
> Steve Sistare (44):
>   migration: fix populate_vfio_info                  ---  reboot mode  ---
>   memory: RAM_NAMED_FILE flag
>   migration: file URI
>   migration: mode parameter
>   migration: migrate-enable-mode option
>   migration: simplify blockers
>   migration: per-mode blockers
>   cpr: relax some blockers
>   cpr: reboot mode
> 
>   qdev-properties: strList                           ---  exec mode ---
>   qapi: strList_from_string
>   qapi: QAPI_LIST_LENGTH
>   qapi: strv_from_strList
>   qapi: strList unit tests
>   migration: cpr-exec-args parameter
>   migration: simplify notifiers
>   migration: check mode in notifiers
>   memory: flat section iterator
>   oslib: qemu_clear_cloexec
>   vl: helper to request re-exec
>   cpr: preserve extra state
>   cpr: exec mode
>   cpr: add exec-mode blockers
>   cpr: ram block blockers
>   cpr: only-cpr-capable
>   cpr: Mismatched GPAs fix
>   hostmem-memfd: cpr support
>   hostmem-epc: cpr support
> 
>   pci: export msix_is_pending                       --- vfio for exec ---
>   vfio-pci: refactor for cpr
>   vfio-pci: cpr part 1 (fd and dma)
>   vfio-pci: cpr part 2 (msi)
>   vfio-pci: cpr part 3 (intx)
>   vfio-pci: recover from unmap-all-vaddr failure
> 
>   chardev: cpr framework                            --- misc for exec ---
>   chardev: cpr for simple devices
>   chardev: cpr for pty
>   python/machine: QEMUMachine full_args
>   python/machine: QEMUMachine reopen_qmp_connection
>   tests/avocado: add cpr regression test
> 
>   vl: start on wakeup request                       --- vfio for reboot ---
>   migration: fix suspended runstate
>   migration: notifier error reporting
>   vfio: allow cpr-reboot migration if suspended
> 
> Mark Kanda, Steve Sistare (2):
>   vhost: reset vhost devices for cpr
>   chardev: cpr for sockets
> 
>  MAINTAINERS                         |  14 ++
>  accel/xen/xen-all.c                 |   3 +
>  backends/hostmem-epc.c              |  18 +-
>  backends/hostmem-file.c             |   1 +
>  backends/hostmem-memfd.c            |  22 ++-
>  backends/tpm/tpm_emulator.c         |  11 +-
>  block/parallels.c                   |   7 +-
>  block/qcow.c                        |   7 +-
>  block/vdi.c                         |   7 +-
>  block/vhdx.c                        |   7 +-
>  block/vmdk.c                        |   7 +-
>  block/vpc.c                         |   7 +-
>  block/vvfat.c                       |   7 +-
>  chardev/char-mux.c                  |   1 +
>  chardev/char-null.c                 |   1 +
>  chardev/char-pty.c                  |  16 +-
>  chardev/char-serial.c               |   1 +
>  chardev/char-socket.c               |  48 +++++
>  chardev/char-stdio.c                |  31 +++
>  chardev/char.c                      |  49 ++++-
>  dump/dump.c                         |   4 +-
>  gdbstub.c                           |   1 +
>  hmp-commands.hx                     |   2 +-
>  hw/9pfs/9p.c                        |  11 +-
>  hw/core/qdev-properties-system.c    |  12 ++
>  hw/core/qdev-properties.c           |  44 +++++
>  hw/display/virtio-gpu-base.c        |   8 +-
>  hw/intc/arm_gic_kvm.c               |   3 +-
>  hw/intc/arm_gicv3_its_kvm.c         |   3 +-
>  hw/intc/arm_gicv3_kvm.c             |   3 +-
>  hw/misc/ivshmem.c                   |   8 +-
>  hw/net/virtio-net.c                 |  10 +-
>  hw/pci/msix.c                       |   2 +-
>  hw/pci/pci.c                        |  12 ++
>  hw/ppc/pef.c                        |   2 +-
>  hw/ppc/spapr.c                      |   2 +-
>  hw/ppc/spapr_events.c               |   2 +-
>  hw/ppc/spapr_rtas.c                 |   2 +-
>  hw/remote/proxy.c                   |   7 +-
>  hw/s390x/s390-virtio-ccw.c          |   9 +-
>  hw/scsi/vhost-scsi.c                |   9 +-
>  hw/vfio/common.c                    | 235 +++++++++++++++++++----
>  hw/vfio/cpr.c                       | 177 ++++++++++++++++++
>  hw/vfio/meson.build                 |   1 +
>  hw/vfio/migration.c                 |  23 +--
>  hw/vfio/pci.c                       | 336 ++++++++++++++++++++++++++++-----
>  hw/vfio/trace-events                |   1 +
>  hw/virtio/vhost-vdpa.c              |   6 +-
>  hw/virtio/vhost.c                   |  32 +++-
>  include/chardev/char-socket.h       |   1 +
>  include/chardev/char.h              |   5 +
>  include/exec/memory.h               |  48 +++++
>  include/exec/ram_addr.h             |   1 +
>  include/exec/ramblock.h             |   1 +
>  include/hw/pci/msix.h               |   1 +
>  include/hw/qdev-properties-system.h |   4 +
>  include/hw/qdev-properties.h        |   3 +
>  include/hw/vfio/vfio-common.h       |  12 ++
>  include/hw/virtio/vhost.h           |   1 +
>  include/migration/blocker.h         |  69 ++++++-
>  include/migration/cpr-state.h       |  30 +++
>  include/migration/cpr.h             |  20 ++
>  include/migration/misc.h            |  13 +-
>  include/migration/vmstate.h         |   2 +
>  include/qapi/util.h                 |  28 +++
>  include/qemu/osdep.h                |   9 +
>  include/sysemu/runstate.h           |   2 +
>  migration/cpr-state.c               | 362 ++++++++++++++++++++++++++++++++++++
>  migration/cpr.c                     |  85 +++++++++
>  migration/file.c                    |  62 ++++++
>  migration/file.h                    |  14 ++
>  migration/meson.build               |   3 +
>  migration/migration.c               | 268 +++++++++++++++++++++++---
>  migration/ram.c                     |  24 ++-
>  migration/target.c                  |   1 +
>  migration/trace-events              |  12 ++
>  monitor/hmp-cmds.c                  |  59 +++---
>  monitor/hmp.c                       |   3 +
>  monitor/qmp.c                       |   4 +
>  python/qemu/machine/machine.py      |  14 ++
>  qapi/char.json                      |   7 +-
>  qapi/migration.json                 |  68 ++++++-
>  qapi/qapi-util.c                    |  37 ++++
>  qemu-options.hx                     |  50 ++++-
>  replay/replay.c                     |   4 +
>  softmmu/memory.c                    |  31 ++-
>  softmmu/physmem.c                   | 100 +++++++++-
>  softmmu/runstate.c                  |  42 ++++-
>  softmmu/vl.c                        |  10 +
>  stubs/cpr-state.c                   |  26 +++
>  stubs/meson.build                   |   2 +
>  stubs/migr-blocker.c                |   9 +-
>  stubs/migration.c                   |  33 ++++
>  target/i386/kvm/kvm.c               |   8 +-
>  target/i386/nvmm/nvmm-all.c         |   4 +-
>  target/i386/sev.c                   |   2 +-
>  target/i386/whpx/whpx-all.c         |   3 +-
>  tests/avocado/cpr.py                | 176 ++++++++++++++++++
>  tests/unit/meson.build              |   1 +
>  tests/unit/test-strlist.c           |  81 ++++++++
>  trace-events                        |   1 +
>  ui/spice-core.c                     |   5 +-
>  ui/vdagent.c                        |   5 +-
>  util/oslib-posix.c                  |   9 +
>  util/oslib-win32.c                  |   4 +
>  105 files changed, 2781 insertions(+), 330 deletions(-)
>  create mode 100644 hw/vfio/cpr.c
>  create mode 100644 include/migration/cpr-state.h
>  create mode 100644 include/migration/cpr.h
>  create mode 100644 migration/cpr-state.c
>  create mode 100644 migration/cpr.c
>  create mode 100644 migration/file.c
>  create mode 100644 migration/file.h
>  create mode 100644 stubs/cpr-state.c
>  create mode 100644 stubs/migration.c
>  create mode 100644 tests/avocado/cpr.py
>  create mode 100644 tests/unit/test-strlist.c
>