From: "Dr. David Alan Gilbert" <dgilbert@redhat.com>
To: "Daniel P. Berrangé" <berrange@redhat.com>
Cc: "Michael S. Tsirkin" <mst@redhat.com>,
"Alex Bennée" <alex.bennee@linaro.org>,
"Juan Quintela" <quintela@redhat.com>,
qemu-devel@nongnu.org, "Markus Armbruster" <armbru@redhat.com>,
"Alex Williamson" <alex.williamson@redhat.com>,
"Steven Sistare" <steven.sistare@oracle.com>,
"Stefan Hajnoczi" <stefanha@redhat.com>,
"Marc-André Lureau" <marcandre.lureau@redhat.com>,
"Paolo Bonzini" <pbonzini@redhat.com>,
"Philippe Mathieu-Daudé" <philmd@redhat.com>
Subject: Re: [PATCH V1 00/32] Live Update
Date: Tue, 11 Aug 2020 20:08:30 +0100 [thread overview]
Message-ID: <20200811190830.GQ2671@work-vm> (raw)
In-Reply-To: <20200731155252.GE3660103@redhat.com>
* Daniel P. Berrangé (berrange@redhat.com) wrote:
> On Fri, Jul 31, 2020 at 11:27:45AM -0400, Steven Sistare wrote:
> > On 7/31/2020 4:53 AM, Daniel P. Berrangé wrote:
> > > On Thu, Jul 30, 2020 at 02:48:44PM -0400, Steven Sistare wrote:
> > >> On 7/30/2020 12:52 PM, Daniel P. Berrangé wrote:
> > >>> On Thu, Jul 30, 2020 at 08:14:04AM -0700, Steve Sistare wrote:
> > >>>> Improve and extend the qemu functions that save and restore VM state so a
> > >>>> guest may be suspended and resumed with minimal pause time. qemu may be
> > >>>> updated to a new version in between.
> > >>>>
> > >>>> The first set of patches adds the cprsave and cprload commands to save and
> > >>>> restore VM state, and allow the host kernel to be updated and rebooted in
> > >>>> between. The VM must create guest RAM in a persistent shared memory file,
> > >>>> such as /dev/dax0.0 or persistant /dev/shm PKRAM as proposed in
> > >>>> https://lore.kernel.org/lkml/1588812129-8596-1-git-send-email-anthony.yznaga@oracle.com/
> > >>>>
> > >>>> cprsave stops the VCPUs and saves VM device state in a simple file, and
> > >>>> thus supports any type of guest image and block device. The caller must
> > >>>> not modify the VM's block devices between cprsave and cprload.
> > >>>>
> > >>>> cprsave and cprload support guests with vfio devices if the caller first
> > >>>> suspends the guest by issuing guest-suspend-ram to the qemu guest agent.
> > >>>> The guest drivers suspend methods flush outstanding requests and re-
> > >>>> initialize the devices, and thus there is no device state to save and
> > >>>> restore.
> > >>>>
> > >>>> 1 savevm: add vmstate handler iterators
> > >>>> 2 savevm: VM handlers mode mask
> > >>>> 3 savevm: QMP command for cprsave
> > >>>> 4 savevm: HMP Command for cprsave
> > >>>> 5 savevm: QMP command for cprload
> > >>>> 6 savevm: HMP Command for cprload
> > >>>> 7 savevm: QMP command for cprinfo
> > >>>> 8 savevm: HMP command for cprinfo
> > >>>> 9 savevm: prevent cprsave if memory is volatile
> > >>>> 10 kvmclock: restore paused KVM clock
> > >>>> 11 cpu: disable ticks when suspended
> > >>>> 12 vl: pause option
> > >>>> 13 gdbstub: gdb support for suspended state
> > >>>>
> > >>>> The next patches add a restart method that eliminates the persistent memory
> > >>>> constraint, and allows qemu to be updated across the restart, but does not
> > >>>> allow host reboot. Anonymous memory segments used by the guest are
> > >>>> preserved across a re-exec of qemu, mapped at the same VA, via a proposed
> > >>>> madvise(MADV_DOEXEC) option in the Linux kernel. See
> > >>>> https://lore.kernel.org/lkml/1595869887-23307-1-git-send-email-anthony.yznaga@oracle.com/
> > >>>>
> > >>>> 14 savevm: VMS_RESTART and cprsave restart
> > >>>> 15 vl: QEMU_START_FREEZE env var
> > >>>> 16 oslib: add qemu_clr_cloexec
> > >>>> 17 util: env var helpers
> > >>>> 18 osdep: import MADV_DOEXEC
> > >>>> 19 memory: ram_block_add cosmetic changes
> > >>>> 20 vl: add helper to request re-exec
> > >>>> 21 exec, memory: exec(3) to restart
> > >>>> 22 char: qio_channel_socket_accept reuse fd
> > >>>> 23 char: save/restore chardev socket fds
> > >>>> 24 ui: save/restore vnc socket fds
> > >>>> 25 char: save/restore chardev pty fds
> > >>>
> > >>> Keeping FDs open across re-exec is a nice trick, but how are you dealing
> > >>> with the state associated with them, most especially the TLS encryption
> > >>> state ? AFAIK, there's no way to serialize/deserialize the TLS state that
> > >>> GNUTLS maintains, and the patches don't show any sign of dealing with
> > >>> this. IOW it looks like while the FD will be preserved, any TLS session
> > >>> running on it will fail.
> > >>
> > >> I had not considered TLS. If a non-qemu library maintains connection state, then
> > >> we won't be able to support it for live update until the library provides interfaces
> > >> to serialize the state.
> > >>
> > >> For qemu objects, so far vmstate has been adequate to represent the devices with
> > >> descriptors that we preserve.
> > >
> > > My main concern about this series is that there is an implicit assumption
> > > that QEMU is *not* configured with certain features that are not handled
> > > If QEMU is using one of the unsupported features, I don't see anything in
> > > the series which attempts to prevent the actions.
> > >
> > > IOW, users can have an arbitrary QEMU config, attempt to use these new features,
> > > the commands may well succeed, but the user is silently left with a broken QEMU.
> > > Such silent failure modes are really undesirable as they'll lead to a never
> > > ending stream of hard to diagnose bug reports for QEMU maintainers.
> > >
> > > TLS is one example of this, the live upgrade will "succeed", but the TLS
> > > connections will be totally non-functional.
> >
> > I agree with all your points and would like to do better in this area. Other than hunting for
> > every use of a descriptor and either supporting it or blocking cpr, do you have any suggestions?
> > Thinking out loud, maybe we can gather all the fds that we support, then look for all fds in the
> > process, and block the cpr if we find an unrecognized fd.
>
> There's no magic easy answer to this problem. Conceptually it is similar to
> the problem of reliably migrating guest device state, but in this case we're
> primarily concerned about the backends instead.
>
> For migration we've got standardized interfaces that devices must implement
> in order to correctly support migration serialization. There is also support
> for devices to register migration "blockers" which prevent any use of the
> migration feature when the device is present.
>
> We lack this kind of concept for the backend, and that's what I think needs
> to be tackled in a more thorough way. There are quite alot of backends,
> but they're grouped into a reasonable small number of sets (UIs, chardevs,
> blockdevs, net devs, etc). We need some standard interface that we can
> plumb into all the backends, along with providing backends the ability to
> block the re-exec. If we plumb the generic infrastructure into each of the
> different types of backend, and make the default behaviour be to reject
> the re-exec. Then we need to carefull consider specific backend impls
> and allow the re-exec only in the very precise cases we can demonstrate
> to be safe.
>
> IOW, have a presumption that re-exec will *not* be permitted. Over time
> we can make it work for an ever expanding set of use cases.
Yes, it does feel like an interface that needs to be implemented on the
chardev; then you don't need to worry about handling them all
individually.
Dave
>
> Regards,
> Daniel
> --
> |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org -o- https://fstop138.berrange.com :|
> |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
next prev parent reply other threads:[~2020-08-11 19:09 UTC|newest]
Thread overview: 118+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-07-30 15:14 [PATCH V1 00/32] Live Update Steve Sistare
2020-07-30 15:14 ` [PATCH V1 01/32] savevm: add vmstate handler iterators Steve Sistare
2020-09-11 16:24 ` Dr. David Alan Gilbert
2020-09-24 21:43 ` Steven Sistare
2020-09-25 9:07 ` Dr. David Alan Gilbert
2020-07-30 15:14 ` [PATCH V1 02/32] savevm: VM handlers mode mask Steve Sistare
2020-07-30 15:14 ` [PATCH V1 03/32] savevm: QMP command for cprsave Steve Sistare
2020-07-30 16:12 ` Eric Blake
2020-07-30 17:52 ` Steven Sistare
2020-09-11 16:43 ` Dr. David Alan Gilbert
2020-09-25 18:43 ` Steven Sistare
2020-09-25 22:22 ` Steven Sistare
2020-07-30 15:14 ` [PATCH V1 04/32] savevm: HMP Command " Steve Sistare
2020-09-11 16:57 ` Dr. David Alan Gilbert
2020-09-24 21:44 ` Steven Sistare
2020-09-25 9:26 ` Dr. David Alan Gilbert
2020-07-30 15:14 ` [PATCH V1 05/32] savevm: QMP command for cprload Steve Sistare
2020-07-30 16:14 ` Eric Blake
2020-07-30 18:00 ` Steven Sistare
2020-09-11 17:18 ` Dr. David Alan Gilbert
2020-09-24 21:49 ` Steven Sistare
2020-07-30 15:14 ` [PATCH V1 06/32] savevm: HMP Command " Steve Sistare
2020-07-30 15:14 ` [PATCH V1 07/32] savevm: QMP command for cprinfo Steve Sistare
2020-07-30 16:17 ` Eric Blake
2020-07-30 18:02 ` Steven Sistare
2020-07-30 15:14 ` [PATCH V1 08/32] savevm: HMP " Steve Sistare
2020-09-11 17:27 ` Dr. David Alan Gilbert
2020-09-24 21:50 ` Steven Sistare
2020-07-30 15:14 ` [PATCH V1 09/32] savevm: prevent cprsave if memory is volatile Steve Sistare
2020-09-11 17:35 ` Dr. David Alan Gilbert
2020-09-24 21:51 ` Steven Sistare
2020-07-30 15:14 ` [PATCH V1 10/32] kvmclock: restore paused KVM clock Steve Sistare
2020-09-11 17:50 ` Dr. David Alan Gilbert
2020-09-25 18:07 ` Steven Sistare
2020-07-30 15:14 ` [PATCH V1 11/32] cpu: disable ticks when suspended Steve Sistare
2020-09-11 17:53 ` Dr. David Alan Gilbert
2020-09-24 20:42 ` Steven Sistare
2020-09-25 9:03 ` Dr. David Alan Gilbert
2020-07-30 15:14 ` [PATCH V1 12/32] vl: pause option Steve Sistare
2020-07-30 16:20 ` Eric Blake
2020-07-30 18:11 ` Steven Sistare
2020-07-31 10:07 ` Daniel P. Berrangé
2020-07-31 15:18 ` Steven Sistare
2020-07-30 17:03 ` Alex Bennée
2020-07-30 18:14 ` Steven Sistare
2020-07-31 9:44 ` Alex Bennée
2020-09-11 17:59 ` Dr. David Alan Gilbert
2020-09-24 21:51 ` Steven Sistare
2020-07-30 15:14 ` [PATCH V1 13/32] gdbstub: gdb support for suspended state Steve Sistare
2020-09-11 18:41 ` Dr. David Alan Gilbert
2020-09-24 21:51 ` Steven Sistare
2020-07-30 15:14 ` [PATCH V1 14/32] savevm: VMS_RESTART and cprsave restart Steve Sistare
2020-07-30 16:22 ` Eric Blake
2020-07-30 18:14 ` Steven Sistare
2020-09-11 18:44 ` Dr. David Alan Gilbert
2020-09-24 21:44 ` Steven Sistare
2020-07-30 15:14 ` [PATCH V1 15/32] vl: QEMU_START_FREEZE env var Steve Sistare
2020-09-11 18:49 ` Dr. David Alan Gilbert
2020-09-24 21:47 ` Steven Sistare
2020-09-25 15:52 ` Dr. David Alan Gilbert
2020-07-30 15:14 ` [PATCH V1 16/32] oslib: add qemu_clr_cloexec Steve Sistare
2020-09-11 18:52 ` Dr. David Alan Gilbert
2020-07-30 15:14 ` [PATCH V1 17/32] util: env var helpers Steve Sistare
2020-09-11 19:00 ` Dr. David Alan Gilbert
2020-09-24 21:52 ` Steven Sistare
2020-07-30 15:14 ` [PATCH V1 18/32] osdep: import MADV_DOEXEC Steve Sistare
2020-08-17 18:30 ` Steven Sistare
2020-08-17 20:48 ` Alex Williamson
2020-08-17 21:20 ` Steven Sistare
2020-08-17 21:44 ` Alex Williamson
2020-08-18 2:42 ` Alex Williamson
2020-08-19 21:52 ` Steven Sistare
2020-08-24 22:30 ` Alex Williamson
2020-10-08 16:32 ` Steven Sistare
2020-10-15 20:36 ` Alex Williamson
2020-10-19 16:33 ` Steven Sistare
2020-10-26 18:28 ` Steven Sistare
2020-07-30 15:14 ` [PATCH V1 19/32] memory: ram_block_add cosmetic changes Steve Sistare
2020-07-30 15:14 ` [PATCH V1 20/32] vl: add helper to request re-exec Steve Sistare
2020-07-30 15:14 ` [PATCH V1 21/32] exec, memory: exec(3) to restart Steve Sistare
2020-07-30 15:14 ` [PATCH V1 22/32] char: qio_channel_socket_accept reuse fd Steve Sistare
2020-09-15 17:33 ` Dr. David Alan Gilbert
2020-09-15 17:53 ` Daniel P. Berrangé
2020-09-24 21:54 ` Steven Sistare
2020-07-30 15:14 ` [PATCH V1 23/32] char: save/restore chardev socket fds Steve Sistare
2020-07-30 15:14 ` [PATCH V1 24/32] ui: save/restore vnc " Steve Sistare
2020-07-31 9:06 ` Daniel P. Berrangé
2020-07-31 16:51 ` Steven Sistare
2020-07-30 15:14 ` [PATCH V1 25/32] char: save/restore chardev pty fds Steve Sistare
2020-07-30 15:14 ` [PATCH V1 26/32] monitor: save/restore QMP negotiation status Steve Sistare
2020-07-30 15:14 ` [PATCH V1 27/32] vhost: reset vhost devices upon cprsave Steve Sistare
2020-07-30 15:14 ` [PATCH V1 28/32] char: restore terminal on restart Steve Sistare
2020-07-30 15:14 ` [PATCH V1 29/32] pci: export pci_update_mappings Steve Sistare
2020-07-30 15:14 ` [PATCH V1 30/32] vfio-pci: save and restore Steve Sistare
2020-08-06 10:22 ` Jason Zeng
2020-08-07 20:38 ` Steven Sistare
2020-08-10 3:50 ` Jason Zeng
2020-08-19 21:15 ` Steven Sistare
2020-08-20 10:33 ` Jason Zeng
2020-10-07 21:25 ` Steven Sistare
2020-07-30 15:14 ` [PATCH V1 31/32] vfio-pci: trace pci config Steve Sistare
2020-07-30 15:14 ` [PATCH V1 32/32] vfio-pci: improved tracing Steve Sistare
2020-09-15 18:49 ` Dr. David Alan Gilbert
2020-09-24 21:52 ` Steven Sistare
2020-07-30 16:52 ` [PATCH V1 00/32] Live Update Daniel P. Berrangé
2020-07-30 18:48 ` Steven Sistare
2020-07-31 8:53 ` Daniel P. Berrangé
2020-07-31 15:27 ` Steven Sistare
2020-07-31 15:52 ` Daniel P. Berrangé
2020-07-31 17:20 ` Steven Sistare
2020-08-11 19:08 ` Dr. David Alan Gilbert [this message]
2020-07-30 17:15 ` Paolo Bonzini
2020-07-30 19:09 ` Steven Sistare
2020-07-30 21:39 ` Paolo Bonzini
2020-07-31 19:22 ` Steven Sistare
2020-07-30 17:49 ` Dr. David Alan Gilbert
2020-07-30 19:31 ` Steven Sistare
2020-08-04 18:18 ` Steven Sistare
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20200811190830.GQ2671@work-vm \
--to=dgilbert@redhat.com \
--cc=alex.bennee@linaro.org \
--cc=alex.williamson@redhat.com \
--cc=armbru@redhat.com \
--cc=berrange@redhat.com \
--cc=marcandre.lureau@redhat.com \
--cc=mst@redhat.com \
--cc=pbonzini@redhat.com \
--cc=philmd@redhat.com \
--cc=qemu-devel@nongnu.org \
--cc=quintela@redhat.com \
--cc=stefanha@redhat.com \
--cc=steven.sistare@oracle.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).