Re: [Qemu-devel] [RFC 00/29] postcopy+vhost-user/shared ram

["Dr. David Alan Gilbert" <dgilbert@redhat.com>]

* Marc-André Lureau (marcandre.lureau@gmail.com) wrote:
> Hi
> 
> On Thu, Jun 29, 2017 at 8:56 PM Dr. David Alan Gilbert <dgilbert@redhat.com>
> wrote:
> 
> > * Dr. David Alan Gilbert (git) (dgilbert@redhat.com) wrote:
> > > From: "Dr. David Alan Gilbert" <dgilbert@redhat.com>
> > >
> > > Hi,
> > >   This is a RFC/WIP series that enables postcopy migration
> > > with shared memory to a vhost-user process.
> > > It's based off current-head + Juan's load_cleanup series, and
> > > Alexey's bitmap series (v4).  It's very lightly tested and seems
> > > to work, but it's quite rough.
> >
> > Marc-André asked if I had a git with it all applied; so here we are:
> > https://github.com/dagrh/qemu/commits/vhost
> > git@github.com:dagrh/qemu.git on the vhost branch
> >
> >
> I started looking at the series, but I am not familiar with ufd/postcopy.

I'm similarly unfamiliar with the vhost code when I started this (which
probably shows!).
The main thing about ufd is that a process registers with the ufd system
and registers an area of memory with it;  accesses to the memory block
until the page is available, a message is sent down the ufd, and whoever
receives that message may then respond by atomically copying a page into
memory, or wakeing the process when it knows the page is there.
This is the first time we've tried to use userfaultfd with shared memory
and it does need a very recent kernel for it (4.11.0 or rhel 7.4 beta)

> Could you update vhost-user.txt to describe the new messages?

See below; I'll add that in.

> Otherwise,
> make check hangs in /x86_64/vhost-user/connect-fail (might be an unrelated
> regression?) Thanks

Entirely possible I broke it; I'll have a look - at the moment I'm more
interested in comments on the structure of this set.

Dave

diff --git a/docs/interop/vhost-user.txt b/docs/interop/vhost-user.txt
index 481ab56e35..fec4cd0ffe 100644
--- a/docs/interop/vhost-user.txt
+++ b/docs/interop/vhost-user.txt
@@ -273,6 +273,14 @@ Once the source has finished migration, rings will be stopped by
 the source. No further update must be done before rings are
 restarted.

+In postcopy migration the slave is started before all the memory has been
+received from the source host, and care must be taken to avoid accessing pages
+that have yet to be received.  The slave opens a 'userfault'-fd and registers
+the memory with it; this fd is then passed back over to the master.
+The master services requests on the userfaultfd for pages that are accessed
+and when the page is available it performs WAKE ioctl's on the userfaultfd
+to wake the stalled slave.
+
 IOMMU support
 -------------

@@ -326,6 +334,7 @@ Protocol features
 #define VHOST_USER_PROTOCOL_F_REPLY_ACK      3
 #define VHOST_USER_PROTOCOL_F_MTU            4
 #define VHOST_USER_PROTOCOL_F_SLAVE_REQ      5
+#define VHOST_USER_PROTOCOL_F_POSTCOPY       6

 Master message types
 --------------------
@@ -402,12 +411,17 @@ Master message types
       Id: 5
       Equivalent ioctl: VHOST_SET_MEM_TABLE
       Master payload: memory regions description
+      Slave payload: (postcopy only) memory regions description

       Sets the memory map regions on the slave so it can translate the vring
       addresses. In the ancillary data there is an array of file descriptors
       for each memory mapped region. The size and ordering of the fds matches
       the number and ordering of memory regions.

+      When postcopy-listening has been received, SET_MEM_TABLE replies with
+      the bases of the memory mapped regions to the master.  It must have mmap'd
+      the regions and enabled userfaultfd on them.
+
  * VHOST_USER_SET_LOG_BASE

       Id: 6
@@ -580,6 +594,29 @@ Master message types
       This request should be send only when VIRTIO_F_IOMMU_PLATFORM feature
       has been successfully negotiated.

+ * VHOST_USER_POSTCOPY_ADVISE
+      Id: 23
+      Master payload: N/A
+      Slave payload: userfault fd + u64
+
+      Master advises slave that a migration with postcopy enabled is underway,
+      the slave must open a userfaultfd for later use.
+      Note that at this stage the migration is still in precopy mode.
+
+ * VHOST_USER_POSTCOPY_LISTEN
+      Id: 24
+      Master payload: N/A
+
+      Master advises slave that a transition to postcopy mode has happened.
+
+ * VHOST_USER_POSTCOPY_END
+      Id: 25
+      Slave payload: u64
+
+      Master advises that postcopy migration has now completed.  The
+      slave must disable the userfaultfd. The response is an acknowledgement
+      only.
+
 Slave message types
 -------------------


> -- 
> Marc-André Lureau
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK