On Thu, Apr 02, 2020 at 11:46:45AM +0100, Daniel P. Berrangé wrote:
> On Thu, Apr 02, 2020 at 11:19:42AM +0100, Stefan Hajnoczi wrote:
> > On Wed, Apr 01, 2020 at 06:58:20PM +0200, Marc-André Lureau wrote:
> > > On Wed, Apr 1, 2020 at 5:51 PM Thanos Makatos
> > > <thanos.makatos@nutanix.com> wrote:
> > > > > > Bear in mind that since this is just a PoC lots of things can break, e.g. some
> > > > > > system call not intercepted etc.
> > > > >
> > > > > Cool, I had a quick look at libvfio and how the transport integrates
> > > > > into libmuser.  The integration on the libmuser side is nice and small.
> > > > >
> > > > > It seems likely that there will be several different implementations of
> > > > > the vfio-over-socket device side (server):
> > > > > 1. libmuser
> > > > > 2. A Rust equivalent to libmuser
> > > > > 3. Maybe a native QEMU implementation for multi-process QEMU (I think JJ
> > > > >    has been investigating this?)
> > > > >
> > > > > In order to interoperate we'll need to maintain a protocol
> > > > > specification.  Mayb You and JJ could put that together and CC the vfio,
> > > > > rust-vmm, and QEMU communities for discussion?
> > > >
> > > > Sure, I can start by drafting a design doc and share it.
> > > 
> > > ok! I am quite amazed you went this far with a ldpreload hack. This
> > > demonstrates some limits of gpl projects, if it was necessary.
> > > 
> > > I think with this work, and the muser experience, you have a pretty
> > > good idea of what the protocol could look like. My approach, as I
> > > remember, was a quite straightforward VFIO over socket translation,
> > > while trying to see if it could share some aspects with vhost-user,
> > > such as memory handling etc.
> > > 
> > > To contrast with the work done on qemu-mp series, I'd also prefer we
> > > focus our work on a vfio-like protocol, before trying to see how qemu
> > > code and interface could be changed over multiple binaries etc. We
> > > will start with some limitations, similar to the one that apply to
> > > VFIO: migration, introspection, managements etc are mostly left out
> > > for now. (iow, qemu-mp is trying to do too many things simultaneously)
> > 
> > qemu-mp has been cut down significantly in order to make it
> > non-invasive.  The model is now much cleaner:
> > 1. No monitor command or command-line option forwarding.  The device
> >    emulation program has its own command-line and monitor that QEMU
> >    doesn't know about.
> > 2. No per-device proxy objects.  A single RemotePCIDevice is added to
> >    QEMU.  In the current patch series it only supports the LSI SCSI
> >    controller but once the socket protocol is changed to
> >    vfio-over-socket it will be possible to use any PCI device.
> > 
> > We recently agreed on dropping live migration to further reduce the
> > patch series.  If you have specific suggestions, please post reviews on
> > the latest patch series.
> 
> To clarify - the decision was to *temporarily* drop live migration, to
> make the initial patch series smaller and thus easier to merge. It does
> ultimately need live migration, so there would be followup patch series
> to provide migration support, after the initial merge.

Yes.  Live migration should come from the VFIO protocol and/or vmstate
DBus.  There is no need to implement it in a qemu-mp-specific way.

Stefan