Re: Out-of-Process Device Emulation session at KVM Forum 2020

[Stefan Hajnoczi <stefanha@gmail.com>]

On Tue, Nov 3, 2020 at 7:53 AM Jason Wang <jasowang@redhat.com> wrote:
> On 2020/11/2 下午6:13, Stefan Hajnoczi wrote:
> > On Mon, Nov 02, 2020 at 10:51:18AM +0800, Jason Wang wrote:
> >> On 2020/10/30 下午9:15, Stefan Hajnoczi wrote:
> >>> On Fri, Oct 30, 2020 at 12:08 PM Jason Wang <jasowang@redhat.com> wrote:
> >>>> On 2020/10/30 下午7:13, Stefan Hajnoczi wrote:
> >>>>> On Fri, Oct 30, 2020 at 9:46 AM Jason Wang <jasowang@redhat.com> wrote:
> >>>>>> On 2020/10/30 下午2:21, Stefan Hajnoczi wrote:
> >>>>>>> On Fri, Oct 30, 2020 at 3:04 AM Alex Williamson
> >>>>>>> <alex.williamson@redhat.com> wrote:
> >>>>>>>> It's great to revisit ideas, but proclaiming a uAPI is bad solely
> >>>>>>>> because the data transfer is opaque, without defining why that's bad,
> >>>>>>>> evaluating the feasibility and implementation of defining a well
> >>>>>>>> specified data format rather than protocol, including cross-vendor
> >>>>>>>> support, or proposing any sort of alternative is not so helpful imo.
> >>>>>>> The migration approaches in VFIO and vDPA/vhost were designed for
> >>>>>>> different requirements and I think this is why there are different
> >>>>>>> perspectives on this. Here is a comparison and how VFIO could be
> >>>>>>> extended in the future. I see 3 levels of device state compatibility:
> >>>>>>>
> >>>>>>> 1. The device cannot save/load state blobs, instead userspace fetches
> >>>>>>> and restores specific values of the device's runtime state (e.g. last
> >>>>>>> processed ring index). This is the vhost approach.
> >>>>>>>
> >>>>>>> 2. The device can save/load state in a standard format. This is
> >>>>>>> similar to #1 except that there is a single read/write blob interface
> >>>>>>> instead of fine-grained get_FOO()/set_FOO() interfaces. This approach
> >>>>>>> pushes the migration state parsing into the device so that userspace
> >>>>>>> doesn't need knowledge of every device type. With this approach it is
> >>>>>>> possible for a device from vendor A to migrate to a device from vendor
> >>>>>>> B, as long as they both implement the same standard migration format.
> >>>>>>> The limitation of this approach is that vendor-specific state cannot
> >>>>>>> be transferred.
> >>>>>>>
> >>>>>>> 3. The device can save/load opaque blobs. This is the initial VFIO
> >>>>>>> approach.
> >>>>>> I still don't get why it must be opaque.
> >>>>> If the device state format needs to be in the VMM then each device
> >>>>> needs explicit enablement in each VMM (QEMU, cloud-hypervisor, etc).
> >>>>>
> >>>>> Let's invert the question: why does the VMM need to understand the
> >>>>> device state of a _passthrough_ device?
> >>>> For better manageability, compatibility and debug-ability. If we depends
> >>>> on a opaque structure, do we encourage device to implement its own
> >>>> migration protocol? It would be very challenge.
> >>>>
> >>>> For VFIO in the kernel, I suspect a uAPI that may result a opaque data
> >>>> to be read or wrote from guest violates the Linux uAPI principle. It
> >>>> will be very hard to maintain uABI or even impossible. It looks to me
> >>>> VFIO is the first subsystem that is trying to do this.
> >>> I think our concepts of uAPI are different. The uAPI of read(2) and
> >>> write(2) does not define the structure of the data buffers. VFIO
> >>> device regions are exactly the same, the structure of the data is not
> >>> defined by the kernel uAPI.
> >>
> >> I think we're talking about different things. It's not about the data
> >> structure, it's about whether to data that reads from kernel can be
> >> understood by userspace.
> >>
> >>
> >>> Maybe microcode and firmware loading is an example we agree on?
> >>
> >> I think not. They are bytecodes that have
> >>
> >> 1) strict ABI definitions
> >> 2) understood by userspace
> > No, they can be proprietary formats that neither the Linux kernel nor
> > userspace can parse. For example, look at linux-firmware
> > (https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/about/)
> > it's just a collection of binary blobs. The format is not necessarily
> > public. The only restriction on that repo is that the binary blob must
> > be redistributable and users must be allowed to run them (i.e.
> > proprietary licenses can be used).
>
>
> I think not. Obviously each firmware should have its own ABI no matter
> whether its public or proprietary. For proprietary firmware, it should
> be understood by the proprietary userspace counterpart.

Userspace does not necessarily need to interpret the contents. The
vendor can ship a binary blob and the driver loads the file onto the
device without interpreting it.

> > Or look at other passthrough device interfaces like /dev/i2c or libusb.
> > They expose data to userspace without requiring a defined format. It's
> > the same as VFIO.
>
>
> Again, it should have an ABI there (either device or spec) no matter
> whether or not it's a transport layer. And there will be an endpoint in
> the userspace know all the format.

VFIO defines how userspace interacts with migration regions, see the
patch series that I linked at the beginning of this discussion.
Userspace has control over pausing/resuming the device and reading
migration blobs.

> > In addition, look at kernel uAPIs where userspace acts simply as a data
> > transport for opaque data (e.g. where a userspace helper facilitates
> > communication but has no visibility of the data). I imagine that memory
> > encryption relies on this because the host kernel and userspace do not
> > have access to encrypted memory or associated state - but they need to
> > help migrate them to other hosts.
>
>
> Which uAPI do you mean here?

Migration of encrypted guests. The host kernel and userspace do not
have access to all guest state. Userspace acts as a transport - same
as VFIO migration.

I'm not sure how much of it is already upstream since it's being
actively developed right now, but it's another example where userspace
does not need to and cannot interpret data.

> > I hope these examples show that such APIs don't pose a problem for the
> > Linux uAPI and are already in use. VFIO device state isn't doing
> > anything new here.
>
>
> I feel that you tried to explain "why it can be" but not "why it must
> be". Trying to find one or two subsystems that have opaque uAPI without
> ABI (though I suspect there will be one) may not convince here.

As I've said from the beginning of the discussion, there are multiple
approaches and they are suited to different use cases.

For passthrough devices I think it's preferable for the VMM not to be
involved in the device state representation. This keeps the VMM
simple, avoids code duplication across VMMs, and allows migration to
work in non-virtualization use cases.

Stefan