Re: [PATCH RFC] vfio: Documentation for the migration region

From: Jason Gunthorpe <jgg@nvidia.com>
To: Cornelia Huck <cohuck@redhat.com>
Cc: Alex Williamson <alex.williamson@redhat.com>,
	Jonathan Corbet <corbet@lwn.net>,
	linux-doc@vger.kernel.org, kvm@vger.kernel.org,
	Kirti Wankhede <kwankhede@nvidia.com>,
	Max Gurtovoy <mgurtovoy@nvidia.com>,
	Shameer Kolothum <shameerali.kolothum.thodi@huawei.com>,
	Yishai Hadas <yishaih@nvidia.com>
Subject: Re: [PATCH RFC] vfio: Documentation for the migration region
Date: Fri, 26 Nov 2021 09:06:08 -0400	[thread overview]
Message-ID: <20211126130608.GR4670@nvidia.com> (raw)
In-Reply-To: <87pmqnhy85.fsf@redhat.com>

On Fri, Nov 26, 2021 at 01:56:26PM +0100, Cornelia Huck wrote:
> On Thu, Nov 25 2021, Jason Gunthorpe <jgg@nvidia.com> wrote:
> 
> > On Thu, Nov 25, 2021 at 01:27:12PM +0100, Cornelia Huck wrote:
> >> On Wed, Nov 24 2021, Jason Gunthorpe <jgg@nvidia.com> wrote:
> >> 
> >> > On Wed, Nov 24, 2021 at 05:55:49PM +0100, Cornelia Huck wrote:
> >> 
> >> >> What I meant to say: If we give userspace the flexibility to operate
> >> >> this, we also must give different device types some flexibility. While
> >> >> subchannels will follow the general flow, they'll probably condense/omit
> >> >> some steps, as I/O is quite different to PCI there.
> >> >
> >> > I would say no - migration is general, no device type should get to
> >> > violate this spec.  Did you have something specific in mind? There is
> >> > very little PCI specific here already
> >> 
> >> I'm not really thinking about violating the spec, but more omitting
> >> things that do not really apply to the hardware. For example, it is
> >> really easy to shut up a subchannel, we don't really need to wait until
> >> nothing happens anymore, and it doesn't even have MMIO. 
> >
> > I've never really looked closely at the s390 mdev drivers..
> >
> > What does something like AP even do anyhow? The ioctl handler doesn't
> > do anything, there is no mmap hook, how does the VFIO userspace
> > interact with this thing?
> 
> For AP, the magic is in the hardware/firmware; the vfio parts are needed
> to configure what is exposed to a given guest, not for operation. Once
> it is up, the hardware will handle any instructions directly, the
> hypervisor will not see them. (Unfortunately, none of the details have
> public documentation.) I have no idea how this would play with migration.

That is kind of what I thought..

VFIO is all about exposing a device to userspace control, sounds like
the S390 drivers skipped that step.

KVM is all about taking what userspace can already control and giving
it to a guest, in an accelerated way.

Making a bypass where a KVM guest has more capability than the user
process because VFIO and KVM have been directly coupled completely
upends the whole logical model.

As we talked with Intel's wbinvd stuff you should have a mental model
where the VFIO userspace process can do anything the KVM guest can do
via ioctls on the mdev. KVM is just an accelerated way to do that same
stuff. Maybe S390 doesn't implement those ioctls, but they are
logically part of the model.

So, for the migration doc, imagine some non-accelerated KVM that was
intercepting the guest operations and calling the logical ioctls on
the mdev instead. When we talk about MMIO/PIO/etc it also includes
mdev operation ioctls too, and by extension any ioctl accelerated
inside KVM.

Jason