RE: [RFC PATCH] vfio: Update/Clarify migration uAPI, add NDMA state

From: "Tian, Kevin" <kevin.tian@intel.com>
To: Jason Gunthorpe <jgg@nvidia.com>
Cc: Alex Williamson <alex.williamson@redhat.com>,
	"cohuck@redhat.com" <cohuck@redhat.com>,
	"corbet@lwn.net" <corbet@lwn.net>,
	"kvm@vger.kernel.org" <kvm@vger.kernel.org>,
	"linux-doc@vger.kernel.org" <linux-doc@vger.kernel.org>,
	"farman@linux.ibm.com" <farman@linux.ibm.com>,
	"mjrosato@linux.ibm.com" <mjrosato@linux.ibm.com>,
	"pasic@linux.ibm.com" <pasic@linux.ibm.com>
Subject: RE: [RFC PATCH] vfio: Update/Clarify migration uAPI, add NDMA state
Date: Tue, 11 Jan 2022 03:14:04 +0000	[thread overview]
Message-ID: <BN9PR11MB5276CD4004B789D004C8A1488C519@BN9PR11MB5276.namprd11.prod.outlook.com> (raw)
In-Reply-To: <20220110181140.GH2328285@nvidia.com>

> From: Jason Gunthorpe <jgg@nvidia.com>
> Sent: Tuesday, January 11, 2022 2:12 AM
> 
> On Mon, Jan 10, 2022 at 07:55:16AM +0000, Tian, Kevin wrote:
> 
> > > > {SAVING} -> {RESUMING}
> > > > 	If not supported, user can achieve this via:
> > > > 		{SAVING}->{RUNNING}->{RESUMING}
> > > > 		{SAVING}-RESET->{RUNNING}->{RESUMING}
> > >
> > > This can be:
> > >
> > > SAVING -> STOP -> RESUMING
> >
> > From Alex's original description the default device state is RUNNING.
> > This supposed to be the initial state on the dest machine for the
> > device assigned to Qemu before Qemu resumes the device state.
> > Then how do we eliminate the RUNNING state in above flow? Who
> > makes STOP as the initial state on the dest node?
> 
> All of this notation should be read with the idea that the
> device_state is already somehow moved away from RESET. Ie the above
> notation is about what is possible once qemu has already moved the
> device to SAVING.

Qemu moves the device to SAVING on the src node.

On the dest the device is in RUNNING (after reset) which can be directly
transitioned to RESUMING. I didn't see the point of adding a STOP here.

> 
> > > This is currently buggy as-is because they cannot DMA map these
> > > things, touch them with the CPU and kmap, or do, really, anything with
> > > them.
> >
> > Can you elaborate why mdev cannot access p2p pages?
> 
> It is just a failure of APIs in the kernel. A p2p page has no 'struct
> page' so it cannot be used in a scatter list, and thus cannot be used in
> dma_map_sg.
> 
> It also cannot be kmap'd, or memcpy'd from.
> 
> So, essentially, everything that a current mdev drivers try to do will
> crash with a non-struct page pfn.
> 
> In principle this could all be made to work someday, but it doesn't
> work now.
> 
> What I want to do is make these APIs correctly use struct page and
> block all non-struct page memory from getting into them.
> 

I see. 

It does make sense for the current sw mdev drivers.

Later when supporting hw mdev (with pasid granular isolation in
iommu), this restriction can be uplifted as it doesn't use dma api
and is pretty much like a pdev regarding to ioas management.

Thanks
Kevin