RE: [PATCH v2 1/1] docs/devel: Add VFIO device migration documentation

From: "Tian, Kevin" <kevin.tian@intel.com>
To: "Dr. David Alan Gilbert" <dgilbert@redhat.com>,
	"Daniel P. Berrangé" <berrange@redhat.com>
Cc: "alex.williamson@redhat.com" <alex.williamson@redhat.com>,
	"cjia@nvidia.com" <cjia@nvidia.com>,
	"quintela@redhat.com" <quintela@redhat.com>,
	"cohuck@redhat.com" <cohuck@redhat.com>,
	"qemu-devel@nongnu.org" <qemu-devel@nongnu.org>,
	"lushenming@huawei.com" <lushenming@huawei.com>,
	Kirti Wankhede <kwankhede@nvidia.com>,
	Tarun Gupta <targupta@nvidia.com>,
	"Zhao, Yan Y" <yan.y.zhao@intel.com>,
	"philmd@redhat.com" <philmd@redhat.com>,
	"dnigam@nvidia.com" <dnigam@nvidia.com>
Subject: RE: [PATCH v2 1/1] docs/devel: Add VFIO device migration documentation
Date: Fri, 12 Mar 2021 02:30:32 +0000	[thread overview]
Message-ID: <MWHPR11MB1886E79EB08B27A46D3AC8E88C6F9@MWHPR11MB1886.namprd11.prod.outlook.com> (raw)
In-Reply-To: <YEpyYY2jE8TAWD8u@work-vm>

> From: Qemu-devel <qemu-devel-bounces+kevin.tian=intel.com@nongnu.org>
> On Behalf Of Dr. David Alan Gilbert
> 
> * Daniel P. Berrangé (berrange@redhat.com) wrote:
> > On Thu, Mar 11, 2021 at 12:50:09AM +0530, Tarun Gupta wrote:
> > > Document interfaces used for VFIO device migration. Added flow of state
> changes
> > > during live migration with VFIO device. Tested by building docs with the
> new
> > > vfio-migration.rst file.
> > >
> > > v2:
> > > - Included the new vfio-migration.rst file in index.rst
> > > - Updated dirty page tracking section, also added details about
> > >   'pre-copy-dirty-page-tracking' opt-out option.
> > > - Incorporated comments around wording of doc.
> > >
> > > Signed-off-by: Tarun Gupta <targupta@nvidia.com>
> > > Signed-off-by: Kirti Wankhede <kwankhede@nvidia.com>
> > > ---
> > >  MAINTAINERS                   |   1 +
> > >  docs/devel/index.rst          |   1 +
> > >  docs/devel/vfio-migration.rst | 135
> ++++++++++++++++++++++++++++++++++
> > >  3 files changed, 137 insertions(+)
> > >  create mode 100644 docs/devel/vfio-migration.rst
> >
> >
> > > +Postcopy
> > > +========
> > > +
> > > +Postcopy migration is not supported for VFIO devices.
> >
> > What is the problem here and is there any plan for how to address it ?
> 
> There's no equivalent to userfaultfd for accesses to RAM made by a
> device.
> There's some potential for this to be doable with an IOMMU or the like,
> but:
>   a) IOMMUs and devices aren't currently happy at recovering from
> failures
>   b) the fragementation you get during a postcopy probably isn't pretty
> when you get to build IOMMU tables.

To overcome such limitations one may adopt a prefault-and-pull scheme if 
the vendor driver has the capability to track pending DMA buffers in the
migration process (with additional uAPI changes in VFIO or userfaultfd), 
as discussed here:

https://static.sched.com/hosted_files/kvmforum2019/7a/kvm-forum-postcopy-final.pdf

> 
> > Postcopy is essentially the only migration mechanism that can reliably
> > complete, so it really should be considered the default approach to
> > migration for all mgmt apps wanting to do migration, except in special
> > cases.   IOW, if we want VFIO migration to be viable, we need postcopy
> > support.
> 
> There's lots of other things postcopy doesn't work with; so hmm.
> 

Agree. Also given the amount of work even for pre-copy migration, it makes 
more sense to do things step-by-step.

Thanks
Kevin