RE: [PATCH V7 mlx5-next 08/15] vfio: Define device migration protocol v2

From: "Tian, Kevin" <kevin.tian@intel.com>
To: Jason Gunthorpe <jgg@nvidia.com>,
	Alex Williamson <alex.williamson@redhat.com>
Cc: Yishai Hadas <yishaih@nvidia.com>,
	"bhelgaas@google.com" <bhelgaas@google.com>,
	"saeedm@nvidia.com" <saeedm@nvidia.com>,
	"linux-pci@vger.kernel.org" <linux-pci@vger.kernel.org>,
	"kvm@vger.kernel.org" <kvm@vger.kernel.org>,
	"netdev@vger.kernel.org" <netdev@vger.kernel.org>,
	"kuba@kernel.org" <kuba@kernel.org>,
	"leonro@nvidia.com" <leonro@nvidia.com>,
	"kwankhede@nvidia.com" <kwankhede@nvidia.com>,
	"mgurtovoy@nvidia.com" <mgurtovoy@nvidia.com>,
	"maorg@nvidia.com" <maorg@nvidia.com>,
	"Raj, Ashok" <ashok.raj@intel.com>,
	"shameerali.kolothum.thodi@huawei.com" 
	<shameerali.kolothum.thodi@huawei.com>
Subject: RE: [PATCH V7 mlx5-next 08/15] vfio: Define device migration protocol v2
Date: Tue, 15 Feb 2022 10:58:58 +0000	[thread overview]
Message-ID: <BN9PR11MB5276C361E686DCAFA8A078788C349@BN9PR11MB5276.namprd11.prod.outlook.com> (raw)
In-Reply-To: 20220209023645.GN4160@nvidia.com

> From: Tian, Kevin
> Sent: Tuesday, February 15, 2022 6:42 PM
> 
> > From: Jason Gunthorpe <jgg@nvidia.com>
> > Sent: Wednesday, February 9, 2022 10:37 AM
> >
> > > >  /* -------- API for Type1 VFIO IOMMU -------- */
> > > >
> > > >  /**
> > >
> > > Otherwise, I'm still not sure how userspace handles the fact that it
> > > can't know how much data will be read from the device and how
> important
> > > that is.  There's no replacement of that feature from the v1 protocol
> > > here.
> >
> > I'm not sure this was part of the v1 protocol either. Yes it had a
> > pending_bytes, but I don't think it was actually expected to be 100%
> > accurate. Computing this value accurately is potentially quite
> > expensive, I would prefer we not enforce this on an implementation
> > without a reason, and qemu currently doesn't make use of it.
> >
> > The ioctl from the precopy patch is probably the best approach, I
> > think it would be fine to allow that for stop copy as well, but also
> > don't see a usage right now.
> >
> > It is not something that needs decision now, it is very easy to detect
> > if an ioctl is supported on the data_fd at runtime to add new things
> > here when needed.
> >
> 
> Another interesting thing (not an immediate concern on this series)
> is how to handle devices which may have long time (e.g. due to
> draining outstanding requests, even w/o vPRI) to enter the STOP
> state. that time is not as deterministic as pending bytes thus cannot
> be reported back to the user before the operation is actually done.
> 
> Similarly to what we discussed for vPRI an eventfd will be beneficial
> so the user can timeout-wait on it, but it also needs an arc to create
> the eventfd between RUNNING->STOP...
> 

type too fast. it doesn’t need a new arc. Just a new capability to say
that STOP returns an event fd for the user to wait for completion,
when supporting such devices is required. 😊

Thanks
Kevin