RE: [PATCH v10 vfio 4/7] vfio/pds: Add VFIO live migration support

From: "Tian, Kevin" <kevin.tian@intel.com>
To: Jason Gunthorpe <jgg@nvidia.com>
Cc: Brett Creeley <brett.creeley@amd.com>,
	"kvm@vger.kernel.org" <kvm@vger.kernel.org>,
	"netdev@vger.kernel.org" <netdev@vger.kernel.org>,
	"alex.williamson@redhat.com" <alex.williamson@redhat.com>,
	"yishaih@nvidia.com" <yishaih@nvidia.com>,
	"shameerali.kolothum.thodi@huawei.com"
	<shameerali.kolothum.thodi@huawei.com>,
	"shannon.nelson@amd.com" <shannon.nelson@amd.com>
Subject: RE: [PATCH v10 vfio 4/7] vfio/pds: Add VFIO live migration support
Date: Wed, 21 Jun 2023 06:49:12 +0000	[thread overview]
Message-ID: <BN9PR11MB52763F3D4F18DAB867D146458C5DA@BN9PR11MB5276.namprd11.prod.outlook.com> (raw)
In-Reply-To: <ZJGcCF2hBGERGUBZ@nvidia.com>

> From: Jason Gunthorpe <jgg@nvidia.com>
> Sent: Tuesday, June 20, 2023 8:31 PM
> 
> On Tue, Jun 20, 2023 at 02:02:44AM +0000, Tian, Kevin wrote:
> > > From: Jason Gunthorpe <jgg@nvidia.com>
> > > Sent: Monday, June 19, 2023 8:47 PM
> > >
> > > On Fri, Jun 16, 2023 at 08:06:21AM +0000, Tian, Kevin wrote:
> > >
> > > > Ideally the VMM has an estimation how long a VM can be paused based
> on
> > > > SLA, to-be-migrated state size, available network bandwidth, etc. and
> that
> > > > hint should be passed to the kernel so any state transition which may
> > > violate
> > > > that expectation can fail quickly to break the migration process and put
> the
> > > > VM back to the running state.
> > > >
> > > > Jason/Shameer, is there similar concern in mlx/hisilicon drivers?
> > >
> > > It is handled through the vfio_device_feature_mig_data_size mechanism..
> >
> > that is only for estimation of copied data.
> >
> > IMHO the stop time when the VM is paused includes both the time of
> > stopping the device and the time of migrating the VM state.
> >
> > For a software-emulated device the time of stopping the device is negligible.
> >
> > But certainly for assigned device the worst-case hard-coded 5s timeout as
> > done in this patch will kill whatever reasonable 'VM dead time' SLA (usually
> > in milliseconds) which CSPs try to meet purely based on the size of copied
> > data.
> 
> There is not alot that can be done here, the stop time cannot be
> predicted in advance on these devices - the system relies on the
> device having a reasonable time window.

What is the criteria for 'reasonable'? How does CSPs judge that such
device can guarantee a *reliable* reasonable window so live migration
can be enabled in the production environment?

I'm afraid that we are hiding a non-deterministic factor in current protocol.

Looking at mlx5 case which has a even larger timeout:

	 [MLX5_TO_CMD_MS] = 60000,

> 
> > Wouldn't a user-specified stop-device timeout be required to at least allow
> > breaking migration early according to the desired SLA?
> 
> Not really, the device is going to still execute the stop regardless
> of the timeout, and when it does the VM will be broken.
> 
> With a FW approach like this it is pretty stuck, we need the FW to
> remain in sync as the highest priority.

This makes some sense.

But still I don't think it's a good situation where the user has ZERO
knowledge about the non-negligible time in the stopping path...

> 
> > > We want new devices to get their architecture right, they need to
> > > support P2P. Didn't we talk about this already and Brett was going to
> > > fix it?
> >
> > Looks it's not fixed since RUNNING_P2P->STOP is a nop in this patch.
> 
> That could be OK, it needs a comment explaining why it is OK
> 

Yes, a comment is welcomed. having RUNNING_P2P->STOP as nop
kind of suggest that the device has been fully stopped in RUNNING_P2P
to meet the definition of the STOP state. But then it violates the
definition of RUNNING_P2P.