Re: [PATCH v10 vfio 4/7] vfio/pds: Add VFIO live migration support

From: Jason Gunthorpe <jgg@nvidia.com>
To: "Tian, Kevin" <kevin.tian@intel.com>
Cc: Brett Creeley <brett.creeley@amd.com>,
	"kvm@vger.kernel.org" <kvm@vger.kernel.org>,
	"netdev@vger.kernel.org" <netdev@vger.kernel.org>,
	"alex.williamson@redhat.com" <alex.williamson@redhat.com>,
	"yishaih@nvidia.com" <yishaih@nvidia.com>,
	"shameerali.kolothum.thodi@huawei.com"
	<shameerali.kolothum.thodi@huawei.com>,
	"shannon.nelson@amd.com" <shannon.nelson@amd.com>
Subject: Re: [PATCH v10 vfio 4/7] vfio/pds: Add VFIO live migration support
Date: Mon, 19 Jun 2023 09:46:46 -0300	[thread overview]
Message-ID: <ZJBONrx5LOgpTr1U@nvidia.com> (raw)
In-Reply-To: <BN9PR11MB5276511543775B852AD1C5A88C58A@BN9PR11MB5276.namprd11.prod.outlook.com>

On Fri, Jun 16, 2023 at 08:06:21AM +0000, Tian, Kevin wrote:

> Ideally the VMM has an estimation how long a VM can be paused based on
> SLA, to-be-migrated state size, available network bandwidth, etc. and that
> hint should be passed to the kernel so any state transition which may violate
> that expectation can fail quickly to break the migration process and put the
> VM back to the running state.
> 
> Jason/Shameer, is there similar concern in mlx/hisilicon drivers? 

It is handled through the vfio_device_feature_mig_data_size mechanism..

> > +	if (cur == VFIO_DEVICE_STATE_RUNNING_P2P && next ==
> > VFIO_DEVICE_STATE_STOP)
> > +		return NULL;
> 
> I'm not sure whether P2P is actually supported here. By definition
> P2P means the device is stopped but still responds to p2p request
> from other devices. If you look at mlx example it uses different
> cmds between RUNNING->RUNNING_P2P and RUNNING_P2P->STOP.
> 
> But in your case seems you simply move what is required in STOP
> into P2P. Probably you can just remove the support of P2P like
> hisilicon does.

We want new devices to get their architecture right, they need to
support P2P. Didn't we talk about this already and Brett was going to
fix it?

Jason