RE: [PATCH v7 09/10] hisi_acc_vfio_pci: Add support for VFIO live migration

[Shameerali Kolothum Thodi <shameerali.kolothum.thodi@huawei.com>]

> -----Original Message-----
> From: Jason Gunthorpe [mailto:jgg@nvidia.com]
> Sent: 03 March 2022 00:22
> To: Shameerali Kolothum Thodi <shameerali.kolothum.thodi@huawei.com>
> Cc: kvm@vger.kernel.org; linux-kernel@vger.kernel.org;
> linux-crypto@vger.kernel.org; linux-pci@vger.kernel.org;
> alex.williamson@redhat.com; cohuck@redhat.com; mgurtovoy@nvidia.com;
> yishaih@nvidia.com; Linuxarm <linuxarm@huawei.com>; liulongfang
> <liulongfang@huawei.com>; Zengtao (B) <prime.zeng@hisilicon.com>;
> Jonathan Cameron <jonathan.cameron@huawei.com>; Wangzhou (B)
> <wangzhou1@hisilicon.com>
> Subject: Re: [PATCH v7 09/10] hisi_acc_vfio_pci: Add support for VFIO live
> migration
> 
> On Wed, Mar 02, 2022 at 05:29:02PM +0000, Shameer Kolothum wrote:
> > +static long hisi_acc_vf_save_unl_ioctl(struct file *filp,
> > +				       unsigned int cmd, unsigned long arg)
> > +{
> > +	struct hisi_acc_vf_migration_file *migf = filp->private_data;
> > +	struct hisi_acc_vf_core_device *hisi_acc_vdev = container_of(migf,
> > +			struct hisi_acc_vf_core_device, saving_migf);
> > +	loff_t *pos = &filp->f_pos;
> > +	struct vfio_precopy_info info;
> > +	unsigned long minsz;
> > +	int ret;
> > +
> > +	if (cmd != VFIO_MIG_GET_PRECOPY_INFO)
> > +		return -ENOTTY;
> > +
> > +	minsz = offsetofend(struct vfio_precopy_info, dirty_bytes);
> > +
> > +	if (copy_from_user(&info, (void __user *)arg, minsz))
> > +		return -EFAULT;
> > +	if (info.argsz < minsz)
> > +		return -EINVAL;
> > +
> > +	mutex_lock(&hisi_acc_vdev->state_mutex);
> > +	if (hisi_acc_vdev->mig_state != VFIO_DEVICE_STATE_PRE_COPY) {
> > +		mutex_unlock(&hisi_acc_vdev->state_mutex);
> > +		return -EINVAL;
> > +	}
> 
> IMHO it is easier just to check the total_length and not grab this
> other lock

The problem with checking the total_length here is that it is possible that
in STOP_COPY the dev is not ready and there are no more data to be transferred 
and the total_length remains at QM_MATCH_SIZE.

This just reminded me that the -ENOMSG setting logic in save_read() is
wrong now as it uses only the total_length to determine the PRE_COPY state. 

I think either we need to get the curr state info at both places or in STOP_COPY,
if there are no additional data, set the total_length = 0 and handle it in save_read().

Looks like setting the total_length = 0 in STOP_COPY is a better solution(If there are
no other issues with that) as it will avoid grabbing the state_mutex as you
mentioned above.

> > +struct acc_vf_data {
> > +#define QM_MATCH_SIZE 32L
> 
> This should be
> 
> #define QM_MATCH_SIZE offsetofend(struct acc_vf_data, qm_rsv_state)

Ok.

> > +	/* QM match information */
> 
> You should probably put an 8 byte random magic number here just to
> make the compatibility more unique.

Ok. Will add one.

> > +	u32 qp_num;
> > +	u32 dev_id;
> > +	u32 que_iso_cfg;
> > +	u32 qp_base;
> > +	/* QM reserved match information */
> > +	u32 qm_rsv_state[4];
> > +
> > +	/* QM RW regs */
> > +	u32 aeq_int_mask;
> > +	u32 eq_int_mask;
> > +	u32 ifc_int_source;
> > +	u32 ifc_int_mask;
> > +	u32 ifc_int_set;
> > +	u32 page_size;
> > +
> > +	/* QM_EQC_DW has 7 regs */
> > +	u32 qm_eqc_dw[7];
> > +
> > +	/* QM_AEQC_DW has 7 regs */
> > +	u32 qm_aeqc_dw[7];
> > +
> > +	/* QM reserved 5 regs */
> > +	u32 qm_rsv_regs[5];
> > +
> > +	/* qm memory init information */
> > +	u64 eqe_dma;
> 
> Am I counting wrong or is there a padding before this? 7+7+5 is not a multiple
> of 2. Be explicit about padding in a structure like this.

That's right. It needs padding before 'eqe_dma'.

Thanks,
Shameer