Re: [PATCH v6 11/20] vfio/mdev: idxd: Add basic driver setup for idxd mdev

[Jason Gunthorpe <jgg@nvidia.com>]

On Fri, May 21, 2021 at 05:20:13PM -0700, Dave Jiang wrote:

> +int idxd_mdev_host_init(struct idxd_device *idxd, struct mdev_driver *drv)
> +{
> +	struct device *dev = &idxd->pdev->dev;
> +	int rc;
> +
> +	if (!idxd->ims_size)
> +		return -EOPNOTSUPP;
> +
> +	rc = iommu_dev_enable_feature(dev, IOMMU_DEV_FEAT_AUX);
> +	if (rc < 0) {
> +		dev_warn(dev, "Failed to enable aux-domain: %d\n", rc);
> +		return rc;
> +	}
> +
> +	rc = mdev_register_device(dev, drv);
> +	if (rc < 0) {
> +		iommu_dev_disable_feature(dev, IOMMU_DEV_FEAT_AUX);
> +		return rc;
> +	}

Don't call mdev_register_device from drivers/dma/idxd/init.c - vfio
stuff all belongs under drivers/vfio.

> +void idxd_mdev_host_release(struct kref *kref)
> +{
> +	struct idxd_device *idxd = container_of(kref, struct idxd_device, mdev_kref);
> +	struct device *dev = &idxd->pdev->dev;
> +
> +	mdev_unregister_device(dev);
> +	iommu_dev_disable_feature(dev, IOMMU_DEV_FEAT_AUX);
> +}
> +EXPORT_SYMBOL_GPL(idxd_mdev_host_release);
> +
>  static int idxd_setup_interrupts(struct idxd_device *idxd)
>  {
>  	struct pci_dev *pdev = idxd->pdev;
> @@ -352,6 +387,9 @@ static int idxd_setup_internals(struct idxd_device *idxd)
>  		goto err_wkq_create;
>  	}
>  
> +	kref_init(&idxd->mdev_kref);
> +	mutex_init(&idxd->kref_lock);
> +
>  	return 0;
>  
>   err_wkq_create:
> @@ -741,6 +779,7 @@ static void idxd_remove(struct pci_dev *pdev)
>  
>  	dev_dbg(&pdev->dev, "%s called\n", __func__);
>  	idxd_shutdown(pdev);
> +	kref_put_mutex(&idxd->mdev_kref, idxd_mdev_host_release, &idxd->kref_lock);

I didn't look closely at why this is like this, but please try to
avoid kref_put_mutex(), it should only be needed in exceptional
cases and this shouldn't be exceptional.

If you need to lock a kref before using it, it isn't a kref anymore,
just use an 'int'.

Especially since the kref is calling mdev_unregister_device(),
something is really upside down to motivate refcounting that.

Jason