Re: [PATCH 2/7] kvm/vfio: detect assigned device via irqbypass manager

Re: [PATCH 2/7] kvm/vfio: detect assigned device via irqbypass manager

[Alex Williamson]

On Sun, 12 Jul 2020 22:49:21 +0800
Zhu Lingshan <lingshan.zhu@intel.com> wrote:

> We used to detect assigned device via VFIO manipulated device
> conters. This is less flexible consider VFIO is not the only
> interface for assigned device. vDPA devices has dedicated
> backed hardware as well. So this patch tries to detect
> the assigned device via irqbypass manager.
> 
> We will increase/decrease the assigned device counter in kvm/x86.
> Both vDPA and VFIO would go through this code path.
> 
> This code path only affect x86 for now.

No it doesn't, it only adds VFIO support to x86, but it removes it from
architecture neutral code.  Also a VFIO device does not necessarily
make use of the irqbypass manager, this depends on platform support and
enablement of this feature.   Therefore, NAK.  Thanks,

Alex
 
> Signed-off-by: Zhu Lingshan <lingshan.zhu@intel.com>
> ---
>  arch/x86/kvm/x86.c | 10 ++++++++--
>  virt/kvm/vfio.c    |  2 --
>  2 files changed, 8 insertions(+), 4 deletions(-)
> 
> diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
> index 00c88c2..20c07d3 100644
> --- a/arch/x86/kvm/x86.c
> +++ b/arch/x86/kvm/x86.c
> @@ -10624,11 +10624,17 @@ int kvm_arch_irq_bypass_add_producer(struct irq_bypass_consumer *cons,
>  {
>  	struct kvm_kernel_irqfd *irqfd =
>  		container_of(cons, struct kvm_kernel_irqfd, consumer);
> +	int ret;
>  
>  	irqfd->producer = prod;
> +	kvm_arch_start_assignment(irqfd->kvm);
> +	ret = kvm_x86_ops.update_pi_irte(irqfd->kvm,
> +					 prod->irq, irqfd->gsi, 1);
> +
> +	if (ret)
> +		kvm_arch_end_assignment(irqfd->kvm);
>  
> -	return kvm_x86_ops.update_pi_irte(irqfd->kvm,
> -					   prod->irq, irqfd->gsi, 1);
> +	return ret;
>  }
>  
>  void kvm_arch_irq_bypass_del_producer(struct irq_bypass_consumer *cons,
> diff --git a/virt/kvm/vfio.c b/virt/kvm/vfio.c
> index 8fcbc50..111da52 100644
> --- a/virt/kvm/vfio.c
> +++ b/virt/kvm/vfio.c
> @@ -226,7 +226,6 @@ static int kvm_vfio_set_group(struct kvm_device *dev, long attr, u64 arg)
>  		list_add_tail(&kvg->node, &kv->group_list);
>  		kvg->vfio_group = vfio_group;
>  
> -		kvm_arch_start_assignment(dev->kvm);
>  
>  		mutex_unlock(&kv->lock);
>  
> @@ -254,7 +253,6 @@ static int kvm_vfio_set_group(struct kvm_device *dev, long attr, u64 arg)
>  				continue;
>  
>  			list_del(&kvg->node);
> -			kvm_arch_end_assignment(dev->kvm);
>  #ifdef CONFIG_SPAPR_TCE_IOMMU
>  			kvm_spapr_tce_release_vfio_group(dev->kvm,
>  							 kvg->vfio_group);

Re: [PATCH 2/7] kvm/vfio: detect assigned device via irqbypass manager

[Michael S. Tsirkin]

On Sun, Jul 12, 2020 at 10:49:21PM +0800, Zhu Lingshan wrote:
> We used to detect assigned device via VFIO manipulated device
> conters. This is less flexible consider VFIO is not the only
> interface for assigned device. vDPA devices has dedicated
> backed hardware as well. So this patch tries to detect
> the assigned device via irqbypass manager.
> 
> We will increase/decrease the assigned device counter in kvm/x86.
> Both vDPA and VFIO would go through this code path.
> 
> This code path only affect x86 for now.
> 
> Signed-off-by: Zhu Lingshan <lingshan.zhu@intel.com>


I think it's best to leave VFIO alone. Add appropriate APIs for VDPA,
worry about converting existing users later.

> ---
>  arch/x86/kvm/x86.c | 10 ++++++++--
>  virt/kvm/vfio.c    |  2 --
>  2 files changed, 8 insertions(+), 4 deletions(-)
> 
> diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
> index 00c88c2..20c07d3 100644
> --- a/arch/x86/kvm/x86.c
> +++ b/arch/x86/kvm/x86.c
> @@ -10624,11 +10624,17 @@ int kvm_arch_irq_bypass_add_producer(struct irq_bypass_consumer *cons,
>  {
>  	struct kvm_kernel_irqfd *irqfd =
>  		container_of(cons, struct kvm_kernel_irqfd, consumer);
> +	int ret;
>  
>  	irqfd->producer = prod;
> +	kvm_arch_start_assignment(irqfd->kvm);
> +	ret = kvm_x86_ops.update_pi_irte(irqfd->kvm,
> +					 prod->irq, irqfd->gsi, 1);
> +
> +	if (ret)
> +		kvm_arch_end_assignment(irqfd->kvm);
>  
> -	return kvm_x86_ops.update_pi_irte(irqfd->kvm,
> -					   prod->irq, irqfd->gsi, 1);
> +	return ret;
>  }
>  
>  void kvm_arch_irq_bypass_del_producer(struct irq_bypass_consumer *cons,
> diff --git a/virt/kvm/vfio.c b/virt/kvm/vfio.c
> index 8fcbc50..111da52 100644
> --- a/virt/kvm/vfio.c
> +++ b/virt/kvm/vfio.c
> @@ -226,7 +226,6 @@ static int kvm_vfio_set_group(struct kvm_device *dev, long attr, u64 arg)
>  		list_add_tail(&kvg->node, &kv->group_list);
>  		kvg->vfio_group = vfio_group;
>  
> -		kvm_arch_start_assignment(dev->kvm);
>  
>  		mutex_unlock(&kv->lock);
>  
> @@ -254,7 +253,6 @@ static int kvm_vfio_set_group(struct kvm_device *dev, long attr, u64 arg)
>  				continue;
>  
>  			list_del(&kvg->node);
> -			kvm_arch_end_assignment(dev->kvm);
>  #ifdef CONFIG_SPAPR_TCE_IOMMU
>  			kvm_spapr_tce_release_vfio_group(dev->kvm,
>  							 kvg->vfio_group);
> -- 
> 1.8.3.1

Re: [PATCH 2/7] kvm/vfio: detect assigned device via irqbypass manager

[Jason Wang]

On 2020/7/12 下午11:29, Alex Williamson wrote:
> On Sun, 12 Jul 2020 22:49:21 +0800
> Zhu Lingshan <lingshan.zhu@intel.com> wrote:
>
>> We used to detect assigned device via VFIO manipulated device
>> conters. This is less flexible consider VFIO is not the only
>> interface for assigned device. vDPA devices has dedicated
>> backed hardware as well. So this patch tries to detect
>> the assigned device via irqbypass manager.
>>
>> We will increase/decrease the assigned device counter in kvm/x86.
>> Both vDPA and VFIO would go through this code path.
>>
>> This code path only affect x86 for now.
> No it doesn't, it only adds VFIO support to x86, but it removes it from
> architecture neutral code.


Do you mean we should introduce a kvm_irq_bypass_add_producer and do 
kvm_arch_start_assignment( ) there?


> Also a VFIO device does not necessarily
> make use of the irqbypass manager, this depends on platform support and
> enablement of this feature.


Yes, we should keep the VFIO part unchanged.

Thanks


>    Therefore, NAK.  Thanks,
>
> Alex
>   
>> Signed-off-by: Zhu Lingshan <lingshan.zhu@intel.com>
>> ---
>>   arch/x86/kvm/x86.c | 10 ++++++++--
>>   virt/kvm/vfio.c    |  2 --
>>   2 files changed, 8 insertions(+), 4 deletions(-)
>>
>> diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
>> index 00c88c2..20c07d3 100644
>> --- a/arch/x86/kvm/x86.c
>> +++ b/arch/x86/kvm/x86.c
>> @@ -10624,11 +10624,17 @@ int kvm_arch_irq_bypass_add_producer(struct irq_bypass_consumer *cons,
>>   {
>>   	struct kvm_kernel_irqfd *irqfd =
>>   		container_of(cons, struct kvm_kernel_irqfd, consumer);
>> +	int ret;
>>   
>>   	irqfd->producer = prod;
>> +	kvm_arch_start_assignment(irqfd->kvm);
>> +	ret = kvm_x86_ops.update_pi_irte(irqfd->kvm,
>> +					 prod->irq, irqfd->gsi, 1);
>> +
>> +	if (ret)
>> +		kvm_arch_end_assignment(irqfd->kvm);
>>   
>> -	return kvm_x86_ops.update_pi_irte(irqfd->kvm,
>> -					   prod->irq, irqfd->gsi, 1);
>> +	return ret;
>>   }
>>   
>>   void kvm_arch_irq_bypass_del_producer(struct irq_bypass_consumer *cons,
>> diff --git a/virt/kvm/vfio.c b/virt/kvm/vfio.c
>> index 8fcbc50..111da52 100644
>> --- a/virt/kvm/vfio.c
>> +++ b/virt/kvm/vfio.c
>> @@ -226,7 +226,6 @@ static int kvm_vfio_set_group(struct kvm_device *dev, long attr, u64 arg)
>>   		list_add_tail(&kvg->node, &kv->group_list);
>>   		kvg->vfio_group = vfio_group;
>>   
>> -		kvm_arch_start_assignment(dev->kvm);
>>   
>>   		mutex_unlock(&kv->lock);
>>   
>> @@ -254,7 +253,6 @@ static int kvm_vfio_set_group(struct kvm_device *dev, long attr, u64 arg)
>>   				continue;
>>   
>>   			list_del(&kvg->node);
>> -			kvm_arch_end_assignment(dev->kvm);
>>   #ifdef CONFIG_SPAPR_TCE_IOMMU
>>   			kvm_spapr_tce_release_vfio_group(dev->kvm,
>>   							 kvg->vfio_group);

Re: [PATCH 2/7] kvm/vfio: detect assigned device via irqbypass manager

[Jason Wang]

On 2020/7/13 上午5:06, Michael S. Tsirkin wrote:
> On Sun, Jul 12, 2020 at 10:49:21PM +0800, Zhu Lingshan wrote:
>> We used to detect assigned device via VFIO manipulated device
>> conters. This is less flexible consider VFIO is not the only
>> interface for assigned device. vDPA devices has dedicated
>> backed hardware as well. So this patch tries to detect
>> the assigned device via irqbypass manager.
>>
>> We will increase/decrease the assigned device counter in kvm/x86.
>> Both vDPA and VFIO would go through this code path.
>>
>> This code path only affect x86 for now.
>>
>> Signed-off-by: Zhu Lingshan <lingshan.zhu@intel.com>
>
> I think it's best to leave VFIO alone. Add appropriate APIs for VDPA,
> worry about converting existing users later.



Just to make sure I understand, did you mean:

1) introduce another bridge for vDPA

or

2) only detect vDPA via bypass manager? (we can leave VFIO code as is, 
then the assigned device counter may increase/decrease twice if VFIO use 
irq bypass manager which should be no harm).

Thanks


>
>> ---
>>   arch/x86/kvm/x86.c | 10 ++++++++--
>>   virt/kvm/vfio.c    |  2 --
>>   2 files changed, 8 insertions(+), 4 deletions(-)
>>
>> diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
>> index 00c88c2..20c07d3 100644
>> --- a/arch/x86/kvm/x86.c
>> +++ b/arch/x86/kvm/x86.c
>> @@ -10624,11 +10624,17 @@ int kvm_arch_irq_bypass_add_producer(struct irq_bypass_consumer *cons,
>>   {
>>   	struct kvm_kernel_irqfd *irqfd =
>>   		container_of(cons, struct kvm_kernel_irqfd, consumer);
>> +	int ret;
>>   
>>   	irqfd->producer = prod;
>> +	kvm_arch_start_assignment(irqfd->kvm);
>> +	ret = kvm_x86_ops.update_pi_irte(irqfd->kvm,
>> +					 prod->irq, irqfd->gsi, 1);
>> +
>> +	if (ret)
>> +		kvm_arch_end_assignment(irqfd->kvm);
>>   
>> -	return kvm_x86_ops.update_pi_irte(irqfd->kvm,
>> -					   prod->irq, irqfd->gsi, 1);
>> +	return ret;
>>   }
>>   
>>   void kvm_arch_irq_bypass_del_producer(struct irq_bypass_consumer *cons,
>> diff --git a/virt/kvm/vfio.c b/virt/kvm/vfio.c
>> index 8fcbc50..111da52 100644
>> --- a/virt/kvm/vfio.c
>> +++ b/virt/kvm/vfio.c
>> @@ -226,7 +226,6 @@ static int kvm_vfio_set_group(struct kvm_device *dev, long attr, u64 arg)
>>   		list_add_tail(&kvg->node, &kv->group_list);
>>   		kvg->vfio_group = vfio_group;
>>   
>> -		kvm_arch_start_assignment(dev->kvm);
>>   
>>   		mutex_unlock(&kv->lock);
>>   
>> @@ -254,7 +253,6 @@ static int kvm_vfio_set_group(struct kvm_device *dev, long attr, u64 arg)
>>   				continue;
>>   
>>   			list_del(&kvg->node);
>> -			kvm_arch_end_assignment(dev->kvm);
>>   #ifdef CONFIG_SPAPR_TCE_IOMMU
>>   			kvm_spapr_tce_release_vfio_group(dev->kvm,
>>   							 kvg->vfio_group);
>> -- 
>> 1.8.3.1

Re: [PATCH 3/7] vhost_vdpa: implement IRQ offloading functions in vhost_vdpa

[Jason Wang]

On 2020/7/12 下午10:49, Zhu Lingshan wrote:
> This patch introduce a set of functions for setup/unsetup
> and update irq offloading respectively by register/unregister
> and re-register the irq_bypass_producer.
>
> Signed-off-by: Zhu Lingshan <lingshan.zhu@intel.com>
> ---
>   drivers/vhost/vdpa.c | 69 ++++++++++++++++++++++++++++++++++++++++++++++++++++
>   1 file changed, 69 insertions(+)
>
> diff --git a/drivers/vhost/vdpa.c b/drivers/vhost/vdpa.c
> index 2fcc422..92683e4 100644
> --- a/drivers/vhost/vdpa.c
> +++ b/drivers/vhost/vdpa.c
> @@ -115,6 +115,63 @@ static irqreturn_t vhost_vdpa_config_cb(void *private)
>   	return IRQ_HANDLED;
>   }
>   
> +static void vhost_vdpa_setup_vq_irq(struct vdpa_device *dev, int qid, int irq)
> +{
> +	struct vhost_vdpa *v = vdpa_get_drvdata(dev);
> +	struct vhost_virtqueue *vq = &v->vqs[qid];
> +	int ret;
> +
> +	vq_err(vq, "setup irq bypass for vq %d with irq = %d\n", qid, irq);
> +	spin_lock(&vq->call_ctx.ctx_lock);
> +	if (!vq->call_ctx.ctx)
> +		return;
> +
> +	vq->call_ctx.producer.token = vq->call_ctx.ctx;
> +	vq->call_ctx.producer.irq = irq;
> +	ret = irq_bypass_register_producer(&vq->call_ctx.producer);
> +	spin_unlock(&vq->call_ctx.ctx_lock);
> +
> +	if (unlikely(ret))
> +		vq_err(vq,
> +		"irq bypass producer (token %p registration fails: %d\n",
> +		vq->call_ctx.producer.token, ret);


Not sure this deserves a vq_err(), irq will be relayed through eventfd 
if irq bypass manager can't work.


> +}
> +
> +static void vhost_vdpa_unsetup_vq_irq(struct vdpa_device *dev, int qid)
> +{
> +	struct vhost_vdpa *v = vdpa_get_drvdata(dev);
> +	struct vhost_virtqueue *vq = &v->vqs[qid];
> +
> +	spin_lock(&vq->call_ctx.ctx_lock);
> +	irq_bypass_unregister_producer(&vq->call_ctx.producer);
> +	spin_unlock(&vq->call_ctx.ctx_lock);
> +
> +	vq_err(vq, "unsetup irq bypass for vq %d\n", qid);


Why call vq_err() here?


> +}
> +
> +static void vhost_vdpa_update_vq_irq(struct vhost_virtqueue *vq)
> +{
> +	struct eventfd_ctx *ctx;
> +	void *token;
> +
> +	spin_lock(&vq->call_ctx.ctx_lock);
> +	ctx = vq->call_ctx.ctx;
> +	token = vq->call_ctx.producer.token;
> +	if (ctx == token)
> +		return;


Need do unlock here.


> +
> +	if (!ctx && token)
> +		irq_bypass_unregister_producer(&vq->call_ctx.producer);
> +
> +	if (ctx && ctx != token) {
> +		irq_bypass_unregister_producer(&vq->call_ctx.producer);
> +		vq->call_ctx.producer.token = ctx;
> +		irq_bypass_register_producer(&vq->call_ctx.producer);
> +	}
> +
> +	spin_unlock(&vq->call_ctx.ctx_lock);


This should be rare so I'd use simple codes just do unregister and register.


> +}
> +
>   static void vhost_vdpa_reset(struct vhost_vdpa *v)
>   {
>   	struct vdpa_device *vdpa = v->vdpa;
> @@ -332,6 +389,7 @@ static long vhost_vdpa_set_config_call(struct vhost_vdpa *v, u32 __user *argp)
>   
>   	return 0;
>   }
> +


Unnecessary change.


>   static long vhost_vdpa_vring_ioctl(struct vhost_vdpa *v, unsigned int cmd,
>   				   void __user *argp)
>   {
> @@ -390,6 +448,16 @@ static long vhost_vdpa_vring_ioctl(struct vhost_vdpa *v, unsigned int cmd,
>   			cb.private = NULL;
>   		}
>   		ops->set_vq_cb(vdpa, idx, &cb);
> +#ifdef CONFIG_HAVE_KVM_IRQ_BYPASS
> +		/*
> +		 * if it has a non-zero irq, means there is a
> +		 * previsouly registered irq_bypass_producer,
> +		 * we should update it when ctx (its token)
> +		 * changes.
> +		 */
> +		if (vq->call_ctx.producer.irq)
> +			vhost_vdpa_update_vq_irq(vq);
> +#endif
>   		break;
>   
>   	case VHOST_SET_VRING_NUM:
> @@ -741,6 +809,7 @@ static int vhost_vdpa_open(struct inode *inode, struct file *filep)
>   		vqs[i] = &v->vqs[i];
>   		vqs[i]->handle_kick = handle_vq_kick;
>   	}
> +


Unnecessary change.

Thanks


>   	vhost_dev_init(dev, vqs, nvqs, 0, 0, 0, false,
>   		       vhost_vdpa_process_iotlb_msg);
>   

Re: [PATCH 4/7] vDPA: implement IRQ offloading helpers in vDPA core

[Jason Wang]

On 2020/7/12 下午10:49, Zhu Lingshan wrote:
> This commit implements IRQ offloading helpers in vDPA core by
> introducing two couple of functions:
>
> vdpa_alloc_vq_irq() and vdpa_free_vq_irq(): request irq and free
> irq, will setup irq offloading if irq_bypass is enabled.
>
> vdpa_setup_irq() and vdpa_unsetup_irq(): supportive functions,
> will call vhost_vdpa helpers.
>
> Signed-off-by: Zhu Lingshan <lingshan.zhu@intel.com>


This patch should go before patch 3.


> ---
>   drivers/vdpa/vdpa.c   | 46 ++++++++++++++++++++++++++++++++++++++++++++++
>   drivers/vhost/Kconfig |  1 +
>   drivers/vhost/vdpa.c  |  2 ++
>   include/linux/vdpa.h  | 11 +++++++++++
>   4 files changed, 60 insertions(+)
>
> diff --git a/drivers/vdpa/vdpa.c b/drivers/vdpa/vdpa.c
> index ff6562f..d8eba01 100644
> --- a/drivers/vdpa/vdpa.c
> +++ b/drivers/vdpa/vdpa.c
> @@ -163,6 +163,52 @@ void vdpa_unregister_driver(struct vdpa_driver *drv)
>   }
>   EXPORT_SYMBOL_GPL(vdpa_unregister_driver);
>   
> +static void vdpa_setup_irq(struct vdpa_device *vdev, int qid, int irq)
> +{
> +	struct vdpa_driver *drv = drv_to_vdpa(vdev->dev.driver);
> +
> +#ifdef CONFIG_HAVE_KVM_IRQ_BYPASS


Let's don't do the check here. It's the responsibility of driver to 
decide what it should do.


> +	if (drv->setup_vq_irq)
> +		drv->setup_vq_irq(vdev, qid, irq);
> +#endif
> +}
> +
> +static void vdpa_unsetup_irq(struct vdpa_device *vdev, int qid)
> +{
> +	struct vdpa_driver *drv = drv_to_vdpa(vdev->dev.driver);
> +
> +#ifdef CONFIG_HAVE_KVM_IRQ_BYPASS
> +	if (drv->unsetup_vq_irq)
> +		drv->unsetup_vq_irq(vdev, qid);
> +#endif
> +}
> +
> +int vdpa_alloc_vq_irq(struct device *dev, struct vdpa_device *vdev,
> +		      unsigned int irq, irq_handler_t handler,
> +		      unsigned long irqflags, const char *devname, void *dev_id,
> +		      int qid)
> +{
> +	int ret;
> +
> +	ret = devm_request_irq(dev, irq, handler, irqflags, devname, dev_id);
> +	if (ret)
> +		dev_err(dev, "Failed to request irq for vq %d\n", qid);
> +	else
> +		vdpa_setup_irq(vdev, qid, irq);


I'd like to squash the vdpa_setup_irq logic here.


> +
> +	return ret;
> +
> +}
> +EXPORT_SYMBOL_GPL(vdpa_alloc_vq_irq);
> +
> +void vdpa_free_vq_irq(struct device *dev, struct vdpa_device *vdev, int irq,
> +			 int qid, void *dev_id)
> +{
> +	devm_free_irq(dev, irq, dev_id);
> +	vdpa_unsetup_irq(vdev, qid);
> +}
> +EXPORT_SYMBOL_GPL(vdpa_free_vq_irq);
> +
>   static int vdpa_init(void)
>   {
>   	return bus_register(&vdpa_bus);
> diff --git a/drivers/vhost/Kconfig b/drivers/vhost/Kconfig


Let squash the vhost patch into patch 3.


> index d3688c6..587fbae 100644
> --- a/drivers/vhost/Kconfig
> +++ b/drivers/vhost/Kconfig
> @@ -65,6 +65,7 @@ config VHOST_VDPA
>   	tristate "Vhost driver for vDPA-based backend"
>   	depends on EVENTFD
>   	select VHOST
> +	select IRQ_BYPASS_MANAGER
>   	depends on VDPA
>   	help
>   	  This kernel module can be loaded in host kernel to accelerate
> diff --git a/drivers/vhost/vdpa.c b/drivers/vhost/vdpa.c
> index 92683e4..6e25158 100644
> --- a/drivers/vhost/vdpa.c
> +++ b/drivers/vhost/vdpa.c
> @@ -1020,6 +1020,8 @@ static void vhost_vdpa_remove(struct vdpa_device *vdpa)
>   	},
>   	.probe	= vhost_vdpa_probe,
>   	.remove	= vhost_vdpa_remove,
> +	.setup_vq_irq = vhost_vdpa_setup_vq_irq,
> +	.unsetup_vq_irq = vhost_vdpa_unsetup_vq_irq,
>   };
>   
>   static int __init vhost_vdpa_init(void)
> diff --git a/include/linux/vdpa.h b/include/linux/vdpa.h
> index 239db79..9f9b245 100644
> --- a/include/linux/vdpa.h
> +++ b/include/linux/vdpa.h
> @@ -220,17 +220,28 @@ struct vdpa_device *__vdpa_alloc_device(struct device *parent,
>   
>   int vdpa_register_device(struct vdpa_device *vdev);
>   void vdpa_unregister_device(struct vdpa_device *vdev);
> +int vdpa_alloc_vq_irq(struct device *dev, struct vdpa_device *vdev,
> +		      unsigned int irq, irq_handler_t handler,
> +		      unsigned long irqflags, const char *devname, void *dev_id,
> +		      int qid);
> +void vdpa_free_vq_irq(struct device *dev, struct vdpa_device *vdev, int irq,
> +		      int qid, void *dev_id);


You need to document the devres implications of those two helpers.


> +
>   
>   /**
>    * vdpa_driver - operations for a vDPA driver
>    * @driver: underlying device driver
>    * @probe: the function to call when a device is found.  Returns 0 or -errno.
>    * @remove: the function to call when a device is removed.
> + * @setup_vq_irq: setup irq bypass for a vhost_vdpa vq.
> + * @unsetup_vq_irq: unsetup irq bypass for a vhost_vdpa vq.


Though irq bypass is used by vhost-vdpa, it's not necessarily to be true 
in the future. So it's better not to mention irqbypass in the doc here.

Thanks


>    */
>   struct vdpa_driver {
>   	struct device_driver driver;
>   	int (*probe)(struct vdpa_device *vdev);
>   	void (*remove)(struct vdpa_device *vdev);
> +	void (*setup_vq_irq)(struct vdpa_device *vdev, int qid, int irq);
> +	void (*unsetup_vq_irq)(struct vdpa_device *vdev, int qid);
>   };
>   
>   #define vdpa_register_driver(drv) \

Re: [PATCH 5/7] virtio_vdpa: init IRQ offloading function pointers to NULL.

[Jason Wang]

On 2020/7/12 下午10:49, Zhu Lingshan wrote:
> This commit initialize IRQ offloading function pointers in
> virtio_vdpa_driver to NULL. Becasue irq offloading only focus
> on VMs for vhost_vdpa.
>
> Signed-off-by: Zhu Lingshan <lingshan.zhu@intel.com>
> ---
>   drivers/virtio/virtio_vdpa.c | 2 ++
>   1 file changed, 2 insertions(+)
>
> diff --git a/drivers/virtio/virtio_vdpa.c b/drivers/virtio/virtio_vdpa.c
> index c30eb55..1e8acb9 100644
> --- a/drivers/virtio/virtio_vdpa.c
> +++ b/drivers/virtio/virtio_vdpa.c
> @@ -386,6 +386,8 @@ static void virtio_vdpa_remove(struct vdpa_device *vdpa)
>   	},
>   	.probe	= virtio_vdpa_probe,
>   	.remove = virtio_vdpa_remove,
> +	.setup_vq_irq = NULL,
> +	.unsetup_vq_irq = NULL,
>   };


Is this really needed consider the it's static?

Thanks


>   
>   module_vdpa_driver(virtio_vdpa_driver);

Re: [PATCH 6/7] ifcvf: replace irq_request/free with helpers in vDPA core.

[Jason Wang]

On 2020/7/12 下午10:49, Zhu Lingshan wrote:
> This commit replaced irq_request/free() with helpers in vDPA
> core, so that it can request/free irq and setup irq offloading
> on order.
>
> Signed-off-by: Zhu Lingshan <lingshan.zhu@intel.com>
> ---
>   drivers/vdpa/ifcvf/ifcvf_main.c | 11 ++++++-----
>   1 file changed, 6 insertions(+), 5 deletions(-)
>
> diff --git a/drivers/vdpa/ifcvf/ifcvf_main.c b/drivers/vdpa/ifcvf/ifcvf_main.c
> index f5a60c1..65b84e1 100644
> --- a/drivers/vdpa/ifcvf/ifcvf_main.c
> +++ b/drivers/vdpa/ifcvf/ifcvf_main.c
> @@ -47,11 +47,12 @@ static void ifcvf_free_irq(struct ifcvf_adapter *adapter, int queues)
>   {
>   	struct pci_dev *pdev = adapter->pdev;
>   	struct ifcvf_hw *vf = &adapter->vf;
> +	struct vdpa_device *vdpa = &adapter->vdpa;
>   	int i;
>   
>   
>   	for (i = 0; i < queues; i++)
> -		devm_free_irq(&pdev->dev, vf->vring[i].irq, &vf->vring[i]);
> +		vdpa_free_vq_irq(&pdev->dev, vdpa, vf->vring[i].irq, i, &vf->vring[i]);
>   
>   	ifcvf_free_irq_vectors(pdev);
>   }
> @@ -60,6 +61,7 @@ static int ifcvf_request_irq(struct ifcvf_adapter *adapter)
>   {
>   	struct pci_dev *pdev = adapter->pdev;
>   	struct ifcvf_hw *vf = &adapter->vf;
> +	struct vdpa_device *vdpa = &adapter->vdpa;
>   	int vector, i, ret, irq;
>   
>   	ret = pci_alloc_irq_vectors(pdev, IFCVF_MAX_INTR,
> @@ -73,6 +75,7 @@ static int ifcvf_request_irq(struct ifcvf_adapter *adapter)
>   		 pci_name(pdev));
>   	vector = 0;
>   	irq = pci_irq_vector(pdev, vector);
> +	/* config interrupt */


Unnecessary changes.

Thanks


>   	ret = devm_request_irq(&pdev->dev, irq,
>   			       ifcvf_config_changed, 0,
>   			       vf->config_msix_name, vf);
> @@ -82,13 +85,11 @@ static int ifcvf_request_irq(struct ifcvf_adapter *adapter)
>   			 pci_name(pdev), i);
>   		vector = i + IFCVF_MSI_QUEUE_OFF;
>   		irq = pci_irq_vector(pdev, vector);
> -		ret = devm_request_irq(&pdev->dev, irq,
> +		ret = vdpa_alloc_vq_irq(&pdev->dev, vdpa, irq,
>   				       ifcvf_intr_handler, 0,
>   				       vf->vring[i].msix_name,
> -				       &vf->vring[i]);
> +				       &vf->vring[i], i);
>   		if (ret) {
> -			IFCVF_ERR(pdev,
> -				  "Failed to request irq for vq %d\n", i);
>   			ifcvf_free_irq(adapter, i);
>   
>   			return ret;

Re: [PATCH 2/7] kvm/vfio: detect assigned device via irqbypass manager

[Michael S. Tsirkin]

On Mon, Jul 13, 2020 at 04:13:35PM +0800, Jason Wang wrote:
> 
> On 2020/7/13 上午5:06, Michael S. Tsirkin wrote:
> > On Sun, Jul 12, 2020 at 10:49:21PM +0800, Zhu Lingshan wrote:
> > > We used to detect assigned device via VFIO manipulated device
> > > conters. This is less flexible consider VFIO is not the only
> > > interface for assigned device. vDPA devices has dedicated
> > > backed hardware as well. So this patch tries to detect
> > > the assigned device via irqbypass manager.
> > > 
> > > We will increase/decrease the assigned device counter in kvm/x86.
> > > Both vDPA and VFIO would go through this code path.
> > > 
> > > This code path only affect x86 for now.
> > > 
> > > Signed-off-by: Zhu Lingshan <lingshan.zhu@intel.com>
> > 
> > I think it's best to leave VFIO alone. Add appropriate APIs for VDPA,
> > worry about converting existing users later.
> 
> 
> 
> Just to make sure I understand, did you mean:
> 
> 1) introduce another bridge for vDPA
> 
> or
> 
> 2) only detect vDPA via bypass manager? (we can leave VFIO code as is, then
> the assigned device counter may increase/decrease twice if VFIO use irq
> bypass manager which should be no harm).
> 
> Thanks

2 is probably easier to justify. 1 would depend on the specific bridge
proposed.

> 
> > 
> > > ---
> > >   arch/x86/kvm/x86.c | 10 ++++++++--
> > >   virt/kvm/vfio.c    |  2 --
> > >   2 files changed, 8 insertions(+), 4 deletions(-)
> > > 
> > > diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
> > > index 00c88c2..20c07d3 100644
> > > --- a/arch/x86/kvm/x86.c
> > > +++ b/arch/x86/kvm/x86.c
> > > @@ -10624,11 +10624,17 @@ int kvm_arch_irq_bypass_add_producer(struct irq_bypass_consumer *cons,
> > >   {
> > >   	struct kvm_kernel_irqfd *irqfd =
> > >   		container_of(cons, struct kvm_kernel_irqfd, consumer);
> > > +	int ret;
> > >   	irqfd->producer = prod;
> > > +	kvm_arch_start_assignment(irqfd->kvm);
> > > +	ret = kvm_x86_ops.update_pi_irte(irqfd->kvm,
> > > +					 prod->irq, irqfd->gsi, 1);
> > > +
> > > +	if (ret)
> > > +		kvm_arch_end_assignment(irqfd->kvm);
> > > -	return kvm_x86_ops.update_pi_irte(irqfd->kvm,
> > > -					   prod->irq, irqfd->gsi, 1);
> > > +	return ret;
> > >   }
> > >   void kvm_arch_irq_bypass_del_producer(struct irq_bypass_consumer *cons,
> > > diff --git a/virt/kvm/vfio.c b/virt/kvm/vfio.c
> > > index 8fcbc50..111da52 100644
> > > --- a/virt/kvm/vfio.c
> > > +++ b/virt/kvm/vfio.c
> > > @@ -226,7 +226,6 @@ static int kvm_vfio_set_group(struct kvm_device *dev, long attr, u64 arg)
> > >   		list_add_tail(&kvg->node, &kv->group_list);
> > >   		kvg->vfio_group = vfio_group;
> > > -		kvm_arch_start_assignment(dev->kvm);
> > >   		mutex_unlock(&kv->lock);
> > > @@ -254,7 +253,6 @@ static int kvm_vfio_set_group(struct kvm_device *dev, long attr, u64 arg)
> > >   				continue;
> > >   			list_del(&kvg->node);
> > > -			kvm_arch_end_assignment(dev->kvm);
> > >   #ifdef CONFIG_SPAPR_TCE_IOMMU
> > >   			kvm_spapr_tce_release_vfio_group(dev->kvm,
> > >   							 kvg->vfio_group);
> > > -- 
> > > 1.8.3.1

Re: [PATCH 2/7] kvm/vfio: detect assigned device via irqbypass manager

[Michael S. Tsirkin]

On Tue, Jul 14, 2020 at 08:52:43AM +0800, Zhu, Lingshan wrote:
> 
> On 7/13/2020 6:52 PM, Michael S. Tsirkin wrote:
> 
>     On Mon, Jul 13, 2020 at 04:13:35PM +0800, Jason Wang wrote:
> 
>         On 2020/7/13 上午5:06, Michael S. Tsirkin wrote:
> 
>             On Sun, Jul 12, 2020 at 10:49:21PM +0800, Zhu Lingshan wrote:
> 
>                 We used to detect assigned device via VFIO manipulated device
>                 conters. This is less flexible consider VFIO is not the only
>                 interface for assigned device. vDPA devices has dedicated
>                 backed hardware as well. So this patch tries to detect
>                 the assigned device via irqbypass manager.
> 
>                 We will increase/decrease the assigned device counter in kvm/x86.
>                 Both vDPA and VFIO would go through this code path.
> 
>                 This code path only affect x86 for now.
> 
>                 Signed-off-by: Zhu Lingshan <lingshan.zhu@intel.com>
> 
>             I think it's best to leave VFIO alone. Add appropriate APIs for VDPA,
>             worry about converting existing users later.
> 
> 
> 
>         Just to make sure I understand, did you mean:
> 
>         1) introduce another bridge for vDPA
> 
>         or
> 
>         2) only detect vDPA via bypass manager? (we can leave VFIO code as is, then
>         the assigned device counter may increase/decrease twice if VFIO use irq
>         bypass manager which should be no harm).
> 
>         Thanks
> 
>     2 is probably easier to justify. 1 would depend on the specific bridge
>     proposed.
> 
> Thanks Michael, so we should just drop changes in vfio, just increase / decrease
> the counter in irq bypass manager. right?
> 
> Thanks

I don't see any issue with that.

> 
> 
>                 ---
>                   arch/x86/kvm/x86.c | 10 ++++++++--
>                   virt/kvm/vfio.c    |  2 --
>                   2 files changed, 8 insertions(+), 4 deletions(-)
> 
>                 diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
>                 index 00c88c2..20c07d3 100644
>                 --- a/arch/x86/kvm/x86.c
>                 +++ b/arch/x86/kvm/x86.c
>                 @@ -10624,11 +10624,17 @@ int kvm_arch_irq_bypass_add_producer(struct irq_bypass_consumer *cons,
>                   {
>                         struct kvm_kernel_irqfd *irqfd =
>                                 container_of(cons, struct kvm_kernel_irqfd, consumer);
>                 +       int ret;
>                         irqfd->producer = prod;
>                 +       kvm_arch_start_assignment(irqfd->kvm);
>                 +       ret = kvm_x86_ops.update_pi_irte(irqfd->kvm,
>                 +                                        prod->irq, irqfd->gsi, 1);
>                 +
>                 +       if (ret)
>                 +               kvm_arch_end_assignment(irqfd->kvm);
>                 -       return kvm_x86_ops.update_pi_irte(irqfd->kvm,
>                 -                                          prod->irq, irqfd->gsi, 1);
>                 +       return ret;
>                   }
>                   void kvm_arch_irq_bypass_del_producer(struct irq_bypass_consumer *cons,
>                 diff --git a/virt/kvm/vfio.c b/virt/kvm/vfio.c
>                 index 8fcbc50..111da52 100644
>                 --- a/virt/kvm/vfio.c
>                 +++ b/virt/kvm/vfio.c
>                 @@ -226,7 +226,6 @@ static int kvm_vfio_set_group(struct kvm_device *dev, long attr, u64 arg)
>                                 list_add_tail(&kvg->node, &kv->group_list);
>                                 kvg->vfio_group = vfio_group;
>                 -               kvm_arch_start_assignment(dev->kvm);
>                                 mutex_unlock(&kv->lock);
>                 @@ -254,7 +253,6 @@ static int kvm_vfio_set_group(struct kvm_device *dev, long attr, u64 arg)
>                                                 continue;
>                                         list_del(&kvg->node);
>                 -                       kvm_arch_end_assignment(dev->kvm);
>                   #ifdef CONFIG_SPAPR_TCE_IOMMU
>                                         kvm_spapr_tce_release_vfio_group(dev->kvm,
>                                                                          kvg->vfio_group);
>                 --
>                 1.8.3.1
> 

Re: [PATCH 6/7] ifcvf: replace irq_request/free with helpers in vDPA core.

[Jason Wang]

On 2020/7/13 下午6:22, Zhu, Lingshan wrote:
>
>
> On 7/13/2020 4:33 PM, Jason Wang wrote:
>>
>> On 2020/7/12 下午10:49, Zhu Lingshan wrote:
>>> This commit replaced irq_request/free() with helpers in vDPA
>>> core, so that it can request/free irq and setup irq offloading
>>> on order.
>>>
>>> Signed-off-by: Zhu Lingshan <lingshan.zhu@intel.com>
>>> ---
>>>   drivers/vdpa/ifcvf/ifcvf_main.c | 11 ++++++-----
>>>   1 file changed, 6 insertions(+), 5 deletions(-)
>>>
>>> diff --git a/drivers/vdpa/ifcvf/ifcvf_main.c 
>>> b/drivers/vdpa/ifcvf/ifcvf_main.c
>>> index f5a60c1..65b84e1 100644
>>> --- a/drivers/vdpa/ifcvf/ifcvf_main.c
>>> +++ b/drivers/vdpa/ifcvf/ifcvf_main.c
>>> @@ -47,11 +47,12 @@ static void ifcvf_free_irq(struct ifcvf_adapter 
>>> *adapter, int queues)
>>>   {
>>>       struct pci_dev *pdev = adapter->pdev;
>>>       struct ifcvf_hw *vf = &adapter->vf;
>>> +    struct vdpa_device *vdpa = &adapter->vdpa;
>>>       int i;
>>>           for (i = 0; i < queues; i++)
>>> -        devm_free_irq(&pdev->dev, vf->vring[i].irq, &vf->vring[i]);
>>> +        vdpa_free_vq_irq(&pdev->dev, vdpa, vf->vring[i].irq, i, 
>>> &vf->vring[i]);
>>>         ifcvf_free_irq_vectors(pdev);
>>>   }
>>> @@ -60,6 +61,7 @@ static int ifcvf_request_irq(struct ifcvf_adapter 
>>> *adapter)
>>>   {
>>>       struct pci_dev *pdev = adapter->pdev;
>>>       struct ifcvf_hw *vf = &adapter->vf;
>>> +    struct vdpa_device *vdpa = &adapter->vdpa;
>>>       int vector, i, ret, irq;
>>>         ret = pci_alloc_irq_vectors(pdev, IFCVF_MAX_INTR,
>>> @@ -73,6 +75,7 @@ static int ifcvf_request_irq(struct ifcvf_adapter 
>>> *adapter)
>>>            pci_name(pdev));
>>>       vector = 0;
>>>       irq = pci_irq_vector(pdev, vector);
>>> +    /* config interrupt */
>>
>>
>> Unnecessary changes.
> This is to show we did not setup this irq offloading for config 
> interrupt, only setup irq offloading for data vq. But can remove this 
> since we have config_msix_name in code to show what it is


Btw, any reason for not making config interrupt work for irq offloading? 
I don't see any thing that blocks this.

Thanks


> Thanks BR Zhu Lingshan
>>

Re: [PATCH 5/7] virtio_vdpa: init IRQ offloading function pointers to NULL.

[Jason Wang]

On 2020/7/13 下午6:20, Zhu, Lingshan wrote:
>
>
> On 7/13/2020 4:28 PM, Jason Wang wrote:
>>
>> On 2020/7/12 下午10:49, Zhu Lingshan wrote:
>>> This commit initialize IRQ offloading function pointers in
>>> virtio_vdpa_driver to NULL. Becasue irq offloading only focus
>>> on VMs for vhost_vdpa.
>>>
>>> Signed-off-by: Zhu Lingshan <lingshan.zhu@intel.com>
>>> ---
>>>   drivers/virtio/virtio_vdpa.c | 2 ++
>>>   1 file changed, 2 insertions(+)
>>>
>>> diff --git a/drivers/virtio/virtio_vdpa.c 
>>> b/drivers/virtio/virtio_vdpa.c
>>> index c30eb55..1e8acb9 100644
>>> --- a/drivers/virtio/virtio_vdpa.c
>>> +++ b/drivers/virtio/virtio_vdpa.c
>>> @@ -386,6 +386,8 @@ static void virtio_vdpa_remove(struct 
>>> vdpa_device *vdpa)
>>>       },
>>>       .probe    = virtio_vdpa_probe,
>>>       .remove = virtio_vdpa_remove,
>>> +    .setup_vq_irq = NULL,
>>> +    .unsetup_vq_irq = NULL,
>>>   };
>>
>>
>> Is this really needed consider the it's static?
>>
>> Thanks
> This is for readability, to show they are NULL, so virtio_vdpa would not go through irq forwarding / offloading.
>
> Does this make sense?


Probably not, please refer what is done by other subsystems.

Thanks


>
> Thanks,
> BR
> Zhu Lingshan
>>
>>
>>> module_vdpa_driver(virtio_vdpa_driver);
>>

Re: [PATCH 3/7] vhost_vdpa: implement IRQ offloading functions in vhost_vdpa

[Jason Wang]

On 2020/7/13 下午5:47, Zhu, Lingshan wrote:
>
>
> On 7/13/2020 4:22 PM, Jason Wang wrote:
>>
>> On 2020/7/12 下午10:49, Zhu Lingshan wrote:
>>> This patch introduce a set of functions for setup/unsetup
>>> and update irq offloading respectively by register/unregister
>>> and re-register the irq_bypass_producer.
>>>
>>> Signed-off-by: Zhu Lingshan <lingshan.zhu@intel.com>
>>> ---
>>>   drivers/vhost/vdpa.c | 69 
>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>   1 file changed, 69 insertions(+)
>>>
>>> diff --git a/drivers/vhost/vdpa.c b/drivers/vhost/vdpa.c
>>> index 2fcc422..92683e4 100644
>>> --- a/drivers/vhost/vdpa.c
>>> +++ b/drivers/vhost/vdpa.c
>>> @@ -115,6 +115,63 @@ static irqreturn_t vhost_vdpa_config_cb(void 
>>> *private)
>>>       return IRQ_HANDLED;
>>>   }
>>>   +static void vhost_vdpa_setup_vq_irq(struct vdpa_device *dev, int 
>>> qid, int irq)
>>> +{
>>> +    struct vhost_vdpa *v = vdpa_get_drvdata(dev);
>>> +    struct vhost_virtqueue *vq = &v->vqs[qid];
>>> +    int ret;
>>> +
>>> +    vq_err(vq, "setup irq bypass for vq %d with irq = %d\n", qid, 
>>> irq);
>>> +    spin_lock(&vq->call_ctx.ctx_lock);
>>> +    if (!vq->call_ctx.ctx)
>>> +        return;
>>> +
>>> +    vq->call_ctx.producer.token = vq->call_ctx.ctx;
>>> +    vq->call_ctx.producer.irq = irq;
>>> +    ret = irq_bypass_register_producer(&vq->call_ctx.producer);
>>> +    spin_unlock(&vq->call_ctx.ctx_lock);
>>> +
>>> +    if (unlikely(ret))
>>> +        vq_err(vq,
>>> +        "irq bypass producer (token %p registration fails: %d\n",
>>> +        vq->call_ctx.producer.token, ret);
>>
>>
>> Not sure this deserves a vq_err(), irq will be relayed through 
>> eventfd if irq bypass manager can't work.
> OK, I see vq_err() will eventfd_signal err_ctx than just print a 
> message, will remove all vq_err().
>>
>>
>>> +}
>>> +
>>> +static void vhost_vdpa_unsetup_vq_irq(struct vdpa_device *dev, int 
>>> qid)
>>> +{
>>> +    struct vhost_vdpa *v = vdpa_get_drvdata(dev);
>>> +    struct vhost_virtqueue *vq = &v->vqs[qid];
>>> +
>>> +    spin_lock(&vq->call_ctx.ctx_lock);
>>> + irq_bypass_unregister_producer(&vq->call_ctx.producer);
>>> +    spin_unlock(&vq->call_ctx.ctx_lock);
>>> +
>>> +    vq_err(vq, "unsetup irq bypass for vq %d\n", qid);
>>
>>
>> Why call vq_err() here?
>>
>>
>>> +}
>>> +
>>> +static void vhost_vdpa_update_vq_irq(struct vhost_virtqueue *vq)
>>> +{
>>> +    struct eventfd_ctx *ctx;
>>> +    void *token;
>>> +
>>> +    spin_lock(&vq->call_ctx.ctx_lock);
>>> +    ctx = vq->call_ctx.ctx;
>>> +    token = vq->call_ctx.producer.token;
>>> +    if (ctx == token)
>>> +        return;
>>
>>
>> Need do unlock here.
> sure!
>>
>>
>>> +
>>> +    if (!ctx && token)
>>> + irq_bypass_unregister_producer(&vq->call_ctx.producer);
>>> +
>>> +    if (ctx && ctx != token) {
>>> + irq_bypass_unregister_producer(&vq->call_ctx.producer);
>>> +        vq->call_ctx.producer.token = ctx;
>>> + irq_bypass_register_producer(&vq->call_ctx.producer);
>>> +    }
>>> +
>>> +    spin_unlock(&vq->call_ctx.ctx_lock);
>>
>>
>> This should be rare so I'd use simple codes just do unregister and 
>> register.
>
> do you mean remove "if (ctx && ctx != token)"? I think this could be 
> useful, we should only update it when ctx!=NULL and ctx!= existing token.
>

I meant something like:

unregister();
vq->call_ctx.producer.token = ctx;
register();


>>
>>
>>> +}
>>> +
>>>   static void vhost_vdpa_reset(struct vhost_vdpa *v)
>>>   {
>>>       struct vdpa_device *vdpa = v->vdpa;
>>> @@ -332,6 +389,7 @@ static long vhost_vdpa_set_config_call(struct 
>>> vhost_vdpa *v, u32 __user *argp)
>>>         return 0;
>>>   }
>>> +
>>
>>
>> Unnecessary change.
> this new blank line is added because there is no blank line between 
> functions, I will double check


The point is not mixing coding style fix with other fixes or enhancement.

Thanks


> THanks, BR Zhu Lingshan
>>
>>
>>>   static long vhost_vdpa_vring_ioctl(struct vhost_vdpa *v, unsigned 
>>> int cmd,
>>>                      void __user *argp)
>>>   {
>>> @@ -390,6 +448,16 @@ static long vhost_vdpa_vring_ioctl(struct 
>>> vhost_vdpa *v, unsigned int cmd,
>>>               cb.private = NULL;
>>>           }
>>>           ops->set_vq_cb(vdpa, idx, &cb);
>>> +#ifdef CONFIG_HAVE_KVM_IRQ_BYPASS
>>> +        /*
>>> +         * if it has a non-zero irq, means there is a
>>> +         * previsouly registered irq_bypass_producer,
>>> +         * we should update it when ctx (its token)
>>> +         * changes.
>>> +         */
>>> +        if (vq->call_ctx.producer.irq)
>>> +            vhost_vdpa_update_vq_irq(vq);
>>> +#endif
>>>           break;
>>>         case VHOST_SET_VRING_NUM:
>>> @@ -741,6 +809,7 @@ static int vhost_vdpa_open(struct inode *inode, 
>>> struct file *filep)
>>>           vqs[i] = &v->vqs[i];
>>>           vqs[i]->handle_kick = handle_vq_kick;
>>>       }
>>> +
>>
>>
>> Unnecessary change.
>>
>> Thanks
>>
>>
>>>       vhost_dev_init(dev, vqs, nvqs, 0, 0, 0, false,
>>>                  vhost_vdpa_process_iotlb_msg);
>>

Re: [PATCH 3/7] vhost_vdpa: implement IRQ offloading functions in vhost_vdpa

[Jason Wang]

On 2020/7/15 下午4:56, Zhu, Lingshan wrote:
>
>
> On 7/15/2020 4:51 PM, Jason Wang wrote:
>>
>> On 2020/7/13 下午5:47, Zhu, Lingshan wrote:
>>>
>>>
>>> On 7/13/2020 4:22 PM, Jason Wang wrote:
>>>>
>>>> On 2020/7/12 下午10:49, Zhu Lingshan wrote:
>>>>> This patch introduce a set of functions for setup/unsetup
>>>>> and update irq offloading respectively by register/unregister
>>>>> and re-register the irq_bypass_producer.
>>>>>
>>>>> Signed-off-by: Zhu Lingshan <lingshan.zhu@intel.com>
>>>>> ---
>>>>>   drivers/vhost/vdpa.c | 69 
>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>   1 file changed, 69 insertions(+)
>>>>>
>>>>> diff --git a/drivers/vhost/vdpa.c b/drivers/vhost/vdpa.c
>>>>> index 2fcc422..92683e4 100644
>>>>> --- a/drivers/vhost/vdpa.c
>>>>> +++ b/drivers/vhost/vdpa.c
>>>>> @@ -115,6 +115,63 @@ static irqreturn_t vhost_vdpa_config_cb(void 
>>>>> *private)
>>>>>       return IRQ_HANDLED;
>>>>>   }
>>>>>   +static void vhost_vdpa_setup_vq_irq(struct vdpa_device *dev, 
>>>>> int qid, int irq)
>>>>> +{
>>>>> +    struct vhost_vdpa *v = vdpa_get_drvdata(dev);
>>>>> +    struct vhost_virtqueue *vq = &v->vqs[qid];
>>>>> +    int ret;
>>>>> +
>>>>> +    vq_err(vq, "setup irq bypass for vq %d with irq = %d\n", qid, 
>>>>> irq);
>>>>> +    spin_lock(&vq->call_ctx.ctx_lock);
>>>>> +    if (!vq->call_ctx.ctx)
>>>>> +        return;
>>>>> +
>>>>> +    vq->call_ctx.producer.token = vq->call_ctx.ctx;
>>>>> +    vq->call_ctx.producer.irq = irq;
>>>>> +    ret = irq_bypass_register_producer(&vq->call_ctx.producer);
>>>>> +    spin_unlock(&vq->call_ctx.ctx_lock);
>>>>> +
>>>>> +    if (unlikely(ret))
>>>>> +        vq_err(vq,
>>>>> +        "irq bypass producer (token %p registration fails: %d\n",
>>>>> +        vq->call_ctx.producer.token, ret);
>>>>
>>>>
>>>> Not sure this deserves a vq_err(), irq will be relayed through 
>>>> eventfd if irq bypass manager can't work.
>>> OK, I see vq_err() will eventfd_signal err_ctx than just print a 
>>> message, will remove all vq_err().
>>>>
>>>>
>>>>> +}
>>>>> +
>>>>> +static void vhost_vdpa_unsetup_vq_irq(struct vdpa_device *dev, 
>>>>> int qid)
>>>>> +{
>>>>> +    struct vhost_vdpa *v = vdpa_get_drvdata(dev);
>>>>> +    struct vhost_virtqueue *vq = &v->vqs[qid];
>>>>> +
>>>>> +    spin_lock(&vq->call_ctx.ctx_lock);
>>>>> + irq_bypass_unregister_producer(&vq->call_ctx.producer);
>>>>> +    spin_unlock(&vq->call_ctx.ctx_lock);
>>>>> +
>>>>> +    vq_err(vq, "unsetup irq bypass for vq %d\n", qid);
>>>>
>>>>
>>>> Why call vq_err() here?
>>>>
>>>>
>>>>> +}
>>>>> +
>>>>> +static void vhost_vdpa_update_vq_irq(struct vhost_virtqueue *vq)
>>>>> +{
>>>>> +    struct eventfd_ctx *ctx;
>>>>> +    void *token;
>>>>> +
>>>>> +    spin_lock(&vq->call_ctx.ctx_lock);
>>>>> +    ctx = vq->call_ctx.ctx;
>>>>> +    token = vq->call_ctx.producer.token;
>>>>> +    if (ctx == token)
>>>>> +        return;
>>>>
>>>>
>>>> Need do unlock here.
>>> sure!
>>>>
>>>>
>>>>> +
>>>>> +    if (!ctx && token)
>>>>> + irq_bypass_unregister_producer(&vq->call_ctx.producer);
>>>>> +
>>>>> +    if (ctx && ctx != token) {
>>>>> + irq_bypass_unregister_producer(&vq->call_ctx.producer);
>>>>> +        vq->call_ctx.producer.token = ctx;
>>>>> + irq_bypass_register_producer(&vq->call_ctx.producer);
>>>>> +    }
>>>>> +
>>>>> +    spin_unlock(&vq->call_ctx.ctx_lock);
>>>>
>>>>
>>>> This should be rare so I'd use simple codes just do unregister and 
>>>> register.
>>>
>>> do you mean remove "if (ctx && ctx != token)"? I think this could be 
>>> useful, we should only update it when ctx!=NULL and ctx!= existing 
>>> token.
>>>
>>
>> I meant something like:
>>
>> unregister();
>> vq->call_ctx.producer.token = ctx;
>> register();
> This is what we are doing now, or I must missed somethig:
> if (ctx && ctx != token) {
> 	irq_bypass_unregister_producer(&vq->call_ctx.producer);
> 	vq->call_ctx.producer.token = ctx;
> 	irq_bypass_register_producer(&vq->call_ctx.producer);
>
> }
>
> It just unregister and register.


I meant there's probably no need for the check and another check and 
unregister before. The whole function is as simple as I suggested above.

Thanks


>
> Thanks,
> BR
> Zhu Lingshan

Re: [PATCH 3/7] vhost_vdpa: implement IRQ offloading functions in vhost_vdpa

[Jason Wang]

On 2020/7/15 下午5:20, Zhu, Lingshan wrote:
>>>>
>>>> I meant something like:
>>>>
>>>> unregister();
>>>> vq->call_ctx.producer.token = ctx;
>>>> register();
>>> This is what we are doing now, or I must missed somethig:
>>> if (ctx && ctx != token) {
>>>     irq_bypass_unregister_producer(&vq->call_ctx.producer);
>>>     vq->call_ctx.producer.token = ctx;
>>>     irq_bypass_register_producer(&vq->call_ctx.producer);
>>>
>>> }
>>>
>>> It just unregister and register.
>>
>>
>> I meant there's probably no need for the check and another check and 
>> unregister before. The whole function is as simple as I suggested above.
>>
>> Thanks
> IMHO we still need the checks, this function handles three cases:
> (1)if the ctx == token, we do nothing. For this unregister and register can work, but waste of time.


But we have a more simple code and we don't care about the performance 
here since the operations is rare.


> (2)if token exists but ctx is NULL, this means user space issued an unbind, so we need to only unregister the producer.


Note that the register/unregister have a graceful check of whether or 
not there's a token.


> (3)if ctx exists and ctx!=token, this means there is a new ctx, we need to update producer by unregister and register.
>
> I think we can not simply handle all these cases by "unregister and register".


So it looks to me the functions are equivalent.

Thanks


>
> Thanks,
> BR
> Zhu Lingshan

Re: [PATCH 6/7] ifcvf: replace irq_request/free with helpers in vDPA core.

[Michael S. Tsirkin]

On Wed, Jul 15, 2020 at 04:40:17PM +0800, Jason Wang wrote:
> 
> On 2020/7/13 下午6:22, Zhu, Lingshan wrote:
> > 
> > 
> > On 7/13/2020 4:33 PM, Jason Wang wrote:
> > > 
> > > On 2020/7/12 下午10:49, Zhu Lingshan wrote:
> > > > This commit replaced irq_request/free() with helpers in vDPA
> > > > core, so that it can request/free irq and setup irq offloading
> > > > on order.
> > > > 
> > > > Signed-off-by: Zhu Lingshan <lingshan.zhu@intel.com>
> > > > ---
> > > >   drivers/vdpa/ifcvf/ifcvf_main.c | 11 ++++++-----
> > > >   1 file changed, 6 insertions(+), 5 deletions(-)
> > > > 
> > > > diff --git a/drivers/vdpa/ifcvf/ifcvf_main.c
> > > > b/drivers/vdpa/ifcvf/ifcvf_main.c
> > > > index f5a60c1..65b84e1 100644
> > > > --- a/drivers/vdpa/ifcvf/ifcvf_main.c
> > > > +++ b/drivers/vdpa/ifcvf/ifcvf_main.c
> > > > @@ -47,11 +47,12 @@ static void ifcvf_free_irq(struct
> > > > ifcvf_adapter *adapter, int queues)
> > > >   {
> > > >       struct pci_dev *pdev = adapter->pdev;
> > > >       struct ifcvf_hw *vf = &adapter->vf;
> > > > +    struct vdpa_device *vdpa = &adapter->vdpa;
> > > >       int i;
> > > >           for (i = 0; i < queues; i++)
> > > > -        devm_free_irq(&pdev->dev, vf->vring[i].irq, &vf->vring[i]);
> > > > +        vdpa_free_vq_irq(&pdev->dev, vdpa, vf->vring[i].irq, i,
> > > > &vf->vring[i]);
> > > >         ifcvf_free_irq_vectors(pdev);
> > > >   }
> > > > @@ -60,6 +61,7 @@ static int ifcvf_request_irq(struct
> > > > ifcvf_adapter *adapter)
> > > >   {
> > > >       struct pci_dev *pdev = adapter->pdev;
> > > >       struct ifcvf_hw *vf = &adapter->vf;
> > > > +    struct vdpa_device *vdpa = &adapter->vdpa;
> > > >       int vector, i, ret, irq;
> > > >         ret = pci_alloc_irq_vectors(pdev, IFCVF_MAX_INTR,
> > > > @@ -73,6 +75,7 @@ static int ifcvf_request_irq(struct
> > > > ifcvf_adapter *adapter)
> > > >            pci_name(pdev));
> > > >       vector = 0;
> > > >       irq = pci_irq_vector(pdev, vector);
> > > > +    /* config interrupt */
> > > 
> > > 
> > > Unnecessary changes.
> > This is to show we did not setup this irq offloading for config
> > interrupt, only setup irq offloading for data vq. But can remove this
> > since we have config_msix_name in code to show what it is
> 
> 
> Btw, any reason for not making config interrupt work for irq offloading? I
> don't see any thing that blocks this.
> 
> Thanks

Well config accesses all go through userspace right?
Doing config interrupt directly would just be messy ...


> 
> > Thanks BR Zhu Lingshan
> > > 

Re: [PATCH 6/7] ifcvf: replace irq_request/free with helpers in vDPA core.

[Jason Wang]

On 2020/7/15 下午6:01, Michael S. Tsirkin wrote:
> On Wed, Jul 15, 2020 at 04:40:17PM +0800, Jason Wang wrote:
>> On 2020/7/13 下午6:22, Zhu, Lingshan wrote:
>>> On 7/13/2020 4:33 PM, Jason Wang wrote:
>>>> On 2020/7/12 下午10:49, Zhu Lingshan wrote:
>>>>> This commit replaced irq_request/free() with helpers in vDPA
>>>>> core, so that it can request/free irq and setup irq offloading
>>>>> on order.
>>>>>
>>>>> Signed-off-by: Zhu Lingshan<lingshan.zhu@intel.com>
>>>>> ---
>>>>>    drivers/vdpa/ifcvf/ifcvf_main.c | 11 ++++++-----
>>>>>    1 file changed, 6 insertions(+), 5 deletions(-)
>>>>>
>>>>> diff --git a/drivers/vdpa/ifcvf/ifcvf_main.c
>>>>> b/drivers/vdpa/ifcvf/ifcvf_main.c
>>>>> index f5a60c1..65b84e1 100644
>>>>> --- a/drivers/vdpa/ifcvf/ifcvf_main.c
>>>>> +++ b/drivers/vdpa/ifcvf/ifcvf_main.c
>>>>> @@ -47,11 +47,12 @@ static void ifcvf_free_irq(struct
>>>>> ifcvf_adapter *adapter, int queues)
>>>>>    {
>>>>>        struct pci_dev *pdev = adapter->pdev;
>>>>>        struct ifcvf_hw *vf = &adapter->vf;
>>>>> +    struct vdpa_device *vdpa = &adapter->vdpa;
>>>>>        int i;
>>>>>            for (i = 0; i < queues; i++)
>>>>> -        devm_free_irq(&pdev->dev, vf->vring[i].irq, &vf->vring[i]);
>>>>> +        vdpa_free_vq_irq(&pdev->dev, vdpa, vf->vring[i].irq, i,
>>>>> &vf->vring[i]);
>>>>>          ifcvf_free_irq_vectors(pdev);
>>>>>    }
>>>>> @@ -60,6 +61,7 @@ static int ifcvf_request_irq(struct
>>>>> ifcvf_adapter *adapter)
>>>>>    {
>>>>>        struct pci_dev *pdev = adapter->pdev;
>>>>>        struct ifcvf_hw *vf = &adapter->vf;
>>>>> +    struct vdpa_device *vdpa = &adapter->vdpa;
>>>>>        int vector, i, ret, irq;
>>>>>          ret = pci_alloc_irq_vectors(pdev, IFCVF_MAX_INTR,
>>>>> @@ -73,6 +75,7 @@ static int ifcvf_request_irq(struct
>>>>> ifcvf_adapter *adapter)
>>>>>             pci_name(pdev));
>>>>>        vector = 0;
>>>>>        irq = pci_irq_vector(pdev, vector);
>>>>> +    /* config interrupt */
>>>> Unnecessary changes.
>>> This is to show we did not setup this irq offloading for config
>>> interrupt, only setup irq offloading for data vq. But can remove this
>>> since we have config_msix_name in code to show what it is
>> Btw, any reason for not making config interrupt work for irq offloading? I
>> don't see any thing that blocks this.
>>
>> Thanks
> Well config accesses all go through userspace right?
> Doing config interrupt directly would just be messy ...


Right, so I think we need a better comment here.

Thanks


>
>

Re: [PATCH 3/7] vhost_vdpa: implement IRQ offloading functions in vhost_vdpa

[Zhu, Lingshan]

On 7/15/2020 5:42 PM, Jason Wang wrote:
>
> On 2020/7/15 下午5:20, Zhu, Lingshan wrote:
>>>>>
>>>>> I meant something like:
>>>>>
>>>>> unregister();
>>>>> vq->call_ctx.producer.token = ctx;
>>>>> register();
>>>> This is what we are doing now, or I must missed somethig:
>>>> if (ctx && ctx != token) {
>>>>     irq_bypass_unregister_producer(&vq->call_ctx.producer);
>>>>     vq->call_ctx.producer.token = ctx;
>>>>     irq_bypass_register_producer(&vq->call_ctx.producer);
>>>>
>>>> }
>>>>
>>>> It just unregister and register.
>>>
>>>
>>> I meant there's probably no need for the check and another check and 
>>> unregister before. The whole function is as simple as I suggested 
>>> above.
>>>
>>> Thanks
>> IMHO we still need the checks, this function handles three cases:
>> (1)if the ctx == token, we do nothing. For this unregister and 
>> register can work, but waste of time.
>
>
> But we have a more simple code and we don't care about the performance 
> here since the operations is rare.
>
>
>> (2)if token exists but ctx is NULL, this means user space issued an 
>> unbind, so we need to only unregister the producer.
>
>
> Note that the register/unregister have a graceful check of whether or 
> not there's a token.
>
>
>> (3)if ctx exists and ctx!=token, this means there is a new ctx, we 
>> need to update producer by unregister and register.
>>
>> I think we can not simply handle all these cases by "unregister and 
>> register".
>
>
> So it looks to me the functions are equivalent.
>
> Thanks
Sounds reasonable, thanks!
>
>
>>
>> Thanks,
>> BR
>> Zhu Lingshan
>

Re: [PATCH 6/7] ifcvf: replace irq_request/free with helpers in vDPA core.

[Zhu, Lingshan]

On 7/15/2020 6:04 PM, Jason Wang wrote:
>
> On 2020/7/15 下午6:01, Michael S. Tsirkin wrote:
>> On Wed, Jul 15, 2020 at 04:40:17PM +0800, Jason Wang wrote:
>>> On 2020/7/13 下午6:22, Zhu, Lingshan wrote:
>>>> On 7/13/2020 4:33 PM, Jason Wang wrote:
>>>>> On 2020/7/12 下午10:49, Zhu Lingshan wrote:
>>>>>> This commit replaced irq_request/free() with helpers in vDPA
>>>>>> core, so that it can request/free irq and setup irq offloading
>>>>>> on order.
>>>>>>
>>>>>> Signed-off-by: Zhu Lingshan<lingshan.zhu@intel.com>
>>>>>> ---
>>>>>>    drivers/vdpa/ifcvf/ifcvf_main.c | 11 ++++++-----
>>>>>>    1 file changed, 6 insertions(+), 5 deletions(-)
>>>>>>
>>>>>> diff --git a/drivers/vdpa/ifcvf/ifcvf_main.c
>>>>>> b/drivers/vdpa/ifcvf/ifcvf_main.c
>>>>>> index f5a60c1..65b84e1 100644
>>>>>> --- a/drivers/vdpa/ifcvf/ifcvf_main.c
>>>>>> +++ b/drivers/vdpa/ifcvf/ifcvf_main.c
>>>>>> @@ -47,11 +47,12 @@ static void ifcvf_free_irq(struct
>>>>>> ifcvf_adapter *adapter, int queues)
>>>>>>    {
>>>>>>        struct pci_dev *pdev = adapter->pdev;
>>>>>>        struct ifcvf_hw *vf = &adapter->vf;
>>>>>> +    struct vdpa_device *vdpa = &adapter->vdpa;
>>>>>>        int i;
>>>>>>            for (i = 0; i < queues; i++)
>>>>>> -        devm_free_irq(&pdev->dev, vf->vring[i].irq, &vf->vring[i]);
>>>>>> +        vdpa_free_vq_irq(&pdev->dev, vdpa, vf->vring[i].irq, i,
>>>>>> &vf->vring[i]);
>>>>>>          ifcvf_free_irq_vectors(pdev);
>>>>>>    }
>>>>>> @@ -60,6 +61,7 @@ static int ifcvf_request_irq(struct
>>>>>> ifcvf_adapter *adapter)
>>>>>>    {
>>>>>>        struct pci_dev *pdev = adapter->pdev;
>>>>>>        struct ifcvf_hw *vf = &adapter->vf;
>>>>>> +    struct vdpa_device *vdpa = &adapter->vdpa;
>>>>>>        int vector, i, ret, irq;
>>>>>>          ret = pci_alloc_irq_vectors(pdev, IFCVF_MAX_INTR,
>>>>>> @@ -73,6 +75,7 @@ static int ifcvf_request_irq(struct
>>>>>> ifcvf_adapter *adapter)
>>>>>>             pci_name(pdev));
>>>>>>        vector = 0;
>>>>>>        irq = pci_irq_vector(pdev, vector);
>>>>>> +    /* config interrupt */
>>>>> Unnecessary changes.
>>>> This is to show we did not setup this irq offloading for config
>>>> interrupt, only setup irq offloading for data vq. But can remove this
>>>> since we have config_msix_name in code to show what it is
>>> Btw, any reason for not making config interrupt work for irq 
>>> offloading? I
>>> don't see any thing that blocks this.
>>>
>>> Thanks
>> Well config accesses all go through userspace right?
>> Doing config interrupt directly would just be messy ...
>
>
> Right, so I think we need a better comment here.
Thanks, will add a better comment than just "config interrupt".
>
> Thanks
>
>
>>
>>
>