[PATCH 0/7] *** IRQ offloading for vDPA ***

[PATCH 0/7] *** IRQ offloading for vDPA ***

[Zhu Lingshan]

Hi All,

This series intends to implement IRQ offloading for
vhost_vdpa.

By the feat of irq forwarding facilities like posted
interrupt on X86, irq bypass can  help deliver
interrupts to vCPU directly.

vDPA devices have dedicated hardware backends like VFIO
pass-throughed devices. So it would be possible to setup
irq offloading(irq bypass) for vDPA devices and gain
performance improvements.

In my testing, with this feature, we can save 0.1ms
in a ping between two VFs on average.


Zhu Lingshan (7):
  vhost: introduce vhost_call_ctx
  kvm/vfio: detect assigned device via irqbypass manager
  vhost_vdpa: implement IRQ offloading functions in vhost_vdpa
  vDPA: implement IRQ offloading helpers in vDPA core
  virtio_vdpa: init IRQ offloading function pointers to NULL.
  ifcvf: replace irq_request/free with helpers in vDPA core.
  irqbypass: do not start consumer or producer when failed to connect

 arch/x86/kvm/x86.c              | 10 ++++--
 drivers/vdpa/ifcvf/ifcvf_main.c | 11 +++---
 drivers/vdpa/vdpa.c             | 46 +++++++++++++++++++++++++
 drivers/vhost/Kconfig           |  1 +
 drivers/vhost/vdpa.c            | 75 +++++++++++++++++++++++++++++++++++++++--
 drivers/vhost/vhost.c           | 22 ++++++++----
 drivers/vhost/vhost.h           |  9 ++++-
 drivers/virtio/virtio_vdpa.c    |  2 ++
 include/linux/vdpa.h            | 11 ++++++
 virt/kvm/vfio.c                 |  2 --
 virt/lib/irqbypass.c            | 16 +++++----
 11 files changed, 181 insertions(+), 24 deletions(-)

-- 
1.8.3.1

Re: [PATCH 0/7] *** IRQ offloading for vDPA ***

[Jason Wang]

On 2020/7/12 下午10:52, Zhu Lingshan wrote:
> Hi All,
>
> This series intends to implement IRQ offloading for
> vhost_vdpa.
>
> By the feat of irq forwarding facilities like posted
> interrupt on X86, irq bypass can  help deliver
> interrupts to vCPU directly.
>
> vDPA devices have dedicated hardware backends like VFIO
> pass-throughed devices. So it would be possible to setup
> irq offloading(irq bypass) for vDPA devices and gain
> performance improvements.
>
> In my testing, with this feature, we can save 0.1ms
> in a ping between two VFs on average.


Hi Lingshan:

During the virtio-networking meeting, Michael spots two possible issues:

1) do we need an new uAPI to stop the irq offloading?
2) can interrupt lost during the eventfd ctx?

For 1) I think we probably not, we can allocate an independent eventfd 
which does not map to MSIX. So the consumer can't match the producer and 
we fallback to eventfd based irq.
For 2) it looks to me guest should deal with the irq synchronization 
when mask or unmask MSIX vectors.

What's your thought?

Thanks


>
>
> Zhu Lingshan (7):
>    vhost: introduce vhost_call_ctx
>    kvm/vfio: detect assigned device via irqbypass manager
>    vhost_vdpa: implement IRQ offloading functions in vhost_vdpa
>    vDPA: implement IRQ offloading helpers in vDPA core
>    virtio_vdpa: init IRQ offloading function pointers to NULL.
>    ifcvf: replace irq_request/free with helpers in vDPA core.
>    irqbypass: do not start consumer or producer when failed to connect
>
>   arch/x86/kvm/x86.c              | 10 ++++--
>   drivers/vdpa/ifcvf/ifcvf_main.c | 11 +++---
>   drivers/vdpa/vdpa.c             | 46 +++++++++++++++++++++++++
>   drivers/vhost/Kconfig           |  1 +
>   drivers/vhost/vdpa.c            | 75 +++++++++++++++++++++++++++++++++++++++--
>   drivers/vhost/vhost.c           | 22 ++++++++----
>   drivers/vhost/vhost.h           |  9 ++++-
>   drivers/virtio/virtio_vdpa.c    |  2 ++
>   include/linux/vdpa.h            | 11 ++++++
>   virt/kvm/vfio.c                 |  2 --
>   virt/lib/irqbypass.c            | 16 +++++----
>   11 files changed, 181 insertions(+), 24 deletions(-)
>

Re: [PATCH 0/7] *** IRQ offloading for vDPA ***

[Jason Wang]

On 2020/7/16 上午9:39, Zhu, Lingshan wrote:
>
>
> On 7/15/2020 9:43 PM, Jason Wang wrote:
>>
>> On 2020/7/12 下午10:52, Zhu Lingshan wrote:
>>> Hi All,
>>>
>>> This series intends to implement IRQ offloading for
>>> vhost_vdpa.
>>>
>>> By the feat of irq forwarding facilities like posted
>>> interrupt on X86, irq bypass can  help deliver
>>> interrupts to vCPU directly.
>>>
>>> vDPA devices have dedicated hardware backends like VFIO
>>> pass-throughed devices. So it would be possible to setup
>>> irq offloading(irq bypass) for vDPA devices and gain
>>> performance improvements.
>>>
>>> In my testing, with this feature, we can save 0.1ms
>>> in a ping between two VFs on average.
>>
>>
>> Hi Lingshan:
>>
>> During the virtio-networking meeting, Michael spots two possible issues:
>>
>> 1) do we need an new uAPI to stop the irq offloading?
>> 2) can interrupt lost during the eventfd ctx?
>>
>> For 1) I think we probably not, we can allocate an independent 
>> eventfd which does not map to MSIX. So the consumer can't match the 
>> producer and we fallback to eventfd based irq.
> Hi Jason,
>
> I wonder why we need to stop irq offloading, but if we need to do so, maybe a new uAPI would be more intuitive to me,
> but why and who(user? qemu?) shall initialize this process, based on what kinda of basis to make the decision?


The reason is we may want to fallback to software datapath for some 
reason (e.g software assisted live migration). In this case we need 
intercept device write to used ring so we can not offloading virtqueue 
interrupt in this case.


>> For 2) it looks to me guest should deal with the irq synchronization 
>> when mask or unmask MSIX vectors.
> Agreed!


It's better to double check for this.

Thanks


>
> Thanks,
> BR
> Zhu Lingshan
>>
>> What's your thought?
>>
>> Thanks
>>
>>
>>>
>>>
>>> Zhu Lingshan (7):
>>>    vhost: introduce vhost_call_ctx
>>>    kvm/vfio: detect assigned device via irqbypass manager
>>>    vhost_vdpa: implement IRQ offloading functions in vhost_vdpa
>>>    vDPA: implement IRQ offloading helpers in vDPA core
>>>    virtio_vdpa: init IRQ offloading function pointers to NULL.
>>>    ifcvf: replace irq_request/free with helpers in vDPA core.
>>>    irqbypass: do not start consumer or producer when failed to connect
>>>
>>>   arch/x86/kvm/x86.c              | 10 ++++--
>>>   drivers/vdpa/ifcvf/ifcvf_main.c | 11 +++---
>>>   drivers/vdpa/vdpa.c             | 46 +++++++++++++++++++++++++
>>>   drivers/vhost/Kconfig           |  1 +
>>>   drivers/vhost/vdpa.c            | 75 
>>> +++++++++++++++++++++++++++++++++++++++--
>>>   drivers/vhost/vhost.c           | 22 ++++++++----
>>>   drivers/vhost/vhost.h           |  9 ++++-
>>>   drivers/virtio/virtio_vdpa.c    |  2 ++
>>>   include/linux/vdpa.h            | 11 ++++++
>>>   virt/kvm/vfio.c                 |  2 --
>>>   virt/lib/irqbypass.c            | 16 +++++----
>>>   11 files changed, 181 insertions(+), 24 deletions(-)
>>>
>>

Re: [PATCH 0/7] *** IRQ offloading for vDPA ***

[Jason Wang]

On 2020/7/16 下午12:13, Zhu, Lingshan wrote:
>
>
> On 7/16/2020 12:02 PM, Jason Wang wrote:
>>
>> On 2020/7/16 上午11:59, Zhu, Lingshan wrote:
>>>
>>> On 7/16/2020 10:59 AM, Jason Wang wrote:
>>>>
>>>> On 2020/7/16 上午9:39, Zhu, Lingshan wrote:
>>>>>
>>>>>
>>>>> On 7/15/2020 9:43 PM, Jason Wang wrote:
>>>>>>
>>>>>> On 2020/7/12 下午10:52, Zhu Lingshan wrote:
>>>>>>> Hi All,
>>>>>>>
>>>>>>> This series intends to implement IRQ offloading for
>>>>>>> vhost_vdpa.
>>>>>>>
>>>>>>> By the feat of irq forwarding facilities like posted
>>>>>>> interrupt on X86, irq bypass can  help deliver
>>>>>>> interrupts to vCPU directly.
>>>>>>>
>>>>>>> vDPA devices have dedicated hardware backends like VFIO
>>>>>>> pass-throughed devices. So it would be possible to setup
>>>>>>> irq offloading(irq bypass) for vDPA devices and gain
>>>>>>> performance improvements.
>>>>>>>
>>>>>>> In my testing, with this feature, we can save 0.1ms
>>>>>>> in a ping between two VFs on average.
>>>>>>
>>>>>>
>>>>>> Hi Lingshan:
>>>>>>
>>>>>> During the virtio-networking meeting, Michael spots two possible 
>>>>>> issues:
>>>>>>
>>>>>> 1) do we need an new uAPI to stop the irq offloading?
>>>>>> 2) can interrupt lost during the eventfd ctx?
>>>>>>
>>>>>> For 1) I think we probably not, we can allocate an independent 
>>>>>> eventfd which does not map to MSIX. So the consumer can't match 
>>>>>> the producer and we fallback to eventfd based irq.
>>>>> Hi Jason,
>>>>>
>>>>> I wonder why we need to stop irq offloading, but if we need to do 
>>>>> so, maybe a new uAPI would be more intuitive to me,
>>>>> but why and who(user? qemu?) shall initialize this process, based 
>>>>> on what kinda of basis to make the decision?
>>>>
>>>>
>>>> The reason is we may want to fallback to software datapath for some 
>>>> reason (e.g software assisted live migration). In this case we need 
>>>> intercept device write to used ring so we can not offloading 
>>>> virtqueue interrupt in this case.
>>> so add a VHOST_VDPA_STOP_IRQ_OFFLOADING? Then do we need a 
>>> VHOST_VDPA_START_IRQ_OFFLOADING, then let userspace fully control 
>>> this? Or any better approaches? 
>>
>>
>> Probably not, it's as simple as allocating another eventfd (but not 
>> irqfd), and pass it to vhost-vdpa. Then the offloading is disabled 
>> since it doesn't have a consumer.
> OK, sounds like QEMU work, no need to take care in this series, right?


That's my understanding.

Thanks


>
> Thanks,
> BR
> Zhu Lingshan
>>
>> Thanks
>>
>>

Re: [PATCH 0/7] *** IRQ offloading for vDPA ***

[Michael S. Tsirkin]

On Thu, Jul 16, 2020 at 09:39:17AM +0800, Zhu, Lingshan wrote:
> 
> On 7/15/2020 9:43 PM, Jason Wang wrote:
> 
> 
>     On 2020/7/12 下午10:52, Zhu Lingshan wrote:
> 
>         Hi All,
> 
>         This series intends to implement IRQ offloading for
>         vhost_vdpa.
> 
>         By the feat of irq forwarding facilities like posted
>         interrupt on X86, irq bypass can  help deliver
>         interrupts to vCPU directly.
> 
>         vDPA devices have dedicated hardware backends like VFIO
>         pass-throughed devices. So it would be possible to setup
>         irq offloading(irq bypass) for vDPA devices and gain
>         performance improvements.
> 
>         In my testing, with this feature, we can save 0.1ms
>         in a ping between two VFs on average.
> 
> 
> 
>     Hi Lingshan:
> 
>     During the virtio-networking meeting, Michael spots two possible issues:
> 
>     1) do we need an new uAPI to stop the irq offloading?
>     2) can interrupt lost during the eventfd ctx?
> 
>     For 1) I think we probably not, we can allocate an independent eventfd
>     which does not map to MSIX. So the consumer can't match the producer and we
>     fallback to eventfd based irq.
> 
> Hi Jason,
> 
> I wonder why we need to stop irq offloading, but if we need to do so, maybe a new uAPI would be more intuitive to me,
> but why and who(user? qemu?) shall initialize this process, based on what kinda of basis to make the decision?
> 
>     For 2) it looks to me guest should deal with the irq synchronization when
>     mask or unmask MSIX vectors.
> 
> Agreed!

Well we need to make sure during a switch each interrupt is reported
*somewhere*: either irq or eventfd - and not lost.


> Thanks,
> BR
> Zhu Lingshan
> 
> 
>     What's your thought?
> 
>     Thanks
> 
> 
> 
> 
> 
>         Zhu Lingshan (7):
>            vhost: introduce vhost_call_ctx
>            kvm/vfio: detect assigned device via irqbypass manager
>            vhost_vdpa: implement IRQ offloading functions in vhost_vdpa
>            vDPA: implement IRQ offloading helpers in vDPA core
>            virtio_vdpa: init IRQ offloading function pointers to NULL.
>            ifcvf: replace irq_request/free with helpers in vDPA core.
>            irqbypass: do not start consumer or producer when failed to connect
> 
>           arch/x86/kvm/x86.c              | 10 ++++--
>           drivers/vdpa/ifcvf/ifcvf_main.c | 11 +++---
>           drivers/vdpa/vdpa.c             | 46 +++++++++++++++++++++++++
>           drivers/vhost/Kconfig           |  1 +
>           drivers/vhost/vdpa.c            | 75
>         +++++++++++++++++++++++++++++++++++++++--
>           drivers/vhost/vhost.c           | 22 ++++++++----
>           drivers/vhost/vhost.h           |  9 ++++-
>           drivers/virtio/virtio_vdpa.c    |  2 ++
>           include/linux/vdpa.h            | 11 ++++++
>           virt/kvm/vfio.c                 |  2 --
>           virt/lib/irqbypass.c            | 16 +++++----
>           11 files changed, 181 insertions(+), 24 deletions(-)
> 
> 
> 
> 

Re: [PATCH 0/7] *** IRQ offloading for vDPA ***

[Michael S. Tsirkin]

On Thu, Jul 16, 2020 at 12:20:09PM +0800, Jason Wang wrote:
> 
> On 2020/7/16 下午12:13, Zhu, Lingshan wrote:
> > 
> > 
> > On 7/16/2020 12:02 PM, Jason Wang wrote:
> > > 
> > > On 2020/7/16 上午11:59, Zhu, Lingshan wrote:
> > > > 
> > > > On 7/16/2020 10:59 AM, Jason Wang wrote:
> > > > > 
> > > > > On 2020/7/16 上午9:39, Zhu, Lingshan wrote:
> > > > > > 
> > > > > > 
> > > > > > On 7/15/2020 9:43 PM, Jason Wang wrote:
> > > > > > > 
> > > > > > > On 2020/7/12 下午10:52, Zhu Lingshan wrote:
> > > > > > > > Hi All,
> > > > > > > > 
> > > > > > > > This series intends to implement IRQ offloading for
> > > > > > > > vhost_vdpa.
> > > > > > > > 
> > > > > > > > By the feat of irq forwarding facilities like posted
> > > > > > > > interrupt on X86, irq bypass can  help deliver
> > > > > > > > interrupts to vCPU directly.
> > > > > > > > 
> > > > > > > > vDPA devices have dedicated hardware backends like VFIO
> > > > > > > > pass-throughed devices. So it would be possible to setup
> > > > > > > > irq offloading(irq bypass) for vDPA devices and gain
> > > > > > > > performance improvements.
> > > > > > > > 
> > > > > > > > In my testing, with this feature, we can save 0.1ms
> > > > > > > > in a ping between two VFs on average.
> > > > > > > 
> > > > > > > 
> > > > > > > Hi Lingshan:
> > > > > > > 
> > > > > > > During the virtio-networking meeting, Michael spots
> > > > > > > two possible issues:
> > > > > > > 
> > > > > > > 1) do we need an new uAPI to stop the irq offloading?
> > > > > > > 2) can interrupt lost during the eventfd ctx?
> > > > > > > 
> > > > > > > For 1) I think we probably not, we can allocate an
> > > > > > > independent eventfd which does not map to MSIX. So
> > > > > > > the consumer can't match the producer and we
> > > > > > > fallback to eventfd based irq.
> > > > > > Hi Jason,
> > > > > > 
> > > > > > I wonder why we need to stop irq offloading, but if we
> > > > > > need to do so, maybe a new uAPI would be more intuitive
> > > > > > to me,
> > > > > > but why and who(user? qemu?) shall initialize this
> > > > > > process, based on what kinda of basis to make the
> > > > > > decision?
> > > > > 
> > > > > 
> > > > > The reason is we may want to fallback to software datapath
> > > > > for some reason (e.g software assisted live migration). In
> > > > > this case we need intercept device write to used ring so we
> > > > > can not offloading virtqueue interrupt in this case.
> > > > so add a VHOST_VDPA_STOP_IRQ_OFFLOADING? Then do we need a
> > > > VHOST_VDPA_START_IRQ_OFFLOADING, then let userspace fully
> > > > control this? Or any better approaches?
> > > 
> > > 
> > > Probably not, it's as simple as allocating another eventfd (but not
> > > irqfd), and pass it to vhost-vdpa. Then the offloading is disabled
> > > since it doesn't have a consumer.
> > OK, sounds like QEMU work, no need to take care in this series, right?
> 
> 
> That's my understanding.
> 
> Thanks

Let's confirm a switch happens atomically so each interrupt
is sent either to eventfd to guest directly though.

> 
> > 
> > Thanks,
> > BR
> > Zhu Lingshan
> > > 
> > > Thanks
> > > 
> > > 

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [PATCH 0/7] *** IRQ offloading for vDPA ***

[Jason Wang]

On 2020/7/16 下午2:15, Michael S. Tsirkin wrote:
> On Thu, Jul 16, 2020 at 12:20:09PM +0800, Jason Wang wrote:
>> On 2020/7/16 下午12:13, Zhu, Lingshan wrote:
>>> On 7/16/2020 12:02 PM, Jason Wang wrote:
>>>> On 2020/7/16 上午11:59, Zhu, Lingshan wrote:
>>>>> On 7/16/2020 10:59 AM, Jason Wang wrote:
>>>>>> On 2020/7/16 上午9:39, Zhu, Lingshan wrote:
>>>>>>> On 7/15/2020 9:43 PM, Jason Wang wrote:
>>>>>>>> On 2020/7/12 下午10:52, Zhu Lingshan wrote:
>>>>>>>>> Hi All,
>>>>>>>>>
>>>>>>>>> This series intends to implement IRQ offloading for
>>>>>>>>> vhost_vdpa.
>>>>>>>>>
>>>>>>>>> By the feat of irq forwarding facilities like posted
>>>>>>>>> interrupt on X86, irq bypass can  help deliver
>>>>>>>>> interrupts to vCPU directly.
>>>>>>>>>
>>>>>>>>> vDPA devices have dedicated hardware backends like VFIO
>>>>>>>>> pass-throughed devices. So it would be possible to setup
>>>>>>>>> irq offloading(irq bypass) for vDPA devices and gain
>>>>>>>>> performance improvements.
>>>>>>>>>
>>>>>>>>> In my testing, with this feature, we can save 0.1ms
>>>>>>>>> in a ping between two VFs on average.
>>>>>>>> Hi Lingshan:
>>>>>>>>
>>>>>>>> During the virtio-networking meeting, Michael spots
>>>>>>>> two possible issues:
>>>>>>>>
>>>>>>>> 1) do we need an new uAPI to stop the irq offloading?
>>>>>>>> 2) can interrupt lost during the eventfd ctx?
>>>>>>>>
>>>>>>>> For 1) I think we probably not, we can allocate an
>>>>>>>> independent eventfd which does not map to MSIX. So
>>>>>>>> the consumer can't match the producer and we
>>>>>>>> fallback to eventfd based irq.
>>>>>>> Hi Jason,
>>>>>>>
>>>>>>> I wonder why we need to stop irq offloading, but if we
>>>>>>> need to do so, maybe a new uAPI would be more intuitive
>>>>>>> to me,
>>>>>>> but why and who(user? qemu?) shall initialize this
>>>>>>> process, based on what kinda of basis to make the
>>>>>>> decision?
>>>>>> The reason is we may want to fallback to software datapath
>>>>>> for some reason (e.g software assisted live migration). In
>>>>>> this case we need intercept device write to used ring so we
>>>>>> can not offloading virtqueue interrupt in this case.
>>>>> so add a VHOST_VDPA_STOP_IRQ_OFFLOADING? Then do we need a
>>>>> VHOST_VDPA_START_IRQ_OFFLOADING, then let userspace fully
>>>>> control this? Or any better approaches?
>>>> Probably not, it's as simple as allocating another eventfd (but not
>>>> irqfd), and pass it to vhost-vdpa. Then the offloading is disabled
>>>> since it doesn't have a consumer.
>>> OK, sounds like QEMU work, no need to take care in this series, right?
>> That's my understanding.
>>
>> Thanks
> Let's confirm a switch happens atomically so each interrupt
> is sent either to eventfd to guest directly though.


I think it's safe since:

1) we don't alloc/free interrupt during the eventfd change
2) The irte is modified automatically through cmpxchg_double() in 
modify_irte(), so the interrupt is either remapping to eventfd or pi 
descriptor

Thanks


>