[Qemu-devel] [PATCH V2] virtio: Fix no interrupt when not creating msi controller

[Qemu-devel] [PATCH V2] virtio: Fix no interrupt when not creating msi controller

[Shannon Zhao]

From: Shannon Zhao <shannon.zhao@linaro.org>

For ARM virt machine, if we use virt-2.7 which will not create ITS node,
the virtio-net can not recieve interrupts so it can't get ip address
through dhcp.
This fixes commit 83d768b(virtio: set ISR on dataplane notifications).

Signed-off-by: Shannon Zhao <shannon.zhao@linaro.org>
---
V2: Factor out a common function instead of duplicating code
---
 hw/virtio/virtio.c | 11 ++++++++---
 1 file changed, 8 insertions(+), 3 deletions(-)

diff --git a/hw/virtio/virtio.c b/hw/virtio/virtio.c
index aa4f38f..00e8f74 100644
--- a/hw/virtio/virtio.c
+++ b/hw/virtio/virtio.c
@@ -1441,6 +1441,12 @@ void virtio_notify_irqfd(VirtIODevice *vdev, VirtQueue *vq)
     event_notifier_set(&vq->guest_notifier);
 }
 
+static void virtio_irq(VirtQueue *vq)
+{
+    virtio_set_isr(vq->vdev, 0x1);
+    virtio_notify_vector(vq->vdev, vq->vector);
+}
+
 void virtio_notify(VirtIODevice *vdev, VirtQueue *vq)
 {
     if (!virtio_should_notify(vdev, vq)) {
@@ -1448,8 +1454,7 @@ void virtio_notify(VirtIODevice *vdev, VirtQueue *vq)
     }
 
     trace_virtio_notify(vdev, vq);
-    virtio_set_isr(vq->vdev, 0x1);
-    virtio_notify_vector(vdev, vq->vector);
+    virtio_irq(vq);
 }
 
 void virtio_notify_config(VirtIODevice *vdev)
@@ -2082,7 +2087,7 @@ static void virtio_queue_guest_notifier_read(EventNotifier *n)
 {
     VirtQueue *vq = container_of(n, VirtQueue, guest_notifier);
     if (event_notifier_test_and_clear(n)) {
-        virtio_notify_vector(vq->vdev, vq->vector);
+        virtio_irq(vq);
     }
 }
 
-- 
2.0.4

Re: [Qemu-devel] [PATCH V2] virtio: Fix no interrupt when not creating msi controller

[Michael S. Tsirkin]

On Mon, Jan 23, 2017 at 08:55:04AM +0800, Shannon Zhao wrote:
> From: Shannon Zhao <shannon.zhao@linaro.org>
> 
> For ARM virt machine, if we use virt-2.7 which will not create ITS node,
> the virtio-net can not recieve interrupts so it can't get ip address
> through dhcp.
> This fixes commit 83d768b(virtio: set ISR on dataplane notifications).
> 
> Signed-off-by: Shannon Zhao <shannon.zhao@linaro.org>

Paolo, could you comment on this pls?


> ---
> V2: Factor out a common function instead of duplicating code
> ---
>  hw/virtio/virtio.c | 11 ++++++++---
>  1 file changed, 8 insertions(+), 3 deletions(-)
> 
> diff --git a/hw/virtio/virtio.c b/hw/virtio/virtio.c
> index aa4f38f..00e8f74 100644
> --- a/hw/virtio/virtio.c
> +++ b/hw/virtio/virtio.c
> @@ -1441,6 +1441,12 @@ void virtio_notify_irqfd(VirtIODevice *vdev, VirtQueue *vq)
>      event_notifier_set(&vq->guest_notifier);
>  }
>  
> +static void virtio_irq(VirtQueue *vq)
> +{
> +    virtio_set_isr(vq->vdev, 0x1);
> +    virtio_notify_vector(vq->vdev, vq->vector);
> +}
> +
>  void virtio_notify(VirtIODevice *vdev, VirtQueue *vq)
>  {
>      if (!virtio_should_notify(vdev, vq)) {
> @@ -1448,8 +1454,7 @@ void virtio_notify(VirtIODevice *vdev, VirtQueue *vq)
>      }
>  
>      trace_virtio_notify(vdev, vq);
> -    virtio_set_isr(vq->vdev, 0x1);
> -    virtio_notify_vector(vdev, vq->vector);
> +    virtio_irq(vq);
>  }
>  
>  void virtio_notify_config(VirtIODevice *vdev)
> @@ -2082,7 +2087,7 @@ static void virtio_queue_guest_notifier_read(EventNotifier *n)
>  {
>      VirtQueue *vq = container_of(n, VirtQueue, guest_notifier);
>      if (event_notifier_test_and_clear(n)) {
> -        virtio_notify_vector(vq->vdev, vq->vector);
> +        virtio_irq(vq);
>      }
>  }
>  
> -- 
> 2.0.4
> 

Re: [Qemu-devel] [PATCH V2] virtio: Fix no interrupt when not creating msi controller

[Paolo Bonzini]

On 23/01/2017 17:07, Michael S. Tsirkin wrote:
> On Mon, Jan 23, 2017 at 08:55:04AM +0800, Shannon Zhao wrote:
>> From: Shannon Zhao <shannon.zhao@linaro.org>
>>
>> For ARM virt machine, if we use virt-2.7 which will not create ITS node,
>> the virtio-net can not recieve interrupts so it can't get ip address
>> through dhcp.
>> This fixes commit 83d768b(virtio: set ISR on dataplane notifications).
>>
>> Signed-off-by: Shannon Zhao <shannon.zhao@linaro.org>
> 
> Paolo, could you comment on this pls?

For virtio-blk, my patch moved the setting of ISR from
virtio_queue_guest_notifier_read to virtio_notify_irqfd.  This is
because the irqfd emulation only needs to trigger the interrupt.
Setting the ISR should have been done elsewhere.

Can vhost set ISR (I thought it couldn't)?  If so, it seems like ARM was
relying on QEMU's irqfd emulation, as a work around for vhost not
setting ISR.  But this only works if irqfd is directed to QEMU and not
to KVM.  So if vhost cannot set ISR, I think vhost should be disabled
unless MSI is active.

Am I missing something?

Paolo

> 
>> ---
>> V2: Factor out a common function instead of duplicating code
>> ---
>>  hw/virtio/virtio.c | 11 ++++++++---
>>  1 file changed, 8 insertions(+), 3 deletions(-)
>>
>> diff --git a/hw/virtio/virtio.c b/hw/virtio/virtio.c
>> index aa4f38f..00e8f74 100644
>> --- a/hw/virtio/virtio.c
>> +++ b/hw/virtio/virtio.c
>> @@ -1441,6 +1441,12 @@ void virtio_notify_irqfd(VirtIODevice *vdev, VirtQueue *vq)
>>      event_notifier_set(&vq->guest_notifier);
>>  }
>>  
>> +static void virtio_irq(VirtQueue *vq)
>> +{
>> +    virtio_set_isr(vq->vdev, 0x1);
>> +    virtio_notify_vector(vq->vdev, vq->vector);
>> +}
>> +
>>  void virtio_notify(VirtIODevice *vdev, VirtQueue *vq)
>>  {
>>      if (!virtio_should_notify(vdev, vq)) {
>> @@ -1448,8 +1454,7 @@ void virtio_notify(VirtIODevice *vdev, VirtQueue *vq)
>>      }
>>  
>>      trace_virtio_notify(vdev, vq);
>> -    virtio_set_isr(vq->vdev, 0x1);
>> -    virtio_notify_vector(vdev, vq->vector);
>> +    virtio_irq(vq);
>>  }
>>  
>>  void virtio_notify_config(VirtIODevice *vdev)
>> @@ -2082,7 +2087,7 @@ static void virtio_queue_guest_notifier_read(EventNotifier *n)
>>  {
>>      VirtQueue *vq = container_of(n, VirtQueue, guest_notifier);
>>      if (event_notifier_test_and_clear(n)) {
>> -        virtio_notify_vector(vq->vdev, vq->vector);
>> +        virtio_irq(vq);
>>      }
>>  }
>>  
>> -- 
>> 2.0.4
>>

Re: [Qemu-devel] [PATCH V2] virtio: Fix no interrupt when not creating msi controller

[Michael S. Tsirkin]

On Mon, Jan 23, 2017 at 05:46:47PM +0100, Paolo Bonzini wrote:
> 
> 
> On 23/01/2017 17:07, Michael S. Tsirkin wrote:
> > On Mon, Jan 23, 2017 at 08:55:04AM +0800, Shannon Zhao wrote:
> >> From: Shannon Zhao <shannon.zhao@linaro.org>
> >>
> >> For ARM virt machine, if we use virt-2.7 which will not create ITS node,
> >> the virtio-net can not recieve interrupts so it can't get ip address
> >> through dhcp.
> >> This fixes commit 83d768b(virtio: set ISR on dataplane notifications).
> >>
> >> Signed-off-by: Shannon Zhao <shannon.zhao@linaro.org>
> > 
> > Paolo, could you comment on this pls?
> 
> For virtio-blk, my patch moved the setting of ISR from
> virtio_queue_guest_notifier_read to virtio_notify_irqfd.  This is
> because the irqfd emulation only needs to trigger the interrupt.
> Setting the ISR should have been done elsewhere.
> 
> Can vhost set ISR (I thought it couldn't)?  If so, it seems like ARM was
> relying on QEMU's irqfd emulation, as a work around for vhost not
> setting ISR.  But this only works if irqfd is directed to QEMU and not
> to KVM.  So if vhost cannot set ISR, I think vhost should be disabled
> unless MSI is active.
> 
> Am I missing something?
> 
> Paolo

vhost doesn't set ISR ATM.  Without MSI we are always bouncing
interrupts through QEMU in particular in order to set ISR.

Disabling vhost when user requested it was what we did
at some point but this just confused people.


> > 
> >> ---
> >> V2: Factor out a common function instead of duplicating code
> >> ---
> >>  hw/virtio/virtio.c | 11 ++++++++---
> >>  1 file changed, 8 insertions(+), 3 deletions(-)
> >>
> >> diff --git a/hw/virtio/virtio.c b/hw/virtio/virtio.c
> >> index aa4f38f..00e8f74 100644
> >> --- a/hw/virtio/virtio.c
> >> +++ b/hw/virtio/virtio.c
> >> @@ -1441,6 +1441,12 @@ void virtio_notify_irqfd(VirtIODevice *vdev, VirtQueue *vq)
> >>      event_notifier_set(&vq->guest_notifier);
> >>  }
> >>  
> >> +static void virtio_irq(VirtQueue *vq)
> >> +{
> >> +    virtio_set_isr(vq->vdev, 0x1);
> >> +    virtio_notify_vector(vq->vdev, vq->vector);
> >> +}
> >> +
> >>  void virtio_notify(VirtIODevice *vdev, VirtQueue *vq)
> >>  {
> >>      if (!virtio_should_notify(vdev, vq)) {
> >> @@ -1448,8 +1454,7 @@ void virtio_notify(VirtIODevice *vdev, VirtQueue *vq)
> >>      }
> >>  
> >>      trace_virtio_notify(vdev, vq);
> >> -    virtio_set_isr(vq->vdev, 0x1);
> >> -    virtio_notify_vector(vdev, vq->vector);
> >> +    virtio_irq(vq);
> >>  }
> >>  
> >>  void virtio_notify_config(VirtIODevice *vdev)
> >> @@ -2082,7 +2087,7 @@ static void virtio_queue_guest_notifier_read(EventNotifier *n)
> >>  {
> >>      VirtQueue *vq = container_of(n, VirtQueue, guest_notifier);
> >>      if (event_notifier_test_and_clear(n)) {
> >> -        virtio_notify_vector(vq->vdev, vq->vector);
> >> +        virtio_irq(vq);
> >>      }
> >>  }
> >>  
> >> -- 
> >> 2.0.4
> >>

Re: [Qemu-devel] [PATCH V2] virtio: Fix no interrupt when not creating msi controller

[Paolo Bonzini]

On 23/01/2017 18:04, Michael S. Tsirkin wrote:
>> For virtio-blk, my patch moved the setting of ISR from
>> virtio_queue_guest_notifier_read to virtio_notify_irqfd.  This is
>> because the irqfd emulation only needs to trigger the interrupt.
>> Setting the ISR should have been done elsewhere.
>>
>> Can vhost set ISR (I thought it couldn't)?  If so, it seems like ARM was
>> relying on QEMU's irqfd emulation, as a work around for vhost not
>> setting ISR.  But this only works if irqfd is directed to QEMU and not
>> to KVM.  So if vhost cannot set ISR, I think vhost should be disabled
>> unless MSI is active.
>>
>> Am I missing something?
>
> vhost doesn't set ISR ATM.  Without MSI we are always bouncing
> interrupts through QEMU in particular in order to set ISR.

Where is the code that disables KVM irqfd?

Paolo

> Disabling vhost when user requested it was what we did
> at some point but this just confused people.
> 
> 

Re: [Qemu-devel] [PATCH V2] virtio: Fix no interrupt when not creating msi controller

[Michael S. Tsirkin]

On Tue, Jan 24, 2017 at 10:25:00AM +0100, Paolo Bonzini wrote:
> 
> 
> On 23/01/2017 18:04, Michael S. Tsirkin wrote:
> >> For virtio-blk, my patch moved the setting of ISR from
> >> virtio_queue_guest_notifier_read to virtio_notify_irqfd.  This is
> >> because the irqfd emulation only needs to trigger the interrupt.
> >> Setting the ISR should have been done elsewhere.
> >>
> >> Can vhost set ISR (I thought it couldn't)?  If so, it seems like ARM was
> >> relying on QEMU's irqfd emulation, as a work around for vhost not
> >> setting ISR.  But this only works if irqfd is directed to QEMU and not
> >> to KVM.  So if vhost cannot set ISR, I think vhost should be disabled
> >> unless MSI is active.
> >>
> >> Am I missing something?
> >
> > vhost doesn't set ISR ATM.  Without MSI we are always bouncing
> > interrupts through QEMU in particular in order to set ISR.
> 
> Where is the code that disables KVM irqfd?
> 
> Paolo


This bit in virtio_pci_set_guest_notifiers does the trick I think:


    bool with_irqfd = msix_enabled(&proxy->pci_dev) &&
        kvm_msi_via_irqfd_enabled();

I don't think we ever used irqfd for level interrupts.

> > Disabling vhost when user requested it was what we did
> > at some point but this just confused people.
> > 
> > 

Re: [Qemu-devel] [PATCH V2] virtio: Fix no interrupt when not creating msi controller

[Paolo Bonzini]

On 24/01/2017 15:44, Michael S. Tsirkin wrote:
> On Tue, Jan 24, 2017 at 10:25:00AM +0100, Paolo Bonzini wrote:
>>
>>
>> On 23/01/2017 18:04, Michael S. Tsirkin wrote:
>>>> For virtio-blk, my patch moved the setting of ISR from
>>>> virtio_queue_guest_notifier_read to virtio_notify_irqfd.  This is
>>>> because the irqfd emulation only needs to trigger the interrupt.
>>>> Setting the ISR should have been done elsewhere.
>>>>
>>>> Can vhost set ISR (I thought it couldn't)?  If so, it seems like ARM was
>>>> relying on QEMU's irqfd emulation, as a work around for vhost not
>>>> setting ISR.  But this only works if irqfd is directed to QEMU and not
>>>> to KVM.  So if vhost cannot set ISR, I think vhost should be disabled
>>>> unless MSI is active.
>>>>
>>>> Am I missing something?
>>>
>>> vhost doesn't set ISR ATM.  Without MSI we are always bouncing
>>> interrupts through QEMU in particular in order to set ISR.
>>
>> Where is the code that disables KVM irqfd?
>
> This bit in virtio_pci_set_guest_notifiers does the trick I think:
> 
> 
>     bool with_irqfd = msix_enabled(&proxy->pci_dev) &&
>         kvm_msi_via_irqfd_enabled();
> 
> I don't think we ever used irqfd for level interrupts.

Ok, thanks!

So we have four cases:

- assign with emulated irqfd + ISR
- assign with emulated irqfd
- assign with irqfd
- deassign

Currently we don't distinguish the first two, so (which I don't like)
Shannon's patch ends up setting ISR twice.  The only (ugly) solution
that comes to mind is making with_irqfd an enum...

Paolo

Re: [Qemu-devel] [PATCH V2] virtio: Fix no interrupt when not creating msi controller

[Michael S. Tsirkin]

On Tue, Jan 24, 2017 at 04:00:55PM +0100, Paolo Bonzini wrote:
> 
> 
> On 24/01/2017 15:44, Michael S. Tsirkin wrote:
> > On Tue, Jan 24, 2017 at 10:25:00AM +0100, Paolo Bonzini wrote:
> >>
> >>
> >> On 23/01/2017 18:04, Michael S. Tsirkin wrote:
> >>>> For virtio-blk, my patch moved the setting of ISR from
> >>>> virtio_queue_guest_notifier_read to virtio_notify_irqfd.  This is
> >>>> because the irqfd emulation only needs to trigger the interrupt.
> >>>> Setting the ISR should have been done elsewhere.
> >>>>
> >>>> Can vhost set ISR (I thought it couldn't)?  If so, it seems like ARM was
> >>>> relying on QEMU's irqfd emulation, as a work around for vhost not
> >>>> setting ISR.  But this only works if irqfd is directed to QEMU and not
> >>>> to KVM.  So if vhost cannot set ISR, I think vhost should be disabled
> >>>> unless MSI is active.
> >>>>
> >>>> Am I missing something?
> >>>
> >>> vhost doesn't set ISR ATM.  Without MSI we are always bouncing
> >>> interrupts through QEMU in particular in order to set ISR.
> >>
> >> Where is the code that disables KVM irqfd?
> >
> > This bit in virtio_pci_set_guest_notifiers does the trick I think:
> > 
> > 
> >     bool with_irqfd = msix_enabled(&proxy->pci_dev) &&
> >         kvm_msi_via_irqfd_enabled();
> > 
> > I don't think we ever used irqfd for level interrupts.
> 
> Ok, thanks!
> 
> So we have four cases:
> 
> - assign with emulated irqfd + ISR
> - assign with emulated irqfd
> - assign with irqfd
> - deassign
> 
> Currently we don't distinguish the first two, so (which I don't like)
> Shannon's patch ends up setting ISR twice.

I don't see this yet - where is it set twice?

> The only (ugly) solution
> that comes to mind is making with_irqfd an enum...
> 
> Paolo

Re: [Qemu-devel] [PATCH V2] virtio: Fix no interrupt when not creating msi controller

[Paolo Bonzini]

On 24/01/2017 16:05, Michael S. Tsirkin wrote:
>>
>> So we have four cases:
>>
>> - assign with emulated irqfd + ISR
>> - assign with emulated irqfd
>> - assign with irqfd
>> - deassign
>>
>> Currently we don't distinguish the first two, so (which I don't like)
>> Shannon's patch ends up setting ISR twice.
> 
> I don't see this yet - where is it set twice?

Once in virtio_notify_irqfd, once in virtio_queue_guest_notifier_read.

Unfortunately, for virtio-blk + MSI + KVM + old Windows drivers we need
the one in virtio_notify_irqfd.  For virtio-net + vhost + INTx we need
the one in virtio_queue_guest_notifier_read.

Re: [Qemu-devel] [PATCH V2] virtio: Fix no interrupt when not creating msi controller

[Michael S. Tsirkin]

On Tue, Jan 24, 2017 at 04:50:44PM +0100, Paolo Bonzini wrote:
> 
> 
> On 24/01/2017 16:05, Michael S. Tsirkin wrote:
> >>
> >> So we have four cases:
> >>
> >> - assign with emulated irqfd + ISR
> >> - assign with emulated irqfd
> >> - assign with irqfd
> >> - deassign
> >>
> >> Currently we don't distinguish the first two, so (which I don't like)
> >> Shannon's patch ends up setting ISR twice.
> > 
> > I don't see this yet - where is it set twice?
> 
> Once in virtio_notify_irqfd, once in virtio_queue_guest_notifier_read.
> 
> Unfortunately, for virtio-blk + MSI + KVM + old Windows drivers we need
> the one in virtio_notify_irqfd.  For virtio-net + vhost + INTx we need
> the one in virtio_queue_guest_notifier_read.

Well in MSI mode virtio_queue_guest_notifier_read so we only do this
once, right? I need to check INTx mode - does virtio_notify_irqfd
get called then?

-- 
MST