Re: [Qemu-devel] [PATCH 1/1] virtio: fallback from irqfd to non-irqfd notify

[Halil Pasic <pasic@linux.vnet.ibm.com>]

On 03/01/2017 01:57 PM, Paolo Bonzini wrote:
> 
> 
> On 01/03/2017 12:50, Halil Pasic wrote:
>> The commits 03de2f527 "virtio-blk: do not use vring in dataplane"  and
>> 9ffe337c08 "virtio-blk: always use dataplane path if ioeventfd is active"
>> changed how notifications are done for virtio-blk substantially. Due to a
>> race condition interrupts are lost when irqfd is torn down after
>> notify_guest_bh was scheduled but before it actually runs.
> 
> I don't think the non-irqfd notification mechanism is thread safe, and
> that would be a problem for this patch.

Sorry, haven't looked into that thoroughly (and speculated people with
more understanding will jump in).

> 
> What is the path that causes the irqfd to be torn down?  Only something

Here a trace:

135871@1488304024.512533:virtio_blk_req_complete req 0x2aa6b117e10 status 0
135871@1488304024.512541:virtio_notify_irqfd vdev 0x2aa6b0e19d8 vq 0x2aa6b4c0870
135871@1488304024.522607:virtio_blk_req_complete req 0x2aa6b118980 status 0
135871@1488304024.522616:virtio_blk_req_complete req 0x2aa6b119260 status 0
135871@1488304024.522627:virtio_notify_irqfd vdev 0x2aa6b0e19d8 vq 0x2aa6b4c0870
135871@1488304024.527386:virtio_blk_req_complete req 0x2aa6b118980 status 0
135871@1488304024.527431:virtio_notify_irqfd vdev 0x2aa6b0e19d8 vq 0x2aa6b4c0870
135871@1488304024.528611:virtio_guest_notifier_read vdev 0x2aa6b0e61c8 vq 0x2aa6b4de880
135871@1488304024.528628:virtio_guest_notifier_read vdev 0x2aa6b0e61c8 vq 0x2aa6b4de8f8
135871@1488304024.528753:virtio_blk_data_plane_stop dataplane 0x2aa6b0e5540
                         ^== DATAPLANE STOP  
135871@1488304024.530709:virtio_blk_req_complete req 0x2aa6b117e10 status 0
135871@1488304024.530752:virtio_guest_notifier_read vdev 0x2aa6b0e19d8 vq 0x2aa6b4c0870
                         ^== comes from k->set_guest_notifiers(qbus->parent, nvqs, false);
                             in virtio_blk_data_plane_stop and done immediately after
                             irqfd is cleaned up by the transport
135871@1488304024.530836:virtio_notify_irqfd vdev 0x2aa6b0e19d8 vq 0x2aa6b4c0870
halil: error in event_notifier_set: Bad file descriptor
                         ^== here we have the problem

If you want a stacktrace that can be arranged to.

> like a reset should cause it (the only call in virtio-blk is from
> virtio_blk_data_plane_stop), and then the guest doesn't care anymore
> about interrupts.

I do not understand this with 'doesn't care anymore about interrupts'.
I was debugging a virtio-blk device being stuck waiting for a host
notification (interrupt) after migration.

> 
> That path also does a qemu_bh_delete, so the notify_guest_bh should not
> be invoked at all.
> 

That's only for destroy. I'm migrating.

Seems I tried to fix this is the wrong way. Was not too confident about it
in the first place. Suggestions welcome!

Cheers,
Halil