aio_wait_bh_oneshot() thread-safety question

aio_wait_bh_oneshot() thread-safety question

[Vladimir Sementsov-Ogievskiy]

Hi all (yes, that's my new address, I hope for a long time. )

I have a doubt about how aio_wait_bh_oneshot() works. Exactly, I see that data->done is not accessed atomically, and doesn't have any barrier protecting it..

Is following possible:

main-loop                           iothread
                                 |
aio_wait_bh_oneshot()           |
     aio_bh_schedule_oneshot()   |
                                 |  handle bh:
                                 | 1. set data->done = true
                                 | 2. call aio_wait_kick(), inserting the
                                 | dummy bh into main context
                                 |
  ... in AIO_WAIT_WHILE():
    handle dummy bh, go to next
    iteration, but still read
    data->done=false due to some
    processor data reordering,
    go to next iteration of polling
    and hang

?

I've seen a following dead-lock on 2.12-based Qemu, but failed to find is it (and how is it) fixed in master:

1. main() thread is stuck in qemu_mutex_lock_iothread()

2. The global mutex is taken by migration_thread(), which has the following stack:

aio_poll ( ctx=qemu_aio_context, blocking=true )
aio_wait_bh_oneshot ( ctx=context_of_iothread, cb=virtio_blk_data_plane_stop_bh )
virtio_blk_data_plane_stop
virtio_bus_stop_ioeventfd
virtio_vmstate_change
vm_state_notify
do_vm_stop
migration_completion

The iothread itself is in qemu_poll_ns() -> ppoll(). data->done of the BH is true, so I assume iothread completely handled the BH. Also, there is no dummy_bh in the main qemu aio context bh-list, so I assume it's either already handled, or aio_wait_kick() was called even before entering AIO_WAIT_WHILE. But still, AIO_WAIT_WHILE somehow go into block aio_poll, like data->done was false.

-- 
Best regards,
Vladimir

Re: aio_wait_bh_oneshot() thread-safety question

[Paolo Bonzini]

On 5/23/22 18:04, Vladimir Sementsov-Ogievskiy wrote:
> 
> I have a doubt about how aio_wait_bh_oneshot() works. Exactly, I see 
> that data->done is not accessed atomically, and doesn't have any barrier 
> protecting it..
> 
> Is following possible:
> 
> main-loop                           iothread
>                                  |
> aio_wait_bh_oneshot()           |
>      aio_bh_schedule_oneshot()   |
>                                  |  handle bh:
>                                  | 1. set data->done = true
>                                  | 2. call aio_wait_kick(), inserting the
>                                  | dummy bh into main context
>                                  |
>   ... in AIO_WAIT_WHILE():
>     handle dummy bh, go to next
>     iteration, but still read
>     data->done=false due to some
>     processor data reordering,
>     go to next iteration of polling
>     and hang
Yes, barriers are missing:

https://lore.kernel.org/qemu-devel/You6FburTi7gVyxy@stefanha-x1.localdomain/T/#md97146c6eae1fce2ddd687fdc3f2215eee03f6f4

It seems like the issue was never observed, at least on x86.

Paolo

Re: aio_wait_bh_oneshot() thread-safety question

[Kevin Wolf]

Am 24.05.2022 um 09:08 hat Paolo Bonzini geschrieben:
> On 5/23/22 18:04, Vladimir Sementsov-Ogievskiy wrote:
> > 
> > I have a doubt about how aio_wait_bh_oneshot() works. Exactly, I see
> > that data->done is not accessed atomically, and doesn't have any barrier
> > protecting it..
> > 
> > Is following possible:
> > 
> > main-loop                           iothread
> >                                  |
> > aio_wait_bh_oneshot()           |
> >      aio_bh_schedule_oneshot()   |
> >                                  |  handle bh:
> >                                  | 1. set data->done = true
> >                                  | 2. call aio_wait_kick(), inserting the
> >                                  | dummy bh into main context
> >                                  |
> >   ... in AIO_WAIT_WHILE():
> >     handle dummy bh, go to next
> >     iteration, but still read
> >     data->done=false due to some
> >     processor data reordering,
> >     go to next iteration of polling
> >     and hang
> Yes, barriers are missing:
> 
> https://lore.kernel.org/qemu-devel/You6FburTi7gVyxy@stefanha-x1.localdomain/T/#md97146c6eae1fce2ddd687fdc3f2215eee03f6f4
> 
> It seems like the issue was never observed, at least on x86.

Why is the barrier in aio_bh_enqueue() not enough? Is the comment there
wrong?

aio_notify() has another barrier. This is a little bit too late, but if
I misunderstood the aio_bh_enqueue() one, it could explain why it was
never observed.

Kevin

Re: aio_wait_bh_oneshot() thread-safety question

[Vladimir Sementsov-Ogievskiy]

On 5/24/22 15:40, Kevin Wolf wrote:
> Am 24.05.2022 um 09:08 hat Paolo Bonzini geschrieben:
>> On 5/23/22 18:04, Vladimir Sementsov-Ogievskiy wrote:
>>>
>>> I have a doubt about how aio_wait_bh_oneshot() works. Exactly, I see
>>> that data->done is not accessed atomically, and doesn't have any barrier
>>> protecting it..
>>>
>>> Is following possible:
>>>
>>> main-loop                           iothread
>>>                                   |
>>> aio_wait_bh_oneshot()           |
>>>       aio_bh_schedule_oneshot()   |
>>>                                   |  handle bh:
>>>                                   | 1. set data->done = true
>>>                                   | 2. call aio_wait_kick(), inserting the
>>>                                   | dummy bh into main context
>>>                                   |
>>>    ... in AIO_WAIT_WHILE():
>>>      handle dummy bh, go to next
>>>      iteration, but still read
>>>      data->done=false due to some
>>>      processor data reordering,
>>>      go to next iteration of polling
>>>      and hang
>> Yes, barriers are missing:
>>
>> https://lore.kernel.org/qemu-devel/You6FburTi7gVyxy@stefanha-x1.localdomain/T/#md97146c6eae1fce2ddd687fdc3f2215eee03f6f4
>>
>> It seems like the issue was never observed, at least on x86.
> 
> Why is the barrier in aio_bh_enqueue() not enough? Is the comment there
> wrong?
> 
> aio_notify() has another barrier. This is a little bit too late, but if
> I misunderstood the aio_bh_enqueue() one, it could explain why it was
> never observed.
> 
> Kevin
> 

I'd consider two cases:

1. aio_wait_kick() reads num_waiters as 0 and don't schedule any BH into main ctx.

In this case aio_wait_kick() only do one atomic operation: qatomic_read(&global_aio_wait.num_waiters), which is not a barrier as I understand.
So, data->done=true may be reordered with this operation.

main-loop                                iothread

   aio_wait_bh_oneshot()          |
      aio_bh_schedule_oneshot()   |
                                  |  atomic read num_waiters = 0 => don't kick
      AIO_WAIT_WHILE              |
       atomic inc num_waiters     |
       read done = false, go      |
       into blocking aio_poll()   |
                                  |  set data->done = true  # reordered to the end
                                  |    - but that doesn't help to wake main loop


For this case, iothread just don't call aio_bh_enqueue() and aio_notify(), so any barriers in them doesn't help


2. aio_wait_kick() reads num_waiters>0 and do schedule BH

In this case it seems you are right: if main-loop dequeued  dummy BH, it should be guaranteed that after handling this BH the main loop will see data->done=true.. That's if the comment is correct, hope it is. At least it corresponds to what I've read here : https://www.kernel.org/doc/Documentation/atomic_t.txt . How much generic this information is - I don't know.

In 2.12 there was no enque() deque() functions, but there was smp_wmb() in aio_bh_schedule_oneshot(), paired with atomic_xchg() in aio_bh_poll(), with similar comment about implicit barrier.


-- 
Best regards,
Vladimir

Re: aio_wait_bh_oneshot() thread-safety question

[Paolo Bonzini]

On 5/24/22 14:40, Kevin Wolf wrote:
> Why is the barrier in aio_bh_enqueue() not enough? Is the comment there
> wrong?
> 
> aio_notify() has another barrier. This is a little bit too late, but if
> I misunderstood the aio_bh_enqueue() one, it could explain why it was
> never observed.

The missing one that I (and I think Vladimir) were talking about is at the
end of the execution of the bottom half, not at the beginning:

/* Context: BH in IOThread */
static void aio_wait_bh(void *opaque)
{
     AioWaitBHData *data = opaque;

     data->cb(data->opaque);

     data->done = true;
     aio_wait_kick();
}

void aio_wait_kick(void)
{
     /* The barrier (or an atomic op) is in the caller.  */
     if (qatomic_read(&global_aio_wait.num_waiters)) {
         aio_bh_schedule_oneshot(qemu_get_aio_context(), dummy_bh_cb, NULL);
     }
}

where there is no barrier in the caller to separate reading data->done
(qatomic_set would be nice, if only for clarity) from reading num_waiters.

Paolo

Re: aio_wait_bh_oneshot() thread-safety question

[Emanuele Giuseppe Esposito]

Am 24/05/2022 um 09:08 schrieb Paolo Bonzini:
> On 5/23/22 18:04, Vladimir Sementsov-Ogievskiy wrote:
>>
>> I have a doubt about how aio_wait_bh_oneshot() works. Exactly, I see
>> that data->done is not accessed atomically, and doesn't have any
>> barrier protecting it..
>>
>> Is following possible:
>>
>> main-loop                           iothread
>>                                  |
>> aio_wait_bh_oneshot()           |
>>      aio_bh_schedule_oneshot()   |
>>                                  |  handle bh:
>>                                  | 1. set data->done = true
>>                                  | 2. call aio_wait_kick(), inserting the
>>                                  | dummy bh into main context
>>                                  |
>>   ... in AIO_WAIT_WHILE():
>>     handle dummy bh, go to next
>>     iteration, but still read
>>     data->done=false due to some
>>     processor data reordering,
>>     go to next iteration of polling
>>     and hang
> Yes, barriers are missing:
> 
> https://lore.kernel.org/qemu-devel/You6FburTi7gVyxy@stefanha-x1.localdomain/T/#md97146c6eae1fce2ddd687fdc3f2215eee03f6f4
> 
> 
> It seems like the issue was never observed, at least on x86.
> 
> Paolo
> 

Sent the fix as a separate patch:
https://patchew.org/QEMU/20220524173054.12651-1-eesposit@redhat.com/

Thank you,
Emanuele