Re: Regression in io_uring, leading to data corruption

[Timothy Pearson <tpearson@raptorengineering.com>]

----- Original Message -----
> From: "Jens Axboe" <axboe@kernel.dk>
> To: "Timothy Pearson" <tpearson@raptorengineering.com>, "regressions" <regressions@lists.linux.dev>, "Pavel Begunkov"
> <asml.silence@gmail.com>
> Sent: Tuesday, November 7, 2023 3:22:16 PM
> Subject: Re: Regression in io_uring, leading to data corruption

> On 11/7/23 9:34 AM, Timothy Pearson wrote:
>> I can make the corruption disappear by adding a 1ms busy wait delay
>> into io_wqe_dec_running().  This appears to alter the timing of
>> something in the io_uring system just enough to make the (presumed)
>> data race disappear.  KASAN and KCSAN do not show any issues, nor does
>> the lock debugger, yet a corruption problem that disappears with a
>> delay is indicative of a race somewhere.  The delay primary impacts
>> how long the IRQ lock is held, if the delay is moved outside of the
>> IRQ locked section the corruption returns.
> 
> This is interesting... This forces the current task to schedule out
> first, rather than potentially create a new worker.

That makes sense -- when debugging, everything seems to center around worker creation and the exact timing / context of that operation.

> If you put that 1 ms sleep at the top of io_wq_worker(), before the
> loop, does it fix it as well? With the previous one removed, of course.

No, it does not.

> Does it do anything if you just put isync() in there instead of the
> sleep in io_wqe_dec_running()?

Unfortunately no, that doesn't fix things either.

My gut says it's something racing with either the I/O worker creation or wake (trying to schedule work onto a half-created or still sleeping worker?) but so far I haven't been able to isolate the root cause.