Re: Regression in io_uring, leading to data corruption

[Timothy Pearson <tpearson@raptorengineering.com>]

----- Original Message -----
> From: "Jens Axboe" <axboe@kernel.dk>
> To: "Timothy Pearson" <tpearson@raptorengineering.com>, "regressions" <regressions@lists.linux.dev>, "Pavel Begunkov"
> <asml.silence@gmail.com>
> Sent: Tuesday, November 7, 2023 10:49:34 AM
> Subject: Re: Regression in io_uring, leading to data corruption

> On 11/7/23 9:34 AM, Timothy Pearson wrote:
>> I have spent some considerable effort tracking down a bug that appears
>> to be present in the io_uring workqueue.  As I have not yet been able
>> to isolate the exact cause, I would like to solicit ideas from the
>> developers / maintainers of the io_uring system.  This regression
>> persists into the latest kernel GIT head, and is only reliably
>> reproduceable under fairly exacting conditions.
>> 
>> In GIT hash 685fe7fe the workqueue manager thread was removed and
>> replaced with code that allows the workqueues to manage their own
>> workers.  This has the unfortunate side effect of exposing what I
>> believe to be an existing timing-dependent race condition somewhere
>> else within the kernel.  On a ppc64el host, I can reliably trigger
>> data corruption on what I believe to be writes by running the
>> following mysql mtr sequence:
>> 
>> ./mtr encryption.innodb-discard-import --repeat=100 --force
>> 
>> This results in corruption of the data being written to disk --
>> reverting 685fe7fe resolves the issue by (I believe) masking it
>> through changes in workqueue inter-thread timing.
>> 
>> I can make the corruption disappear by adding a 1ms busy wait delay
>> into io_wqe_dec_running().  This appears to alter the timing of
>> something in the io_uring system just enough to make the (presumed)
>> data race disappear.  KASAN and KCSAN do not show any issues, nor does
>> the lock debugger, yet a corruption problem that disappears with a
>> delay is indicative of a race somewhere.  The delay primary impacts
>> how long the IRQ lock is held, if the delay is moved outside of the
>> IRQ locked section the corruption returns.
>> 
>> I have already tried adding memory barriers etc. to the code paths in
>> question, with no effect.  The exact same issue persists on the latest
>> kernel versions.
>> 
>> Thoughts welcome -- this is a serious issue causing data corruption on
>> production systems.
> 
> I looked into this for quite a while back in March, see my initial
> postings on it here:
> 
> https://lore.kernel.org/all/2b015a34-220e-674e-7301-2cf17ef45ed9@kernel.dk/
> 
> it unfortunately never got anywhere, and as far as I can tell, this is
> most likely a page cache or ordering issue on the ppc side. I no longer
> have hardware to test with, and not really a huge inclination to dive
> into this again as it's hugely time consuming and doesn't seem to be an
> io_uring issue to begin with, but I'd be happy to help out with this.
> 
> Back then I looked into getting some ppc hardware to test with for
> this very reason, and even reached out to various manufacturers to see
> if they would be able to lend/give me some. Didn't pan out, and ended
> up using a university vm for it.
> 
> --
> Jens Axboe

Understood.  I think between the pinning and the findings above, plus the fact that (IIRC) this seemed to disappear in SMT1 mode, I may have some better idea of where to look.  The pinning "fixing" things is something I wasn't aware of and will significantly reduce debug effort on this end, thanks for the pointer!

In the future, Raptor is more than willing to offer bare metal access to test machines for ppc64el at no cost.  I was unaware of the need so couldn't respond.

Thanks again!