Re: Regression in io_uring, leading to data corruption

[Timothy Pearson <tpearson@raptorengineering.com>]

----- Original Message -----
> From: "Timothy Pearson" <tpearson@raptorengineering.com>
> To: "Timothy Pearson" <tpearson@raptorengineering.com>
> Cc: "Jens Axboe" <axboe@kernel.dk>, "regressions" <regressions@lists.linux.dev>, "Pavel Begunkov"
> <asml.silence@gmail.com>
> Sent: Saturday, November 11, 2023 3:57:23 PM
> Subject: Re: Regression in io_uring, leading to data corruption
> 
> Unfortunately I got led down a rabbit hole here.  With the tests I was running,
> MariaDB writes the encrypted data separately from the normal un-encrypted page
> header and checksums, and the internal encrpyted data on disk was corrupt while
> the outer page checksum was still valid.
> 
> I have since switched to the main.xa_prepared_binlog_off test, which shows the
> corruption more easily in the on-disk format.  We are still apparently dealing
> with a write path issue, which makes more sense given the nature of the
> corruption observed on production systems.

Quick status update -- after considerable effort applied I've managed to narrow down what is going wrong, but still need to locate the root cause.  The bug is incredibly timing-dependent, therefore it is difficult to instrument the code paths I need without causing it to disappear.

What we're dealing with is a wild write to RAM of some sort, provoked by the exact timing of some of the encryption tests in mariadb.  I've caught the wild write a few times now, it is not in the standard io_uring write path but instead appears to be triggered (somehow) by the io worker punting process.

When the bug is hit, and if all other conditions are exactly correct, *something* (still to be identified) writes 32 bytes of gibberish into one of the mariadb in-RAM database pages at a random offset.  This wild write occurs right before the page is encrypted for write to disk via io_uring.  I have confirmed that the post-encryption data in RAM is written to disk without any additional corruption, and is then read back out from disk into the page verification routine also without any additional corruption.  The page verification routine decrypts the data from disk, thus restoring the decrypted data that contains the wild write data stamped somewhere on it, where we then hit the corruption warning and halt the test run.

Irritatingly, if I try to instrument the data flow in the application right before the encryption routine, the bug disappears (or, more precisely, is masked).  If I had to guess from these symptoms, I'd suspect the application io worker thread is waking up, grabbing wrong context from somewhere, and scribbling some kind of status data into memory, which rarely ends up being on top of one of the in-RAM database pages.  This could be an application issue or a kernel issue, I'm not sure yet, but given the precise timing requirements I'm less and less surprised this is only showing on ppc64 right now.

As always, investigation continues...