Re: Regression in io_uring, leading to data corruption

[Timothy Pearson <tpearson@raptorengineering.com>]

----- Original Message -----
> From: "Jens Axboe" <axboe@kernel.dk>
> To: "Timothy Pearson" <tpearson@raptorengineering.com>
> Cc: "regressions" <regressions@lists.linux.dev>, "Pavel Begunkov" <asml.silence@gmail.com>
> Sent: Tuesday, November 7, 2023 4:16:51 PM
> Subject: Re: Regression in io_uring, leading to data corruption

> On 11/7/23 3:07 PM, Timothy Pearson wrote:
>> Interestingly enough that's where my current investigation is leading
>> as well.  After instrumenting and re-instrumenting the codebase far
>> more times than I'd like to admit, I've noticed that the workers don't
>> time out on kernel builds that show the corruption, they are directly
>> terminated via signal (SIGKILL).  On kernel builds with the delay, the
>> workers time out and self-terminate.  I'm still trying to parse out
>> what the exact difference between these two mechanisms would be and
>> how it plays into the corruption, but at least it's a start...
> 
> Maybe poke at how they are exiting - you say timeout, so they've been
> idle for a while and then go away? This would then cause worker creation
> again later on if, say, we have 1 worker left and it goes to sleep. So
> the timeout itself may not tell you much, outside of then causing that
> other condition to happen. You could even try and shrink the timeout to
> HZ / 10 or something like that to make it more likely to happen.

Agreed.  As of right now I can confirm that with the delay in place (no corruption) the workers are exiting on their own, no signals and no IO_EXIT bit being set.  When I remove the delay (reintroducing the corruption) I see signal 9 being sent to the workers, and a mix of IO_EXIT being set and not being set.

Ignoring the signal 9 does not fix the corruption, which makes me wonder more about IO_EXIT and whether things are not fully committed / properly torn down when the worker thread terminates.  This also dovetails nicely with the fact that the observed write corruption always seems to be in the latter portions of the page, never at the beginning of the page, also indicating rapid / unclean termination of the writer process.

Will keep digging...