Re: Regression in io_uring, leading to data corruption

[Jens Axboe <axboe@kernel.dk>]

On 11/7/23 3:29 PM, Timothy Pearson wrote:
> 
> 
> ----- Original Message -----
>> From: "Jens Axboe" <axboe@kernel.dk>
>> To: "Timothy Pearson" <tpearson@raptorengineering.com>
>> Cc: "regressions" <regressions@lists.linux.dev>, "Pavel Begunkov" <asml.silence@gmail.com>
>> Sent: Tuesday, November 7, 2023 4:16:51 PM
>> Subject: Re: Regression in io_uring, leading to data corruption
> 
>> On 11/7/23 3:07 PM, Timothy Pearson wrote:
>>> Interestingly enough that's where my current investigation is leading
>>> as well.  After instrumenting and re-instrumenting the codebase far
>>> more times than I'd like to admit, I've noticed that the workers don't
>>> time out on kernel builds that show the corruption, they are directly
>>> terminated via signal (SIGKILL).  On kernel builds with the delay, the
>>> workers time out and self-terminate.  I'm still trying to parse out
>>> what the exact difference between these two mechanisms would be and
>>> how it plays into the corruption, but at least it's a start...
>>
>> Maybe poke at how they are exiting - you say timeout, so they've been
>> idle for a while and then go away? This would then cause worker creation
>> again later on if, say, we have 1 worker left and it goes to sleep. So
>> the timeout itself may not tell you much, outside of then causing that
>> other condition to happen. You could even try and shrink the timeout to
>> HZ / 10 or something like that to make it more likely to happen.
> 
> Agreed.  As of right now I can confirm that with the delay in place
> (no corruption) the workers are exiting on their own, no signals and
> no IO_EXIT bit being set.  When I remove the delay (reintroducing the
> corruption) I see signal 9 being sent to the workers, and a mix of
> IO_EXIT being set and not being set.
> 
> Ignoring the signal 9 does not fix the corruption, which makes me
> wonder more about IO_EXIT and whether things are not fully committed /
> properly torn down when the worker thread terminates.  This also
> dovetails nicely with the fact that the observed write corruption
> always seems to be in the latter portions of the page, never at the
> beginning of the page, also indicating rapid / unclean termination of
> the writer process.
> 
> Will keep digging...

This is useful. If the workers are exiting, they will try and process
work that is still pending. And it obviously does, or the process would
hang on exit or ring exit. But it'll also cancel said work, which
obviously did not happen for the old kthread scheme, as there was no way
to do that. So you'd just wait for it. Hence maybe what's happening here
is that mtr/mysql/mariadb isn't properly waiting for pending writes to
finish? It's just assuming that previously submitted writes will finish
if the task is killed?

What page size are you using?

-- 
Jens Axboe