Re: Regression in io_uring, leading to data corruption

[Jens Axboe <axboe@kernel.dk>]

On 11/8/23 10:40 AM, Timothy Pearson wrote:
> 
> 
> ----- Original Message -----
>> From: "Jens Axboe" <axboe@kernel.dk>
>> To: "Timothy Pearson" <tpearson@raptorengineering.com>
>> Cc: "regressions" <regressions@lists.linux.dev>, "Pavel Begunkov" <asml.silence@gmail.com>
>> Sent: Wednesday, November 8, 2023 11:26:53 AM
>> Subject: Re: Regression in io_uring, leading to data corruption
> 
>> On 11/8/23 10:10 AM, Timothy Pearson wrote:
>>> ----- Original Message -----
>>>> From: "Jens Axboe" <axboe@kernel.dk>
>>>> To: "Timothy Pearson" <tpearson@raptorengineering.com>
>>>> Cc: "regressions" <regressions@lists.linux.dev>, "Pavel Begunkov"
>>>> <asml.silence@gmail.com>
>>>> Sent: Wednesday, November 8, 2023 9:14:55 AM
>>>> Subject: Re: Regression in io_uring, leading to data corruption
>>>
>>>> On 11/8/23 8:10 AM, Jens Axboe wrote:
>>>>> It could also be a task that has pending IO and is doing exec() (and
>>>>> friends), this would also cancel inflight IO.
>>>>
>>>> If this is the case, then you could try with this one to just disable
>>>> that and see if the corruption goes away:
>>>
>>> Unfortunately that had no effect on the corruption.  I've also traced
>>> the signal generation into the get_signal() call, which is apparently
>>> sending SIGKILL when the thread group is marked for termination --
>>> this is in turn why the PID fields etc. are all zero.
>>
>> That's good news though, because I'm continually pondering why powerpc
>> is different here.
>>
>>> Investigation continues.
>>
>> If it's not exec, then it has to be a signal. I'm assuming you're
>> hitting this in get_signal():
>>
>> 		/* Has this task already been marked for death? */
>> 		if ((signal->flags & SIGNAL_GROUP_EXIT) ||
>> 		     signal->group_exec_task) {
>> 			clear_siginfo(&ksig->info);
>> 			ksig->info.si_signo = signr = SIGKILL;
>> 			sigdelset(&current->pending.signal, SIGKILL);
>> 			trace_signal_deliver(SIGKILL, SEND_SIG_NOINFO,
>> 				&sighand->action[SIGKILL - 1]);
>> 			recalc_sigpending();
>> 			goto fatal;
>> 		}
>>
>> which is either exec (which we verified it is not), so I don't see
>> anything other than this being a signal sent to mtr/mariadb for exit.
>>
>> Does this trigger? Doesn't necessarily indicate a bug as it would be
>> valid, but if it does trigger, perhaps io-wq has unstarted requests at
>> this point and they get canceled and hence never written. If this does
>> trigger, maybe try and do your sleep trick there too and see if that
>> gets rid of it.
> 
> Yes, it does indeed trigger.  Is there a way to directly check for the
> unstarted requests?

Let me hack up a debug patch for this, give me a minute.

-- 
Jens Axboe