Re: Regression in io_uring, leading to data corruption

[Timothy Pearson <tpearson@raptorengineering.com>]

----- Original Message -----
> From: "Timothy Pearson" <tpearson@raptorengineeringinc.com>
> To: "Jens Axboe" <axboe@kernel.dk>
> Cc: "regressions" <regressions@lists.linux.dev>, "Pavel Begunkov" <asml.silence@gmail.com>
> Sent: Tuesday, November 7, 2023 9:27:13 PM
> Subject: Re: Regression in io_uring, leading to data corruption

> ----- Original Message -----
>> From: "Jens Axboe" <axboe@kernel.dk>
>> To: "Timothy Pearson" <tpearson@raptorengineering.com>
>> Cc: "regressions" <regressions@lists.linux.dev>, "Pavel Begunkov"
>> <asml.silence@gmail.com>
>> Sent: Tuesday, November 7, 2023 6:09:39 PM
>> Subject: Re: Regression in io_uring, leading to data corruption
> 
>> On 11/7/23 5:02 PM, Timothy Pearson wrote:
>>> 
>>> 
>>> ----- Original Message -----
>>>> From: "Jens Axboe" <axboe@kernel.dk>
>>>> To: "Timothy Pearson" <tpearson@raptorengineering.com>
>>>> Cc: "regressions" <regressions@lists.linux.dev>, "Pavel Begunkov"
>>>> <asml.silence@gmail.com>
>>>> Sent: Tuesday, November 7, 2023 5:52:54 PM
>>>> Subject: Re: Regression in io_uring, leading to data corruption
>>> 
>>>> On 11/7/23 4:34 PM, Timothy Pearson wrote:
>>>>>
>>>>>
>>>>> ----- Original Message -----
>>>>>> From: "Jens Axboe" <axboe@kernel.dk>
>>>>>> To: "Timothy Pearson" <tpearson@raptorengineering.com>
>>>>>> Cc: "regressions" <regressions@lists.linux.dev>, "Pavel Begunkov"
>>>>>> <asml.silence@gmail.com>
>>>>>> Sent: Tuesday, November 7, 2023 5:16:24 PM
>>>>>> Subject: Re: Regression in io_uring, leading to data corruption
>>>>>>
>>>>>> But since you said you're seeing exits on signal 9, that would seem to
>>>>>> indicate that someone ran SIGKILL with potentially pending IO.
>>>>>
>>>>> Just to make sure we're talking about the same thing, when I refer to
>>>>> signal 9 I am referring to instrumentation I added to the
>>>>> get_signal(&ksig) call.  I am seeing 9 in the ksig.sig field.  I
>>>>> assume the SIGKILL is not expected, i.e. not coming from another
>>>>> location in the io_uring kernel code?
>>>>
>>>> ksig.sig is set to signr, which is the number of the signal. So that
>>>> would seem to indicate that _someone_ is sending SIGKILL. But at the
>>>> same time, you don't see any of the cancel-on-exit triggering. Puzzling!
>>>> io_uring or io-wq doesn't send any signals.
>>>>
>>>> In your instrumentation, are you checking where the signal is coming
>>>> from?Is it being dequeued as an actual signal, or is it some other
>>>> condition in get_signal() that ends up setting it to SIGKILL?
>>> 
>>> I still need to check this in more detail.  What I do know at the
>>> moment is that the kill seems to be related to the entire workqueue
>>> going down, i.e. it is sent to the workers of a given workqueue after
>>> io_wq_put_and_exit() and io_wq_exit_workers() are called for that
>>> workqueue.  Not sure if that helps any, will keep digging...
>> 
>> Another option is that it's doing exec() with pending IO, which will
>> cancel it. But I'm also assuming that mtr/friends will check result
>> values of writes and would've caught that there.
> 
> So I ran it a few more times and we do actually hit the WARN_ON_ONCE(do_kill) in
> io_worker_handle_work().  It's possible we hit it much earlier in the boot
> process before and I simply missed it, since it's only printed once per boot.
> If you have a potential patch or avenue of testing based on that information
> I'm happy to try it.

Actually, I take that back.  I had other debugging in place that tripped that condition.  Investigation continues...