Re: Regression in io_uring, leading to data corruption

[Jens Axboe <axboe@kernel.dk>]

On 11/9/23 10:24 AM, Timothy Pearson wrote:
> 
> 
> ----- Original Message -----
>> From: "Jens Axboe" <axboe@kernel.dk>
>> To: "Timothy Pearson" <tpearson@raptorengineering.com>
>> Cc: "regressions" <regressions@lists.linux.dev>, "Pavel Begunkov" <asml.silence@gmail.com>
>> Sent: Thursday, November 9, 2023 11:17:03 AM
>> Subject: Re: Regression in io_uring, leading to data corruption
> 
>> On 11/9/23 10:00 AM, Timothy Pearson wrote:
>>>
>>>
>>> ----- Original Message -----
>>>> From: "Jens Axboe" <axboe@kernel.dk>
>>>> To: "Timothy Pearson" <tpearson@raptorengineering.com>
>>>> Cc: "regressions" <regressions@lists.linux.dev>, "Pavel Begunkov"
>>>> <asml.silence@gmail.com>
>>>> Sent: Thursday, November 9, 2023 9:12:07 AM
>>>> Subject: Re: Regression in io_uring, leading to data corruption
>>>
>>>> On 11/8/23 4:58 PM, Jens Axboe wrote:
>>>>> On 11/8/23 3:18 PM, Jens Axboe wrote:
>>>>>> OK, let me actually test this thing and see if I can make it solid
>>>>>> first...
>>>>>
>>>>> Here's a suitable hack - it just creates a new io worker for each item,
>>>>> ensuring that that worker is run on the same CPU.
>>>>
>>>> Turns out I didn't send you the current one, this one was older and
>>>> untested. Please try this one instead.
>>>
>>> I'm still seeing an oops with this newer patchset applied:
>>> [   76.419788] Kernel attempted to read user page (9c) - exploit attempt? (uid:
>>> 0)
>>> [   76.419853] BUG: Kernel NULL pointer dereference on read at 0x0000009c
>>> [   76.419887] Faulting instruction address: 0xc0000000008374d4
>>> [   76.419919] Oops: Kernel access of bad area, sig: 11 [#1]
>>> <snip>
>>> [   76.429125] Call Trace:
>>> [   76.429142] [c000000009a8f800] [c000000000ea4e7c] schedule+0xfc/0x150
>>> (unreliable)
>>> [   76.429190] [c000000009a8f870] [c000000000ead8f0]
>>> schedule_timeout+0x170/0x1e0
>>> [   76.429238] [c000000009a8f940] [c000000000ea3fc8]
>>> io_schedule_timeout+0x68/0xa0
>>> [   76.429285] [c000000009a8f970] [c0000000007adcbc] blk_io_schedule+0x3c/0x70
>>> [   76.429326] [c000000009a8f990] [c00000000065da84] __iomap_dio_rw+0x8d4/0x9f0
>>> [   76.429374] [c000000009a8fb70] [c00000000065dea0] iomap_dio_rw+0x20/0x80
>>> [   76.429415] [c000000009a8fb90] [c008000000514eac]
>>> ext4_file_write_iter+0x814/0xdb8 [ext4]
>>> [   76.429475] [c000000009a8fc60] [c000000000833aa0] io_write+0x280/0x540
>>> [   76.429514] [c000000009a8fd80] [c0000000008183bc] io_issue_sqe+0xdc/0x390
>>> [   76.429554] [c000000009a8fdd0] [c000000000818ab4]
>>> io_wq_submit_work+0x1c4/0x2c0
>>> [   76.429601] [c000000009a8fe20] [c000000000835578]
>>> io_wq_single_worker+0x88/0xb0
>>> [   76.429652] [c000000009a8fe50] [c00000000000df3c]
>>> ret_from_kernel_user_thread+0x14/0x1c
>>>
>>> If you're not seeing an oops, that's an interesting difference that I
>>> can investigate further.
>>
>> Are you sure that's with the latest patch? Because that looks like
>> something that'd happen with the buggy one I sent out, as the sleep/wake
>> handling doesn't properly handle the specialized workers.
>>
>> --
>> Jens Axboe
> 
> You're right, the new patch wasn't applied, somehow it didn't copy
> over from my clipboard.  Apologies for the noise.
> 
> That said, the new patch is still showing the data corruption.  Maybe
> the pinning was just introducing the same timing alterations that my
> udelay() does on specific kernel builds?

Hmm ok, that is odd. How quickly does it trigger for you?

-- 
Jens Axboe