From: Daniel Harding <dharding@living180.net>
To: Pavel Begunkov <asml.silence@gmail.com>, Jens Axboe <axboe@kernel.dk>
Cc: regressions@lists.linux.dev, io-uring@vger.kernel.org,
linux-kernel@vger.kernel.org
Subject: Re: [REGRESSION] lxc-stop hang on 5.17.x kernels
Date: Tue, 3 May 2022 10:37:41 +0300 [thread overview]
Message-ID: <a204ba93-7261-5c6e-1baf-e5427e26b124@living180.net> (raw)
In-Reply-To: <fb0dbd71-9733-0208-48f2-c5d22ed17510@gmail.com>
[-- Attachment #1: Type: text/plain, Size: 4271 bytes --]
[Resend with a smaller trace]
On 5/3/22 02:14, Pavel Begunkov wrote:
> On 5/2/22 19:49, Daniel Harding wrote:
>> On 5/2/22 20:40, Pavel Begunkov wrote:
>>> On 5/2/22 18:00, Jens Axboe wrote:
>>>> On 5/2/22 7:59 AM, Jens Axboe wrote:
>>>>> On 5/2/22 7:36 AM, Daniel Harding wrote:
>>>>>> On 5/2/22 16:26, Jens Axboe wrote:
>>>>>>> On 5/2/22 7:17 AM, Daniel Harding wrote:
>>>>>>>> I use lxc-4.0.12 on Gentoo, built with io-uring support
>>>>>>>> (--enable-liburing), targeting liburing-2.1. My kernel config is a
>>>>>>>> very lightly modified version of Fedora's generic kernel
>>>>>>>> config. After
>>>>>>>> moving from the 5.16.x series to the 5.17.x kernel series, I
>>>>>>>> started
>>>>>>>> noticed frequent hangs in lxc-stop. It doesn't happen 100% of the
>>>>>>>> time, but definitely more than 50% of the time. Bisecting narrowed
>>>>>>>> down the issue to commit aa43477b040251f451db0d844073ac00a8ab66ee:
>>>>>>>> io_uring: poll rework. Testing indicates the problem is still
>>>>>>>> present
>>>>>>>> in 5.18-rc5. Unfortunately I do not have the expertise with the
>>>>>>>> codebases of either lxc or io-uring to try to debug the problem
>>>>>>>> further on my own, but I can easily apply patches to any of the
>>>>>>>> involved components (lxc, liburing, kernel) and rebuild for
>>>>>>>> testing or
>>>>>>>> validation. I am also happy to provide any further information
>>>>>>>> that
>>>>>>>> would be helpful with reproducing or debugging the problem.
>>>>>>> Do you have a recipe to reproduce the hang? That would make it
>>>>>>> significantly easier to figure out.
>>>>>>
>>>>>> I can reproduce it with just the following:
>>>>>>
>>>>>> sudo lxc-create --n lxc-test --template download --bdev dir
>>>>>> --dir /var/lib/lxc/lxc-test/rootfs -- -d ubuntu -r bionic -a amd64
>>>>>> sudo lxc-start -n lxc-test
>>>>>> sudo lxc-stop -n lxc-test
>>>>>>
>>>>>> The lxc-stop command never exits and the container continues running.
>>>>>> If that isn't sufficient to reproduce, please let me know.
>>>>>
>>>>> Thanks, that's useful! I'm at a conference this week and hence have
>>>>> limited amount of time to debug, hopefully Pavel has time to take
>>>>> a look
>>>>> at this.
>>>>
>>>> Didn't manage to reproduce. Can you try, on both the good and bad
>>>> kernel, to do:
>>>
>>> Same here, it doesn't reproduce for me
>> OK, sorry it wasn't something simple.
>>> # echo 1 > /sys/kernel/debug/tracing/events/io_uring/enable
>>>>
>>>> run lxc-stop
>>>>
>>>> # cp /sys/kernel/debug/tracing/trace ~/iou-trace
>>>>
>>>> so we can see what's going on? Looking at the source, lxc is just using
>>>> plain POLL_ADD, so I'm guessing it's not getting a notification when it
>>>> expects to, or it's POLL_REMOVE not doing its job. If we have a trace
>>>> from both a working and broken kernel, that might shed some light
>>>> on it.
>> It's late in my timezone, but I'll try to work on getting those
>> traces tomorrow.
>
> I think I got it, I've attached a trace.
>
> What's interesting is that it issues a multi shot poll but I don't
> see any kind of cancellation, neither cancel requests nor task/ring
> exit. Perhaps have to go look at lxc to see how it's supposed
> to work
Yes, that looks exactly like my bad trace. I've attached good trace
(captured with linux-5.16.19) and a bad trace (captured with
linux-5.17.5). These are the differences I noticed with just a visual scan:
* Both traces have three io_uring_submit_sqe calls at the very
beginning, but in the good trace, there are further io_uring_submit_sqe
calls throughout the trace, while in the bad trace, there are none.
* The good trace uses a mask of c3 for io_uring_task_add much more often
than the bad trace: the bad trace uses a mask of c3 only for the very
last call to io_uring_task_add, but a mask of 41 for the other calls.
* In the good trace, many of the io_uring_complete calls have a result
of 195, while in the bad trace, they all have a result of 1.
I don't know whether any of those things are significant or not, but
that's what jumped out at me.
I have also attached a copy of the script I used to generate the
traces. If there is anything further I can to do help debug, please let
me know.
--
Regards,
Daniel Harding
[-- Attachment #2: lxc-trace-good --]
[-- Type: text/plain, Size: 19774 bytes --]
# tracer: nop
#
# entries-in-buffer/entries-written: 145/145 #P:16
#
# _-----=> irqs-off
# / _----=> need-resched
# | / _---=> hardirq/softirq
# || / _--=> preempt-depth
# ||| / _-=> migrate-disable
# |||| / delay
# TASK-PID CPU# ||||| TIMESTAMP FUNCTION
# | | | ||||| | |
lxc-start-5701 [008] ..... 537.440889: io_uring_create: ring 00000000b31688a5, fd 3 sq size 512, cq size 1024, flags 0
lxc-start-5701 [008] ..... 537.440918: io_uring_create: ring 0000000076cc4f29, fd 4 sq size 512, cq size 1024, flags 0
lxc-start-5701 [008] ..... 537.440928: io_uring_file_get: ring 00000000b31688a5, fd 7
lxc-start-5701 [008] ..... 537.440928: io_uring_submit_sqe: ring 00000000b31688a5, req 000000002a7a691b, op 6, data 0x55dbcf28cd80, flags 524288, non block 1, sq_thread 0
lxc-start-5701 [008] ..... 537.440931: io_uring_file_get: ring 00000000b31688a5, fd 53
lxc-start-5701 [008] ..... 537.440931: io_uring_submit_sqe: ring 00000000b31688a5, req 00000000105632b5, op 6, data 0x55dbcf28d420, flags 524288, non block 1, sq_thread 0
lxc-start-5701 [008] ..... 537.440933: io_uring_file_get: ring 00000000b31688a5, fd 5
lxc-start-5701 [008] ..... 537.440933: io_uring_submit_sqe: ring 00000000b31688a5, req 00000000dcef163d, op 6, data 0x55dbcf28d470, flags 524288, non block 1, sq_thread 0
lxc-start-5701 [008] ..... 537.440935: io_uring_cqring_wait: ring 00000000b31688a5, min_events 1
kworker/u32:3-138 [008] d..1. 537.442660: io_uring_task_add: ring 00000000b31688a5, op 6, data 0x55dbcf28d420, mask 41
lxc-start-5701 [010] ...1. 537.442693: io_uring_complete: ring 00000000b31688a5, user_data 0x55dbcf28d420, result 65, cflags 0
lxc-start-5701 [010] ..... 537.442708: io_uring_file_get: ring 00000000b31688a5, fd 53
lxc-start-5701 [010] ..... 537.442709: io_uring_submit_sqe: ring 00000000b31688a5, req 00000000435a6b59, op 6, data 0x55dbcf28d420, flags 524288, non block 1, sq_thread 0
lxc-start-5701 [010] ..... 537.442713: io_uring_cqring_wait: ring 00000000b31688a5, min_events 1
kworker/u32:3-138 [008] d..1. 537.442882: io_uring_task_add: ring 00000000b31688a5, op 6, data 0x55dbcf28d420, mask 41
lxc-start-5701 [010] ...1. 537.442907: io_uring_complete: ring 00000000b31688a5, user_data 0x55dbcf28d420, result 65, cflags 0
lxc-start-5701 [010] ..... 537.442914: io_uring_file_get: ring 00000000b31688a5, fd 53
lxc-start-5701 [010] ..... 537.442915: io_uring_submit_sqe: ring 00000000b31688a5, req 00000000df1dac20, op 6, data 0x55dbcf28d420, flags 524288, non block 1, sq_thread 0
lxc-start-5701 [010] ..... 537.442917: io_uring_cqring_wait: ring 00000000b31688a5, min_events 1
lxc-stop-5751 [006] d..1. 537.448788: io_uring_task_add: ring 00000000b31688a5, op 6, data 0x55dbcf28d470, mask c3
lxc-start-5701 [010] ...1. 537.448814: io_uring_complete: ring 00000000b31688a5, user_data 0x55dbcf28d470, result 195, cflags 2
lxc-start-5701 [010] ..... 537.448836: io_uring_file_get: ring 00000000b31688a5, fd 24
lxc-start-5701 [010] ..... 537.448837: io_uring_submit_sqe: ring 00000000b31688a5, req 00000000c3421fbe, op 6, data 0x55dbcf28d4c0, flags 524288, non block 1, sq_thread 0
lxc-start-5701 [010] ...1. 537.448838: io_uring_complete: ring 00000000b31688a5, user_data 0x55dbcf28d4c0, result 1, cflags 0
lxc-start-5701 [010] ..... 537.448857: io_uring_cqring_wait: ring 00000000b31688a5, min_events 1
lxc-stop-5751 [012] d..1. 537.448888: io_uring_task_add: ring 00000000b31688a5, op 6, data 0x55dbcf28d470, mask c3
lxc-start-5701 [010] ...1. 537.448908: io_uring_complete: ring 00000000b31688a5, user_data 0x55dbcf28d470, result 195, cflags 2
lxc-start-5701 [010] ..... 537.448916: io_uring_file_get: ring 00000000b31688a5, fd 24
lxc-start-5701 [010] ..... 537.448916: io_uring_submit_sqe: ring 00000000b31688a5, req 0000000091e8a675, op 6, data 0x55dbcf28a380, flags 524288, non block 1, sq_thread 0
lxc-start-5701 [010] ...1. 537.448916: io_uring_complete: ring 00000000b31688a5, user_data 0x55dbcf28a380, result 1, cflags 0
lxc-start-5701 [010] ..... 537.448924: io_uring_cqring_wait: ring 00000000b31688a5, min_events 1
lxc-stop-5751 [012] d..1. 537.448965: io_uring_task_add: ring 00000000b31688a5, op 6, data 0x55dbcf28d470, mask c3
lxc-start-5701 [010] ...1. 537.448982: io_uring_complete: ring 00000000b31688a5, user_data 0x55dbcf28d470, result 195, cflags 2
lxc-start-5701 [010] ..... 537.448991: io_uring_file_get: ring 00000000b31688a5, fd 24
lxc-start-5701 [010] ..... 537.448991: io_uring_submit_sqe: ring 00000000b31688a5, req 00000000404191e3, op 6, data 0x55dbcf28a3d0, flags 524288, non block 1, sq_thread 0
lxc-start-5701 [010] ...1. 537.448992: io_uring_complete: ring 00000000b31688a5, user_data 0x55dbcf28a3d0, result 1, cflags 0
lxc-start-5701 [010] ..... 537.449002: io_uring_cqring_wait: ring 00000000b31688a5, min_events 1
lxc-stop-5751 [012] d..1. 537.449018: io_uring_task_add: ring 00000000b31688a5, op 6, data 0x55dbcf28d470, mask c3
lxc-start-5701 [010] ...1. 537.449038: io_uring_complete: ring 00000000b31688a5, user_data 0x55dbcf28d470, result 195, cflags 2
lxc-start-5701 [010] ..... 537.449046: io_uring_file_get: ring 00000000b31688a5, fd 24
lxc-start-5701 [010] ..... 537.449046: io_uring_submit_sqe: ring 00000000b31688a5, req 000000004c87bdb8, op 6, data 0x55dbcf28a420, flags 524288, non block 1, sq_thread 0
lxc-start-5701 [010] ...1. 537.449047: io_uring_complete: ring 00000000b31688a5, user_data 0x55dbcf28a420, result 1, cflags 0
lxc-start-5701 [010] ..... 537.449057: io_uring_cqring_wait: ring 00000000b31688a5, min_events 1
lxc-stop-5751 [012] d..1. 537.449095: io_uring_task_add: ring 00000000b31688a5, op 6, data 0x55dbcf28d470, mask c3
lxc-start-5701 [010] ...1. 537.449113: io_uring_complete: ring 00000000b31688a5, user_data 0x55dbcf28d470, result 195, cflags 2
lxc-start-5701 [010] ..... 537.449120: io_uring_file_get: ring 00000000b31688a5, fd 24
lxc-start-5701 [010] ..... 537.449120: io_uring_submit_sqe: ring 00000000b31688a5, req 00000000b8df304d, op 6, data 0x55dbcf28a470, flags 524288, non block 1, sq_thread 0
lxc-start-5701 [010] ...1. 537.449121: io_uring_complete: ring 00000000b31688a5, user_data 0x55dbcf28a470, result 1, cflags 0
lxc-start-5701 [010] ..... 537.449131: io_uring_cqring_wait: ring 00000000b31688a5, min_events 1
lxc-stop-5751 [012] d..1. 537.449207: io_uring_task_add: ring 00000000b31688a5, op 6, data 0x55dbcf28d470, mask c3
lxc-start-5701 [010] ...1. 537.449223: io_uring_complete: ring 00000000b31688a5, user_data 0x55dbcf28d470, result 195, cflags 2
lxc-start-5701 [010] ..... 537.449231: io_uring_file_get: ring 00000000b31688a5, fd 24
lxc-start-5701 [010] ..... 537.449231: io_uring_submit_sqe: ring 00000000b31688a5, req 0000000028c2db2d, op 6, data 0x55dbcf28a4c0, flags 524288, non block 1, sq_thread 0
lxc-start-5701 [010] ...1. 537.449232: io_uring_complete: ring 00000000b31688a5, user_data 0x55dbcf28a4c0, result 1, cflags 0
lxc-start-5701 [010] ..... 537.449241: io_uring_file_get: ring 00000000b31688a5, fd 24
lxc-start-5701 [010] ..... 537.449241: io_uring_submit_sqe: ring 00000000b31688a5, req 00000000a0ddd401, op 6, data 0x55dbcf28a4c0, flags 524288, non block 1, sq_thread 0
lxc-start-5701 [010] ..... 537.449242: io_uring_cqring_wait: ring 00000000b31688a5, min_events 1
kworker/u32:3-138 [012] d..1. 537.535100: io_uring_task_add: ring 00000000b31688a5, op 6, data 0x55dbcf28d420, mask 41
lxc-start-5701 [010] ...1. 537.535155: io_uring_complete: ring 00000000b31688a5, user_data 0x55dbcf28d420, result 65, cflags 0
lxc-start-5701 [010] ..... 537.535170: io_uring_file_get: ring 00000000b31688a5, fd 53
lxc-start-5701 [010] ..... 537.535171: io_uring_submit_sqe: ring 00000000b31688a5, req 000000004b218709, op 6, data 0x55dbcf28d420, flags 524288, non block 1, sq_thread 0
lxc-start-5701 [010] ..... 537.535175: io_uring_cqring_wait: ring 00000000b31688a5, min_events 1
kworker/u32:3-138 [012] d..1. 537.535942: io_uring_task_add: ring 00000000b31688a5, op 6, data 0x55dbcf28d420, mask 41
lxc-start-5701 [010] ...1. 537.535969: io_uring_complete: ring 00000000b31688a5, user_data 0x55dbcf28d420, result 65, cflags 0
lxc-start-5701 [010] ..... 537.535976: io_uring_file_get: ring 00000000b31688a5, fd 53
lxc-start-5701 [010] ..... 537.535976: io_uring_submit_sqe: ring 00000000b31688a5, req 0000000006afc786, op 6, data 0x55dbcf28d420, flags 524288, non block 1, sq_thread 0
kworker/u32:3-138 [012] d..1. 537.536000: io_uring_task_add: ring 00000000b31688a5, op 6, data 0x55dbcf28d420, mask 41
lxc-start-5701 [010] ...1. 537.536030: io_uring_complete: ring 00000000b31688a5, user_data 0x55dbcf28d420, result 65, cflags 0
lxc-start-5701 [010] ..... 537.536036: io_uring_file_get: ring 00000000b31688a5, fd 53
lxc-start-5701 [010] ..... 537.536037: io_uring_submit_sqe: ring 00000000b31688a5, req 00000000deda76cd, op 6, data 0x55dbcf28d420, flags 524288, non block 1, sq_thread 0
lxc-start-5701 [010] ..... 537.536038: io_uring_cqring_wait: ring 00000000b31688a5, min_events 1
kworker/u32:3-138 [012] d..1. 537.536129: io_uring_task_add: ring 00000000b31688a5, op 6, data 0x55dbcf28d420, mask 41
lxc-start-5701 [010] ...1. 537.536155: io_uring_complete: ring 00000000b31688a5, user_data 0x55dbcf28d420, result 65, cflags 0
lxc-start-5701 [010] ..... 537.536161: io_uring_file_get: ring 00000000b31688a5, fd 53
lxc-start-5701 [010] ..... 537.536162: io_uring_submit_sqe: ring 00000000b31688a5, req 00000000d59ff773, op 6, data 0x55dbcf28d420, flags 524288, non block 1, sq_thread 0
lxc-start-5701 [010] ..... 537.536164: io_uring_cqring_wait: ring 00000000b31688a5, min_events 1
kworker/u32:3-138 [012] d..1. 537.536199: io_uring_task_add: ring 00000000b31688a5, op 6, data 0x55dbcf28d420, mask 41
lxc-start-5701 [010] ...1. 537.536225: io_uring_complete: ring 00000000b31688a5, user_data 0x55dbcf28d420, result 65, cflags 0
lxc-start-5701 [010] ..... 537.536234: io_uring_file_get: ring 00000000b31688a5, fd 53
lxc-start-5701 [010] ..... 537.536235: io_uring_submit_sqe: ring 00000000b31688a5, req 00000000ecd22ed2, op 6, data 0x55dbcf28d420, flags 524288, non block 1, sq_thread 0
kworker/u32:3-138 [012] d..1. 537.536246: io_uring_task_add: ring 00000000b31688a5, op 6, data 0x55dbcf28d420, mask 41
lxc-start-5701 [010] ...1. 537.536262: io_uring_complete: ring 00000000b31688a5, user_data 0x55dbcf28d420, result 65, cflags 0
lxc-start-5701 [010] ..... 537.536264: io_uring_file_get: ring 00000000b31688a5, fd 53
lxc-start-5701 [010] ..... 537.536264: io_uring_submit_sqe: ring 00000000b31688a5, req 000000007e28f103, op 6, data 0x55dbcf28d420, flags 524288, non block 1, sq_thread 0
lxc-start-5701 [010] ..... 537.536265: io_uring_cqring_wait: ring 00000000b31688a5, min_events 1
kworker/u32:3-138 [012] d..1. 537.536273: io_uring_task_add: ring 00000000b31688a5, op 6, data 0x55dbcf28d420, mask 41
lxc-start-5701 [010] ...1. 537.536286: io_uring_complete: ring 00000000b31688a5, user_data 0x55dbcf28d420, result 65, cflags 0
lxc-start-5701 [010] ..... 537.536288: io_uring_file_get: ring 00000000b31688a5, fd 53
lxc-start-5701 [010] ..... 537.536288: io_uring_submit_sqe: ring 00000000b31688a5, req 000000002a8f1fc3, op 6, data 0x55dbcf28d420, flags 524288, non block 1, sq_thread 0
lxc-start-5701 [010] ..... 537.536289: io_uring_cqring_wait: ring 00000000b31688a5, min_events 1
kworker/u32:3-138 [012] d..1. 537.536666: io_uring_task_add: ring 00000000b31688a5, op 6, data 0x55dbcf28d420, mask 41
lxc-start-5701 [010] ...1. 537.536694: io_uring_complete: ring 00000000b31688a5, user_data 0x55dbcf28d420, result 65, cflags 0
lxc-start-5701 [010] ..... 537.536701: io_uring_file_get: ring 00000000b31688a5, fd 53
lxc-start-5701 [010] ..... 537.536701: io_uring_submit_sqe: ring 00000000b31688a5, req 00000000228168af, op 6, data 0x55dbcf28d420, flags 524288, non block 1, sq_thread 0
lxc-start-5701 [010] ..... 537.536703: io_uring_cqring_wait: ring 00000000b31688a5, min_events 1
kworker/u32:3-138 [012] d..1. 537.537049: io_uring_task_add: ring 00000000b31688a5, op 6, data 0x55dbcf28d420, mask 41
lxc-start-5701 [010] ...1. 537.537077: io_uring_complete: ring 00000000b31688a5, user_data 0x55dbcf28d420, result 65, cflags 0
lxc-start-5701 [010] ..... 537.537084: io_uring_file_get: ring 00000000b31688a5, fd 53
lxc-start-5701 [010] ..... 537.537084: io_uring_submit_sqe: ring 00000000b31688a5, req 000000003f0080b8, op 6, data 0x55dbcf28d420, flags 524288, non block 1, sq_thread 0
lxc-start-5701 [010] ..... 537.537087: io_uring_cqring_wait: ring 00000000b31688a5, min_events 1
kworker/u32:3-138 [012] d..1. 537.537813: io_uring_task_add: ring 00000000b31688a5, op 6, data 0x55dbcf28d420, mask 41
lxc-start-5701 [010] ...1. 537.537842: io_uring_complete: ring 00000000b31688a5, user_data 0x55dbcf28d420, result 65, cflags 0
lxc-start-5701 [010] ..... 537.537851: io_uring_file_get: ring 00000000b31688a5, fd 53
lxc-start-5701 [010] ..... 537.537851: io_uring_submit_sqe: ring 00000000b31688a5, req 000000001baf187d, op 6, data 0x55dbcf28d420, flags 524288, non block 1, sq_thread 0
lxc-start-5701 [010] ..... 537.537854: io_uring_cqring_wait: ring 00000000b31688a5, min_events 1
kworker/u32:3-138 [012] d..1. 537.538189: io_uring_task_add: ring 00000000b31688a5, op 6, data 0x55dbcf28d420, mask 41
lxc-start-5701 [013] ...1. 537.538203: io_uring_complete: ring 00000000b31688a5, user_data 0x55dbcf28d420, result 65, cflags 0
lxc-start-5701 [013] ..... 537.538208: io_uring_file_get: ring 00000000b31688a5, fd 53
lxc-start-5701 [013] ..... 537.538208: io_uring_submit_sqe: ring 00000000b31688a5, req 000000000fb0d9a9, op 6, data 0x55dbcf28d420, flags 524288, non block 1, sq_thread 0
lxc-start-5701 [013] ..... 537.538210: io_uring_cqring_wait: ring 00000000b31688a5, min_events 1
kworker/u32:3-138 [012] d..1. 537.538614: io_uring_task_add: ring 00000000b31688a5, op 6, data 0x55dbcf28d420, mask 41
lxc-start-5701 [013] ...1. 537.538625: io_uring_complete: ring 00000000b31688a5, user_data 0x55dbcf28d420, result 65, cflags 0
lxc-start-5701 [013] ..... 537.538634: io_uring_file_get: ring 00000000b31688a5, fd 53
lxc-start-5701 [013] ..... 537.538635: io_uring_submit_sqe: ring 00000000b31688a5, req 00000000c9d4b035, op 6, data 0x55dbcf28d420, flags 524288, non block 1, sq_thread 0
lxc-start-5701 [013] ..... 537.538637: io_uring_cqring_wait: ring 00000000b31688a5, min_events 1
kworker/u32:3-138 [012] d..1. 537.538718: io_uring_task_add: ring 00000000b31688a5, op 6, data 0x55dbcf28d420, mask 41
lxc-start-5701 [013] ...1. 537.538729: io_uring_complete: ring 00000000b31688a5, user_data 0x55dbcf28d420, result 65, cflags 0
lxc-start-5701 [013] ..... 537.538736: io_uring_file_get: ring 00000000b31688a5, fd 53
lxc-start-5701 [013] ..... 537.538736: io_uring_submit_sqe: ring 00000000b31688a5, req 000000009f6c3d45, op 6, data 0x55dbcf28d420, flags 524288, non block 1, sq_thread 0
lxc-start-5701 [013] ..... 537.538737: io_uring_cqring_wait: ring 00000000b31688a5, min_events 1
kworker/u32:3-138 [010] d..1. 537.539270: io_uring_task_add: ring 00000000b31688a5, op 6, data 0x55dbcf28d420, mask 41
lxc-start-5701 [013] ...1. 537.539272: io_uring_complete: ring 00000000b31688a5, user_data 0x55dbcf28d420, result 65, cflags 0
lxc-start-5701 [013] ..... 537.539278: io_uring_file_get: ring 00000000b31688a5, fd 53
lxc-start-5701 [013] ..... 537.539278: io_uring_submit_sqe: ring 00000000b31688a5, req 00000000658d5a67, op 6, data 0x55dbcf28d420, flags 524288, non block 1, sq_thread 0
lxc-start-5701 [013] ..... 537.539279: io_uring_cqring_wait: ring 00000000b31688a5, min_events 1
kworker/u32:3-138 [012] d..1. 537.540199: io_uring_task_add: ring 00000000b31688a5, op 6, data 0x55dbcf28d420, mask 41
lxc-start-5701 [013] ...1. 537.540213: io_uring_complete: ring 00000000b31688a5, user_data 0x55dbcf28d420, result 65, cflags 0
lxc-start-5701 [013] ..... 537.540221: io_uring_file_get: ring 00000000b31688a5, fd 53
lxc-start-5701 [013] ..... 537.540222: io_uring_submit_sqe: ring 00000000b31688a5, req 000000003ad51d84, op 6, data 0x55dbcf28d420, flags 524288, non block 1, sq_thread 0
lxc-start-5701 [013] ..... 537.540223: io_uring_cqring_wait: ring 00000000b31688a5, min_events 1
kworker/u32:3-138 [010] d..1. 537.540415: io_uring_task_add: ring 00000000b31688a5, op 6, data 0x55dbcf28d420, mask 41
lxc-start-5701 [013] ...1. 537.540424: io_uring_complete: ring 00000000b31688a5, user_data 0x55dbcf28d420, result 65, cflags 0
lxc-start-5701 [013] ..... 537.540428: io_uring_file_get: ring 00000000b31688a5, fd 53
lxc-start-5701 [013] ..... 537.540428: io_uring_submit_sqe: ring 00000000b31688a5, req 00000000297b5537, op 6, data 0x55dbcf28d420, flags 524288, non block 1, sq_thread 0
lxc-start-5701 [013] ..... 537.540430: io_uring_cqring_wait: ring 00000000b31688a5, min_events 1
systemd-shutdow-5702 [008] dN.3. 537.572892: io_uring_task_add: ring 00000000b31688a5, op 6, data 0x55dbcf28cd80, mask 0
lxc-start-5701 [013] ...1. 537.572909: io_uring_complete: ring 00000000b31688a5, user_data 0x55dbcf28cd80, result 1, cflags 2
lxc-start-5701 [013] ..... 537.572929: io_uring_file_get: ring 0000000076cc4f29, fd 53
lxc-start-5701 [013] ..... 537.572930: io_uring_submit_sqe: ring 0000000076cc4f29, req 00000000e482f889, op 6, data 0x55dbcf28a510, flags 524288, non block 1, sq_thread 0
lxc-start-5701 [013] ..... 537.572938: io_uring_cqring_wait: ring 0000000076cc4f29, min_events 1
lxc-start-5701 [013] ...1. 537.572967: io_uring_complete: ring 0000000076cc4f29, user_data 0x55dbcf28a510, result -125, cflags 0
lxc-start-5701 [013] ...1. 537.572984: io_uring_complete: ring 00000000b31688a5, user_data 0x55dbcf28a4c0, result -125, cflags 0
lxc-start-5701 [013] ...1. 537.572984: io_uring_complete: ring 00000000b31688a5, user_data 0x55dbcf28d470, result -125, cflags 0
lxc-start-5701 [013] ...1. 537.572984: io_uring_complete: ring 00000000b31688a5, user_data 0x55dbcf28d420, result -125, cflags 0
lxc-start-5701 [013] ...1. 537.572984: io_uring_complete: ring 00000000b31688a5, user_data 0x55dbcf28cd80, result -125, cflags 0
[-- Attachment #3: lxc-trace-bad --]
[-- Type: text/plain, Size: 23533 bytes --]
# tracer: nop
#
# entries-in-buffer/entries-written: 183/183 #P:16
#
# _-----=> irqs-off/BH-disabled
# / _----=> need-resched
# | / _---=> hardirq/softirq
# || / _--=> preempt-depth
# ||| / _-=> migrate-disable
# |||| / delay
# TASK-PID CPU# ||||| TIMESTAMP FUNCTION
# | | | ||||| | |
lxc-start-2249 [007] ..... 47.766086: io_uring_create: ring 00000000ad366d59, fd 3 sq size 512, cq size 1024, flags 0
lxc-start-2249 [007] ..... 47.766128: io_uring_create: ring 00000000a3f46d45, fd 4 sq size 512, cq size 1024, flags 0
lxc-start-2249 [007] ..... 47.766143: io_uring_submit_sqe: ring 00000000ad366d59, req 00000000c89e5524, op 6, data 0x56049a674b80, flags 524288, non block 1, sq_thread 0
lxc-start-2249 [007] ..... 47.766144: io_uring_file_get: ring 00000000ad366d59, fd 7
lxc-start-2249 [007] ..... 47.766146: io_uring_submit_sqe: ring 00000000ad366d59, req 000000004114f7f5, op 6, data 0x56049a674bd0, flags 524288, non block 1, sq_thread 0
lxc-start-2249 [007] ..... 47.766147: io_uring_file_get: ring 00000000ad366d59, fd 53
lxc-start-2249 [007] ..... 47.766152: io_uring_submit_sqe: ring 00000000ad366d59, req 00000000476a2670, op 6, data 0x56049a671380, flags 524288, non block 1, sq_thread 0
lxc-start-2249 [007] ..... 47.766152: io_uring_file_get: ring 00000000ad366d59, fd 5
lxc-start-2249 [007] ..... 47.766155: io_uring_cqring_wait: ring 00000000ad366d59, min_events 1
kworker/u32:9-1078 [011] d..1. 47.792070: io_uring_task_add: ring 00000000ad366d59, op 6, data 0x56049a674bd0, mask 41
lxc-start-2249 [007] ...1. 47.792101: io_uring_complete: ring 00000000ad366d59, user_data 0x56049a674bd0, result 1, cflags 2
lxc-start-2249 [007] ..... 47.792114: io_uring_cqring_wait: ring 00000000ad366d59, min_events 1
kworker/u32:6-148 [007] d..1. 47.792800: io_uring_task_add: ring 00000000ad366d59, op 6, data 0x56049a674bd0, mask 41
lxc-start-2249 [000] ...1. 47.792834: io_uring_complete: ring 00000000ad366d59, user_data 0x56049a674bd0, result 1, cflags 2
lxc-start-2249 [000] ..... 47.792840: io_uring_cqring_wait: ring 00000000ad366d59, min_events 1
kworker/u32:6-148 [007] d..1. 47.792933: io_uring_task_add: ring 00000000ad366d59, op 6, data 0x56049a674bd0, mask 41
lxc-start-2249 [000] ...1. 47.792959: io_uring_complete: ring 00000000ad366d59, user_data 0x56049a674bd0, result 1, cflags 2
lxc-start-2249 [000] ..... 47.792965: io_uring_cqring_wait: ring 00000000ad366d59, min_events 1
kworker/u32:9-1078 [003] d..1. 47.902763: io_uring_task_add: ring 00000000ad366d59, op 6, data 0x56049a674bd0, mask 41
lxc-start-2249 [000] ...1. 47.903010: io_uring_complete: ring 00000000ad366d59, user_data 0x56049a674bd0, result 1, cflags 2
lxc-start-2249 [000] ..... 47.903023: io_uring_cqring_wait: ring 00000000ad366d59, min_events 1
kworker/u32:9-1078 [003] d..1. 47.903040: io_uring_task_add: ring 00000000ad366d59, op 6, data 0x56049a674bd0, mask 41
lxc-start-2249 [000] ...1. 47.903049: io_uring_complete: ring 00000000ad366d59, user_data 0x56049a674bd0, result 1, cflags 2
lxc-start-2249 [000] ..... 47.903051: io_uring_cqring_wait: ring 00000000ad366d59, min_events 1
kworker/u32:9-1078 [003] d..1. 47.903718: io_uring_task_add: ring 00000000ad366d59, op 6, data 0x56049a674bd0, mask 41
lxc-start-2249 [000] ...1. 47.903742: io_uring_complete: ring 00000000ad366d59, user_data 0x56049a674bd0, result 1, cflags 2
lxc-start-2249 [000] ..... 47.903749: io_uring_cqring_wait: ring 00000000ad366d59, min_events 1
kworker/u32:9-1078 [003] d..1. 47.904085: io_uring_task_add: ring 00000000ad366d59, op 6, data 0x56049a674bd0, mask 41
lxc-start-2249 [000] ...1. 47.904107: io_uring_complete: ring 00000000ad366d59, user_data 0x56049a674bd0, result 1, cflags 2
lxc-start-2249 [000] ..... 47.904113: io_uring_cqring_wait: ring 00000000ad366d59, min_events 1
kworker/u32:9-1078 [003] d..1. 47.904175: io_uring_task_add: ring 00000000ad366d59, op 6, data 0x56049a674bd0, mask 41
lxc-start-2249 [000] ...1. 47.904198: io_uring_complete: ring 00000000ad366d59, user_data 0x56049a674bd0, result 1, cflags 2
lxc-start-2249 [000] ..... 47.904204: io_uring_cqring_wait: ring 00000000ad366d59, min_events 1
kworker/u32:9-1078 [003] d..1. 47.904525: io_uring_task_add: ring 00000000ad366d59, op 6, data 0x56049a674bd0, mask 41
lxc-start-2249 [000] ...1. 47.904548: io_uring_complete: ring 00000000ad366d59, user_data 0x56049a674bd0, result 1, cflags 2
lxc-start-2249 [000] ..... 47.904554: io_uring_cqring_wait: ring 00000000ad366d59, min_events 1
kworker/u32:9-1078 [003] d..1. 47.904586: io_uring_task_add: ring 00000000ad366d59, op 6, data 0x56049a674bd0, mask 41
lxc-start-2249 [000] ...1. 47.904609: io_uring_complete: ring 00000000ad366d59, user_data 0x56049a674bd0, result 1, cflags 2
lxc-start-2249 [000] ..... 47.904614: io_uring_cqring_wait: ring 00000000ad366d59, min_events 1
kworker/u32:9-1078 [003] d..1. 47.904700: io_uring_task_add: ring 00000000ad366d59, op 6, data 0x56049a674bd0, mask 41
lxc-start-2249 [000] ...1. 47.904725: io_uring_complete: ring 00000000ad366d59, user_data 0x56049a674bd0, result 1, cflags 2
lxc-start-2249 [000] ..... 47.904731: io_uring_cqring_wait: ring 00000000ad366d59, min_events 1
kworker/u32:9-1078 [003] d..1. 47.905406: io_uring_task_add: ring 00000000ad366d59, op 6, data 0x56049a674bd0, mask 41
lxc-start-2249 [000] ...1. 47.905414: io_uring_complete: ring 00000000ad366d59, user_data 0x56049a674bd0, result 1, cflags 2
lxc-start-2249 [000] ..... 47.905421: io_uring_cqring_wait: ring 00000000ad366d59, min_events 1
kworker/u32:8-156 [005] d..1. 47.906170: io_uring_task_add: ring 00000000ad366d59, op 6, data 0x56049a674bd0, mask 41
lxc-start-2249 [000] ...1. 47.906202: io_uring_complete: ring 00000000ad366d59, user_data 0x56049a674bd0, result 1, cflags 2
lxc-start-2249 [000] ..... 47.906212: io_uring_cqring_wait: ring 00000000ad366d59, min_events 1
kworker/u32:8-156 [005] d..1. 47.908746: io_uring_task_add: ring 00000000ad366d59, op 6, data 0x56049a674bd0, mask 41
lxc-start-2249 [000] ...1. 47.908768: io_uring_complete: ring 00000000ad366d59, user_data 0x56049a674bd0, result 1, cflags 2
lxc-start-2249 [000] ..... 47.908778: io_uring_cqring_wait: ring 00000000ad366d59, min_events 1
kworker/u32:8-156 [005] d..1. 47.909269: io_uring_task_add: ring 00000000ad366d59, op 6, data 0x56049a674bd0, mask 41
lxc-start-2249 [000] ...1. 47.909291: io_uring_complete: ring 00000000ad366d59, user_data 0x56049a674bd0, result 1, cflags 2
lxc-start-2249 [000] ..... 47.909299: io_uring_cqring_wait: ring 00000000ad366d59, min_events 1
kworker/u32:8-156 [005] d..1. 47.911856: io_uring_task_add: ring 00000000ad366d59, op 6, data 0x56049a674bd0, mask 41
lxc-start-2249 [000] ...1. 47.911893: io_uring_complete: ring 00000000ad366d59, user_data 0x56049a674bd0, result 1, cflags 2
lxc-start-2249 [000] ..... 47.911909: io_uring_cqring_wait: ring 00000000ad366d59, min_events 1
kworker/u32:8-156 [005] d..1. 47.912580: io_uring_task_add: ring 00000000ad366d59, op 6, data 0x56049a674bd0, mask 41
lxc-start-2249 [002] ...1. 47.912592: io_uring_complete: ring 00000000ad366d59, user_data 0x56049a674bd0, result 1, cflags 2
lxc-start-2249 [002] ..... 47.912601: io_uring_cqring_wait: ring 00000000ad366d59, min_events 1
kworker/u32:2-107 [012] d..1. 47.918821: io_uring_task_add: ring 00000000ad366d59, op 6, data 0x56049a674bd0, mask 41
lxc-start-2249 [002] ...1. 47.918835: io_uring_complete: ring 00000000ad366d59, user_data 0x56049a674bd0, result 1, cflags 2
lxc-start-2249 [002] ..... 47.918844: io_uring_cqring_wait: ring 00000000ad366d59, min_events 1
kworker/u32:2-107 [012] d..1. 47.919048: io_uring_task_add: ring 00000000ad366d59, op 6, data 0x56049a674bd0, mask 41
lxc-start-2249 [002] ...1. 47.919055: io_uring_complete: ring 00000000ad366d59, user_data 0x56049a674bd0, result 1, cflags 2
lxc-start-2249 [002] ..... 47.919056: io_uring_cqring_wait: ring 00000000ad366d59, min_events 1
kworker/u32:2-107 [012] d..1. 47.919090: io_uring_task_add: ring 00000000ad366d59, op 6, data 0x56049a674bd0, mask 41
lxc-start-2249 [002] ...1. 47.919097: io_uring_complete: ring 00000000ad366d59, user_data 0x56049a674bd0, result 1, cflags 2
lxc-start-2249 [002] ..... 47.919098: io_uring_cqring_wait: ring 00000000ad366d59, min_events 1
kworker/u32:2-107 [012] d..1. 47.919752: io_uring_task_add: ring 00000000ad366d59, op 6, data 0x56049a674bd0, mask 41
lxc-start-2249 [002] ...1. 47.919760: io_uring_complete: ring 00000000ad366d59, user_data 0x56049a674bd0, result 1, cflags 2
lxc-start-2249 [002] ..... 47.919761: io_uring_cqring_wait: ring 00000000ad366d59, min_events 1
kworker/u32:11-2181 [000] d..1. 47.921724: io_uring_task_add: ring 00000000ad366d59, op 6, data 0x56049a674bd0, mask 41
lxc-start-2249 [002] ...1. 47.921733: io_uring_complete: ring 00000000ad366d59, user_data 0x56049a674bd0, result 1, cflags 2
lxc-start-2249 [002] ..... 47.921734: io_uring_cqring_wait: ring 00000000ad366d59, min_events 1
kworker/u32:2-107 [012] d..1. 47.922163: io_uring_task_add: ring 00000000ad366d59, op 6, data 0x56049a674bd0, mask 41
lxc-start-2249 [002] ...1. 47.922171: io_uring_complete: ring 00000000ad366d59, user_data 0x56049a674bd0, result 1, cflags 2
lxc-start-2249 [002] ..... 47.922172: io_uring_cqring_wait: ring 00000000ad366d59, min_events 1
kworker/u32:2-107 [012] d..1. 47.922643: io_uring_task_add: ring 00000000ad366d59, op 6, data 0x56049a674bd0, mask 41
lxc-start-2249 [014] ...1. 47.922654: io_uring_complete: ring 00000000ad366d59, user_data 0x56049a674bd0, result 1, cflags 2
lxc-start-2249 [014] ..... 47.922662: io_uring_cqring_wait: ring 00000000ad366d59, min_events 1
kworker/u32:2-107 [012] d..1. 47.928228: io_uring_task_add: ring 00000000ad366d59, op 6, data 0x56049a674bd0, mask 41
lxc-start-2249 [014] ...1. 47.928235: io_uring_complete: ring 00000000ad366d59, user_data 0x56049a674bd0, result 1, cflags 2
lxc-start-2249 [014] ..... 47.928240: io_uring_cqring_wait: ring 00000000ad366d59, min_events 1
kworker/u32:2-107 [012] d..1. 47.928339: io_uring_task_add: ring 00000000ad366d59, op 6, data 0x56049a674bd0, mask 41
lxc-start-2249 [014] ...1. 47.928344: io_uring_complete: ring 00000000ad366d59, user_data 0x56049a674bd0, result 1, cflags 2
lxc-start-2249 [014] ..... 47.928345: io_uring_cqring_wait: ring 00000000ad366d59, min_events 1
kworker/u32:2-107 [012] d..1. 47.928370: io_uring_task_add: ring 00000000ad366d59, op 6, data 0x56049a674bd0, mask 41
lxc-start-2249 [014] ...1. 47.928375: io_uring_complete: ring 00000000ad366d59, user_data 0x56049a674bd0, result 1, cflags 2
lxc-start-2249 [014] ..... 47.928376: io_uring_cqring_wait: ring 00000000ad366d59, min_events 1
kworker/u32:9-1078 [003] d..1. 47.930346: io_uring_task_add: ring 00000000ad366d59, op 6, data 0x56049a674bd0, mask 41
lxc-start-2249 [014] ...1. 47.930358: io_uring_complete: ring 00000000ad366d59, user_data 0x56049a674bd0, result 1, cflags 2
lxc-start-2249 [014] ..... 47.930363: io_uring_cqring_wait: ring 00000000ad366d59, min_events 1
kworker/u32:7-151 [010] d..1. 47.930820: io_uring_task_add: ring 00000000ad366d59, op 6, data 0x56049a674bd0, mask 41
lxc-start-2249 [014] ...1. 47.930831: io_uring_complete: ring 00000000ad366d59, user_data 0x56049a674bd0, result 1, cflags 2
lxc-start-2249 [014] ..... 47.930832: io_uring_cqring_wait: ring 00000000ad366d59, min_events 1
kworker/u32:6-148 [005] d..1. 47.937915: io_uring_task_add: ring 00000000ad366d59, op 6, data 0x56049a674bd0, mask 41
lxc-start-2249 [006] ...1. 47.937946: io_uring_complete: ring 00000000ad366d59, user_data 0x56049a674bd0, result 1, cflags 2
lxc-start-2249 [006] ..... 47.937958: io_uring_cqring_wait: ring 00000000ad366d59, min_events 1
kworker/u32:6-148 [005] d..1. 47.938370: io_uring_task_add: ring 00000000ad366d59, op 6, data 0x56049a674bd0, mask 41
lxc-start-2249 [006] ...1. 47.938399: io_uring_complete: ring 00000000ad366d59, user_data 0x56049a674bd0, result 1, cflags 2
lxc-start-2249 [006] ..... 47.938406: io_uring_cqring_wait: ring 00000000ad366d59, min_events 1
kworker/u32:6-148 [005] d..1. 47.942927: io_uring_task_add: ring 00000000ad366d59, op 6, data 0x56049a674bd0, mask 41
lxc-start-2249 [006] ...1. 47.942955: io_uring_complete: ring 00000000ad366d59, user_data 0x56049a674bd0, result 1, cflags 2
lxc-start-2249 [006] ..... 47.942966: io_uring_cqring_wait: ring 00000000ad366d59, min_events 1
kworker/u32:7-151 [010] d..1. 47.943029: io_uring_task_add: ring 00000000ad366d59, op 6, data 0x56049a674bd0, mask 41
lxc-start-2249 [006] ...1. 47.943057: io_uring_complete: ring 00000000ad366d59, user_data 0x56049a674bd0, result 1, cflags 2
lxc-start-2249 [006] ..... 47.943063: io_uring_cqring_wait: ring 00000000ad366d59, min_events 1
kworker/u32:7-151 [010] d..1. 47.943247: io_uring_task_add: ring 00000000ad366d59, op 6, data 0x56049a674bd0, mask 41
lxc-start-2249 [006] ...1. 47.943310: io_uring_complete: ring 00000000ad366d59, user_data 0x56049a674bd0, result 1, cflags 2
lxc-start-2249 [006] ..... 47.943317: io_uring_cqring_wait: ring 00000000ad366d59, min_events 1
kworker/u32:7-151 [010] d..1. 47.943323: io_uring_task_add: ring 00000000ad366d59, op 6, data 0x56049a674bd0, mask 41
lxc-start-2249 [006] ...1. 47.943335: io_uring_complete: ring 00000000ad366d59, user_data 0x56049a674bd0, result 1, cflags 2
lxc-start-2249 [006] ..... 47.943337: io_uring_cqring_wait: ring 00000000ad366d59, min_events 1
kworker/u32:7-151 [010] d..1. 47.943389: io_uring_task_add: ring 00000000ad366d59, op 6, data 0x56049a674bd0, mask 41
lxc-start-2249 [006] ...1. 47.943418: io_uring_complete: ring 00000000ad366d59, user_data 0x56049a674bd0, result 1, cflags 2
lxc-start-2249 [006] ..... 47.943424: io_uring_cqring_wait: ring 00000000ad366d59, min_events 1
kworker/u32:7-151 [012] d..1. 47.943452: io_uring_task_add: ring 00000000ad366d59, op 6, data 0x56049a674bd0, mask 41
lxc-start-2249 [006] ...1. 47.943479: io_uring_complete: ring 00000000ad366d59, user_data 0x56049a674bd0, result 1, cflags 2
lxc-start-2249 [006] ..... 47.943485: io_uring_cqring_wait: ring 00000000ad366d59, min_events 1
kworker/u32:7-151 [012] d..1. 47.943656: io_uring_task_add: ring 00000000ad366d59, op 6, data 0x56049a674bd0, mask 41
lxc-start-2249 [006] ...1. 47.943684: io_uring_complete: ring 00000000ad366d59, user_data 0x56049a674bd0, result 1, cflags 2
lxc-start-2249 [006] ..... 47.943691: io_uring_cqring_wait: ring 00000000ad366d59, min_events 1
kworker/u32:7-151 [012] d..1. 47.943709: io_uring_task_add: ring 00000000ad366d59, op 6, data 0x56049a674bd0, mask 41
lxc-start-2249 [006] ...1. 47.943724: io_uring_complete: ring 00000000ad366d59, user_data 0x56049a674bd0, result 1, cflags 2
lxc-start-2249 [006] ..... 47.943727: io_uring_cqring_wait: ring 00000000ad366d59, min_events 1
kworker/u32:6-148 [005] d..1. 47.943742: io_uring_task_add: ring 00000000ad366d59, op 6, data 0x56049a674bd0, mask 41
lxc-start-2249 [006] ...1. 47.943768: io_uring_complete: ring 00000000ad366d59, user_data 0x56049a674bd0, result 1, cflags 2
lxc-start-2249 [006] ..... 47.943774: io_uring_cqring_wait: ring 00000000ad366d59, min_events 1
kworker/u32:6-148 [005] d..1. 47.944355: io_uring_task_add: ring 00000000ad366d59, op 6, data 0x56049a674bd0, mask 41
lxc-start-2249 [000] ...1. 47.944382: io_uring_complete: ring 00000000ad366d59, user_data 0x56049a674bd0, result 1, cflags 2
lxc-start-2249 [000] ..... 47.944389: io_uring_cqring_wait: ring 00000000ad366d59, min_events 1
kworker/u32:6-148 [005] d..1. 47.944882: io_uring_task_add: ring 00000000ad366d59, op 6, data 0x56049a674bd0, mask 41
lxc-start-2249 [002] ...1. 47.944908: io_uring_complete: ring 00000000ad366d59, user_data 0x56049a674bd0, result 1, cflags 2
lxc-start-2249 [002] ..... 47.944914: io_uring_cqring_wait: ring 00000000ad366d59, min_events 1
kworker/u32:6-148 [005] d..1. 47.945357: io_uring_task_add: ring 00000000ad366d59, op 6, data 0x56049a674bd0, mask 41
lxc-start-2249 [002] ...1. 47.945382: io_uring_complete: ring 00000000ad366d59, user_data 0x56049a674bd0, result 1, cflags 2
lxc-start-2249 [002] ..... 47.945388: io_uring_cqring_wait: ring 00000000ad366d59, min_events 1
kworker/u32:9-1078 [003] d..1. 47.954651: io_uring_task_add: ring 00000000ad366d59, op 6, data 0x56049a674bd0, mask 41
lxc-start-2249 [002] ...1. 47.954665: io_uring_complete: ring 00000000ad366d59, user_data 0x56049a674bd0, result 1, cflags 2
lxc-start-2249 [002] ..... 47.954676: io_uring_cqring_wait: ring 00000000ad366d59, min_events 1
kworker/u32:7-151 [006] d..1. 47.955367: io_uring_task_add: ring 00000000ad366d59, op 6, data 0x56049a674bd0, mask 41
lxc-start-2249 [002] ...1. 47.955381: io_uring_complete: ring 00000000ad366d59, user_data 0x56049a674bd0, result 1, cflags 2
lxc-start-2249 [002] ..... 47.955392: io_uring_cqring_wait: ring 00000000ad366d59, min_events 1
kworker/u32:9-1078 [000] d..1. 47.955791: io_uring_task_add: ring 00000000ad366d59, op 6, data 0x56049a674bd0, mask 41
lxc-start-2249 [002] ...1. 47.955798: io_uring_complete: ring 00000000ad366d59, user_data 0x56049a674bd0, result 1, cflags 2
lxc-start-2249 [002] ..... 47.955800: io_uring_cqring_wait: ring 00000000ad366d59, min_events 1
kworker/u32:7-151 [006] d..1. 47.956563: io_uring_task_add: ring 00000000ad366d59, op 6, data 0x56049a674bd0, mask 41
lxc-start-2249 [002] ...1. 47.956574: io_uring_complete: ring 00000000ad366d59, user_data 0x56049a674bd0, result 1, cflags 2
lxc-start-2249 [002] ..... 47.956585: io_uring_cqring_wait: ring 00000000ad366d59, min_events 1
kworker/u32:7-151 [006] d..1. 47.956808: io_uring_task_add: ring 00000000ad366d59, op 6, data 0x56049a674bd0, mask 41
lxc-start-2249 [002] ...1. 47.956816: io_uring_complete: ring 00000000ad366d59, user_data 0x56049a674bd0, result 1, cflags 2
lxc-start-2249 [002] ..... 47.956819: io_uring_cqring_wait: ring 00000000ad366d59, min_events 1
kworker/u32:7-151 [006] d..1. 47.957256: io_uring_task_add: ring 00000000ad366d59, op 6, data 0x56049a674bd0, mask 41
lxc-start-2249 [002] ...1. 47.957265: io_uring_complete: ring 00000000ad366d59, user_data 0x56049a674bd0, result 1, cflags 2
lxc-start-2249 [002] ..... 47.957275: io_uring_cqring_wait: ring 00000000ad366d59, min_events 1
kworker/u32:7-151 [006] d..1. 47.957353: io_uring_task_add: ring 00000000ad366d59, op 6, data 0x56049a674bd0, mask 41
lxc-start-2249 [002] ...1. 47.957361: io_uring_complete: ring 00000000ad366d59, user_data 0x56049a674bd0, result 1, cflags 2
lxc-start-2249 [002] ..... 47.957362: io_uring_cqring_wait: ring 00000000ad366d59, min_events 1
kworker/u32:7-151 [004] d..1. 47.960507: io_uring_task_add: ring 00000000ad366d59, op 6, data 0x56049a674bd0, mask 41
lxc-start-2249 [005] ...1. 47.960518: io_uring_complete: ring 00000000ad366d59, user_data 0x56049a674bd0, result 1, cflags 2
lxc-start-2249 [005] ..... 47.960529: io_uring_cqring_wait: ring 00000000ad366d59, min_events 1
kworker/u32:9-1078 [000] d..1. 47.961340: io_uring_task_add: ring 00000000ad366d59, op 6, data 0x56049a674bd0, mask 41
lxc-start-2249 [001] ...1. 47.961355: io_uring_complete: ring 00000000ad366d59, user_data 0x56049a674bd0, result 1, cflags 2
lxc-start-2249 [001] ..... 47.961360: io_uring_cqring_wait: ring 00000000ad366d59, min_events 1
kworker/u32:11-2181 [014] d..1. 48.029152: io_uring_task_add: ring 00000000ad366d59, op 6, data 0x56049a674bd0, mask 41
lxc-start-2249 [005] ...1. 48.029182: io_uring_complete: ring 00000000ad366d59, user_data 0x56049a674bd0, result 1, cflags 2
lxc-start-2249 [005] ..... 48.029196: io_uring_cqring_wait: ring 00000000ad366d59, min_events 1
kworker/u32:11-2181 [000] d..1. 48.029815: io_uring_task_add: ring 00000000ad366d59, op 6, data 0x56049a674bd0, mask 41
lxc-start-2249 [005] ...1. 48.029828: io_uring_complete: ring 00000000ad366d59, user_data 0x56049a674bd0, result 1, cflags 2
lxc-start-2249 [005] ..... 48.029832: io_uring_cqring_wait: ring 00000000ad366d59, min_events 1
kworker/u32:11-2181 [000] d..1. 48.033023: io_uring_task_add: ring 00000000ad366d59, op 6, data 0x56049a674bd0, mask 41
lxc-start-2249 [005] ...1. 48.033040: io_uring_complete: ring 00000000ad366d59, user_data 0x56049a674bd0, result 1, cflags 2
lxc-start-2249 [005] ..... 48.033048: io_uring_cqring_wait: ring 00000000ad366d59, min_events 1
kworker/u32:8-156 [006] d..1. 49.035170: io_uring_task_add: ring 00000000ad366d59, op 6, data 0x56049a674bd0, mask 41
lxc-start-2249 [005] ...1. 49.035232: io_uring_complete: ring 00000000ad366d59, user_data 0x56049a674bd0, result 1, cflags 2
lxc-start-2249 [005] ..... 49.035261: io_uring_cqring_wait: ring 00000000ad366d59, min_events 1
kworker/u32:8-156 [006] d..1. 49.042316: io_uring_task_add: ring 00000000ad366d59, op 6, data 0x56049a674bd0, mask 41
lxc-start-2249 [005] ...1. 49.042361: io_uring_complete: ring 00000000ad366d59, user_data 0x56049a674bd0, result 1, cflags 2
kworker/u32:7-151 [004] d..1. 49.042367: io_uring_task_add: ring 00000000ad366d59, op 6, data 0x56049a674bd0, mask 41
lxc-start-2249 [005] ...1. 49.042368: io_uring_complete: ring 00000000ad366d59, user_data 0x56049a674bd0, result 1, cflags 2
lxc-stop-2534 [003] d..1. 52.782291: io_uring_task_add: ring 00000000ad366d59, op 6, data 0x56049a671380, mask c3
lxc-start-2249 [005] ...1. 52.782314: io_uring_complete: ring 00000000ad366d59, user_data 0x56049a671380, result 1, cflags 2
[-- Attachment #4: lxc-record-trace --]
[-- Type: text/plain, Size: 372 bytes --]
#!/bin/bash
sudo -v
echo 0 | sudo dd status=none of=/sys/kernel/debug/tracing/trace
echo 1 | sudo dd status=none of=/sys/kernel/debug/tracing/events/io_uring/enable
sudo lxc-start -n lxc-test
sudo lxc-stop -n lxc-test &
sleep 1
sudo cat /sys/kernel/debug/tracing/trace > ~/lxc-trace
echo 0 | sudo dd status=none of=/sys/kernel/debug/tracing/events/io_uring/enable
next prev parent reply other threads:[~2022-05-03 7:37 UTC|newest]
Thread overview: 28+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-05-02 13:17 [REGRESSION] lxc-stop hang on 5.17.x kernels Daniel Harding
2022-05-02 13:26 ` Jens Axboe
2022-05-02 13:36 ` Daniel Harding
2022-05-02 13:59 ` Jens Axboe
2022-05-02 17:00 ` Jens Axboe
2022-05-02 17:40 ` Pavel Begunkov
2022-05-02 18:49 ` Daniel Harding
2022-05-02 23:14 ` Pavel Begunkov
2022-05-03 7:13 ` Daniel Harding
2022-05-03 7:37 ` Daniel Harding [this message]
2022-05-03 14:14 ` Pavel Begunkov
2022-05-04 6:54 ` Daniel Harding
2022-05-15 8:20 ` Thorsten Leemhuis
2022-05-15 18:34 ` Daniel Harding
2022-05-16 12:12 ` Pavel Begunkov
2022-05-16 13:25 ` Pavel Begunkov
2022-05-16 13:57 ` Daniel Harding
2022-05-16 15:13 ` Daniel Harding
2022-05-16 18:13 ` Pavel Begunkov
2022-05-17 8:19 ` Christian Brauner
2022-05-17 10:31 ` Pavel Begunkov
2022-05-16 18:17 ` Thorsten Leemhuis
2022-05-16 18:22 ` Jens Axboe
2022-05-16 18:34 ` Thorsten Leemhuis
2022-05-16 18:39 ` Jens Axboe
2022-05-16 19:07 ` Thorsten Leemhuis
2022-05-16 19:14 ` Jens Axboe
2022-05-17 9:08 ` [REGRESSION] lxc-stop hang on 5.17.x kernels #forregzbot Thorsten Leemhuis
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=a204ba93-7261-5c6e-1baf-e5427e26b124@living180.net \
--to=dharding@living180.net \
--cc=asml.silence@gmail.com \
--cc=axboe@kernel.dk \
--cc=io-uring@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=regressions@lists.linux.dev \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).