From: Sagi Grimberg <sagi@grimberg.me>
To: "linux-block@vger.kernel.org" <linux-block@vger.kernel.org>
Subject: soft lockup with io_uring
Date: Wed, 21 Aug 2019 15:48:14 -0700 [thread overview]
Message-ID: <2f87ee3f-d61f-e572-08f5-96a8ef8843b0@grimberg.me> (raw)
Hey,
Just ran io-uring-bench on my VM to /dev/nullb0 and got the following
soft lockup [1], the reproducer is as simple as:
modprobe null_blk
tools/io_uring/io_uring-bench /dev/nullb0
It looks like io_iopoll_getevents() can hog the cpu, however I don't
yet really know what is preventing it from quickly exceeding min and
punting back...
Adding this makes the problem go away:
--
diff --git a/fs/io_uring.c b/fs/io_uring.c
index 8b9dbf3b2298..aba03eee5c81 100644
--- a/fs/io_uring.c
+++ b/fs/io_uring.c
@@ -779,6 +779,7 @@ static int io_iopoll_getevents(struct io_ring_ctx
*ctx, unsigned int *nr_events,
return ret;
if (!min || *nr_events >= min)
return 0;
+ cond_resched();
}
return 1;
--
But I do not know if this is the correct way to fix this, or what
exactly is the issue, but thought I send it out given its so
easy to reproduce.
[1]:
--
[1032930.615999] watchdog: BUG: soft lockup - CPU#2 stuck for 22s!
[io_uring-bench:21221]
[1032930.626182] RIP: 0010:io_iopoll_getevents+0xaf/0x260
[...]
[1032930.626942] Call Trace:
[1032930.627598] io_iopoll_check+0x34/0x70
[1032930.627601] __x64_sys_io_uring_enter+0x183/0x300
[1032930.627729] do_syscall_64+0x55/0x130
[1032930.627911] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[1032930.627927] RIP: 0033:0x7fa4641d7839
[...]
[1032958.615873] watchdog: BUG: soft lockup - CPU#2 stuck for 22s!
[io_uring-bench:21221]
[1032958.720822] RIP: 0010:blkdev_iopoll+0x0/0x30
[...]
[1032958.721116] Call Trace:
[1032958.721624] io_iopoll_getevents+0xaa/0x260
[1032958.721635] io_iopoll_check+0x34/0x70
[1032958.721637] __x64_sys_io_uring_enter+0x183/0x300
[1032958.721642] do_syscall_64+0x55/0x130
[1032958.721649] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[1032958.721654] RIP: 0033:0x7fa4641d7839
[...]
--
next reply other threads:[~2019-08-21 22:48 UTC|newest]
Thread overview: 3+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-08-21 22:48 Sagi Grimberg [this message]
2019-08-22 0:55 ` soft lockup with io_uring Jens Axboe
[not found] ` <e14439c4-b59a-2aa0-6cf1-1ef54e70b14b@grimberg.me>
2019-08-22 1:33 ` Jens Axboe
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=2f87ee3f-d61f-e572-08f5-96a8ef8843b0@grimberg.me \
--to=sagi@grimberg.me \
--cc=linux-block@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.