From: Pavel Begunkov <asml.silence@gmail.com>
To: Jens Axboe <axboe@kernel.dk>, io-uring@vger.kernel.org
Subject: [PATCH 10/12] io_uring: move uring_lock location
Date: Mon, 14 Jun 2021 23:37:29 +0100 [thread overview]
Message-ID: <dea5e845caee4c98aa0922b46d713154d81f7bd8.1623709150.git.asml.silence@gmail.com> (raw)
In-Reply-To: <cover.1623709150.git.asml.silence@gmail.com>
->uring_lock is prevalently used for submission, even though it protects
many other things like iopoll, registeration, selected bufs, and more.
And it's placed together with ->cq_wait poked on completion and CQ
waiting sides. Move them apart, ->uring_lock goes to the submission
data, and cq_wait to completion related chunk. The last one requires
some reshuffling so everything needed by io_cqring_ev_posted*() is in
one cacheline.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
---
fs/io_uring.c | 16 +++++++---------
1 file changed, 7 insertions(+), 9 deletions(-)
diff --git a/fs/io_uring.c b/fs/io_uring.c
index e9bf26fbf65d..1b6cfc6b79c5 100644
--- a/fs/io_uring.c
+++ b/fs/io_uring.c
@@ -356,6 +356,8 @@ struct io_ring_ctx {
/* submission data */
struct {
+ struct mutex uring_lock;
+
/*
* Ring buffer of indices into array of io_uring_sqe, which is
* mmapped by the application using the IORING_OFF_SQES offset.
@@ -392,11 +394,6 @@ struct io_ring_ctx {
unsigned sq_thread_idle;
} ____cacheline_aligned_in_smp;
- struct {
- struct mutex uring_lock;
- wait_queue_head_t cq_wait;
- } ____cacheline_aligned_in_smp;
-
/* IRQ completion list, under ->completion_lock */
struct list_head locked_free_list;
unsigned int locked_free_nr;
@@ -412,12 +409,13 @@ struct io_ring_ctx {
struct {
unsigned cached_cq_tail;
unsigned cq_entries;
- atomic_t cq_timeouts;
- unsigned cq_last_tm_flush;
- unsigned cq_extra;
+ struct eventfd_ctx *cq_ev_fd;
struct wait_queue_head poll_wait;
+ struct wait_queue_head cq_wait;
+ unsigned cq_extra;
+ atomic_t cq_timeouts;
struct fasync_struct *cq_fasync;
- struct eventfd_ctx *cq_ev_fd;
+ unsigned cq_last_tm_flush;
} ____cacheline_aligned_in_smp;
struct {
--
2.31.1
next prev parent reply other threads:[~2021-06-14 22:38 UTC|newest]
Thread overview: 15+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-06-14 22:37 [PATCH 5.14 00/12] for-next optimisations Pavel Begunkov
2021-06-14 22:37 ` [PATCH 01/12] io_uring: keep SQ pointers in a single cacheline Pavel Begunkov
2021-06-14 22:37 ` [PATCH 02/12] io_uring: move ctx->flags from SQ cacheline Pavel Begunkov
2021-06-14 22:37 ` [PATCH 03/12] io_uring: shuffle more fields into SQ ctx section Pavel Begunkov
2021-06-14 22:37 ` [PATCH 04/12] io_uring: refactor io_get_sqe() Pavel Begunkov
2021-06-14 22:37 ` [PATCH 05/12] io_uring: don't cache number of dropped SQEs Pavel Begunkov
2021-06-14 22:37 ` [PATCH 06/12] io_uring: optimise completion timeout flushing Pavel Begunkov
2021-06-14 22:37 ` [PATCH 07/12] io_uring: small io_submit_sqe() optimisation Pavel Begunkov
2021-06-14 22:37 ` [PATCH 08/12] io_uring: clean up check_overflow flag Pavel Begunkov
2021-06-14 22:37 ` [PATCH 09/12] io_uring: wait heads renaming Pavel Begunkov
2021-06-14 22:37 ` Pavel Begunkov [this message]
2021-06-14 22:37 ` [PATCH 11/12] io_uring: refactor io_req_defer() Pavel Begunkov
2021-06-14 22:37 ` [PATCH 12/12] io_uring: optimise non-drain path Pavel Begunkov
2021-06-15 12:27 ` [PATCH 5.14 00/12] for-next optimisations Pavel Begunkov
2021-06-15 21:38 ` Jens Axboe
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=dea5e845caee4c98aa0922b46d713154d81f7bd8.1623709150.git.asml.silence@gmail.com \
--to=asml.silence@gmail.com \
--cc=axboe@kernel.dk \
--cc=io-uring@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.