All of lore.kernel.org
 help / color / mirror / Atom feed
From: Pavel Begunkov <asml.silence@gmail.com>
To: Jens Axboe <axboe@kernel.dk>, io-uring@vger.kernel.org
Subject: [PATCH 12/14] io_uring: help inlining of io_req_complete()
Date: Tue, 19 Jan 2021 13:32:45 +0000	[thread overview]
Message-ID: <4ca07fa0795c558ca59f8e053bcb4cff996607c9.1611062505.git.asml.silence@gmail.com> (raw)
In-Reply-To: <cover.1611062505.git.asml.silence@gmail.com>

__io_req_complete() inlining is a bit weird, some compilers don't
optimise out the non-NULL branch of it even when called as
io_req_complete(). Help it a bit by extracting state and stateless
helpers out of __io_req_complete().

Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
---
 fs/io_uring.c | 36 +++++++++++++++++++++---------------
 1 file changed, 21 insertions(+), 15 deletions(-)

diff --git a/fs/io_uring.c b/fs/io_uring.c
index a004102fbbde..b0f54f4495c7 100644
--- a/fs/io_uring.c
+++ b/fs/io_uring.c
@@ -1868,7 +1868,8 @@ static void io_cqring_fill_event(struct io_kiocb *req, long res)
 	__io_cqring_fill_event(req, res, 0);
 }
 
-static void io_cqring_add_event(struct io_kiocb *req, long res, long cflags)
+static void io_req_complete_nostate(struct io_kiocb *req, long res,
+				    unsigned int cflags)
 {
 	struct io_ring_ctx *ctx = req->ctx;
 	unsigned long flags;
@@ -1879,6 +1880,7 @@ static void io_cqring_add_event(struct io_kiocb *req, long res, long cflags)
 	spin_unlock_irqrestore(&ctx->completion_lock, flags);
 
 	io_cqring_ev_posted(ctx);
+	io_put_req(req);
 }
 
 static void io_submit_flush_completions(struct io_comp_state *cs)
@@ -1914,23 +1916,27 @@ static void io_submit_flush_completions(struct io_comp_state *cs)
 	cs->nr = 0;
 }
 
-static void __io_req_complete(struct io_kiocb *req, long res, unsigned cflags,
-			      struct io_comp_state *cs)
+static void io_req_complete_state(struct io_kiocb *req, long res,
+				  unsigned int cflags, struct io_comp_state *cs)
 {
-	if (!cs) {
-		io_cqring_add_event(req, res, cflags);
-		io_put_req(req);
-	} else {
-		io_clean_op(req);
-		req->result = res;
-		req->compl.cflags = cflags;
-		list_add_tail(&req->compl.list, &cs->list);
-		if (++cs->nr >= 32)
-			io_submit_flush_completions(cs);
-	}
+	io_clean_op(req);
+	req->result = res;
+	req->compl.cflags = cflags;
+	list_add_tail(&req->compl.list, &cs->list);
+	if (++cs->nr >= 32)
+		io_submit_flush_completions(cs);
+}
+
+static inline void __io_req_complete(struct io_kiocb *req, long res,
+				     unsigned cflags, struct io_comp_state *cs)
+{
+	if (!cs)
+		io_req_complete_nostate(req, res, cflags);
+	else
+		io_req_complete_state(req, res, cflags, cs);
 }
 
-static void io_req_complete(struct io_kiocb *req, long res)
+static inline void io_req_complete(struct io_kiocb *req, long res)
 {
 	__io_req_complete(req, res, 0, NULL);
 }
-- 
2.24.0


  parent reply	other threads:[~2021-01-19 14:57 UTC|newest]

Thread overview: 18+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-01-19 13:32 [PATCH for-next 00/14] mostly cleanups for 5.12 Pavel Begunkov
2021-01-19 13:32 ` [PATCH 01/14] io_uring: optimise io_rw_reissue() Pavel Begunkov
2021-01-19 13:32 ` [PATCH 02/14] io_uring: refactor io_resubmit_prep() Pavel Begunkov
2021-01-19 13:32 ` [PATCH 03/14] io_uring: cleanup personalities under uring_lock Pavel Begunkov
2021-01-19 13:32 ` [PATCH 04/14] io_uring: inline io_async_submit() Pavel Begunkov
2021-01-19 13:32 ` [PATCH 05/14] io_uring: inline __io_commit_cqring() Pavel Begunkov
2021-01-19 13:32 ` [PATCH 06/14] io_uring: further deduplicate #CQ events calc Pavel Begunkov
2021-01-19 13:32 ` [PATCH 07/14] io_uring: simplify io_alloc_req() Pavel Begunkov
2021-01-19 13:32 ` [PATCH 08/14] io_uring: remove __io_state_file_put Pavel Begunkov
2021-01-19 13:32 ` [PATCH 09/14] io_uring: deduplicate failing task_work_add Pavel Begunkov
2021-01-19 13:32 ` [PATCH 10/14] io_uring: don't block resource recycle by oveflows Pavel Begunkov
2021-01-19 22:59   ` Jens Axboe
2021-01-19 23:08     ` Pavel Begunkov
2021-01-19 13:32 ` [PATCH 11/14] io_uring: add a helper timeout mode calculation Pavel Begunkov
2021-01-19 13:32 ` Pavel Begunkov [this message]
2021-01-19 13:32 ` [PATCH 13/14] io_uring: don't flush CQEs deep down the stack Pavel Begunkov
2021-01-19 13:32 ` [PATCH 14/14] io_uring: save atomic dec for inline executed reqs Pavel Begunkov
2021-01-19 23:01 ` [PATCH for-next 00/14] mostly cleanups for 5.12 Jens Axboe

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=4ca07fa0795c558ca59f8e053bcb4cff996607c9.1611062505.git.asml.silence@gmail.com \
    --to=asml.silence@gmail.com \
    --cc=axboe@kernel.dk \
    --cc=io-uring@vger.kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body. 
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.