From: Jens Axboe <axboe@kernel.dk>
To: Pavel Begunkov <asml.silence@gmail.com>, io-uring@vger.kernel.org
Subject: Re: [PATCH RFC 00/17] playing around req alloc
Date: Tue, 9 Feb 2021 19:08:16 -0700 [thread overview]
Message-ID: <ffaf0640-7747-553d-9baf-4d41777a4d4d@kernel.dk> (raw)
In-Reply-To: <cover.1612915326.git.asml.silence@gmail.com>
On 2/9/21 5:03 PM, Pavel Begunkov wrote:
> Unfolding previous ideas on persistent req caches. 4-7 including
> slashed ~20% of overhead for nops benchmark, haven't done benchmarking
> personally for this yet, but according to perf should be ~30-40% in
> total. That's for IOPOLL + inline completion cases, obviously w/o
> async/IRQ completions.
And task_work, which is sort-of inline.
> Jens,
> 1. 11/17 removes deallocations on end of submit_sqes. Looks you
> forgot or just didn't do that.
>
> 2. lists are slow and not great cache-wise, that why at I want at least
> a combined approach from 12/17.
This is only true if you're browsing a full list. If you do add-to-front
for a cache, and remove-from-front, then cache footprint of lists are
good.
> 3. Instead of lists in "use persistent request cache" I had in mind a
> slightly different way: to grow the req alloc cache to 32-128 (or hint
> from the userspace), batch-alloc by 8 as before, and recycle _all_ reqs
> right into it. If overflows, do kfree().
> It should give probabilistically high hit rate, amortising out most of
> allocations. Pros: it doesn't grow ~infinitely as lists can. Cons: there
> are always counter examples. But as I don't have numbers to back it, I
> took your implementation. Maybe, we'll reconsider later.
It shouldn't grow bigger than what was used, but the downside is that
it will grow as big as the biggest usage ever. We could prune, if need
be, of course.
As far as I'm concerned, the hint from user space is the submit count.
> I'll revise tomorrow on a fresh head + do some performance testing,
> and is leaving it RFC until then.
I'll look too and test this, thanks!
--
Jens Axboe
next prev parent reply other threads:[~2021-02-10 2:11 UTC|newest]
Thread overview: 27+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-02-10 0:03 [PATCH RFC 00/17] playing around req alloc Pavel Begunkov
2021-02-10 0:03 ` [PATCH 01/17] io_uring: replace force_nonblock with flags Pavel Begunkov
2021-02-10 0:03 ` [PATCH 02/17] io_uring: make op handlers always take issue flags Pavel Begunkov
2021-02-10 0:03 ` [PATCH 03/17] io_uring: don't propagate io_comp_state Pavel Begunkov
2021-02-10 14:00 ` Pavel Begunkov
2021-02-10 14:27 ` Jens Axboe
2021-02-10 0:03 ` [PATCH 04/17] io_uring: don't keep submit_state on stack Pavel Begunkov
2021-02-10 0:03 ` [PATCH 05/17] io_uring: remove ctx from comp_state Pavel Begunkov
2021-02-10 0:03 ` [PATCH 06/17] io_uring: don't reinit submit state every time Pavel Begunkov
2021-02-10 0:03 ` [PATCH 07/17] io_uring: replace list with array for compl batch Pavel Begunkov
2021-02-10 0:03 ` [PATCH 08/17] io_uring: submit-completion free batching Pavel Begunkov
2021-02-10 0:03 ` [PATCH 09/17] io_uring: remove fallback_req Pavel Begunkov
2021-02-10 0:03 ` [PATCH 10/17] io_uring: count ctx refs separately from reqs Pavel Begunkov
2021-02-10 0:03 ` [PATCH 11/17] io_uring: persistent req cache Pavel Begunkov
2021-02-10 0:03 ` [PATCH 12/17] io_uring: feed reqs back into alloc cache Pavel Begunkov
2021-02-10 0:03 ` [PATCH 13/17] io_uring: use persistent request cache Pavel Begunkov
2021-02-10 2:14 ` Jens Axboe
2021-02-10 0:03 ` [PATCH 14/17] io_uring: provide FIFO ordering for task_work Pavel Begunkov
2021-02-10 0:03 ` [PATCH 15/17] io_uring: enable req cache for task_work items Pavel Begunkov
2021-02-10 0:03 ` [PATCH 16/17] io_uring: take comp_state from ctx Pavel Begunkov
2021-02-10 0:03 ` [PATCH 17/17] io_uring: defer flushing cached reqs Pavel Begunkov
2021-02-10 2:10 ` Jens Axboe
2021-02-10 2:08 ` Jens Axboe [this message]
2021-02-10 3:14 ` [PATCH RFC 00/17] playing around req alloc Pavel Begunkov
2021-02-10 3:23 ` Jens Axboe
2021-02-10 11:53 ` Pavel Begunkov
2021-02-10 14:27 ` Jens Axboe
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=ffaf0640-7747-553d-9baf-4d41777a4d4d@kernel.dk \
--to=axboe@kernel.dk \
--cc=asml.silence@gmail.com \
--cc=io-uring@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).