* [PATCH 0/2] optimise blk_try_enter_queue()
@ 2021-10-20 20:03 Pavel Begunkov
2021-10-20 20:03 ` [PATCH 1/2] percpu_ref: percpu_ref_tryget_live() version holding RCU Pavel Begunkov
2021-10-20 20:03 ` [PATCH 2/2] block: kill extra rcu lock/unlock in queue enter Pavel Begunkov
0 siblings, 2 replies; 5+ messages in thread
From: Pavel Begunkov @ 2021-10-20 20:03 UTC (permalink / raw)
To: linux-block
Cc: Jens Axboe, linux-kernel, linux-mm, Christoph Lameter, Tejun Heo,
Dennis Zhou, Pavel Begunkov
Kill extra rcu_read_lock/unlock() pair in blk_try_enter_queue().
Testing with io_uring (high batching) with nullblk:
Before:
3.20% io_uring [kernel.vmlinux] [k] __rcu_read_unlock
3.05% io_uring [kernel.vmlinux] [k] __rcu_read_lock
After:
2.52% io_uring [kernel.vmlinux] [k] __rcu_read_unlock
2.28% io_uring [kernel.vmlinux] [k] __rcu_read_lock
Doesn't necessarily translates into 1.4% perfofrmance improvement
but nice to have.
Pavel Begunkov (2):
percpu_ref: percpu_ref_tryget_live() version holding RCU
block: kill extra rcu lock/unlock in queue enter
block/blk-core.c | 2 +-
include/linux/percpu-refcount.h | 31 +++++++++++++++++++++----------
2 files changed, 22 insertions(+), 11 deletions(-)
--
2.33.1
^ permalink raw reply [flat|nested] 5+ messages in thread
* [PATCH 1/2] percpu_ref: percpu_ref_tryget_live() version holding RCU
2021-10-20 20:03 [PATCH 0/2] optimise blk_try_enter_queue() Pavel Begunkov
@ 2021-10-20 20:03 ` Pavel Begunkov
2021-10-20 20:10 ` Tejun Heo
2021-10-20 20:03 ` [PATCH 2/2] block: kill extra rcu lock/unlock in queue enter Pavel Begunkov
1 sibling, 1 reply; 5+ messages in thread
From: Pavel Begunkov @ 2021-10-20 20:03 UTC (permalink / raw)
To: linux-block
Cc: Jens Axboe, linux-kernel, linux-mm, Christoph Lameter, Tejun Heo,
Dennis Zhou, Pavel Begunkov
Add percpu_ref_tryget_live_rcu(), which is a version of
percpu_ref_tryget_live() but the user is responsible for enclosing it in
a RCU read lock section.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
---
include/linux/percpu-refcount.h | 31 +++++++++++++++++++++----------
1 file changed, 21 insertions(+), 10 deletions(-)
diff --git a/include/linux/percpu-refcount.h b/include/linux/percpu-refcount.h
index ae16a9856305..13e26add7d5c 100644
--- a/include/linux/percpu-refcount.h
+++ b/include/linux/percpu-refcount.h
@@ -266,6 +266,26 @@ static inline bool percpu_ref_tryget(struct percpu_ref *ref)
return percpu_ref_tryget_many(ref, 1);
}
+/**
+ * percpu_ref_tryget_live_rcu - same as percpu_ref_tryget_live() but the
+ * caller is responsible for taking RCU.
+ *
+ * This function is safe to call as long as @ref is between init and exit.
+ */
+static inline bool percpu_ref_tryget_live_rcu(struct percpu_ref *ref)
+{
+ unsigned long __percpu *percpu_count;
+ bool ret = false;
+
+ if (likely(__ref_is_percpu(ref, &percpu_count))) {
+ this_cpu_inc(*percpu_count);
+ ret = true;
+ } else if (!(ref->percpu_count_ptr & __PERCPU_REF_DEAD)) {
+ ret = atomic_long_inc_not_zero(&ref->data->count);
+ }
+ return ret;
+}
+
/**
* percpu_ref_tryget_live - try to increment a live percpu refcount
* @ref: percpu_ref to try-get
@@ -283,20 +303,11 @@ static inline bool percpu_ref_tryget(struct percpu_ref *ref)
*/
static inline bool percpu_ref_tryget_live(struct percpu_ref *ref)
{
- unsigned long __percpu *percpu_count;
bool ret = false;
rcu_read_lock();
-
- if (__ref_is_percpu(ref, &percpu_count)) {
- this_cpu_inc(*percpu_count);
- ret = true;
- } else if (!(ref->percpu_count_ptr & __PERCPU_REF_DEAD)) {
- ret = atomic_long_inc_not_zero(&ref->data->count);
- }
-
+ ret = percpu_ref_tryget_live_rcu(ref);
rcu_read_unlock();
-
return ret;
}
--
2.33.1
^ permalink raw reply related [flat|nested] 5+ messages in thread
* Re: [PATCH 1/2] percpu_ref: percpu_ref_tryget_live() version holding RCU
2021-10-20 20:03 ` [PATCH 1/2] percpu_ref: percpu_ref_tryget_live() version holding RCU Pavel Begunkov
@ 2021-10-20 20:10 ` Tejun Heo
2021-10-21 8:41 ` Pavel Begunkov
0 siblings, 1 reply; 5+ messages in thread
From: Tejun Heo @ 2021-10-20 20:10 UTC (permalink / raw)
To: Pavel Begunkov
Cc: linux-block, Jens Axboe, linux-kernel, linux-mm,
Christoph Lameter, Dennis Zhou
On Wed, Oct 20, 2021 at 09:03:18PM +0100, Pavel Begunkov wrote:
> +/**
> + * percpu_ref_tryget_live_rcu - same as percpu_ref_tryget_live() but the
> + * caller is responsible for taking RCU.
> + *
> + * This function is safe to call as long as @ref is between init and exit.
> + */
> +static inline bool percpu_ref_tryget_live_rcu(struct percpu_ref *ref)
> +{
> + unsigned long __percpu *percpu_count;
> + bool ret = false;
> +
> + if (likely(__ref_is_percpu(ref, &percpu_count))) {
> + this_cpu_inc(*percpu_count);
> + ret = true;
> + } else if (!(ref->percpu_count_ptr & __PERCPU_REF_DEAD)) {
> + ret = atomic_long_inc_not_zero(&ref->data->count);
> + }
> + return ret;
> +}
Can we please add rcu_read_lock_held() assertion? Other than that, looks
fine to me.
Thanks.
--
tejun
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH 1/2] percpu_ref: percpu_ref_tryget_live() version holding RCU
2021-10-20 20:10 ` Tejun Heo
@ 2021-10-21 8:41 ` Pavel Begunkov
0 siblings, 0 replies; 5+ messages in thread
From: Pavel Begunkov @ 2021-10-21 8:41 UTC (permalink / raw)
To: Tejun Heo
Cc: linux-block, Jens Axboe, linux-kernel, linux-mm,
Christoph Lameter, Dennis Zhou
On 10/20/21 21:10, Tejun Heo wrote:
> On Wed, Oct 20, 2021 at 09:03:18PM +0100, Pavel Begunkov wrote:
>> +/**
>> + * percpu_ref_tryget_live_rcu - same as percpu_ref_tryget_live() but the
>> + * caller is responsible for taking RCU.
>> + *
>> + * This function is safe to call as long as @ref is between init and exit.
>> + */
>> +static inline bool percpu_ref_tryget_live_rcu(struct percpu_ref *ref)
>> +{
>> + unsigned long __percpu *percpu_count;
>> + bool ret = false;
>> +
>> + if (likely(__ref_is_percpu(ref, &percpu_count))) {
>> + this_cpu_inc(*percpu_count);
>> + ret = true;
>> + } else if (!(ref->percpu_count_ptr & __PERCPU_REF_DEAD)) {
>> + ret = atomic_long_inc_not_zero(&ref->data->count);
>> + }
>> + return ret;
>> +}
>
> Can we please add rcu_read_lock_held() assertion? Other than that, looks
> fine to me.
Will add, thanks
--
Pavel Begunkov
^ permalink raw reply [flat|nested] 5+ messages in thread
* [PATCH 2/2] block: kill extra rcu lock/unlock in queue enter
2021-10-20 20:03 [PATCH 0/2] optimise blk_try_enter_queue() Pavel Begunkov
2021-10-20 20:03 ` [PATCH 1/2] percpu_ref: percpu_ref_tryget_live() version holding RCU Pavel Begunkov
@ 2021-10-20 20:03 ` Pavel Begunkov
1 sibling, 0 replies; 5+ messages in thread
From: Pavel Begunkov @ 2021-10-20 20:03 UTC (permalink / raw)
To: linux-block
Cc: Jens Axboe, linux-kernel, linux-mm, Christoph Lameter, Tejun Heo,
Dennis Zhou, Pavel Begunkov
blk_try_enter_queue() already takes rcu_read_lock/unlock, so we can
avoid the second pair in percpu_ref_tryget_live(), use a newly added
percpu_ref_tryget_live_rcu().
As rcu_read_lock/unlock imply barrier()s, it's pretty noticeable,
especially for for !CONFIG_PREEMPT_RCU (default for some distributions),
where __rcu_read_lock/unlock() are not inlined.
3.20% io_uring [kernel.vmlinux] [k] __rcu_read_unlock
3.05% io_uring [kernel.vmlinux] [k] __rcu_read_lock
2.52% io_uring [kernel.vmlinux] [k] __rcu_read_unlock
2.28% io_uring [kernel.vmlinux] [k] __rcu_read_lock
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
---
block/blk-core.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/block/blk-core.c b/block/blk-core.c
index 88752e51d2b6..20e76aeb50f5 100644
--- a/block/blk-core.c
+++ b/block/blk-core.c
@@ -389,7 +389,7 @@ EXPORT_SYMBOL(blk_cleanup_queue);
static bool blk_try_enter_queue(struct request_queue *q, bool pm)
{
rcu_read_lock();
- if (!percpu_ref_tryget_live(&q->q_usage_counter))
+ if (!percpu_ref_tryget_live_rcu(&q->q_usage_counter))
goto fail;
/*
--
2.33.1
^ permalink raw reply related [flat|nested] 5+ messages in thread
end of thread, other threads:[~2021-10-21 8:42 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-10-20 20:03 [PATCH 0/2] optimise blk_try_enter_queue() Pavel Begunkov
2021-10-20 20:03 ` [PATCH 1/2] percpu_ref: percpu_ref_tryget_live() version holding RCU Pavel Begunkov
2021-10-20 20:10 ` Tejun Heo
2021-10-21 8:41 ` Pavel Begunkov
2021-10-20 20:03 ` [PATCH 2/2] block: kill extra rcu lock/unlock in queue enter Pavel Begunkov
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).