* [PATCH 0/2] optimise blk_try_enter_queue()
@ 2021-10-20 20:03 Pavel Begunkov
2021-10-20 20:03 ` [PATCH 1/2] percpu_ref: percpu_ref_tryget_live() version holding RCU Pavel Begunkov
2021-10-20 20:03 ` [PATCH 2/2] block: kill extra rcu lock/unlock in queue enter Pavel Begunkov
0 siblings, 2 replies; 5+ messages in thread
From: Pavel Begunkov @ 2021-10-20 20:03 UTC (permalink / raw)
To: linux-block
Cc: Jens Axboe, linux-kernel, linux-mm, Christoph Lameter, Tejun Heo,
Dennis Zhou, Pavel Begunkov
Kill extra rcu_read_lock/unlock() pair in blk_try_enter_queue().
Testing with io_uring (high batching) with nullblk:
Before:
3.20% io_uring [kernel.vmlinux] [k] __rcu_read_unlock
3.05% io_uring [kernel.vmlinux] [k] __rcu_read_lock
After:
2.52% io_uring [kernel.vmlinux] [k] __rcu_read_unlock
2.28% io_uring [kernel.vmlinux] [k] __rcu_read_lock
Doesn't necessarily translates into 1.4% perfofrmance improvement
but nice to have.
Pavel Begunkov (2):
percpu_ref: percpu_ref_tryget_live() version holding RCU
block: kill extra rcu lock/unlock in queue enter
block/blk-core.c | 2 +-
include/linux/percpu-refcount.h | 31 +++++++++++++++++++++----------
2 files changed, 22 insertions(+), 11 deletions(-)
--
2.33.1
^ permalink raw reply [flat|nested] 5+ messages in thread
* [PATCH 1/2] percpu_ref: percpu_ref_tryget_live() version holding RCU
2021-10-20 20:03 [PATCH 0/2] optimise blk_try_enter_queue() Pavel Begunkov
@ 2021-10-20 20:03 ` Pavel Begunkov
2021-10-20 20:10 ` Tejun Heo
2021-10-20 20:03 ` [PATCH 2/2] block: kill extra rcu lock/unlock in queue enter Pavel Begunkov
1 sibling, 1 reply; 5+ messages in thread
From: Pavel Begunkov @ 2021-10-20 20:03 UTC (permalink / raw)
To: linux-block
Cc: Jens Axboe, linux-kernel, linux-mm, Christoph Lameter, Tejun Heo,
Dennis Zhou, Pavel Begunkov
Add percpu_ref_tryget_live_rcu(), which is a version of
percpu_ref_tryget_live() but the user is responsible for enclosing it in
a RCU read lock section.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
---
include/linux/percpu-refcount.h | 31 +++++++++++++++++++++----------
1 file changed, 21 insertions(+), 10 deletions(-)
diff --git a/include/linux/percpu-refcount.h b/include/linux/percpu-refcount.h
index ae16a9856305..13e26add7d5c 100644
--- a/include/linux/percpu-refcount.h
+++ b/include/linux/percpu-refcount.h
@@ -266,6 +266,26 @@ static inline bool percpu_ref_tryget(struct percpu_ref *ref)
return percpu_ref_tryget_many(ref, 1);
}
+/**
+ * percpu_ref_tryget_live_rcu - same as percpu_ref_tryget_live() but the
+ * caller is responsible for taking RCU.
+ *
+ * This function is safe to call as long as @ref is between init and exit.
+ */
+static inline bool percpu_ref_tryget_live_rcu(struct percpu_ref *ref)
+{
+ unsigned long __percpu *percpu_count;
+ bool ret = false;
+
+ if (likely(__ref_is_percpu(ref, &percpu_count))) {
+ this_cpu_inc(*percpu_count);
+ ret = true;
+ } else if (!(ref->percpu_count_ptr & __PERCPU_REF_DEAD)) {
+ ret = atomic_long_inc_not_zero(&ref->data->count);
+ }
+ return ret;
+}
+
/**
* percpu_ref_tryget_live - try to increment a live percpu refcount
* @ref: percpu_ref to try-get
@@ -283,20 +303,11 @@ static inline bool percpu_ref_tryget(struct percpu_ref *ref)
*/
static inline bool percpu_ref_tryget_live(struct percpu_ref *ref)
{
- unsigned long __percpu *percpu_count;
bool ret = false;
rcu_read_lock();
-
- if (__ref_is_percpu(ref, &percpu_count)) {
- this_cpu_inc(*percpu_count);
- ret = true;
- } else if (!(ref->percpu_count_ptr & __PERCPU_REF_DEAD)) {
- ret = atomic_long_inc_not_zero(&ref->data->count);
- }
-
+ ret = percpu_ref_tryget_live_rcu(ref);
rcu_read_unlock();
-
return ret;
}
--
2.33.1
^ permalink raw reply related [flat|nested] 5+ messages in thread
* Re: [PATCH 1/2] percpu_ref: percpu_ref_tryget_live() version holding RCU
2021-10-20 20:03 ` [PATCH 1/2] percpu_ref: percpu_ref_tryget_live() version holding RCU Pavel Begunkov
@ 2021-10-20 20:10 ` Tejun Heo
2021-10-21 8:41 ` Pavel Begunkov
0 siblings, 1 reply; 5+ messages in thread
From: Tejun Heo @ 2021-10-20 20:10 UTC (permalink / raw)
To: Pavel Begunkov
Cc: linux-block, Jens Axboe, linux-kernel, linux-mm,
Christoph Lameter, Dennis Zhou
On Wed, Oct 20, 2021 at 09:03:18PM +0100, Pavel Begunkov wrote:
> +/**
> + * percpu_ref_tryget_live_rcu - same as percpu_ref_tryget_live() but the
> + * caller is responsible for taking RCU.
> + *
> + * This function is safe to call as long as @ref is between init and exit.
> + */
> +static inline bool percpu_ref_tryget_live_rcu(struct percpu_ref *ref)
> +{
> + unsigned long __percpu *percpu_count;
> + bool ret = false;
> +
> + if (likely(__ref_is_percpu(ref, &percpu_count))) {
> + this_cpu_inc(*percpu_count);
> + ret = true;
> + } else if (!(ref->percpu_count_ptr & __PERCPU_REF_DEAD)) {
> + ret = atomic_long_inc_not_zero(&ref->data->count);
> + }
> + return ret;
> +}
Can we please add rcu_read_lock_held() assertion? Other than that, looks
fine to me.
Thanks.
--
tejun
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH 1/2] percpu_ref: percpu_ref_tryget_live() version holding RCU
2021-10-20 20:10 ` Tejun Heo
@ 2021-10-21 8:41 ` Pavel Begunkov
0 siblings, 0 replies; 5+ messages in thread
From: Pavel Begunkov @ 2021-10-21 8:41 UTC (permalink / raw)
To: Tejun Heo
Cc: linux-block, Jens Axboe, linux-kernel, linux-mm,
Christoph Lameter, Dennis Zhou
On 10/20/21 21:10, Tejun Heo wrote:
> On Wed, Oct 20, 2021 at 09:03:18PM +0100, Pavel Begunkov wrote:
>> +/**
>> + * percpu_ref_tryget_live_rcu - same as percpu_ref_tryget_live() but the
>> + * caller is responsible for taking RCU.
>> + *
>> + * This function is safe to call as long as @ref is between init and exit.
>> + */
>> +static inline bool percpu_ref_tryget_live_rcu(struct percpu_ref *ref)
>> +{
>> + unsigned long __percpu *percpu_count;
>> + bool ret = false;
>> +
>> + if (likely(__ref_is_percpu(ref, &percpu_count))) {
>> + this_cpu_inc(*percpu_count);
>> + ret = true;
>> + } else if (!(ref->percpu_count_ptr & __PERCPU_REF_DEAD)) {
>> + ret = atomic_long_inc_not_zero(&ref->data->count);
>> + }
>> + return ret;
>> +}
>
> Can we please add rcu_read_lock_held() assertion? Other than that, looks
> fine to me.
Will add, thanks
--
Pavel Begunkov
^ permalink raw reply [flat|nested] 5+ messages in thread
* [PATCH 2/2] block: kill extra rcu lock/unlock in queue enter
2021-10-20 20:03 [PATCH 0/2] optimise blk_try_enter_queue() Pavel Begunkov
2021-10-20 20:03 ` [PATCH 1/2] percpu_ref: percpu_ref_tryget_live() version holding RCU Pavel Begunkov
@ 2021-10-20 20:03 ` Pavel Begunkov
1 sibling, 0 replies; 5+ messages in thread
From: Pavel Begunkov @ 2021-10-20 20:03 UTC (permalink / raw)
To: linux-block
Cc: Jens Axboe, linux-kernel, linux-mm, Christoph Lameter, Tejun Heo,
Dennis Zhou, Pavel Begunkov
blk_try_enter_queue() already takes rcu_read_lock/unlock, so we can
avoid the second pair in percpu_ref_tryget_live(), use a newly added
percpu_ref_tryget_live_rcu().
As rcu_read_lock/unlock imply barrier()s, it's pretty noticeable,
especially for for !CONFIG_PREEMPT_RCU (default for some distributions),
where __rcu_read_lock/unlock() are not inlined.
3.20% io_uring [kernel.vmlinux] [k] __rcu_read_unlock
3.05% io_uring [kernel.vmlinux] [k] __rcu_read_lock
2.52% io_uring [kernel.vmlinux] [k] __rcu_read_unlock
2.28% io_uring [kernel.vmlinux] [k] __rcu_read_lock
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
---
block/blk-core.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/block/blk-core.c b/block/blk-core.c
index 88752e51d2b6..20e76aeb50f5 100644
--- a/block/blk-core.c
+++ b/block/blk-core.c
@@ -389,7 +389,7 @@ EXPORT_SYMBOL(blk_cleanup_queue);
static bool blk_try_enter_queue(struct request_queue *q, bool pm)
{
rcu_read_lock();
- if (!percpu_ref_tryget_live(&q->q_usage_counter))
+ if (!percpu_ref_tryget_live_rcu(&q->q_usage_counter))
goto fail;
/*
--
2.33.1
^ permalink raw reply related [flat|nested] 5+ messages in thread
end of thread, other threads:[~2021-10-21 8:42 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-10-20 20:03 [PATCH 0/2] optimise blk_try_enter_queue() Pavel Begunkov
2021-10-20 20:03 ` [PATCH 1/2] percpu_ref: percpu_ref_tryget_live() version holding RCU Pavel Begunkov
2021-10-20 20:10 ` Tejun Heo
2021-10-21 8:41 ` Pavel Begunkov
2021-10-20 20:03 ` [PATCH 2/2] block: kill extra rcu lock/unlock in queue enter Pavel Begunkov
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.