* [PATCH 2/4] io_uring: initialize percpu refcounters using PERCU_REF_ALLOW_REINIT
2019-05-07 17:01 [PATCH 1/4] percpu_ref: introduce PERCPU_REF_ALLOW_REINIT flag Roman Gushchin
@ 2019-05-07 17:01 ` Roman Gushchin
2019-05-07 17:01 ` [PATCH 3/4] md: " Roman Gushchin
` (2 subsequent siblings)
3 siblings, 0 replies; 6+ messages in thread
From: Roman Gushchin @ 2019-05-07 17:01 UTC (permalink / raw)
To: Tejun Heo
Cc: Jens Axboe, Song Liu, Dennis Zhou, linux-kernel, kernel-team,
Roman Gushchin
Percpu reference counters should now be initialized with the
PERCPU_REF_ALLOW_REINIT in order to allow switching them to the
percpu mode from the atomic mode. This is exactly what
percpu_ref_reinit() called from __io_uring_register() is supposed to
do. So let's initialize percpu refcounters with the
PERCU_REF_ALLOW_REINIT flag.
Signed-off-by: Roman Gushchin <guro@fb.com>
---
fs/io_uring.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/fs/io_uring.c b/fs/io_uring.c
index 452e35357865..9a9df175af31 100644
--- a/fs/io_uring.c
+++ b/fs/io_uring.c
@@ -397,7 +397,8 @@ static struct io_ring_ctx *io_ring_ctx_alloc(struct io_uring_params *p)
if (!ctx)
return NULL;
- if (percpu_ref_init(&ctx->refs, io_ring_ctx_ref_free, 0, GFP_KERNEL)) {
+ if (percpu_ref_init(&ctx->refs, io_ring_ctx_ref_free,
+ PERCPU_REF_ALLOW_REINIT, GFP_KERNEL)) {
kfree(ctx);
return NULL;
}
--
2.20.1
^ permalink raw reply related [flat|nested] 6+ messages in thread
* [PATCH 3/4] md: initialize percpu refcounters using PERCU_REF_ALLOW_REINIT
2019-05-07 17:01 [PATCH 1/4] percpu_ref: introduce PERCPU_REF_ALLOW_REINIT flag Roman Gushchin
2019-05-07 17:01 ` [PATCH 2/4] io_uring: initialize percpu refcounters using PERCU_REF_ALLOW_REINIT Roman Gushchin
@ 2019-05-07 17:01 ` Roman Gushchin
2019-05-07 17:01 ` [PATCH 4/4] percpu_ref: release percpu memory early without PERCPU_REF_ALLOW_REINIT Roman Gushchin
2019-05-09 16:53 ` [PATCH 1/4] percpu_ref: introduce PERCPU_REF_ALLOW_REINIT flag Tejun Heo
3 siblings, 0 replies; 6+ messages in thread
From: Roman Gushchin @ 2019-05-07 17:01 UTC (permalink / raw)
To: Tejun Heo
Cc: Jens Axboe, Song Liu, Dennis Zhou, linux-kernel, kernel-team,
Roman Gushchin
Percpu reference counters should now be initialized with the
PERCPU_REF_ALLOW_REINIT in order to allow switching them to the
percpu mode from the atomic mode.
To make percpu_ref_switch_to_percpu() call in set_in_sync()
succeed,let's initialize percpu refcounters with the
PERCU_REF_ALLOW_REINIT flag.
Signed-off-by: Roman Gushchin <guro@fb.com>
---
drivers/md/md.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/drivers/md/md.c b/drivers/md/md.c
index 45ffa23fa85d..6b60211d0b3e 100644
--- a/drivers/md/md.c
+++ b/drivers/md/md.c
@@ -5245,7 +5245,8 @@ int mddev_init_writes_pending(struct mddev *mddev)
{
if (mddev->writes_pending.percpu_count_ptr)
return 0;
- if (percpu_ref_init(&mddev->writes_pending, no_op, 0, GFP_KERNEL) < 0)
+ if (percpu_ref_init(&mddev->writes_pending, no_op,
+ PERCPU_REF_ALLOW_REINIT, GFP_KERNEL) < 0)
return -ENOMEM;
/* We want to start with the refcount at zero */
percpu_ref_put(&mddev->writes_pending);
--
2.20.1
^ permalink raw reply related [flat|nested] 6+ messages in thread
* [PATCH 4/4] percpu_ref: release percpu memory early without PERCPU_REF_ALLOW_REINIT
2019-05-07 17:01 [PATCH 1/4] percpu_ref: introduce PERCPU_REF_ALLOW_REINIT flag Roman Gushchin
2019-05-07 17:01 ` [PATCH 2/4] io_uring: initialize percpu refcounters using PERCU_REF_ALLOW_REINIT Roman Gushchin
2019-05-07 17:01 ` [PATCH 3/4] md: " Roman Gushchin
@ 2019-05-07 17:01 ` Roman Gushchin
2019-05-09 16:53 ` [PATCH 1/4] percpu_ref: introduce PERCPU_REF_ALLOW_REINIT flag Tejun Heo
3 siblings, 0 replies; 6+ messages in thread
From: Roman Gushchin @ 2019-05-07 17:01 UTC (permalink / raw)
To: Tejun Heo
Cc: Jens Axboe, Song Liu, Dennis Zhou, linux-kernel, kernel-team,
Roman Gushchin
Release percpu memory after finishing the switch to the atomic mode
if only PERCPU_REF_ALLOW_REINIT isn't set.
Signed-off-by: Roman Gushchin <guro@fb.com>
---
include/linux/percpu-refcount.h | 1 +
lib/percpu-refcount.c | 13 +++++++++++--
2 files changed, 12 insertions(+), 2 deletions(-)
diff --git a/include/linux/percpu-refcount.h b/include/linux/percpu-refcount.h
index 0f0240af8520..7aef0abc194a 100644
--- a/include/linux/percpu-refcount.h
+++ b/include/linux/percpu-refcount.h
@@ -102,6 +102,7 @@ struct percpu_ref {
percpu_ref_func_t *release;
percpu_ref_func_t *confirm_switch;
bool force_atomic:1;
+ bool allow_reinit:1;
struct rcu_head rcu;
};
diff --git a/lib/percpu-refcount.c b/lib/percpu-refcount.c
index da54318d3b55..47f0aeb136c4 100644
--- a/lib/percpu-refcount.c
+++ b/lib/percpu-refcount.c
@@ -69,11 +69,14 @@ int percpu_ref_init(struct percpu_ref *ref, percpu_ref_func_t *release,
return -ENOMEM;
ref->force_atomic = flags & PERCPU_REF_INIT_ATOMIC;
+ ref->allow_reinit = flags & PERCPU_REF_ALLOW_REINIT;
- if (flags & (PERCPU_REF_INIT_ATOMIC | PERCPU_REF_INIT_DEAD))
+ if (flags & (PERCPU_REF_INIT_ATOMIC | PERCPU_REF_INIT_DEAD)) {
ref->percpu_count_ptr |= __PERCPU_REF_ATOMIC;
- else
+ ref->allow_reinit = true;
+ } else {
start_count += PERCPU_COUNT_BIAS;
+ }
if (flags & PERCPU_REF_INIT_DEAD)
ref->percpu_count_ptr |= __PERCPU_REF_DEAD;
@@ -119,6 +122,9 @@ static void percpu_ref_call_confirm_rcu(struct rcu_head *rcu)
ref->confirm_switch = NULL;
wake_up_all(&percpu_ref_switch_waitq);
+ if (!ref->allow_reinit)
+ percpu_ref_exit(ref);
+
/* drop ref from percpu_ref_switch_to_atomic() */
percpu_ref_put(ref);
}
@@ -194,6 +200,9 @@ static void __percpu_ref_switch_to_percpu(struct percpu_ref *ref)
if (!(ref->percpu_count_ptr & __PERCPU_REF_ATOMIC))
return;
+ if (WARN_ON_ONCE(!ref->allow_reinit))
+ return;
+
atomic_long_add(PERCPU_COUNT_BIAS, &ref->count);
/*
--
2.20.1
^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: [PATCH 1/4] percpu_ref: introduce PERCPU_REF_ALLOW_REINIT flag
2019-05-07 17:01 [PATCH 1/4] percpu_ref: introduce PERCPU_REF_ALLOW_REINIT flag Roman Gushchin
` (2 preceding siblings ...)
2019-05-07 17:01 ` [PATCH 4/4] percpu_ref: release percpu memory early without PERCPU_REF_ALLOW_REINIT Roman Gushchin
@ 2019-05-09 16:53 ` Tejun Heo
2019-05-09 17:52 ` Dennis Zhou
3 siblings, 1 reply; 6+ messages in thread
From: Tejun Heo @ 2019-05-09 16:53 UTC (permalink / raw)
To: Roman Gushchin
Cc: Jens Axboe, Song Liu, Dennis Zhou, linux-kernel, kernel-team
On Tue, May 07, 2019 at 10:01:47AM -0700, Roman Gushchin wrote:
> In most cases percpu reference counters are not switched to the
> percpu mode after they reach the atomic mode. Some obvious exceptions
> are reference counters which are initialized into the atomic
> mode (using PERCPU_REF_INIT_ATOMIC and PERCPU_REF_INIT_DEAD flags),
> and there are few other exceptions.
>
> But in most cases there is no way back, and once the reference counter
> is switched to the atomic mode, there is no reason to wait for
> percpu_ref_exit() to release the percpu memory. Of course, the size
> of a single counter is not so big, but because it can pin the whole
> percpu block in memory, the memory footprint can be noticeable
> (e.g. on my 32 CPUs machine a percpu block is 8Mb large).
>
> To make releasing of the percpu memory as early as possible, let's
> introduce the PERCPU_REF_ALLOW_REINIT flag with the following semantics:
> it has to be set in order to switch a percpu reference counter to the
> percpu mode after the initialization. PERCPU_REF_INIT_ATOMIC and
> PERCPU_REF_INIT_DEAD flags will implicitly assume PERCPU_REF_ALLOW_REINIT.
>
> This patch doesn't introduce any functional change to avoid any
> regressions. It will be done later in the patchset after adjusting
> all call sites, which are reviving percpu counters.
>
> Signed-off-by: Roman Gushchin <guro@fb.com>
For all patches in the series:
Acked-by: Tejun Heo <tj@kenrel.org>
Thanks.
--
tejun
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH 1/4] percpu_ref: introduce PERCPU_REF_ALLOW_REINIT flag
2019-05-09 16:53 ` [PATCH 1/4] percpu_ref: introduce PERCPU_REF_ALLOW_REINIT flag Tejun Heo
@ 2019-05-09 17:52 ` Dennis Zhou
0 siblings, 0 replies; 6+ messages in thread
From: Dennis Zhou @ 2019-05-09 17:52 UTC (permalink / raw)
To: Tejun Heo
Cc: Roman Gushchin, Jens Axboe, Song Liu, Dennis Zhou, linux-kernel,
kernel-team
On Thu, May 09, 2019 at 09:53:43AM -0700, Tejun Heo wrote:
> On Tue, May 07, 2019 at 10:01:47AM -0700, Roman Gushchin wrote:
> > In most cases percpu reference counters are not switched to the
> > percpu mode after they reach the atomic mode. Some obvious exceptions
> > are reference counters which are initialized into the atomic
> > mode (using PERCPU_REF_INIT_ATOMIC and PERCPU_REF_INIT_DEAD flags),
> > and there are few other exceptions.
> >
> > But in most cases there is no way back, and once the reference counter
> > is switched to the atomic mode, there is no reason to wait for
> > percpu_ref_exit() to release the percpu memory. Of course, the size
> > of a single counter is not so big, but because it can pin the whole
> > percpu block in memory, the memory footprint can be noticeable
> > (e.g. on my 32 CPUs machine a percpu block is 8Mb large).
> >
> > To make releasing of the percpu memory as early as possible, let's
> > introduce the PERCPU_REF_ALLOW_REINIT flag with the following semantics:
> > it has to be set in order to switch a percpu reference counter to the
> > percpu mode after the initialization. PERCPU_REF_INIT_ATOMIC and
> > PERCPU_REF_INIT_DEAD flags will implicitly assume PERCPU_REF_ALLOW_REINIT.
> >
> > This patch doesn't introduce any functional change to avoid any
> > regressions. It will be done later in the patchset after adjusting
> > all call sites, which are reviving percpu counters.
> >
> > Signed-off-by: Roman Gushchin <guro@fb.com>
>
> For all patches in the series:
>
> Acked-by: Tejun Heo <tj@kenrel.org>
>
> Thanks.
>
> --
> tejun
Great, I've applied this to for-5.3.
Thanks,
Dennis
^ permalink raw reply [flat|nested] 6+ messages in thread