[tip/core/rcu,2/3] rcu: Provide polling interfaces for Tiny RCU grace periods
diff mbox series

Message ID 20210304002632.23870-2-paulmck@kernel.org
State New, archived
Headers show
Series
  • Polling RCU grace-period interfaces for v5.13
Related show

Commit Message

Paul E. McKenney March 4, 2021, 12:26 a.m. UTC
From: "Paul E. McKenney" <paulmck@kernel.org>

There is a need for a non-blocking polling interface for RCU grace
periods, so this commit supplies start_poll_synchronize_rcu() and
poll_state_synchronize_rcu() for this purpose.  Note that the existing
get_state_synchronize_rcu() may be used if future grace periods are
inevitable (perhaps due to a later call_rcu() invocation).  The new
start_poll_synchronize_rcu() is to be used if future grace periods
might not otherwise happen.  Finally, poll_state_synchronize_rcu()
provides a lockless check for a grace period having elapsed since
the corresponding call to either of the get_state_synchronize_rcu()
or start_poll_synchronize_rcu().

As with get_state_synchronize_rcu(), the return value from either
get_state_synchronize_rcu() or start_poll_synchronize_rcu() is passed in
to a later call to either poll_state_synchronize_rcu() or the existing
(might_sleep) cond_synchronize_rcu().

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
---
 include/linux/rcutiny.h | 11 ++++++-----
 kernel/rcu/tiny.c       | 40 ++++++++++++++++++++++++++++++++++++++++
 2 files changed, 46 insertions(+), 5 deletions(-)

Comments

Frederic Weisbecker March 21, 2021, 10:28 p.m. UTC | #1
On Wed, Mar 03, 2021 at 04:26:31PM -0800, paulmck@kernel.org wrote:
> From: "Paul E. McKenney" <paulmck@kernel.org>
> 
> There is a need for a non-blocking polling interface for RCU grace
> periods, so this commit supplies start_poll_synchronize_rcu() and
> poll_state_synchronize_rcu() for this purpose.  Note that the existing
> get_state_synchronize_rcu() may be used if future grace periods are
> inevitable (perhaps due to a later call_rcu() invocation).  The new
> start_poll_synchronize_rcu() is to be used if future grace periods
> might not otherwise happen.  Finally, poll_state_synchronize_rcu()
> provides a lockless check for a grace period having elapsed since
> the corresponding call to either of the get_state_synchronize_rcu()
> or start_poll_synchronize_rcu().
> 
> As with get_state_synchronize_rcu(), the return value from either
> get_state_synchronize_rcu() or start_poll_synchronize_rcu() is passed in
> to a later call to either poll_state_synchronize_rcu() or the existing
> (might_sleep) cond_synchronize_rcu().
> 
> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
> ---
>  include/linux/rcutiny.h | 11 ++++++-----
>  kernel/rcu/tiny.c       | 40 ++++++++++++++++++++++++++++++++++++++++
>  2 files changed, 46 insertions(+), 5 deletions(-)
> 
> diff --git a/include/linux/rcutiny.h b/include/linux/rcutiny.h
> index 2a97334..69108cf4 100644
> --- a/include/linux/rcutiny.h
> +++ b/include/linux/rcutiny.h
> @@ -17,14 +17,15 @@
>  /* Never flag non-existent other CPUs! */
>  static inline bool rcu_eqs_special_set(int cpu) { return false; }
>  
> -static inline unsigned long get_state_synchronize_rcu(void)
> -{
> -	return 0;
> -}
> +unsigned long get_state_synchronize_rcu(void);
> +unsigned long start_poll_synchronize_rcu(void);
> +bool poll_state_synchronize_rcu(unsigned long oldstate);
>  
>  static inline void cond_synchronize_rcu(unsigned long oldstate)
>  {
> -	might_sleep();
> +	if (poll_state_synchronize_rcu(oldstate))
> +		return;
> +	synchronize_rcu();

Perhaps cond_synchronize_rcu() could stay as it was. If it might
call synchronize_rcu() then it inherits its constraint to be
called from a quiescent state.

Thanks.
Paul E. McKenney March 22, 2021, 3:47 p.m. UTC | #2
On Sun, Mar 21, 2021 at 11:28:55PM +0100, Frederic Weisbecker wrote:
> On Wed, Mar 03, 2021 at 04:26:31PM -0800, paulmck@kernel.org wrote:
> > From: "Paul E. McKenney" <paulmck@kernel.org>
> > 
> > There is a need for a non-blocking polling interface for RCU grace
> > periods, so this commit supplies start_poll_synchronize_rcu() and
> > poll_state_synchronize_rcu() for this purpose.  Note that the existing
> > get_state_synchronize_rcu() may be used if future grace periods are
> > inevitable (perhaps due to a later call_rcu() invocation).  The new
> > start_poll_synchronize_rcu() is to be used if future grace periods
> > might not otherwise happen.  Finally, poll_state_synchronize_rcu()
> > provides a lockless check for a grace period having elapsed since
> > the corresponding call to either of the get_state_synchronize_rcu()
> > or start_poll_synchronize_rcu().
> > 
> > As with get_state_synchronize_rcu(), the return value from either
> > get_state_synchronize_rcu() or start_poll_synchronize_rcu() is passed in
> > to a later call to either poll_state_synchronize_rcu() or the existing
> > (might_sleep) cond_synchronize_rcu().
> > 
> > Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
> > ---
> >  include/linux/rcutiny.h | 11 ++++++-----
> >  kernel/rcu/tiny.c       | 40 ++++++++++++++++++++++++++++++++++++++++
> >  2 files changed, 46 insertions(+), 5 deletions(-)
> > 
> > diff --git a/include/linux/rcutiny.h b/include/linux/rcutiny.h
> > index 2a97334..69108cf4 100644
> > --- a/include/linux/rcutiny.h
> > +++ b/include/linux/rcutiny.h
> > @@ -17,14 +17,15 @@
> >  /* Never flag non-existent other CPUs! */
> >  static inline bool rcu_eqs_special_set(int cpu) { return false; }
> >  
> > -static inline unsigned long get_state_synchronize_rcu(void)
> > -{
> > -	return 0;
> > -}
> > +unsigned long get_state_synchronize_rcu(void);
> > +unsigned long start_poll_synchronize_rcu(void);
> > +bool poll_state_synchronize_rcu(unsigned long oldstate);
> >  
> >  static inline void cond_synchronize_rcu(unsigned long oldstate)
> >  {
> > -	might_sleep();
> > +	if (poll_state_synchronize_rcu(oldstate))
> > +		return;
> > +	synchronize_rcu();
> 
> Perhaps cond_synchronize_rcu() could stay as it was. If it might
> call synchronize_rcu() then it inherits its constraint to be
> called from a quiescent state.

As in leave the might_sleep()?  How about something like this?

static inline void cond_synchronize_rcu(unsigned long oldstate)
{
	if (!poll_state_synchronize_rcu(oldstate))
		synchronize_rcu();
	else
		might_sleep();
}

One advantage of this is that the Tiny and Tree implementations
become identical and can then be consolidated.

Or did I miss your point?

						Thanx, Paul
Frederic Weisbecker March 22, 2021, 7 p.m. UTC | #3
On Mon, Mar 22, 2021 at 08:47:44AM -0700, Paul E. McKenney wrote:
> On Sun, Mar 21, 2021 at 11:28:55PM +0100, Frederic Weisbecker wrote:
> > On Wed, Mar 03, 2021 at 04:26:31PM -0800, paulmck@kernel.org wrote:
> > > From: "Paul E. McKenney" <paulmck@kernel.org>
> > > 
> > > There is a need for a non-blocking polling interface for RCU grace
> > > periods, so this commit supplies start_poll_synchronize_rcu() and
> > > poll_state_synchronize_rcu() for this purpose.  Note that the existing
> > > get_state_synchronize_rcu() may be used if future grace periods are
> > > inevitable (perhaps due to a later call_rcu() invocation).  The new
> > > start_poll_synchronize_rcu() is to be used if future grace periods
> > > might not otherwise happen.  Finally, poll_state_synchronize_rcu()
> > > provides a lockless check for a grace period having elapsed since
> > > the corresponding call to either of the get_state_synchronize_rcu()
> > > or start_poll_synchronize_rcu().
> > > 
> > > As with get_state_synchronize_rcu(), the return value from either
> > > get_state_synchronize_rcu() or start_poll_synchronize_rcu() is passed in
> > > to a later call to either poll_state_synchronize_rcu() or the existing
> > > (might_sleep) cond_synchronize_rcu().
> > > 
> > > Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
> > > ---
> > >  include/linux/rcutiny.h | 11 ++++++-----
> > >  kernel/rcu/tiny.c       | 40 ++++++++++++++++++++++++++++++++++++++++
> > >  2 files changed, 46 insertions(+), 5 deletions(-)
> > > 
> > > diff --git a/include/linux/rcutiny.h b/include/linux/rcutiny.h
> > > index 2a97334..69108cf4 100644
> > > --- a/include/linux/rcutiny.h
> > > +++ b/include/linux/rcutiny.h
> > > @@ -17,14 +17,15 @@
> > >  /* Never flag non-existent other CPUs! */
> > >  static inline bool rcu_eqs_special_set(int cpu) { return false; }
> > >  
> > > -static inline unsigned long get_state_synchronize_rcu(void)
> > > -{
> > > -	return 0;
> > > -}
> > > +unsigned long get_state_synchronize_rcu(void);
> > > +unsigned long start_poll_synchronize_rcu(void);
> > > +bool poll_state_synchronize_rcu(unsigned long oldstate);
> > >  
> > >  static inline void cond_synchronize_rcu(unsigned long oldstate)
> > >  {
> > > -	might_sleep();
> > > +	if (poll_state_synchronize_rcu(oldstate))
> > > +		return;
> > > +	synchronize_rcu();
> > 
> > Perhaps cond_synchronize_rcu() could stay as it was. If it might
> > call synchronize_rcu() then it inherits its constraint to be
> > called from a quiescent state.
> 
> As in leave the might_sleep()?  How about something like this?
> 
> static inline void cond_synchronize_rcu(unsigned long oldstate)
> {
> 	if (!poll_state_synchronize_rcu(oldstate))
> 		synchronize_rcu();
> 	else
> 		might_sleep();
> }
> 
> One advantage of this is that the Tiny and Tree implementations
> become identical and can then be consolidated.
> 
> Or did I miss your point?

But poll_state_synchronize_rcu() checks that the gp_num has changed,
which is not needed for cond_synchronize_rcu() since this it is
only allowed to be called from a QS.

> 
> 						Thanx, Paul
Paul E. McKenney March 22, 2021, 7:45 p.m. UTC | #4
On Mon, Mar 22, 2021 at 08:00:35PM +0100, Frederic Weisbecker wrote:
> On Mon, Mar 22, 2021 at 08:47:44AM -0700, Paul E. McKenney wrote:
> > On Sun, Mar 21, 2021 at 11:28:55PM +0100, Frederic Weisbecker wrote:
> > > On Wed, Mar 03, 2021 at 04:26:31PM -0800, paulmck@kernel.org wrote:
> > > > From: "Paul E. McKenney" <paulmck@kernel.org>
> > > > 
> > > > There is a need for a non-blocking polling interface for RCU grace
> > > > periods, so this commit supplies start_poll_synchronize_rcu() and
> > > > poll_state_synchronize_rcu() for this purpose.  Note that the existing
> > > > get_state_synchronize_rcu() may be used if future grace periods are
> > > > inevitable (perhaps due to a later call_rcu() invocation).  The new
> > > > start_poll_synchronize_rcu() is to be used if future grace periods
> > > > might not otherwise happen.  Finally, poll_state_synchronize_rcu()
> > > > provides a lockless check for a grace period having elapsed since
> > > > the corresponding call to either of the get_state_synchronize_rcu()
> > > > or start_poll_synchronize_rcu().
> > > > 
> > > > As with get_state_synchronize_rcu(), the return value from either
> > > > get_state_synchronize_rcu() or start_poll_synchronize_rcu() is passed in
> > > > to a later call to either poll_state_synchronize_rcu() or the existing
> > > > (might_sleep) cond_synchronize_rcu().
> > > > 
> > > > Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
> > > > ---
> > > >  include/linux/rcutiny.h | 11 ++++++-----
> > > >  kernel/rcu/tiny.c       | 40 ++++++++++++++++++++++++++++++++++++++++
> > > >  2 files changed, 46 insertions(+), 5 deletions(-)
> > > > 
> > > > diff --git a/include/linux/rcutiny.h b/include/linux/rcutiny.h
> > > > index 2a97334..69108cf4 100644
> > > > --- a/include/linux/rcutiny.h
> > > > +++ b/include/linux/rcutiny.h
> > > > @@ -17,14 +17,15 @@
> > > >  /* Never flag non-existent other CPUs! */
> > > >  static inline bool rcu_eqs_special_set(int cpu) { return false; }
> > > >  
> > > > -static inline unsigned long get_state_synchronize_rcu(void)
> > > > -{
> > > > -	return 0;
> > > > -}
> > > > +unsigned long get_state_synchronize_rcu(void);
> > > > +unsigned long start_poll_synchronize_rcu(void);
> > > > +bool poll_state_synchronize_rcu(unsigned long oldstate);
> > > >  
> > > >  static inline void cond_synchronize_rcu(unsigned long oldstate)
> > > >  {
> > > > -	might_sleep();
> > > > +	if (poll_state_synchronize_rcu(oldstate))
> > > > +		return;
> > > > +	synchronize_rcu();
> > > 
> > > Perhaps cond_synchronize_rcu() could stay as it was. If it might
> > > call synchronize_rcu() then it inherits its constraint to be
> > > called from a quiescent state.
> > 
> > As in leave the might_sleep()?  How about something like this?
> > 
> > static inline void cond_synchronize_rcu(unsigned long oldstate)
> > {
> > 	if (!poll_state_synchronize_rcu(oldstate))
> > 		synchronize_rcu();
> > 	else
> > 		might_sleep();
> > }
> > 
> > One advantage of this is that the Tiny and Tree implementations
> > become identical and can then be consolidated.
> > 
> > Or did I miss your point?
> 
> But poll_state_synchronize_rcu() checks that the gp_num has changed,
> which is not needed for cond_synchronize_rcu() since this it is
> only allowed to be called from a QS.

Good catch, and thank you!  Back to a single might_sleep() it is!

						Thanx, Paul
Frederic Weisbecker March 23, 2021, 2:02 p.m. UTC | #5
On Mon, Mar 22, 2021 at 12:45:22PM -0700, Paul E. McKenney wrote:
> On Mon, Mar 22, 2021 at 08:00:35PM +0100, Frederic Weisbecker wrote:
> > But poll_state_synchronize_rcu() checks that the gp_num has changed,
> > which is not needed for cond_synchronize_rcu() since this it is
> > only allowed to be called from a QS.
> 
> Good catch, and thank you!  Back to a single might_sleep() it is!

And then: Reviewed-by: Frederic Weisbecker <frederic@kernel.org>

Thanks!
Paul E. McKenney March 23, 2021, 4:45 p.m. UTC | #6
On Tue, Mar 23, 2021 at 03:02:07PM +0100, Frederic Weisbecker wrote:
> On Mon, Mar 22, 2021 at 12:45:22PM -0700, Paul E. McKenney wrote:
> > On Mon, Mar 22, 2021 at 08:00:35PM +0100, Frederic Weisbecker wrote:
> > > But poll_state_synchronize_rcu() checks that the gp_num has changed,
> > > which is not needed for cond_synchronize_rcu() since this it is
> > > only allowed to be called from a QS.
> > 
> > Good catch, and thank you!  Back to a single might_sleep() it is!
> 
> And then: Reviewed-by: Frederic Weisbecker <frederic@kernel.org>

Thank you!  I will apply this on my next rebase.

							Thanx, Paul

Patch
diff mbox series

diff --git a/include/linux/rcutiny.h b/include/linux/rcutiny.h
index 2a97334..69108cf4 100644
--- a/include/linux/rcutiny.h
+++ b/include/linux/rcutiny.h
@@ -17,14 +17,15 @@ 
 /* Never flag non-existent other CPUs! */
 static inline bool rcu_eqs_special_set(int cpu) { return false; }
 
-static inline unsigned long get_state_synchronize_rcu(void)
-{
-	return 0;
-}
+unsigned long get_state_synchronize_rcu(void);
+unsigned long start_poll_synchronize_rcu(void);
+bool poll_state_synchronize_rcu(unsigned long oldstate);
 
 static inline void cond_synchronize_rcu(unsigned long oldstate)
 {
-	might_sleep();
+	if (poll_state_synchronize_rcu(oldstate))
+		return;
+	synchronize_rcu();
 }
 
 extern void rcu_barrier(void);
diff --git a/kernel/rcu/tiny.c b/kernel/rcu/tiny.c
index aa897c3..c8a029f 100644
--- a/kernel/rcu/tiny.c
+++ b/kernel/rcu/tiny.c
@@ -32,12 +32,14 @@  struct rcu_ctrlblk {
 	struct rcu_head *rcucblist;	/* List of pending callbacks (CBs). */
 	struct rcu_head **donetail;	/* ->next pointer of last "done" CB. */
 	struct rcu_head **curtail;	/* ->next pointer of last CB. */
+	unsigned long gp_seq;		/* Grace-period counter. */
 };
 
 /* Definition for rcupdate control block. */
 static struct rcu_ctrlblk rcu_ctrlblk = {
 	.donetail	= &rcu_ctrlblk.rcucblist,
 	.curtail	= &rcu_ctrlblk.rcucblist,
+	.gp_seq		= 0 - 300UL,
 };
 
 void rcu_barrier(void)
@@ -56,6 +58,7 @@  void rcu_qs(void)
 		rcu_ctrlblk.donetail = rcu_ctrlblk.curtail;
 		raise_softirq_irqoff(RCU_SOFTIRQ);
 	}
+	WRITE_ONCE(rcu_ctrlblk.gp_seq, rcu_ctrlblk.gp_seq + 1);
 	local_irq_restore(flags);
 }
 
@@ -177,6 +180,43 @@  void call_rcu(struct rcu_head *head, rcu_callback_t func)
 }
 EXPORT_SYMBOL_GPL(call_rcu);
 
+/*
+ * Return a grace-period-counter "cookie".  For more information,
+ * see the Tree RCU header comment.
+ */
+unsigned long get_state_synchronize_rcu(void)
+{
+	return READ_ONCE(rcu_ctrlblk.gp_seq);
+}
+EXPORT_SYMBOL_GPL(get_state_synchronize_rcu);
+
+/*
+ * Return a grace-period-counter "cookie" and ensure that a future grace
+ * period completes.  For more information, see the Tree RCU header comment.
+ */
+unsigned long start_poll_synchronize_rcu(void)
+{
+	unsigned long gp_seq = get_state_synchronize_rcu();
+
+	if (unlikely(is_idle_task(current))) {
+		/* force scheduling for rcu_qs() */
+		resched_cpu(0);
+	}
+	return gp_seq;
+}
+EXPORT_SYMBOL_GPL(start_poll_synchronize_rcu);
+
+/*
+ * Return true if the grace period corresponding to oldstate has completed
+ * and false otherwise.  For more information, see the Tree RCU header
+ * comment.
+ */
+bool poll_state_synchronize_rcu(unsigned long oldstate)
+{
+	return READ_ONCE(rcu_ctrlblk.gp_seq) != oldstate;
+}
+EXPORT_SYMBOL_GPL(poll_state_synchronize_rcu);
+
 void __init rcu_init(void)
 {
 	open_softirq(RCU_SOFTIRQ, rcu_process_callbacks);