From: "Paul E. McKenney" <paulmck@linux.ibm.com>
To: Joel Fernandes <joel@joelfernandes.org>
Cc: Nikolay Borisov <nborisov@suse.com>,
linux-kernel@vger.kernel.org, Jonathan Corbet <corbet@lwn.net>,
Josh Triplett <josh@joshtriplett.org>,
Lai Jiangshan <jiangshanlai@gmail.com>,
linux-doc@vger.kernel.org,
Mathieu Desnoyers <mathieu.desnoyers@efficios.com>,
Steven Rostedt <rostedt@goodmis.org>
Subject: Re: [PATCH RFC] doc: rcu: remove obsolete (non-)requirement about disabling preemption
Date: Wed, 17 Oct 2018 09:11:00 -0700 [thread overview]
Message-ID: <20181017161100.GP2674@linux.ibm.com> (raw)
In-Reply-To: <20181016204122.GA8176@joelaf.mtv.corp.google.com>
On Tue, Oct 16, 2018 at 01:41:22PM -0700, Joel Fernandes wrote:
> On Tue, Oct 16, 2018 at 04:26:11AM -0700, Paul E. McKenney wrote:
> > On Mon, Oct 15, 2018 at 02:08:56PM -0700, Paul E. McKenney wrote:
> > > On Mon, Oct 15, 2018 at 01:15:56PM -0700, Joel Fernandes wrote:
> > > > On Mon, Oct 15, 2018 at 12:54:26PM -0700, Paul E. McKenney wrote:
> > > > [...]
> > > > > > > In any case, please don't spin for milliseconds with preemption disabled.
> > > > > > > The real-time guys are unlikely to be happy with you if you do this!
> > > > > >
> > > > > > Well just to clarify, I was just running Oleg's test which did this. This
> > > > > > test was mentioned in the original documentation that I deleted. Ofcourse I
> > > > > > would not dare do such a thing in production code :-D. I guess to Oleg's
> > > > > > defense, he did it to very that synchronize_rcu() was not blocked on
> > > > > > preempt-disable sections which was a different test.
> > > > >
> > > > > Understood! Just pointing out that RCU's tolerating a given action does
> > > > > not necessarily mean that it is a good idea to take that action. ;-)
> > > >
> > > > Makes sense :-) thanks.
> > >
> > > Don't worry, that won't happen again. ;-)
> > >
> > > > > > > > > + pr_crit("SPIN done!\n");
> > > > > > > > > + preempt_enable();
> > > > > > > > > + break;
> > > > > > > > > + case 777:
> > > > > > > > > + pr_crit("SYNC start\n");
> > > > > > > > > + synchronize_rcu();
> > > > > > > > > + pr_crit("SYNC done!\n");
> > > > > > > >
> > > > > > > > But you are using the console printing infrastructure which is rather
> > > > > > > > heavyweight. Try replacing pr_* calls with trace_printk so that you
> > > > > > > > write to the lock-free ring buffer, this will reduce the noise from the
> > > > > > > > heavy console printing infrastructure.
> > > > > > >
> > > > > > > And this might be a problem as well.
> > > > > >
> > > > > > This was not the issue (or atleast not fully the issue) since I saw the same
> > > > > > thing with trace_printk. It was exactly what you said - which is the
> > > > > > excessively long preempt disabled times.
> > > > >
> > > > > One approach would be to apply this patch against (say) v4.18, which
> > > > > does not have consolidated grace periods. You might then be able to
> > > > > tell if the pr_crit() calls make any difference.
> > > >
> > > > I could do that, yeah. But since the original problem went away due to
> > > > disabling preempts for a short while, I will move on and continue to focus on
> > > > updating other parts of the documenation. Just to mention I
> > > > brought this up because I thought its better to do that than not to, just
> > > > incase there is any lurking issue with the consolidation. Sorry if that ended
> > > > up with me being noisy.
> > >
> > > Not a problem, no need to apologize!
> >
> > Besides, digging through the code did point out a reasonable optimization.
> > In the common case, this would buy 100s of microseconds rather than
> > milliseconds, but it seems simple enough to be worthwhile. Thoughts?
>
> Cool, thanks. One comment below:
>
> > ------------------------------------------------------------------------
> >
> > commit 07921e8720907f58f82b142f2027fc56d5abdbfd
> > Author: Paul E. McKenney <paulmck@linux.ibm.com>
> > Date: Tue Oct 16 04:12:58 2018 -0700
> >
> > rcu: Speed up expedited GPs when interrupting RCU reader
> >
> > In PREEMPT kernels, an expedited grace period might send an IPI to a
> > CPU that is executing an RCU read-side critical section. In that case,
> > it would be nice if the rcu_read_unlock() directly interacted with the
> > RCU core code to immediately report the quiescent state. And this does
> > happen in the case where the reader has been preempted. But it would
> > also be a nice performance optimization if immediate reporting also
> > happened in the preemption-free case.
> >
> > This commit therefore adds an ->exp_hint field to the task_struct structure's
> > ->rcu_read_unlock_special field. The IPI handler sets this hint when
> > it has interrupted an RCU read-side critical section, and this causes
> > the outermost rcu_read_unlock() call to invoke rcu_read_unlock_special(),
> > which, if preemption is enabled, reports the quiescent state immediately.
> > If preemption is disabled, then the report is required to be deferred
> > until preemption (or bottom halves or interrupts or whatever) is re-enabled.
> >
> > Because this is a hint, it does nothing for more complicated cases. For
> > example, if the IPI interrupts an RCU reader, but interrupts are disabled
> > across the rcu_read_unlock(), but another rcu_read_lock() is executed
> > before interrupts are re-enabled, the hint will already have been cleared.
> > If you do crazy things like this, reporting will be deferred until some
> > later RCU_SOFTIRQ handler, context switch, cond_resched(), or similar.
> >
> > Reported-by: Joel Fernandes <joel@joelfernandes.org>
> > Signed-off-by: Paul E. McKenney <paulmck@linux.ibm.com>
> >
> > diff --git a/include/linux/sched.h b/include/linux/sched.h
> > index 004ca21f7e80..64ce751b5fe9 100644
> > --- a/include/linux/sched.h
> > +++ b/include/linux/sched.h
> > @@ -571,8 +571,10 @@ union rcu_special {
> > struct {
> > u8 blocked;
> > u8 need_qs;
> > + u8 exp_hint; /* Hint for performance. */
> > + u8 pad; /* No garbage from compiler! */
> > } b; /* Bits. */
> > - u16 s; /* Set of bits. */
> > + u32 s; /* Set of bits. */
> > };
> >
> > enum perf_event_task_context {
> > diff --git a/kernel/rcu/tree_exp.h b/kernel/rcu/tree_exp.h
> > index e669ccf3751b..928fe5893a57 100644
> > --- a/kernel/rcu/tree_exp.h
> > +++ b/kernel/rcu/tree_exp.h
> > @@ -692,8 +692,10 @@ static void sync_rcu_exp_handler(void *unused)
> > */
> > if (t->rcu_read_lock_nesting > 0) {
> > raw_spin_lock_irqsave_rcu_node(rnp, flags);
> > - if (rnp->expmask & rdp->grpmask)
> > + if (rnp->expmask & rdp->grpmask) {
> > rdp->deferred_qs = true;
> > + WRITE_ONCE(t->rcu_read_unlock_special.b.exp_hint, true);
> > + }
> > raw_spin_unlock_irqrestore_rcu_node(rnp, flags);
> > }
> >
> > diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h
> > index 8b48bb7c224c..d6286eb6e77e 100644
> > --- a/kernel/rcu/tree_plugin.h
> > +++ b/kernel/rcu/tree_plugin.h
> > @@ -643,8 +643,9 @@ static void rcu_read_unlock_special(struct task_struct *t)
> > local_irq_save(flags);
> > irqs_were_disabled = irqs_disabled_flags(flags);
> > if ((preempt_bh_were_disabled || irqs_were_disabled) &&
> > - t->rcu_read_unlock_special.b.blocked) {
> > + t->rcu_read_unlock_special.s) {
> > /* Need to defer quiescent state until everything is enabled. */
> > + WRITE_ONCE(t->rcu_read_unlock_special.b.exp_hint, false);
> > raise_softirq_irqoff(RCU_SOFTIRQ);
>
> Still going through this patch, but it seems to me like the fact that
> rcu_read_unlock_special is called means someone has requested for a grace
> period. Then in that case, does it not make sense to raise the softirq
> for processing anyway?
Not necessarily. Another reason that rcu_read_unlock_special() might
be called is if the RCU read-side critical section had been preempted,
in which case there might not even be a grace period in progress.
In addition, if interrupts, bottom halves, and preemption are all enabled,
the code in rcu_preempt_deferred_qs_irqrestore() doesn't need to bother
raising softirq, as it can instead just immediately report the quiescent
state.
Thanx, Paul
next prev parent reply other threads:[~2018-10-17 16:11 UTC|newest]
Thread overview: 32+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-10-14 21:29 [PATCH RFC] doc: rcu: remove obsolete (non-)requirement about disabling preemption Joel Fernandes (Google)
2018-10-14 23:17 ` Paul E. McKenney
2018-10-15 2:08 ` Joel Fernandes
2018-10-15 2:13 ` Joel Fernandes
2018-10-15 2:33 ` Paul E. McKenney
2018-10-15 2:47 ` Joel Fernandes
2018-10-15 2:50 ` Joel Fernandes
2018-10-15 6:05 ` Nikolay Borisov
2018-10-15 11:21 ` Paul E. McKenney
2018-10-15 19:39 ` Joel Fernandes
2018-10-15 19:54 ` Paul E. McKenney
2018-10-15 20:15 ` Joel Fernandes
2018-10-15 21:08 ` Paul E. McKenney
2018-10-16 11:26 ` Paul E. McKenney
2018-10-16 20:41 ` Joel Fernandes
2018-10-17 16:11 ` Paul E. McKenney [this message]
2018-10-17 18:15 ` Joel Fernandes
2018-10-17 20:33 ` Paul E. McKenney
2018-10-18 2:07 ` Joel Fernandes
2018-10-18 14:46 ` Paul E. McKenney
2018-10-19 0:03 ` Joel Fernandes
2018-10-19 0:19 ` Paul E. McKenney
2018-10-19 1:12 ` Steven Rostedt
2018-10-19 1:27 ` Joel Fernandes
2018-10-19 1:26 ` Joel Fernandes
2018-10-19 1:50 ` Steven Rostedt
2018-10-19 2:25 ` Joel Fernandes
2018-10-19 2:52 ` Steven Rostedt
2018-10-19 3:58 ` Joel Fernandes
2018-10-19 12:07 ` Paul E. McKenney
2018-10-19 17:24 ` Joel Fernandes
2018-10-19 18:11 ` Paul E. McKenney
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20181017161100.GP2674@linux.ibm.com \
--to=paulmck@linux.ibm.com \
--cc=corbet@lwn.net \
--cc=jiangshanlai@gmail.com \
--cc=joel@joelfernandes.org \
--cc=josh@joshtriplett.org \
--cc=linux-doc@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=mathieu.desnoyers@efficios.com \
--cc=nborisov@suse.com \
--cc=rostedt@goodmis.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).