From: Byungchul Park <byungchul.park@lge.com>
To: "Paul E. McKenney" <paulmck@linux.ibm.com>
Cc: Joel Fernandes <joel@joelfernandes.org>,
Byungchul Park <max.byungchul.park@gmail.com>,
rcu <rcu@vger.kernel.org>, LKML <linux-kernel@vger.kernel.org>,
kernel-team@lge.com
Subject: Re: [PATCH] rcu: Make jiffies_till_sched_qs writable
Date: Wed, 24 Jul 2019 16:58:41 +0900 [thread overview]
Message-ID: <20190724075841.GA14712@X58A-UD3R> (raw)
In-Reply-To: <20190723165403.GA7239@linux.ibm.com>
On Tue, Jul 23, 2019 at 09:54:03AM -0700, Paul E. McKenney wrote:
> On Tue, Jul 23, 2019 at 06:47:17AM -0700, Paul E. McKenney wrote:
> > On Tue, Jul 23, 2019 at 08:05:21PM +0900, Byungchul Park wrote:
> > > On Fri, Jul 19, 2019 at 04:33:56PM -0400, Joel Fernandes wrote:
> > > > On Fri, Jul 19, 2019 at 3:57 PM Paul E. McKenney <paulmck@linux.ibm.com> wrote:
> > > > >
> > > > > On Fri, Jul 19, 2019 at 06:57:58PM +0900, Byungchul Park wrote:
> > > > > > On Fri, Jul 19, 2019 at 4:43 PM Paul E. McKenney <paulmck@linux.ibm.com> wrote:
> > > > > > >
> > > > > > > On Thu, Jul 18, 2019 at 08:52:52PM -0400, Joel Fernandes wrote:
> > > > > > > > On Thu, Jul 18, 2019 at 8:40 PM Byungchul Park <byungchul.park@lge.com> wrote:
> > > > > > > > [snip]
> > > > > > > > > > - There is a bug in the CPU stopper machinery itself preventing it
> > > > > > > > > > from scheduling the stopper on Y. Even though Y is not holding up the
> > > > > > > > > > grace period.
> > > > > > > > >
> > > > > > > > > Or any thread on Y is busy with preemption/irq disabled preventing the
> > > > > > > > > stopper from being scheduled on Y.
> > > > > > > > >
> > > > > > > > > Or something is stuck in ttwu() to wake up the stopper on Y due to any
> > > > > > > > > scheduler locks such as pi_lock or rq->lock or something.
> > > > > > > > >
> > > > > > > > > I think what you mentioned can happen easily.
> > > > > > > > >
> > > > > > > > > Basically we would need information about preemption/irq disabled
> > > > > > > > > sections on Y and scheduler's current activity on every cpu at that time.
> > > > > > > >
> > > > > > > > I think all that's needed is an NMI backtrace on all CPUs. An ARM we
> > > > > > > > don't have NMI solutions and only IPI or interrupt based backtrace
> > > > > > > > works which should at least catch and the preempt disable and softirq
> > > > > > > > disable cases.
> > > > > > >
> > > > > > > True, though people with systems having hundreds of CPUs might not
> > > > > > > thank you for forcing an NMI backtrace on each of them. Is it possible
> > > > > > > to NMI only the ones that are holding up the CPU stopper?
> > > > > >
> > > > > > What a good idea! I think it's possible!
> > > > > >
> > > > > > But we need to think about the case NMI doesn't work when the
> > > > > > holding-up was caused by IRQ disabled.
> > > > > >
> > > > > > Though it's just around the corner of weekend, I will keep thinking
> > > > > > on it during weekend!
> > > > >
> > > > > Very good!
> > > >
> > > > Me too will think more about it ;-) Agreed with point about 100s of
> > > > CPUs usecase,
> > > >
> > > > Thanks, have a great weekend,
> > >
> > > BTW, if there's any long code section with irq/preemption disabled, then
> > > the problem would be not only about RCU stall. And we can also use
> > > latency tracer or something to detect the bad situation.
> > >
> > > So in this case, sending ipi/nmi to the CPUs where the stoppers cannot
> > > to be scheduled does not give us additional meaningful information.
> > >
> > > I think Paul started to think about this to solve some real problem. I
> > > seriously love to help RCU and it's my pleasure to dig deep into kind of
> > > RCU stuff, but I've yet to define exactly what problem is. Sorry.
> > >
> > > Could you share the real issue? I think you don't have to reproduce it.
> > > Just sharing the issue that you got inspired from is enough. Then I
> > > might be able to develop 'how' with Joel! :-) It's our pleasure!
> >
> > It is unfortunately quite intermittent. I was hoping to find a way
> > to make it happen more often. Part of the underlying problem appears
> > to be lock contention, in that reducing contention made it even more
> > intermittent. Which is good in general, but not for exercising the
> > CPU-stopper issue.
> >
> > But perhaps your hardware will make this happen more readily than does
> > mine. The repeat-by is simple, namely run TREE04 on branch "dev" on an
> > eight-CPU system. It appear that the number of CPUs used by the test
> > should match the number available on the system that you are running on,
> > though perhaps affinity could allow mismatches.
> >
> > So why not try it and see what happens?
>
> And another potential issue causing this is a CONFIG_NO_HZ_FULL=y
I see. This provides more insight into the problem.
Thanks,
Byungchul
> kernel running in kernel mode (rcutorture on the one hand and callback
> invocation on the other) for extended periods of time with the scheduling
> clock disabled. Just started the tests for this. They will be running
> for quite some time, which this week is a good thing. ;-)
>
> Thanx, Paul
next prev parent reply other threads:[~2019-07-24 7:59 UTC|newest]
Thread overview: 53+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-07-08 6:00 [PATCH] rcu: Make jiffies_till_sched_qs writable Byungchul Park
2019-07-08 12:50 ` Paul E. McKenney
2019-07-08 13:03 ` Joel Fernandes
2019-07-08 13:19 ` Paul E. McKenney
2019-07-08 14:15 ` Joel Fernandes
2019-07-09 6:05 ` Byungchul Park
2019-07-09 12:43 ` Paul E. McKenney
2019-07-09 5:58 ` Byungchul Park
2019-07-09 6:45 ` Byungchul Park
2019-07-09 12:41 ` Paul E. McKenney
2019-07-10 1:20 ` Byungchul Park
2019-07-11 12:30 ` Paul E. McKenney
2019-07-11 13:08 ` Joel Fernandes
2019-07-11 15:02 ` Paul E. McKenney
2019-07-11 16:48 ` Joel Fernandes
2019-07-11 19:58 ` Joel Fernandes
2019-07-12 6:32 ` Byungchul Park
2019-07-12 12:51 ` Joel Fernandes
2019-07-12 13:02 ` Paul E. McKenney
2019-07-12 13:43 ` Joel Fernandes
2019-07-12 14:53 ` Paul E. McKenney
2019-07-13 8:47 ` Byungchul Park
2019-07-13 14:20 ` Joel Fernandes
2019-07-13 15:13 ` Paul E. McKenney
2019-07-13 15:42 ` Joel Fernandes
2019-07-13 17:41 ` Paul E. McKenney
2019-07-14 13:39 ` Byungchul Park
2019-07-14 13:56 ` Paul E. McKenney
2019-07-15 17:39 ` Joel Fernandes
2019-07-15 20:09 ` Paul E. McKenney
2019-07-18 16:14 ` Joel Fernandes
2019-07-18 16:15 ` Joel Fernandes
2019-07-18 21:34 ` Paul E. McKenney
2019-07-19 0:48 ` Joel Fernandes
2019-07-19 0:54 ` Byungchul Park
2019-07-19 0:39 ` Byungchul Park
2019-07-19 0:52 ` Joel Fernandes
2019-07-19 1:10 ` Byungchul Park
2019-07-19 7:43 ` Paul E. McKenney
2019-07-19 9:57 ` Byungchul Park
2019-07-19 19:57 ` Paul E. McKenney
2019-07-19 20:33 ` Joel Fernandes
2019-07-23 11:05 ` Byungchul Park
2019-07-23 13:47 ` Paul E. McKenney
2019-07-23 16:54 ` Paul E. McKenney
2019-07-24 7:58 ` Byungchul Park [this message]
2019-07-24 7:59 ` Byungchul Park
2019-07-12 13:01 ` Paul E. McKenney
2019-07-12 13:40 ` Joel Fernandes
2019-07-12 6:00 ` Byungchul Park
2019-07-12 5:52 ` Byungchul Park
2019-07-12 5:48 ` Byungchul Park
2019-07-13 9:08 ` Byungchul Park
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20190724075841.GA14712@X58A-UD3R \
--to=byungchul.park@lge.com \
--cc=joel@joelfernandes.org \
--cc=kernel-team@lge.com \
--cc=linux-kernel@vger.kernel.org \
--cc=max.byungchul.park@gmail.com \
--cc=paulmck@linux.ibm.com \
--cc=rcu@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).