All of lore.kernel.org
 help / color / mirror / Atom feed
* Re: [PATCH tip/core/rcu 1/2] rcu: Parallelize and economize NOCB kthread wakeups
@ 2014-08-23  7:43 Pranith Kumar
  2014-08-23 16:51 ` Paul E. McKenney
  2014-08-27  4:43 ` Amit Shah
  0 siblings, 2 replies; 48+ messages in thread
From: Pranith Kumar @ 2014-08-23  7:43 UTC (permalink / raw)
  To: Paul McKenney
  Cc: Amit Shah, LKML, Rik van Riel, Ingo Molnar, Lai Jiangshan,
	Dipankar Sarma, Andrew Morton, Mathieu Desnoyers, Josh Triplett,
	Thomas Gleixner, Peter Zijlstra, Steven Rostedt, David Howells,
	Eric Dumazet, dvhart, Frédéric Weisbecker,
	Oleg Nesterov, Silas Boyd-Wickizer

On Fri, Aug 22, 2014 at 5:53 PM, Paul E. McKenney
<paulmck@linux.vnet.ibm.com> wrote:
>
> Hmmm...  Please try replacing the synchronize_rcu() in
> __sysrq_swap_key_ops() with (say) schedule_timeout_interruptible(HZ / 10).
> I bet that gets rid of the hang.  (And also introduces a low-probability
> bug, but should be OK for testing.)
>
> The other thing to try is to revert your patch that turned my event
> traces into printk()s, then put an ftrace_dump(DUMP_ALL); just after
> the synchronize_rcu() -- that might make it so that the ftrace data
> actually gets dumped out.
>

I was able to reproduce this error on my Ubuntu 14.04 machine. I think
I found the root cause of the problem after several kvm runs.

The problem is that earlier we were waiting on nocb_head and now we
are waiting on nocb_leader_wake.

So there are a lot of nocb callbacks which are enqueued before the
nocb thread is spawned. This sets up nocb_head to be non-null, because
of which the nocb kthread used to wake up immediately after sleeping.

Now that we have switched to nocb_leader_wake, this is not being set
when there are pending callbacks, unless the callbacks overflow the
qhimark. The pending callbacks were around 7000 when the boot hangs.

So setting the qhimark using the boot parameter rcutree.qhimark=5000
is one way to allow us to boot past the point by forcefully waking up
the nocb kthread. I am not sure this is fool-proof.

Another option to start the nocb kthreads with nocb_leader_wake set,
so that it can handle any pending callbacks. The following patch also
allows us to boot properly.

Phew! Let me know if this makes any sense :)

diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h
index 00dc411..4c397aa 100644
--- a/kernel/rcu/tree_plugin.h
+++ b/kernel/rcu/tree_plugin.h
@@ -2386,6 +2386,9 @@ static int rcu_nocb_kthread(void *arg)
        struct rcu_head **tail;
        struct rcu_data *rdp = arg;

+       if (rdp->nocb_leader == rdp)
+               rdp->nocb_leader_wake = true;
+
        /* Each pass through this loop invokes one batch of callbacks */
        for (;;) {
                /* Wait for callbacks. */


-- 
Pranith

^ permalink raw reply related	[flat|nested] 48+ messages in thread
* [PATCH tip/core/rcu 0/2] Callback-offloading changes for 3.17
@ 2014-07-07 22:48 Paul E. McKenney
  2014-07-11 13:35 ` [PATCH tip/core/rcu 1/2] rcu: Parallelize and economize NOCB kthread wakeups Paul E. McKenney
  0 siblings, 1 reply; 48+ messages in thread
From: Paul E. McKenney @ 2014-07-07 22:48 UTC (permalink / raw)
  To: linux-kernel
  Cc: mingo, laijs, dipankar, akpm, mathieu.desnoyers, josh, niv, tglx,
	peterz, rostedt, dhowells, edumazet, dvhart, fweisbec, oleg, sbw

Hello!

This series provides a couple of callback-offloading changes:

1.	Parallelize and economize NOCB kthread wakeups to better handle
	cases where callback offloading is used on workloads that have
	many context switches and thus many grace periods.

2.	Don't offload callbacks for a given CPU unless it has been
	explicitly requested for that CPU, either directly at build
	time or boot time, or indirectly at boot time due to that
	CPU being a nohz_full CPU.

							Thanx, Paul

------------------------------------------------------------------------

 b/Documentation/kernel-parameters.txt |    7 
 b/init/Kconfig                        |    4 
 b/kernel/rcu/tree.h                   |   28 +++
 b/kernel/rcu/tree_plugin.h            |  255 ++++++++++++++++++++++++++++------
 4 files changed, 249 insertions(+), 45 deletions(-)


^ permalink raw reply	[flat|nested] 48+ messages in thread

end of thread, other threads:[~2014-08-27 17:08 UTC | newest]

Thread overview: 48+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2014-08-23  7:43 [PATCH tip/core/rcu 1/2] rcu: Parallelize and economize NOCB kthread wakeups Pranith Kumar
2014-08-23 16:51 ` Paul E. McKenney
2014-08-24  0:26   ` Pranith Kumar
2014-08-24  3:23     ` Paul E. McKenney
2014-08-24  3:39       ` Pranith Kumar
2014-08-24 14:36         ` Paul E. McKenney
2014-08-27  4:43 ` Amit Shah
2014-08-27 16:21   ` Paul E. McKenney
2014-08-27 16:43     ` Pranith Kumar
2014-08-27 17:08       ` Paul E. McKenney
  -- strict thread matches above, loose matches on Subject: below --
2014-07-07 22:48 [PATCH tip/core/rcu 0/2] Callback-offloading changes for 3.17 Paul E. McKenney
2014-07-11 13:35 ` [PATCH tip/core/rcu 1/2] rcu: Parallelize and economize NOCB kthread wakeups Paul E. McKenney
2014-08-08  8:40   ` Amit Shah
2014-08-08 16:25     ` Paul E. McKenney
2014-08-08 17:37       ` Amit Shah
2014-08-08 18:18         ` Paul E. McKenney
2014-08-08 18:34           ` Amit Shah
2014-08-08 21:43             ` Paul E. McKenney
2014-08-08 21:46               ` Paul E. McKenney
2014-08-11  7:13                 ` Amit Shah
2014-08-11 16:28                   ` Paul E. McKenney
2014-08-11 19:41                     ` Amit Shah
2014-08-11 20:11                       ` Paul E. McKenney
2014-08-11 20:18                         ` Amit Shah
2014-08-11 20:34                           ` Paul E. McKenney
2014-08-12  3:45                             ` Paul E. McKenney
2014-08-12  5:33                               ` Amit Shah
2014-08-12 16:06                                 ` Paul E. McKenney
2014-08-12 21:39                                   ` Paul E. McKenney
2014-08-12 21:41                                     ` Paul E. McKenney
2014-08-12 21:44                                       ` Paul E. McKenney
2014-08-13  5:44                                       ` Amit Shah
2014-08-13 13:00                                         ` Paul E. McKenney
2014-08-13 14:18                                           ` Paul E. McKenney
2014-08-15  5:24                                           ` Amit Shah
2014-08-15 15:04                                             ` Paul E. McKenney
2014-08-18 17:53                                               ` Amit Shah
2014-08-19  4:01                                                 ` Paul E. McKenney
2014-08-22 12:24                                                   ` Amit Shah
2014-08-22 12:36                                                     ` Amit Shah
2014-08-22 12:56                                                       ` Amit Shah
2014-08-22 14:48                                                         ` Paul E. McKenney
2014-08-22 17:14                                                           ` Amit Shah
2014-08-22 17:37                                                             ` Amit Shah
2014-08-22 21:53                                                             ` Paul E. McKenney
2014-08-22 21:57                                                               ` Paul E. McKenney
2014-08-22 14:43                                                     ` Paul E. McKenney
2014-08-12  5:27                             ` Amit Shah
2014-08-12 16:08                               ` Paul E. McKenney

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.