From: Ingo Molnar <mingo@elte.hu>
To: Peter Zijlstra <peterz@infradead.org>
Cc: David Ahern <dsahern@gmail.com>,
Linus Torvalds <torvalds@linux-foundation.org>,
Eric Dumazet <eric.dumazet@gmail.com>,
Thomas Gleixner <tglx@linutronix.de>,
Martin Schwidefsky <schwidefsky@de.ibm.com>,
linux-kernel <linux-kernel@vger.kernel.org>,
Frederic Weisbecker <fweisbec@gmail.com>,
Suresh Siddha <suresh.b.siddha@intel.com>
Subject: Re: [BUG] kernel freezes with latest tree
Date: Wed, 11 Jan 2012 17:14:32 +0100 [thread overview]
Message-ID: <20120111161431.GA1233@elte.hu> (raw)
In-Reply-To: <1326297936.2442.157.camel@twins>
* Peter Zijlstra <peterz@infradead.org> wrote:
> On Wed, 2012-01-11 at 16:56 +0100, Ingo Molnar wrote:
>
> > Well, what happens if every CPU runs load_balance() and we keep
> > triggering:
> >
> > if (loops++ > sysctl_sched_nr_migrate) {
> > *lb_flags |= LBF_NEED_BREAK;
> > break;
> > }
> >
> > in this case load_balance() will do the retry:
> >
> > if (lb_flags & LBF_NEED_BREAK) {
> > lb_flags &= ~LBF_NEED_BREAK;
> > goto redo;
> > }
> >
> > but the retry starts the loop again:
> >
> > list_for_each_entry_safe(p, n, &busiest_cfs_rq->tasks, se.group_node) {
> >
> > so nobody is able to make progress: livelock/lockup.
>
> Ah, right! Silly me. One possibility is to rotate that list, except that
> won't work for the cgroup case where we have another iteration.
>
> OK, here's an updated patch..
>
> ---
> Subject: sched: Limit load-balance retries on lock-break
> From: Peter Zijlstra <a.p.zijlstra@chello.nl>
> Date: Wed Jan 11 13:11:12 CET 2012
>
> Eric and David reported dead machines and traced it to commit a195f004 ("sched:
> Fix load-balance lock-breaking"), it turns out there's still a
> scenario where we can end up re-trying forever.
>
> Since there is no strict forward progress guarantee in the
> load-balance iteration we can get stuck re-retrying the same task-set
> over and over.
>
> Creating a forward progress guarantee with the existing structure is
> somewhat non-trivial, for now simply terminate the retry loop after a
> few tries.
>
> Reported-by: Eric Dumazet <eric.dumazet@gmail.com>
> Reported-by: David Ahern <dsahern@gmail.com>
> Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
> [eric: logic cleanup]
> Tested-by: Eric Dumazet <eric.dumazet@gmail.com>
> Link: http://lkml.kernel.org/n/tip-ya9m8grb9wfc26uqnviq2wjq@git.kernel.org
> ---
> kernel/sched/fair.c | 10 +++++++---
> 1 file changed, 7 insertions(+), 3 deletions(-)
Thanks Peter, i'll get this fix to Linus ASAP.
Thanks,
Ingo
next prev parent reply other threads:[~2012-01-11 16:14 UTC|newest]
Thread overview: 28+ messages / expand[flat|nested] mbox.gz Atom feed top
2012-01-10 4:57 [BUG] kernel freezes with latest tree Eric Dumazet
2012-01-10 5:03 ` Eric Dumazet
2012-01-10 8:16 ` Eric Dumazet
2012-01-10 16:13 ` Eric Dumazet
2012-01-10 16:28 ` Linus Torvalds
2012-01-10 16:37 ` Eric Dumazet
2012-01-10 16:49 ` Linus Torvalds
2012-01-10 16:53 ` Eric Dumazet
2012-01-10 17:00 ` Linus Torvalds
2012-01-10 19:32 ` Ingo Molnar
2012-01-10 22:23 ` Eric Dumazet
2012-01-10 23:44 ` Linus Torvalds
2012-01-11 6:35 ` David Ahern
2012-01-11 9:04 ` Peter Zijlstra
2012-01-11 10:28 ` Eric Dumazet
2012-01-11 11:29 ` Eric Dumazet
2012-01-11 12:25 ` Peter Zijlstra
2012-01-11 13:24 ` Eric Dumazet
2012-01-11 15:56 ` Ingo Molnar
2012-01-11 16:05 ` Peter Zijlstra
2012-01-11 16:14 ` Ingo Molnar [this message]
2012-01-11 16:31 ` Linus Torvalds
2012-01-11 16:58 ` Peter Zijlstra
2012-01-12 6:17 ` [tip:sched/urgent] sched: Fix lockup by limiting load-balance retries on lock-break tip-bot for Peter Zijlstra
2012-01-11 8:22 ` [BUG] kernel freezes with latest tree Eric Dumazet
2012-01-11 14:20 ` Frederic Weisbecker
2012-01-10 16:16 ` Linus Torvalds
2012-01-10 16:33 ` Eric Dumazet
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20120111161431.GA1233@elte.hu \
--to=mingo@elte.hu \
--cc=dsahern@gmail.com \
--cc=eric.dumazet@gmail.com \
--cc=fweisbec@gmail.com \
--cc=linux-kernel@vger.kernel.org \
--cc=peterz@infradead.org \
--cc=schwidefsky@de.ibm.com \
--cc=suresh.b.siddha@intel.com \
--cc=tglx@linutronix.de \
--cc=torvalds@linux-foundation.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).