From: Eric Dumazet <eric.dumazet@gmail.com>
To: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: David Ahern <dsahern@gmail.com>,
Linus Torvalds <torvalds@linux-foundation.org>,
Ingo Molnar <mingo@elte.hu>, Thomas Gleixner <tglx@linutronix.de>,
Martin Schwidefsky <schwidefsky@de.ibm.com>,
linux-kernel <linux-kernel@vger.kernel.org>,
Frederic Weisbecker <fweisbec@gmail.com>,
Suresh Siddha <suresh.b.siddha@intel.com>
Subject: Re: [BUG] kernel freezes with latest tree
Date: Wed, 11 Jan 2012 11:28:49 +0100 [thread overview]
Message-ID: <1326277729.2767.5.camel@edumazet-HP-Compaq-6005-Pro-SFF-PC> (raw)
In-Reply-To: <1326272685.2442.120.camel@twins>
Le mercredi 11 janvier 2012 à 10:04 +0100, Peter Zijlstra a écrit :
> On Tue, 2012-01-10 at 23:35 -0700, David Ahern wrote:
> > On 01/10/2012 04:44 PM, Linus Torvalds wrote:
> > > Anybody? Any ideas? Clearly there can be a merge problem that doesn't
> > > actually show as a real data conflict, just some semantic conflict,
> > > but I don't see what such issues would be brouht in by the scheduler
> > > merge anyway.
> >
> > This is really easy to reproduce in a KVM hosted VM.
> >
> > Using the gdb stub one cpu is spinning here:
> >
> > (gdb) bt
> > #0 try_to_wake_up (p=0xf529b200, state=<optimized out>, wake_flags=1)
> > at /mnt/sw/kernel-2.6.git/kernel/sched/core.c:1575
> > #1 0xc0470ab0 in default_wake_function (curr=<optimized out>,
> > mode=<optimized out>,
> > wake_flags=<optimized out>, key=0xc3) at
> > /mnt/sw/kernel-2.6.git/kernel/sched/core.c:3364
> >
> > So basically:
> > while (p->on_cpu) {
> > ...
> > cpu_relax();
> > }
> >
> >
> > And the other vcpu is here:
> >
> > #0 tg_load_down (tg=0xf55b9c00, data=<optimized out>) at
> > /mnt/sw/kernel-2.6.git/kernel/sched/fair.c:3351
> > #1 0xc0470049 in walk_tg_tree_from (from=0xc0ba5400, down=0xc04753c0
> > <tg_load_down>, up=0xc046a3b0 <tg_nop>,
> > data=0x0) at /mnt/sw/kernel-2.6.git/kernel/sched/core.c:664
> > #2 0xc04793f7 in walk_tg_tree (data=<optimized out>, up=<optimized
> > out>, down=0xc04753c0 <tg_load_down>)
> > at /mnt/sw/kernel-2.6.git/kernel/sched/sched.h:175
> > #3 update_h_load (cpu=<optimized out>) at
> > /mnt/sw/kernel-2.6.git/kernel/sched/fair.c:3361
> > #4 load_balance_fair (lb_flags=<synthetic pointer>,
> > idle=CPU_NEWLY_IDLE, sd=0xf5c30800, max_load_move=278,
> > busiest=0xf6607d00, this_cpu=1, this_rq=0xf6707d00) at
> > /mnt/sw/kernel-2.6.git/kernel/sched/fair.c:3374
> > #5 move_tasks (lb_flags=<synthetic pointer>, idle=CPU_NEWLY_IDLE,
> > sd=0xf5c30800, max_load_move=278,
> > busiest=<optimized out>, this_cpu=1, this_rq=0xf6707d00)
> > at /mnt/sw/kernel-2.6.git/kernel/sched/fair.c:3444
> > #6 load_balance (this_cpu=1, this_rq=0xf6707d00, sd=0xf5c30800,
> > idle=CPU_NEWLY_IDLE, balance=0xf5217cb4)
> > at /mnt/sw/kernel-2.6.git/kernel/sched/fair.c:4496
> > #7 0xc0479be2 in idle_balance (this_cpu=1, this_rq=0xf6707d00)
> > at /mnt/sw/kernel-2.6.git/kernel/sched/fair.c:4640
> >
> >
> > Based on the file in question (sched/fair.c) I took a stab at guessing
> > the commit: without a195f004 I was not able to lock it up. With the
> > patch the VM spins after a few hackbench iterations.
> >
> > I don't have time for a proper bisect tonight. I can do that in the a.m.
> > if I am not totally off base here. Peter: any chance this commit could
> > explain the spinning cpus / system freeze?
>
> It could, I certainly ran into similar issues while developing that
> patch. I ran into all sorts of weird stuff but had hoped I'd cured all
> of it.
>
> If Eric can confirm this is indeed what is causing his pain, I'm fine
> with reverting it and having another go at it later. How easy is it to
> reproduce using your KVM thing?
>
> I think simply replacing the one |= LBF_NEED_BREAK with a LBF_ABORT or
> removing that condition all-together should make the hang go away.
>
> I'll try and figure out how it ends up in the infinite retry loop after
> my brain wakes up a bit more. Maybe adding a few more NEED_BREAK bits
> and making it a counter and overflowing it into ABORT might be good.
>
> /me off to get breakfast and morning-juice..
>
I let my machine run 3.2 kernel for 37 minutes with no hang, so at least
we know there is no hardware glitch.
$ uptime
11:26:45 up 37 min, 2 users, load average: 201.80, 206.06, 170.68
I'll try now latest tree with a195f004 reverted.
Thanks
next prev parent reply other threads:[~2012-01-11 10:28 UTC|newest]
Thread overview: 28+ messages / expand[flat|nested] mbox.gz Atom feed top
2012-01-10 4:57 [BUG] kernel freezes with latest tree Eric Dumazet
2012-01-10 5:03 ` Eric Dumazet
2012-01-10 8:16 ` Eric Dumazet
2012-01-10 16:13 ` Eric Dumazet
2012-01-10 16:28 ` Linus Torvalds
2012-01-10 16:37 ` Eric Dumazet
2012-01-10 16:49 ` Linus Torvalds
2012-01-10 16:53 ` Eric Dumazet
2012-01-10 17:00 ` Linus Torvalds
2012-01-10 19:32 ` Ingo Molnar
2012-01-10 22:23 ` Eric Dumazet
2012-01-10 23:44 ` Linus Torvalds
2012-01-11 6:35 ` David Ahern
2012-01-11 9:04 ` Peter Zijlstra
2012-01-11 10:28 ` Eric Dumazet [this message]
2012-01-11 11:29 ` Eric Dumazet
2012-01-11 12:25 ` Peter Zijlstra
2012-01-11 13:24 ` Eric Dumazet
2012-01-11 15:56 ` Ingo Molnar
2012-01-11 16:05 ` Peter Zijlstra
2012-01-11 16:14 ` Ingo Molnar
2012-01-11 16:31 ` Linus Torvalds
2012-01-11 16:58 ` Peter Zijlstra
2012-01-12 6:17 ` [tip:sched/urgent] sched: Fix lockup by limiting load-balance retries on lock-break tip-bot for Peter Zijlstra
2012-01-11 8:22 ` [BUG] kernel freezes with latest tree Eric Dumazet
2012-01-11 14:20 ` Frederic Weisbecker
2012-01-10 16:16 ` Linus Torvalds
2012-01-10 16:33 ` Eric Dumazet
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1326277729.2767.5.camel@edumazet-HP-Compaq-6005-Pro-SFF-PC \
--to=eric.dumazet@gmail.com \
--cc=a.p.zijlstra@chello.nl \
--cc=dsahern@gmail.com \
--cc=fweisbec@gmail.com \
--cc=linux-kernel@vger.kernel.org \
--cc=mingo@elte.hu \
--cc=schwidefsky@de.ibm.com \
--cc=suresh.b.siddha@intel.com \
--cc=tglx@linutronix.de \
--cc=torvalds@linux-foundation.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).