From: Ingo Molnar <mingo@elte.hu>
To: Kevin Shanahan <kmshanah@ucwb.org.au>
Cc: Avi Kivity <avi@redhat.com>, "Rafael J. Wysocki" <rjw@sisk.pl>,
Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
Kernel Testers List <kernel-testers@vger.kernel.org>,
Mike Galbraith <efault@gmx.de>,
Peter Zijlstra <a.p.zijlstra@chello.nl>
Subject: Re: [Bug #12465] KVM guests stalling on 2.6.28 (bisected)
Date: Tue, 20 Jan 2009 15:25:15 +0100 [thread overview]
Message-ID: <20090120142515.GC10224@elte.hu> (raw)
In-Reply-To: <1232461380.4895.33.camel@kulgan.wumi.org.au>
* Kevin Shanahan <kmshanah@ucwb.org.au> wrote:
> On Tue, 2009-01-20 at 13:56 +0100, Ingo Molnar wrote:
> > * Kevin Shanahan <kmshanah@ucwb.org.au> wrote:
> > > > This suggests some sort of KVM-specific problem. Scheduler latencies
> > > > in the seconds that occur under normal load situations are noticed and
> > > > reported quickly - and there are no such open regressions currently.
> > >
> > > It at least suggests a problem with interaction between the scheduler
> > > and kvm, otherwise reverting that scheduler patch wouldn't have made the
> > > regression go away.
> >
> > the scheduler affects almost everything, so almost by definition a
> > scheduler change can tickle a race or other timing bug in just about any
> > code - and reverting that change in the scheduler can make the bug go
> > away. But yes, it could also be a genuine scheduler bug - that is always a
> > possibility.
>
> Okay, I understand.
>
> > Could you please run a cfs-debug-info.sh session on a CONFIG_SCHED_DEBUG=y
> > and CONFIG_SCHEDSTATS=y kernel, while you are experiencing those
> > latencies:
> >
> > http://people.redhat.com/mingo/cfs-scheduler/tools/cfs-debug-info.sh
> >
> > and post that (relatively large) somewhere, or send it as a reply after
> > bzip2 -9 compressing it? It will include a lot of information about the
> > delays your tasks are experiencing.
>
> Running it while the problem is occuring will be tricky, as it only
> lasts for a few seconds at a time. Is it going to be useful at all to
> just see those statistics if the system is running normally?
>
> I might need to modify the script a little. Am I right that everything
> above "gathering statistics..." is pretty much static information?
Correct.
> I could run top, vmstat and cat /proc/sched_debug in a loop until the
> problem occurs and then trim it. Something like:
>
> while true; do
> date >> $FILE
> echo "-- top: --" >> $FILE
> top -H -c -b -d 1 -n 0.5 >> $FILE 2>/dev/null
> echo "-- vmstat: --" >> $FILE
> vmstat >> $FILE 2>/dev/null
> echo "-- sched_debug #$i: --" >> $FILE
> cat /proc/sched_debug >> $FILE 2>/dev/null
> done
>
> That should take a snapshot every half second or so.
Yeah, that would be lovely. You dont even have to trim it much - just give
us a timestamp to look at for the delay incident. You might also want to
start the kvm session while the script is already running - that way we'll
get fresh statistics and see the whole thing.
Ingo
next prev parent reply other threads:[~2009-01-20 14:25 UTC|newest]
Thread overview: 133+ messages / expand[flat|nested] mbox.gz Atom feed top
2009-01-19 21:41 2.6.29-rc2-git1: Reported regressions 2.6.27 -> 2.6.28 Rafael J. Wysocki
2009-01-19 21:41 ` [Bug #11849] default IRQ affinity change in v2.6.27 (breaking several SMP PPC based systems) Rafael J. Wysocki
2009-01-19 21:45 ` [Bug #12061] snd_hda_intel: power_save: sound cracks on powerdown Rafael J. Wysocki
2009-01-19 21:45 ` [Bug #12208] uml is very slow on 2.6.28 host Rafael J. Wysocki
2009-01-26 11:35 ` Miklos Szeredi
2009-01-19 21:45 ` [Bug #12160] networking oops after resume from s2ram (2.6.28-rc6) Rafael J. Wysocki
2009-01-19 21:45 ` [Bug #12159] 2.6.28-rc6-git1 -- No sound produced from Intel HDA ALSA driver Rafael J. Wysocki
2009-01-19 21:45 ` [Bug #12263] Sata soft reset filling log Rafael J. Wysocki
2009-01-19 21:45 ` [Bug #12260] Regression due to commit 2b80848e3818fb1c (p54usb: support LM87 firmwares) Rafael J. Wysocki
2009-01-20 22:11 ` [PATCH -stable] p54usb: fix traffic stalls / packet drop Christian Lamparter
2009-01-20 22:36 ` Rafael J. Wysocki
2009-01-20 22:39 ` Greg KH
2009-01-20 23:56 ` John W. Linville
2009-01-21 14:03 ` Christian Lamparter
2009-01-19 21:45 ` [Bug #12224] journal activity on inactive partition causes inactive harddrive spinup Rafael J. Wysocki
2009-01-20 13:03 ` Theodore Tso
2009-01-19 21:45 ` [Bug #12209] oldish top core dumps (in its meminfo() function) Rafael J. Wysocki
2009-01-19 21:45 ` [Bug #12265] FPU emulation broken in 2.6.28-rc8 ? Rafael J. Wysocki
2009-01-19 21:45 ` [Bug #12264] i915: switching from kwin in opengl mode to a VT then back to x11, x11 freezes Rafael J. Wysocki
2009-01-20 18:13 ` Caleb Cushing
2009-01-19 21:45 ` [Bug #12337] ~100 extra wakeups reported by powertop Rafael J. Wysocki
2009-01-20 9:38 ` Alberto Gonzalez
2009-01-19 21:45 ` [Bug #12391] Processor does not go below C2 state until usb.autosuspend is enabled Rafael J. Wysocki
2009-01-27 10:27 ` Pavel Machek
2009-01-19 21:45 ` [Bug #12395] 2.6.28-rc9: oprofile regression Rafael J. Wysocki
2009-01-19 21:45 ` [Bug #12393] debugging in dosemu causes lots of 'scheduling while atomic' Rafael J. Wysocki
2009-01-20 9:58 ` Michal Suchanek
2009-01-19 21:45 ` [Bug #12396] hwinfo problem since 2.6.28 Rafael J. Wysocki
2009-01-26 14:00 ` Beschorner Daniel
2009-01-19 21:45 ` [Bug #12404] Oops in 2.6.28-rc9 and -rc8 -- mtrr issues / e1000e Rafael J. Wysocki
2009-01-19 21:45 ` [Bug #12403] TTY problem on linux-2.6.28-rc7 Rafael J. Wysocki
2009-01-19 21:45 ` [Bug #12401] 2.6.28 regression: xbacklight broken on ThinkPad X61s Rafael J. Wysocki
2009-01-20 7:30 ` Tino Keitel
2009-01-19 21:45 ` [Bug #12406] 2.6.28 thinks that my PS/2 mouse is a touchpad Rafael J. Wysocki
2009-01-20 1:45 ` Arjan Opmeer
2009-01-20 9:19 ` Dmitry Torokhov
2009-01-22 6:29 ` Alexander E. Patrakov
2009-01-19 21:45 ` [Bug #12407] Kernel 2.6.28 regression: Hang after hibernate Rafael J. Wysocki
2009-01-19 21:45 ` [Bug #12408] Funny problem with 2.6.28: Kernel stalls Rafael J. Wysocki
2009-01-19 21:45 ` [Bug #12405] oops in __bounce_end_io_read under kvm Rafael J. Wysocki
2009-01-19 21:45 ` [Bug #12409] NULL pointer dereference at get_stats() Rafael J. Wysocki
2009-01-21 16:18 ` Frederik Deweerdt
2009-01-24 0:39 ` Tetsuo Handa
2009-02-07 2:34 ` Tetsuo Handa
2009-02-09 11:19 ` Tetsuo Handa
2009-02-11 22:54 ` Alok Kataria
2009-02-11 23:02 ` Alok Kataria
2009-02-13 11:54 ` Tetsuo Handa
2009-01-19 21:45 ` [Bug #12465] KVM guests stalling on 2.6.28 (bisected) Rafael J. Wysocki
2009-01-20 0:12 ` Kevin Shanahan
2009-01-20 11:35 ` Ingo Molnar
2009-01-20 12:37 ` Avi Kivity
2009-01-20 12:42 ` Kevin Shanahan
2009-01-20 12:56 ` Ingo Molnar
2009-01-20 13:07 ` Ingo Molnar
2009-01-20 14:59 ` Steven Rostedt
2009-01-20 15:04 ` Ingo Molnar
2009-01-20 17:53 ` Steven Rostedt
2009-01-20 18:39 ` Ingo Molnar
2009-01-20 17:47 ` Avi Kivity
2009-01-21 14:25 ` Kevin Shanahan
2009-01-21 14:34 ` Avi Kivity
2009-01-21 14:51 ` Kevin Shanahan
2009-01-21 14:59 ` Avi Kivity
2009-01-21 15:13 ` Steven Rostedt
2009-01-22 1:48 ` Steven Rostedt
2009-01-21 15:10 ` Steven Rostedt
2009-01-21 15:18 ` Ingo Molnar
2009-01-22 19:57 ` Kevin Shanahan
2009-01-22 20:31 ` Ingo Molnar
2009-01-26 9:55 ` Kevin Shanahan
2009-01-26 11:35 ` Peter Zijlstra
2009-01-26 14:38 ` [RFC][PATCH] ftrace: function graph trace context switches Peter Zijlstra
2009-01-26 15:39 ` Frédéric Weisbecker
2009-01-26 15:41 ` Steven Rostedt
2009-03-16 17:57 ` Frederic Weisbecker
2009-01-26 15:00 ` [Bug #12465] KVM guests stalling on 2.6.28 (bisected) Ingo Molnar
2009-01-20 14:23 ` Kevin Shanahan
2009-01-20 14:25 ` Ingo Molnar [this message]
2009-01-20 15:51 ` Kevin Shanahan
2009-01-20 16:06 ` Ingo Molnar
2009-01-20 16:19 ` Peter Zijlstra
2009-01-20 14:46 ` Frédéric Weisbecker
2009-01-20 13:04 ` Avi Kivity
2009-01-20 17:54 ` Kevin Shanahan
2009-01-20 18:42 ` Ingo Molnar
2009-01-19 21:45 ` [Bug #12411] 2.6.28: BUG in r8169 Rafael J. Wysocki
2009-01-19 21:45 ` [Bug #12426] TMDC Joystick no longer works in kernel 2.6.28 Rafael J. Wysocki
2009-01-21 0:48 ` Andrew S. Johnson
2009-01-22 13:34 ` Jiri Kosina
2009-01-23 2:06 ` Andrew S. Johnson
2009-01-26 11:49 ` Jiri Kosina
2009-01-19 21:45 ` [Bug #12483] Reference to inexistent struct dmi_device_id breaks the build Rafael J. Wysocki
2009-01-20 8:15 ` Jean Delvare
2009-01-19 21:45 ` [Bug #12500] r8169: NETDEV WATCHDOG: eth0 (r8169): transmit timed out Rafael J. Wysocki
2009-01-22 16:43 ` 2.6.29-rc2-git1: Reported regressions 2.6.27 -> 2.6.28 Jörg-Volker Peetz
2009-01-24 13:25 ` Rolf Eike Beer
2009-02-04 10:55 2.6.29-rc3-git6: " Rafael J. Wysocki
2009-02-04 10:58 ` [Bug #12465] KVM guests stalling on 2.6.28 (bisected) Rafael J. Wysocki
2009-02-05 19:35 ` Kevin Shanahan
2009-02-05 22:37 ` Rafael J. Wysocki
2009-02-14 20:48 2.6.29-rc5: Reported regressions 2.6.27 -> 2.6.28 Rafael J. Wysocki
2009-02-14 20:50 ` [Bug #12465] KVM guests stalling on 2.6.28 (bisected) Rafael J. Wysocki
2009-02-23 22:00 2.6.29-rc6: Reported regressions 2.6.27 -> 2.6.28 Rafael J. Wysocki
2009-02-23 22:03 ` [Bug #12465] KVM guests stalling on 2.6.28 (bisected) Rafael J. Wysocki
2009-02-24 0:59 ` Kevin Shanahan
2009-02-24 1:37 ` Rafael J. Wysocki
2009-02-24 12:09 ` Avi Kivity
2009-02-24 22:11 ` Kevin Shanahan
2009-03-03 19:34 2.6.29-rc6-git7: Reported regressions 2.6.27 -> 2.6.28 Rafael J. Wysocki
2009-03-03 19:41 ` [Bug #12465] KVM guests stalling on 2.6.28 (bisected) Rafael J. Wysocki
2009-03-04 3:08 ` Kevin Shanahan
2009-03-08 10:04 ` Avi Kivity
2009-03-14 19:11 2.6.29-rc8: Reported regressions 2.6.27 -> 2.6.28 Rafael J. Wysocki
2009-03-14 19:20 ` [Bug #12465] KVM guests stalling on 2.6.28 (bisected) Rafael J. Wysocki
2009-03-15 9:03 ` Kevin Shanahan
2009-03-15 9:18 ` Avi Kivity
2009-03-15 9:48 ` Ingo Molnar
2009-03-15 9:56 ` Avi Kivity
2009-03-15 10:03 ` Ingo Molnar
2009-03-15 10:13 ` Avi Kivity
2009-03-16 9:49 ` Avi Kivity
2009-03-16 12:46 ` Kevin Shanahan
2009-03-16 20:07 ` Frederic Weisbecker
2009-03-16 22:55 ` Kevin Shanahan
2009-03-18 0:20 ` Frederic Weisbecker
2009-03-18 1:16 ` Kevin Shanahan
2009-03-18 2:24 ` Frederic Weisbecker
2009-03-18 21:24 ` Kevin Shanahan
2009-03-21 5:00 ` Kevin Shanahan
2009-03-21 14:08 ` Frederic Weisbecker
2009-03-24 11:44 ` Frederic Weisbecker
2009-03-24 11:47 ` Frederic Weisbecker
2009-03-25 23:40 ` Kevin Shanahan
2009-03-25 23:48 ` Frederic Weisbecker
2009-03-26 20:22 ` Kevin Shanahan
2009-03-21 17:01 2.6.29-rc8-git5: Reported regressions 2.6.27 -> 2.6.28 Rafael J. Wysocki
2009-03-21 17:07 ` [Bug #12465] KVM guests stalling on 2.6.28 (bisected) Rafael J. Wysocki
2009-03-21 19:50 ` Ingo Molnar
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20090120142515.GC10224@elte.hu \
--to=mingo@elte.hu \
--cc=a.p.zijlstra@chello.nl \
--cc=avi@redhat.com \
--cc=efault@gmx.de \
--cc=kernel-testers@vger.kernel.org \
--cc=kmshanah@ucwb.org.au \
--cc=linux-kernel@vger.kernel.org \
--cc=rjw@sisk.pl \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).